Profiles/manifests-related command line interface enhancements

Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2370 bytes --]

Hi!

While playing with multiple profiles and manifests and discussing with
a couple of people in the community, I collected a number of usability
issues with Guix when it comes to managing multiple profiles and dealing
with manifests.

Ideas for new features, more or less from most important to least
important:

- Activate a profile with =guix activate /path/to/profile=.
  Right now, the most appropriate way to activate a profile is

--8<---------------cut here---------------start------------->8---
GUIX_PROFILE="$profile" ; . "$profile"/etc/profile
--8<---------------cut here---------------end--------------->8---

  But this is flawed: it exposes implementation details plus it fails to
  set important variables like MANPATH or INFOPATH if man-db /
  info-reader or not installed in the said profile.  See /etc/profile
  for the full list of what =guix activate= should do.

- The inverse command, =guix deactivate /path/to/profile=.
  This can be useful to deactivate a profile that was activated during login.

- All commands that accept manifests should be able to compose manifests
  by passing multiple =--manifest FILE= options.
  
- Auto-update manifests when calling =guix package -m manifest.scm -i
  foo -r bar=.  This would make users more inclined to user manifests
  even for "quick and dirty" installs.  See next point.

  This means we need to edit the code, but that's doable since the last
  form of manifest.scm must evaluate to a manifest.  So we could proceed
  as follows:

  + If the last form is a spec, edit the list directly.
  + If not, then wrap the last form in the appropriate `manifest-add' /
  `manifest-remove' calls, e.g.

--8<---------------cut here---------------start------------->8---
(manifest-add "foo"
  (manifest-delete "bar"
    my-manifest))
--8<---------------cut here---------------end--------------->8---

- Use a default manifest by default with =guix install= and =guix
  remove=.  This way would basically get rid of "ad-hoc" profiles which
  has so many flaws (non-reproducible, non-trackable, etc.).

Last, one that's at the Scheme level:

- A Scheme function to create a manifest for the necessary inputs of a
  package, like =guix environment PACKAGE= does.  (Maybe it's already possible?)


Thoughts?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Mark H Weaver]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> While playing with multiple profiles and manifests and discussing with
> a couple of people in the community, I collected a number of usability
> issues with Guix when it comes to managing multiple profiles and dealing
> with manifests.
>
> Ideas for new features, more or less from most important to least
> important:
>
> - Activate a profile with =guix activate /path/to/profile=.
>   Right now, the most appropriate way to activate a profile is
>
> GUIX_PROFILE="$profile" ; . "$profile"/etc/profile

Due to the way environment variables work, there's no way to implement
your proposed interface, at least not without changing 'guix' from a
normal program into a Bash shell function.  Programs are unable to
change the environment variable settings of the parent shell that
launched them.

However, we could provide a Bash shell function (with a different name)
that provides conveniences like this.  Users of other shells would not
be able to use the Bash shell function, though.  Ideally, we'd provide
shell functions with a similar interface for several popular shells.
I'm not sure if it's worth it, but it could be done.

      Thanks,
        Mark

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

Hi Mark,

Good points!

Mark H Weaver <mhw@netris.org> writes:

> However, we could provide a Bash shell function (with a different name)
> that provides conveniences like this.  Users of other shells would not
> be able to use the Bash shell function, though.  Ideally, we'd provide
> shell functions with a similar interface for several popular shells.
> I'm not sure if it's worth it, but it could be done.

We already suffer from "Bash-specificism" since our "etc/profile" files
are written in Bash.  It will work for Zsh, but I'm not sure it works
for POSIX shell and it does not work for tcsh or Fish shell.

As you suggested `guix-activate` could be a shell function that's
defined in /etc/profile or anywhere appropriate.  We can "switch-case"
on the running shell and define `guix-activate` appropriately.  This way
we could support different shells.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

Forgot to mention one of the stated goals of these enhancements (in
particular profile activation): fix bug #37790 "MANPATH missing from
non-default profile".

https://issues.guix.gnu.org/issue/37790

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

Hi Pierre,

On Thu, 24 Oct 2019 11:32:55 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> As you suggested `guix-activate` could be a shell function that's
> defined in /etc/profile or anywhere appropriate.

Yes, but what's the advantage of that as opposed to just doing it like anyone
else does it: Just start a new shell.

For example

  ssh foo ls

starts ls on the remote host at our terminal

but

  ssh foo

starts an interactive shell on the remote host at our terminal.

Likewise, we could (and do) have 

  guix environment --ad-hoc package1 package2 ls

which starts ls in the guix-environment

but

  guix environment --ad-hoc package1 package2

starts interactive shell in the guix-environment.

Why should it be different with profiles?

I for one *like* it that the shell environment can't be
modified by random programs.

See Windows for how annoying it would be if it were possible for
random programs to change the interactive cmd's environment.

>- The inverse command, =guix deactivate /path/to/profile=.
> This can be useful to deactivate a profile that was activated during login.

That is not composable and sounds like a HUGE effort for what would otherwise
one "exit" command.

>- All commands that accept manifests should be able to compose manifests
>  by passing multiple =--manifest FILE= options.

Hmm that would be nice!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2318 bytes --]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi Pierre,
>
> On Thu, 24 Oct 2019 11:32:55 +0200
> Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
>> As you suggested `guix-activate` could be a shell function that's
>> defined in /etc/profile or anywhere appropriate.
>
> Yes, but what's the advantage of that as opposed to just doing it like anyone
> else does it: Just start a new shell.

If I understand correctly, you are saying we should use `guix
environment...` everywhere.

Currently, this does not support multiple manifests, so it does not
compose.  But that's an issue that could (should?) be fixed, as we are
discussing below.

There can be some benefits in editing the current environment for the
running application.
Say with EXWM, if I want to enable a profile for the running
instance of Emacs.

Moreover, I believe this does not work on login, e.g. from your
~/.bash_profile.

I'd like to streamline the way profiles are loaded, i.e. it should work
the same way on login and from a random shell.

> I for one *like* it that the shell environment can't be
> modified by random programs.

You mean that you prefer running sub-shells instead of changing the
current shell environment?

>>- The inverse command, =guix deactivate /path/to/profile=.
>> This can be useful to deactivate a profile that was activated during login.
>
> That is not composable and sounds like a HUGE effort for what would otherwise
> one "exit" command.

Why isn't it composable?

It does not seem too hard: if I'm not mistaken, it boils down to
removing all environment entries that match a given profile.

This could be brittle though and would need testing.

An alternative to this, once we have the ability to compose manifests
from command line, is to simply start a new shell with the desired
manifests.

With a good shell (e.g. Eshell) it's easy to complete against a list of
manifests and filter out the few that we don't want.

We could also leverage Emacs-Guix for that.

>>- All commands that accept manifests should be able to compose manifests
>>  by passing multiple =--manifest FILE= options.
>
> Hmm that would be nice!

This simple feature would fix many of the current limitations of Guix.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Mark H Weaver]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
>> On Thu, 24 Oct 2019 11:32:55 +0200
>> Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>>
>>>- The inverse command, =guix deactivate /path/to/profile=.
>>> This can be useful to deactivate a profile that was activated during login.
>>
>> That is not composable and sounds like a HUGE effort for what would otherwise
>> one "exit" command.
>
> Why isn't it composable?
>
> It does not seem too hard: if I'm not mistaken, it boils down to
> removing all environment entries that match a given profile.

It wouldn't be sufficient to remove them.  You'd have to restore the
previous settings.  For example, if we remove the settings for PATH,
MANPATH, etc, without restoring the previous settings, I doubt that you
would be pleased with the results.

It seems to me that the most natural approach to allow restoring the
previous environment settings is to launch a subshell when you activate
a profile.  That subshell can then simply be exited to restore the
previous settings.

Does that make sense?

      Regards,
        Mark

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

Mark H Weaver <mhw@netris.org> writes:

> It wouldn't be sufficient to remove them.  You'd have to restore the
> previous settings.  For example, if we remove the settings for PATH,
> MANPATH, etc, without restoring the previous settings, I doubt that you
> would be pleased with the results.

I don't understand.
Which previous settings?

As far as I understand, activating a profile never _removes_ any value
from environment variables.

> It seems to me that the most natural approach to allow restoring the
> previous environment settings is to launch a subshell when you activate
> a profile.  That subshell can then simply be exited to restore the
> previous settings.

I gave the example of EXWM, for which the user would like to change the
profile but a subshell won't do since it won't reflect on the parent
process, Emacs.

All that said, this is just a suggestion, I can live without this feature.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Mark H Weaver]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> It wouldn't be sufficient to remove them.  You'd have to restore the
>> previous settings.  For example, if we remove the settings for PATH,
>> MANPATH, etc, without restoring the previous settings, I doubt that you
>> would be pleased with the results.
>
> I don't understand.
> Which previous settings?

I'm talking about the previous values of the environment variables, as
they existed before activating the new profile.

> As far as I understand, activating a profile never _removes_ any value
> from environment variables.

It doesn't remove them, but it overwrites them, thus losing the
information of what the previous environment was.  Now, it's true that
for *most* of those environment variables, the previous value is a
suffix of the new value, but that's not always the case.  One notable
counter-example is GUIX_PROFILE itself, which is simply overwritten.
Others include GIT_EXEC_PATH and ASPELL_DICT_DIR.

>> It seems to me that the most natural approach to allow restoring the
>> previous environment settings is to launch a subshell when you activate
>> a profile.  That subshell can then simply be exited to restore the
>> previous settings.
>
> I gave the example of EXWM, for which the user would like to change the
> profile but a subshell won't do since it won't reflect on the parent
> process, Emacs.

There's no way to change the environment variable in Emacs without
running Emacs lisp code, e.g. by running M-x setenv.

In general, every process has a complete copy of its own environment
variables, which live in its own address space.  When a process launches
a child process, it gets to decide what the child's initial environment
will be.  Usually, but not always, programs pass a copy of their own
environment to child processes, and there are convenience functions that
make this common case easier.  From the kernel's point of view,
environment variables are passed to child processes exactly the same way
that command-line arguments are, although by convention they are treated
differently by user interfaces.

There's no mechanism for child processes to ask their parent processes
to modify their environment.  Each program needs to provide its own
mechanism for modifying its internal environment, and most don't even
provide such a mechanism.  For example, Ratpoison provides a command to
modify its own environment (C-t setenv, iirc), but I have no idea how to
modify gnome-session's environment.

Note that I'm not passing judgement on the merits of your proposals, I'm
merely letting you know that they cannot be implemented as envisioned,
at least not if I understand correctly.

     Regards,
       Mark

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2394 bytes --]

Hi Mark!

Mark H Weaver <mhw@netris.org> writes:

> It doesn't remove them, but it overwrites them, thus losing the
> information of what the previous environment was.  Now, it's true that
> for *most* of those environment variables, the previous value is a
> suffix of the new value, but that's not always the case.  One notable
> counter-example is GUIX_PROFILE itself, which is simply overwritten.
> Others include GIT_EXEC_PATH and ASPELL_DICT_DIR.

- ASPELL_DICT_DIR is broken: #29686.  We should fix it anyways.

- What's the use of GUIX_PROFILE once a profile is loaded?  I believe
  it's only there for the user to know which was the last loaded
  profile.  Which is a bit broken in my opinion: I believe GUIX_PROFILE
  should point to the default profile, but this is not possible when
  using multiple profiles.

  With my `guix-activate` proposal, GUIX_PROFILE would not have to be
  overridden.

  Although we could also fix this by modifying the etc/profile file so
  that they don't depend on GUIX_PROFILE being set to themselves (which
  is the actual bug here).

- You are right about GIT_EXEC_PATH, any clue why it has to be overridden?

> There's no way to change the environment variable in Emacs without
> running Emacs lisp code, e.g. by running M-x setenv.

Indeed, for Emacs `guix-activate` would run some Elisp code.  It would
essentially work for every program that can edit its own environment.

> Note that I'm not passing judgement on the merits of your proposals, I'm
> merely letting you know that they cannot be implemented as envisioned,
> at least not if I understand correctly.

With the shell/Emacs/etc. specialization we discussed, I believe it
would be :)


To clarify: what I'm proposing is a more universal, better encapsulated
way to activate profiles.

Currently we can do

--8<---------------cut here---------------start------------->8---
GUIX_PROFILE=/path/to/profile ; . $GUIX_PROFILE/etc/profile
--8<---------------cut here---------------end--------------->8---

which has some shortcomings:

- It only works for Bourne-style shells.  Activating profiles for
  different shells / programs require some work from the user; we should
  do this work upstream I believe.
- It exposes too many implementation details.
- It's too long ;)

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]


> --8<---------------cut here---------------start------------->8---
> GUIX_PROFILE=/path/to/profile ; . $GUIX_PROFILE/etc/profile
> --8<---------------cut here---------------end--------------->8---

If we want to activate a profile in a subshell, it would be enough to
add a `--profile` argument to `guix environment`, wouldn't it?

--8<---------------cut here---------------start------------->8---
guix environment --profile=/path/to/profile
--8<---------------cut here---------------end--------------->8---

It's simpler and hides the implementation details.

Better: support _multiple profiles_ so that we can compose environments
more easily.

Regarding the shell functions to activate a profile: I think with the
above it would not be really useful beyond the ~/.bash_profile settings.
So documenting the required snippet for ~/.bash_profile would be enough.

I can think of just one case where `guix environment' does not work:
Emacs' Eshell.  Here we could provide an Emacs function (maybe in
emacs-guix.el?) to load a profile.  Note that every eshell has a
let-bound `process-environment`, so activating a profile in an Eshell
only affects the current Eshell.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Hi Pierre!

Thanks for looking into this!  (And sorry for being late to the party. :-))

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> - Activate a profile with =guix activate /path/to/profile=.
>   Right now, the most appropriate way to activate a profile is
>
> GUIX_PROFILE="$profile" ; . "$profile"/etc/profile
>

Another way is:

  eval `guix package --search-paths=prefix`

or similar.

Another one is:

  guix environment …

>   But this is flawed: it exposes implementation details plus it fails to
>   set important variables like MANPATH or INFOPATH if man-db /
>   info-reader or not installed in the said profile.  See /etc/profile
>   for the full list of what =guix activate= should do.

The MANPATH issue is orthogonal so I don’t think we should bring it up
in this context.  See <https://issues.guix.gnu.org/issue/22138> for
context.

> - The inverse command, =guix deactivate /path/to/profile=.
>   This can be useful to deactivate a profile that was activated during login.

The activate/deactivate is similar to what CONDA does (from what I
understand; I’ve never used it) and similar to “module load” with
<http://modules.sourceforge.net/>.  It’s a nice pattern, though I think
it’s easy to lose track of what’s activated and what not, and it’s
tricky to define ‘deactivate’ correctly.

Regarding environment variables, “module” uses a trick where (roughly)
“module” itself is a shell function, such that it can adjust the
environment variables of the shell that invokes it.  We could do
something similar, though that would be limited to POSIX shells I guess.

> - All commands that accept manifests should be able to compose manifests
>   by passing multiple =--manifest FILE= options.

Do you have a use case?  :-)

Sounds like something quite easy to do but I’ve never heard anyone
request it before so I’m curious.

> - Auto-update manifests when calling =guix package -m manifest.scm -i
>   foo -r bar=.  This would make users more inclined to user manifests
>   even for "quick and dirty" installs.  See next point.
>
>   This means we need to edit the code, but that's doable since the last
>   form of manifest.scm must evaluate to a manifest.  So we could proceed
>   as follows:
>
>   + If the last form is a spec, edit the list directly.
>   + If not, then wrap the last form in the appropriate `manifest-add' /
>   `manifest-remove' calls, e.g.
>
> (manifest-add "foo"
>   (manifest-delete "bar"
>     my-manifest))

Could be fun, but… needs more thought.

> - Use a default manifest by default with =guix install= and =guix
>   remove=.  This way would basically get rid of "ad-hoc" profiles which
>   has so many flaws (non-reproducible, non-trackable, etc.).

Ambitious!

Note that “ad-hoc” profiles (imperatively-managed profiles) contain
provenance info in their ‘manifest’ file, so it’s not black and white.

> Last, one that's at the Scheme level:
>
> - A Scheme function to create a manifest for the necessary inputs of a
>   package, like =guix environment PACKAGE= does.  (Maybe it's already possible?)

Like ‘specifications->manifest’?

I think we should isolate different actionable items from this list and
work on them independently, to the extent possible.

Note that another option is to overhaul ‘guix environment’ like David
proposed a while back:

  https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html

I very much like those ideas, and I like the idea of making ‘guix
environment’ central (as opposed to offering more ways to manage
environments.)  It’s not completely the same goals as what you’re
looking for, but there’s some overlap.  Food for thought!

Thanks,
Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2802 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Another way is:
>
>   eval `guix package --search-paths=prefix`
>
> or similar.

Note that this suffers from the shell compatibility issue, e.g. it won't
work with Fish / Eshell.

> Another one is:
>
>   guix environment …

This is a bit different since it spawns a subshell.
Depends on the use case.

> The MANPATH issue is orthogonal so I don’t think we should bring it up
> in this context.  See <https://issues.guix.gnu.org/issue/22138> for
> context.

Oh, this is exactly it!  Thanks for the link, I had no clue this had
been under discussion... since 2015 O.o :)

>> - All commands that accept manifests should be able to compose manifests
>>   by passing multiple =--manifest FILE= options.
>
> Do you have a use case?  :-)
>
> Sounds like something quite easy to do but I’ve never heard anyone
> request it before so I’m curious.

I'm actually surprised you find it surprising! :)
I can think of Simon, maybe Konrad(?) and myself who mentioned it
before.

Examples:

- `guix pack` can only pack ` manifest at a time.  Impossible to combine
  2 manifests.

- Same for guix weather if you want to avoid calling guix weather
  multiple times in a row.

- `guix environment` can work around this limitation by calling the
  manifest from the generated subshells.  It's not super convenient, and
  it does not work for scripts.  For instance, we can't do this at the moment:

   guix environment -m manifest1 -m manifest2 -- my command line...

I'm sure we can come up with tons of examples :)

>> - Use a default manifest by default with =guix install= and =guix
>>   remove=.  This way would basically get rid of "ad-hoc" profiles which
>>   has so many flaws (non-reproducible, non-trackable, etc.).
>
> Ambitious!
>
> Note that “ad-hoc” profiles (imperatively-managed profiles) contain
> provenance info in their ‘manifest’ file, so it’s not black and white.

I actually never noticed this :p

>> Last, one that's at the Scheme level:
>>
>> - A Scheme function to create a manifest for the necessary inputs of a
>>   package, like =guix environment PACKAGE= does.  (Maybe it's already possible?)
>
> Like ‘specifications->manifest’?

Can specifications->manifest do this?

Concrete example:  You want a profile to work on IceCat, so you need a
manifest that has all the necessary inputs required to work with Icecat.
How can we automate the generation of the manifest?

> Note that another option is to overhaul ‘guix environment’ like David
> proposed a while back:
>
>   https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html

Nice thread, lots of good ideas in there.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[zimoun]

Hi,

On Mon, 4 Nov 2019 at 11:39, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> >> - A Scheme function to create a manifest for the necessary inputs of a
> >>   package, like =guix environment PACKAGE= does.  (Maybe it's already possible?)
> >
> > Like ‘specifications->manifest’?
>
> Can specifications->manifest do this?

I do not think so.
From my point of view [1], something is missing.
And to quote Ludo's answer [2]: "it’s a fact of life IMO.  :-)" ;-)

All the best,
simon


[1] https://lists.gnu.org/archive/html/guix-devel/2019-09/msg00193.html
[2] https://lists.gnu.org/archive/html/guix-devel/2019-09/msg00219.html

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> I'm actually surprised you find it surprising! :)
> I can think of Simon, maybe Konrad(?) and myself who mentioned it
> before.

Yes, me too. I could add to Pierre's list of use cases, but I prefer to
shift the discussion to a higher level.

What we have been discussing here recently is the organization of
software one level above packages. The vague idea is "groups of packages
that go together". Outside of the Guix universe, this is the realm of
(Docker) containers. A quick look at what happens in that universe
shows that composing such groups of packages corresponds to a frequent
need (see docker compose, Kubernetes, ...).

In Guix we have ephemeral (environment) vs. persistent (profile), ad-hoc
(package -i, environment from package lists, ...) and declarative
(manifests). It's a bit of a mess. Hacks such as "environment -r" show
that we ought to think about a better overall design. In addition to the
two dimensions I mentioned, this should include shareable/re-usable
vs. personal. People publish and share Docker images, so why wouldn't
they publish and share Guix super-packages? This third dimension also
raises the question of where the information (profiles, manifests, ...)
are stored and managed (version control?), and how they are referred to
(name, filename, ...). And of course how they are composed - in Guile,
at the shell level, or yet something else?

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Hartmut Goebel]

Am 05.11.19 um 07:26 schrieb Konrad Hinsen:
> In Guix we have ephemeral (environment) vs. persistent (profile), ad-hoc
> (package -i, environment from package lists, ...) and declarative
> (manifests). It's a bit of a mess.

+1

This becomes even messier if we would implement a "guix develop"
command, which is yet just another version of environment/profile.

And adding another dimension: spawning a sub-shell (environment) or not
(profile).

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Hartmut,

> This becomes even messier if we would implement a "guix develop"
> command, which is yet just another version of environment/profile.

Yes, I worry also about making more of a mess by implementing
special-purpose variants.

> And adding another dimension: spawning a sub-shell (environment) or not
> (profile).

How is this different from the ephemeral vs. persistent dimension?
Creating an ephemeral package set makes sense only if you spwan a
process in it (not necessarily a shell).

Cheers,
  Konrad

Re: Profiles/manifests-related command line interface enhancements

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

Am 05.11.19 um 10:03 schrieb Konrad Hinsen:
>> And adding another dimension: spawning a sub-shell (environment) or not
>> (profile).
> How is this different from the ephemeral vs. persistent dimension?
> Creating an ephemeral package set makes sense only if you spwan a
> process in it (not necessarily a shell).

You are right, these are (almost?) the same.

Ephemeral could also be imagined without spawning a some process, but also without registering a "root". But this would not work, since as soon as one runs "guix gc" while such an environment is active, the environment would be destroyed.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1498 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

Very good points!

We are having very constructive discussions these days about `guix
environment` and guix profiles ;)

Exciting times ahead!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[zimoun]

Hi Konrad,

On Tue, 5 Nov 2019 at 07:27, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> two dimensions I mentioned, this should include shareable/re-usable
> vs. personal. People publish and share Docker images, so why wouldn't
> they publish and share Guix super-packages?

I am not sure to see which feature is missing.

 - Channels allow to share package definition;
 - Publish allows to share pre-built binaries;
 - Pack allows to share images.

So what should be missing should be a central server collecting all to
ease, à la DockerHub.

From my point of view, ephemeral (environment) vs persistent (profile)
depends on use-case but at the end the only concern is: how to
populate them? And the only answer should be: declarative (manifest);
the ad-hoc (package -i) should not be the usual way but only a
quick&dirt test.

Therefore ephemeral (environment) vs persistent (package --profile)
should be compliant. Especially about composition. However, as
discussed elsewhere about manifest and the time-machine patch -- I
have not tried yet --, the manifest file perhaps needs more fine
control to better describe the channels, substitutes etc..

> This third dimension also
> raises the question of where the information (profiles, manifests, ...)
> are stored and managed (version control?),

Profiles are managed by Guix, isn't it?
I mean their information are in generations, from my understanding.

Manifests are managed by the user. And they does what they wants. ;-)

> and how they are referred to
> (name, filename, ...).

Do you mean that
  guix environment -m https://example.com/my-manifest.scm
should be possible?


All the best,
simon

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Simon,

> I am not sure to see which feature is missing.

My main point is that the future evolution of functionality (and user
interfaces) in this space should take into account usage scenarii,
rather than adding features in an ad-hoc fashion.

I can of course share a manifest file by putting it in a public
repository. But that isn't necessarily the best way.

Take the typical example from Docker tutorials: bundling a Web server
with some Web application and a database. It's easy to make a manifest
file for collecting the required packages. But it would make more sense
to me to have a module in a Guix channel that defines a parametrizable
super-package for the Web application that has no source code of its own
but various inputs and perhaps configuration files. Users can then
install several Web apps bundled in this fashion, sharing the Web
server. This kind of composition is not possible (currently) with
manifest files.

> From my point of view, ephemeral (environment) vs persistent (profile)
> depends on use-case but at the end the only concern is: how to
> populate them? And the only answer should be: declarative (manifest);
> the ad-hoc (package -i) should not be the usual way but only a
> quick&dirt test.

I agree. So maybe each profile should have an associated manifest file,
with "guix install" merely adding to it and the updating the profile.
But then it would make sense for guix to manage manifest files in git
repositories by default, to encourage good habits.

>> This third dimension also
>> raises the question of where the information (profiles, manifests, ...)
>> are stored and managed (version control?),
>
> Profiles are managed by Guix, isn't it?

Sure, but how exactly? Right now, a profile is a directory anywhere in
the file system that Guix knows about. Recent discussions have proposed
alternatives, such as keeping all of a user's profile in some directory
defined by convention and referring to them by name. What's the better
way to use as a default? I don't know, but I think we should discuss it
rather than adding new sub-commands with different behavior and thus
adding to the mess.

> Manifests are managed by the user. And they does what they wants. ;-)

True again, but again, is that the best way to do it?

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[zimoun]

Hi Konrad,

On Tue, 5 Nov 2019 at 17:05, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> > I am not sure to see which feature is missing.
>
> My main point is that the future evolution of functionality (and user
> interfaces) in this space should take into account usage scenarii,
> rather than adding features in an ad-hoc fashion.

I agree.

To me, scenarii is how we combine the features.

Speaking about your example of "à la Docker" usage, except the
manifest composition, we could already write an usage scenario, I
guess.


> I can of course share a manifest file by putting it in a public
> repository. But that isn't necessarily the best way.

From my opinion, it is not the problem of Guix. Guix should not force
a practise, as good it should be. The way the users share manifests is
the problem of users, not the Guix's one, IMHO.


> Take the typical example from Docker tutorials: bundling a Web server
> with some Web application and a database. It's easy to make a manifest
> file for collecting the required packages. But it would make more sense
> to me to have a module in a Guix channel that defines a parametrizable
> super-package for the Web application that has no source code of its own
> but various inputs and perhaps configuration files. Users can then
> install several Web apps bundled in this fashion, sharing the Web
> server. This kind of composition is not possible (currently) with
> manifest files.

I do not understand very well what "guix deploy" is, but it is not
what it is doing?

(except the manifest composition that we agree both is missing; Pierre
mentioned us ;-))

From my opinion, Docker tutorials seems really easy because 1.
everything is centralized and 2. they cheat (it is hard to replay a
random tuto couple of months later; from my experience.)


> > From my point of view, ephemeral (environment) vs persistent (profile)
> > depends on use-case but at the end the only concern is: how to
> > populate them? And the only answer should be: declarative (manifest);
> > the ad-hoc (package -i) should not be the usual way but only a
> > quick&dirt test.
>
> I agree. So maybe each profile should have an associated manifest file,
> with "guix install" merely adding to it and the updating the profile.

A kind of plumbing manifest already exists. See <your-profile>/manifest.
But it is not compliant with the "--manifest" option.

Let consider the scenario:

  guix package --manifest manif.scm -p foo #1
  guix pull #2
  guix package -i pkg -p foo #3


Initially, I was thinking [1] that it should be nice to create again
the exact same profile than at the end of the step #3 using
foo/manifest. For example, sending to colleagues the file foo/manifest
and they just run:

  guix package --manifest foo-manifest.scm -p bar

So packages comes from 2 different commits: one from the Guix state at
step #1 or provided by manif.scm, another one from the Guix updated
state of step #2.

But reading thread elsewhere about manifest and so on, yours and
Pierre's arguments convinced me that it is the wrong way. Other said,
it will become a big mess with too much inferiors (because it does not
force the user to use the manifests).

Instead, after testing (or whatever) at the step #3, the user should
write the correct manifest corresponding to this step #3, say
other.scm. Then sending manif.scm and other.scm to colleagues and they
run:

   guix package -m manif.scm -m other.scm -p baz

And the profiles foo and baz contain the exact same binaries (hope so :-))


What is missing is an easy way to track the Guix commit state. If the
user runs the step #1 and the file manif.scm does not provide any
commit, then they needs to write down the Guix commit. And it is not
straightforward if they forgot and already run the step #2. However,
this information is currently tracked by Guix in the file
foo/manifest.


From what I understand, "guix time-machine" is the answer to this kind
of scenario. Right?


[1] https://lists.gnu.org/archive/html/guix-devel/2019-09/msg00228.html


> But then it would make sense for guix to manage manifest files in git
> repositories by default, to encourage good habits.

I agree. The same way than channel does.


> >> This third dimension also
> >> raises the question of where the information (profiles, manifests, ...)
> >> are stored and managed (version control?),
> >
> > Profiles are managed by Guix, isn't it?
>
> Sure, but how exactly? Right now, a profile is a directory anywhere in
> the file system that Guix knows about. Recent discussions have proposed
> alternatives, such as keeping all of a user's profile in some directory
> defined by convention and referring to them by name. What's the better
> way to use as a default? I don't know, but I think we should discuss it
> rather than adding new sub-commands with different behavior and thus
> adding to the mess.

I agree about "adding to the mess". :-)

Personally, I do not like the "directory defined by convention and
referring to them by name" approach. The "package manager" Conda does
that and after some years using it (before switching to Guix \o/),
this approach appears to be also a mess. I never remembered the name
of the profile so I was always doing "conda list -e" oups! wrong
command the other one "conda info -e" to list all the "environment"
names and then source the one I want.

I prefer the Guix way and I store the profile (symbolic link) in the
project directory. This way, I always remember the name, I can quickly
create another one with a cryptic name -- naming is hard :-) -- and
weeks later I remember that it is attached to this project.

I understand the use-case described in the Pierre's blog post but this
use-case is mainly not mine and I am happy with the "scattered"
approach. :-)


About the subcommand mess, what should help IMHO should be to be able
to easily create "alias" and/or new subcommands distributed with
channels. I did not know before reading this bug report [2].


[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37399



All the best,
simon

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Simon,

>> I can of course share a manifest file by putting it in a public
>> repository. But that isn't necessarily the best way.
>
> From my opinion, it is not the problem of Guix. Guix should not force
> a practise, as good it should be. The way the users share manifests is
> the problem of users, not the Guix's one, IMHO.

Guix should support good practices in my opinion. Sharing files that can
contain arbitrary code (such as manifests) is not a good practice, for
security reasons. For personal use, on the other hand, that flexibility
is very valuable.

> From my opinion, Docker tutorials seems really easy because 1.
> everything is centralized and 2. they cheat (it is hard to replay a
> random tuto couple of months later; from my experience.)

I agree. I certainly don't want Guix to follow Docker in everything.
We can always do better.

> From what I understand, "guix time-machine" is the answer to this kind
> of scenario. Right?

Yes, in combination with "guix describe".

> Personally, I do not like the "directory defined by convention and
> referring to them by name" approach. The "package manager" Conda does
> that and after some years using it (before switching to Guix \o/),
> this approach appears to be also a mess. I never remembered the name

That's valuable feedback. I have never used conda myself systematically,
just a few times for following tutorials.

> I prefer the Guix way and I store the profile (symbolic link) in the
> project directory. This way, I always remember the name, I can quickly
> create another one with a cryptic name -- naming is hard :-) -- and
> weeks later I remember that it is attached to this project.

That's how I work as well. But Pierre's recent document convinced me
that profiles can also be a good way to modularize the global software
collection of my account. It would indeed be nice to be able to update
some parts (e.g. scientific software) more often than others
(e.g. TeXLive).

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Take the typical example from Docker tutorials: bundling a Web server
> with some Web application and a database. It's easy to make a manifest
> file for collecting the required packages. But it would make more sense
> to me to have a module in a Guix channel that defines a parametrizable
> super-package for the Web application that has no source code of its own
> but various inputs and perhaps configuration files. Users can then
> install several Web apps bundled in this fashion, sharing the Web
> server. This kind of composition is not possible (currently) with
> manifest files.

I agree this is an important use case.  It seems to me that the problem
here is being able to aggregate more than just packages.

A manifest can contain any lowerable object, not just packages, but
then, does it make much sense to use a manifest (and a profile) in this
case?

The Web application example above seems to be somewhere between a mere
<manifest> and an full-blown <operating-system>.  Perhaps we need an
intermediate abstraction that bundles together a profile along with some
lightweight services?

Or, to put it differently, how do we define “super-package”?

>> Profiles are managed by Guix, isn't it?
>
> Sure, but how exactly? Right now, a profile is a directory anywhere in
> the file system that Guix knows about. Recent discussions have proposed
> alternatives, such as keeping all of a user's profile in some directory
> defined by convention and referring to them by name. What's the better
> way to use as a default? I don't know, but I think we should discuss it
> rather than adding new sub-commands with different behavior and thus
> adding to the mess.

‘guix package --list-profiles’ was added to improve this situation where
there’s no enforced convention.

The question seems to be whether we should add more convention and less
configuration.

I’m often reluctant to that, preferring self-contained commands where
all the files involved appear on the command line.  However, I agree
that it’s a tradeoff and there’s a balance to be found to make the CLI
practical and convenient without making it obscure.  To that end, we
should probably usefully move from “zero conventions” to “some
conventions enforced” to improve over the current situation.

Examples of conventions discussed before:

  • Referring to profiles by name, meaning that users would no longer
    explicitly manipulate symlinks that live anywhere on the file
    system.

  • Having “guix environment” without arguments load an
    ‘environment.scm’ file (or similar) from the current directory.

Are these things you have in mind?  Anything else?

We should boil that discussion down to a list of things to implement.
:-)

Thanks!

Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Bengt Richter]

Hi all,

On +2019-11-06 18:07:09 +0100, Ludovic Courtès wrote:
> Hi Konrad,
> 
> Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
> 
> > Take the typical example from Docker tutorials: bundling a Web server
> > with some Web application and a database. It's easy to make a manifest
> > file for collecting the required packages. But it would make more sense
> > to me to have a module in a Guix channel that defines a parametrizable
> > super-package for the Web application that has no source code of its own
> > but various inputs and perhaps configuration files. Users can then
> > install several Web apps bundled in this fashion, sharing the Web
> > server. This kind of composition is not possible (currently) with
> > manifest files.
> 
> I agree this is an important use case.  It seems to me that the problem
> here is being able to aggregate more than just packages.
> 
> A manifest can contain any lowerable object, not just packages, but
> then, does it make much sense to use a manifest (and a profile) in this
> case?
> 
> The Web application example above seems to be somewhere between a mere
> <manifest> and an full-blown <operating-system>.  Perhaps we need an
> intermediate abstraction that bundles together a profile along with some
> lightweight services?
> 
> Or, to put it differently, how do we define “super-package”?
> 
> >> Profiles are managed by Guix, isn't it?
> >
> > Sure, but how exactly? Right now, a profile is a directory anywhere in
> > the file system that Guix knows about. Recent discussions have proposed
> > alternatives, such as keeping all of a user's profile in some directory
> > defined by convention and referring to them by name. What's the better
> > way to use as a default? I don't know, but I think we should discuss it
> > rather than adding new sub-commands with different behavior and thus
> > adding to the mess.
> 
> ‘guix package --list-profiles’ was added to improve this situation where
> there’s no enforced convention.
> 
> The question seems to be whether we should add more convention and less
> configuration.
> 
> I’m often reluctant to that, preferring self-contained commands where
> all the files involved appear on the command line.  However, I agree
> that it’s a tradeoff and there’s a balance to be found to make the CLI
> practical and convenient without making it obscure.  To that end, we
> should probably usefully move from “zero conventions” to “some
> conventions enforced” to improve over the current situation.
> 
> Examples of conventions discussed before:
> 
>   • Referring to profiles by name, meaning that users would no longer
>     explicitly manipulate symlinks that live anywhere on the file
>     system.
> 
>   • Having “guix environment” without arguments load an
>     ‘environment.scm’ file (or similar) from the current directory.
> 
> Are these things you have in mind?  Anything else?
> 
> We should boil that discussion down to a list of things to implement.
> :-)
> 
> Thanks!
> 
> Ludo’.
> 

1. guix explain EXAMPLE
as a quick way to get a definition of EXAMPLE from a guix glossary of terms and acronyms,
as well as references to example snippets etc. Also add easy ways to launch relevant links
to info or list archives or a wiki of troubleshooting experience, etc. in arbitrary URLs.

2. When implementing something, consider whether it could be useful as a separate script
that can be invoked from bash or on a region in emacs, vs being inaccessibly embedded,
and if useful, expose it in the current profile (or as directed).

As an example, putting snip-delimiters around some text could be done in elisp,
but it's more accessible and customizable as a script, e.g.,

--8<----(original did not do this)-----------cut here---------------start------------->8---
#!/usr/bin/env -S guile -e gxsnip -s
!#

(use-modules (ice-9 format))
(use-modules (ice-9 textual-ports))

(define (gxsnip args)
  (begin
    (let*((opt (cdr args))
	  (tag (if (pair? opt)
		   (string-append "(" (string-join opt " ") ")")
		   "")))
      (format #t 
       "--8<----~a~a\n~a\n--8<----~a~a\n"
       tag
       "-----------cut here---------------start------------->8---"
       (get-string-all (current-input-port))
       tag
       "-----------cut here---------------end--------------->8---"
       ))))

--8<----(original did not do this)-----------cut here---------------end--------------->8---

Oops. still using /usr/bin/... there ;-)
And I see easy improvements, but my point is it would have been less shareable as an emacs-only macro.

Where guix is factorable into separately useful components, I think the same idea applies.
The unix way of small things :)
HTH
--
Regards,
Bengt Richter

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> I agree this is an important use case.  It seems to me that the problem
> here is being able to aggregate more than just packages.

Yes, it's a bit more than that. We'd probably have to look at a couple
of use cases to see what the most general content would be.

> The Web application example above seems to be somewhere between a mere
> <manifest> and an full-blown <operating-system>.  Perhaps we need an
> intermediate abstraction that bundles together a profile along with some
> lightweight services?
>
> Or, to put it differently, how do we define “super-package”?

That's the question. I don't pretend to have an answer, I just wanted to
raise the question.

There are use cases, such as Web servers, that move towards the service
or even system abstraction. Other use cases would be just packages
plus configuration, with no automatic execution implied.

> ‘guix package --list-profiles’ was added to improve this situation where
> there’s no enforced convention.

Unfortunately it makes no distinction between package profiles
and channel profiles, so it's hard to use in scripts.

> The question seems to be whether we should add more convention and less
> configuration.

Or have all configuration in a single file per user, with convenient
defaults encoding a set of conventions. Trying to please everyone :-)

Right now the only conventions I can think of are the two default
profile names (~/.guix-profile for packages and ~/.config/guix/current
for channels). And I wonder BTW if the latter is not an abuse of
~/.config, given that it contains a ton of software rather than just
small configuration files, but that's a different question.

> We should boil that discussion down to a list of things to implement.
> :-)

Definitely. But I'd like to start by compiling a list of use cases,
to see how well they are served by what we have and then how they
could be served better.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Hey!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>
>> I'm actually surprised you find it surprising! :)
>> I can think of Simon, maybe Konrad(?) and myself who mentioned it
>> before.
>
> Yes, me too. I could add to Pierre's list of use cases, but I prefer to
> shift the discussion to a higher level.

Just to be clear: I think supporting multiple ‘--manifest’ flags in all
the commands would be welcome.  It’s probably a rather simple change,
and if it’s useful, go for it!

> What we have been discussing here recently is the organization of
> software one level above packages. The vague idea is "groups of packages
> that go together". Outside of the Guix universe, this is the realm of
> (Docker) containers. A quick look at what happens in that universe
> shows that composing such groups of packages corresponds to a frequent
> need (see docker compose, Kubernetes, ...).

Yes, I see.

> In Guix we have ephemeral (environment) vs. persistent (profile), ad-hoc
> (package -i, environment from package lists, ...) and declarative
> (manifests). It's a bit of a mess. Hacks such as "environment -r" show
> that we ought to think about a better overall design. In addition to the
> two dimensions I mentioned, this should include shareable/re-usable
> vs. personal. People publish and share Docker images, so why wouldn't
> they publish and share Guix super-packages? This third dimension also
> raises the question of where the information (profiles, manifests, ...)
> are stored and managed (version control?), and how they are referred to
> (name, filename, ...). And of course how they are composed - in Guile,
> at the shell level, or yet something else?

I think having ephemeral + persistent and declarative + imperative is
cool—I’d call that “flexible” rather than “messy”.  :-)

It’s great to have “guix install” as an entry point; I’m sure it allows
us to reach out to more people, and that matters too.  (I actually use
it, BTW, so it’s not an expert vs. newbie thing!)

I agree that sharing and publishing is important, and I think manifests
support that.  I think we need to support “unions” of manifests, and
that means (as always) supporting it at the API level and at the
command-line level (allowing for multiple ‘--manifest’ flags).

What we’re now saying is “look, you can write a manifest, and then you
can have it under version-control and publish it and it’s all up to you
how you do that”; but you seem to suggest that we should offer a
higher-level, more integrated solution, is that correct?

Like, we would enforce certain conventions by default, perhaps have
direct Git integration so that one can refer to a “manifest” just like
they refer to a channel, things like that.  Do I get it right?

I haven’t thought much about this but that sounds like an exciting
direction!

Thanks,
Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> I think having ephemeral + persistent and declarative + imperative is
> cool—I’d call that “flexible” rather than “messy”.  :-)

I agree. What's messy is how the concepts map to commands. How many Guix
users understand that profiles are persistent environments, or
environments ephemeral profiles? And how many understand what "guix
environment -r" does exactly, and how it should be used?  It took me a
while to figure this out.

What we have is two commands (package and environment) each designed for
one main usage pattern, plus options to get something else. But even
those options don't overlap completely in functionality. For example,
how do I make a profile with the dependencies of a package?

The current discussion started with adding more commands for different
specific usage patterns. If we go that route the mess will become worse.

> It’s great to have “guix install” as an entry point; I’m sure it allows
> us to reach out to more people, and that matters too.  (I actually use
> it, BTW, so it’s not an expert vs. newbie thing!)

Me too :-) It's "guix package" that is the worst offender in my
opinion. It does two distinct things: querying the package database and
managing profiles. And now that we have "guix search" for queries, I'd
like to see "guix package" go away, to be replaced by either "guix
profile" for profile management, with as much overlap as possible in
options with "guix environment", or by a single command that handles
environments and profiles in a unified way.

> I agree that sharing and publishing is important, and I think
> manifests support that.

They do, but not very well in my opinion. I think everything meant to be
shared, published, and maintained should be accessible by name in a
database. A channel, for example.

Some ideas that could make this possible (and convenient):

 - Better support for adding/managing channels at the user account
   level. Users shouldn't have to edit Guile code (unless they want to).

 - Support for creating and managing channels without having to
   descend to the file and repository level.

 - Support for something one could call super-packages or named
   manifests. Named package groups that live in a channel (or a guix.scm
   distributed with software external to Guix). Perhaps parametrizable
   and/or containing configuration files, so you could put something
   like (nginx #:port 8080) into a manifest file.

> I think we need to support “unions” of manifests, and
> that means (as always) supporting it at the API level and at the
> command-line level (allowing for multiple ‘--manifest’ flags).

Yes, I agree. That's the easiest part.

> What we’re now saying is “look, you can write a manifest, and then you
> can have it under version-control and publish it and it’s all up to you
> how you do that”; but you seem to suggest that we should offer a
> higher-level, more integrated solution, is that correct?

Exactly. Every functionality that requires end-users to manage
Guile code will always be restricted to expert users. Manifest
files may look simple and understandable for simple cases,
but if you expect users to install manifests downloaded from
someone else, they need to be able to inspect them and be sure
that installing the manifest file won't delete their user account.
And that means they have to understand a lot about Guile.

> Like, we would enforce certain conventions by default, perhaps have
> direct Git integration so that one can refer to a “manifest” just like
> they refer to a channel, things like that.  Do I get it right?

Yes, that's the idea!

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> I agree. What's messy is how the concepts map to commands. How many Guix
> users understand that profiles are persistent environments, or
> environments ephemeral profiles? And how many understand what "guix
> environment -r" does exactly, and how it should be used?  It took me a
> while to figure this out.

Yup!  Before I got down to write the article, I must confess I had been
using Guix wrong for more than a year :p

> What we have is two commands (package and environment) each designed for
> one main usage pattern, plus options to get something else. But even
> those options don't overlap completely in functionality. For example,
> how do I make a profile with the dependencies of a package?
>
> The current discussion started with adding more commands for different
> specific usage patterns. If we go that route the mess will become worse.

Agreed.

> Me too :-) It's "guix package" that is the worst offender in my
> opinion. It does two distinct things: querying the package database and
> managing profiles. And now that we have "guix search" for queries, I'd
> like to see "guix package" go away, to be replaced by either "guix
> profile" for profile management, with as much overlap as possible in
> options with "guix environment", or by a single command that handles
> environments and profiles in a unified way.

I like this idea a lot!

> They do, but not very well in my opinion. I think everything meant to be
> shared, published, and maintained should be accessible by name in a
> database. A channel, for example.
>
> Some ideas that could make this possible (and convenient):
>
>  - Better support for adding/managing channels at the user account
>    level. Users shouldn't have to edit Guile code (unless they want to).

This is a great idea.
By the way, the same should be possible from the installer:

- Specify which channels you want to use.
- Specify which config.scm you want to use.  Indeed, the config.scm
  might rely on channels.

This way, the Guix installation process would boil down to the above 2
items, that is to say... 1 minute of the user time!  Great, no?


Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> By the way, the same should be possible from the installer:
>
> - Specify which channels you want to use.
> - Specify which config.scm you want to use.  Indeed, the config.scm
>   might rely on channels.
>
> This way, the Guix installation process would boil down to the above 2
> items, that is to say... 1 minute of the user time!  Great, no?

This does open some interesting possibilities, such as experts defining
personalized system configurations which are then easy to install for
anyone. Although I am not terribly fond of the idea of typing URLs into
an installer.

The bigger issue is config.scm - again unrestricted Guile code, like
manifests. That's not good for publishing because we shouldn't encourage
anyone to run unrestricted code from untrusted sources.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> This does open some interesting possibilities, such as experts defining
> personalized system configurations which are then easy to install for
> anyone. Although I am not terribly fond of the idea of typing URLs into
> an installer.

This should not be an issue once we have Live Guix with a graphical
installer and all :)

> The bigger issue is config.scm - again unrestricted Guile code, like
> manifests. That's not good for publishing because we shouldn't encourage
> anyone to run unrestricted code from untrusted sources.

I think the most common use case here is to provide your own config.scm
that you have in some git repository for instance.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> The bigger issue is config.scm - again unrestricted Guile code, like
> manifests. That's not good for publishing because we shouldn't encourage
> anyone to run unrestricted code from untrusted sources.

It’s a conscious design choice to have configuration as code everywhere.
The alternative is to define DSLs embedded in XML/JSON/YAML, and we know
what this often leads to: feature creep, half-baked semi-general-purpose
languages.

Of course, using a general-purpose language upfront also comes at a
price, as you note.  But I think that what it has to offer to users
outweighs the costs, and that’s a lesson learned from Emacs.  Just to
say I’m not willing to replace ‘config.scm’ with ‘config.yaml’, if
that’s what you had in mind.  :-)

So I think we should address the use cases we’ve been discussing without
compromising on this fundamental choice that gives users all the power,
flexibility, and transparency of a consistent and extensible system.
And I’m sure there are ways to achieve that!

Thanks,
Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> Of course, using a general-purpose language upfront also comes at a
> price, as you note.  But I think that what it has to offer to users
> outweighs the costs, and that’s a lesson learned from Emacs.  Just to
> say I’m not willing to replace ‘config.scm’ with ‘config.yaml’, if
> that’s what you had in mind.  :-)

YAML is for kids. Real managers won't settle for less than full XML. ;-)

Seriously, as a power user, I am perfectly happy with Guile for
everything. I certainly don't want less. And for now, it's safe to
assume that most Guix users are power users. The question is if we want
Guix to remain exclusively a power tool for power users. If not, we need
to make sure that it won't become a malware platform, by making it safe
to use for people who don't read Guile code. In particular, common use
cases should not require users do download unrestricted Guile code from
untrusted sources.

Emacs is an interesting comparison in many ways, but also a much less
interesting target for malware than Guix. An attack on Guix can
undermine all the guarantees it provides through reproducible builds.
Maybe Ken Thompson should do an update of his famous "Trusing trust"
that extends the discussion of compilers to build tools in general.

One direction could be to add a sandboxing feature to Guile, which would
be nice-to-have for other uses as well if Guile is to become a
general-purpose systems scripting language. There are some interesting
ideas in shill (http://shill.seas.harvard.edu/) for this scenario.

Cheers,
  Konrad.

A better XML, config is code (was Re: Profiles/manifests-related command line...)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2798 bytes --]

Hello Konrad,

this thread is slowly going OT from the initial subject, anyway I found
an interesting comment from you I would like to reply

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Hi Ludo,

[...]

>> Just to say I’m not willing to replace ‘config.scm’ with
>> ‘config.yaml’, if that’s what you had in mind.  :-)
>
> YAML is for kids. Real managers won't settle for less than full
> XML. ;-)

When I started studying Guix/Scheme (and I'm still far away from
understanding) I stubled on an article advocating Lisp to XML-minded
people like I was: http:/www.defmacro.org/ramblings/lisp.html

The point is: config.scm is better than config.xml that is better than
config.yaml that is better than config.json :-D

The real question is: a configure file is code or data?  IMHO is code,
especially when it comes to system configuration; for this reason every
user learn to *code* in some way or another (even when using a GUI to
configure software)

Systems are complex, we should avoid complicating them (think any
configuration management system you like :-O ) *and* we should also
avoid oversimplifying them [1]; this also means users must know what they
are doing

[...]

> The question is if we want Guix to remain exclusively a power tool for
> power users.

Mumble... but every user *is* a power user when installing and
configuring a system, no? The "only" difference is what tools *and*
binary distribution (or binary building) system we decide to trust when
installing and configuring our system, and our decision should be
informed

...so yes, if it's not a channel under your control - or of someone you
decide to trust - you should better not use it (and do not copy/paste
configuration files you do not understand)

I recently read this "Curl to shell isn't so bad" article (thanks ARota)
https://arp242.net/curl-to-sh.html

«In the end it’s still just running code you didn’t personally audit on
your computer, and a matter of trust.»

Guix is the best candidate to build a software ecosystem which we can
trust (and we already have), starting from "almost zero" binary depends
via GNU Mes and stage0; and we can always /challenge/ any substitute
server :-)

This obviously does not solve the "trust in source code" problem [2],
but this is another story

[...]


Thanks! Gio'


[1] Docker "containers" are easier from an "end user" POV than
installing and configuring via a "classic" package manager... but have
you ever tried to understand what's installed in a 7 layer filesystems
container?!? So what is *seems* simple is often _complicated_ :-O 

[2] and that was also *not* the scope of the famous Trusting Trust speech

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: A better XML, config is code (was Re: Profiles/manifests-related command line...)

[Konrad Hinsen]

Hi Giovanni,

> The real question is: a configure file is code or data?  IMHO is code,

Code is data with execution semantics, so "code" is a subset of "data".

I'd reformulate the question as: should configuration data be
literal data, or the result of a computation? The second opton
is more general, and therefore more powerful. If that is good or bad
depends on the application. If you are writing the configuration, you
appreciate more power. If you use someone else's, you might well prefer
it not being more powerful than what you can understand.

> Mumble... but every user *is* a power user when installing and
> configuring a system, no?

"Is", no. That would assume that everybody knows their limits. Not true
in my experience with human nature.

> ...so yes, if it's not a channel under your control - or of someone you
> decide to trust - you should better not use it (and do not copy/paste
> configuration files you do not understand)

Fine with me, but then we should (1) say so somewhere in the manual and
(2) not recommend using such configuration files for performing tasks
that ought to be accessible to ordinary users.

> I recently read this "Curl to shell isn't so bad" article (thanks ARota)
> https://arp242.net/curl-to-sh.html
>
> «In the end it’s still just running code you didn’t personally audit on
> your computer, and a matter of trust.»

Exactly. It's OK for us to ask users to trust the Guix team, which they
have to do anyway if they use Guix. So configuration files provided by
Guix itself are not a problem.

But if we tell people that Guix is great for reproducing someone else's
computation, and that the best way to share a computation is publishing
a manifest file, then we are encouraging people to run code from
untrusted sources. Which leaves three options:
 1. Provide a safe way to re-create environments from untrusted
    sources.
 2. Don't recommend reproducing someone else's computation using Guix.
 3. Explain why reproducing someone else's computation is
    a risky procedure that should be reserved to power users.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Andy Wingo]

On Sun 10 Nov 2019 10:36, Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> One direction could be to add a sandboxing feature to Guile, which would
> be nice-to-have for other uses as well if Guile is to become a
> general-purpose systems scripting language. There are some interesting
> ideas in shill (http://shill.seas.harvard.edu/) for this scenario.

I wrote this for that purpose:

  https://www.gnu.org/software/guile/manual/html_node/Sandboxed-Evaluation.html

However I can't recommend it as a robust security layer because of the
weaknesses in the heap allocation limit; discussed in the page above.

I agree that Shill has some great patterns that go beyond what Guile or
Guix has, and that adopting some of them is a really interesting idea
:-)

I admit that I was a bit depressed at the impact that Spectre et al has
had on language-level sandboxing abstractions :-( and haven't much
pursued this line since then.  In practice Guix's "containerized" build
jobs are much more effective than in-language barriers.

Cheers,

Andy

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Andy,

> I wrote this for that purpose:
>
>   https://www.gnu.org/software/guile/manual/html_node/Sandboxed-Evaluation.html

Right, I had found this when searching for something. It seems to solve
a couple of problems that I don't quite understand, but not so much
those I do (file/network access). Would be nice to see this extended.

> In practice Guix's "containerized" build jobs are much more effective
> than in-language barriers.

Indeed, but if Guix is compromised by malware, the build jobs may build
code that has already been tampered with.

Maybe one could have config and manifest files interpreted by the build
daemon for safety. Except that some manifest files (read: mine) need
read access to the file system.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Bengt Richter]

Hi Andy, Guix...

On +2019-11-12 09:55:27 +0100, Andy Wingo wrote:
> On Sun 10 Nov 2019 10:36, Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
> 
> > One direction could be to add a sandboxing feature to Guile, which would
> > be nice-to-have for other uses as well if Guile is to become a
> > general-purpose systems scripting language. There are some interesting
> > ideas in shill (http://shill.seas.harvard.edu/) for this scenario.
> 
> I wrote this for that purpose:
> 
>   https://www.gnu.org/software/guile/manual/html_node/Sandboxed-Evaluation.html
> 
> However I can't recommend it as a robust security layer because of the
> weaknesses in the heap allocation limit; discussed in the page above.
> 
> I agree that Shill has some great patterns that go beyond what Guile or
> Guix has, and that adopting some of them is a really interesting idea
> :-)
> 
> I admit that I was a bit depressed at the impact that Spectre et al has
> had on language-level sandboxing abstractions :-( and haven't much
┌───────────────────────────────────────────────────────────────────────────┐
│ > pursued this line since then.  In practice Guix's "containerized" build │
│ > jobs are much more effective than in-language barriers.                 │
└───────────────────────────────────────────────────────────────────────────┘
> 
> Cheers,
> 
> Andy
> 
Would it be possible to have a sand-box daemon like the build daemon
which could run sandboxed guile expressions safely?

If designed for the future, maybe such a daemon's interface could anticipate
replacing the daemon and talking to a hypervisor dom0 as in Qubes-OS?

-- 
Regards,
Bengt Richter
--8<----(OT PS)-----------cut here---------------start------------->8---
Andy, have you looked at glTF and sketchfab?
Would you be interested in bringing that kind of 3D graphics into
the Guix package world? (or are you or someone already doing something? :)

https://sketchfab.com/features/gltf

Have a look with firefox (my icecat on top of weston-launch shows the static
images beautifully, but no dynamics, need to get js going).
Play with rotating and zooming, really nice, plus animated stuff ;-)
I think it would be super-cool to have this 3D modeling capability
for Guix presentations, toys and fun ;-)
--8<----(OT PS)-----------cut here---------------end--------------->8---

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> YAML is for kids. Real managers won't settle for less than full XML. ;-)
>
> Seriously, as a power user, I am perfectly happy with Guile for
> everything. I certainly don't want less. And for now, it's safe to
> assume that most Guix users are power users. The question is if we want
> Guix to remain exclusively a power tool for power users.

I’d like to think that writing Guile declarations for the OS config,
manifest, etc. is not just for “power users”.  After all people, or
rather “computer-savvy” people in a broad sense, write JSON, YAML,
custom config files etc. routinely, and I don’t think the typical config
we propose is any “harder”.  You may say I’m a dreamer, but I’m not the
only one. 𝅗𝅥𝅘𝅥  ;-)

> If not, we need to make sure that it won't become a malware platform,
> by making it safe to use for people who don't read Guile code. In
> particular, common use cases should not require users do download
> unrestricted Guile code from untrusted sources.

Definitely!

I think we need to focus on specific scenarios though.

For example, when you add a channel to ~/.config/guix, you’ll end up
running its unrestricted Guile code as soon as ‘guix pull’ is done.
However, independently of that, you’ll most likely be installing
packages defined in that channel, and then running them unrestricted.
IOW, users of a channel have to trust it to not be malicious, regardless
of the fact that its Guile code runs unrestricted.

For manifests shared over the net, the situation may be different: a
manifest could refer to packages in the channels you trust, and thus
there’s value in not having to trust the manifest code itself.

It’s still a bit too abstract, but for the purposes of sharing and
publishing “super packages” as you wrote, we could define a
purely-declarative format (could be JSON, could be Guile code that can
run under (ice-9 sandbox) or with ‘eval/container’) that people could
use instead unrestricted as is currently the case.

Thanks,
Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> I’d like to think that writing Guile declarations for the OS config,
> manifest, etc. is not just for “power users”.  After all people, or
> rather “computer-savvy” people in a broad sense, write JSON, YAML,
> custom config files etc. routinely, and I don’t think the typical config
> we propose is any “harder”.  You may say I’m a dreamer, but I’m not the
> only one. 𝅗𝅥𝅘𝅥  ;-)

The problem with powerful formal languages (read: Turing-complete) is
not writing, but (1) debugging and (2) reading.

1. Writing a manifest file in Guile is no harder than writing a list
   in YAML or whatever. But leave out the closing quote behind a
   package name, and you will get an error message that will make
   no sense to someone unfamiliar with the *complete* Scheme syntax.
   For a small and simple file, you can usually spot the problem by
   inspection (i.e. without the error message), but for more complex
   files, it makes sense to use a more constrained language in order
   to provide better error reporting.

   BTW, the Racket team makes that point to argue for their rather
   complex macro system. It takes much longer to master than traditional
   Lisp or Scheme macros, but it supports writing macros with error
   reporting that makes sense to someone who didn't read the macro code.

2. Power users will always write code in powerful languages that exceed
   what less advanced users can deal with. And since power users are not
   necessarily benevolent, this creates a trust issue for the less
   advanced ones.

For a long version of these arguments, see
   https://hal.archives-ouvertes.fr/hal-01966145/document

> I think we need to focus on specific scenarios though.

Definitely!

> IOW, users of a channel have to trust it to not be malicious,
> regardless of the fact that its Guile code runs unrestricted.

Yes. That's perhaps something that the manual should point out
explicitly. Also, a more specific user interface ("guix channel add
URL") could show a warning.

> For manifests shared over the net, the situation may be different: a
> manifest could refer to packages in the channels you trust, and thus
> there’s value in not having to trust the manifest code itself.

Exactly, and that's the idea that got me into this thread.

> It’s still a bit too abstract, but for the purposes of sharing and
> publishing “super packages” as you wrote, we could define a
> purely-declarative format (could be JSON, could be Guile code that can
> run under (ice-9 sandbox) or with ‘eval/container’) that people could
> use instead unrestricted as is currently the case.

Yes, that's one way to go. BTW, I didn't know about eval/container,
I'll have to look at it!

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[zimoun]

Hi Konrad,

On Sun, 17 Nov 2019 at 11:45, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> For a long version of these arguments, see
>    https://hal.archives-ouvertes.fr/hal-01966145/document

Thank you for the pointer.
I missed it months ago. :-)

So, you propose to "enrich" the DSL describing the manifest files, right?


All the best,
simon

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Simon,

> So, you propose to "enrich" the DSL describing the manifest files, right?

Perhaps. I suppose there are several possible approaches, and I don't
claim to know what works best before actually trying.

One way is to "harden" manifest files, providing richer constructs for
defining and transforming packages but putting them into a
sandbox. Another way is to push as much information as possible into
channels and make manifests little more than lists of package
references.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Howdy!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> I’d like to think that writing Guile declarations for the OS config,
>> manifest, etc. is not just for “power users”.  After all people, or
>> rather “computer-savvy” people in a broad sense, write JSON, YAML,
>> custom config files etc. routinely, and I don’t think the typical config
>> we propose is any “harder”.  You may say I’m a dreamer, but I’m not the
>> only one. 𝅗𝅥𝅘𝅥  ;-)
>
> The problem with powerful formal languages (read: Turing-complete) is
> not writing, but (1) debugging and (2) reading.

Yes and no.  Guile has a debugger, whereas XML/JSON/YAML don’t.  As for
reading, it’s subjective, but I don’t think a full-blown language is
necessarily harder to read.

  (define lst (list 1 2 3))

  lst = [1, 2, 3]

  lst:
    - 1
    - 2
    - 3

The goal for Guix was to have the ‘package’ and ‘operating-system’
forms, for instance, look exactly like what you’d write in JSON etc.,
only with a different syntax.

> 1. Writing a manifest file in Guile is no harder than writing a list
>    in YAML or whatever. But leave out the closing quote behind a
>    package name, and you will get an error message that will make
>    no sense to someone unfamiliar with the *complete* Scheme syntax.
>    For a small and simple file, you can usually spot the problem by
>    inspection (i.e. without the error message), but for more complex
>    files, it makes sense to use a more constrained language in order
>    to provide better error reporting.

Error-reporting is tricky, but it’s tricky also (perhaps even more!)
when parsing YAML.

I agree there’s room for improvement in Guile in that area, but it’s not
that bad either if you compare to error reporting with most YAML, JSON,
etc. libraries.  Also Guile can report errors about semantics.

>    BTW, the Racket team makes that point to argue for their rather
>    complex macro system. It takes much longer to master than traditional
>    Lisp or Scheme macros, but it supports writing macros with error
>    reporting that makes sense to someone who didn't read the macro code.

‘syntax-parse’ is wonderful for that, I agree.

> 2. Power users will always write code in powerful languages that exceed
>    what less advanced users can deal with. And since power users are not
>    necessarily benevolent, this creates a trust issue for the less
>    advanced ones.

Good point.  I tend to (naively?) view it the other way around: that it
gives people an incentive to try and write actual code rather than mere
declarations.

> For a long version of these arguments, see
>    https://hal.archives-ouvertes.fr/hal-01966145/document

Interesting!  The conclusion you seem to draw is that embedded DSLs as
found in Racket fit the bill, and to me, that’s pretty much what we’ve
been trying to do with Guix as well, no?

What do we disagree on, actually?  :-)

Thanks again for your feedback!

Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

I'll start from the end:

> What do we disagree on, actually?  :-)

This:

>> 2. Power users will always write code in powerful languages that exceed
>>    what less advanced users can deal with. And since power users are not
>>    necessarily benevolent, this creates a trust issue for the less
>>    advanced ones.
>
> Good point.  I tend to (naively?) view it the other way around: that it
> gives people an incentive to try and write actual code rather than mere
> declarations.

I'd say we should encourage people to write declarations as much as
possible and resort to executable code only when declarations become
too messy. As a corollary, we should support most configuration-style
use cases with suitable declarative forms, much like these:

> The goal for Guix was to have the ‘package’ and ‘operating-system’
> forms, for instance, look exactly like what you’d write in JSON etc.,
> only with a different syntax.

For better illustration, I'll try to rewrite my own manifests in the
way I'd like to be able to write them. That's probably more useful
than theory (a tough statement to make for a theoretician ;-)

The main reason why I want to see more declarative style is:

>> The problem with powerful formal languages (read: Turing-complete) is
>> not writing, but (1) debugging and (2) reading.
>
> Yes and no.  Guile has a debugger, whereas XML/JSON/YAML don’t.  As for
> reading, it’s subjective, but I don’t think a full-blown language is
> necessarily harder to read.

It's harder to read because you need to understand the language's
execution model if there is one. YAML etc. don't, there is nothing
but literals. Which is also why they don't need a debugger.

>   (define lst (list 1 2 3))
>
>   lst = [1, 2, 3]

Fine. But then a power user comes along and writes

   (define lst (cons* 1 2 '(3)))

That may be bad style, but as a reader I have to be able to deal with it
nevertheless. And bad style may in fact serve to obfuscate evil
intentions.

> Interesting!  The conclusion you seem to draw is that embedded DSLs as
> found in Racket fit the bill, and to me, that’s pretty much what we’ve
> been trying to do with Guix as well, no?

Indeed, just not enough for my taste!

Cheers,
  Konrad.

On DSLs

[Ludovic Courtès]

Hello,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> What do we disagree on, actually?  :-)
>
> This:
>
>>> 2. Power users will always write code in powerful languages that exceed
>>>    what less advanced users can deal with. And since power users are not
>>>    necessarily benevolent, this creates a trust issue for the less
>>>    advanced ones.
>>
>> Good point.  I tend to (naively?) view it the other way around: that it
>> gives people an incentive to try and write actual code rather than mere
>> declarations.
>
> I'd say we should encourage people to write declarations as much as
> possible and resort to executable code only when declarations become
> too messy. As a corollary, we should support most configuration-style
> use cases with suitable declarative forms, much like these:
>
>> The goal for Guix was to have the ‘package’ and ‘operating-system’
>> forms, for instance, look exactly like what you’d write in JSON etc.,
>> only with a different syntax.

I agree.

> For better illustration, I'll try to rewrite my own manifests in the
> way I'd like to be able to write them. That's probably more useful
> than theory (a tough statement to make for a theoretician ;-)

Agreed!

> The main reason why I want to see more declarative style is:
>
>>> The problem with powerful formal languages (read: Turing-complete) is
>>> not writing, but (1) debugging and (2) reading.
>>
>> Yes and no.  Guile has a debugger, whereas XML/JSON/YAML don’t.  As for
>> reading, it’s subjective, but I don’t think a full-blown language is
>> necessarily harder to read.
>
> It's harder to read because you need to understand the language's
> execution model if there is one. YAML etc. don't, there is nothing
> but literals. Which is also why they don't need a debugger.

That’s not true.  In some cases, people write something that’s actually
code (in YAML, in JSON, etc.) and there’s an interpreter running it.
There’s usually no tooling coming with that interpreter, in particular
no debugger, error reporting may not be optimal either, and the
semantics may be ill-defined (I’m getting close to Greenspun’s tenth
rule :-)).  It’s just that it’s not presented that way, but that’s what
it is.

IOW, I think you can have a declarative _style_ in a full-blown
language, like:

  (define coreutils
    (package
      (name "coreutils")
      ;; …
      ))

And you can have “arbitrary code” in things that are presented as “pure
declarations” or not-a-programming-language.  For an example, see
‘eval-cabal’ in (guix import cabal), which evaluates Cabal “things”
(code or not?).  Or see the ‘syntax-rules’ “declarations”.

This is just to say that we should not conflate the style and the
language.  I think what we care about is supporting a declarative style,
and making it expressive enough that people don’t feel the need to
resort to complicated code.

>>   (define lst (list 1 2 3))
>>
>>   lst = [1, 2, 3]
>
> Fine. But then a power user comes along and writes
>
>    (define lst (cons* 1 2 '(3)))
>
> That may be bad style, but as a reader I have to be able to deal with it
> nevertheless. And bad style may in fact serve to obfuscate evil
> intentions.

I agree, but I think that this is hardly avoidable.  Sure, a pet DSL
will initially have nothing but ‘list’ so the problem won’t exist.  But
sooner or later, it’ll get ‘cons*’ as well.

Thanks,
Ludo’.

Re: On DSLs

[zimoun]

Hi Ludo,

On Tue, 26 Nov 2019 at 10:52, Ludovic Courtès <ludo@gnu.org> wrote:

> That’s not true.  In some cases, people write something that’s actually
> code (in YAML, in JSON, etc.) and there’s an interpreter running it.
> There’s usually no tooling coming with that interpreter, in particular
> no debugger, error reporting may not be optimal either, and the
> semantics may be ill-defined (I’m getting close to Greenspun’s tenth
> rule :-)).  It’s just that it’s not presented that way, but that’s what
> it is.

Out of curiosity, do you have examples about YAML or JSON with an
interpreter running it?


Thank for this interesting discussion. :-)

Cheers,
simon

Re: On DSLs

[Julien Lepiller]

Le 2 décembre 2019 20:05:57 GMT+01:00, zimoun <zimon.toutoune@gmail.com> a écrit :
>Hi Ludo,
>
>On Tue, 26 Nov 2019 at 10:52, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> That’s not true.  In some cases, people write something that’s
>actually
>> code (in YAML, in JSON, etc.) and there’s an interpreter running it.
>> There’s usually no tooling coming with that interpreter, in
>particular
>> no debugger, error reporting may not be optimal either, and the
>> semantics may be ill-defined (I’m getting close to Greenspun’s tenth
>> rule :-)).  It’s just that it’s not presented that way, but that’s
>what
>> it is.
>
>Out of curiosity, do you have examples about YAML or JSON with an
>interpreter running it?
>
>
>Thank for this interesting discussion. :-)
>
>Cheers,
>simon

Could this discussion be saved in the cookbook for instance? I'd like to have this kind of discussion on the approach of guix and ideas behind it somewhere more accessible than the ML archive. Does it make sense?

Re: On DSLs

[Konrad Hinsen]

Hi Julien,

> Could this discussion be saved in the cookbook for instance? I'd like
> to have this kind of discussion on the approach of guix and ideas
> behind it somewhere more accessible than the ML archive. Does it make
> sense?

I think it makes sense to archive summaries of such discussions. I am
less convinced that the cookbook is the right place, because design
principles are pretty much the opposite of what I'd expect in a
cookbook. But I don't see any better place and multiplying the parts of
the documentation isn't great either. So... I am undecided ;-)

Cheers,
  Konrad.

Re: On DSLs

[Ricardo Wurmus]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

>> Could this discussion be saved in the cookbook for instance? I'd like
>> to have this kind of discussion on the approach of guix and ideas
>> behind it somewhere more accessible than the ML archive. Does it make
>> sense?
>
> I think it makes sense to archive summaries of such discussions. I am
> less convinced that the cookbook is the right place, because design
> principles are pretty much the opposite of what I'd expect in a
> cookbook. But I don't see any better place and multiplying the parts of
> the documentation isn't great either. So... I am undecided ;-)

We can rename the cookbook to make it fit ;)

It was a placeholder name, because we all agreed for some time that a
“document” like that would be useful and that it shouldn’t be a wiki.
Beyond that we really didn’t have any plans.  I went ahead and
started it to enable collaboration on this resource — whatever it may
become eventually.

Re: On DSLs

[zimoun]

Hi,

On Tue, 3 Dec 2019 at 15:36, Ricardo Wurmus <rekado@elephly.net> wrote:

> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> >> Could this discussion be saved in the cookbook for instance? I'd like
> >> to have this kind of discussion on the approach of guix and ideas
> >> behind it somewhere more accessible than the ML archive. Does it make
> >> sense?
> >
> > I think it makes sense to archive summaries of such discussions. I am
> > less convinced that the cookbook is the right place, because design
> > principles are pretty much the opposite of what I'd expect in a
> > cookbook. But I don't see any better place and multiplying the parts of
> > the documentation isn't great either. So... I am undecided ;-)
>
> We can rename the cookbook to make it fit ;)
>
> It was a placeholder name, because we all agreed for some time that a
> “document” like that would be useful and that it shouldn’t be a wiki.
> Beyond that we really didn’t have any plans.  I went ahead and
> started it to enable collaboration on this resource — whatever it may
> become eventually.

IMHO, Julien has opened an interesting discussion about the Cookbook
in this thread [1].

From my point view, we have not defined what the Cookbook is i.e., its
aim and we all have different definitions, so we do not know where to
document what.

[1] https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00478.html


Cheers,
simon

Re: On DSLs

[Bengt Richter]

On +2019-12-03 15:12:32 +0100, Ricardo Wurmus wrote:
> 
> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
> 
> >> Could this discussion be saved in the cookbook for instance? I'd like
> >> to have this kind of discussion on the approach of guix and ideas
> >> behind it somewhere more accessible than the ML archive. Does it make
> >> sense?
> >
> > I think it makes sense to archive summaries of such discussions. I am
> > less convinced that the cookbook is the right place, because design
> > principles are pretty much the opposite of what I'd expect in a
> > cookbook. But I don't see any better place and multiplying the parts of
> > the documentation isn't great either. So... I am undecided ;-)
> 

I would like to see useful info and tested guix (and related) usage examples
accumulate in a knowledge base for easy searching and extraction, however it's named.

I regularly snip out examples from developer postings as grist for my personal
how-to notes, but that doesn't help others, and no common resource grows.

I think as much as possible ought to be automated, therefore I am proposing
... (well, sketching out half-baked proposal ideas :) ...
that some sort of delimiting syntax be defined to make automated extraction
and re-presentation of search results easy and effective.

I think the mailing list archives could serve as a knowledgebase if
desirable posting content in our emails were delimited with a MIME-inspired extension
of the established: "--8<---------------cut here---------------start------------->8---"
delimiter.

When a thread comes to an agreed conclusion, it should not be hard to wrap a few
paragraphs in appropriate delimiters. Maybe use rc-n versioning in leadup?

I would suggest making delimited text unique too, by leveraging some prefix characters
of base32 of content, along with some identifying info. Side benefit: big search
engines will find them for you, wherever copies wind up.

In fact, if we adopted real MIME multipart delimiters with the final one ending with
"--" appended, it would open many possibilities (and probably tap into lots of existing parsing tools).

But guix-specifically, for guile processing of text with mime-delimited sections,
maybe someone --( I tried some time ago, but got no sa-tis-faction ;-/ )--
could make a guile module that could be imported to define a reader macro e.g.
like #--"boundary" which would chew through input and convert
it into a list of (header . content) pairs until the final "\n--boundary--" is encountered.
Think of it as a super-quote that can quote anything, even broken quote syntax from other
languages and of course unbalanced quoting and bracketing from lispy languages.
Hm. maybe a special #L language switch to do the parsing?

Anyway, both the header and the content of the macro output pair would be string,
since I am just sketching a way to quote and identify desirable text in emails
and media already rendered as text (though of course lots can be done with the raw email).

This would serve as a way to quote "foreign" language source for use
embedded in guile code. You could quote make files and bash scripts and whatever.
I am working on something that incorporates very similar functionality, but I offer
this misleading mish-mosh so you won't die holding your breath :)

--proposed sketch for mime reader macro
Comment: part 1: reader macro usage example

(define mime-stuff #--"ad lib"
  garbage up to first delimiter is first header like rfc0822
--ad lib
first header

first content after blank line
no newline
--ad lib

content with default header
content ends with newline

--ad lib--
;; reader macro exits, normal reader resumes
)

--proposed sketch for mime reader macro
Comment: part 2: net effect of reader macro
Comment: just out of my head, totally untested, just to get the idea out

;; so the above reader macro incantation becomes equivalent
;; in result as if this had been written instead
(define mime-stuff '(\
("  garbage up to first delimiter is first header like rfc0822" . "") \
("first header" . "first content after blank line\nno newline") \
("" . "content with default header\ncontent ends with newline\n" ))
)

--proposed sketch for mime reader macro--

see (proposal) below for another way of tagging snippets ;-)

--8<---------------cut here---------------start------------->8---
Comment: The immeadiatly above typical guix snippet delimiter
 might be a legal MIME delimiter, even though it's not paired with the
 identical line with "--" appended to form the ending delimiter.
Comment: per rfc0822 the first lines after the starting delimiter constitute
 a header, terminated by a blank line. Header lines may be folded
 by breaking linear whitespace to a new line starting with at least one linear
 white space char, like these two Comments: lines.

So this line is the first content line, following the blank line.

This is still content, including blank lines, up to the final delimiter.

--8<----(proposal)-----------cut here---------------start------------->8---

This is nested and tagged as "proposal"
I think using the existing guix delimiter style, we could add tags and headers designed
to make extraction of cookbook (knowledge-base?) materials from posts and archives
automatable, generating nice snippets with footnoted urls and attributions etc
from email headers.

--8<----(proposal)-----------cut here---------------end--------------->8---

--8<----(code contribution: untested)-----------cut here---------------start------------->8---
Comment: bash invocation of guile.

guile -c $'(display "I am a code snippet")(newline)'

--8<----(code contribution: untested)-----------cut here---------------end--------------->8---

--8<----(possible markup for extraction)-----------cut here---------------start------------->8---
Comment: would X-Content-Type: dangerous?;
 be bad here? Not sure of the rules. Email should just encapsulate what I write, so
 we should be able to process as we wish with custom tools, IWT

┌────────────────────────────────────────────────┐
│         not sure boxes are a good idea         │
├────────────────────────────────────────────────┤
│ > We can rename the cookbook to make it fit ;) │
└────────────────────────────────────────────────┘
> 
+-------------------------------------------------------------------------+
|             not sure boxes are a good idea -- ascii safer?              |
+-------------------------------------------------------------------------+
| > It was a placeholder name, because we all agreed for some time that a |
| > “document” like that would be useful and that it shouldn’t be a wiki. |
| > Beyond that we really didn’t have any plans.  I went ahead and        |
| > started it to enable collaboration on this resource — whatever it may |
| > become eventually.                                                    |
+-------------------------------------------------------------------------+

--8<----(possible markup for extraction)-----------cut here---------------end--------------->8---

HTH trigger some ideas :)
Here comes the final delimiter, guix style, not real mime terminal delim,
recognized by having skipped paired  "--8<----(something ..." delims as well as plain.
--8<---------------cut here---------------end--------------->8---

-- 
Regards,
Bengt Richter

Re: On DSLs

[Ludovic Courtès]

Hi!

Julien Lepiller <julien@lepiller.eu> skribis:

> Could this discussion be saved in the cookbook for instance? I'd like to have this kind of discussion on the approach of guix and ideas behind it somewhere more accessible than the ML archive. Does it make sense?

I sort of see the cookbook as a very practical thing, so like Konrad I’m
not sure it’s the right place for high-level design discussions.

That said, the EDSL approach for Guix is discussed in two wonderful
papers:

  https://hal.inria.fr/hal-01580582/en
  https://hal.inria.fr/hal-00824004/en

:-)

Ludo’.

Re: On DSLs

[Konrad Hinsen]

Hi Simon,

> Out of curiosity, do you have examples about YAML or JSON with an
> interpreter running it?

I just encountered an example: GitHub actions.

Cheers,
  Konrad.

Re: On DSLs

[Konrad Hinsen]

Hi Ludo,

>> For better illustration, I'll try to rewrite my own manifests in the
>> way I'd like to be able to write them. That's probably more useful
>> than theory (a tough statement to make for a theoretician ;-)
>
> Agreed!

Just to be clear: I actually intend to implement some infrastructure to
make this happen, so this will take a while. One of my convictions after
decades in computational science is that ideas without implementations
should not be discussed extensively, so I try to live by my own
standards ;-)

> That’s not true.  In some cases, people write something that’s actually
> code (in YAML, in JSON, etc.) and there’s an interpreter running it.

I have been fortunate enough not to have seen such things yet!

> IOW, I think you can have a declarative _style_ in a full-blown
> language, like:

Definitely, and that's what I am aiming for. There remains the security
issue of malevolent power users sneaking in innocuous-looking
non-declarative code that non-expert users might run without suspicion.
But I'd say we can make a lot of progress by having declarative style
for all routine configuration data.

> This is just to say that we should not conflate the style and the
> language.  I think what we care about is supporting a declarative style,
> and making it expressive enough that people don’t feel the need to
> resort to complicated code.

Exactly!

Cheers,
  Konrad.

Re: On DSLs

[zimoun]

Hi Konrad,

On Tue, 3 Dec 2019 at 11:29, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> > IOW, I think you can have a declarative _style_ in a full-blown
> > language, like:
>
> Definitely, and that's what I am aiming for. There remains the security
> issue of malevolent power users sneaking in innocuous-looking
> non-declarative code that non-expert users might run without suspicion.
> But I'd say we can make a lot of progress by having declarative style
> for all routine configuration data.

I am sure you are not talking about parenthesis-style, but if you need
an example of DSL based on Guix with YAML-flavour, give a look at
these examples [1].

[1] https://guixwl.org/beyond-started


From my perspective, I find hard:

 - to know which function does what
 - where this function is located (module to import)
 - and sometimes this function is not exported, so I need to tweak
Guix, recompile, not-friendly time.

For example, I have in mind ‘package-environment-inputs’.

It will help to list what the features the manifest file should
contain. I mean, except the example [2] providing 'packages->manifest'
or 'specifications->manifest' or the other one in Inferiors [3] using
a specific commit, then it is a bit sparse.


[2] http://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#profile_002dmanifest
[3] http://guix.gnu.org/manual/devel/en/html_node/Inferiors.html#Inferiors



All the best,
simon

Re: Profiles/manifests-related command line interface enhancements

[Hartmut Goebel]

Hi,

just want to say, that I like the concept, Konrad and Ludo are
discussing. Please continue and make it into code.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Howdy!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> I think having ephemeral + persistent and declarative + imperative is
>> cool—I’d call that “flexible” rather than “messy”.  :-)
>
> I agree. What's messy is how the concepts map to commands. How many Guix
> users understand that profiles are persistent environments, or
> environments ephemeral profiles? And how many understand what "guix
> environment -r" does exactly, and how it should be used?  It took me a
> while to figure this out.
>
> What we have is two commands (package and environment) each designed for
> one main usage pattern, plus options to get something else. But even
> those options don't overlap completely in functionality. For example,
> how do I make a profile with the dependencies of a package?
>
> The current discussion started with adding more commands for different
> specific usage patterns. If we go that route the mess will become worse.

OK, I understand.

>> It’s great to have “guix install” as an entry point; I’m sure it allows
>> us to reach out to more people, and that matters too.  (I actually use
>> it, BTW, so it’s not an expert vs. newbie thing!)
>
> Me too :-) It's "guix package" that is the worst offender in my
> opinion. It does two distinct things: querying the package database and
> managing profiles. And now that we have "guix search" for queries,

We also have ‘guix show’, but there’s no standalone command equivalent
to ‘--list-installed’, ‘--list-available’, ‘--list-profiles’, and
‘--list-generations’.

> I'd like to see "guix package" go away, to be replaced by either "guix
> profile" for profile management, with as much overlap as possible in
> options with "guix environment", or by a single command that handles
> environments and profiles in a unified way.

To me it’s not entirely clear that a unified command would be easier to
use for newcomers.  For example, I’m not fond of “guix profile” as a
command name because that would mean users have to know what a “profile”
is before they’ve even installed their first package.

So I definitely agree we need to homogenize what ‘guix environment’ and
‘guix package’ currently provide, but I’m not sure where to go from
there.  Perhaps we need to discuss on concrete CLI proposals to have a
better idea?  Dunno!

I have an actionable wishlist though:  :-)

  1. Arrange so that ‘--ad-hoc’ becomes the default in ‘guix
     environment’, and add a ‘--devel’ (?) option for what’s currently
     the default.

     Tricky because of compatibility considerations, but would be an
     improvement, and goes in the direction of shortening the distance
     between ‘guix environment’ and ‘guix package’.

  2. Add that same ‘--devel’ option to ‘guix package’.

Thoughts?

>> I agree that sharing and publishing is important, and I think
>> manifests support that.
>
> They do, but not very well in my opinion. I think everything meant to be
> shared, published, and maintained should be accessible by name in a
> database. A channel, for example.
>
> Some ideas that could make this possible (and convenient):
>
>  - Better support for adding/managing channels at the user account
>    level. Users shouldn't have to edit Guile code (unless they want to).

You mean like ‘guix channel add URL’?  Are you looking at this from a UI
perspective?

Are you also suggesting a “GuixHub” (I already hate that name :-)) that
would collect channels?  In Strasbourg in June we discussed the
possibility to have a service to register free software channels, we
even came up with a name for that (was it “delta”?), and I think it
could be useful.  The “potluck” that Andy worked on a while back goes in
that direction too.

>  - Support for creating and managing channels without having to
>    descend to the file and repository level.

Not entirely sure what you mean.  Could you give examples of what you’d
type to create a channel, say?

>  - Support for something one could call super-packages or named
>    manifests. Named package groups that live in a channel (or a guix.scm
>    distributed with software external to Guix). Perhaps parametrizable
>    and/or containing configuration files, so you could put something
>    like (nginx #:port 8080) into a manifest file.

So this thing would be a profile + services?

>> What we’re now saying is “look, you can write a manifest, and then you
>> can have it under version-control and publish it and it’s all up to you
>> how you do that”; but you seem to suggest that we should offer a
>> higher-level, more integrated solution, is that correct?
>
> Exactly. Every functionality that requires end-users to manage
> Guile code will always be restricted to expert users. Manifest
> files may look simple and understandable for simple cases,
> but if you expect users to install manifests downloaded from
> someone else, they need to be able to inspect them and be sure
> that installing the manifest file won't delete their user account.
> And that means they have to understand a lot about Guile.

OK, I see what you mean now regarding security.

(I don’t really agree with the “expert user” bit but we discussed that
in another fiber of this thread. :-))

Thank!

Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Ludovic Courtès <ludo@gnu.org> writes:

> To me it’s not entirely clear that a unified command would be easier to
> use for newcomers.  For example, I’m not fond of “guix profile” as a

Indeed. It's also not clear to me if a single command line package
for beginners and power users is the best way to go. Maybe we should
have two levels of commands, with tutorials using only the simpler one.

> So I definitely agree we need to homogenize what ‘guix environment’ and
> ‘guix package’ currently provide, but I’m not sure where to go from
> there.  Perhaps we need to discuss on concrete CLI proposals to have a
> better idea?  Dunno!

If enough people are willing to work on this (beginners included!), we
could run a "CLI working group" that plays with alternatives to the
current CLI, implemented as a separate Guile package so it won't perturb
business as usual. That requires a few people working on implementation,
and a much larger group willing to test-drive the design(s) and provide
feedback.

> I have an actionable wishlist though:  :-)
>
>   1. Arrange so that ‘--ad-hoc’ becomes the default in ‘guix
>      environment’, and add a ‘--devel’ (?) option for what’s currently
>      the default.

That sounds good to me, with

>   2. Add that same ‘--devel’ option to ‘guix package’.

being the obvious (and less controversial) complement.

I'd probably prefer the more descriptive –-inputs-of for the option name.

>      Tricky because of compatibility considerations, but would be an

That's the eternal question!

>>  - Better support for adding/managing channels at the user account
>>    level. Users shouldn't have to edit Guile code (unless they want to).
>
> You mean like ‘guix channel add URL’?  Are you looking at this from a UI
> perspective?

Yes.

> Are you also suggesting a “GuixHub” (I already hate that name :-)) that

That's not something I personally care about (I am all for the
distributed Web), but if there's a demand, why not.

>>  - Support for creating and managing channels without having to
>>    descend to the file and repository level.
>
> Not entirely sure what you mean.  Could you give examples of what you’d
> type to create a channel, say?

For management, something like git's remotes:

 guix channel add my-channel URL
 guix channel remove my-channel
 guix channel describe my-channel
 guix channel show-all

For channel creation, something like git's init:

 guix make-channel directory-name

That would create the directory name, run "git init", add minimal code
making a template (something like the "hello" package), and perhaps a
file with a summary of instructions.

>>  - Support for something one could call super-packages or named
>>    manifests. Named package groups that live in a channel (or a guix.scm
>>    distributed with software external to Guix). Perhaps parametrizable
>>    and/or containing configuration files, so you could put something
>>    like (nginx #:port 8080) into a manifest file.
>
> So this thing would be a profile + services?

A bit less than a service, because no process would be started or
managed. I might want to run that nginx instance manually and/or
under my user account. Also on a foreign distribution that doesn't
know about Guix services. Think of something like a Jupyter server.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[zimoun]

Hi,

On Sun, 17 Nov 2019 at 12:30, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> If enough people are willing to work on this (beginners included!), we
> could run a "CLI working group" that plays with alternatives to the
> current CLI, implemented as a separate Guile package so it won't perturb
> business as usual. That requires a few people working on implementation,
> and a much larger group willing to test-drive the design(s) and provide
> feedback.

Do we open a Bug to track the discussion? Maybe first by synthesizing
what have already been proposed?
Or a Gitlab/GitHub/Name-it project?


> > Are you also suggesting a “GuixHub” (I already hate that name :-)) that
>
> That's not something I personally care about (I am all for the
> distributed Web), but if there's a demand, why not.

Do you propose that my channel repo lives on say Gitlab and it will be
registered on say GuixHub?
And GuixHub just eases the searching part.
Or something else?



>  guix channel add my-channel URL
>  guix channel remove my-channel
>  guix channel describe my-channel
>  guix channel show-all
>
> For channel creation, something like git's init:
>
>  guix make-channel directory-name

guix channel init directory-name





All the best,
simon

Re: Profiles/manifests-related command line interface enhancements

[Andreas Enge]

Hello,

I am reading up on old threads during the holiday season...

On Sun, Nov 17, 2019 at 12:30:11PM +0100, Konrad Hinsen wrote:
> If enough people are willing to work on this (beginners included!), we
> could run a "CLI working group" that plays with alternatives to the
> current CLI, implemented as a separate Guile package so it won't perturb
> business as usual. That requires a few people working on implementation,
> and a much larger group willing to test-drive the design(s) and provide
> feedback.

Could we reach out to some experienced UI designers? Is there actually
any such thing as "CLUI designers"?

Andreas

Re: Profiles/manifests-related command line interface enhancements

[Pjotr Prins]

On Sun, Dec 22, 2019 at 08:40:25PM +0100, Andreas Enge wrote:
> Could we reach out to some experienced UI designers? Is there actually
> any such thing as "CLUI designers"?

These things grow over time. I think with our current knowledge we
could put it to the vote. Also some of the default behaviour may be up
for discussion (such as using guixbuild user by default for the
daemon).  One way to handle it is to have a new CLUI and a guix-legacy
CLI to handle the old one for the time being noting that a switch is
going obsolete.

A phased approach, in other words.

Pj.

Re: Profiles/manifests-related command line interface enhancements

[zimoun]

On Sat, 16 Nov 2019 at 23:27, Ludovic Courtès <ludo@gnu.org> wrote:

> > Me too :-) It's "guix package" that is the worst offender in my
> > opinion. It does two distinct things: querying the package database and
> > managing profiles. And now that we have "guix search" for queries,
>
> We also have ‘guix show’, but there’s no standalone command equivalent
> to ‘--list-installed’, ‘--list-available’, ‘--list-profiles’, and
> ‘--list-generations’.

IMHO, we should have a generic alias system "à la" Git or an UI
modifier via channel.

It is not reasonable to duplicate all the commands just for
convenience. Because convenience mainly depends on personal
preference.

The same way that we add "guix workflow" or "guix home", a forward
move should to package some "guix flavor", e.g., add the command "guix
profile" via channel (or package). Then simply "guix install
guix-profile" will add the very command.
Konrad already proposed similar.


>   1. Arrange so that ‘--ad-hoc’ becomes the default in ‘guix
>      environment’, and add a ‘--devel’ (?) option for what’s currently
>      the default.

I agree. David proposed that in "The future of 'guix environment'".

https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html

And as Konrad mentioned, I prefer "--inputs-of" instead of "--devel".


>      Tricky because of compatibility considerations, but would be an
>      improvement, and goes in the direction of shortening the distance
>      between ‘guix environment’ and ‘guix package’.

Let bump the version and break compatibility. :-p


>   2. Add that same ‘--devel’ option to ‘guix package’.

And to be consistent, why not add a similar option to manifest (via
"package-environment-inputs" exporting as "<option-name>" same than
"--option-name"; I opt for option-name equal inputs-of instead of
devel :-))



All the best,
simon

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 16 Nov 2019 at 23:27, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> > Me too :-) It's "guix package" that is the worst offender in my
>> > opinion. It does two distinct things: querying the package database and
>> > managing profiles. And now that we have "guix search" for queries,
>>
>> We also have ‘guix show’, but there’s no standalone command equivalent
>> to ‘--list-installed’, ‘--list-available’, ‘--list-profiles’, and
>> ‘--list-generations’.
>
> IMHO, we should have a generic alias system "à la" Git or an UI
> modifier via channel.

I agree, that’d be nice.

>>   1. Arrange so that ‘--ad-hoc’ becomes the default in ‘guix
>>      environment’, and add a ‘--devel’ (?) option for what’s currently
>>      the default.
>
> I agree. David proposed that in "The future of 'guix environment'".
>
> https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html
>
> And as Konrad mentioned, I prefer "--inputs-of" instead of "--devel".

OK, fine with me!  :-)

>>      Tricky because of compatibility considerations, but would be an
>>      improvement, and goes in the direction of shortening the distance
>>      between ‘guix environment’ and ‘guix package’.
>
> Let bump the version and break compatibility. :-p

Heh, I think we owe our users a smooth transition.

>>   2. Add that same ‘--devel’ option to ‘guix package’.
>
> And to be consistent, why not add a similar option to manifest (via
> "package-environment-inputs" exporting as "<option-name>" same than
> "--option-name"; I opt for option-name equal inputs-of instead of
> devel :-))

You mean we’d have a way to create a manifest and pass it the same
options as on the command line, as strings?

Thanks,
Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Another way is:
>>
>>   eval `guix package --search-paths=prefix`
>>
>> or similar.
>
> Note that this suffers from the shell compatibility issue, e.g. it won't
> work with Fish / Eshell.

That can be fixed, as proposed in
<https://issues.guix.gnu.org/issue/36021>.

>> Another one is:
>>
>>   guix environment …
>
> This is a bit different since it spawns a subshell.

Not if you do “-- some command”.

>>> - All commands that accept manifests should be able to compose manifests
>>>   by passing multiple =--manifest FILE= options.
>>
>> Do you have a use case?  :-)
>>
>> Sounds like something quite easy to do but I’ve never heard anyone
>> request it before so I’m curious.
>
> I'm actually surprised you find it surprising! :)
> I can think of Simon, maybe Konrad(?) and myself who mentioned it
> before.
>
> Examples:
>
> - `guix pack` can only pack ` manifest at a time.  Impossible to combine
>   2 manifests.
>
> - Same for guix weather if you want to avoid calling guix weather
>   multiple times in a row.
>
> - `guix environment` can work around this limitation by calling the
>   manifest from the generated subshells.  It's not super convenient, and
>   it does not work for scripts.  For instance, we can't do this at the moment:
>
>    guix environment -m manifest1 -m manifest2 -- my command line...
>
> I'm sure we can come up with tons of examples :)

I understand what it is we cannot do; I just hadn’t felt the need for
it.  But yeah, let’s do that!  :-)

>> Note that “ad-hoc” profiles (imperatively-managed profiles) contain
>> provenance info in their ‘manifest’ file, so it’s not black and white.
>
> I actually never noticed this :p

It’s sort of a “hidden” feature waiting to be put to good use.  :-)

>>> Last, one that's at the Scheme level:
>>>
>>> - A Scheme function to create a manifest for the necessary inputs of a
>>>   package, like =guix environment PACKAGE= does.  (Maybe it's already possible?)
>>
>> Like ‘specifications->manifest’?
>
> Can specifications->manifest do this?
>
> Concrete example:  You want a profile to work on IceCat, so you need a
> manifest that has all the necessary inputs required to work with Icecat.
> How can we automate the generation of the manifest?

Ah, that no.  But (guix scripts environment) has
‘package-environment-inputs’, which does exactly that, and I guess we
could expose it and document it.

More generally, ‘guix environment’ was initially designed in a quite
monolithic fashion, and I think we could expose its functionality in
sizable bites.  For instance, Dave suggested long ago exposing an
<environment> data type, and that’s one thing that could be very useful.

Thanks for starting this insightful brainstorming session!  :-)

Ludo’.

Re: Profiles/manifests-related command line interface enhancements

[zimoun]

Hi Ludo,

On Wed, 6 Nov 2019 at 17:42, Ludovic Courtès <ludo@gnu.org> wrote:

> >>> - A Scheme function to create a manifest for the necessary inputs of a
> >>>   package, like =guix environment PACKAGE= does.  (Maybe it's already possible?)
> >>
> >> Like ‘specifications->manifest’?
> >
> > Can specifications->manifest do this?
> >
> > Concrete example:  You want a profile to work on IceCat, so you need a
> > manifest that has all the necessary inputs required to work with Icecat.
> > How can we automate the generation of the manifest?
>
> Ah, that no.  But (guix scripts environment) has
> ‘package-environment-inputs’, which does exactly that, and I guess we
> could expose it and document it.

Maybe I am doing wrong but I get:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix package -p /tmp/guix-dev -m ~/tmp/manif-dep.scm

installing new manifest from '/home/simon/tmp/manif-dep.scm' with 53 entries
guix package: error: profile contains conflicting entries for gzip
guix package: error:   first entry: gzip@1.10
/gnu/store/py3k9zla9fj3z7430v4crqj5pyrsd3qj-gzip-1.10
guix package: error:   second entry: gzip@1.10
/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10
hint: You cannot have two different versions or variants of `gzip' in
the same profile.
--8<---------------cut here---------------end--------------->8---

where I tweak guix/scripts/environment.scm to export
`package-environment-inputs' and ~/tmp/manif-dep.scm contains:

--8<---------------cut here---------------start------------->8---
(use-modules (guix)
             (guix profiles)
             (guix scripts environment)
             (gnu packages package-management))

(make-manifest (package-environment-inputs guix))
--8<---------------cut here---------------end--------------->8---


> More generally, ‘guix environment’ was initially designed in a quite
> monolithic fashion, and I think we could expose its functionality in
> sizable bites.  For instance, Dave suggested long ago exposing an
> <environment> data type, and that’s one thing that could be very useful.

What do you mean by an <environment> data type?



All the best,
simon

Package inputs in manifests

[Ludovic Courtès]

Hi Simon!

zimoun <zimon.toutoune@gmail.com> skribis:

>> Ah, that no.  But (guix scripts environment) has
>> ‘package-environment-inputs’, which does exactly that, and I guess we
>> could expose it and document it.
>
> Maybe I am doing wrong but I get:
>
> $ ./pre-inst-env guix package -p /tmp/guix-dev -m ~/tmp/manif-dep.scm
>
> installing new manifest from '/home/simon/tmp/manif-dep.scm' with 53 entries
> guix package: error: profile contains conflicting entries for gzip
> guix package: error:   first entry: gzip@1.10
> /gnu/store/py3k9zla9fj3z7430v4crqj5pyrsd3qj-gzip-1.10
> guix package: error:   second entry: gzip@1.10
> /gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10
> hint: You cannot have two different versions or variants of `gzip' in
> the same profile.
>
>
> where I tweak guix/scripts/environment.scm to export
> `package-environment-inputs' and ~/tmp/manif-dep.scm contains:
>
> (use-modules (guix)
>              (guix profiles)
>              (guix scripts environment)
>              (gnu packages package-management))
>
> (make-manifest (package-environment-inputs guix))

You’re doing it right!  It turns out that two different variants of
“gzip” appear: one in ‘%final-inputs’ (the implicit inputs of
‘gnu-build-system’) and another one somewhere else, hence the collision.

The solution is to pass ‘--allow-collisions’, which is what ‘guix
environment’ does by default precisely for that reason (see commit
afd06f605bf88a796acefc7ed598b43879346a6b).

We could check exactly why we end up with several ‘gzip’ instances, but
overall I think this kind of thing is hard to avoid for package inputs.

Thanks,
Ludo’.

Re: Package inputs in manifests

[Bengt Richter]

Hi Ludo, Guix,

On +2019-11-17 11:35:07 +0100, Ludovic Courtès wrote:
> Hi Simon!
> 
> zimoun <zimon.toutoune@gmail.com> skribis:
> 
> >> Ah, that no.  But (guix scripts environment) has
> >> ‘package-environment-inputs’, which does exactly that, and I guess we
> >> could expose it and document it.
> >
> > Maybe I am doing wrong but I get:
> >
> > $ ./pre-inst-env guix package -p /tmp/guix-dev -m ~/tmp/manif-dep.scm
> >
> > installing new manifest from '/home/simon/tmp/manif-dep.scm' with 53 entries
> > guix package: error: profile contains conflicting entries for gzip
> > guix package: error:   first entry: gzip@1.10
> > /gnu/store/py3k9zla9fj3z7430v4crqj5pyrsd3qj-gzip-1.10
> > guix package: error:   second entry: gzip@1.10
> > /gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10
> > hint: You cannot have two different versions or variants of `gzip' in
> > the same profile.
> >
> >
> > where I tweak guix/scripts/environment.scm to export
> > `package-environment-inputs' and ~/tmp/manif-dep.scm contains:
> >
> > (use-modules (guix)
> >              (guix profiles)
> >              (guix scripts environment)
> >              (gnu packages package-management))
> >
> > (make-manifest (package-environment-inputs guix))
> 
> You’re doing it right!  It turns out that two different variants of
> “gzip” appear: one in ‘%final-inputs’ (the implicit inputs of
> ‘gnu-build-system’) and another one somewhere else, hence the collision.
> 
> The solution is to pass ‘--allow-collisions’, which is what ‘guix
> environment’ does by default precisely for that reason (see commit
> afd06f605bf88a796acefc7ed598b43879346a6b).
> 
> We could check exactly why we end up with several ‘gzip’ instances, but
> overall I think this kind of thing is hard to avoid for package inputs.
> 
> Thanks,
> Ludo’.
> 

Can "collisions" be collisions even if the targets are bit-identical?

E.g.

find /gnu -name '*gzip'|xargs sha1sum|gxsnip
--8<---------------cut here---------------start------------->8---
3a69ba705f2a8674fe6f0dfbdbf47de95bf72438  /gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9/bin/gzip
3a69ba705f2a8674fe6f0dfbdbf47de95bf72438  /gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9/bin/gzip
c944319913ed9006b819abceda783ceee0985972  /gnu/store/c6z3v1s9ky7h3wsc6da470s35iiciidd-profile/bin/gzip
c944319913ed9006b819abceda783ceee0985972  /gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin/gzip
c944319913ed9006b819abceda783ceee0985972  /gnu/store/py3k9zla9fj3z7430v4crqj5pyrsd3qj-gzip-1.10/bin/gzip
--8<---------------cut here---------------end--------------->8---

It seems like it would be possible to automate an item-specific --allow-collisions
where the "collision" is really an aliasing?

(IOW, why use different names for the same thing, except for local convenience abbreviations?)

I see the de-duping gc does when it converts to hard links, and wonder there too.
What am I missing?

RFE:
(request for enlightenment :)
Exactly how are the /gnu/store prefixes computed? What _exactly_ is/are the input(s) that get hashed??

TIA
-- 
Regards,
Bengt Richter

Re: Package inputs in manifests

[zimoun]

Hi,

On Mon, 18 Nov 2019 at 00:12, Bengt Richter <bokr@bokr.com> wrote:

> Can "collisions" be collisions even if the targets are bit-identical?

I think yes.

--8<---------------cut here---------------end--------------->8---
$ find -P /gnu/store  -maxdepth 1 -name "*gzip-1.9" -type d -print
/gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9
/gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9
/gnu/store/1qgxnwxn7kqg6hmqc5hajc9g2z2bbaik-gzip-1.9
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------end--------------->8---
$ diff -r --no-dereference
/gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9
/gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9
diff -r --no-dereference
/gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9/bin/gunzip
/gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9/bin/gunzip
57c57
< exec /gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9/bin/gzip -d "$@"
---
> exec /gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9/bin/gzip -d "$@"
diff -r --no-dereference
/gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9/bin/uncompress
/gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9/bin/uncompress
57c57
< exec /gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9/bin/gzip -d "$@"
---
> exec /gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9/bin/gzip -d "$@"

--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------end--------------->8---
$ guix gc --requisites
/gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9 | sort
/gnu/store/02k245xy33cvcnr8vm3lagm9zmb1s2wa-grep-3.1
/gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib
/gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28
/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23
/gnu/store/zzakf905mzla4csi1dn9qpcwmgbxj29b-bash-static-4.4.23
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------end--------------->8---
$ guix gc --requisites
/gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9 | sort
/gnu/store/02k245xy33cvcnr8vm3lagm9zmb1s2wa-grep-3.1
/gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28
/gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9
/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23
/gnu/store/zzakf905mzla4csi1dn9qpcwmgbxj29b-bash-static-4.4.23
--8<---------------cut here---------------end--------------->8---


But the derivations differs.


--8<---------------cut here---------------end--------------->8---
$ guix gc --derivers /gnu/store/h0c398zan9ibhk4w0c944vp5pwgzkfpd-gzip-1.9 | sort
/gnu/store/3c9spvq9cgack8g0sznr6mv4b2vbag8v-gzip-1.9.drv
/gnu/store/534rm9gn29d7dd88wjkvmy9qf9w6j3b6-gzip-1.9.drv
/gnu/store/5gjql669pszp6wjg80wx0l7b6bkdirlj-gzip-1.9.drv
/gnu/store/5l2d67j1h0riax42a0kljnn6ghg4qm3s-gzip-1.9.drv
/gnu/store/xpv9bnrfvz5z6xwh5ypziz6waxzfa9jk-gzip-1.9.drv
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------end--------------->8---
$ guix gc --derivers /gnu/store/q09sy224qnxrp982z4xfaxi19721mjx8-gzip-1.9 | sort
/gnu/store/c0ngi4rm0avjlp3nb8j4yvmdlf598vr2-gzip-1.9.drv
/gnu/store/ib28wjw95572psh6z9nm9a6rf05ndcqw-gzip-1.9.drv
/gnu/store/jbnidakwvhr84rp8p8icdg1lnczjvi6w-gzip-1.9.drv
/gnu/store/rmqarqss73akz9yaad4a02zdvp62ijx2-gzip-1.9.drv
/gnu/store/xwrwib0yi7d2n1fp1zaxydw5kn704qjd-gzip-1.9.drv
--8<---------------cut here---------------end--------------->8---



I do not know, maybe it is about grafts.


All the best,
simon

Re: Package inputs in manifests

[Ludovic Courtès]

Hi,

Bengt Richter <bokr@bokr.com> skribis:

> Can "collisions" be collisions even if the targets are bit-identical?

Collisions are when the same package appears several times with
different version strings, or when the same package/version appears
several times with a different store item.

Ludo’.

Re: Package inputs in manifests

[Bengt Richter]

Hi Ludo,

On +2019-11-23 15:05:49 +0100, Ludovic Courtès wrote:
> Hi,
> 
> Bengt Richter <bokr@bokr.com> skribis:
> 
> > Can "collisions" be collisions even if the targets are bit-identical?
> 
> Collisions are when the same package appears several times with
> different version strings, or when the same package/version appears
> several times with a different store item.
> 
> Ludo’.

In this case, the "Inode: 1966255" entries below say the gzips are not different store items,
so what am I missing about "version strings?" :)

Why would there be different prefixes? Are transient-state link counts somehow
entering into prefix hash calculations? But that's directory state, isn't it? ...

┌─────────────────────────────────────────────────────────────────────────────────────┐
│ So again, what exactly goes into computing those /gnu/store/.../file prefixes?? ;-) │
└─────────────────────────────────────────────────────────────────────────────────────┘

find /gnu -name '*gzip'|grep gzip-1.10
--8<---------------cut here---------------start------------->8---
/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin/gzip
/gnu/store/py3k9zla9fj3z7430v4crqj5pyrsd3qj-gzip-1.10/bin/gzip
--8<---------------cut here---------------end--------------->8---


find /gnu -name '*gzip'|grep gzip-1.10|xargs file  # (manually wrapped after gzip: and before /gnu)
--8<---------------cut here---------------start------------->8---
/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin/gzip:
    ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter
    /gnu/store/ahqgl4h89xqj695lgqvsaf6zh2nhy4pj-glibc-2.29/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
/gnu/store/py3k9zla9fj3z7430v4crqj5pyrsd3qj-gzip-1.10/bin/gzip:
    ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter
    /gnu/store/ahqgl4h89xqj695lgqvsaf6zh2nhy4pj-glibc-2.29/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
--8<---------------cut here---------------end--------------->8---

find /gnu -name '*gzip'|grep gzip-1.10|xargs stat
--8<---------------cut here---------------start------------->8---
  File: /gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin/gzip
  Size: 95928           Blocks: 192        IO Block: 4096   regular file
Device: 10304h/66308d   Inode: 1966255     Links: 3
Access: (0555/-r-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-23 19:48:56.570075367 -0800
Modify: 1969-12-31 16:00:01.000000000 -0800
Change: 2019-10-08 18:17:58.417674370 -0700
 Birth: -
  File: /gnu/store/py3k9zla9fj3z7430v4crqj5pyrsd3qj-gzip-1.10/bin/gzip
  Size: 95928           Blocks: 192        IO Block: 4096   regular file
Device: 10304h/66308d   Inode: 1966255     Links: 3
Access: (0555/-r-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-23 19:48:56.570075367 -0800
Modify: 1969-12-31 16:00:01.000000000 -0800
Change: 2019-10-08 18:17:58.417674370 -0700
 Birth: -
--8<---------------cut here---------------end--------------->8---

-- 
Regards,
Bengt Richter

Re: Package inputs in manifests

[Timothy Sample]

Hi Bengt,

Bengt Richter <bokr@bokr.com> writes:

> On +2019-11-23 15:05:49 +0100, Ludovic Courtès wrote:
>> 
>> Bengt Richter <bokr@bokr.com> skribis:
>> 
>> > Can "collisions" be collisions even if the targets are bit-identical?
>> 
>> Collisions are when the same package appears several times with
>> different version strings, or when the same package/version appears
>> several times with a different store item.
>
> In this case, the "Inode: 1966255" entries below say the gzips are not
> different store items,
> so what am I missing about "version strings?" :)
>
> Why would there be different prefixes? Are transient-state link counts somehow
> entering into prefix hash calculations? But that's directory state, isn't it? ...
>
> ┌─────────────────────────────────────────────────────────────────────────────────────┐
> │ So again, what exactly goes into computing those /gnu/store/.../file prefixes?? ;-) │
> └─────────────────────────────────────────────────────────────────────────────────────┘

A store prefix is the hash of all the inputs used to build a store item.
It has nothing to do with the contents of the output [1].  Two store
items may have identical contents but different prefixes.

Say we have a package A.  If we build A we might get a file called
“/gnu/store/abc123-a” with contents “this is a”.  Now imagine that we
make a change to A that doesn’t change its output.  If we build this
changed package A′, we would get a file called “/gnu/store/xyz456-a”
(note the different prefix) but its contents would still be “this is a”.

What’s more is that Guix will notice that these two files are the same
and deduplicate them through hard linking.  From the manual:

    By default, files added to the store are automatically
    “deduplicated”: if a newly added file is identical to another one
    found in the store, the daemon makes the new file a hard link to the
    other file.

This explains why you have only one inode.

As an aside, this is called the “extensional” model of functional
package management.  There is also an “intensional” model where the
prefixes are hashes of the content (i.e., everything in the store is
content addressable).  It has some neat properties, but also some
complications.  This is all discussed in Eelco Dolstra’s Ph.D. thesis
which introduces Nix: <https://nixos.org/~eelco/pubs/phd-thesis.pdf>.

Hope that helps!

[1] There are “fixed-output” derivations where the prefix is the hash of
the contents.  They are used for things like a source archive, where you
don’t care how you get it, as long as you have the right one.


-- Tim

Re: Package inputs in manifests

[Bengt Richter]

Hi Timothy,

On +2019-11-24 02:17:20 -0500, Timothy Sample wrote:
> Hi Bengt,
> 
> Bengt Richter <bokr@bokr.com> writes:
> 
[...]
> > ┌─────────────────────────────────────────────────────────────────────────────────────┐
> > │ So again, what exactly goes into computing those /gnu/store/.../file prefixes?? ;-) │
> > └─────────────────────────────────────────────────────────────────────────────────────┘
> 
> A store prefix is the hash of all the inputs used to build a store item.
> It has nothing to do with the contents of the output [1].  Two store
> items may have identical contents but different prefixes.
> 
> Say we have a package A.  If we build A we might get a file called
> “/gnu/store/abc123-a” with contents “this is a”.  Now imagine that we
> make a change to A that doesn’t change its output.  If we build this
> changed package A′, we would get a file called “/gnu/store/xyz456-a”
> (note the different prefix) but its contents would still be “this is a”.
> 
> What’s more is that Guix will notice that these two files are the same
> and deduplicate them through hard linking.  From the manual:
> 
>     By default, files added to the store are automatically
>     “deduplicated”: if a newly added file is identical to another one
>     found in the store, the daemon makes the new file a hard link to the
>     other file.
> 
> This explains why you have only one inode.
> 
> As an aside, this is called the “extensional” model of functional
> package management.  There is also an “intensional” model where the
> prefixes are hashes of the content (i.e., everything in the store is
> content addressable).  It has some neat properties, but also some
> complications.  This is all discussed in Eelco Dolstra’s Ph.D. thesis
> which introduces Nix: <https://nixos.org/~eelco/pubs/phd-thesis.pdf>.
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
			 a really good read, glad you added your "aside"
> 
> Hope that helps!

Very much so!

My mistaken assumption was that the current guix was an implementation
of an "intensional" model whose implementation I just wasn't understanding ;-p

Crystal lear now, thanks :) Preconceptions die hard ;-/
 
> 
> [1] There are “fixed-output” derivations where the prefix is the hash of
> the contents.  They are used for things like a source archive, where you
> don’t care how you get it, as long as you have the right one.

I found that explored and explained in the thesis too.

I hope there's an incrementally growing index of guix reference docs
like that, with urls and synopses in the citations.

Your post here (before I cluttered it up :) could be a model entry.
> 
> 
> -- Tim

Thanks again.
-- 
Regards,
Bengt Richter

Re: Package inputs in manifests

[zimoun]

Hi,

On Sun, 17 Nov 2019 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:

> You’re doing it right!  It turns out that two different variants of
> “gzip” appear: one in ‘%final-inputs’ (the implicit inputs of
> ‘gnu-build-system’) and another one somewhere else, hence the collision.
>
> The solution is to pass ‘--allow-collisions’, which is what ‘guix
> environment’ does by default precisely for that reason (see commit
> afd06f605bf88a796acefc7ed598b43879346a6b).

Nice!
I would be able to find myself! ;-)

However, in the commit afd06f605bf88a796acefc7ed598b43879346a6b, the
behaviour appears to me inconsistent. On one hand, the default is #f
in guix/profiles.scm or in guix/scripts/package.scm, on the other hand
the default is #t in guix/scripts/environment.scm. Does a rationale
exist for that?

Why not turn on #t all the defaults and instead have the option
"--no-allow-collisions". Keeping the option "--allow-collisions" for
compatibility.


> We could check exactly why we end up with several ‘gzip’ instances, but
> overall I think this kind of thing is hard to avoid for package inputs.

Yes, we could investigate from where comes this collisions.


Thank you for the explanations and the pointers.


All the best,
simon