[RFC] A simple draft for channels

[RFC] A simple draft for channels

[Ricardo Wurmus]

Hi Guix,

I’d like to retire GUIX_PACKAGE_PATH as the main way to get third-party
packages, because we can’t really keep track of packages that were added
or redefined in this way.  I want to replace it with slightly more
formal “channels”.

As a first implementation of channels I’d just like to have a channel
description file that records at least the following things:

* the channel name (all lower case, no spaces)
* a URL from where package definitions can be loaded (initially, this
  can be restricted to git repositories)

Optional fields:

* a description of the channel

* a URL from where substitutes for the packages can be obtained (this
  will be backed by “guix publish”)

* a mail address or URL to contact the maintainers of the channel, or to
  view the status of the channel

* the Guix git commit that was used when this channel was last
  updated.  This is useful when Guix upstream breaks the ABI or moves
  packages between modules.

On the Guix side we’d need to add the “guix channel” command, which
allows for adding, removing, updating, and downgrading channels.  Adding
a channel means fetching the channel description from a URL and storing
state in ~/.config/guix/channels/, and fetching the git repo it
specifies (just like what guix pull does: it’s a git frontend).  It also
authorizes the the substitute server’s public key.

Internally, it’s just like GUIX_PACKAGE_PATH in that the repos are used
to extend the modules that Guix uses.  Unlike GUIX_PACKAGE_PATH,
however, we now have a way to record the complete state of Guix,
including any extensions: the version of Guix and all active channels
with their versions.  We would also have a way to fetch substitutes from
channels without having to “globally” enable new substitute servers and
authorize their keys.  (Is this safe?  Can we have per-user extensions
to the set of public keys that are accepted?)

Downsides: Guix has no stable ABI, so channels that are not up-to-date
will break with newer versions of Guix.  Moving around packages to
different modules might break channels.  That’s okay.  It’s still an
improvement over plain GUIX_PACKAGE_PATH.

I don’t think it has to be more complicated than that.  What do you
think?

--
Ricardo

Re: [RFC] A simple draft for channels

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 3439 bytes --]

2018-01-19 9:24 GMT+01:00 Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>:

> Hi Guix,
>
> I’d like to retire GUIX_PACKAGE_PATH as the main way to get third-party
> packages, because we can’t really keep track of packages that were added
> or redefined in this way.  I want to replace it with slightly more
> formal “channels”.
>
> As a first implementation of channels I’d just like to have a channel
> description file that records at least the following things:
>
> * the channel name (all lower case, no spaces)
> * a URL from where package definitions can be loaded (initially, this
>   can be restricted to git repositories)
>
> Optional fields:
>
> * a description of the channel
>
> * a URL from where substitutes for the packages can be obtained (this
>   will be backed by “guix publish”)
>
> * a mail address or URL to contact the maintainers of the channel, or to
>   view the status of the channel
>
> * the Guix git commit that was used when this channel was last
>   updated.  This is useful when Guix upstream breaks the ABI or moves
>   packages between modules.
>
> On the Guix side we’d need to add the “guix channel” command, which
> allows for adding, removing, updating, and downgrading channels.  Adding
> a channel means fetching the channel description from a URL and storing
> state in ~/.config/guix/channels/, and fetching the git repo it
> specifies (just like what guix pull does: it’s a git frontend).  It also
> authorizes the the substitute server’s public key.
>
> Internally, it’s just like GUIX_PACKAGE_PATH in that the repos are used
> to extend the modules that Guix uses.  Unlike GUIX_PACKAGE_PATH,
> however, we now have a way to record the complete state of Guix,
> including any extensions: the version of Guix and all active channels
> with their versions.  We would also have a way to fetch substitutes from
> channels without having to “globally” enable new substitute servers and

authorize their keys.

[...]

> (Is this safe?  Can we have per-user extensions
> to the set of public keys that are accepted?)
>
> I am not sure, but I think we need to be able to ensure that these 'new'
substitute servers
will only be used to get substitutes for the derivations in that specific
channel.

I am not sure how easy it will be to make sure this will be the case, but I
guess we do not
want to give any user-defined the possibility to 'overwrite' substitutes
for existing derivations
from system-trusted substitute servers.

Downsides: Guix has no stable ABI, so channels that are not up-to-date
> will break with newer versions of Guix.  Moving around packages to
> different modules might break channels.  That’s okay.  It’s still an
> improvement over plain GUIX_PACKAGE_PATH.
>
> We might be able to mitigate this by using by using Semantic Versioning [1]
on a best-effort basis. Perhaps (some) changes to the abi could even be
picked up and warned
about by a tool not unlike the one used to generate the package listings
for new releases. I am
thinking of things like:
- A package was renamed (so the previous named version no longer exists)
- A package was moved



> I don’t think it has to be more complicated than that.  What do you
> think?
>
> --
> Ricardo
>
> In general, I like it and would love to play around with this soon.

- Jelle

[1]: https://semver.org/

[-- Attachment #2: Type: text/html, Size: 4708 bytes --]

Re: [RFC] A simple draft for channels

[Pjotr Prins]

On Fri, Jan 19, 2018 at 09:24:27AM +0100, Ricardo Wurmus wrote:
> Hi Guix,
> 
> I’d like to retire GUIX_PACKAGE_PATH as the main way to get third-party
> packages, because we can’t really keep track of packages that were added
> or redefined in this way.  I want to replace it with slightly more
> formal “channels”.
> 
> As a first implementation of channels I’d just like to have a channel
> description file that records at least the following things:
> 
> * the channel name (all lower case, no spaces)
> * a URL from where package definitions can be loaded (initially, this
>   can be restricted to git repositories)

Name and URL can be one. I would think the URL+branch or git tag is
unique for the channel.

A channel can still be named by a user, but it need not be specified.
Like git does with branches and remote repos. E.g.

  guix channel add bioinfo1 git-URL

So the user choses the name.

> Optional fields:
> 
> * a description of the channel
> 
> * a URL from where substitutes for the packages can be obtained (this
>   will be backed by “guix publish”)
> 
> * a mail address or URL to contact the maintainers of the channel, or to
>   view the status of the channel
> 
> * the Guix git commit that was used when this channel was last
>   updated.  This is useful when Guix upstream breaks the ABI or moves
>   packages between modules.

The last is critical. One thing we need to think through is that we
don't want to build the full guix repo since it slow and will take
more and more time. Would it be possible to package the binary .go
files and use those on the go since they are fixated anyway? I presume
they are system independent (but not arch independent - but I think
different arch will have different channels anyway).

The package definition can be hosted inside the git repo as a
guix-channel.scm file. When the user adds a channel we can fetch this
info.

> On the Guix side we’d need to add the “guix channel” command, which
> allows for adding, removing, updating, and downgrading channels.  Adding
> a channel means fetching the channel description from a URL and storing
> state in ~/.config/guix/channels/, and fetching the git repo it
> specifies (just like what guix pull does: it’s a git frontend).  It also
> authorizes the the substitute server’s public key.

That would be wonderful. Maybe make it an --auto-authorize switch for
those who choose to live dangerously.

> Internally, it’s just like GUIX_PACKAGE_PATH in that the repos are used
> to extend the modules that Guix uses.  Unlike GUIX_PACKAGE_PATH,
> however, we now have a way to record the complete state of Guix,
> including any extensions: the version of Guix and all active channels
> with their versions.  

Exactly.

> We would also have a way to fetch substitutes from
> channels without having to “globally” enable new substitute servers and
> authorize their keys.  (Is this safe?  Can we have per-user extensions
> to the set of public keys that are accepted?)
> 
> Downsides: Guix has no stable ABI, so channels that are not up-to-date
> will break with newer versions of Guix.  

Popular channels will move along, I have no doubt.

> Moving around packages to
> different modules might break channels.  That’s okay.  It’s still an
> improvement over plain GUIX_PACKAGE_PATH.
> 
> I don’t think it has to be more complicated than that.  What do you
> think?

I think it is spot on. Some devil in details, I am sure.

We can also build a web page where we can list channels, test-run
them, and their libre-state. In principle build-farm support could be
added for libre-channels. It would help GNU Guix move forward as a
generalized solution. You will not believe the pain I went through
with Fred just to install a huge GUIX_PACKAGE_PATH package. And we
both already use Guix and can write packages! 

In the next phase I can add support for relocatable packages attached
to a channel. We could get to the point that:

1. Install Guix with proot support
2. Fetch channel
3. Install binary package from channel (native compiled and optimized)
   into proot

So far will work for all software that runs in proot. Next

4. Relocate binary

And we have a local optimized software without ever getting root
because the relocated binary won't need proot.

It won't work for all software though. But for most computational
tools it will. Anyway, separate story. Even a native channel is a
huge win.

We can start hacking at FOSDEM. I am happy to participate.

Pj.

Re: [RFC] A simple draft for channels

[Ludovic Courtès]

Hi!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> As a first implementation of channels I’d just like to have a channel
> description file that records at least the following things:
>
> * the channel name (all lower case, no spaces)
> * a URL from where package definitions can be loaded (initially, this
>   can be restricted to git repositories)
>
> Optional fields:
>
> * a description of the channel
>
> * a URL from where substitutes for the packages can be obtained (this
>   will be backed by “guix publish”)
>
> * a mail address or URL to contact the maintainers of the channel, or to
>   view the status of the channel
>
> * the Guix git commit that was used when this channel was last
>   updated.  This is useful when Guix upstream breaks the ABI or moves
>   packages between modules.

Sounds good.

> On the Guix side we’d need to add the “guix channel” command, which
> allows for adding, removing, updating, and downgrading channels.  Adding
> a channel means fetching the channel description from a URL and storing
> state in ~/.config/guix/channels/, and fetching the git repo it
> specifies (just like what guix pull does: it’s a git frontend).

I think what you described above is “config” rather than “state.”

To me, “state” would be the current channel commit being used and the
list of previous commits that were used (akin to the Git reflog).  That
way, one could always roll back a “guix pull” or “guix channel update”
operation.

The reflog thing is a feature we can already add to ‘guix pull’.  I
think the migration to channels can be incremental.

> It also authorizes the the substitute server’s public key.

This would require root access.

> Internally, it’s just like GUIX_PACKAGE_PATH in that the repos are used
> to extend the modules that Guix uses.  Unlike GUIX_PACKAGE_PATH,
> however, we now have a way to record the complete state of Guix,
> including any extensions: the version of Guix and all active channels
> with their versions.  We would also have a way to fetch substitutes from
> channels without having to “globally” enable new substitute servers and
> authorize their keys.  (Is this safe?  Can we have per-user extensions
> to the set of public keys that are accepted?)

Authorizing keys is necessarily limited to root since the store is
shared among all users of the machine.  I don’t see any way around that
(other than switching to the other model presented in Eelco’s thesis,
which is to content-address derivation outputs—but I don’t imagine us
playing with that idea in the near future :-)).

> Downsides: Guix has no stable ABI, so channels that are not up-to-date
> will break with newer versions of Guix.  Moving around packages to
> different modules might break channels.  That’s okay.  It’s still an
> improvement over plain GUIX_PACKAGE_PATH.
>
> I don’t think it has to be more complicated than that.  What do you
> think?

I agree!

One thing that’s still an open question is how we should treat Guix
itself in that channelized world.

Should Guix be a “normal” channel?  It’s tempting to think of it as a
regular channel; however, it’s definitely “special” in that it can
update the ‘guix’ command, maybe guix-daemon & co., locale data, etc.
How does that affect ‘guix channel’?

Thanks,
Ludo’.

Re: [RFC] A simple draft for channels

[Pjotr Prins]

On Fri, Jan 19, 2018 at 02:41:42PM +0100, Ludovic Courtès wrote:
> Authorizing keys is necessarily limited to root since the store is
> shared among all users of the machine.  I don’t see any way around that

Well, the daemon could update itself with its own privileges. How
about maintaining authentication for a channel at runtime in RAM. When
the daemon restarts it is lost. The channel will not be shared with
other users. So every user maintains their own channels. When a
channel reconnects it authenticates itself again.

There really is no reason to share individual channels between users
(other then their outputs).

Pj.

Re: [RFC] A simple draft for channels

[宋文武]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Hi Guix,
>
> I’d like to retire GUIX_PACKAGE_PATH as the main way to get third-party
> packages, because we can’t really keep track of packages that were added
> or redefined in this way.  I want to replace it with slightly more
> formal “channels”.

Hello, as a way to get and manage third-party guix repositories, it
sounds more like “overlays” (in Gentoo world) rather than “channels” (in
Nix world, to ensure binaries availability and updates stability) to me.

https://wiki.gentoo.org/wiki/Overlay
https://nixos.org/nix/manual/index.html#sec-channels

How about call it as “overlay”?

Re: [RFC] A simple draft for channels

[Ricardo Wurmus]

Hi Pjotr,

> On Fri, Jan 19, 2018 at 02:41:42PM +0100, Ludovic Courtès wrote:
>> Authorizing keys is necessarily limited to root since the store is
>> shared among all users of the machine.  I don’t see any way around that
>
> Well, the daemon could update itself with its own privileges.

I think Ludo’s point is that this is a security issue, not a technical
limitation.

> How
> about maintaining authentication for a channel at runtime in RAM. When
> the daemon restarts it is lost. The channel will not be shared with
> other users. So every user maintains their own channels. When a
> channel reconnects it authenticates itself again.

It all ends up in the store though and is thus available to everybody.

> There really is no reason to share individual channels between users
> (other then their outputs).

Yes, channel configuration and state is kept in the user’s home
directory.  But authorization for downloading and installing substitutes
in /gnu/store currently still falls to root.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [RFC] A simple draft for channels

[Pjotr Prins]

On Tue, Jan 23, 2018 at 07:38:46AM +0100, Ricardo Wurmus wrote:
> 
> Hi Pjotr,
> 
> > On Fri, Jan 19, 2018 at 02:41:42PM +0100, Ludovic Courtès wrote:
> >> Authorizing keys is necessarily limited to root since the store is
> >> shared among all users of the machine.  I don’t see any way around that
> >
> > Well, the daemon could update itself with its own privileges.
> 
> I think Ludo’s point is that this is a security issue, not a technical
> limitation.

How is it a security issue? If authentication is user bound it becomes
a two-step or a one-step choice. User still has to do something
explicit to open a channel. Installed software is running in user land
so it is no different from compiling your own. But yes, devil is in
the details.

> > How
> > about maintaining authentication for a channel at runtime in RAM. When
> > the daemon restarts it is lost. The channel will not be shared with
> > other users. So every user maintains their own channels. When a
> > channel reconnects it authenticates itself again.
> 
> It all ends up in the store though and is thus available to everybody.

Hmmm. Is that a true concern? I ususally have no trouble software
installed by other people on systems (in their HOME or not). If an SA
does not want to allow for it we can tell the daemon not to support
channels with auto-authentication.

> > There really is no reason to share individual channels between users
> > (other then their outputs).
> 
> Yes, channel configuration and state is kept in the user’s home
> directory.  But authorization for downloading and installing substitutes
> in /gnu/store currently still falls to root.

I don't see why the daemon can not handle it. But maybe we should just
do a round-table at FOSDEM.

Pj.

Re: [RFC] A simple draft for channels

[myglc2]

On 01/19/2018 at 14:41 Ludovic Courtès writes:

> Hi!
>
> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>
>> As a first implementation of channels I’d just like to have a channel
>> description file that records at least the following things:
>>
>> * the channel name (all lower case, no spaces)
>> * a URL from where package definitions can be loaded (initially, this
>>   can be restricted to git repositories)
>>
>> Optional fields:
>>
>> * a description of the channel
>>
>> * a URL from where substitutes for the packages can be obtained (this
>>   will be backed by “guix publish”)
>>
>> * a mail address or URL to contact the maintainers of the channel, or to
>>   view the status of the channel
>>
>> * the Guix git commit that was used when this channel was last
>>   updated.  This is useful when Guix upstream breaks the ABI or moves
>>   packages between modules.
>
> Sounds good.
>
>> On the Guix side we’d need to add the “guix channel” command, which
>> allows for adding, removing, updating, and downgrading channels.  Adding
>> a channel means fetching the channel description from a URL and storing
>> state in ~/.config/guix/channels/, and fetching the git repo it
>> specifies (just like what guix pull does: it’s a git frontend).
>
> I think what you described above is “config” rather than “state.”
>
> To me, “state” would be the current channel commit being used and the
> list of previous commits that were used (akin to the Git reflog).  That
> way, one could always roll back a “guix pull” or “guix channel update”
> operation.
>
> The reflog thing is a feature we can already add to ‘guix pull’.  I
> think the migration to channels can be incremental.
>
>> It also authorizes the the substitute server’s public key.
>
> This would require root access.
>
>> Internally, it’s just like GUIX_PACKAGE_PATH in that the repos are used
>> to extend the modules that Guix uses.  Unlike GUIX_PACKAGE_PATH,
>> however, we now have a way to record the complete state of Guix,
>> including any extensions: the version of Guix and all active channels
>> with their versions.  We would also have a way to fetch substitutes from
>> channels without having to “globally” enable new substitute servers and
>> authorize their keys.  (Is this safe?  Can we have per-user extensions
>> to the set of public keys that are accepted?)
>
> Authorizing keys is necessarily limited to root since the store is
> shared among all users of the machine.  I don’t see any way around that
> (other than switching to the other model presented in Eelco’s thesis,
> which is to content-address derivation outputs—but I don’t imagine us
> playing with that idea in the near future :-)).
>
>> Downsides: Guix has no stable ABI, so channels that are not up-to-date
>> will break with newer versions of Guix.  Moving around packages to
>> different modules might break channels.  That’s okay.  It’s still an
>> improvement over plain GUIX_PACKAGE_PATH.
>>
>> I don’t think it has to be more complicated than that.  What do you
>> think?
>
> I agree!
>
> One thing that’s still an open question is how we should treat Guix
> itself in that channelized world.
>
> Should Guix be a “normal” channel?  It’s tempting to think of it as a
> regular channel; however, it’s definitely “special” in that it can
> update the ‘guix’ command, maybe guix-daemon & co., locale data, etc.
> How does that affect ‘guix channel’?

ISTM this design allows channels to inject non-free &/or non-safe
components into other user's Guix systems. Is that true?

If so, how will it impact the Guix promise of software freedom/safety?

WDYT? - George

Re: [RFC] A simple draft for channels

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

myglc2 writes:
> On 01/19/2018 at 14:41 Ludovic Courtès writes:
>
>> Hi!
>>
>> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>>
>>> As a first implementation of channels I’d just like to have a channel
>>> description file that records at least the following things:
[…]
>> One thing that’s still an open question is how we should treat Guix
>> itself in that channelized world.
>>
>> Should Guix be a “normal” channel?  It’s tempting to think of it as a
>> regular channel; however, it’s definitely “special” in that it can
>> update the ‘guix’ command, maybe guix-daemon & co., locale data, etc.
>> How does that affect ‘guix channel’?
>
> ISTM this design allows channels to inject non-free &/or non-safe
> components into other user's Guix systems. Is that true?
>
> If so, how will it impact the Guix promise of software freedom/safety?
>
> WDYT? - George

Just commenting on this one for now until I got my mail fixed:

Why is this a problem? Already today you can run Guix with as
many modifications as you like to, and you are free to install
whatever you want.
That's one of the very good aspects of Guix - you can use it to
create whatever you like.
Or maybe you need to expand a bit on the sentences you wrote
George.

-- 
ng0 :: https://ea.n0.is
A88C8ADD129828D7EAC02E52E22F9BBFEE348588 :: https://ea.n0.is/keys/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [RFC] A simple draft for channels

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> One thing that’s still an open question is how we should treat Guix
> itself in that channelized world.
>
> Should Guix be a “normal” channel?

Guix as such should not be a “normal” channel; “(gnu packages *)”,
however, could be considered a channel.  I don’t see any use of
disabling it, though.

But maybe there’s potential here to split up that namespace into core
packages and non-core packages.  Then we could allow users to disable
all but the core packages (such as the bootstrap) and possibly drop in
replacements from their own channels.  This would give users a little
more control over the set of available packages, but this all sounds a
little too speculative to me at this point.

My initial goal is much simpler.

--
Ricardo

Re: [RFC] A simple draft for channels

[Ricardo Wurmus]

myglc2@gmail.com writes:

>> One thing that’s still an open question is how we should treat Guix
>> itself in that channelized world.
>>
>> Should Guix be a “normal” channel?  It’s tempting to think of it as a
>> regular channel; however, it’s definitely “special” in that it can
>> update the ‘guix’ command, maybe guix-daemon & co., locale data, etc.
>> How does that affect ‘guix channel’?
>
> ISTM this design allows channels to inject non-free &/or non-safe
> components into other user's Guix systems. Is that true?

If one user could override the daemon or the guix command *for other
users* that wouldn’t be okay.  The simple channel draft would not make
this possible.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [RFC] A simple draft for channels

[Carlo Zancanaro]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On Tue, Jan 23 2018, Pjotr Prins wrote:
> How is it a security issue?

If I can authorise any substitute server key that I want, then I 
can authorise my own server's key. I can then create a malicious 
substitute that doesn't correspond to the build recipe in Guix. I 
could inject whatever code I want into this substitute, and have 
it placed in the store as the output for the derivation. When 
another user attempts to install the same package into their 
profile they will then use my malicious substitute (even though 
they never authorised my server's key).

Carlo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [RFC] A simple draft for channels

[myglc2]

On 01/23/2018 at 16:50 ng0@n0.is writes:

> myglc2 writes:
>> On 01/19/2018 at 14:41 Ludovic Courtès writes:
>>
>>> Hi!
>>>
>>> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>>>
>>>> As a first implementation of channels I’d just like to have a channel
>>>> description file that records at least the following things:
> […]
>>> One thing that’s still an open question is how we should treat Guix
>>> itself in that channelized world.
>>>
>>> Should Guix be a “normal” channel?  It’s tempting to think of it as a
>>> regular channel; however, it’s definitely “special” in that it can
>>> update the ‘guix’ command, maybe guix-daemon & co., locale data, etc.
>>> How does that affect ‘guix channel’?
>>
>> ISTM this design allows channels to inject non-free &/or non-safe
>> components into other user's Guix systems. Is that true?
>>
>> If so, how will it impact the Guix promise of software freedom/safety?
>>
>> WDYT? - George
>
> Just commenting on this one for now until I got my mail fixed:
>
> Why is this a problem? Already today you can run Guix with as many
> modifications as you like to, and you are free to install whatever you
> want.  That's one of the very good aspects of Guix - you can use it to
> create whatever you like.  Or maybe you need to expand a bit on the
> sentences you wrote George.

Yes, and this is important to the current user base. But in the future
the majority of our users will be end-users that do not directly use FSF
freedoms & Guix hackability. Still, they will choose Guix because this
freedom and hackability provides indirect benefits such as enhanced
security and safety.

Yes, FSF freedom means we must permit any user to shoot themselves in
the foot. But with GUIX_PACKAGE_PATH, this is not a worry.

Channels dramatically increases the ease with which an end-user can harm
themselves by e.g. using a channel that delivers non-free &/or non-safe
software. This raises the question: are we obliged to, and if so, how do
we help end-users protect themselves from this risk?

Re: [RFC] A simple draft for channels

[Pjotr Prins]

On Tue, Jan 23, 2018 at 09:37:04PM +0100, Ricardo Wurmus wrote:
> My initial goal is much simpler.

Let's stick to simple for now. Sorry about raising the noise level. I
would be extremely pleased with a simple replacement of
GUIX_PACKAGE_PATH and I am happy to contribute.

Keys etc. we can probably allow through root rights.

Pj.

Re: [RFC] A simple draft for channels

[ng0]

myglc2@gmail.com:
> On 01/23/2018 at 16:50 ng0@n0.is writes:
>
>> myglc2 writes:
>>> On 01/19/2018 at 14:41 Ludovic Courtès writes:
>>>
>>>> Hi!
>>>>
>>>> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>>>>
>>>>> As a first implementation of channels I’d just like to have a channel
>>>>> description file that records at least the following things:
>> […]
>>>> One thing that’s still an open question is how we should treat Guix
>>>> itself in that channelized world.
>>>>
>>>> Should Guix be a “normal” channel?  It’s tempting to think of it as a
>>>> regular channel; however, it’s definitely “special” in that it can
>>>> update the ‘guix’ command, maybe guix-daemon & co., locale data, etc.
>>>> How does that affect ‘guix channel’?
>>>
>>> ISTM this design allows channels to inject non-free &/or non-safe
>>> components into other user's Guix systems. Is that true?
>>>
>>> If so, how will it impact the Guix promise of software freedom/safety?
>>>
>>> WDYT? - George
>>
>> Just commenting on this one for now until I got my mail fixed:
>>
>> Why is this a problem? Already today you can run Guix with as many
>> modifications as you like to, and you are free to install whatever you
>> want.  That's one of the very good aspects of Guix - you can use it to
>> create whatever you like.  Or maybe you need to expand a bit on the
>> sentences you wrote George.
>
> Yes, and this is important to the current user base. But in the future
> the majority of our users will be end-users that do not directly use FSF
> freedoms & Guix hackability. Still, they will choose Guix because this
> freedom and hackability provides indirect benefits such as enhanced
> security and safety.
>
> Yes, FSF freedom means we must permit any user to shoot themselves in
> the foot. But with GUIX_PACKAGE_PATH, this is not a worry.
>
> Channels dramatically increases the ease with which an end-user can harm
> themselves by e.g. using a channel that delivers non-free &/or non-safe
> software. This raises the question: are we obliged to, and if so, how do
> we help end-users protect themselves from this risk?

In my honest opinion: No. We can not prevent this. All we can do
is to provide a list of *official* channels. Beyond that I don't
think we should try to regulate what's in an unofficial channel
and what's allowed.
It would be waste of time better spent in other parts of the
project.

-- 
ng0 :: https://ea.n0.is
A88C8ADD129828D7EAC02E52E22F9BBFEE348588 :: https://ea.n0.is/keys/

Re: [RFC] A simple draft for channels

[Ludovic Courtès]

Hello,

iyzsong@member.fsf.org (宋文武) skribis:

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
>
>> Hi Guix,
>>
>> I’d like to retire GUIX_PACKAGE_PATH as the main way to get third-party
>> packages, because we can’t really keep track of packages that were added
>> or redefined in this way.  I want to replace it with slightly more
>> formal “channels”.
>
> Hello, as a way to get and manage third-party guix repositories, it
> sounds more like “overlays” (in Gentoo world) rather than “channels” (in
> Nix world, to ensure binaries availability and updates stability) to me.
>
> https://wiki.gentoo.org/wiki/Overlay
> https://nixos.org/nix/manual/index.html#sec-channels
>
> How about call it as “overlay”?

I think “overlay” has the notion that it’s a layer on top of another
layer, which is not exactly the case here.  It’s not exactly like what
Nix channels either, but closer to that, I think.

Ludo’.

Re: [RFC] A simple draft for channels

[Konrad Hinsen]

On 24/01/2018 13:33, ng0@n0.is wrote:

> In my honest opinion: No. We can not prevent this. All we can do
> is to provide a list of *official* channels. Beyond that I don't
> think we should try to regulate what's in an unofficial channel
> and what's allowed.

+1

The best option in my opinion is to make it as easy as possible for 
users to find out the policy of any channel, and to do automatic 
filtering (for the day when there will be hundreds of channels to manage).

One simple measure would be a field in the channel description that says 
"complies with Guix' rules for free software yes/no". An added benefit: 
it forces channel designers to think about the question right from the 
start.

Konrad.

Re: [RFC] A simple draft for channels

[myglc2]

On 01/24/2018 at 15:08 Ludovic Courtès writes:

> Hello,
>
> iyzsong@member.fsf.org (宋文武) skribis:
>
>> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
>>
>>> Hi Guix,
>>>
>>> I’d like to retire GUIX_PACKAGE_PATH as the main way to get third-party
>>> packages, because we can’t really keep track of packages that were added
>>> or redefined in this way.  I want to replace it with slightly more
>>> formal “channels”.
>>
>> Hello, as a way to get and manage third-party guix repositories, it
>> sounds more like “overlays” (in Gentoo world) rather than “channels” (in
>> Nix world, to ensure binaries availability and updates stability) to me.
>>
>> https://wiki.gentoo.org/wiki/Overlay
>> https://nixos.org/nix/manual/index.html#sec-channels
>>
>> How about call it as “overlay”?

> I think “overlay” has the notion that it’s a layer on top of another
> layer, which is not exactly the case here.  It’s not exactly like what
> Nix channels either, but closer to that, I think.

Because Nix is not well known and their definition[1] is obscure I
looked for mainstream uses of "channels". I found Chrome "Release
Channels"[2] and Red Hat Network Satellite "Channels"[3,4]. A Chrome
user can use only one channel at a time, e.g., Stable, Beta,
Developer. The Red Hat model is more complex:

CHANNEL
A channel is a list of packages. Channels are used to choose packages to
be installed from client systems. Every client system must be subscribed
to one Base Channel and can be subscribed to one or more Child Channel .

BASE CHANNEL
A base channel is a type of Channel that consists of a list of packages
based on a specific architecture and Red Hat release. For example, all
the packages in Red Hat Enterprise Linux AS 3 for the x86 architecture
make a base channel.

CHILD CHANNEL
A child channel is a Channel associated with a Base Channel but contains
extra packages.

Ricardo, AIUI, your channels don't work like this. But I am at a loss to
say how they will work in practice from the user's POV. If you could
write such a description it would sure help us understand the proposal
better.

[1] https://nixos.org/nix/manual/#sec-channels:

    "A Nix channel is just a URL that points to a place that contains a
    set of Nix expressions and a manifest."

[2] https://www.chromium.org/getting-involved/dev-channel

[3]
https://access.redhat.com/documentation/en-us/red_hat_network_satellite/5.3/html/reference_guide/glossary

[4] https://access.redhat.com/documentation/en-us/red_hat_network_satellite/5.3/html/reference_guide/s1-sm-channels-packages

Re: [RFC] A simple draft for channels

[Ricardo Wurmus]

myglc2@gmail.com writes:

> Ricardo, AIUI, your channels don't work like this. But I am at a loss to
> say how they will work in practice from the user's POV. If you could
> write such a description it would sure help us understand the proposal
> better.

This draft is similar to how Conda channels[1] work:

    Conda packages are downloaded from remote channels, which are URLs
    to directories containing conda packages. The conda command searches
    a default set of channels, and packages are automatically downloaded
    and updated from http://repo.continuum.io/pkgs/. You can modify what
    remote channels are automatically searched. You might want to do
    this to maintain a private or internal channel.

[1]: https://conda.io/docs/user-guide/concepts.html#conda-packages

In our case a channel consists of a git repository with Guile module
definitions and a channel description file.  Optionally, it points to a
URL from where substitutes can be fetched.

Other than that it behaves like a Conda user would expect:

   “Guix does not yet include ‘foolicious 2000’, but I know it has been
    packaged by someone else and made available on the
    ‘garlic-onion-sweat’ channel.  So I enable this channel, authorize
    its public key, and then install ‘foolicious 2000’.”

Users can add any number of channels; their modules will be downloaded
and built (TODO: this might be dangerous, so try sandboxing), and then
added to GUIX_PACKAGE_PATH.

The second iteration of this would involve some potluck ideas for extra
safety and convenience.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [RFC] A simple draft for channels

[myglc2]

On 01/24/2018 at 19:20 Ricardo Wurmus writes:

> myglc2@gmail.com writes:
>
>> Ricardo, AIUI, your channels don't work like this. But I am at a loss to
>> say how they will work in practice from the user's POV. If you could
>> write such a description it would sure help us understand the proposal
>> better.
>
> This draft is similar to how Conda channels[1] work:
>
>     Conda packages are downloaded from remote channels, which are URLs
>     to directories containing conda packages. The conda command searches
>     a default set of channels, and packages are automatically downloaded
>     and updated from http://repo.continuum.io/pkgs/. You can modify what
>     remote channels are automatically searched. You might want to do
>     this to maintain a private or internal channel.
>
> [1]: https://conda.io/docs/user-guide/concepts.html#conda-packages
>
> In our case a channel consists of a git repository with Guile module
> definitions and a channel description file.  Optionally, it points to a
> URL from where substitutes can be fetched.
>
> Other than that it behaves like a Conda user would expect:
>
>    “Guix does not yet include ‘foolicious 2000’, but I know it has been
>     packaged by someone else and made available on the
>     ‘garlic-onion-sweat’ channel.  So I enable this channel, authorize
>     its public key, and then install ‘foolicious 2000’.”
>
> Users can add any number of channels; their modules will be downloaded
> and built (TODO: this might be dangerous, so try sandboxing), and then
> added to GUIX_PACKAGE_PATH.
>
> The second iteration of this would involve some potluck ideas for extra
> safety and convenience.
>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net

Thanks Ricardo.

So here is the most brief intro I can come up with:

'guix Channel' allows you to use software packages provided by any
third-party git repo.  To use substitutes that may be provided by the
channel, root must authorize the third-party server. Warning: Channels
may expose you to non-free or harmful software.

WDYT?

I don't think you should call them "channels." Why? Only advanced Nix or
Conda users will know what it means.  For anyone familiar with Chrome,
Red Hat, or TV channels, the "channels" label works against you: they
have to learn and remember that a Guix channel works differently. These
labels might work better: "Guix Sources" (ala Debian sources list[1]),
"third party sources" or "third party repos".

WDYT?

[1] https://wiki.debian.org/SourcesList

Re: [RFC] A simple draft for channels

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

myglc2@gmail.com writes:

[...]

> I don't think you should call them "channels." Why? Only advanced Nix or
> Conda users will know what it means.  For anyone familiar with Chrome,
> Red Hat, or TV channels, the "channels" label works against you: they
> have to learn and remember that a Guix channel works differently. These
> labels might work better: "Guix Sources" (ala Debian sources list[1]),
> "third party sources" or "third party repos".
>
> WDYT?
>
> [1] https://wiki.debian.org/SourcesList

"Sources" is a very bad name.  Especially for search engines.  And it
also bad for the reference.  You will need to specify what is the 'Guix
source' all the time you mention it.  I think about '~/src/guix' Git
repository as I see this.

I think the best for understand phrase is a "third party package
repository" or "package repositories" for short.  Is it too general or
could we stick with it?

I also like Ricardo's mention 'Guix package collection' on the IRC.
Maybe 'Guix collections' will be a good candidate?  It's unique unlike
'package repositories', which is good for search.

Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [RFC] A simple draft for channels

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4473 bytes --]

Hi Ricardo,

This is a great start - I'm excited to playing with a prototype!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> As a first implementation of channels I’d just like to have a channel
> description file that records at least the following things:
>
> * the channel name (all lower case, no spaces)
> * a URL from where package definitions can be loaded (initially, this
>   can be restricted to git repositories)

Should a channel have to declare its own version?  Is this even
necessary/useful, or will this be taken care of in some other way, for
example by hashing the fetched copy of the channel's package
definitions?

> * a URL from where substitutes for the packages can be obtained (this
>   will be backed by “guix publish”)

Do you intend to couple a channel with its substitute URL?  Since this
is optional, it doesn't sound that way to me, but I just want to double
check.  Even if a channel chooses to provide its own substitutes, I
think that in general a substitute for a package should be able to come
from any authorized substitute server, not just the one declared by the
channel.  They should be decoupled.

> * the Guix git commit that was used when this channel was last
>   updated.  This is useful when Guix upstream breaks the ABI or moves
>   packages between modules.

It sounds like you intend for HUMANS to use this information (when
present) in a purely ADVISORY fashion, like this: if the version of the
user's installed Guix matches (or is close) the one specified here, then
the channel's package definitions will (or are likely to) work
correctly; otherwise, all bets are off.  Is that right?

What do you mean by "when this channel was last updated"?  I can think
of at least two interpretations: when the package definitions change,
and when the substitutes change (because the channel owner built them
using a different version of Guix).  To maximize this information's
usefulness, we should be clear about what it means.

> On the Guix side we’d need to add the “guix channel” command, which
> allows for adding, removing, updating, and downgrading channels.

How are channels created?  In other words, how will one prepare all the
things that must be present at the channel URL?  Will there be a command
for this?

> Adding a channel means fetching the channel description from a URL

Will there be any restrictions on the relationship between this URL and
the URL from where the channel's package definitions can be loaded?

> and storing state in ~/.config/guix/channels/, and fetching the git
> repo it specifies (just like what guix pull does: it’s a git
> frontend).

As an administrator, how will I add a channel system-wide, for all
users?  Will the method differ for GuixSD and foreign distros?

> It also authorizes the the substitute server’s public key.

Please correct me if I'm wrong, but my understanding is that for
security reasons, we must not let an unprivileged user authorize a
substitute server's public key.  If the intensional model were
implemented, this might be different, but since we currently use the
extensional model, we must not allow it, since those substitutes will be
shared by all users.  To allow an unprivileged user to authorize a
substitute server's public key would be to allow her to make a trust
decision on behalf of all users of the system.

Additional questions follow:

Do you intend for this initial design to be orthogonal to the
improvements for "guix pull" that have been discussed elsewhere (e.g.,
bug 22629)?

What is the intended behavior when a channel defines a package with the
same name as a "core" Guix package or a package from another channel?
(Will we use some kind of prioritization or namespacing to avoid this?)

For creation, deletion, upgrading, and downgrading of channels, what
sort of mechanism do we plan to use?  Will it be similar to profiles
(e.g., flip a symlink)?  Will the channel's package definitions be
stored in the Guix store?

How will this differ, and how will this be similar, to Nix's channel
mechanism?  Nix has had "channels" for a long time, so if we can copy
any good ideas or techniques from there, we should.  I haven't
personally reviewed Nix's channel implementation in detail, so I'm not
really sure how your proposal is different or similar.

Thank you for getting the ball rolling on this!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[Orchestration][RFC] A simple draft for channels

[Pjotr Prins]

Let's start up again on this conversation in the context of
deployment. I have a simple use case. For GeneNetwork we maintain
GUIX_PACKAGE_PATH packages. It contains packages that ought to go in
main line (such as python-gunicorn), but also packages that will never
make it (such as a javascript twitter feed).

Now we deploy multiple setups, which I'll simplify to development,
testing and production here (there are more!). Each of these has a
combination of a Guix git checkout and a GUIX_PACKAGE_PATH checkout.

These git repo's are supposedly in sync with each other.

Moving from one to the other, however, is too complicated and error
prone. I can do it, but no one else really wants to. Even with my
explanations it proves to be a royal pain.

Especially with multiple people developing. The other thing is that it
takes too long to rebuild the Guix repo. Even on 20+ cores we have to
wait a significant amount of time. And things go wrong... 

Otherwise, only good stuff. I can provide binary packages, that is
great. It is just this git juggling and Guix repo building that is
killing me.

Maybe the short term solution for us is to no longer use
GUIX_PACKAGE_PATH, but to merge the differences in branches of the GNU
Guix tree. That takes care of the syncing problem (though it will
still be a headache). Maybe channels ought to be Guix git trees
anyway, so that is one step in the right direction.

Now I need a way to no longer rebuild all .go files for Guix tree
updates/changes. Not only between switching branches, but also when
just running 'git pull' from Guix savannah. I find I have to do that
very often. So often that I don't even try running make anymore
without make clean. Anyone here share that experience?

One thing I could do is split out 3 git repos for every use case and
update these individually not triggering rebuilds. And when I deploy
on other machines move the complete repo across with .go files.

Maybe also stick it into a container so I can deploy all dependencies
with it and update any substitute keys in user land.

HEY. Did I just invent a channel? 

If we allow channels to be architecture specific we could ship
dependencies and .go files with the git tree. No rebuilds.

If we allow using containers we can update substitute keys for
channels (like mine on http://guix.genenetwork.org).

The key updates are minor though. Main thing is to speed up deployment
and make it less error prone.

Maybe we should start thinking that a channel is simply an
architecture dependent Guix 'pack' of substitutes that includes the
pre-built Guix git repo. When deployed in a container we can inject
the keys. When this works we can design a pack repository, making the
channels searchable.

Pj.

Re: [Orchestration][RFC] A simple draft for channels

[ng0]

Pjotr Prins transcribed 2.7K bytes:
> Let's start up again on this conversation in the context of
> deployment. I have a simple use case. For GeneNetwork we maintain
> GUIX_PACKAGE_PATH packages. It contains packages that ought to go in
> main line (such as python-gunicorn), but also packages that will never
> make it (such as a javascript twitter feed).
> 
> Now we deploy multiple setups, which I'll simplify to development,
> testing and production here (there are more!). Each of these has a
> combination of a Guix git checkout and a GUIX_PACKAGE_PATH checkout.

This reads a bit like what I'm working on and experimenting with
indepently at the moment.
I'm still waiting on what the turnout of this channels RFC will be.

So: I have a combination of guix.git on some-commit + a couple of
package-path repositories (put into classes "ports" and "pkgs")
and another couple of repositories which include other data.
All of this is used to construct a specified variant of an operating
system. Updates happen with a simple command so far, nothing's worked
out in a way that can be exposed to public. GUIX_PACKAGE_PATH for me
locally is super long by now. Essentially parts that never make their
way into Guix for various reasons are kept in modular pieces outside
of Guix (there's more to it, but that would be derailing the thread).

> These git repo's are supposedly in sync with each other.
> 
> Moving from one to the other, however, is too complicated and error
> prone. I can do it, but no one else really wants to. Even with my
> explanations it proves to be a royal pain.
> 
> Especially with multiple people developing.

> The other thing is that it
> takes too long to rebuild the Guix repo. Even on 20+ cores we have to
> wait a significant amount of time. And things go wrong... 

Yep. This is not really good in the very long term.

> Otherwise, only good stuff. I can provide binary packages, that is
> great. It is just this git juggling and Guix repo building that is
> killing me.

Didn't we talk about building Guix itself on the build farm? What's
keeping us from doing so?

> Maybe the short term solution for us is to no longer use
> GUIX_PACKAGE_PATH, but to merge the differences in branches of the GNU
> Guix tree. That takes care of the syncing problem (though it will
> still be a headache). Maybe channels ought to be Guix git trees
> anyway, so that is one step in the right direction.

I think this (just guix trees) is not really good, at least for my goals.
I'd rather have a modular layout where you can define multiple repositories
and channels are an addition to Guix. Unless we have good reasons to remove
the current extension-like behavior I'd like to keep them.

> Now I need a way to no longer rebuild all .go files for Guix tree
> updates/changes. Not only between switching branches, but also when
> just running 'git pull' from Guix savannah. I find I have to do that
> very often. So often that I don't even try running make anymore
> without make clean. Anyone here share that experience?
> 
> One thing I could do is split out 3 git repos for every use case and
> update these individually not triggering rebuilds. And when I deploy
> on other machines move the complete repo across with .go files.
> 
> Maybe also stick it into a container so I can deploy all dependencies
> with it and update any substitute keys in user land.
> 
> HEY. Did I just invent a channel? 
> 
> If we allow channels to be architecture specific we could ship
> dependencies and .go files with the git tree. No rebuilds.
> 
> If we allow using containers we can update substitute keys for
> channels (like mine on http://guix.genenetwork.org).
> 
> The key updates are minor though. Main thing is to speed up deployment
> and make it less error prone.
> 
> Maybe we should start thinking that a channel is simply an
> architecture dependent Guix 'pack' of substitutes that includes the
> pre-built Guix git repo. When deployed in a container we can inject
> the keys. When this works we can design a pack repository, making the
> channels searchable.
> 
> Pj.
> 
> 
> 

-- 
A88C8ADD129828D7EAC02E52E22F9BBFEE348588
https://n0.is

Re: [RFC] A simple draft for channels

[ng0]

Ricardo Wurmus transcribed 1.7K bytes:
> 
> myglc2@gmail.com writes:
> 
> > Ricardo, AIUI, your channels don't work like this. But I am at a loss to
> > say how they will work in practice from the user's POV. If you could
> > write such a description it would sure help us understand the proposal
> > better.
> 
> This draft is similar to how Conda channels[1] work:
> 
>     Conda packages are downloaded from remote channels, which are URLs
>     to directories containing conda packages. The conda command searches
>     a default set of channels, and packages are automatically downloaded
>     and updated from http://repo.continuum.io/pkgs/. You can modify what
>     remote channels are automatically searched. You might want to do
>     this to maintain a private or internal channel.
> 
> [1]: https://conda.io/docs/user-guide/concepts.html#conda-packages
> 
> In our case a channel consists of a git repository with Guile module
> definitions and a channel description file.  Optionally, it points to a
> URL from where substitutes can be fetched.
> 
> Other than that it behaves like a Conda user would expect:
> 
>    “Guix does not yet include ‘foolicious 2000’, but I know it has been
>     packaged by someone else and made available on the
>     ‘garlic-onion-sweat’ channel.  So I enable this channel, authorize
>     its public key, and then install ‘foolicious 2000’.”
> 
> Users can add any number of channels; their modules will be downloaded
> and built (TODO: this might be dangerous, so try sandboxing), and then
> added to GUIX_PACKAGE_PATH.

Couldn't we include the use of GUILE_LOAD_PATH + GUILE_LOAD_COMPILED_PATH
as well, for custom definitions of services and other elements?
This would make the use of channels more independent from just being about
packages. Or does this fit into the second iteration already?

> The second iteration of this would involve some potluck ideas for extra
> safety and convenience.
> 
> --
> Ricardo
> 
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
> 
> 
> 
> 

-- 
A88C8ADD129828D7EAC02E52E22F9BBFEE348588
https://n0.is

Re: [Orchestration][RFC] A simple draft for channels

[myglc2]

Hi Pjotr,

On 03/19/2018 at 13:04 Pjotr Prins writes:

> Let's start up again on this conversation in the context of
> deployment. I have a simple use case. For GeneNetwork we maintain
> GUIX_PACKAGE_PATH packages. It contains packages that ought to go in
> main line (such as python-gunicorn), but also packages that will never
> make it (such as a javascript twitter feed).
>
> Now we deploy multiple setups, which I'll simplify to development,
> testing and production here (there are more!). Each of these has a
> combination of a Guix git checkout and a GUIX_PACKAGE_PATH checkout.
>
> These git repo's are supposedly in sync with each other.
>
> Moving from one to the other, however, is too complicated and error
> prone. I can do it, but no one else really wants to. Even with my
> explanations it proves to be a royal pain.

How about making guix a submodule of the GeneNetwork repo?

> Now I need a way to no longer rebuild all .go files for Guix tree
> updates/changes. Not only between switching branches, but also when
> just running 'git pull' from Guix savannah. I find I have to do that
> very often. So often that I don't even try running make anymore
> without make clean. Anyone here share that experience?

Yes the guix make does seem rather fragile ;-) So I usually do ...

guix environment guix  -M 4 -c 4 --ad-hoc help2man git strace
rm -fr /home/g1/.cache/guile/ccache/*
sudo git clean -dfx
git pull
./bootstrap
./configure --localstatedir=/var --sysconfdir=/etc
make -j 10
make -j 10 check

This takes a while but it avoids me chasing spurious errors caused by
clashes between the state of my build directory and the upstream
changes ;-)

> One thing I could do is split out 3 git repos for every use case and
> update these individually not triggering rebuilds. And when I deploy
> on other machines move the complete repo across with .go files.

Have you considered a git-worktree for each of the development, testing
and production branches?

HTH - George

Re: [Orchestration][RFC] A simple draft for channels

[Pjotr Prins]

On Mon, Mar 19, 2018 at 02:21:35PM -0400, myglc2@gmail.com wrote:
> > Moving from one to the other, however, is too complicated and error
> > prone. I can do it, but no one else really wants to. Even with my
> > explanations it proves to be a royal pain.
> 
> How about making guix a submodule of the GeneNetwork repo?

I don't like git submodules, unfortunately. I have plenty experience
there, and often not good. It works as long as you don't update the
modules ;)

I am OK with two git trees, there is no tight coupling between
GeneNetwork and Guix. But there is tight coupling between
GUIX_PACKAGE_PATH (guix-bioinformatics tree) and guix. I could
consider making guix-bioinformatics a module of guix. But I am sure I
am in for pain there too.

> > Now I need a way to no longer rebuild all .go files for Guix tree
> > updates/changes. Not only between switching branches, but also when
> > just running 'git pull' from Guix savannah. I find I have to do that
> > very often. So often that I don't even try running make anymore
> > without make clean. Anyone here share that experience?
> 
> Yes the guix make does seem rather fragile ;-) So I usually do ...
> 
> guix environment guix  -M 4 -c 4 --ad-hoc help2man git strace
> rm -fr /home/g1/.cache/guile/ccache/*
> sudo git clean -dfx
> git pull
> ./bootstrap
> ./configure --localstatedir=/var --sysconfdir=/etc
> make -j 10
> make -j 10 check

Mine is comparable, but even more rigorous:

  screen -S guix-build # I tend to build in screen
  env -i /bin/bash --login --noprofile --norc
  ./pre-inst-env guix environment guix --ad-hoc help2man git strace \
    pkg-config less vim binutils coreutils grep guile guile-git guile-json \
    gcc nss-certs --no-grafts
  bash # you may want this shell
  rm -rf autom4te.cache/ # to be sure
  make clean
  ./bootstrap
  ./configure --localstatedir=/var
  make clean    # to be really sure
  make clean-go # to be even surer
  make -j 

(forget the make check)

but, yes, the point is that I have to do this too often and it takes a
long time. So much that I thrust my hand through the monitor every
time I have to start again. It is costing me monitors.

And there are problems, usually with package updates that go out of
sync between my trees.

> This takes a while but it avoids me chasing spurious errors caused by
> clashes between the state of my build directory and the upstream
> changes ;-)

I think we agree.

> > One thing I could do is split out 3 git repos for every use case and
> > update these individually not triggering rebuilds. And when I deploy
> > on other machines move the complete repo across with .go files.
> 
> Have you considered a git-worktree for each of the development, testing
> and production branches?

Hmmm. That may be helpful. I should try that. 

Still does not solve my deployment problems.

Pj.
-- 

Re: [Orchestration][RFC] A simple draft for channels

[myglc2]

On 03/19/2018 at 19:31 Pjotr Prins writes:

> On Mon, Mar 19, 2018 at 02:21:35PM -0400, myglc2@gmail.com wrote:
>> > Moving from one to the other, however, is too complicated and error
>> > prone. I can do it, but no one else really wants to. Even with my
>> > explanations it proves to be a royal pain.
>> 
>> How about making guix a submodule of the GeneNetwork repo?
>
> I don't like git submodules, unfortunately. I have plenty experience
> there, and often not good. It works as long as you don't update the
> modules ;)
>
> I am OK with two git trees, there is no tight coupling between
> GeneNetwork and Guix. But there is tight coupling between
> GUIX_PACKAGE_PATH (guix-bioinformatics tree) and guix. I could
> consider making guix-bioinformatics a module of guix. But I am sure I
> am in for pain there too.

Other people bash them, but I have used git modules a lot for
hierarchical bio analysis and never hit a real issue.  Maybe you could
say more about the specific problem you see in this application?

>> > Now I need a way to no longer rebuild all .go files for Guix tree
>> > updates/changes. Not only between switching branches, but also when
>> > just running 'git pull' from Guix savannah. I find I have to do that
>> > very often. So often that I don't even try running make anymore
>> > without make clean. Anyone here share that experience?
>> 
>> Yes the guix make does seem rather fragile ;-) So I usually do ...
>> 
>> guix environment guix  -M 4 -c 4 --ad-hoc help2man git strace
>> rm -fr /home/g1/.cache/guile/ccache/*
>> sudo git clean -dfx
>> git pull
>> ./bootstrap
>> ./configure --localstatedir=/var --sysconfdir=/etc
>> make -j 10
>> make -j 10 check
>
> Mine is comparable, but even more rigorous:
>
>   screen -S guix-build # I tend to build in screen
>   env -i /bin/bash --login --noprofile --norc
>   ./pre-inst-env guix environment guix --ad-hoc help2man git strace \
>     pkg-config less vim binutils coreutils grep guile guile-git guile-json \
>     gcc nss-certs --no-grafts
>   bash # you may want this shell
>   rm -rf autom4te.cache/ # to be sure
>   make clean
>   ./bootstrap
>   ./configure --localstatedir=/var
>   make clean    # to be really sure
>   make clean-go # to be even surer
>   make -j 
>
> (forget the make check)

OK, but I prefer 'sudo git clean -dfx' because it innoculates me against
any errors in 'make clean' logic. I should use ./pre-inst-env more ;-)

> but, yes, the point is that I have to do this too often and it takes a
> long time. So much that I thrust my hand through the monitor every
> time I have to start again. It is costing me monitors.

I agree it is annoying, but maybe this is the cost of complete artistic
source code freedom? And MIPS are so cheap these days, no?

> And there are problems, usually with package updates that go out of
> sync between my trees.

guix as a submodule would avoid this, no?

>> This takes a while but it avoids me chasing spurious errors caused by
>> clashes between the state of my build directory and the upstream
>> changes ;-)
>
> I think we agree.
>
>> > One thing I could do is split out 3 git repos for every use case and
>> > update these individually not triggering rebuilds. And when I deploy
>> > on other machines move the complete repo across with .go files.
>> 
>> Have you considered a git-worktree for each of the development, testing
>> and production branches?
>
> Hmmm. That may be helpful. I should try that. 
>
> Still does not solve my deployment problems.

If it were me, I would do source deployment w/ 'git clone
--recurse-submodules', source update w/ 'git pull; git submodule update'
and try to use bespoke project code to check out development and testing
branches with git-subtree.

You can capture the .go files by checking them into disposable "deploy"
branches which you cam either pull to the target machine or a push to a
non-naked repo.

Am I missing something? - George

Re: [Orchestration][RFC] A simple draft for channels

[Pjotr Prins]

On Mon, Mar 19, 2018 at 04:18:29PM -0400, myglc2@gmail.com wrote:
> Other people bash them, but I have used git modules a lot for
> hierarchical bio analysis and never hit a real issue.  Maybe you could
> say more about the specific problem you see in this application?

No modules, please. Like I wrote, it only works if you don't really
change the modules. It is particularly bad when working with other
people.

> OK, but I prefer 'sudo git clean -dfx' because it innoculates me against
> any errors in 'make clean' logic. I should use ./pre-inst-env more ;-)

dfx. I'll look into that, thanks.

> I agree it is annoying, but maybe this is the cost of complete artistic
> source code freedom? And MIPS are so cheap these days, no?

My time is not exactly cheap. I can do something else, but it breaks
the flow. Which means I start looking at ways of avoiding updates. And
it happens too often.

> You can capture the .go files by checking them into disposable "deploy"
> branches which you cam either pull to the target machine or a push to a
> non-naked repo.

Awesome, that is a really bad-ass idea. I may just try it for fun!

Using branches that way is not exactly what git was meant for ;). 

> Am I missing something? - George

Yeah, the discussion is about channels. I.e., sharing bespoke binary
deployment. And making it really easy. And saving me and others from
pain. That is my itch to scratch.

Even so, I'll try sub-tree and dfx - they may help me ;).

Pj.

-- 

Re: [Orchestration][RFC] A simple draft for channels

[Pjotr Prins]

On Mon, Mar 19, 2018 at 01:04:00PM +0100, Pjotr Prins wrote:
> Maybe we should start thinking that a channel is simply an
> architecture dependent Guix 'pack' of substitutes that includes the
> pre-built Guix git repo. When deployed in a container we can inject
> the keys. When this works we can design a pack repository, making the
> channels searchable.

Continuing my line of thought: a binary channel in my mind is now a
compiled Guix package tree with Guix and possible adaptations (say a
special package for Ruby). In other words, a special (timed) version
of Guix sitting in /gnu/store/*named-guix-channel/bin/guix. Installing
it I would simply create my own profile

    guix -i named-guix-channel -p ~/opt/named-guix-channel

and use that to install new software using the running guix-daemon. I.e.

    ~/opt/named-guix-channel/bin/guix package -A

or maybe better

    ~/opt/named-guix-channel/bin/named-guix-channel package -A

so it can actually live in a shared profile and pick up its own
substitute server. Anyone can create a channel and they can even live
on trunk as packages:

    guix package -i bio-ruby-guix-channel
    (admin add key)
    bio-ruby-guix-channel package -i bio-ruby

Simple but effective. Wouldn't that be cool?

To add a channel we may need some mechanism - possibly something like
loading a .scm on the command line which fetches the guix-channel
binary from a substitute server. The more flexible the better. So

    (admin add key)
    guix channel -i URI/my-channel.scm
    (which does a 'guix package -i my-bio-ruby-guix-channel')
    my-ruby-guix-channel package -i bio-ruby

a version would be

    (admin add key)
    guix channel -i URI/my-channel.scm GITHASH/GITTAG
    (which does a 'guix package -i my-bio-ruby-guix-channel')
    my-ruby-guix-channel package -i bio-ruby

We can start without the 'guix channel' bit. Early channel adopters
can work from a compiled source tree.

The only immediate adaptation to Guix required is that named-guix-channel
can pick up its own compiled package definitions and we have
fast binary-channels.

Right?

Pj.


-- 

Re: [Orchestration][RFC] A simple draft for channels

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Mon, Mar 19, 2018 at 01:04:00PM +0100, Pjotr Prins wrote:
>> Maybe we should start thinking that a channel is simply an
>> architecture dependent Guix 'pack' of substitutes that includes the
>> pre-built Guix git repo. When deployed in a container we can inject
>> the keys. When this works we can design a pack repository, making the
>> channels searchable.
>
> Continuing my line of thought: a binary channel in my mind is now a
> compiled Guix package tree with Guix and possible adaptations (say a
> special package for Ruby). In other words, a special (timed) version
> of Guix sitting in /gnu/store/*named-guix-channel/bin/guix.

What you describe is essentially “guix pull”.  “guix pull” can already
use a different branch or a different repository.

It has at least two problems in this context:

1) it only keeps a link to the “latest” Guix.  There is no way to
   declare the use of a different variant of Guix.

2) it builds everything from source (Ludo’s branch for splitting the
   target of “guix pull” into several derivations addresses this)

Maybe we should address the first problem next and allow for not only
the latest Guix version to be retained.  This is not quite the same as a
regular profile, because these different variants of Guix surely would
provide colliding files.

The guix command would need to be changed to pick “latest” by default,
but use a different variant if specified.

The workflow would be something like that:

    # fetch Guix from somewhere and record as “foobar”
    guix pull --url=https://somewhere foobar

    # use that variant of Guix
    guix --variant=foobar build hello

I don’t think this should be the only mechanism through which people can
provide channels.  I wouldn’t want to have to essentially fork Guix.
For a user this is a problem, too, because channels would no longer be
composable (today I can compose multiple package collections with
GUIX_PACKAGE_PATH).

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [Orchestration][RFC] A simple draft for channels

[Pjotr Prins]

Thanks Ricardo:

On Tue, Mar 20, 2018 at 11:41:33AM +0100, Ricardo Wurmus wrote:
> 
> Pjotr Prins <pjotr.public12@thebird.nl> writes:
> 
> > On Mon, Mar 19, 2018 at 01:04:00PM +0100, Pjotr Prins wrote:
> >> Maybe we should start thinking that a channel is simply an
> >> architecture dependent Guix 'pack' of substitutes that includes the
> >> pre-built Guix git repo. When deployed in a container we can inject
> >> the keys. When this works we can design a pack repository, making the
> >> channels searchable.
> >
> > Continuing my line of thought: a binary channel in my mind is now a
> > compiled Guix package tree with Guix and possible adaptations (say a
> > special package for Ruby). In other words, a special (timed) version
> > of Guix sitting in /gnu/store/*named-guix-channel/bin/guix.
> 
> What you describe is essentially “guix pull”.  “guix pull” can already
> use a different branch or a different repository.
>
> It has at least two problems in this context:
> 
> 1) it only keeps a link to the “latest” Guix.  There is no way to
>    declare the use of a different variant of Guix.

No git hash, no tag, but could be added later.

> 2) it builds everything from source (Ludo’s branch for splitting the
>    target of “guix pull” into several derivations addresses this)

That is not good enough. But it seems to me that substitution is only
an implementation question.

> Maybe we should address the first problem next and allow for not only
> the latest Guix version to be retained.  This is not quite the same as a
> regular profile, because these different variants of Guix surely would
> provide colliding files.
> 
> The guix command would need to be changed to pick “latest” by default,
> but use a different variant if specified.
> 
> The workflow would be something like that:
> 
>     # fetch Guix from somewhere and record as “foobar”
>     guix pull --url=https://somewhere foobar
> 
>     # use that variant of Guix
>     guix --variant=foobar build hello

If we can substitute, that would work. And guix pull would have to be
binary too. But yes, with binary instant gratification, that would
really be a good start. Different versions can live on different URLs.

(hmm, what did I just write?)

> I don’t think this should be the only mechanism through which people can
> provide channels.  I wouldn’t want to have to essentially fork Guix.
> For a user this is a problem, too, because channels would no longer be
> composable (today I can compose multiple package collections with
> GUIX_PACKAGE_PATH).

I am not sure composability is required for most use cases. I think we
should keep it simple. I am happy to have channels act independently
if we can get it this year.

Pj.

Re: [Orchestration][RFC] A simple draft for channels

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

>> I don’t think this should be the only mechanism through which people can
>> provide channels.  I wouldn’t want to have to essentially fork Guix.
>> For a user this is a problem, too, because channels would no longer be
>> composable (today I can compose multiple package collections with
>> GUIX_PACKAGE_PATH).
>
> I am not sure composability is required for most use cases. I think we
> should keep it simple. I am happy to have channels act independently
> if we can get it this year.

Yes, it’s not a big feature as there are practical limits to composing
multiple channels.  However, simple composition can come in handy.

At the MDC, for example, we strictly separate non-free software from
free software variants that are not available in Guix upstream.  Users
can add one or both of these package repositories.  This would not be
easy with the proposed “Guix fork as channel” implementation.

That said, I think that having a more featureful “guix pull” is a good
goal in itself.  Even better if this happens to scratch your itch wrt
channels already.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net