Medium-term road map

Medium-term road map

[Ludovic Courtès]

Hello Guix!

We released 1.1.0, but what’s coming next?  What would you like to see?

There are many exciting things being developed and great ideas floating
around.  For myself, I feel like focusing on “consolidating” what we
have in the coming weeks.  Here are the areas I hope to focus on (and
embarking as many of you as possible :-)):

  1. Authentication.  I want to finally provide a standardized mechanism
     to allow channels to be authenticated upon ‘guix pull’.  “make
     authenticate” was a first milestone, let’s get it done.  See
     <https://issues.guix.gnu.org/issue/22883>.

  2. Performance.  There are many things we can improve there, first and
     foremost: the “Computing derivation” part of ‘guix pull’, Guile’s
     compiler terrible time and space requirements, further optimizing
     core operations like ‘package-derivation’, as well as low-level
     stuff as described at <https://issues.guix.gnu.org/issue/40626>.

     Related to that is the question of substitute availability, another
     major hindrance to usability.  We should address this both in the
     build farm (reducing the
     time-to-push-to-time-of-substitute-availability, tracking random
     build failures), and on the client side (can we provide ways for
     users to pull to a commit that won’t require them to build
     everything from source, while not compromising on their security?).

  3. G-exps.  We should really finish the migration to gexps, as in the
     ‘wip-build-system-gexp’ branch, and adjust our APIs accordingly.

  4. User interface.  Let’s get our act together with ‘guix shell’ and
     ‘guix run-script’, and let’s address other annoyances that
     newcomers keep stumbling upon!

Thoughts?

Happy hacking!  :-)

Ludo’.

Re: Medium-term road map

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

Hi!

>   2. Performance.  

At a higher level, we can work on parallelization of builds and
downloads (all combinations).

Many improvements were suggested in this thread:

  https://lists.gnu.org/archive/html/guix-devel/2019-10/msg00694.html

Julien sent a patch to enable parallel downloads:

  https://issues.guix.info/issue/39728

This could significantly improve the user experience.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Medium-term road map

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

On Sat, 25 Apr 2020, Ludovic Courtès wrote:

> Hello Guix!
>
> We released 1.1.0, but what’s coming next?  What would you like to see?

…

> Thoughts?

Hi Ludo’ et al,

Thanks for your hard work on the 1.1.0 release!

Your proposed direction sounds great to me. To those, I would like to add:

* Improvements in the Go and Rust build systems. Are the any commonalities 
that we can factor our for providing the source of dependencies?

* Continued development of guix deploy. Figuring out how to deploy secrets 
to remote machines would be great.

* Accessibility for the installer. I don't know where we stand currently, 
but it would be great to know and make improvements.

* Sandboxing of applications. What would a Guix System with SELinux 
enforcement look like?

* GDM and GNOME-Shell Wayland sessions.

* Continued improvements to bootstrapping and GNU Hurd efforts.

I certainly don't have the energy to work on all of these at once. Goals 
for my involvement in Guix are to help review more patches, and contribute 
to areas of Guix outside of package definition.

Best,
Jack

Re: Medium-term road map

[Josh Marshall]

[-- Attachment #1: Type: text/plain, Size: 203 bytes --]

I'll echo Jack's suggestions for golang and rust build systems and secrets
deployments.  Another quick bit is having a better or documented way to
debug packages and in keeping more packages up to date.

[-- Attachment #2: Type: text/html, Size: 246 bytes --]

Re: Medium-term road map

[zimoun]

Hi Ludo,

On Sat, 25 Apr 2020 at 15:38, Ludovic Courtès <ludo@gnu.org> wrote:

>   4. User interface.  Let’s get our act together with ‘guix shell’ and
>      ‘guix run-script’, and let’s address other annoyances that
>      newcomers keep stumbling upon!

Adding some items to the whishlist. :-)

1. Access to the internal checkout without knowing the very internals.

When using "guix time-machine", I am regularly navigating in the Git
history. Therefore, I have to clone <somewhere>, going from the
<workdir> to <somewhere>, run Git commands, going back, etc. Well, it
is not handy since the checkout already exists (in
'~/.cache/guix/<hash>/').

Something like "guix git <command> <options>" should be useful. Even
if I am not clear on how to do with several channels. Maybe:

guix git <guix-options> -- <git-command> <git-options>
guix git -C chan.scm -- --no-pager log --oneline


Considering this new subcommand, it should become easy to add Git tags
and so ease the navigation (for reference [1]).

[1] https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00513.html


2. Search on all the packages included in Guix since the Big Bang.

It is difficult to find the Guix commit where one package goes in and
the commit where it goes out. The Guix Data Service (GDS) helps a lot
for that! But AFAIK, it is not possible from the CLI and I do not find
it handy when I need it (just because I need to open a webbrowser,
etc.).

IMHO, there is 3 questions:
 - how to build such full historical index locally? Even if it is expensive
 - how to fetch it from GDS? Or any other substitue?
 - how to append new packages locally? For example with packages from channels.

The bonus question is feeding SWH to be "guix time-machine" bullet-proof. :-)


Cheers,
simon

Re: Medium-term road map

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4220 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> We released 1.1.0, but what’s coming next?  What would you like to see?

Thanks for starting this thread Ludo, I love this kind of thinking :)

> There are many exciting things being developed and great ideas floating
> around.  For myself, I feel like focusing on “consolidating” what we
> have in the coming weeks.  Here are the areas I hope to focus on (and
> embarking as many of you as possible :-)):
>
>   1. Authentication.  I want to finally provide a standardized mechanism
>      to allow channels to be authenticated upon ‘guix pull’.  “make
>      authenticate” was a first milestone, let’s get it done.  See
>      <https://issues.guix.gnu.org/issue/22883>.

Sounds useful!

>   2. Performance.  There are many things we can improve there, first and
>      foremost: the “Computing derivation” part of ‘guix pull’, Guile’s
>      compiler terrible time and space requirements, further optimizing
>      core operations like ‘package-derivation’, as well as low-level
>      stuff as described at <https://issues.guix.gnu.org/issue/40626>.

I remember saying that the Guix Data Service times various things, and
that it could time the "Computing derivation" bit, but it doesn't store
that data currently. While the performance is dependent on what else is
going on at the same time, it might be interesting to store the data to
see if there's a trend going forward.

>      Related to that is the question of substitute availability, another
>      major hindrance to usability.  We should address this both in the
>      build farm (reducing the
>      time-to-push-to-time-of-substitute-availability, tracking random
>      build failures), and on the client side (can we provide ways for
>      users to pull to a commit that won’t require them to build
>      everything from source, while not compromising on their security?).

Substitute availability as well as building things is something I've
been interested in recently. With your guidance, I think Bayfront is
working more reliably now, although like Berlin there is a disk space
issue to be worked on.

The Guix Data Service has some awareness of substitute availablility,
but the data is not as up to date as it could be, and there isn't a
great way of looking at it either. I hope to address that over the next
few weeks.

In terms of building things, I'm working on making it possible to use
the "Guix Build Coordinator" thing I've been building [1] to provide
substitues, and I'll hopefully send an email out about that soon.

1: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html

>   3. G-exps.  We should really finish the migration to gexps, as in the
>      ‘wip-build-system-gexp’ branch, and adjust our APIs accordingly.
>
>   4. User interface.  Let’s get our act together with ‘guix shell’ and
>      ‘guix run-script’, and let’s address other annoyances that
>      newcomers keep stumbling upon!
>
> Thoughts?

The other things on my mind are:

Patch submission and review, some information in [2]. I at least want to
do some more work on the Guix Data Service to make the comparisons more
useful for reviewing patches, as well as seeing if I can get the Guix
Build Coordinator to build the affected things, and report back. Then
there are all the issues around submitting packages, like how do I know
if someone is working on this already, or how do I know this hasn't been
updated on core-updates, and how can actually submitting patches be made
easier.

2: https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00476.html

More sophisticated policies on accepting substitutes, this means being
able to say things like "I trust substitutes if there signed by all
these keys/entities". While most of the work on reproducible builds has
been on getting things to build reproducibly, Guix could lead the way on
doing the user facing side of this. A majority of the substitutes for
x86_64-linux match between Bayfront and Berlin, so we already have a
minimially viable "rebuilder" setup to actually make use of this.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Medium-term road map

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> 2. Search on all the packages included in Guix since the Big Bang.
>
> It is difficult to find the Guix commit where one package goes in and
> the commit where it goes out. The Guix Data Service (GDS) helps a lot
> for that! But AFAIK, it is not possible from the CLI and I do not find
> it handy when I need it (just because I need to open a webbrowser,
> etc.).
>
> IMHO, there is 3 questions:
>  - how to build such full historical index locally? Even if it is expensive
>  - how to fetch it from GDS? Or any other substitue?

It's far from "the Big Bang", but the data.guix.gnu.org instance of the
Guix Data Service has data going back to the start of 2019, with some
big gaps that are still being filled in. Once that's happened, it'll be
feasible to start looking at going further back.

On your point about fetching data, the Guix Data Service can return data
in JSON, and it's been mentioned that there could be something like the
(guix ci) module, but to access the Guix Data Service. Once the data is
available, it would at least be feasible to use that for a command line
interface.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Medium-term road map

[Andy Wingo]

On Sat 25 Apr 2020 15:37, Ludovic Courtès <ludo@gnu.org> writes:

>   2. Performance.  There are many things we can improve there, first and
>      foremost: the “Computing derivation” part of ‘guix pull’, Guile’s
>      compiler terrible time and space requirements

I think I have a solution here, as discussed on IRC.  Basic idea is to
make a direct compiler from Tree-IL to bytecode, skipping the CPS step.
The result won't be optimal in terms of generated code quality (I
estimate on average 2x slowdown) but it will be very cheap to produce (I
estimate 10x improvement in time, and going from O(n log n) space to
O(n), with better constant factors.  We'd use this compiler for -O0 or
maybe even -O1 code.

Got to spend a few days working it up tho!

Andy

Re: Medium-term road map

[Christopher Lemmer Webber]

Andy Wingo writes:

> On Sat 25 Apr 2020 15:37, Ludovic Courtès <ludo@gnu.org> writes:
>
>>   2. Performance.  There are many things we can improve there, first and
>>      foremost: the “Computing derivation” part of ‘guix pull’, Guile’s
>>      compiler terrible time and space requirements
>
> I think I have a solution here, as discussed on IRC.  Basic idea is to
> make a direct compiler from Tree-IL to bytecode, skipping the CPS step.
> The result won't be optimal in terms of generated code quality (I
> estimate on average 2x slowdown) but it will be very cheap to produce (I
> estimate 10x improvement in time, and going from O(n log n) space to
> O(n), with better constant factors.  We'd use this compiler for -O0 or
> maybe even -O1 code.
>
> Got to spend a few days working it up tho!
>
> Andy

That sound like a nice path to an improvement!

Re: Medium-term road map

[Joshua Branson]

Andy Wingo <wingo@igalia.com> writes:

>
> I think I have a solution here, as discussed on IRC.  Basic idea is to
> make a direct compiler from Tree-IL to bytecode, skipping the CPS step.
> The result won't be optimal in terms of generated code quality (I
> estimate on average 2x slowdown) but it will be very cheap to produce (I
> estimate 10x improvement in time, and going from O(n log n) space to
> O(n), with better constant factors.  We'd use this compiler for -O0 or
> maybe even -O1 code.
>
> Got to spend a few days working it up tho!
>
> Andy

I love how Andy can say something like "yeah I've thought of this
awesome idea that will result in a 10x performance gain"...and "oh yeah,
it'll only take me a few days to do it."  What's awesome is he is such a
genius, that he'll probably pull it off.  :)

--
Joshua Branson
Sent from Emacs and Gnus

Re: Medium-term road map

[Ludovic Courtès]

Hey!

Christopher Baines <mail@cbaines.net> skribis:

> zimoun <zimon.toutoune@gmail.com> writes:
>
>> 2. Search on all the packages included in Guix since the Big Bang.
>>
>> It is difficult to find the Guix commit where one package goes in and
>> the commit where it goes out. The Guix Data Service (GDS) helps a lot
>> for that! But AFAIK, it is not possible from the CLI and I do not find
>> it handy when I need it (just because I need to open a webbrowser,
>> etc.).
>>
>> IMHO, there is 3 questions:
>>  - how to build such full historical index locally? Even if it is expensive
>>  - how to fetch it from GDS? Or any other substitue?
>
> It's far from "the Big Bang", but the data.guix.gnu.org instance of the
> Guix Data Service has data going back to the start of 2019, with some
> big gaps that are still being filled in. Once that's happened, it'll be
> feasible to start looking at going further back.
>
> On your point about fetching data, the Guix Data Service can return data
> in JSON, and it's been mentioned that there could be something like the
> (guix ci) module, but to access the Guix Data Service. Once the data is
> available, it would at least be feasible to use that for a command line
> interface.

Yup, that’s the most practical approach.

We could write a program to build a database locally (I very much like
the idea of “substitutes” viewed as an optimization compared to local
computation) but that would be expensive, although it could be
incremental.

I suppose we could reuse relevant bits of the Data Service as a library.

Hmm thinking about it, by building the package-cache derivation of each
Guix revision, one already gets a lot of information, and that is
substitutable.  Food for thought!

Ludo’.

Re: Medium-term road map

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> We released 1.1.0, but what’s coming next?  What would you like to see?
>
> Thanks for starting this thread Ludo, I love this kind of thinking :)

Though I realize now that I have an email problem that introduces a
significant delay in my response time, and I apologize for it!

>>   2. Performance.  There are many things we can improve there, first and
>>      foremost: the “Computing derivation” part of ‘guix pull’, Guile’s
>>      compiler terrible time and space requirements, further optimizing
>>      core operations like ‘package-derivation’, as well as low-level
>>      stuff as described at <https://issues.guix.gnu.org/issue/40626>.
>
> I remember saying that the Guix Data Service times various things, and
> that it could time the "Computing derivation" bit, but it doesn't store
> that data currently. While the performance is dependent on what else is
> going on at the same time, it might be interesting to store the data to
> see if there's a trend going forward.

Yup!

Performance monitoring in general would be nice, and could have been a
subject for GSoC/Outreachy.

> The other things on my mind are:
>
> Patch submission and review, some information in [2]. I at least want to
> do some more work on the Guix Data Service to make the comparisons more
> useful for reviewing patches, as well as seeing if I can get the Guix
> Build Coordinator to build the affected things, and report back. Then
> there are all the issues around submitting packages, like how do I know
> if someone is working on this already, or how do I know this hasn't been
> updated on core-updates, and how can actually submitting patches be made
> easier.
>
> 2: https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00476.html

Would be nice.

> More sophisticated policies on accepting substitutes, this means being
> able to say things like "I trust substitutes if there signed by all
> these keys/entities". While most of the work on reproducible builds has
> been on getting things to build reproducibly, Guix could lead the way on
> doing the user facing side of this. A majority of the substitutes for
> x86_64-linux match between Bayfront and Berlin, so we already have a
> minimially viable "rebuilder" setup to actually make use of this.

+1!

Thanks,
Ludo’.

Re: Medium-term road map

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>>   2. Performance.  
>
> At a higher level, we can work on parallelization of builds and
> downloads (all combinations).
>
> Many improvements were suggested in this thread:
>
>   https://lists.gnu.org/archive/html/guix-devel/2019-10/msg00694.html
>
> Julien sent a patch to enable parallel downloads:
>
>   https://issues.guix.info/issue/39728
>
> This could significantly improve the user experience.

Yes definitely, thanks for the reminder.  I hope Julien will follow up
on the issue above.  :-)

Ludo’.

Re: Medium-term road map

[Ludovic Courtès]

Hi,

Jack Hill <jackhill@jackhill.us> skribis:

> * Continued development of guix deploy. Figuring out how to deploy
> secrets to remote machines would be great.

Right.  I think ‘guix deploy’ development will be fueled by people using
it to do things beyond the simple use cases (the build farm,
single-machine deployments) and reporting on what’s missing.

> * Accessibility for the installer. I don't know where we stand
> currently, but it would be great to know and make improvements.

Yup!

> * Sandboxing of applications. What would a Guix System with SELinux
> enforcement look like?

Related to that there were discussions on taking advantage of user
namespaces and ‘call-with-container’:

  https://lists.gnu.org/archive/html/help-guix/2018-01/msg00118.html

I’m sure we could provide a nice answer.

> * GDM and GNOME-Shell Wayland sessions.
>
> * Continued improvements to bootstrapping and GNU Hurd efforts.

+1!

Ludo’.

Re: Medium-term road map

[zimoun]

Dear,

On Sun, 3 May 2020 at 22:07, Ludovic Courtès <ludo@gnu.org> wrote:

> We could write a program to build a database locally (I very much like
> the idea of “substitutes” viewed as an optimization compared to local
> computation) but that would be expensive, although it could be
> incremental.

I agree that the "substitutes" mechanism seems the right approach.
Because it allow the user to build the something locally (if they do
not trust the server or use Guix for another set packages etc.).

However, the "issue" with substitutes when pulling is that they are
rarely available.  Because Guix moves faster than the build-farm
builds; I mean my laptop is building locally -- more more than often
-- guix-manual, guix-cli, guix-extra, guix-cli-modules,
guix-extra-modules, guix-packages-base, etc. and some expensive ones ,
e.g., guix-system.


> I suppose we could reuse relevant bits of the Data Service as a library.
>
> Hmm thinking about it, by building the package-cache derivation of each
> Guix revision, one already gets a lot of information, and that is
> substitutable.  Food for thought!

Yes.  What I have in mind is: substitute a big index once and then
update it locally (incrementally, similarly as 'news' does).
The questions are:
  what does that? pull or explicit command or something else?
  how to deal with channels?


All the best,
simon

Re: Medium-term road map

[raingloom]

On Sat, 25 Apr 2020 15:37:44 +0200
Ludovic Courtès <ludo@gnu.org> wrote:

> Hello Guix!
> 
> We released 1.1.0, but what’s coming next?  What would you like to
> see?
> 
> There are many exciting things being developed and great ideas
> floating around.  For myself, I feel like focusing on “consolidating”
> what we have in the coming weeks.  Here are the areas I hope to focus
> on (and embarking as many of you as possible :-)):
> 
>   1. Authentication.  I want to finally provide a standardized
> mechanism to allow channels to be authenticated upon ‘guix pull’.
> “make authenticate” was a first milestone, let’s get it done.  See
>      <https://issues.guix.gnu.org/issue/22883>.
> 
>   2. Performance.  There are many things we can improve there, first
> and foremost: the “Computing derivation” part of ‘guix pull’, Guile’s
>      compiler terrible time and space requirements, further optimizing
>      core operations like ‘package-derivation’, as well as low-level
>      stuff as described at <https://issues.guix.gnu.org/issue/40626>.
> 
>      Related to that is the question of substitute availability,
> another major hindrance to usability.  We should address this both in
> the build farm (reducing the
>      time-to-push-to-time-of-substitute-availability, tracking random
>      build failures), and on the client side (can we provide ways for
>      users to pull to a commit that won’t require them to build
>      everything from source, while not compromising on their
> security?).
> 
>   3. G-exps.  We should really finish the migration to gexps, as in
> the ‘wip-build-system-gexp’ branch, and adjust our APIs accordingly.
> 
>   4. User interface.  Let’s get our act together with ‘guix shell’ and
>      ‘guix run-script’, and let’s address other annoyances that
>      newcomers keep stumbling upon!
> 
> Thoughts?
> 
> Happy hacking!  :-)
> 
> Ludo’.
> 

I would love to see better tools for debugging boot time issues and a
way to not have to rewrite the entire target directory during a
repeated `guix system init` that only changes a few things. I don't
want to waste my SSD's write cycles. And it's also pointlessly slow.

Right now the debugging experience in the Guile rescue shell is...
ugh. Not great. GRUB 0.9 has a CLI that's easier to navigate.
What I'd like to see:
 - tab autocomplete
 - recovery from errors
The latter is quite hard, because certain steps don't work well when
run multiple times. For example, if you want to re-run the procedure
that mounts essential file systems, it will complain that /dev is
already mounted, instead of just realizing that it can skip mounting it.

Something equivalent to `arch-chroot` would also be nice, again for
debugging. I'm sure there is some way to make modifications to a store
on a different device, but it's not easy to find.

I've written something simple that adresses the overwriting issue, but
since it's a modification to Guix itself, it triggered a full rebuild
of the world, and I did not have the patience to sit through that, so I
just installed a more minimal system instead.

Re: [EXT] Re: Medium-term road map

[Thompson, David]

On Sat, Apr 25, 2020 at 5:38 PM Jack Hill <jackhill@jackhill.us> wrote:
>
> * Continued development of guix deploy. Figuring out how to deploy secrets
> to remote machines would be great.

I used to think this was a problem that guix deploy had to deal with
but after many years doing devops full-time I no longer think this is
a concern. Industry best practice is to use a secrets management
service to fetch secrets at application boot time.  For example, you
could write a shepherd service that downloads and installs an SSH host
key from AWS Secrets Manager (or a self-hosted free tool or another
cloud provider's service, you get the idea) before the SSH service
starts.  In my experience, every application requires a slightly
different strategy: Maybe you need to put a key into a specific file,
maybe you need to set environment variables, maybe you need to
templatize the config file, etc. There's no single general solution to
the problem, but I strongly the believe that the guix client that is
doing the deployment should never access such secrets.

Long story short: Guix need not worry about this.

- Dave

Re: [EXT] Re: Medium-term road map

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]

On Wed, May 06, 2020 at 01:03:39PM -0400, Thompson, David wrote:
> On Sat, Apr 25, 2020 at 5:38 PM Jack Hill <jackhill@jackhill.us> wrote:
> >
> > * Continued development of guix deploy. Figuring out how to deploy secrets
> > to remote machines would be great.
> 
> I used to think this was a problem that guix deploy had to deal with
> but after many years doing devops full-time I no longer think this is
> a concern. Industry best practice is to use a secrets management
> service to fetch secrets at application boot time.  For example, you
> could write a shepherd service that downloads and installs an SSH host
> key from AWS Secrets Manager (or a self-hosted free tool or another
> cloud provider's service, you get the idea) before the SSH service
> starts.  In my experience, every application requires a slightly
> different strategy: Maybe you need to put a key into a specific file,
> maybe you need to set environment variables, maybe you need to
> templatize the config file, etc. There's no single general solution to
> the problem, but I strongly the believe that the guix client that is
> doing the deployment should never access such secrets.
> 
> Long story short: Guix need not worry about this.
> 
> - Dave
> 

For the SSH example, imagine a one-shot service that fetches a private
and public keypair¹, replaces the pair already inside /etc/ssh and then
restarts the openssh service.

¹ Using magic or ssh or from a thumbdrive, etc

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [EXT] Re: Medium-term road map

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]

Dave,

On Wed, 6 May 2020, Thompson, David wrote:

> On Sat, Apr 25, 2020 at 5:38 PM Jack Hill <jackhill@jackhill.us> wrote:
>>
>> * Continued development of guix deploy. Figuring out how to deploy secrets
>> to remote machines would be great.
>
> I used to think this was a problem that guix deploy had to deal with
> but after many years doing devops full-time I no longer think this is
> a concern. Industry best practice is to use a secrets management
> service to fetch secrets at application boot time.  For example, you
> could write a shepherd service that downloads and installs an SSH host
> key from AWS Secrets Manager (or a self-hosted free tool or another
> cloud provider's service, you get the idea) before the SSH service
> starts.  In my experience, every application requires a slightly
> different strategy: Maybe you need to put a key into a specific file,
> maybe you need to set environment variables, maybe you need to
> templatize the config file, etc. There's no single general solution to
> the problem, but I strongly the believe that the guix client that is
> doing the deployment should never access such secrets.

Good idea, thanks for sharing. That sounds like a reasonable path forward 
to me. However, …

> Long story short: Guix need not worry about this.

I think we may want to do some work in Guix to support this workflow 
conveniently. That work could include having a secrets management service, 
bootstrapping new hosts for access to the service, or writing system 
services that can be easily configured for different secret management at 
deploy time. It's fun to think about what we could do, but as Ludo’ 
suggested elsewhere in the thread, we'll find out by trying to deploy more 
hosts with more complex configurations. I hope to be able to do so soon.

Best,
Jack

Re: [EXT] Re: [EXT] Re: Medium-term road map

[Thompson, David]

On Wed, May 6, 2020 at 3:46 PM Jack Hill <jackhill@jackhill.us> wrote:
>
> > Long story short: Guix need not worry about this.
>
> I think we may want to do some work in Guix to support this workflow
> conveniently. That work could include having a secrets management service,
> bootstrapping new hosts for access to the service, or writing system
> services that can be easily configured for different secret management at
> deploy time. It's fun to think about what we could do, but as Ludo’
> suggested elsewhere in the thread, we'll find out by trying to deploy more
> hosts with more complex configurations. I hope to be able to do so soon.

To that end, I think a good starting place would be to research the
available free secrets management applications (my knowledge is a few
years out of date), package it, and write a shepherd service for it.
From there, we could see what additional integration would be useful
for clients (your other servers being clients of the secrets
management server.)  I don't know if this would actually work, but I
can picture a world where service configuration objects are aware of
secret fields (some new Scheme data type) and will arrange to lazily
generate config files in a just-in-time fashion on the server when
shepherd starts the service.  Sounds like a real fun project, IMO!

Okay, so I take it back: Guix *should* worry about this, but in a very
specific way that is orders of magnitude better than every other
configuration management system out there, just like the rest of Guix.
:)

- Dave