Guix on macOS

Guix on macOS

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

Hi Guix,

I want to get Guix working on macOS.  I recently had a need to do this,
and I was sad to find that although Nix works on macOS, Guix isn't quite
there yet.  The manual makes it sound like this should be fairly
straightforward, and I intend to give it a shot [1].  But before I
begin, I wanted to know: has anyone done this already?  Is there
interest?  I've checked the email lists, and I didn't find much
discussion about this.

I know a lot of developers who use macOS as their primary workstation,
and most of them use a combination of Homebrew [2] and manual
installation for package management.  It'd be great if Guix were easy to
use on macOS!  Not only is it the best package manager (of course! :-)),
but it would be a great way to encourage the use and growth of free
software among this group of people.

Additionally, if you're interested in helping out, please let me know.
To facilitate the work, I'm temporarily renting a Mac from MacStadium
[3] (it's just a weak little Mac Mini [4] for now), and it would be
trivial to give interested parties administrative access (via VNC and
SSH) to facilitate the work.

Footnotes: 
[1]  info '(guix) Porting'

[2]  https://brew.sh/

[3]  https://www.macstadium.com/

[4]  https://www.macstadium.com/mac-mini/

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Guix on macOS

[Konrad Hinsen]

On 12/10/2017 05:29, Chris Marusich wrote:

> But before I
> begin, I wanted to know: has anyone done this already?  Is there
> interest?

Interest, yes, from at least one more person: me. But I suspect this is 
a really big effort, in particular to boostrap the whole system based on 
Apple's XCode, which is all you can count on initially on a Mac. In 
other words, the issue is not porting the package manager but porting 
the package definitions.

Konrad.

Re: Guix on macOS

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> I want to get Guix working on macOS.  I recently had a need to do this,
> and I was sad to find that although Nix works on macOS, Guix isn't quite
> there yet.  The manual makes it sound like this should be fairly
> straightforward, and I intend to give it a shot [1].  But before I
> begin, I wanted to know: has anyone done this already?  Is there
> interest?  I've checked the email lists, and I didn't find much
> discussion about this.

First of all, it’s never been a goal of Guix to run on non-GNU systems.
Now, I have nothing against it in principle, as long as (1) this can be
achieved in a maintainable way, and (2) the targeted user-land software
is free and buildable from source.

I suspect macOS fails criterion #2.  Back in the day (not sure if that’s
still the case), Nix would bootstrap using the system’s compiler and C
library (which meant that things were likely to break in subtle ways on
macOS upgrades.)

As for criterion #1, to me, that pretty much means sticking to the GNU
libc.  From my experience on Nixpkgs, having to deal with different C
libraries is a real burden.  It also leads to a situation where you have
second-class systems because they use an alternate libc and it’s not
uncommon for packages to fail to build against that libc.  To put it
differently: it’s already difficult enough to have *one* OS working.

I’m afraid this is not the answer you were looking for.  WDYT?

Ludo’.

Re: Guix on macOS

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1664 bytes --]

On Wed, 11 Oct 2017 20:29:57 -0700
Chris Marusich <cmmarusich@gmail.com> wrote:

> Hi Guix,
> 
> I want to get Guix working on macOS.  I recently had a need to do
> this, and I was sad to find that although Nix works on macOS, Guix
> isn't quite there yet.  The manual makes it sound like this should be
> fairly straightforward, and I intend to give it a shot [1].  But
> before I begin, I wanted to know: has anyone done this already?  Is
> there interest?  I've checked the email lists, and I didn't find much
> discussion about this.

Hey Chris,

I'm interested! My workplace hands out laptops running MacOS, and while
I don't use one, and most people using them do development in a Ubuntu
VM, I'd love to have the option to help my colleagues use Guix on their
machines.

> I know a lot of developers who use macOS as their primary workstation,
> and most of them use a combination of Homebrew [2] and manual
> installation for package management.  It'd be great if Guix were easy
> to use on macOS!  Not only is it the best package manager (of
> course! :-)), but it would be a great way to encourage the use and
> growth of free software among this group of people.
> 
> Additionally, if you're interested in helping out, please let me know.
> To facilitate the work, I'm temporarily renting a Mac from MacStadium
> [3] (it's just a weak little Mac Mini [4] for now), and it would be
> trivial to give interested parties administrative access (via VNC and
> SSH) to facilitate the work.

I'd be interested in helping out, but I have never seriously used
MacOS, so this would be quite a learning curve.

Thanks,

Chris

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Guix on macOS

[Christopher Allan Webber]

Ludovic Courtès writes:

> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> I want to get Guix working on macOS.  I recently had a need to do this,
>> and I was sad to find that although Nix works on macOS, Guix isn't quite
>> there yet.  The manual makes it sound like this should be fairly
>> straightforward, and I intend to give it a shot [1].  But before I
>> begin, I wanted to know: has anyone done this already?  Is there
>> interest?  I've checked the email lists, and I didn't find much
>> discussion about this.
>
> First of all, it’s never been a goal of Guix to run on non-GNU systems.
> Now, I have nothing against it in principle, as long as (1) this can be
> achieved in a maintainable way, and (2) the targeted user-land software
> is free and buildable from source.
>
> I suspect macOS fails criterion #2.  Back in the day (not sure if that’s
> still the case), Nix would bootstrap using the system’s compiler and C
> library (which meant that things were likely to break in subtle ways on
> macOS upgrades.)
>
> As for criterion #1, to me, that pretty much means sticking to the GNU
> libc.  From my experience on Nixpkgs, having to deal with different C
> libraries is a real burden.  It also leads to a situation where you have
> second-class systems because they use an alternate libc and it’s not
> uncommon for packages to fail to build against that libc.  To put it
> differently: it’s already difficult enough to have *one* OS working.
>
> I’m afraid this is not the answer you were looking for.  WDYT?
>
> Ludo’.

Is there a way to maybe run Guix in some sort of namespaced or some
variant of "virtualized" or "contained" way that we could recommend for
OSX users, without having to bend over backwards to accomodate a
different libc and etc?

 - Chris

Re: Guix on macOS

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> First of all, it’s never been a goal of Guix to run on non-GNU systems.
> Now, I have nothing against it in principle, as long as (1) this can be
> achieved in a maintainable way, and (2) the targeted user-land software
> is free and buildable from source.
>
> I suspect macOS fails criterion #2.  Back in the day (not sure if that’s
> still the case), Nix would bootstrap using the system’s compiler and C
> library (which meant that things were likely to break in subtle ways on
> macOS upgrades.)
>
> As for criterion #1, to me, that pretty much means sticking to the GNU
> libc.  From my experience on Nixpkgs, having to deal with different C
> libraries is a real burden.  It also leads to a situation where you have
> second-class systems because they use an alternate libc and it’s not
> uncommon for packages to fail to build against that libc.  To put it
> differently: it’s already difficult enough to have *one* OS working.

I’m quoting all of what Ludo wrote here, because I absolutely agree.

I investigated this in 2014 and once again in 2015 as I had access to
some Macs at the office, but the very fact that there is no legal way to
build the software in freedom makes this whole business very
unattractive.  I also looked at PureDarwin and the defunct OpenDarwin,
as well as inofficial ports of the GNU C library to macOS — none of
these roads looked promising as the software is hardly even buildable.

The only way I could see this working is to support a mechanism for
package graph sections, i.e. a way for a user to cheat and “graft” a
section of a package graph onto a binary blob that the user says is
equivalent.  Obviously, that’s crude and we would sanction a hack as an
official misfeature.  We’d lose all guarantees that we’re working hard
to provide.  It would be regrettable to “support” Guix on macOS if that
thing running on macOS hasn’t really much in common with Guix on GNU.

Christopher wrote this:

> Is there a way to maybe run Guix in some sort of namespaced or some
> variant of "virtualized" or "contained" way that we could recommend for
> OSX users, without having to bend over backwards to accomodate a
> different libc and etc?

In order to use the Hypervisor framework that macOS 10.10 and higher
provide we would need to use Xcode, so we couldn’t even build a tool
like that without relying on proprietary software.  But we don’t have to
build a tool like that, because it already exists in the form of virtual
machine applications (or even Docker for Mac).

The best we could do on macOS is to run applications in a GNU virtual
machine and try hard to make them blend in.  That’s not really
appealing, neither technically nor practically, in my opinion.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Guix on macOS

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4860 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> First of all, it’s never been a goal of Guix to run on non-GNU systems.
> Now, I have nothing against it in principle, as long as (1) this can be
> achieved in a maintainable way, and (2) the targeted user-land software
> is free and buildable from source.

I understand, and I agree with your criteria.

I don't want to use Guix on macOS to package, promote, or make it easy
to use non-free software.  I also don't want to port Guix to macOS in a
way that's difficult to maintain.

I want users and developers who are currently using macOS to be able to
easily use Guix and all the free software that it provides.  To borrow
the language used in the "GNU Emacs FAQ for MS Windows", I hope that the
experience of using GNU Guix on macOS will give programmers a taste of
freedom, and that this will later inspire them to move to a free
operating system such as GNU/Linux (hopefully one that uses Guix, like
GuixSD!) [1].  I think this is a reasonable motivation.

> I suspect macOS fails criterion #2.  Back in the day (not sure if that’s
> still the case), Nix would bootstrap using the system’s compiler and C
> library (which meant that things were likely to break in subtle ways on
> macOS upgrades.)
>
> As for criterion #1, to me, that pretty much means sticking to the GNU
> libc.  From my experience on Nixpkgs, having to deal with different C
> libraries is a real burden.  It also leads to a situation where you have
> second-class systems because they use an alternate libc and it’s not
> uncommon for packages to fail to build against that libc.  To put it
> differently: it’s already difficult enough to have *one* OS working.

I haven't yet looked at how Nix bootstraps on macOS.  I'll do that and
update this thread if I find any useful information to share.

Currently, I hope that we can get Guix working on macOS via a plan like
the following:

  1) On an x86_64-linux GuixSD system, use Guix to cross-build Guix for
     the x86_64-darwin target [2].  We would use GNU libc.

  2) Install the output of (1) on a macOS system, following a procedure
     similar to the one in the manual for binary installation ((guix)
     Binary Installation).

Is this plan feasible?  Please understand that I'm genuinely curious,
and I just want to help.  I might be missing some information that's
obvious to others.  If there are gaps in my understanding, please help
me to fill them.

I imagine that there might be other ways to get Guix working on macOS.
Here are some possibilities that I've thought of or that others have
already mentioned:

  * Compile Guix (and its bootstrap binaries, I guess?) natively for
    x86_64-darwin on macOS using Xcode, etc.  This seems undesirable for
    a lot of reasons.  Some reasons I can think of are: the build
    process would rely on non-free software, it probably wouldn't be
    easily reproducible, and it would probably place a significant
    additional burden on the Guix maintainers.  I suppose the only
    saving grace in this case might be that once we had a working Guix
    (with bootstrap binaries) for macOS, call it G1, we might be in a
    position to use G1 to reproducibly build Guix (with GNU libc) on
    macOS going forward.

  * Run Guix compiled for x86_64-linux on macOS using some kind of a
    shim layer.  This is pretty vague, but not without precedent:
    consider virtual machines, WINE, and similar technologies.  This
    seems undesirable for a lot of reasons.  Some reasons I can think of
    are: any solution like this would probably be fragile, and as far as
    I know there is no turn-key solution for running ELF executables on
    macOS, so we'd have to build our own, and building our own would
    entail the same kinds of problems as mentioned in the previous
    bullet point.

I'd love to hear any other ideas anyone might have!  Cross-compiling
seems like one possible way forward; however, I don't know if
cross-compiling is feasible.  I hope it is.

> I’m afraid this is not the answer you were looking for.  WDYT?

This is exactly what I was hoping for: the start of a discussion!  If we
can get Guix working on macOS while meeting the criteria you mentioned,
I think it'd be great for the reasons I mentioned at the start of this
email.  If it isn't feasible, then I'd like to understand why.

Footnotes: 
[1]  https://www.gnu.org/software/emacs/manual/html_node/efaq-w32/Why-Emacs-on-Windows.html#Why-Emacs-on-Windows

[2] The string "x86_64-darwin" is used by Nix (it also shows up in some
Guix files), and it seems to be synonymous with "x86_64-apple-darwin".
If you run a command like "gcc -dumpmachine" on a recent version of
macOS, you'll see something like "x86_64-apple-darwin16.7.0".

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Guix on macOS

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

> Currently, I hope that we can get Guix working on macOS via a plan like
> the following:
>
>   1) On an x86_64-linux GuixSD system, use Guix to cross-build Guix for
>      the x86_64-darwin target [2].  We would use GNU libc.
>
>   2) Install the output of (1) on a macOS system, following a procedure
>      similar to the one in the manual for binary installation ((guix)
>      Binary Installation).
>
> Is this plan feasible?

I don’t think it is.   You wrote:

> [2] The string "x86_64-darwin" is used by Nix (it also shows up in some
> Guix files), and it seems to be synonymous with "x86_64-apple-darwin".
> If you run a command like "gcc -dumpmachine" on a recent version of
> macOS, you'll see something like "x86_64-apple-darwin16.7.0".

AFAIK on macOS “gcc” and “g++” are just aliases for clang.  In order to
build a version of GCC that can build binaries for macOS you need XCode
or parts of it (the libraries it provides).

There is no usable port of the GNU C library for Darwin, so you must
link with the non-free C library provided by XCode.  This means that
step 1 won’t work.

The C library abstracts away the peculiarities of the kernel.  Without a
maintained port of the GNU C library for Darwin I see no way to build
software targeting macOS without XCode.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Guix on macOS

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> First of all, it’s never been a goal of Guix to run on non-GNU systems.
>> Now, I have nothing against it in principle, as long as (1) this can be
>> achieved in a maintainable way, and (2) the targeted user-land software
>> is free and buildable from source.
>
> I understand, and I agree with your criteria.
>
> I don't want to use Guix on macOS to package, promote, or make it easy
> to use non-free software.  I also don't want to port Guix to macOS in a
> way that's difficult to maintain.
>
> I want users and developers who are currently using macOS to be able to
> easily use Guix and all the free software that it provides.  To borrow
> the language used in the "GNU Emacs FAQ for MS Windows", I hope that the
> experience of using GNU Guix on macOS will give programmers a taste of
> freedom, and that this will later inspire them to move to a free
> operating system such as GNU/Linux (hopefully one that uses Guix, like
> GuixSD!) [1].  I think this is a reasonable motivation.

Understood.

At the same time, one could hope that, if freedom is not enough, the
nifty features of GuixSD, GNOME, the GNU toolchain, etc. would be enough
of an incentive to switch.  But hey, it’s complicated!

> Currently, I hope that we can get Guix working on macOS via a plan like
> the following:
>
>   1) On an x86_64-linux GuixSD system, use Guix to cross-build Guix for
>      the x86_64-darwin target [2].  We would use GNU libc.
>
>   2) Install the output of (1) on a macOS system, following a procedure
>      similar to the one in the manual for binary installation ((guix)
>      Binary Installation).
>
> Is this plan feasible? 

In theory, yes.  That’s roughly the plan outlined at
<https://www.gnu.org/software/guix/manual/html_node/Porting.html>.

Now, glibc proper only supports Linux and the Hurd.  Debian has a port
of glibc to the kernel of FreeBSD (“kFreeBSD”).  But AFAIK, these are
the only working ports of glibc.  Ricardo mentioned a glibc port to
Darwin (or XNU?), but I suspect that was very experimental no?

So your best bet would be to cross-compile using the macOS libc, similar
to what Jan did with MinGW cross-compilation support.  But again, this
is assuming that the macOS libc is free, and that both that libc and the
GNU toolchain support cross-compilation.  I don’t know if this is the
case.

> Please understand that I'm genuinely curious, and I just want to help.
> I might be missing some information that's obvious to others.  If
> there are gaps in my understanding, please help me to fill them.

Sure, I understand.  It’s a perfectly valid question to ask!

> I imagine that there might be other ways to get Guix working on macOS.
> Here are some possibilities that I've thought of or that others have
> already mentioned:
>
>   * Compile Guix (and its bootstrap binaries, I guess?) natively for
>     x86_64-darwin on macOS using Xcode, etc.  This seems undesirable for
>     a lot of reasons.  Some reasons I can think of are: the build
>     process would rely on non-free software, it probably wouldn't be
>     easily reproducible, and it would probably place a significant
>     additional burden on the Guix maintainers.  I suppose the only
>     saving grace in this case might be that once we had a working Guix
>     (with bootstrap binaries) for macOS, call it G1, we might be in a
>     position to use G1 to reproducibly build Guix (with GNU libc) on
>     macOS going forward.

I agree that this is undesirable.  :-)

>   * Run Guix compiled for x86_64-linux on macOS using some kind of a
>     shim layer.  This is pretty vague, but not without precedent:
>     consider virtual machines, WINE, and similar technologies.  This
>     seems undesirable for a lot of reasons.  Some reasons I can think of
>     are: any solution like this would probably be fragile, and as far as
>     I know there is no turn-key solution for running ELF executables on
>     macOS, so we'd have to build our own, and building our own would
>     entail the same kinds of problems as mentioned in the previous
>     bullet point.

Windows recently gained an in-kernel Linux syscall emulation, which
means that (GNU/)Linux binaries can run unmodified on Windows.

If macOS had a similar feature, that’d be perfect: we wouldn’t have
anything to do.  Perhaps Docker-for-Mac actually provides something
close to that?  I really don’t know.

>> I’m afraid this is not the answer you were looking for.  WDYT?
>
> This is exactly what I was hoping for: the start of a discussion!  If we
> can get Guix working on macOS while meeting the criteria you mentioned,
> I think it'd be great for the reasons I mentioned at the start of this
> email.  If it isn't feasible, then I'd like to understand why.

Yes, let’s see!

Thanks,
Ludo’.

Re: Guix on macOS

[Konrad Hinsen]

On 13/10/2017 14:55, Ludovic Courtès wrote:

> If macOS had a similar feature, that’d be perfect: we wouldn’t have
> anything to do.  Perhaps Docker-for-Mac actually provides something
> close to that?  I really don’t know.

Docker for Mac uses the macOS user-space virtualization facilities (via 
xhyve: https://github.com/mist64/xhyve) to run a standard Linux kernel.

Targeting xhyve is an approach worth exploring in my opinion. It looks 
feasible to generate all the required files via "guix system", using 
nothing but tools that are already there. In fact, all it takes is some 
repackaging because xhyve cannot (yet?) boot from a bootable disk image.

Advantage compared to VirtualBox: access to macOS file systems from the 
guest (which VirtualBox can only achieve via non-free extension code).

Advantage compared to qemu: better performance, access to macOS 
filesystems from the guest.

On the other hand, it is not obvious if there is a significant advantage 
compared to just running Guix in a Docker container (which I haven't 
tried yet).

Konrad.

Re: Guix on macOS

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Windows recently gained an in-kernel Linux syscall emulation, which
> means that (GNU/)Linux binaries can run unmodified on Windows.
>
> If macOS had a similar feature, that’d be perfect: we wouldn’t have
> anything to do.  Perhaps Docker-for-Mac actually provides something
> close to that?  I really don’t know.

IIUC, Docker for Mac uses macOS virtualization libraries.  Previously,
it would require an installation of VirtualBox, but this is no longer
the case.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Guix on macOS

[Konrad Hinsen]

On 13/10/2017 14:55, Ludovic Courtès wrote:

> At the same time, one could hope that, if freedom is not enough, the
> nifty features of GuixSD, GNOME, the GNU toolchain, etc. would be enough
> of an incentive to switch.  But hey, it’s complicated!

There are those nasty real-world constraints, indeed. I'd be happy to 
run 100% free software on personal hardware. But my employer restricts 
the hardware I am allowed to buy and requires me to use Microsoft 
Windows (no, LibreOffice won't do). That makes macOS an attractive 
choice, the alternative being a Windows/Linux dual boot system.

A "free software subsystem" inside macOS is definitely attractive for 
me, because I could then run strictly the same software as my 
Linux-using colleagues. With the features of Guix, it would be even better.

Konrad.

Re: Guix on macOS

[Christopher Allan Webber]

Ricardo Wurmus writes:

> Christopher wrote this:
>
>> Is there a way to maybe run Guix in some sort of namespaced or some
>> variant of "virtualized" or "contained" way that we could recommend for
>> OSX users, without having to bend over backwards to accomodate a
>> different libc and etc?
>
> In order to use the Hypervisor framework that macOS 10.10 and higher
> provide we would need to use Xcode, so we couldn’t even build a tool
> like that without relying on proprietary software.  But we don’t have to
> build a tool like that, because it already exists in the form of virtual
> machine applications (or even Docker for Mac).
>
> The best we could do on macOS is to run applications in a GNU virtual
> machine and try hard to make them blend in.  That’s not really
> appealing, neither technically nor practically, in my opinion.

I think someone told me this is exactly how Docker does it, or used to
do it, for quite some time.  So we'd hardly be alone... :)

Re: Guix on macOS

[Christopher Allan Webber]

Ricardo Wurmus writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Windows recently gained an in-kernel Linux syscall emulation, which
>> means that (GNU/)Linux binaries can run unmodified on Windows.
>>
>> If macOS had a similar feature, that’d be perfect: we wouldn’t have
>> anything to do.  Perhaps Docker-for-Mac actually provides something
>> close to that?  I really don’t know.
>
> IIUC, Docker for Mac uses macOS virtualization libraries.  Previously,
> it would require an installation of VirtualBox, but this is no longer
> the case.

Oh, I see you already replied with essentially the same message I
replied to you with... :)

So effectively "if it is (or was) good enough for Docker, why couldn't
this be good enough for us as a path forward?"

Re: Guix on macOS

[Adonay Felipe Nogueira]

Another thing that just came to my mind is the hard-coding of file name
separators instead of using a combination of checkers to see if the
'file-name-separator-string matches the *target* system (as is briefly
told in `info guile', the GNU Guile Scheme infopage). Although I don't
know if macOS uses "\" or "/". If macOS does use "\", besides having to
make all the attempts to change paths lookup for that separator
evaluation, thus potentially causing extra work, this could require a
major rewrite of package definitions and rebuilds.

Besides, I agree that things should be built entirely with free/libre
software. And I don't know if macOS allows us to do that (thanks to,
along other things, the non-existance of community-oriented copyleft
enforcement by the copyright holders of the XNU/BSD kernel projects).

Chris Marusich <cmmarusich@gmail.com> writes:

> Hi Guix,
>
> I want to get Guix working on macOS.  I recently had a need to do this,
> and I was sad to find that although Nix works on macOS, Guix isn't quite
> there yet.  The manual makes it sound like this should be fairly
> straightforward, and I intend to give it a shot [1].  But before I
> begin, I wanted to know: has anyone done this already?  Is there
> interest?  I've checked the email lists, and I didn't find much
> discussion about this.
>
> I know a lot of developers who use macOS as their primary workstation,
> and most of them use a combination of Homebrew [2] and manual
> installation for package management.  It'd be great if Guix were easy to
> use on macOS!  Not only is it the best package manager (of course! :-)),
> but it would be a great way to encourage the use and growth of free
> software among this group of people.
>
> Additionally, if you're interested in helping out, please let me know.
> To facilitate the work, I'm temporarily renting a Mac from MacStadium
> [3] (it's just a weak little Mac Mini [4] for now), and it would be
> trivial to give interested parties administrative access (via VNC and
> SSH) to facilitate the work.
>
> Footnotes: 
> [1]  info '(guix) Porting'
>
> [2]  https://brew.sh/
>
> [3]  https://www.macstadium.com/
>
> [4]  https://www.macstadium.com/mac-mini/

-- 
- https://libreplanet.org/wiki/User:Adfeno
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre. Por favor, veja formas de se comunicar
  instantaneamente comigo no endereço abaixo.
- Contato: https://libreplanet.org/wiki/User:Adfeno#vCard
- Arquivos comuns aceitos (apenas sem DRM): Corel Draw, Microsoft
  Office, MP3, MP4, WMA, WMV.
- Arquivos comuns aceitos e enviados: CSV, GNU Dia, GNU Emacs Org, GNU
  GIMP, Inkscape SVG, JPG, LibreOffice (padrão ODF), OGG, OPUS, PDF
  (apenas sem DRM), PNG, TXT, WEBM.

Re: Guix on macOS

[Adonay Felipe Nogueira]

I hope they get more used to referring to GNU operating system, or
GNU/Linux operating system too.

I find it puzzling that most people call it Linux, the kernel (Linux)
alone isn't an operating system, let alone usable without one. ;)

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> There are those nasty real-world constraints, indeed. I'd be happy to
> run 100% free software on personal hardware. But my employer restricts
> the hardware I am allowed to buy and requires me to use Microsoft
> Windows (no, LibreOffice won't do). That makes macOS an attractive
> choice, the alternative being a Windows/Linux dual boot system.
>
> A "free software subsystem" inside macOS is definitely attractive for
> me, because I could then run strictly the same software as my
> Linux-using colleagues. With the features of Guix, it would be even
> better.
>
> Konrad.

Re: Guix on macOS

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 139 bytes --]

Adonay Felipe Nogueira <adfeno@hyperbola.info> writes:

> Although I don't know if macOS uses "\" or "/".

It uses "/".

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Guix on macOS

[Chris Marusich]

[-- Attachment #1.1: Type: text/plain, Size: 7304 bytes --]

Hi Guix,

I've looked further into the possibility of running Guix on macOS.  I'm
convinced now that running Guix natively on macOS (i.e., compiling it
for x86_64-apple-darwin) isn't a feasible goal at this time.  GNU libc
doesn't target Darwin, so it can't be used as-is to compile programs to
run on macOS, and as Ricardo mentioned, it seems there are currently no
viable ports of GNU libc that target Darwin.  In addition, I checked the
NixPkgs source code and spoke with a Nix developer who helped get Nix
working on macOS, and I can confirm that the following are true:

* In NixPkgs, some impurities are explicitly allowed here and there to
  make Darwin builds succeed.  Specifically, certain macOS system files
  are exposed to and used by the build processes.  This means that not
  all Darwin package builds are pure, and I'm not sure what the
  licensing implications might be.

* Nix's Darwin bootstrap tools are maintained in Nix, but they aren't
  cross-compiled (e.g., from x86_64-linux).  They're built from a
  previous generation of the stdenv for Darwin.  I suspect, but do not
  know for sure, that originally the bootstrap binaries were compiled
  using the usual impure and non-free macOS toolchain.

* Nix doesn't use GNU libc for Darwin software.  Instead, it uses
  libSystem's libc.  In addition, I'm told that Nix relies pretty
  heavily on opensource.apple.com, and I'm told that many of those
  pieces of software don't have clear free software analogs.

For these reasons, it seems infeasible at this time to port Guix to
macOS natively while satisfying the two criteria that Ludo mentioned,
which were: "(1) this can be achieved in a maintainable way, and (2) the
targeted user-land software is free and buildable from source."  That's
unfortunate, but at least I did my homework, and now we know.

In lieu of running Guix natively on macOS, I'm now interested in adding
a "guix system docker-image" feature which would enable us to invoke a
command like "guix system docker-image my-os-config.scm" and receive as
output a docker image containing the GuixSD system defined by
my-os-config.scm.  This seems potentially very useful.  If we had this
feature, then anyone capable of running Docker images could run GuixSD.
This is similar to making a VM image or a regular disk image.  Docker is
popular, so I think it makes sense to make it easy for people to
experiment with GuixSD using Docker, too.

Personally, I'd like to empower people to use GuixSD on macOS in a
docker container to manage installed software in a way that is,
hopefully, almost as good as running Guix natively.  However, I can
imagine other uses for this feature, as well.  If you're a DevOps shop
and you're already heavily invested in the Docker ecosystem, this could
make it possible to deploy GuixSD easily in your environment.  For the
use case of managing software on a laptop, I think it'd be OK to mutate
the state of a GuixSD docker container by installing packages in it.
Hopefully the container can be integrated well enough to make it feel
like you've actually installed Guix on your laptop.  For a production
service, though, I would not want to deploy stateful Docker containers;
I'd prefer to use stateless containers if possible.

If we had the ability to build a docker image from a GuixSD operating
system configuration file, then it could potentially accommodate both
needs: a person could use a stateful GuixSD docker image on their laptop
for day to day tasks, and they could also leverage their company's
existing Docker infrastructure to deploy (hopefully stateless) GuixSD
docker images to production.  Instead of writing a Dockerfile based on a
base image blob that might be difficult to audit, validate, or
reproduce, and instead of compounding the issue by adding "layers" on
top of your base image blob like Docker encourages its users to do,
you'd write a GuixSD operating system configuration file to
declaratively define the service(s) in your GuixSD docker image.  You
would then build precisely the image you want, every time, using Guix.
You would then deploy it using your existing Docker tools.  This is
nothing new for GuixSD users, really, but Docker is popular and being
used today by many companies and individuals, so it might be a good way
to encourage adoption of GuixSD, Guix, the functional software
deployment model, and free software.

I've tried to implement such a feature, but I'm stuck.  In the attached
patch, I've added a command that lets you do "guix system disk-image -t
docker my-config.scm".  I've tried to merge some of the logic for
running a gexp in a linux vm with the logic for building a docker
container, but I can't get it to work.  I can successfully build Guix
with "make -j", but when I run it, I get the following error:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix system disk-image -t docker gnu/system/install.scm
...
Welcome, this is GNU's early boot Guile.
Use '--repl' for an initrd REPL.

loading kernel modules...
[    1.151988] usbcore: registered new interface driver usb-storage
[    1.158506] usbcore: registered new interface driver uas
[    1.169030] hidraw: raw HID events driver (C) Jiri Kosina
[    1.174144] usbcore: registered new interface driver usbhid
[    1.175537] usbhid: USB HID core driver
[    1.221664] isci: Intel(R) C600 SAS Controller Driver - version 1.2.0
[    1.253086] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[    1.282881] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
[    1.312644] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[    1.345843] FS-Cache: Loaded
[    1.362140] 9pnet: Installing 9P2000 support
[    1.366118] 9p: Installing v9fs 9p2000 file system support
[    1.368730] FS-Cache: Netfs '9p' registered for caching
configuring QEMU networking...
loading '/gnu/store/jy509dgcsz82y13fmizv2sqaj90s1vfg-linux-vm-loader'...
ERROR: In procedure dynamic-link:
ERROR: In procedure dynamic-link: file: "/gnu/store/hwygv5jwd47amhp1m67iy3bkvxqjlbhm-libgcrypt-1.8.1/lib/libgcrypt", message: "file not found"

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
GNU Guile 2.2.2
...
--8<---------------cut here---------------end--------------->8---

What's wrong?  How can I fix it?  I'm stuck and need help.

This appears to be an error on the build side, inside of the linux VM.
It seems /gnu/store/...-libgcrypt-1.8.1/lib/libgcrypt.so is not
available to the code running inside the VM.  The shared object file
(with the .so extension) exists in my store on the host side.  As far as
I can tell, I'm correctly adding libgcrypt to the gexp's closure of
inputs, so I don't understand why the error is happening.  Any tips on
how to debug this or fix it would be great.  Part of the reason I'm
stuck might be because I don't really understand how (guix config) is
used or why the (guix docker) module needs to override it.  I just
cargo-culted that part of the code in an attempt to quickly get
something that worked.  I plan to clean up the patch a lot later.

The attached patch should apply cleanly to commit
3b2fa4787938a408fab27ef7b3bc1302b6b6a805.

Thank you,

-- 
Chris

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-Try-to-build-a-GuixSD-docker-image-currently-fails.patch --]
[-- Type: text/x-patch, Size: 14620 bytes --]

From 6f5f43702f10a15f58df386d107d63843f788fe5 Mon Sep 17 00:00:00 2001
From: Chris Marusich <cmmarusich@gmail.com>
Date: Sat, 21 Oct 2017 14:40:58 -0700
Subject: [PATCH] Try to build a GuixSD docker image - currently fails

---
 gnu/build/linux-boot.scm |   5 +-
 gnu/build/vm.scm         |  14 +++--
 gnu/system/vm.scm        | 158 ++++++++++++++++++++++++++++++++++++++++-------
 guix/docker.scm          |   6 +-
 guix/scripts/pack.scm    |   5 +-
 guix/scripts/system.scm  |   1 +
 6 files changed, 161 insertions(+), 28 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 3712abe91..37da5b217 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -117,8 +117,9 @@ with the given MAJOR number, starting with MINOR."
                        "/")
                    dir))
 
+  (display "just before dev\n")
   (unless (file-exists? (scope "dev"))
-    (mkdir (scope "dev")))
+    (mkdir (pk (scope "dev"))))
 
   ;; Make the device nodes for SCSI disks.
   (make-disk-device-nodes (scope "dev/sda") 8)
@@ -138,6 +139,7 @@ with the given MAJOR number, starting with MINOR."
   (mknod (scope "dev/kmem") 'char-special #o640 (device-number 1 2))
 
   ;; Inputs (used by Xorg.)
+  (display "just before dev/input\n")
   (unless (file-exists? (scope "dev/input"))
     (mkdir (scope "dev/input")))
   (mknod (scope "dev/input/mice") 'char-special #o640 (device-number 13 63))
@@ -171,6 +173,7 @@ with the given MAJOR number, starting with MINOR."
   (chmod (scope "dev/ptmx") #o666)
 
   ;; Create /dev/pts; it will be mounted later, at boot time.
+  (display "just before dev/pts\n")
   (unless (file-exists? (scope "dev/pts"))
     (mkdir (scope "dev/pts")))
 
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 7537f8150..19c47e1ff 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -294,11 +294,14 @@ it, run its initializer, and unmount it."
 (define* (root-partition-initializer #:key (closures '())
                                      copy-closures?
                                      (register-closures? #t)
-                                     system-directory)
+                                     system-directory
+                                     (deduplicate? #t))
   "Return a procedure to initialize a root partition.
 
-If REGISTER-CLOSURES? is true, register all of CLOSURES is the partition's
-store.  If COPY-CLOSURES? is true, copy all of CLOSURES to the partition.
+If REGISTER-CLOSURES? is true, register all of CLOSURES in the partition's
+store.  If DEDUPLICATE? is true, then also deduplicate files common to
+CLOSURES and the rest of the store when registering the closures.  If
+COPY-CLOSURES? is true, copy all of CLOSURES to the partition.
 SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
   (lambda (target)
     (define target-store
@@ -317,13 +320,16 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
       (unless copy-closures?
         ;; XXX: 'guix-register' wants to palpate the things it registers, so
         ;; bind-mount the store on the target.
+        (display "making target store directory\n")
         (mkdir-p target-store)
+        (display "bind-mounting\n")
         (mount (%store-directory) target-store "" MS_BIND))
 
       (display "registering closures...\n")
       (for-each (lambda (closure)
                   (register-closure target
-                                    (string-append "/xchg/" closure)))
+                                    (string-append "/xchg/" closure)
+                                    #:deduplicate? deduplicate?))
                 closures)
       (unless copy-closures?
         (umount target-store)))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 3127b305e..5e3db23b3 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -22,6 +22,7 @@
 
 (define-module (gnu system vm)
   #:use-module (guix config)
+  #:use-module (guix docker)
   #:use-module (guix store)
   #:use-module (guix gexp)
   #:use-module (guix derivations)
@@ -29,13 +30,16 @@
   #:use-module (guix monads)
   #:use-module (guix records)
   #:use-module (guix modules)
+  #:use-module (guix scripts pack)
 
   #:use-module ((gnu build vm)
                 #:select (qemu-command))
   #:use-module (gnu packages base)
+
   #:use-module (gnu packages bootloaders)
   #:use-module (gnu packages cdrom)
   #:use-module (gnu packages guile)
+  #:autoload   (gnu packages gnupg) (libgcrypt)
   #:use-module (gnu packages gawk)
   #:use-module (gnu packages bash)
   #:use-module (gnu packages less)
@@ -349,6 +353,111 @@ the image."
    #:disk-image-format disk-image-format
    #:references-graphs inputs))
 
+(define* (os-docker-image #:key
+                          (name "guixsd-docker-image")
+                          os-drv
+                          (system (%current-system))
+                          (compressor (first %compressors))
+                          localstatedir?
+                          (symlinks '())
+                          (tar tar)
+                          (register-closures? #t)
+                          (inputs '()))
+  "Build a docker image.  OS-DRV is a derivation which builds the
+operating system profile."
+    ;; FIXME: Honor LOCALSTATEDIR?.
+  (define not-config?
+    (match-lambda
+      (('guix 'config) #f)
+      (('guix rest ...) #t)
+      (('gnu rest ...) #t)
+      (rest #f)))
+
+  (define config
+    ;; (guix config) module for consumption by (guix gcrypt).
+    (scheme-file "gcrypt-config.scm"
+                 #~(begin
+                     (define-module (guix config)
+                       #:export (%libgcrypt))
+
+                     ;; XXX: Work around <http://bugs.gnu.org/15602>.
+                     (eval-when (expand load eval)
+                       (define %libgcrypt
+                         #+(file-append libgcrypt "/lib/libgcrypt"))))))
+
+  (define json
+    ;; Pick the guile-json package that corresponds to the Guile used to build
+    ;; derivations.
+    (if (string-prefix? "2.0" (package-version (default-guile)))
+        guile2.0-json
+        guile-json))
+  
+  (let ((name (string-append name ".tar" (compressor-extension compressor))))
+    (define build
+      (with-imported-modules `(,@(source-module-closure '((guix docker)
+                                                          (gnu build vm)
+                                                          (guix build utils)
+                                                          (guix build syscalls))
+                                                      #:select? not-config?)
+                             ((guix config) => ,config))
+      #~(begin
+          ;; Guile-JSON is required by (guix docker).
+          (add-to-load-path
+           (string-append #+json "/share/guile/site/"
+                          (effective-version)))
+         (use-modules (gnu build vm)
+                      (guix build utils)
+                      (guix build syscalls)
+                      (srfi srfi-26)
+                      (ice-9 match)
+                      (guix docker)
+                      (srfi srfi-19))
+
+         (let* ((inputs
+                 '#$(append (list tree parted e2fsprogs dosfstools tar)
+                            (map canonical-package
+                                 (list sed grep coreutils findutils gawk))
+                            (if register-closures? (list guix) '())))
+
+                 ;; This variable is unused but allows us to add INPUTS-TO-COPY
+                 ;; as inputs.
+                 (to-register
+                  '#$(map (match-lambda
+                            ((name thing) thing)
+                            ((name thing output) `(,thing ,output)))
+                          inputs))
+                 (graphs '#$(match inputs
+                              (((names . _) ...)
+                                names)))
+                 (initialize (root-partition-initializer
+                               #:closures graphs
+                               #:copy-closures? #f
+                               #:register-closures? #$register-closures?
+                               #:system-directory #$os-drv
+                               #:deduplicate? #f))
+                 (root "/tmp/root"))
+
+           (display "before set path\n")
+           (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           (system* "id")
+           (display "before initializing root\n")
+           (mkdir-p root)
+           (initialize root)
+           (display "after initializing root\n")
+           (build-docker-image (string-append "/xchg/" #$name)
+                               #$os-drv
+                               #:closure "system"
+                               #:symlinks '#$symlinks
+                               #:compressor '#$(compressor-command compressor)
+                               #:creation-time (make-time time-utc 0 1))))))
+  (expression->derivation-in-linux-vm
+   name
+   build
+   #:system system
+   #:make-disk-image? #f
+   #:single-file-output? #t
+   #:references-graphs inputs)))
+
 \f
 ;;;
 ;;; VM and disk images.
@@ -443,31 +552,38 @@ to USB sticks meant to be read-only."
 
     (mlet* %store-monad ((os-drv   (operating-system-derivation os))
                          (bootcfg  (operating-system-bootcfg os)))
-      (if (string=? "iso9660" file-system-type)
-          (iso9660-image #:name name
-                         #:file-system-label root-label
-                         #:file-system-uuid root-uuid
+      (cond ((string=? "iso9660" file-system-type)
+             (iso9660-image #:name name
+                            #:file-system-label root-label
+                            #:file-system-uuid root-uuid
+                            #:os-drv os-drv
+                            #:register-closures? #t
+                            #:bootcfg-drv bootcfg
+                            #:bootloader (bootloader-configuration-bootloader
+                                           (operating-system-bootloader os))
+                            #:inputs `(("system" ,os-drv)
+                                       ("bootcfg" ,bootcfg))))
+            ((string=? "docker")
+             (display "made it to docker image part\n")
+             (os-docker-image #:name name
+                              #:os-drv os-drv
+                              #:register-closures? #t
+                              #:inputs `(("system" ,os-drv))))
+            (else
+             (qemu-image #:name name
                          #:os-drv os-drv
-                         #:register-closures? #t
                          #:bootcfg-drv bootcfg
                          #:bootloader (bootloader-configuration-bootloader
-                                        (operating-system-bootloader os))
+                                       (operating-system-bootloader os))
+                         #:disk-image-size disk-image-size
+                         #:disk-image-format "raw"
+                         #:file-system-type file-system-type
+                         #:file-system-label root-label
+                         #:file-system-uuid root-uuid
+                         #:copy-inputs? #t
+                         #:register-closures? #t
                          #:inputs `(("system" ,os-drv)
-                                    ("bootcfg" ,bootcfg)))
-          (qemu-image #:name name
-                      #:os-drv os-drv
-                      #:bootcfg-drv bootcfg
-                      #:bootloader (bootloader-configuration-bootloader
-                                    (operating-system-bootloader os))
-                      #:disk-image-size disk-image-size
-                      #:disk-image-format "raw"
-                      #:file-system-type file-system-type
-                      #:file-system-label root-label
-                      #:file-system-uuid root-uuid
-                      #:copy-inputs? #t
-                      #:register-closures? #t
-                      #:inputs `(("system" ,os-drv)
-                                 ("bootcfg" ,bootcfg)))))))
+                                    ("bootcfg" ,bootcfg))))))))
 
 (define* (system-qemu-image os
                             #:key
diff --git a/guix/docker.scm b/guix/docker.scm
index 060232148..64d92ce16 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -28,7 +28,8 @@
   #:use-module (srfi srfi-19)
   #:use-module (rnrs bytevectors)
   #:use-module (ice-9 match)
-  #:export (build-docker-image))
+  #:export (build-docker-image
+            raw-disk-image->docker-image))
 
 ;; Load Guile-JSON at run time to simplify the job of 'imported-modules' & co.
 (module-use! (current-module) (resolve-interface '(json)))
@@ -181,3 +182,6 @@ CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
                                     '())
                               ".")))
               (begin (delete-file-recursively directory) #t)))))
+
+(define* (raw-disk-image->docker-image raw-image)
+  (display "Doing the docker stuff!"))
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 21fea446a..8d8053fca 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -41,7 +41,10 @@
   #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-37)
   #:use-module (ice-9 match)
-  #:export (compressor?
+  #:export (%compressors
+            compressor-extension
+            compressor-command
+            compressor?
             lookup-compressor
             self-contained-tarball
             guix-pack))
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index e50f1d8ac..392d2a3cc 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -640,6 +640,7 @@ any, are available.  Raise an error if they're not."
      (system-disk-image os
                         #:name (match file-system-type
                                  ("iso9660" "image.iso")
+                                 ("docker" "docker-image")
                                  (_         "disk-image"))
                         #:disk-image-size image-size
                         #:file-system-type file-system-type))))
-- 
2.14.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Guix on macOS

[Hartmut Goebel]

Am 27.10.2017 um 06:11 schrieb Chris Marusich:
> * Nix's Darwin bootstrap tools are maintained in Nix, but they aren't
>   cross-compiled (e.g., from x86_64-linux).  They're built from a
>   previous generation of the stdenv for Darwin.  I suspect, but do not
>   know for sure, that originally the bootstrap binaries were compiled
>   using the usual impure and non-free macOS toolchain.

For cross-building the PyInstaller bootloader we are using the ideas
based on OSXcross [1]. Basically we are extracting parts of the SDK from
Xcode and using this cross-build the bootloader. The SDK as a whole must
only be used on a Apple device, but parts of the SDK *may* not have this
limitation.[*] If you are interested who we do this, plase have a look
at [2] and [3].

[*] I did no check the license in detail. If Apple stepps up and
complains, we'll simply stop delivering a pre-compiled bootloader.

[1] https://github.com/tpoechtrager//osxcross
/[2]
https://github.com/pyinstaller/pyinstaller/blob/v3.3/doc/bootloader-building.rst
[3]
https://github.com/pyinstaller/pyinstaller/blob/v3.3/bootloader/Vagrantfile#L71
//

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Building Docker images of GuixSD

[Ludovic Courtès]

Hi Chris,

Nice work on building Docker images of GuixSD!

Chris Marusich <cmmarusich@gmail.com> skribis:

> [    1.345843] FS-Cache: Loaded
> [    1.362140] 9pnet: Installing 9P2000 support
> [    1.366118] 9p: Installing v9fs 9p2000 file system support
> [    1.368730] FS-Cache: Netfs '9p' registered for caching
> configuring QEMU networking...
> loading '/gnu/store/jy509dgcsz82y13fmizv2sqaj90s1vfg-linux-vm-loader'...
> ERROR: In procedure dynamic-link:
> ERROR: In procedure dynamic-link: file: "/gnu/store/hwygv5jwd47amhp1m67iy3bkvxqjlbhm-libgcrypt-1.8.1/lib/libgcrypt", message: "file not found"

The code above is running in the initrd, which means that it’s executed
by ‘guile-static-stripped’, which does not support dlopening:

--8<---------------cut here---------------start------------->8---
$ guix environment -C --ad-hoc guile-static-stripped libgcrypt -- guile

[...]

GNU Guile 2.2.2
Copyright (C) 1995-2017 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (file-exists? (string-append (getenv "GUIX_ENVIRONMENT") "/lib/libgcrypt.so"))
$1 = #t
scheme@(guile-user)> (dynamic-link (string-append (getenv "GUIX_ENVIRONMENT") "/lib/libgcrypt"))
ERROR: In procedure dynamic-link:
ERROR: In procedure dynamic-link: file: "/gnu/store/wplxvw0mxxy35j7019j6mkjvpgl0hs1g-profile/lib/libgcrypt", message: "file not found"

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
scheme@(guile-user) [1]>
--8<---------------cut here---------------end--------------->8---

(The “file not found” message is misleading.)

So we should either make a big initrd with the dynamically-linked Guile,
but then we may need to pass “-m 512” or similar to qemu…

Or we mount the host store over 9p and exec a dynamically-linked Guile
from there.

I realize it’s a bit sketchy, but I hope it makes sense.

Thanks,
Ludo’.

Re: Building Docker images of GuixSD

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> [    1.345843] FS-Cache: Loaded
>> [    1.362140] 9pnet: Installing 9P2000 support
>> [    1.366118] 9p: Installing v9fs 9p2000 file system support
>> [    1.368730] FS-Cache: Netfs '9p' registered for caching
>> configuring QEMU networking...
>> loading '/gnu/store/jy509dgcsz82y13fmizv2sqaj90s1vfg-linux-vm-loader'...
>> ERROR: In procedure dynamic-link:
>> ERROR: In procedure dynamic-link: file: "/gnu/store/hwygv5jwd47amhp1m67iy3bkvxqjlbhm-libgcrypt-1.8.1/lib/libgcrypt", message: "file not found"
>
> The code above is running in the initrd, which means that it’s executed
> by ‘guile-static-stripped’, which does not support dlopening:

Glad I asked!  It would have required much more fumbling around before I
had even though about a possibility like that.  It makes sense now.

> So we should either make a big initrd with the dynamically-linked
>Guile,
> but then we may need to pass “-m 512” or similar to qemu…

I will try this.  I'm running the gexp in a linux vm is only because I
need permission to create some of the device files.  If you try to run
this as a normal builder (e.g., using gexp->derivation), the build will
fail because the builder tries to create device nodes and lacks
permission to do so.

I might also look into building the image in a container, since we also
have some build-side container logic available (e.g.,
call-with-container).

I'll let you know how it goes.

> Or we mount the host store over 9p and exec a dynamically-linked Guile
> from there.

I understand hat by "9p" you mean to share the file system with the VM,
but I don't quite understand what this option entails.  If the initrd
doesn't use the right Guile, why will sharing the store via 9p work?  Do
you mean to exec directly somehow, instead of using an initrd...?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Building Docker images of GuixSD

[Ludovic Courtès]

Hi!

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Or we mount the host store over 9p and exec a dynamically-linked Guile
>> from there.
>
> I understand hat by "9p" you mean to share the file system with the VM,
> but I don't quite understand what this option entails.  If the initrd
> doesn't use the right Guile, why will sharing the store via 9p work?  Do
> you mean to exec directly somehow, instead of using an initrd...?

9p support is in the kernel, so we can always mount a 9p file system,
provided the relevant 9p kernel modules are loaded.

HTH!

Ludo’.

Re: Building Docker images of GuixSD

[Chris Marusich]

[-- Attachment #1.1: Type: text/plain, Size: 6984 bytes --]

Hi Ludo and others following along,

I've run GuixSD in a Docker container and returned to tell the tale!
The attached patch requires a lot of cleaning up (e.g., proper ChangeLog
entry, update documentation, remove some unnecessary imports and debug
messages that are probably still in there), so I'm taking a moment to
share my results and ask for feedback before committing to spending more
time on this.

Run GuixSD in Docker
====================

The attached patch makes it possible to build a GuixSD Docker image from
an operating system configuration file.

You can build your own like this:

1) Apply this patch to 3b2fa4787938a408fab27ef7b3bc1302b6b6a805.

2) Build an image (I used the attached file "very-bare-bones.scm"):

    ./pre-inst-env guix system disk-image -t docker very-bare-bones.scm

3) Copy the resulting image onto a host that has Docker installed.

4) On the host with Docker, load the image and note the image ID:

    docker load < pw3d4r4m1x9yc3d1kg9x3y6abdzq9z7g-docker-image.tar.gz

5) Run a Docker container from the image, and note the container ID:

    docker run --privileged -d -e GUIX_NEW_SYSTEM=/var/guix/profiles/system --net host --entrypoint /var/guix/profiles/system/profile/bin/guile dcaa8fb677c7 /var/guix/profiles/system/boot

6) Run a shell in the container, install a package, and use it:

    docker exec -it -e USER=alice -u alice fb06fdcd3a0d /run/current-system/profile/bin/bash --login

7) Install a package and use it:

    alice@komputilo /$ guix package -i hello
    ...
    Creating manual page database for 1 packages... done in 0.110 s
    1 package in profile
    alice@komputilo /$ guix package --list-installed
    hello	2.10	out	/gnu/store/wf65hjwqwpz4wllasn63zysi5irql2sx-hello-2.10
    alice@komputilo /$ hello
    Hello, world!

Pretty neat!

How Useful Is This?
===================

Using Guix, it was already possible to generate Docker images using
"guix pack".  For example, I could have just generated a Docker image
from the GNU Hello package, created a container from that, and then run
"hello" from that container.  What does running GuixSD in Docker give us
that we don't have already?  At a minimum, it gives us the following:

* The ability to define what service(s) should run in the resulting
  Docker container, including their configs and start/stop scripts.
  
* Since the Docker image is generated from a GuixSD operating system
  configuration file, the rules for defining and configuring services
  are the same as always.  You don't have to learn anything new.

* If you want to run Guix on a system to which Guix hasn't been ported
  (like macOS) but your system does run Docker, now you can run Guix on
  that system by running it from a GuixSD Docker container.

Is this helpful?  Is it worth polishing up and maintaining?  I'm not
entirely sure, and I'd like to know what you think.

For the first two bullet points, that's nice, but instead of using a
full-blown OS and relying on the Shepherd for process management in this
case, would it be simpler to just provide a way to easily bundle
start/stop scripts inside of the packs produced by "guix pack"?  An
enterprising user can probably do this today by simply defining a
package that builds start/stop scripts for a given service; the user
would then just need to include that package in the pack.  The downside,
I guess, is that you can't re-use the service-specific stuff that you
can normally use in a GuixSD operating system configuration file.

For the third bullet point, I don't know of any other reasonable way to
get Guix working in Docker (although one could certainly run Guix in a
VM using a technology other than Docker, such as QEMU).  To run Guix,
you need the Guix daemon running somewhere, right?  And the Guix daemon
requires that certain build users exist.  It might require other things
from its environment, too.  In any case, you can't just run "guix pack
-t docker guix" and expect the "guix" command to work in the container
(I tried, and it doesn't work).  You have to take additional measures,
like create build users, at which point it seems easier to just put all
of GuixSD into a Docker image.  That's what my patch lets you do.

What do you think?  Is this worth polishing up and maintaining?

Problems I Noticed
==================

Now I'll mention some specific problems I've noticed while running
GuixSD in a Docker container.  First, I saw this while the Docker image
was being generated:

    tar: Removing leading `/' from member names
    tar: Removing leading `/' from hard link targets
    tar: ./dev/log: socket ignored

It's fine that we remove the leading '/' from member names, since it
looks like the tarball will be extracted relative to '/'.  I think the
same is true for the hard link targets.  However, because tar ignored
'/dev/log', that socket is missing in the Docker image.  I don't know if
that will interfere with syslogd, but it sure doesn't sound good.

Second, I noticed the following error in the Guix daemon's logs.  It
might be benign, since package installation worked fine, but I'm not
sure what it means or how to debug it:

    error in finalization thread: Bad file descriptor

Third, I noticed that the shepherd failed to start syslogd and nscd (and
user-homes, although I wasn't as concerned about that because the home
directory for alice did in fact get created).  I understand that, due to
the way Docker works, some services are either not required (like
networking) or might require modifications to "behave well" in a Docker
container.  However, I didn't think syslogd and nscd would fall into
either of those categories, so I was surprised that they failed to
start.  The only relevant debug information appears to be the following
messages in the Shepherd logs (/var/log):

    2017-11-09 06:41:27 Service user-homes could not be started.
    2017-11-09 06:41:32 Service nscd could not be started.
    2017-11-09 06:41:37 Service syslogd could not be started.

I thought maybe syslogd wasn't working because /dev/log hadn't been
created in the Docker image, so I tried creating it manually.  However,
that didn't help; the Shepherd still couldn't start syslogd.

Fourth, I wasn't able to run GuixSD in a Docker container without
supplying the "--privileged" option.  GuixSD writes to sysfs during boot
(I don't know why, but the details are apparently in
guix/gnu/build/activation.scm), so the only way to get GuixSD to start
is to run the container in privileged mode.  This is unfortunate,
because privileged mode sounds quite dangerous for a lot of reasons.
For example, if both GuixSD in the Docker container and the host
operating system attempt to control the underlying hardware at the same
time, bad things might happen.

Thanks for reading this far.  I look forward to hearing your thoughts!

-- 
Chris

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-Make-it-possible-to-build-GuixSD-docker-images.patch --]
[-- Type: text/x-patch, Size: 20793 bytes --]

From 25d5527b14302fc835af5c338bf37cf621c63a4e Mon Sep 17 00:00:00 2001
From: Chris Marusich <cmmarusich@gmail.com>
Date: Sat, 21 Oct 2017 14:40:58 -0700
Subject: [PATCH] Make it possible to build GuixSD docker images

---
 gnu/build/linux-boot.scm    |   5 +-
 gnu/build/vm.scm            |  14 ++--
 gnu/system/linux-initrd.scm |  12 ++--
 gnu/system/vm.scm           | 169 ++++++++++++++++++++++++++++++++++++++------
 guix/docker.scm             |  23 ++++--
 guix/scripts/pack.scm       |   5 +-
 guix/scripts/system.scm     |   3 +-
 7 files changed, 191 insertions(+), 40 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 3712abe91..37da5b217 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -117,8 +117,9 @@ with the given MAJOR number, starting with MINOR."
                        "/")
                    dir))
 
+  (display "just before dev\n")
   (unless (file-exists? (scope "dev"))
-    (mkdir (scope "dev")))
+    (mkdir (pk (scope "dev"))))
 
   ;; Make the device nodes for SCSI disks.
   (make-disk-device-nodes (scope "dev/sda") 8)
@@ -138,6 +139,7 @@ with the given MAJOR number, starting with MINOR."
   (mknod (scope "dev/kmem") 'char-special #o640 (device-number 1 2))
 
   ;; Inputs (used by Xorg.)
+  (display "just before dev/input\n")
   (unless (file-exists? (scope "dev/input"))
     (mkdir (scope "dev/input")))
   (mknod (scope "dev/input/mice") 'char-special #o640 (device-number 13 63))
@@ -171,6 +173,7 @@ with the given MAJOR number, starting with MINOR."
   (chmod (scope "dev/ptmx") #o666)
 
   ;; Create /dev/pts; it will be mounted later, at boot time.
+  (display "just before dev/pts\n")
   (unless (file-exists? (scope "dev/pts"))
     (mkdir (scope "dev/pts")))
 
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 7537f8150..19c47e1ff 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -294,11 +294,14 @@ it, run its initializer, and unmount it."
 (define* (root-partition-initializer #:key (closures '())
                                      copy-closures?
                                      (register-closures? #t)
-                                     system-directory)
+                                     system-directory
+                                     (deduplicate? #t))
   "Return a procedure to initialize a root partition.
 
-If REGISTER-CLOSURES? is true, register all of CLOSURES is the partition's
-store.  If COPY-CLOSURES? is true, copy all of CLOSURES to the partition.
+If REGISTER-CLOSURES? is true, register all of CLOSURES in the partition's
+store.  If DEDUPLICATE? is true, then also deduplicate files common to
+CLOSURES and the rest of the store when registering the closures.  If
+COPY-CLOSURES? is true, copy all of CLOSURES to the partition.
 SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
   (lambda (target)
     (define target-store
@@ -317,13 +320,16 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
       (unless copy-closures?
         ;; XXX: 'guix-register' wants to palpate the things it registers, so
         ;; bind-mount the store on the target.
+        (display "making target store directory\n")
         (mkdir-p target-store)
+        (display "bind-mounting\n")
         (mount (%store-directory) target-store "" MS_BIND))
 
       (display "registering closures...\n")
       (for-each (lambda (closure)
                   (register-closure target
-                                    (string-append "/xchg/" closure)))
+                                    (string-append "/xchg/" closure)
+                                    #:deduplicate? deduplicate?))
                 closures)
       (unless copy-closures?
         (umount target-store)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 948c543a1..698f0aa70 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -155,7 +155,8 @@ MODULES and taken from LINUX."
                       (mapped-devices '())
                       (helper-packages '())
                       qemu-networking?
-                      volatile-root?)
+                      volatile-root?
+                      (guile %guile-static-stripped))
   "Return a monadic derivation that builds a raw initrd, with kernel
 modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
 mounted by the initrd, possibly in addition to the root file system specified
@@ -217,7 +218,8 @@ to it are lost."
                       #:linux-module-directory '#$kodir
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?)))
-   #:name "raw-initrd"))
+   #:name "raw-initrd"
+   #:guile guile))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))
   "Return the list of statically-linked, stripped packages to check
@@ -246,7 +248,8 @@ FILE-SYSTEMS."
                       qemu-networking?
                       volatile-root?
                       (virtio? #t)
-                      (extra-modules '()))
+                      (extra-modules '())
+                      (guile %guile-static-stripped))
   "Return a monadic derivation that builds a generic initrd, with kernel
 modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
 mounted by the initrd, possibly in addition to the root file system specified
@@ -321,6 +324,7 @@ loaded at boot time in the order in which they appear."
               #:mapped-devices mapped-devices
               #:helper-packages helper-packages
               #:qemu-networking? qemu-networking?
-              #:volatile-root? volatile-root?))
+              #:volatile-root? volatile-root?
+              #:guile guile))
 
 ;;; linux-initrd.scm ends here
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 3127b305e..b48a9a962 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -22,6 +22,7 @@
 
 (define-module (gnu system vm)
   #:use-module (guix config)
+  #:use-module (guix docker)
   #:use-module (guix store)
   #:use-module (guix gexp)
   #:use-module (guix derivations)
@@ -29,13 +30,16 @@
   #:use-module (guix monads)
   #:use-module (guix records)
   #:use-module (guix modules)
+  #:use-module (guix scripts pack)
 
   #:use-module ((gnu build vm)
                 #:select (qemu-command))
   #:use-module (gnu packages base)
+
   #:use-module (gnu packages bootloaders)
   #:use-module (gnu packages cdrom)
   #:use-module (gnu packages guile)
+  #:autoload   (gnu packages gnupg) (libgcrypt)
   #:use-module (gnu packages gawk)
   #:use-module (gnu packages bash)
   #:use-module (gnu packages less)
@@ -116,7 +120,8 @@
                                              (references-graphs #f)
                                              (memory-size 256)
                                              (disk-image-format "qcow2")
-                                             (disk-image-size 'guess))
+                                             (disk-image-size 'guess)
+                                             (guile-for-initrd %guile-static-stripped))
   "Evaluate EXP in a QEMU virtual machine running LINUX with INITRD (a
 derivation).  The virtual machine runs with MEMORY-SIZE MiB of memory.  In the
 virtual machine, EXP has access to all its inputs from the store; it should
@@ -143,7 +148,8 @@ made available under the /xchg CIFS share."
                          (base-initrd %linux-vm-file-systems
                                       #:linux linux
                                       #:virtio? #t
-                                      #:qemu-networking? #t))))
+                                      #:qemu-networking? #t
+                                      #:guile guile-for-initrd))))
 
     (define builder
       ;; Code that launches the VM that evaluates EXP.
@@ -349,6 +355,117 @@ the image."
    #:disk-image-format disk-image-format
    #:references-graphs inputs))
 
+(define* (os-docker-image #:key
+                          (name "guixsd-docker-image")
+                          os-drv
+                          (system (%current-system))
+                          (compressor (first %compressors))
+                          localstatedir?
+                          (symlinks '())
+                          (tar tar)
+                          (register-closures? #t))
+  "Build a docker image.  OS-DRV is a derivation which builds the
+operating system profile."
+  ;; FIXME: Honor LOCALSTATEDIR?.
+  (define not-config?
+    (match-lambda
+      (('guix 'config) #f)
+      (('guix rest ...) #t)
+      (('gnu rest ...) #t)
+      (rest #f)))
+
+  (define config
+    ;; (guix config) module for consumption by (guix gcrypt).
+    (scheme-file "gcrypt-config.scm"
+                 #~(begin
+                     (define-module (guix config)
+                       #:export (%libgcrypt))
+
+                     ;; XXX: Work around <http://bugs.gnu.org/15602>.
+                     (eval-when (expand load eval)
+                       (define %libgcrypt
+                         #+(file-append libgcrypt "/lib/libgcrypt"))))))
+
+  (define json
+    ;; Pick the guile-json package that corresponds to the Guile used to build
+    ;; derivations.
+    (if (string-prefix? "2.0" (package-version (default-guile)))
+        guile2.0-json
+        guile-json))
+  
+  (let ((name (string-append name ".tar" (compressor-extension compressor)))
+        (system-graph-name "system"))
+    (define build
+      (with-imported-modules `(,@(source-module-closure '((guix docker)
+                                                          (gnu build vm)
+                                                          (guix build utils)
+                                                          (guix build syscalls))
+                                                        #:select? not-config?)
+                               ((guix config) => ,config))
+        #~(begin
+            ;; Guile-JSON is required by (guix docker).
+            (add-to-load-path
+             (string-append #+json "/share/guile/site/"
+                            (effective-version)))
+            (use-modules (gnu build vm)
+                         (guix build utils)
+                         (guix build syscalls)
+                         (srfi srfi-26)
+                         (ice-9 match)
+                         (guix docker)
+                         (srfi srfi-19))
+
+            (let* ((inputs
+                    '#$(append (list tree parted e2fsprogs dosfstools tar)
+                               (map canonical-package
+                                    (list sed grep coreutils findutils gawk))
+                               (if register-closures? (list guix) '())))
+
+                   ;; This variable is unused but allows us to add INPUTS-TO-COPY
+                   ;; as inputs.
+                   (to-register '#$os-drv)
+                   (initialize (root-partition-initializer
+                                #:closures '(#$system-graph-name)
+                                #:copy-closures? #f
+                                #:register-closures? #$register-closures?
+                                #:system-directory #$os-drv
+                                #:deduplicate? #f))
+                   (root "/tmp/root"))
+
+              (display "before set path\n")
+              (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+              (system* "id")
+              (display "before initializing root\n")
+              (system* "df")
+              (mkdir-p root)
+              (initialize root)
+              (display "after initializing root, building docker image\n")
+              ;; Use a temporary directory inside xchg to avoid hitting space
+              ;; limitations in the initrd's root file system.
+              (let ((tmpdir "/xchg/tmp"))
+                (mkdir tmpdir)
+                ;; TODO: Put paths from outside of the store into the docker image.
+                ;; For example, /var/guix, /home, etc.
+                (build-docker-image
+                 (string-append "/xchg/" #$name) ;; The output file.
+                 #$os-drv
+                 #:closure (string-append "/xchg/" #$system-graph-name)
+                 #:symlinks '#$symlinks
+                 #:compressor '#$(compressor-command compressor)
+                 #:creation-time (make-time time-utc 0 1)
+                 #:tmpdir tmpdir
+                 #:extra-items-dir root)
+                (delete-file-recursively tmpdir))))))
+    (expression->derivation-in-linux-vm
+     name
+     build
+     #:system system
+     #:make-disk-image? #f
+     #:single-file-output? #t
+     #:references-graphs `((,system-graph-name ,os-drv))
+     #:guile-for-initrd guile-2.2
+     #:memory-size 512)))
+
 \f
 ;;;
 ;;; VM and disk images.
@@ -443,31 +560,37 @@ to USB sticks meant to be read-only."
 
     (mlet* %store-monad ((os-drv   (operating-system-derivation os))
                          (bootcfg  (operating-system-bootcfg os)))
-      (if (string=? "iso9660" file-system-type)
-          (iso9660-image #:name name
-                         #:file-system-label root-label
-                         #:file-system-uuid root-uuid
+      (cond ((string=? "iso9660" file-system-type)
+             (iso9660-image #:name name
+                            #:file-system-label root-label
+                            #:file-system-uuid root-uuid
+                            #:os-drv os-drv
+                            #:register-closures? #t
+                            #:bootcfg-drv bootcfg
+                            #:bootloader (bootloader-configuration-bootloader
+                                           (operating-system-bootloader os))
+                            #:inputs `(("system" ,os-drv)
+                                       ("bootcfg" ,bootcfg))))
+            ((string=? "docker" file-system-type)
+             (display "made it to docker image part\n")
+             (os-docker-image #:name name
+                              #:os-drv os-drv
+                              #:register-closures? #t))
+            (else
+             (qemu-image #:name name
                          #:os-drv os-drv
-                         #:register-closures? #t
                          #:bootcfg-drv bootcfg
                          #:bootloader (bootloader-configuration-bootloader
-                                        (operating-system-bootloader os))
+                                       (operating-system-bootloader os))
+                         #:disk-image-size disk-image-size
+                         #:disk-image-format "raw"
+                         #:file-system-type file-system-type
+                         #:file-system-label root-label
+                         #:file-system-uuid root-uuid
+                         #:copy-inputs? #t
+                         #:register-closures? #t
                          #:inputs `(("system" ,os-drv)
-                                    ("bootcfg" ,bootcfg)))
-          (qemu-image #:name name
-                      #:os-drv os-drv
-                      #:bootcfg-drv bootcfg
-                      #:bootloader (bootloader-configuration-bootloader
-                                    (operating-system-bootloader os))
-                      #:disk-image-size disk-image-size
-                      #:disk-image-format "raw"
-                      #:file-system-type file-system-type
-                      #:file-system-label root-label
-                      #:file-system-uuid root-uuid
-                      #:copy-inputs? #t
-                      #:register-closures? #t
-                      #:inputs `(("system" ,os-drv)
-                                 ("bootcfg" ,bootcfg)))))))
+                                    ("bootcfg" ,bootcfg))))))))
 
 (define* (system-qemu-image os
                             #:key
diff --git a/guix/docker.scm b/guix/docker.scm
index 060232148..98914f1a1 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -28,7 +28,8 @@
   #:use-module (srfi srfi-19)
   #:use-module (rnrs bytevectors)
   #:use-module (ice-9 match)
-  #:export (build-docker-image))
+  #:export (build-docker-image
+            raw-disk-image->docker-image))
 
 ;; Load Guile-JSON at run time to simplify the job of 'imported-modules' & co.
 (module-use! (current-module) (resolve-interface '(json)))
@@ -106,7 +107,9 @@ return \"a\"."
                              #:key closure compressor
                              (symlinks '())
                              (system (utsname:machine (uname)))
-                             (creation-time (current-time time-utc)))
+                             (creation-time (current-time time-utc))
+                             (tmpdir "/tmp")
+                             extra-items-dir)
   "Write to IMAGE a Docker image archive from the given store PATH.  The image
 contains the closure of PATH, as specified in CLOSURE (a file produced by
 #:references-graphs).  SYMLINKS must be a list of (SOURCE -> TARGET) tuples
@@ -116,7 +119,7 @@ binaries at PATH are for; it is used to produce metadata in the image.
 
 Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE.  Use
 CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
-  (let ((directory "/tmp/docker-image")           ;temporary working directory
+  (let ((directory (string-append tmpdir "/docker-image")) ;temporary working directory
         (closure (canonicalize-path closure))
         (id (docker-id path))
         (time (date->string (time-utc->date creation-time) "~4"))
@@ -159,9 +162,14 @@ CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
                                   (append %tar-determinism-options
                                           items
                                           (map symlink-source symlinks))))
-                    (for-each delete-file-recursively
-                              (map (compose topmost-component symlink-source)
-                                   symlinks)))))
+                    (begin
+                      (for-each delete-file-recursively
+                                (map (compose topmost-component symlink-source)
+                                     symlinks))
+                      (zero? (apply system* "tar" "-C" extra-items-dir
+                                    "-rf" "layer.tar"
+                                    (append %tar-determinism-options
+                                            '("."))))))))
 
            (with-output-to-file "config.json"
              (lambda ()
@@ -181,3 +189,6 @@ CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
                                     '())
                               ".")))
               (begin (delete-file-recursively directory) #t)))))
+
+(define* (raw-disk-image->docker-image raw-image)
+  (display "Doing the docker stuff!"))
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 21fea446a..8d8053fca 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -41,7 +41,10 @@
   #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-37)
   #:use-module (ice-9 match)
-  #:export (compressor?
+  #:export (%compressors
+            compressor-extension
+            compressor-command
+            compressor?
             lookup-compressor
             self-contained-tarball
             guix-pack))
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index e50f1d8ac..a319692d7 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -638,8 +638,9 @@ any, are available.  Raise an error if they're not."
                                             #:mappings mappings))
     ((disk-image)
      (system-disk-image os
-                        #:name (match file-system-type
+                        #:name (match (pk file-system-type)
                                  ("iso9660" "image.iso")
+                                 ("docker" "docker-image")
                                  (_         "disk-image"))
                         #:disk-image-size image-size
                         #:file-system-type file-system-type))))
-- 
2.14.2


[-- Attachment #1.3: very-bare-bones.scm --]
[-- Type: application/octet-stream, Size: 1533 bytes --]

(use-modules (gnu))

(operating-system
  ;; These defaults are fine.
  (host-name "komputilo")
  (timezone "Europe/Berlin")
  (locale "en_US.utf8")

  ;; These values will be ignored when making the Docker image, just
  ;; like how they will be ignored when making a VM image.
  (bootloader (bootloader-configuration
               (bootloader grub-bootloader)
               (target "/dev/sdX")))
  (file-systems (cons (file-system
                        (device "my-root")
                        (title 'label)
                        (mount-point "/")
                        (type "ext4"))
                      %base-file-systems))

  ;; Default user is fine for testing.
  (users (cons (user-account
                (name "alice")
                (comment "Bob's sister")
                (group "users")

                (supplementary-groups '("wheel"
                                        "audio" "video"))
                (home-directory "/home/alice"))
               %base-user-accounts))

  ;; Default base packages are fine.
  (packages %base-packages)

  ;; Extremely bare bones.  We don't need things like mingetty or udev
  ;; or a login service in our container; we just probably need these.
  ;; We don't even need networking, since Docker takes care of that
  ;; for us.
  (services
   (list
    ;; Herd fails to start syslogd - /var/log contained no hints.
    (syslog-service)
    (urandom-seed-service)
    (guix-service)
    ;; Herd fails to start nscd - /var/log contained no hints.
    (nscd-service))))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Building Docker images of GuixSD

[Pjotr Prins]

Chris, this is very interesting! Even with privileged mode it makes it
much easier to experiment. 

Pj.

On Wed, Nov 08, 2017 at 10:15:38PM -0800, Chris Marusich wrote:
> Hi Ludo and others following along,
> 
> I've run GuixSD in a Docker container and returned to tell the tale!
> The attached patch requires a lot of cleaning up (e.g., proper ChangeLog
> entry, update documentation, remove some unnecessary imports and debug
> messages that are probably still in there), so I'm taking a moment to
> share my results and ask for feedback before committing to spending more
> time on this.
> 
> Run GuixSD in Docker
> ====================
> 
> The attached patch makes it possible to build a GuixSD Docker image from
> an operating system configuration file.
> 
> You can build your own like this:
> 
> 1) Apply this patch to 3b2fa4787938a408fab27ef7b3bc1302b6b6a805.
> 
> 2) Build an image (I used the attached file "very-bare-bones.scm"):
> 
>     ./pre-inst-env guix system disk-image -t docker very-bare-bones.scm
> 
> 3) Copy the resulting image onto a host that has Docker installed.
> 
> 4) On the host with Docker, load the image and note the image ID:
> 
>     docker load < pw3d4r4m1x9yc3d1kg9x3y6abdzq9z7g-docker-image.tar.gz
> 
> 5) Run a Docker container from the image, and note the container ID:
> 
>     docker run --privileged -d -e GUIX_NEW_SYSTEM=/var/guix/profiles/system --net host --entrypoint /var/guix/profiles/system/profile/bin/guile dcaa8fb677c7 /var/guix/profiles/system/boot
> 
> 6) Run a shell in the container, install a package, and use it:
> 
>     docker exec -it -e USER=alice -u alice fb06fdcd3a0d /run/current-system/profile/bin/bash --login
> 
> 7) Install a package and use it:
> 
>     alice@komputilo /$ guix package -i hello
>     ...
>     Creating manual page database for 1 packages... done in 0.110 s
>     1 package in profile
>     alice@komputilo /$ guix package --list-installed
>     hello	2.10	out	/gnu/store/wf65hjwqwpz4wllasn63zysi5irql2sx-hello-2.10
>     alice@komputilo /$ hello
>     Hello, world!
> 
> Pretty neat!
> 
> How Useful Is This?
> ===================
> 
> Using Guix, it was already possible to generate Docker images using
> "guix pack".  For example, I could have just generated a Docker image
> from the GNU Hello package, created a container from that, and then run
> "hello" from that container.  What does running GuixSD in Docker give us
> that we don't have already?  At a minimum, it gives us the following:
> 
> * The ability to define what service(s) should run in the resulting
>   Docker container, including their configs and start/stop scripts.
>   
> * Since the Docker image is generated from a GuixSD operating system
>   configuration file, the rules for defining and configuring services
>   are the same as always.  You don't have to learn anything new.
> 
> * If you want to run Guix on a system to which Guix hasn't been ported
>   (like macOS) but your system does run Docker, now you can run Guix on
>   that system by running it from a GuixSD Docker container.
> 
> Is this helpful?  Is it worth polishing up and maintaining?  I'm not
> entirely sure, and I'd like to know what you think.
> 
> For the first two bullet points, that's nice, but instead of using a
> full-blown OS and relying on the Shepherd for process management in this
> case, would it be simpler to just provide a way to easily bundle
> start/stop scripts inside of the packs produced by "guix pack"?  An
> enterprising user can probably do this today by simply defining a
> package that builds start/stop scripts for a given service; the user
> would then just need to include that package in the pack.  The downside,
> I guess, is that you can't re-use the service-specific stuff that you
> can normally use in a GuixSD operating system configuration file.
> 
> For the third bullet point, I don't know of any other reasonable way to
> get Guix working in Docker (although one could certainly run Guix in a
> VM using a technology other than Docker, such as QEMU).  To run Guix,
> you need the Guix daemon running somewhere, right?  And the Guix daemon
> requires that certain build users exist.  It might require other things
> from its environment, too.  In any case, you can't just run "guix pack
> -t docker guix" and expect the "guix" command to work in the container
> (I tried, and it doesn't work).  You have to take additional measures,
> like create build users, at which point it seems easier to just put all
> of GuixSD into a Docker image.  That's what my patch lets you do.
> 
> What do you think?  Is this worth polishing up and maintaining?
> 
> Problems I Noticed
> ==================
> 
> Now I'll mention some specific problems I've noticed while running
> GuixSD in a Docker container.  First, I saw this while the Docker image
> was being generated:
> 
>     tar: Removing leading `/' from member names
>     tar: Removing leading `/' from hard link targets
>     tar: ./dev/log: socket ignored
> 
> It's fine that we remove the leading '/' from member names, since it
> looks like the tarball will be extracted relative to '/'.  I think the
> same is true for the hard link targets.  However, because tar ignored
> '/dev/log', that socket is missing in the Docker image.  I don't know if
> that will interfere with syslogd, but it sure doesn't sound good.
> 
> Second, I noticed the following error in the Guix daemon's logs.  It
> might be benign, since package installation worked fine, but I'm not
> sure what it means or how to debug it:
> 
>     error in finalization thread: Bad file descriptor
> 
> Third, I noticed that the shepherd failed to start syslogd and nscd (and
> user-homes, although I wasn't as concerned about that because the home
> directory for alice did in fact get created).  I understand that, due to
> the way Docker works, some services are either not required (like
> networking) or might require modifications to "behave well" in a Docker
> container.  However, I didn't think syslogd and nscd would fall into
> either of those categories, so I was surprised that they failed to
> start.  The only relevant debug information appears to be the following
> messages in the Shepherd logs (/var/log):
> 
>     2017-11-09 06:41:27 Service user-homes could not be started.
>     2017-11-09 06:41:32 Service nscd could not be started.
>     2017-11-09 06:41:37 Service syslogd could not be started.
> 
> I thought maybe syslogd wasn't working because /dev/log hadn't been
> created in the Docker image, so I tried creating it manually.  However,
> that didn't help; the Shepherd still couldn't start syslogd.
> 
> Fourth, I wasn't able to run GuixSD in a Docker container without
> supplying the "--privileged" option.  GuixSD writes to sysfs during boot
> (I don't know why, but the details are apparently in
> guix/gnu/build/activation.scm), so the only way to get GuixSD to start
> is to run the container in privileged mode.  This is unfortunate,
> because privileged mode sounds quite dangerous for a lot of reasons.
> For example, if both GuixSD in the Docker container and the host
> operating system attempt to control the underlying hardware at the same
> time, bad things might happen.
> 
> Thanks for reading this far.  I look forward to hearing your thoughts!
> 
> -- 
> Chris

> From 25d5527b14302fc835af5c338bf37cf621c63a4e Mon Sep 17 00:00:00 2001
> From: Chris Marusich <cmmarusich@gmail.com>
> Date: Sat, 21 Oct 2017 14:40:58 -0700
> Subject: [PATCH] Make it possible to build GuixSD docker images
> 
> ---
>  gnu/build/linux-boot.scm    |   5 +-
>  gnu/build/vm.scm            |  14 ++--
>  gnu/system/linux-initrd.scm |  12 ++--
>  gnu/system/vm.scm           | 169 ++++++++++++++++++++++++++++++++++++++------
>  guix/docker.scm             |  23 ++++--
>  guix/scripts/pack.scm       |   5 +-
>  guix/scripts/system.scm     |   3 +-
>  7 files changed, 191 insertions(+), 40 deletions(-)
> 
> diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
> index 3712abe91..37da5b217 100644
> --- a/gnu/build/linux-boot.scm
> +++ b/gnu/build/linux-boot.scm
> @@ -117,8 +117,9 @@ with the given MAJOR number, starting with MINOR."
>                         "/")
>                     dir))
>  
> +  (display "just before dev\n")
>    (unless (file-exists? (scope "dev"))
> -    (mkdir (scope "dev")))
> +    (mkdir (pk (scope "dev"))))
>  
>    ;; Make the device nodes for SCSI disks.
>    (make-disk-device-nodes (scope "dev/sda") 8)
> @@ -138,6 +139,7 @@ with the given MAJOR number, starting with MINOR."
>    (mknod (scope "dev/kmem") 'char-special #o640 (device-number 1 2))
>  
>    ;; Inputs (used by Xorg.)
> +  (display "just before dev/input\n")
>    (unless (file-exists? (scope "dev/input"))
>      (mkdir (scope "dev/input")))
>    (mknod (scope "dev/input/mice") 'char-special #o640 (device-number 13 63))
> @@ -171,6 +173,7 @@ with the given MAJOR number, starting with MINOR."
>    (chmod (scope "dev/ptmx") #o666)
>  
>    ;; Create /dev/pts; it will be mounted later, at boot time.
> +  (display "just before dev/pts\n")
>    (unless (file-exists? (scope "dev/pts"))
>      (mkdir (scope "dev/pts")))
>  
> diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
> index 7537f8150..19c47e1ff 100644
> --- a/gnu/build/vm.scm
> +++ b/gnu/build/vm.scm
> @@ -294,11 +294,14 @@ it, run its initializer, and unmount it."
>  (define* (root-partition-initializer #:key (closures '())
>                                       copy-closures?
>                                       (register-closures? #t)
> -                                     system-directory)
> +                                     system-directory
> +                                     (deduplicate? #t))
>    "Return a procedure to initialize a root partition.
>  
> -If REGISTER-CLOSURES? is true, register all of CLOSURES is the partition's
> -store.  If COPY-CLOSURES? is true, copy all of CLOSURES to the partition.
> +If REGISTER-CLOSURES? is true, register all of CLOSURES in the partition's
> +store.  If DEDUPLICATE? is true, then also deduplicate files common to
> +CLOSURES and the rest of the store when registering the closures.  If
> +COPY-CLOSURES? is true, copy all of CLOSURES to the partition.
>  SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
>    (lambda (target)
>      (define target-store
> @@ -317,13 +320,16 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
>        (unless copy-closures?
>          ;; XXX: 'guix-register' wants to palpate the things it registers, so
>          ;; bind-mount the store on the target.
> +        (display "making target store directory\n")
>          (mkdir-p target-store)
> +        (display "bind-mounting\n")
>          (mount (%store-directory) target-store "" MS_BIND))
>  
>        (display "registering closures...\n")
>        (for-each (lambda (closure)
>                    (register-closure target
> -                                    (string-append "/xchg/" closure)))
> +                                    (string-append "/xchg/" closure)
> +                                    #:deduplicate? deduplicate?))
>                  closures)
>        (unless copy-closures?
>          (umount target-store)))
> diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
> index 948c543a1..698f0aa70 100644
> --- a/gnu/system/linux-initrd.scm
> +++ b/gnu/system/linux-initrd.scm
> @@ -155,7 +155,8 @@ MODULES and taken from LINUX."
>                        (mapped-devices '())
>                        (helper-packages '())
>                        qemu-networking?
> -                      volatile-root?)
> +                      volatile-root?
> +                      (guile %guile-static-stripped))
>    "Return a monadic derivation that builds a raw initrd, with kernel
>  modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
>  mounted by the initrd, possibly in addition to the root file system specified
> @@ -217,7 +218,8 @@ to it are lost."
>                        #:linux-module-directory '#$kodir
>                        #:qemu-guest-networking? #$qemu-networking?
>                        #:volatile-root? '#$volatile-root?)))
> -   #:name "raw-initrd"))
> +   #:name "raw-initrd"
> +   #:guile guile))
>  
>  (define* (file-system-packages file-systems #:key (volatile-root? #f))
>    "Return the list of statically-linked, stripped packages to check
> @@ -246,7 +248,8 @@ FILE-SYSTEMS."
>                        qemu-networking?
>                        volatile-root?
>                        (virtio? #t)
> -                      (extra-modules '()))
> +                      (extra-modules '())
> +                      (guile %guile-static-stripped))
>    "Return a monadic derivation that builds a generic initrd, with kernel
>  modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
>  mounted by the initrd, possibly in addition to the root file system specified
> @@ -321,6 +324,7 @@ loaded at boot time in the order in which they appear."
>                #:mapped-devices mapped-devices
>                #:helper-packages helper-packages
>                #:qemu-networking? qemu-networking?
> -              #:volatile-root? volatile-root?))
> +              #:volatile-root? volatile-root?
> +              #:guile guile))
>  
>  ;;; linux-initrd.scm ends here
> diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
> index 3127b305e..b48a9a962 100644
> --- a/gnu/system/vm.scm
> +++ b/gnu/system/vm.scm
> @@ -22,6 +22,7 @@
>  
>  (define-module (gnu system vm)
>    #:use-module (guix config)
> +  #:use-module (guix docker)
>    #:use-module (guix store)
>    #:use-module (guix gexp)
>    #:use-module (guix derivations)
> @@ -29,13 +30,16 @@
>    #:use-module (guix monads)
>    #:use-module (guix records)
>    #:use-module (guix modules)
> +  #:use-module (guix scripts pack)
>  
>    #:use-module ((gnu build vm)
>                  #:select (qemu-command))
>    #:use-module (gnu packages base)
> +
>    #:use-module (gnu packages bootloaders)
>    #:use-module (gnu packages cdrom)
>    #:use-module (gnu packages guile)
> +  #:autoload   (gnu packages gnupg) (libgcrypt)
>    #:use-module (gnu packages gawk)
>    #:use-module (gnu packages bash)
>    #:use-module (gnu packages less)
> @@ -116,7 +120,8 @@
>                                               (references-graphs #f)
>                                               (memory-size 256)
>                                               (disk-image-format "qcow2")
> -                                             (disk-image-size 'guess))
> +                                             (disk-image-size 'guess)
> +                                             (guile-for-initrd %guile-static-stripped))
>    "Evaluate EXP in a QEMU virtual machine running LINUX with INITRD (a
>  derivation).  The virtual machine runs with MEMORY-SIZE MiB of memory.  In the
>  virtual machine, EXP has access to all its inputs from the store; it should
> @@ -143,7 +148,8 @@ made available under the /xchg CIFS share."
>                           (base-initrd %linux-vm-file-systems
>                                        #:linux linux
>                                        #:virtio? #t
> -                                      #:qemu-networking? #t))))
> +                                      #:qemu-networking? #t
> +                                      #:guile guile-for-initrd))))
>  
>      (define builder
>        ;; Code that launches the VM that evaluates EXP.
> @@ -349,6 +355,117 @@ the image."
>     #:disk-image-format disk-image-format
>     #:references-graphs inputs))
>  
> +(define* (os-docker-image #:key
> +                          (name "guixsd-docker-image")
> +                          os-drv
> +                          (system (%current-system))
> +                          (compressor (first %compressors))
> +                          localstatedir?
> +                          (symlinks '())
> +                          (tar tar)
> +                          (register-closures? #t))
> +  "Build a docker image.  OS-DRV is a derivation which builds the
> +operating system profile."
> +  ;; FIXME: Honor LOCALSTATEDIR?.
> +  (define not-config?
> +    (match-lambda
> +      (('guix 'config) #f)
> +      (('guix rest ...) #t)
> +      (('gnu rest ...) #t)
> +      (rest #f)))
> +
> +  (define config
> +    ;; (guix config) module for consumption by (guix gcrypt).
> +    (scheme-file "gcrypt-config.scm"
> +                 #~(begin
> +                     (define-module (guix config)
> +                       #:export (%libgcrypt))
> +
> +                     ;; XXX: Work around <http://bugs.gnu.org/15602>.
> +                     (eval-when (expand load eval)
> +                       (define %libgcrypt
> +                         #+(file-append libgcrypt "/lib/libgcrypt"))))))
> +
> +  (define json
> +    ;; Pick the guile-json package that corresponds to the Guile used to build
> +    ;; derivations.
> +    (if (string-prefix? "2.0" (package-version (default-guile)))
> +        guile2.0-json
> +        guile-json))
> +  
> +  (let ((name (string-append name ".tar" (compressor-extension compressor)))
> +        (system-graph-name "system"))
> +    (define build
> +      (with-imported-modules `(,@(source-module-closure '((guix docker)
> +                                                          (gnu build vm)
> +                                                          (guix build utils)
> +                                                          (guix build syscalls))
> +                                                        #:select? not-config?)
> +                               ((guix config) => ,config))
> +        #~(begin
> +            ;; Guile-JSON is required by (guix docker).
> +            (add-to-load-path
> +             (string-append #+json "/share/guile/site/"
> +                            (effective-version)))
> +            (use-modules (gnu build vm)
> +                         (guix build utils)
> +                         (guix build syscalls)
> +                         (srfi srfi-26)
> +                         (ice-9 match)
> +                         (guix docker)
> +                         (srfi srfi-19))
> +
> +            (let* ((inputs
> +                    '#$(append (list tree parted e2fsprogs dosfstools tar)
> +                               (map canonical-package
> +                                    (list sed grep coreutils findutils gawk))
> +                               (if register-closures? (list guix) '())))
> +
> +                   ;; This variable is unused but allows us to add INPUTS-TO-COPY
> +                   ;; as inputs.
> +                   (to-register '#$os-drv)
> +                   (initialize (root-partition-initializer
> +                                #:closures '(#$system-graph-name)
> +                                #:copy-closures? #f
> +                                #:register-closures? #$register-closures?
> +                                #:system-directory #$os-drv
> +                                #:deduplicate? #f))
> +                   (root "/tmp/root"))
> +
> +              (display "before set path\n")
> +              (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
> +              (system* "id")
> +              (display "before initializing root\n")
> +              (system* "df")
> +              (mkdir-p root)
> +              (initialize root)
> +              (display "after initializing root, building docker image\n")
> +              ;; Use a temporary directory inside xchg to avoid hitting space
> +              ;; limitations in the initrd's root file system.
> +              (let ((tmpdir "/xchg/tmp"))
> +                (mkdir tmpdir)
> +                ;; TODO: Put paths from outside of the store into the docker image.
> +                ;; For example, /var/guix, /home, etc.
> +                (build-docker-image
> +                 (string-append "/xchg/" #$name) ;; The output file.
> +                 #$os-drv
> +                 #:closure (string-append "/xchg/" #$system-graph-name)
> +                 #:symlinks '#$symlinks
> +                 #:compressor '#$(compressor-command compressor)
> +                 #:creation-time (make-time time-utc 0 1)
> +                 #:tmpdir tmpdir
> +                 #:extra-items-dir root)
> +                (delete-file-recursively tmpdir))))))
> +    (expression->derivation-in-linux-vm
> +     name
> +     build
> +     #:system system
> +     #:make-disk-image? #f
> +     #:single-file-output? #t
> +     #:references-graphs `((,system-graph-name ,os-drv))
> +     #:guile-for-initrd guile-2.2
> +     #:memory-size 512)))
> +
> 
>  ;;;
>  ;;; VM and disk images.
> @@ -443,31 +560,37 @@ to USB sticks meant to be read-only."
>  
>      (mlet* %store-monad ((os-drv   (operating-system-derivation os))
>                           (bootcfg  (operating-system-bootcfg os)))
> -      (if (string=? "iso9660" file-system-type)
> -          (iso9660-image #:name name
> -                         #:file-system-label root-label
> -                         #:file-system-uuid root-uuid
> +      (cond ((string=? "iso9660" file-system-type)
> +             (iso9660-image #:name name
> +                            #:file-system-label root-label
> +                            #:file-system-uuid root-uuid
> +                            #:os-drv os-drv
> +                            #:register-closures? #t
> +                            #:bootcfg-drv bootcfg
> +                            #:bootloader (bootloader-configuration-bootloader
> +                                           (operating-system-bootloader os))
> +                            #:inputs `(("system" ,os-drv)
> +                                       ("bootcfg" ,bootcfg))))
> +            ((string=? "docker" file-system-type)
> +             (display "made it to docker image part\n")
> +             (os-docker-image #:name name
> +                              #:os-drv os-drv
> +                              #:register-closures? #t))
> +            (else
> +             (qemu-image #:name name
>                           #:os-drv os-drv
> -                         #:register-closures? #t
>                           #:bootcfg-drv bootcfg
>                           #:bootloader (bootloader-configuration-bootloader
> -                                        (operating-system-bootloader os))
> +                                       (operating-system-bootloader os))
> +                         #:disk-image-size disk-image-size
> +                         #:disk-image-format "raw"
> +                         #:file-system-type file-system-type
> +                         #:file-system-label root-label
> +                         #:file-system-uuid root-uuid
> +                         #:copy-inputs? #t
> +                         #:register-closures? #t
>                           #:inputs `(("system" ,os-drv)
> -                                    ("bootcfg" ,bootcfg)))
> -          (qemu-image #:name name
> -                      #:os-drv os-drv
> -                      #:bootcfg-drv bootcfg
> -                      #:bootloader (bootloader-configuration-bootloader
> -                                    (operating-system-bootloader os))
> -                      #:disk-image-size disk-image-size
> -                      #:disk-image-format "raw"
> -                      #:file-system-type file-system-type
> -                      #:file-system-label root-label
> -                      #:file-system-uuid root-uuid
> -                      #:copy-inputs? #t
> -                      #:register-closures? #t
> -                      #:inputs `(("system" ,os-drv)
> -                                 ("bootcfg" ,bootcfg)))))))
> +                                    ("bootcfg" ,bootcfg))))))))
>  
>  (define* (system-qemu-image os
>                              #:key
> diff --git a/guix/docker.scm b/guix/docker.scm
> index 060232148..98914f1a1 100644
> --- a/guix/docker.scm
> +++ b/guix/docker.scm
> @@ -28,7 +28,8 @@
>    #:use-module (srfi srfi-19)
>    #:use-module (rnrs bytevectors)
>    #:use-module (ice-9 match)
> -  #:export (build-docker-image))
> +  #:export (build-docker-image
> +            raw-disk-image->docker-image))
>  
>  ;; Load Guile-JSON at run time to simplify the job of 'imported-modules' & co.
>  (module-use! (current-module) (resolve-interface '(json)))
> @@ -106,7 +107,9 @@ return \"a\"."
>                               #:key closure compressor
>                               (symlinks '())
>                               (system (utsname:machine (uname)))
> -                             (creation-time (current-time time-utc)))
> +                             (creation-time (current-time time-utc))
> +                             (tmpdir "/tmp")
> +                             extra-items-dir)
>    "Write to IMAGE a Docker image archive from the given store PATH.  The image
>  contains the closure of PATH, as specified in CLOSURE (a file produced by
>  #:references-graphs).  SYMLINKS must be a list of (SOURCE -> TARGET) tuples
> @@ -116,7 +119,7 @@ binaries at PATH are for; it is used to produce metadata in the image.
>  
>  Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE.  Use
>  CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
> -  (let ((directory "/tmp/docker-image")           ;temporary working directory
> +  (let ((directory (string-append tmpdir "/docker-image")) ;temporary working directory
>          (closure (canonicalize-path closure))
>          (id (docker-id path))
>          (time (date->string (time-utc->date creation-time) "~4"))
> @@ -159,9 +162,14 @@ CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
>                                    (append %tar-determinism-options
>                                            items
>                                            (map symlink-source symlinks))))
> -                    (for-each delete-file-recursively
> -                              (map (compose topmost-component symlink-source)
> -                                   symlinks)))))
> +                    (begin
> +                      (for-each delete-file-recursively
> +                                (map (compose topmost-component symlink-source)
> +                                     symlinks))
> +                      (zero? (apply system* "tar" "-C" extra-items-dir
> +                                    "-rf" "layer.tar"
> +                                    (append %tar-determinism-options
> +                                            '("."))))))))
>  
>             (with-output-to-file "config.json"
>               (lambda ()
> @@ -181,3 +189,6 @@ CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
>                                      '())
>                                ".")))
>                (begin (delete-file-recursively directory) #t)))))
> +
> +(define* (raw-disk-image->docker-image raw-image)
> +  (display "Doing the docker stuff!"))
> diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
> index 21fea446a..8d8053fca 100644
> --- a/guix/scripts/pack.scm
> +++ b/guix/scripts/pack.scm
> @@ -41,7 +41,10 @@
>    #:use-module (srfi srfi-9)
>    #:use-module (srfi srfi-37)
>    #:use-module (ice-9 match)
> -  #:export (compressor?
> +  #:export (%compressors
> +            compressor-extension
> +            compressor-command
> +            compressor?
>              lookup-compressor
>              self-contained-tarball
>              guix-pack))
> diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
> index e50f1d8ac..a319692d7 100644
> --- a/guix/scripts/system.scm
> +++ b/guix/scripts/system.scm
> @@ -638,8 +638,9 @@ any, are available.  Raise an error if they're not."
>                                              #:mappings mappings))
>      ((disk-image)
>       (system-disk-image os
> -                        #:name (match file-system-type
> +                        #:name (match (pk file-system-type)
>                                   ("iso9660" "image.iso")
> +                                 ("docker" "docker-image")
>                                   (_         "disk-image"))
>                          #:disk-image-size image-size
>                          #:file-system-type file-system-type))))
> -- 
> 2.14.2
> 





-- 

Re: Building Docker images of GuixSD

[Konrad Hinsen]

Hi Chris,

> I've run GuixSD in a Docker container and returned to tell the tale!
Congratulations! And thanks for exploring all this.

 > Is this helpful?  Is it worth polishing up and maintaining?  I'm not
 > entirely sure, and I'd like to know what you think.

I think it is useful, mainly for reason 3:

> * If you want to run Guix on a system to which Guix hasn't been ported
>    (like macOS) but your system does run Docker, now you can run Guix on
>    that system by running it from a GuixSD Docker container.

To which I might add a less obvious one: with GuixSD nicely integrated 
into the Docker universe, it has a better chance of adoption by people 
committed to Docker, and thus a better chance of becoming a/the 
preferred way of constructing Docker images. In other words, an attempt 
to take over the (Docker) world from inside.

> For the third bullet point, I don't know of any other reasonable way to
> get Guix working in Docker (although one could certainly run Guix in a
> VM using a technology other than Docker, such as QEMU).  To run Guix,

I have been trying this approach for a while, but I am still much 
further away from running Guix on my Mac than you are.

First, QEMU: forget it on the Mac. It's not nearly as advanced/stable as 
it is under Linux. You can run Guix with QEMU under macOS, but it's slow 
and crashes a bit too often to rely on it.

I moved on to VirtualBox, which runs a basic GuixSD without any major 
problem. But what you get is an isolated virtual machine. I haven't yet 
found a usable strategy for accessing the macOS file system from GuixSD. 
VirtualBox relies on its proprietary guest OS add-ins. I suspect they 
could be ported to GuixSD from a technical point of view, but it's not a 
trivial job and you'd have to remove the term "free software" from your 
brain for a while to do it. Currently I am trying NFS, exporting my Mac 
home directory via an NFS server on the Mac (easy) and mounting it from 
GuixSD (no success so far, because of the very incomplete NFS support in 
GuixSD). I expect this will work eventually, but in terms of performance 
it will probably never get to what you can achieve with Docker.

Konrad.

Re: Building Docker images of GuixSD

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> Run GuixSD in Docker
> ====================
>
> The attached patch makes it possible to build a GuixSD Docker image from
> an operating system configuration file.

For some reason, I had overlooked this message, but it’s awesome!

> Pretty neat!

Yup!

> Problems I Noticed
> ==================

[...]

> Second, I noticed the following error in the Guix daemon's logs.  It
> might be benign, since package installation worked fine, but I'm not
> sure what it means or how to debug it:
>
>     error in finalization thread: Bad file descriptor

I’ve noticed this since we use Shepherd on Guile 2.2, but I haven’t
checked where that comes from; it doesn’t seem to be a serious issue.
;-)  Anyway, it’s not related to your experiment.

> Third, I noticed that the shepherd failed to start syslogd and nscd (and
> user-homes, although I wasn't as concerned about that because the home
> directory for alice did in fact get created).

[...]

> I thought maybe syslogd wasn't working because /dev/log hadn't been
> created in the Docker image, so I tried creating it manually.  However,
> that didn't help; the Shepherd still couldn't start syslogd.

Hmm, I would have thought /dev/log was the issue.  Any other hints?

> Fourth, I wasn't able to run GuixSD in a Docker container without
> supplying the "--privileged" option.  GuixSD writes to sysfs during boot
> (I don't know why, but the details are apparently in
> guix/gnu/build/activation.scm), so the only way to get GuixSD to start
> is to run the container in privileged mode.  This is unfortunate,
> because privileged mode sounds quite dangerous for a lot of reasons.

I don’t think so: there’s a special case for when one creates a
container with ‘guix system container’ that disables this kind of thing.
I guess we should use it here.  It’s mostly about passing #:container? #f 
somewhere.

> From 25d5527b14302fc835af5c338bf37cf621c63a4e Mon Sep 17 00:00:00 2001
> From: Chris Marusich <cmmarusich@gmail.com>
> Date: Sat, 21 Oct 2017 14:40:58 -0700
> Subject: [PATCH] Make it possible to build GuixSD docker images
>
> ---
>  gnu/build/linux-boot.scm    |   5 +-
>  gnu/build/vm.scm            |  14 ++--
>  gnu/system/linux-initrd.scm |  12 ++--
>  gnu/system/vm.scm           | 169 ++++++++++++++++++++++++++++++++++++++------
>  guix/docker.scm             |  23 ++++--
>  guix/scripts/pack.scm       |   5 +-
>  guix/scripts/system.scm     |   3 +-
>  7 files changed, 191 insertions(+), 40 deletions(-)

[...]

> +      (cond ((string=? "iso9660" file-system-type)
> +             (iso9660-image #:name name
> +                            #:file-system-label root-label
> +                            #:file-system-uuid root-uuid
> +                            #:os-drv os-drv
> +                            #:register-closures? #t
> +                            #:bootcfg-drv bootcfg
> +                            #:bootloader (bootloader-configuration-bootloader
> +                                           (operating-system-bootloader os))
> +                            #:inputs `(("system" ,os-drv)
> +                                       ("bootcfg" ,bootcfg))))
> +            ((string=? "docker" file-system-type)
> +             (display "made it to docker image part\n")
> +             (os-docker-image #:name name
> +                              #:os-drv os-drv
> +                              #:register-closures? #t))

I’m not sure this is the right place for it since “docker” is not a file
system type.  Perhaps we need a separate procedure instead?

> @@ -106,7 +107,9 @@ return \"a\"."
>                               #:key closure compressor
>                               (symlinks '())
>                               (system (utsname:machine (uname)))
> -                             (creation-time (current-time time-utc)))
> +                             (creation-time (current-time time-utc))
> +                             (tmpdir "/tmp")
> +                             extra-items-dir)
>    "Write to IMAGE a Docker image archive from the given store PATH.  The image
>  contains the closure of PATH, as specified in CLOSURE (a file produced by
>  #:references-graphs).  SYMLINKS must be a list of (SOURCE -> TARGET) tuples
> @@ -116,7 +119,7 @@ binaries at PATH are for; it is used to produce metadata in the image.
>  
>  Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE.  Use
>  CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata."
> -  (let ((directory "/tmp/docker-image")           ;temporary working directory
> +  (let ((directory (string-append tmpdir "/docker-image")) ;temporary working directory

Why do we need that?  Would it be enough to honor $TMPDIR?

> --- a/guix/scripts/system.scm
> +++ b/guix/scripts/system.scm
> @@ -638,8 +638,9 @@ any, are available.  Raise an error if they're not."
>                                              #:mappings mappings))
>      ((disk-image)
>       (system-disk-image os
> -                        #:name (match file-system-type
> +                        #:name (match (pk file-system-type)
>                                   ("iso9660" "image.iso")
> +                                 ("docker" "docker-image")
>                                   (_         "disk-image"))
>                          #:disk-image-size image-size
>                          #:file-system-type file-system-type))))

Perhaps we could have a separate ‘guix system docker-image’ command?  Or
‘guix system container -f docker’?  WDYT?

Once we’ve sorted out these minor issues, it would be great if you could
send polished patches.  This is something we should add!

Thank you,
Ludo’.

Re: Building Docker images of GuixSD

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2709 bytes --]


Chris Marusich writes:

> Hi Ludo and others following along,

...

> Thanks for reading this far.  I look forward to hearing your thoughts!

Awesome stuff Chris, I've tried this myself, on a Debian machine with
Docker installed.

I struggled getting root, as su and sudo didn't seem to work, until I
realised I could just replace alice with root in the "docker exec"
command...

Anyway, when I got root, I could tell that the system hadn't come up
correctly. The problem seemed to be related to cgroups.

  root@komputilo /# mount -t cgroup /sys/fs/cgroup/cpu
  mount: /sys/fs/cgroup/cpu: cgroup already mounted on
  /sys/fs/cgroup/systemd.

  root@komputilo /# herd status
  Started:
   + file-system-/sys/fs/cgroup/perf_event
   + file-system-/dev/shm
   + host-name
   + root
   + file-system-/sys/fs/cgroup
   + file-system-/sys/fs/cgroup/cpuset
   + file-system-/dev/pts
   + user-file-systems
   + root-file-system
   + file-system-/gnu/store
   + file-system-/sys/fs/cgroup/freezer
   + file-system-/sys/fs/cgroup/memory
   + file-system-/sys/fs/cgroup/devices
   + file-system-/sys/fs/cgroup/blkio
  Stopped:
   - file-system-/sys/fs/cgroup/hugetlb
   - file-system-/sys/fs/cgroup/cpuacct
   - file-system-/sys/fs/cgroup/cpu
   - guix-daemon
   - file-systems
   - syslogd
   - urandom-seed
   - nscd
   - user-homes
   - user-processes

  root@komputilo /# herd start guix-daemon
  herd: exception caught while executing 'start' on service 'file-system-/sys/fs/cgroup/cpu':
  ERROR: In procedure mount: mount "cgroup" on "///sys/fs/cgroup/cpu": Device or resource busy


I changed the %base-file-systems in the very-bare-bones system with
%container-file-systems, and then things started working.

I tried without privileged mode, and got a error related to the firmware
service. This isn't included when you build call
operating-system-derivation with the #:container? #t argument, and sure
enough I was able to get the system up without the Docker --privileged
flag. I think Ludo mentioned this in his reply.

Unfortunately, while I could get a shell using "docker exec ...", I had
to start the guix-daemon manually as the shepherd service didn't seem to
work, at least initially. Also, when I had started it, I tried
installing a package, and there was some promising output to start off
with, but then it failed with:

  guix package: error: build failed: cloning builder process: Operation
  not permitted

Anyway, this is all pretty great! Awesome work getting this far. I'm
very excited to see what services will run this way, as Docker could
provide, albeit with some overhead, a layer of interoperability between
software that can handle Docker containers, and Guix.

Thanks again,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Building Docker images of GuixSD

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> Unfortunately, while I could get a shell using "docker exec ...", I had
> to start the guix-daemon manually as the shepherd service didn't seem to
> work, at least initially. Also, when I had started it, I tried
> installing a package, and there was some promising output to start off
> with, but then it failed with:
>
>   guix package: error: build failed: cloning builder process: Operation
>   not permitted

Presumably this is about clone(2) creating a child process with separate
namespaces.  At first sight I can’t think of an obvious reason why it
wouldn’t work.

Anyway, that’s great progress already, I think GuixSD containers are
useful even without guix-daemon, for the (hopefully common) case of
stateless containers.

Cheers,
Ludo’.

Re: Building Docker images of GuixSD

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

Hi,

Christopher Baines <mail@cbaines.net> writes:

> Awesome stuff Chris, I've tried this myself, on a Debian machine with
> Docker installed.

Great!  It's heartening to know that I'm not the only one tinkering with
this.  :-)

> I changed the %base-file-systems in the very-bare-bones system with
> %container-file-systems, and then things started working.

I see.  I'll have to try this, too.

> I tried without privileged mode, and got a error related to the firmware
> service. This isn't included when you build call
> operating-system-derivation with the #:container? #t argument, and sure
> enough I was able to get the system up without the Docker --privileged
> flag. I think Ludo mentioned this in his reply.

Excellent!  I'll have to do that, too.

> Unfortunately, while I could get a shell using "docker exec ...", I had
> to start the guix-daemon manually as the shepherd service didn't seem to
> work, at least initially. Also, when I had started it, I tried
> installing a package, and there was some promising output to start off
> with, but then it failed with:
>
>   guix package: error: build failed: cloning builder process: Operation
>   not permitted

Huh.  I'll look into this.

> Anyway, this is all pretty great! Awesome work getting this far. I'm
> very excited to see what services will run this way, as Docker could
> provide, albeit with some overhead, a layer of interoperability between
> software that can handle Docker containers, and Guix.

I agree!  Thank you for taking the time to test this out.  It's
extremely helpful to get a second pair of eyes on it.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Building Docker images of GuixSD

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2310 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> Unfortunately, while I could get a shell using "docker exec ...", I had
> to start the guix-daemon manually as the shepherd service didn't seem to
> work, at least initially. Also, when I had started it, I tried
> installing a package, and there was some promising output to start off
> with, but then it failed with:
>
>   guix package: error: build failed: cloning builder process: Operation
>   not permitted
>
> Anyway, this is all pretty great! Awesome work getting this far. I'm
> very excited to see what services will run this way, as Docker could
> provide, albeit with some overhead, a layer of interoperability between
> software that can handle Docker containers, and Guix.

I tried making the changes you suggested.  I launched a container
without using docker's --privileged option.  However, the "boot" script
failed because something couldn't mount something in the container.  I
am not excited about the idea of trying to figure out which esoteric
combination of capabilities [1] are needed to run without the
--privileged option, but I suppose that is necessary if I want to
minimize the container's access to the host system.

To be honest, I find it puzzling that Docker requires me to think so
much about the capabilities in the first place.  Perhaps I'm naive, but
I had hoped that within a Docker container, everything would be
"private" in the sense that, as root in the container, I can do anything
and everything, including mounting, including creating device nodes, and
no changes will be visible outside of the container.  The fact that that
is not the case (at least by default) comes as quite a surprise to me.

I also noticed that some services, like nscd, failed to start.  No error
messages anywhere except Shepherd saying something like "could not start
service nscd".  How do I begin to debug something like that?  All I can
think of is to inspect the Guix code that runs the service, and the
Shepherd code, commit time to learning about it, and then hopefully find
a way to insert debug statements of some kind that give me a hint about
what's going wrong.  Is there a faster, better way?

[1] https://docs.docker.com/engine/security/security/#linux-kernel-capabilities

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]