GSoC ideas

GSoC ideas

[Pjotr Prins]

I have posted a GNU Guix project idea to SciRuby Foundation:

  https://groups.google.com/forum/?fromgroups#!topic/sciruby-dev/CZt75MuUxhg

Basically a student can work on

  - Add JRuby support to Guix
  - Add rbx support to Guix
  - Add all testing frameworks to Guix (e.g. cucumber)
  - Add all web development frameworks to Guix (e.g. RoR)
  - Add Sciruby and related modules to Guix
  - Provide support to Travis-CI (probably through Docker)
  - Provide general Docker support
  - Integrate with IRuby notebook and nyaplot 

Guix has support for containers built-in, so we can even live without
Docker.

If you are interested, sign up with the SciRuby mailing list (sorry
they use Google groups).

Also I encourage other packagers to post project ideas to relevant
organisations (R, SciPy anyone?). 

Pj.

Re: GSoC ideas

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> I have posted a GNU Guix project idea to SciRuby Foundation:
>
>   https://groups.google.com/forum/?fromgroups#!topic/sciruby-dev/CZt75MuUxhg

Nice!   (Well, except for Google Groups. ;-))

For the record, GNU is applying as an organization.  Guix will be
participating under the aegis of GNU as in previous years, so interested
parties are welcome to start brainstorming!

Ludo’.

Re: GSoC ideas

[Christopher Allan Webber]

Maybe we should have a place to collect ideas?

Anyway, here would be my top items for GSoC:

 - An installer wizard (written in Guile of course!)
 - g-expressions which generate packages of the Guix package manager for
   .deb and .rpm based distros, and work to get those upstream in Debian
   and Fedora
 - Perhaps some kind of non-emacs, non-cli package (and system
   management?) interface.  Maybe GTK based?

Re: GSoC ideas

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

On Sat, 06 Feb 2016 10:54:34 -0800
Christopher Allan Webber <cwebber@dustycloud.org> wrote:

> Maybe we should have a place to collect ideas?
> 
> Anyway, here would be my top items for GSoC:
> 
>  - An installer wizard (written in Guile of course!)
>  - g-expressions which generate packages of the Guix package manager for
>    .deb and .rpm based distros, and work to get those upstream in Debian
>    and Fedora
>  - Perhaps some kind of non-emacs, non-cli package (and system
>    management?) interface.  Maybe GTK based?
> 

In the interest of full disclosure, I should note that in all likelyhood I
will still qualify as a student for this years GSoC.

The three ideas that I had so far that are floating around in a text file are:

- arm64 port -- This could probably be extended into getting a working system
  into place to have a u-boot -> grub loader, and/or using kexec to get a
  booted system to reboot into an arm libre-kernel. (for armv7 and arm64?)

- ncurses installer -- my idea was Debian's installer can be considered ugly,
  but it does have everything needed to install a system.

- GaPMaCt - Guix's amazing package manager and configure tool -- Same
  non-emacs package & system manager as above. Maybe a backend with a GTK or
  ncurses frontend.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: GSoC ideas

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> The three ideas that I had so far that are floating around in a text file are:
>
> - arm64 port -- This could probably be extended into getting a working system
>   into place to have a u-boot -> grub loader, and/or using kexec to get a
>   booted system to reboot into an arm libre-kernel. (for armv7 and arm64?)

For aarch64 the first task would be to port Guix (not GuixSD) to that
architecture (info "(guix) Bootstrapping").  If everything goes well, it
can be relatively simple, but sometimes things don’t go well.  ;-)

Otherwise it’s surely an eligible project IMO.

> - ncurses installer -- my idea was Debian's installer can be considered ugly,
>   but it does have everything needed to install a system.

Right, that would be nice.

> - GaPMaCt - Guix's amazing package manager and configure tool -- Same
>   non-emacs package & system manager as above. Maybe a backend with a GTK or
>   ncurses frontend.

+1!

Ludo’.

Re: GSoC ideas

[myglc2]

Efraim Flashner <efraim@flashner.co.il> writes:
[...]
>
> - ncurses installer -- my idea was Debian's installer can be considered ugly,
>   but it does have everything needed to install a system.

Couldn't agree more. Don't know if you are still considering this idea,
but reading your comment and a couple more recent GSoC GuixSD installer
proposals prompted me to write some general thoughts about Guix
installation which you might find interesting:

http://article.gmane.org/gmane.comp.gnu.guix.devel/18206

Best - George

Re: GSoC ideas

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On Sat, Feb 06, 2016 at 10:54:34AM -0800, Christopher Allan Webber wrote:
>Maybe we should have a place to collect ideas?
>
>Anyway, here would be my top items for GSoC:
>
> - An installer wizard (written in Guile of course!)
> - g-expressions which generate packages of the Guix package manager for
>   .deb and .rpm based distros, and work to get those upstream in Debian
>   and Fedora

What would be purpose of that? Do you plan to use some other store path so it
won't collide with Guix maintained installation on the same machine?

OTOH it may be interesting to generate only (in my case) RPM metadata
so native system's package manager can track Guix generated files and
use pre/post install scripts for interactions with Guix (to
install/remove/protect them). That would glue Guix to system even
better. Just wild idea when thinking about your idea...

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: GSoC ideas

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1379 bytes --]

On Mon, 8 Feb 2016 11:45:30 +0100
Tomáš Čech <sleep_walker@gnu.org> wrote:

> On Sat, Feb 06, 2016 at 10:54:34AM -0800, Christopher Allan Webber wrote:
> >Maybe we should have a place to collect ideas?
> >
> >Anyway, here would be my top items for GSoC:
> >
> > - An installer wizard (written in Guile of course!)
> > - g-expressions which generate packages of the Guix package manager for
> >   .deb and .rpm based distros, and work to get those upstream in Debian
> >   and Fedora  
> 
> What would be purpose of that? Do you plan to use some other store path so it
> won't collide with Guix maintained installation on the same machine?
> 
> OTOH it may be interesting to generate only (in my case) RPM metadata
> so native system's package manager can track Guix generated files and
> use pre/post install scripts for interactions with Guix (to
> install/remove/protect them). That would glue Guix to system even
> better. Just wild idea when thinking about your idea...
> 
> S_W

like checkinstall? It would be nice if a foreign distro to have graphical
programs just work, integrated into the system like their own native programs.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: GSoC ideas

[Christopher Allan Webber]

Tomáš Čech writes:

> On Sat, Feb 06, 2016 at 10:54:34AM -0800, Christopher Allan Webber wrote:
>>Maybe we should have a place to collect ideas?
>>
>>Anyway, here would be my top items for GSoC:
>>
>> - An installer wizard (written in Guile of course!)
>> - g-expressions which generate packages of the Guix package manager for
>>   .deb and .rpm based distros, and work to get those upstream in Debian
>>   and Fedora
>
> What would be purpose of that? Do you plan to use some other store path so it
> won't collide with Guix maintained installation on the same machine?
>
> OTOH it may be interesting to generate only (in my case) RPM metadata
> so native system's package manager can track Guix generated files and
> use pre/post install scripts for interactions with Guix (to
> install/remove/protect them). That would glue Guix to system even
> better. Just wild idea when thinking about your idea...
>
> S_W

Sorry, I was unclear.  What I'm talking about is Guix *as a package
manager* to be able to be available on foreigh distros so I can
encourage friends to do:

  apt-get install guix

... and then do user-space hacking with Guix, and use "guix environment"
for hacking on my projects...  I think we'd increase our userbase
dramatically by getting Guix as a user-space pacakge manager into
various distros.

The gexp part was because of the gexp that (iirc) generates the "tarball
install" method of things as inspiration... could we build something
similar that builds a .deb or .rpm of Guix as an install method?

 - Chris

Re: GSoC ideas

[Thompson, David]

On Mon, Feb 8, 2016 at 12:24 PM, Christopher Allan Webber
<cwebber@dustycloud.org> wrote:
> What I'm talking about is Guix *as a package
> manager* to be able to be available on foreigh distros so I can
> encourage friends to do:
>
>   apt-get install guix
>
> ... and then do user-space hacking with Guix, and use "guix environment"
> for hacking on my projects...  I think we'd increase our userbase
> dramatically by getting Guix as a user-space pacakge manager into
> various distros.

Such packages would be convenient for users of these distros, but
getting them into Debian and Fedora proper is unlikely because Guix
does not conform to the FHS.

- Dave

Re: GSoC ideas

[Christopher Allan Webber]

Thompson, David writes:

> On Mon, Feb 8, 2016 at 12:24 PM, Christopher Allan Webber
> <cwebber@dustycloud.org> wrote:
>> What I'm talking about is Guix *as a package
>> manager* to be able to be available on foreigh distros so I can
>> encourage friends to do:
>>
>>   apt-get install guix
>>
>> ... and then do user-space hacking with Guix, and use "guix environment"
>> for hacking on my projects...  I think we'd increase our userbase
>> dramatically by getting Guix as a user-space pacakge manager into
>> various distros.
>
> Such packages would be convenient for users of these distros, but
> getting them into Debian and Fedora proper is unlikely because Guix
> does not conform to the FHS.
>
> - Dave

So, I talked about this with Stefano Zacchiroli while at FOSDEM.  He
suggested that we could do /opt/g/ or something like that and we *could*
get Guix in as a package manager to Debian.  Sure, it wouldn't be able
to use the Hydra packages, but it's probably okayish.

(We *might* be able to get Guix in even using the Hydra packages if the
Guix daemon could have a wrapper which bind-mounts it to /gnu/ or
something, when it starts up?  That might be a bit over the top, but
maybe would allow us to have both.  I'm not so sure about this strategy
though, I've just been bouncing it around in my head.)

We could also build apt / rpm packages which we just provide ourselves,
on the Guix website, or by providing some PPA'ish thing.  I would have
preferred this myself when I started running Debian on Guix... and in
this route we could keep everything as it already is.  (But people will
have to take an extra step to "get to" those nice packages.)

 - Chris

Re: GSoC ideas

[Pjotr Prins]

On Mon, Feb 08, 2016 at 11:47:31AM -0800, Christopher Allan Webber wrote:
> So, I talked about this with Stefano Zacchiroli while at FOSDEM.  He
> suggested that we could do /opt/g/ or something like that and we *could*
> get Guix in as a package manager to Debian.  Sure, it wouldn't be able
> to use the Hydra packages, but it's probably okayish.

Unless users get binaries they won't be happy.

> We could also build apt / rpm packages which we just provide ourselves,
> on the Guix website, or by providing some PPA'ish thing.  I would have
> preferred this myself when I started running Debian on Guix... and in
> this route we could keep everything as it already is.  (But people will
> have to take an extra step to "get to" those nice packages.)

That looks like the way forward to me. Such a package can setup and
start the daemon - which is enough. No need to get the blessing from
the distributions themselves (will take time, but it will come - there
really is no difference with allowing foreign packages to work anyway).

Pj.

Installing Guix on foreign distros

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Mon, Feb 08, 2016 at 11:47:31AM -0800, Christopher Allan Webber wrote:
>> So, I talked about this with Stefano Zacchiroli while at FOSDEM.  He
>> suggested that we could do /opt/g/ or something like that and we *could*
>> get Guix in as a package manager to Debian.  Sure, it wouldn't be able
>> to use the Hydra packages, but it's probably okayish.
>
> Unless users get binaries they won't be happy.

+1

>> We could also build apt / rpm packages which we just provide ourselves,
>> on the Guix website, or by providing some PPA'ish thing.  I would have
>> preferred this myself when I started running Debian on Guix... and in
>> this route we could keep everything as it already is.  (But people will
>> have to take an extra step to "get to" those nice packages.)
>
> That looks like the way forward to me. Such a package can setup and
> start the daemon - which is enough. No need to get the blessing from
> the distributions themselves (will take time, but it will come - there
> really is no difference with allowing foreign packages to work anyway).

Yeah, I think you’re right.

There are already people who wrote Arch/Parabola and RPM (openSuSE)
packages, AFAIK.  I’m not sure what the status is?

If someone familiar with dpkg could chime in, we could host the
infrastructure to build the .deb somewhere.

Ludo’.

Re: Installing Guix on foreign distros

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

On Tue, Feb 09, 2016 at 11:36:24AM +0100, Ludovic Courtès wrote:
>There are already people who wrote Arch/Parabola and RPM (openSuSE)
>packages, AFAIK.  I’m not sure what the status is?

For openSUSE ecosystem I maintain package in system:packagemanagers
project of our Open Build Service. It means it doesn't live in the
main package repository but it is easy to install.

 https://software.opensuse.org/download/package?project=system:packagemanager&package=guix

I'm trying to push that to the main distribution but for that the
package may need some more work because I need to get rid of foreign
binaries (the ones in
$PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
guix' for all openSUSE users is still my goal.

Any help on that from Guix side would be appreciated.

Besides this, thanks to our the OBS I should be able to setup build
for other RPM distributions (Fedora, CentOS) but it wasn't my focus
now.

Best regards,

Tomas

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Installing Guix on foreign distros

[Ricardo Wurmus]

Tomáš Čech <sleep_walker@gnu.org> writes:

> On Tue, Feb 09, 2016 at 11:36:24AM +0100, Ludovic Courtès wrote:
>>There are already people who wrote Arch/Parabola and RPM (openSuSE)
>>packages, AFAIK.  I’m not sure what the status is?
>
> For openSUSE ecosystem I maintain package in system:packagemanagers
> project of our Open Build Service. It means it doesn't live in the
> main package repository but it is easy to install.
>
>  https://software.opensuse.org/download/package?project=system:packagemanager&package=guix
>
> I'm trying to push that to the main distribution but for that the
> package may need some more work because I need to get rid of foreign
> binaries (the ones in
> $PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
> guix' for all openSUSE users is still my goal.

The bootstrap binaries form the roots of the dependency graph.
Modifying them results in different hashes for all packages.  I learned
this when a colleague accidentally modified these files in our
deployment, which made Guix refuse all of the substitutes from Hydra and
instead rebuilt the world locally.

I don’t think you can realistically get rid of them.

~~ Ricardo

Re: Installing Guix on foreign distros

[Jookia]

On Wed, Feb 10, 2016 at 09:53:01AM +0100, Tomáš Čech wrote:
> I'm trying to push that to the main distribution but for that the
> package may need some more work because I need to get rid of foreign
> binaries (the ones in
> $PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
> guix' for all openSUSE users is still my goal.
> 
> Any help on that from Guix side would be appreciated.
> 
> Besides this, thanks to our the OBS I should be able to setup build
> for other RPM distributions (Fedora, CentOS) but it wasn't my focus
> now.
> 
> Best regards,
> 
> Tomas

I brought this up a bit earlier in IRC but you could rebuild the bootstrap
binaries- but then they'd just be identical to the ones shipped. So are they
really foreign then?

Jookia.

Re: Installing Guix on foreign distros

[Ricardo Wurmus]

Jookia <166291@gmail.com> writes:

> On Wed, Feb 10, 2016 at 09:53:01AM +0100, Tomáš Čech wrote:
>> I'm trying to push that to the main distribution but for that the
>> package may need some more work because I need to get rid of foreign
>> binaries (the ones in
>> $PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
>> guix' for all openSUSE users is still my goal.
>> 
>> Any help on that from Guix side would be appreciated.
>> 
>> Besides this, thanks to our the OBS I should be able to setup build
>> for other RPM distributions (Fedora, CentOS) but it wasn't my focus
>> now.
>> 
>> Best regards,
>> 
>> Tomas
>
> I brought this up a bit earlier in IRC but you could rebuild the bootstrap
> binaries- but then they'd just be identical to the ones shipped. So are they
> really foreign then?

I wonder: can you *really* rebuild them?  Would that not require
reproducible builds, something that we try to achieve with Guix?  These
tarballs come from their upstream release servers, if I’m not mistaken.

Since the bootstrapping step starts with statically linked binaries of
tar (and some other tools) and the tarballs that are then unpacked with
that binary of tar, I don’t see how these binaries could possibly
be reproduced *exactly* on a foreign distro and seamlessly replace the
bundled binaries.

It would be amazing and quite a surprise if you *were* able to rebuild
them and thus reproduce them bit for bit.

~~

I wished we could reduce the number of bootstrap binaries down to little
more than just Guile and an old C compiler, i.e. implement a subset of
tar in Guile and use that to unpack the tarballs, then use the simple C
compiler to compile an old version of GCC (without C++), then use that
version of GCC to compile a GCC capable of building C++, then use that
GCC to build a modern GCC...

Since Guile is mostly written in Guile (and it is probably somewhat
easier to come to a point to trust it) there may be value in trying to
bootstrap from even less.

~~

Practically speaking, though, you probably cannot get around the
bootstrap binaries — not unless there was a way to arbitrarily
manipulate hashes or sneakily “seed” hashes at a certain level into the
graph (e.g. tell Guix that the binaries of GCC that are in the system
always have a particular hash even if they do not).

I don’t think a system like this would be desirable in general as it
undermines reproducibility and trust.  (I do see that something like
this might be useful for a shoddy port to MacOS, but ... ugh!  What’s
the point in that!)

~~ Rambling Ricardo

Bootstrapping

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Jookia <166291@gmail.com> writes:

[...]

>> I brought this up a bit earlier in IRC but you could rebuild the bootstrap
>> binaries- but then they'd just be identical to the ones shipped. So are they
>> really foreign then?
>
> I wonder: can you *really* rebuild them?

First, you’d need to build them as explained at
<http://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries>,
but starting from the exact same commit that was used to build them.

> Would that not require reproducible builds, something that we try to
> achieve with Guix?

Indeed, some of the packages in the bootstrap binaries are reproducible,
some are not.  See
<https://lists.gnu.org/archive/html/guix-devel/2013-09/msg00159.html>
for a previous experiment in that area.

> I wished we could reduce the number of bootstrap binaries down to little
> more than just Guile and an old C compiler, i.e. implement a subset of
> tar in Guile and use that to unpack the tarballs, then use the simple C
> compiler to compile an old version of GCC (without C++), then use that
> version of GCC to compile a GCC capable of building C++, then use that
> GCC to build a modern GCC...
>
> Since Guile is mostly written in Guile (and it is probably somewhat
> easier to come to a point to trust it) there may be value in trying to
> bootstrap from even less.

Definitely!  Just like we have cpio, FTP and HTTP clients, etc. in
Scheme, we could incrementally have tiny replacements for some of the
things currently found in the bootstrap binaries.  They don’t need to be
full-featured.

This is one of the topics discussed as the reproducible summit:

  https://lists.gnu.org/archive/html/guix-devel/2015-12/msg00107.html
  https://reproducible-builds.org/events/athens2015/bootstrapping/

Ludo’.

Re: GSoC ideas

[Diane Trout]

> That looks like the way forward to me. Such a package can setup and
> start the daemon - which is enough. No need to get the blessing from
> the distributions themselves (will take time, but it will come -
> there
> really is no difference with allowing foreign packages to work
> anyway).


I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.

https://github.com/detrout/debian-guix

Currently I've only split the guix package into the emacs components
and everything else. I'd thought about splitting the daemon out into
its own package, but I wasn't sure what the daemon depended on. 

The daemon is still using the default /gnu/store path, and the user
needs to manually run guix authorize if they want to use hydra
binaries. The package is currently based on the stable 0.9.0 release,
and I'm not sure how security updates make it into a guix store if you
without updating the scheme packaging source tree.

It might be nice to prompt the user if they wanted to authorize hydra
on install but that's not implemented.

Currently its unlikely to go into Debian because Debian policy requires
everything to be built from source, and currently the Guix build
process downloads some bootstrap binaries.

However with the current packaging "guix environment --pure bash --
bash" does give me a clean guix environment, and the guix info docs get
installed when Debian emacs can see them.

Diane

Guix on Debian (was: GSoC ideas)

[Christopher Allan Webber]

Diane Trout writes:

>> That looks like the way forward to me. Such a package can setup and
>> start the daemon - which is enough. No need to get the blessing from
>> the distributions themselves (will take time, but it will come -
>> there
>> really is no difference with allowing foreign packages to work
>> anyway).
>
>
> I wrote a basic Debian recipe to build guix, create the build users,
> and install the systemd config file.
>
> https://github.com/detrout/debian-guix
>
> Currently I've only split the guix package into the emacs components
> and everything else. I'd thought about splitting the daemon out into
> its own package, but I wasn't sure what the daemon depended on.
>
> The daemon is still using the default /gnu/store path, and the user
> needs to manually run guix authorize if they want to use hydra
> binaries. The package is currently based on the stable 0.9.0 release,
> and I'm not sure how security updates make it into a guix store if you
> without updating the scheme packaging source tree.
>
> It might be nice to prompt the user if they wanted to authorize hydra
> on install but that's not implemented.
>
> Currently its unlikely to go into Debian because Debian policy requires
> everything to be built from source, and currently the Guix build
> process downloads some bootstrap binaries.
>
> However with the current packaging "guix environment --pure bash --
> bash" does give me a clean guix environment, and the guix info docs get
> installed when Debian emacs can see them.
>
> Diane

Great work Diane!

Are those bootstrapping binaries really necessary for getting Guix
going?  I guess for some reason I thought if you did the whole
configure/make/etc dance it wouldn't be but maybe I'm wrong.

Maybe this is a good step towards getting a Guix .deb we self-host on
the Guix website?

 - Chris

Re: Guix on Debian (was: GSoC ideas)

[Leo Famulari]

On Tue, Feb 23, 2016 at 03:52:30PM -0800, Christopher Allan Webber wrote:
> Diane Trout writes:
> 
> >> That looks like the way forward to me. Such a package can setup and
> >> start the daemon - which is enough. No need to get the blessing from
> >> the distributions themselves (will take time, but it will come -
> >> there
> >> really is no difference with allowing foreign packages to work
> >> anyway).
> >
> >
> > I wrote a basic Debian recipe to build guix, create the build users,
> > and install the systemd config file.
> >
> > https://github.com/detrout/debian-guix
> >
> > Currently I've only split the guix package into the emacs components
> > and everything else. I'd thought about splitting the daemon out into
> > its own package, but I wasn't sure what the daemon depended on.
> >
> > The daemon is still using the default /gnu/store path, and the user
> > needs to manually run guix authorize if they want to use hydra
> > binaries. The package is currently based on the stable 0.9.0 release,
> > and I'm not sure how security updates make it into a guix store if you
> > without updating the scheme packaging source tree.
> >
> > It might be nice to prompt the user if they wanted to authorize hydra
> > on install but that's not implemented.
> >
> > Currently its unlikely to go into Debian because Debian policy requires
> > everything to be built from source, and currently the Guix build
> > process downloads some bootstrap binaries.
> >
> > However with the current packaging "guix environment --pure bash --
> > bash" does give me a clean guix environment, and the guix info docs get
> > installed when Debian emacs can see them.
> >
> > Diane
> 
> Great work Diane!
> 
> Are those bootstrapping binaries really necessary for getting Guix
> going?  I guess for some reason I thought if you did the whole
> configure/make/etc dance it wouldn't be but maybe I'm wrong.

My understanding is that if you alter the bootstrap binaries, the entire
dependency graph will change, forcing a rebuild of everything. And of
course, the altered binaries may present different interfaces, breaking
things as well.

Re: Guix on Debian (was: GSoC ideas)

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

>> Are those bootstrapping binaries really necessary for getting Guix
>> going?  I guess for some reason I thought if you did the whole
>> configure/make/etc dance it wouldn't be but maybe I'm wrong.
>
> My understanding is that if you alter the bootstrap binaries, the entire
> dependency graph will change, forcing a rebuild of everything. And of
> course, the altered binaries may present different interfaces, breaking
> things as well.

This is correct.  Back then we ran into trouble with our Guix
installation at work when someone modified permission bits on the
bootstrap binaries, causing a rebuild of everything.  It took us a while
to find out the cause and revert the change.

Some of the bootstrap binaries can be reproduced from source (if you
make sure to follow the Guix recipes), but others (like Guile IIRC)
don’t have reproducible build systems, so reproducing the exact same
binaries without using Guix is going to be very challenging.

I don’t know if it is possible and if it would make sense to cheat,
i.e. just lie to Guix about the hashes of the bootstrap binaries.

~~ Ricardo

Re: Guix on Debian (was: GSoC ideas)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Wed, 24 Feb 2016 10:03:34 +0100
Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> wrote:

> Leo Famulari <leo@famulari.name> writes:
> 
>  [...]  
>  [...]  
> 
> This is correct.  Back then we ran into trouble with our Guix
> installation at work when someone modified permission bits on the
> bootstrap binaries, causing a rebuild of everything.  It took us a while
> to find out the cause and revert the change.
> 
> Some of the bootstrap binaries can be reproduced from source (if you
> make sure to follow the Guix recipes), but others (like Guile IIRC)
> don’t have reproducible build systems, so reproducing the exact same
> binaries without using Guix is going to be very challenging.
> 
> I don’t know if it is possible and if it would make sense to cheat,
> i.e. just lie to Guix about the hashes of the bootstrap binaries.
> 
> ~~ Ricardo
> 

What about taking it a step further and having a multi-level bootstrap
process like when we have the core-updates? If we bootstrap away enough times
would we end up with the bootstrap binaries we have now?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Guix on Debian (was: GSoC ideas)

[Jookia]

On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
> What about taking it a step further and having a multi-level bootstrap
> process like when we have the core-updates? If we bootstrap away enough times
> would we end up with the bootstrap binaries we have now?

From what I understand the bootstrap binaries aren't reproducible yet. If they
were, it would solve this issue as we could build them again.

Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>> What about taking it a step further and having a multi-level bootstrap
>> process like when we have the core-updates? If we bootstrap away enough times
>> would we end up with the bootstrap binaries we have now?
>
> From what I understand the bootstrap binaries aren't reproducible yet.

Depends on what kind of reproducibility we’re talking about.  It’s
simple to build bootstrap binaries:

  https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries

I think they are all bit-for-bit reproducible (that is, you can use
--rounds=3 and everything is fine), except for Guile due to
<http://bugs.gnu.org/20272>.)

However, if you build them today, you’ll obviously get something
different from the bootstrap binaries we currently use, which were from
Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.

HTH,
Ludo’.

Re: Bootstrap binaries

[Christopher Allan Webber]

Ludovic Courtès writes:

> Jookia <166291@gmail.com> skribis:
>
>> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>>> What about taking it a step further and having a multi-level bootstrap
>>> process like when we have the core-updates? If we bootstrap away enough times
>>> would we end up with the bootstrap binaries we have now?
>>
>> From what I understand the bootstrap binaries aren't reproducible yet.
>
> Depends on what kind of reproducibility we’re talking about.  It’s
> simple to build bootstrap binaries:
>
>   https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
>
> I think they are all bit-for-bit reproducible (that is, you can use
> --rounds=3 and everything is fine), except for Guile due to
> <http://bugs.gnu.org/20272>.)
>
> However, if you build them today, you’ll obviously get something
> different from the bootstrap binaries we currently use, which were from
> Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
>
> HTH,
> Ludo’.

It seems like a good idea, once that bug in Guile is fixed, to move over
to a new set of bootstrap binaries... even if this involves some
difficulty for Guix users today.  It would certainly be a good thing to
do before we hit 1.0, whenever that is.

Re: Bootstrap binaries

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:
>
>> Jookia <166291@gmail.com> skribis:
>>
>>> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>>>> What about taking it a step further and having a multi-level bootstrap
>>>> process like when we have the core-updates? If we bootstrap away enough times
>>>> would we end up with the bootstrap binaries we have now?
>>>
>>> From what I understand the bootstrap binaries aren't reproducible yet.
>>
>> Depends on what kind of reproducibility we’re talking about.  It’s
>> simple to build bootstrap binaries:
>>
>>   https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
>>
>> I think they are all bit-for-bit reproducible (that is, you can use
>> --rounds=3 and everything is fine), except for Guile due to
>> <http://bugs.gnu.org/20272>.)
>>
>> However, if you build them today, you’ll obviously get something
>> different from the bootstrap binaries we currently use, which were from
>> Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
>>
>> HTH,
>> Ludo’.
>
> It seems like a good idea, once that bug in Guile is fixed, to move over
> to a new set of bootstrap binaries... even if this involves some
> difficulty for Guix users today.  It would certainly be a good thing to
> do before we hit 1.0, whenever that is.

I prefer to change those binaries as rarely as possible.  Intuitively
(and unscientifically), it gives more confidence to keep using the same
old binaries wrt. Ken Thompson attacks.

Ludo’.

Re: Bootstrap binaries

[Jookia]

On Sat, Feb 27, 2016 at 12:19:04AM +0100, Ludovic Courtès wrote:
> I prefer to change those binaries as rarely as possible.  Intuitively
> (and unscientifically), it gives more confidence to keep using the same
> old binaries wrt. Ken Thompson attacks.

I'm not sure about that, if we could establish the binaries could be
reproducibly built using the current bootstrap binaries it sounds like it could
be fine. Having reproducible bootstrap binaries seems like something incredibly
useful especially for packagers that for whatever reason want to verify that the
binaries can be built with Guix before signing them.

> Ludo’.

Jookia.

Re: Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Sat, Feb 27, 2016 at 12:19:04AM +0100, Ludovic Courtès wrote:
>> I prefer to change those binaries as rarely as possible.  Intuitively
>> (and unscientifically), it gives more confidence to keep using the same
>> old binaries wrt. Ken Thompson attacks.
>
> I'm not sure about that, if we could establish the binaries could be
> reproducibly built using the current bootstrap binaries it sounds like it could
> be fine. Having reproducible bootstrap binaries seems like something incredibly
> useful especially for packagers that for whatever reason want to verify that the
> binaries can be built with Guix before signing them.

We would have to update them every time we change GCC, Guile, Coreutils,
etc. or one of their dependencies, which sounds impractical or even
infeasible to me.

Ludo’.

Re: Bootstrap binaries

[Jookia]

On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
> We would have to update them every time we change GCC, Guile, Coreutils,
> etc. or one of their dependencies, which sounds impractical or even
> infeasible to me.

Perhaps there's some miscommunication here- the problem right now is that the
binaries aren't reproducible at all. Having them switched to reproducible builds
once and then tagging the Guix commit used would probably be enough to fix the
problem. I don't think we need to rebuild them every time.

> Ludo’.

Jookia.

Re: Bootstrap binaries

[Christopher Allan Webber]

Jookia writes:

> On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
>> We would have to update them every time we change GCC, Guile, Coreutils,
>> etc. or one of their dependencies, which sounds impractical or even
>> infeasible to me.
>
> Perhaps there's some miscommunication here- the problem right now is that the
> binaries aren't reproducible at all. Having them switched to reproducible builds
> once and then tagging the Guix commit used would probably be enough to fix the
> problem. I don't think we need to rebuild them every time.

Yes, this is what I was originally suggesting!

Re: Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
>> We would have to update them every time we change GCC, Guile, Coreutils,
>> etc. or one of their dependencies, which sounds impractical or even
>> infeasible to me.
>
> Perhaps there's some miscommunication here- the problem right now is that the
> binaries aren't reproducible at all. Having them switched to reproducible builds
> once and then tagging the Guix commit used would probably be enough to fix the
> problem. I don't think we need to rebuild them every time.

Ah sure, I agree!  Though to be on the safe side, we’d need to do
diverse double compilation (DDC) of these binaries, à la Wheeler.

Ludo’.

Re: Guix on Debian (was: GSoC ideas)

[Diane Trout]

> Great work Diane!
> 
> Are those bootstrapping binaries really necessary for getting Guix
> going?  I guess for some reason I thought if you did the whole
> configure/make/etc dance it wouldn't be but maybe I'm wrong.
> 
> Maybe this is a good step towards getting a Guix .deb we self-host on
> the Guix website?

The bootstrap binaries are used to compute the hashes for Guix's
initial packages which then influence all of the downstream hashes.

I know that shipping sourceless binaries is against Debian policy. I do
wonder if there's a way to produce the same bootstrap binaries from
Debian components.

Also one of your other messages suggested the /gnu directory is also
against Debian policy.

If there's a way resolve those things, there's probably a much better
chance to get it into Debian.

But other than that I think my postrm script is incomplete and
certainly needs some more testing but this can certainly turn into
.debs hosted by guix.

Diane

Re: Guix on Debian (was: GSoC ideas)

[Pjotr Prins]

On Tue, Feb 23, 2016 at 04:32:30PM -0800, Diane Trout wrote:
> Also one of your other messages suggested the /gnu directory is also
> against Debian policy.
>
> If there's a way resolve those things, there's probably a much better
> chance to get it into Debian.
 
Yes, this in old discussion (also in Nix). For sure, if guix built in
/usr/local and $HOME it would be much easier to get by the policy
folks (also in HPC environments).

There is, however, a good reason to make it a blatant /gnu outside the
FHS. The store contains everything that normally goes into the full
FHS. It lives outside the FHS. If we make it part of the FHS (even in
one subdirectory) it would confuse things badly. I think the clarity
matters to system administrators AND users.

Also, for true reproducibility, we need one path. Arguably it could be
a different one, but to make sure a package is exactly the same in its
binary form it needs to be running from a root point.

It took me quite some time to come round to this idea. But now I am
really convinced it is the only way forward. Administrators can use
chroot and containers/VM to mount /gnu - so, there is no real
technical concern.

Guix is not tied to one server. If they think it important, the Debian
people can, at some point, decide to take Guix, create their own
caching server, and distribute /usr/local/guix. They have done far
more intrusive things ;)

For us acceptance in Debian is not a prime concern either. An
installable .deb package is good enough to help regular users.
Therefore I don't think you need to fix the binary bootstrapping as
long as we can make it part of the automated build farm.

> But other than that I think my postrm script is incomplete and
> certainly needs some more testing but this can certainly turn into
> .debs hosted by guix.
> 
> Diane

Thanks for the great work! A .deb package would be very useful for
people to start using Guix.

Pj.
-- 

Re: Guix on Debian (was: GSoC ideas)

[Thompson, David]

On Wed, Feb 24, 2016 at 2:04 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:

> For us acceptance in Debian is not a prime concern either. An
> installable .deb package is good enough to help regular users.
> Therefore I don't think you need to fix the binary bootstrapping as
> long as we can make it part of the automated build farm.

+1

Getting a Guix package into Debian is really not important, especially
given the amount of nasty hacks it would require to attempt to appease
the Debian developers.

Hosting a .deb file on our own server that users could download and
install with dpkg would be perfect for us.

Thanks for the hard work, Diane!

- Dave

Re: Guix on Debian (was: GSoC ideas)

[Diane Trout]

> Hosting a .deb file on our own server that users could download and
> install with dpkg would be perfect for us.

Actually the best thing to do would be to put the debs into a signed 
repository, for example https://mirrorer.alioth.debian.org/ is a utility that 
lets you create your own apt-gettable repositories.

The guix on debian instructions would then be:

apt-key add (signing key)
add "deb http://<server>/debian guix main" to 
/etc/apt/sources.list.d/guix.list
apt-get update
apt-get install guix

this allows apt-get update to get new versions.

The nix hydra paper implies that it can build packages and installers for non-
nix systems. I assume the guix version can as well?

> Thanks for the hard work, Diane!

You're welcome, I'm glad this may be useful to others. :)

Diane

Re: Guix on Debian (was: GSoC ideas)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1619 bytes --]

On Wed, Feb 24, 2016 at 11:36:52AM -0800, Diane Trout wrote:
> > Hosting a .deb file on our own server that users could download and
> > install with dpkg would be perfect for us.
> 
> Actually the best thing to do would be to put the debs into a signed 
> repository, for example https://mirrorer.alioth.debian.org/ is a utility that 
> lets you create your own apt-gettable repositories.
> 
> The guix on debian instructions would then be:
> 
> apt-key add (signing key)
> add "deb http://<server>/debian guix main" to 
> /etc/apt/sources.list.d/guix.list
> apt-get update
> apt-get install guix
> 
> this allows apt-get update to get new versions.

Another option would be to include in the pre/post install script to add
the repo if it's not already there. I know some third party repos delete
and recreate their files, my `tor+http` setting keeps on getting
overwritten.

> The nix hydra paper implies that it can build packages and installers for non-
> nix systems. I assume the guix version can as well?
> 
> > Thanks for the hard work, Diane!
> 
> You're welcome, I'm glad this may be useful to others. :)
> 
> Diane
> 

I haven't looked too closely at the code yet, so appologies if you've
already taken care of it. In the guix manual it says not to install the
binary installer over an existing install, does the deb you've created
make sure to not do that?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Guix on Debian (was: GSoC ideas)

[Diane Trout]

[-- Attachment #1: Type: text/plain, Size: 1539 bytes --]

On Thursday, February 25, 2016 12:30:47 PM Efraim Flashner wrote:

> Another option would be to include in the pre/post install script to add
> the repo if it's not already there. I know some third party repos delete
> and recreate their files, my `tor+http` setting keeps on getting
> overwritten.

I know some packages automatically install repository keys, I don't really 
like it.

> 
> I haven't looked too closely at the code yet, so appologies if you've
> already taken care of it. In the guix manual it says not to install the
> binary installer over an existing install, does the deb you've created
> make sure to not do that?

No. Other than the basic checks from dpkg where it will complain if it tries 
to overwrite a file.

This package is somewhat inferior to the binary installation instructions as 
it currently stands the guix scheme libraries go into 
/usr/share/guile/site/2.0/guix.

I discovered this causes trouble by trying to use a guix git tree I'd 
compiled. That didn't work right until after I'd rebuilt it with:
$ guix environment guix 
$ ./bootstrap ; ./configure --prefix=/usr --localstatedir=/var

I'm wondering if the debian build script should build guix, then and then do 

$ guix package --no-substitutes -i guix

And install what ended up in /gnu/store

However I haven't checked to see if there's a way to start the guix-daemon in 
temporary location. The Debian build utilities assume it can cause things to 
be built in a subdirectory of the source tree by manipulating environment 
variables.

Diane

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: GSoC ideas

[Jan Nieuwenhuizen]

Diane Trout writes:

> I wrote a basic Debian recipe to build guix, create the build users,
> and install the systemd config file.
>
> https://github.com/detrout/debian-guix

Thank you, this is /really/ useful.  I want to introduce Guix at work
and was thinking about a debian package.

Greetings, Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: GSoC ideas

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

> Diane Trout writes:
>
>> I wrote a basic Debian recipe to build guix, create the build users,
>> and install the systemd config file.
>>
>> https://github.com/detrout/debian-guix
>
> Thank you, this is /really/ useful.  I want to introduce Guix at work
> and was thinking about a debian package.

For Ubuntu, I used the patch below.

Greetings,
Jan

diff --git a/README.rst b/README.rst
index 7092389..0e6b228 100644
--- a/README.rst
+++ b/README.rst
@@ -22,7 +22,7 @@ To build you need to do something like the following:
     cd guix
     git clone https://github.com/detrout/debian-guix.git debian
     uscan --download-current-version
-    tar xavf ../guix-$(dpkg-parsechangelog -S Version | cut -f 1 -d -).orig.tar.gz --strip-components=1
+    tar xavf ../guix_$(dpkg-parsechangelog -S Version | cut -f 1 -d -).orig.tar.gz --strip-components=1
     sudo apt-get install build-essential dh-autoreconf dh-systemd autotools-dev graphviz guile-2.0-dev guile-json help2man libgcrypt20-dev libsqlite3-dev libbz2-dev texinfo
     dpkg-buildpackage
     sudo dpkg -i ../guix_0.9.0-1_amd64.deb ../emacs-guix_0.9.0-1_all.deb
diff --git a/watch b/watch
index 5e7fecb..972498e 100644
--- a/watch
+++ b/watch
@@ -1,3 +1,2 @@
-version=4
-opts="pgpsigurlmangle=s/$/.sig/" \
+version=3
 http://alpha.gnu.org/gnu/guix/ guix-([\d\.]+)\.tar\.gz debian uupdate

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: GSoC ideas

[Diane Trout]

> For Ubuntu, I used the patch below.
> 

Thank you.

I fixed the typo and switched to using a uscan version=3 config file with the 
Ludovic Courtès signing key in upstream/signing-key.asc so the uscan key 
validation works correctly.

Does the updated uscan file work correctly on *buntu?

I also committed a change to the postrm script as I learned of a few more 
directories that should be removed on --purge.

Diane

Re: GSoC ideas

[Ludovic Courtès]

Diane Trout <diane@ghic.org> skribis:

> I wrote a basic Debian recipe to build guix, create the build users,
> and install the systemd config file.
>
> https://github.com/detrout/debian-guix

Nice!

> Currently I've only split the guix package into the emacs components
> and everything else. I'd thought about splitting the daemon out into
> its own package, but I wasn't sure what the daemon depended on. 

What you describe sounds good enough.

> Currently its unlikely to go into Debian because Debian policy requires
> everything to be built from source, and currently the Guix build
> process downloads some bootstrap binaries.

Without a fully automated process to build .deb for several distros, I
don’t think we can offer to distribute .deb ourselves.  :-/

If you’re versed in Guix, you might be interested in writing a
derivation that runs Debian in QEMU and invokes the commands that build
.deb files, as Chris suggested.  That would be awesome.

Ludo’.

Re: GSoC ideas

[Diane Trout]

> 
> If you’re versed in Guix, you might be interested in writing a
> derivation that runs Debian in QEMU and invokes the commands that build
> .deb files, as Chris suggested.  That would be awesome.

I don't currently know enough about guix to do that, but I think it'd be great 
if I did.

Diane

Re: GSoC ideas

[myglc2]

Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
> Sorry, I was unclear.  What I'm talking about is Guix *as a package
> manager* to be able to be available on foreigh distros so I can
> encourage friends to do:
>
>   apt-get install guix
>
> ... and then do user-space hacking with Guix, and use "guix environment"
> for hacking on my projects...  I think we'd increase our userbase
> dramatically by getting Guix as a user-space pacakge manager into
> various distros.

I strongly agree with this. To my mind this is the key selling point of
Guix and we need to make it easy to experience this.

But you know, given how well the Guix Binary install + a lame makefile
has worked to install/uninstall on my Debian servers, and the way Guix
kind of includes it's own world, I'm not so sure a .deb package
necessary.

> The gexp part was because of the gexp that (iirc) generates the "tarball
> install" method of things as inspiration... could we build something
> similar that builds a .deb or .rpm of Guix as an install method?
>
>  - Chris

Maybe a downloadable install package that does a binary install and is
routinely verified to run on 3 or 4 popular distros is just as good and
has more band for the buck.

Anyway, reading your comments and a couple other recent GSoC GuixSD
installer proposals prompted me to write some general thoughts about
Guix installation which you might find interesting:

http://lists.gnu.org/archive/html/guix-devel/2016-03/msg01024.html

Best - George

Re: GSoC ideas

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Maybe we should have a place to collect ideas?

What about creating a page at <https://libreplanet.org/wiki/Group:Guix>?

> Anyway, here would be my top items for GSoC:
>
>  - An installer wizard (written in Guile of course!)

For GuixSD I guess?  Sounds like good idea, of course.

>  - g-expressions which generate packages of the Guix package manager for
>    .deb and .rpm based distros, and work to get those upstream in Debian
>    and Fedora

That would be useful, and CheckInstall may be good enough for that.  I
don’t think it’s “enough” for GSoC, though.  WDYT?

>  - Perhaps some kind of non-emacs, non-cli package (and system
>    management?) interface.  Maybe GTK based?

Definitely.  Both a guile-curses-based and a GTK+-based thing would be
nice.  For the latter we need to check whether PackageKit can be of any
use (this has been discussed in the past and my impression was that it
is biased towards “traditional” distros with just a single root-managed
profile.)

Thanks,
Ludo’.

Re: GSoC ideas

[Christopher Allan Webber]

Ludovic Courtès writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>> Maybe we should have a place to collect ideas?
>
> What about creating a page at <https://libreplanet.org/wiki/Group:Guix>?

https://libreplanet.org/wiki/Group:Guix/GSoC-2016

Here you go!

Just commenting on one comment:

>>  - g-expressions which generate packages of the Guix package manager for
>>    .deb and .rpm based distros, and work to get those upstream in Debian
>>    and Fedora
>
> That would be useful, and CheckInstall may be good enough for that.  I
> don’t think it’s “enough” for GSoC, though.  WDYT?

If the student was really to package Guix for many distributions, it
might be enough.  Especially if that student was to explore packaging
with upstream.  Anyway, the payout could be *huge*.

If they finish early, I'm sure we could find other things for them to
do?  :)

 - Chris

Re: GSoC ideas

[Pjotr Prins]

On Tue, Feb 09, 2016 at 02:17:08PM -0800, Christopher Allan Webber wrote:
> >>  - g-expressions which generate packages of the Guix package manager for
> >>    .deb and .rpm based distros, and work to get those upstream in Debian
> >>    and Fedora
> >
> > That would be useful, and CheckInstall may be good enough for that.  I
> > don’t think it’s “enough” for GSoC, though.  WDYT?
> 
> If the student was really to package Guix for many distributions, it
> might be enough.  Especially if that student was to explore packaging
> with upstream.  Anyway, the payout could be *huge*.

One thing to keep in mind is that GSoC is about programming. So, we
have to be clear about that in the proposal.

> If they finish early, I'm sure we could find other things for them to
> do?  :)

Better to put a secondary job in - such as improving the web-ui or
creating a tool for importing deb packages, or something. Students
will come if they think the job challenging/interesting enough.

Pj.

Re: GSoC ideas

[Christopher Allan Webber]

Pjotr Prins writes:

> On Tue, Feb 09, 2016 at 02:17:08PM -0800, Christopher Allan Webber wrote:
>> >>  - g-expressions which generate packages of the Guix package manager for
>> >>    .deb and .rpm based distros, and work to get those upstream in Debian
>> >>    and Fedora
>> >
>> > That would be useful, and CheckInstall may be good enough for that.  I
>> > don’t think it’s “enough” for GSoC, though.  WDYT?
>> 
>> If the student was really to package Guix for many distributions, it
>> might be enough.  Especially if that student was to explore packaging
>> with upstream.  Anyway, the payout could be *huge*.
>
> One thing to keep in mind is that GSoC is about programming. So, we
> have to be clear about that in the proposal.
>
>> If they finish early, I'm sure we could find other things for them to
>> do?  :)
>
> Better to put a secondary job in - such as improving the web-ui or
> creating a tool for importing deb packages, or something. Students
> will come if they think the job challenging/interesting enough.
>
> Pj.

So, while we're on the topic of hopes and dreams that I might
*personally* love to see solved through GSoC, but maybe people will be
like "That's not enough of a GSoC programming project"....

npm packaging and importer.  Even better: enough npm packaging and
importing to where we package pump.io :)

It might not be the most fun GSoC project though?

Re: GSoC ideas

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Pjotr Prins writes:
>
>> On Tue, Feb 09, 2016 at 02:17:08PM -0800, Christopher Allan Webber wrote:
>>> >>  - g-expressions which generate packages of the Guix package manager for
>>> >>    .deb and .rpm based distros, and work to get those upstream in Debian
>>> >>    and Fedora
>>> >
>>> > That would be useful, and CheckInstall may be good enough for that.  I
>>> > don’t think it’s “enough” for GSoC, though.  WDYT?
>>> 
>>> If the student was really to package Guix for many distributions, it
>>> might be enough.  Especially if that student was to explore packaging
>>> with upstream.  Anyway, the payout could be *huge*.
>>
>> One thing to keep in mind is that GSoC is about programming. So, we
>> have to be clear about that in the proposal.
>>
>>> If they finish early, I'm sure we could find other things for them to
>>> do?  :)
>>
>> Better to put a secondary job in - such as improving the web-ui or
>> creating a tool for importing deb packages, or something. Students
>> will come if they think the job challenging/interesting enough.
>>
>> Pj.
>
> So, while we're on the topic of hopes and dreams that I might
> *personally* love to see solved through GSoC, but maybe people will be
> like "That's not enough of a GSoC programming project"....

We could frame it in terms of programming: providing a Guix API for the
creation of Debian etc. packages.

There is (used to be?) something in Nixpkgs that produces a derivation
to build stuff in a Debian VM, build the .deb with CheckInstall, and
return that .deb as the derivation’s output.  Something like this is
definitely programming, and possibly fun as well.  :-)

Could you maybe add more details for this one on the wiki page?

> npm packaging and importer.  Even better: enough npm packaging and
> importing to where we package pump.io :)

Woow, why not!  Point the candidate to your blog post on that topic.
;-)

Ludo’.

Re: GSoC ideas

[Christopher Allan Webber]

Ludovic Courtès writes:

>> So, while we're on the topic of hopes and dreams that I might
>> *personally* love to see solved through GSoC, but maybe people will be
>> like "That's not enough of a GSoC programming project"....
>
> We could frame it in terms of programming: providing a Guix API for the
> creation of Debian etc. packages.
>
> There is (used to be?) something in Nixpkgs that produces a derivation
> to build stuff in a Debian VM, build the .deb with CheckInstall, and
> return that .deb as the derivation’s output.  Something like this is
> definitely programming, and possibly fun as well.  :-)
>
> Could you maybe add more details for this one on the wiki page?
>
>> npm packaging and importer.  Even better: enough npm packaging and
>> importing to where we package pump.io :)
>
> Woow, why not!  Point the candidate to your blog post on that topic.
> ;-)
>
> Ludo’.

Okay, I updated the page:

  https://libreplanet.org/wiki/Group:Guix/GSoC-2016

Re: GSoC ideas

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:

[...]

>>> npm packaging and importer.  Even better: enough npm packaging and
>>> importing to where we package pump.io :)
>>
>> Woow, why not!  Point the candidate to your blog post on that topic.
>> ;-)
>>
>> Ludo’.
>
> Okay, I updated the page:
>
>   https://libreplanet.org/wiki/Group:Guix/GSoC-2016

Cool, thanks!

Now is really the time to add new ideas or expound items already on this
page!

There are a couple of things regarding the Shepherd that could be done,
notably
<http://www.gnu.org/software/soc-projects/ideas-2015.html#dmd_units>.

It could be useful to continue on the GNUnet idea at
<http://www.gnu.org/software/soc-projects/ideas-2015.html#guix>, or to
look at alternative approaches or P2P stacks.

Someone wants add new items to the wiki page starting from that?  :-)

Thanks,
Ludo’.

Re: GSoC ideas

[myglc2]

ludo@gnu.org (Ludovic Courtès) writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>> Maybe we should have a place to collect ideas?
>
> What about creating a page at <https://libreplanet.org/wiki/Group:Guix>?
>
>> Anyway, here would be my top items for GSoC:
>>
>>  - An installer wizard (written in Guile of course!)
>
> For GuixSD I guess?  Sounds like good idea, of course.

Possibly out of scope, but, I have found that there are many ways to
install Guix. It would help to simplify this. what if this installer
could do something like this:

Examine the environment & determine the possible ways that Guix and/or
GuixSD can be installed and then walk the user through choices:

- netboot?
 - fdisk & GuixSD?
- GNU/Linux system?
 - ?root user?
  - install Guix?
  - put GuixSD netboot on USB?
  - install GuixSD to disk?
  - install GuixSD dual boot?
 - running in user space?
  - guix installed?
   - set up git clone?
  - guix not present?
   - can you sudo?
    - yes - got to '?root user' above
    - no
     - install Guix in user space?
     - put GuixSD netboot on USB?

At the end of the process, the user-lever user should have a functioning
Guix setup in which all guix features described in the manual are
working, (except those requiring root, e.g. guix system reconfigure).

Re: GSoC ideas

[Manolis Ragkousis]

Hello everyone,

As I am still a student I would like to suggest continuing the project
of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
having a working GuixSD/Hurd by the end of the summer.

After discussions with Justus and Samuel in FOSDEM about the Guix-Hurd
port I am starting to have a good idea on what is left to be done and
the solution needed for the problems we have.

So here's a preliminary timeline as well as a list of things to be done:

Before May:

1) Merge wip-hurd branch.
2) Make the daemon handle chroot builds on the Hurd.
Note here that on the Hurd, one does not need to be root to achieve
isolation, so I should change the daemon to use this capability.
3) Instead of using the Linux syscall guile wrappers, I should modify
Guix to use a more Hurdish way (i.e settrans) so later on we can handle
translators and bootstrap a GNU/Hurd system.
4) Better understand how GuixSD startup works.

May-August:
1) Implement the mechanisms for creating and mounting the initial
filesystem and starting the system.
2) Implement the mechanisms to handle the Hurd servers in /hurd. I
remember I was told that there may be an issue when the servers are not
actually there (i.e all the binaries are in the /gnu/store). Would love
if somebody could tell us more on that.
3) Isolate Linuxisms around Guix.
4) David Michael has made an excelent work on getting Shepherd working
with Hurd so I don't think we will have any serious issues with the init
daemon.
5) Others that will come up.. In the following weeks I will work closely
with the Hurd guys to make sure we get a better understanding of
everything and a detailed timeline.

End of GSoC:
Have a working GuixSD with Hurd as the kernel.

Currently Justus (and others soon :-)) are testing Guix on their
machines. We have some issues but we are working on them.

Please feel free to add/correct anything from the above. Your
comment/opinion may help to get things up and running faster :-)

Finally I want to say that I will continue my work on this, regardless
of GSoC or not, but I would like it if it was. :-)

Thank you,
Manolis

Re: GSoC ideas

[Andreas Enge]

On Thu, Feb 11, 2016 at 01:46:30PM +0200, Manolis Ragkousis wrote:
> As I am still a student I would like to suggest continuing the project
> of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
> having a working GuixSD/Hurd by the end of the summer.

Sounds great, I hope it will go through!

Andreas

Re: GSoC ideas

[Alex Sassmannshausen]

Andreas Enge writes:

> On Thu, Feb 11, 2016 at 01:46:30PM +0200, Manolis Ragkousis wrote:
>> As I am still a student I would like to suggest continuing the project
>> of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
>> having a working GuixSD/Hurd by the end of the summer.
>
> Sounds great, I hope it will go through!
>
> Andreas

+1 :-)

Re: GSoC ideas

[Christopher Allan Webber]

Manolis Ragkousis writes:

> Hello everyone,
>
> As I am still a student I would like to suggest continuing the project
> of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
> having a working GuixSD/Hurd by the end of the summer.
>
> After discussions with Justus and Samuel in FOSDEM about the Guix-Hurd
> port I am starting to have a good idea on what is left to be done and
> the solution needed for the problems we have.
>
> So here's a preliminary timeline as well as a list of things to be done:
>
> Before May:
>
> 1) Merge wip-hurd branch.
> 2) Make the daemon handle chroot builds on the Hurd.
> Note here that on the Hurd, one does not need to be root to achieve
> isolation, so I should change the daemon to use this capability.
> 3) Instead of using the Linux syscall guile wrappers, I should modify
> Guix to use a more Hurdish way (i.e settrans) so later on we can handle
> translators and bootstrap a GNU/Hurd system.
> 4) Better understand how GuixSD startup works.
>
> May-August:
> 1) Implement the mechanisms for creating and mounting the initial
> filesystem and starting the system.
> 2) Implement the mechanisms to handle the Hurd servers in /hurd. I
> remember I was told that there may be an issue when the servers are not
> actually there (i.e all the binaries are in the /gnu/store). Would love
> if somebody could tell us more on that.
> 3) Isolate Linuxisms around Guix.
> 4) David Michael has made an excelent work on getting Shepherd working
> with Hurd so I don't think we will have any serious issues with the init
> daemon.
> 5) Others that will come up.. In the following weeks I will work closely
> with the Hurd guys to make sure we get a better understanding of
> everything and a detailed timeline.
>
> End of GSoC:
> Have a working GuixSD with Hurd as the kernel.
>
> Currently Justus (and others soon :-)) are testing Guix on their
> machines. We have some issues but we are working on them.
>
> Please feel free to add/correct anything from the above. Your
> comment/opinion may help to get things up and running faster :-)
>
> Finally I want to say that I will continue my work on this, regardless
> of GSoC or not, but I would like it if it was. :-)
>
> Thank you,
> Manolis

I can't speak for everyone else but it feels like this would be a great
project for you to contininue.  The guile-emacs project ran for a few
years, so I don't see any blocker on continuing, right?

Re: GSoC ideas

[Samuel Thibault]

Hello,

Just sending my big +1 on this project :)

Samuel

Re: GSoC ideas

[Richard Braun]

On Sun, Feb 28, 2016 at 11:12:25PM +0100, Samuel Thibault wrote:
> Just sending my big +1 on this project :)

Same here.

I'd like to note that I'm personally pushing so that we have less
students but with better projects. This one definitely fits.

-- 
Richard Braun

Re: GSoC ideas

[Ludovic Courtès]

Hi!

I realize I hadn’t responded to Manolis.  Another +1 from me!

Some random comments:

Manolis Ragkousis <manolis837@gmail.com> skribis:

> Before May:
>
> 1) Merge wip-hurd branch.
> 2) Make the daemon handle chroot builds on the Hurd.
> Note here that on the Hurd, one does not need to be root to achieve
> isolation, so I should change the daemon to use this capability.

I think an ideal situation would be if libc provides ‘mount’ and
‘umount’, with MS_BIND implemented in terms of firmlinks.

I remember Roland was not thrilled with the idea of adding ‘mount’ and
‘umount’, but it would clearly help porting.  I don’t know if this task
should be part of the GSoC project though, and you may need guidance
from the Hurd folks.  What do people think?

> 3) Instead of using the Linux syscall guile wrappers, I should modify
> Guix to use a more Hurdish way (i.e settrans) so later on we can handle
> translators and bootstrap a GNU/Hurd system.

Possibly, depending on whether libc provides ‘mount’ and ‘umount’, which
would make things simpler.

> End of GSoC:
> Have a working GuixSD with Hurd as the kernel.

\o/

Cheers,
Ludo’.

Re: GSoC ideas

[Samuel Thibault]

Ludovic Courtès, on Wed 02 Mar 2016 11:09:28 +0100, wrote:
> Manolis Ragkousis <manolis837@gmail.com> skribis:
> > 1) Merge wip-hurd branch.
> > 2) Make the daemon handle chroot builds on the Hurd.
> > Note here that on the Hurd, one does not need to be root to achieve
> > isolation, so I should change the daemon to use this capability.
> 
> I think an ideal situation would be if libc provides ‘mount’ and
> ‘umount’, with MS_BIND implemented in terms of firmlinks.
> 
> I remember Roland was not thrilled with the idea of adding ‘mount’ and
> ‘umount’, but it would clearly help porting.

It can both help and hurt. Roland's concern is that we will probably not
want to provide a mount() that behaves like Linux, because in some cases
it may just not even make sense. So the "ported" software that would try
to use it would just fail.

Samuel

Re: GSoC ideas

[Ludovic Courtès]

Samuel Thibault <samuel.thibault@gnu.org> skribis:

> Ludovic Courtès, on Wed 02 Mar 2016 11:09:28 +0100, wrote:
>> Manolis Ragkousis <manolis837@gmail.com> skribis:
>> > 1) Merge wip-hurd branch.
>> > 2) Make the daemon handle chroot builds on the Hurd.
>> > Note here that on the Hurd, one does not need to be root to achieve
>> > isolation, so I should change the daemon to use this capability.
>> 
>> I think an ideal situation would be if libc provides ‘mount’ and
>> ‘umount’, with MS_BIND implemented in terms of firmlinks.
>> 
>> I remember Roland was not thrilled with the idea of adding ‘mount’ and
>> ‘umount’, but it would clearly help porting.
>
> It can both help and hurt. Roland's concern is that we will probably not
> want to provide a mount() that behaves like Linux, because in some cases
> it may just not even make sense. So the "ported" software that would try
> to use it would just fail.

Do you have examples of GNU/Linux uses that would make no sense?

Other options include calling out to the ‘settrans’ command (inelegant
IMO), writing Guile bindings for some of the Hurd libraries, and/or some
sort of a MiG in Scheme (neat but takes some time!).

Ludo’.

Re: GSoC ideas

[Samuel Thibault]

Ludovic Courtès, on Wed 02 Mar 2016 22:31:41 +0100, wrote:
> Samuel Thibault <samuel.thibault@gnu.org> skribis:
> > It can both help and hurt. Roland's concern is that we will probably not
> > want to provide a mount() that behaves like Linux, because in some cases
> > it may just not even make sense. So the "ported" software that would try
> > to use it would just fail.
> 
> Do you have examples of GNU/Linux uses that would make no sense?

I haven't investigated, I just report what kind of fear Roland was
having.

Samuel

[GSoC] Porting GuixSD to GNU Hurd draft

[Manolis Ragkousis]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

Hello everyone,

Although I have uploaded and shared my draft to the GSoC website, I am
also resending it to the lists as per Ludovic's instruction.

I am sharing it as a txt + pdf.

Here is also the link that I shared to the GSoC website as per Google's
instructions.
https://docs.google.com/document/d/1NEEz_PHM2YZTm8okd2BgPRSc4c89I1fGnwbyeeEUrhw/edit?usp=sharing

This is the time to share your ideas/corrections :-)

Manolis

[-- Attachment #2: PortingGuixSDtoHurd.txt --]
[-- Type: text/plain, Size: 6403 bytes --]

Porting GuixSD to GNU/Hurd

Manolis Ragkousis
manolis837@gmail.com

March 17, 2016


1. Summary

In this project, we would like to port the Guix Software Distribution to GNU Hurd.  By the end of the project, 
Guix will be able to handle and use the available Hurd mechanisms, have the required tools to produce a working, 
bootable, system vm/image with GNU Shepherd as the init daemon and offer the same functionality as you would 
expect from booting into a GNU/Linux GuixSD system. 


2. The Project

The project consists of four main stages

1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
3. Successfully boot into one such system using GNU Shepherd with pid 1.
4. Modify the new Guix system to take advantage of Hurd specific mechanisms.


Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module. 
This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system. 
In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the 
relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs for example. This module will be called 
(guix build hurd). This allows us to re-implement the functionality of the 'settrans' command, as described 
here[1], in Scheme and use that in place of 'mount()', where applicable.

Additionally, we need to make sure that all modules in (guix build *) and (gnu build *) can offer the same 
functionalities on a GNU/Hurd system.  For example (gnu build vm) relies on QEMU's '-kernel' command-line 
option, which is Linux-specific.  On the other hand, in the case of modules like (gnu build linux-boot) or 
(gnu build linux-initrd), which by design are Linux-specific, we will need to provide a separate module with 
equivalent functionality. (gnu system *) modules will require changes as well, in order to be able to use 
modifications that will happen in the (guix build *) and (gnu build *) modules. 

At this point we will be able to generate a working vm image and boot into it.  Guix will be able to use the 
Hurd servers (i.e. /hurd/init) to start the init manager and the system itself.  We will also have to account 
for the fact that servers in /hurd will be symlinks to /gnu/store/*, and modify Guix and/or the Hurd libraries 
accordingly to achieve functionality.  Regarding the init daemon, please note that a precedent already exists 
for starting GuixSD/Hurd with GNU Shepherd as pid 1 (David Michael [2015][2]).

Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific 
mechanisms. 
This includes but is not limited to:
1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer 
isolation similar to Linux containers as described here[3].
2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root 
privileges in order to setup chrooted environments for the builds.  In the Hurd this can be done by setting up a 
translator as described here[4]. 
3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory 
structure from all the files in the individual package directories.

 
3. Estimated Project Timeline

Before March 31
Finish merging the wip-hurd branch to upstream.
Write a (guix build hurd) module which will contain Guile bindings to the RPC stubs like hurd/fs.defs or hurd/exec.defs .

April 1 - April 15
Package missing dependencies (Hurd libs).
Re-implement 'settrans' in scheme.

April 16 - May 1
At this point we will have the tools needed to build a Hurd based file-system. (Milestone 1)
Start working on getting (guix build *) and (gnu build *) modules to work on Hurd.
Make sure '%base-packages' in (gnu system) module work as expected on Hurd.

May 2 - May 22
Create (gnu build hurd-boot) and (gnu build hurd-initrd).
Start working on describing the GNU/Hurd system in (gnu system).

May 23 - 12 June
Modify (gnu system *) modules as needed.
All the modules (guix build *) and (gnu build *) will be working as expected by now.
Try building a GuixSD image. (Milestone 2)

13 June - 23 June
Solve any problems with booting into the system and running GNU Shepherd.

24 June - 9 July
Have a fully working GNU/Hurd system. (Milestone 3)
Make sure all the services/packages run correctly on the new GuixSD/Hurd system and solve any issues.

10 July - 8 August
Start working on getting Hurd specific mechanisms integrated to Guix.

9 August - 23 August
By now all the objectives will have been achieved.
Merge patches, code refactoring, documentation.
Deliver a working GuixSD system image with Hurd as the kernel.



4. About me

My name is Manolis Ragkousis and you can contact me via email or irc ('phant0mas').  I am 22 years old and I am 
studying Informatics Engineering (Computer Science curriculum) in the Department of Informatics Engineering at the 
Technological Educational Institute of Crete.  I am currently in the process of writing my Bachelor's Thesis on 
running L4Linux and L4re/Fiasco-based realtime applications concurrently on a ZedBoard.

I am a Guix contributor since 2014.  My main contribution to the project includes adding GNU/Hurd support to GNU 
Guix which was also part of GSoC 2015. 

I am a supporter of Free Software and I want to help the GNU Guix and GNU Hurd projects in any way I possibly can.  
My belief is that my work could attract more people to get involved into both projects and help in increasing the 
number of developers.  I will work on this project regardless of being accepted or not as I have kept doing for the last 2 years.

For a more detailed self-portrait, please refer to my online CV[6].

This project will be my only activity during the summer and I will work full time on it.

[1]https://www.gnu.org/software/hurd/hurd/settrans.html
[2]https://lists.gnu.org/archive/html/bug-hurd/2015-01/msg00027.html
[3]https://www.gnu.org/software/hurd/hurd/subhurd.html
[4]https://www.gnu.org/software/hurd/hurd/running/chroot.html
[5]https://www.gnu.org/software/hurd/hurd/translator/unionfs.html#stowfs
[6]http://www.manolisragkousis.com/about-me/

[-- Attachment #3: PortingGuixSDtoHurd.pdf --]
[-- Type: application/pdf, Size: 60734 bytes --]

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Samuel Thibault]

Hello,

Manolis Ragkousis, on Mon 21 Mar 2016 14:09:36 +0200, wrote:
> Although I have uploaded and shared my draft to the GSoC website, I am
> also resending it to the lists as per Ludovic's instruction.

Looks very promising :)

Samuel

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Ludovic Courtès]

Hello!

Manolis Ragkousis <manolis837@gmail.com> skribis:

> Although I have uploaded and shared my draft to the GSoC website, I am
> also resending it to the lists as per Ludovic's instruction.

Yeah, the GSoC website makes it possible to provide a link to a text
file, so let’s do that instead of using their SaaSS.

> 2. The Project
>
> The project consists of four main stages
>
> 1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
> 2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
> 3. Successfully boot into one such system using GNU Shepherd with pid 1.
> 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
>
>
> Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module. 
> This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system. 
> In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the 
> relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
> for example.

Note that technically the ‘file_set_translator’ function is in libc:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (dynamic-func "file_set_translator" (dynamic-link))
$1 = #<pointer 0x1675660>
--8<---------------cut here---------------end--------------->8---

> This module will be called (guix build hurd). This allows us to
> re-implement the functionality of the 'settrans' command, as described
> here[1], in Scheme and use that in place of 'mount()', where
> applicable.

Good!  (We might as well call it (hurd …) in fact: (hurd fs) would
correspond to fs.defs, (hurd io) for io.defs, and so on.)

> Additionally, we need to make sure that all modules in (guix build *) and (gnu build *) can offer the same 
> functionalities on a GNU/Hurd system.  For example (gnu build vm) relies on QEMU's '-kernel' command-line 
> option, which is Linux-specific.  On the other hand, in the case of modules like (gnu build linux-boot) or 
> (gnu build linux-initrd), which by design are Linux-specific, we will need to provide a separate module with 
> equivalent functionality. (gnu system *) modules will require changes as well, in order to be able to use 
> modifications that will happen in the (guix build *) and (gnu build *) modules. 

I think it’s important to think about:

  1. How functionality equivalent to linux-{initrd,boot} will be
     implemented on the Hurd.

     In my experience the equivalent thing is simpler on the Hurd,
     esp. if relying on passive translators (see daemons/runsystem.sh in
     the Hurd), though here we’ll probably explicitly activate
     translators at boot time.

  2. How to structure GuixSD code in a way that abstracts the underlying
     OS kernel.

     For instance, we could have a (guix build os) module providing an
     API that works for both kernels, probably close to the Linux one,
     and that would dispatch to the right implementation, the Linux or
     Hurd one.

Both of these are quite a bit of design and implementation work that we
should not underestimate.

> Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific 
> mechanisms. 
> This includes but is not limited to:
> 1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer 
> isolation similar to Linux containers as described here[3].

This is really super optional IMO.  This module is only used by ‘guix
environment --container’, which is an optional feature.

> 2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root 
> privileges in order to setup chrooted environments for the builds.  In the Hurd this can be done by setting up a 
> translator as described here[4]. 

This is more important, and already non-trivial work.  If libc provided
‘mount’ with support for MS_BIND (which I think it should), it would
help a bit, but there’s still the problem of the other Linux name spaces
that are used (the CLONE_NEW* clone(2) flags.)

Thus it may make more sense to write this functionality in guix-daemon
using directly the Hurd interfaces.  Separate PID name spaces, UID name
spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
“just” a matter of giving the child process ports to separate proc,
auth, etc. translators.

In itself this is also a bit of work.  I wonder what the Hurd folks
think about priorities here?

> 3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory 
> structure from all the files in the individual package directories.

Fun but optional.  ;-)

> 3. Estimated Project Timeline
>
> Before March 31
> Finish merging the wip-hurd branch to upstream.

Note that this task depends on, ahem, other people as well.  ;-)

> Write a (guix build hurd) module which will contain Guile bindings to the RPC stubs like hurd/fs.defs or hurd/exec.defs .
>
> April 1 - April 15
> Package missing dependencies (Hurd libs).
> Re-implement 'settrans' in scheme.

Don’t just straight into low-level “details.”  I’d recommend
familiarizing yourself with GuixSD on GNU/Linux, fiddling with bits of
code and so on (you can try safely in a VM with ‘guix system vm’), and
then, familiarizing yourself with the GNU/Hurd startup process (looking
at daemons/runsystem.sh already gives a good idea.)

> April 16 - May 1
> At this point we will have the tools needed to build a Hurd based file-system. (Milestone 1)

“Hurd-based file system”?

> May 2 - May 22
> Create (gnu build hurd-boot) and (gnu build hurd-initrd).
> Start working on describing the GNU/Hurd system in (gnu system).

I know Debian at some point added initrd support to GNU Mach for some
reason, but fundamentally, the whole point of multiboot is to provide a
solution more flexible than initrds.  So, hopefully, no initrds here.
Since there’s no initrd, there’s also no ‘switch_root’ dance (on
Linux-based system the initrd is the initial root file system, so you
need to switch to the real root file system as soon as you’re able to
mount it), which simplifies things.

Justus, Samuel, WDYT?

> May 23 - 12 June
> Modify (gnu system *) modules as needed.
> All the modules (guix build *) and (gnu build *) will be working as expected by now.
> Try building a GuixSD image. (Milestone 2)

This is the crux of the matter.  :-)

Before starting, it would be nice to have a rough idea of how GuixSD
startup will work on the Hurd, and what changes this entails.

For debugging purposes, it would be very helpful to say the least to
have a working ‘guix system vm’: it would allow you to test your changes
very quickly, without rebooting and so on.

This in itself requires some thought: currently (guix system vm) relies
on QEMU’s ‘-kernel’ option to boot the kernel Linux.  For GNU/Hurd, we
instead need to boot a complete image with GRUB, like ‘guix system
vm-image’ does.  You’ll have to investigate how to port this.

> 13 June - 23 June
> Solve any problems with booting into the system and running GNU Shepherd.
>
> 24 June - 9 July
> Have a fully working GNU/Hurd system. (Milestone 3)
> Make sure all the services/packages run correctly on the new GuixSD/Hurd system and solve any issues.
>
> 10 July - 8 August
> Start working on getting Hurd specific mechanisms integrated to Guix.

What do you mean?

> 9 August - 23 August
> By now all the objectives will have been achieved.
> Merge patches, code refactoring, documentation.
> Deliver a working GuixSD system image with Hurd as the kernel.

Victory!  :-)

I think this is super cool, and also super ambitious!  I’d expect that
we’d be able to reach milestone #2 if everything goes well (and assuming
we don’t try to address isolation in guix-daemon), but milestone #3 is
most likely for later.

What do people think?

The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts.  I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.

Justus, Samuel: could you add yourselves as official co-mentors on
Google’s web site, if not already done?  :-)

Thanks, Manolis!

Ludo’.

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Justus Winter]

Hi,

Quoting Ludovic Courtès (2016-03-23 14:40:38)
> > 2. The Project
> >
> > The project consists of four main stages
> >
> > 1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
> > 2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
> > 3. Successfully boot into one such system using GNU Shepherd with pid 1.
> > 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.

For me, 4. is the most important bit, so we can build packages in
isolation.

> > Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module. 
> > This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system. 
> > In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the 
> > relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
> > for example.
> 
> Note that technically the ‘file_set_translator’ function is in libc:
> 
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> (dynamic-func "file_set_translator" (dynamic-link))
> $1 = #<pointer 0x1675660>
> --8<---------------cut here---------------end--------------->8---
> 
> > This module will be called (guix build hurd). This allows us to
> > re-implement the functionality of the 'settrans' command, as described
> > here[1], in Scheme and use that in place of 'mount()', where
> > applicable.

Right.  In fact, what settrans (or mount) does isn't that hard to
reproduce, though I wouldn't mind moving the c implementation to
libhurdutil or the like and binding to that.

> I think it’s important to think about:
> 
>   1. How functionality equivalent to linux-{initrd,boot} will be
>      implemented on the Hurd.
> 
>      In my experience the equivalent thing is simpler on the Hurd,
>      esp. if relying on passive translators (see daemons/runsystem.sh in
>      the Hurd), though here we’ll probably explicitly activate
>      translators at boot time.

Currently, there is not really a reason to have an initrd-like
solution on the Hurd.  Yes, one has to start the default pager
explicitly, but that's about it.

> > Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific 
> > mechanisms. 
> > This includes but is not limited to:
> > 1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer 
> > isolation similar to Linux containers as described here[3].
> 
> This is really super optional IMO.  This module is only used by ‘guix
> environment --container’, which is an optional feature.

Yes, subhurds are more on the experimental side imho.

> > 2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root 
> > privileges in order to setup chrooted environments for the builds.  In the Hurd this can be done by setting up a 
> > translator as described here[4]. 
> 
> This is more important, and already non-trivial work.  If libc provided
> ‘mount’ with support for MS_BIND (which I think it should), it would
> help a bit, but there’s still the problem of the other Linux name spaces
> that are used (the CLONE_NEW* clone(2) flags.)
> 
> Thus it may make more sense to write this functionality in guix-daemon
> using directly the Hurd interfaces.  Separate PID name spaces, UID name
> spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
> “just” a matter of giving the child process ports to separate proc,
> auth, etc. translators.
> 
> In itself this is also a bit of work.  I wonder what the Hurd folks
> think about priorities here?

I'd go for specializing guix-daemon over trying too hard to implement
Linux-compatible interfaces in the libc.  I consider the
filesystem-isolation part easy, UID isolation relatively easy, PID
isolation is a bit tricky.  Currently one cannot simply start another
proc server and expect it to work as it needs privileged kernel
interfaces.  I created an RPC to allow nested proc servers for
unprivileged subhurds.  That should do the trick, it is merely a
matter of making it actually work I guess.

> > 3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory 
> > structure from all the files in the individual package directories.
> 
> Fun but optional.  ;-)

Agreed.

> > May 2 - May 22
> > Create (gnu build hurd-boot) and (gnu build hurd-initrd).
> > Start working on describing the GNU/Hurd system in (gnu system).
> 
> I know Debian at some point added initrd support to GNU Mach for some
> reason, but fundamentally, the whole point of multiboot is to provide a
> solution more flexible than initrds.  So, hopefully, no initrds here.
> Since there’s no initrd, there’s also no ‘switch_root’ dance (on
> Linux-based system the initrd is the initial root file system, so you
> need to switch to the real root file system as soon as you’re able to
> mount it), which simplifies things.
> 
> Justus, Samuel, WDYT?

Debian has the initrd for the live cds, but that is a bit hacky.  I
don't believe we need an initrd at this point.

> > May 23 - 12 June
> > Modify (gnu system *) modules as needed.
> > All the modules (guix build *) and (gnu build *) will be working as expected by now.
> > Try building a GuixSD image. (Milestone 2)
> 
> This is the crux of the matter.  :-)
> 
> Before starting, it would be nice to have a rough idea of how GuixSD
> startup will work on the Hurd, and what changes this entails.
> 
> For debugging purposes, it would be very helpful to say the least to
> have a working ‘guix system vm’: it would allow you to test your changes
> very quickly, without rebooting and so on.
> 
> This in itself requires some thought: currently (guix system vm) relies
> on QEMU’s ‘-kernel’ option to boot the kernel Linux.  For GNU/Hurd, we
> instead need to boot a complete image with GRUB, like ‘guix system
> vm-image’ does.  You’ll have to investigate how to port this.

qemu can boot multiboot operating systems.

> > Deliver a working GuixSD system image with Hurd as the kernel.

Hurd is not a kernel.

> Victory!  :-)
> 
> I think this is super cool, and also super ambitious!  I’d expect that
> we’d be able to reach milestone #2 if everything goes well (and assuming
> we don’t try to address isolation in guix-daemon), but milestone #3 is
> most likely for later.
> 
> What do people think?
> 
> The main question is whether you should implement build isolation in
> guix-daemon, in which case that would leave little time for the GuixSD
> parts.  I think I would rather let you focus on the GuixSD stuff as you
> wrote, but I’d like to hear what the Hurd folks think.

I consider isolation more important.

> Justus, Samuel: could you add yourselves as official co-mentors on
> Google’s web site, if not already done?  :-)

I clicked on 'want to mentor', do I need to do more?

Justus

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Manolis Ragkousis]

Hello Justus, Hello Ludo

On 03/23/16 18:55, Justus Winter wrote:
> Quoting Ludovic Courtès (2016-03-23 14:40:38)
>>> 2. The Project
>>>
>>> The project consists of four main stages
>>>
>>> 1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
>>> 2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
>>> 3. Successfully boot into one such system using GNU Shepherd with pid 1.
>>> 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
> 
> For me, 4. is the most important bit, so we can build packages in
> isolation.

I think our priority should be to first get GuixSD working and then
concentrate to achieving isolation.  From what I understand none of the
two is trivial but in the long run the ability to spawn GNU/Hurd system
vms on the fly will make it easier to work on it.

>>> Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module. 
>>> This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system. 
>>> In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the 
>>> relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
>>> for example.
>>
>> Note that technically the ‘file_set_translator’ function is in libc:
>>
>> --8<---------------cut here---------------start------------->8---
>> scheme@(guile-user)> (dynamic-func "file_set_translator" (dynamic-link))
>> $1 = #<pointer 0x1675660>
>> --8<---------------cut here---------------end--------------->8---
>>
>>> This module will be called (guix build hurd). This allows us to
>>> re-implement the functionality of the 'settrans' command, as described
>>> here[1], in Scheme and use that in place of 'mount()', where
>>> applicable.
> 
> Right.  In fact, what settrans (or mount) does isn't that hard to
> reproduce, though I wouldn't mind moving the c implementation to
> libhurdutil or the like and binding to that.

Then I think it will be wiser to move the c implementation of settrans
to libhurdutil and binding to that.  Then we will only need a (guix
build hurd) module which will include that binding.  I will update the
proposal if you agree.

>> I think it’s important to think about:
>>
>>   1. How functionality equivalent to linux-{initrd,boot} will be
>>      implemented on the Hurd.
>>
>>      In my experience the equivalent thing is simpler on the Hurd,
>>      esp. if relying on passive translators (see daemons/runsystem.sh in
>>      the Hurd), though here we’ll probably explicitly activate
>>      translators at boot time.
> 
> Currently, there is not really a reason to have an initrd-like
> solution on the Hurd.  Yes, one has to start the default pager
> explicitly, but that's about it.

Then it seems we only need a hurd-boot module which will take care of
activating translators and booting the system.

>   2. How to structure GuixSD code in a way that abstracts the
> underlying OS kernel.
>
>      For instance, we could have a (guix build os) module providing an
>      API that works for both kernels, probably close to the Linux one,
>      and that would dispatch to the right implementation, the Linux or
>      Hurd one.

This module sounds like a great idea. I will add it to the proposal.


>>> Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific 
>>> mechanisms. 
>>> This includes but is not limited to:
>>> 1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer 
>>> isolation similar to Linux containers as described here[3].
>>
>> This is really super optional IMO.  This module is only used by ‘guix
>> environment --container’, which is an optional feature.
> 
> Yes, subhurds are more on the experimental side imho.

So Adding subhurd support -> if there is time.
> 
>>> 2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root 
>>> privileges in order to setup chrooted environments for the builds.  In the Hurd this can be done by setting up a 
>>> translator as described here[4]. 
>>
>> This is more important, and already non-trivial work.  If libc provided
>> ‘mount’ with support for MS_BIND (which I think it should), it would
>> help a bit, but there’s still the problem of the other Linux name spaces
>> that are used (the CLONE_NEW* clone(2) flags.)
>>
>> Thus it may make more sense to write this functionality in guix-daemon
>> using directly the Hurd interfaces.  Separate PID name spaces, UID name
>> spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
>> “just” a matter of giving the child process ports to separate proc,
>> auth, etc. translators.
>>
>> In itself this is also a bit of work.  I wonder what the Hurd folks
>> think about priorities here?
> 
> I'd go for specializing guix-daemon over trying too hard to implement
> Linux-compatible interfaces in the libc.  I consider the
> filesystem-isolation part easy, UID isolation relatively easy, PID
> isolation is a bit tricky.  Currently one cannot simply start another
> proc server and expect it to work as it needs privileged kernel
> interfaces.  I created an RPC to allow nested proc servers for
> unprivileged subhurds.  That should do the trick, it is merely a
> matter of making it actually work I guess.

As I said above I believe that first getting GuixSD to work will make it
easier to work on this and test any changes.

>>> 3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory 
>>> structure from all the files in the individual package directories.
>>
>> Fun but optional.  ;-)
> 
> Agreed.

Stowfs support -> if there is time.

>> Before March 31
>> Finish merging the wip-hurd branch to upstream.
>
> Note that this task depends on, ahem, other people as well.  ;-)

Well one can only hope :-)

>>> May 2 - May 22
>>> Create (gnu build hurd-boot) and (gnu build hurd-initrd).
>>> Start working on describing the GNU/Hurd system in (gnu system).
>>
>> I know Debian at some point added initrd support to GNU Mach for some
>> reason, but fundamentally, the whole point of multiboot is to provide a
>> solution more flexible than initrds.  So, hopefully, no initrds here.
>> Since there’s no initrd, there’s also no ‘switch_root’ dance (on
>> Linux-based system the initrd is the initial root file system, so you
>> need to switch to the real root file system as soon as you’re able to
>> mount it), which simplifies things.
>>
>> Justus, Samuel, WDYT?
> 
> Debian has the initrd for the live cds, but that is a bit hacky.  I
> don't believe we need an initrd at this point.

(gnu build hurd-initrd) -> dropped

>>> May 23 - 12 June
>>> Modify (gnu system *) modules as needed.
>>> All the modules (guix build *) and (gnu build *) will be working as expected by now.
>>> Try building a GuixSD image. (Milestone 2)
>>
>> This is the crux of the matter.  :-)
>>
>> Before starting, it would be nice to have a rough idea of how GuixSD
>> startup will work on the Hurd, and what changes this entails.

Currently I am working on familiarizing myself with both how Hurd and
GuixSD booting works. I am already using GuixSD as a system and I am
going though the source code so I can better understand what is going on.
>>
>> For debugging purposes, it would be very helpful to say the least to
>> have a working ‘guix system vm’: it would allow you to test your changes
>> very quickly, without rebooting and so on.
>>
>> This in itself requires some thought: currently (guix system vm) relies
>> on QEMU’s ‘-kernel’ option to boot the kernel Linux.  For GNU/Hurd, we
>> instead need to boot a complete image with GRUB, like ‘guix system
>> vm-image’ does.  You’ll have to investigate how to port this.
> 
> qemu can boot multiboot operating systems.

Justus is correct here. I found this
https://www.gnu.org/software/hurd/hurd/running/qemu.html#index9h1 which
explains how to make qemu start with gnumach.  This way guix system vm
can work, just with the proper modifications.

>>> Deliver a working GuixSD system image with Hurd as the kernel.
> 
> Hurd is not a kernel.

I will rephrase it correctly.  I am sorry again.

>> The main question is whether you should implement build isolation in
>> guix-daemon, in which case that would leave little time for the GuixSD
>> parts.  I think I would rather let you focus on the GuixSD stuff as you
>> wrote, but I’d like to hear what the Hurd folks think.
> 
> I consider isolation more important.

I understand that isolation is more important but I think first getting
GuixSD will be more beneficial in the long run.  If everything goes
well, even though we may not be able to fully implement isolation in the
duration of this gsoc, we can bootstrap the process so we continue after
the end of the summer.

WDYT?

Thank you :-)
Manolis

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Justus Winter]

Quoting Manolis Ragkousis (2016-03-24 12:18:25)
> >>> The project consists of four main stages
> >>>
> >>> 1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
> >>> 2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
> >>> 3. Successfully boot into one such system using GNU Shepherd with pid 1.
> >>> 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
> > 
> > For me, 4. is the most important bit, so we can build packages in
> > isolation.
> 
> I think our priority should be to first get GuixSD working and then
> concentrate to achieving isolation.  From what I understand none of the
> two is trivial but in the long run the ability to spawn GNU/Hurd system
> vms on the fly will make it easier to work on it.

Otoh if we could properly build packages, we could provide the
substitutes, so people could try Guix on the Hurd without going
through the 12h+ bootstrap procedure.

> Currently I am working on familiarizing myself with both how Hurd and
> GuixSD booting works. I am already using GuixSD as a system and I am
> going though the source code so I can better understand what is going on.

Here is an overview of the early server bootstrap in the Hurd.  It is
slightly outdated, but still the best description that I know of:

http://teythoon.cryptobitch.de/posts/bootstrapping-the-hurd/

> >> For debugging purposes, it would be very helpful to say the least to
> >> have a working ‘guix system vm’: it would allow you to test your changes
> >> very quickly, without rebooting and so on.
> >>
> >> This in itself requires some thought: currently (guix system vm) relies
> >> on QEMU’s ‘-kernel’ option to boot the kernel Linux.  For GNU/Hurd, we
> >> instead need to boot a complete image with GRUB, like ‘guix system
> >> vm-image’ does.  You’ll have to investigate how to port this.
> > 
> > qemu can boot multiboot operating systems.
> 
> Justus is correct here. I found this
> https://www.gnu.org/software/hurd/hurd/running/qemu.html#index9h1 which
> explains how to make qemu start with gnumach.  This way guix system vm
> can work, just with the proper modifications.

Hmmm, so one still needs a filesystem, right?  That's going to be a
bit tricky too, since whatever tool you use for that purpose, it
surely does not support creating hurdish passive translator records.
Without passive translator records things get indeed more interesting.
I have a patch for some tool for creating ext2 filesystems that could
help, or we create all the passive translator records on first boot
similar to how Samuels Debian/Hurd live cds deal with that.  Or I
finish my bootshell work that can boot from filesystems without
passive translator records.

Justus

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Manolis Ragkousis]

On 03/24/16 13:38, Justus Winter wrote:
> Otoh if we could properly build packages, we could provide the
> substitutes, so people could try Guix on the Hurd without going
> through the 12h+ bootstrap procedure.

Here the problem is not isolation but the fact that we don't have a
substitutes server.  The machine I had at my house crashed and I cannot
bring it back up remotely. When I get back to my home island and revive
it, this will not be an issue.

> Here is an overview of the early server bootstrap in the Hurd.  It is
> slightly outdated, but still the best description that I know of:
> 
> http://teythoon.cryptobitch.de/posts/bootstrapping-the-hurd/
>

It definitely is!! Nice one!!

> Hmmm, so one still needs a filesystem, right?  That's going to be a
> bit tricky too, since whatever tool you use for that purpose, it
> surely does not support creating hurdish passive translator records.
> Without passive translator records things get indeed more interesting.
> I have a patch for some tool for creating ext2 filesystems that could
> help, or we create all the passive translator records on first boot
> similar to how Samuels Debian/Hurd live cds deal with that.  Or I
> finish my bootshell work that can boot from filesystems without
> passive translator records.

Where can I find more info on those?

Manolis

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Justus Winter]

Quoting Manolis Ragkousis (2016-03-24 13:36:04)
> > Hmmm, so one still needs a filesystem, right?  That's going to be a
> > bit tricky too, since whatever tool you use for that purpose, it
> > surely does not support creating hurdish passive translator records.
> > Without passive translator records things get indeed more interesting.
> > I have a patch for some tool for creating ext2 filesystems that could
> > help, or we create all the passive translator records on first boot
> > similar to how Samuels Debian/Hurd live cds deal with that.  Or I
> > finish my bootshell work that can boot from filesystems without
> > passive translator records.
> 
> Where can I find more info on those?

https://www.gnu.org/software/hurd/hurd-paper.html#translator

Passive translator records are stored in ext2 using the Hurd on-disk
format:

http://git.sceen.net/hurd/hurd.git/blob/HEAD:/ext2fs/ext2_fs.h#l252

Justus

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Ludovic Courtès]

Hi Justus,

Justus Winter <4winter@informatik.uni-hamburg.de> skribis:

> Quoting Ludovic Courtès (2016-03-23 14:40:38)
>> > 2. The Project
>> >
>> > The project consists of four main stages
>> >
>> > 1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
>> > 2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
>> > 3. Successfully boot into one such system using GNU Shepherd with pid 1.
>> > 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
>
> For me, 4. is the most important bit, so we can build packages in
> isolation.

I guess you mean isolation in guix-daemon.

>> This is more important, and already non-trivial work.  If libc provided
>> ‘mount’ with support for MS_BIND (which I think it should), it would
>> help a bit, but there’s still the problem of the other Linux name spaces
>> that are used (the CLONE_NEW* clone(2) flags.)
>> 
>> Thus it may make more sense to write this functionality in guix-daemon
>> using directly the Hurd interfaces.  Separate PID name spaces, UID name
>> spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
>> “just” a matter of giving the child process ports to separate proc,
>> auth, etc. translators.
>> 
>> In itself this is also a bit of work.  I wonder what the Hurd folks
>> think about priorities here?
>
> I'd go for specializing guix-daemon over trying too hard to implement
> Linux-compatible interfaces in the libc.

So this would become maybe half of the GSoC coding part maybe.  WDYT?

> I consider the filesystem-isolation part easy, UID isolation
> relatively easy, PID isolation is a bit tricky.  Currently one cannot
> simply start another proc server and expect it to work as it needs
> privileged kernel interfaces.  I created an RPC to allow nested proc
> servers for unprivileged subhurds.  That should do the trick, it is
> merely a matter of making it actually work I guess.

“Merely”, hmm…  :-)

So, let’s say PID isolation will be optional, and we can always adjust
later on.  Sounds good?

Manolis, make sure to read about how the various Hurd servers provides
these parts of POSIX personality: file systems, UIDs, PIDs, networking,
and so on.

As far as code integration code, I think we won’t bother syncing this
work with nix-daemon; guix-daemon has already diverged, and for instance
it does not have OS X sandbox support.

You’ll have to arrange to have the Hurd-specific bits in a separate
file, so that ‘#ifdef HURD’ are not scattered all over the place.

This is C(++) as you know.  WDYT, Manolis?

>> This in itself requires some thought: currently (guix system vm) relies
>> on QEMU’s ‘-kernel’ option to boot the kernel Linux.  For GNU/Hurd, we
>> instead need to boot a complete image with GRUB, like ‘guix system
>> vm-image’ does.  You’ll have to investigate how to port this.
>
> qemu can boot multiboot operating systems.

Great, didn’t know that.

>> > Deliver a working GuixSD system image with Hurd as the kernel.
>
> Hurd is not a kernel.

I think we use “kernel” to mean “the thing(s) that libc talks to.”

>> The main question is whether you should implement build isolation in
>> guix-daemon, in which case that would leave little time for the GuixSD
>> parts.  I think I would rather let you focus on the GuixSD stuff as you
>> wrote, but I’d like to hear what the Hurd folks think.
>
> I consider isolation more important.

OK.

So, Manolis, what about reframing the agenda such that porting
guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the
programming effort), followed by GuixSD stuff?

Thanks,
Ludo’.

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Manolis Ragkousis]

Hey

On 03/24/16 15:22, Ludovic Courtès wrote:
> So, let’s say PID isolation will be optional, and we can always adjust
> later on.  Sounds good?
> 
> Manolis, make sure to read about how the various Hurd servers provides
> these parts of POSIX personality: file systems, UIDs, PIDs, networking,
> and so on.

Already started. :-)

> As far as code integration code, I think we won’t bother syncing this
> work with nix-daemon; guix-daemon has already diverged, and for instance
> it does not have OS X sandbox support.
> 
> You’ll have to arrange to have the Hurd-specific bits in a separate
> file, so that ‘#ifdef HURD’ are not scattered all over the place.
> 
> This is C(++) as you know.  WDYT, Manolis?

I have to start studying the daemon's code more. From what I know the
part that handles builds is in libstore/build.cc. I will start there.

I think I can do it.

>>> The main question is whether you should implement build isolation in
>>> guix-daemon, in which case that would leave little time for the GuixSD
>>> parts.  I think I would rather let you focus on the GuixSD stuff as you
>>> wrote, but I’d like to hear what the Hurd folks think.
>>
>> I consider isolation more important.
> 
> OK.

Isolation first it is then.

> So, Manolis, what about reframing the agenda such that porting
> guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the
> programming effort), followed by GuixSD stuff?
> 

Current objectives then:
1) Achieve build isolation in the daemon on the Hurd.
2) Modify Guix so it can produce a working image, while isolating any
cases of Linux assumptions.
3) Boot to GuixSD

Ludo, Justus do you agree with this?

Manolis

Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Justus Winter]

Hi,

Quoting Manolis Ragkousis (2016-03-24 14:55:31)
> On 03/24/16 15:22, Ludovic Courtès wrote:
> >>> The main question is whether you should implement build isolation in
> >>> guix-daemon, in which case that would leave little time for the GuixSD
> >>> parts.  I think I would rather let you focus on the GuixSD stuff as you
> >>> wrote, but I’d like to hear what the Hurd folks think.
> >>
> >> I consider isolation more important.
> > 
> > OK.
> 
> Isolation first it is then.
> 
> > So, Manolis, what about reframing the agenda such that porting
> > guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the
> > programming effort), followed by GuixSD stuff?
> > 
> 
> Current objectives then:
> 1) Achieve build isolation in the daemon on the Hurd.
> 2) Modify Guix so it can produce a working image, while isolating any
> cases of Linux assumptions.
> 3) Boot to GuixSD

Well, I didn't ment to stomp over your priorities like that, it is just
that I personally consider it more important and from my point of view
it should be easier to achieve.

Justus

Re: GSoC ideas

[Manolis Ragkousis]

Hey

On 03/02/16 23:31, Ludovic Courtès wrote:
> Do you have examples of GNU/Linux uses that would make no sense?
> 

After a quick look in mount's man page I think that maybe Roland was
referring to some flags (like relatime for example) that when passed to
mount, expect the Linux kernel to handle the filesystem in a specific way.

> Other options include calling out to the ‘settrans’ command (inelegant
> IMO), writing Guile bindings for some of the Hurd libraries, and/or some
> sort of a MiG in Scheme (neat but takes some time!).

We could have a guile wrapper for the ‘settrans’ command that could
cover our needs. Isn't this feasible? WDYT?

Manolis

Re: GSoC ideas

[Ludovic Courtès]

Manolis Ragkousis <manolis837@gmail.com> skribis:

> On 03/02/16 23:31, Ludovic Courtès wrote:

[...]

>> Other options include calling out to the ‘settrans’ command (inelegant
>> IMO), writing Guile bindings for some of the Hurd libraries, and/or some
>> sort of a MiG in Scheme (neat but takes some time!).
>
> We could have a guile wrapper for the ‘settrans’ command that could
> cover our needs. Isn't this feasible? WDYT?

It’s definitely feasible, and probably the easiest option.  It scores
poorly in terms of elegance though.  :-)

Ludo’.

Re: GSoC ideas

[Ben Woodcroft]

On 06/02/16 21:38, Pjotr Prins wrote:
[..]
>    - Add all testing frameworks to Guix (e.g. cucumber)
Hey just fyi I think I just got cucumber working here
https://github.com/wwood/guix_mine/blob/master/ben/packages/rails.scm

but the (7?) patches need finalisation (updating to newest 
versions/synopses/description/lint/reproducibility etc.) before 
submission here.

For Ruby, circular dependencies seem to be an ongoing issue. I'm not 
sure how hard this is, but an error message more useful than "stack 
overflow" or whatever it is would be helpful, perhaps this is something 
a student could work on.

Re: GSoC ideas

[Pjotr Prins]

On Sun, Feb 07, 2016 at 09:53:00AM +1000, Ben Woodcroft wrote:
> 
> 
> On 06/02/16 21:38, Pjotr Prins wrote:
> [..]
> >   - Add all testing frameworks to Guix (e.g. cucumber)
> Hey just fyi I think I just got cucumber working here
> https://github.com/wwood/guix_mine/blob/master/ben/packages/rails.scm
> 
> but the (7?) patches need finalisation (updating to newest
> versions/synopses/description/lint/reproducibility etc.) before
> submission here.

Cool! Your packaging of Rails comes a long way too. 

Of course there is a lot more to package still ;)

> For Ruby, circular dependencies seem to be an ongoing issue. I'm not
> sure how hard this is, but an error message more useful than "stack
> overflow" or whatever it is would be helpful, perhaps this is
> something a student could work on.

Interesting point. Question is whether it is all laziness of upstream,
or that it is a form of bootstrapping the test systems ;). I'll
factor it in if anyone is interested in this project in the SciRuby
world. 

Pj.


--