Guix as a non-optional dependency in another project, and Guix resources requirements.

Guix as a non-optional dependency in another project, and Guix resources requirements.

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 3761 bytes --]

Hi,

I'm one of the co-maintainers of GNU Boot, a project aimed at replace
nonfree boot firmwares that comes with computers with fully free
software.

For that we reuse software like Coreboot, GRUB, SeaBIOS, etc that are
combined into installable images. Most of the code is still inherited
from the latest release(s) of Libreboot that produced fully free
software releases and currently consists in shell scripts that build
these software components (by running ./configure, make etc directly in
the shell scripts).

Right now we only use Guix as an optional dependency to build the
website: if people use './configure --enable-guix', it uses 'guix
time-machine [...] -- shell [...]' under the hood to make the build
reproducible, install all dependencies, etc.

Having Guix optional and not used by default means that people who use
--enable-guix already have Guix installed and take know how to update
it with guix pull.

So if we have Guix as a non-optional dependency, if we take care of
automatizing the installation and update of Guix, users would then be
able to build GNU Boot directly without having to lean Guix basics
first, and worry about learning Guix later on when they contribute to
GNU Boot or Guix directly.

My first tentative was to automatically install Guix from the host
distribution package manager (if Guix is not already there) and update
it to the revision we need (if it's not already at a more recent
revision).

So I took care (in script(s) that are run automatically) of different
distros package managers (apt, pacman, yum, etc) and exporting the right
variables / sourcing the right scripts, and I still need to handle
distributions without a guix package and to enable the substitutes for
new Guix installations.

But beside that I'm not sure I'm on the right track because:
- First and most importantly, running 'guix pull' can fail sometimes,
  especially if there is not enough RAM per core. A fix that work is to
  lower the number of cores used with 'guix pull -c 1 -M 1' for
  instance.

  But the issue is that I'm unsure how much RAM per core is needed.
  Maybe 2GiB or 3GiB? Also if users have heavy desktop usage
  (browser(s) with a ton of tabs open) could that value change?

  I could ask help from GNU Boot users for testing to find that value
  but maybe the Guix community already knows more about this.

  The big downside of my approach here is that if for some reason the
  script I made uses a 'memory per core' value that is too low, it
  would probably makes things way more difficult for contributors than
  having to read the Guix manual upfront in the first place, because it
  would fail and they won't understand why. So they'd need to
  understand both Guix, the code I wrote, and what caused the issue.

- If Guix was already installed on a host distribution, I'm not sure
  how to handle substitutes. Ideally I'd like to be able to enable them
  just for GNU Boot and not change the way they are handled for other
  Guix usage. Also if substitutes are not used it probably affect the
  amount of RAM per core required.

  One way around here could be to build Guix ourselves and somehow find
  a way to have different defaults, but building that Guix could
  probably a long time.

Are there projects that had similar issues / questions?  Or is what I'm
trying to do a can of worms and I'd better point users to the Guix
manual instead?

PS: Is this mail posted on the right mailing list? In one hand these
    questions are about Guix usage, but in another the people who might
    know the answer probably need to have a very good understanding of
    Guix internals and/or of how to write the Guix manual.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[pelzflorian (Florian Pelz)]

Hello Denis.

I believe automatically invoking package managers for users does not
give users control.  Telling them what to run, yes, but running the
commands for them does not seem like good practice, if it can be
avoided.  If I were a user, I would mostly like to understand.

In particular, the GNU Boot description on Savannah says it aims to
replace boot code with 100% free software; the Canoeboot FAQ says it is
impossible to get completely rid of Intel ME.  I have never flashed
Intel/AMD machines, so what do I know.

Last time I checked, guix operations like pull needed almost 2GB RAM,
because of Guile compilation not optimizing for memory use yet.

Regards,
Florian

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Adrien 'neox' Bourmault]

[-- Attachment #1: Type: text/plain, Size: 1673 bytes --]

Le mercredi 20 mars 2024 à 13:15 +0100, pelzflorian (Florian Pelz) a écrit :
> Hello Denis.
> 
> I believe automatically invoking package managers for users does not
> give users control.  Telling them what to run, yes, but running the
> commands for them does not seem like good practice, if it can be
> avoided.  If I were a user, I would mostly like to understand.

Hi Florian, and thanks for your feedback about that. I personally do not agree
with your opinion, because I see Guix like a package manager but also as a
reproductible build system and environment, which is a really good thing.

> 
> In particular, the GNU Boot description on Savannah says it aims to
> replace boot code with 100% free software; the Canoeboot FAQ says it is
> impossible to get completely rid of Intel ME.  I have never flashed
> Intel/AMD machines, so what do I know.

I absolutely don't understand what you're trying to tell here. GNU Boot does not
support any board where Intel ME is mandatory to operate the board. Indeed, it's
impossible to remove Intel ME from boards that requires it, and this is why we
can never support these boards unless someone retroengineers these boards and is
able to find a way to fully get rid of ME.

> 
> Last time I checked, guix operations like pull needed almost 2GB RAM,
> because of Guile compilation not optimizing for memory use yet.

Yes, I know that and this is something I measured myself.

Thanks again for your feedback and happy hacking! 
-- 
Adrien Bourmault
Maintainer, GNU Boot project
Associate member, Free Software Foundation
GPG : 393D4CC68136F39799DA75F295F65F55F682A17A









[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Adrien 'neox' Bourmault]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

Le jeudi 21 mars 2024 à 17:26 +0100, Adrien 'neox' Bourmault a écrit :
> > 
> > Last time I checked, guix operations like pull needed almost 2GB RAM,
> > because of Guile compilation not optimizing for memory use yet.
> 
> Yes, I know that and this is something I measured myself.

Sorry, I did not see earlier this was a question Denis asked. I measured that
recently and did not have the time to tell him. Thanks for your answer.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On Thu, 21 Mar 2024 17:35:25 +0100
Adrien 'neox' Bourmault <neox@gnu.org> wrote:

> Sorry, I did not see earlier this was a question Denis asked. I
> measured that recently and did not have the time to tell him. Thanks
> for your answer.
Thanks a lot for the test. 

The issue here is also that a given measure is very important as it
gives some indication, but I'm also not sure that 2GiB is enough with
all the situations. For instance a user with a browser and tons of open
tabs may need more RAM for Guix.

This is also why my initial plan was to ask many people to test, but I
then figured out that it was probably best to ask Guix users and
contributors first as they might already a precise enough value
that we could use, or have insight on if what I am trying to do is a
good idea or design.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Andreas Enge]

Hello,

Am Fri, Mar 22, 2024 at 01:16:28AM +0100 schrieb Denis 'GNUtoo' Carikli:
> The issue here is also that a given measure is very important as it
> gives some indication, but I'm also not sure that 2GiB is enough with
> all the situations. For instance a user with a browser and tons of open
> tabs may need more RAM for Guix.

I have a virtual machine that essentially runs an nginx instance and had
1GB of memory. This was not enough to do a "guix pull". I asked the people
who provide the machine to me to ramp up to 2GB, but this was still not
enough. Now I dare not ask for more, as I already felt ridiculous enough
asking for 2GB for what essentially amounts to "apt-get update".
This is not exactly good advertisement for our system...

Andreas

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Felix Lechner via]

Hi Andreas,

On Mon, Apr 08 2024, Andreas Enge wrote:

> I have a virtual machine that ... had 1GB of memory. This was not
> enough to do a "guix pull".

I operated a 1 GB web server that ran Guix on Linode for a while.  Did
you configure a swap partition?

Kind regards
Felix

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Andreas Enge]

Hello,

Am Mon, Apr 08, 2024 at 01:32:25PM -0700 schrieb Felix Lechner:
> I operated a 1 GB web server that ran Guix on Linode for a while.  Did
> you configure a swap partition?

actually I did not, good idea!

Am Tue, Apr 09, 2024 at 05:23:56AM -0700 schrieb Felix Lechner:
> There is also 'guix deploy' from a stronger machine, which I recommend
> for any VPS.

And this is another good idea, thanks!

I tried to reproduce the problem, but so far could not; maybe something
changed, maybe I made a mistake previously. I tried to monitor memory
usage by a very crude "cat /proc/meminfo|head -3"; the minimum values
I saw were around these:
MemTotal:        2024516 kB
MemFree:           31768 kB
MemAvailable:     140064 kB
This happened when building guix-base, which seems to be particularly
memory hungry.

So if I understand correctly, 2GB is about the amount of memory needed
for "guix pull".

Now I have been running out of disk space, but this is another problem.

Andreas

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Felix Lechner via]

Hi Andreas,

On Thu, Apr 11 2024, Andreas Enge wrote:

> So if I understand correctly, 2GB is about the amount of memory needed
> for "guix pull".

Probably.  By the way, nothing in my messages should be interpreted to
say that Guix's memory usage is acceptable.  I am merely trying to help.

> Now I have been running out of disk space, but this is another
> problem.

Please have a look at zram [1] which compresses many of your memory
pages and requires no device at all (thus saving valuable disk space),
or zswap [2] which compresses your swap pages.  The latter reduces the
size of your swap partition while keeping the same capacity. [3]

Kind regards
Felix

[1] https://en.wikipedia.org/wiki/Zram
[2] https://en.wikipedia.org/wiki/Zswap
[3] https://stackoverflow.com/questions/18437205/difference-between-zram-and-zswap

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[pelzflorian (Florian Pelz)]

Andreas Enge <andreas@enge.fr> writes:
> I have a virtual machine that essentially runs an nginx instance and had
> 1GB of memory. This was not enough to do a "guix pull". I asked the people
> who provide the machine to me to ramp up to 2GB, but this was still not
> enough. Now I dare not ask for more, as I already felt ridiculous enough
> asking for 2GB for what essentially amounts to "apt-get update".
> This is not exactly good advertisement for our system...
>
> Andreas

If other processeses do indeed consume much of the 2GB, you could make
Guix pull an older commit for which substitutes are already available:

info "(guix)Channels with Substitutes"

Actually maybe 1GB is already enough then.

Regards,
Florian

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Felix Lechner via]

Hi,

On Tue, Apr 09 2024, pelzflorian (Florian Pelz) wrote:

> you could make Guix pull an older commit

There is also 'guix deploy' from a stronger machine, which I recommend
for any VPS.

It's a breeze to set up but does currently come with some security
implications.  It requires automated SSH access to the root account on
the targeted equipment.

Kind regards
Felix

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 3028 bytes --]

On Wed, 20 Mar 2024 13:15:17 +0100
"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> wrote:

> Hello Denis.
Hi,

> I believe automatically invoking package managers for users does not
> give users control.  Telling them what to run, yes, but running the
> commands for them does not seem like good practice, if it can be
> avoided.  If I were a user, I would mostly like to understand.
Here it's more a question of design and what to do or not to do
automatically. For instance command line users do understand 'ls' or
'find' without needing to look at its source code or to change it. Many
scripts also use 'ls' or 'find'.

Here running 'guix build' commands automatically makes sense and as I
understand many people sharing their Guix configurations do that, at
least to setup the load path (-L, --load-path).

Since we want to reuse the output of guix build (because we want to
integrate guix progressively) there is no other way. 

The question here is more how far to go with the rest (guix pull,
substitutes, etc), and you get a point with that.

When there is setup-specific configuration to do, your advise is usually
good as documentation is easier to get right than a very specific tool
that works only in a subset of the cases. An example here is if you
want to install GNU/Linux on a WiFi access point and that requires you
to setup a TFTP server, in that case a documentation is better than
fragile scripts.

But here Guix is very reproducible there is not a lot to configure for
the users here, so it somewhat makes automatizing updates and
configuration a bit more relevant than other cases.

Though your answer also make me think once we have everything as
Guix packages, what we automatize (updates, etc) would still need to be
kept working not to break contributors or users habits. So the
automatizing would most likely need to be moved in a Makefile.

> In particular, the GNU Boot description on Savannah says it aims to
> replace boot code with 100% free software; the Canoeboot FAQ says it
> is impossible to get completely rid of Intel ME.
GNU Boot description talks about boot software, not boot code (more on
that below).

The devices we support either have no Management Engine (so we don't
need to get rid of something that isn't there), or have a management
engine hardware with its OS being completely removed.

What we remove is the OS (which is software) of the management engine
hardware. We don't remove hardware. 

In practice there is a bootrom inside the Management Engine (that
contains some code that is read-only as the name implies), but we can
consider that as hardware as it is not modifiable by anybody.

In any case we don't have to redistribute any nonfree software so that
bootrom has no impact on Guix usage or in our strategy to upstream
packages inside Guix. And since that bootrom cannot be updated anyway,
we don't have the risk of having to later on provide (nonfree) updates
for users either.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[pelzflorian (Florian Pelz)]

Hello.  Sorry for not being precise.  Invoking guix build or guix
shell is fine in my opinion, but “apt-get install” is not.

Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> writes:
> Here running 'guix build' commands automatically makes sense and as I
> understand many people sharing their Guix configurations do that, at
> least to setup the load path (-L, --load-path).

Yes.  It can be trusted to continue working.

Guix also supports i686 more or less; it will not be dropped, so memory
use can be trusted to not be more than i686 can do.

> The devices we support either have no Management Engine (so we don't
> need to get rid of something that isn't there),

Ohh.

> or have a management
> engine hardware with its OS being completely removed.
>
> What we remove is the OS (which is software) of the management engine
> hardware. We don't remove hardware. 
>
> In practice there is a bootrom inside the Management Engine (that
> contains some code that is read-only as the name implies), but we can
> consider that as hardware as it is not modifiable by anybody.
Adrien 'neox' Bourmault <neox@gnu.org> writes:
> GNU Boot does not
> support any board where Intel ME is mandatory to operate the board. Indeed, it's
> impossible to remove Intel ME from boards that requires it, and this is why we
> can never support these boards unless someone retroengineers these boards and is
> able to find a way to fully get rid of ME.

Thank you, neox and Denis, for telling about the extent of the removal.

Side note, as an observer from outside, it also seems relevant how the
project goal is different from Canoeboot.  Is the difference that GNU
Boot seeks to integrate more with FSDG distros like Guix?  That the
build system is more traditional?  Leah takes it personal, but on the
Canoeboot FAQ, I find “GNU Boot does support downloading, deblobbing and
re-compressing U-Boot archives, and in fact does a better job of *that*
than Canoeboot in some ways, but it does not yet actually build U-Boot,
and it does not boot U-Boot in any way, on any actual mainboards.”

Regards,
Florian

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 5309 bytes --]

On Guix integration:
--------------------

On Fri, 22 Mar 2024 10:17:39 +0100
"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> wrote:
> Side note, as an observer from outside, it also seems relevant how the
> project goal is different from Canoeboot.  Is the difference that GNU
> Boot seeks to integrate more with FSDG distros like Guix?
Right now our goal is more to work with upstream Guix to add packages in
Guix for what we need, and then use these packages to build the images
that we distribute in ftp.gnu.org/gnu/gnuboot. 

We already added grub-coreboot in Guix but we need to add more
(at least seabios-coreboot, and the Coreboot memtest86+, and Coreboot
itself if possible). Since both GNU Boot maintainers are also Guix
users, it also enable us to collaborate with Guix by sending patches and
through that somewhat share the maintenance with Guix of a very small
number of packages.

GNU Boot is also based on an older version of the Libreboot build
system that does everything with shell scripts, and we already adapted
it to be able to easily replace build commands (./configure, make) by
'guix shell' commands.

Since writing Guix packages is relatively easy and that Guix is well
documented, if contributors can use 'guix build', then the rest is
pretty straightforward and they can learn it step by step, by looking
at examples, asking other GNU Boot or Guix contributors for help, etc.

So this is why the integration of Guix as a dependency in GNU Boot is
something really crucial and that it has to be done right: we don't
want to loose contributors because there is too much setup, or because
our automatic way to install and update Guix fails.

What you suggested could indeed be the way to go: replace the code to
at least install and configure Guix with a pointer to the installation
section of the official Guix manual, and improve it along the way if we
need to (by mentioning more distro packages for instance). This should
take care of the substitutes configuration issue.

But then we would still like to somehow make guix time-machine
--commit=<hash> work automatically, so we need to somehow fetch that
revision, and if possible without touching the current guix revision
not to mess up the system of people that also use Guix for other
things. For that we probably need to add an option to guix pull and
upstream that. 

Since that could take some time, we could start by detecting if the
revision we need is not there and in that case at the same time point
users to the Guix manual to run guix pull, and warn about the issue of
changing guix revision, store the older guix revision in a well known
place and provide instructions to restore it.

As for supporting various guix build options (like '-c, --cores=N',
'--max-jobs=N'), we could probably make that configurable in GNU Boot
with the help of autotools.

> Guix also supports i686 more or less; it will not be dropped, so
> memory use can be trusted to not be more than i686 can do.
That's a good point. Once we build all our software with Guix,
doing that would also enable building on i686 for free[1].

Answers to your other questions:
--------------------------------
Another key difference with Canoeboot is that we also want to work
directly with various upstream projects like GRUB or Coreboot to make
sure we can keep maintaining the same devices over time. This decision
also probably aligns well with Guix goals as having too much extra
patch can be a burden to maintain and many distributions have formal or
informal policies to limit that to the strict minimum.

There are probably other differences as well (website build system,
etc) but they might not be very relevant to the use of Guix has a
mandatory dependency. Also note that GNU Boot is still pretty new (and
doesn't even have an official release yet) so over time there might be
more technical differences.

> Leah takes it personal, but on the Canoeboot FAQ, I find “GNU Boot
> does support downloading, deblobbing and re-compressing U-Boot
> archives, and in fact does a better job of *that* than Canoeboot in
> some ways, but it does not yet actually build U-Boot, and it does not
> boot U-Boot in any way, on any actual mainboards.”
If I understood right, Canoeboot dropped that functionality, and it's
still in the code of GNU Boot, but I wouldn't rely on that until GNU
Boot starts releasing deblobbed u-boot archives or that it's integrated
upstream in Guix in some form.

References:
-----------
[1]The FSDG distributions versions mentioned in the build documentation
   (https://www.gnu.org/software/gnuboot/web/docs/build/) do not
   support i686. Using Guix would fix that. All that is because we
   want to continue supporting the KGPE-D16 (that Guix also uses for its
   infrastructure), and that right now we use an older Coreboot
   revision for it, which requires older compiler revisions.

   We have plans for re-upstreaming this mainboard in Coreboot in a
   sustainable way (so it doesn't get dropped again) but in the future
   we might need help to complete that task that either with coding
   (we might be able to provide reverse engineered documentation to
   write maintainable code) or funding.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[pelzflorian (Florian Pelz)]

Hello, what you intend does sound very interesting.  As for “guix
time-machine”, I do not see the problem:

Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> writes:
> But then we would still like to somehow make guix time-machine
> --commit=<hash> work automatically, so we need to somehow fetch that
> revision, and if possible without touching the current guix revision
> not to mess up the system of people that also use Guix for other
> things. For that we probably need to add an option to guix pull and
> upstream that. 
>
> Since that could take some time, we could start by detecting if the
> revision we need is not there and in that case at the same time point
> users to the Guix manual to run guix pull, and warn about the issue of
> changing guix revision, store the older guix revision in a well known
> place and provide instructions to restore it.

“guix time-machine” is confined to the store and does not leave traces,
like “guix build” and unlike “guix pull”.  It is basically “guix pull”
without traces.


> We already added grub-coreboot in Guix

Ahh that was your work, GNU Booters; great job!  (Although I have no
machine to try yet.)

> What you suggested could indeed be the way to go: replace the code to
> at least install and configure Guix with a pointer to the installation
> section of the official Guix manual, and improve it along the way if we
> need to (by mentioning more distro packages for instance). This should
> take care of the substitutes configuration issue.

Simplifying install docs is being discussed and we would like more
feedback:

https://issues.guix.gnu.org/69977

At the same time, me citing the Arch Wiki’s negative stance on distros’
guix packages
<https://lists.gnu.org/archive/html/guix-devel/2024-03/msg00068.html>
and the dealing with the recent Guix local privilege escalation
vulnerability
<https://lists.gnu.org/archive/html/guix-devel/2024-03/msg00238.html>
hopefully will not cost us our Debian package.


> As for supporting various guix build options (like '-c, --cores=N',
> '--max-jobs=N'), we could probably make that configurable in GNU Boot
> with the help of autotools.

I do not know, but maybe the Autotools of Guix itself use something like
this to deal with “make -j4”.

I’m looking forward to reading much of the info you gave in this mail on
a GNU Boot website, or if the info is there already I just missed it.

Regards,
Florian

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 5837 bytes --]

On Mon, 25 Mar 2024 18:34:18 +0100
"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> wrote:

> Hello, what you intend does sound very interesting.  As for “guix
> time-machine”, I do not see the problem [...]
Let's say a user install Guix 1.4.0 and GNU Boot use a guix commit after
v1.4.0, as I understand guix time-machine will fail.

Here we have code to detect that situation already, the issue is more
what to do when this situation happens.

A second problem is if a user install guix and runs guix pull right
after but doesn't run guix pull again, and that in the future we start
using commits/revisions newer than the ones the user had right after
running guix pull.

Especially in the second case, running an additional 'guix pull'
behind the back of the user can have some bad consequences if the user
is also using Guix for other things and for some reasons didn't plan to
update guix yet.

So my current plan is just to detect that the commit is not there and
tell the user to run guix pull and also give the user a way to restore
the old guix revision afterward if needed. It's not ideal but it could
work right now for all use cases.

> Simplifying install docs is being discussed and we would like more
> feedback:
> 
> https://issues.guix.gnu.org/69977
>
> At the same time, me citing the Arch Wiki’s negative stance on
> distros’ guix packages
> <https://lists.gnu.org/archive/html/guix-devel/2024-03/msg00068.html>
> and the dealing with the recent Guix local privilege escalation
> vulnerability
> <https://lists.gnu.org/archive/html/guix-devel/2024-03/msg00238.html>
> hopefully will not cost us our Debian package.
Thanks, I'll read all these threads.

> > As for supporting various guix build options (like '-c, --cores=N',
> > '--max-jobs=N'), we could probably make that configurable in GNU
> > Boot with the help of autotools.
> 
> I do not know, but maybe the Autotools of Guix itself use something
> like this to deal with “make -j4”.
My question was more about the user interface and if it was the right
thing to do. As for the code implementing it[1], it was pretty easy to
do for me and it integrates fine with the current GNU Boot structure: if
users run './autogen.sh && ./configure' they can still use the scripts
manually, so this avoids too much invasive changes.

I still need to do some cleanups though and complete that work (as some
things are still missing, like handling 'guix pull' to make sure that
guix-time-machine works).

[1]https://git.savannah.gnu.org/cgit/gnuboot.git/log/?h=GNUtoo/guix-configure

> I’m looking forward to reading much of the info you gave in this mail
> on a GNU Boot website, or if the info is there already I just missed
> it.
The issue is that there is a chicken and egg issue as for the
code/documentation to be merged in GNU Boot, we need to figure out the
questions I asked in this thread.

And there is also a second chicken and egg: we don't want to add a
dependency on Guix without a real use case that really requires Guix in
some non-optional way (more on that below).

As for the code that is actually merged, building the GNU Boot website
can be done with guix shell but to do that the user needs to pass
--enable-guix to the ./configure in that directory:
https://git.savannah.gnu.org/cgit/gnuboot.git/tree/website-build

We used Guix shell here because it makes Guix optional, so we didn't
need to have the Guix part being ultra robust/polished/documented. If
it worked for users already familiar with Guix, it was good enough.

And we also already merged code to update Guix:
https://git.savannah.gnu.org/cgit/gnuboot.git/tree/resources/dependencies/guix
https://git.savannah.gnu.org/cgit/gnuboot.git/tree/resources/scripts/misc/guix.sh
but this is not run automatically, and not mentioned in the
documentation either. So users that know about it could run it manually
but that's pretty much it.

So in the meantime the code/documentation we have on Guix
non-optional integration in GNU Boot is available in various branches as
it is not ready yet.

And we don't want to make Guix non-optional for the website right now,
as it works fine in the way it is.

And the idea is to try to integrate Guix as a required dependency with
the least amount of changes. So it will probably be done to build some
tools first like with this branch:
https://git.savannah.gnu.org/cgit/gnuboot.git/log/?h=GNUtoo/guix-configure

This kind of changes really makes sense as it would enable us to fix
some installation instructions for some devices which would make the
first release closer, and it limits the risk of breakages since it
doesn't modify in any way the binaries to install.

As for deeper integration that can build GRUB with Guix, it can be found
here: https://git.savannah.gnu.org/cgit/gnuboot.git/log/?h=GNUtoo/guix
Note that this branch is older and will probably be rebased / cleaned
up much later (probably after the release).

To get there we converted the Libreboot directory structures to
resemble more packages with tasks[2], and that is merged already. So
that enables us to then just remove the shell commands that build grub
and replace that with a call to guix build and keep the rest of the
shell commands that reuses the output.

[2]https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?id=857afa42a8ade870115391b09d712b110e6a1066

It's done in this commit for instance:
https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?h=GNUtoo/guix&id=f96d93160d6c29cb45b999c56f03ec8a4312140d
But as explained before, this commit need to be rebased, split, cleaned
up, etc. For instance it was made at a time where grub-coreboot wasn't
in Guix yet, so now we can simply use the upstream package instead.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[pelzflorian (Florian Pelz)]

Hi there.  Some corrections.

Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> writes:
> On Mon, 25 Mar 2024 18:34:18 +0100
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> wrote:
>> Hello, what you intend does sound very interesting.  As for “guix
>> time-machine”, I do not see the problem [...]
> Let's say a user install Guix 1.4.0 and GNU Boot use a guix commit after
> v1.4.0, as I understand guix time-machine will fail.

No, it does not fail.

I just tried from a pre-v1.4.0 installer to time-machine
--commit=9f11c33b6d2c16e710fa2ef3c74f985e2b885350,

commit 9f11c33b6d2c16e710fa2ef3c74f985e2b885350
Author: Sou Bunnbu (宋文武) <iyzsong@gmail.com>
Date:   Sat Mar 16 13:14:37 2024 +0800

    gnu: Add logtalk.
    
    * gnu/packages/prolog.scm (logtalk): New package.
    
    Change-Id: I99e04e70b32c498b5da1f094752279e4668f3865

I can run logtalk binaries.


>> I do not know, but maybe the Autotools of Guix itself use something
>> like this to deal with “make -j4”.
> My question was more about the user interface and if it was the right
> thing to do. As for the code implementing it[1], it was pretty easy to
> do for me and it integrates fine with the current GNU Boot structure: if
> users run './autogen.sh && ./configure' they can still use the scripts
> manually, so this avoids too much invasive changes.

So from reading the Guix build machinery, Makefile.am runs

https://git.savannah.gnu.org/cgit/guix.git/plain/build-aux/compile-all.scm

and they use (getenv "MAKEFLAGS") to check if make gets any --jobserver
flag that specifies the number of parallel compilations.  This seems
slightly nicer than a non-standard configure option, but is also more
complicated in that Makefile.am calls out to compile-all.scm to do the
work.

Presumably Guix developers like writing Scheme more than writing shell
scripts.


> And we also already merged code to update Guix:
> https://git.savannah.gnu.org/cgit/gnuboot.git/tree/resources/dependencies/guix
> https://git.savannah.gnu.org/cgit/gnuboot.git/tree/resources/scripts/misc/guix.sh
> but this is not run automatically, and not mentioned in the
> documentation either. So users that know about it could run it manually
> but that's pretty much it.

Yes, I read in your scripts you check the Guix caches.  Of course guix
shell and time-machine modify not only the store but also the caches,
I’ve said it wrong.

But GNU Boot calling “guix time-machine” and thereby modifying the user
.cache directory is OK, in my opinion.

Why do you need so many different Guix versions?  Are there regressions
or is it just that you have tried building different parts of Libreboot
with different versions of Guix?


> And the idea is to try to integrate Guix as a required dependency with
> the least amount of changes. So it will probably be done to build some
> tools first like with this branch:
> https://git.savannah.gnu.org/cgit/gnuboot.git/log/?h=GNUtoo/guix-configure

That is, you move Libreboot’s build system stepwise towards using guix
build instead.  Sounds good from outside; I have not perused all the
links throughly.  It is very interesting, though.

Regards,
Florian

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 5191 bytes --]

On Wed, 27 Mar 2024 12:22:40 +0100
"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> wrote:

> Hi there.  [...]
Hi

> Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> writes:
> > On Mon, 25 Mar 2024 18:34:18 +0100
> > "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> wrote:  
> >> Hello, what you intend does sound very interesting.  As for “guix
> >> time-machine”, I do not see the problem [...]  
> > Let's say a user install Guix 1.4.0 and GNU Boot use a guix commit
> > after v1.4.0, as I understand guix time-machine will fail.  
> 
> No, it does not fail.
Thanks a lot, this is very interesting as it could simplify a lot the
code. In addition running guix time-machine doesn't modify the current
Guix version so it's not invasive.

I recall that when I tried it on older versions of Guix, it did fail,
but maybe more recent versions don't fail anymore. I'd need to do more
tests with that to understand when it fails or doesn't fail and
somehow get/require the right conditions and then remove all the code
that isn't needed anymore.

> >> I do not know, but maybe the Autotools of Guix itself use something
> >> like this to deal with “make -j4”.  
> > My question was more about the user interface and if it was the
> > right thing to do. As for the code implementing it[1], it was
> > pretty easy to do for me and it integrates fine with the current
> > GNU Boot structure: if users run './autogen.sh && ./configure' they
> > can still use the scripts manually, so this avoids too much
> > invasive changes.  
> 
> So from reading the Guix build machinery, Makefile.am runs
> 
> https://git.savannah.gnu.org/cgit/guix.git/plain/build-aux/compile-all.scm
> 
> and they use (getenv "MAKEFLAGS") to check if make gets any
> --jobserver flag that specifies the number of parallel compilations.
> This seems slightly nicer than a non-standard configure option, but
> is also more complicated in that Makefile.am calls out to
> compile-all.scm to do the work.
Thanks for that information. This could make things work out of the box
without more configuration. Though I still need to think about that one
as the code is much harder to understand than with configuration
options.

> Why do you need so many different Guix versions?  Are there
> regressions or is it just that you have tried building different
> parts of Libreboot with different versions of Guix?
We don't necessarily need multiple Guix versions but I think it fits
better in the current design if we allow that.

The old structure of Libreboot we inherited was converted to look more
like packages. Now each "package" has a directory in resources/packages/
like resources/packages/grub for instance. And inside the package
directory there are shell scripts with specific names like download,
build that are tasks.

Because of that it makes sense to treat each "package" separately and
allow the use of different Guix versions. So practically speaking if I
want to replace the code that builds SeaBIOS with shell scripts I could
first write a local guix seabios-coreboot package that I add inside
resources/packages/seabios/seabios.scm and call guix build inside
resources/packages/seabios/payload (which builds SeaBIOS for Coreboot).
At this point I could use Guix 1.4.0 if there are no security issues in
SeaBIOS or its dependencies for Guix 1.4.0.

But once seabios-coreboot is upstreamed in Guix it would make sense to
get rid of resources/packages/seabios/seabios.scm completely if
possible (if we use the same build options, don't have extra patches,
etc) and instead update the shell script that calls guix build to use
the Guix version where seabios-coreboot was added.

Thanks to that it's easier to update each package without affecting the
rest.

When everything will have been converted to use Guix packages we would
need to think again if using a single Guix revision could make sense or
not depending on the amount of risk of breakages. The main advantage
of a single revision would be to have way faster builds.

> That is, you move Libreboot’s build system stepwise towards using guix
> build instead.  Sounds good from outside; I have not perused all the
> links throughly.  It is very interesting, though.
The main rationale behind that is to have at any point something that
still works. Using Guix just for specific packages like GRUB enable to
test the resulting images like with any other changes, and also
consistently update the documentation along the way and so on. We can
even write some automatic tests that work with images made before Guix
is integrated and after without needing any special consideration.

And finally it also lowers a lot the odds of having never-ending work to
do the conversion, and never finishing nor making any releases,
especially because some of the tasks (like building Coreboot) are not
trivial (it requires an ada compiler and Guix doesn't have one yet).

And all that also makes it easier to collaborate with upstream as we
don't need to rush to convert everything to Guix packages, so it leaves
time to upstream things.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix as a non-optional dependency in another project, and Guix resources requirements.

[Simon Tournier]

Hi Denis,

On sam., 16 mars 2024 at 02:03, Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> wrote:

> - First and most importantly, running 'guix pull' can fail sometimes,
>   especially if there is not enough RAM per core. A fix that work is to
>   lower the number of cores used with 'guix pull -c 1 -M 1' for
>   instance.
>
>   But the issue is that I'm unsure how much RAM per core is needed.
>   Maybe 2GiB or 3GiB? Also if users have heavy desktop usage
>   (browser(s) with a ton of tabs open) could that value change?
>
>   I could ask help from GNU Boot users for testing to find that value
>   but maybe the Guix community already knows more about this.
>
>   The big downside of my approach here is that if for some reason the
>   script I made uses a 'memory per core' value that is too low, it
>   would probably makes things way more difficult for contributors than
>   having to read the Guix manual upfront in the first place, because it
>   would fail and they won't understand why. So they'd need to
>   understand both Guix, the code I wrote, and what caused the issue.
>
> - If Guix was already installed on a host distribution, I'm not sure
>   how to handle substitutes. Ideally I'd like to be able to enable them
>   just for GNU Boot and not change the way they are handled for other
>   Guix usage. Also if substitutes are not used it probably affect the
>   amount of RAM per core required.
>
>   One way around here could be to build Guix ourselves and somehow find
>   a way to have different defaults, but building that Guix could
>   probably a long time.

Well, from my understanding, with or without substitutes is the key
here.  Without substitutes, yeah we hit the Guile compilation
boundaries, especially about RAM.

With substitutes, the story is different. :-)

The only operation that is expensive is: “Computing Guix derivation”.
Then the rest is substitutable.  Here, we could also speak network
requirements but that’s another story. ;-)

I have never measured the RAM requirement of this step.  I would be
surprised if it’s more than 2GiB… but who know! :-)

Your users do not need to run a complete Guix but only something that
can be determined beforehand and compliant with the rest of your
project.

Somehow, you have two choices:

 + Rely on substitutes from Guix project.  The advantage is that you
 have nothing to do, just invoke the right command with your custom
 script.  The main drawback is that your script is subject to Guix
 project policy when deleting substitutes.  To my knowledge, this policy
 is clear but the effects are hard to predict.

 + Serve your own substitutes.  The advantage is that you control which
 substitutes are available.  The drawback is that you need to setup
 “guix publish”.

Do not take me wrong about the substitutes.  You only need the result of
some derivations (see ’compile-guix’ procedure in (guix self) module);
the ones that “guix pull” builds.

Doing that, your script might pinpoint a specific Guix revision and you
also serve the substitute requirements for that revision.

Bah the first question is the RAM of “Computing Guix derivation”?

Cheers,
simon