What’s next?

What’s next?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2421 bytes --]

Hello Guix!

I think it’s time to think about what we want to work on next, ideally
before the next release, and ideally said release should be in a couple
of months rather than in 5 months.  Here are some important items I can
think of:

  • Merge the ncurses installer (the ‘wip-installer’) branch.

  • We’re supposed to freeze ‘core-updates’ in a couple of days and we
    have defined a couple of important goals:
    <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00120.html>.
    I have a vague feeling that the ‘wip-build-systems-gexp’ merge is
    not for this time yet…

  • ‘guix offload’ terrible bug: <https://bugs.gnu.org/26976>.

  • Guile 2.2 compiler terrible issue:
    <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

  • Adjust the release process so we don’t find ourselves without
    substitutes for the ‘guix’ package, as reported at
    <https://bugs.gnu.org/27032>.

  • Prepare to migrate to the new build farm: the hardware of
    bayfront.guixsd.org has been fixed (it was unreliable until we
    changed its CPUs), so now we need to fix a couple of issues in
    Cuirass and generally improve it so we can, via its HTTP interface,
    add new jobsets and so on.  Of course we’ll have to work
    on that incrementally.

  • Finalize the new bootloader API and support for new bootloaders:
    <https://bugs.gnu.org/26339>.

  • Merge the potluck!  <https://bugs.gnu.org/26645>

I think everything above could pretty much go into a 0.13.1 release not
far away.  And the missing bits, IMO, for a “1.0” label would be:

  • Improve the UI: add colors (yay!), hide build logs by default for
    most commands, improve progress reports, display how much will be
    downloaded, etc.

  • Implement channels: <https://bugs.gnu.org/22629>.  A first step may
    be to fix some of the obvious shortcomings of ‘guix pull’: use (guix
    git) to optimize bandwidth usage, and compute the derivation of the
    new ‘guix’ and use that.

  • Authenticate Git checkouts: <https://bugs.gnu.org/22883>.

What do people think?

I think the key idea is that we should fix all the annoyances and bugs,
many of which seasoned Guix hackers more or less got used to, but which
are nevertheless a serious hindrance when using Guix “normally.”

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.  Here are some important items I can
> think of:

[…]

Excellent points!

> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”

Here are some other annoyances:

* the verbosity of reporting hash mismatches.  You posted a neat little
  change for that some time ago, but I cannot find it any more.

* texlive.  I’m working on splitting the package up.  (An aggressive
  firewall at work put a stop to that work, but I’ll try to resume very
  soon.)

* very explicit but cryptic stack traces.  I think that all errors
  should be reported more nicely.  The error can still be printed (on
  demand?), but Guix probably should not spill its innards too readily.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Brendan Tildesley]

Ludovic Courtès 於 2017-05-24 23:11 寫道:
> Hello Guix!
>
> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.  Here are some important items I can
> think of:
[...]
> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”
>
> Ludo’.

One little annoyance I have is that guix takes every opportunity to
update the list of substitutes when guix build, guix package -u, etc...
is run, and fills my terminal with output like:

substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
...

It will output this line a different number each time based on some rule
I couldn't guess, and renders the percentage measurement meaningless.
Currently I am running `guix system init config.scm
--substitute-urls="http://10.1.1.61:8080 https://mirror.hydra.gnu.org/"
--fallback' from the 0.13 installer, and it his been outputting hundreds
of these lines for the last half an hour. I'm not sure if it is ever
going to stop, perhaps I've stumbled across a genuine bug? In any case,
the minor annoyance version is something I've noticed for a while.

Re: What’s next?

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

2017-05-24 15:11 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:

> Hello Guix!
>
> I think it’s time to think about what we want to work on next, ideally
>

one little ting I'd love is Janneke's and Jlicht's solution to use "binary"
nodejs packages

That would allow me to play with some projects that depend on nodejs

The problem of evaluating the adherence of the nodejs collection to the
Guix diistribution guidelines could be faced later ( I posted a thread that
I think is relevant in this regard)



>   • ‘guix offload’ terrible bug: <https://bugs.gnu.org/26976>.
>
>   • Guile 2.2 compiler terrible issue:
>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html
> >.
>

terrible bugs are terrible bugs. They should be regarded as priorities


>   • Improve the UI: add colors (yay!), hide build logs by default for
>     most commands, improve progress reports, display how much will be
>     downloaded, etc.
>

This might be regarded as a small thing but to me this is very important
I use to watch Fedora's colorful terminals with longing


> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”
>

yeah

[-- Attachment #2: Type: text/html, Size: 2401 bytes --]

Re: What’s next?

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.  Here are some important items I can
> think of:

Great points!

>  • Implement channels: <https://bugs.gnu.org/22629>.  A first step may
>    be to fix some of the obvious shortcomings of ‘guix pull’: use (guix
>    git) to optimize bandwidth usage, and compute the derivation of the
>    new ‘guix’ and use that.
>
> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”

+1

A friend of mine is having a second look at Guix (not SD yet) and one of
the most confusing things initially is `guix pull'.  "When/how do I use
that," he asks...and I can only say: I'm not using that...I think we
want this to work--or something like this, we talked about this at
FOSDEM, but AFAIK everyone is using Guix with Git.

He responds with: then *why* is it in the manual.  I have no answer.
Possibly I'm wrong and/or my information is outdated?

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: What’s next?

[Adonay Felipe Nogueira]

As an average Guix user, I use `guix pull`.

In fact, I avoid `git pull and the pre-inst-env stuff`. I prefer to work
with GUIX_PACKAGE_PATH in order to work only with `guix package` and
`guix build` if I do have the need to install or build something that I
changed.

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: What’s next?

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

2017-05-24 18:25 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:

>
>
> +1
>
> A friend of mine is having a second look at Guix (not SD yet) and one of
> the most confusing things initially is `guix pull'.  "When/how do I use
> that," he asks...and I can only say: I'm not using that...I think we
> want this to work--or something like this, we talked about this at
> FOSDEM, but AFAIK everyone is using Guix with Git.
>
> He responds with: then *why* is it in the manual.  I have no answer.
> Possibly I'm wrong and/or my information is outdated?
>

This is an important point for me too

I realized that everyone is using git and not guix pull just yesterday

I have been using guix pull until today because I didn' t know any better

The manual mentions something that only beginners use and doesn' t mention
something that the majority of the community is using.

Probably in many of the footages in which Ludo appears, the use of git over
guix pull is assumed.

I think this is a problem. It' s unfair to newcomers and it damages Guix as
a project because it makes the learning curve steeper with not so much of a
point why.

I kinda agree with myglc2, on this.

[-- Attachment #2: Type: text/html, Size: 1702 bytes --]

Re: What’s next?

[Ricardo Wurmus]

Catonano <catonano@gmail.com> writes:

> 2017-05-24 18:25 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:
[…]
>> A friend of mine is having a second look at Guix (not SD yet) and one of
>> the most confusing things initially is `guix pull'.  "When/how do I use
>> that," he asks...and I can only say: I'm not using that...I think we
>> want this to work--or something like this, we talked about this at
>> FOSDEM, but AFAIK everyone is using Guix with Git.
>>
>> He responds with: then *why* is it in the manual.  I have no answer.
>> Possibly I'm wrong and/or my information is outdated?
>>
>
> This is an important point for me too
>
> I realized that everyone is using git and not guix pull just yesterday
[…]
> I think this is a problem. It' s unfair to newcomers and it damages Guix as
> a project because it makes the learning curve steeper with not so much of a
> point why.

There are two reasons why developers use Guix from git:

* it allows them to add new packages and features to Guix itself.  This
  is something “guix pull” doesn’t support well.

* it doesn’t require compiling all of Guix on each update.

The latter is a defect in “guix pull” that we are aware of and working
on.  Progress has already been made in improving “guix pull”, but it
takes time.  It is not by design that “guix pull” is currently less
desirable than working from a git checkout.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Wed, May 24, 2017 at 06:25:40PM +0200, Jan Nieuwenhuizen wrote:
> A friend of mine is having a second look at Guix (not SD yet) and one of
> the most confusing things initially is `guix pull'.  "When/how do I use
> that," he asks...and I can only say: I'm not using that...I think we
> want this to work--or something like this, we talked about this at
> FOSDEM, but AFAIK everyone is using Guix with Git.

`guix pull` is one of the primary tools of Guix. For those who are new
to Guix, it should be described as a per-user `apt-get update`. That is,
it updates the list of available packages. The finer differences and
extra features are not important for new users to learn at the
beginning.

With the recent commit adding '--fallback' to `guix pull` [0], the main
reason for Guix users who are not Guix developers to resort to Git has
been removed.

So, I use and recommend `guix pull`!

Do you think the manual can be more clear about this? I'd really like to
hear which parts of the manual your friend read. Maybe we need to
rearrange or rewrite some sections.

I think the most immediate problem with `guix pull` is that it doesn't
support Git commit signature verification. So, you end up trusting
different things: basically, a subset of the X.509 PKI vs PGP+SHA1 [1,2].
I think we can fix this while making `guix pull` use (guix git).

Building Guix from Git is the normal way to develop Guix, and it avoids
downloading a Guix tarball from Savannah in the default case, so
developers will learn and use it, but it brings its own pitfalls.

[0]
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4902d3c4e0376974356481f222583580b49f39e1

[1] `guix pull` verifies the certificate of <git.savannah.gnu.org>
against the Let's Encrypt trust chain *only*.

[2] If I understand correctly, Git commit signatures are of the SHA1
hash, not the actual commit data. So... not great if I'm correct, but it
will get better as Git introduces a new hash function. And SHA1
collisions are rather obvious to detect, at least according the public
research. An attempt at collision detection was added in Git 2.13.0.

> He responds with: then *why* is it in the manual.  I have no answer.
> Possibly I'm wrong and/or my information is outdated?

Since we are all Guix developers, we talk about developing Guix, but not
as much the day-to-day use. So our impressions may not match actual
usage patterns.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What’s next?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

On Wed, May 24, 2017 at 09:34:14PM +0200, Catonano wrote:
> I think this is a problem. It' s unfair to newcomers and it damages Guix as
> a project because it makes the learning curve steeper with not so much of a
> point why.

It's true, we collectively worked around some problems with `guix pull`
for too long. Let's all eat the dog food when we don't actually need to
work from Git. I promise it tastes okay :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What???s next?

[Pjotr Prins]

On Wed, May 24, 2017 at 05:45:39PM -0400, Leo Famulari wrote:
> [1] `guix pull` verifies the certificate of <git.savannah.gnu.org>
> against the Let's Encrypt trust chain *only*.

This brings up another annoyance. Before a first 'git pull' as a
newbie you have to go through a number of steps which are, arguably,
redundant.

I am talking about installing a first key to trust the guix server.
Well, if we have installed guix AND we use guix pull, I think we can
assume the guix server is trusted (by the user). Therefore, that key
should work out of the box (it is what people install from the tree
anyway!). It is a redundant step. Debian also uses keys and works
out of the box.

The other thing is permissions. Sometimes the user profile needs
explicit permission settings. This is not right. I can see it is
useful on a server setup controlled by an administrator, but arguably
it should just work. The administrator can revert on that. So, if
possible, the default should be allowing guix to work once the daemon
runs.

Pj.

Re: What’s next?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Leo Famulari <leo@famulari.name> writes:

> So, I use and recommend `guix pull`!

I use it too.  Statements by others in this thread that "nobody" uses it
or that "everyone" is using Git are mistaken.

I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
IMO, the biggest problem with 'guix pull' is that there is no easy
rollback.  I can live with long execution times (--fallback is fine, but
it'd be nice if substitutes were available more often), and I can live
with 'guix pull' causing me to get a version of guix that's broken
somehow, but the inability to easily roll back when things go south
makes me hesitant to run 'guix pull' regularly.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: What’s next?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

On Thu, May 25, 2017 at 07:57:41AM -0700, Chris Marusich wrote:
> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
> IMO, the biggest problem with 'guix pull' is that there is no easy
> rollback.  I can live with long execution times (--fallback is fine, but
> it'd be nice if substitutes were available more often), and I can live
> with 'guix pull' causing me to get a version of guix that's broken
> somehow, but the inability to easily roll back when things go south
> makes me hesitant to run 'guix pull' regularly.

It's not "easy rollback", but you can at least use it to deploy any Guix
commit, from any source, with the '--url=' argument.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What’s next?

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

> Leo Famulari <leo@famulari.name> writes:
>
>> So, I use and recommend `guix pull`!
>
> I use it too.  Statements by others in this thread that "nobody" uses it
> or that "everyone" is using Git are mistaken.
>
> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
> IMO, the biggest problem with 'guix pull' is that there is no easy
> rollback.  I can live with long execution times (--fallback is fine, but
> it'd be nice if substitutes were available more often), and I can live
> with 'guix pull' causing me to get a version of guix that's broken
> somehow, but the inability to easily roll back when things go south
> makes me hesitant to run 'guix pull' regularly.

I believe this can be fixed by adding more links to “.config/guix”,
i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
and then link from there to “latest”.  On update it would create a new
link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.

(“latest” should be renamed to “current” to better match the intent in
this case.)

Timestamped names like this are ugly, but that’s what’s at the top of my
head.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Adonay Felipe Nogueira]

I think we can try either using:

- ISO dates: [Full year number]-[Two digit month number]-[Two digit day
  number]T[Two digit hour in 24-hour clock]:[Two digit minutes]:[Two digit
  seconds]

- A numeric sufix: Like "latest.1" for the first generation, "latest.2"
  for the second, and so on, and "latest" for the numbered backup being
  considered as current. Thus, the date and time information would be
  kept in the file metadata, not in the name, specially considering that
  names with ":" might be problematic in some file systems.

I of course don't know, from a developer point of view, what is the best
option, but I thought about giving my "personal" two cents on the
matter. :)

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: What’s next?

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Here are some other annoyances:
>
> * the verbosity of reporting hash mismatches.  You posted a neat little
>   change for that some time ago, but I cannot find it any more.

Oh right, see attached.

> * texlive.  I’m working on splitting the package up.  (An aggressive
>   firewall at work put a stop to that work, but I’ll try to resume very
>   soon.)

Good point!  Looking forward to that.

> * very explicit but cryptic stack traces.  I think that all errors
>   should be reported more nicely.  The error can still be printed (on
>   demand?), but Guix probably should not spill its innards too readily.

I agree with this direction, but we’ll have to work on concrete cases
here.  Bug reports of the form “when I do this I get a stack trace
instead of an error message” are welcome!

Thanks,
Ludo’.

Re: What’s next?

[Ludovic Courtès]

Hi Brendan,

Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:

> One little annoyance I have is that guix takes every opportunity to
> update the list of substitutes when guix build, guix package -u, etc...
> is run, and fills my terminal with output like:
>
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> ...

I agree, it’s actually a huge annoyance.  It relates to grafts and how
they are implemented; I took a stab at fixing this a while back but the
approach turned out to be (mostly?) misguided:

  https://bugs.gnu.org/22990

Would be worth thinking through it again!

Thanks,
Ludo’.

Re: What’s next?

[Ludovic Courtès]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> A friend of mine is having a second look at Guix (not SD yet) and one of
> the most confusing things initially is `guix pull'.  "When/how do I use
> that," he asks...and I can only say: I'm not using that...I think we
> want this to work--or something like this, we talked about this at
> FOSDEM, but AFAIK everyone is using Guix with Git.
>
> He responds with: then *why* is it in the manual.  I have no answer.
> Possibly I'm wrong and/or my information is outdated?

I think there’s a lot of (well deserved, as far as its implementation
goes) badmouthing against ‘guix pull’ on #guix ;-), but nevertheless, I
think it’s a useful tool for users who don’t want to manage their own
Guix checkout etc. by themselves.

The manual does mention in several places what it’s for, I think.

Ludo’.

Re: What’s next?

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> Leo Famulari <leo@famulari.name> writes:
>>
>>> So, I use and recommend `guix pull`!
>>
>> I use it too.  Statements by others in this thread that "nobody" uses it
>> or that "everyone" is using Git are mistaken.
>>
>> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
>> IMO, the biggest problem with 'guix pull' is that there is no easy
>> rollback.  I can live with long execution times (--fallback is fine, but
>> it'd be nice if substitutes were available more often), and I can live
>> with 'guix pull' causing me to get a version of guix that's broken
>> somehow, but the inability to easily roll back when things go south
>> makes me hesitant to run 'guix pull' regularly.
>
> I believe this can be fixed by adding more links to “.config/guix”,
> i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
> and then link from there to “latest”.  On update it would create a new
> link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
> be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.

There would be some similarity with profiles.  Should we simply use
profiles, and effectively turn ~/.config/guix/latest into a profile,
with generations etc.?

Food for thought…  :-)

Ludo’.

Re: What???s next?

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Wed, May 24, 2017 at 05:45:39PM -0400, Leo Famulari wrote:
>> [1] `guix pull` verifies the certificate of <git.savannah.gnu.org>
>> against the Let's Encrypt trust chain *only*.
>
> This brings up another annoyance. Before a first 'git pull' as a
> newbie you have to go through a number of steps which are, arguably,
> redundant.

Note that the Let’s Encrypt certificate check by ‘guix pull’ works out
of the box: users don’t need to install ‘nss-certs’, define a bunch of
environment variables, etc.

> I am talking about installing a first key to trust the guix server.
> Well, if we have installed guix AND we use guix pull, I think we can
> assume the guix server is trusted (by the user). Therefore, that key
> should work out of the box (it is what people install from the tree
> anyway!). It is a redundant step. Debian also uses keys and works
> out of the box.

Substitute servers are fundamentally different from servers that provide
Guix packages, which is why it’s treated differently.

On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
registered by default.  We cannot do that for someone installing Guix on
a foreign distro because that involves creating a file in /etc.

> The other thing is permissions. Sometimes the user profile needs
> explicit permission settings.

What do you mean?

Thanks,
Ludo’.

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Here are some other annoyances:
>>
>> * the verbosity of reporting hash mismatches.  You posted a neat little
>>   change for that some time ago, but I cannot find it any more.
>
> Oh right, see attached.

I think you forgot to attach it.

>> * texlive.  I’m working on splitting the package up.  (An aggressive
>>   firewall at work put a stop to that work, but I’ll try to resume very
>>   soon.)
>
> Good point!  Looking forward to that.

Me too!

>> * very explicit but cryptic stack traces.  I think that all errors
>>   should be reported more nicely.  The error can still be printed (on
>>   demand?), but Guix probably should not spill its innards too readily.
>
> I agree with this direction, but we’ll have to work on concrete cases
> here.  Bug reports of the form “when I do this I get a stack trace
> instead of an error message” are welcome!

Danny submitted bug #27100 for this with two examples.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What's next?

[Pjotr Prins]

On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
> registered by default.  We cannot do that for someone installing Guix on
> a foreign distro because that involves creating a file in /etc.

Many installs are not on GuixSD. Can't we use the key that is stored
in the store itself? If /etc does not exist then use what comes
with the installation.

> > The other thing is permissions. Sometimes the user profile needs
> > explicit permission settings.
> 
> What do you mean?

I encountered that /var/guix/profiles/ permissions need to be set for
every user. But maybe that has changed, or I got it wrong.

Pj.

Re: What???s next?

[Maxim Cournoyer]

Hello,

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
> registered by default.  We cannot do that for someone installing Guix on
> a foreign distro because that involves creating a file in /etc.

The bayfront key is now registered by default? Does it mean that
bayfront is ready to be part of the `%default-substitute-urls'? It
doesn't seem to be the case currently:

--8<---------------cut here---------------start------------->8---
(define %default-substitute-urls
  ;; Default list of substituters.  This is *not* the list baked in
  ;; 'guix-daemon', but it is used by 'guix-service-type' and and a couple of
  ;; clients ('guix build --log-file' uses it.)
  (map (if (false-if-exception (resolve-interface '(gnutls)))
           (cut string-append "https://" <>)
           (cut string-append "http://" <>))
       '("mirror.hydra.gnu.org")))
--8<---------------cut here---------------end--------------->8---

Maxim

Re: What’s next?

[Maxim Cournoyer]

Hi,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Brendan,
>
> Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:
>
>> One little annoyance I have is that guix takes every opportunity to
>> update the list of substitutes when guix build, guix package -u, etc...
>> is run, and fills my terminal with output like:
>>
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> ...
>
> I agree, it’s actually a huge annoyance.  It relates to grafts and how
> they are implemented; I took a stab at fixing this a while back but the
> approach turned out to be (mostly?) misguided:
>
>   https://bugs.gnu.org/22990
>
> Would be worth thinking through it again!

Maybe we could make a timer variable configurable, so that at least it
doesn't try to refresh this information at *every* guix command? Default
could be at least one hour. I care about my security, but refreshing
this at every command seems pretty wasteful in terms of resources.

Maxim

Re: What’s next?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello!
>>
>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>
>>> Here are some other annoyances:
>>>
>>> * the verbosity of reporting hash mismatches.  You posted a neat little
>>>   change for that some time ago, but I cannot find it any more.
>>
>> Oh right, see attached.
>
> I think you forgot to attach it.

Oops.  Here we go:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1380 bytes --]

modified   nix/libstore/build.cc
@@ -2449,8 +2449,11 @@ void DerivationGoal::registerOutputs()
             Hash h2 = recursive ? hashPath(ht, actualPath).first : hashFile(ht, actualPath);
             if (h != h2)
                 throw BuildError(
-                    format("output path `%1%' should have %2% hash `%3%', instead has `%4%'")
-                    % path % i->second.hashAlgo % printHash16or32(h) % printHash16or32(h2));
+                    format("%1% hash mismatch for output path `%2%'\n"
+			   "  expected: %3%\n"
+			   "  actual:   %4%")
+                    % i->second.hashAlgo % path
+		    % printHash16or32(h) % printHash16or32(h2));
         }
 
         /* Get rid of all weird permissions.  This also checks that
@@ -3096,7 +3099,9 @@ void SubstitutionGoal::finished()
             Hash expectedHash = parseHash16or32(hashType, string(expectedHashStr, n + 1));
             Hash actualHash = hashType == htSHA256 ? hash.first : hashPath(hashType, destPath).first;
             if (expectedHash != actualHash)
-                throw SubstError(format("hash mismatch in downloaded path `%1%': expected %2%, got %3%")
+                throw SubstError(format("hash mismatch in downloaded path `%1%'\n"
+					"  expected: %2%\n"
+					"  actual:   %3%")
                     % storePath % printHash(expectedHash) % printHash(actualHash));
         }

[-- Attachment #3: Type: text/plain, Size: 314 bytes --]


Should we apply it?

>> I agree with this direction, but we’ll have to work on concrete cases
>> here.  Bug reports of the form “when I do this I get a stack trace
>> instead of an error message” are welcome!
>
> Danny submitted bug #27100 for this with two examples.

Great.

Thanks,
Ludo’.

Re: What's next?

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>> registered by default.  We cannot do that for someone installing Guix on
>> a foreign distro because that involves creating a file in /etc.
>
> Many installs are not on GuixSD. Can't we use the key that is stored
> in the store itself? If /etc does not exist then use what comes
> with the installation.

The current behavior is to print a warning when /etc/guix/acl (the list
of authorized keys) is empty or nonexistent.

Your suggestion would be to automatically populate it, right?

I’m mildly reluctant to that, because we’d stealthily force every user
into trusting our substitute servers.  OTOH I agree that the current
situation is not optimal.

What do people think?

Ludo’.

Re: What???s next?

[Ricardo Wurmus]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hello,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>> registered by default.  We cannot do that for someone installing Guix on
>> a foreign distro because that involves creating a file in /etc.
>
> The bayfront key is now registered by default? Does it mean that
> bayfront is ready to be part of the `%default-substitute-urls'? It
> doesn't seem to be the case currently:

No, but the mirror may soon be distributing substitutes that have been
created on and signed by bayfront.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hello!
>>>
>>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>>
>>>> Here are some other annoyances:
>>>>
>>>> * the verbosity of reporting hash mismatches.  You posted a neat little
>>>>   change for that some time ago, but I cannot find it any more.
>>>
>>> Oh right, see attached.
>>
>> I think you forgot to attach it.
>
> Oops.  Here we go:
>
> modified   nix/libstore/build.cc
> @@ -2449,8 +2449,11 @@ void DerivationGoal::registerOutputs()
>              Hash h2 = recursive ? hashPath(ht, actualPath).first : hashFile(ht, actualPath);
>              if (h != h2)
>                  throw BuildError(
> -                    format("output path `%1%' should have %2% hash `%3%', instead has `%4%'")
> -                    % path % i->second.hashAlgo % printHash16or32(h) % printHash16or32(h2));
> +                    format("%1% hash mismatch for output path `%2%'\n"
> +			   "  expected: %3%\n"
> +			   "  actual:   %4%")
> +                    % i->second.hashAlgo % path
> +		    % printHash16or32(h) % printHash16or32(h2));
>          }
>
>          /* Get rid of all weird permissions.  This also checks that
> @@ -3096,7 +3099,9 @@ void SubstitutionGoal::finished()
>              Hash expectedHash = parseHash16or32(hashType, string(expectedHashStr, n + 1));
>              Hash actualHash = hashType == htSHA256 ? hash.first : hashPath(hashType, destPath).first;
>              if (expectedHash != actualHash)
> -                throw SubstError(format("hash mismatch in downloaded path `%1%': expected %2%, got %3%")
> +                throw SubstError(format("hash mismatch in downloaded path `%1%'\n"
> +					"  expected: %2%\n"
> +					"  actual:   %3%")
>                      % storePath % printHash(expectedHash) % printHash(actualHash));
>          }
>
> Should we apply it?

Yes, please.  This looks much better!  Thank you!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What's next?

[Roel Janssen]

Ludovic Courtès writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>>> registered by default.  We cannot do that for someone installing Guix on
>>> a foreign distro because that involves creating a file in /etc.
>>
>> Many installs are not on GuixSD. Can't we use the key that is stored
>> in the store itself? If /etc does not exist then use what comes
>> with the installation.
>
> The current behavior is to print a warning when /etc/guix/acl (the list
> of authorized keys) is empty or nonexistent.
>
> Your suggestion would be to automatically populate it, right?
>
> I’m mildly reluctant to that, because we’d stealthily force every user
> into trusting our substitute servers.  OTOH I agree that the current
> situation is not optimal.
>
> What do people think?

Maybe we could find a mid-way here by doing the same as Fedora does with
RPMfusion repositories: It asks the user for trusting the signing keys
before enabling the repository.

So in our case it would be something like:
$ guix package -i emacs
A `substitute' is available for this package on
https://mirror.hydra.gnu.org.  This means we can download the binary
output for this package, instead of compiling it from its source code.
Do you want to use this substitute server with key ... for this package,
and for future packages? [y/N]

We need to find the proper wording for this message.  Using this, we can
still let the user decide, but we can make it a lot easier for the user
to make a decision -- a 'yes' or 'no' answer to a question is easier
than a paragraph in the manual with instructions to enable it.

Kind regards,
Roel Janssen

Re: What's next?

[Maxim Cournoyer]

ludo@gnu.org (Ludovic Courtès) writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>>> registered by default.  We cannot do that for someone installing Guix on
>>> a foreign distro because that involves creating a file in /etc.
>>
>> Many installs are not on GuixSD. Can't we use the key that is stored
>> in the store itself? If /etc does not exist then use what comes
>> with the installation.
>
> The current behavior is to print a warning when /etc/guix/acl (the list
> of authorized keys) is empty or nonexistent.
>
> Your suggestion would be to automatically populate it, right?
>
> I’m mildly reluctant to that, because we’d stealthily force every user
> into trusting our substitute servers.  OTOH I agree that the current
> situation is not optimal.
>

Maybe there could be a prompt that tells the user the current message
(no keys in /etc/guix/acl) and then asks them if they'd like to register
the default Guix substitute server keys? That'd be a middle ground
solution.

Maxim

Re: What’s next?

[myglc2]

On 05/27/2017 at 12:13 Ludovic Courtès writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>> Leo Famulari <leo@famulari.name> writes:
>>>
>>>> So, I use and recommend `guix pull`!
>>>
>>> I use it too.  Statements by others in this thread that "nobody" uses it
>>> or that "everyone" is using Git are mistaken.
>>>
>>> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
>>> IMO, the biggest problem with 'guix pull' is that there is no easy
>>> rollback.  I can live with long execution times (--fallback is fine, but
>>> it'd be nice if substitutes were available more often), and I can live
>>> with 'guix pull' causing me to get a version of guix that's broken
>>> somehow, but the inability to easily roll back when things go south
>>> makes me hesitant to run 'guix pull' regularly.
>>
>> I believe this can be fixed by adding more links to “.config/guix”,
>> i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
>> and then link from there to “latest”.  On update it would create a new
>> link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
>> be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.
>
> There would be some similarity with profiles.  Should we simply use
> profiles, and effectively turn ~/.config/guix/latest into a profile,
> with generations etc.?

+1

Re: What’s next?

[myglc2]

On 05/24/2017 at 21:56 Ricardo Wurmus writes:

> Catonano <catonano@gmail.com> writes:
>
>> 2017-05-24 18:25 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:
> […]
>>> A friend of mine is having a second look at Guix (not SD yet) and one of
>>> the most confusing things initially is `guix pull'.  "When/how do I use
>>> that," he asks...and I can only say: I'm not using that...I think we
>>> want this to work--or something like this, we talked about this at
>>> FOSDEM, but AFAIK everyone is using Guix with Git.
>>>
>>> He responds with: then *why* is it in the manual.  I have no answer.
>>> Possibly I'm wrong and/or my information is outdated?
>>>
>>
>> This is an important point for me too
>>
>> I realized that everyone is using git and not guix pull just yesterday
> […]
>> I think this is a problem. It' s unfair to newcomers and it damages Guix as
>> a project because it makes the learning curve steeper with not so much of a
>> point why.
>
> There are two reasons why developers use Guix from git:
>
> * it allows them to add new packages and features to Guix itself.  This
>   is something “guix pull” doesn’t support well.
>
> * it doesn’t require compiling all of Guix on each update.

Hmm, that explains how it feels. Why doesn't it just pull the compiled guix files?

Re: What???s next?

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>> registered by default.  We cannot do that for someone installing Guix on
>> a foreign distro because that involves creating a file in /etc.
>
> The bayfront key is now registered by default?

Yes!

> Does it mean that bayfront is ready to be part of the
> `%default-substitute-urls'?

Not yet!

Currently it only builds for x86_64/i686, it doesn’t offload anywhere,
etc.  But the fact that it’s registered means that we should eventually
be able to transition transparently (for GuixSD users at least).

Ludo’.

Re: What’s next?

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Hi,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hi Brendan,
>>
>> Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:
>>
>>> One little annoyance I have is that guix takes every opportunity to
>>> update the list of substitutes when guix build, guix package -u, etc...
>>> is run, and fills my terminal with output like:
>>>
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> ...
>>
>> I agree, it’s actually a huge annoyance.  It relates to grafts and how
>> they are implemented; I took a stab at fixing this a while back but the
>> approach turned out to be (mostly?) misguided:
>>
>>   https://bugs.gnu.org/22990
>>
>> Would be worth thinking through it again!
>
> Maybe we could make a timer variable configurable, so that at least it
> doesn't try to refresh this information at *every* guix command?

That’s not what’s happening.  When you see repeated lines that go
directly to 100%, that’s because it’s fetching metadata for just a
single package, and does that one by one.

In other cases, it fetches as many as possible at once, and uses HTTP
pipelining to send all these requests to hydra.gnu.org.

Ludo’.

Re: What's next?

[Ludovic Courtès]

Roel Janssen <roel@gnu.org> skribis:

> Ludovic Courtès writes:
>
>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>>
>>> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>>>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>>>> registered by default.  We cannot do that for someone installing Guix on
>>>> a foreign distro because that involves creating a file in /etc.
>>>
>>> Many installs are not on GuixSD. Can't we use the key that is stored
>>> in the store itself? If /etc does not exist then use what comes
>>> with the installation.
>>
>> The current behavior is to print a warning when /etc/guix/acl (the list
>> of authorized keys) is empty or nonexistent.
>>
>> Your suggestion would be to automatically populate it, right?
>>
>> I’m mildly reluctant to that, because we’d stealthily force every user
>> into trusting our substitute servers.  OTOH I agree that the current
>> situation is not optimal.
>>
>> What do people think?
>
> Maybe we could find a mid-way here by doing the same as Fedora does with
> RPMfusion repositories: It asks the user for trusting the signing keys
> before enabling the repository.
>
> So in our case it would be something like:
> $ guix package -i emacs
> A `substitute' is available for this package on
> https://mirror.hydra.gnu.org.  This means we can download the binary
> output for this package, instead of compiling it from its source code.
> Do you want to use this substitute server with key ... for this package,
> and for future packages? [y/N]

It cannot work this way because the decision has to be made by the
sysadmin, not by unprivileged users.

Also, I like that ‘guix package’ is non-interactive.

Ludo’.

Re: What’s next?

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Oops.  Here we go:
>>
>> modified   nix/libstore/build.cc
>> @@ -2449,8 +2449,11 @@ void DerivationGoal::registerOutputs()
>>              Hash h2 = recursive ? hashPath(ht, actualPath).first : hashFile(ht, actualPath);
>>              if (h != h2)
>>                  throw BuildError(
>> -                    format("output path `%1%' should have %2% hash `%3%', instead has `%4%'")
>> -                    % path % i->second.hashAlgo % printHash16or32(h) % printHash16or32(h2));
>> +                    format("%1% hash mismatch for output path `%2%'\n"
>> +			   "  expected: %3%\n"
>> +			   "  actual:   %4%")
>> +                    % i->second.hashAlgo % path
>> +		    % printHash16or32(h) % printHash16or32(h2));
>>          }
>>
>>          /* Get rid of all weird permissions.  This also checks that
>> @@ -3096,7 +3099,9 @@ void SubstitutionGoal::finished()
>>              Hash expectedHash = parseHash16or32(hashType, string(expectedHashStr, n + 1));
>>              Hash actualHash = hashType == htSHA256 ? hash.first : hashPath(hashType, destPath).first;
>>              if (expectedHash != actualHash)
>> -                throw SubstError(format("hash mismatch in downloaded path `%1%': expected %2%, got %3%")
>> +                throw SubstError(format("hash mismatch in downloaded path `%1%'\n"
>> +					"  expected: %2%\n"
>> +					"  actual:   %3%")
>>                      % storePath % printHash(expectedHash) % printHash(actualHash));
>>          }
>>
>> Should we apply it?
>
> Yes, please.  This looks much better!  Thank you!

Applied!

Ludo’.

Re: What's next?

[Pjotr Prins]

On Tue, May 30, 2017 at 05:19:09PM +0200, Ludovic Courtes wrote:
> > A `substitute' is available for this package on
> > https://mirror.hydra.gnu.org.  This means we can download the binary
> > output for this package, instead of compiling it from its source code.
> > Do you want to use this substitute server with key ... for this package,
> > and for future packages? [y/N]
> 
> It cannot work this way because the decision has to be made by the
> sysadmin, not by unprivileged users.

The first time guix is run it is by a system administrator (almost
100% certain), typically from a binary installation.

> Also, I like that 'guix package' is non-interactive.

Sure. But it is a step that is not required on GuixSD. The key comes
with the base install of the guix package. 

Can't we just fetch the key from the store? Why does it have to be
/etc? First look in /etc. If that does not exist, fetch from the guix
package itself. It is there. /etc overrides the key in the store.

Now it is a needless manual step. I say, it is an easy thing to
automate. Automate if you can.

Maybe not the highest priority, but if we come with a solution it
should be easy to do.

Pj.

-- 

Re: What’s next?

[Maxim Cournoyer]

Hello Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Hi,
>>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Hi Brendan,
>>>
>>> Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:
>>>
>>>> One little annoyance I have is that guix takes every opportunity to
>>>> update the list of substitutes when guix build, guix package -u, etc...
>>>> is run, and fills my terminal with output like:
>>>>
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> ...
>>>
>>> I agree, it’s actually a huge annoyance.  It relates to grafts and how
>>> they are implemented; I took a stab at fixing this a while back but the
>>> approach turned out to be (mostly?) misguided:
>>>
>>>   https://bugs.gnu.org/22990
>>>
>>> Would be worth thinking through it again!
>>
>> Maybe we could make a timer variable configurable, so that at least it
>> doesn't try to refresh this information at *every* guix command?
>
> That’s not what’s happening.  When you see repeated lines that go
> directly to 100%, that’s because it’s fetching metadata for just a
> single package, and does that one by one.
>
> In other cases, it fetches as many as possible at once, and uses HTTP
> pipelining to send all these requests to hydra.gnu.org.

My observation was that for the same exact command (I just tried: "guix
environment emacs-el-mock") entered twice:

--8<---------------cut here---------------start------------->8---
$ guix environment emacs-el-mock
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'...   0.0
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'...  33.3
[...] # (downloads a bunch of substitute)
[env]$ exit
$ guix environment emacs-el-mock
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
[env]$
--8<---------------cut here---------------end--------------->8---

We can see that a connection is made to the substitute server(s) and
data is retrieved each time the command is entered. Why does it need to
refresh the list of substitutes the second time? Everything needed is
already in the store.

Maxim

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>> Leo Famulari <leo@famulari.name> writes:
>>>
>>>> So, I use and recommend `guix pull`!
>>>
>>> I use it too.  Statements by others in this thread that "nobody" uses it
>>> or that "everyone" is using Git are mistaken.
>>>
>>> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
>>> IMO, the biggest problem with 'guix pull' is that there is no easy
>>> rollback.  I can live with long execution times (--fallback is fine, but
>>> it'd be nice if substitutes were available more often), and I can live
>>> with 'guix pull' causing me to get a version of guix that's broken
>>> somehow, but the inability to easily roll back when things go south
>>> makes me hesitant to run 'guix pull' regularly.
>>
>> I believe this can be fixed by adding more links to “.config/guix”,
>> i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
>> and then link from there to “latest”.  On update it would create a new
>> link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
>> be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.
>
> There would be some similarity with profiles.  Should we simply use
> profiles, and effectively turn ~/.config/guix/latest into a profile,
> with generations etc.?

That’s not a bad idea!  It sure beats messing with a single link and it
makes it possible to more easily manage different versions of Guix.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Release!

[Ludovic Courtès]

Hello Guix!

ludo@gnu.org (Ludovic Courtès) skribis:

> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.

Oops, that was 5 months ago…  :-)

> Here are some important items I can think of:
>
>   • Merge the ncurses installer (the ‘wip-installer’) branch.
>
>   • We’re supposed to freeze ‘core-updates’ in a couple of days and we
>     have defined a couple of important goals:
>     <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00120.html>.
>     I have a vague feeling that the ‘wip-build-systems-gexp’ merge is
>     not for this time yet…
>
>   • ‘guix offload’ terrible bug: <https://bugs.gnu.org/26976>.
>
>   • Guile 2.2 compiler terrible issue:
>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>
>   • Adjust the release process so we don’t find ourselves without
>     substitutes for the ‘guix’ package, as reported at
>     <https://bugs.gnu.org/27032>.
>
>   • Prepare to migrate to the new build farm: the hardware of
>     bayfront.guixsd.org has been fixed (it was unreliable until we
>     changed its CPUs), so now we need to fix a couple of issues in
>     Cuirass and generally improve it so we can, via its HTTP interface,
>     add new jobsets and so on.  Of course we’ll have to work
>     on that incrementally.
>
>   • Finalize the new bootloader API and support for new bootloaders:
>     <https://bugs.gnu.org/26339>.
>
>   • Merge the potluck!  <https://bugs.gnu.org/26645>

Good news is that we made progress on some of these issues, but not all;
we also made progress on other things which are really cool, such as the
ISO installer.

Regardless, I think we should be planning for a release within a couple
of weeks.  An important question to me is ‘wip-installer’: what do we
do, Danny?  John?

I would have loved to have a fix to at least mitigate Guile’s compiler
resource consumption issue.  I made some progress at
<https://bugs.gnu.org/28590> notably, but it’s not over yet.

In the meantime, it would be great if people could run an installation
from an ISO image and report bugs and glitches:

  https://www.gnu.org/software/guix/manual/html_node/Building-the-Installation-Image.html

Thoughts?

Ludo’.

Re: Release!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

On Wed, 04 Oct 2017 17:12:36 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> In the meantime, it would be great if people could run an installation
> from an ISO image and report bugs and glitches:
> 
>   https://www.gnu.org/software/guix/manual/html_node/Building-the-Installation-Image.html
> 
> Thoughts?

Unfortunately, building the ISO image has broken since I was
successfully using it [1].

One other issue I was thinking of recently was the size of the image. I
checked an ISO of the installation system I generated a while ago, and
it's 1.1GB in size. A actual CD can only hold 0.7GB of data. This might
be something that just needs documenting, saying that this is for a
DVD, and not a CD, or, it the size could be reduced so that it fits on
a CD.

When I build the installation system, and run guix size, it says 1031.0
MiB, so I guess looking at the breakdown there and trying to reduce it
would be a way to start, if this is a worthwhile objective.

I don't have any use for a physical CD installation image, so if anyone
is interested in having this, now is the time to say so?


1: guix system disk-image --file-system-type=iso9660
gnu/system/install.scm

error: changing mode of
`/tmp/root//gnu/store/jjna4ivyidxfq40mq78g97yzg0wfzcqy-shadow-4.5/bin/su'
to 100555: Operation not permitted ERROR: In procedure scm-error:
ERROR: failed to register store items "/xchg/system"

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Release!

[Ludovic Courtès]

Heya!

Christopher Baines <mail@cbaines.net> skribis:

> Unfortunately, building the ISO image has broken since I was
> successfully using it [1].

Oops, weird.  Let’s investigate.

> One other issue I was thinking of recently was the size of the image. I
> checked an ISO of the installation system I generated a while ago, and
> it's 1.1GB in size. A actual CD can only hold 0.7GB of data. This might
> be something that just needs documenting, saying that this is for a
> DVD, and not a CD, or, it the size could be reduced so that it fits on
> a CD.
>
> When I build the installation system, and run guix size, it says 1031.0
> MiB, so I guess looking at the breakdown there and trying to reduce it
> would be a way to start, if this is a worthwhile objective.

We can do both.  :-)

I think reducing the size of package closures in general is
worthwhile—there’s room for progress.

Yet, we probably won’t make it fit in 650 MiB this easily, so we should
document this.

Thanks for your feedback,
Ludo’.

> 1: guix system disk-image --file-system-type=iso9660
> gnu/system/install.scm
>
> error: changing mode of
> `/tmp/root//gnu/store/jjna4ivyidxfq40mq78g97yzg0wfzcqy-shadow-4.5/bin/su'
> to 100555: Operation not permitted ERROR: In procedure scm-error:
> ERROR: failed to register store items "/xchg/system"

Re: Release!

[Ricardo Wurmus]

Hi Ludo,

>> Here are some important items I can think of:
[…]
>>   • Guile 2.2 compiler terrible issue:
>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

One way to side-step this issue for the upcoming release is to use one
Guile process per file when running “guix pull”.  This will make it run
a lot slower, but that would be better than the current situation.

Maybe we could make parallel compilation within the same Guile process
an option, so that it won’t be needlessly slow on machines within the
RAM goldilocks zone.

I’ve reverted f07041f7d25badb7d74b8fad6ee446a12af04f63 locally on my
i686 netbook with 1GB RAM and tested it with “guix pull
--url=/path/to/guix”.  This seems to work — it’s still running… :)

One annoyance is that after compiling one file it prints this message:

   Some deprecated features have been used.  Set the environment
   variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
   program to get more information.  Set it to "no" to suppress
   this message.

I suppose we should suppress this for “guix pull”.

Do you want to make this behaviour optional, e.g. through a command line
switch like “--sloth”?  Or should this be the default until the problem
in Guile/libgc is fixed?

>>   • Prepare to migrate to the new build farm: the hardware of
>>     bayfront.guixsd.org has been fixed (it was unreliable until we
>>     changed its CPUs), so now we need to fix a couple of issues in
>>     Cuirass and generally improve it so we can, via its HTTP interface,
>>     add new jobsets and so on.  Of course we’ll have to work
>>     on that incrementally.

I guess we’ll be using berlin.guixsd.org instead of bayfront, no?  I’m
currently installing additional servers (we’re now at 13 build servers,
I’ll get this up to 18 this week).

>>   • Merge the potluck!  <https://bugs.gnu.org/26645>

About that… We now have a JSON importer, so maybe it’s worth using the
even simpler JSON package format instead of the simplified S-expression
format that Andy proposed.  What do you think?  Should we discuss this
at <https://bugs.gnu.org/26645>?

Who would like to help us squash some bugs before the release?  Let’s
try to fix at least 5 more bugs!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Hi Recardo,
Hi Ludo,

> >>   • Merge the potluck!  <https://bugs.gnu.org/26645>  

> About that… We now have a JSON importer, so maybe it’s worth using the
> even simpler JSON package format instead of the simplified S-expression
> format that Andy proposed.  What do you think?  Should we discuss this
> at <https://bugs.gnu.org/26645>?

FWIW, I much prefer s-exp, and the generated file is a scheme file right? I very
much doubt guix and potluck package developers can't easily read (and write :))
s-exp, so why would we 'abandon' s-exp, what would we win here? These files will
never be processed by anything else but guix and/or potluck, and the 'package
developers' do all know scheme perfectly well...

my 2c :)
Thanks for the fantastic work, both Guix and potluck!

David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Mark H Weaver]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Ludo,
>
>>> Here are some important items I can think of:
> […]
>>>   • Guile 2.2 compiler terrible issue:
>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>
> One way to side-step this issue for the upcoming release is to use one
> Guile process per file when running “guix pull”.  This will make it run
> a lot slower, but that would be better than the current situation.

I've attached a workaround that I've been using for the last 6 weeks on
my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
otherwise it would not be able to successfully build the 'guix' package.

Note that I never use 'guix pull', so I'm not sure off-hand whether this
solves the problem there, but it certainly greatly reduces the memory
needed to run 'make' and thus to build the 'guix' package.

This patch modifies build-aux/compile-all.scm to work as follows: after
loading all modules in the parent process, it then forks off a child and
compiles 20 modules in parallel within that child while the parent
waits.  When the child is finished compiling those 20 modules, the child
exits (thus freeing the memory leaked during compilation), and then the
parent spawns a new child to compile the next 20 modules, and so on,
until all the modules are compiled.

We should probably consider applying this to master.  Thoughts?

      Mark



[-- Attachment #2: [PATCH] DRAFT: build: Compile scheme modules in batches --]
[-- Type: text/x-patch, Size: 3684 bytes --]

From 05d5581ff71eb3b48773a5d46b612202de0492fb Mon Sep 17 00:00:00 2001
From: Mark H Weaver <mhw@netris.org>
Date: Tue, 22 Aug 2017 03:26:10 -0400
Subject: [PATCH] DRAFT: build: Compile scheme modules in batches.

---
 build-aux/compile-all.scm | 48 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 147bb8019..96658e069 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
 ;;; Copyright © 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Mark H Weaver <mhw@netris.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -19,6 +20,7 @@
 
 (use-modules (system base target)
              (system base message)
+             (srfi srfi-1)
              (ice-9 match)
              (ice-9 threads)
              (guix build utils))
@@ -118,13 +120,45 @@
   ((_ . files)
    (let ((files (filter file-needs-compilation? files)))
      (for-each load-module-file files)
-     (let ((mutex (make-mutex)))
-       ;; Make sure compilation related modules are loaded before starting to
-       ;; compile files in parallel.
-       (compile #f)
-       (par-for-each (lambda (file)
-                       (compile-file* file mutex))
-                     files)))))
+     ;; Make sure compilation related modules are loaded before starting to
+     ;; compile files in parallel.
+     (compile #f)
+     ;; Flush all ports before entering the fork loop, to avoid flushing them
+     ;; more than once within the child processes created below.
+     (flush-all-ports)
+
+     ;; FIXME The following loop works around the apparent memory leak in the
+     ;; compiler of guile-2.2.2, where compiling scheme modules requires
+     ;; increasing amounts of memory, up to nearly 2 gigabytes when all guix
+     ;; sources are compiled within a single process.
+     ;;
+     ;; Ideally, we would simply apply 'par-for-each' to the entire set of
+     ;; files.  For now, to work around the memory leak, we spawn subprocesses
+     ;; to compile the files in batches of up to 20 files each.
+     (let fork-loop ((files files))
+       (unless (null? files)
+         (call-with-values (lambda ()
+                             (split-at files (min 20 (length files))))
+           (lambda (current-batch remaining-files)
+             ;; IMPORTANT: as noted in the Guile manual, it is unsafe to fork a
+             ;; process that has multiple threads running.  Here we avoid this
+             ;; difficulty by spawning threads only within the child processes,
+             ;; which never call fork.
+             (match (primitive-fork)
+               (0
+                ;; This is the child.  It spawns threads but never forks.
+                (let ((mutex (make-mutex)))
+                  (par-for-each (lambda (file)
+                                  (compile-file* file mutex))
+                                current-batch))
+                (primitive-exit))
+               (child-pid
+                ;; This is the parent.  It forks but never spawns threads.
+                (match (waitpid child-pid)
+                  ((_ . 0)
+                   (fork-loop remaining-files))
+                  ((_ . status)
+                   (primitive-exit (or (status:exit-val status) 1)))))))))))))
 
 ;;; Local Variables:
 ;;; eval: (put 'with-target 'scheme-indent-function 1)
-- 
2.14.1

Re: Release!

[Hartmut Goebel]

Am 07.10.2017 um 01:31 schrieb David Pirotte:
> so why would we 'abandon' s-exp, what would we win here?

It might be interesting to *create* these files using tools written in
other programming languages. And modules for creating *JSON* are
available for most programming languages.

(OTOH TOML[]1] could be a better format than JSON – it's much like
.ini-files, but more formal specification. A scheme implementation is
available [2].)

[1] https://en.wikipedia.org/wiki/TOML
[2] 0f52bab3e3cfeb261110f9217c39eb03e78d026a

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Release!

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

Hello Hartmut,

> > so why would we 'abandon' s-exp, what would we win here?  

> It might be interesting to *create* these files using tools written in
> other programming languages. And modules for creating *JSON* are
> available for most programming languages.

But that would be another tool, another package manager, not potluck... written by
we don't know who, neither when ... and that would be a totally separate effort,
that would not contribute to potluck ... which needs help (I wish I had the time...)

Potluck package definitions are generated, adjusted 'by hand' - mostly to update
the description, sometimes the copyright - then that is used to generated a Guix
package, which is a Guile scheme module: It makes no sense to me that the
potluck package representation would be anything but s-expr:

	actually, most guilers do the exact opposite :) -  when an app or a lib
	either produces or needs xml, html, json ... the first thing they do is to
	transform these into s-expr, so these become (a lot more) readable and
	hackable ...

> (OTOH TOML[]1] could be a better format than JSON – it's much like
> .ini-files, but more formal specification...

Absolutely terrible :):) I hope we never do that, at least not for the 'official'
and maintained potluck package representation.

Again, my 2c.
David


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]

On Sat, Oct 07, 2017 at 12:06:51AM -0400, Mark H Weaver wrote:
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
> > Hi Ludo,
> >
> >>> Here are some important items I can think of:
> > […]
> >>>   • Guile 2.2 compiler terrible issue:
> >>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
> >
> > One way to side-step this issue for the upcoming release is to use one
> > Guile process per file when running “guix pull”.  This will make it run
> > a lot slower, but that would be better than the current situation.
> 
> I've attached a workaround that I've been using for the last 6 weeks on
> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
> otherwise it would not be able to successfully build the 'guix' package.
> 
> Note that I never use 'guix pull', so I'm not sure off-hand whether this
> solves the problem there, but it certainly greatly reduces the memory
> needed to run 'make' and thus to build the 'guix' package.
> 
> This patch modifies build-aux/compile-all.scm to work as follows: after
> loading all modules in the parent process, it then forks off a child and
> compiles 20 modules in parallel within that child while the parent
> waits.  When the child is finished compiling those 20 modules, the child
> exits (thus freeing the memory leaked during compilation), and then the
> parent spawns a new child to compile the next 20 modules, and so on,
> until all the modules are compiled.
> 
> We should probably consider applying this to master.  Thoughts?
> 
>       Mark
> 
> 

Can we give it a set to compile in order? For example building
guix/build[-system] and then gnu/build, and then only at the end build
gnu/services {guix,gnu}/tests? It might be slightly faster to build the
modules that don't pull in gnu/packages/* first and then end with those
that would use those modules.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release!

[Ricardo Wurmus]

Hi David,

>> >>   • Merge the potluck!  <https://bugs.gnu.org/26645>
>
>> About that… We now have a JSON importer, so maybe it’s worth using the
>> even simpler JSON package format instead of the simplified S-expression
>> format that Andy proposed.  What do you think?  Should we discuss this
>> at <https://bugs.gnu.org/26645>?
>
> FWIW, I much prefer s-exp, and the generated file is a scheme file right? I very
> much doubt guix and potluck package developers can't easily read (and write :))
> s-exp, so why would we 'abandon' s-exp, what would we win here? These files will
> never be processed by anything else but guix and/or potluck, and the 'package
> developers' do all know scheme perfectly well...

I’m not saying that JSON is “better” than s-exps.

Part of the potluck effort as I understood it is to simplify packaging.
The potluck package definition is strictly simpler than Guix package
definitions in that there’s less boilerplate and inputs are really just
strings.  Taking that aspect of simplifying packaging even farther we
can reduce the syntax even more.

The target audience here has little overlap with Guix developers.  Guix
won’t adopt JSON as a packaging format; that’s not what this is about.
The goal I had in mind when I worked on the JSON importer was to make
packaging even simpler for people who don’t really care all that much
about packaging — if they did they would probably want to learn about
how to contribute to Guix, and thus would want to learn the S-expr
syntax we use in Guix.

There are users of Guix who benefit from its features as a personal
package manager.  Users can already add their own packages via
GUIX_PACKAGE_PATH.  We support those who don’t feel comfortable writing
Scheme by offering a JSON importer; with just a minor change to “guix
build” we can even build JSON packages directly, without making people
convert them to Scheme modules first.

I think that this feature can be useful within the context of the
potluck.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Hi Mark,

> I've attached a workaround that I've been using for the last 6 weeks on
> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
> otherwise it would not be able to successfully build the 'guix' package.
>
> Note that I never use 'guix pull', so I'm not sure off-hand whether this
> solves the problem there, but it certainly greatly reduces the memory
> needed to run 'make' and thus to build the 'guix' package.

thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
My own attempt to revert the commit to build all files in one process
resulted in a guix pull that would only go to 20.4% and then sit there
doing nothing.

I’ll report back once I manage to run “guix pull --url=/path/to/guix”
with this patch.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>> I've attached a workaround that I've been using for the last 6 weeks on
>> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
>> otherwise it would not be able to successfully build the 'guix' package.
>>
>> Note that I never use 'guix pull', so I'm not sure off-hand whether this
>> solves the problem there, but it certainly greatly reduces the memory
>> needed to run 'make' and thus to build the 'guix' package.
>
> thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
> My own attempt to revert the commit to build all files in one process
> resulted in a guix pull that would only go to 20.4% and then sit there
> doing nothing.
>
> I’ll report back once I manage to run “guix pull --url=/path/to/guix”
> with this patch.

20 files at once is still too much when running “guix pull” on my i686
netbook.  I tried lowering this to 10 files and still ran out of memory
(after 70 minutes of compiling).  I’m now trying with 3 files at a time.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[Hartmut Goebel]

Am 07.10.2017 um 23:30 schrieb Ricardo Wurmus:
> The target audience here has little overlap with Guix developers.  […]
> The goal I had in mind when I worked on the JSON importer was to make
> packaging even simpler for people who don’t really care all that much
> about packaging – […]

+1

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>>> I've attached a workaround that I've been using for the last 6 weeks on
>>> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
>>> otherwise it would not be able to successfully build the 'guix' package.
>>>
>>> Note that I never use 'guix pull', so I'm not sure off-hand whether this
>>> solves the problem there, but it certainly greatly reduces the memory
>>> needed to run 'make' and thus to build the 'guix' package.
>>
>> thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
>> My own attempt to revert the commit to build all files in one process
>> resulted in a guix pull that would only go to 20.4% and then sit there
>> doing nothing.
>>
>> I’ll report back once I manage to run “guix pull --url=/path/to/guix”
>> with this patch.

Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
uses build-self.scm, so this patch has no effect on “guix pull”.

I’ll try to come up with another fix for guix pull.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

On Fri, 06 Oct 2017 15:01:42 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Heya!
> 
> Christopher Baines <mail@cbaines.net> skribis:
> 
> > Unfortunately, building the ISO image has broken since I was
> > successfully using it [1].  
> 
> Oops, weird.  Let’s investigate.

I'm very glad that now I've removed the setuid binaries from the store,
the building of the ISO image, and the associated iso-image-installer
system test are now working for me :)

Thanks for investigating and fixing this Ludo :D

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>>>> I've attached a workaround that I've been using for the last 6 weeks on
>>>> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
>>>> otherwise it would not be able to successfully build the 'guix' package.
>>>>
>>>> Note that I never use 'guix pull', so I'm not sure off-hand whether this
>>>> solves the problem there, but it certainly greatly reduces the memory
>>>> needed to run 'make' and thus to build the 'guix' package.
>>>
>>> thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
>>> My own attempt to revert the commit to build all files in one process
>>> resulted in a guix pull that would only go to 20.4% and then sit there
>>> doing nothing.
>>>
>>> I’ll report back once I manage to run “guix pull --url=/path/to/guix”
>>> with this patch.
>
> Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
> uses build-self.scm, so this patch has no effect on “guix pull”.

Right, this should be tested with “make”.

If it works fine here, we can port it to (guix build pull).

(And factorize it eventually…)

Ludo’.

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ludovic Courtès]

Hi,

Mark H Weaver <mhw@netris.org> skribis:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Hi Ludo,
>>
>>>> Here are some important items I can think of:
>> […]
>>>>   • Guile 2.2 compiler terrible issue:
>>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>>
>> One way to side-step this issue for the upcoming release is to use one
>> Guile process per file when running “guix pull”.  This will make it run
>> a lot slower, but that would be better than the current situation.
>
> I've attached a workaround that I've been using for the last 6 weeks on
> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
> otherwise it would not be able to successfully build the 'guix' package.
>
> Note that I never use 'guix pull', so I'm not sure off-hand whether this
> solves the problem there, but it certainly greatly reduces the memory
> needed to run 'make' and thus to build the 'guix' package.
>
> This patch modifies build-aux/compile-all.scm to work as follows: after
> loading all modules in the parent process, it then forks off a child and
> compiles 20 modules in parallel within that child while the parent
> waits.  When the child is finished compiling those 20 modules, the child
> exits (thus freeing the memory leaked during compilation), and then the
> parent spawns a new child to compile the next 20 modules, and so on,
> until all the modules are compiled.
>
> We should probably consider applying this to master.  Thoughts?

That sounds like a good idea.  If it works, I’m all for it.

We’ll need to do something similar in (guix scripts pull).

I’ve been working on having ‘guix pull’ build modules as several
derivations as part of <https://bugs.gnu.org/27284>.  I’ll try to post a
prototype hopefully later today.

Thanks,
Ludo’.

Re: Release!

[Ludovic Courtès]

Heya,

Ricardo Wurmus <rekado@elephly.net> skribis:

>>> Here are some important items I can think of:
> […]
>>>   • Guile 2.2 compiler terrible issue:
>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

I should mention that I spent entire days on this, and some of the
conclusions are described here:

  https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html
  https://bugs.gnu.org/28590

I invite the Guilers among us to join the fun.  :-)  A bug that serious
is a problem for Guile.

From a Guix perspective, we have to find alternate solutions to improve
scalability anyway:

  https://bugs.gnu.org/27284

>>>   • Prepare to migrate to the new build farm: the hardware of
>>>     bayfront.guixsd.org has been fixed (it was unreliable until we
>>>     changed its CPUs), so now we need to fix a couple of issues in
>>>     Cuirass and generally improve it so we can, via its HTTP interface,
>>>     add new jobsets and so on.  Of course we’ll have to work
>>>     on that incrementally.
>
> I guess we’ll be using berlin.guixsd.org instead of bayfront, no?

Yes.

> I’m currently installing additional servers (we’re now at 13 build
> servers, I’ll get this up to 18 this week).

Woohoo, thank you!

>>>   • Merge the potluck!  <https://bugs.gnu.org/26645>
>
> About that… We now have a JSON importer, so maybe it’s worth using the
> even simpler JSON package format instead of the simplified S-expression
> format that Andy proposed.  What do you think?  Should we discuss this
> at <https://bugs.gnu.org/26645>?

Yeah I think we need to reopen the discussion.  We discussed it at the
GHM but this should take place here on guix-devel I guess.

> Who would like to help us squash some bugs before the release?  Let’s
> try to fix at least 5 more bugs!

Yes, let’s squash bugs!

There are many ways people can help, and one of them is triage: close
bugs that are already fixed, determine whether old bugs are still
relevant and close those that are obsolete, retitle bugs to better
reflect what the problem is, ping people asking for more information.
After that, there are more-or-less “easy” packaging bugs in the list
that can be a good starting point.

Thanks,
Ludo’.

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

>> Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
>> uses build-self.scm, so this patch has no effect on “guix pull”.
>
> Right, this should be tested with “make”.
>
> If it works fine here, we can port it to (guix build pull).

Okay.  I’ll try this some time this week, as soon as I can.  Maybe
tonight.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> On Fri, 06 Oct 2017 15:01:42 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Heya!
>> 
>> Christopher Baines <mail@cbaines.net> skribis:
>> 
>> > Unfortunately, building the ISO image has broken since I was
>> > successfully using it [1].  
>> 
>> Oops, weird.  Let’s investigate.
>
> I'm very glad that now I've removed the setuid binaries from the store,
> the building of the ISO image, and the associated iso-image-installer
> system test are now working for me :)

Awesome, thanks for checking!

I really didn’t expect such a bug to crop up…

Ludo’.

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>>> Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
>>> uses build-self.scm, so this patch has no effect on “guix pull”.
>>
>> Right, this should be tested with “make”.
>>
>> If it works fine here, we can port it to (guix build pull).
>
> Okay.  I’ll try this some time this week, as soon as I can.  Maybe
> tonight.

“make” crashed when I ran this on the netbook.  It ran out of memory
when compiling python.scm, as it always does.

I cannot discern if this is an improvement, because the result is
ultimately the same on my hardware; but since Mark uses this
successfully on a Yeeloong with 1GB RAM, I have no objections to pushing
this to master.

Thanks!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[Ludovic Courtès]

Hello Guix,

I think we can start preparing the release for good now!

The situation of ‘guix pull’ is mitigated by the split of python.scm and
other big files into several pieces.  This is of course not ideal, but
it’s an improvement over 0.13.0 anyway.

I think the main tasks to prepare the release are (1) testing the GuixSD
ISO, and (2) updating ‘NEWS’.

Let’s get that done!

Ludo’.

Re: Release!

[Ludovic Courtès]

Hello!

Time flies, but I think we’re about ready for the release.  I’d like to
aim for next Tuesday (Dec. 5th) and fix small hickups and annoyances by
then, particularly regarding GuixSD installation.

Ricardo Wurmus <rekado@elephly.net> skribis:

>>> Here are some important items I can think of:
> […]
>>>   • Guile 2.2 compiler terrible issue:
>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

There’s more going on on the Guile side, but for now the recent split of
the big package files has helped reduce peak memory consumption
noticeably.

> One annoyance is that after compiling one file it prints this message:
>
>    Some deprecated features have been used.  Set the environment
>    variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
>    program to get more information.  Set it to "no" to suppress
>    this message.

This was also showing up a lot in ‘guix system init’ et al, when
compiling modules.  I think a912c723f76d9762072ce27204a9227a64bcb625
removes most of these.

>>>   • Prepare to migrate to the new build farm: the hardware of
>>>     bayfront.guixsd.org has been fixed (it was unreliable until we
>>>     changed its CPUs), so now we need to fix a couple of issues in
>>>     Cuirass and generally improve it so we can, via its HTTP interface,
>>>     add new jobsets and so on.  Of course we’ll have to work
>>>     on that incrementally.
>
> I guess we’ll be using berlin.guixsd.org instead of bayfront, no?

Yes.  Questions:

  1. Do we pre-register berlin’s key on GuixSD?

  2. Do we add berlin.guixsd.org to the list of substitute servers on
     GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
     both servers when retrieve substitute lists, which can be slightly
     slower/annoying, but otherwise I think it’s a win.

Thoughts?

The main GuixSD annoying messages that I think we should address by
Tuesday are:

  1. “Error in finalization thread: Bad file descriptor” coming from the
     Shepherd;

  2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”

Both are harmless but they may give users a false sense that something’s
broken.

Ludo’.

Re: Release!

[Maxim Cournoyer]

Hi!

ludo@gnu.org (Ludovic Courtès) writes:

[...]

>>
>> I guess we’ll be using berlin.guixsd.org instead of bayfront, no?
>
> Yes.  Questions:
>
>   1. Do we pre-register berlin’s key on GuixSD?

Isn't this already the case?

[...]

>
> The main GuixSD annoying messages that I think we should address by
> Tuesday are:

[...]

>
>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>

I had researched about that error a bit and it seems that it caused by
eudev lacking support for 'uaccess'[0]; so not something to fix in Guix
proper.

[0]  https://github.com/gentoo/eudev/issues/145

Maxim

Re: Release!

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>   1. Do we pre-register berlin’s key on GuixSD?

I vote yes.

>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>      both servers when retrieve substitute lists, which can be slightly
>      slower/annoying, but otherwise I think it’s a win.

I think it's worth the extra source of substitutes. Since adding
berlin.guixsd.org to my list of substitute URLs, it seems like I need to
build less often.

In my experience, it's annoying to query multiple servers when my
network connection is slow or unreliable. But, it's also annoying in
that situation to need to download source files from a variety of
upstream sites.

> The main GuixSD annoying messages that I think we should address by
> Tuesday are:
> 
>   1. “Error in finalization thread: Bad file descriptor” coming from the
>      Shepherd;

I'm not sure how to start debugging this :/ If I get some advice, I
could try to fix it on Sunday and Monday.

>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”

I haven't seen this one before.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release!

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>>   1. Do we pre-register berlin’s key on GuixSD?
>
> I vote yes.
>
>>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>>      both servers when retrieve substitute lists, which can be slightly
>>      slower/annoying, but otherwise I think it’s a win.
>
> I think it's worth the extra source of substitutes. Since adding
> berlin.guixsd.org to my list of substitute URLs, it seems like I need to
> build less often.
>
> In my experience, it's annoying to query multiple servers when my
> network connection is slow or unreliable. But, it's also annoying in
> that situation to need to download source files from a variety of
> upstream sites.

I’d like to add berlin.guix.org to the list of default substitute
servers, but before I can allow this with a good conscience I need to
increase space on the head node.  We were very busy, so the new head
node and storage aren’t ready yet, and the old node that’s currently
coordinating builds on berlin.guix.org has very limited storage.

I’ll try to get the new storage ready on Monday.

>>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>
> I haven't seen this one before.

I have seen it, but only when checking the output of dmesg.  I guess we
could just remove that rule.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[Ludovic Courtès]

Hello!

Leo Famulari <leo@famulari.name> skribis:

> On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>>   1. Do we pre-register berlin’s key on GuixSD?
>
> I vote yes.
>
>>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>>      both servers when retrieve substitute lists, which can be slightly
>>      slower/annoying, but otherwise I think it’s a win.
>
> I think it's worth the extra source of substitutes. Since adding
> berlin.guixsd.org to my list of substitute URLs, it seems like I need to
> build less often.
>
> In my experience, it's annoying to query multiple servers when my
> network connection is slow or unreliable. But, it's also annoying in
> that situation to need to download source files from a variety of
> upstream sites.

Sounds reasonable.  Let’s wait for a green light from Ricardo.

>> The main GuixSD annoying messages that I think we should address by
>> Tuesday are:
>> 
>>   1. “Error in finalization thread: Bad file descriptor” coming from the
>>      Shepherd;
>
> I'm not sure how to start debugging this :/ If I get some advice, I
> could try to fix it on Sunday and Monday.

I investigated and fixed it yesterday:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4bd70904c7f555a953808a9a4f892f462ffd352f

The thing is that there’s a separate finalization thread in Guile, which
takes care of finalizing dead objects.  For file ports, finalization
means closing the underlying file descriptor.  As it turns out,
‘close-port’ in Guile 2.0 would ignore EBADF errors, whereas
‘close-port’ in 2.2 reports them, hence the error.

The fix is to take extra care to close ports and not just the underlying
file descriptors.

>>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>
> I haven't seen this one before.

I compared the source of eudev and systemd, and indeed, like Maxim
wrote, the issue is that eudev does not support a “uaccess” built-in
tag.  So I pushed a fix that simply comments out that line:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f8446df663fecb5aa34e5c6dfa477544d3271d1e

There’s still room for improvement, but the console output is less messy
now.  :-)

Thanks,
Ludo’.

Re: Release!

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Leo Famulari <leo@famulari.name> writes:
>
>> On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>>>   1. Do we pre-register berlin’s key on GuixSD?
>>
>> I vote yes.
>>
>>>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>>>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>>>      both servers when retrieve substitute lists, which can be slightly
>>>      slower/annoying, but otherwise I think it’s a win.
>>
>> I think it's worth the extra source of substitutes. Since adding
>> berlin.guixsd.org to my list of substitute URLs, it seems like I need to
>> build less often.
>>
>> In my experience, it's annoying to query multiple servers when my
>> network connection is slow or unreliable. But, it's also annoying in
>> that situation to need to download source files from a variety of
>> upstream sites.
>
> I’d like to add berlin.guix.org to the list of default substitute
> servers, but before I can allow this with a good conscience I need to
> increase space on the head node.  We were very busy, so the new head
> node and storage aren’t ready yet, and the old node that’s currently
> coordinating builds on berlin.guix.org has very limited storage.
>
> I’ll try to get the new storage ready on Monday.

Awesome.  Let us know how it goes and what we should do!

Thanks,
Ludo’.

ISO image available for testing!

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Hello there!

I’ve uploaded a GuixSD installation ISO image for testing:

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz
  SHA256: 7e04017494c419bb6ca654cff05dc7e5ba4fdfd3c2812fba9ba6d929b319e93b

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz.sig

It was built with “guix system disk-image -t iso9660
gnu/system/install.scm” on commit 8d7f1d736.

Note that it’s 1.1G uncompressed so it won’t fit on a CD.

Feedback welcome!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: ISO image available for testing!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]


Ludovic Courtès writes:

> Hello there!
>
> I’ve uploaded a GuixSD installation ISO image for testing:
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz
>   SHA256: 7e04017494c419bb6ca654cff05dc7e5ba4fdfd3c2812fba9ba6d929b319e93b
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz.sig
>
> It was built with “guix system disk-image -t iso9660
> gnu/system/install.scm” on commit 8d7f1d736.
>
> Note that it’s 1.1G uncompressed so it won’t fit on a CD.
>
> Feedback welcome!

Hey,

I've attempted to use this to install GuixSD on a Bytemark
VM. Unfortunately I haven't succeeded yet. I managed to get as far as
running guix system init, but when I did, it started downloading the
bootstrap binaries, and then building binutils.

When I run guix system build, or guix build with the --dry-run options,
it says that it will download, rather than building, but it doesn't.

I also made a few other notes during the installation process:

 - The documentation mentions ipconfig to get the wired interface up,
   but it isn't available in the image. I had to run ip link set eth0
   up.
 - The documentation says that the ssh server needs to be started, but
   it was running without me starting it.

Any tips on debugging the building from source issue?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: ISO image available for testing!

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> I've attempted to use this to install GuixSD on a Bytemark
> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
> running guix system init, but when I did, it started downloading the
> bootstrap binaries, and then building binutils.
>
> When I run guix system build, or guix build with the --dry-run options,
> it says that it will download, rather than building, but it doesn't.

I’m pretty sure the building-from-source comes from grafts where the
package replacements, for some reason, are not getting built by Hydra.
I thought this had been fixed by
b574cee361bdabf21f5506252b6a183dcfcd028d, but clearly it hasn’t.

I’ll investigate tomorrow.

> I also made a few other notes during the installation process:
>
>  - The documentation mentions ipconfig to get the wired interface up,
>    but it isn't available in the image. I had to run ip link set eth0
>    up.

It mentions “ifconfig” (with an “f”) and that one is present.  Or am I
missing something?

>  - The documentation says that the ssh server needs to be started, but
>    it was running without me starting it.

Indeed.  I’ve pushed a fix as aab322d909c0b4abec132ef7aff31c31a1208841
in the new ‘version-0.14.0’ branch (note that it changes the ABI of (gnu
services ssh)).

Thanks,
Ludo’.

Re: Release!

[Maxim Cournoyer]

Hi!

ludo@gnu.org (Ludovic Courtès) writes:

[...] (Great fix! :)

>>>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>>
>> I haven't seen this one before.
>
> I compared the source of eudev and systemd, and indeed, like Maxim
> wrote, the issue is that eudev does not support a “uaccess” built-in
> tag.  So I pushed a fix that simply comments out that line:
>
>   https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f8446df663fecb5aa34e5c6dfa477544d3271d1e

Nitpick: Maybe we could reference to the issue so that a maintainer can
decide to remove that alteration after the "uaccess" issue gets properly
fixed (implemented).

> There’s still room for improvement, but the console output is less messy
> now.  :-)

Thank you for it!

Maxim

Re: ISO image available for testing!

[Ludovic Courtès]

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> I've attempted to use this to install GuixSD on a Bytemark
> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
> running guix system init, but when I did, it started downloading the
> bootstrap binaries, and then building binutils.
>
> When I run guix system build, or guix build with the --dry-run options,
> it says that it will download, rather than building, but it doesn't.

It turned out to be issues related to grafts and to what Hydra builds,
fixed with these commits:

3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable.
91c9b5d01 * packages: 'package-grafts' trims native inputs.
ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls.
f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages.

The Binutils issue is fixed by f00b85ff8.

Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in
its dependency graph, via gettext.  Thus, the expat graft was picked up
as a candidate graft.  However, expat itself was subject to the glibc
graft, and since there was no substitute for this particular expat, we’d
have to build it first, just to throw it away later on because coreutils
does not refer to it at run time.

Long story short: we were flagging native inputs as potential sources of
grafts even though, by definition, native inputs are not referred to at
run time.

The last commit ensures that Hydra builds the replacement for
‘ghostscript-with-cups’.

What’s tricky is that one doesn’t notice these issues unless starting
from a fresh store.

I’ve uploaded an updated ISO image, which I used to test substitute
availability and grafts.  If you have time in the coming hours, feedback
welcome:

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz
  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz.sig

Ludo’.

Re: ISO image available for testing!

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:
> 91c9b5d01 * packages: 'package-grafts' trims native inputs.

[...]

> Long story short: we were flagging native inputs as potential sources of
> grafts even though, by definition, native inputs are not referred to at
> run time.

I agree that this *should* never happen, but I see little reason for
confidence that it never happens in actual fact.

What would happen if a reference to a native-input *was* present in the
build outputs?  The reason I ask is that, for security reasons, it's
obviously very important to reliably avoid using ungrafted software at
run time.

I'm concerned that this recent change could cause minor
nearly-undetectable packaging mistakes to become major security holes.

One solution would be to explicitly check build outputs for references
to native-inputs, and to force a build failure in that case.

What do you think?

    Regards,
      Mark

Re: ISO image available for testing!

[Ben Woodcroft]

On 06/12/17 10:52, Mark H Weaver wrote:
> Hi Ludovic,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
> [...]
>
>> Long story short: we were flagging native inputs as potential sources of
>> grafts even though, by definition, native inputs are not referred to at
>> run time.
> I agree that this *should* never happen, but I see little reason for
> confidence that it never happens in actual fact.
>
> What would happen if a reference to a native-input *was* present in the
> build outputs?  The reason I ask is that, for security reasons, it's
> obviously very important to reliably avoid using ungrafted software at
> run time.
>
> I'm concerned that this recent change could cause minor
> nearly-undetectable packaging mistakes to become major security holes.
>
> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.
I believe there are a number of cases of this that happen when binaries 
are wrapped with paths derived from getenv, e.g. this phase in bamm:

          (add-after 'install 'wrap-executable
            (lambda* (#:key outputs #:allow-other-keys)
             (let* ((out  (assoc-ref outputs "out"))
                    (path (getenv "PATH")))
               (wrap-program (string-append out "/bin/bamm")
                 `("PATH" ":" prefix (,path))))
             #t))

It would be good to stop using getenv for this, for the reasons 
described here and others e.g. non-reproducibility, unnecessary 
dependencies etc.

Is there some easy way to "getenv" excluding unwanted components of an 
environment variable?

Thanks, ben

Re: native-inputs ending up as run-time references [was: ISO image available for testing!]

[Tobias Geerinckx-Rice]

Mark!
Ludovic!

Mark H Weaver wrote on 06/12/17 at 01:52:
> ludo@gnu.org (Ludovic Courtès) writes:
>> Long story short: we were flagging native inputs as potential 
>> sources of grafts even though, by definition, native inputs are
>> not referred to at run time.
> 
> I agree that this *should* never happen, but I see little reason for 
> confidence that it never happens in actual fact.

Hold on. I thought this happened *all the actual time*.

To me, the output of ‘guix graph’ implies that ghc[*] refers directly to
perl, and ghc-haddock-library to hspec-discover, and that both of those
are native inputs.

These are just the first two examples of packages with native inputs
that I happened to pull out of my haskell.scm. While Haskell does seem
particularly naughty, I've no reason to believe it's unique.

Are these not ‘run-time references’? Is your use of the term narrower
than mine?

> One solution would be to explicitly check build outputs for 
> references to native-inputs, and to force a build failure in that 
> case.

I was surprised to learn this was not already the case (before I started
slowly dragging hissing Haskell packages into the present). I suggest we
don't make any security assumptions about it until it is.

Kind regards,

T G-R

Re: native-inputs ending up as run-time references [was: ISO image available for testing!]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

On Wed, Dec 06, 2017 at 03:16:45AM +0100, Tobias Geerinckx-Rice wrote:
> Hold on. I thought this happened *all the actual time*.
> 
> To me, the output of ‘guix graph’ implies that ghc[*] refers directly to
> perl, and ghc-haddock-library to hspec-discover, and that both of those
> are native inputs.
> 
> These are just the first two examples of packages with native inputs
> that I happened to pull out of my haskell.scm. While Haskell does seem
> particularly naughty, I've no reason to believe it's unique.
> 
> Are these not ‘run-time references’? Is your use of the term narrower
> than mine?

I think of "run-time references" as explicit store references
(file-names) found in the build output by the post-build reference
scanner and recorded in /var/guix/db. These references are what grafting
rewrites.

`guix gc --references` can be used to query the database for a given
store item.

`guix graph`, on the other hand, shows the graphs of package
definitions. That is, the list of inputs, native-inputs,
propagated-inputs, etc. Some of these inputs could be totally
superfluous, and the build output would ideally not contain any
explicit references to them at all.

The term "reference" has a few meanings in Guix, but I think that Mark
is talking about explicit store paths.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: native-inputs ending up as run-time references [was: ISO image available for testing!]

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 1986 bytes --]

Leo,

Leo Famulari wrote on 06/12/17 at 04:18:
> I think of "run-time references" as explicit store references
> (file-names) found in the build output by the post-build reference
> scanner and recorded in /var/guix/db. These references are what grafting
> rewrites.

All right, that was my idea too (though there might be some sloppiness
in my mental model yet).

> `guix gc --references` can be used to query the database for a given
> store item.
>
> `guix graph`, on the other hand, shows the graphs of package
> definitions. That is, the list of inputs, native-inputs,
> propagated-inputs, etc. Some of these inputs could be totally
> superfluous, and the build output would ideally not contain any
> explicit references to them at all.

Thanks for the clarification! I've never actually found a use for ‘guix
graph’ myself, but can't imagine a Guix talk without it. :-)

Unfortunately, I sent you after a red herring: ‘graph’ was a typo.
I did, in fact, mean ‘guix gc’:

  $ guix gc --references `guix build ghc` | grep perl
  /gnu/store/9g4fpfz86vjkx83v5696vm5dzg2sc9qj-perl-5.26.0
  $ grep -r /gnu/store/9g4fpfz86vjkx83v5696vm5dzg2sc9qj-perl-5.26.0 \
    `guix build ghc`
  .../ghc-split:#!/gnu/store/...-perl-5.26.0/bin/perl
  .../settings: ("perl command", "/gnu/store/...-perl-5.26.0/bin/perl"),

And:

  $ guix gc --references `guix build ghc-haddock-library` | grep hspec-d
  /gnu/store/xfh89fmdn2xvd4aw76164zqz0941xw01-hspec-discover-2.2.0
  $ grep -r xfh89fmdn2xvd4aw76164zqz0941xw01-hspec-discover-2.2.0 \
    `guix build ghc-haddock-library`
  [very long paths snipped with great prejudice]
  Binary file /gnu/store/.../package.cache matches
  Binary file /gnu/store/.../libHShaddock-library-1.2.1-...-ghc7.10.2.so
  matches

> The term "reference" has a few meanings in Guix, but I think that Mark
> is talking about explicit store paths.

I'm afraid we're on the same page then.

Kind regards,

T G-R


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

Re: ISO image available for testing!

[Mark H Weaver]

I wrote:
> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.

More precisely, we could force a build failure when any build output
contains a reference to a component (store path) in the following set:

  requisites(native-inputs) - requisites(inputs)

       Mark

Re: ISO image available for testing!

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
>
> [...]
>
>> Long story short: we were flagging native inputs as potential sources of
>> grafts even though, by definition, native inputs are not referred to at
>> run time.
>
> I agree that this *should* never happen, but I see little reason for
> confidence that it never happens in actual fact.
>
> What would happen if a reference to a native-input *was* present in the
> build outputs?  The reason I ask is that, for security reasons, it's
> obviously very important to reliably avoid using ungrafted software at
> run time.
>
> I'm concerned that this recent change could cause minor
> nearly-undetectable packaging mistakes to become major security holes.

Given the examples that Tobias and Ben were quick to find, I’m afraid
you’re right and I was overconfident.  I’m reverting the change.

> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.

We could do that, though I suppose a lot of packages would break.

Thanks to the quick reply,
Ludo’.

Re: ISO image available for testing!

[Tobias Geerinckx-Rice]

Ludovic Courtès wrote on 06/12/17 at 09:14:
>> One solution would be to explicitly check build outputs for references
>> to native-inputs, and to force a build failure in that case.

I started a rudimentary PoC for this last week, after realising how bad
the situation is in ghc-* alone. We'll see what comes of it.

> We could do that, though I suppose a lot of packages would break.

We *should* do it, though. Just not before the pending release.

Kind regards,

T G-R

Re: ISO image available for testing!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2953 bytes --]


Ludovic Courtès writes:

> Hi Chris,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I've attempted to use this to install GuixSD on a Bytemark
>> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
>> running guix system init, but when I did, it started downloading the
>> bootstrap binaries, and then building binutils.
>>
>> When I run guix system build, or guix build with the --dry-run options,
>> it says that it will download, rather than building, but it doesn't.
>
> It turned out to be issues related to grafts and to what Hydra builds,
> fixed with these commits:
>
> 3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable.
> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
> ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls.
> f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages.
>
> The Binutils issue is fixed by f00b85ff8.
>
> Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in
> its dependency graph, via gettext.  Thus, the expat graft was picked up
> as a candidate graft.  However, expat itself was subject to the glibc
> graft, and since there was no substitute for this particular expat, we’d
> have to build it first, just to throw it away later on because coreutils
> does not refer to it at run time.
>
> Long story short: we were flagging native inputs as potential sources of
> grafts even though, by definition, native inputs are not referred to at
> run time.
>
> The last commit ensures that Hydra builds the replacement for
> ‘ghostscript-with-cups’.
>
> What’s tricky is that one doesn’t notice these issues unless starting
> from a fresh store.
>
> I’ve uploaded an updated ISO image, which I used to test substitute
> availability and grafts.  If you have time in the coming hours, feedback
> welcome:

Thanks for fixing this Ludo, and congratulations on the release. I'm
glad to say that I've now managed to install GuixSD using the 0.14.0
x86_64 ISO, however I did encounter some difficulties.

I tried a few times with both the ISO you replied with here, and the
released ISO, but each time the virtual machine I was installing on to
appeared to restart while guix system init was running. It's difficult
to get more information, but the last messages I got out of guix system
init relate to grafting and collisions.

This evening when I tried again I passed --no-grafts to guix system
init, and this time it successfully finished the
installation. Interestingly, this is also what is actually tested by the
iso-image-installer system test, as it sets the GUIX_BUILD_OPTIONS
environment variable.

This isn't conclusive, but I'd be very interested to hear from anyone
that has had similar issues, or successes in using the ISO installer,
both with and without the --no-grafts option.

Thanks again,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: ISO image available for testing!

[Ludovic Courtès]

Hello Chris,

Christopher Baines <mail@cbaines.net> skribis:

> Thanks for fixing this Ludo, and congratulations on the release. I'm
> glad to say that I've now managed to install GuixSD using the 0.14.0
> x86_64 ISO, however I did encounter some difficulties.
>
> I tried a few times with both the ISO you replied with here, and the
> released ISO, but each time the virtual machine I was installing on to
> appeared to restart while guix system init was running. It's difficult
> to get more information, but the last messages I got out of guix system
> init relate to grafting and collisions.

I wonder how the VM can restart.  Could it be an out-of-memory
condition?  Though even that should not lead to a reboot.

Could you see if you can get more details from the VM?

Thanks,
Ludo’.

Re: Release ?

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> I think we do release and then core-updates, before we get bogged down
> in fixing packages in core-updates.

Agreed!

Ludo’.