What’s next?

What’s next?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2421 bytes --]

Hello Guix!

I think it’s time to think about what we want to work on next, ideally
before the next release, and ideally said release should be in a couple
of months rather than in 5 months.  Here are some important items I can
think of:

  • Merge the ncurses installer (the ‘wip-installer’) branch.

  • We’re supposed to freeze ‘core-updates’ in a couple of days and we
    have defined a couple of important goals:
    <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00120.html>.
    I have a vague feeling that the ‘wip-build-systems-gexp’ merge is
    not for this time yet…

  • ‘guix offload’ terrible bug: <https://bugs.gnu.org/26976>.

  • Guile 2.2 compiler terrible issue:
    <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

  • Adjust the release process so we don’t find ourselves without
    substitutes for the ‘guix’ package, as reported at
    <https://bugs.gnu.org/27032>.

  • Prepare to migrate to the new build farm: the hardware of
    bayfront.guixsd.org has been fixed (it was unreliable until we
    changed its CPUs), so now we need to fix a couple of issues in
    Cuirass and generally improve it so we can, via its HTTP interface,
    add new jobsets and so on.  Of course we’ll have to work
    on that incrementally.

  • Finalize the new bootloader API and support for new bootloaders:
    <https://bugs.gnu.org/26339>.

  • Merge the potluck!  <https://bugs.gnu.org/26645>

I think everything above could pretty much go into a 0.13.1 release not
far away.  And the missing bits, IMO, for a “1.0” label would be:

  • Improve the UI: add colors (yay!), hide build logs by default for
    most commands, improve progress reports, display how much will be
    downloaded, etc.

  • Implement channels: <https://bugs.gnu.org/22629>.  A first step may
    be to fix some of the obvious shortcomings of ‘guix pull’: use (guix
    git) to optimize bandwidth usage, and compute the derivation of the
    new ‘guix’ and use that.

  • Authenticate Git checkouts: <https://bugs.gnu.org/22883>.

What do people think?

I think the key idea is that we should fix all the annoyances and bugs,
many of which seasoned Guix hackers more or less got used to, but which
are nevertheless a serious hindrance when using Guix “normally.”

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.  Here are some important items I can
> think of:

[…]

Excellent points!

> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”

Here are some other annoyances:

* the verbosity of reporting hash mismatches.  You posted a neat little
  change for that some time ago, but I cannot find it any more.

* texlive.  I’m working on splitting the package up.  (An aggressive
  firewall at work put a stop to that work, but I’ll try to resume very
  soon.)

* very explicit but cryptic stack traces.  I think that all errors
  should be reported more nicely.  The error can still be printed (on
  demand?), but Guix probably should not spill its innards too readily.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Here are some other annoyances:
>
> * the verbosity of reporting hash mismatches.  You posted a neat little
>   change for that some time ago, but I cannot find it any more.

Oh right, see attached.

> * texlive.  I’m working on splitting the package up.  (An aggressive
>   firewall at work put a stop to that work, but I’ll try to resume very
>   soon.)

Good point!  Looking forward to that.

> * very explicit but cryptic stack traces.  I think that all errors
>   should be reported more nicely.  The error can still be printed (on
>   demand?), but Guix probably should not spill its innards too readily.

I agree with this direction, but we’ll have to work on concrete cases
here.  Bug reports of the form “when I do this I get a stack trace
instead of an error message” are welcome!

Thanks,
Ludo’.

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Here are some other annoyances:
>>
>> * the verbosity of reporting hash mismatches.  You posted a neat little
>>   change for that some time ago, but I cannot find it any more.
>
> Oh right, see attached.

I think you forgot to attach it.

>> * texlive.  I’m working on splitting the package up.  (An aggressive
>>   firewall at work put a stop to that work, but I’ll try to resume very
>>   soon.)
>
> Good point!  Looking forward to that.

Me too!

>> * very explicit but cryptic stack traces.  I think that all errors
>>   should be reported more nicely.  The error can still be printed (on
>>   demand?), but Guix probably should not spill its innards too readily.
>
> I agree with this direction, but we’ll have to work on concrete cases
> here.  Bug reports of the form “when I do this I get a stack trace
> instead of an error message” are welcome!

Danny submitted bug #27100 for this with two examples.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello!
>>
>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>
>>> Here are some other annoyances:
>>>
>>> * the verbosity of reporting hash mismatches.  You posted a neat little
>>>   change for that some time ago, but I cannot find it any more.
>>
>> Oh right, see attached.
>
> I think you forgot to attach it.

Oops.  Here we go:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1380 bytes --]

modified   nix/libstore/build.cc
@@ -2449,8 +2449,11 @@ void DerivationGoal::registerOutputs()
             Hash h2 = recursive ? hashPath(ht, actualPath).first : hashFile(ht, actualPath);
             if (h != h2)
                 throw BuildError(
-                    format("output path `%1%' should have %2% hash `%3%', instead has `%4%'")
-                    % path % i->second.hashAlgo % printHash16or32(h) % printHash16or32(h2));
+                    format("%1% hash mismatch for output path `%2%'\n"
+			   "  expected: %3%\n"
+			   "  actual:   %4%")
+                    % i->second.hashAlgo % path
+		    % printHash16or32(h) % printHash16or32(h2));
         }
 
         /* Get rid of all weird permissions.  This also checks that
@@ -3096,7 +3099,9 @@ void SubstitutionGoal::finished()
             Hash expectedHash = parseHash16or32(hashType, string(expectedHashStr, n + 1));
             Hash actualHash = hashType == htSHA256 ? hash.first : hashPath(hashType, destPath).first;
             if (expectedHash != actualHash)
-                throw SubstError(format("hash mismatch in downloaded path `%1%': expected %2%, got %3%")
+                throw SubstError(format("hash mismatch in downloaded path `%1%'\n"
+					"  expected: %2%\n"
+					"  actual:   %3%")
                     % storePath % printHash(expectedHash) % printHash(actualHash));
         }

[-- Attachment #3: Type: text/plain, Size: 314 bytes --]


Should we apply it?

>> I agree with this direction, but we’ll have to work on concrete cases
>> here.  Bug reports of the form “when I do this I get a stack trace
>> instead of an error message” are welcome!
>
> Danny submitted bug #27100 for this with two examples.

Great.

Thanks,
Ludo’.

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hello!
>>>
>>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>>
>>>> Here are some other annoyances:
>>>>
>>>> * the verbosity of reporting hash mismatches.  You posted a neat little
>>>>   change for that some time ago, but I cannot find it any more.
>>>
>>> Oh right, see attached.
>>
>> I think you forgot to attach it.
>
> Oops.  Here we go:
>
> modified   nix/libstore/build.cc
> @@ -2449,8 +2449,11 @@ void DerivationGoal::registerOutputs()
>              Hash h2 = recursive ? hashPath(ht, actualPath).first : hashFile(ht, actualPath);
>              if (h != h2)
>                  throw BuildError(
> -                    format("output path `%1%' should have %2% hash `%3%', instead has `%4%'")
> -                    % path % i->second.hashAlgo % printHash16or32(h) % printHash16or32(h2));
> +                    format("%1% hash mismatch for output path `%2%'\n"
> +			   "  expected: %3%\n"
> +			   "  actual:   %4%")
> +                    % i->second.hashAlgo % path
> +		    % printHash16or32(h) % printHash16or32(h2));
>          }
>
>          /* Get rid of all weird permissions.  This also checks that
> @@ -3096,7 +3099,9 @@ void SubstitutionGoal::finished()
>              Hash expectedHash = parseHash16or32(hashType, string(expectedHashStr, n + 1));
>              Hash actualHash = hashType == htSHA256 ? hash.first : hashPath(hashType, destPath).first;
>              if (expectedHash != actualHash)
> -                throw SubstError(format("hash mismatch in downloaded path `%1%': expected %2%, got %3%")
> +                throw SubstError(format("hash mismatch in downloaded path `%1%'\n"
> +					"  expected: %2%\n"
> +					"  actual:   %3%")
>                      % storePath % printHash(expectedHash) % printHash(actualHash));
>          }
>
> Should we apply it?

Yes, please.  This looks much better!  Thank you!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Oops.  Here we go:
>>
>> modified   nix/libstore/build.cc
>> @@ -2449,8 +2449,11 @@ void DerivationGoal::registerOutputs()
>>              Hash h2 = recursive ? hashPath(ht, actualPath).first : hashFile(ht, actualPath);
>>              if (h != h2)
>>                  throw BuildError(
>> -                    format("output path `%1%' should have %2% hash `%3%', instead has `%4%'")
>> -                    % path % i->second.hashAlgo % printHash16or32(h) % printHash16or32(h2));
>> +                    format("%1% hash mismatch for output path `%2%'\n"
>> +			   "  expected: %3%\n"
>> +			   "  actual:   %4%")
>> +                    % i->second.hashAlgo % path
>> +		    % printHash16or32(h) % printHash16or32(h2));
>>          }
>>
>>          /* Get rid of all weird permissions.  This also checks that
>> @@ -3096,7 +3099,9 @@ void SubstitutionGoal::finished()
>>              Hash expectedHash = parseHash16or32(hashType, string(expectedHashStr, n + 1));
>>              Hash actualHash = hashType == htSHA256 ? hash.first : hashPath(hashType, destPath).first;
>>              if (expectedHash != actualHash)
>> -                throw SubstError(format("hash mismatch in downloaded path `%1%': expected %2%, got %3%")
>> +                throw SubstError(format("hash mismatch in downloaded path `%1%'\n"
>> +					"  expected: %2%\n"
>> +					"  actual:   %3%")
>>                      % storePath % printHash(expectedHash) % printHash(actualHash));
>>          }
>>
>> Should we apply it?
>
> Yes, please.  This looks much better!  Thank you!

Applied!

Ludo’.

Re: What’s next?

[Brendan Tildesley]

Ludovic Courtès 於 2017-05-24 23:11 寫道:
> Hello Guix!
>
> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.  Here are some important items I can
> think of:
[...]
> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”
>
> Ludo’.

One little annoyance I have is that guix takes every opportunity to
update the list of substitutes when guix build, guix package -u, etc...
is run, and fills my terminal with output like:

substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from
'https://mirror.hydra.gnu.org'... 100.0%
...

It will output this line a different number each time based on some rule
I couldn't guess, and renders the percentage measurement meaningless.
Currently I am running `guix system init config.scm
--substitute-urls="http://10.1.1.61:8080 https://mirror.hydra.gnu.org/"
--fallback' from the 0.13 installer, and it his been outputting hundreds
of these lines for the last half an hour. I'm not sure if it is ever
going to stop, perhaps I've stumbled across a genuine bug? In any case,
the minor annoyance version is something I've noticed for a while.

Re: What’s next?

[Ludovic Courtès]

Hi Brendan,

Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:

> One little annoyance I have is that guix takes every opportunity to
> update the list of substitutes when guix build, guix package -u, etc...
> is run, and fills my terminal with output like:
>
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> substitute: updating list of substitutes from
> 'https://mirror.hydra.gnu.org'... 100.0%
> ...

I agree, it’s actually a huge annoyance.  It relates to grafts and how
they are implemented; I took a stab at fixing this a while back but the
approach turned out to be (mostly?) misguided:

  https://bugs.gnu.org/22990

Would be worth thinking through it again!

Thanks,
Ludo’.

Re: What’s next?

[Maxim Cournoyer]

Hi,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Brendan,
>
> Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:
>
>> One little annoyance I have is that guix takes every opportunity to
>> update the list of substitutes when guix build, guix package -u, etc...
>> is run, and fills my terminal with output like:
>>
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> substitute: updating list of substitutes from
>> 'https://mirror.hydra.gnu.org'... 100.0%
>> ...
>
> I agree, it’s actually a huge annoyance.  It relates to grafts and how
> they are implemented; I took a stab at fixing this a while back but the
> approach turned out to be (mostly?) misguided:
>
>   https://bugs.gnu.org/22990
>
> Would be worth thinking through it again!

Maybe we could make a timer variable configurable, so that at least it
doesn't try to refresh this information at *every* guix command? Default
could be at least one hour. I care about my security, but refreshing
this at every command seems pretty wasteful in terms of resources.

Maxim

Re: What’s next?

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Hi,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hi Brendan,
>>
>> Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:
>>
>>> One little annoyance I have is that guix takes every opportunity to
>>> update the list of substitutes when guix build, guix package -u, etc...
>>> is run, and fills my terminal with output like:
>>>
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> substitute: updating list of substitutes from
>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>> ...
>>
>> I agree, it’s actually a huge annoyance.  It relates to grafts and how
>> they are implemented; I took a stab at fixing this a while back but the
>> approach turned out to be (mostly?) misguided:
>>
>>   https://bugs.gnu.org/22990
>>
>> Would be worth thinking through it again!
>
> Maybe we could make a timer variable configurable, so that at least it
> doesn't try to refresh this information at *every* guix command?

That’s not what’s happening.  When you see repeated lines that go
directly to 100%, that’s because it’s fetching metadata for just a
single package, and does that one by one.

In other cases, it fetches as many as possible at once, and uses HTTP
pipelining to send all these requests to hydra.gnu.org.

Ludo’.

Re: What’s next?

[Maxim Cournoyer]

Hello Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Hi,
>>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Hi Brendan,
>>>
>>> Brendan Tildesley <brendan.tildesley@openmailbox.org> skribis:
>>>
>>>> One little annoyance I have is that guix takes every opportunity to
>>>> update the list of substitutes when guix build, guix package -u, etc...
>>>> is run, and fills my terminal with output like:
>>>>
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> substitute: updating list of substitutes from
>>>> 'https://mirror.hydra.gnu.org'... 100.0%
>>>> ...
>>>
>>> I agree, it’s actually a huge annoyance.  It relates to grafts and how
>>> they are implemented; I took a stab at fixing this a while back but the
>>> approach turned out to be (mostly?) misguided:
>>>
>>>   https://bugs.gnu.org/22990
>>>
>>> Would be worth thinking through it again!
>>
>> Maybe we could make a timer variable configurable, so that at least it
>> doesn't try to refresh this information at *every* guix command?
>
> That’s not what’s happening.  When you see repeated lines that go
> directly to 100%, that’s because it’s fetching metadata for just a
> single package, and does that one by one.
>
> In other cases, it fetches as many as possible at once, and uses HTTP
> pipelining to send all these requests to hydra.gnu.org.

My observation was that for the same exact command (I just tried: "guix
environment emacs-el-mock") entered twice:

--8<---------------cut here---------------start------------->8---
$ guix environment emacs-el-mock
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'...   0.0
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'...  33.3
[...] # (downloads a bunch of substitute)
[env]$ exit
$ guix environment emacs-el-mock
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
[env]$
--8<---------------cut here---------------end--------------->8---

We can see that a connection is made to the substitute server(s) and
data is retrieved each time the command is entered. Why does it need to
refresh the list of substitutes the second time? Everything needed is
already in the store.

Maxim

Re: What’s next?

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

2017-05-24 15:11 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:

> Hello Guix!
>
> I think it’s time to think about what we want to work on next, ideally
>

one little ting I'd love is Janneke's and Jlicht's solution to use "binary"
nodejs packages

That would allow me to play with some projects that depend on nodejs

The problem of evaluating the adherence of the nodejs collection to the
Guix diistribution guidelines could be faced later ( I posted a thread that
I think is relevant in this regard)



>   • ‘guix offload’ terrible bug: <https://bugs.gnu.org/26976>.
>
>   • Guile 2.2 compiler terrible issue:
>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html
> >.
>

terrible bugs are terrible bugs. They should be regarded as priorities


>   • Improve the UI: add colors (yay!), hide build logs by default for
>     most commands, improve progress reports, display how much will be
>     downloaded, etc.
>

This might be regarded as a small thing but to me this is very important
I use to watch Fedora's colorful terminals with longing


> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”
>

yeah

[-- Attachment #2: Type: text/html, Size: 2401 bytes --]

Re: What’s next?

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.  Here are some important items I can
> think of:

Great points!

>  • Implement channels: <https://bugs.gnu.org/22629>.  A first step may
>    be to fix some of the obvious shortcomings of ‘guix pull’: use (guix
>    git) to optimize bandwidth usage, and compute the derivation of the
>    new ‘guix’ and use that.
>
> I think the key idea is that we should fix all the annoyances and bugs,
> many of which seasoned Guix hackers more or less got used to, but which
> are nevertheless a serious hindrance when using Guix “normally.”

+1

A friend of mine is having a second look at Guix (not SD yet) and one of
the most confusing things initially is `guix pull'.  "When/how do I use
that," he asks...and I can only say: I'm not using that...I think we
want this to work--or something like this, we talked about this at
FOSDEM, but AFAIK everyone is using Guix with Git.

He responds with: then *why* is it in the manual.  I have no answer.
Possibly I'm wrong and/or my information is outdated?

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: What’s next?

[Adonay Felipe Nogueira]

As an average Guix user, I use `guix pull`.

In fact, I avoid `git pull and the pre-inst-env stuff`. I prefer to work
with GUIX_PACKAGE_PATH in order to work only with `guix package` and
`guix build` if I do have the need to install or build something that I
changed.

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: What’s next?

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

2017-05-24 18:25 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:

>
>
> +1
>
> A friend of mine is having a second look at Guix (not SD yet) and one of
> the most confusing things initially is `guix pull'.  "When/how do I use
> that," he asks...and I can only say: I'm not using that...I think we
> want this to work--or something like this, we talked about this at
> FOSDEM, but AFAIK everyone is using Guix with Git.
>
> He responds with: then *why* is it in the manual.  I have no answer.
> Possibly I'm wrong and/or my information is outdated?
>

This is an important point for me too

I realized that everyone is using git and not guix pull just yesterday

I have been using guix pull until today because I didn' t know any better

The manual mentions something that only beginners use and doesn' t mention
something that the majority of the community is using.

Probably in many of the footages in which Ludo appears, the use of git over
guix pull is assumed.

I think this is a problem. It' s unfair to newcomers and it damages Guix as
a project because it makes the learning curve steeper with not so much of a
point why.

I kinda agree with myglc2, on this.

[-- Attachment #2: Type: text/html, Size: 1702 bytes --]

Re: What’s next?

[Ricardo Wurmus]

Catonano <catonano@gmail.com> writes:

> 2017-05-24 18:25 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:
[…]
>> A friend of mine is having a second look at Guix (not SD yet) and one of
>> the most confusing things initially is `guix pull'.  "When/how do I use
>> that," he asks...and I can only say: I'm not using that...I think we
>> want this to work--or something like this, we talked about this at
>> FOSDEM, but AFAIK everyone is using Guix with Git.
>>
>> He responds with: then *why* is it in the manual.  I have no answer.
>> Possibly I'm wrong and/or my information is outdated?
>>
>
> This is an important point for me too
>
> I realized that everyone is using git and not guix pull just yesterday
[…]
> I think this is a problem. It' s unfair to newcomers and it damages Guix as
> a project because it makes the learning curve steeper with not so much of a
> point why.

There are two reasons why developers use Guix from git:

* it allows them to add new packages and features to Guix itself.  This
  is something “guix pull” doesn’t support well.

* it doesn’t require compiling all of Guix on each update.

The latter is a defect in “guix pull” that we are aware of and working
on.  Progress has already been made in improving “guix pull”, but it
takes time.  It is not by design that “guix pull” is currently less
desirable than working from a git checkout.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[myglc2]

On 05/24/2017 at 21:56 Ricardo Wurmus writes:

> Catonano <catonano@gmail.com> writes:
>
>> 2017-05-24 18:25 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:
> […]
>>> A friend of mine is having a second look at Guix (not SD yet) and one of
>>> the most confusing things initially is `guix pull'.  "When/how do I use
>>> that," he asks...and I can only say: I'm not using that...I think we
>>> want this to work--or something like this, we talked about this at
>>> FOSDEM, but AFAIK everyone is using Guix with Git.
>>>
>>> He responds with: then *why* is it in the manual.  I have no answer.
>>> Possibly I'm wrong and/or my information is outdated?
>>>
>>
>> This is an important point for me too
>>
>> I realized that everyone is using git and not guix pull just yesterday
> […]
>> I think this is a problem. It' s unfair to newcomers and it damages Guix as
>> a project because it makes the learning curve steeper with not so much of a
>> point why.
>
> There are two reasons why developers use Guix from git:
>
> * it allows them to add new packages and features to Guix itself.  This
>   is something “guix pull” doesn’t support well.
>
> * it doesn’t require compiling all of Guix on each update.

Hmm, that explains how it feels. Why doesn't it just pull the compiled guix files?

Re: What’s next?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

On Wed, May 24, 2017 at 09:34:14PM +0200, Catonano wrote:
> I think this is a problem. It' s unfair to newcomers and it damages Guix as
> a project because it makes the learning curve steeper with not so much of a
> point why.

It's true, we collectively worked around some problems with `guix pull`
for too long. Let's all eat the dog food when we don't actually need to
work from Git. I promise it tastes okay :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What’s next?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Wed, May 24, 2017 at 06:25:40PM +0200, Jan Nieuwenhuizen wrote:
> A friend of mine is having a second look at Guix (not SD yet) and one of
> the most confusing things initially is `guix pull'.  "When/how do I use
> that," he asks...and I can only say: I'm not using that...I think we
> want this to work--or something like this, we talked about this at
> FOSDEM, but AFAIK everyone is using Guix with Git.

`guix pull` is one of the primary tools of Guix. For those who are new
to Guix, it should be described as a per-user `apt-get update`. That is,
it updates the list of available packages. The finer differences and
extra features are not important for new users to learn at the
beginning.

With the recent commit adding '--fallback' to `guix pull` [0], the main
reason for Guix users who are not Guix developers to resort to Git has
been removed.

So, I use and recommend `guix pull`!

Do you think the manual can be more clear about this? I'd really like to
hear which parts of the manual your friend read. Maybe we need to
rearrange or rewrite some sections.

I think the most immediate problem with `guix pull` is that it doesn't
support Git commit signature verification. So, you end up trusting
different things: basically, a subset of the X.509 PKI vs PGP+SHA1 [1,2].
I think we can fix this while making `guix pull` use (guix git).

Building Guix from Git is the normal way to develop Guix, and it avoids
downloading a Guix tarball from Savannah in the default case, so
developers will learn and use it, but it brings its own pitfalls.

[0]
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4902d3c4e0376974356481f222583580b49f39e1

[1] `guix pull` verifies the certificate of <git.savannah.gnu.org>
against the Let's Encrypt trust chain *only*.

[2] If I understand correctly, Git commit signatures are of the SHA1
hash, not the actual commit data. So... not great if I'm correct, but it
will get better as Git introduces a new hash function. And SHA1
collisions are rather obvious to detect, at least according the public
research. An attempt at collision detection was added in Git 2.13.0.

> He responds with: then *why* is it in the manual.  I have no answer.
> Possibly I'm wrong and/or my information is outdated?

Since we are all Guix developers, we talk about developing Guix, but not
as much the day-to-day use. So our impressions may not match actual
usage patterns.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What???s next?

[Pjotr Prins]

On Wed, May 24, 2017 at 05:45:39PM -0400, Leo Famulari wrote:
> [1] `guix pull` verifies the certificate of <git.savannah.gnu.org>
> against the Let's Encrypt trust chain *only*.

This brings up another annoyance. Before a first 'git pull' as a
newbie you have to go through a number of steps which are, arguably,
redundant.

I am talking about installing a first key to trust the guix server.
Well, if we have installed guix AND we use guix pull, I think we can
assume the guix server is trusted (by the user). Therefore, that key
should work out of the box (it is what people install from the tree
anyway!). It is a redundant step. Debian also uses keys and works
out of the box.

The other thing is permissions. Sometimes the user profile needs
explicit permission settings. This is not right. I can see it is
useful on a server setup controlled by an administrator, but arguably
it should just work. The administrator can revert on that. So, if
possible, the default should be allowing guix to work once the daemon
runs.

Pj.

Re: What???s next?

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Wed, May 24, 2017 at 05:45:39PM -0400, Leo Famulari wrote:
>> [1] `guix pull` verifies the certificate of <git.savannah.gnu.org>
>> against the Let's Encrypt trust chain *only*.
>
> This brings up another annoyance. Before a first 'git pull' as a
> newbie you have to go through a number of steps which are, arguably,
> redundant.

Note that the Let’s Encrypt certificate check by ‘guix pull’ works out
of the box: users don’t need to install ‘nss-certs’, define a bunch of
environment variables, etc.

> I am talking about installing a first key to trust the guix server.
> Well, if we have installed guix AND we use guix pull, I think we can
> assume the guix server is trusted (by the user). Therefore, that key
> should work out of the box (it is what people install from the tree
> anyway!). It is a redundant step. Debian also uses keys and works
> out of the box.

Substitute servers are fundamentally different from servers that provide
Guix packages, which is why it’s treated differently.

On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
registered by default.  We cannot do that for someone installing Guix on
a foreign distro because that involves creating a file in /etc.

> The other thing is permissions. Sometimes the user profile needs
> explicit permission settings.

What do you mean?

Thanks,
Ludo’.

Re: What's next?

[Pjotr Prins]

On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
> registered by default.  We cannot do that for someone installing Guix on
> a foreign distro because that involves creating a file in /etc.

Many installs are not on GuixSD. Can't we use the key that is stored
in the store itself? If /etc does not exist then use what comes
with the installation.

> > The other thing is permissions. Sometimes the user profile needs
> > explicit permission settings.
> 
> What do you mean?

I encountered that /var/guix/profiles/ permissions need to be set for
every user. But maybe that has changed, or I got it wrong.

Pj.

Re: What's next?

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>> registered by default.  We cannot do that for someone installing Guix on
>> a foreign distro because that involves creating a file in /etc.
>
> Many installs are not on GuixSD. Can't we use the key that is stored
> in the store itself? If /etc does not exist then use what comes
> with the installation.

The current behavior is to print a warning when /etc/guix/acl (the list
of authorized keys) is empty or nonexistent.

Your suggestion would be to automatically populate it, right?

I’m mildly reluctant to that, because we’d stealthily force every user
into trusting our substitute servers.  OTOH I agree that the current
situation is not optimal.

What do people think?

Ludo’.

Re: What's next?

[Roel Janssen]

Ludovic Courtès writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>>> registered by default.  We cannot do that for someone installing Guix on
>>> a foreign distro because that involves creating a file in /etc.
>>
>> Many installs are not on GuixSD. Can't we use the key that is stored
>> in the store itself? If /etc does not exist then use what comes
>> with the installation.
>
> The current behavior is to print a warning when /etc/guix/acl (the list
> of authorized keys) is empty or nonexistent.
>
> Your suggestion would be to automatically populate it, right?
>
> I’m mildly reluctant to that, because we’d stealthily force every user
> into trusting our substitute servers.  OTOH I agree that the current
> situation is not optimal.
>
> What do people think?

Maybe we could find a mid-way here by doing the same as Fedora does with
RPMfusion repositories: It asks the user for trusting the signing keys
before enabling the repository.

So in our case it would be something like:
$ guix package -i emacs
A `substitute' is available for this package on
https://mirror.hydra.gnu.org.  This means we can download the binary
output for this package, instead of compiling it from its source code.
Do you want to use this substitute server with key ... for this package,
and for future packages? [y/N]

We need to find the proper wording for this message.  Using this, we can
still let the user decide, but we can make it a lot easier for the user
to make a decision -- a 'yes' or 'no' answer to a question is easier
than a paragraph in the manual with instructions to enable it.

Kind regards,
Roel Janssen

Re: What's next?

[Ludovic Courtès]

Roel Janssen <roel@gnu.org> skribis:

> Ludovic Courtès writes:
>
>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>>
>>> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>>>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>>>> registered by default.  We cannot do that for someone installing Guix on
>>>> a foreign distro because that involves creating a file in /etc.
>>>
>>> Many installs are not on GuixSD. Can't we use the key that is stored
>>> in the store itself? If /etc does not exist then use what comes
>>> with the installation.
>>
>> The current behavior is to print a warning when /etc/guix/acl (the list
>> of authorized keys) is empty or nonexistent.
>>
>> Your suggestion would be to automatically populate it, right?
>>
>> I’m mildly reluctant to that, because we’d stealthily force every user
>> into trusting our substitute servers.  OTOH I agree that the current
>> situation is not optimal.
>>
>> What do people think?
>
> Maybe we could find a mid-way here by doing the same as Fedora does with
> RPMfusion repositories: It asks the user for trusting the signing keys
> before enabling the repository.
>
> So in our case it would be something like:
> $ guix package -i emacs
> A `substitute' is available for this package on
> https://mirror.hydra.gnu.org.  This means we can download the binary
> output for this package, instead of compiling it from its source code.
> Do you want to use this substitute server with key ... for this package,
> and for future packages? [y/N]

It cannot work this way because the decision has to be made by the
sysadmin, not by unprivileged users.

Also, I like that ‘guix package’ is non-interactive.

Ludo’.

Re: What's next?

[Pjotr Prins]

On Tue, May 30, 2017 at 05:19:09PM +0200, Ludovic Courtes wrote:
> > A `substitute' is available for this package on
> > https://mirror.hydra.gnu.org.  This means we can download the binary
> > output for this package, instead of compiling it from its source code.
> > Do you want to use this substitute server with key ... for this package,
> > and for future packages? [y/N]
> 
> It cannot work this way because the decision has to be made by the
> sysadmin, not by unprivileged users.

The first time guix is run it is by a system administrator (almost
100% certain), typically from a binary installation.

> Also, I like that 'guix package' is non-interactive.

Sure. But it is a step that is not required on GuixSD. The key comes
with the base install of the guix package. 

Can't we just fetch the key from the store? Why does it have to be
/etc? First look in /etc. If that does not exist, fetch from the guix
package itself. It is there. /etc overrides the key in the store.

Now it is a needless manual step. I say, it is an easy thing to
automate. Automate if you can.

Maybe not the highest priority, but if we come with a solution it
should be easy to do.

Pj.

-- 

Re: What's next?

[Maxim Cournoyer]

ludo@gnu.org (Ludovic Courtès) writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> On Sat, May 27, 2017 at 12:16:45PM +0200, Ludovic Court??s wrote:
>>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>>> registered by default.  We cannot do that for someone installing Guix on
>>> a foreign distro because that involves creating a file in /etc.
>>
>> Many installs are not on GuixSD. Can't we use the key that is stored
>> in the store itself? If /etc does not exist then use what comes
>> with the installation.
>
> The current behavior is to print a warning when /etc/guix/acl (the list
> of authorized keys) is empty or nonexistent.
>
> Your suggestion would be to automatically populate it, right?
>
> I’m mildly reluctant to that, because we’d stealthily force every user
> into trusting our substitute servers.  OTOH I agree that the current
> situation is not optimal.
>

Maybe there could be a prompt that tells the user the current message
(no keys in /etc/guix/acl) and then asks them if they'd like to register
the default Guix substitute server keys? That'd be a middle ground
solution.

Maxim

Re: What???s next?

[Maxim Cournoyer]

Hello,

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
> registered by default.  We cannot do that for someone installing Guix on
> a foreign distro because that involves creating a file in /etc.

The bayfront key is now registered by default? Does it mean that
bayfront is ready to be part of the `%default-substitute-urls'? It
doesn't seem to be the case currently:

--8<---------------cut here---------------start------------->8---
(define %default-substitute-urls
  ;; Default list of substituters.  This is *not* the list baked in
  ;; 'guix-daemon', but it is used by 'guix-service-type' and and a couple of
  ;; clients ('guix build --log-file' uses it.)
  (map (if (false-if-exception (resolve-interface '(gnutls)))
           (cut string-append "https://" <>)
           (cut string-append "http://" <>))
       '("mirror.hydra.gnu.org")))
--8<---------------cut here---------------end--------------->8---

Maxim

Re: What???s next?

[Ricardo Wurmus]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hello,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>> registered by default.  We cannot do that for someone installing Guix on
>> a foreign distro because that involves creating a file in /etc.
>
> The bayfront key is now registered by default? Does it mean that
> bayfront is ready to be part of the `%default-substitute-urls'? It
> doesn't seem to be the case currently:

No, but the mirror may soon be distributing substitutes that have been
created on and signed by bayfront.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What???s next?

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> On GuixSD, the key of hydra.gnu.org and bayfront.guixsd.org are always
>> registered by default.  We cannot do that for someone installing Guix on
>> a foreign distro because that involves creating a file in /etc.
>
> The bayfront key is now registered by default?

Yes!

> Does it mean that bayfront is ready to be part of the
> `%default-substitute-urls'?

Not yet!

Currently it only builds for x86_64/i686, it doesn’t offload anywhere,
etc.  But the fact that it’s registered means that we should eventually
be able to transition transparently (for GuixSD users at least).

Ludo’.

Re: What’s next?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Leo Famulari <leo@famulari.name> writes:

> So, I use and recommend `guix pull`!

I use it too.  Statements by others in this thread that "nobody" uses it
or that "everyone" is using Git are mistaken.

I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
IMO, the biggest problem with 'guix pull' is that there is no easy
rollback.  I can live with long execution times (--fallback is fine, but
it'd be nice if substitutes were available more often), and I can live
with 'guix pull' causing me to get a version of guix that's broken
somehow, but the inability to easily roll back when things go south
makes me hesitant to run 'guix pull' regularly.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: What’s next?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

On Thu, May 25, 2017 at 07:57:41AM -0700, Chris Marusich wrote:
> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
> IMO, the biggest problem with 'guix pull' is that there is no easy
> rollback.  I can live with long execution times (--fallback is fine, but
> it'd be nice if substitutes were available more often), and I can live
> with 'guix pull' causing me to get a version of guix that's broken
> somehow, but the inability to easily roll back when things go south
> makes me hesitant to run 'guix pull' regularly.

It's not "easy rollback", but you can at least use it to deploy any Guix
commit, from any source, with the '--url=' argument.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What’s next?

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

> Leo Famulari <leo@famulari.name> writes:
>
>> So, I use and recommend `guix pull`!
>
> I use it too.  Statements by others in this thread that "nobody" uses it
> or that "everyone" is using Git are mistaken.
>
> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
> IMO, the biggest problem with 'guix pull' is that there is no easy
> rollback.  I can live with long execution times (--fallback is fine, but
> it'd be nice if substitutes were available more often), and I can live
> with 'guix pull' causing me to get a version of guix that's broken
> somehow, but the inability to easily roll back when things go south
> makes me hesitant to run 'guix pull' regularly.

I believe this can be fixed by adding more links to “.config/guix”,
i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
and then link from there to “latest”.  On update it would create a new
link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.

(“latest” should be renamed to “current” to better match the intent in
this case.)

Timestamped names like this are ugly, but that’s what’s at the top of my
head.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Adonay Felipe Nogueira]

I think we can try either using:

- ISO dates: [Full year number]-[Two digit month number]-[Two digit day
  number]T[Two digit hour in 24-hour clock]:[Two digit minutes]:[Two digit
  seconds]

- A numeric sufix: Like "latest.1" for the first generation, "latest.2"
  for the second, and so on, and "latest" for the numbered backup being
  considered as current. Thus, the date and time information would be
  kept in the file metadata, not in the name, specially considering that
  names with ":" might be problematic in some file systems.

I of course don't know, from a developer point of view, what is the best
option, but I thought about giving my "personal" two cents on the
matter. :)

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: What’s next?

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> Leo Famulari <leo@famulari.name> writes:
>>
>>> So, I use and recommend `guix pull`!
>>
>> I use it too.  Statements by others in this thread that "nobody" uses it
>> or that "everyone" is using Git are mistaken.
>>
>> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
>> IMO, the biggest problem with 'guix pull' is that there is no easy
>> rollback.  I can live with long execution times (--fallback is fine, but
>> it'd be nice if substitutes were available more often), and I can live
>> with 'guix pull' causing me to get a version of guix that's broken
>> somehow, but the inability to easily roll back when things go south
>> makes me hesitant to run 'guix pull' regularly.
>
> I believe this can be fixed by adding more links to “.config/guix”,
> i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
> and then link from there to “latest”.  On update it would create a new
> link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
> be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.

There would be some similarity with profiles.  Should we simply use
profiles, and effectively turn ~/.config/guix/latest into a profile,
with generations etc.?

Food for thought…  :-)

Ludo’.

Re: What’s next?

[myglc2]

On 05/27/2017 at 12:13 Ludovic Courtès writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>> Leo Famulari <leo@famulari.name> writes:
>>>
>>>> So, I use and recommend `guix pull`!
>>>
>>> I use it too.  Statements by others in this thread that "nobody" uses it
>>> or that "everyone" is using Git are mistaken.
>>>
>>> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
>>> IMO, the biggest problem with 'guix pull' is that there is no easy
>>> rollback.  I can live with long execution times (--fallback is fine, but
>>> it'd be nice if substitutes were available more often), and I can live
>>> with 'guix pull' causing me to get a version of guix that's broken
>>> somehow, but the inability to easily roll back when things go south
>>> makes me hesitant to run 'guix pull' regularly.
>>
>> I believe this can be fixed by adding more links to “.config/guix”,
>> i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
>> and then link from there to “latest”.  On update it would create a new
>> link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
>> be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.
>
> There would be some similarity with profiles.  Should we simply use
> profiles, and effectively turn ~/.config/guix/latest into a profile,
> with generations etc.?

+1

Re: What’s next?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>> Leo Famulari <leo@famulari.name> writes:
>>>
>>>> So, I use and recommend `guix pull`!
>>>
>>> I use it too.  Statements by others in this thread that "nobody" uses it
>>> or that "everyone" is using Git are mistaken.
>>>
>>> I use Git when I want to hack on Guix.  Otherwise, I use 'guix pull'.
>>> IMO, the biggest problem with 'guix pull' is that there is no easy
>>> rollback.  I can live with long execution times (--fallback is fine, but
>>> it'd be nice if substitutes were available more often), and I can live
>>> with 'guix pull' causing me to get a version of guix that's broken
>>> somehow, but the inability to easily roll back when things go south
>>> makes me hesitant to run 'guix pull' regularly.
>>
>> I believe this can be fixed by adding more links to “.config/guix”,
>> i.e. before creating “latest” it would create “2017-05-24:08:21:01.123”
>> and then link from there to “latest”.  On update it would create a new
>> link “2017-05-25:17:45:45.123” and link that to latest.  Roll back would
>> be a matter of pointing “2017-05-24:08:21:01.123” to “latest”.
>
> There would be some similarity with profiles.  Should we simply use
> profiles, and effectively turn ~/.config/guix/latest into a profile,
> with generations etc.?

That’s not a bad idea!  It sure beats messing with a single link and it
makes it possible to more easily manage different versions of Guix.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What’s next?

[Ludovic Courtès]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> A friend of mine is having a second look at Guix (not SD yet) and one of
> the most confusing things initially is `guix pull'.  "When/how do I use
> that," he asks...and I can only say: I'm not using that...I think we
> want this to work--or something like this, we talked about this at
> FOSDEM, but AFAIK everyone is using Guix with Git.
>
> He responds with: then *why* is it in the manual.  I have no answer.
> Possibly I'm wrong and/or my information is outdated?

I think there’s a lot of (well deserved, as far as its implementation
goes) badmouthing against ‘guix pull’ on #guix ;-), but nevertheless, I
think it’s a useful tool for users who don’t want to manage their own
Guix checkout etc. by themselves.

The manual does mention in several places what it’s for, I think.

Ludo’.

Release!

[Ludovic Courtès]

Hello Guix!

ludo@gnu.org (Ludovic Courtès) skribis:

> I think it’s time to think about what we want to work on next, ideally
> before the next release, and ideally said release should be in a couple
> of months rather than in 5 months.

Oops, that was 5 months ago…  :-)

> Here are some important items I can think of:
>
>   • Merge the ncurses installer (the ‘wip-installer’) branch.
>
>   • We’re supposed to freeze ‘core-updates’ in a couple of days and we
>     have defined a couple of important goals:
>     <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00120.html>.
>     I have a vague feeling that the ‘wip-build-systems-gexp’ merge is
>     not for this time yet…
>
>   • ‘guix offload’ terrible bug: <https://bugs.gnu.org/26976>.
>
>   • Guile 2.2 compiler terrible issue:
>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>
>   • Adjust the release process so we don’t find ourselves without
>     substitutes for the ‘guix’ package, as reported at
>     <https://bugs.gnu.org/27032>.
>
>   • Prepare to migrate to the new build farm: the hardware of
>     bayfront.guixsd.org has been fixed (it was unreliable until we
>     changed its CPUs), so now we need to fix a couple of issues in
>     Cuirass and generally improve it so we can, via its HTTP interface,
>     add new jobsets and so on.  Of course we’ll have to work
>     on that incrementally.
>
>   • Finalize the new bootloader API and support for new bootloaders:
>     <https://bugs.gnu.org/26339>.
>
>   • Merge the potluck!  <https://bugs.gnu.org/26645>

Good news is that we made progress on some of these issues, but not all;
we also made progress on other things which are really cool, such as the
ISO installer.

Regardless, I think we should be planning for a release within a couple
of weeks.  An important question to me is ‘wip-installer’: what do we
do, Danny?  John?

I would have loved to have a fix to at least mitigate Guile’s compiler
resource consumption issue.  I made some progress at
<https://bugs.gnu.org/28590> notably, but it’s not over yet.

In the meantime, it would be great if people could run an installation
from an ISO image and report bugs and glitches:

  https://www.gnu.org/software/guix/manual/html_node/Building-the-Installation-Image.html

Thoughts?

Ludo’.

Re: Release!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

On Wed, 04 Oct 2017 17:12:36 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> In the meantime, it would be great if people could run an installation
> from an ISO image and report bugs and glitches:
> 
>   https://www.gnu.org/software/guix/manual/html_node/Building-the-Installation-Image.html
> 
> Thoughts?

Unfortunately, building the ISO image has broken since I was
successfully using it [1].

One other issue I was thinking of recently was the size of the image. I
checked an ISO of the installation system I generated a while ago, and
it's 1.1GB in size. A actual CD can only hold 0.7GB of data. This might
be something that just needs documenting, saying that this is for a
DVD, and not a CD, or, it the size could be reduced so that it fits on
a CD.

When I build the installation system, and run guix size, it says 1031.0
MiB, so I guess looking at the breakdown there and trying to reduce it
would be a way to start, if this is a worthwhile objective.

I don't have any use for a physical CD installation image, so if anyone
is interested in having this, now is the time to say so?


1: guix system disk-image --file-system-type=iso9660
gnu/system/install.scm

error: changing mode of
`/tmp/root//gnu/store/jjna4ivyidxfq40mq78g97yzg0wfzcqy-shadow-4.5/bin/su'
to 100555: Operation not permitted ERROR: In procedure scm-error:
ERROR: failed to register store items "/xchg/system"

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Release!

[Ludovic Courtès]

Heya!

Christopher Baines <mail@cbaines.net> skribis:

> Unfortunately, building the ISO image has broken since I was
> successfully using it [1].

Oops, weird.  Let’s investigate.

> One other issue I was thinking of recently was the size of the image. I
> checked an ISO of the installation system I generated a while ago, and
> it's 1.1GB in size. A actual CD can only hold 0.7GB of data. This might
> be something that just needs documenting, saying that this is for a
> DVD, and not a CD, or, it the size could be reduced so that it fits on
> a CD.
>
> When I build the installation system, and run guix size, it says 1031.0
> MiB, so I guess looking at the breakdown there and trying to reduce it
> would be a way to start, if this is a worthwhile objective.

We can do both.  :-)

I think reducing the size of package closures in general is
worthwhile—there’s room for progress.

Yet, we probably won’t make it fit in 650 MiB this easily, so we should
document this.

Thanks for your feedback,
Ludo’.

> 1: guix system disk-image --file-system-type=iso9660
> gnu/system/install.scm
>
> error: changing mode of
> `/tmp/root//gnu/store/jjna4ivyidxfq40mq78g97yzg0wfzcqy-shadow-4.5/bin/su'
> to 100555: Operation not permitted ERROR: In procedure scm-error:
> ERROR: failed to register store items "/xchg/system"

Re: Release!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

On Fri, 06 Oct 2017 15:01:42 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Heya!
> 
> Christopher Baines <mail@cbaines.net> skribis:
> 
> > Unfortunately, building the ISO image has broken since I was
> > successfully using it [1].  
> 
> Oops, weird.  Let’s investigate.

I'm very glad that now I've removed the setuid binaries from the store,
the building of the ISO image, and the associated iso-image-installer
system test are now working for me :)

Thanks for investigating and fixing this Ludo :D

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Release!

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> On Fri, 06 Oct 2017 15:01:42 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Heya!
>> 
>> Christopher Baines <mail@cbaines.net> skribis:
>> 
>> > Unfortunately, building the ISO image has broken since I was
>> > successfully using it [1].  
>> 
>> Oops, weird.  Let’s investigate.
>
> I'm very glad that now I've removed the setuid binaries from the store,
> the building of the ISO image, and the associated iso-image-installer
> system test are now working for me :)

Awesome, thanks for checking!

I really didn’t expect such a bug to crop up…

Ludo’.

Re: Release!

[Ricardo Wurmus]

Hi Ludo,

>> Here are some important items I can think of:
[…]
>>   • Guile 2.2 compiler terrible issue:
>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

One way to side-step this issue for the upcoming release is to use one
Guile process per file when running “guix pull”.  This will make it run
a lot slower, but that would be better than the current situation.

Maybe we could make parallel compilation within the same Guile process
an option, so that it won’t be needlessly slow on machines within the
RAM goldilocks zone.

I’ve reverted f07041f7d25badb7d74b8fad6ee446a12af04f63 locally on my
i686 netbook with 1GB RAM and tested it with “guix pull
--url=/path/to/guix”.  This seems to work — it’s still running… :)

One annoyance is that after compiling one file it prints this message:

   Some deprecated features have been used.  Set the environment
   variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
   program to get more information.  Set it to "no" to suppress
   this message.

I suppose we should suppress this for “guix pull”.

Do you want to make this behaviour optional, e.g. through a command line
switch like “--sloth”?  Or should this be the default until the problem
in Guile/libgc is fixed?

>>   • Prepare to migrate to the new build farm: the hardware of
>>     bayfront.guixsd.org has been fixed (it was unreliable until we
>>     changed its CPUs), so now we need to fix a couple of issues in
>>     Cuirass and generally improve it so we can, via its HTTP interface,
>>     add new jobsets and so on.  Of course we’ll have to work
>>     on that incrementally.

I guess we’ll be using berlin.guixsd.org instead of bayfront, no?  I’m
currently installing additional servers (we’re now at 13 build servers,
I’ll get this up to 18 this week).

>>   • Merge the potluck!  <https://bugs.gnu.org/26645>

About that… We now have a JSON importer, so maybe it’s worth using the
even simpler JSON package format instead of the simplified S-expression
format that Andy proposed.  What do you think?  Should we discuss this
at <https://bugs.gnu.org/26645>?

Who would like to help us squash some bugs before the release?  Let’s
try to fix at least 5 more bugs!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Hi Recardo,
Hi Ludo,

> >>   • Merge the potluck!  <https://bugs.gnu.org/26645>  

> About that… We now have a JSON importer, so maybe it’s worth using the
> even simpler JSON package format instead of the simplified S-expression
> format that Andy proposed.  What do you think?  Should we discuss this
> at <https://bugs.gnu.org/26645>?

FWIW, I much prefer s-exp, and the generated file is a scheme file right? I very
much doubt guix and potluck package developers can't easily read (and write :))
s-exp, so why would we 'abandon' s-exp, what would we win here? These files will
never be processed by anything else but guix and/or potluck, and the 'package
developers' do all know scheme perfectly well...

my 2c :)
Thanks for the fantastic work, both Guix and potluck!

David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Release!

[Hartmut Goebel]

Am 07.10.2017 um 01:31 schrieb David Pirotte:
> so why would we 'abandon' s-exp, what would we win here?

It might be interesting to *create* these files using tools written in
other programming languages. And modules for creating *JSON* are
available for most programming languages.

(OTOH TOML[]1] could be a better format than JSON – it's much like
.ini-files, but more formal specification. A scheme implementation is
available [2].)

[1] https://en.wikipedia.org/wiki/TOML
[2] 0f52bab3e3cfeb261110f9217c39eb03e78d026a

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Release!

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

Hello Hartmut,

> > so why would we 'abandon' s-exp, what would we win here?  

> It might be interesting to *create* these files using tools written in
> other programming languages. And modules for creating *JSON* are
> available for most programming languages.

But that would be another tool, another package manager, not potluck... written by
we don't know who, neither when ... and that would be a totally separate effort,
that would not contribute to potluck ... which needs help (I wish I had the time...)

Potluck package definitions are generated, adjusted 'by hand' - mostly to update
the description, sometimes the copyright - then that is used to generated a Guix
package, which is a Guile scheme module: It makes no sense to me that the
potluck package representation would be anything but s-expr:

	actually, most guilers do the exact opposite :) -  when an app or a lib
	either produces or needs xml, html, json ... the first thing they do is to
	transform these into s-expr, so these become (a lot more) readable and
	hackable ...

> (OTOH TOML[]1] could be a better format than JSON – it's much like
> .ini-files, but more formal specification...

Absolutely terrible :):) I hope we never do that, at least not for the 'official'
and maintained potluck package representation.

Again, my 2c.
David


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Release!

[Ricardo Wurmus]

Hi David,

>> >>   • Merge the potluck!  <https://bugs.gnu.org/26645>
>
>> About that… We now have a JSON importer, so maybe it’s worth using the
>> even simpler JSON package format instead of the simplified S-expression
>> format that Andy proposed.  What do you think?  Should we discuss this
>> at <https://bugs.gnu.org/26645>?
>
> FWIW, I much prefer s-exp, and the generated file is a scheme file right? I very
> much doubt guix and potluck package developers can't easily read (and write :))
> s-exp, so why would we 'abandon' s-exp, what would we win here? These files will
> never be processed by anything else but guix and/or potluck, and the 'package
> developers' do all know scheme perfectly well...

I’m not saying that JSON is “better” than s-exps.

Part of the potluck effort as I understood it is to simplify packaging.
The potluck package definition is strictly simpler than Guix package
definitions in that there’s less boilerplate and inputs are really just
strings.  Taking that aspect of simplifying packaging even farther we
can reduce the syntax even more.

The target audience here has little overlap with Guix developers.  Guix
won’t adopt JSON as a packaging format; that’s not what this is about.
The goal I had in mind when I worked on the JSON importer was to make
packaging even simpler for people who don’t really care all that much
about packaging — if they did they would probably want to learn about
how to contribute to Guix, and thus would want to learn the S-expr
syntax we use in Guix.

There are users of Guix who benefit from its features as a personal
package manager.  Users can already add their own packages via
GUIX_PACKAGE_PATH.  We support those who don’t feel comfortable writing
Scheme by offering a JSON importer; with just a minor change to “guix
build” we can even build JSON packages directly, without making people
convert them to Scheme modules first.

I think that this feature can be useful within the context of the
potluck.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[Hartmut Goebel]

Am 07.10.2017 um 23:30 schrieb Ricardo Wurmus:
> The target audience here has little overlap with Guix developers.  […]
> The goal I had in mind when I worked on the JSON importer was to make
> packaging even simpler for people who don’t really care all that much
> about packaging – […]

+1

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

[PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Mark H Weaver]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Ludo,
>
>>> Here are some important items I can think of:
> […]
>>>   • Guile 2.2 compiler terrible issue:
>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>
> One way to side-step this issue for the upcoming release is to use one
> Guile process per file when running “guix pull”.  This will make it run
> a lot slower, but that would be better than the current situation.

I've attached a workaround that I've been using for the last 6 weeks on
my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
otherwise it would not be able to successfully build the 'guix' package.

Note that I never use 'guix pull', so I'm not sure off-hand whether this
solves the problem there, but it certainly greatly reduces the memory
needed to run 'make' and thus to build the 'guix' package.

This patch modifies build-aux/compile-all.scm to work as follows: after
loading all modules in the parent process, it then forks off a child and
compiles 20 modules in parallel within that child while the parent
waits.  When the child is finished compiling those 20 modules, the child
exits (thus freeing the memory leaked during compilation), and then the
parent spawns a new child to compile the next 20 modules, and so on,
until all the modules are compiled.

We should probably consider applying this to master.  Thoughts?

      Mark



[-- Attachment #2: [PATCH] DRAFT: build: Compile scheme modules in batches --]
[-- Type: text/x-patch, Size: 3684 bytes --]

From 05d5581ff71eb3b48773a5d46b612202de0492fb Mon Sep 17 00:00:00 2001
From: Mark H Weaver <mhw@netris.org>
Date: Tue, 22 Aug 2017 03:26:10 -0400
Subject: [PATCH] DRAFT: build: Compile scheme modules in batches.

---
 build-aux/compile-all.scm | 48 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 147bb8019..96658e069 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
 ;;; Copyright © 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Mark H Weaver <mhw@netris.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -19,6 +20,7 @@
 
 (use-modules (system base target)
              (system base message)
+             (srfi srfi-1)
              (ice-9 match)
              (ice-9 threads)
              (guix build utils))
@@ -118,13 +120,45 @@
   ((_ . files)
    (let ((files (filter file-needs-compilation? files)))
      (for-each load-module-file files)
-     (let ((mutex (make-mutex)))
-       ;; Make sure compilation related modules are loaded before starting to
-       ;; compile files in parallel.
-       (compile #f)
-       (par-for-each (lambda (file)
-                       (compile-file* file mutex))
-                     files)))))
+     ;; Make sure compilation related modules are loaded before starting to
+     ;; compile files in parallel.
+     (compile #f)
+     ;; Flush all ports before entering the fork loop, to avoid flushing them
+     ;; more than once within the child processes created below.
+     (flush-all-ports)
+
+     ;; FIXME The following loop works around the apparent memory leak in the
+     ;; compiler of guile-2.2.2, where compiling scheme modules requires
+     ;; increasing amounts of memory, up to nearly 2 gigabytes when all guix
+     ;; sources are compiled within a single process.
+     ;;
+     ;; Ideally, we would simply apply 'par-for-each' to the entire set of
+     ;; files.  For now, to work around the memory leak, we spawn subprocesses
+     ;; to compile the files in batches of up to 20 files each.
+     (let fork-loop ((files files))
+       (unless (null? files)
+         (call-with-values (lambda ()
+                             (split-at files (min 20 (length files))))
+           (lambda (current-batch remaining-files)
+             ;; IMPORTANT: as noted in the Guile manual, it is unsafe to fork a
+             ;; process that has multiple threads running.  Here we avoid this
+             ;; difficulty by spawning threads only within the child processes,
+             ;; which never call fork.
+             (match (primitive-fork)
+               (0
+                ;; This is the child.  It spawns threads but never forks.
+                (let ((mutex (make-mutex)))
+                  (par-for-each (lambda (file)
+                                  (compile-file* file mutex))
+                                current-batch))
+                (primitive-exit))
+               (child-pid
+                ;; This is the parent.  It forks but never spawns threads.
+                (match (waitpid child-pid)
+                  ((_ . 0)
+                   (fork-loop remaining-files))
+                  ((_ . status)
+                   (primitive-exit (or (status:exit-val status) 1)))))))))))))
 
 ;;; Local Variables:
 ;;; eval: (put 'with-target 'scheme-indent-function 1)
-- 
2.14.1

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]

On Sat, Oct 07, 2017 at 12:06:51AM -0400, Mark H Weaver wrote:
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
> > Hi Ludo,
> >
> >>> Here are some important items I can think of:
> > […]
> >>>   • Guile 2.2 compiler terrible issue:
> >>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
> >
> > One way to side-step this issue for the upcoming release is to use one
> > Guile process per file when running “guix pull”.  This will make it run
> > a lot slower, but that would be better than the current situation.
> 
> I've attached a workaround that I've been using for the last 6 weeks on
> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
> otherwise it would not be able to successfully build the 'guix' package.
> 
> Note that I never use 'guix pull', so I'm not sure off-hand whether this
> solves the problem there, but it certainly greatly reduces the memory
> needed to run 'make' and thus to build the 'guix' package.
> 
> This patch modifies build-aux/compile-all.scm to work as follows: after
> loading all modules in the parent process, it then forks off a child and
> compiles 20 modules in parallel within that child while the parent
> waits.  When the child is finished compiling those 20 modules, the child
> exits (thus freeing the memory leaked during compilation), and then the
> parent spawns a new child to compile the next 20 modules, and so on,
> until all the modules are compiled.
> 
> We should probably consider applying this to master.  Thoughts?
> 
>       Mark
> 
> 

Can we give it a set to compile in order? For example building
guix/build[-system] and then gnu/build, and then only at the end build
gnu/services {guix,gnu}/tests? It might be slightly faster to build the
modules that don't pull in gnu/packages/* first and then end with those
that would use those modules.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Hi Mark,

> I've attached a workaround that I've been using for the last 6 weeks on
> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
> otherwise it would not be able to successfully build the 'guix' package.
>
> Note that I never use 'guix pull', so I'm not sure off-hand whether this
> solves the problem there, but it certainly greatly reduces the memory
> needed to run 'make' and thus to build the 'guix' package.

thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
My own attempt to revert the commit to build all files in one process
resulted in a guix pull that would only go to 20.4% and then sit there
doing nothing.

I’ll report back once I manage to run “guix pull --url=/path/to/guix”
with this patch.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>> I've attached a workaround that I've been using for the last 6 weeks on
>> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
>> otherwise it would not be able to successfully build the 'guix' package.
>>
>> Note that I never use 'guix pull', so I'm not sure off-hand whether this
>> solves the problem there, but it certainly greatly reduces the memory
>> needed to run 'make' and thus to build the 'guix' package.
>
> thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
> My own attempt to revert the commit to build all files in one process
> resulted in a guix pull that would only go to 20.4% and then sit there
> doing nothing.
>
> I’ll report back once I manage to run “guix pull --url=/path/to/guix”
> with this patch.

20 files at once is still too much when running “guix pull” on my i686
netbook.  I tried lowering this to 10 files and still ran out of memory
(after 70 minutes of compiling).  I’m now trying with 3 files at a time.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>>> I've attached a workaround that I've been using for the last 6 weeks on
>>> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
>>> otherwise it would not be able to successfully build the 'guix' package.
>>>
>>> Note that I never use 'guix pull', so I'm not sure off-hand whether this
>>> solves the problem there, but it certainly greatly reduces the memory
>>> needed to run 'make' and thus to build the 'guix' package.
>>
>> thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
>> My own attempt to revert the commit to build all files in one process
>> resulted in a guix pull that would only go to 20.4% and then sit there
>> doing nothing.
>>
>> I’ll report back once I manage to run “guix pull --url=/path/to/guix”
>> with this patch.

Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
uses build-self.scm, so this patch has no effect on “guix pull”.

I’ll try to come up with another fix for guix pull.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>>>> I've attached a workaround that I've been using for the last 6 weeks on
>>>> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
>>>> otherwise it would not be able to successfully build the 'guix' package.
>>>>
>>>> Note that I never use 'guix pull', so I'm not sure off-hand whether this
>>>> solves the problem there, but it certainly greatly reduces the memory
>>>> needed to run 'make' and thus to build the 'guix' package.
>>>
>>> thank you for this patch.  I’m trying it on my i686 with 1GB of RAM now.
>>> My own attempt to revert the commit to build all files in one process
>>> resulted in a guix pull that would only go to 20.4% and then sit there
>>> doing nothing.
>>>
>>> I’ll report back once I manage to run “guix pull --url=/path/to/guix”
>>> with this patch.
>
> Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
> uses build-self.scm, so this patch has no effect on “guix pull”.

Right, this should be tested with “make”.

If it works fine here, we can port it to (guix build pull).

(And factorize it eventually…)

Ludo’.

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

>> Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
>> uses build-self.scm, so this patch has no effect on “guix pull”.
>
> Right, this should be tested with “make”.
>
> If it works fine here, we can port it to (guix build pull).

Okay.  I’ll try this some time this week, as soon as I can.  Maybe
tonight.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>>> Well… I just noticed that guix pull doesn’t even use compile-all.scm; it
>>> uses build-self.scm, so this patch has no effect on “guix pull”.
>>
>> Right, this should be tested with “make”.
>>
>> If it works fine here, we can port it to (guix build pull).
>
> Okay.  I’ll try this some time this week, as soon as I can.  Maybe
> tonight.

“make” crashed when I ran this on the netbook.  It ran out of memory
when compiling python.scm, as it always does.

I cannot discern if this is an improvement, because the result is
ultimately the same on my hardware; but since Mark uses this
successfully on a Yeeloong with 1GB RAM, I have no objections to pushing
this to master.

Thanks!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Ludovic Courtès]

Hi,

Mark H Weaver <mhw@netris.org> skribis:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Hi Ludo,
>>
>>>> Here are some important items I can think of:
>> […]
>>>>   • Guile 2.2 compiler terrible issue:
>>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>>
>> One way to side-step this issue for the upcoming release is to use one
>> Guile process per file when running “guix pull”.  This will make it run
>> a lot slower, but that would be better than the current situation.
>
> I've attached a workaround that I've been using for the last 6 weeks on
> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
> otherwise it would not be able to successfully build the 'guix' package.
>
> Note that I never use 'guix pull', so I'm not sure off-hand whether this
> solves the problem there, but it certainly greatly reduces the memory
> needed to run 'make' and thus to build the 'guix' package.
>
> This patch modifies build-aux/compile-all.scm to work as follows: after
> loading all modules in the parent process, it then forks off a child and
> compiles 20 modules in parallel within that child while the parent
> waits.  When the child is finished compiling those 20 modules, the child
> exits (thus freeing the memory leaked during compilation), and then the
> parent spawns a new child to compile the next 20 modules, and so on,
> until all the modules are compiled.
>
> We should probably consider applying this to master.  Thoughts?

That sounds like a good idea.  If it works, I’m all for it.

We’ll need to do something similar in (guix scripts pull).

I’ve been working on having ‘guix pull’ build modules as several
derivations as part of <https://bugs.gnu.org/27284>.  I’ll try to post a
prototype hopefully later today.

Thanks,
Ludo’.

Re: Release!

[Ludovic Courtès]

Heya,

Ricardo Wurmus <rekado@elephly.net> skribis:

>>> Here are some important items I can think of:
> […]
>>>   • Guile 2.2 compiler terrible issue:
>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

I should mention that I spent entire days on this, and some of the
conclusions are described here:

  https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html
  https://bugs.gnu.org/28590

I invite the Guilers among us to join the fun.  :-)  A bug that serious
is a problem for Guile.

From a Guix perspective, we have to find alternate solutions to improve
scalability anyway:

  https://bugs.gnu.org/27284

>>>   • Prepare to migrate to the new build farm: the hardware of
>>>     bayfront.guixsd.org has been fixed (it was unreliable until we
>>>     changed its CPUs), so now we need to fix a couple of issues in
>>>     Cuirass and generally improve it so we can, via its HTTP interface,
>>>     add new jobsets and so on.  Of course we’ll have to work
>>>     on that incrementally.
>
> I guess we’ll be using berlin.guixsd.org instead of bayfront, no?

Yes.

> I’m currently installing additional servers (we’re now at 13 build
> servers, I’ll get this up to 18 this week).

Woohoo, thank you!

>>>   • Merge the potluck!  <https://bugs.gnu.org/26645>
>
> About that… We now have a JSON importer, so maybe it’s worth using the
> even simpler JSON package format instead of the simplified S-expression
> format that Andy proposed.  What do you think?  Should we discuss this
> at <https://bugs.gnu.org/26645>?

Yeah I think we need to reopen the discussion.  We discussed it at the
GHM but this should take place here on guix-devel I guess.

> Who would like to help us squash some bugs before the release?  Let’s
> try to fix at least 5 more bugs!

Yes, let’s squash bugs!

There are many ways people can help, and one of them is triage: close
bugs that are already fixed, determine whether old bugs are still
relevant and close those that are obsolete, retitle bugs to better
reflect what the problem is, ping people asking for more information.
After that, there are more-or-less “easy” packaging bugs in the list
that can be a good starting point.

Thanks,
Ludo’.

Re: Release!

[Ludovic Courtès]

Hello Guix,

I think we can start preparing the release for good now!

The situation of ‘guix pull’ is mitigated by the split of python.scm and
other big files into several pieces.  This is of course not ideal, but
it’s an improvement over 0.13.0 anyway.

I think the main tasks to prepare the release are (1) testing the GuixSD
ISO, and (2) updating ‘NEWS’.

Let’s get that done!

Ludo’.

Re: Release!

[Ludovic Courtès]

Hello!

Time flies, but I think we’re about ready for the release.  I’d like to
aim for next Tuesday (Dec. 5th) and fix small hickups and annoyances by
then, particularly regarding GuixSD installation.

Ricardo Wurmus <rekado@elephly.net> skribis:

>>> Here are some important items I can think of:
> […]
>>>   • Guile 2.2 compiler terrible issue:
>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

There’s more going on on the Guile side, but for now the recent split of
the big package files has helped reduce peak memory consumption
noticeably.

> One annoyance is that after compiling one file it prints this message:
>
>    Some deprecated features have been used.  Set the environment
>    variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
>    program to get more information.  Set it to "no" to suppress
>    this message.

This was also showing up a lot in ‘guix system init’ et al, when
compiling modules.  I think a912c723f76d9762072ce27204a9227a64bcb625
removes most of these.

>>>   • Prepare to migrate to the new build farm: the hardware of
>>>     bayfront.guixsd.org has been fixed (it was unreliable until we
>>>     changed its CPUs), so now we need to fix a couple of issues in
>>>     Cuirass and generally improve it so we can, via its HTTP interface,
>>>     add new jobsets and so on.  Of course we’ll have to work
>>>     on that incrementally.
>
> I guess we’ll be using berlin.guixsd.org instead of bayfront, no?

Yes.  Questions:

  1. Do we pre-register berlin’s key on GuixSD?

  2. Do we add berlin.guixsd.org to the list of substitute servers on
     GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
     both servers when retrieve substitute lists, which can be slightly
     slower/annoying, but otherwise I think it’s a win.

Thoughts?

The main GuixSD annoying messages that I think we should address by
Tuesday are:

  1. “Error in finalization thread: Bad file descriptor” coming from the
     Shepherd;

  2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”

Both are harmless but they may give users a false sense that something’s
broken.

Ludo’.

Re: Release!

[Maxim Cournoyer]

Hi!

ludo@gnu.org (Ludovic Courtès) writes:

[...]

>>
>> I guess we’ll be using berlin.guixsd.org instead of bayfront, no?
>
> Yes.  Questions:
>
>   1. Do we pre-register berlin’s key on GuixSD?

Isn't this already the case?

[...]

>
> The main GuixSD annoying messages that I think we should address by
> Tuesday are:

[...]

>
>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>

I had researched about that error a bit and it seems that it caused by
eudev lacking support for 'uaccess'[0]; so not something to fix in Guix
proper.

[0]  https://github.com/gentoo/eudev/issues/145

Maxim

Re: Release!

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>   1. Do we pre-register berlin’s key on GuixSD?

I vote yes.

>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>      both servers when retrieve substitute lists, which can be slightly
>      slower/annoying, but otherwise I think it’s a win.

I think it's worth the extra source of substitutes. Since adding
berlin.guixsd.org to my list of substitute URLs, it seems like I need to
build less often.

In my experience, it's annoying to query multiple servers when my
network connection is slow or unreliable. But, it's also annoying in
that situation to need to download source files from a variety of
upstream sites.

> The main GuixSD annoying messages that I think we should address by
> Tuesday are:
> 
>   1. “Error in finalization thread: Bad file descriptor” coming from the
>      Shepherd;

I'm not sure how to start debugging this :/ If I get some advice, I
could try to fix it on Sunday and Monday.

>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”

I haven't seen this one before.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release!

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>>   1. Do we pre-register berlin’s key on GuixSD?
>
> I vote yes.
>
>>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>>      both servers when retrieve substitute lists, which can be slightly
>>      slower/annoying, but otherwise I think it’s a win.
>
> I think it's worth the extra source of substitutes. Since adding
> berlin.guixsd.org to my list of substitute URLs, it seems like I need to
> build less often.
>
> In my experience, it's annoying to query multiple servers when my
> network connection is slow or unreliable. But, it's also annoying in
> that situation to need to download source files from a variety of
> upstream sites.

I’d like to add berlin.guix.org to the list of default substitute
servers, but before I can allow this with a good conscience I need to
increase space on the head node.  We were very busy, so the new head
node and storage aren’t ready yet, and the old node that’s currently
coordinating builds on berlin.guix.org has very limited storage.

I’ll try to get the new storage ready on Monday.

>>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>
> I haven't seen this one before.

I have seen it, but only when checking the output of dmesg.  I guess we
could just remove that rule.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Release!

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Leo Famulari <leo@famulari.name> writes:
>
>> On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>>>   1. Do we pre-register berlin’s key on GuixSD?
>>
>> I vote yes.
>>
>>>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>>>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>>>      both servers when retrieve substitute lists, which can be slightly
>>>      slower/annoying, but otherwise I think it’s a win.
>>
>> I think it's worth the extra source of substitutes. Since adding
>> berlin.guixsd.org to my list of substitute URLs, it seems like I need to
>> build less often.
>>
>> In my experience, it's annoying to query multiple servers when my
>> network connection is slow or unreliable. But, it's also annoying in
>> that situation to need to download source files from a variety of
>> upstream sites.
>
> I’d like to add berlin.guix.org to the list of default substitute
> servers, but before I can allow this with a good conscience I need to
> increase space on the head node.  We were very busy, so the new head
> node and storage aren’t ready yet, and the old node that’s currently
> coordinating builds on berlin.guix.org has very limited storage.
>
> I’ll try to get the new storage ready on Monday.

Awesome.  Let us know how it goes and what we should do!

Thanks,
Ludo’.

ISO image available for testing!

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Hello there!

I’ve uploaded a GuixSD installation ISO image for testing:

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz
  SHA256: 7e04017494c419bb6ca654cff05dc7e5ba4fdfd3c2812fba9ba6d929b319e93b

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz.sig

It was built with “guix system disk-image -t iso9660
gnu/system/install.scm” on commit 8d7f1d736.

Note that it’s 1.1G uncompressed so it won’t fit on a CD.

Feedback welcome!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: ISO image available for testing!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]


Ludovic Courtès writes:

> Hello there!
>
> I’ve uploaded a GuixSD installation ISO image for testing:
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz
>   SHA256: 7e04017494c419bb6ca654cff05dc7e5ba4fdfd3c2812fba9ba6d929b319e93b
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz.sig
>
> It was built with “guix system disk-image -t iso9660
> gnu/system/install.scm” on commit 8d7f1d736.
>
> Note that it’s 1.1G uncompressed so it won’t fit on a CD.
>
> Feedback welcome!

Hey,

I've attempted to use this to install GuixSD on a Bytemark
VM. Unfortunately I haven't succeeded yet. I managed to get as far as
running guix system init, but when I did, it started downloading the
bootstrap binaries, and then building binutils.

When I run guix system build, or guix build with the --dry-run options,
it says that it will download, rather than building, but it doesn't.

I also made a few other notes during the installation process:

 - The documentation mentions ipconfig to get the wired interface up,
   but it isn't available in the image. I had to run ip link set eth0
   up.
 - The documentation says that the ssh server needs to be started, but
   it was running without me starting it.

Any tips on debugging the building from source issue?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: ISO image available for testing!

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> I've attempted to use this to install GuixSD on a Bytemark
> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
> running guix system init, but when I did, it started downloading the
> bootstrap binaries, and then building binutils.
>
> When I run guix system build, or guix build with the --dry-run options,
> it says that it will download, rather than building, but it doesn't.

I’m pretty sure the building-from-source comes from grafts where the
package replacements, for some reason, are not getting built by Hydra.
I thought this had been fixed by
b574cee361bdabf21f5506252b6a183dcfcd028d, but clearly it hasn’t.

I’ll investigate tomorrow.

> I also made a few other notes during the installation process:
>
>  - The documentation mentions ipconfig to get the wired interface up,
>    but it isn't available in the image. I had to run ip link set eth0
>    up.

It mentions “ifconfig” (with an “f”) and that one is present.  Or am I
missing something?

>  - The documentation says that the ssh server needs to be started, but
>    it was running without me starting it.

Indeed.  I’ve pushed a fix as aab322d909c0b4abec132ef7aff31c31a1208841
in the new ‘version-0.14.0’ branch (note that it changes the ABI of (gnu
services ssh)).

Thanks,
Ludo’.

Re: ISO image available for testing!

[Ludovic Courtès]

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> I've attempted to use this to install GuixSD on a Bytemark
> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
> running guix system init, but when I did, it started downloading the
> bootstrap binaries, and then building binutils.
>
> When I run guix system build, or guix build with the --dry-run options,
> it says that it will download, rather than building, but it doesn't.

It turned out to be issues related to grafts and to what Hydra builds,
fixed with these commits:

3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable.
91c9b5d01 * packages: 'package-grafts' trims native inputs.
ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls.
f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages.

The Binutils issue is fixed by f00b85ff8.

Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in
its dependency graph, via gettext.  Thus, the expat graft was picked up
as a candidate graft.  However, expat itself was subject to the glibc
graft, and since there was no substitute for this particular expat, we’d
have to build it first, just to throw it away later on because coreutils
does not refer to it at run time.

Long story short: we were flagging native inputs as potential sources of
grafts even though, by definition, native inputs are not referred to at
run time.

The last commit ensures that Hydra builds the replacement for
‘ghostscript-with-cups’.

What’s tricky is that one doesn’t notice these issues unless starting
from a fresh store.

I’ve uploaded an updated ISO image, which I used to test substitute
availability and grafts.  If you have time in the coming hours, feedback
welcome:

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz
  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz.sig

Ludo’.

Re: ISO image available for testing!

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:
> 91c9b5d01 * packages: 'package-grafts' trims native inputs.

[...]

> Long story short: we were flagging native inputs as potential sources of
> grafts even though, by definition, native inputs are not referred to at
> run time.

I agree that this *should* never happen, but I see little reason for
confidence that it never happens in actual fact.

What would happen if a reference to a native-input *was* present in the
build outputs?  The reason I ask is that, for security reasons, it's
obviously very important to reliably avoid using ungrafted software at
run time.

I'm concerned that this recent change could cause minor
nearly-undetectable packaging mistakes to become major security holes.

One solution would be to explicitly check build outputs for references
to native-inputs, and to force a build failure in that case.

What do you think?

    Regards,
      Mark

Re: ISO image available for testing!

[Ben Woodcroft]

On 06/12/17 10:52, Mark H Weaver wrote:
> Hi Ludovic,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
> [...]
>
>> Long story short: we were flagging native inputs as potential sources of
>> grafts even though, by definition, native inputs are not referred to at
>> run time.
> I agree that this *should* never happen, but I see little reason for
> confidence that it never happens in actual fact.
>
> What would happen if a reference to a native-input *was* present in the
> build outputs?  The reason I ask is that, for security reasons, it's
> obviously very important to reliably avoid using ungrafted software at
> run time.
>
> I'm concerned that this recent change could cause minor
> nearly-undetectable packaging mistakes to become major security holes.
>
> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.
I believe there are a number of cases of this that happen when binaries 
are wrapped with paths derived from getenv, e.g. this phase in bamm:

          (add-after 'install 'wrap-executable
            (lambda* (#:key outputs #:allow-other-keys)
             (let* ((out  (assoc-ref outputs "out"))
                    (path (getenv "PATH")))
               (wrap-program (string-append out "/bin/bamm")
                 `("PATH" ":" prefix (,path))))
             #t))

It would be good to stop using getenv for this, for the reasons 
described here and others e.g. non-reproducibility, unnecessary 
dependencies etc.

Is there some easy way to "getenv" excluding unwanted components of an 
environment variable?

Thanks, ben

Re: native-inputs ending up as run-time references [was: ISO image available for testing!]

[Tobias Geerinckx-Rice]

Mark!
Ludovic!

Mark H Weaver wrote on 06/12/17 at 01:52:
> ludo@gnu.org (Ludovic Courtès) writes:
>> Long story short: we were flagging native inputs as potential 
>> sources of grafts even though, by definition, native inputs are
>> not referred to at run time.
> 
> I agree that this *should* never happen, but I see little reason for 
> confidence that it never happens in actual fact.

Hold on. I thought this happened *all the actual time*.

To me, the output of ‘guix graph’ implies that ghc[*] refers directly to
perl, and ghc-haddock-library to hspec-discover, and that both of those
are native inputs.

These are just the first two examples of packages with native inputs
that I happened to pull out of my haskell.scm. While Haskell does seem
particularly naughty, I've no reason to believe it's unique.

Are these not ‘run-time references’? Is your use of the term narrower
than mine?

> One solution would be to explicitly check build outputs for 
> references to native-inputs, and to force a build failure in that 
> case.

I was surprised to learn this was not already the case (before I started
slowly dragging hissing Haskell packages into the present). I suggest we
don't make any security assumptions about it until it is.

Kind regards,

T G-R

Re: native-inputs ending up as run-time references [was: ISO image available for testing!]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

On Wed, Dec 06, 2017 at 03:16:45AM +0100, Tobias Geerinckx-Rice wrote:
> Hold on. I thought this happened *all the actual time*.
> 
> To me, the output of ‘guix graph’ implies that ghc[*] refers directly to
> perl, and ghc-haddock-library to hspec-discover, and that both of those
> are native inputs.
> 
> These are just the first two examples of packages with native inputs
> that I happened to pull out of my haskell.scm. While Haskell does seem
> particularly naughty, I've no reason to believe it's unique.
> 
> Are these not ‘run-time references’? Is your use of the term narrower
> than mine?

I think of "run-time references" as explicit store references
(file-names) found in the build output by the post-build reference
scanner and recorded in /var/guix/db. These references are what grafting
rewrites.

`guix gc --references` can be used to query the database for a given
store item.

`guix graph`, on the other hand, shows the graphs of package
definitions. That is, the list of inputs, native-inputs,
propagated-inputs, etc. Some of these inputs could be totally
superfluous, and the build output would ideally not contain any
explicit references to them at all.

The term "reference" has a few meanings in Guix, but I think that Mark
is talking about explicit store paths.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: native-inputs ending up as run-time references [was: ISO image available for testing!]

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 1986 bytes --]

Leo,

Leo Famulari wrote on 06/12/17 at 04:18:
> I think of "run-time references" as explicit store references
> (file-names) found in the build output by the post-build reference
> scanner and recorded in /var/guix/db. These references are what grafting
> rewrites.

All right, that was my idea too (though there might be some sloppiness
in my mental model yet).

> `guix gc --references` can be used to query the database for a given
> store item.
>
> `guix graph`, on the other hand, shows the graphs of package
> definitions. That is, the list of inputs, native-inputs,
> propagated-inputs, etc. Some of these inputs could be totally
> superfluous, and the build output would ideally not contain any
> explicit references to them at all.

Thanks for the clarification! I've never actually found a use for ‘guix
graph’ myself, but can't imagine a Guix talk without it. :-)

Unfortunately, I sent you after a red herring: ‘graph’ was a typo.
I did, in fact, mean ‘guix gc’:

  $ guix gc --references `guix build ghc` | grep perl
  /gnu/store/9g4fpfz86vjkx83v5696vm5dzg2sc9qj-perl-5.26.0
  $ grep -r /gnu/store/9g4fpfz86vjkx83v5696vm5dzg2sc9qj-perl-5.26.0 \
    `guix build ghc`
  .../ghc-split:#!/gnu/store/...-perl-5.26.0/bin/perl
  .../settings: ("perl command", "/gnu/store/...-perl-5.26.0/bin/perl"),

And:

  $ guix gc --references `guix build ghc-haddock-library` | grep hspec-d
  /gnu/store/xfh89fmdn2xvd4aw76164zqz0941xw01-hspec-discover-2.2.0
  $ grep -r xfh89fmdn2xvd4aw76164zqz0941xw01-hspec-discover-2.2.0 \
    `guix build ghc-haddock-library`
  [very long paths snipped with great prejudice]
  Binary file /gnu/store/.../package.cache matches
  Binary file /gnu/store/.../libHShaddock-library-1.2.1-...-ghc7.10.2.so
  matches

> The term "reference" has a few meanings in Guix, but I think that Mark
> is talking about explicit store paths.

I'm afraid we're on the same page then.

Kind regards,

T G-R


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

Re: ISO image available for testing!

[Mark H Weaver]

I wrote:
> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.

More precisely, we could force a build failure when any build output
contains a reference to a component (store path) in the following set:

  requisites(native-inputs) - requisites(inputs)

       Mark

Re: ISO image available for testing!

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
>
> [...]
>
>> Long story short: we were flagging native inputs as potential sources of
>> grafts even though, by definition, native inputs are not referred to at
>> run time.
>
> I agree that this *should* never happen, but I see little reason for
> confidence that it never happens in actual fact.
>
> What would happen if a reference to a native-input *was* present in the
> build outputs?  The reason I ask is that, for security reasons, it's
> obviously very important to reliably avoid using ungrafted software at
> run time.
>
> I'm concerned that this recent change could cause minor
> nearly-undetectable packaging mistakes to become major security holes.

Given the examples that Tobias and Ben were quick to find, I’m afraid
you’re right and I was overconfident.  I’m reverting the change.

> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.

We could do that, though I suppose a lot of packages would break.

Thanks to the quick reply,
Ludo’.

Re: ISO image available for testing!

[Tobias Geerinckx-Rice]

Ludovic Courtès wrote on 06/12/17 at 09:14:
>> One solution would be to explicitly check build outputs for references
>> to native-inputs, and to force a build failure in that case.

I started a rudimentary PoC for this last week, after realising how bad
the situation is in ghc-* alone. We'll see what comes of it.

> We could do that, though I suppose a lot of packages would break.

We *should* do it, though. Just not before the pending release.

Kind regards,

T G-R

Re: ISO image available for testing!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2953 bytes --]


Ludovic Courtès writes:

> Hi Chris,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I've attempted to use this to install GuixSD on a Bytemark
>> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
>> running guix system init, but when I did, it started downloading the
>> bootstrap binaries, and then building binutils.
>>
>> When I run guix system build, or guix build with the --dry-run options,
>> it says that it will download, rather than building, but it doesn't.
>
> It turned out to be issues related to grafts and to what Hydra builds,
> fixed with these commits:
>
> 3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable.
> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
> ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls.
> f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages.
>
> The Binutils issue is fixed by f00b85ff8.
>
> Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in
> its dependency graph, via gettext.  Thus, the expat graft was picked up
> as a candidate graft.  However, expat itself was subject to the glibc
> graft, and since there was no substitute for this particular expat, we’d
> have to build it first, just to throw it away later on because coreutils
> does not refer to it at run time.
>
> Long story short: we were flagging native inputs as potential sources of
> grafts even though, by definition, native inputs are not referred to at
> run time.
>
> The last commit ensures that Hydra builds the replacement for
> ‘ghostscript-with-cups’.
>
> What’s tricky is that one doesn’t notice these issues unless starting
> from a fresh store.
>
> I’ve uploaded an updated ISO image, which I used to test substitute
> availability and grafts.  If you have time in the coming hours, feedback
> welcome:

Thanks for fixing this Ludo, and congratulations on the release. I'm
glad to say that I've now managed to install GuixSD using the 0.14.0
x86_64 ISO, however I did encounter some difficulties.

I tried a few times with both the ISO you replied with here, and the
released ISO, but each time the virtual machine I was installing on to
appeared to restart while guix system init was running. It's difficult
to get more information, but the last messages I got out of guix system
init relate to grafting and collisions.

This evening when I tried again I passed --no-grafts to guix system
init, and this time it successfully finished the
installation. Interestingly, this is also what is actually tested by the
iso-image-installer system test, as it sets the GUIX_BUILD_OPTIONS
environment variable.

This isn't conclusive, but I'd be very interested to hear from anyone
that has had similar issues, or successes in using the ISO installer,
both with and without the --no-grafts option.

Thanks again,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: ISO image available for testing!

[Ludovic Courtès]

Hello Chris,

Christopher Baines <mail@cbaines.net> skribis:

> Thanks for fixing this Ludo, and congratulations on the release. I'm
> glad to say that I've now managed to install GuixSD using the 0.14.0
> x86_64 ISO, however I did encounter some difficulties.
>
> I tried a few times with both the ISO you replied with here, and the
> released ISO, but each time the virtual machine I was installing on to
> appeared to restart while guix system init was running. It's difficult
> to get more information, but the last messages I got out of guix system
> init relate to grafting and collisions.

I wonder how the VM can restart.  Could it be an out-of-memory
condition?  Though even that should not lead to a reboot.

Could you see if you can get more details from the VM?

Thanks,
Ludo’.

Re: Release!

[Ludovic Courtès]

Hello!

Leo Famulari <leo@famulari.name> skribis:

> On Thu, Nov 30, 2017 at 11:40:16AM +0100, Ludovic Courtès wrote:
>>   1. Do we pre-register berlin’s key on GuixSD?
>
> I vote yes.
>
>>   2. Do we add berlin.guixsd.org to the list of substitute servers on
>>      GuixSD?  On Guix?  The drawback is that ‘guix’ sometimes talks to
>>      both servers when retrieve substitute lists, which can be slightly
>>      slower/annoying, but otherwise I think it’s a win.
>
> I think it's worth the extra source of substitutes. Since adding
> berlin.guixsd.org to my list of substitute URLs, it seems like I need to
> build less often.
>
> In my experience, it's annoying to query multiple servers when my
> network connection is slow or unreliable. But, it's also annoying in
> that situation to need to download source files from a variety of
> upstream sites.

Sounds reasonable.  Let’s wait for a green light from Ricardo.

>> The main GuixSD annoying messages that I think we should address by
>> Tuesday are:
>> 
>>   1. “Error in finalization thread: Bad file descriptor” coming from the
>>      Shepherd;
>
> I'm not sure how to start debugging this :/ If I get some advice, I
> could try to fix it on Sunday and Monday.

I investigated and fixed it yesterday:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4bd70904c7f555a953808a9a4f892f462ffd352f

The thing is that there’s a separate finalization thread in Guile, which
takes care of finalizing dead objects.  For file ports, finalization
means closing the underlying file descriptor.  As it turns out,
‘close-port’ in Guile 2.0 would ignore EBADF errors, whereas
‘close-port’ in 2.2 reports them, hence the error.

The fix is to take extra care to close ports and not just the underlying
file descriptors.

>>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>
> I haven't seen this one before.

I compared the source of eudev and systemd, and indeed, like Maxim
wrote, the issue is that eudev does not support a “uaccess” built-in
tag.  So I pushed a fix that simply comments out that line:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f8446df663fecb5aa34e5c6dfa477544d3271d1e

There’s still room for improvement, but the console output is less messy
now.  :-)

Thanks,
Ludo’.

Re: Release!

[Maxim Cournoyer]

Hi!

ludo@gnu.org (Ludovic Courtès) writes:

[...] (Great fix! :)

>>>   2. “udevd[304]: RUN{builtin}: 'uaccess' unknown /gnu/store/q7c8yayywf76ai3sgvz16pmbz07gj4bp-udev-rules/lib/udev/rules.d/73-seat-late.rules:15”
>>
>> I haven't seen this one before.
>
> I compared the source of eudev and systemd, and indeed, like Maxim
> wrote, the issue is that eudev does not support a “uaccess” built-in
> tag.  So I pushed a fix that simply comments out that line:
>
>   https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f8446df663fecb5aa34e5c6dfa477544d3271d1e

Nitpick: Maybe we could reference to the issue so that a maintainer can
decide to remove that alteration after the "uaccess" issue gets properly
fixed (implemented).

> There’s still room for improvement, but the console output is less messy
> now.  :-)

Thank you for it!

Maxim

What’s next?

[Brendan Tildesley]

[-- Attachment #1: Type: text/plain, Size: 3271 bytes --]

Since you asked I'll dump my nebulous wishes here. Sorry that I don't have very
concrete suggestions and solutions, just things I think could be better.  I
should also state that 99% of my thoughts about Guix are through the filter of
"I want to build a Guix based desktop distribution that can be installed on
library, school, and home computers and Just Work better than Microsoft Windows"

1. Improve guix pull and updating reliability.
Updating guix for me is often like this:
$ guix pull
retrieving git objects....
TLS error in stream blah blah blah
$ guix pull
retrieving git objects..34%......... *dies but doesn't return to the prompt*
ctrl-c
$ guix pull
$ guix upgrade
$ sudo guix system reconfigure...
GUI desktop crashes back to TTY mysteriously.

Basically if guix pull can become as reliable as 'guix pull; guix pull; guix
pull; guix pull;, that would be great.  I think guix has many code paths
pertaining to downloading files where it chooses "Give Up" where "Try 2 or 3
times first" might be better.

Often downloads to source tarballs will download at ~30KiB/s. maybe some
axel-like system where both upstream an mirror urls are used might be good. Also
this may be wrong but from memory I think if the hash is wrong on a source
tarball, it will not bother trying to use the guix ci mirror of it, only if it's
404. So people with --no-substitutes can get completely stuck.

2. 'guix system reconfigure' without instantiating the new system until reboot.
I think the fact that reconfigure changes the whole system while its running is
a bit of a party trick, like pulling the table cloth out without any glasses
falling over.  For a Desktop GNU/Linux it kinda just accidentally works much of
the time. I've always thought I'd prefer the default to simply install the new
configuration in to GRUB and tell the user to reboot or add some flag like
--instatiate-now.


3. CLI consistency, e.g.,
$ guix package => nothing
$ guix system =>
guix system: missing command name
Try 'guix system --help' for more information.

$ guix package --list-generations, but
$ guix system list-generations ???\

Also https://issues.guix.gnu.org/47618


4. Make guix simpler and more performant.

Guix is complex. It has features that make it theoretically superior in many
ways, but in practice hasn't reached the reliability of simpler systems.  Every
aspect of Guix seems to take more time, cpu, ram, hd space than say pacman.
It's a bit beyond my skills to figure out how to actually improve these things
though.

$ guix system foobar

takes 1.5-3.0 seconds on SSD, stats 2374 files, with 345 No such file or
directories in order to determine the command doesn't exist and do nothing.
A fast guix could have 'guix time-machine -- environment --ad-hoc emacs --
emacs' run instantly after its been ran once, and some kind of 'guix run'
command could be added.

5. I think Guix should have some simple system to only update to the latest
commit where behemoths like webkit/chromium already have substitutes available
and that should maybe be the default way to update. It's not a big deal if we
are making people build stuff that only takes 30 seconds and uses minimal
ram. Doesn't need to be perfect. Ricardo made some simple scripts that did stuff
like that in the past.


[-- Attachment #2: Type: text/html, Size: 3958 bytes --]

Re: What’s next?

[Ludovic Courtès]

Hi Brendan,

Brendan Tildesley <btild@mailbox.org> skribis:

> Since you asked I'll dump my nebulous wishes here. Sorry that I don't have very
> concrete suggestions and solutions, just things I think could be better.  I
> should also state that 99% of my thoughts about Guix are through the filter of
> "I want to build a Guix based desktop distribution that can be installed on
> library, school, and home computers and Just Work better than Microsoft Windows"

This is great.  I agree that polishing what already exists should remain
a major goal.

> 1. Improve guix pull and updating reliability.
> Updating guix for me is often like this:
> $ guix pull
> retrieving git objects....
> TLS error in stream blah blah blah
> $ guix pull
> retrieving git objects..34%......... *dies but doesn't return to the prompt*
> ctrl-c

I’ve literally never experienced this.  Could you file a bug or two to
bug-guix@gnu.org, with at least the command and all the output?

> $ guix pull
> $ guix upgrade
> $ sudo guix system reconfigure...
> GUI desktop crashes back to TTY mysteriously.

Likewise, please!

> Often downloads to source tarballs will download at ~30KiB/s. maybe some
> axel-like system where both upstream an mirror urls are used might be
> good.

Could you report as many details when that happens?

> Also this may be wrong but from memory I think if the hash is wrong on
> a source tarball, it will not bother trying to use the guix ci mirror
> of it, only if it's 404. So people with --no-substitutes can get
> completely stuck.

True: <https://issues.guix.gnu.org/28659>.

> 2. 'guix system reconfigure' without instantiating the new system until reboot.
> I think the fact that reconfigure changes the whole system while its running is
> a bit of a party trick, like pulling the table cloth out without any glasses
> falling over.  For a Desktop GNU/Linux it kinda just accidentally works much of
> the time. I've always thought I'd prefer the default to simply install the new
> configuration in to GRUB and tell the user to reboot or add some flag like
> --instatiate-now.

I’m satisfied with the current default, but we could have an option to
delay activation until reboot.  Likewise, please file as a “wishlist”
item to bug-guix@gnu.org.

> 3. CLI consistency, e.g.,
> $ guix package => nothing
> $ guix system =>
> guix system: missing command name
> Try 'guix system --help' for more information.
>
> $ guix package --list-generations, but
> $ guix system list-generations ???\
>
> Also https://issues.guix.gnu.org/47618

All good points!

> 4. Make guix simpler and more performant.
>
> Guix is complex. It has features that make it theoretically superior in many
> ways, but in practice hasn't reached the reliability of simpler systems.  Every
> aspect of Guix seems to take more time, cpu, ram, hd space than say pacman.
> It's a bit beyond my skills to figure out how to actually improve these things
> though.
>
> $ guix system foobar
>
> takes 1.5-3.0 seconds on SSD, stats 2374 files, with 345 No such file or
> directories in order to determine the command doesn't exist and do nothing.

Agreed.  There have been improvements over time, such as the ld.so cache
in ‘core-updates’ (not merged yet), but there’s still a long way.

It’s frustrating for all of us, but one way to help is by pinpointing
specific bottlenecks and gathering as much data as possible so we have a
good starting point for optimization work.

> A fast guix could have 'guix time-machine -- environment --ad-hoc emacs --
> emacs' run instantly after its been ran once, and some kind of 'guix run'
> command could be added.

Yeah.

> 5. I think Guix should have some simple system to only update to the latest
> commit where behemoths like webkit/chromium already have substitutes available
> and that should maybe be the default way to update. It's not a big deal if we
> are making people build stuff that only takes 30 seconds and uses minimal
> ram. Doesn't need to be perfect. Ricardo made some simple scripts that did stuff
> like that in the past.

Yes, this is being worked on (you may have seen
‘channel-with-substitutes-available’ in 1.3.0, which is a first step in
that direction.)

Thanks for sharing!  I guess my message here is that all these should be
broken down into individual bug reports/wishes that can be more easily
addressed.  :-)

Ludo’.

What’s next?

[Ludovic Courtès]

Hello Guix!

So, now that 1.3.0 is out the door, what’s next?!

Here’s my wish list of things that look achievable within 4 to 6 months
(I hope to help on some of these):

  • Merging Guix Home.

  • Completing and consolidating Disarchive support (see
    <https://issues.guix.gnu.org/47336>): continuously building the
    Disarchive database, making sure it’s replicated or backed up by
    SWH, and having a blog post or two explaining the whole endeavor
    (I’m looking at you, Timothy ;-)).

  • Merging Magali’s ‘guix git log’ work.

  • Merging ‘core-updates’, perhaps with a switch to GCC 10?  Perhaps
    with support for “simplified packages” (getting rid of input
    labels)?  We’ll see how much can go in there, but the sooner the
    better.

  • On ‘core-updates’, merging either the full-source bootstrap or
    the reduced bootstrap on ARM, whichever comes first.  :-)

  • Consolidating the POWER9 and AArch64 ports (more build machines
    behind ci.guix!).

  • Maybe a step towards “early cutoffs” to reduce the amount of
    rebuilding (more on that in a future post).

  • Maybe optimized substitute downloads based on
    <https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html>
    or at the store item level (same as for early cutoffs).

  • Maybe parameterized packages based on
    <https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html>.

What do people think?  What’s your wish list?  What do you feel an urge
to hack on?  :-)

Ludo’.

Re: What’s next?

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 2106 bytes --]

In the short term I'd like to merge the python optimisations. We discussed it before release and thought it might be nice to merge them before c-u, from a separate branch.

Maybe before next release we'll have our separate repo on savannah for translations (it's currently a repo I manage on framagit). I'd also like to add some automation so I don't have to manually update the pot files :)

Le 15 mai 2021 13:47:19 GMT-04:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hello Guix!
>
>So, now that 1.3.0 is out the door, what’s next?!
>
>Here’s my wish list of things that look achievable within 4 to 6 months
>(I hope to help on some of these):
>
>  • Merging Guix Home.
>
>  • Completing and consolidating Disarchive support (see
>    <https://issues.guix.gnu.org/47336>): continuously building the
>    Disarchive database, making sure it’s replicated or backed up by
>    SWH, and having a blog post or two explaining the whole endeavor
>    (I’m looking at you, Timothy ;-)).
>
>  • Merging Magali’s ‘guix git log’ work.
>
>  • Merging ‘core-updates’, perhaps with a switch to GCC 10?  Perhaps
>    with support for “simplified packages” (getting rid of input
>    labels)?  We’ll see how much can go in there, but the sooner the
>    better.
>
>  • On ‘core-updates’, merging either the full-source bootstrap or
>    the reduced bootstrap on ARM, whichever comes first.  :-)
>
>  • Consolidating the POWER9 and AArch64 ports (more build machines
>    behind ci.guix!).
>
>  • Maybe a step towards “early cutoffs” to reduce the amount of
>    rebuilding (more on that in a future post).
>
>  • Maybe optimized substitute downloads based on
>  <https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html>
>    or at the store item level (same as for early cutoffs).
>
>  • Maybe parameterized packages based on
> <https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html>.
>
>What do people think?  What’s your wish list?  What do you feel an urge
>to hack on?  :-)
>
>Ludo’.

[-- Attachment #2: Type: text/html, Size: 2643 bytes --]

Re: What’s next?

[Leo Famulari]

On Sat, May 15, 2021 at 02:08:50PM -0400, Julien Lepiller wrote:
> In the short term I'd like to merge the python optimisations. We discussed it before release and thought it might be nice to merge them before c-u, from a separate branch.

+1

Have you tested them at all yet? If so, and they don't seem to break
anything, maybe they could be squeezed onto the wip-ungrafting branch.

The 'ungrafting' jobset (based on the wip-ungrafting branch) entails a
complete rebuild of Python 2 and 3. If that branch requires another
iteration due to broken builds, and we are confident that the Python
optimisations won't break things, I'd say we could include them.

Otherwise, we could do a python-updates branch that includes these
optimizations, and updates to Python 3.8.10, along with any existing bug
fixes for Python 2. It would be completed more quickly than
core-updates.

Re: What’s next?

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

I've rebuilt only up to python 3, and a handful pytgon packages, not a lot more. I'm very confident for the one that enables optimisation, less so for the one that removes files (supposedly useless, like windows binaries and test files).

I'm go for both options, maybe python-update is more wise.

Le 18 mai 2021 15:30:24 GMT-04:00, Leo Famulari <leo@famulari.name> a écrit :
>On Sat, May 15, 2021 at 02:08:50PM -0400, Julien Lepiller wrote:
>> In the short term I'd like to merge the python optimisations. We
>discussed it before release and thought it might be nice to merge them
>before c-u, from a separate branch.
>
>+1
>
>Have you tested them at all yet? If so, and they don't seem to break
>anything, maybe they could be squeezed onto the wip-ungrafting branch.
>
>The 'ungrafting' jobset (based on the wip-ungrafting branch) entails a
>complete rebuild of Python 2 and 3. If that branch requires another
>iteration due to broken builds, and we are confident that the Python
>optimisations won't break things, I'd say we could include them.
>
>Otherwise, we could do a python-updates branch that includes these
>optimizations, and updates to Python 3.8.10, along with any existing
>bug
>fixes for Python 2. It would be completed more quickly than
>core-updates.

[-- Attachment #2: Type: text/html, Size: 1688 bytes --]

Re: What’s next?

[Ludovic Courtès]

Bonjour !

Julien Lepiller <julien@lepiller.eu> skribis:

> In the short term I'd like to merge the python optimisations. We
> discussed it before release and thought it might be nice to merge them
> before c-u, from a separate branch.

That’d be great!

BTW, another thing that’d be nice it to use Guile-Netlink to provide a
more capable static-networking service.

> Maybe before next release we'll have our separate repo on savannah for
> translations (it's currently a repo I manage on framagit). I'd also
> like to add some automation so I don't have to manually update the pot
> files :)

+1  :-)

Ludo’.

Re: What’s next?

[Katherine Cox-Buday]

Ludovic Courtès <ludo@gnu.org> writes:

> BTW, another thing that’d be nice it to use Guile-Netlink to provide a
> more capable static-networking service.

Thank you for the reminder! This is blocking me from standing up a few
edge-routers.

Also, more focus on aarch64-linux substitutes!

-- 
Katherine

Re: What’s next?

[Ricardo Wurmus]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> BTW, another thing that’d be nice it to use Guile-Netlink to 
>> provide a
>> more capable static-networking service.
>
> Thank you for the reminder! This is blocking me from standing up 
> a few
> edge-routers.
>
> Also, more focus on aarch64-linux substitutes!

We’re currently waiting for three new 16-core Honeycomb LX2 boards 
to arrive Berlin, which should help with aarch64-linux 
substitutes.

-- 
Ricardo

Re: What’s next?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2065 bytes --]

On Sat, May 15, 2021 at 07:47:19PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> So, now that 1.3.0 is out the door, what’s next?!
> 
> Here’s my wish list of things that look achievable within 4 to 6 months
> (I hope to help on some of these):
> 
>   • Merging Guix Home.
> 
>   • Completing and consolidating Disarchive support (see
>     <https://issues.guix.gnu.org/47336>): continuously building the
>     Disarchive database, making sure it’s replicated or backed up by
>     SWH, and having a blog post or two explaining the whole endeavor
>     (I’m looking at you, Timothy ;-)).
> 
>   • Merging Magali’s ‘guix git log’ work.
> 
>   • Merging ‘core-updates’, perhaps with a switch to GCC 10?  Perhaps
>     with support for “simplified packages” (getting rid of input
>     labels)?  We’ll see how much can go in there, but the sooner the
>     better.
> 
>   • On ‘core-updates’, merging either the full-source bootstrap or
>     the reduced bootstrap on ARM, whichever comes first.  :-)
> 
>   • Consolidating the POWER9 and AArch64 ports (more build machines
>     behind ci.guix!).
> 
>   • Maybe a step towards “early cutoffs” to reduce the amount of
>     rebuilding (more on that in a future post).
> 
>   • Maybe optimized substitute downloads based on
>     <https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html>
>     or at the store item level (same as for early cutoffs).
> 
>   • Maybe parameterized packages based on
>     <https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html>.
> 
> What do people think?  What’s your wish list?  What do you feel an urge
> to hack on?  :-)
> 
> Ludo’.
> 

+1 on core-updates

package-transformations applied to the operating-system field of the
os-config.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What’s next?

[raingloom]

On Sat, 15 May 2021 23:24:41 +0300
Efraim Flashner <efraim@flashner.co.il> wrote:

> package-transformations applied to the operating-system field of the
> os-config.
> 

Gosh, yes please. I keep bumping into this while debugging services.

Also:
* substitutes over bittorrent might be cool.
* split packages more, similar to how Alpine does it. (i said i'd work
  on this and i intend to)
* deduplicate similar but not identical files (if the file system
  allows it)
* better GDB integration
* better initramfs environment, so debugging isn't such a horrible
  experience
* integrate environments with GUIs. if you've used nested rio on Plan
  9, you know what I'm talking about. I already have some utility
  programs for this, but they are pretty basic and only tested on my
  setup.
* accessible installer! even Arch has this now. (also something I want
  to work on eventually)

Re: What’s next?

[Joshua Branson]

Finally finish my endlessh service.

There's a version that works here:  http://issues.guix.gnu.org/39136

But it runs as root and IS NOT containerized.


P.S.  Also there were some really great suggestions in the previous
emails!

--
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar

Re: What’s next?

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> package-transformations applied to the operating-system field of the
> os-config.

Ah, that’s a good one, but possibly tricky!  What would it operate on?
Any package?  Only those showing up in the system profile?

The former is not really possible; the latter is.

Ludo’.

Re: What’s next?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

On Mon, May 17, 2021 at 10:13:10PM +0200, Ludovic Courtès wrote:
> Hi,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > package-transformations applied to the operating-system field of the
> > os-config.
> 
> Ah, that’s a good one, but possibly tricky!  What would it operate on?
> Any package?  Only those showing up in the system profile?
> 
> The former is not really possible; the latter is.
> 
> Ludo’.
> 

The idea is everything in the system declaration. So the global
packages, the guix-daemon from the guix-daemon service, but not packages
installed using /run/current-system/profile/bin/guix.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What’s next?

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Mon, May 17, 2021 at 10:13:10PM +0200, Ludovic Courtès wrote:
>> Hi,
>> 
>> Efraim Flashner <efraim@flashner.co.il> skribis:
>> 
>> > package-transformations applied to the operating-system field of the
>> > os-config.
>> 
>> Ah, that’s a good one, but possibly tricky!  What would it operate on?
>> Any package?  Only those showing up in the system profile?
>> 
>> The former is not really possible; the latter is.
>> 
>> Ludo’.
>> 
>
> The idea is everything in the system declaration. So the global
> packages, the guix-daemon from the guix-daemon service,

Yeah, that’s not easily possible, in the sense that there’s no place to
“hook” to perform such transforms.  The reason is that references to
packages may be buried down anywhere in gexps or files produced by
services, an obvious example being:

  #~(make-forkexec-constructor
      (list #$(file-append some-package "/bin/xyz")
            …))

The reference to ‘some-package’ here is deep down.

An option would be to have a way to provide ‘lower-object’ a customize
way to lower all the <package> records that it sees, for instance.  But
then, how does that interact with caches, etc.  Tricky!

Ludo’.

Re: What’s next?

[Maxim Cournoyer]

Hi Ludovic,

Those are all good ideas, but to be brief:

+1 for core-updates & newer GCC.

Some other things I'd like to see that seem achievable in a 4-6 months
time frame:

* Add .deb and .rpm formats for 'guix pack'.
* Better Texlive importer
* Easier way to debug (guix debug package) -> leaves you in an
environment including everything needed to debug (sources, symbols,
GDB).
* Proper support to mount NFS shares at boot
* guix environment --fhs (file hierachy standard)
* Support for Go modules in the go-build-system
* Better tooling (perhaps contribute to fix perf issues in Geiser/Emacs)
* A bootsplash to finally hide our "Error in finalizer: Success" message ;-)

I'll stop there :-).

Maxim

Re: What’s next?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

Hi,

Off the top of my head:

- Optimize the man page database update during profile builds.

- File search.

  (I did work on it a year ago, it's stuck, would need help from people
  familiar with Cuirass.)

- A GUI.

  https://gitlab.com/daym/guix-gui.git seems to be a good starting point.

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> * Add .deb and .rpm formats for 'guix pack'.
[...]
> * guix environment --fhs (file hierachy standard)

Oh my, awesome ideas!!!  (These 2 in particular.)

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: What’s next?

[Christopher Lemmer Webber]

Pierre Neidhardt writes:

> Hi,
>
> Off the top of my head:
>
> - Optimize the man page database update during profile builds.
>
> - File search.
>
>   (I did work on it a year ago, it's stuck, would need help from people
>   familiar with Cuirass.)

That sounds interesting.  What does it mean?  Would it help me figure
out which commands come from which package?

A feature I remember being cool they had enabled by default on Ubuntu
back in the day was:

  $ crawl-tiles
  Not found.  Maybe "sudo apt-get install crawl-tiles"?

So for us:

  $ crawl-tiles
  Not found.  Maybe "guix package -i crawl-tiles"?

The curious thing about this though... it seems to require knowledge of
the *outputs*.

To invoke this manually, a user could type:

  # Queries curiass or some sort of build system, similar to guix weather
  $ guix search --binary crawl-tiles
  Found in:
   - crawl-tiles

> - A GUI.
>
>   https://gitlab.com/daym/guix-gui.git seems to be a good starting point.

Oh wow!  I'm going to give this a try.

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> * Add .deb and .rpm formats for 'guix pack'.
> [...]
>> * guix environment --fhs (file hierachy standard)
>
> Oh my, awesome ideas!!!  (These 2 in particular.)
>
> Cheers!

I'll add one that I *should* finish: I should get the setuid stuff
done.  IIRC it's like, two minor tweaks away from being done and
bitrotting... I've been busy...

Maybe I could do it towards the end of the following week?  Would be
good to get the damned thing done... and then finally get postfix in,
too.

Re: What’s next?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

Hi Chris,

>> - File search.
>>
>>   (I did work on it a year ago, it's stuck, would need help from people
>>   familiar with Cuirass.)
>
> That sounds interesting.  What does it mean?  Would it help me figure
> out which commands come from which package?

Yes, and more ;)

> A feature I remember being cool they had enabled by default on Ubuntu
> back in the day was:
>
>   $ crawl-tiles
>   Not found.  Maybe "sudo apt-get install crawl-tiles"?
>
> So for us:
>
>   $ crawl-tiles
>   Not found.  Maybe "guix package -i crawl-tiles"?

Guix file-search can do that, but it wouldn't be limited to executables,
it would work for any file.

So `guix filesearch myimage.png`
would return a list of matches:

--8<---------------cut here---------------start------------->8---
emacs-myimage-mode:out  share/assets/myimage.png
...
some-game:assets  share/some-game/subfolder/myimage.png
--8<---------------cut here---------------end--------------->8---

It works by crawling the files and maintaining a database.

The work-in-progress on the wip-filesearch branch works for locally
built packages.  It's obviously very limited, the next thing we need to
do is sync the database from substitute servers.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: What’s next?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

Ludovic Courtès schreef op za 15-05-2021 om 19:47 [+0200]:
> Hello Guix!
> 
> So, now that 1.3.0 is out the door, what’s next?!
> 
> Here’s my wish list of things that look achievable within 4 to 6 months
> (I hope to help on some of these): [...]

Distributing substitutes over IPFS.

The original patch series dates from 2018
<https://issues.guix.gnu.org/33899>. We now have an IPFS service
(<https://issues.guix.gnu.org/45905>, merged) and a patch to expose the the gateway
and not only the API (<https://issues.guix.gnu.org/48299>, unmerged).
It seems the only part missing is the IPFS substituter (and relevant "guix publish"
code) itself.

The discussion on IPLD and Unixfs is rather esoteric to me, so I can't help
there.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: What’s next?

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On 2021-05-15, Ludovic Courtès wrote:
> So, now that 1.3.0 is out the door, what’s next?!
>
> Here’s my wish list of things that look achievable within 4 to 6 months
> (I hope to help on some of these):

   • Add a "make dist" job to ci.guix.gnu.org to produce prerelease
     source tarballs.

   • guix lint: support for spellchecker or basic grammar
     https://issues.guix.gnu.org/44675
     Let's have fewer spelling/typo/grammar issues end up in the next
     release tarballs! :)

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: What’s next?

[Svante Signell]

On Sat, 2021-05-15 at 19:47 +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> So, now that 1.3.0 is out the door, what’s next?!
> 
> Here’s my wish list of things that look achievable within 4 to 6
> months
> (I hope to help on some of these):

What about Hurd?

Re: What’s next?

[Leo Famulari]

On Sun, May 16, 2021 at 06:26:57PM +0200, Svante Signell wrote:
> What about Hurd?

I think the answer will be: what about it? :)

We always welcome Hurd-related work here in Guix.

Re: What’s next?

[Joshua Branson]

Leo Famulari <leo@famulari.name> writes:

> On Sun, May 16, 2021 at 06:26:57PM +0200, Svante Signell wrote:
>> What about Hurd?
>
> I think the answer will be: what about it? :)
>
> We always welcome Hurd-related work here in Guix.
>

I suppose someone should fix the Hurd vulnerabilities as reported here:

https://lists.gnu.org/archive/html/bug-hurd/2021-05/msg00079.html

I don't think the vulnerabilities have been disclosed yet nor has there
been a fix yet.

--
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar

Re: What’s next?

[Leo Famulari]

On Mon, May 17, 2021 at 10:35:22PM -0400, Joshua Branson wrote:
> I suppose someone should fix the Hurd vulnerabilities as reported here:
> 
> https://lists.gnu.org/archive/html/bug-hurd/2021-05/msg00079.html
> 
> I don't think the vulnerabilities have been disclosed yet nor has there
> been a fix yet.

That message is the disclosure.

Re: What’s next?

[raingloom]

On Tue, 18 May 2021 10:05:05 -0400
Leo Famulari <leo@famulari.name> wrote:

> On Mon, May 17, 2021 at 10:35:22PM -0400, Joshua Branson wrote:
> > I suppose someone should fix the Hurd vulnerabilities as reported
> > here:
> > 
> > https://lists.gnu.org/archive/html/bug-hurd/2021-05/msg00079.html
> > 
> > I don't think the vulnerabilities have been disclosed yet nor has
> > there been a fix yet.  
> 
> That message is the disclosure.
> 

Why not put our eggs in a few more baskets with way fewer holes and
more, uh, basket inspectors looking at them, like maybe packaging Minix,
or OpenBSD, or MirageOS, or whatever? I think I stretched that metaphor
but yknow what I mean.
They have seen way more scrutiny than Hurd and also run on more
architectures and while not GPL licensed, AFAIK they are all libre.

Maybe they can't be used in the operating-system-kernel struct field,
but I don't see anything wrong with using Guix to deploy Mirage
unikernel images for instance.
There is even a nascent Scheme unikernel project with Loko Scheme.

Ooor maybe compile some things to WASM and use a WASM+WASI runtime. I
hate webshit but at least there is already tooling and major porting
efforts for WASM.

Re: What’s next?

[Joshua Branson]

Some other cool thoughts:

Package the PHP bits of wordpress.  Then create a wordpress theme that
uses either no or bootstrapable (reproducible?) javascript.  Wordpress
writes some really minimal simple wordpress themes that have minimal
javascript that could be used as a start.  I think there might be a way
that one could then use Emacs to make a really simple blog.  And we
could have a really simple WordPress service.

Or just go crazy a create a competitor to wordpress in GNU guile or
racket.

Package the PHP bits of mediagoblin.  I think this is already done in
one of the WIP branches.  Then create a front-end of media goblin that
uses no or minimal amount bootstrapable (reproducible?) javascript.

One of the guix developers has a WIP git repo web viewer.  And it looks
amazing!  He's got some of his guile code on there. I think it would be
really awesome if savannah's guix git repos could point to that for the
guix repos.  Then we (who am I kidding you, I'm not a developer...yet)
could somehow combine bugs.guix.gnu.org with that guile git repo web
viewer.  The eventual goal could be to create a competitor to sourcehut!

--
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar

Re: What’s next?

[zimoun]

Hi,

>   • Completing and consolidating Disarchive support (see
>     <https://issues.guix.gnu.org/47336>): continuously building the
>     Disarchive database, making sure it’s replicated or backed up by
>     SWH, and having a blog post or two explaining the whole endeavor
>     (I’m looking at you, Timothy ;-)).

Adding items here:

 - SVN support.  Chunks are there [1] but missing glue.
 - other (Hg, CVS, etc.)
 - coverage (by the Data Service?)

 - guix time -C channels.scm -- help
   where channels.scm describes missing Git repo, fallback to SWH.

   (Corollary, “guix channel” subcommand helping to add, remove, save,
   etc.)

1: <http://issues.guix.gnu.org/43442#11>

>   • Merging Magali’s ‘guix git log’ work.

About reviewing, feedback welcome. :-) From my understanding, some
commits should be rewritten (split or merged).

In the same direction, I add 2 items:

 - guix time-machine --commit=<OID> / guix pull --commit=<OID>
 - guix git tag: for locally tagging a specific commit

On the next-next step, be able to download information (JSON?) from the
Data Service and use it to easy the “navigation” in the history.  


> What do people think?  What’s your wish list?  What do you feel an urge
> to hack on?  :-)

Add another item:

 - Julia importer (I am working on it, slowly! :-))

Cheers,
simon

PS:  I hope that I have not messed up the In-Reply-To. :-)

Re: What’s next?

[Bone Baboon]

Ludovic Courtès writes:
> So, now that 1.3.0 is out the door, what’s next?!

> What’s your wish list?  What do you feel an urge to hack on?  :-)

There are two improvements on my Guix wish list.

1) Make the core parts of Guix reproducible
** I do not know if this fits into the 4-6 month time frame mentioned.

2) Alternative kernel
** Motivated by 1.
** Longer term beyond 6 months.

1) Make the core parts of Guix reproducible

Many core parts of Guix are not reproducible.  If more core parts of
Guix were reproducible it would benefit all Guix users.

There are several core parts of Guix that are not reproducible
including:

* Linux-libre
  https://issues.guix.gnu.org/24028#2
  Note: I like what the Linux-libre project is doing.
  	This is likely a result of Linux not being reproducible.

* Many guix-*
  https://issues.guix.gnu.org/48487#0

* Guile
  https://issues.guix.gnu.org/48490#0

* nss 3.59 on the master branch
  https://issues.guix.gnu.org/40316#5

* Emacs
  https://issues.guix.gnu.org/35085#7
  Note: A good text editor is important.
        nvi, vim and neovim are reproducible for me.
	Emacs is more than a text editor and that is a part of why it is
        not reproducible. 

2) Alternative kernel

It is important to have a reproducible kernel.  Linux-libre is not
reproducible (see 1 above).  Linux-libre has not been reproducible for
an extended period of time.  Linux-libre not being reproducible was
reported in 2016 <https://issues.guix.gnu.org/24028#0>.

<https://odysee.com/@Lunduke:e/LinuxSucks2021:1> provides an interesting
thought exercise.  What free libre kernel would Guix use if Linux was
no longer a viable option?  I do not agree with all their points. The
point on Linux complexity increasing rapidly (13:29-17:56) is the one I
would be most concerned about.

Both Linux-libre not being reproducible and the idea that Linux might
not be viable in the future highlight the importance (and potential
urgency) of having an alternative free libre kernel that Guix can run
on.

It is great that work is already underway to get Guix to run on the Gnu
Hurd microkernel.

I think the design concept of a microkernel make them  more resistant to
the problem of increasing complexity at the kernel level when compared
to monolithic kernels.  With microkernels the increased complexity is
pushes out to user processes.  This allows the user (or their operating
system) to choose the level of complexity.

<http://www.microkernel.info/> is a listing of microkernel projects.

Re: What’s next?

[Bone Baboon]

Bone Baboon writes:
> 1) Make the core parts of Guix reproducible
>
> Many core parts of Guix are not reproducible.  If more core parts of
> Guix were reproducible it would benefit all Guix users.
>
> There are several core parts of Guix that are not reproducible
> including:
>
> * Linux-libre
>   https://issues.guix.gnu.org/24028#2
>   Note: I like what the Linux-libre project is doing.
>   	This is likely a result of Linux not being reproducible.
>
> * Many guix-*
>   https://issues.guix.gnu.org/48487#0
>
> * Guile
>   https://issues.guix.gnu.org/48490#0
>
> * nss 3.59 on the master branch
>   https://issues.guix.gnu.org/40316#5
>
> * Emacs
>   https://issues.guix.gnu.org/35085#7
>   Note: A good text editor is important.
>         nvi, vim and neovim are reproducible for me.
> 	Emacs is more than a text editor and that is a part of why it is
>         not reproducible. 

I have one addition to this.  mariadb is failing to build from source
because of a failing test suite.  As mariadb is a dependency of
diffoscope this is causing diffoscope's build from source to also fail.
diffoscope is a useful tool for gathering information about why a
package is not reproducible.

`guix graph --path diffoscope mariadb` outputs:

```
diffoscope@174
ffmpeg@4.3.2
sdl2@2.0.12
fcitx@4.2.9.8
extra-cmake-modules@5.70.0
qtbase@5.15.2
mariadb@10.5.8
```

More details on how mariadb is failing to build can be seen here
<https://issues.guix.gnu.org/48151#3>.

Re: What’s next?

[Ludovic Courtès]

Hi,

Bone Baboon <bone.baboon@disroot.org> skribis:

> 1) Make the core parts of Guix reproducible
>
> Many core parts of Guix are not reproducible.  If more core parts of
> Guix were reproducible it would benefit all Guix users.
>
> There are several core parts of Guix that are not reproducible
> including:
>
> * Linux-libre
>   https://issues.guix.gnu.org/24028#2
>   Note: I like what the Linux-libre project is doing.
>   	This is likely a result of Linux not being reproducible.
>
> * Many guix-*
>   https://issues.guix.gnu.org/48487#0
>
> * Guile
>   https://issues.guix.gnu.org/48490#0
>
> * nss 3.59 on the master branch
>   https://issues.guix.gnu.org/40316#5
>
> * Emacs
>   https://issues.guix.gnu.org/35085#7
>   Note: A good text editor is important.
>         nvi, vim and neovim are reproducible for me.
> 	Emacs is more than a text editor and that is a part of why it is
>         not reproducible. 

All these are great (apart from the Guile and guix-* ones, which are
“known” issues).  I think it takes investigation, to pinpoint the source
of non-determinism, and also checking what other distros do, possibly
asking on the <rb-general@lists.reproducible-builds.org>.

> 2) Alternative kernel

[...]

> It is great that work is already underway to get Guix to run on the Gnu
> Hurd microkernel.

Yup!  Help welcome on this front, too.  :-)

> I think the design concept of a microkernel make them  more resistant to
> the problem of increasing complexity at the kernel level when compared
> to monolithic kernels.  With microkernels the increased complexity is
> pushes out to user processes.  This allows the user (or their operating
> system) to choose the level of complexity.
>
> <http://www.microkernel.info/> is a listing of microkernel projects.

Note that once you have a microkernel, you have next to nothing, by
definition.  The Hurd (+ libc) provides everything you need to provide a
POSIX personality.

Thanks,
Ludo’.

Re: What’s next?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

Ludovic Courtès schreef op za 15-05-2021 om 19:47 [+0200]:
> [...]
> Here’s my wish list of things that look achievable within 4 to 6 months
> (I hope to help on some of these):
> 
>   • [...]
> 
>   • Merging ‘core-updates’, perhaps with a switch to GCC 10?  Perhaps
>     with support for “simplified packages” (getting rid of input
>     labels)?  We’ll see how much can go in there, but the sooner the
>     better.
> 
>   • On ‘core-updates’, merging either the full-source bootstrap or
>     the reduced bootstrap on ARM, whichever comes first.  :-)
> 
>   [...]
> 

While we are talking about ‘core-updates’, could someone have a look
at a patch series I posted exactly a month ago that fixes a class of
cross-compilation bugs? See <https://issues.guix.gnu.org/47869>.
(No replies so far.)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

What’s next?

[Ludovic Courtès]

Hello!

So, I think we now want to start working on a bootable distro.

However, I would add a less ambitious 0.3 milestone, first, ideally to
be released before July.  For 0.3, I would like to add actual
cross-compilation support.  If everything goes well, that would allow
the MIPS/N64 port to be bootstrap (and perhaps an ARM port too?).  0.3
would also add interesting packages (GTK+, for instance), and bug fixes.
WDYT?

Working on support for a bootable distro will require some thought and
design, and that should be started ASAP.  I’ll send an email in the next
few days to discuss my current thoughts.

Ludo’.

Re: What’s next?

[Andreas Enge]

Am Donnerstag, 16. Mai 2013 schrieb Ludovic Courtès:
> However, I would add a less ambitious 0.3 milestone, first, ideally to
> be released before July.  For 0.3, I would like to add actual
> cross-compilation support.  If everything goes well, that would allow
> the MIPS/N64 port to be bootstrap (and perhaps an ARM port too?).

This looks like an attractive goal to me.

> 0.3 would also add interesting packages (GTK+, for instance), and bug 
fixes.

I started some work on gtk+, with a few tentative recipes for cairo and 
pango. The cairo tests fail for the time being, and I will not be able to 
investigate during the next two weeks.

Andreas

Re: What’s next?

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Donnerstag, 16. Mai 2013 schrieb Ludovic Courtès:
>> However, I would add a less ambitious 0.3 milestone, first, ideally to
>> be released before July.  For 0.3, I would like to add actual
>> cross-compilation support.  If everything goes well, that would allow
>> the MIPS/N64 port to be bootstrap (and perhaps an ARM port too?).
>
> This looks like an attractive goal to me.

Good.  :-)

>> 0.3 would also add interesting packages (GTK+, for instance), and bug 
> fixes.
>
> I started some work on gtk+, with a few tentative recipes for cairo and 
> pango. The cairo tests fail for the time being, and I will not be able to 
> investigate during the next two weeks.

Could you perhaps commit Cairo with #:tests? #f for now?  That would
“unblock” a few packages, and hopefully one of us can fix it eventually.

Thanks,
Ludo’.

Re: What’s next?

[Ludovic Courtès]

Hi!

ludo@gnu.org (Ludovic Courtès) skribis:

> So, I think we now want to start working on a bootable distro.
>
> However, I would add a less ambitious 0.3 milestone, first, ideally to
> be released before July.  For 0.3, I would like to add actual
> cross-compilation support.  If everything goes well, that would allow
> the MIPS/N64 port to be bootstrap (and perhaps an ARM port too?).  0.3
> would also add interesting packages (GTK+, for instance), and bug fixes.
> WDYT?

It’s been almost two months since the 0.2 release.  We achieved some of
the important items above (cross-compilation, and cross-compilation of
the bootstrap tools); other items are not ready yet (MIPS port, GTK+);
there are also bonuses, such as debugging info support.

So I would like to merge ‘core-updates’ soon, and release 0.3 within one
or two weeks.

Thoughts?

Ludo’.