[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Hi!

With the release of Mes 0.21, the wip-bootstrap branch is ready for a
first review.  This is a big one, so I am opening this bug to discuss
and keep track of things.

Please note that wip-bootstrap uses development snapshots of Gash and
Gash Core Utils; we still expect some changes here and probably hard
resets of `wip-bootstrap'.

Gash has now reached pretty good posix compliance, which really helps to
run configure scripts and sh snippets in autotool-generated Makefiles.

Gash Core Utils currently comes with

   [, awk, basename, cat, chmod, cmp, compress, cp, cut, diff, dirname,
   egrep, expr, false, fgrep, find, gawk, grep, gzip, head, ln, ls,
   mkdir, mv, pwd, reboot, rm, rmdir, sed, sleep, sort, tar, test,
   touch, tr, true, uname, uniq, wc, and which.

that in comparison with Gash are much flakier.  While basic
functionality is generally OK, most Gash tools cannot be used to
bootstrap the entire system yet.  This holds especially for awk, grep,
and sed.

Between autoconf 2.61 (2006) and autoconf-2.63 (2008), much
functionality that was written in sed was moved to awk.  The NEWS file
states performance reasons.  Gash's sed is much mature than Gash's awk,
so currently it makes sense to target C versions of around 2004.

However, some of the 2004 C tools are not good enough anymore to
bootstrap the entire system; so we need a second round for them.  The
Mes C Library is another constraint.  With Mes 0.21 it gained support
for many early C tools; but more recent tools have C library
requirements that are not yet covered.

Another thing to note is that we do not have bzip2, lzip or xz and that
after 2009 some crucial tools (coreutils, diffutils, grep, sed, ...)
start shipping .xz or .lz tarballs only.  While bzip2 can be built early
in the bootstrap, I only managed to build xz with a fairly recent gcc
(4.6).

Even with these considerations, there still is quite some room to change
build order and versions of the C versions of coreutils&co.  The current
choices are mostly made by "what works".  We could invest in fixing
Gash's awk or sed or enrich the Mes C library or ..., if that seems a
helpful thing to do.

When we manage to merge this, we will have halved the bootstrap seed
again, reducing the bootstrap seed to under 60MB.

--8<---------------cut here---------------start------------->8---
10:40:02 janneke@dundal:~/tmp [env]
$ cd guix-sob/
10:40:04 janneke@dundal:~/tmp/guix-sob [env]
$ du -schx $(readlink $(~/src/guix/wip-bootstrap/pre-inst-env guix build bootstrap-tarballs)/*)
388K	/gnu/store/49giv6b94zbv2pjl6a9ycgy5ny9x3jbc-gash-bootstrap-guile-tarball-0.1-9.32188ac/gash-bootstrap-guile-0.1-9.32188ac-x86_64-linux.tar.xz
5.7M	/gnu/store/47qxvqs7rm7agdp86lxr2jzvval5hkqc-guile-static-stripped-tarball-2.2.6/guile-static-stripped-2.2.6-x86_64-linux.tar.xz
80K	/gnu/store/l6m2rmqs31mc9w7rl99xxl6m35x7fg6v-linux-libre-headers-stripped-tarball-4.19.56/linux-libre-headers-stripped-4.19.56-x86_64-linux.tar.xz
280K	/gnu/store/gnz5mlnzcb0nkaiyad5smblalfjvi8xc-mescc-tools-static-stripped-tarball-0.6.1/mescc-tools-static-stripped-0.6.1-x86_64-linux.tar.xz
352K	/gnu/store/p32p46x5iic0sff2wcf5gilddvfibb19-mes-minimal-stripped-tarball-0.21/mes-minimal-stripped-0.21-x86_64-linux.tar.xz
6.7M	total
10:40:14 janneke@dundal:~/tmp/guix-sob [env]
$ for i in $(readlink $(~/src/guix/wip-bootstrap/pre-inst-env guix build bootstrap-tarballs)/*); do sudo tar xf $i; done
10:45:29 janneke@dundal:~/tmp/guix-sob [env]
$ du -schx *
6.2M	bin
988K	include
42M	lib
8.3M	share
58M	total
--8<---------------cut here---------------end--------------->8---

Just when I thought the branch was functionally done; I already found one
small problem; I have pushed this ugly workaround

--8<---------------cut here---------------start------------->8---
bootstrap: ACL: disable tests.  FIXME: LD_PRELOAD.

Running

    ./pre-inst-env build hello

fails to build acl:

ERROR: ld.so: object '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so' from LD_PRELOAD cannot be preloaded: ignored. != ~
ERROR: ld.so: object '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so' from LD_PRELOAD cannot be preloaded: ignored. != ~
ERROR: ld.so: object '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so' from LD_PRELOAD cannot be preloaded: ignored. != ~
ERROR: ld.so: object '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so' from LD_PRELOAD cannot be preloaded: ignored. != ~
[7] $ rm large-file -- failed
ERROR: ld.so: object '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so' from LD_PRELOAD cannot be preloaded: ignored. != ~
4 commands (2 passed, 2 failed)
FAIL test/getfacl-lfs.test (exit status: 2)

* gnu/packages/acl.scm (acl): Disable tests.
--8<---------------cut here---------------end--------------->8---

It looks like a coreutils is getting built with a too early bootstrap
gcc/glibc, but I could use some help here.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Ludovic Courtès]

Hello Janneke & Timothy!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Gash has now reached pretty good posix compliance, which really helps to
> run configure scripts and sh snippets in autotool-generated Makefiles.
>
> Gash Core Utils currently comes with
>
>    [, awk, basename, cat, chmod, cmp, compress, cp, cut, diff, dirname,
>    egrep, expr, false, fgrep, find, gawk, grep, gzip, head, ln, ls,
>    mkdir, mv, pwd, reboot, rm, rmdir, sed, sleep, sort, tar, test,
>    touch, tr, true, uname, uniq, wc, and which.

Woow, excellent!  Do I get it right that “Gash Core Utils” refers to the
‘gash-core-utils’ branch at <https://gitlab.com/janneke/gash>?

So this is meant to be a Gash extension but separate from Gash.  Looks
really cool!

Do you plan to eventually merge it on Savannah?  (I expected to find the
code there.  :-)  BTW, I posted (unimportant) patches to
<https://lists.gnu.org/archive/html/gash-devel/2019-06/threads.html>.)

> that in comparison with Gash are much flakier.  While basic
> functionality is generally OK, most Gash tools cannot be used to
> bootstrap the entire system yet.  This holds especially for awk, grep,
> and sed.

(gash core-utils awk *) look quite fancy though!  I thought ‘configure’
only used a couple of trivial Awk snippets and after checking, I see
that there are in fact relatively fancy Awk programs in there.  Bah.

> Between autoconf 2.61 (2006) and autoconf-2.63 (2008), much
> functionality that was written in sed was moved to awk.  The NEWS file
> states performance reasons.  Gash's sed is much mature than Gash's awk,
> so currently it makes sense to target C versions of around 2004.
>
> However, some of the 2004 C tools are not good enough anymore to
> bootstrap the entire system; so we need a second round for them.  The
> Mes C Library is another constraint.  With Mes 0.21 it gained support
> for many early C tools; but more recent tools have C library
> requirements that are not yet covered.

How much work would be necessary at first sign to implement what’s
missing from (gash core-utils awk) to cover what modern-time ‘configure’
scripts need?

I suppose having a good(-enough) Awk implementation in Gash would be
more fruitful in the long run than reviving old C packages.  Notably, I
think you’d rather keep Mes’ libc as small as possible IMO, because it’s
cumbersome to write and maintain, which means more Scheme and less C.
But obviously, this all depends on the difficulty of implementing the
missing bits of Awk.

> Another thing to note is that we do not have bzip2, lzip or xz and that
> after 2009 some crucial tools (coreutils, diffutils, grep, sed, ...)
> start shipping .xz or .lz tarballs only.  While bzip2 can be built early
> in the bootstrap, I only managed to build xz with a fairly recent gcc
> (4.6).

At worst, we could host gzipped versions of these tarballs (or ask the
maintainers to do so).

Or we could have a fixed-output derivation that does the lzip->gzip
conversion (creating a “soft” circular dependency), which is kinda
equivalent to hosting a gzipped versions when substitutes are available.

Longer-term, we could also consider having a derivation built-in that
would allow us to “cheat” (i.e., take gz/lzip/xz/bzip2 for granted),
though it’s not so nice.

> Even with these considerations, there still is quite some room to change
> build order and versions of the C versions of coreutils&co.  The current
> choices are mostly made by "what works".  We could invest in fixing
> Gash's awk or sed or enrich the Mes C library or ..., if that seems a
> helpful thing to do.
>
> When we manage to merge this, we will have halved the bootstrap seed
> again, reducing the bootstrap seed to under 60MB.

Woohoo!

> Just when I thought the branch was functionally done; I already found one
> small problem; I have pushed this ugly workaround
>
> bootstrap: ACL: disable tests.  FIXME: LD_PRELOAD.
>
> Running
>
>     ./pre-inst-env build hello
>
> fails to build acl:
>
> ERROR: ld.so: object '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so' from LD_PRELOAD cannot be preloaded: ignored. != ~

We should have a debugging session for this one.  :-)

What about building the ‘core’ subset of ‘wip-bootstrap’ on berlin?
That would allow others to experiment (and debug!) without having to
build it all by themselves.

At some point we should consider “cleaning up” the history of that
branch.  For instance, I see commit “326d45561c gnu: Add gash.  WIP”,
which adds ‘guile-gash’ when there’s already a ‘gash’ package (and it
also modifies ‘jupyter-guile-kernel’).

We’ll also have to collectively review the new bootstrap tarballs.

Anyway, great perspectives and much excitement!  :-)

Thank you,
Ludo’.

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Timothy Sample]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Janneke & Timothy!
>
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Gash has now reached pretty good posix compliance, which really helps to
>> run configure scripts and sh snippets in autotool-generated Makefiles.
>>
>> Gash Core Utils currently comes with
>>
>>    [, awk, basename, cat, chmod, cmp, compress, cp, cut, diff, dirname,
>>    egrep, expr, false, fgrep, find, gawk, grep, gzip, head, ln, ls,
>>    mkdir, mv, pwd, reboot, rm, rmdir, sed, sleep, sort, tar, test,
>>    touch, tr, true, uname, uniq, wc, and which.
>
> Woow, excellent!  Do I get it right that “Gash Core Utils” refers to the
> ‘gash-core-utils’ branch at <https://gitlab.com/janneke/gash>?

For now (see below).

> So this is meant to be a Gash extension but separate from Gash.  Looks
> really cool!

My long term plan for it is to be a really, really souped-up version of
“(guix build utils)”.  Hopefully we can get reasonably nice Scheme
interfaces for common shell idioms and then build the utilities on top
of those interfaces.  Then, the library could be useful independent of
the external utilities, Gash, and bootstrapping.  It’s nowhere near
there yet, but that’s the goal.

> Do you plan to eventually merge it on Savannah?  (I expected to find the
> code there.  :-)

Yes!  I plan to add another repo as part of the Gash project.  This will
happen when it’s closer to an initial release.  (I’ll have to audit all
of the copyright notices, etc., which I find easier to do while getting
ready for a release.)

> BTW, I posted (unimportant) patches to
> <https://lists.gnu.org/archive/html/gash-devel/2019-06/threads.html>.)

I was not subscribed to gash-devel!  :P  I was an admin, but that
doesn’t mean I was subscribed.  Sorry!

I noticed the install path patch that you added to the Guix package, so
it’s already fixed.  So far I’ve been too busy to test on Guile 3....

> [...]

> How much work would be necessary at first sign to implement what’s
> missing from (gash core-utils awk) to cover what modern-time ‘configure’
> scripts need?
>
> I suppose having a good(-enough) Awk implementation in Gash would be
> more fruitful in the long run than reviving old C packages.  Notably, I
> think you’d rather keep Mes’ libc as small as possible IMO, because it’s
> cumbersome to write and maintain, which means more Scheme and less C.
> But obviously, this all depends on the difficulty of implementing the
> missing bits of Awk.

I agree this is the right way to go in the long run.  I’m pretty sure
we’ll get there, but I haven’t worked on the Awk implementation yet.

In around one week (with the usual caveats around software estimates), I
will make another release of Gash, at which point it will be “good
enough” for the bootstrapping task.  After that, I plan to turn my
attention to the other utilities.  There’s a lot of administrative work
and cleaning to do there, so it will take some time to get a release
out.  Once that happens I’ll move on to removing some intermediate
bootstrapping packages like early versions of Gawk, etc.


-- Tim

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Timothy Sample writes:

Hi Ludo, Timothy,

>>> Gash Core Utils currently comes with
>>>
>>>    [, awk, basename, cat, chmod, cmp, compress, cp, cut, diff, dirname,
>>>    egrep, expr, false, fgrep, find, gawk, grep, gzip, head, ln, ls,
>>>    mkdir, mv, pwd, reboot, rm, rmdir, sed, sleep, sort, tar, test,
>>>    touch, tr, true, uname, uniq, wc, and which.
>>
>> Woow, excellent!  Do I get it right that “Gash Core Utils” refers to the
>> ‘gash-core-utils’ branch at <https://gitlab.com/janneke/gash>?
>
> For now (see below).

Right.  I think that while Timothy mostly worked on the Gash core to
make it a full sh alternative, I have been working in parallel to add
all utilities that are needed during bootstrap.  That worked out pretty
OK, but now things need to come together again.

>> So this is meant to be a Gash extension but separate from Gash.  Looks
>> really cool!
>
> My long term plan for it is to be a really, really souped-up version of
> “(guix build utils)”.  Hopefully we can get reasonably nice Scheme
> interfaces for common shell idioms and then build the utilities on top
> of those interfaces.  Then, the library could be useful independent of
> the external utilities, Gash, and bootstrapping.  It’s nowhere near
> there yet, but that’s the goal.

Yes, a nice shell library for Guile was Rutger's initial goal for the
pre-merger Gash.  I have diverged quite a bit from that path by
focussing on the actual shell tools -- so while some of the library
functionality is starting to fall in place, an actual nice library
interface is still mostly missing...

>> Do you plan to eventually merge it on Savannah?  (I expected to find the
>> code there.  :-)
>
> Yes!  I plan to add another repo as part of the Gash project.  This will
> happen when it’s closer to an initial release.  (I’ll have to audit all
> of the copyright notices, etc., which I find easier to do while getting
> ready for a release.)

Great!

>> BTW, I posted (unimportant) patches to
>> <https://lists.gnu.org/archive/html/gash-devel/2019-06/threads.html>.)
>
> I was not subscribed to gash-devel!  :P  I was an admin, but that
> doesn’t mean I was subscribed.  Sorry!
>
> I noticed the install path patch that you added to the Guix package, so
> it’s already fixed.  So far I’ve been too busy to test on Guile 3....

Oh, I only subscribed in September 22nd...oops!

>> I suppose having a good(-enough) Awk implementation in Gash would be
>> more fruitful in the long run than reviving old C packages.  Notably, I
>> think you’d rather keep Mes’ libc as small as possible IMO, because it’s
>> cumbersome to write and maintain, which means more Scheme and less C.
>> But obviously, this all depends on the difficulty of implementing the
>> missing bits of Awk.
>
> I agree this is the right way to go in the long run.  I’m pretty sure
> we’ll get there, but I haven’t worked on the Awk implementation yet.

I think the parser is OK, and I also thought the Awk we needed was only
assignments and printing columns.  I started with a naive AST-interpreter
that worked a bit too well but has a flawed design (I don't remember the
details).  We need some good tests and probably a re-thinking and
re-implementatation of the `run-commands' and `awk-expression'
functions.

> In around one week (with the usual caveats around software estimates), I
> will make another release of Gash, at which point it will be “good
> enough” for the bootstrapping task.  After that, I plan to turn my
> attention to the other utilities.  There’s a lot of administrative work
> and cleaning to do there, so it will take some time to get a release
> out.  Once that happens I’ll move on to removing some intermediate
> bootstrapping packages like early versions of Gawk, etc.

Great!
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Ludovic Courtès]

Hi, Timothy!

Timothy Sample <samplet@ngyro.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> So this is meant to be a Gash extension but separate from Gash.  Looks
>> really cool!
>
> My long term plan for it is to be a really, really souped-up version of
> “(guix build utils)”.  Hopefully we can get reasonably nice Scheme
> interfaces for common shell idioms and then build the utilities on top
> of those interfaces.  Then, the library could be useful independent of
> the external utilities, Gash, and bootstrapping.  It’s nowhere near
> there yet, but that’s the goal.

I agree with the goal!  Having a nice Scheme interface for all these
things, perhaps with inspiration from scsh, would be great, and
certainly nicer than the very ad-hoc (guix build utils).

>> Do you plan to eventually merge it on Savannah?  (I expected to find the
>> code there.  :-)
>
> Yes!  I plan to add another repo as part of the Gash project.  This will
> happen when it’s closer to an initial release.  (I’ll have to audit all
> of the copyright notices, etc., which I find easier to do while getting
> ready for a release.)

Alright!

>> BTW, I posted (unimportant) patches to
>> <https://lists.gnu.org/archive/html/gash-devel/2019-06/threads.html>.)
>
> I was not subscribed to gash-devel!  :P  I was an admin, but that
> doesn’t mean I was subscribed.  Sorry!
>
> I noticed the install path patch that you added to the Guix package, so
> it’s already fixed.  So far I’ve been too busy to test on Guile 3....

Heheh, no problem.  :-)

> In around one week (with the usual caveats around software estimates), I
> will make another release of Gash, at which point it will be “good
> enough” for the bootstrapping task.  After that, I plan to turn my
> attention to the other utilities.  There’s a lot of administrative work
> and cleaning to do there, so it will take some time to get a release
> out.  Once that happens I’ll move on to removing some intermediate
> bootstrapping packages like early versions of Gawk, etc.

Sounds good, thanks a lot for all the work!

Ludo’.

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> (gash core-utils awk *) look quite fancy though!  I thought ‘configure’
> only used a couple of trivial Awk snippets and after checking, I see
> that there are in fact relatively fancy Awk programs in there.  Bah.

Yeah, that's what I thought...and then it grew.

> How much work would be necessary at first sign to implement what’s
> missing from (gash core-utils awk) to cover what modern-time ‘configure’
> scripts need?
>
> I suppose having a good(-enough) Awk implementation in Gash would be
> more fruitful in the long run than reviving old C packages.  Notably, I
> think you’d rather keep Mes’ libc as small as possible IMO, because it’s
> cumbersome to write and maintain, which means more Scheme and less C.
> But obviously, this all depends on the difficulty of implementing the
> missing bits of Awk.
>
>> Another thing to note is that we do not have bzip2, lzip or xz and that
>> after 2009 some crucial tools (coreutils, diffutils, grep, sed, ...)
>> start shipping .xz or .lz tarballs only.  While bzip2 can be built early
>> in the bootstrap, I only managed to build xz with a fairly recent gcc
>> (4.6).
>
> At worst, we could host gzipped versions of these tarballs (or ask the
> maintainers to do so).

Hmm.  Yes.  Somehow I would prefer if we could convince upstream to help
us bootstrap their package (dare I say to "do the right thing"?).

> Or we could have a fixed-output derivation that does the lzip->gzip
> conversion (creating a “soft” circular dependency), which is kinda
> equivalent to hosting a gzipped versions when substitutes are available.
>
> Longer-term, we could also consider having a derivation built-in that
> would allow us to “cheat” (i.e., take gz/lzip/xz/bzip2 for granted),
> though it’s not so nice.

Okay, that's many options.  Maybe we could brainstorm a bit about
possible attack vectors against the different "solutions", WDYT?

>> When we manage to merge this, we will have halved the bootstrap seed
>> again, reducing the bootstrap seed to under 60MB.
>
> Woohoo!

Yes, I feel so too; thanks!

>> ERROR: ld.so: object '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so' from LD_PRELOAD cannot be preloaded: ignored. != ~
>
> We should have a debugging session for this one.  :-)

I would love that; I looked into it today but there is some dependency
that I do not grok yet...

> What about building the ‘core’ subset of ‘wip-bootstrap’ on berlin?
> That would allow others to experiment (and debug!) without having to
> build it all by themselves.

Yes, that would be great!

> At some point we should consider “cleaning up” the history of that
> branch.  For instance, I see commit “326d45561c gnu: Add gash.  WIP”,
> which adds ‘guile-gash’ when there’s already a ‘gash’ package (and it
> also modifies ‘jupyter-guile-kernel’).

Hmm, I cannot find that commit; are you looking at `wip-bootstrap' on
savannah?  I remember something like that and probably rewrote it.

Anyway, the Gash and Gash Core Utils commits are marked WIP because I
want to replace them with a proper release by Timothy :-)

> We’ll also have to collectively review the new bootstrap tarballs.

Yes, sure.

> Anyway, great perspectives and much excitement!  :-)

Very happily enjoying the excitement you share, thanks :-)
Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

>>> ERROR: ld.so: object
>>> '/tmp/guix-build-acl-2.2.53.drv-0/acl-2.2.53/.libs/libtestlookup.so'
>>> from LD_PRELOAD cannot be preloaded: ignored. != ~
>>
>> We should have a debugging session for this one.  :-)
>
> I would love that; I looked into it today but there is some dependency
> that I do not grok yet...

Just a heads-up: In a productive and fun debugging session yesterday in
Marrakech Ludo and I got a fix in for this!  This means that I now
consider `wip-bootstrap' functionally correct.

Earlier this week, Ludo and I found a way to not add Gash or Gash Core
Utils to the Bootstrap Seed and instead build them as first packages
using %bootstrap-guile.

We will be working on a rewrite of wip-bootstrap to have it use Gash
wip-0.2.0+ and include a number of cleanups.

Here is my current TODO list

  * base bootstrap on Gash wip-0.20.0 (plus janneke's 2 patches)
  * remove %bootstrap-gash (with gash core utils) from bootstrap seed
  * look at possibility/cost to avoid updating the mescc-tools and mes
    bootstrap binaries
  * remove any generated (gitlab/github) tarballs
  * look into awkward combined bash+gash dependency of glibc-mesboot0
  * add some %bootX-input stages, at least when reached gcc-mesboot1
  * commit messages: Use "Use Gash instead of coretutils&co." rather than
    "Scheme-only bootstrap."
  * some smaller cleanups and nitpicks here and there

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Ludovic Courtès]

Hello!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Just a heads-up: In a productive and fun debugging session yesterday in
> Marrakech Ludo and I got a fix in for this!  This means that I now
> consider `wip-bootstrap' functionally correct.
>
> Earlier this week, Ludo and I found a way to not add Gash or Gash Core
> Utils to the Bootstrap Seed and instead build them as first packages
> using %bootstrap-guile.
>
> We will be working on a rewrite of wip-bootstrap to have it use Gash
> wip-0.2.0+ and include a number of cleanups.

It looks like this is all shaping up nicely, and I’m happy to see we’ve
already (!) reached that point where we can again strip some of the
binary seeds!  Kudos, comrades!

Ludo’.

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

Hi!

> We will be working on a rewrite of wip-bootstrap to have it use Gash
> wip-0.2.0+ and include a number of cleanups.

I have just hard-reset wip-bootstrap for the next iteration!

> Here is my current TODO list
>
>   * base bootstrap on Gash wip-0.20.0 (plus janneke's 2 patches)

Done.  I just built `hello', so Gash 0.2.0 here we come (afaic).

>   * remove %bootstrap-gash (with gash core utils) from bootstrap seed

Done; thanks to Ludo's insights about %boostrap-guile+guild, we were
able to do this.  The `gash-boot' and `gash-core-utils-boot' packages no
longer appear in shells.scm; they are built during bootstrap using the
guile-build-system.  I currently add lalr.upstream as an extra origin
to gash core utils from

--8<---------------cut here---------------start------------->8---
"http://git.savannah.gnu.org/cgit/guile.git/plain/module/system/base/lalr.upstream.scm?h=v2.0.9"
--8<---------------cut here---------------end--------------->8---

>   * look at possibility/cost to avoid updating the mescc-tools and mes
>     bootstrap binaries

This proved possible (dare I say feasible?).  What was needed is

 - allow mescc (0.21+) to work with old mescc-tools 0.5.2
 - add %bootstrap-mes-rewired, to remove MesCC from mes-0.19 and
   enable it to run 0.21+ MesCC

%bootstrap-mes-rewired uses this hack
--8<---------------cut here---------------start------------->8---
         (add-after 'unpack 'patch-%moduledir
           (lambda _
             (copy-file "module/mescc.scm" "module/mescc.scm.orig")
             (substitute* "module/mescc.scm"
               (("^  \\(mes-use-module \\(mescc mescc\\)" all)
                (string-append "
  ;; MesCC from mes-0.21
  (let* ((self (car (command-line)))
         (prefix (dirname (dirname self))))
    (set! %moduledir (string-append prefix \"/mes/module/\"))
    (setenv \"%numbered_arch\" \"true\"))

  ;; A fixed map, from mes-0.21
  (define (map f h . t)
    (if (or (null? h)
            (and (pair? t) (null? (car t)))
            (and (pair? t) (pair? (cdr t)) (null? (cadr t)))) '()
        (if (null? t) (cons (f (car h)) (map f (cdr h)))
            (if (null? (cdr t))
                (cons (f (car h) (caar t)) (map f (cdr h) (cdar t)))
                (if (null? (cddr t))
                    (cons (f (car h) (caar t) (caadr t)) (map f (cdr h) (cdar t) (cdadr t)))
                    (if (null? (cdddr t))
                        (cons (f (car h) (caar t) (caadr t) (car (caddr t))) (map f (cdr h) (cdar t) (cdadr t) (cdr (caddr t))))
                        (error 'unsupported (cons* 'map-5: f h t))) )))))
" all)))
             #t))
--8<---------------cut here---------------end--------------->8---

...not particularly nice/clean/auditable; but "it works".  WDYT?
On a related note, should mescc 0.21 include some kind of `hook' make
this kind of change easier to make?

This snippet could also be pushed "upstream", to mes-0.21+...but meh...

>   * remove any generated (gitlab/github) tarballs

Done (but please check!).

>   * look into awkward combined bash+gash dependency of glibc-mesboot0

Haven't addressed this.  I quickly looked with Ludo at this, not really
into it though.  WYDT?

>   * add some %bootX-input stages, at least when reached gcc-mesboot1

Done.  Numbering of bootX, mesbootX could possibly made to make more
sense.  However, I also would like to get rid of the whole 2.95 story
some time soon and then many things change again.  I could do with some
help/inspiration here.

>   * commit messages: Use "Use Gash instead of coretutils&co." rather than
>     "Scheme-only bootstrap."

Done.

>   * some smaller cleanups and nitpicks here and there

Removed "ed-1.4" again (using a revert commit, that we can drop).

I kept most earlier commits that add to the bootstrap seed and added
revert commits.  If we like what we have, we can remove both the commits
that introduced these and their reverts.

We are really getting there!  I would like to do a rewrite once Gash
0.2.0 is out.  What to do about Gash Core Utils?  I also plan to do a
rewrite once MES 0.22 is out.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

Howdy!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

>> We will be working on a rewrite of wip-bootstrap to have it use Gash
>> wip-0.2.0+ and include a number of cleanups.
>
> I have just hard-reset wip-bootstrap for the next iteration!

Yay!

>> Here is my current TODO list
>>
>>   * base bootstrap on Gash wip-0.20.0 (plus janneke's 2 patches)
>
> Done.  I just built `hello', so Gash 0.2.0 here we come (afaic).
>
>>   * remove %bootstrap-gash (with gash core utils) from bootstrap seed
>
> Done; thanks to Ludo's insights about %boostrap-guile+guild, we were
> able to do this.  The `gash-boot' and `gash-core-utils-boot' packages no
> longer appear in shells.scm; they are built during bootstrap using the
> guile-build-system.  I currently add lalr.upstream as an extra origin
> to gash core utils from
>
> "http://git.savannah.gnu.org/cgit/guile.git/plain/module/system/base/lalr.upstream.scm?h=v2.0.9"

Awesome.

BTW, I’d suggest this for clarity:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 641 bytes --]

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 9e3cecc174..d4cf58a7eb 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -266,6 +266,7 @@
                       ,(origin
                          (method url-fetch)
                          (uri (string-append "http://git.savannah.gnu.org/cgit/guile.git/plain/module/system/base/lalr.upstream.scm?h=v2.0.9"))
+                         (file-name "lalr.upstream.scm")
                          (sha256
                           (base32
                            "0h7gyjj8nr2qrgzwma146s7l22scp8bbcqzdy9wqf12bgyhbw7d5"))))))

[-- Attachment #3: Type: text/plain, Size: 4820 bytes --]


>>   * look at possibility/cost to avoid updating the mescc-tools and mes
>>     bootstrap binaries
>
> This proved possible (dare I say feasible?).  What was needed is
>
>  - allow mescc (0.21+) to work with old mescc-tools 0.5.2
>  - add %bootstrap-mes-rewired, to remove MesCC from mes-0.19 and
>    enable it to run 0.21+ MesCC
>
> %bootstrap-mes-rewired uses this hack
>
>          (add-after 'unpack 'patch-%moduledir
>            (lambda _
>              (copy-file "module/mescc.scm" "module/mescc.scm.orig")
>              (substitute* "module/mescc.scm"
>                (("^  \\(mes-use-module \\(mescc mescc\\)" all)
>                 (string-append "
>   ;; MesCC from mes-0.21
>   (let* ((self (car (command-line)))
>          (prefix (dirname (dirname self))))
>     (set! %moduledir (string-append prefix \"/mes/module/\"))
>     (setenv \"%numbered_arch\" \"true\"))
>
>   ;; A fixed map, from mes-0.21
>   (define (map f h . t)
>     (if (or (null? h)
>             (and (pair? t) (null? (car t)))
>             (and (pair? t) (pair? (cdr t)) (null? (cadr t)))) '()
>         (if (null? t) (cons (f (car h)) (map f (cdr h)))
>             (if (null? (cdr t))
>                 (cons (f (car h) (caar t)) (map f (cdr h) (cdar t)))
>                 (if (null? (cddr t))
>                     (cons (f (car h) (caar t) (caadr t)) (map f (cdr h) (cdar t) (cdadr t)))
>                     (if (null? (cdddr t))
>                         (cons (f (car h) (caar t) (caadr t) (car (caddr t))) (map f (cdr h) (cdar t) (cdadr t) (cdr (caddr t))))
>                         (error 'unsupported (cons* 'map-5: f h t))) )))))
> " all)))
>              #t))
>
> ...not particularly nice/clean/auditable; but "it works".  WDYT?

‘caddr’, loooove it!  ;-)

It seems to me that mescc.scm in 0.21 only use the two-argument ‘map’,
no?  In that case I guess we could go with a two-argument, fixed-arity
version that would have fewer cdadrs caddrs and caadrs?  Just sayin’.
:-)

> On a related note, should mescc 0.21 include some kind of `hook' make
> this kind of change easier to make?

Good question!  During the summit, Vagrant (I think?) asked whether Mes
and MesCC should be separated.  I think in this case it would have been
beneficial to have them distributed separately, no?  It might be worth
looking at which parts change frequently to determine what to keep as
part of the ‘mes’ package itself.

Thoughts?

>>   * remove any generated (gitlab/github) tarballs
>
> Done (but please check!).

LGTM!

>>   * look into awkward combined bash+gash dependency of glibc-mesboot0
>
> Haven't addressed this.  I quickly looked with Ludo at this, not really
> into it though.  WYDT?

Hmm, dunno.  I can take a look later.

>>   * add some %bootX-input stages, at least when reached gcc-mesboot1
>
> Done.  Numbering of bootX, mesbootX could possibly made to make more
> sense.  However, I also would like to get rid of the whole 2.95 story
> some time soon and then many things change again.  I could do with some
> help/inspiration here.

I think it’s good to have ‘%bootX-inputs’ for when we’re going to reuse
more or less the same input set several times, because it clarifies that
there’s a “plateau” of sorts, but I guess it’s not always appropriate
early on in the graph.

Another thing we discussed was the growth of the dependency graph:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix graph -e '(@@ (gnu packages commencement) coreutils-final)' | grep 'label =' | wc -l
177
$ guix graph -e '(@@ (gnu packages commencement) coreutils-final)' | grep 'label =' | wc -l
76
--8<---------------cut here---------------end--------------->8---

Clearly we’re adding way more nodes than we mean to.

For example, you can remove the call to ‘package-with-bootstrap-guile’
for ‘bash-mesboot’ (down to 153 nodes) and the one for
‘gcc-core-mesboot1’ (down to 121) and the one for ‘binutils-mesboot’
(down to 87), and I think at this point we’re done and the graph has a
lovely shape.  :-)

It makes a noticeable difference from a performance viewpoint (see
<https://lists.gnu.org/archive/html/guix-devel/2019-10/msg00350.html>.)

Also, you can leave out calls to ‘bootstrap-origin’ for origins without
a snippet and without patches (it’s purely stylistic though.)

> We are really getting there!  I would like to do a rewrite once Gash
> 0.2.0 is out.  What to do about Gash Core Utils?  I also plan to do a
> rewrite once MES 0.22 is out.

I think we should probably wait for the Gash and Gash Core Utils
releases so we can refer to them directly.  How does that sound to you,
Timothy?

Thanks!

Ludo’.

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Timothy Sample]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> We are really getting there!  I would like to do a rewrite once Gash
>> 0.2.0 is out.  What to do about Gash Core Utils?  I also plan to do a
>> rewrite once MES 0.22 is out.
>
> I think we should probably wait for the Gash and Gash Core Utils
> releases so we can refer to them directly.  How does that sound to you,
> Timothy?

I’m planning on releasing Gash tonight, so that should be okay.  :D

For the utilities, it might take some time (especially since we are
running up against the holidays).  My current plan is to do it as
quickly as possible, meaning that I will focus on the bare minimum for a
first release.  Even then, it won’t be speedy: besides cleaning up
what’s there and setting up the build scripts, I will have to coordinate
with the folks at Savannah to get another Git repo.

Even if it takes a few weeks for a release, I think it makes sense to
wait.  This whole project has taken years!  What’s a couple of weeks?
:)


-- Tim

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Brett Gilio]

Timothy Sample <samplet@ngyro.com> writes:

>
> I’m planning on releasing Gash tonight, so that should be okay.  :D


Yes!

-- 
Brett M. Gilio
Homepage -- https://scm.pw/
GNU Guix -- https://guix.gnu.org/

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Timothy Sample writes:

Hi All,

>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>> We are really getting there!  I would like to do a rewrite once Gash
>>> 0.2.0 is out.  What to do about Gash Core Utils?  I also plan to do a
>>> rewrite once MES 0.22 is out.
>>
>> I think we should probably wait for the Gash and Gash Core Utils
>> releases so we can refer to them directly.  How does that sound to you,
>> Timothy?
>
> I’m planning on releasing Gash tonight, so that should be okay.  :D

\o/

> For the utilities, it might take some time (especially since we are
> running up against the holidays).  My current plan is to do it as
> quickly as possible, meaning that I will focus on the bare minimum for a
> first release.  Even then, it won’t be speedy: besides cleaning up
> what’s there and setting up the build scripts, I will have to coordinate
> with the folks at Savannah to get another Git repo.
>
> Even if it takes a few weeks for a release, I think it makes sense to
> wait.  This whole project has taken years!  What’s a couple of weeks?
> :)

I agree; no need to hurry, there is time.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

>> I have just hard-reset wip-bootstrap for the next iteration!
>
> Yay!

and I did it again.

> Awesome.
>
> BTW, I’d suggest this for clarity:
> +                         (file-name "lalr.upstream.scm")

Ah, yes.  Added.

>>>   * look at possibility/cost to avoid updating the mescc-tools and mes
>>>     bootstrap binaries
>>
>> This proved possible (dare I say feasible?).  What was needed is
>>
>>  - allow mescc (0.21+) to work with old mescc-tools 0.5.2
>>  - add %bootstrap-mes-rewired, to remove MesCC from mes-0.19 and
>>    enable it to run 0.21+ MesCC
>>
>> %bootstrap-mes-rewired uses this hack
>>   ;; MesCC from mes-0.21
>>     (set! %moduledir (string-append prefix \"/mes/module/\"))
>>
>>   ;; A fixed map, from mes-0.21
>>   (define (map f h . t)
>
> ‘caddr’, loooove it!  ;-)
>
> It seems to me that mescc.scm in 0.21 only use the two-argument ‘map’,
> no?  In that case I guess we could go with a two-argument, fixed-arity
> version that would have fewer cdadrs caddrs and caadrs?  Just sayin’.
> :-)

Ah yes...well almost.  MesCC uses map3; so I cut only map4.

>> On a related note, should mescc 0.21 include some kind of `hook' make
>> this kind of change easier to make?
>
> Good question!  During the summit, Vagrant (I think?) asked whether Mes
> and MesCC should be separated.  I think in this case it would have been
> beneficial to have them distributed separately, no?  It might be worth
> looking at which parts change frequently to determine what to keep as
> part of the ‘mes’ package itself.
>
> Thoughts?

I've been "struggling" with this question a bit myself.  In this
particular instance it would not have helped much, since map was
broken and I would not have wanted to add a fix for that in mescc.

It would have helped switching to the newer mescc.  Keeping them in one
archive for now is a bit less work for me I think; it saves me managing
another dependency.  Possibly until mes can boot guile modules.  Until
mes can do that, mes needs to come with a "shadow *.mes" tree for module
inclusion, like it has for Nyacc too.  That's less than great, but
works...

I am open to other perspectives, though.

>>>   * look into awkward combined bash+gash dependency of glibc-mesboot0
>>
>> Haven't addressed this.  I quickly looked with Ludo at this, not really
>> into it though.  WYDT?
>
> Hmm, dunno.  I can take a look later.

Okay, great.  This issue still remains.  I will try to create a bug
report for Gash, I think Gash hangs while running configure, while
bash-mesboot* have trouble running make-syscalls.sh correctly.

I just realise that the mixture of bash/gawk/sed here is possibly one of
the things that was replaced by Python.  Bootstrapping-wise that seemed
like a very bad idea, but I may be starting to see the sanity in such a
choice.  It would be great if it were Guile, though, or else if Guile
could run this Python.

> Another thing we discussed was the growth of the dependency graph:
>
> $ ./pre-inst-env guix graph -e '(@@ (gnu packages commencement) coreutils-final)' | grep 'label =' | wc -l
> 177
> $ guix graph -e '(@@ (gnu packages commencement) coreutils-final)' | grep 'label =' | wc -l
> 76
>
> Clearly we’re adding way more nodes than we mean to.
>
> For example, you can remove the call to ‘package-with-bootstrap-guile’
> for ‘bash-mesboot’ (down to 153 nodes) and the one for
> ‘gcc-core-mesboot1’ (down to 121) and the one for ‘binutils-mesboot’
> (down to 87), and I think at this point we’re done and the graph has a
> lovely shape.  :-)

Oh, thank you!  Incorporated those changes and verified I got the 87
nodes too.

> Also, you can leave out calls to ‘bootstrap-origin’ for origins without
> a snippet and without patches (it’s purely stylistic though.)

Ah, I didn't realise that.  Removed most, if not all of them.

> I think we should probably wait for the Gash and Gash Core Utils
> releases so we can refer to them directly.  How does that sound to you,
> Timothy?

Incoroprated Gash 0.2.0!

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

Hi,

A new step forward.

>>>>   * look into awkward combined bash+gash dependency of glibc-mesboot0
>>>
>>> Haven't addressed this.  I quickly looked with Ludo at this, not really
>>> into it though.  WYDT?
>>
>> Hmm, dunno.  I can take a look later.
>
> Okay, great.  This issue still remains.  I will try to create a bug
> report for Gash, I think Gash hangs while running configure, while
> bash-mesboot* have trouble running make-syscalls.sh correctly.

Good news.  bash-mesboot0 now compiles with either gash+gash-core-utils,
or with bash-mesboot0.

Gash' "test -L FILE" used to crash on non-existing files, not sure why
that made configure hang; but that's how I found and fixed it.

The problem with bash-mesboot0 turned out to be a Mes C Library problem,
related to buffered reads.  Buffered reads were introduced when working
on the Hurd.

Not clearing the read buffer on lseek, when lseek is not allowed (bash
uses the same: lseek (FD, 0, SEEK_CUR) to find out if it may seek),
fixes the problem.  That took me a couple of days to find, but very
happy 

--8<---------------cut here---------------start------------->8---
diff --git a/lib/linux/lseek.c b/lib/linux/lseek.c
index 94f2f9f7a..f71af59f5 100644
--- a/lib/linux/lseek.c
+++ b/lib/linux/lseek.c
@@ -24,9 +24,21 @@
 #include <stdio.h>
 #include <sys/types.h>
 
+#if !__MESC__ /* FIXME: We want bin/mes-mescc's x86-linux sha256sum to stay the same. */
+off_t
+_lseek (int filedes, off_t offset, int whence)
+{
+  return _sys_call3 (SYS_lseek, (int) filedes, (long) offset, (int) whence);
+}
+#endif
+
 off_t
 lseek (int filedes, off_t offset, int whence)
 {
+#if !__MESC__ /* FIXME: We want bin/mes-mescc's x86-linux sha256sum to stay the same. */
+  if (_lseek (filedes, 0, SEEK_CUR) == -1)
+    return -1;
+#endif
   size_t skip = __buffered_read_clear (filedes);
   if (whence == SEEK_CUR)
     offset -= skip;
--8<---------------cut here---------------end--------------->8---

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap

[Ludovic Courtès]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Good news.  bash-mesboot0 now compiles with either gash+gash-core-utils,
> or with bash-mesboot0.
>
> Gash' "test -L FILE" used to crash on non-existing files, not sure why
> that made configure hang; but that's how I found and fixed it.
>
> The problem with bash-mesboot0 turned out to be a Mes C Library problem,
> related to buffered reads.  Buffered reads were introduced when working
> on the Hurd.
>
> Not clearing the read buffer on lseek, when lseek is not allowed (bash
> uses the same: lseek (FD, 0, SEEK_CUR) to find out if it may seek),
> fixes the problem.  That took me a couple of days to find, but very
> happy 

Woow, wild!  Great that you found out!

Thank you,
Ludo’.

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Timothy Sample]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

Hi Ludo,

(CC’ing Jan and bug#38390.)

We’re nearing a release for Gash-Utils, and while working on it I made
some changes to how the bootstrap Gash and Gash-Utils are built (on the
wip-bootstrap branch.)  Jan suggested that we get your opinion on the
changes, as you are behind the current '%bootstrap-guile+guild'
approach.

The very short story is that I found that the Scheme GZip code is being
maintained and works out-of-the-box on Guile 2.0.9 [1].  This made me
unsure about copying the code into Gash-Utils when maybe we could just
use it as a dependency.  Then, I realized that the library included a
simple Tar reader, and wondered how simple a program could be that could
handle 'tar xvf gash.tar.gz'.  It turns out pretty simple!  So I put the
program and all its dependencies in an a-list with a little loop that
writes them to disk, and made a self-extracting Scheme script that can
unpack compressed tarballs [2].

[1] https://github.com/weinholt/compression
[2] https://git.ngyro.com/bootar/

The other thing is that I always intended for Gash and Gash-Utils to be
built with a loop calling “compile-file”.  This avoids the need for
“guild” which in turn avoids “bash” (AIUI).

These patches update Gash-Utils, replace the binary “tar” with my
self-extracting Scheme implementation, and replace “guild” with
“compile-file”.  I think this simplifies the bootstrap processes, and I
really like getting rid of the references to “tar”, “bash”, and “xz”
(even though they are needed for the bootstrap Guile, it feels nice to
quarantine them there).

WDYT?


-- Tim


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Rename-gash-core-utils-to-gash-utils.patch --]
[-- Type: text/x-patch, Size: 2413 bytes --]

From 33614593c2a212ac99cc126755efebcb0574909c Mon Sep 17 00:00:00 2001
From: Timothy Sample <samplet@ngyro.com>
Date: Mon, 3 Feb 2020 11:26:35 -0500
Subject: [PATCH 1/3] Rename 'gash-core-utils' to 'gash-utils'.

* gnu/packages/shells.scm (gash-core-utils): Rename this...
(gash-utils): ...to this.
* gnu/packages/commencement.scm (gash-core-utils): Rename this...
(gash-utils): ...to this and update the parent package reference.
(%boot-gash-inputs): Update reference.
---
 gnu/packages/commencement.scm | 10 +++++-----
 gnu/packages/shells.scm       |  4 ++--
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 1c62fcfae7..43533b4525 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -89,7 +89,7 @@
 (define %bootstrap-guile+guild
   ;; This package combines %bootstrap-guile with guild, which is not included
   ;; in %bootstrap-guile.  Guild is needed to build gash-boot and
-  ;; gash-core-utils-boot because it is dependency of the Guile build system.
+  ;; gash-utils-boot because it is dependency of the Guile build system.
   (package
     (name "guile-bootstrap+guild")
     (version "2.0")
@@ -225,10 +225,10 @@
                (install-file "scripts/bash" bin)
                #t))))))))
 
-(define gash-core-utils-boot
+(define gash-utils-boot
   (package
-    (inherit gash-core-utils)
-    (name "gash-core-utils-boot")
+    (inherit gash-utils)
+    (name "gash-utils-boot")
     (version "0.0.214-fc1b")
     (source (bootstrap-origin
              (origin
@@ -387,7 +387,7 @@
 
 (define (%boot-gash-inputs)
   `(("bash" , gash-boot)                ; gnu-build-system wants "bash"
-    ("coreutils" , gash-core-utils-boot)
+    ("coreutils" , gash-utils-boot)
     ("guile" ,%bootstrap-guile)
     ("guile+guild" ,%bootstrap-guile+guild)))
 
diff --git a/gnu/packages/shells.scm b/gnu/packages/shells.scm
index 3d2975fe14..ea61e7b2f8 100644
--- a/gnu/packages/shells.scm
+++ b/gnu/packages/shells.scm
@@ -827,9 +827,9 @@ library for parsing shell scripts.  Gash is designed to bootstrap Bash
 as part of the Guix bootstrap process.")
     (license gpl3+)))
 
-(define-public gash-core-utils
+(define-public gash-utils
   (package
-    (name "gash-core-utils")
+    (name "gash-utils")
     (version "0.0.211-789c3")
     (source (origin
               (method url-fetch)
-- 
2.24.1


[-- Attachment #3: 0002-Simplify-bootstrap-Gash-and-Gash-Utils.patch --]
[-- Type: text/x-patch, Size: 26252 bytes --]

From 2cec50928a4ff67df363322d2adfb6aaa5aedc83 Mon Sep 17 00:00:00 2001
From: Timothy Sample <samplet@ngyro.com>
Date: Mon, 3 Feb 2020 10:51:07 -0500
Subject: [PATCH 2/3] Simplify bootstrap Gash and Gash-Utils.

This change does three things.  First Gash-Utils is updated to
0.1.0-preview.  Then, the bootstrap Gash and Gash-Utils packages are
arranged to be built without using 'guild'.  Finally, instead of
using a binary 'tar' via 'bootstrap-executable' to extract Gash and
Gash-Utils, a self-extracting Scheme implementation of 'tar' and
'gzip' is used instead.

These three changes need to happen in concert to avoid breaking the
bootstrap process.

* gnu/packages/shells.scm (gash-utils): Update to 0.1.0-preview.
* gnu/packages/commencement.scm (make-bootstrap-phases): New procedure.
(bootar): New variable.
(gash-boot): Replace 'tar', and 'xz' with 'bootar'; remove 'bash' and
'guile-source'; do not use 'guile-build-system' but rather
'gnu-build-system' with its phases processed by 'make-bootstrap-phases';
disable tests.
(gash-utils-boot): Likewise.
(%boot-gash-inputs): Add 'bootar'; replace '%bootstrap-guile+guild' with
'%bootstrap-guile'.
(mes-boot): In the configure phase, do not reference 'GUILE_LOAD_PATH'
as it is no longer set.
---
 gnu/packages/commencement.scm | 431 ++++++++++++++++------------------
 gnu/packages/shells.scm       |   8 +-
 2 files changed, 206 insertions(+), 233 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 43533b4525..914b3f2a8d 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -7,6 +7,7 @@
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2018, 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2019 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -53,7 +54,6 @@
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
-  #:use-module (guix build-system guile)
   #:use-module (guix build-system trivial)
   #:use-module (guix memoization)
   #:use-module (guix utils)
@@ -144,252 +144,226 @@
             (variable "GUILE_LOAD_COMPILED_PATH")
             (files '("lib/guile/2.0/site-ccache")))))))
 
+(define (make-bootstrap-phases version scripts modules)
+  "Create a form that modifies the standard GNU build phases so that
+they build simple Guile programs using only the bootstrap Guile.  The
+'.in' files in the directory MODULES are configured with VERSION, the
+'.in' files in the directory SCRIPTS are configured with the bootstrap
+Guile and its module and object directories, and the Scheme files in the
+directory MODULES are compiled and installed."
+  `(modify-phases %standard-phases
+     (replace 'configure
+       (lambda* (#:key inputs outputs #:allow-other-keys)
+         (let* ((out (assoc-ref outputs "out"))
+                (guile-dir (assoc-ref inputs "guile"))
+                (guile (string-append guile-dir "/bin/guile"))
+                (moddir (string-append out "/share/guile/site/"
+                                       (effective-version)))
+                (godir (string-append out "/lib/guile/"
+                                      (effective-version)
+                                      "/site-ccache")))
+           (for-each (lambda (template)
+                       (format #t "Configuring ~a~%" template)
+                       (let ((target (string-drop-right template 3)))
+                         (copy-file template target)
+                         (substitute* target
+                           (("@VERSION@") ,version))))
+                     (find-files ,modules
+                                 (lambda (fn st)
+                                   (string-suffix? ".in" fn))))
+           (for-each (lambda (template)
+                       (format #t "Configuring ~a~%" template)
+                       (let ((target (string-drop-right template 3)))
+                         (copy-file template target)
+                         (substitute* target
+                           (("@GUILE@") guile)
+                           (("@MODDIR@") moddir)
+                           (("@GODIR@") godir))
+                         (chmod target #o755)))
+                     (find-files ,scripts
+                                 (lambda (fn st)
+                                   (string-suffix? ".in" fn))))
+           #t)))
+     (replace 'build
+       (lambda _
+         (add-to-load-path (getcwd))
+         (for-each (lambda (scm)
+                     (let* ((base (string-drop-right scm 4))
+                            (go (string-append base ".go"))
+                            (dir (dirname scm)))
+                       (format #t "Compiling ~a~%" scm)
+                       (compile-file scm #:output-file go)))
+                   (find-files ,modules "\\.scm$"))
+         #t))
+     (replace 'install
+       (lambda* (#:key inputs outputs #:allow-other-keys)
+         (let* ((out (assoc-ref outputs "out"))
+                (guile-dir (assoc-ref inputs "guile"))
+                (guile (string-append guile-dir "/bin/guile"))
+                (moddir (string-append out "/share/guile/site/"
+                                       (effective-version)))
+                (godir (string-append out "/lib/guile/"
+                                      (effective-version)
+                                      "/site-ccache")))
+           (for-each (lambda (scm)
+                       (let* ((base (string-drop-right scm 4))
+                              (go (string-append base ".go"))
+                              (dir (dirname scm)))
+                         (format #t "Installing ~a~%" scm)
+                         (install-file scm (string-append moddir "/" dir))
+                         (format #t "Installing ~a~%" go)
+                         (install-file go (string-append godir "/" dir))))
+                     (find-files ,modules "\\.scm$"))
+           (for-each (lambda (script)
+                       (format #t "Installing ~a~%" script)
+                       (install-file script (string-append out "/bin")))
+                     (find-files ,scripts
+                                 (lambda (fn st)
+                                   (executable-file? fn))))
+           #t)))))
+
+(define bootar
+  (package
+    (name "bootar")
+    (version "1")
+    (source (origin
+              (method url-fetch)
+              (uri "https://files.ngyro.com/tmp/bootar-1.ses")
+              (sha256
+               (base32
+                "0f37f1lav7y9nf2c6s783z7w4jfrpqrgcksqb028al82f0dn1glv"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:implicit-inputs? #f
+       #:tests? #f
+       #:guile ,%bootstrap-guile
+       #:phases
+       (modify-phases ,(make-bootstrap-phases version "scripts" ".")
+         (replace 'unpack
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let* ((source (assoc-ref inputs "source"))
+                    (guile-dir (assoc-ref inputs "guile"))
+                    (guile (string-append guile-dir "/bin/guile")))
+               (invoke guile "--no-auto-compile" source)
+               (chdir "bootar")
+               #t))))))
+    (inputs `(("guile" ,%bootstrap-guile)))
+    (home-page "https://git.ngyro.com/bootar")
+    (synopsis "Tar decompression and extraction in Guile Scheme")
+    (description "Bootar is a simple Tar extractor written in Guile
+Scheme.  It supports running 'tar xvf' on uncompressed tarballs or
+tarballs that are compressed with BZip2, GZip, or XZ.  It also provides
+standalone scripts for 'bzip2', 'gzip', and 'xz' that each support
+decompression to standard output.
+
+What makes this special is that Bootar is distributed as a
+self-extracting Scheme (SES) program.  That is, a little script that
+outputs the source code of Bootar.  This makes it possible to go from
+pure Scheme to Tar and decompression in one easy step.")
+    (license gpl3+)))
+
 (define gash-boot
   (package
     (inherit gash)
     (name "gash-boot")
-    (version "0.2.0")
-    (source (bootstrap-origin
-             (origin (inherit (package-source gash))
-                     (modules '((guix build utils)
-                                (srfi srfi-26)))
-                     (snippet
-                      '(begin
-                         ;; Remove Guix'y files that we cannot compile.
-                         (delete-file "guix.scm")
-                         (delete-file-recursively "tests")
-                         #t)))))
-    (build-system guile-build-system)
-    (native-inputs `(("bash" ,(bootstrap-executable "bash" (%current-system)))
-                     ("tar" ,(bootstrap-executable "tar" (%current-system)))
-                     ("xz" ,(bootstrap-executable "xz" (%current-system)))
-                     ("guile-source" ,(bootstrap-origin
-                                       (package-source guile-2.0)))))
-    (inputs `(("guile" ,%bootstrap-guile+guild)))
+    (source (origin
+              (inherit (package-source gash))
+              (modules '())))
     (arguments
      `(#:implicit-inputs? #f
-       #:guile ,%bootstrap-guile+guild
+       #:tests? #f
+       #:guile ,%bootstrap-guile
        #:phases
-       (modify-phases %standard-phases
-         (add-after 'unpack 'unpack-guile-source
-           (lambda _
-             (let ((guile-source (assoc-ref %build-inputs "guile-source"))
-                   (bin (string-append (getcwd) "/zee-bin")))
-               (mkdir-p bin)
-               (with-directory-excursion bin
-                 (invoke "tar" "--strip-components=2"
-
-                         "-xvf" guile-source
-                         (string-append "guile-"
-                                        ,(package-version guile-2.0)
-                                        "/meta/guild.in"))
-                 (copy-file "guild.in" "guild")
-                 (chmod "guild" #o555))
-               #t)))
-         (add-before 'unpack 'set-path
-           (lambda _
-             (let ((bash (assoc-ref %build-inputs "bash"))
-                   (tar (assoc-ref %build-inputs "tar"))
-                   (xz (assoc-ref %build-inputs "xz"))
-                   (bin (string-append (getcwd) "/zee-bin")))
-               (mkdir-p bin)
-               (setenv "PATH" (string-append bin ":" (getenv "PATH")))
-               (copy-file bash (string-append bin "/bash"))
-               (copy-file bash (string-append bin "/sh"))
-               (copy-file tar (string-append bin "/tar"))
-               (copy-file xz (string-append bin "/xz"))
-               #t)))
-         (add-after 'build 'build-scripts
-           (lambda _
-             (let* ((guile (assoc-ref %build-inputs "guile"))
-                    (guile (string-append guile "/bin/guile"))
-                    (out (assoc-ref %outputs "out"))
-                    (effective "2.0")
-                    (moddir (string-append out "/share/guile/site/" effective "/"))
-                    (godir (string-append out "/lib/guile/" effective "/site-ccache/")))
-               (copy-file "scripts/gash.in" "scripts/gash")
-               (chmod "scripts/gash" #o555)
-               (substitute* "scripts/gash"
-                 (("@GUILE@") guile)
-                 (("@MODDIR@") moddir)
-                 (("@GODIR") godir))
-               #t)))
-         (add-after 'install 'install-scripts
-           (lambda _
-             (let* ((out (assoc-ref %outputs "out"))
-                    (bin (string-append out "/bin")))
-               (install-file "scripts/gash" bin)
-               (copy-file "scripts/gash" "scripts/sh")
-               (install-file "scripts/sh" bin)
-               (copy-file "scripts/gash" "scripts/bash")
-               (install-file "scripts/bash" bin)
-               #t))))))))
+       (modify-phases ,(make-bootstrap-phases (version) "scripts" "gash")
+         (add-after 'install 'install-symlinks
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((out (assoc-ref outputs "out")))
+               (symlink (string-append out "/bin/gash")
+                        (string-append out "/bin/sh"))
+               (symlink (string-append out "/bin/gash")
+                        (string-append out "/bin/bash"))
+               #t))))))
+    (inputs `(("guile" ,%bootstrap-guile)))
+    (native-inputs `(("bootar" ,bootar)))))
 
 (define gash-utils-boot
   (package
     (inherit gash-utils)
     (name "gash-utils-boot")
-    (version "0.0.214-fc1b")
-    (source (bootstrap-origin
-             (origin
-               (method url-fetch)
-               (uri (string-append "http://lilypond.org/janneke/"
-                                   "/gash-core-utils-" version ".tar.gz"))
-               (modules '((guix build utils)))
-               (snippet
-                '(begin
-                   ;; The Guile build system compiles *.scm; avoid
-                   ;; compiling included lalr.
-                   (delete-file "guix.scm")
-                   (delete-file-recursively "tests")
-                   (substitute* "system/base/lalr.scm"
-                     (("system/base/lalr.upstream.scm") "lalr.upstream.scm"))
-                   #t))
-               (sha256
-                (base32
-                 "090d8m0b165jf9381nhqpljc5zk22jd6bw03xnyf0na5snk9xp6v")))))
-    (build-system guile-build-system)
-    (native-inputs `(("bash" ,(bootstrap-executable "bash" (%current-system)))
-                     ("tar" ,(bootstrap-executable "tar" (%current-system)))
-                     ("xz" ,(bootstrap-executable "xz" (%current-system)))
-                     ("guile-source" ,(bootstrap-origin
-                                       (package-source guile-2.0)))
-                     ;; We need the 2.0.9 lalr for %bootstrap-guile
-                     ("lalr.upstream"
-                      ,(origin
-                         (method url-fetch)
-                         (uri (string-append "http://git.savannah.gnu.org/cgit/guile.git/plain/module/system/base/lalr.upstream.scm?h=v2.0.9"))
-                         (file-name "lalr.upstream.scm")
-                         (sha256
-                          (base32
-                           "0h7gyjj8nr2qrgzwma146s7l22scp8bbcqzdy9wqf12bgyhbw7d5"))))))
-    (inputs `(("guile" ,%bootstrap-guile+guild)
-              ("gash" ,gash-boot)))
     (arguments
      `(#:implicit-inputs? #f
-       #:guile ,%bootstrap-guile+guild
-       #:not-compiled-file-regexp "upstream\\.scm$"
+       #:tests? #f
+       #:guile ,%bootstrap-guile
        #:phases
-       (modify-phases %standard-phases
-         (add-after 'unpack 'unpack-guile-source
-           (lambda _
-             (let ((guile-source (assoc-ref %build-inputs "guile-source"))
-                   (bin (string-append (getcwd) "/zee-bin")))
-               (mkdir-p bin)
-               (with-directory-excursion bin
-                 (invoke "tar" "--strip-components=2"
-
-                         "-xvf" guile-source
-                         (string-append "guile-"
-                                        ,(package-version guile-2.0)
-                                        "/meta/guild.in"))
-                 (copy-file "guild.in" "guild")
-                 (chmod "guild" #o555))
-               #t)))
-         (add-before 'unpack 'set-path
-           (lambda _
-             (let ((bash (assoc-ref %build-inputs "bash"))
-                   (tar (assoc-ref %build-inputs "tar"))
-                   (xz (assoc-ref %build-inputs "xz"))
-                   (bin (string-append (getcwd) "/zee-bin")))
-               (mkdir-p bin)
-               (setenv "PATH" (string-append bin ":" (getenv "PATH")))
-               (copy-file bash (string-append bin "/bash"))
-               (copy-file bash (string-append bin "/sh"))
-               (copy-file tar (string-append bin "/tar"))
-               (copy-file xz (string-append bin "/xz"))
-               #t)))
-         (add-before 'build 'set-env
-           (lambda _
-             (let ((gash (assoc-ref %build-inputs "gash")))
-               (setenv "LANG" "C")
-               (setenv "LC_ALL" "C")
-               (setenv "GUILE_LOAD_PATH"
-                       (string-append (getcwd)
-                                      ":" (getcwd) "/system/base"
-                                      ":" gash "/share/guile/2.0"))
-               (setenv "GUILE_LOAD_COMPILED_PATH"
-                       (string-append ".:" gash "/lib/guile/2.0/site-ccache/"))
-               (format (current-error-port)
-                       "GUILE_LOAD_PATH=~s\n" (getenv "GUILE_LOAD_PATH"))
-               #t)))
-         (add-before 'build 'replace-lalr.upstream
-           (lambda _
-             (let ((lalr.upstream (assoc-ref %build-inputs "lalr.upstream")))
-               (copy-file lalr.upstream "system/base/lalr.upstream.scm")
-               #t)))
-         (add-after 'build 'build-scripts
+       (modify-phases ,(make-bootstrap-phases (version) "scripts" "gash")
+         (add-after 'unpack 'set-load-path
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let ((gash (assoc-ref inputs "gash")))
+               (add-to-load-path (string-append gash "/share/guile/site/"
+                                                (effective-version))))
+             #t))
+         (add-before 'configure 'pre-configure
            (lambda _
-             (let* ((guile (assoc-ref %build-inputs "guile"))
-                    (guile (string-append guile "/bin/guile"))
-                    (gash (string-append guile "gash"))
-                    (out (assoc-ref %outputs "out"))
-                    (effective "2.0")
-                    (guilemoduledir (string-append gash "/share/guile/site/" effective "/"))
-                    (guileobjectdir (string-append gash "/lib/guile/" effective "/site-ccache/"))
-                    (gashmoduledir (string-append out "/share/guile/site/" effective "/"))
-                    (gashobjectdir (string-append out "/lib/guile/" effective "/site-ccache/"))
-                    (bin (string-append out "/bin")))
-               (define (wrap name)
-                 (copy-file "command.in" name)
-                 (chmod name #o555)
-                 (substitute* name
-                   (("@GUILE@") guile)
-                   (("@guilemoduledir@") guilemoduledir)
-                   (("@guileobjectdir") guileobjectdir)
-                   (("@gashmoduledir@") gashmoduledir)
-                   (("@gashobjectdir") gashobjectdir)
-                   (("@command@") name))
-                 (install-file name bin))
-               (mkdir-p bin)
-               (with-directory-excursion "bin"
-                 (for-each wrap '("awk"
-                                  "basename"
-                                  "cat"
-                                  "chmod"
-                                  "cmp"
-                                  "compress"
-                                  "cp"
-                                  "cut"
-                                  "diff"
-                                  "dirname"
-                                  "expr"
-                                  "false"
-                                  "find"
-                                  "grep"
-                                  "gzip"
-                                  "head"
-                                  "ln"
-                                  "ls"
-                                  "mkdir"
-                                  "mv"
-                                  "pwd"
-                                  "reboot"
-                                  "rm"
-                                  "rmdir"
-                                  "sed"
-                                  "sleep"
-                                  "sort"
-                                  "tar"
-                                  "test"
-                                  "touch"
-                                  "tr"
-                                  "true"
-                                  "uname"
-                                  "uniq"
-                                  "wc"
-                                  "which")))
-               (with-directory-excursion bin
-                 (copy-file "grep" "fgrep")
-                 (copy-file "grep" "egrep")
-                 (copy-file "test" "["))
-               #t))))))))
+             (format #t "Creating gash/commands/testb.scm~%")
+             (copy-file "gash/commands/test.scm"
+                        "gash/commands/testb.scm")
+             (substitute* "gash/commands/testb.scm"
+               (("gash commands test") "gash commands testb")
+               (("apply test [(]cdr") "apply test/bracket (cdr"))
+             (for-each (lambda (script)
+                         (let ((target (string-append "scripts/"
+                                                      script ".in")))
+                           (format #t "Creating scripts/~a~%" target)
+                           (copy-file "scripts/template.in" target)
+                           (substitute* target
+                             (("@UTILITY@") script))))
+                       '("awk" "basename" "cat" "chmod" "cmp" "command"
+                         "compress" "cp" "cut" "diff" "dirname" "expr"
+                         "false" "find" "grep" "head" "ln" "ls" "mkdir"
+                         "mv" "printf" "pwd" "reboot" "rm" "rmdir"
+                         "sed" "sleep" "sort" "tar" "test" "touch" "tr"
+                         "true" "uname" "uniq" "wc" "which"))
+             (format #t "Creating scripts/[.in~%")
+             (copy-file "scripts/template.in" "scripts/[.in")
+             (substitute* "scripts/[.in"
+               (("@UTILITY@") "testb"))
+             #t))
+         ;; XXX: The scripts should add Gash to their load paths and
+         ;; this phase should not exist.  Although ultimately, Gash
+         ;; should depend on Gash-Utils, so that it can treat its
+         ;; commands as built-ins.
+         (add-after 'install 'copy-gash
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (moddir (string-append out "/share/guile/site/"
+                                           (effective-version)))
+                    (godir (string-append out "/lib/guile/"
+                                          (effective-version)
+                                          "/site-ccache"))
+                    (gash (assoc-ref inputs "gash"))
+                    (gash-moddir (string-append gash "/share/guile/site/"
+                                                (effective-version)))
+                    (gash-godir (string-append gash "/lib/guile/"
+                                               (effective-version)
+                                               "/site-ccache")))
+               (copy-file (string-append gash-moddir "/gash/compat.scm")
+                          (string-append moddir "/gash/compat.scm"))
+               (copy-recursively (string-append gash-moddir "/gash/compat")
+                                 (string-append moddir "/gash/compat"))
+               (copy-file (string-append gash-godir "/gash/compat.go")
+                          (string-append godir "/gash/compat.go"))
+               (copy-recursively (string-append gash-godir "/gash/compat")
+                                 (string-append godir "/gash/compat"))
+               #t))))))
+    (inputs `(("gash" ,gash-boot)
+              ("guile" ,%bootstrap-guile)))
+    (native-inputs `(("bootar" ,bootar)))))
 
 (define (%boot-gash-inputs)
   `(("bash" , gash-boot)                ; gnu-build-system wants "bash"
     ("coreutils" , gash-utils-boot)
-    ("guile" ,%bootstrap-guile)
-    ("guile+guild" ,%bootstrap-guile+guild)))
+    ("bootar" ,bootar)
+    ("guile" ,%bootstrap-guile)))
 
 (define %bootstrap-mes-rewired
   (package
@@ -542,8 +516,7 @@ $MES -e '(mescc)' module/mescc.scm -- \"$@\"
                (setenv "GUILE_LOAD_PATH"
                        (string-append
                         mes "/share/mes/module"
-                        ":" dir "/nyacc-0.99.0/module"
-                        ":" (getenv "GUILE_LOAD_PATH")))
+                        ":" dir "/nyacc-0.99.0/module"))
                (invoke "gash" "configure.sh"
                        (string-append "--prefix=" out)
                        (string-append "--host=i686-linux-gnu")))))
diff --git a/gnu/packages/shells.scm b/gnu/packages/shells.scm
index ea61e7b2f8..718908e5ce 100644
--- a/gnu/packages/shells.scm
+++ b/gnu/packages/shells.scm
@@ -830,14 +830,14 @@ as part of the Guix bootstrap process.")
 (define-public gash-utils
   (package
     (name "gash-utils")
-    (version "0.0.211-789c3")
+    (version "0.1.0-preview")
     (source (origin
               (method url-fetch)
-              (uri (string-append "http://lilypond.org/janneke/guix/20191123/"
-                                  "/gash-core-utils-" version ".tar.gz"))
+              (uri (string-append "https://files.ngyro.com/tmp/gash-utils-"
+                                  version ".tar.gz"))
               (sha256
                (base32
-                "1ycf0ivyn9q3lc9hwq8vxg6wkwpsykvh48a5lrzsmganh0ma8wrr"))))
+                "1d6rmhhsgqbwjbi0fp383ac6gnxqd8yaq43mbc63xalni13g3mbc"))))
     (build-system gnu-build-system)
     (native-inputs
      `(("autoconf" ,autoconf)
-- 
2.24.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Remove-bootstrap-guile-guild.patch --]
[-- Type: text/x-patch, Size: 3809 bytes --]

From b363e2c3dfcf2906b181c0120e35f6b701455122 Mon Sep 17 00:00:00 2001
From: Timothy Sample <samplet@ngyro.com>
Date: Mon, 3 Feb 2020 11:46:25 -0500
Subject: [PATCH 3/3] Remove %bootstrap-guile+guild.

* gnu/packages/commencement.scm (%bootstrap-guile+guild): Remove variable.
---
 gnu/packages/commencement.scm | 58 -----------------------------------
 1 file changed, 58 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 914b3f2a8d..b797af1585 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -86,64 +86,6 @@
 ;;;
 ;;; Code:
 
-(define %bootstrap-guile+guild
-  ;; This package combines %bootstrap-guile with guild, which is not included
-  ;; in %bootstrap-guile.  Guild is needed to build gash-boot and
-  ;; gash-utils-boot because it is dependency of the Guile build system.
-  (package
-    (name "guile-bootstrap+guild")
-    (version "2.0")
-    (source (bootstrap-origin (package-source guile-2.0)))
-    (native-inputs `(("bash" ,(bootstrap-executable "bash" (%current-system)))
-                     ("tar" ,(bootstrap-executable "tar" (%current-system)))
-                     ("xz" ,(bootstrap-executable "xz" (%current-system)))
-                     ("guile" ,%bootstrap-guile)))
-    (build-system trivial-build-system)
-    (arguments
-     `(#:guile ,%bootstrap-guile
-       #:modules ((guix build utils))
-       #:builder (begin
-                   (use-modules (guix build utils))
-                   (let ((guile-source (assoc-ref %build-inputs "source"))
-                         (bin (string-append (getcwd) "/bin"))
-                         (tar (assoc-ref %build-inputs "tar"))
-                         (xz (assoc-ref %build-inputs "xz")))
-                     (mkdir-p bin)
-                     (setenv "PATH" bin)
-                     (with-directory-excursion bin
-                       (copy-file tar "tar")
-                       (copy-file xz "xz")
-                       (setenv "PATH" bin))
-                     (let* ((out (assoc-ref %outputs "out"))
-                            (out-bin (string-append out "/bin"))
-                            (guile (assoc-ref %build-inputs "guile"))
-                            (bash (assoc-ref %build-inputs "bash")))
-                       (mkdir-p out-bin)
-                       (with-directory-excursion out-bin
-                         (symlink (string-append guile "/bin/guile")
-                                  "guile")
-                         (invoke "tar" "--strip-components=2"
-                                 "-xvf" guile-source
-                                 (string-append "guile-"
-                                                ,(package-version guile-2.0)
-                                                "/meta/guild.in"))
-                         (copy-file "guild.in" "guild")
-                         (substitute* "guild"
-                           (("#!/bin/sh") (string-append "#! " bash))
-                           (("@installed_guile@") (string-append out-bin "/guile")))
-                         (chmod "guild" #o555)))))))
-    (synopsis "Bootstrap Guile plus Guild")
-    (description "Bootstrap Guile with added Guild")
-    (home-page #f)
-    (license (package-license guile-2.0))
-    (native-search-paths
-     (list (search-path-specification
-            (variable "GUILE_LOAD_PATH")
-            (files '("share/guile/site/2.0")))
-           (search-path-specification
-            (variable "GUILE_LOAD_COMPILED_PATH")
-            (files '("lib/guile/2.0/site-ccache")))))))
-
 (define (make-bootstrap-phases version scripts modules)
   "Create a form that modifies the standard GNU build phases so that
 they build simple Guile programs using only the bootstrap Guile.  The
-- 
2.24.1

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Ludovic Courtès]

Hello Timothy!

Timothy Sample <samplet@ngyro.com> skribis:

> We’re nearing a release for Gash-Utils, and while working on it I made
> some changes to how the bootstrap Gash and Gash-Utils are built (on the
> wip-bootstrap branch.)  Jan suggested that we get your opinion on the
> changes, as you are behind the current '%bootstrap-guile+guild'
> approach.

Thanks for reaching out to me!

> The very short story is that I found that the Scheme GZip code is being
> maintained and works out-of-the-box on Guile 2.0.9 [1].  This made me
> unsure about copying the code into Gash-Utils when maybe we could just
> use it as a dependency.  Then, I realized that the library included a
> simple Tar reader, and wondered how simple a program could be that could
> handle 'tar xvf gash.tar.gz'.  It turns out pretty simple!  So I put the
> program and all its dependencies in an a-list with a little loop that
> writes them to disk, and made a self-extracting Scheme script that can
> unpack compressed tarballs [2].

Woow, that is very nice!  So all of a sudden there’s an extra bunch of
binary seeds we can get rid of, woohoo!  I didn’t expect that even bzip2
and xz would be implemented.

> [2] https://git.ngyro.com/bootar/

Clean, short, and elegant; brilliant!

> The other thing is that I always intended for Gash and Gash-Utils to be
> built with a loop calling “compile-file”.  This avoids the need for
> “guild” which in turn avoids “bash” (AIUI).
>
> These patches update Gash-Utils, replace the binary “tar” with my
> self-extracting Scheme implementation, and replace “guild” with
> “compile-file”.  I think this simplifies the bootstrap processes, and I
> really like getting rid of the references to “tar”, “bash”, and “xz”
> (even though they are needed for the bootstrap Guile, it feels nice to
> quarantine them there).
>
> WDYT?

I think it’s perfect.

One comment that can be addressed later:

> From 2cec50928a4ff67df363322d2adfb6aaa5aedc83 Mon Sep 17 00:00:00 2001
> From: Timothy Sample <samplet@ngyro.com>
> Date: Mon, 3 Feb 2020 10:51:07 -0500
> Subject: [PATCH 2/3] Simplify bootstrap Gash and Gash-Utils.
>
> This change does three things.  First Gash-Utils is updated to
> 0.1.0-preview.  Then, the bootstrap Gash and Gash-Utils packages are
> arranged to be built without using 'guild'.  Finally, instead of
> using a binary 'tar' via 'bootstrap-executable' to extract Gash and
> Gash-Utils, a self-extracting Scheme implementation of 'tar' and
> 'gzip' is used instead.

[...]

> +(define (make-bootstrap-phases version scripts modules)
> +  "Create a form that modifies the standard GNU build phases so that
> +they build simple Guile programs using only the bootstrap Guile.  The
> +'.in' files in the directory MODULES are configured with VERSION, the
> +'.in' files in the directory SCRIPTS are configured with the bootstrap
> +Guile and its module and object directories, and the Scheme files in the
> +directory MODULES are compiled and installed."
> +  `(modify-phases %standard-phases
> +     (replace 'configure

Should this be factorized out in a (guix build gnu-bootstrap) module or
similar?  That would keep build-side code separate and would avoid
making ‘commencement.scm’ bigger.

I guess the only thing that remains to be done is changing the temporary
URLs to the self-extracting script & co., right?

Thanks a lot!

Ludo’.

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Timothy Sample]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Timothy Sample <samplet@ngyro.com> skribis:
>
> [...]
>
>> So I put the program and all its dependencies in an a-list with a
>> little loop that writes them to disk, and made a self-extracting
>> Scheme script that can unpack compressed tarballs [2].
>
> Woow, that is very nice!  So all of a sudden there’s an extra bunch of
> binary seeds we can get rid of, woohoo!  I didn’t expect that even bzip2
> and xz would be implemented.

I don’t think we could get rid of any binaries just yet.  We need Guile
to run Bootar and we can’t get Guile without the statically linked
“tar”, “xz”, “mkdir”, and “bash”.  This just removes some references to
them.  Although, it occurred to me that we could get something like
Bootar (perhaps further simplified) to run on Mes, in which case we
could use a statically linked Mes to unpack and wrap Guile.  That would
let us get rid of “%bootstrap-executables”.  (Note however that I tried
running Bootar in Mes for fun, and the extractor script – once it was
simplified – caused a segfault.)

BZip2 and XZ just barely work, by the way.  I implemented BZip2, but I
skipped over all the CRC checking.  XZ has a bug when called with input
from stdin, which is how Gash-Utils tries to call it.

>> [2] https://git.ngyro.com/bootar/
>
> Clean, short, and elegant; brilliant!

Thanks!

>> [...]
>>
>> WDYT?
>
> I think it’s perfect.

\o/

> One comment that can be addressed later:
>
>> From 2cec50928a4ff67df363322d2adfb6aaa5aedc83 Mon Sep 17 00:00:00 2001
>> From: Timothy Sample <samplet@ngyro.com>
>> Date: Mon, 3 Feb 2020 10:51:07 -0500
>> Subject: [PATCH 2/3] Simplify bootstrap Gash and Gash-Utils.
>>
>> This change does three things.  First Gash-Utils is updated to
>> 0.1.0-preview.  Then, the bootstrap Gash and Gash-Utils packages are
>> arranged to be built without using 'guild'.  Finally, instead of
>> using a binary 'tar' via 'bootstrap-executable' to extract Gash and
>> Gash-Utils, a self-extracting Scheme implementation of 'tar' and
>> 'gzip' is used instead.
>
> [...]
>
>> +(define (make-bootstrap-phases version scripts modules)
>> +  "Create a form that modifies the standard GNU build phases so that
>> +they build simple Guile programs using only the bootstrap Guile.  The
>> +'.in' files in the directory MODULES are configured with VERSION, the
>> +'.in' files in the directory SCRIPTS are configured with the bootstrap
>> +Guile and its module and object directories, and the Scheme files in the
>> +directory MODULES are compiled and installed."
>> +  `(modify-phases %standard-phases
>> +     (replace 'configure
>
> Should this be factorized out in a (guix build gnu-bootstrap) module or
> similar?  That would keep build-side code separate and would avoid
> making ‘commencement.scm’ bigger.

I would be happy to do that.  It’s nice having everything in one place,
but having a bootstrap build system would certainly make the packages
clearer.  I suppose it could also get rid of the implicit inputs for us
and use “%bootstrap-guile” by default.

> I guess the only thing that remains to be done is changing the temporary
> URLs to the self-extracting script & co., right?

Yup.  I’ll release both packages soon.

> Thanks a lot!

My pleasure!


-- Tim

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Ludovic Courtès]

Hello!

Timothy Sample <samplet@ngyro.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Woow, that is very nice!  So all of a sudden there’s an extra bunch of
>> binary seeds we can get rid of, woohoo!  I didn’t expect that even bzip2
>> and xz would be implemented.
>
> I don’t think we could get rid of any binaries just yet.  We need Guile
> to run Bootar and we can’t get Guile without the statically linked
> “tar”, “xz”, “mkdir”, and “bash”.  This just removes some references to
> them.  Although, it occurred to me that we could get something like
> Bootar (perhaps further simplified) to run on Mes, in which case we
> could use a statically linked Mes to unpack and wrap Guile.  That would
> let us get rid of “%bootstrap-executables”.  (Note however that I tried
> running Bootar in Mes for fun, and the extractor script – once it was
> simplified – caused a segfault.)

Oh cool.  Being able to run Guile code on Mes sounds like a worthy goal
longer-term anyway, so hopefully we’ll get there!

> BZip2 and XZ just barely work, by the way.  I implemented BZip2, but I
> skipped over all the CRC checking.  XZ has a bug when called with input
> from stdin, which is how Gash-Utils tries to call it.

Heh, not so bad.

>>> +(define (make-bootstrap-phases version scripts modules)
>>> +  "Create a form that modifies the standard GNU build phases so that
>>> +they build simple Guile programs using only the bootstrap Guile.  The
>>> +'.in' files in the directory MODULES are configured with VERSION, the
>>> +'.in' files in the directory SCRIPTS are configured with the bootstrap
>>> +Guile and its module and object directories, and the Scheme files in the
>>> +directory MODULES are compiled and installed."
>>> +  `(modify-phases %standard-phases
>>> +     (replace 'configure
>>
>> Should this be factorized out in a (guix build gnu-bootstrap) module or
>> similar?  That would keep build-side code separate and would avoid
>> making ‘commencement.scm’ bigger.
>
> I would be happy to do that.  It’s nice having everything in one place,
> but having a bootstrap build system would certainly make the packages
> clearer.  I suppose it could also get rid of the implicit inputs for us
> and use “%bootstrap-guile” by default.

I doesn’t have to be a full-blown build system, because there might be
as much boilerplate as actual code, but simply moving the definitions of
phases in a separate file could help keep commencement.scm clear.

>> I guess the only thing that remains to be done is changing the temporary
>> URLs to the self-extracting script & co., right?
>
> Yup.  I’ll release both packages soon.

Great, thank you!

Ludo’.

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello!

I have rebased wip-bootstrap on core-updates that include your
commencements updates.

With your explanations on irc and studying your patch, I was able to
keep using your cleanups and do without the most ugly 'setenv stages.

So, wip-bootstrap on savannah is reset once again.

> Timothy Sample <samplet@ngyro.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:

>> I would be happy to do that.  It’s nice having everything in one place,
>> but having a bootstrap build system would certainly make the packages
>> clearer.  I suppose it could also get rid of the implicit inputs for us
>> and use “%bootstrap-guile” by default.
>
> I doesn’t have to be a full-blown build system, because there might be
> as much boilerplate as actual code, but simply moving the definitions of
> phases in a separate file could help keep commencement.scm clear.
>
>>> I guess the only thing that remains to be done is changing the temporary
>>> URLs to the self-extracting script & co., right?
>>
>> Yup.  I’ll release both packages soon.
>
> Great, thank you!

Yes, that's great!

@Timothy: I haven't included your previous patches on `wip-bootstrap',
feel free to push them to wip-bootstrap.

We are getting real close to merging this, I think.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Ludovic Courtès]

Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> I have rebased wip-bootstrap on core-updates that include your
> commencements updates.
>
> With your explanations on irc and studying your patch, I was able to
> keep using your cleanups and do without the most ugly 'setenv stages.
>
> So, wip-bootstrap on savannah is reset once again.

Great, thanks for taking the time to rebase and adjust the code, and
apologies for all the trouble!  (Actually making this change to
commencement.scm wasn’t on my agenda, it’s one of the things that pop up
and give you a good excuse to avoid more boring stuff, I suppose…)

> @Timothy: I haven't included your previous patches on `wip-bootstrap',
> feel free to push them to wip-bootstrap.
>
> We are getting real close to merging this, I think.

This is exciting.  Let’s synchronize with Marius once the branch has
stabilized to see if this can go in ‘core-updates’ this time.

Thanks,
Ludo’.

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Timothy Sample]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> @Timothy: I haven't included your previous patches on `wip-bootstrap',
>> feel free to push them to wip-bootstrap.
>>
>> We are getting real close to merging this, I think.
>
> This is exciting.  Let’s synchronize with Marius once the branch has
> stabilized to see if this can go in ‘core-updates’ this time.

Indeed!

I just pushed five commits to “wip-bootstrap”.  They are a little
different from the patches I sent.

For one, I followed Ludo’s advice and added a “gnu-bootstrap” module for
the build code.  That means I got rid of “make-bootstrap-phases” and
adjusted the packages accordingly.  It’s a little janky, but it does
clear up “commencement.scm”.

Bootar is released and saw some cosmetic improvements.  Notably, it
disables “escape-newlines” and uses “pretty-print”, making the SES file
easy to read for humans.  The SES file is still hosted on my server, but
at a stable URL.  Is it worrisome that it’s on my server?  I put a
mirror on Gitlab <https://gitlab.com/samplet/bootar/-/tags/v1>, but the
URL to the actual file is kinda ugly.  Thoughts?

Gash-Utils is released!  The Git repo and tarball are on Savannah.  I
updated how the base package builds a little bit, and fiddled with the
home-page, synopsis, and description.  The bootstrap version was
installing a useless “template” binary, which I fixed.

I noticed a little issue with “%boot-mesboot1-inputs”, so I fixed it and
made sure that “bootar” does not get propagated past that point.

I have not tested the full bootstrap yet, but I did test “hello-mesboot”
on basically this same code, and it was fine.  Since almost nothing
substantial has been changed, it should be OK, but you never know until
you try!  I’ll build the full thing as fast as my little computer can.
:)


-- Tim

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Jan Nieuwenhuizen]

Timothy Sample writes:

Hello,

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>> @Timothy: I haven't included your previous patches on `wip-bootstrap',
>>> feel free to push them to wip-bootstrap.
>>>
>>> We are getting real close to merging this, I think.
>>
>> This is exciting.  Let’s synchronize with Marius once the branch has
>> stabilized to see if this can go in ‘core-updates’ this time.
>
> Indeed!
>
> I just pushed five commits to “wip-bootstrap”.  They are a little
> different from the patches I sent.

Very nice.

> For one, I followed Ludo’s advice and added a “gnu-bootstrap” module for
> the build code.  That means I got rid of “make-bootstrap-phases” and
> adjusted the packages accordingly.  It’s a little janky, but it does
> clear up “commencement.scm”.

I like it!

> Bootar is released and saw some cosmetic improvements.  Notably, it
> disables “escape-newlines” and uses “pretty-print”, making the SES file
> easy to read for humans.  The SES file is still hosted on my server, but
> at a stable URL.  Is it worrisome that it’s on my server?  I put a
> mirror on Gitlab <https://gitlab.com/samplet/bootar/-/tags/v1>, but the
> URL to the actual file is kinda ugly.  Thoughts?
>
> Gash-Utils is released!  The Git repo and tarball are on Savannah.  I
> updated how the base package builds a little bit, and fiddled with the
> home-page, synopsis, and description.  The bootstrap version was
> installing a useless “template” binary, which I fixed.

Oh, congrats!

> I noticed a little issue with “%boot-mesboot1-inputs”, so I fixed it and
> made sure that “bootar” does not get propagated past that point.
>
> I have not tested the full bootstrap yet, but I did test “hello-mesboot”
> on basically this same code, and it was fine.  Since almost nothing
> substantial has been changed, it should be OK, but you never know until
> you try!  I’ll build the full thing as fast as my little computer can.
> :)

I found that the new file-boot0 5.38 does not build with the new bzip
feature; I added a squash! commit to disable that.

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Timothy Sample]

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> I found that the new file-boot0 5.38 does not build with the new bzip
> feature; I added a squash! commit to disable that.

I managed to build GNU Hello, but I had to disable “bzlib” for the
“file” package in “%final-inputs” as well.  I assume that’s a problem.

You may have already noticed, but I thought I would mention it just in
case.


-- Tim

[bug#38390] [bug #38390] Building bootstrap Gash and Gash-Utils

[Ludovic Courtès]

Hi Timothy,

Timothy Sample <samplet@ngyro.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>> @Timothy: I haven't included your previous patches on `wip-bootstrap',
>>> feel free to push them to wip-bootstrap.
>>>
>>> We are getting real close to merging this, I think.
>>
>> This is exciting.  Let’s synchronize with Marius once the branch has
>> stabilized to see if this can go in ‘core-updates’ this time.
>
> Indeed!
>
> I just pushed five commits to “wip-bootstrap”.  They are a little
> different from the patches I sent.

Awesome, thank you!

> For one, I followed Ludo’s advice and added a “gnu-bootstrap” module for
> the build code.  That means I got rid of “make-bootstrap-phases” and
> adjusted the packages accordingly.  It’s a little janky, but it does
> clear up “commencement.scm”.
>
> Bootar is released and saw some cosmetic improvements.  Notably, it
> disables “escape-newlines” and uses “pretty-print”, making the SES file
> easy to read for humans.  The SES file is still hosted on my server, but
> at a stable URL.  Is it worrisome that it’s on my server?  I put a
> mirror on Gitlab <https://gitlab.com/samplet/bootar/-/tags/v1>, but the
> URL to the actual file is kinda ugly.  Thoughts?

We can host a copy at <https://ftp.gnu.org/gnu/guix/mirror>, which will
be picked up by Software Heritage and archived for eternity.  :-)
Let’s do that when we merge the branch.

> Gash-Utils is released!  The Git repo and tarball are on Savannah.  I
> updated how the base package builds a little bit, and fiddled with the
> home-page, synopsis, and description.  The bootstrap version was
> installing a useless “template” binary, which I fixed.
>
> I noticed a little issue with “%boot-mesboot1-inputs”, so I fixed it and
> made sure that “bootar” does not get propagated past that point.
>
> I have not tested the full bootstrap yet, but I did test “hello-mesboot”
> on basically this same code, and it was fine.  Since almost nothing
> substantial has been changed, it should be OK, but you never know until
> you try!  I’ll build the full thing as fast as my little computer can.
> :)

Thank you for all the great work!

Ludo’.