Building Bash with Geesh

Building Bash with Geesh

[Timothy Sample]

Hi Guix,

Over the last year, I built Geesh <https://gitlab.com/samplet/geesh>,
which is a simple Shell interpreter written in Guile Scheme.  As of the
other day, it is capable enough to build Bash!

(That’s the main point of this message.  Everything that follows is just
details.)

Geesh is very simple, and is missing a lot of features.  It does not
even support all of POSIX.  If you are looking for a new shell, Geesh is
not for you!  (It doesn’t even prompt for input!)

However, thanks to the robustness of Autotools, Geesh is capable of
being the shell used to run Bash’s “configure” script and do everything
required (of a shell) to build Bash.  (As an aside, “configure” scripts
are amazing!  Geesh does not support “$LINENO”, but the script works
around this by running itself through “sed” and replacing every
“$LINENO” with the current line number.  It then restarts itself as this
new script that doesn’t use “$LINENO”.)  Actually, I’m only 99% sure of
this, because I have not worked on actual bootstrapping yet (see the
caveats below).  Checkout “tests/bash-without-bash.scm” for how I’m
currently testing this claim.

Now, there are some caveats with respect to actual bootstrapping:

    • It does not have a non-Autotools build script;

    • It requires Guile 2.2, while (I assume) the current bootstrap
      Guile is version 2.0;

    • It still requires Core Utilities, though many of the utilities are
      available as Guile code in Gash.

Of these, I think fixing the first two will be trivial, while fixing the
third will require a bit of work (though hopefully not too much).  Once
these things are done, we will be pretty close to being able to retire
the “coreutils&co” bootstrap binary.  We certainly could make it a lot
smaller.  (AFAIK, we are still missing a few things, but Jan knows
better than I do.)

You may be thinking at this point, “what about Gash?”  (For those who
don’t know, Gash is also a Shell interpreter written in Guile
<https://gitlab.com/janneke/gash>.)  The Gash folks and I have been
thinking for a while about the best way to share work and combine
efforts.

Finally, and I’m just putting this out there ;), if you’re new to Guile
and want a simple project learn with, Geesh might be right for you!  It
was a Guile-learning project for me, and it served its purpose
admirably.


-- Tim

Re: Building Bash with Geesh

[Jan Nieuwenhuizen]

Timothy Sample writes:

Hi Timothy!

> Over the last year, I built Geesh <https://gitlab.com/samplet/geesh>,
> which is a simple Shell interpreter written in Guile Scheme.  As of the
> other day, it is capable enough to build Bash!

That is just amazing, congrats!

> Geesh is very simple, and is missing a lot of features.  It does not
> even support all of POSIX.  If you are looking for a new shell, Geesh is
> not for you!  (It doesn’t even prompt for input!)

> Now, there are some caveats with respect to actual bootstrapping:
>
>     • It does not have a non-Autotools build script;

That's OK, we can add a bootstrap-geesh if we want.

>     • It requires Guile 2.2, while (I assume) the current bootstrap
>       Guile is version 2.0;

That should be OK too, I'll leave Ludo' to comment on this, but Gash
also needs Guile 2.2; on my Guix wip-bootstrap branch I already upgraded
Guile.

>     • It still requires Core Utilities, though many of the utilities are
>       available as Guile code in Gash.

Yes!

> Of these, I think fixing the first two will be trivial, while fixing the
> third will require a bit of work (though hopefully not too much).  Once
> these things are done, we will be pretty close to being able to retire
> the “coreutils&co” bootstrap binary.  We certainly could make it a lot
> smaller.  (AFAIK, we are still missing a few things, but Jan knows
> better than I do.)

Just yesterday, I managed to build a minimal bash and make using Gash
and a very simple boot script, so without coreutils&co.  A Scheme-only
bootstrap is coming closer...

> You may be thinking at this point, “what about Gash?”  (For those who
> don’t know, Gash is also a Shell interpreter written in Guile
> <https://gitlab.com/janneke/gash>.)  The Gash folks and I have been
> thinking for a while about the best way to share work and combine
> efforts.

Yeah...Gash has an option to use the Geesh LALR parser, however it's not
very well integrated (in fact I fear that the integration may have
bitrotted).

I think we should either cherry-pick eachother's goodies for a while, or
somehow merge into one project that has two parsers (LALR and PEG), or
spawn a new Guile library that provides the backend (the core-utils and
co).  It's a real interesting puzzle.  WDYT?

> Finally, and I’m just putting this out there ;), if you’re new to Guile
> and want a simple project learn with, Geesh might be right for you!  It
> was a Guile-learning project for me, and it served its purpose
> admirably.

Thanks!
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Building Bash with Geesh

[Ludovic Courtès]

Hello Timothy & Jan!

You folks are amazing!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

>> Over the last year, I built Geesh <https://gitlab.com/samplet/geesh>,
>> which is a simple Shell interpreter written in Guile Scheme.  As of the
>> other day, it is capable enough to build Bash!
>
> That is just amazing, congrats!

Indeed, that’s a major achievement!

>>     • It requires Guile 2.2, while (I assume) the current bootstrap
>>       Guile is version 2.0;
>
> That should be OK too, I'll leave Ludo' to comment on this, but Gash
> also needs Guile 2.2; on my Guix wip-bootstrap branch I already upgraded
> Guile.

Which parts of 2.2 that 2.0 lacks does it need?  PEG?

Overall I’m in favor of changing the bootstrap seeds as rarely as
possible.  Also, if our horizon is a full Scheme bootstrap has Jan
proposed (and I think that’s a great plan!), then perhaps we’ll have to
arrange to not rely on fancy Guile features in build-side code meant to
run early on during bootstrap.  So far it was easy to keep (guix build …) 
valid for both 2.2 and 2.0, but these are simple modules; I don’t know
whether that’s reasonably feasible for more complex pieces of software
like Geesh and Gash.

All that said, the benefit of removing Bash from the seeds may well
outweigh the “cost” of upgrading to Guile 2.2.

>> Of these, I think fixing the first two will be trivial, while fixing the
>> third will require a bit of work (though hopefully not too much).  Once
>> these things are done, we will be pretty close to being able to retire
>> the “coreutils&co” bootstrap binary.  We certainly could make it a lot
>> smaller.  (AFAIK, we are still missing a few things, but Jan knows
>> better than I do.)
>
> Just yesterday, I managed to build a minimal bash and make using Gash
> and a very simple boot script, so without coreutils&co.  A Scheme-only
> bootstrap is coming closer...

Woohoo!

>> You may be thinking at this point, “what about Gash?”  (For those who
>> don’t know, Gash is also a Shell interpreter written in Guile
>> <https://gitlab.com/janneke/gash>.)  The Gash folks and I have been
>> thinking for a while about the best way to share work and combine
>> efforts.
>
> Yeah...Gash has an option to use the Geesh LALR parser, however it's not
> very well integrated (in fact I fear that the integration may have
> bitrotted).
>
> I think we should either cherry-pick eachother's goodies for a while, or
> somehow merge into one project that has two parsers (LALR and PEG), or
> spawn a new Guile library that provides the backend (the core-utils and
> co).  It's a real interesting puzzle.  WDYT?

It’d be great if both projects could converge; there’ll still be plenty
of challenges to satisfy your playfulness anyway.  :-)  (Like, say, a
shell→tree-il front-end, hint hint ;-)).

That said I can imagine it’s not that easy and maybe also less fun but
it would help the longer-term goal of building a solid foundation for
bootstrapped distros.

Anyway, kudos on these achievements!  I guess we at least need Geesh and
Gash packages now!  :-)

Ludo’.

Re: Building Bash with Geesh

[Timothy Sample]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> ...
>
> Which parts of 2.2 that 2.0 lacks does it need?  PEG?

For Geesh, I think it is mostly just trivialities like “peek-char”
vs. “lookahead-char”.  I started adapting it once, and fixing one or two
small things like that would make the front-end (lexer and parser) work
for Guile 2.0.  I’m less sure about the back-end.  Being relatively new
to Guile means that Guile 2.2 just *is* Guile as far as I know.  :)

> Overall I’m in favor of changing the bootstrap seeds as rarely as
> possible.  Also, if our horizon is a full Scheme bootstrap has Jan
> proposed (and I think that’s a great plan!), then perhaps we’ll have to
> arrange to not rely on fancy Guile features in build-side code meant to
> run early on during bootstrap.  So far it was easy to keep (guix build …) 
> valid for both 2.2 and 2.0, but these are simple modules; I don’t know
> whether that’s reasonably feasible for more complex pieces of software
> like Geesh and Gash.
>
> All that said, the benefit of removing Bash from the seeds may well
> outweigh the “cost” of upgrading to Guile 2.2.

This makes sense.  I’m pretty sure we could stick with version 2.0 if
we’re careful.

> ...
>
> It’d be great if both projects could converge; there’ll still be plenty
> of challenges to satisfy your playfulness anyway.  :-)  (Like, say, a
> shell→tree-il front-end, hint hint ;-)).

Agreed!  Imagining a “(language shell spec)” module was what got me
motivated in the first place.  :)

> That said I can imagine it’s not that easy and maybe also less fun but
> it would help the longer-term goal of building a solid foundation for
> bootstrapped distros.
>
> Anyway, kudos on these achievements!  I guess we at least need Geesh and
> Gash packages now!  :-)

Forthcoming!

> Ludo’.


-- Tim

Re: Building Bash with Geesh

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>>>     • It requires Guile 2.2, while (I assume) the current bootstrap
>>>       Guile is version 2.0;
>>
>> That should be OK too, I'll leave Ludo' to comment on this, but Gash
>> also needs Guile 2.2; on my Guix wip-bootstrap branch I already upgraded
>> Guile.
>
> Which parts of 2.2 that 2.0 lacks does it need?  PEG?

I was a bit quick here, as I thought that we wanted to move to 2.2
anyway.  Until recently we needed PEG, but Gash now includes a modified
copy of it.  So I suspect that staying on 2.0 for a bit might be
feasible (I haven't looked at details).

> Overall I’m in favor of changing the bootstrap seeds as rarely as
> possible.  Also, if our horizon is a full Scheme bootstrap has Jan
> proposed (and I think that’s a great plan!), then perhaps we’ll have to
> arrange to not rely on fancy Guile features in build-side code meant to
> run early on during bootstrap.  So far it was easy to keep (guix build …) 
> valid for both 2.2 and 2.0, but these are simple modules; I don’t know
> whether that’s reasonably feasible for more complex pieces of software
> like Geesh and Gash.

OK.  In any case would be nice if the 2.0->2.2 upgrade can be a separate
decision rather than a dependency for Gash (or Geesh).

> All that said, the benefit of removing Bash from the seeds may well
> outweigh the “cost” of upgrading to Guile 2.2.

That's great, we'll see!

> It’d be great if both projects could converge; there’ll still be plenty
> of challenges to satisfy your playfulness anyway.  :-)  (Like, say, a
> shell→tree-il front-end, hint hint ;-)).

Well, there's another challenge I have tried not to think of :)

> That said I can imagine it’s not that easy and maybe also less fun but
> it would help the longer-term goal of building a solid foundation for
> bootstrapped distros.

It's hard to see for me beyond the horizon of a Scheme-only bootstrap.
Once we get there, we may want to make the bootstrap Scheme more tiny,
either by making a more tiny, bootstrappable Guile or replacing Guile
with Mes.

> Anyway, kudos on these achievements!  I guess we at least need Geesh and
> Gash packages now!  :-)

Hehe, there's a Gash package on wip-bootstrap.  However, Gash is still
failing 19 (of 120) shell test; we'll want to fix those before releasing
0.1.  Also the concept of `export' is missing entirely.

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Building Bash with Geesh

[Timothy Sample]

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Timothy Sample writes:
>
> Hi Timothy!
>
> ...
>
> Just yesterday, I managed to build a minimal bash and make using Gash
> and a very simple boot script, so without coreutils&co.  A Scheme-only
> bootstrap is coming closer...

That’s excellent!  I am excited to take a closer look at Gash with all
the progress made over the last month.

>> You may be thinking at this point, “what about Gash?”  (For those who
>> don’t know, Gash is also a Shell interpreter written in Guile
>> <https://gitlab.com/janneke/gash>.)  The Gash folks and I have been
>> thinking for a while about the best way to share work and combine
>> efforts.
>
> Yeah...Gash has an option to use the Geesh LALR parser, however it's not
> very well integrated (in fact I fear that the integration may have
> bitrotted).
>
> I think we should either cherry-pick eachother's goodies for a while, or
> somehow merge into one project that has two parsers (LALR and PEG), or
> spawn a new Guile library that provides the backend (the core-utils and
> co).  It's a real interesting puzzle.  WDYT?

Hmm....  A puzzle indeed.  Spinning off the core-utils stuff makes
sense.  I always assumed that there would be some way in Geesh to
dynamically add built-ins.  The Geesh front-end is isolated from the
rest of it, and could be pulled out with no trouble.

Ultimately, I would like us to coordinate and consolidate what we have
into a unified whole (even if it that whole is made of multiple
packages).  I will take a close look at Gash, and then I will message
you with any plans I come up with for moving forward.  Feel free to
message me with whatever ideas you have, too.  I’m sure we can figure
this out.  :)

>> Finally, and I’m just putting this out there ;), if you’re new to Guile
>> and want a simple project learn with, Geesh might be right for you!  It
>> was a Guile-learning project for me, and it served its purpose
>> admirably.
>
> Thanks!
> janneke

-- Tim

Re: Building Bash with Geesh

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

Timothy Sample <samplet@ngyro.com> writes:

> Over the last year, I built Geesh <https://gitlab.com/samplet/geesh>,
> which is a simple Shell interpreter written in Guile Scheme.  As of the
> other day, it is capable enough to build Bash!

That's awesome!  I always enjoy reading the emails on this list that are
related to bootstrapping.  It's super important work, and I'm glad you
and the others are making progress!

Thank you,

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Building Bash with Geesh

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

Timothy Sample writes:

Hello,

Attached are two small patches that allow me to run the Gash test suite
with Geesh, like so

--8<---------------cut here---------------start------------->8---
PATH=$PATH:bin SHELL='../geesh/pre-inst-env geesh' ./check.sh 
--8<---------------cut here---------------end--------------->8---

assuming that both projects live in the same parent directory.

Geesh does much better than Gash, it only fails 11/126 tests.

Moreover, all POSIX tests pass!  The ones that fail involve `echo -e'
and Bash-isms, notably substitutions like ${var/foo} (and all %, %%, #,
##, ...etc).

What a great job!

Interestingly, Gash passes all tests that Geesh fails, and Gash fails
about 20 other tests.

It looks like that once we manage to combine Geesh and Gash, stealing
eachothers code or otherwise, we have a shell that can replace bash and
coreutils&co in the initial bootstrap.

Greetings,
janneke

Also put up on: https://gitlab.com/janneke/geesh


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-Use-getcwd-instead-of-hard-coded-directory.patch --]
[-- Type: text/x-patch, Size: 816 bytes --]

From 0aca7e71517b63fc3f67f3a72757f69d7a91158a Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sun, 9 Dec 2018 07:35:18 +0100
Subject: [PATCH 1/2] guix: Use getcwd instead of hard-coded directory.

* guix.scm (make-select): Use getcwd instead of hard-coded directory.
---
 guix.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/guix.scm b/guix.scm
index 04973a9..3b647dd 100644
--- a/guix.scm
+++ b/guix.scm
@@ -24,7 +24,7 @@
       version)))
 
 (define (make-select)
-  (let* ((directory (repository-discover "/home/samplet/code/geesh"))
+  (let* ((directory (repository-discover (getcwd)))
          (repository (repository-open directory))
          (oid (reference-target (repository-head repository)))
          (commit (commit-lookup repository oid))
-- 
2.19.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Support-e-x-in-geesh-script.patch --]
[-- Type: text/x-patch, Size: 1393 bytes --]

From 52d3b6ea3c47a84f58e06e0d09db5ce8f9cf383e Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sun, 9 Dec 2018 07:37:22 +0100
Subject: [PATCH 2/2] Support -e, -x in geesh script.

* scripts/geesh.in (options-spec): Support -e, -x.
---
 scripts/geesh.in | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/scripts/geesh.in b/scripts/geesh.in
index fa8001f..e7a0051 100644
--- a/scripts/geesh.in
+++ b/scripts/geesh.in
@@ -24,17 +24,26 @@
 (set! %load-compiled-path (cons "@GODIR@" %load-compiled-path))
 
 (use-modules (geesh repl)
+             (geesh environment)
              (ice-9 getopt-long))
 
 (define options-spec
   '((command (single-char #\c) (value #t))
-    (stdin (single-char #\s))))
+    (errexit (single-char #\e))
+    (stdin (single-char #\s))
+    (xtrace (single-char #\x))))
 
 (let* ((options (getopt-long (program-arguments) options-spec
                              #:stop-at-first-non-option #t))
        (command (option-ref options 'command #f))
+       (errexit? (option-ref options 'errexit #f))
        (stdin (option-ref options 'stdin #f))
+       (xtrace? (option-ref options 'xtrace #f))
        (args (option-ref options '() '())))
+  (when errexit?
+    (setopt! 'errexit #t))
+  (when xtrace?
+    (setopt! 'xtrace #t))
   (cond
    ((and command stdin)
     (format (current-error-port)
-- 
2.19.1

Re: Building Bash with Geesh

[Timothy Sample]

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Timothy Sample writes:
>
> Hello,
>
> Attached are two small patches that allow me to run the Gash test suite
> with Geesh, like so
>
> PATH=$PATH:bin SHELL='../geesh/pre-inst-env geesh' ./check.sh 
>
> assuming that both projects live in the same parent directory.
>
> Geesh does much better than Gash, it only fails 11/126 tests.
>
> Moreover, all POSIX tests pass!  The ones that fail involve `echo -e'
> and Bash-isms, notably substitutions like ${var/foo} (and all %, %%, #,
> ##, ...etc).
>
> What a great job!

Thanks!  I had a lot of help from the Oil shell test suite that we
talked about before.  Right now I am working on sed, but integrating the
Oil tests into Gash is something that I want to do soon.

With respect to Bash-isms, I’ve been deliberately avoiding them.  POSIX
seemed complicated enough for a first go!  The substring operations
(“%%”, etc.) are all POSIX, but Geesh raises a “not implemented” error
whenever it sees them.  :)

> Interestingly, Gash passes all tests that Geesh fails, and Gash fails
> about 20 other tests.

What an interesting coincidence!

> It looks like that once we manage to combine Geesh and Gash, stealing
> eachothers code or otherwise, we have a shell that can replace bash and
> coreutils&co in the initial bootstrap.

I am still of the opinion that Geesh’s parser is its strongest asset
(hint, hint).  The back-end code is all pretty ad-hoc.

> Greetings,
> janneke
>
> Also put up on: https://gitlab.com/janneke/geesh
>
> From 0aca7e71517b63fc3f67f3a72757f69d7a91158a Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Sun, 9 Dec 2018 07:35:18 +0100
> Subject: [PATCH 1/2] guix: Use getcwd instead of hard-coded directory.
>
> * guix.scm (make-select): Use getcwd instead of hard-coded directory.
> ---
>  guix.scm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/guix.scm b/guix.scm
> index 04973a9..3b647dd 100644
> --- a/guix.scm
> +++ b/guix.scm
> @@ -24,7 +24,7 @@
>        version)))
>  
>  (define (make-select)
> -  (let* ((directory (repository-discover "/home/samplet/code/geesh"))
> +  (let* ((directory (repository-discover (getcwd)))

Definitely didn’t mean to commit that!  :p

>           (repository (repository-open directory))
>           (oid (reference-target (repository-head repository)))
>           (commit (commit-lookup repository oid))
> -- 
> 2.19.1
>
> From 52d3b6ea3c47a84f58e06e0d09db5ce8f9cf383e Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Sun, 9 Dec 2018 07:37:22 +0100
> Subject: [PATCH 2/2] Support -e, -x in geesh script.
>
> * scripts/geesh.in (options-spec): Support -e, -x.
> ---
>  scripts/geesh.in | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/geesh.in b/scripts/geesh.in
> index fa8001f..e7a0051 100644
> --- a/scripts/geesh.in
> +++ b/scripts/geesh.in
> @@ -24,17 +24,26 @@
>  (set! %load-compiled-path (cons "@GODIR@" %load-compiled-path))
>  
>  (use-modules (geesh repl)
> +             (geesh environment)
>               (ice-9 getopt-long))
>  
>  (define options-spec
>    '((command (single-char #\c) (value #t))
> -    (stdin (single-char #\s))))
> +    (errexit (single-char #\e))
> +    (stdin (single-char #\s))
> +    (xtrace (single-char #\x))))
>  
>  (let* ((options (getopt-long (program-arguments) options-spec
>                               #:stop-at-first-non-option #t))
>         (command (option-ref options 'command #f))
> +       (errexit? (option-ref options 'errexit #f))
>         (stdin (option-ref options 'stdin #f))
> +       (xtrace? (option-ref options 'xtrace #f))
>         (args (option-ref options '() '())))
> +  (when errexit?
> +    (setopt! 'errexit #t))
> +  (when xtrace?
> +    (setopt! 'xtrace #t))
>    (cond
>     ((and command stdin)
>      (format (current-error-port)


-- Tim