[GSoC] Bourne-shell compiler front-end for Guile

[GSoC] Bourne-shell compiler front-end for Guile

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 396 bytes --]

I'm a student still this year and Guix is the only project I'm considering.

If there's anything I can do/change to make it more clear or thorough I'd
love to have some feedback.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: GSoC_2016_Efraim.txt --]
[-- Type: text/plain, Size: 2905 bytes --]

Write a Bourne-shell compiler front-end for Guile

Efraim Flashner
efraim@flashner.co.il

Who am I?

My name is Efraim, I’m currently living in Jerusalem, Israel. I’m a 3rd year Computer Science student at the Jerusalem College of Technology (Israel has the college/university split). One of my classes was on functional programming, where we covered prolog and lisp. I found the idea of functional programming interesting, and at the time even tried writing small programs using them.

I switched full-time to Linux in 2010, trying out Ubuntu and then switching to Debian, and currently I’m waiting on a blob-free wireless mini-pcie/usb before switching my laptop over to GuixSD. I haven’t had a lot of opportunity in Israel to meet up with Free Software or Linux people, and I’ve worked hard to make up for that by reading lots of online tutorials and experimenting on my machines. I first discovered Guix around June of last year and got involved shortly after.

The Plan:

As it currently stands, looking in gnu/packages/bootstrap/$ARCH/, in addition to guile there’s bash, mkdir, tar and xz. From the manual we have the bootstrapping methodology (https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html) which also uses gcc, binutils, coreutils, libc, etc. We also have guix/build/bournish.scm to act as a rescue-shell of sorts and in use in the initrd.

By extending/building upon bournish we can lessen the number of bootstrap binaries that we need to bootstrap a working system AND we get a more powerful shell to use when something™ happens during boot and we’re dropped down to a rescue shell.

Currently the idea on the GSoC idea wiki is to extend bournish so that it can cover everything needed by a configure script. This would of course allow us to decrease the number of binaries we need to get started. Looking at the list of bootstrap binaries, I’m most interested in including enough functionality to remove our dependance on bash which to me seems like a good security idea, as surely we wouldn’t want to have a vulnerability found for our bootstrap version of bash.

The Plan, Part 2:

From now until the start of GSoC I’d work on bettering my knowledge of guile and more closely analyzing the bootstrap process. I’ve had a look at Wikipedia’s entry on Unix shells and I’ll be grabbing the source of a few of them, including ash, scsh and busybox, for inspiration.

For the coding I plan on implementing the different features that we’d need for replacing a number of the bootstrap binaries. As a stretch-goal I’d like to see if it is possible to switch to using it as my user shell in place of bash. Part of this would of course come with deciding on how to use it to replace bootstrap binaries; whether to have additional binary outputs for the different binaries being replaced or to have one monolithic binary.

[-- Attachment #1.3: GSoC_proposal_Efraim_Flashner.pdf --]
[-- Type: application/pdf, Size: 34564 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [GSoC] Bourne-shell compiler front-end for Guile

[Ludovic Courtès]

Hi Efraim,

Thanks for your proposal!

This project can be pretty fun, I think, and quite rewarding if it
indeed allows us to get rid of a bunch of bootstrap binaries.

As I see it, the project covers mostly two areas:

  1. Lexing & parsing.  This compiler front-end will need a real parser
     for the Bourne shell syntax, notably semicolon-separated command
     sequences, pipes, brace-delimited blocks, functions, variables, and
     ‘test’ and ‘[’.

     The available tools could be SILex + (system base lalr) (the latter
     is part of Guile, the former is maintained separately; a copy can
     be found in Guile-RPC for instance.)

  2. Run-time support: redirections, pipes, background execution, C-c
     (SIGINT) and C-z (SIGTSTP), globbing, and so on.

     Guile provides just the basic here, and a little more with (ice-9
     popen).  Scsh and its Guile port(s) provide a lot of that, so it
     might be a good idea to see what can be stolen from these.

It may be that some of your CS classes cover some of this ground (a
fairly common programming project here at university is to implement a
small shell precisely because it covers several areas.)

How does that sound?

Perhaps one way to get started would be by looking at the available
tools for #1, and experimenting with hacks for #2, possibly borrowing
code from Scsh.

Thoughts?

Thanks,
Ludo’.

PS: Would be nice to have a second mentor on this, either officially or
    unofficially (if you don’t want to have a to be known to Google…).
    Any takers?  :-)

Re: [GSoC] Bourne-shell compiler front-end for Guile

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2127 bytes --]

On Mon, 21 Mar 2016 23:27:25 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Hi Efraim,
> 
> Thanks for your proposal!
> 
> This project can be pretty fun, I think, and quite rewarding if it
> indeed allows us to get rid of a bunch of bootstrap binaries.
> 
> As I see it, the project covers mostly two areas:
> 
>   1. Lexing & parsing.  This compiler front-end will need a real parser
>      for the Bourne shell syntax, notably semicolon-separated command
>      sequences, pipes, brace-delimited blocks, functions, variables, and
>      ‘test’ and ‘[’.
> 
>      The available tools could be SILex + (system base lalr) (the latter
>      is part of Guile, the former is maintained separately; a copy can
>      be found in Guile-RPC for instance.)
> 
>   2. Run-time support: redirections, pipes, background execution, C-c
>      (SIGINT) and C-z (SIGTSTP), globbing, and so on.
> 
>      Guile provides just the basic here, and a little more with (ice-9
>      popen).  Scsh and its Guile port(s) provide a lot of that, so it
>      might be a good idea to see what can be stolen from these.
> 
> It may be that some of your CS classes cover some of this ground (a
> fairly common programming project here at university is to implement a
> small shell precisely because it covers several areas.)
> 
> How does that sound?
> 
> Perhaps one way to get started would be by looking at the available
> tools for #1, and experimenting with hacks for #2, possibly borrowing
> code from Scsh.
> 
> Thoughts?

Sounds like lots of fun, I'm really looking forward to it. And it sounds like
I'll need to dive deeper into bash to see what I get to implement

> 
> Thanks,
> Ludo’.
> 
> PS: Would be nice to have a second mentor on this, either officially or
>     unofficially (if you don’t want to have a to be known to Google…).
>     Any takers?  :-)



-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [GSoC] Bourne-shell compiler front-end for Guile

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Mon, 21 Mar 2016 23:27:25 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Hi Efraim,
>> 
>> Thanks for your proposal!
>> 
>> This project can be pretty fun, I think, and quite rewarding if it
>> indeed allows us to get rid of a bunch of bootstrap binaries.
>> 
>> As I see it, the project covers mostly two areas:
>> 
>>   1. Lexing & parsing.  This compiler front-end will need a real parser
>>      for the Bourne shell syntax, notably semicolon-separated command
>>      sequences, pipes, brace-delimited blocks, functions, variables, and
>>      ‘test’ and ‘[’.
>> 
>>      The available tools could be SILex + (system base lalr) (the latter
>>      is part of Guile, the former is maintained separately; a copy can
>>      be found in Guile-RPC for instance.)
>> 
>>   2. Run-time support: redirections, pipes, background execution, C-c
>>      (SIGINT) and C-z (SIGTSTP), globbing, and so on.
>> 
>>      Guile provides just the basic here, and a little more with (ice-9
>>      popen).  Scsh and its Guile port(s) provide a lot of that, so it
>>      might be a good idea to see what can be stolen from these.
>> 
>> It may be that some of your CS classes cover some of this ground (a
>> fairly common programming project here at university is to implement a
>> small shell precisely because it covers several areas.)
>> 
>> How does that sound?
>> 
>> Perhaps one way to get started would be by looking at the available
>> tools for #1, and experimenting with hacks for #2, possibly borrowing
>> code from Scsh.
>> 
>> Thoughts?
>
> Sounds like lots of fun, I'm really looking forward to it. And it sounds like
> I'll need to dive deeper into bash to see what I get to implement

I guess a related question is how confident/familiar you feel with the
two areas above.  The agenda could be adjusted as a function of that.

Thanks,
Ludo’.

Re: [GSoC] Bourne-shell compiler front-end for Guile

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2743 bytes --]

On Wed, Mar 23, 2016 at 03:05:08PM +0100, Ludovic Courtès wrote:
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > On Mon, 21 Mar 2016 23:27:25 +0100
> > ludo@gnu.org (Ludovic Courtès) wrote:
> >
> >> Hi Efraim,
> >> 
> >> Thanks for your proposal!
> >> 
> >> This project can be pretty fun, I think, and quite rewarding if it
> >> indeed allows us to get rid of a bunch of bootstrap binaries.
> >> 
> >> As I see it, the project covers mostly two areas:
> >> 
> >>   1. Lexing & parsing.  This compiler front-end will need a real parser
> >>      for the Bourne shell syntax, notably semicolon-separated command
> >>      sequences, pipes, brace-delimited blocks, functions, variables, and
> >>      ‘test’ and ‘[’.
> >> 
> >>      The available tools could be SILex + (system base lalr) (the latter
> >>      is part of Guile, the former is maintained separately; a copy can
> >>      be found in Guile-RPC for instance.)
> >> 
> >>   2. Run-time support: redirections, pipes, background execution, C-c
> >>      (SIGINT) and C-z (SIGTSTP), globbing, and so on.
> >> 
> >>      Guile provides just the basic here, and a little more with (ice-9
> >>      popen).  Scsh and its Guile port(s) provide a lot of that, so it
> >>      might be a good idea to see what can be stolen from these.
> >> 
> >> It may be that some of your CS classes cover some of this ground (a
> >> fairly common programming project here at university is to implement a
> >> small shell precisely because it covers several areas.)
> >> 
> >> How does that sound?
> >> 
> >> Perhaps one way to get started would be by looking at the available
> >> tools for #1, and experimenting with hacks for #2, possibly borrowing
> >> code from Scsh.
> >> 
> >> Thoughts?
> >
> > Sounds like lots of fun, I'm really looking forward to it. And it sounds like
> > I'll need to dive deeper into bash to see what I get to implement
> 
> I guess a related question is how confident/familiar you feel with the
> two areas above.  The agenda could be adjusted as a function of that.
> 
> Thanks,
> Ludo’.

I feel pretty confident about the lexing and parsing; not too sure what
would go into implementing the run-time support.

To summarize a bit, there are two mostly overlapping parts here, one more
busybox-esque where we can drop bootstrap binaries, and a second with a
shell implemented in guile. At this stage I have more in the
confidence/excitement side and less on the familiarity side.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [GSoC] Bourne-shell compiler front-end for Guile

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> I feel pretty confident about the lexing and parsing; not too sure what
> would go into implementing the run-time support.
>
> To summarize a bit, there are two mostly overlapping parts here, one more
> busybox-esque where we can drop bootstrap binaries, and a second with a
> shell implemented in guile. At this stage I have more in the
> confidence/excitement side and less on the familiarity side.

OK, excitement is an important boost anyway.  :-)

Ludo’.