summer of code ideas

summer of code ideas

[Andy Wingo]

Hey all,

We need to get together a few Guile-proposed projects for SoC students
to hack on.  How about we collect them here in this thread.  I'll kick
it off with one idea:

  * A CPAN for Guile.  My initial thoughts are that you want a
    combination of APT-like discovery and download of sources of
    software, and a stowfs-like local storage, allowing you to update a
    tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
    back changes, etc.  See
    http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
    for more on $XDG_DATA_DIRS.

    C extensions are outside the scope of this project.  IMO a CPAN for
    Guile should not compile C helper libraries.

    You can start here by doing a survey of what other schemes do:
    chicken and racket being the obvious ones, and also perhaps taking a
    look at cabal and other CPAN-alikes developed in the last 10 years.

    You'd need to define a format for code -- tarballs or something.
    You need to think about signing too, and the possibility both of
    dumb servers and smart servers.  You'll need to write a web client
    in Guile, if one hasn't been written yet.  In short, a large project
    for an ambitious and careful hacker!

Regards,

Andy
-- 
http://wingolog.org/

Re: summer of code ideas

[Noah Lavine]

I've been toying with the idea of an AOT compiler for Guile, and I
think that would make a good summer project.

I think you'd want to compile from Tree-IL to GCC's GIMPLE format (the
highest-level intermediate representation they have, I think). One
interesting possibility would be basing it on Stalin, which compiles
R4RS to C and is apparently extremely good. It's GPL-licensed, but as
far as I know not FSF-copyrighted, so someone would either have to ask
the person for a copyright assignment or reimplement things. Either
way, there would be a substantial amount of work to be done just
getting basic translation to work and integrating with both GCC and
Guile, and then of course we'd like it to optimize things.

Noah

On Mon, Mar 7, 2011 at 2:51 PM, Andy Wingo <wingo@pobox.com> wrote:
> Hey all,
>
> We need to get together a few Guile-proposed projects for SoC students
> to hack on.  How about we collect them here in this thread.  I'll kick
> it off with one idea:
>
>  * A CPAN for Guile.  My initial thoughts are that you want a
>    combination of APT-like discovery and download of sources of
>    software, and a stowfs-like local storage, allowing you to update a
>    tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
>    back changes, etc.  See
>    http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
>    for more on $XDG_DATA_DIRS.
>
>    C extensions are outside the scope of this project.  IMO a CPAN for
>    Guile should not compile C helper libraries.
>
>    You can start here by doing a survey of what other schemes do:
>    chicken and racket being the obvious ones, and also perhaps taking a
>    look at cabal and other CPAN-alikes developed in the last 10 years.
>
>    You'd need to define a format for code -- tarballs or something.
>    You need to think about signing too, and the possibility both of
>    dumb servers and smart servers.  You'll need to write a web client
>    in Guile, if one hasn't been written yet.  In short, a large project
>    for an ambitious and careful hacker!
>
> Regards,
>
> Andy
> --
> http://wingolog.org/
>
>

Re: summer of code ideas

[Ludovic Courtès]

Hi!

Noah Lavine <noah.b.lavine@gmail.com> writes:

> I've been toying with the idea of an AOT compiler for Guile, and I
> think that would make a good summer project.

Really?  I thought you were toying with JIT.  :-)

> I think you'd want to compile from Tree-IL to GCC's GIMPLE format (the
> highest-level intermediate representation they have, I think).

Actually, I suspect there would be a lot of technical problems to solve,
notably because GCC is not a library and because passing it such complex
data structures may be far from trivial (see MELT).

Besides, for AOT, I was beguiled by Andy’s initial plan of using
exclusively Scheme tools such as Sassy.

Thanks,
Ludo’.

Re: summer of code ideas

[Noah Lavine]

Hello,

> Really?  I thought you were toying with JIT.  :-)

Yes, it's been a while since we talked about that :-). I still want to
do it, though. The current status as I see it is that I have a
prototype JIT that works, and a plausible way to integrate it into
Guile's VM. What needs to happen now is to write all of the opcodes as
JIT functions (the prototype had five or so opcodes). Since writing
them by hand would cause an absurd amount of code duplication, I
wanted to do it automatically. We talked about having a Scheme-based
language that could compile to both plain C and JIT, but decided that
would make the VM too complicated, so the current idea is to use a C
parser to parse the VM code and generate a JIT compiler from that.
That's one reason I've been contributing patches to the PEG branch. I
also started work on a C parser, which I was planning to publish once
the PEG branch was merged (and once the parser was done, of course).
So the JIT is actually moving along.

>> I think you'd want to compile from Tree-IL to GCC's GIMPLE format (the
>> highest-level intermediate representation they have, I think).
>
> Actually, I suspect there would be a lot of technical problems to solve,
> notably because GCC is not a library and because passing it such complex
> data structures may be far from trivial (see MELT).

> Besides, for AOT, I was beguiled by Andy’s initial plan of using
> exclusively Scheme tools such as Sassy.

Ah, I see. I was tempted by the idea of using GCC's already-excellent
optimizer as the backend to whatever Scheme-specific stuff we do. If
there's a way to do that, I think it would be worth quite a bit of
pain to use that.

I notice that GCC now has the ability to use plugins, so if we could
write one of those then perhaps it wouldn't be so bad. I also imagine
that if we approached them and said that a GSoC student would like to
write a frontend for them, they might be willing to help out a bit. I
think the goal should be to make the compiler part of a GCC frontend
developed with the rest of the compiler, so it would stay up to date
if interfaces changed, and also so that parts of it could be reused
for other things, leading to GCC developers making our compiler
faster.

Noah

Re: summer of code ideas

[Ludovic Courtès]

Hi!

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Yes, it's been a while since we talked about that :-). I still want to
> do it, though. The current status as I see it is that I have a
> prototype JIT that works, and a plausible way to integrate it into
> Guile's VM. What needs to happen now is to write all of the opcodes as
> JIT functions (the prototype had five or so opcodes). Since writing
> them by hand would cause an absurd amount of code duplication, I
> wanted to do it automatically.

Yes.

> We talked about having a Scheme-based language that could compile to
> both plain C and JIT, but decided that would make the VM too
> complicated,

Maybe I forgot to feed that thread, but I think it might be easier to
have a high-level representation the VM opcodes as sexps (possibly a
subset of Scheme), and then write the C files from that.

Though the main difficulty is to find a sufficiently expressive way to
write opcodes in.

> I notice that GCC now has the ability to use plugins,

The plug-in API doesn’t allow plug-ins to add new front-ends
(info "(gccint) Plugins").

Thanks,
Ludo’.

Re: summer of code ideas

[Noah Lavine]

Hello,

>> We talked about having a Scheme-based language that could compile to
>> both plain C and JIT, but decided that would make the VM too
>> complicated,
>
> Maybe I forgot to feed that thread, but I think it might be easier to
> have a high-level representation the VM opcodes as sexps (possibly a
> subset of Scheme), and then write the C files from that.
>
> Though the main difficulty is to find a sufficiently expressive way to
> write opcodes in.

Oh dear. My impression was that we had decided to use a C parser. If
not, I think the hardest part of making this work would be making the
C output look nice. Libjit and C are such similar languages that
making the translator would be quite simple.

>> I notice that GCC now has the ability to use plugins,
>
> The plug-in API doesn’t allow plug-ins to add new front-ends
> (info "(gccint) Plugins").

Ah, I see. I read their wiki a bit. It does look like using GCC would
be difficult, but I still think it would be worthwhile to do just
because they have such a nice compiler already written. It looks like
what you need to build a frontend is the ability to manipulate GCC's
data structures and the ability to write functions that C code can
call back to. I believe the FFI can do both of these things now (not
that it wouldn't be a pain to fit everything together).

Also, after looking at the MELT project and the Modula-2 frontend, I'd
say they might be receptive to the idea of using a high-level language
to write parts of GCC.

Noah

Re: summer of code ideas

[Ludovic Courtès]

Hello,

Noah Lavine <noah.b.lavine@gmail.com> writes:

>>> We talked about having a Scheme-based language that could compile to
>>> both plain C and JIT, but decided that would make the VM too
>>> complicated,
>>
>> Maybe I forgot to feed that thread, but I think it might be easier to
>> have a high-level representation the VM opcodes as sexps (possibly a
>> subset of Scheme), and then write the C files from that.
>>
>> Though the main difficulty is to find a sufficiently expressive way to
>> write opcodes in.
>
> Oh dear. My impression was that we had decided to use a C parser. If
> not, I think the hardest part of making this work would be making the
> C output look nice. Libjit and C are such similar languages that
> making the translator would be quite simple.

Well I think Andy preferred parsing C code and generating whatever the
JIT needs from that.  My intuition is that doing it the other way around
may be simpler and possibly more elegant, but I haven’t looked into the
details.

Anyway, you’re the one doing the actual work, so if you find a solution
that works well for everyone, then you’re done.  :-)

> Also, after looking at the MELT project and the Modula-2 frontend, I'd
> say they might be receptive to the idea of using a high-level language
> to write parts of GCC.

I think there’s no “they”—i.e., there are many hackers, and probably
diverging views.  In practice, MELT is still in a separate branch and
isn’t used internally, though.

Technically, there are a few issues with MELT: currently it requires
both the source and build tree of GCC because of its tight integration
with GCC’s internals (for instance it requires ‘gengtype’.)  This is
just to illustrate that using GCC as a library may prove to be
difficult.

Thanks,
Ludo’.

CPAN for Guile

[Andreas Rottmann]

Andy Wingo <wingo@pobox.com> writes:

> Hey all,
>
> We need to get together a few Guile-proposed projects for SoC students
> to hack on.  How about we collect them here in this thread.  I'll kick
> it off with one idea:
>
>   * A CPAN for Guile.  My initial thoughts are that you want a
>     combination of APT-like discovery and download of sources of
>     software, and a stowfs-like local storage, allowing you to update a
>     tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
>     back changes, etc.  See
>     http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
>     for more on $XDG_DATA_DIRS.
>
>     C extensions are outside the scope of this project.  IMO a CPAN for
>     Guile should not compile C helper libraries.
>
I'm (very slowly) working on getting dorodango[0] to work on Guile.  In
principle, nothing in Dorodango is inherently tied to R6RS -- it should
very reasonably be possible to use it for Guile-specific packages as
well.

[0] http://home.gna.org/dorodango/

It currently doesn't use a symlinking approach like GNU Stow or scsh
packages, but installs real files.  However, it keeps track of all
installed files (like APT/dpkg) and allows upgrades, uninstalling, etc.
It even contains a full-blown dependency resolver, transcribed from
aptitude's C++ implementation (in the process shrinking ~9KLoC to ~3KLoC
;-)).

>     You can start here by doing a survey of what other schemes do:
>     chicken and racket being the obvious ones, and also perhaps taking a
>     look at cabal and other CPAN-alikes developed in the last 10 years.
>
>     You'd need to define a format for code -- tarballs or something.
>     You need to think about signing too, and the possibility both of
>     dumb servers and smart servers.  You'll need to write a web client
>     in Guile, if one hasn't been written yet.  In short, a large project
>     for an ambitious and careful hacker!
>
Well, I certainly don't want to stop anyone from coming up with
something entirely Guile-specific, but I want to point out that there's
already a good amount of code out there, and even some documentation
;-).  I'd be thrilled however, if some ambitious, careful hacker would
join me and help me enhance dorodango to work perfectly as Guile's
package manager!

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: CPAN for Guile

[Andy Wingo]

On Mon 07 Mar 2011 21:13, Andreas Rottmann <a.rottmann@gmx.at> writes:

> I'm (very slowly) working on getting dorodango[0] to work on Guile.  In
> principle, nothing in Dorodango is inherently tied to R6RS -- it should
> very reasonably be possible to use it for Guile-specific packages as
> well.
>
> [0] http://home.gna.org/dorodango/

Looks very nice, and well thought-out.  Dorodango would be an excellent
starting point.

Alaric Snell-Pym gave us an overview of Chicken's system here:
http://rotty.yi.org/irclogs/freenode/#guile/2011-02-10/#e212

I agree with him that one of the most important aspects about
CPAN-alikes is the social one: how the system enables the hack, and with
what flavor.  For example right now Guile has a tight center, enforced
by the stability needs of Guile, as well as the copyright assignment
policy, then a wide-orbiting set of comet- and asteroid-like libraries.
A good CPAN-alike would smooth this out into a gradient of more and less
experimental hacks, to enable participation of folks with useful
libraries, but which don't need to be in Guile proper.

It would be good to also think about modules' test suites,
documentation, code coverage, code analysis and such, and be opinionated
about that.

It would be interesting to have the ability to add on, as an additional
source of code, repos with portable R6RS code.  I wonder how this would
affect us socially though.

Anyway, just words into the ether.  Happy hacking!

Andy
-- 
http://wingolog.org/

Re: CPAN for Guile

[Andreas Rottmann]

Andy Wingo <wingo@pobox.com> writes:

> On Mon 07 Mar 2011 21:13, Andreas Rottmann <a.rottmann@gmx.at> writes:
>
>> I'm (very slowly) working on getting dorodango[0] to work on Guile.  In
>> principle, nothing in Dorodango is inherently tied to R6RS -- it should
>> very reasonably be possible to use it for Guile-specific packages as
>> well.
>>
>> [0] http://home.gna.org/dorodango/
>
> Looks very nice, and well thought-out.  Dorodango would be an excellent
> starting point.
>
> Alaric Snell-Pym gave us an overview of Chicken's system here:
> http://rotty.yi.org/irclogs/freenode/#guile/2011-02-10/#e212
>
> I agree with him that one of the most important aspects about
> CPAN-alikes is the social one: how the system enables the hack, and with
> what flavor.  For example right now Guile has a tight center, enforced
> by the stability needs of Guile, as well as the copyright assignment
> policy, then a wide-orbiting set of comet- and asteroid-like libraries.
> A good CPAN-alike would smooth this out into a gradient of more and less
> experimental hacks, to enable participation of folks with useful
> libraries, but which don't need to be in Guile proper.
>
> It would be good to also think about modules' test suites,
> documentation, code coverage, code analysis and such, and be opinionated
> about that.
>
Yeah, that are all areas I envisioned for dorodango, but have not yet
tackled.

> It would be interesting to have the ability to add on, as an additional
> source of code, repos with portable R6RS code.  
>
Indeed. Dorodango already has the capability to deal with multiple
repositories, and a Guile-specific version of Dorodango would probably
come with a default configuration that points to Guile's (default)
reprository; users would then add additional ones according to their
likes.

> I wonder how this would affect us socially though.
>
No idea :-).

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: summer of code ideas

[Noah Lavine]

Also, about your CPAN for guile (CGAN?) idea - it seems like with what
you said about stowfs that what you're looking for is similar to Nix,
but used to install Guile packages. (www.nixos.org)

Do you think it would be useful to just compile a version of Nix with
a different root directory, make some Guile bindings, and use that?

Noah

On Mon, Mar 7, 2011 at 2:51 PM, Andy Wingo <wingo@pobox.com> wrote:
> Hey all,
>
> We need to get together a few Guile-proposed projects for SoC students
> to hack on.  How about we collect them here in this thread.  I'll kick
> it off with one idea:
>
>  * A CPAN for Guile.  My initial thoughts are that you want a
>    combination of APT-like discovery and download of sources of
>    software, and a stowfs-like local storage, allowing you to update a
>    tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
>    back changes, etc.  See
>    http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
>    for more on $XDG_DATA_DIRS.
>
>    C extensions are outside the scope of this project.  IMO a CPAN for
>    Guile should not compile C helper libraries.
>
>    You can start here by doing a survey of what other schemes do:
>    chicken and racket being the obvious ones, and also perhaps taking a
>    look at cabal and other CPAN-alikes developed in the last 10 years.
>
>    You'd need to define a format for code -- tarballs or something.
>    You need to think about signing too, and the possibility both of
>    dumb servers and smart servers.  You'll need to write a web client
>    in Guile, if one hasn't been written yet.  In short, a large project
>    for an ambitious and careful hacker!
>
> Regards,
>
> Andy
> --
> http://wingolog.org/
>
>

Re: summer of code ideas

[Andy Wingo]

On Mon 07 Mar 2011 21:40, Noah Lavine <noah.b.lavine@gmail.com> writes:

> Also, about your CPAN for guile (CGAN?) idea - it seems like with what
> you said about stowfs that what you're looking for is similar to Nix,
> but used to install Guile packages. (www.nixos.org)

Yes indeed!

> Do you think it would be useful to just compile a version of Nix with
> a different root directory, make some Guile bindings, and use that?

I want any user that has Guile to be able to install packages off the
net easily.  Requiring them to use Nix is probably too much (though Nix
is indeed impressive, as Ludovic will tell you! :-))

Andy
-- 
http://wingolog.org/

Re: summer of code ideas

[Ludovic Courtès]

Hello!

Actually, a fun project would be to implement Nix storage model, its
build primitive (‘derivation’), and assorted tools (garbage collector,
etc.) in Guile Scheme.  It doesn’t seem unrealistic, though it’s perhaps
another project; OTOH you mentioned rollback so it’s not completely
irrelevant either.  ;-)

Another (difficult) question is how that would integrate with the rest
of GNU—the GNU Build System, GSRC, ELPA, Stowfs [0], etc.

Thanks,
Ludo’.

[0] http://www.gnu.org/software/hurd/hurd/translator/unionfs.html