summer of code ideas

summer of code ideas

[Andy Wingo]

Hey all,

We need to get together a few Guile-proposed projects for SoC students
to hack on.  How about we collect them here in this thread.  I'll kick
it off with one idea:

  * A CPAN for Guile.  My initial thoughts are that you want a
    combination of APT-like discovery and download of sources of
    software, and a stowfs-like local storage, allowing you to update a
    tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
    back changes, etc.  See
    http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
    for more on $XDG_DATA_DIRS.

    C extensions are outside the scope of this project.  IMO a CPAN for
    Guile should not compile C helper libraries.

    You can start here by doing a survey of what other schemes do:
    chicken and racket being the obvious ones, and also perhaps taking a
    look at cabal and other CPAN-alikes developed in the last 10 years.

    You'd need to define a format for code -- tarballs or something.
    You need to think about signing too, and the possibility both of
    dumb servers and smart servers.  You'll need to write a web client
    in Guile, if one hasn't been written yet.  In short, a large project
    for an ambitious and careful hacker!

Regards,

Andy
-- 
http://wingolog.org/

Re: summer of code ideas

[Noah Lavine]

I've been toying with the idea of an AOT compiler for Guile, and I
think that would make a good summer project.

I think you'd want to compile from Tree-IL to GCC's GIMPLE format (the
highest-level intermediate representation they have, I think). One
interesting possibility would be basing it on Stalin, which compiles
R4RS to C and is apparently extremely good. It's GPL-licensed, but as
far as I know not FSF-copyrighted, so someone would either have to ask
the person for a copyright assignment or reimplement things. Either
way, there would be a substantial amount of work to be done just
getting basic translation to work and integrating with both GCC and
Guile, and then of course we'd like it to optimize things.

Noah

On Mon, Mar 7, 2011 at 2:51 PM, Andy Wingo <wingo@pobox.com> wrote:
> Hey all,
>
> We need to get together a few Guile-proposed projects for SoC students
> to hack on.  How about we collect them here in this thread.  I'll kick
> it off with one idea:
>
>  * A CPAN for Guile.  My initial thoughts are that you want a
>    combination of APT-like discovery and download of sources of
>    software, and a stowfs-like local storage, allowing you to update a
>    tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
>    back changes, etc.  See
>    http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
>    for more on $XDG_DATA_DIRS.
>
>    C extensions are outside the scope of this project.  IMO a CPAN for
>    Guile should not compile C helper libraries.
>
>    You can start here by doing a survey of what other schemes do:
>    chicken and racket being the obvious ones, and also perhaps taking a
>    look at cabal and other CPAN-alikes developed in the last 10 years.
>
>    You'd need to define a format for code -- tarballs or something.
>    You need to think about signing too, and the possibility both of
>    dumb servers and smart servers.  You'll need to write a web client
>    in Guile, if one hasn't been written yet.  In short, a large project
>    for an ambitious and careful hacker!
>
> Regards,
>
> Andy
> --
> http://wingolog.org/
>
>

Re: summer of code ideas

[Ludovic Courtès]

Hi!

Noah Lavine <noah.b.lavine@gmail.com> writes:

> I've been toying with the idea of an AOT compiler for Guile, and I
> think that would make a good summer project.

Really?  I thought you were toying with JIT.  :-)

> I think you'd want to compile from Tree-IL to GCC's GIMPLE format (the
> highest-level intermediate representation they have, I think).

Actually, I suspect there would be a lot of technical problems to solve,
notably because GCC is not a library and because passing it such complex
data structures may be far from trivial (see MELT).

Besides, for AOT, I was beguiled by Andy’s initial plan of using
exclusively Scheme tools such as Sassy.

Thanks,
Ludo’.

Re: summer of code ideas

[Noah Lavine]

Hello,

> Really?  I thought you were toying with JIT.  :-)

Yes, it's been a while since we talked about that :-). I still want to
do it, though. The current status as I see it is that I have a
prototype JIT that works, and a plausible way to integrate it into
Guile's VM. What needs to happen now is to write all of the opcodes as
JIT functions (the prototype had five or so opcodes). Since writing
them by hand would cause an absurd amount of code duplication, I
wanted to do it automatically. We talked about having a Scheme-based
language that could compile to both plain C and JIT, but decided that
would make the VM too complicated, so the current idea is to use a C
parser to parse the VM code and generate a JIT compiler from that.
That's one reason I've been contributing patches to the PEG branch. I
also started work on a C parser, which I was planning to publish once
the PEG branch was merged (and once the parser was done, of course).
So the JIT is actually moving along.

>> I think you'd want to compile from Tree-IL to GCC's GIMPLE format (the
>> highest-level intermediate representation they have, I think).
>
> Actually, I suspect there would be a lot of technical problems to solve,
> notably because GCC is not a library and because passing it such complex
> data structures may be far from trivial (see MELT).

> Besides, for AOT, I was beguiled by Andy’s initial plan of using
> exclusively Scheme tools such as Sassy.

Ah, I see. I was tempted by the idea of using GCC's already-excellent
optimizer as the backend to whatever Scheme-specific stuff we do. If
there's a way to do that, I think it would be worth quite a bit of
pain to use that.

I notice that GCC now has the ability to use plugins, so if we could
write one of those then perhaps it wouldn't be so bad. I also imagine
that if we approached them and said that a GSoC student would like to
write a frontend for them, they might be willing to help out a bit. I
think the goal should be to make the compiler part of a GCC frontend
developed with the rest of the compiler, so it would stay up to date
if interfaces changed, and also so that parts of it could be reused
for other things, leading to GCC developers making our compiler
faster.

Noah

Re: summer of code ideas

[Ludovic Courtès]

Hi!

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Yes, it's been a while since we talked about that :-). I still want to
> do it, though. The current status as I see it is that I have a
> prototype JIT that works, and a plausible way to integrate it into
> Guile's VM. What needs to happen now is to write all of the opcodes as
> JIT functions (the prototype had five or so opcodes). Since writing
> them by hand would cause an absurd amount of code duplication, I
> wanted to do it automatically.

Yes.

> We talked about having a Scheme-based language that could compile to
> both plain C and JIT, but decided that would make the VM too
> complicated,

Maybe I forgot to feed that thread, but I think it might be easier to
have a high-level representation the VM opcodes as sexps (possibly a
subset of Scheme), and then write the C files from that.

Though the main difficulty is to find a sufficiently expressive way to
write opcodes in.

> I notice that GCC now has the ability to use plugins,

The plug-in API doesn’t allow plug-ins to add new front-ends
(info "(gccint) Plugins").

Thanks,
Ludo’.

Re: summer of code ideas

[Noah Lavine]

Hello,

>> We talked about having a Scheme-based language that could compile to
>> both plain C and JIT, but decided that would make the VM too
>> complicated,
>
> Maybe I forgot to feed that thread, but I think it might be easier to
> have a high-level representation the VM opcodes as sexps (possibly a
> subset of Scheme), and then write the C files from that.
>
> Though the main difficulty is to find a sufficiently expressive way to
> write opcodes in.

Oh dear. My impression was that we had decided to use a C parser. If
not, I think the hardest part of making this work would be making the
C output look nice. Libjit and C are such similar languages that
making the translator would be quite simple.

>> I notice that GCC now has the ability to use plugins,
>
> The plug-in API doesn’t allow plug-ins to add new front-ends
> (info "(gccint) Plugins").

Ah, I see. I read their wiki a bit. It does look like using GCC would
be difficult, but I still think it would be worthwhile to do just
because they have such a nice compiler already written. It looks like
what you need to build a frontend is the ability to manipulate GCC's
data structures and the ability to write functions that C code can
call back to. I believe the FFI can do both of these things now (not
that it wouldn't be a pain to fit everything together).

Also, after looking at the MELT project and the Modula-2 frontend, I'd
say they might be receptive to the idea of using a high-level language
to write parts of GCC.

Noah

Re: summer of code ideas

[Ludovic Courtès]

Hello,

Noah Lavine <noah.b.lavine@gmail.com> writes:

>>> We talked about having a Scheme-based language that could compile to
>>> both plain C and JIT, but decided that would make the VM too
>>> complicated,
>>
>> Maybe I forgot to feed that thread, but I think it might be easier to
>> have a high-level representation the VM opcodes as sexps (possibly a
>> subset of Scheme), and then write the C files from that.
>>
>> Though the main difficulty is to find a sufficiently expressive way to
>> write opcodes in.
>
> Oh dear. My impression was that we had decided to use a C parser. If
> not, I think the hardest part of making this work would be making the
> C output look nice. Libjit and C are such similar languages that
> making the translator would be quite simple.

Well I think Andy preferred parsing C code and generating whatever the
JIT needs from that.  My intuition is that doing it the other way around
may be simpler and possibly more elegant, but I haven’t looked into the
details.

Anyway, you’re the one doing the actual work, so if you find a solution
that works well for everyone, then you’re done.  :-)

> Also, after looking at the MELT project and the Modula-2 frontend, I'd
> say they might be receptive to the idea of using a high-level language
> to write parts of GCC.

I think there’s no “they”—i.e., there are many hackers, and probably
diverging views.  In practice, MELT is still in a separate branch and
isn’t used internally, though.

Technically, there are a few issues with MELT: currently it requires
both the source and build tree of GCC because of its tight integration
with GCC’s internals (for instance it requires ‘gengtype’.)  This is
just to illustrate that using GCC as a library may prove to be
difficult.

Thanks,
Ludo’.

CPAN for Guile

[Andreas Rottmann]

Andy Wingo <wingo@pobox.com> writes:

> Hey all,
>
> We need to get together a few Guile-proposed projects for SoC students
> to hack on.  How about we collect them here in this thread.  I'll kick
> it off with one idea:
>
>   * A CPAN for Guile.  My initial thoughts are that you want a
>     combination of APT-like discovery and download of sources of
>     software, and a stowfs-like local storage, allowing you to update a
>     tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
>     back changes, etc.  See
>     http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
>     for more on $XDG_DATA_DIRS.
>
>     C extensions are outside the scope of this project.  IMO a CPAN for
>     Guile should not compile C helper libraries.
>
I'm (very slowly) working on getting dorodango[0] to work on Guile.  In
principle, nothing in Dorodango is inherently tied to R6RS -- it should
very reasonably be possible to use it for Guile-specific packages as
well.

[0] http://home.gna.org/dorodango/

It currently doesn't use a symlinking approach like GNU Stow or scsh
packages, but installs real files.  However, it keeps track of all
installed files (like APT/dpkg) and allows upgrades, uninstalling, etc.
It even contains a full-blown dependency resolver, transcribed from
aptitude's C++ implementation (in the process shrinking ~9KLoC to ~3KLoC
;-)).

>     You can start here by doing a survey of what other schemes do:
>     chicken and racket being the obvious ones, and also perhaps taking a
>     look at cabal and other CPAN-alikes developed in the last 10 years.
>
>     You'd need to define a format for code -- tarballs or something.
>     You need to think about signing too, and the possibility both of
>     dumb servers and smart servers.  You'll need to write a web client
>     in Guile, if one hasn't been written yet.  In short, a large project
>     for an ambitious and careful hacker!
>
Well, I certainly don't want to stop anyone from coming up with
something entirely Guile-specific, but I want to point out that there's
already a good amount of code out there, and even some documentation
;-).  I'd be thrilled however, if some ambitious, careful hacker would
join me and help me enhance dorodango to work perfectly as Guile's
package manager!

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: CPAN for Guile

[Andy Wingo]

On Mon 07 Mar 2011 21:13, Andreas Rottmann <a.rottmann@gmx.at> writes:

> I'm (very slowly) working on getting dorodango[0] to work on Guile.  In
> principle, nothing in Dorodango is inherently tied to R6RS -- it should
> very reasonably be possible to use it for Guile-specific packages as
> well.
>
> [0] http://home.gna.org/dorodango/

Looks very nice, and well thought-out.  Dorodango would be an excellent
starting point.

Alaric Snell-Pym gave us an overview of Chicken's system here:
http://rotty.yi.org/irclogs/freenode/#guile/2011-02-10/#e212

I agree with him that one of the most important aspects about
CPAN-alikes is the social one: how the system enables the hack, and with
what flavor.  For example right now Guile has a tight center, enforced
by the stability needs of Guile, as well as the copyright assignment
policy, then a wide-orbiting set of comet- and asteroid-like libraries.
A good CPAN-alike would smooth this out into a gradient of more and less
experimental hacks, to enable participation of folks with useful
libraries, but which don't need to be in Guile proper.

It would be good to also think about modules' test suites,
documentation, code coverage, code analysis and such, and be opinionated
about that.

It would be interesting to have the ability to add on, as an additional
source of code, repos with portable R6RS code.  I wonder how this would
affect us socially though.

Anyway, just words into the ether.  Happy hacking!

Andy
-- 
http://wingolog.org/

Re: CPAN for Guile

[Andreas Rottmann]

Andy Wingo <wingo@pobox.com> writes:

> On Mon 07 Mar 2011 21:13, Andreas Rottmann <a.rottmann@gmx.at> writes:
>
>> I'm (very slowly) working on getting dorodango[0] to work on Guile.  In
>> principle, nothing in Dorodango is inherently tied to R6RS -- it should
>> very reasonably be possible to use it for Guile-specific packages as
>> well.
>>
>> [0] http://home.gna.org/dorodango/
>
> Looks very nice, and well thought-out.  Dorodango would be an excellent
> starting point.
>
> Alaric Snell-Pym gave us an overview of Chicken's system here:
> http://rotty.yi.org/irclogs/freenode/#guile/2011-02-10/#e212
>
> I agree with him that one of the most important aspects about
> CPAN-alikes is the social one: how the system enables the hack, and with
> what flavor.  For example right now Guile has a tight center, enforced
> by the stability needs of Guile, as well as the copyright assignment
> policy, then a wide-orbiting set of comet- and asteroid-like libraries.
> A good CPAN-alike would smooth this out into a gradient of more and less
> experimental hacks, to enable participation of folks with useful
> libraries, but which don't need to be in Guile proper.
>
> It would be good to also think about modules' test suites,
> documentation, code coverage, code analysis and such, and be opinionated
> about that.
>
Yeah, that are all areas I envisioned for dorodango, but have not yet
tackled.

> It would be interesting to have the ability to add on, as an additional
> source of code, repos with portable R6RS code.  
>
Indeed. Dorodango already has the capability to deal with multiple
repositories, and a Guile-specific version of Dorodango would probably
come with a default configuration that points to Guile's (default)
reprository; users would then add additional ones according to their
likes.

> I wonder how this would affect us socially though.
>
No idea :-).

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: summer of code ideas

[Noah Lavine]

Also, about your CPAN for guile (CGAN?) idea - it seems like with what
you said about stowfs that what you're looking for is similar to Nix,
but used to install Guile packages. (www.nixos.org)

Do you think it would be useful to just compile a version of Nix with
a different root directory, make some Guile bindings, and use that?

Noah

On Mon, Mar 7, 2011 at 2:51 PM, Andy Wingo <wingo@pobox.com> wrote:
> Hey all,
>
> We need to get together a few Guile-proposed projects for SoC students
> to hack on.  How about we collect them here in this thread.  I'll kick
> it off with one idea:
>
>  * A CPAN for Guile.  My initial thoughts are that you want a
>    combination of APT-like discovery and download of sources of
>    software, and a stowfs-like local storage, allowing you to update a
>    tree of symlinks in $XDG_DATA_DIRS/guile/2.0/ to add packages, roll
>    back changes, etc.  See
>    http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
>    for more on $XDG_DATA_DIRS.
>
>    C extensions are outside the scope of this project.  IMO a CPAN for
>    Guile should not compile C helper libraries.
>
>    You can start here by doing a survey of what other schemes do:
>    chicken and racket being the obvious ones, and also perhaps taking a
>    look at cabal and other CPAN-alikes developed in the last 10 years.
>
>    You'd need to define a format for code -- tarballs or something.
>    You need to think about signing too, and the possibility both of
>    dumb servers and smart servers.  You'll need to write a web client
>    in Guile, if one hasn't been written yet.  In short, a large project
>    for an ambitious and careful hacker!
>
> Regards,
>
> Andy
> --
> http://wingolog.org/
>
>

Re: summer of code ideas

[Andy Wingo]

On Mon 07 Mar 2011 21:40, Noah Lavine <noah.b.lavine@gmail.com> writes:

> Also, about your CPAN for guile (CGAN?) idea - it seems like with what
> you said about stowfs that what you're looking for is similar to Nix,
> but used to install Guile packages. (www.nixos.org)

Yes indeed!

> Do you think it would be useful to just compile a version of Nix with
> a different root directory, make some Guile bindings, and use that?

I want any user that has Guile to be able to install packages off the
net easily.  Requiring them to use Nix is probably too much (though Nix
is indeed impressive, as Ludovic will tell you! :-))

Andy
-- 
http://wingolog.org/

Re: summer of code ideas

[Ludovic Courtès]

Hello!

Actually, a fun project would be to implement Nix storage model, its
build primitive (‘derivation’), and assorted tools (garbage collector,
etc.) in Guile Scheme.  It doesn’t seem unrealistic, though it’s perhaps
another project; OTOH you mentioned rollback so it’s not completely
irrelevant either.  ;-)

Another (difficult) question is how that would integrate with the rest
of GNU—the GNU Build System, GSRC, ELPA, Stowfs [0], etc.

Thanks,
Ludo’.

[0] http://www.gnu.org/software/hurd/hurd/translator/unionfs.html

Re: summer of code ideas

[Mike Gran]

>Hey all,
>
>We need to get together a few Guile-proposed projects for SoC students
>to hack on.  How about we collect them here in this thread.  I'll kick
>it off with one idea:
>
>  * A CPAN for Guile. 

A similar idea.

Define a syntax for use-modules to pull modules off the net, something 
like

   (use-modules (url://a-url.com library module #:optional a-rev-number))

Update modules on startup.  Cache modules and use last cached version if
internet access is not available.  Create server solution for modules.
Use openID or related to control access to Guile module server.  Use
revision-control-like system to server up proper revision of module.

Also, add syntax for procedure that, when called on local system,
actually execute on remote system.

(use-modules ((url://a-url.com library module #:optional a-rev-number)
              #:rpc external-func)) 

-Mike

Re: summer of code ideas

[Jose A. Ortega Ruiz]

On Mon, Mar 07 2011, Mike Gran wrote:

[...]

> A similar idea.
>
> Define a syntax for use-modules to pull modules off the net, something 
> like
>
>    (use-modules (url://a-url.com library module #:optional a-rev-number))
>

FWIW, i think this is a bad idea.  It intermingles two concerns that are
othogonal, namely installing a package and using it.  I don't want to
have to specify in my code where a module comes from.  This is how
PLaneT currently works, and their experience has been so negative that
they're redesigning the whole thing now.

jao
-- 
Computer games don't affect kids; I mean if Pac Man affected us
as kids, we would all be running around in darkened rooms, munching
magic pills, and listening to repetitive electronic music.
 - Kristian Wilson, Nintendo Inc.

Re: summer of code ideas

[Mark H Weaver]

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:
>>    (use-modules (url://a-url.com library module #:optional a-rev-number))
>
> FWIW, i think this is a bad idea.  It intermingles two concerns that are
> othogonal, namely installing a package and using it.

I very strongly agree with jao.  Systems like this, e.g. Python Eggs,
have been a major headache for distributions to deal with.  Debian
actually takes the time to disable this automatic downloading and
installing functionality from their Python packages, and I'm glad for
it.  I am very security conscious, and the thought of software being
automatically installed "on-demand" from untrusted sources when I run a
program, or maybe even when I first use some particular functionality of
that program, is very disconcerting.

Maybe not everyone wants this, but as a Debian and gNewSense user, I
want my distribution to be an intermediary for most of the software I
use.  I trust them more than I trust most upstreams to ensure that the
software has been somewhat vetted for license issues, security problems,
anti-features, etc.  I want my distribution to be able to modify the
packages as necessary to make them work well together and with the rest
of the system.  I also want experimental distributions to be able to
make significant changes to packages to fit within their new ideas of
how the system should be set up.

Furthermore, there are many thorny issues involved with package
management that are very hard to get right, and most of the new crop of
language-specific package systems like this are half-baked at best.

For example, how do you ensure security?  Debian has a reasonably well
thought out system for using digital signatures for this.  How will we
handle it?

Also, how will we handle versioning?  It is very hard to do this
properly.  Sometimes you want the cutting-edge version of something, and
sometimes you want stability.  It is not enough to simply designate a
stable version of each individual package.  A stable system requires
that all the individual pieces have been tested together as a whole, as
is done in Debian and other distros.  How will we handle it?

I don't mean to be a wet blanket, because I can certainly see the appeal
of a system like this, but let's please be careful not to repeat the
many mistakes that other similar systems have made.  It is a very thorny
problem.

    Regards,
      Mark

Re: summer of code ideas

[Mike Gran]

From: Mark H Weaver <mhw@netris.org>
>To: Jose A. Ortega Ruiz <jao@gnu.org>
>Cc: guile-devel@gnu.org
>Sent: Monday, March 7, 2011 5:10 PM
>Subject: Re: summer of code ideas
>
>>"Jose A. Ortega Ruiz" <jao@gnu.org> writes:
>>>    (use-modules (url://a-url.com library module #:optional a-rev-number))
>>
>> FWIW, i think this is a bad idea.  It intermingles two concerns that are
>> othogonal, namely installing a package and using it.
>
>I very strongly agree with jao.  Systems like this, e.g. Python Eggs,
>have been a major headache for distributions to deal with.  Debian
>actually takes the time to disable this automatic downloading and
>installing functionality from their Python packages, and I'm glad for
>it.  I am very security conscious, and the thought of software being
>automatically installed "on-demand" from untrusted sources when I run a
>program, or maybe even when I first use some particular functionality of
>that program, is very disconcerting.

I think people are conflating 'an idea' with 'the idea' here.  The call was
for possible SOC ideas.

At my work, we sometimes operate this way: field engineers use client
programs that are rather like AJAX webapps.  The head office updates a
server.  It works for us.

I didn't suggest that for all time Guile must officially such a system.

Sheesh.

-Mike