Minimal Guile

Minimal Guile

[Mike Gran]

Hi-

I've been keeping up with some of the discussion on other gnu lists.

One of the criticisms at the moment is that Guile is
 
1. bloated
2. hard to distribute.
 
Re point 2: hard to distribute.
 
A while ago I was looking at the idea of minimizing the number of
files needed to ship Guile as a dependency.  At the time, I thought
that one could retool the build so that it produced
- a tar.gz of the distributable header files
- libguile as a monolithic .so or .dll with no dependencies.
  Dependencies would be wrapped into the monolithic .so or .dll.
- all the compiled .go files as a tar file
- and a Guile executable that used both of the above
 
Guile and libguile would read the .go without unpacking the tar file
by using libtar.
 
There's no real advantage in this for the GNU system, but, it would
make it easier to make a distributable cross-build for non-GNU
systems.  This would make it easier to set up a download repository
of prebuilt .so, .dll, and go-lib tar files.
 
Re point 1: bloated
 
Once the infrastructure for the above is in place, one could set
up an alternate cross-build system that produced a version of
Guile that is
 
- ASCII only, "C" locale only
- Fixnum integers only
- No ability to load ltdl or ffi
- no ability to use whatever libcrypt is used for
 
This would be for the purpose of being a minimalist extension
engine.
 
What do you think?
 
-Mike Gran

Re: Minimal Guile

[Noah Lavine]

This is an interesting point.

I was thinking about it though, and I think there is another thing
that is causing a lot of this "bloat": all of the Scheme modules we
are adding.

There is a tension here, because having lots of modules is very
important for using Guile as a language to write programs in. But when
it is used as an extension to other languages, most of them are not
useful. So if I were dreaming and not thinking about how hard it is to
write code, I would get a system that lets me pick which modules I
want to distribute and bundle them up with libguile into a package
that can be distributed and built on its own.

This would also end up addressing some of your examples, if this
hypothetical packager grew the ability to know about C as well as
Scheme modules. Then it could automatically remove the C code that
implements functions that were not wanted.

This is just an idea, though. This seems like a huge pain to code.

Noah

On Mon, Dec 19, 2011 at 3:35 PM, Mike Gran <spk121@yahoo.com> wrote:
> Hi-
>
> I've been keeping up with some of the discussion on other gnu lists.
>
> One of the criticisms at the moment is that Guile is
>
> 1. bloated
> 2. hard to distribute.
>
> Re point 2: hard to distribute.
>
> A while ago I was looking at the idea of minimizing the number of
> files needed to ship Guile as a dependency.  At the time, I thought
> that one could retool the build so that it produced
> - a tar.gz of the distributable header files
> - libguile as a monolithic .so or .dll with no dependencies.
>   Dependencies would be wrapped into the monolithic .so or .dll.
> - all the compiled .go files as a tar file
> - and a Guile executable that used both of the above
>
> Guile and libguile would read the .go without unpacking the tar file
> by using libtar.
>
> There's no real advantage in this for the GNU system, but, it would
> make it easier to make a distributable cross-build for non-GNU
> systems.  This would make it easier to set up a download repository
> of prebuilt .so, .dll, and go-lib tar files.
>
> Re point 1: bloated
>
> Once the infrastructure for the above is in place, one could set
> up an alternate cross-build system that produced a version of
> Guile that is
>
> - ASCII only, "C" locale only
> - Fixnum integers only
> - No ability to load ltdl or ffi
> - no ability to use whatever libcrypt is used for
>
> This would be for the purpose of being a minimalist extension
> engine.
>
> What do you think?
>
> -Mike Gran
>

Re: Minimal Guile

[Andy Wingo]

Hi Noah,

On Mon 19 Dec 2011 23:32, Noah Lavine <noah.b.lavine@gmail.com> writes:

> I was thinking about it though, and I think there is another thing
> that is causing a lot of this "bloat": all of the Scheme modules we
> are adding.

Do you really think so?  It's more installed size, but it doesn't
affect runtime speed, runtime size, or compile-time portability.

> There is a tension here, because having lots of modules is very
> important for using Guile as a language to write programs in. But when
> it is used as an extension to other languages, most of them are not
> useful.

A point of clarification: as an extension language, Guile is meant to be
a language to extend programs, not a language to extend other languages.

> So if I were dreaming and not thinking about how hard it is to
> write code, I would get a system that lets me pick which modules I
> want to distribute and bundle them up with libguile into a package
> that can be distributed and built on its own.

It sounds more interesting to bundle up a stripped-down build product,
rather than a stripped-down source package.

YMMV of course :)

Andy
-- 
http://wingolog.org/

Re: Minimal Guile

[Mike Gran]

> From: Mike Gran <spk121@yahoo.com>
> 
> Hi-
> 
> Re point 2: hard to distribute.
>  
> A while ago I was looking at the idea of minimizing the number of
> files needed to ship Guile as a dependency.  At the time, I thought
> that one could retool the build so that it produced
> - a tar.gz of the distributable header files
> - libguile as a monolithic .so or .dll with no dependencies.
>   Dependencies would be wrapped into the monolithic .so or .dll.
> - all the compiled .go files as a tar file
> - and a Guile executable that used both of the above

I'm replying to myself.  LOL.  But I agree with myself that this
sounds like a good idea to me.
 
I think as soon as I get a spare cycle, I'm going to make an
experimental branch where all the core .scm and .go files are in
a single archive file that will act like a read-only filesystem.
 
It'll be fun to try to minimize it down to just
the guile executable, libguile-*, and a scheme archive file.  And it
might help with distribution of prebuilt versions.
 
To keep it GNU, I looked around to see what library-style GNU
solutions exist for treating an archive file like a filesystem.
Oddly, no good documented (L)GPL GNU options dealing with tar, cpio,
pax, or 'ar' in an API way.  But, there is a solution for .iso 
files: libiso9660 from GNU libcdio.
 
OK?  Any objections?
 
Thanks,
 
Mike Gran

Re: Minimal Guile

[Andy Wingo]

On Wed 21 Dec 2011 11:03, Mike Gran <spk121@yahoo.com> writes:

>> A while ago I was looking at the idea of minimizing the number of
>> files needed to ship Guile as a dependency.  At the time, I thought
>> that one could retool the build so that it produced
>> - a tar.gz of the distributable header files
>> - libguile as a monolithic .so or .dll with no dependencies.
>>   Dependencies would be wrapped into the monolithic .so or .dll.
>> - all the compiled .go files as a tar file
>> - and a Guile executable that used both of the above
>
> I'm replying to myself.  LOL.  But I agree with myself that this
> sounds like a good idea to me.

That's good that you're replying to yourself, because I (for one)
overlooked this part of your message.  Too much mail lately! :-)

> I think as soon as I get a spare cycle, I'm going to make an
> experimental branch where all the core .scm and .go files are in
> a single archive file that will act like a read-only filesystem.

Sounds like an interesting experiment.  If you find yourself getting
close to something workable, feel free to make a new branch in Guile
git.

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: Minimal Guile

[Noah Lavine]

Hello,

> To keep it GNU, I looked around to see what library-style GNU
> solutions exist for treating an archive file like a filesystem.
> Oddly, no good documented (L)GPL GNU options dealing with tar, cpio,
> pax, or 'ar' in an API way.  But, there is a solution for .iso
> files: libiso9660 from GNU libcdio.

Emacs has the ability to look inside tar files, so there might be
something in there.

That is just a thought, though. I can think of reasons you might not
want to do it.

This project sounds very interesting. Good luck!

Noah

Re: Minimal Guile

[David Kastrup]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hello,
>
>> To keep it GNU, I looked around to see what library-style GNU
>> solutions exist for treating an archive file like a filesystem.
>> Oddly, no good documented (L)GPL GNU options dealing with tar, cpio,
>> pax, or 'ar' in an API way.  But, there is a solution for .iso
>> files: libiso9660 from GNU libcdio.
>
> Emacs has the ability to look inside tar files, so there might be
> something in there.

No, it doesn't.  It just reads the whole tar file into a buffer, and
picks out parts on demand.  Convenient, but a memory hog.

-- 
David Kastrup

Re: Minimal Guile

[Antono Vasiljev]

On Wed, 2011-12-21 at 08:03 -0800, Mike Gran wrote:
 
> To keep it GNU, I looked around to see what library-style GNU
> solutions exist for treating an archive file like a filesystem.
> Oddly, no good documented (L)GPL GNU options dealing with tar, cpio,
> pax, or 'ar' in an API way.  But, there is a solution for .iso 
> files: libiso9660 from GNU libcdio.

You may be interested in tool made by Alexander Larson:

http://blogs.gnome.org/alexl/2011/09/30/rethinking-the-linux-distibution/
http://people.gnome.org/~alexl/glick2/
http://git.gnome.org/browse/glick2


-- 
Antono Vasiljev <self@antono.info>
Red Brigade

Re: Minimal Guile

[Ludovic Courtès]

Hi Mike!

Mike Gran <spk121@yahoo.com> skribis:

> It'll be fun to try to minimize it down to just
> the guile executable, libguile-*, and a scheme archive file.  And it
> might help with distribution of prebuilt versions.

Are you saying that, say, Lilypond’s tarball would contain libguile.so
and the .go files?  (I guess not but then I don’t get it.  ;-))

> To keep it GNU, I looked around to see what library-style GNU
> solutions exist for treating an archive file like a filesystem.
> Oddly, no good documented (L)GPL GNU options dealing with tar, cpio,
> pax, or 'ar' in an API way.  But, there is a solution for .iso 
> files: libiso9660 from GNU libcdio.

I would borrow code from Bigloo, Chicken, or some other Scheme
implementation that comes with that, or write it anew in Scheme.

Thanks,
Ludo’.

Re: Minimal Guile

[Mike Gran]

> From: Ludovic Courtès <ludo@gnu.org>
 > Hi Mike!
> 
> Mike Gran <spk121@yahoo.com> skribis:
> 
>>  It'll be fun to try to minimize it down to just
>>  the guile executable, libguile-*, and a scheme archive file.  And it
>>  might help with distribution of prebuilt versions.
> 
> Are you saying that, say, Lilypond’s tarball would contain libguile.so
> and the .go files?  (I guess not but then I don’t get it.  ;-))
 
I'm saying that you could create a Guile build that would make only
three non-documentation output products: libguile.so, all the .go files
in a single .iso or .tar file, and the Guile executable.
 
The libguile would be modified so that the .iso or .tar file would
never be unpacked.  Guile would look inside the .tar or .iso for the 
compiled .go files.
 
(I'm assuming this is for use as an extension, so I'm ignoring
complications like guile-readline for now.)
 
This way, if one wanted to distribute compiled versions of Guile along with
an application, one would only need to add the libguile.so and the
go-tar or go-iso file.
 
If you wanted to get really minimal, you could actually put the .go
tar archive or .go iso archive within a binary segment of the libguile.so
itself, so that you'd only need to distribute libguile.so. 
 
> I would borrow code from Bigloo, Chicken, or some other Scheme
> implementation that comes with that, or write it anew in Scheme.

Wouldn't work in this case, since Guile needs to read .go files
before it can interpret scheme.
 
Thanks,
 
Mike 

Re: Minimal Guile

[Ludovic Courtès]

Hi Mike!

Mike Gran <spk121@yahoo.com> skribis:

>> From: Ludovic Courtès <ludo@gnu.org>
>  > Hi Mike!
>> 
>> Mike Gran <spk121@yahoo.com> skribis:
>> 
>>>  It'll be fun to try to minimize it down to just
>>>  the guile executable, libguile-*, and a scheme archive file.  And it
>>>  might help with distribution of prebuilt versions.
>> 
>> Are you saying that, say, Lilypond’s tarball would contain libguile.so
>> and the .go files?  (I guess not but then I don’t get it.  ;-))
>  

[...]

> This way, if one wanted to distribute compiled versions of Guile along with
> an application, one would only need to add the libguile.so and the
> go-tar or go-iso file.

OK, but back to the example above, you wouldn’t want Lilypond’s tarball
to contain these binaries, would you?

Ludo’.

Re: Minimal Guile

[Mike Gran]

> From: Ludovic Courtès <ludo@gnu.org>
 > OK, but back to the example above, you wouldn’t want Lilypond’s tarball
> to contain these binaries, would you?
 
For Lilypond on Win32, where the culture is to download
compiled binaries, I'd want to be able to provide a download location
for a pre-compiled Win7 x86 libguile.dll and go-archive file which Lilypond
could reference or perhaps include in their binary installer.
 
-Mike 

Re: Minimal Guile

[Mark H Weaver]

Mike Gran <spk121@yahoo.com> writes:

> The libguile would be modified so that the .iso or .tar file would
> never be unpacked.  Guile would look inside the .tar or .iso for the 
> compiled .go files.

What is the advantage of including our own little read-only filesystem,
when every OS already provides this functionality?  Is it really
significantly easier to install 3 files than to install 300?

Admittedly, I can see how it might make a psychological difference.
Somehow, people get the feeling that a package is huge and bloated when
it contains a large directory structure, whereas a single file of the
same size (or even larger) seems significantly less obtrusive.

However, I'm not sure that this psychological difference is enough to
justify the reduced flexibility of such an approach.

Is there an advantage that's not merely psychological?

    Thanks,
      Mark

Re: Minimal Guile

[Mike Gran]

> From: Mark H Weaver <mhw@netris.org>

> What is the advantage of including our own little read-only filesystem,
> when every OS already provides this functionality?  Is it really
> significantly easier to install 3 files than to install 300?
> 
> Admittedly, I can see how it might make a psychological difference.
> Somehow, people get the feeling that a package is huge and bloated when
> it contains a large directory structure, whereas a single file of the
> same size (or even larger) seems significantly less obtrusive.
> 
> However, I'm not sure that this psychological difference is enough to
> justify the reduced flexibility of such an approach.
> 
> Is there an advantage that's not merely psychological?

No, there is no advantage beyond the psychological for any system that
uses the standard Unix-like filesystem hierarchy and has a decent
package manager.

There are some theoretical corner cases where it could be useful.
Don't know if these would ever occur in practice.
- Systems that don't use a Unix-like filesystem heirarchy
- Programs that are distributed in a folder whose root location
  can change
- Programs that use Guile as an extension but want to limit its
  library for some reason.

It is all branding, or marketing, I guess.  And fun, of course. I thought
that, as a hack, it would be fun to try.

Regards,

Mike

Re: Minimal Guile

[Mark H Weaver]

Mike Gran <spk121@yahoo.com> writes:

>> From: Mark H Weaver <mhw@netris.org>
>
>> What is the advantage of including our own little read-only filesystem,
>> when every OS already provides this functionality?  Is it really
>> significantly easier to install 3 files than to install 300?
>> 
>> Admittedly, I can see how it might make a psychological difference.
>> Somehow, people get the feeling that a package is huge and bloated when
>> it contains a large directory structure, whereas a single file of the
>> same size (or even larger) seems significantly less obtrusive.
>> 
>> However, I'm not sure that this psychological difference is enough to
>> justify the reduced flexibility of such an approach.
>> 
>> Is there an advantage that's not merely psychological?
>
> No, there is no advantage beyond the psychological for any system that
> uses the standard Unix-like filesystem hierarchy and has a decent
> package manager.

How does the absence of a package manager affect the desirability of 3
files vs 300?  I would think that any archiver such as tar or zip would
be sufficient to make unpacking a directory of 300 files as easy as
installing 3 files.

It seems to me that the problem caused by lack of a decent package
manager has to do with Guile's external dependencies.  This is a
separate issue from 3 files vs 300.

The obvious solution is to provide pre-built Guile packages for popular
platforms that contain all dependencies bundled.  This is the standard
practice on MacOS for example, and probably Windows too I guess.

> There are some theoretical corner cases where it could be useful.
> Don't know if these would ever occur in practice.
> - Systems that don't use a Unix-like filesystem heirarchy

I also don't see how Unix-like filesystem semantics affect this
question.  If the semantics are significantly different, that might
require minor changes to Guile's loading of .go files, but I don't see
how it would require us to combine all the .go files into one file, as
long as the filesystem supports directories.

> - Programs that are distributed in a folder whose root location
>   can change

This is a real problem, I agree.  For example on MacOS, applications are
actually entire directories that are made to look like single files to
the user.  This is Apple's solution to the psychological problem of
programs with too many files :)

In any case, users are accustomed to being able to freely move these
applications to any folder they wish.  Therefore, we should make sure
that applications using Guile can include Guile within their application
directory, and that this directory can be moved without breaking
anything, without any special action on the user's part.

However, it seems to me that this problem is strictly orthogonal to the
question of 3 files vs 300.  The number of files doesn't matter here.
Even if we have to find only one file, we still need to solve the
problem of how to find this file if the user moves it.

> - Programs that use Guile as an extension but want to limit its
>   library for some reason.

I don't understand what you mean here.  By limiting the library, do you
mean removing some of the modules that are currently distributed with
Guile by default?  If so, this also seems like a separate issue.

> It is all branding, or marketing, I guess.  And fun, of course. I
> thought that, as a hack, it would be fun to try.

I agree that marketing is important, and perhaps this might help :)

I would recommend against tar format, because it is a stream format with
no index.  The only way to find an individual file in a tar file is to
read it from the beginning until you reach the file.

If this single file is to be built only during the package build
process, and never modified after that, then it should be super-easy to
invent your own little homebrew format.  If we assume that the library,
executable, and go-bundle files are always distributed together, that
also means that we're free to change the format at will.  I guess it
could be as simple as this:

* Concatenate all of the .go files together, making sure to align each
  .go file on a page boundary (4 kilobytes) so that each one can be
  individually mmapped.

* Generate a C file that includes the entire directory of files within
  an initialized array.  It could be as simple as an array of structs
  with three fields: module name, start offset in the big file, and
  length.  If you sort this array by module name before generating the
  file, then you could use binary search to find the file.

* Once the C file is generated, relink libguile with this file included.
  (The first time libguile is built, it could include a stub version of
  the C file with an empty directory, so that it knows to use the
  traditional module-loading mechanism).

* To allow graceful error messages in case the wrong go-bundle is used
  with the wrong library, there should be some kind of fingerprint put
  in both the go-bundle and the C file index.  Check the fingerprint
  before trying to load things from the go-bundle.

Just an idea.  Another possibility is to use something like TinyCDB
<http://www.corpit.ru/mjt/tinycdb.html>, but be aware that each .go file
should be aligned on a page boundary to allow fast mapping.  This might
require some changes to the format.

   Happy hacking,
       Mark

Re: Minimal Guile

[Mike Gran]

>>  There are some theoretical corner cases where it could be useful.
>>  Don't know if these would ever occur in practice.
>>  - Systems that don't use a Unix-like filesystem heirarchy
> 
> I also don't see how Unix-like filesystem semantics affect this
> question.
 
If there were a go archive, I could finally make a port to
MS-DOS because I could ignore the 8.3 filesystem limitation.  ;-)
 
> If this single file is to be built only during the package build
> process, and never modified after that, then it should be super-easy to
> invent your own little homebrew format.  If we assume that the library,
> executable, and go-bundle files are always distributed together, that
> also means that we're free to change the format at will.  I guess it
> could be as simple as this:
> 
> * Concatenate all of the .go files together, making sure to align each
>   .go file on a page boundary (4 kilobytes) so that each one can be
>   individually mmapped.
> 
> * Generate a C file that includes the entire directory of files within
>   an initialized array.  It could be as simple as an array of structs
>   with three fields: module name, start offset in the big file, and
>   length.  If you sort this array by module name before generating the
>   file, then you could use binary search to find the file.
> 
> * Once the C file is generated, relink libguile with this file included.
>   (The first time libguile is built, it could include a stub version of
>   the C file with an empty directory, so that it knows to use the
>   traditional module-loading mechanism).
> 
> * To allow graceful error messages in case the wrong go-bundle is used
>   with the wrong library, there should be some kind of fingerprint put
>   in both the go-bundle and the C file index.  Check the fingerprint
>   before trying to load things from the go-bundle.

That's good advice.
 
Thanks much,
 
Mike Gran

Re: Minimal Guile

[Andy Wingo]

On Tue 03 Jan 2012 22:44, Mark H Weaver <mhw@netris.org> writes:

> If this single file is to be built only during the package build
> process, and never modified after that, then it should be super-easy to
> invent your own little homebrew format.

Indeed.  It would be nice to use ELF, though.  I'd like to change the
format of .go files to ELF in 2.2.  That would allow lots of things,
like having multiple entry points (as for different modules).  It would
also help to further separate debug information from code (helping
performance), statically allocating data (lowering allocation,
increasing cross-process sharing), and allowing for extensibility,
including sections for natively-compiled code.

But if you (or anyone) wants to work on something simpler, that's cool
too :)  I do have a Scheme ELF parser around somewhere, and intend to
get back to the "linker" sometime soon.

Andy
-- 
http://wingolog.org/

Re: Minimal Guile

[Andreas Rottmann]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 03 Jan 2012 22:44, Mark H Weaver <mhw@netris.org> writes:
>
>> If this single file is to be built only during the package build
>> process, and never modified after that, then it should be super-easy to
>> invent your own little homebrew format.
>
> Indeed.  It would be nice to use ELF, though.  I'd like to change the
> format of .go files to ELF in 2.2.  That would allow lots of things,
> like having multiple entry points (as for different modules).  It would
> also help to further separate debug information from code (helping
> performance), statically allocating data (lowering allocation,
> increasing cross-process sharing), and allowing for extensibility,
> including sections for natively-compiled code.
>
> But if you (or anyone) wants to work on something simpler, that's cool
> too :)  I do have a Scheme ELF parser around somewhere, and intend to
> get back to the "linker" sometime soon.
>
FWIW, I have written a simple ELF generator in R6RS, which can be found
here:

http://rotty.xx.vu/gitweb/?p=scheme/avrth.git;a=blob;f=elf.sls;hb=master

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: Minimal Guile

[Mark H Weaver]

Mike Gran <spk121@yahoo.com> writes:
> If there were a go archive, I could finally make a port to
> MS-DOS because I could ignore the 8.3 filesystem limitation.  ;-)

A much simpler solution would be to modify the logic near the top of
`try-module-autoload' in boot-9.scm, which translates a module name to a
pathname.

     Mark