r6rs libraries, round two

r6rs libraries, round two

[Julian Graham]

Hi Guilers,

I'd like to take another stab at getting R6RS library support in, this
time by extending the capabilities of the module system.  Here's what
I've got in mind to start with:

1. Add an optional `version' field to the module record type

* What's a good format here?  We could mirror the requirements of R6RS
here (i.e., (v1 v2 ...) where vx is a whole number) or be more
flexible.  For example, Apache Maven (I've got Java on the brain from
being at work) lets you specify the version of your project however
you want, but if you follow the convention its docs set out, it can do
things choose the "latest" version or match a version within a range.
We could do likewise.


2. Add support for optional version arguments to `use-modules',
`resolve-module', etc.

* Again, do we want to adhere strictly to R6RS-format version
references here or extend their syntax?

* Should we establish some rules for what you get when you don't
specify a version?  Given what we've decided about loading multiple
versions of a library (i.e., you can't) and the existing behavior of
the module-loading functions (you get an already-loaded module if
one's available), conflicts seem possible:

E.g., if not specifying a version equates to getting the "first
module" in the search path matching the name, then subsequent calls to
those functions that *do* specify a version reference may succeed or
fail based on the result of a prior call.


Regards,
Julian

Re: r6rs libraries, round two

[Neil Jerram]

Julian Graham <joolean@gmail.com> writes:

> Hi Guilers,
>
> I'd like to take another stab at getting R6RS library support in, this
> time by extending the capabilities of the module system.  Here's what
> I've got in mind to start with:
>
> 1. Add an optional `version' field to the module record type

Sounds good.

> * What's a good format here?  We could mirror the requirements of R6RS
> here (i.e., (v1 v2 ...) where vx is a whole number) or be more
> flexible.

Given that your objective is to get R6RS library support in, I'd say
just stick to the R6RS format for now.  It sounds like it will be
fairly easy to extend this in future, if we want to.

>  For example, Apache Maven (I've got Java on the brain from
> being at work) lets you specify the version of your project however
> you want, but if you follow the convention its docs set out, it can do
> things choose the "latest" version or match a version within a range.
> We could do likewise.

I can see the attraction of that, since I was experimenting two days
ago with adding the SHA-1 of the Git HEAD commit to Guile's micro
version - and it didn't work because of some restriction in
libguile/version.c (which I didn't investigate).

However, I still suggest just basic R6RS for now, because I'm sure we
could compatibly add more options later.

> 2. Add support for optional version arguments to `use-modules',
> `resolve-module', etc.
>
> * Again, do we want to adhere strictly to R6RS-format version
> references here or extend their syntax?

As above, I suggest R6RS for now.

> * Should we establish some rules for what you get when you don't
> specify a version?

Yes!  The latest available?

>  Given what we've decided about loading multiple
> versions of a library (i.e., you can't)

I didn't follow why we decided that, but it feels wrong to me.  (It
seems to me that Guile should be able to handle loading ((foo) v1) and
((foo) v2) simultaneously as easily as it could handle loading
((foo-v1)) and ((foo-v2)) simultaneously.) I guess I should look up
the previous thread, please let me know if you have a convenient
reference.

> and the existing behavior of
> the module-loading functions (you get an already-loaded module if
> one's available), conflicts seem possible:
>
> E.g., if not specifying a version equates to getting the "first
> module" in the search path matching the name, then subsequent calls to
> those functions that *do* specify a version reference may succeed or
> fail based on the result of a prior call.

Agree, and agree that this feels wrong.

Regards,
        Neil

Re: r6rs libraries, round two

[Julian Graham]

Hi Neil,

> I didn't follow why we decided that, but it feels wrong to me.  (It
> seems to me that Guile should be able to handle loading ((foo) v1) and
> ((foo) v2) simultaneously as easily as it could handle loading
> ((foo-v1)) and ((foo-v2)) simultaneously.) I guess I should look up
> the previous thread, please let me know if you have a convenient
> reference.

Here's a link [1].  It's possible I misinterpreted Andy's comment and
Ludo's follow-up.


Regards,
Julian

[1] - http://www.mail-archive.com/guile-devel@gnu.org/msg03541.html

Re: r6rs libraries, round two

[Andy Wingo]

Hey Julian,

On Sun 31 May 2009 01:22, Neil Jerram <neil@ossau.uklinux.net> writes:

> Julian Graham <joolean@gmail.com> writes:
>
>> 1. Add an optional `version' field to the module record type
>
> Sounds good.

Agreed.

>> * What's a good format here?  We could mirror the requirements of R6RS
>> here (i.e., (v1 v2 ...) where vx is a whole number) or be more
>> flexible.
>
> Given that your objective is to get R6RS library support in, I'd say
> just stick to the R6RS format for now.  It sounds like it will be
> fairly easy to extend this in future, if we want to.

Also agreed.

>> * Should we establish some rules for what you get when you don't
>> specify a version?
>
> Yes!  The latest available?

I don't know. To me this could be a distro decision, like Debian's
"alternatives" system. It would be nice to minimize the number of `stat'
calls it takes to load a library -- which would fall out nicely if when
asking for a module without specifying a version, like `(foo bar)', we
give foo/bar.scm.

In the presence of multiple versions, installation rules could handle
making the symlink so there is a default version -- typically the
version that was installed most recently.

>>  Given what we've decided about loading multiple
>> versions of a library (i.e., you can't)
>
> I didn't follow why we decided that, but it feels wrong to me.  (It
> seems to me that Guile should be able to handle loading ((foo) v1) and
> ((foo) v2) simultaneously as easily as it could handle loading
> ((foo-v1)) and ((foo-v2)) simultaneously.) I guess I should look up
> the previous thread, please let me know if you have a convenient
> reference.

I agree it would be nice, but as I said in the thread that Julian
referenced, that would take some more thought -- more than the R6RS
editors were willing to give the problem. And for us, I suspect we would
need some changes to our hierarchical namespaces. We probably shouldn't
let this be a sticking point for Guile's R6RS libraries support.

Regards,

Andy
-- 
http://wingolog.org/

Re: r6rs libraries, round two

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

>>> * Should we establish some rules for what you get when you don't
>>> specify a version?
>>
>> Yes!  The latest available?
>
> I don't know.

This is IMO a terrific part of version handling in R6 modules.  What
will it mean for a 2009 program to import `(rnrs base)' when R27RS is
released?

This was discussed and questioned back then:
http://lists.r6rs.org/pipermail/r6rs-discuss/2007-May/002332.html .

I think such issues should lead us to have a `:version' option that does
just what's needed for R6RS, and does not try to do anything smart
(which could seem to be non-deterministic).  We could even go as far as
discouraging its use in the manual.

Thanks,
Ludo'.

Re: r6rs libraries, round two

[Neil Jerram]

ludo@gnu.org (Ludovic Courtès) writes:

> This is IMO a terrific part of version handling in R6 modules.  What
> will it mean for a 2009 program to import `(rnrs base)' when R27RS is
> released?

By `terrific' do you mean good or bad?  Normally terrific means good,
and `terrible' means bad, and I suspect you meant `terrible'.

     Neil

Re: r6rs libraries, round two

[Ludovic Courtès]

Hi Neil,

Neil Jerram <neil@ossau.uklinux.net> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> This is IMO a terrific part of version handling in R6 modules.  What
>> will it mean for a 2009 program to import `(rnrs base)' when R27RS is
>> released?
>
> By `terrific' do you mean good or bad?  Normally terrific means good,
> and `terrible' means bad, and I suspect you meant `terrible'.

Yes, I meant `terrible'.  Sorry for the confusion!

Thanks,
Ludo'.

Re: r6rs libraries, round two

[Julian Graham]

Hi Guilers,

With the 1.9 series launched, I wanted to start thinking about R6RS
libraries again, since it would be awesome to have some semblance of
an implementation ready by October.


> I think such issues should lead us to have a `:version' option that does
> just what's needed for R6RS, and does not try to do anything smart
> (which could seem to be non-deterministic).  We could even go as far as
> discouraging its use in the manual.

Alright -- so, to summarize what (I think) has been decided:

1. Having more than one version of a module loaded at a time in a
single Guile process / VM is not going to be supported.

2. Mixing load calls that specify version with load calls that do not
should produce deterministic behavior independent of the set of
already-loaded modules.  (As described here: [1].)

R6RS says that the mechanism by which implementations match
version-less library references is implementation-dependent, but, in
order to avoid breaking rule number two, I think there's really only
one way to do it:

The expander has to recursively examine all of the imports, starting
with the top-level program, and keep track of the version specifiers
it finds.  In order for a program to be consistent with regard to
version, all of the version specifiers for a particular library name
must be prefixes of the most fully-specified version of that library.
If this is true, then once the most fully-specified version is
resolved, wildcard version specifiers can be resolved to that version.

Unfortunately, depending on the order in which imports are processed,
this mechanism might need to examine the headers of several versions
of a particular library.  So it's the most correct, but it may be
non-optimal.

A lazier but potentially less load-thrashy alternative would be to
have wildcard versions match the latest available version of a library
OR an already-loaded version of that library if present.

What do people think?


Regards,
Julian

[1] - http://www.mail-archive.com/guile-devel@gnu.org/msg03648.html

Re: r6rs libraries, round two

[Neil Jerram]

Julian Graham <joolean@gmail.com> writes:

> Hi Guilers,
>
> With the 1.9 series launched, I wanted to start thinking about R6RS
> libraries again, since it would be awesome to have some semblance of
> an implementation ready by October.

Indeed, yes.

I assume the objective here is to allow a Guile program or module to
use a portable R6RS library; i.e., specifically, allowing
`(use-modules ...)' or `#:use-module (...)' to resolve to an R6RS
library.  Is that correct?

(In addition, I guess we could support `import' directly in contexts
other than R6RS library code, i.e. in a top level program or Guile
module.  But it's not clear to me if that would provide any extra
benefit.)

>> I think such issues should lead us to have a `:version' option that does
>> just what's needed for R6RS, and does not try to do anything smart
>> (which could seem to be non-deterministic).  We could even go as far as
>> discouraging its use in the manual.

(I'm assuming this means a :version option to use-modules or
#:use-module.)

> Alright -- so, to summarize what (I think) has been decided:
>
> 1. Having more than one version of a module loaded at a time in a
> single Guile process / VM is not going to be supported.

Yes, at least as far as R6RS library modules are concerned.  I've
reviewed the relevant discussions on this now and am happy that
there's no desire to support multiple live versions of R6RS libraries.

I'd prefer not to rule this out, though, for any future versioning
that we might add to (define-module ...).  Is that feasible?

> 2. Mixing load calls that specify version with load calls that do not
> should produce deterministic behavior independent of the set of
> already-loaded modules.  (As described here: [1].)

I think that's only possible if we impose some restrictions on how
R6RS libraries can be imported...

> R6RS says that the mechanism by which implementations match
> version-less library references is implementation-dependent, but, in
> order to avoid breaking rule number two, I think there's really only
> one way to do it:
>
> The expander has to recursively examine all of the imports, starting
> with the top-level program, and keep track of the version specifiers
> it finds.

That sounds like it requires us to `read' all of the program code, and
all imported modules and libraries, before deciding which versions to
use.  Possible problems with that would be (i) reader macros that have
non-trivial side effects; (ii) different possible versions of a
library that themselves import different versions of another library,
and so on.

But on the other hand:

- Once we're into R6RS library code, we're OK, because the R6RS layout
  requires all of the imports to be declared upfront; so we don't need
  to read the rest of the library code.

- In a Guile module, we could specify that versioned imports can only
  be done by #:use-module expressions as part of the (define-module
  ...) form, and not support versioned imports by (use-modules ...).
  Then we'd only have to read the (define-module ...) form.

- That just leaves the main program.  For that we could invent a new
  form, like define-module but for the main program, and say that it
  has to cover all versioned imports.

Hmm.. it seems this boils down to saying that we would partially
deprecate `(use-modules ...)'.

>  In order for a program to be consistent with regard to
> version, all of the version specifiers for a particular library name
> must be prefixes of the most fully-specified version of that library.
> If this is true, then once the most fully-specified version is
> resolved, wildcard version specifiers can be resolved to that version.
>
> Unfortunately, depending on the order in which imports are processed,
> this mechanism might need to examine the headers of several versions
> of a particular library.  So it's the most correct, but it may be
> non-optimal.
>
> A lazier but potentially less load-thrashy alternative would be to
> have wildcard versions match the latest available version of a library
> OR an already-loaded version of that library if present.

I can imagine a less clever approach, in which each import is
considered as we come to it, and

- if there's already a live module with that name

  - if it's version is compatible with the new import, it's used

  - if it's version isn't compatible, error

- else load a live module as indicated by the new import (somehow -
  e.g. Andy's symlink approach).

Now obviously this could render some programs unstartable.  But that
could be OK, so long as there was an easy fix for the program author,
such as adding

(use-modules ((the troublesome module) #:version (the right version)))

near the top of the main program.  It would be even more OK if we
could provide a tool to help work out (the right version).

But even with that tool, this less clever approach is a lot less nice
than something that Just Works - which is what your suggestion above
is.  So I'd prefer your suggestion if we can make it work.

Regards,
        Neil

Re: r6rs libraries, round two

[Julian Graham]

Hi Neil,


> I assume the objective here is to allow a Guile program or module to
> use a portable R6RS library; i.e., specifically, allowing
> `(use-modules ...)' or `#:use-module (...)' to resolve to an R6RS
> library.  Is that correct?

Actually, my immediate-term goal was to add versioning info to Guile's
"native" modules in an R6RS-compatible way.  Once that's done, the
level of compatibility you're describing would be feasible, although I
think it'd be up for discussion as to whether it's something we want.
I kind of assumed that in the first iteration there'd be an explicit
bridge into R6RS library territory, e.g., `(rnrs-import ...)' or
something like that.


> Yes, at least as far as R6RS library modules are concerned.  I've
> reviewed the relevant discussions on this now and am happy that
> there's no desire to support multiple live versions of R6RS libraries.
>
> I'd prefer not to rule this out, though, for any future versioning
> that we might add to (define-module ...).  Is that feasible?

Not sure I understand -- the impression I got from earlier messages in
this thread was that we would be implementing R6RS libraries on top of
Guile modules, after first extending the module form to include
version metadata.


> - Once we're into R6RS library code, we're OK, because the R6RS layout
>  requires all of the imports to be declared upfront; so we don't need
>  to read the rest of the library code.

Yes -- from what I can tell, R6RS effectively prohibits "dynamic"
access to the import system.  That is, an import call always results
directly from a previous import, never from the evaluation of an
expression.

(This strikes me as a little weird, since it means that building an
application with a "plugin" architecture would require you to roll
your own import system, but I suppose that's not too far afield from
what languages like C# make you do in that regard.)


> - In a Guile module, we could specify that versioned imports can only
>  be done by #:use-module expressions as part of the (define-module
>  ...) form, and not support versioned imports by (use-modules ...).
>  Then we'd only have to read the (define-module ...) form.

So what would the semantics of the version-less (use-modules ...) be
in the context of versioned modules already loaded via #:use-module
expressions?  To minimize version-clash, I'm guessing we'd defer to
already-loaded modules when possible -- although this wouldn't address
the case in which a module loaded at runtime via (use-modules ...)
introduces a #:use-module dependency that conflicts with an
already-loaded module.


> Hmm.. it seems this boils down to saying that we would partially
> deprecate `(use-modules ...)'.

Or at least point out that its use prevents Guile from being able to
guarantee conflict-less execution.


> I can imagine a less clever approach, in which each import is
> considered as we come to it, and

[snip]

I think this is what I was trying to suggest, stated more clearly.
And unless I'm misunderstanding, I think this makes sense as behavior
for (use-modules ...), with the caveats mentioned above.


Regards,
Julian

Re: r6rs libraries, round two

[Andy Wingo]

Hi Julian,

On Sun 28 Jun 2009 02:20, Julian Graham <joolean@gmail.com> writes:

> With the 1.9 series launched, I wanted to start thinking about R6RS
> libraries again, since it would be awesome to have some semblance of
> an implementation ready by October.

Yes!

>> I think such issues should lead us to have a `:version' option that does
>> just what's needed for R6RS, and does not try to do anything smart
>> (which could seem to be non-deterministic).  We could even go as far as
>> discouraging its use in the manual.
>
> Alright -- so, to summarize what (I think) has been decided:
>
> 1. Having more than one version of a module loaded at a time in a
> single Guile process / VM is not going to be supported.

I agree.

> 2. Mixing load calls that specify version with load calls that do not
> should produce deterministic behavior independent of the set of
> already-loaded modules.  (As described here: [1].)

I disagree. I don't think that you can do (2) without (1).

Your solution of doing whole-program analysis is very much in the spirit
of R6RS, but it is not in the spirit of Lisp, in my opinion at least.

Larry Wall was right. Lisp is like oatmeal: of a uniform, mushy
consistency. A Lisp system is composed of a procedures and data,
permeating each other. You can add (or remove!) procedures and data, and
it's still basically the same system.

Of course we have modules to keep things manageable, but Guile's take is
that modules are mutable entities. You can enter modules and make new
definitions, or even remove old ones. Indeed you may have a long-running
process that you hack for months, mutating the system over time in
response to needs that crop up over time.

Of course, with so much mutation -- definition is mutation, after all,
in a live system -- you just have to have good practices to keep things
under control.

Any program that imports unqualified module names (without versions) is
inherently not "future-portable" anyway, for the reasons that Ludovic
and others discussed on r6rs-discuss[0]. So /we don't even need to worry
about it/. We have to let "good practices" take care of us here.

[0] http://lists.r6rs.org/pipermail/r6rs-discuss/2007-May/002332.html.

The only fully specified program is one that has all of the versions
declared, but even that program may not have deterministic results -- if
an incompatible library is already loaded in the Guile session, the r6rs
program or library will fail to load.

I actually think that doing a whole-program analysis is actively harmful
to the future of Scheme. That's a bit hyperbolic, but it is my opinion.
To me, Lisp is about living, adaptable systems -- not about static
programs. There's no need to let mistakes on the part of the R6RS
editors take us farther down the static path.

Fortunately, we can comply to the letter of the standard, and not the
spirit.

Back to your question though, what did you think about my symlink
solution[1]?

[1] http://article.gmane.org/gmane.lisp.guile.devel/8585

Cheers,

Andy
-- 
http://wingolog.org/

Re: r6rs libraries, round two

[Julian Graham]

Hi Andy,

> Your solution of doing whole-program analysis is very much in the spirit
> of R6RS, but it is not in the spirit of Lisp, in my opinion at least.

Well, to be fair, it's not whole-program analysis -- as Neil pointed
out, we only need to analyze the library and program "headers."  But,
sure, point taken.


> I actually think that doing a whole-program analysis is actively harmful
> to the future of Scheme. That's a bit hyperbolic, but it is my opinion.
> To me, Lisp is about living, adaptable systems -- not about static
> programs. There's no need to let mistakes on the part of the R6RS
> editors take us farther down the static path.
>
> Fortunately, we can comply to the letter of the standard, and not the
> spirit.
>
> Back to your question though, what did you think about my symlink
> solution[1]?

It's a fine way of handling version-less dependencies, but I don't see
how it solves the determinism issue.  Let's say my program (or code
stream or script or whatever you want to call it) uses libraries from
two different authors.  Author A trusts his dependencies to remain
stable and doesn't use version specifiers in his library references;
author B does use them.

If my code imports author A's library before author B's, and there's a
version clash in their dependencies, what's the take-away for me, the
user?  Does this mean my code is not well-formed until I reverse the
order of my imports?  Should I not depend on author A's code at all?
On author B's code?  Do I have to patch my libraries before importing
them?

I'm approaching this project in my capacity as a developer with a fair
number of dependencies (that I want manage as abstractly as I can)
wanting to distribute and test my code on as many platforms as
possible.  From my point of view, the fact that there hasn't been a
unified "module" format for Scheme is pretty frustrating.  To the
extent that R6RS has some traction among Scheme implementations, its
library spec is attractive -- although, I recognize, for political
more than technical reasons.

At the same time, after months of poring over the minutiae of R6RS,
I've come around to the position that it's got some serious defects.

Is there some middle ground possible?  The intentions of the R6RS
editors with regard to libraries and programs being "static" seems
pretty clear.  And for certain use cases, that may be appropriate.
But that doesn't mean that Guile modules need to behave that way.
Would people be amenable to having two different ways of working with
libraries / programs?  I.e., the "mode" that Guile would be in when
you ran an R6RS program would be different than when you were in the
REPL.  Maybe it'd even be a different language in the VM.  And we
could establish some prescribed mechanisms by which non-R6RS Guile
code could use R6RS libraries (with caveats attached), and just not
support the converse.

I don't know -- this problem seems to get thornier the more we discuss
it, but I still think it's worth solving.  Sorry if I've missed the
point on any of your comments.


Regards,
Julian

Re: r6rs libraries, round two

[Ludovic Courtès]

Hello!

Julian Graham <joolean@gmail.com> writes:

> It's a fine way of handling version-less dependencies, but I don't see
> how it solves the determinism issue.  Let's say my program (or code
> stream or script or whatever you want to call it) uses libraries from
> two different authors.  Author A trusts his dependencies to remain
> stable and doesn't use version specifiers in his library references;
> author B does use them.

IMO this example is not realistic.  Libraries with a well defined API
stability policy provide ad hoc versioning mechanisms at the source
level, e.g., "(use-modules (gnome-2))".  Likewise, applications
distributed as a source package and that aim to build on a wide range of
platforms are likely to explicitly check for the version of their
dependencies.

By saying Guile should "comply to the letter of the standard", I think
Andy meant to say it should do the minimum required to be
"standard-compliant".

Thanks,
Ludo'.

Re: r6rs libraries, round two

[Julian Graham]

Hi Andy,


> Back to your question though, what did you think about my symlink
> solution[1]?
>
> [1] http://article.gmane.org/gmane.lisp.guile.devel/8585

It's fine, although any length suffix of a version specifier can be
omitted in the case of a match -- that is, you can do without the
version altogether, or you can just partially specify a version; in
both cases, the system needs to find a matching version for you.
Would the symlinking strategy you describe happen at every node in the
directory tree of versions for a particular library?

E.g., let's say you've got version 1.2.3 of R6RS library `(foo bar)'
installed.  Does the installation directory look like:

foo/1/2/3/bar.scm
foo/1/2/bar.default -> foo/1/2/3/bar.scm
foo/1/bar.default -> foo/1/2/3/bar.scm
foo/bar.default -> foo/1/2/3/bar.scm


Regards,
Julian