Guile release planning

Guile release planning

[Neil Jerram]

Andy recently wondered - in connection with his VM implementation and
docs - about when a 1.9.x or 1.10.x release might happen, and that
reminded me about the following thoughts that I've been trying to
crystallize for a while.  How should we organize future Guile
releases?  I'm interested to hear both developer and user views on
this, hence the cross-posting.

In my view, the most important thing for Guile's near-to-medium-term
future is focus.  By that I mean having developers working on, and
users using, as far as possible, a similar level of code.  In the
past, we did big jumps - from 1.4.x to 1.6.x, and from 1.6.x to 1.8.x
- which I think left users unable easily to upgrade, or perhaps just
unsure of whether to upgrade.  From the developer point of view, they
increased the support burden (because of some users staying with the
old series).  Also the big jump model can be frustrating for
developers, because it tends to mean that there is a long time between
when a shiny new feature is ready, and when it gets released and so
into the hands of most users.

Those past jumps were probably justified, but I'm not sure they are in
future.  I wonder if a better model would be to have a single ongoing
series of releases, and to feed new features one by one into that.  In
principle the jump from one release to the next would always be small,
and so should allow everyone to upgrade easily.  I think this will
allow the community to stay closer together (in code terms), and will
allow developers to get interesting new features out into the wild
more quickly.

I also think it will help us manage API incompatibilities better.  I
think our default position from now on should be to maintain
source-level (API) compatibility, but it is inevitable that there will
be exceptions to this.  When we did a big jump in the past, we did
document all the API changes, but perhaps not as well as could have
been done.  If, in future, each individual release contains less API
change, I think we can do a better job of fully describing that, and
how to cope with it.

So, what do you think?  There have been discussions of release
strategy in the past, which I've seen as 50/50 between the split
stable and development model (which we have now) and the steady new
feature model (described above), but I don't recall them considering
the overall community focus angle before.  In my view, when we add in
that angle, the steady new feature model is better.

Regards,
        Neil

Re: Guile release planning

[Mike Gran]

If the base Guile C API remains stable, it doesn't matter to me how the releases occur, because they won't break my libraries or projects. 

If the Guile C API needs to change, some sort of notification and beta pre-release would be preferred, so that I can test my projects before the new Guile gets yum'ed out to my group.

-Mike Gran

Re: Guile release planning

[Neil Jerram]

Hi Mike, thanks for your response.

2008/11/11 Mike Gran <spk121@yahoo.com>:
> If the base Guile C API remains stable, it doesn't matter to me how the releases occur, because they won't break my libraries or projects.

OK.

> If the Guile C API needs to change, some sort of notification and beta pre-release would be preferred, so that I can test my projects before the new Guile gets yum'ed out to my group.

How exactly would a "beta pre-release" help?  It seems you have in
mind people who are building your project from source, using a
distro-updated libguile.  Even with notification/pre-release, and with
you having updated your code accordingly, one of your users might not
have downloaded your updated code.

I guess I can see, though, that it's nice if you have a bit of notice,
and hence time to prepare an update.  And then I can also see that to
do that you will want real code to work with, not just an English
description of the API change.

I would propose, then, that we clearly flag (on the mailing list) an
API change at the time when the relevant commit is made to the
repository, and make sure that some minimum period of time elapses
before the subsequent release.  I would hope that you could then work
on the basis of the commit, without needing a formal pre-release.
(Any kind of release takes a bit of time, and pre-releases might
confuse the overall release picture.)

Would that work?

     Neil

Re: Guile release planning

[Mike Gran]

> From: Neil Jerram <neiljerram@googlemail.com>
> Hi Mike, thanks for your response.
> 
> I would propose, then, that we clearly flag (on the mailing list) an
> API change at the time when the relevant commit is made to the
> repository, and make sure that some minimum period of time elapses
> before the subsequent release.  I would hope that you could then work
> on the basis of the commit, without needing a formal pre-release.
> (Any kind of release takes a bit of time, and pre-releases might
> confuse the overall release picture.)
> 
> Would that work?

Works for me. 

Re: Guile release planning

[Jon Wilson]

Hi Neil,
Thanks for working on this.  I'd favor the steady new features model, 
but I would recommend periodically marking a release as one that will be 
supported past the next release, similar to Ubuntu's LTS releases.
Regards,
Jon

Neil Jerram wrote:
> Andy recently wondered - in connection with his VM implementation and
> docs - about when a 1.9.x or 1.10.x release might happen, and that
> reminded me about the following thoughts that I've been trying to
> crystallize for a while.  How should we organize future Guile
> releases?  I'm interested to hear both developer and user views on
> this, hence the cross-posting.
> 
> In my view, the most important thing for Guile's near-to-medium-term
> future is focus.  By that I mean having developers working on, and
> users using, as far as possible, a similar level of code.  In the
> past, we did big jumps - from 1.4.x to 1.6.x, and from 1.6.x to 1.8.x
> - which I think left users unable easily to upgrade, or perhaps just
> unsure of whether to upgrade.  From the developer point of view, they
> increased the support burden (because of some users staying with the
> old series).  Also the big jump model can be frustrating for
> developers, because it tends to mean that there is a long time between
> when a shiny new feature is ready, and when it gets released and so
> into the hands of most users.
> 
> Those past jumps were probably justified, but I'm not sure they are in
> future.  I wonder if a better model would be to have a single ongoing
> series of releases, and to feed new features one by one into that.  In
> principle the jump from one release to the next would always be small,
> and so should allow everyone to upgrade easily.  I think this will
> allow the community to stay closer together (in code terms), and will
> allow developers to get interesting new features out into the wild
> more quickly.
> 
> I also think it will help us manage API incompatibilities better.  I
> think our default position from now on should be to maintain
> source-level (API) compatibility, but it is inevitable that there will
> be exceptions to this.  When we did a big jump in the past, we did
> document all the API changes, but perhaps not as well as could have
> been done.  If, in future, each individual release contains less API
> change, I think we can do a better job of fully describing that, and
> how to cope with it.
> 
> So, what do you think?  There have been discussions of release
> strategy in the past, which I've seen as 50/50 between the split
> stable and development model (which we have now) and the steady new
> feature model (described above), but I don't recall them considering
> the overall community focus angle before.  In my view, when we add in
> that angle, the steady new feature model is better.
> 
> Regards,
>         Neil
> 
> 

Re: Guile release planning

[Neil Jerram]

2008/11/11 Jon Wilson <jsw@wilsonjc.us>:
> Hi Neil,
> Thanks for working on this.  I'd favor the steady new features model, but I
> would recommend periodically marking a release as one that will be supported
> past the next release, similar to Ubuntu's LTS releases.

Why?  It seems to me that that would encourage exactly the kind of
community fragmentation that I'm aiming to reduce.

     Neil

Re: Guile release planning

[Jon Wilson]

Hi Neil,

Neil Jerram wrote:
> 2008/11/11 Jon Wilson <jsw@wilsonjc.us>:
>> Hi Neil,
>> Thanks for working on this.  I'd favor the steady new features model, but I
>> would recommend periodically marking a release as one that will be supported
>> past the next release, similar to Ubuntu's LTS releases.
> 
> Why?  It seems to me that that would encourage exactly the kind of
> community fragmentation that I'm aiming to reduce.

Well, now that you ask, and what with me in the middle of some evil E&M 
homework (ref: J.D. Jackson), I really can't remember.  It is possible 
that it simply wasn't well thought out: "It seems to work well for 
Ubuntu, so it should work for everything!", but I'm not sure.  :-/

Regards,
Jon

Re: Guile release planning

[Linas Vepstas]

2008/11/10 Neil Jerram <neiljerram@googlemail.com>:
>
> I also think it will help us manage API incompatibilities better.  I
> think our default position from now on should be to maintain
> source-level (API) compatibility, but it is inevitable that there will
> be exceptions to this.

Any ideas for binary compatibility for the "micro" revisions?
I recently discovered that a library compiled against 1.8.3
would core dump when used with an application compiled
against 1.8.5.  Operationally, not a big deal, really; I just
recompiled the lib, but emotionally, it did give me that
sinking feeling for a while, of maybe having  yet another
hard-to-find bug, or a  system I cannot fully trust. :-(

>  the steady new feature model is better.

The linux kernel got rid of the stable/unstable branch idea,
and it's worked really really well. (the reasons why are
widely documented) I'm for it.

--linas

Re: Guile release planning

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]


  Any ideas for binary compatibility for the "micro" revisions?
  I recently discovered that a library compiled against 1.8.3
  would core dump when used with an application compiled
  against 1.8.5.  Operationally, not a big deal, really; I just
  recompiled the lib, but emotionally, it did give me that
  sinking feeling for a while, of maybe having  yet another
  hard-to-find bug, or a  system I cannot fully trust. :-(

The first question is if there was a shlib minor version bump.  But, if
it were the other way around it would be bad.  It is not expected in
general that you can use older libraries than what you built against.
But, building against 1.8.3 and then updating to 1.8.5 should be ok.

[-- Attachment #2: Type: application/pgp-signature, Size: 193 bytes --]

Re: Guile release planning

[Andy Wingo]

Hi Linas,

On Tue 11 Nov 2008 04:44, "Linas Vepstas" <linasvepstas@gmail.com> writes:

> 2008/11/10 Neil Jerram <neiljerram@googlemail.com>:
>>
>> I also think it will help us manage API incompatibilities better.  I
>> think our default position from now on should be to maintain
>> source-level (API) compatibility, but it is inevitable that there will
>> be exceptions to this.
>
> Any ideas for binary compatibility for the "micro" revisions?

I think it needs to be guaranteed.

> I recently discovered that a library compiled against 1.8.3
> would core dump when used with an application compiled
> against 1.8.5.

Ooh, bummer. The 1.8 series is binary-compatible (i.e. 1.8.x is
compatible with 1.8.y if x >= y), *but* only if compiled in the same
way.

An example of compiling in different ways is if you build against a
guile with --disable-threads, but then rebuild guile with
--enable-threads, or vice versa. Probably what happened to you?

> The linux kernel got rid of the stable/unstable branch idea,
> and it's worked really really well. (the reasons why are
> widely documented) I'm for it.

The linux kernel doesn't guarantee ABI /or/ API stability -- it's a
different kettle of fish.

Cheers,

Andy
-- 
http://wingolog.org/

Re: Guile release planning

[Linas Vepstas]

2008/11/11 Andy Wingo <wingo@pobox.com>:

>> Any ideas for binary compatibility for the "micro" revisions?
>
> I think it needs to be guaranteed.
>
>> I recently discovered that a library compiled against 1.8.3
>> would core dump when used with an application compiled
>> against 1.8.5.

Ludovic asked:
> Do you remember what caused it?

I didn't bother to investigate; I recomipled and the
problem went away.

> --enable-threads, or vice versa. Probably what happened to you?

Don't think so. The 1.8.3 was from Ubuntu Hardy. I assume
it had threads turned on; but I wasn't yet using threads at that
time, so I don't know. The 1.8.5 was self-built. I did not
specify --enable-threads, this seemed to be on automatically.

--linas

Re: Guile release planning

[Andy Wingo]

Hi,

On Tue 11 Nov 2008 22:05, "Linas Vepstas" <linasvepstas@gmail.com> writes:

> 2008/11/11 Andy Wingo <wingo@pobox.com>:
>> --enable-threads, or vice versa. Probably what happened to you?
>
> Don't think so. The 1.8.3 was from Ubuntu Hardy. I assume
> it had threads turned on

Nope, Debian builds --without-threads.

> The 1.8.5 was self-built. I did not specify --enable-threads, this
> seemed to be on automatically.

Voilà.

Andy
-- 
http://wingolog.org/

Re: Guile release planning

[Ludovic Courtès]

Hi Linas,

"Linas Vepstas" <linasvepstas@gmail.com> writes:

> Any ideas for binary compatibility for the "micro" revisions?
> I recently discovered that a library compiled against 1.8.3
> would core dump when used with an application compiled
> against 1.8.5.

Do you remember what caused it?

I don't remember of any ABI change in the 1.8.x series (apart from
function additions).  The with-threads and without-threads Guiles are
not ABI-compatible, though.

Thanks,
Ludo'.

Re: Guile release planning

[Neil Jerram]

2008/11/11 Linas Vepstas <linasvepstas@gmail.com>:
>
> Any ideas for binary compatibility for the "micro" revisions?

At our "upstream" level (i.e. not trying to solve all of the
distribution-level issues), I think the theory is that this is covered
by library interface numbering.  In other words, if a change in guile
causes the new libguile not to be compatible with the previous one,
libguile's revision number should be incremented.

> I recently discovered that a library compiled against 1.8.3
> would core dump when used with an application compiled
> against 1.8.5.

And I assume the actually loaded libguile was version 1.8.5?  So there
should have been something in the library saying that it needed 1.8.3,
which would have caused the link to fail.  Does the libtool scheme
cover this; I'm afraid I just don't know.

> The linux kernel got rid of the stable/unstable branch idea,
> and it's worked really really well. (the reasons why are
> widely documented) I'm for it.

Yes, I guess my suggestion is quite similar to what the kernel did.

Thanks,
   Neil

Re: Guile release planning

[Ludovic Courtès]

Hello,

"Neil Jerram" <neiljerram@googlemail.com> writes:

> And I assume the actually loaded libguile was version 1.8.5?  So there
> should have been something in the library saying that it needed 1.8.3,
> which would have caused the link to fail.  Does the libtool scheme
> cover this; I'm afraid I just don't know.

Yes it does (at the level of "supported interface numbers", not at the
level of version numbers, though), provided the person who prepares the
release correctly updates the Libtool version info---which has always
been the case, of course.  :-)

Ludo'.

Re: Guile release planning

[Andy Wingo]

Hi,

On Mon 17 Nov 2008 00:33, ludo@gnu.org (Ludovic Courtès) writes:

>> And I assume the actually loaded libguile was version 1.8.5?  So there
>> should have been something in the library saying that it needed 1.8.3,
>> which would have caused the link to fail.  Does the libtool scheme
>> cover this; I'm afraid I just don't know.
>
> Yes it does (at the level of "supported interface numbers", not at the
> level of version numbers, though)

Beating a dead horse, 1.8.5 should be compatible with 1.8.3, via
whatever mechanism. (The actual problem in this case was elsewhere.)

I am skeptical of pushing significant changes into stable branches. I
really don't want to be in the situation in which 1.8.4 works for
someone, but 1.8.7 does not. You might be right, Neil, about that path,
but it does not give me warm fuzzy feelings.

Andy
-- 
http://wingolog.org/

Re: Guile release planning

[Ludovic Courtès]

Hi,

Andy Wingo <wingo@pobox.com> writes:

> Beating a dead horse, 1.8.5 should be compatible with 1.8.3, via
> whatever mechanism. (The actual problem in this case was elsewhere.)

1.8.5 *is* compatible with 1.8.x.

> I am skeptical of pushing significant changes into stable branches. I
> really don't want to be in the situation in which 1.8.4 works for
> someone, but 1.8.7 does not. You might be right, Neil, about that path,
> but it does not give me warm fuzzy feelings.

Same for me, but we can surely find a reasonable trade-off.

Thanks,
Ludo'.

Re: Guile release planning

[Neil Jerram]

2008/11/18 Ludovic Courtès <ludo@gnu.org>:
> Hi,
>
> Andy Wingo <wingo@pobox.com> writes:
>
>> Beating a dead horse, 1.8.5 should be compatible with 1.8.3, via
>> whatever mechanism. (The actual problem in this case was elsewhere.)
>
> 1.8.5 *is* compatible with 1.8.x.
>
>> I am skeptical of pushing significant changes into stable branches. I
>> really don't want to be in the situation in which 1.8.4 works for
>> someone, but 1.8.7 does not. You might be right, Neil, about that path,
>> but it does not give me warm fuzzy feelings.
>
> Same for me, but we can surely find a reasonable trade-off.

Many thanks everyone for your replies to this thread, and sorry for my
delay in following up.

OK, it's clear the consensus on 1.8.x is against my suggestion, so
I'll accept that.  And I can understand the reasons too.  I think
perhaps it comes down to Ludovic's point about the version number
being a hint - i.e. people already have expectations about what should
be in the change from 1.8.x to 1.8.x+1, and mostly those expectations
seem to be API and ABI compatibility, so it makes sense to comply with
that.

One of my background concerns, when suggesting that we try to get more
into 1.8.x, was that if we have lots of release branches (e.g. 1.6.x,
1.8.x and 1.10.x) in use at the same time, we have to do more work to
apply fixes to all branches.  But in fact that is a lot easier now
that we use Git, so probably isn't a big worry.

For new features, then, the question becomes: what is our general
plan, from here on, for going from 1.y.x to 1.y+2.0 ?  As Greg has
said, we need to get new stuff out quicker than we have done in the
past.  Right now we have an growing pile of new stuff in Git, should
_all_ of that go into 1.10?

I guess the answer is that 1.10.0 - and more generally any new 1.y+2.0
- should include everything that we have which

- we believe is ready, i.e. in a sufficiently final form that its
API/ABI is unlikely to be broken in future

- is working.

For 1.10.0, then, we just need to check that there isn't anything in
master that isn't ready/working; if there is, it should be moved into
a branch.  And from here on we should adopt the rule that new features
are always developed on branches, and only merged into master when
ready and working.  Then we should be able to release master as a new
1.y+2.0 whenever we see that enough new features have accumulated.

How does that sound?

        Neil

Re: Guile release planning

[Ludovic Courtès]

Hi Neil!

"Neil Jerram" <neiljerram@googlemail.com> writes:

> OK, it's clear the consensus on 1.8.x is against my suggestion, so
> I'll accept that.  And I can understand the reasons too.  I think
> perhaps it comes down to Ludovic's point about the version number
> being a hint - i.e. people already have expectations about what should
> be in the change from 1.8.x to 1.8.x+1, and mostly those expectations
> seem to be API and ABI compatibility, so it makes sense to comply with
> that.

Yep, it's just a matter of labeling, really.

> For 1.10.0, then, we just need to check that there isn't anything in
> master that isn't ready/working; if there is, it should be moved into
> a branch.

In a similar vein, some of the things developed in the 1.7 series, like
futures, were never stabilized and are simply commented out for now.
Thread support is also somewhat sloppy, as shown my recent reports
(e.g., parallel memoization).  We need to think about these as well.

> And from here on we should adopt the rule that new features
> are always developed on branches, and only merged into master when
> ready and working.  Then we should be able to release master as a new
> 1.y+2.0 whenever we see that enough new features have accumulated.
>
> How does that sound?

Sounds good to me!

Thanks,
Ludo'.

Re: Guile release planning

[Sebastian Tennant]

Quoth "Neil Jerram" <neiljerram@googlemail.com>:
> In my view, when we add in [the community focus] angle, the steady new
> feature model is better.

As a non-developer, but committed user, I couldn't agree more.

Sebastian

Re: Guile release planning

[Ludovic Courtès]

Hello!

"Neil Jerram" <neiljerram@googlemail.com> writes:

> In my view, the most important thing for Guile's near-to-medium-term
> future is focus.  By that I mean having developers working on, and
> users using, as far as possible, a similar level of code.  In the
> past, we did big jumps - from 1.4.x to 1.6.x, and from 1.6.x to 1.8.x
> - which I think left users unable easily to upgrade, or perhaps just
> unsure of whether to upgrade.  From the developer point of view, they
> increased the support burden (because of some users staying with the
> old series).  Also the big jump model can be frustrating for
> developers, because it tends to mean that there is a long time between
> when a shiny new feature is ready, and when it gets released and so
> into the hands of most users.

I fully agree with this, but...

Your note doesn't take binary compatibility into account, and I think
it's an important thing, too.  I think the ideal is to maintain binary
compatibility within a major series, as we've done (or tried to do) in
the 1.8.x series.  This is very convenient for binary distributions like
Debian, and for users in general.  Thus, introducing ABI
incompatibilities should imply increasing the first or second digit of
the version number, IMO.

Maintaining ABI compatibility doesn't mean we can't add new features,
though.  In the course of the 1.8.x series, several Scheme modules were
added.  I think enhancements like the lazy symbol binding in modules
could have been in theory added in 1.8.x since it breaks neither the API
nor the ABI.  Things like `libguile-i18n' could have been added too, but
the issue when adding C code is portability: it may be that this module
would have caused build issues on some platforms.  Now, with more
frequent releases, it would be reasonable to hope that such regressions
would quickly be fixed.

Another thing is actual big jumps.  I think the addition of the VM is a
big jump.  A GC change, or a rewrite of the string/char code with
Unicode support would be a big jump, too.  Such big jumps surely need a
new major release.

BTW, we need to consider releasing 1.8.6 one of these days!  ;-)

Thanks!

Ludo'.

Re: Guile release planning

[Neil Jerram]

2008/11/11 Ludovic Courtès <ludo@gnu.org>:
>
> Your note doesn't take binary compatibility into account, and I think
> it's an important thing, too.  I think the ideal is to maintain binary
> compatibility within a major series, as we've done (or tried to do) in
> the 1.8.x series.

(And Andy wrote "I think it needs to be guaranteed.")

But this isn't obvious to me.  _If_ the libtool versioning system
works in practice, in the senses of

- permitting linking when it ought to be permitted

- failing linking when it ought to fail

- providing a clear error message in the failure cases, so that the
user knows what to do next

then it seems to me that a reasonable division of labour is for us
(upstream) to take care of API compatibility, and ensuring that the
libtool numbers are a correct description of ABI state, and for
distributions/users to take care of any consequences of mismatched
libtool numbers.

I think the latter "consequences" are just having to recompile
something against the new libguile version.  For a user who has
already decided to use the source code version of an application, that
should be trivial; for distributions, that's just what they do all the
time, isn't it?

Are there other consequences that I'm missing?

Sure, we could also take on ABI compatibility ourselves, but I think
that overconstrains our development, and/or makes it harder.

> This is very convenient for binary distributions like
> Debian, and for users in general.

Are you sure about that?  I would expect Debian already to have
completely automatic ways of coping with this kind of thing (i.e.
libtool numbers changing).  And I've described what I think the impact
on users is above; seems quite small to me.

> I think enhancements like the lazy symbol binding in modules
> could have been in theory added in 1.8.x since it breaks neither the API
> nor the ABI.

Agreed; and I think they still can be added in 1.8.x.

> Things like `libguile-i18n' could have been added too, but
> the issue when adding C code is portability: it may be that this module
> would have caused build issues on some platforms.  Now, with more
> frequent releases, it would be reasonable to hope that such regressions
> would quickly be fixed.

Agreed, I think we can handle this.

> Another thing is actual big jumps.  I think the addition of the VM is a
> big jump.

Yes, but it's an addition.  As far as I understand, it's completely
back-compatible, so I would be perfectly happy to feed this in to
1.8.x at some point.

Of course, we should take care that we are 99% happy with the new APIs
before releasing them, as it wouldn't be good to make lots of changes
later.  But that's no different from the first public release of
anything - in my view, it should work much better for us to come to
this determination feature-by-feature, than to say arbitrarily at some
point "everything in master" is now ready.

>  A GC change, or a rewrite of the string/char code with
> Unicode support would be a big jump, too.  Such big jumps surely need a
> new major release.

Not necessarily, in my view.  We have an extensive test suite, and I
think we can have some confidence in that.  After sufficient testing,
I would see no problem with your proposed BDW-GC changes going into a
1.8.x release.

Same for Unicode support - if the API stayed compatible.  If the API
could not be compatible, then I would agree that we might need a new
major release.

> BTW, we need to consider releasing 1.8.6 one of these days!  ;-)

Yes.  Do we have any particular more things to get into this?  (I
don't think I have anything.)

     Neil

Re: Guile release planning

[Linas Vepstas]

2008/11/15 Neil Jerram <neiljerram@googlemail.com>:
> 2008/11/11 Ludovic Courtès <ludo@gnu.org>:

>> BTW, we need to consider releasing 1.8.6 one of these days!  ;-)
>
> Yes.  Do we have any particular more things to get into this?  (I
> don't think I have anything.)

I'm seeing frequent and wide-spread deadlocking in the
thread-enabled version of guile. I'm debugging this, but
do not yet have a patch.  I sure would like to see this
fixed in 1.8.6, and am laboring under the optimistic
assumption that I can find a patch in a few days, and
that there won't be yet another problem that shows up
after that.

--linas

Re: Guile release planning

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 6555 bytes --]


"Neil Jerram" <neiljerram@googlemail.com> writes:

>> 2008/11/11 Ludovic Courtès <ludo@gnu.org>:
> But this isn't obvious to me.  _If_ the libtool versioning system
> works in practice, in the senses of
>
> - permitting linking when it ought to be permitted
>
> - failing linking when it ought to fail
>
> - providing a clear error message in the failure cases, so that the
> user knows what to do next

Except for 'clear error message', I think shared library linking (in ELF
at least) satisfies that.

> then it seems to me that a reasonable division of labour is for us
> (upstream) to take care of API compatibility, and ensuring that the
> libtool numbers are a correct description of ABI state, and for
> distributions/users to take care of any consequences of mismatched
> libtool numbers.

That is the standard model, and I think it's fine, modulo the frequency
of ABI breaks (and thus shlib major version increases).

ABI is not just shared library versions - it's all the scheme-level
interfaces that a guile-using program might call.

> I think the latter "consequences" are just having to recompile
> something against the new libguile version.  For a user who has
> already decided to use the source code version of an application, that
> should be trivial; for distributions, that's just what they do all the
> time, isn't it?

Yes, except that it's a pain if it happens often, and it more or less
constrains all the guile-using components in a system that aren't leaves
to use the same guile version.

> Are there other consequences that I'm missing?

Frequent API and ABI breaks are in my view a sign of an immature
development process, and a warning not to rely on something.  guile aims
to be a scripting language to be added to various programs, so we should
hav as a major goal being stable enough to be relied on.

> Sure, we could also take on ABI compatibility ourselves, but I think
> that overconstrains our development, and/or makes it harder.

I think that we should aim to have infrequent ABI breaks.  Every year or
so is probably ok.

>> I think enhancements like the lazy symbol binding in modules
>> could have been in theory added in 1.8.x since it breaks neither the API
>> nor the ABI.
>
> Agreed; and I think they still can be added in 1.8.x.

Sure - this is totally fine.  Programs that worked against 1.8.x, both
compile time and run time, will still work against 1.8.x+1.  But, the
reason we want to add to 1.8 is that 1.10 wasn't release 18 months ago,
about which I'll rant more later.

>> A GC change, or a rewrite of the string/char code with
>> Unicode support would be a big jump, too.  Such big jumps surely need a
>> new major release.
>
> Not necessarily, in my view.  We have an extensive test suite, and I
> think we can have some confidence in that.  After sufficient testing,
> I would see no problem with your proposed BDW-GC changes going into a
> 1.8.x release.
>
> Same for Unicode support - if the API stayed compatible.  If the API
> could not be compatible, then I would agree that we might need a new
> major release.

The biggest issue is API breaks without adequate warning.  To remove
something from the API, it should be marked deprecated in a stable
release, and then have probably 2 years pass during which all depending
programs to update and release.  The key is that one has to be able to
say in good conscience that those depending programs are lame or
unmaintained if they didn't update.

If API breaks other than via the above rules happen too often, then
guile really should support (natively) the installation of multiple
vesrions at once.  pkgsrc has "guile" which is 1.8.x and "guile16", and
guile16 installs to /usr/pkg/guile/1.6 as prefix instead of /usr/pkg.
guile simply doesn't have the mindshare to support this kind of
complexity.

An example of API problems is Linux itself.  I am working on a project
using the kernel and going from 2.6.20 to 2.6.24 is a huge change that
seems to have significant API breaks (for kernel modules).  If these
changes are as bad as they seem, this just isn't adult software
engineering and I would have to recommend against relying on such code.


I think guile's real problem has been that while people hack on head,
this hacking doesn't seem to have the clear goal of producing a stable
release.  Of course all the hard-core guile hackers run guile from
(choose-VCS-weekly :-), but programs that depend on guile should not --
they need to depend on versions which are packaged.  So my general rule
for software is that as soon as people say "you should run HEAD instead
of RELEASE, it's better", a release from head is overdue.

So I don't think there's anything wrong with the
hack-on-head-cut-stable-branches-periodically approach, except:

  1) try really hard to avoid API breaks, and take the ugliness of new
  function names with the new semantics/signatures as a lesser pain

  2) release a new stable version from head every year or so.

For 2, I suggest calling feature freeze and asking for testing of head,
kicking out 1.9.x releases and encouraging fast-moving distributions to
package them as a replacement for 1.8.x so packaging system maintainer
types can test, and then branching for 1.10.0 and unfreezing head.  Even
if there were 3 months a year of feature freeze and 9 months of open
hacking, I think that would be good.  There's a negative feedback cycle,
where people don't want to freeze for release because they want to get
in one more thing before the stable code is frozen for the next 4 years,
and as the stable frequency gets lower the urge to not freeze gets
higher, leading to longer intervals.  NetBSD has been struggling with
this too.

Here's the history

1.4	2000-06
1.6     2002-09
1.8     2006-02
1.10    >> 2008-11 :-(

and these intervals are all significantly too long.  Probably a freeze
is in order 9 months after the last branch cut, to head for 1 year.


Speaking from my experience being the maintainer in pkgsrc of guile, new
versions that don't have API or ABI breaks are nearly trivial to update,
and cause no issues.  ABI breaks but not API breaks are not as bad, but
require touching all the depending packages to ensure cross-package
binary compatibility.  API breaks without all the depending packages
getting fixed leads to choosing between dropping depending packages or
having multiple versions of guile in pkgsrc, both of which are ugly.


[-- Attachment #2: Type: application/pgp-signature, Size: 193 bytes --]

Re: Guile release planning

[Ludovic Courtès]

Hello!

"Neil Jerram" <neiljerram@googlemail.com> writes:

> But this isn't obvious to me.  _If_ the libtool versioning system
> works in practice, in the senses of
>
> - permitting linking when it ought to be permitted
>
> - failing linking when it ought to fail

It partly fails for the second point.  Examples: changing the
number/type of arguments of a function, changing the layout of a public
structure, changing the implementation of public macros, etc.  It really
requires special care to check whether a change is breaking ABI
compatibility.

> then it seems to me that a reasonable division of labour is for us
> (upstream) to take care of API compatibility, and ensuring that the
> libtool numbers are a correct description of ABI state, and for
> distributions/users to take care of any consequences of mismatched
> libtool numbers.

I consider ABI compatibility as important, too, because people don't
want to recompile everything every time a minor release is out.

Like Andy said, we should guarantee it within a given series, e.g.,
1.8.x.  After all, the version number is just a label giving users a
hint about the type of changes introduced in a version and level of
compatibility provided.  It's common for free software libraries to
remain ABI-compatible within a major series (first and second digits
unchanged), to change to the second digit when ABI breaks, and possibly
the first digit when API breaks *or* important changes are introduced.

>> This is very convenient for binary distributions like
>> Debian, and for users in general.
>
> Are you sure about that?

Yes.  Even for the few Guile C libraries I develop myself, it's boring
to recompile them when switching from 1.8 to 1.9 or one of the other
branches.  For a distribution, it means burning a lot of CPU power.

>> I think enhancements like the lazy symbol binding in modules
>> could have been in theory added in 1.8.x since it breaks neither the API
>> nor the ABI.
>
> Agreed; and I think they still can be added in 1.8.x.

Yes, but one con could be that it introduces changes that could be
visible to users that access Guile's undocumented module-fiddling API.

> Yes, but it's an addition.  As far as I understand, it's completely
> back-compatible, so I would be perfectly happy to feed this in to
> 1.8.x at some point.

From a marketing viewpoint, the VM deserves a new version number.  :-)

In practice, it also requires users to adapt their build system to take
advantage of it.

> Of course, we should take care that we are 99% happy with the new APIs
> before releasing them, as it wouldn't be good to make lots of changes
> later.  But that's no different from the first public release of
> anything - in my view, it should work much better for us to come to
> this determination feature-by-feature, than to say arbitrarily at some
> point "everything in master" is now ready.

I agree that a feature-by-feature approach would be more reasonable.

My impression, gathered since the time when Kevin was still with us, was
that the goal was zero-breakage within a stable series, such that
1.8.x+1 would always be more portable and reliable that 1.8.x.  This
approach rules out things like the lazy duplicate symbol resolution, and
`libguile-i18n'.  Surely we could find an "in-between".

> Not necessarily, in my view.  We have an extensive test suite, and I
> think we can have some confidence in that.  After sufficient testing,
> I would see no problem with your proposed BDW-GC changes going into a
> 1.8.x release.

The BDW-GC change is ABI-incompatible, it introduces a new
dependency, augments the API and potentially introduces subtle changes
in behavior, so I would definitely change the version number for that.
Again, that version number is just a hint for users.

>> BTW, we need to consider releasing 1.8.6 one of these days!  ;-)
>
> Yes.  Do we have any particular more things to get into this?  (I
> don't think I have anything.)

I'm interested in the scm_c_read issue I raised on the list, and Linas
is interested in threading issues.  The former seems more important to
me (eh eh ;-)) because it introduces a regression.  Then, I don't think
we have to solve every single bug right now, it's already been too long
since 1.8.5.

Thanks,
Ludo'.

Re: Guile release planning

[Andy Wingo]

Howdy Neil,

Great post!

On Tue 11 Nov 2008 02:23, "Neil Jerram" <neiljerram@googlemail.com> writes:

> How should we organize future Guile releases?
>
> In my view, the most important thing for Guile's near-to-medium-term
> future is focus.  By that I mean having developers working on, and
> users using, as far as possible, a similar level of code.  In the
> past, we did big jumps - from 1.4.x to 1.6.x, and from 1.6.x to 1.8.x
> - which I think left users unable easily to upgrade, or perhaps just
> unsure of whether to upgrade.  From the developer point of view, they
> increased the support burden (because of some users staying with the
> old series).

So, my history with Guile does not go back to the 1.4 days, but let's
take the data point that I think there are people out there doing
conditional compilation against 1.4 *and* 1.6 *and* 1.8.

What this says to me is that there is a class of users that was
satisfied with the facilities that 1.4 offered them, and has been
reluctantly adding #ifdefs ever since.

Then, you have users interested in new versions: those who want to use
pthreads, who want internationalization, who perceive benefit in API
cleanups -- in short, those with stockholm syndrome ;-)

Then you have the Manly Men Schemers, and it is a testosterone-laden
group for some reason. (I am pleased that this does not describe most
Guilers.) These are those that judge an implementation against other
implementations of a Scheme standard, together with the extensions
provided, and who snub their noses at Copying Call/CC and Conservative
Collection. But some of them deign to use Guile, usually for its
portability, Ubiquity (hah!), and the extensions that it provides.


The question is, what does change do to these groups of users?

To the first it is a burden. Continuity is key, certainly on the source
level, and to a lesser extent on the binary level. They can recompile
their software but they don't want to spend time adjusting to API
cleanups in Guile. That's cool.

I think we went wrong in the 1.4->1.6 and 1.6->1.8 transitions, by
leaving /source files/ to bitrot. We should have provided
guile-1.6-compat.[ch] and guile-1.8-compat.[ch] files for users to
include in their source trees, wrapping e.g. the gh_* API, or SCM_INUM.
That way their code stays usable, ours stays maintainable, and whenever
they decide to port to the new API, they already know how -- it's in
their source tree.

So in the future, as we make API breaks, we should provide these people
with breadcrumbs in the form of compatibility shims, and think of the
body of source code that is out there as we make our changes.


Then for the second kind of user, well, our job is to be Consistent: to
have powerful, orthogonal, and well-documented APIs, so that they can
get their work done. I think that we serve these people well, and that
over time we serve them better and better, with occaisional regressions
(e.g. the COW string stuff in the leadup to 1.8).


For the third group... they have been leaving us, because other Schemes
are getting faster and more capable over time. But here we have some
secret weapons: the wonderful Emacs integration, the VM, the possibility
for JIT compilation, possiblity for R6 integration as well -- not to
mention the broad range of extensions that Guile still has. So here our
job is to extend the boundaries of Guile as a Scheme, to make it a
system that you want to inhabit and write *all* of your program, and all
of your programs, in.


(Did you like the dangling preposition?)


So for me, and I'm babbling I know, the things are:

 * Shims for those that don't have time to deal with changes
   - will get people off of guile 1.6
 * Quicker binary-compatible development cycles for all
   - we should be up to 1.8.20 by now ;)
   - limited by developer resources of course, though now we are doing
     well
 * Break when you have something new to offer the second or third group
   - it's about time :)


Finally, a bit of marketing: in my opinion, the next Guile should be
2.0. The meaning ABI/API-wise is the same as 1.10, but it can have a
very nice impact on people's perceptions. It's like the presentation at
the recent GUADEC:

   GNOME 2.30

   => GNOME 30

   => GNOME 3.0

   Guile 1.10

   = > Guile 2.0

Imperfect, but delightful, no?

Happy hack,

Andy
-- 
http://wingolog.org/

Re: Guile release planning

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

> I think we went wrong in the 1.4->1.6 and 1.6->1.8 transitions, by
> leaving /source files/ to bitrot. We should have provided
> guile-1.6-compat.[ch] and guile-1.8-compat.[ch] files for users to
> include in their source trees, wrapping e.g. the gh_* API, or SCM_INUM.
> That way their code stays usable, ours stays maintainable, and whenever
> they decide to port to the new API, they already know how -- it's in
> their source tree.

Hmm, we have `gh.c' and `deprecated.c', which are kind enough to (i)
provide compatibility wrappers, and (ii) tell users how to upgrade to
the new API.  What do you dislike about it?

Or maybe you're referring to things like `guile-compatibility.h' in
G-Wrap?  Basically, it allows G-Wrap to use the latest API and have it
work with the older versions of Guile.

> Finally, a bit of marketing: in my opinion, the next Guile should be
> 2.0. The meaning ABI/API-wise is the same as 1.10, but it can have a
> very nice impact on people's perceptions.

I concur.  :-)

Thanks,
Ludo'.

Re: Guile release planning

[Andy Wingo]

Greets,

On Wed 12 Nov 2008 11:11, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> I think we went wrong in the 1.4->1.6 and 1.6->1.8 transitions, by
>> leaving /source files/ to bitrot. We should have provided
>> guile-1.6-compat.[ch] and guile-1.8-compat.[ch] files for users to
>> include in their source trees, wrapping e.g. the gh_* API, or SCM_INUM.
>> That way their code stays usable, ours stays maintainable, and whenever
>> they decide to port to the new API, they already know how -- it's in
>> their source tree.
>
> Hmm, we have `gh.c' and `deprecated.c', which are kind enough to (i)
> provide compatibility wrappers, and (ii) tell users how to upgrade to
> the new API.  What do you dislike about it?

Oh, I think they're great :) But for example, Clinton was complaining
earlier this week in IRC (#guile on freenode, for those that don't know)
about the removal of the gh API in 1.9. I think Bruno did too some
months back. Etc. It's totally reasonable to remove dead code like that
from Guile itself, but at every bump we should push those shims *out to
the user*, so they can have the compatibility shims in their source
code.

> Or maybe you're referring to things like `guile-compatibility.h' in
> G-Wrap? Basically, it allows G-Wrap to use the latest API and have it
> work with the older versions of Guile.

That is another nice option, for those in group (2) who actively follow
Guile development.

Cheers,

Andy
-- 
http://wingolog.org/

Re: Guile release planning

[Han-Wen Nienhuys]

Neil Jerram escreveu:
> So, what do you think?  There have been discussions of release
> strategy in the past, which I've seen as 50/50 between the split
> stable and development model (which we have now) and the steady new
> feature model (described above), but I don't recall them considering
> the overall community focus angle before.  In my view, when we add in
> that angle, the steady new feature model is better.

One angle that we could take is time based release planning, like GNOME and 
Fedora do: plan to do one or two releases per year on a rigid schedule.  The 
LilyPond 2.11 vs. 2.12 jump has been delaying for too long, but I generally do 
a biweekly release, which is stable enough to reasonably be called 'stable', and
it has worked very well so far.

The precondition for this is that there is a good test-suite so we can be
sure that a release that passes the tests is good.


-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen

Re: Guile release planning

[Andy Wingo]

Hi Han-Wen,

On Wed 12 Nov 2008 05:41, Han-Wen Nienhuys <hanwen@xs4all.nl> writes:

> One angle that we could take is time based release planning, like GNOME and 
> Fedora do: plan to do one or two releases per year on a rigid
> schedule.

With GNOME there is also a difference, as you know: they do not add API
in micro series, only in minor series, and never change the API or ABI
in incompatible ways. While that should be our intent, it's quite a
burden for a language implementation that exposes as much of its guts as
Guile does.

I think we should preserve Guile's ability to change API or ABI
incompatibly (while providing shims: next mail).

Releasing more often would be great, though.

Andy
-- 
http://wingolog.org/