Policies about shared libs (directly affects .deb packaging)

Policies about shared libs (directly affects .deb packaging)

[Rob Browning]

While packaging Guile 1.6, I've tried to keep things as simple as
possible.  In accordance with that, and contrary to my previous
discussions with people, I've been trying to see if we could get away
without having to have a bunch of packages -- including one for each
shared library in guile.

My current Debian packaging arrangement includes four packages:

  Package: guile-1.6
  Package: guile-1.6-libs
  Package: guile-1.6-dev
  Package: guile-1.6-doc

The biggest questions surround guile-1.6-libs and guile-1.6-dev.  With
this arrangement, *all* the shared libs are in guile-1.6-libs, and
guile-1.6-dev contains all the .a files and .so links (and the .la
files for any libs that don't have their major soname in the library
name i.e. libguile and libqthreads for example -- this is OK because
these aren't candidates for dynamic-link).

The problem is that it's OK to put all the libs in one package if *and
only if* we agree to some fairly stringent restrictions upstream.  If
I'm thinking straight, the restrictions are:

  1) We may never change the major number of any lib during a stable
     release cycle.

     Rationale: as an example, if we bumped libguile-srfi-srfi-4-v-1
     to libguile-srfi-srfi-4-v-2 during the 1.6 series, and if there
     are applications that have been compiled against v-1, then when I
     release the new guile-1.6-libs package with v-2, all of those
     applications will be missing libraries.  That's not allowed.

  2) We must change the major number of *all* libs with every new
     guile release, i.e. when we go from 1.6 to 1.8.

     Rationale: imagine we didn't.  Imagine we only changed libguile
     from libguile.so.12 to libguile.so.13 when we went from guile 1.6
     to guile 1.8 but left all the other lib versions alone.  Then the
     guile-1.8-libs package would contain libraries with file names
     that collided with file names from the guile-1.6-libs package
     (which would need to stay around to support apps that still
     required guile 1.6).  Even if I made guile-1.8-libs
     conflict/replace guile-1.6-libs in this case, then you'd could
     have older apps that were still linked against libguile 12, but
     were also using the newer libguile-srfi-srfi-4 from guile-1.8
     that's linked against libguile 13.

Note that these two restrictions *are* fairly Draconian, and so we may
not want to accept them.  In that case, guile will have to be packaged
for debian with one package per shared lib.  However, bear in mind
that this is not a panacea.  We will still have upstream restrictions,
they'll just be less obvious.

No matter what we do, any time we change any .scm file, we have to
make sure that it doesn't directly or indirectly alter the API of
*any* lib inside guile in a way that requires that lib to change its
major number.  This includes alterations to any data structures
returned or modified by any function in the API.  The same goes for
any change to a .c file in libguile.  If that change alters anything
visible to the outside world via one of our libraries, then each
affected library's major number has to be changed.

Of course our development behaviors coupled these two restrictions
could have the practical effect of making us comply with requirements
(1) and (2) anyway.

If we don't want to promise to accommodate restrictions (1) and (2),
then guile will have to be comprised of at least:

  Package: guile-1.6
  Package: libguile-12
  Package: libqthreads-12
  Package: libguile-ltdl-1
  Package: libguilereadline-v-12
  Package: libguile-srfi-srfi-4-v-1
  Package: libguile-srfi-srfi-13-14-v-1
  Package: guile-1.6-dev
  Package: guile-1.6-doc

During packaging I will have to be careful to split up the .scm files
appropriately among the various lib packages, and we'll have to be
very careful here upstream not to create inter-dependencies among the
non-libguile .scm files that could cause problems across releases of
different versions of a given lib.

Thoughts?

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Policies about shared libs (directly affects .deb packaging)

[Neil Jerram]

>>>>> "Rob" == Rob Browning <rlb@defaultvalue.org> writes:

    Rob>   1) We may never change the major number of any lib during a stable
    Rob>      release cycle.

The one problem scenario would be a lib in the core distro that wanted
to change much faster than the rest.  In this case, could we unbundle
just that one library?  In other words, if we try out these
restrictions for a while and they don't work for some library, is
there a recovery path?

Otherwise, this seems fine.  I strongly support restricting what we
can change within a stable release.  (Partly because it's right for
developers/users, partly because I don't like parallel development and
there are lots of goodies in 1.7 that I want to see released :-)

Just to be sure, though, what are the changes that should cause us to
increment the major number of a library?

    Rob>   2) We must change the major number of *all* libs with every new
    Rob>      guile release, i.e. when we go from 1.6 to 1.8.

This seems self-evident.  If there's a library in the distro that is
so non-dependent on libguile that this would be burdensome, why is
that library in the core distro?

    Rob> No matter what we do, any time we change any .scm file, we have to
    Rob> make sure that it doesn't directly or indirectly alter the API of
    Rob> *any* lib inside guile in a way that requires that lib to change its
    Rob> major number.

What about scm_eval_string("(proc-whose-behaviour-just-changed)") ?

    Rob> If we don't want to promise to accommodate restrictions (1) and (2),
    Rob> then guile will have to be comprised of at least:

    Rob>   Package: guile-1.6
    Rob>   Package: libguile-12
    Rob>   Package: libqthreads-12
    Rob>   Package: libguile-ltdl-1
    Rob>   Package: libguilereadline-v-12
    Rob>   Package: libguile-srfi-srfi-4-v-1
    Rob>   Package: libguile-srfi-srfi-13-14-v-1
    Rob>   Package: guile-1.6-dev
    Rob>   Package: guile-1.6-doc

    Rob> During packaging I will have to be careful to split up the .scm files
    Rob> appropriately among the various lib packages, and we'll have to be
    Rob> very careful here upstream not to create inter-dependencies among the
    Rob> non-libguile .scm files that could cause problems across releases of
    Rob> different versions of a given lib.

This sounds a lot like hard work, and probably a worse nightmare than
having your proposed restrictions.

        Neil



_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Policies about shared libs (directly affects .deb packaging)

[Rob Browning]

Neil Jerram <neil@ossau.uklinux.net> writes:

> The one problem scenario would be a lib in the core distro that wanted
> to change much faster than the rest.  In this case, could we unbundle
> just that one library?  In other words, if we try out these
> restrictions for a while and they don't work for some library, is
> there a recovery path?

I'd have to think carefully to be sure, but I *think* that might be
OK.  If we unbundle a lib, then the next micro revision of
guile-X.Y-libs would come out along with a libguile-foo-Z package.
This package would Depend on guile-X.Y-libs (>= new-revision), but
would also have to "Conflicts: guile-X.Y-libs" and "Replaces:
guile-X.Y-libs".  One drawback is that this could get messy in the
control file over time if we do it very often.

> Just to be sure, though, what are the changes that should cause us
> to increment the major number of a library?

libFOO's major number would need to be bumped given any changes that
could break any app or lib that used to be linked against the previous
copy of libFOO, and as I mentioned later, I think that in our case it
may sometimes be more than just the .c files since some lib's
corresponding .scm files could change their semantics in ways that
break backward compatibility wrt the *C* API, even if the C code
hadn't changed (see your and my comments below).  Note that this is
only true if the .c code is allowed to call functions or use variables
provided by the .scm files (or perhaps pass them out as data).  If we
wanted to make the restriction that that's not allowed, that .c code
may not depend on .scm files, then it should only be changes to the .c
files that can affect a lib's binary compatibility.  (Actually it's
the headers too since of course they can actually embed code in
*other* libs or apps).

The libtool info pages have a semi-decent explanation of why/when you
have to bump the major soname, though they call it the "interface
number", and as I recall they may not specifically address the fact
that even if the lib in question's code doesn't change, if that lib
re-exports data structures or functions from a sub-lib, then the lib
must to bump its soname whenever anything in the sub-lib changes "in a
relevant way" that the lib passes on via its API to its clients.

The above is my current understanding -- so everyone, please correct
me if I'm wrong.

>     Rob> No matter what we do, any time we change any .scm file, we have to
>     Rob> make sure that it doesn't directly or indirectly alter the API of
>     Rob> *any* lib inside guile in a way that requires that lib to change its
>     Rob> major number.
>
> What about scm_eval_string("(proc-whose-behaviour-just-changed)") ?

Good question, and one I think we need to answer, or even more
specifically imagine libguile-foo with:

   SCM scm_public_function_do_foo ()
   {
      return scm_eval_string ("(make-a-foo)");
   }

I've wondered about this for a while, and I think I talked to Marius
about it, but I can't recall what if anything was decided.

> This sounds a lot like hard work, and probably a worse nightmare than
> having your proposed restrictions.

It's not actually a ton of work since I've got a decent amount of it
automated, but it *is* more a more complicated arrangement for people
to understand when coming to guile.  It may, however, be the thing we
need to do...

And on a somewhat related note, I do worry a little that we may be
setting ourselves up for some unpleasant impressions from the broader
unix community with our library approach, though I don't have any good
solutions that aren't going to aggravate someone.

Our current approach makes sense, but imagine we do become a lot more
popular.  Is the broader non-scheme, non-guile community going to be
tolerant of the placement of hundreds of libguile-* libraries into
/usr/lib?  If we ever get to even a fraction of the number of add-on
packages in CPAN, that's where we'd be headed.  Not sure it matters,
and I'm not saying we should change our behavior, but I wanted to make
sure everyone's thought about some of the these potential "mostly
non-technical" issues.

It's my impression that perl and python, etc. don't suffer from this
nearly so much b/c they don't support/encourage the direct linking we
do.  Actually let me rephrase that -- they *do* suffer from the
"bazillion small packages problem", i.e. search debian for lib*-perl
packages, but they don't suffer from the "bazillion files in /usr/lib"
issue.

FWIW.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Policies about shared libs (directly affects .deb packaging)

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> My current Debian packaging arrangement includes four packages:
>
>   Package: guile-1.6
>   Package: guile-1.6-libs
>   Package: guile-1.6-dev
>   Package: guile-1.6-doc

OK.  Don't waste any more brainpower on this for now.  I think I'm
going to package the libs separately.  If that doesn't work out, then
we can continue the discussion.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Policies about shared libs (directly affects .deb packaging)

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> OK.  Don't waste any more brainpower on this for now.  I think I'm
> going to package the libs separately.  If that doesn't work out, then
> we can continue the discussion.

OK.  I've decided to break out libqthreads and libguile-ltdl since
they're fairly independent libs with no link against libguile, and
I've left the rest of the libs in guile-1.6-libs.

We'll see how that goes.  We will still have the restriction that we
cannot change the major number of any of the libguile linked libs
during a stable series, and that we *must* change all the major
numbers of the libguile linked libs during a new release, but for the
most part, that just enforces good behavior.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel