Compiled load path issues

Compiled load path issues

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

Hello!

Once upon a time, if ‘chbouib.go’ is in the vicinity of ‘chbouib.scm’,
then ‘(use-modules (chbouib))’ would load ‘chbouib.go’.  This is no
longer the case, but can be remedied with something like this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 697 bytes --]

diff --git a/libguile/load.c b/libguile/load.c
index 50af256..194d1e5 100644
--- a/libguile/load.c
+++ b/libguile/load.c
@@ -726,7 +726,9 @@ SCM_DEFINE (scm_primitive_load_path, "primitive-load-path", 0, 0, 1,
     exception_on_not_found = SCM_BOOL_T;
 
   full_filename = scm_sys_search_load_path (filename);
-  compiled_filename = scm_search_path (*scm_loc_load_compiled_path,
+  compiled_filename = scm_search_path (scm_append
+				        (scm_list_2 (*scm_loc_load_compiled_path,
+						     *scm_loc_load_path)),
                                        filename,
                                        *scm_loc_load_compiled_extensions,
                                        SCM_BOOL_T);

[-- Attachment #3: Type: text/plain, Size: 445 bytes --]


Andy: can you comment?  What was the idea behind ‘%load-compiled-path’?

Besides, ‘scm_search_path ()’ was changed incompatibly compared to 1.8
in 22f4ee48822db5e30df3abf9a11b6066f2bab9d3.  I’m wary about such
incompatibilities and would like it if we could (1) list them, and
(2) avoid them unless we really really can’t think of any other way.  In
this particular case, do you have an idea on how to avoid it?

Thanks,
Ludo’.

Re: Compiled load path issues

[Andy Wingo]

On Sun 18 Oct 2009 17:36, ludo@gnu.org (Ludovic Courtès) writes:

> Once upon a time, if ‘chbouib.go’ is in the vicinity of ‘chbouib.scm’,
> then ‘(use-modules (chbouib))’ would load ‘chbouib.go’.  This is no
> longer the case, but can be remedied with something like this:
>
> diff --git a/libguile/load.c b/libguile/load.c
> index 50af256..194d1e5 100644
> --- a/libguile/load.c
> +++ b/libguile/load.c
> @@ -726,7 +726,9 @@ SCM_DEFINE (scm_primitive_load_path, "primitive-load-path", 0, 0, 1,
>      exception_on_not_found = SCM_BOOL_T;
>  
>    full_filename = scm_sys_search_load_path (filename);
> -  compiled_filename = scm_search_path (*scm_loc_load_compiled_path,
> +  compiled_filename = scm_search_path (scm_append
> +				        (scm_list_2 (*scm_loc_load_compiled_path,
> +						     *scm_loc_load_path)),
>                                         filename,
>                                         *scm_loc_load_compiled_extensions,
>                                         SCM_BOOL_T);
>
> Andy: can you comment?  What was the idea behind
> ‘%load-compiled-path’?

The idea is that given that the compiled files are
architecture-dependent, that they should go in $libdir instead of
$datadir. We can add $libdir, but I don't think it's a good idea -- not
only for reasons of excessive stat, but because I don't think we should
be putting binaries in with installed source.

> Besides, ‘scm_search_path ()’ was changed incompatibly compared to 1.8
> in 22f4ee48822db5e30df3abf9a11b6066f2bab9d3.  I’m wary about such
> incompatibilities and would like it if we could (1) list them, and
> (2) avoid them unless we really really can’t think of any other way.  In
> this particular case, do you have an idea on how to avoid it?

I don't really know. I'm sure it could be worked around somehow, but
it's not very fun work.

Andy
-- 
http://wingolog.org/

Re: Compiled load path issues

[Ludovic Courtès]

Hi!

Andy Wingo <wingo@pobox.com> writes:

> On Sun 18 Oct 2009 17:36, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Andy: can you comment?  What was the idea behind
>> ‘%load-compiled-path’?
>
> The idea is that given that the compiled files are
> architecture-dependent,

In theory, we could interpret the ‘.go’ cookie and byte-swap things if
needed...

> that they should go in $libdir instead of $datadir.  We can add
> $libdir, but I don't think it's a good idea -- not only for reasons of
> excessive stat, but because I don't think we should be putting
> binaries in with installed source.

By now people may have started to update their packages to run
“guile-tools compile” and install ‘.go’ files, so we really need to get
this issue settled.

I’m in favor of ‘.go’ alongside ‘.scm’: that’s what happens with
.elc/.el and .pyc/.py and it had been the plan from 1.9.0 until
recently.

>> Besides, ‘scm_search_path ()’ was changed incompatibly compared to 1.8
>> in 22f4ee48822db5e30df3abf9a11b6066f2bab9d3.  I’m wary about such
>> incompatibilities and would like it if we could (1) list them, and
>> (2) avoid them unless we really really can’t think of any other way.  In
>> this particular case, do you have an idea on how to avoid it?
>
> I don't really know. I'm sure it could be worked around somehow, but
> it's not very fun work.

It’s not, but there’s a fair amount of not very fun work in this vain to
be done by 2.0.  :-)

I think we must pay close attention to backwards compatibility, at least
to honor long time promises
(http://lists.gnu.org/archive/html/guile-devel/2003-02/msg00074.html).

Thanks,
Ludo’.

Re: Compiled load path issues

[Andy Wingo]

Hello ludovic,

On Tue 20 Oct 2009 10:27, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Sun 18 Oct 2009 17:36, ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>> Andy: can you comment?  What was the idea behind
>>> ‘%load-compiled-path’?
>>
>> The idea is that given that the compiled files are
>> architecture-dependent,
>
> In theory, we could interpret the ‘.go’ cookie and byte-swap things if
> needed...

In theory yes. In practice we map things read-only so they can be cached
and not copied, and we'd have to instrument individual VM ops with
checks based on the current objcode, flags for the objcode to say their
format, etc. I really think it's too complicated. If this is really the
way we want to go, we should give up on having endian-specific
bytecode -- which is a bigger task, not to mention the tail wagging the
dog.

>> that they should go in $libdir instead of $datadir.  We can add
>> $libdir, but I don't think it's a good idea -- not only for reasons of
>> excessive stat, but because I don't think we should be putting
>> binaries in with installed source.
>
> By now people may have started to update their packages to run
> “guile-tools compile” and install ‘.go’ files, so we really need to get
> this issue settled.
>
> I’m in favor of ‘.go’ alongside ‘.scm’: that’s what happens with
> .elc/.el and .pyc/.py and it had been the plan from 1.9.0 until
> recently.

For python, pyc files are in $libdir, for exactly this reason. Plus, you
might have some source files that you want to compile with multiple
versions of Guile. I don't think we should be encouraging this.

>>> Besides, ‘scm_search_path ()’ was changed incompatibly compared to 1.8
>>> in 22f4ee48822db5e30df3abf9a11b6066f2bab9d3.  I’m wary about such
>>> incompatibilities and would like it if we could (1) list them, and
>>> (2) avoid them unless we really really can’t think of any other way.  In
>>> this particular case, do you have an idea on how to avoid it?
>>
>> I don't really know. I'm sure it could be worked around somehow, but
>> it's not very fun work.
>
> It’s not, but there’s a fair amount of not very fun work in this vain to
> be done by 2.0.  :-)
>
> I think we must pay close attention to backwards compatibility, at least
> to honor long time promises
> (http://lists.gnu.org/archive/html/guile-devel/2003-02/msg00074.html).

So for the list, we did have a chat about this on IRC. We both agree
that we should not needlessly introduce incompatibilities, especially on
the C level. This is especially a problem because e.g. gnucash,
configuring as it is with guile.m4 and guile-config, will simply pick up
the new version of Guile when it's installed -- which is like upgrading
when you didn't choose to. Gnucash should only have to be concerned with
Guile when it chooses to.

For that reason we also think that Guile should be parallel-installable,
at least on the library level. That means that we should have the
version in the library name, and the version in the include path; so
pkg-config --cflags guile-2.0 will say e.g. -I/usr/include/guile-2.0,
and that pkg-config --libs guile-2.0 will be e.g. -lguile-2-0, or
something. A more detailed manifesto is here:

  http://www106.pair.com/rhp/parallel.html

  Removing barriers to new version adoption

  The big benefit of parallel installation is that you remove the reason
  why people are reluctant to upgrade to a new version of Foo, because
  upgrading to Foo 5 has no effect on users of Foo 4. This means that
  the packages in [the distro] can be upgraded by their upstream
  maintainers, one at a time. It means that GNOME packages can upgrade
  to Foo 5 one at a time. It means that if I use a text editor dependent
  on Foo 4, but want my package to use Foo 5, I can do that since I can
  install both versions of Foo at the same time.

  In short, parallel install nukes the chicken and egg problem, and it
  saves everyone a bunch of time and energy.

  (A side effect: suddenly you have far more freedom to break backward
  compatibility; your new incompatible version is in fact a different
  library, not the same library. So if you need to clean up a big nest
  of cruft, no big deal. No one is forced to upgrade until they have
  time to deal with the breakage.)

That last paragraph is key for me. Sure, we need to make
well-thought-out changes -- but our current policy of very extended
C-level compatibility is very, very limiting, and a big energy drain. If
we think we need to change a function interface, well, we just change
it, and document the change as well -- perhaps even with a Coccinelle[0]
patch. So whenever Lilypond realizes that Guile 2.2 gives them native
Scheme compilation, but changes a couple of functions, they can weigh
their choice, and if they decide to upgrade, they know what to do.

[0] http://lwn.net/Articles/315686/

Anyway, comments welcome. Ludovic let me know if this actually
represents what you think, or if I'm just putting bytes in your mouth :)

Cheers,

Andy
-- 
http://wingolog.org/

Re: Compiled load path issues

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

> On Tue 20 Oct 2009 10:27, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> In theory, we could interpret the ‘.go’ cookie and byte-swap things if
>> needed...
>
> In theory yes. In practice we map things read-only so they can be cached
> and not copied, and we'd have to instrument individual VM ops

Or we could copy the bytecode (when it’s foreign) instead of mmapping
it, and translate instructions that are endianness-sensitive.

Now, I agree it’s a big task, and one I’d rather avoid 2 months before
2.0.

>> I’m in favor of ‘.go’ alongside ‘.scm’: that’s what happens with
>> .elc/.el and .pyc/.py and it had been the plan from 1.9.0 until
>> recently.
>
> For python, pyc files are in $libdir, for exactly this reason.

Out of curiosity, are .pyc endianness- and word-size-sensitive?

What about .elc?

Guile itself currently installs ‘.go’ under $pkgdatadir.  That would
need to be changed to $pkglibdir/1.9, right?

Also we now have $GUILE_LOAD_PATH and $GUILE_LOAD_COMPILED_PATH.  Should
we also have an equivalent to ‘-L’?

Thanks,
Ludo’.

Re: Compiled load path issues

[Andy Wingo]

Hi Ludovic,

On Wed 21 Oct 2009 18:22, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Tue 20 Oct 2009 10:27, ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>> In theory, we could interpret the ‘.go’ cookie and byte-swap things if
>>> needed...
>>
>> In theory yes. In practice we map things read-only so they can be cached
>> and not copied, and we'd have to instrument individual VM ops
>
> Or we could copy the bytecode (when it’s foreign) instead of mmapping
> it, and translate instructions that are endianness-sensitive.
>
> Now, I agree it’s a big task, and one I’d rather avoid 2 months before
> 2.0.

I agree. (I'd rather avoid it altogether.)

>>> I’m in favor of ‘.go’ alongside ‘.scm’: that’s what happens with
>>> .elc/.el and .pyc/.py and it had been the plan from 1.9.0 until
>>> recently.
>>
>> For python, pyc files are in $libdir, for exactly this reason.
>
> Out of curiosity, are .pyc endianness- and word-size-sensitive?

I think so, yes. I could be wrong.

> What about .elc?

Don't know.

> Guile itself currently installs ‘.go’ under $pkgdatadir.  That would
> need to be changed to $pkglibdir/1.9, right?

.scm files got to $pkgdatadir, .go files to $pkglibdir.

> Also we now have $GUILE_LOAD_PATH and $GUILE_LOAD_COMPILED_PATH.  Should
> we also have an equivalent to ‘-L’?

Hm, good question. I don't know the answer to that one.

Regards,

Andy
-- 
http://wingolog.org/

Parallel installability, API stability

[Ludovic Courtès]

Hi again!

Andy Wingo <wingo@pobox.com> writes:

> On Tue 20 Oct 2009 10:27, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> I think we must pay close attention to backwards compatibility, at least
>> to honor long time promises
>> (http://lists.gnu.org/archive/html/guile-devel/2003-02/msg00074.html).

[...]

> For that reason we also think that Guile should be parallel-installable,
> at least on the library level. That means that we should have the
> version in the library name, and the version in the include path; so
> pkg-config --cflags guile-2.0 will say e.g. -I/usr/include/guile-2.0,
> and that pkg-config --libs guile-2.0 will be e.g. -lguile-2-0, or
> something.

Agreed.

[...]

> Sure, we need to make well-thought-out changes -- but our current
> policy of very extended C-level compatibility is very, very limiting,
> and a big energy drain. If we think we need to change a function
> interface, well, we just change it, and document the change as well --
> perhaps even with a Coccinelle[0] patch.

I’d probably be more conservative than you on API changes.  For
instance, I think ‘scm_search_path()’ and ‘scm_primitive_load()’ should
be the same in 1.8 and 2.0 (at least at the C level, because in Scheme
it’s easy to retain compatibility with optional args.)

Guile is a niche, and so is Scheme.  Among those who develop
applications using Guile, I bet the vast majority does it on their free
time.  Free time is scarce and better spent writing application code
than upgrading to the latest APIs.  Thus, I think Guile core should
evolve hands in hands with its users, making, indeed, only
well-thought-out API changes.

Actually, all GNU libraries I can think of, as well as Emacs, follow
this scheme.  I don’t think this is pure coincidence.  GTK+, Python,
etc. can afford disruptive changes, but their “ecosystem” is much
different.

Thanks,
Ludo’.

Re: Parallel installability, API stability

[Andy Wingo]

On Wed 21 Oct 2009 18:45, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> Sure, we need to make well-thought-out changes -- but our current
>> policy of very extended C-level compatibility is very, very limiting,
>> and a big energy drain. If we think we need to change a function
>> interface, well, we just change it, and document the change as well --
>> perhaps even with a Coccinelle[0] patch.
>
> I’d probably be more conservative than you on API changes.  For
> instance, I think ‘scm_search_path()’ and ‘scm_primitive_load()’ should
> be the same in 1.8 and 2.0 (at least at the C level, because in Scheme
> it’s easy to retain compatibility with optional args.)

I'm OK with changing these ones back. I think we agree here.

> Guile is a niche, and so is Scheme.  Among those who develop
> applications using Guile, I bet the vast majority does it on their free
> time.

(Me too.)

But regarding user counts -- I still think we will have many more users
in the future than we have had in the past. We should think of them,
too...

> Thus, I think Guile core should evolve hands in hands with its users,
> making, indeed, only well-thought-out API changes.

OK. I really do think we agree for the most part on this. I'd rather not
spend any more energy on the question.

Regards,

Andy
-- 
http://wingolog.org/