Cross-compiling Guile 2.0

Cross-compiling Guile 2.0

[Neil Jerram]

In principle, how should Guile 2.0 be cross-compiled?  I'm thinking
mostly of the part of the build that compiles all the installed modules.

I think target emulation is needed, using QEMU, and in particular that
it wouldn't work to set GUILE_FOR_BUILD to a build-system-native guile,
because that would generate .go files with the wrong architecture.

Alternatively, I suppose another approach would be to defer the module
compilation part of the build until after installation on the target.

Does that sound right?  Are there any other possibilities?

(The target is my Openmoko Freerunner phone.)

Thanks,
        Neil

Re: Cross-compiling Guile 2.0

[Andy Wingo]

On Sat 05 Mar 2011 20:47, Neil Jerram <neil@ossau.uklinux.net> writes:

> In principle, how should Guile 2.0 be cross-compiled?  I'm thinking
> mostly of the part of the build that compiles all the installed
> modules.

I have never cross-compiled anything, so I really don't know.

Ideally we could make a cross-compiling Guile that executed on the host,
but produced .go files for the target.

Hummmmmm.  Does that mean that for cross-compile builds, that we should
not add the $builddir to the LOAD_COMPILED_PATH?  Probably so, right?  A
cross-compiler probably doesn't autocompile either.  So then the
cross-compiler receives as its inputs the source files from a project,
then produces .go files for the target, but does not load .go files from
the target.

Does that sound right to you?

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Neil Jerram]

Andy Wingo <wingo@pobox.com> writes:

> On Sat 05 Mar 2011 20:47, Neil Jerram <neil@ossau.uklinux.net> writes:
>
>> In principle, how should Guile 2.0 be cross-compiled?  I'm thinking
>> mostly of the part of the build that compiles all the installed
>> modules.
>
> I have never cross-compiled anything, so I really don't know.
>
> Ideally we could make a cross-compiling Guile that executed on the host,
> but produced .go files for the target.
>
> Hummmmmm.  Does that mean that for cross-compile builds, that we should
> not add the $builddir to the LOAD_COMPILED_PATH?  Probably so, right?  A
> cross-compiler probably doesn't autocompile either.  So then the
> cross-compiler receives as its inputs the source files from a project,
> then produces .go files for the target, but does not load .go files from
> the target.
>
> Does that sound right to you?

Thanks for your input on this.

Yes.  In principle I think the cross-compiler could simultaneously
auto-compile, for itself, and non-auto-compile for the target, but
that's a bit mind-blowing and would require careful handling of what's
being written and read from where.  Instead of that, we could do two
separate steps: one where the cross-compiler compiles for itself, and
one to compile for the target.

What architectural dependencies are there in the .go format?  If there
aren't too many, a cross-compiler is probably the nicest solution,
because the other two options (QEMU, and compiling after installation on
the target) are probably going to be ridiculously slow.

       Neil

Re: Cross-compiling Guile 2.0

[Andy Wingo]

Hi,

On Sun 06 Mar 2011 12:03, Neil Jerram <neil@ossau.uklinux.net> writes:

> What architectural dependencies are there in the .go format?

Only endianness and word size.

Of course a macro could check something about the system at
expansion-time, for example a value in a header; and there are things
like the mapping of "long" to int32 or int64 in foreign.c.  But I guess
anything that explicitly depends on such things should know about
cross-compilation.

Cheers,

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Ludovic Courtès]

Hi Neil,

Neil Jerram <neil@ossau.uklinux.net> writes:

> In principle, how should Guile 2.0 be cross-compiled?  I'm thinking
> mostly of the part of the build that compiles all the installed modules.

Guile 2.0 can only be cross-compiled when the endianness and word size
of the host and target match (because the bytecode generation cannot be
parameterized by these.)

When that is the case, it can’t hurt to have $builddir/module in
$LOAD_COMPILED_PATH.

Of course, $GUILE_FOR_BUILD must point to a native Guile 2.0.

Hope this helps,
Ludo’.

Re: Cross-compiling Guile 2.0

[Jan Nieuwenhuizen]

Neil Jerram schreef op za 05-03-2011 om 19:47 [+0000]:
> In principle, how should Guile 2.0 be cross-compiled? 

We're using GUB -- http://lilypond.org/gub and are
sucessfully cross-building guile-2.0.  You may want
to look at some patches

    https://github.com/janneke/gub/tree/guile-2.0/patches

and te recipe

    https://github.com/janneke/gub/blob/guile-2.0/gub/specs/guile.py

> (The target is my Openmoko Freerunner phone.)

Ah, GUB used to have arm architectures, not sure
how up to date they are.

Jan.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: Cross-compiling Guile 2.0

[Andy Wingo]

On Sun 06 Mar 2011 23:12, ludo@gnu.org (Ludovic Courtès) writes:

> Neil Jerram <neil@ossau.uklinux.net> writes:
>
>> In principle, how should Guile 2.0 be cross-compiled?  I'm thinking
>> mostly of the part of the build that compiles all the installed modules.
>
> Guile 2.0 can only be cross-compiled when the endianness and word size
> of the host and target match (because the bytecode generation cannot be
> parameterized by these.)

You know, I think it's actually just endianness right now; I don't see
where word size comes into it, except for the cookie.

Anyway, in the future when we do AOT compilation, this question will be
more serious.  (I would like to do AOT stuff this year.)  So, following
the autoconf manual ("Specifying Target Triplets"), we would have build
== host, but target == the arm triple; so we would need
`target-endianness' (and possibly `target-word-size') in the compiler.

Where do you think we should put these, given that you don't want them
in (rnrs bytevectors)?  :-)

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Ludovic Courtès]

Hello Andy!

Andy Wingo <wingo@pobox.com> writes:

> On Sun 06 Mar 2011 23:12, ludo@gnu.org (Ludovic Courtès) writes:
>
>> Neil Jerram <neil@ossau.uklinux.net> writes:
>>
>>> In principle, how should Guile 2.0 be cross-compiled?  I'm thinking
>>> mostly of the part of the build that compiles all the installed modules.
>>
>> Guile 2.0 can only be cross-compiled when the endianness and word size
>> of the host and target match (because the bytecode generation cannot be
>> parameterized by these.)
>
> You know, I think it's actually just endianness right now; I don't see
> where word size comes into it, except for the cookie.

Oh, good.

> Anyway, in the future when we do AOT compilation, this question will be
> more serious.  (I would like to do AOT stuff this year.)  So, following
> the autoconf manual ("Specifying Target Triplets"), we would have build
> == host, but target == the arm triple; so we would need
> `target-endianness' (and possibly `target-word-size') in the compiler.

Yes.

> Where do you think we should put these, given that you don't want them
> in (rnrs bytevectors)?  :-)

I would add a ‘current-bytecode-endianness’ fluid in (language assembly
compile-bytecode).  The ‘compile’ procedure would have an optional
‘endianness’ parameter, which would set this fluid.  And ‘guile-tools
compile’ would have a command-line argument to specify the target
endianness.

WDYT?

Thanks,
Ludo’.

Re: Cross-compiling Guile 2.0

[Andy Wingo]

On Fri 18 Mar 2011 11:17, ludo@gnu.org (Ludovic Courtès) writes:

> I would add a ‘current-bytecode-endianness’ fluid in (language assembly
> compile-bytecode).  The ‘compile’ procedure would have an optional
> ‘endianness’ parameter, which would set this fluid.  And ‘guile-tools
> compile’ would have a command-line argument to specify the target
> endianness.

This sounds fine, but I would like to actually build a Guile
cross-compiler.  So build = host = my x86-64 laptop, but target = ARM
(for example).  Then you would use that Guile cross-compiler as
GUILE_FOR_BUILD, as you cross-compile Guile itself, or other packages
based on Guile.

It's true that a simple command-line argument and fluid could work, but
the situation will get more complicated, so we will need some part of
Guile to define the host and target triplets.  That's the questions I
was really asking: where in Guile to define those?

I am leaning towards some other module; (system machine-type) or
something, which would load some previously registered extension.

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

> It's true that a simple command-line argument and fluid could work, but
> the situation will get more complicated, so we will need some part of
> Guile to define the host and target triplets.  That's the questions I
> was really asking: where in Guile to define those?

There’s already ‘%host-type’.  However, I don’t think defining
‘%target-type’ would make sense since:

  1. Of the GNU triplet, only the $target_arch matters for bytecode;

  2. You can really choose at run-time what target you want to build
     for, by just setting the endianness fluid.

Now, if we want to produce something comparable to cross-GCC &
cross-Binutils[*], we could install, say, arm-linux-gnueabi-guile-tools
in addition to guile-tools; the former would somehow set the right
endianness for that target when running ‘arm-linux-gnueabi-guile-tools
compile’.

WDYT?

Thanks,
Ludo’.

[*] Installed cross-GCC and cross-Binutils provide binaries called
    TARGET-gcc, TARGET-ld, etc.

Re: Cross-compiling Guile 2.0

[Andy Wingo]

Hi :)

On Sun 20 Mar 2011 14:50, ludo@gnu.org (Ludovic Courtès) writes:

> There’s already ‘%host-type’.

Ah, cool.

> However, I don’t think defining ‘%target-type’ would make sense
> since:
>
>   1. Of the GNU triplet, only the $target_arch matters for bytecode;
>
>   2. You can really choose at run-time what target you want to build
>      for, by just setting the endianness fluid.

Can I convince you otherwise?  Right now it's endianness, but I would
like to hack an ARM native compiler sometime soonish, and that's going
to be easier as a cross-compiler; and at that point you do need the
%target-type.

I guess that's my point: right now it's endianness, in the future it
will be more things, might as well go ahead and add %target-type now.

> Now, if we want to produce something comparable to cross-GCC &
> cross-Binutils[*], we could install, say, arm-linux-gnueabi-guile-tools

Yes, that sounds good, though I would be happier if we did this simply
with --program-prefix and --target.

I really don't want to add an endianness kwarg to `compile', for
example, though an endianness fluid in (language assembly) is OK for
me.

Cheers,

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Ludovic Courtès]

Hi!

Andy Wingo <wingo@pobox.com> writes:

> On Sun 20 Mar 2011 14:50, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> However, I don’t think defining ‘%target-type’ would make sense
>> since:
>>
>>   1. Of the GNU triplet, only the $target_arch matters for bytecode;
>>
>>   2. You can really choose at run-time what target you want to build
>>      for, by just setting the endianness fluid.

What I meant to say here, is that via a couple of knobs akin to
‘current-target-endianness’, you could actually cross-build for any
target.

This is different from Binutils/GCC: a cross-GCC can only cross-build
for the target that was specified when that cross-GCC was built.
Conversely, a single Guile instance could compile code for any target.

Thus ‘%target-type’ would be inappropriate IMO because the target could
be chosen at run-time and it could be anything.

Does that make sense?

Besides, the UI could be made familiar.  For instance
‘guile-tools compile --target=TRIPLET’ would be fine with me.

> Can I convince you otherwise?  Right now it's endianness, but I would
> like to hack an ARM native compiler sometime soonish,

Ooh, cool!  :-)

>> Now, if we want to produce something comparable to cross-GCC &
>> cross-Binutils[*], we could install, say, arm-linux-gnueabi-guile-tools
>
> Yes, that sounds good, though I would be happier if we did this simply
> with --program-prefix and --target.

The prefix is added automatically for cross-Binutils and cross-GCC,
hence this example.

Thanks,
Ludo’.

Re: Cross-compiling Guile 2.0

[Andy Wingo]

On Sun 20 Mar 2011 22:31, ludo@gnu.org (Ludovic Courtès) writes:

> What I meant to say here, is that via a couple of knobs akin to
> ‘current-target-endianness’, you could actually cross-build for any
> target.

Yes, provided you have the compiler of course.

> Thus ‘%target-type’ would be inappropriate IMO because the target could
> be chosen at run-time and it could be anything.
>
> ‘guile-tools compile --target=TRIPLET’ would be fine with me.

I see.  Yes that sounds fine; perhaps we could have %target-type, but
have it be a fluid.  Does that make sense?

Cheers,

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

> On Sun 20 Mar 2011 22:31, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Thus ‘%target-type’ would be inappropriate IMO because the target could
>> be chosen at run-time and it could be anything.
>>
>> ‘guile-tools compile --target=TRIPLET’ would be fine with me.
>
> I see.  Yes that sounds fine; perhaps we could have %target-type, but
> have it be a fluid.  Does that make sense?

Yes, agreed.

At some point there will have to be a triplet → arch → endianness
conversion.  I’d rather have that conversion occur as close to the UI as
possible—i.e., close to ‘scripts/compile.scm’—rather than deep down in
(system base compile) because the triplet string is really a UI notion IMO.

Thanks,
Ludo’.

Re: Cross-compiling Guile 2.0

[Andy Wingo]

On Mon 21 Mar 2011 20:58, ludo@gnu.org (Ludovic Courtès) writes:

> At some point there will have to be a triplet → arch → endianness
> conversion.

Indeed.

>  I’d rather have that conversion occur as close to the UI as
> possible—i.e., close to ‘scripts/compile.scm’—rather than deep down in
> (system base compile) because the triplet string is really a UI notion IMO.

Actually I think it's more fundamental than that.  The compilation
process tries, generically, to find a path through the language tower
path from the source language to the target language.  There is nothing
bytecode-specific in (scripts compile), except for a couple default
target languages.

It seems to me that the value of the target-type fluid at the time that
`compile-passes' is called should be what determines the compilation
path and target-specific settings.  For example, to continue the ARM
compilation example, perhaps it would cause a different target language
to be selected.

Dunno.  All of that could just be too complicated, and maybe you are
right.

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Andy Wingo]

So!

On Mon 21 Mar 2011 21:42, Andy Wingo <wingo@pobox.com> writes:

> On Mon 21 Mar 2011 20:58, ludo@gnu.org (Ludovic Courtès) writes:
>
>> At some point there will have to be a triplet → arch → endianness
>> conversion.
>
> Indeed.
>
>>  I’d rather have that conversion occur as close to the UI as
>> possible—i.e., close to ‘scripts/compile.scm’—rather than deep down in
>> (system base compile) because the triplet string is really a UI notion IMO.
>
> Actually I think it's more fundamental than that.  The compilation
> process tries, generically, to find a path through the language tower
> path from the source language to the target language.  There is nothing
> bytecode-specific in (scripts compile), except for a couple default
> target languages.
>
> It seems to me that the value of the target-type fluid at the time that
> `compile-passes' is called should be what determines the compilation
> path and target-specific settings.  For example, to continue the ARM
> compilation example, perhaps it would cause a different target language
> to be selected.
>
> Dunno.  All of that could just be too complicated, and maybe you are
> right.

To re-take this topic...  I looked at this, and had a patch that looked
for a #:target-endianness option in the compile options.  I'm fairly
sure that it does the bytecode compilation OK, but then to write the
objcode out to disk we need to attach the right cookie on the beginning
(word size and endianness), and we don't have the #:target-endianness at
that point.

After having read "Using System Type" again in the autoconf manual, I am
convinced again that the right thing to do is to add a %target-type
fluid or mutable parameter, defaulting to the %host-type.  We select how
to compile for the target by making decisions based on %target-type.
Since in Guile those decisions don't currently need to be made anywhere
except in compile-bytecode.scm and in bytecode->objcode in objcodes.c,
this is easy.  In the future when the %target-type could select an
entirely different assembler et al, the "where" of the decisions could
be moved somewhat; but I think that the mechanism sounds right to me.

Let me know if you have objections.  Thanks!

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> skribas:

> After having read "Using System Type" again in the autoconf manual, I am
> convinced again that the right thing to do is to add a %target-type
> fluid or mutable parameter, defaulting to the %host-type.  We select how
> to compile for the target by making decisions based on %target-type.
> Since in Guile those decisions don't currently need to be made anywhere
> except in compile-bytecode.scm and in bytecode->objcode in objcodes.c,
> this is easy.  In the future when the %target-type could select an
> entirely different assembler et al, the "where" of the decisions could
> be moved somewhat; but I think that the mechanism sounds right to me.

Sounds good, but you’d need additional data that maps triplets (well,
$host_cpu) to low-level info such as endianness and word size, right?

Thanks,
Ludo’.

Re: Cross-compiling Guile 2.0

[Andy Wingo]

Hi :)

On Fri 27 May 2011 16:51, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribas:
>
>> After having read "Using System Type" again in the autoconf manual, I am
>> convinced again that the right thing to do is to add a %target-type
>> fluid or mutable parameter, defaulting to the %host-type.  We select how
>> to compile for the target by making decisions based on %target-type.
>> Since in Guile those decisions don't currently need to be made anywhere
>> except in compile-bytecode.scm and in bytecode->objcode in objcodes.c,
>> this is easy.  In the future when the %target-type could select an
>> entirely different assembler et al, the "where" of the decisions could
>> be moved somewhat; but I think that the mechanism sounds right to me.
>
> Sounds good, but you’d need additional data that maps triplets (well,
> $host_cpu) to low-level info such as endianness and word size, right?

Yes, which caused me a bit of agony on Friday after writing that mail.
Not resolved yet.  Durnit!

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Andy Wingo]

On Sun 29 May 2011 20:10, Andy Wingo <wingo@pobox.com> writes:

> On Fri 27 May 2011 16:51, ludo@gnu.org (Ludovic Courtès) writes:
>
>> Sounds good, but you’d need additional data that maps triplets (well,
>> $host_cpu) to low-level info such as endianness and word size, right?
>
> Yes, which caused me a bit of agony on Friday after writing that mail.
> Not resolved yet.  Durnit!

In the end I decided to punt.  I added a stub (system base target)
module and wired it up to the compiler, but didn't implement the
endianness / word-size database.  Patches accepted :)

Cheers,

Andy
-- 
http://wingolog.org/

Re: Cross-compiling Guile 2.0

[Ludovic Courtès]

Hi!

Andy Wingo <wingo@pobox.com> skribas:

> On Sun 29 May 2011 20:10, Andy Wingo <wingo@pobox.com> writes:
>
>> On Fri 27 May 2011 16:51, ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Sounds good, but you’d need additional data that maps triplets (well,
>>> $host_cpu) to low-level info such as endianness and word size, right?
>>
>> Yes, which caused me a bit of agony on Friday after writing that mail.
>> Not resolved yet.  Durnit!
>
> In the end I decided to punt.  I added a stub (system base target)
> module and wired it up to the compiler, but didn't implement the
> endianness / word-size database.

It might be possible to extract it semi-automatically from
gcc/config/*/*.h...

Thanks,
Ludo'.