64-bit Guile on Windows

64-bit Guile on Windows

[Thomas Thiriez via General Guile related discussions]

Hi,

I am currently porting a macOS application to Windows, and need to 
have libguile working.

I can see GnuCash works with Guile. I could borrow their 
libguile.dll, but it is a 32-bit build, and I am making a 64-bit 
application. I tried building Guile 3, but had more success with 
Guile 2.2, so I will keep that version.

I have cross compiled it on my mac with mingw, and it kind of 
works. The only thing is that if I compile my .scm files to .go, 
then my fixnums are not automatically converted to big nums, and I 
get things like that:

(* 999 999 999) -> -76738825

My fix is to keep the .scm files, and avoid compiling to .go.

I believe this has something to do with the fact that, on that 
platform, we have:

sizeof(SCM)  == 8
sizeof(long) == 4

Is there something to do about that problem? Is there something I 
can do to have a working 64-bit Guile on Windows?

Thanks a lot for any help you could provide.

Regards,
Thomas Thiriez

Re: 64-bit Guile on Windows

[Jan Nieuwenhuizen]

Thomas Thiriez via General Guile related discussions writes:

Hello,

> I am currently porting a macOS application to Windows, and need to
> have libguile working.
[..]
> I tried building Guile 3, but had more success with Guile 2.2, so I will
> keep that version.

There is a 64-bit port on the wip-mingw branch

    https://git.savannah.gnu.org/cgit/guile.git/log/?h=wip-mingw

but that is based on guile-3.0.8.

> I believe this has something to do with the fact that, on that
> platform, we have:
>
> sizeof(SCM)  == 8
> sizeof(long) == 4

Yes, indeed.

> Is there something to do about that problem? Is there something I can
> do to have a working 64-bit Guile on Windows?

It shouldn't be too much work to backport the fixes to guile-2.2.

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://lilypond.org
Freelance IT https://JoyOfSource.com | Avatar® https://AvatarAcademy.com

Re: 64-bit Guile on Windows

[Thomas Thiriez via General Guile related discussions]

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Thomas Thiriez via General Guile related discussions writes:
>
> Hello,
>
>> I am currently porting a macOS application to Windows, and need 
>> to
>> have libguile working.
> [..]
>> I tried building Guile 3, but had more success with Guile 2.2, 
>> so I will
>> keep that version.
>
> There is a 64-bit port on the wip-mingw branch
>
>     https://git.savannah.gnu.org/cgit/guile.git/log/?h=wip-mingw
>
> but that is based on guile-3.0.8.

Ah, that's good news. I have installed it with Cygwin, and it is 
working fine.

If I understand correctly, in order to use that version of 
libguile, I would have to redistribute the Cygwin libs with it, 
and the license would not work well with my proprietary 
application, am I right?

Regards,
Thomas Thiriez

Re: 64-bit Guile on Windows

[Jean Abou Samra]

Le 27/06/2022 à 15:56, Thomas Thiriez via General Guile related 
discussions a écrit :
> Hi,
>
> I am currently porting a macOS application to Windows, and need to 
> have libguile working.
>
> I can see GnuCash works with Guile. I could borrow their libguile.dll, 
> but it is a 32-bit build, and I am making a 64-bit application. I 
> tried building Guile 3, but had more success with Guile 2.2, so I will 
> keep that version.
>
> I have cross compiled it on my mac with mingw, and it kind of works. 
> The only thing is that if I compile my .scm files to .go, then my 
> fixnums are not automatically converted to big nums, and I get things 
> like that:
>
> (* 999 999 999) -> -76738825
>
> My fix is to keep the .scm files, and avoid compiling to .go.
>
> I believe this has something to do with the fact that, on that 
> platform, we have:
>
> sizeof(SCM)  == 8
> sizeof(long) == 4
>
> Is there something to do about that problem? Is there something I can 
> do to have a working 64-bit Guile on Windows?
>
> Thanks a lot for any help you could provide.


We had exactly the same problem at LilyPond, and this was the fix:

https://gitlab.com/lilypond/lilypond/-/blob/master/release/binaries/lib/dependencies.py#L721

Namely, you need to patch libguile/conv-integer.i.c and
conv-uinteger.i.c to replace "SIZEOF_TYPE < SIZEOF_SCM_T_BITS"
with "SIZEOF_TYPE < SIZEOF_LONG".

HTH,
Jean

Re: 64-bit Guile on Windows

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]

Jean Abou Samra schreef op di 28-06-2022 om 10:38 [+0200]:
> We had exactly the same problem at LilyPond, and this was the fix:
> 
> https://gitlab.com/lilypond/lilypond/-/blob/master/release/binaries/lib/dependencies.py#L721
> 

For security, shouldn't this check the hash of the downloaded tarballls
and patches?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: 64-bit Guile on Windows

[Jean Abou Samra]

On 6/28/22 12:52, Maxime Devos wrote:
> Jean Abou Samra schreef op di 28-06-2022 om 10:38 [+0200]:
>> We had exactly the same problem at LilyPond, and this was the fix:
>>
>> https://gitlab.com/lilypond/lilypond/-/blob/master/release/binaries/lib/dependencies.py#L721
>>
> For security, shouldn't this check the hash of the downloaded tarballls
> and patches?


Sorry, I forgot to reply to this. Yes, it likely should. On the other
hand, LilyPond has a lot of much more pressing security issues to care 
about…

Jean

Re: 64-bit Guile on Windows

[Thomas Thiriez via General Guile related discussions]

Jean Abou Samra <jean@abou-samra.fr> writes:

> Le 27/06/2022 à 15:56, Thomas Thiriez via General Guile related
> discussions a écrit :
[...]
>> (* 999 999 999) -> -76738825
[...]
>
> We had exactly the same problem at LilyPond, and this was the 
> fix:
>
> https://gitlab.com/lilypond/lilypond/-/blob/master/release/binaries/lib/dependencies.py#L721
>
> Namely, you need to patch libguile/conv-integer.i.c and
> conv-uinteger.i.c to replace "SIZEOF_TYPE < SIZEOF_SCM_T_BITS"
> with "SIZEOF_TYPE < SIZEOF_LONG".
>
> HTH,
> Jean

Thanks for the info. I have tried this, but it doesn't appear to 
be helping.

I did a few tests with lilypond, and here is what I found. I have 
a test.scm file containing:

(display (* 999)) (newline)
(display (* 999 999)) (newline)
(display (* 999 999 999)) (newline)

lilypond.exe -e '(load \"test.scm\")' test.ly
GNU LilyPond 2.23.10 (running Guile 2.2)
999
998001
997002999

That is fine. Now, if I try compiling test.scm to test.go, I get:

lilypond.exe -e '(use-modules (system base compile))(compile-file 
\"test.scm\" #:output-file \"test.go\")' test.ly
GNU LilyPond 2.23.10 (running Guile 2.2)
ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
In procedure bytevector-u64-set!: Value out of range: -149659645

That doesn't work, so I compile test.scm to test.go with guile on 
my Mac. If I then try loading the generated test.go with lilypond, 
I get:

lilypond.exe -e '(load \"test.go\")' test.ly
GNU LilyPond 2.23.10 (running Guile 2.2)
999
998001
-76738825

Now, it is possible that the problem is that I am trying to run a 
.go file compiled on macOS on a Windows machine, and that the two 
platforms are different enough that the .go file can't be reused. 
And because I get that "Value out of range" error when compiling 
on Windows, I am probably stuck with .scm files instead of 
compiled .go files.

Regards,
Thomas Thiriez

Re: 64-bit Guile on Windows

[Thomas Thiriez via General Guile related discussions]

Thomas Thiriez via General Guile related discussions 
<guile-user@gnu.org> writes:

> That doesn't work, so I compile test.scm to test.go with guile 
> on my
> Mac.

I have also tried compiling with this command on my Mac to 
generate bytecode for the right architecture, but the result is 
the same:

guild compile test.scm -T x86_64-w64-mingw32 -o test.go

Re: 64-bit Guile on Windows

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1817 bytes --]

Thomas Thiriez via General Guile related discussions schreef op di 28-
06-2022 om 16:41 [+0200]:
> Jean Abou Samra <jean@abou-samra.fr> writes:
> 
> > Le 27/06/2022 à 15:56, Thomas Thiriez via General Guile related
> > discussions a écrit :
> [...]
> > > (* 999 999 999) -> -76738825
> [...]
> > 
> > We had exactly the same problem at LilyPond, and this was the 
> > fix:
> > 
> > https://gitlab.com/lilypond/lilypond/-/blob/master/release/binaries/lib/dependencies.py#L721
> > 
> > Namely, you need to patch libguile/conv-integer.i.c and
> > conv-uinteger.i.c to replace "SIZEOF_TYPE < SIZEOF_SCM_T_BITS"
> > with "SIZEOF_TYPE < SIZEOF_LONG".
> > 
> > HTH,
> > Jean
> 
> Thanks for the info. I have tried this, but it doesn't appear to 
> be helping.
> 
> I did a few tests with lilypond, and here is what I found. I have 
> a test.scm file containing:
> 
> (display (* 999)) (newline)
> (display (* 999 999)) (newline)
> (display (* 999 999 999)) (newline)
> 
> lilypond.exe -e '(load \"test.scm\")' test.ly
> GNU LilyPond 2.23.10 (running Guile 2.2)
> 999
> 998001
> 997002999
> 
> That is fine. Now, if I try compiling test.scm to test.go, I get:
> 
> lilypond.exe -e '(use-modules (system base compile))(compile-file 
> \"test.scm\" #:output-file \"test.go\")' test.ly
> GNU LilyPond 2.23.10 (running Guile 2.2)
> ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
> In procedure bytevector-u64-set!: Value out of range: -149659645

You can find some possibly related bugs on debbugs:

Possibly fixnum-related, so maybe:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42060

Maybe 32-bit <-> 64-bit related, so maybe:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28920

Also, guile@2.2 is not developed anymore AFAICT.

Greetings,
Maixme.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: 64-bit Guile on Windows

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2201 bytes --]


Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:No public key for 49E3EE22191725EE created at 2022-06-28T18:14:22+0200 using EDDSA]]
> Thomas Thiriez via General Guile related discussions schreef op di 28-
> 06-2022 om 16:41 [+0200]:
>> Jean Abou Samra <jean@abou-samra.fr> writes:
>> 
>> > Le 27/06/2022 à 15:56, Thomas Thiriez via General Guile related
>> > discussions a écrit :
>> [...]
>> > > (* 999 999 999) -> -76738825
>> [...]
>> > 
>> > We had exactly the same problem at LilyPond, and this was the 
>> > fix:
>> > 
>> > https://gitlab.com/lilypond/lilypond/-/blob/master/release/binaries/lib/dependencies.py#L721
>> > 
>> > Namely, you need to patch libguile/conv-integer.i.c and
>> > conv-uinteger.i.c to replace "SIZEOF_TYPE < SIZEOF_SCM_T_BITS"
>> > with "SIZEOF_TYPE < SIZEOF_LONG".
>> > 
>> > HTH,
>> > Jean
>> 
>> Thanks for the info. I have tried this, but it doesn't appear to 
>> be helping.
>> 
>> I did a few tests with lilypond, and here is what I found. I have 
>> a test.scm file containing:
>> 
>> (display (* 999)) (newline)
>> (display (* 999 999)) (newline)
>> (display (* 999 999 999)) (newline)
>> 
>> lilypond.exe -e '(load \"test.scm\")' test.ly
>> GNU LilyPond 2.23.10 (running Guile 2.2)
>> 999
>> 998001
>> 997002999
>> 
>> That is fine. Now, if I try compiling test.scm to test.go, I get:
>> 
>> lilypond.exe -e '(use-modules (system base compile))(compile-file 
>> \"test.scm\" #:output-file \"test.go\")' test.ly
>> GNU LilyPond 2.23.10 (running Guile 2.2)
>> ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
>> In procedure bytevector-u64-set!: Value out of range: -149659645
>
> Possibly fixnum-related, so maybe:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42060
>
> Maybe 32-bit <-> 64-bit related, so maybe:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28920
>
> Also, guile@2.2 is not developed anymore AFAICT.

From what I see, Guile 2.2 does still get fixes, but no larger changes.
See https://git.savannah.gnu.org/cgit/guile.git/log/?h=stable-2.2

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: 64-bit Guile on Windows

[Thomas Thiriez via General Guile related discussions]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

[...]
>> Possibly fixnum-related, so maybe:
>>
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42060
>>
>> Maybe 32-bit <-> 64-bit related, so maybe:
>>
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28920

Thanks a lot. I will check those out.

>> Also, guile@2.2 is not developed anymore AFAICT.
>
> From what I see, Guile 2.2 does still get fixes, but no larger 
> changes.
> See 
> https://git.savannah.gnu.org/cgit/guile.git/log/?h=stable-2.2

I was able to have something at least partially working with Guile 
2.2, but if I could have a working Guile 3 on Windows, that would 
be even better!

Regards,
Thomas Thiriez

Re: 64-bit Guile on Windows

[Jean Abou Samra]

[-- Attachment #1.1: Type: text/plain, Size: 2246 bytes --]



Le 28/06/2022 à 16:41, Thomas Thiriez via General Guile related 
discussions a écrit :
>
> Jean Abou Samra <jean@abou-samra.fr> writes:
>
>> Le 27/06/2022 à 15:56, Thomas Thiriez via General Guile related
>> discussions a écrit :
> [...]
>>> (* 999 999 999) -> -76738825
> [...]
>>
>> We had exactly the same problem at LilyPond, and this was the fix:
>>
>> https://gitlab.com/lilypond/lilypond/-/blob/master/release/binaries/lib/dependencies.py#L721 
>>
>>
>> Namely, you need to patch libguile/conv-integer.i.c and
>> conv-uinteger.i.c to replace "SIZEOF_TYPE < SIZEOF_SCM_T_BITS"
>> with "SIZEOF_TYPE < SIZEOF_LONG".
>>
>> HTH,
>> Jean
>
> Thanks for the info. I have tried this, but it doesn't appear to be 
> helping.
>
> I did a few tests with lilypond, and here is what I found. I have a 
> test.scm file containing:
>
> (display (* 999)) (newline)
> (display (* 999 999)) (newline)
> (display (* 999 999 999)) (newline)
>
> lilypond.exe -e '(load \"test.scm\")' test.ly
> GNU LilyPond 2.23.10 (running Guile 2.2)
> 999
> 998001
> 997002999
>
> That is fine. Now, if I try compiling test.scm to test.go, I get:
>
> lilypond.exe -e '(use-modules (system base compile))(compile-file 
> \"test.scm\" #:output-file \"test.go\")' test.ly
> GNU LilyPond 2.23.10 (running Guile 2.2)
> ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
> In procedure bytevector-u64-set!: Value out of range: -149659645
>
> That doesn't work, so I compile test.scm to test.go with guile on my 
> Mac. If I then try loading the generated test.go with lilypond, I get:
>
> lilypond.exe -e '(load \"test.go\")' test.ly
> GNU LilyPond 2.23.10 (running Guile 2.2)
> 999
> 998001
> -76738825
>
> Now, it is possible that the problem is that I am trying to run a .go 
> file compiled on macOS on a Windows machine, and that the two 
> platforms are different enough that the .go file can't be reused. And 
> because I get that "Value out of range" error when compiling on 
> Windows, I am probably stuck with .scm files instead of compiled .go 
> files.


This bug is now hitting us too, we're investigating it here:

https://gitlab.com/lilypond/lilypond/-/issues/6474

Best,
Jean


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: 64-bit Guile on Windows

[Mike Gran]

On Tue, Nov 22, 2022 at 01:44:47PM +0100, Jean Abou Samra wrote:
> 
>> > Now, it is possible that the problem is that I am trying to run a
> > .go file compiled on macOS on a Windows machine, and that the two
> > platforms are different enough that the .go file can't be reused.
> > And because I get that "Value out of range" error when compiling
> > on Windows, I am probably stuck with .scm files instead of
> > compiled .go files.
> 
> 
> This bug is now hitting us too, we're investigating it here:
> 
> https://gitlab.com/lilypond/lilypond/-/issues/6474
> 
> Best,
> Jean
> 


For what it is worth, I do have functional 64-bit Guile 3.0.x for
Windows at the link below, but that tree is still full of missteps and
experiments and needs to made into a cleaner patchset.  And I'm still
actively hacking there because I'm trying to a Guile that, once
installed, lives in a single directory like most Windows applications
so I can make an MSIX installer. So it is a bit of a mess, but,
I'm fairly confident the integers are fixed.

https://github.com/spk121/guile/commits/wip-mingw-bleh

That tree was based on, and will eventually contribute
back to

https://git.savannah.gnu.org/cgit/guile.git/log/?h=wip-mingw

Both of these trees require that you  --enable-mini-gmp 
--enable-jit=no and --disable-lto at configure time.

Regards,
Mike Gran

Re: 64-bit Guile on Windows

[Thomas Thiriez via General Guile related discussions]

Hi Mike,

I was able to build a 64-bit libguile 3 based on the branch 
wip-mingw. It appears to be working fine so far, and I am grateful 
for your work.

The only glitch I have found is that (gettimeofday) does not 
return correct values. It should return a pair (seconds . 
microseconds). The seconds part is correct, but the microseconds 
are often greater than 1000000, and can even be negative.

I am not sure how to fix it, but I believe I have found the root 
cause of the problem. The mingw implementation of the timeval 
struct is:

struct timeval
{
	long tv_sec;
	long tv_usec;
};

But the file guile/lib/sys/time.h replaces it with that:

struct timeval
{
  time_t tv_sec;
  long int tv_usec;
};

long is 32-bit on Windows, and time_t is 64-bit. When 
scm_gettimeofday calls gettimeofday(), that function expects a 
timeval with two longs, and the seconds and microseconds end up in 
the low and high parts of tv_sec. When scm_from_long is called on 
tv_sec, the 64-bit value is truncated, and the seconds are 
correctly converted. When converting the tv_usec member, however, 
we only get what happend to be there on the stack.

I have found a test in guile/m4/sys_time_h.m4 that enables the 
timeval replacement mentioned above when sizeof(tv_sec) is smaller 
than sizeof(time_t), with this accompanying comment:

 On native Windows with a 64-bit 'time_t', 'struct timeval' is 
 defined
 (in <sys/time.h> and <winsock2.h> for mingw64, in <winsock2.h> 
 only
 for MSVC) with a tv_sec field of type 'long' (32-bit!), which is
 smaller than the 'time_t' type mandated by POSIX.
 On OpenBSD 5.1 amd64, tv_sec is 64 bits and time_t 32 bits, but
 that is good enough.

So I am left wondering how it is supposed to work if the timeval 
is replaced with members of a different size. Is scm_gettimeofday 
not calling the correct version of gettimeofday?


Another question. Are there more issues I can expect, while 
working with this wip-mingw branch? The name suggests it is not 
finished yet.

Regards,
Thomas Thiriez

Mike Gran <spk121@yahoo.com> writes:

> On Tue, Nov 22, 2022 at 01:44:47PM +0100, Jean Abou Samra wrote:
>> 
>>> > Now, it is possible that the problem is that I am trying to 
>>> > run a
>> > .go file compiled on macOS on a Windows machine, and that the 
>> > two
>> > platforms are different enough that the .go file can't be 
>> > reused.
>> > And because I get that "Value out of range" error when 
>> > compiling
>> > on Windows, I am probably stuck with .scm files instead of
>> > compiled .go files.
>> 
>> 
>> This bug is now hitting us too, we're investigating it here:
>> 
>> https://gitlab.com/lilypond/lilypond/-/issues/6474
>> 
>> Best,
>> Jean
>> 
>
>
> For what it is worth, I do have functional 64-bit Guile 3.0.x 
> for
> Windows at the link below, but that tree is still full of 
> missteps and
> experiments and needs to made into a cleaner patchset.  And I'm 
> still
> actively hacking there because I'm trying to a Guile that, once
> installed, lives in a single directory like most Windows 
> applications
> so I can make an MSIX installer. So it is a bit of a mess, but,
> I'm fairly confident the integers are fixed.
>
> https://github.com/spk121/guile/commits/wip-mingw-bleh
>
> That tree was based on, and will eventually contribute
> back to
>
> https://git.savannah.gnu.org/cgit/guile.git/log/?h=wip-mingw
>
> Both of these trees require that you  --enable-mini-gmp 
> --enable-jit=no and --disable-lto at configure time.
>
> Regards,
> Mike Gran

Re: 64-bit Guile on Windows

[Jean Abou Samra]



> Le 22 nov. 2022 à 14:22, Mike Gran <spk121@yahoo.com> a écrit :
> For what it is worth, I do have functional 64-bit Guile 3.0.x for
> Windows at the link below, but that tree is still full of missteps and
> experiments and needs to made into a cleaner patchset.


Thanks, that is good to know, but it doesn’t help LilyPond in the immediate future because we have long branched our next stable release, which is supposed to happen next month, and we require Guile 2.2 for this release.

So we need to live with the fact that fixnums are ~32 bits at most because GMP expects long, unlike what is done in the wip-mingw branch with mini-GMP.


Best,
Jean