Guile-Git, bytestructures, libgit2 and Guix 0.14.0

Guile-Git, bytestructures, libgit2 and Guix 0.14.0

[Paul Boddie]

Hello,

I thought I'd try and see if the latest Guix release might help with the 
problems I described a few days ago, but it looks like there are a few 
additional dependencies, and I am struggling to install them. I'm using Debian 
Jessie, but have also been trying with Debian Stretch.

Guile-Git seems to want scheme-bytestructures, but this seems to be something 
published on GitHub. I can seemingly compile and install it starting out as 
follows:

git clone https://github.com/TaylanUB/scheme-bytestructures.git
cd scheme-bytestructures

At this point, it seems that a modification to SOURCES in Makefile.am is 
required: bytestructures/r6/bytevectors.scm needs to be added. Otherwise, 
later attempts by Guile-Git to use the code will fail. Then...

autoreconf
./configure
make
sudo make install

This puts the library in /usr/local/lib/guile/2.0/site-ccache.

I also found that the packaged version of libgit2 on Debian Jessie doesn't 
export the git_libgit2_init symbol for some reason:

ice-9/boot-9.scm:106:20: In procedure dynamic-pointer: Symbol not found: 
git_libgit2_init

So, libgit2 needs to be installed from source as well (having removed the 
packaged version):

git clone https://github.com/libgit2/libgit2.git
cd libgit2
git checkout v0.26.0
mkdir build
cd build
cmake ..
cmake --build .
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local
sudo cmake --build . --target install

This also puts the library in /usr/local/lib/guile/2.0/site-ccache.

Trying to build Guile-Git is more awkward. Here is what I did:

git clone https://gitlab.com/guile-git/guile-git.git
cd guile-git
./bootstrap
./configure GUILE_LOAD_COMPILED_PATH=/usr/local/lib/guile/2.0/site-ccache

It is worth noting that without this extra setting, configure cannot find the 
other libraries. I did wonder whether there might be explicit --with-... 
options for configure, but there doesn't appear to be, at least from the help 
text.

I also found that git/config.scm had some apparently malformed code, possibly 
generated incorrectly, yielding this error when running make:

git/config.scm:23:0: source expression failed to match any pattern in form 
(define %libgit2 "" /usr/local "/lib/libgit2")

So, I edited that file to have the following instead:

(define %libgit2 "/usr/local/lib/libgit2")

Running make also seemed to need the additional setting:

make GUILE_LOAD_COMPILED_PATH=/usr/local/lib/guile/2.0/site-ccache

At this point, Guile-Git, scheme-bytestructures and libgit2 should be 
available and an attempt to compile the Guix tools should be possible. Here is 
what I try and do:

tar zxf guix-0.14.0.tar.gz
cd guix-0.14.0
./bootstrap
./configure GUILE_LOAD_COMPILED_PATH=/usr/local/lib/guile/2.0/site-ccache

Again, without the setting, configure fails to find things. Also, for my 
purposes, I add to gnu/packages/bootstrap.scm a definition for the mipsel 
dynamic linker name:

((string=? system "mipsel-linux") "/lib/ld.so.1")

Then I try and make everything, but the library search path is still a 
problem, so I first have to change the Makefile and remove the following from 
the make-go rule:

unset GUILE_LOAD_COMPILED_PATH ;

Then I actually try and make everything:

make GUILE_LOAD_COMPILED_PATH=/usr/local/lib/guile/2.0/site-ccache

This seems promising until I get an error like this:

  LOAD     guix/scripts/pack.scm
Backtrace:
In ice-9/r4rs.scm:
  90: 19 [dynamic-wind #<procedure 8684db0 at ice-9/eval.scm:416:20 ()> ...]
  90: 18 [dynamic-wind #<procedure 8684d68 at ice-9/eval.scm:416:20 ()> ...]
In ice-9/eval.scm:
 432: 17 [eval # #]
 432: 16 [eval # #]
 481: 15 [lp (#<fluid 16>) (#<output: void 80d2d20>)]
In ice-9/boot-9.scm:
2864: 14 [resolve-interface (guix scripts pack) #:select ...]
2789: 13 [#<procedure 8191660 at ice-9/boot-9.scm:2777:4 (name #:optional 
autoload version #:key ensure)> # ...]
3065: 12 [try-module-autoload (guix scripts pack) #f]
2401: 11 [save-module-excursion #<procedure 9e9e4e0 at 
ice-9/boot-9.scm:3066:17 ()>]
3085: 10 [#<procedure 9e9e4e0 at ice-9/boot-9.scm:3066:17 ()>]
In unknown file:
   ?: 9 [primitive-load-path "guix/scripts/pack" ...]
In ice-9/eval.scm:
 453: 8 [eval # ()]
 387: 7 [eval # ()]
 387: 6 [eval # ()]
 387: 5 [eval # ()]
 387: 4 [eval # ()]
 387: 3 [eval # ()]
 387: 2 [eval # ()]
 393: 1 [eval #<memoized gzip> ()]
In unknown file:
   ?: 0 [memoize-variable-access! #<memoized gzip> #<directory # 9f23f78>]

ERROR: In procedure memoize-variable-access!:
ERROR: gzip: unbound variable
Makefile:5295: recipe for target 'make-go' failed

I'm clearly doing something wrong or at least inconvenient with the library 
search path, but I don't really know how to troubleshoot this error, either. 
It appears to be the result of running the build-aux/compile-all.scm script on 
the different script files, but I don't see any variable called gzip in the 
guix/scripts/pack.scm script, although it must be said that I don't know 
Scheme that well, so maybe the variable is in another file and I misunderstood 
the backtrace.

Here's hoping that I've done something obviously wrong and that someone might 
be able to tell me what that is!

Paul

Re: Guile-Git, bytestructures, libgit2 and Guix 0.14.0

[Ludovic Courtès]

Hi Paul,

Paul Boddie <paul@boddie.org.uk> skribis:

> I also found that the packaged version of libgit2 on Debian Jessie doesn't 
> export the git_libgit2_init symbol for some reason:
>
> ice-9/boot-9.scm:106:20: In procedure dynamic-pointer: Symbol not found: 
> git_libgit2_init

This suggests a bug in the Debian packaging (or an old version?).

But yeah, I can sympathize with all the pain you’ve gone through to
build all this by hand.  That’s a chicken-and-egg problem: the package
manager is here to help you sort this out, but you’re precisely trying
to build it.

Pjotr Prins reported earlier that we don’t have a good bootstrapping
story here, and I come to realize what it means.  ;-)

The lesson here, I guess, is that it would be beneficial to make Guix’s
modularity more apparent, as I did in
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#121>, so that one
can build Guix piecemeal, without having to install all the dependencies
upfront.

> Again, without the setting, configure fails to find things. Also, for my 
> purposes, I add to gnu/packages/bootstrap.scm a definition for the mipsel 
> dynamic linker name:
>
> ((string=? system "mipsel-linux") "/lib/ld.so.1")
>
> Then I try and make everything, but the library search path is still a 
> problem, so I first have to change the Makefile and remove the following from 
> the make-go rule:
>
> unset GUILE_LOAD_COMPILED_PATH ;
>
> Then I actually try and make everything:
>
> make GUILE_LOAD_COMPILED_PATH=/usr/local/lib/guile/2.0/site-ccache
>
> This seems promising until I get an error like this:
>
>   LOAD     guix/scripts/pack.scm
> Backtrace:
> In ice-9/r4rs.scm:
>   90: 19 [dynamic-wind #<procedure 8684db0 at ice-9/eval.scm:416:20 ()> ...]
>   90: 18 [dynamic-wind #<procedure 8684d68 at ice-9/eval.scm:416:20 ()> ...]
> In ice-9/eval.scm:
>  432: 17 [eval # #]
>  432: 16 [eval # #]
>  481: 15 [lp (#<fluid 16>) (#<output: void 80d2d20>)]
> In ice-9/boot-9.scm:
> 2864: 14 [resolve-interface (guix scripts pack) #:select ...]
> 2789: 13 [#<procedure 8191660 at ice-9/boot-9.scm:2777:4 (name #:optional 
> autoload version #:key ensure)> # ...]
> 3065: 12 [try-module-autoload (guix scripts pack) #f]
> 2401: 11 [save-module-excursion #<procedure 9e9e4e0 at 
> ice-9/boot-9.scm:3066:17 ()>]
> 3085: 10 [#<procedure 9e9e4e0 at ice-9/boot-9.scm:3066:17 ()>]
> In unknown file:
>    ?: 9 [primitive-load-path "guix/scripts/pack" ...]
> In ice-9/eval.scm:
>  453: 8 [eval # ()]
>  387: 7 [eval # ()]
>  387: 6 [eval # ()]
>  387: 5 [eval # ()]
>  387: 4 [eval # ()]
>  387: 3 [eval # ()]
>  387: 2 [eval # ()]
>  393: 1 [eval #<memoized gzip> ()]
> In unknown file:
>    ?: 0 [memoize-variable-access! #<memoized gzip> #<directory # 9f23f78>]
>
> ERROR: In procedure memoize-variable-access!:
> ERROR: gzip: unbound variable
> Makefile:5295: recipe for target 'make-go' failed

I’m pretty sure we solved it recently but I forgot how.  Do you have the
problem on current master?

Thanks,
Ludo’.

Re: Guile-Git, bytestructures, libgit2 and Guix 0.14.0

[Paul Boddie]

On Tuesday 12. December 2017 17.29.43 Ludovic Courtès wrote:
> 
> Paul Boddie <paul@boddie.org.uk> skribis:
> > I also found that the packaged version of libgit2 on Debian Jessie
> > doesn't export the git_libgit2_init symbol for some reason:
> > 
> > ice-9/boot-9.scm:106:20: In procedure dynamic-pointer: Symbol not found:
> > git_libgit2_init
> 
> This suggests a bug in the Debian packaging (or an old version?).

Yes, it looks pretty awkward. Debian packaging for Guile seems all over the 
place. I don't see the Guile-TLS package for Stretch (guile-gnutls), for 
instance, which has kept me from just switching to that distribution version:

https://packages.debian.org/search?keywords=guile-gnutls

> But yeah, I can sympathize with all the pain you’ve gone through to
> build all this by hand.  That’s a chicken-and-egg problem: the package
> manager is here to help you sort this out, but you’re precisely trying
> to build it.

Right. If one is already on board, it's just a matter of staying on board, but 
getting on board and also doing it from scratch gives us some opportunities to 
see whether the approach still works conveniently.

> Pjotr Prins reported earlier that we don’t have a good bootstrapping
> story here, and I come to realize what it means.  ;-)

Debian has a similar experience, which is why there are efforts to improve 
bootstrapping. As you know, I was also interested in cross-bootstrapping, and 
there has also been some more interest in that within the Debian realm, 
perhaps because there have been a few new to-be-supported architectures 
recently and people have probably become more easily frustrated about the 
problems they have encountered than they might have been before.

I have some interest in building distributions from scratch, but I find the 
process rather opaque with Debian, which rather explains my interest in 
looking at Guix instead.

> The lesson here, I guess, is that it would be beneficial to make Guix’s
> modularity more apparent, as I did in
> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#121>, so that one
> can build Guix piecemeal, without having to install all the dependencies
> upfront.

Perhaps. I don't have enough experience here to weigh in on that. :-)

[...]

> > ERROR: In procedure memoize-variable-access!:
> > ERROR: gzip: unbound variable
> > Makefile:5295: recipe for target 'make-go' failed
> 
> I’m pretty sure we solved it recently but I forgot how.  Do you have the
> problem on current master?

This is with 0.14.0. I could try with the current master when I'm done trying 
to build 0.13.0 bootstrap tarballs, hopefully successfully. ;-)

Paul

Re: Guile-Git, bytestructures, libgit2 and Guix 0.14.0

[Ludovic Courtès]

Paul Boddie <paul@boddie.org.uk> skribis:

> On Tuesday 12. December 2017 17.29.43 Ludovic Courtès wrote:
>> 
>> Paul Boddie <paul@boddie.org.uk> skribis:
>> > I also found that the packaged version of libgit2 on Debian Jessie
>> > doesn't export the git_libgit2_init symbol for some reason:
>> > 
>> > ice-9/boot-9.scm:106:20: In procedure dynamic-pointer: Symbol not found:
>> > git_libgit2_init
>> 
>> This suggests a bug in the Debian packaging (or an old version?).
>
> Yes, it looks pretty awkward. Debian packaging for Guile seems all over the 
> place. I don't see the Guile-TLS package for Stretch (guile-gnutls), for 
> instance, which has kept me from just switching to that distribution version:
>
> https://packages.debian.org/search?keywords=guile-gnutls

IIRC they removed it because of a bug that didn’t get fix in time.

>> But yeah, I can sympathize with all the pain you’ve gone through to
>> build all this by hand.  That’s a chicken-and-egg problem: the package
>> manager is here to help you sort this out, but you’re precisely trying
>> to build it.
>
> Right. If one is already on board, it's just a matter of staying on board, but 
> getting on board and also doing it from scratch gives us some opportunities to 
> see whether the approach still works conveniently.

Yeah.

>> Pjotr Prins reported earlier that we don’t have a good bootstrapping
>> story here, and I come to realize what it means.  ;-)
>
> Debian has a similar experience, which is why there are efforts to improve 
> bootstrapping. As you know, I was also interested in cross-bootstrapping, and 
> there has also been some more interest in that within the Debian realm, 
> perhaps because there have been a few new to-be-supported architectures 
> recently and people have probably become more easily frustrated about the 
> problems they have encountered than they might have been before.
>
> I have some interest in building distributions from scratch, but I find the 
> process rather opaque with Debian, which rather explains my interest in 
> looking at Guix instead.

Nice.  Though to me there are really two sorts of bootstrapping issue:

  1. Whole-distro bootstrapping as discussed at
     <https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html>,
     and which is what Mes, stage0, and friends are attempting to solve
     (see <http://bootstrappable.org/projects.html>).

     I also think that the purely functional approach (Guix and Nix)
     makes the bootstrapping path much clearer than with the imperative
     approach (Debian, etc.).

  2. Building Guix itself on a system that doesn’t have it yet, which is
     what we’re discussing here.

Clearly we could solve #1 without solving #2 and vice versa.  These are
fairly orthogonal issues.

>> > ERROR: In procedure memoize-variable-access!:
>> > ERROR: gzip: unbound variable
>> > Makefile:5295: recipe for target 'make-go' failed
>> 
>> I’m pretty sure we solved it recently but I forgot how.  Do you have the
>> problem on current master?
>
> This is with 0.14.0. I could try with the current master when I'm done trying 
> to build 0.13.0 bootstrap tarballs, hopefully successfully. ;-)

OK, let us know how it goes!

Ludo’.

Re: Guile-Git, bytestructures, libgit2 and Guix 0.14.0

[Paul Boddie]

(I hope it's OK to send this to the list given that one of the recipients was 
- presumably by mistake - the "request" address for the list, not the list 
address.)

On Wednesday 13. December 2017 01.35.57 rennes@openmailbox.org wrote:
> Hello,
> 
> >> I?m pretty sure we solved it recently but I forgot how.  Do you have the
> >> problem on current master?
> > 
> > This is with 0.14.0. I could try with the current master when I'm done
> > trying to build 0.13.0 bootstrap tarballs, hopefully successfully. ;-)

I'm still having my machine work through this at the moment.

> I followed your tutorial in Debian/Hurd.
> What worked for me is installing the packages manually:
> 
>   * gnutls-3.6.1
>   * scheme-bytestructures
>   * libgit2
>   * guile-git
>   * guile-2.2.3
>   * guix (master branch)

I'll take another look at this in a while, but having the above list of 
ingredients is very helpful, thanks!

> The directory '/usr/local/lib/guile/2.2/site-ccache' should look like this:
>   * bytestructures
>   * git
>   * git.go
> 
> For each 'configure' and 'make' use 'GUILE_LOAD_COMPILED_PATH
> =/usr/local/lib/guile/2.2/site-ccache'

Not being a Guile user, I fortunately managed to figure out the path variable 
and to get the site-ccache (which I guess is like Python's site-packages but 
with only compiled objects) to contain those things. So this is a nice 
reassurance that I wasn't doing something completely stupid. :-)

Paul

P.S. Using Debian GNU/Hurd (or a Guix variant) is something else I would like 
to try at some point, where being stuck on i386/i686/ia32 with old hardware 
might even be helpful. Indeed, doing microkernel-related stuff is also on the 
long list of tasks I find myself juggling.