bug#63331: Guile-GnuTLS/Git circular dependency

bug#63331: Guile-GnuTLS/Git circular dependency

[Ludovic Courtès]

Hi,

‘git-download’ needs to depend on guile-gnutls to implement its fallback
mechanism (downloading from mirrors or from SWH over HTTPS).  Commit
c625e5b64d0a6cb7ffbf2ef971d4c990b1f5c5c1 restored this.  However, it
also introduced a circular dependency: the origin of guile-gnutls relies
on 'git-download', which would now depend on guile-gnutls.  Thus, I
reverted it right away.

We need to solve that.  For now, the only fix I can think of is having
‘guile-gnutls’ built from a “make dist”-provided tarballs.  Apparently
we can add assets at <https://gitlab.com/gnutls/guile/-/tags>; would you
like to upload a tarball and accompanying signature, Simon?

Unfortunately, that means doing away with all the packaging work by
Vivien, in particular proper bootstrapping with Gnulib.

The longer-term solution is to add a “builtin:git-download” derivation
builder, just like we have “builtin:download”.  The implementation
should be relatively easy, but we’ll have to be able to deal with
daemons that lack this builtin possibly for several years.

Thoughts?

Ludo’.

bug#63331: Guile-GnuTLS/Git circular dependency

[Vivien Kraus via Bug reports for GNU Guix]

Hi!

Le samedi 06 mai 2023 à 19:20 +0200, Ludovic Courtès a écrit :
> We need to solve that.  For now, the only fix I can think of is
> having
> ‘guile-gnutls’ built from a “make dist”-provided tarballs. 
> Apparently
> we can add assets at <https://gitlab.com/gnutls/guile/-/tags>; would
> you
> like to upload a tarball and accompanying signature, Simon?

If the problem is with git-download, couldn’t we just use a "git-
archive"-provided tarball, and keep the bootstrapping process? Or are
there further dependencies with the autotools that require a dist-
provided tarball?

Vivien 

bug#63331: Guile-GnuTLS/Git circular dependency

[Ludovic Courtès]

Hi,

Vivien Kraus <vivien@planete-kraus.eu> skribis:

> Le samedi 06 mai 2023 à 19:20 +0200, Ludovic Courtès a écrit :
>> We need to solve that.  For now, the only fix I can think of is
>> having
>> ‘guile-gnutls’ built from a “make dist”-provided tarballs. 
>> Apparently
>> we can add assets at <https://gitlab.com/gnutls/guile/-/tags>; would
>> you
>> like to upload a tarball and accompanying signature, Simon?
>
> If the problem is with git-download, couldn’t we just use a "git-
> archive"-provided tarball, and keep the bootstrapping process? Or are
> there further dependencies with the autotools that require a dist-
> provided tarball?

The ‘gnulib’ package also uses ‘git-fetch’, so we’d at least need to get
rid of it.

More importantly, tarballs generated by GitLab & co. are usually built
on the fly and change over time (details about the tarball headers
etc. may change).  So we cannot depend them.

A tarball produced with ‘make dist’ will have everything we need,
including Gnulib.

Ludo’.

bug#63331: Guile-GnuTLS/Git circular dependency

[Simon Tournier]

Hi,

On Sat, 06 May 2023 at 22:17, Ludovic Courtès <ludo@gnu.org> wrote:

> More importantly, tarballs generated by GitLab & co. are usually built
> on the fly and change over time (details about the tarball headers
> etc. may change).  So we cannot depend them.

We could just store this tarball, no?

Well, I am somehow surprise… it appears to me better to find a way for
fixing this circular dependency without introducing a hard dependency on
Git pushed as a daemon dependency.

Cheers,
simon

bug#63331: Guile-GnuTLS/Git circular dependency

[Simon Josefsson via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> We need to solve that.  For now, the only fix I can think of is having
> ‘guile-gnutls’ built from a “make dist”-provided tarballs.  Apparently
> we can add assets at <https://gitlab.com/gnutls/guile/-/tags>; would you
> like to upload a tarball and accompanying signature, Simon?

The tarballs I created are available here:
https://gitlab.com/gnutls/guile/-/releases

Is a new releases necessary, or does the 3.7.11 release work?  I can do
a release tonight or tomorrow, but I'm also happy to help someone else
to do the release -- see README-release for the process, skip 'make
upload' if you don't have ftp.gnu.org gnutls credentials.

I'm not sure I exactly what the real problem is here -- but would one
solution be to publish a source-only tarball with the source code files
from git, together with a signature?  That would include any gnulib
files, but not autogenerated ./configure etc.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

bug#63331: Guile-GnuTLS/Git circular dependency

[Simon Josefsson via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> We need to solve that.  For now, the only fix I can think of is having
> ‘guile-gnutls’ built from a “make dist”-provided tarballs.  Apparently
> we can add assets at <https://gitlab.com/gnutls/guile/-/tags>; would you
> like to upload a tarball and accompanying signature, Simon?

I published a release of gnutls-guile 3.7.12, this time built on my Guix
development machine to test that the release machinery (README-release)
works under Guix as well; the only "interesting" dependency was ncftp
but you had that packaged and it worked fine.

https://gitlab.com/gnutls/guile/-/releases/v3.7.12

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

bug#63331: Guile-GnuTLS/Git circular dependency

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3330 bytes --]


Simon Josefsson via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> [[PGP Signed Part:Undecided]]
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> We need to solve that.  For now, the only fix I can think of is having
>> ‘guile-gnutls’ built from a “make dist”-provided tarballs.  Apparently
>> we can add assets at <https://gitlab.com/gnutls/guile/-/tags>; would you
>> like to upload a tarball and accompanying signature, Simon?
>
> I published a release of gnutls-guile 3.7.12, this time built on my Guix
> development machine to test that the release machinery (README-release)
> works under Guix as well; the only "interesting" dependency was ncftp
> but you had that packaged and it worked fine.
>
> https://gitlab.com/gnutls/guile/-/releases/v3.7.12

Thanks so much for this Simon.

I've had a go at updating the Guix guile-gnutls package and sent an
initial patch to https://issues.guix.gnu.org/63388 .

It seems to build for me, but I'm having problems cross building. There
were warnings before about protocol/ssl3 being undefined, but now this
seems to result in an error when building extra.scm:


  GUILEC   modules/gnutls.go
gnutls.scm:608:23: warning: possibly unbound variable `protocol/ssl3'
gnutls.scm:609:25: warning: possibly unbound variable `protocol/tls1-0'
gnutls.scm:610:25: warning: possibly unbound variable `protocol/tls1-1'
  GUILEC   modules/gnutls/extra.go
Backtrace:
In ice-9/psyntax.scm:
  1229:36 19 (expand-top-sequence (#<syntax:extra.scm:21:0 (#<synt?>) ?)
  1221:19 18 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
   259:10 17 (parse _ (("placeholder" placeholder)) (()) _ c&e (# #) #)
In ice-9/eval.scm:
   293:34 16 (_ #<module (#{ g106}#) 7ffff786d500>)
In ice-9/boot-9.scm:
   3411:4 15 (define-module* _ #:filename _ #:pure _ #:version _ # _ ?)
  2595:24 14 (call-with-deferred-observers #<procedure 7ffff778ac30 ?>)
  3424:24 13 (_)
   222:17 12 (map1 (((gnutls))))
  3327:17 11 (resolve-interface (gnutls) #:select _ #:hide _ #:prefix ?)
In ice-9/threads.scm:
    390:8 10 (_ _)
In ice-9/boot-9.scm:
  3253:13  9 (_)
In ice-9/threads.scm:
    390:8  8 (_ _)
In ice-9/boot-9.scm:
  3544:20  7 (_)
   2836:4  6 (save-module-excursion #<procedure 7ffff78660f0 at ice-?>)
  3564:26  5 (_)
In unknown file:
           4 (primitive-load-path "gnutls" #<procedure 7ffff27384a0 ?>)
In ice-9/eval.scm:
   626:19  3 (_ #<directory (gnutls) 7ffff786d320>)
   223:20  2 (proc #<directory (gnutls) 7ffff786d320>)
In unknown file:
           1 (%resolve-variable (7 . protocol/ssl3) #<directory (gnu?>)
In ice-9/boot-9.scm:
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Unbound variable: protocol/ssl3
make[3]: *** [Makefile:1295: modules/gnutls/extra.go] Error 1
make[3]: Leaving directory '/tmp/guix-build-guile-gnutls-3.7.12.drv-0/guile-gnutls-3.7.12/guile'
make[2]: *** [Makefile:754: all-recursive] Error 1
make[2]: Leaving directory '/tmp/guix-build-guile-gnutls-3.7.12.drv-0/guile-gnutls-3.7.12/guile'
make[1]: *** [Makefile:471: all-recursive] Error 1
make[1]: Leaving directory '/tmp/guix-build-guile-gnutls-3.7.12.drv-0/guile-gnutls-3.7.12'
make: *** [Makefile:403: all] Error 2
error: in phase 'build': uncaught exception:

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#63331: Guile-GnuTLS/Git circular dependency

[Simon Josefsson via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 430 bytes --]

> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> Unbound variable: protocol/ssl3

Maybe ssl3 is disabled (as it probably should be) in guix's gnutls?

While I built the package on a Guix system using system GnuTLS, I
didn't build it through Guix's packaging, so maybe there is some
difference?

A GitLab CI/CD build check on Guix would be nice, does anyone publish
docker images for a Guix system?

/Simon

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

bug#63331: Guile-GnuTLS/Git circular dependency

[Vivien Kraus via Bug reports for GNU Guix]

Le mardi 09 mai 2023 à 14:23 +0200, Simon Josefsson a écrit :
> A GitLab CI/CD build check on Guix would be nice, does anyone publish
> docker images for a Guix system?

The guix builder uses linux tools to provide an isolated build
environment. It is possible to run the guix build daemon without this
protection, so as to run it within a docker container, but build
scripts may behave incorrectly if they run outside of the sandbox. They
could see libraries that they should not be able to see and by that
configure incorrectly, or install things where they should not. Guix
packagers do not usually care if a build script writes files outside of
its correct store directory, because of the isolation provided by the
daemon. Such problems are thus hard to detect, and broken packages
could be anywhere. This is mostly a hypothetical issue, but opam (for
ocaml) warns about build scripts doing unpredictable things:

https://opam.ocaml.org/doc/FAQ.html#Why-does-opam-require-bwrap

Aside from that, guix is painfully slow in a container, and uses a lot
of disk space.

Vivien

bug#63331: Guile-GnuTLS/Git circular dependency

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> It seems to build for me, but I'm having problems cross building. There
> were warnings before about protocol/ssl3 being undefined, but now this
> seems to result in an error when building extra.scm:
>
>
>   GUILEC   modules/gnutls.go
> gnutls.scm:608:23: warning: possibly unbound variable `protocol/ssl3'
> gnutls.scm:609:25: warning: possibly unbound variable `protocol/tls1-0'
> gnutls.scm:610:25: warning: possibly unbound variable `protocol/tls1-1'
>   GUILEC   modules/gnutls/extra.go

[...]

> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> Unbound variable: protocol/ssl3
> make[3]: *** [Makefile:1295: modules/gnutls/extra.go] Error 1

Is it a regression or did we already have that problem?

That comes from this bit in (gnutls):

  ;; Renaming.
  (define protocol/ssl-3 protocol/ssl3)
  (define protocol/tls-1.0 protocol/tls1-0)
  (define protocol/tls-1.1 protocol/tls1-1)

When cross-compiling, the .so cannot be loaded (understandably; see also
GNUTLS_GUILE_CROSS_COMPILING) so ‘protocol/ssl3’ above is undefined.
The problem is that when compiling (gnutls extra), we end up loading
(gnutls) and thus evaluating the lines above, which fail.

In Guile-Avahi I worked around it like so:

  (define protocol/unspecified
    (and (defined? 'protocol/unspec) protocol/unspec))

I guess we could do that as well here.

HTH,
Ludo’.

bug#63331: Guile-GnuTLS/Git circular dependency

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1972 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> It seems to build for me, but I'm having problems cross building. There
>> were warnings before about protocol/ssl3 being undefined, but now this
>> seems to result in an error when building extra.scm:
>>
>>
>>   GUILEC   modules/gnutls.go
>> gnutls.scm:608:23: warning: possibly unbound variable `protocol/ssl3'
>> gnutls.scm:609:25: warning: possibly unbound variable `protocol/tls1-0'
>> gnutls.scm:610:25: warning: possibly unbound variable `protocol/tls1-1'
>>   GUILEC   modules/gnutls/extra.go
>
> [...]
>
>> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>> Unbound variable: protocol/ssl3
>> make[3]: *** [Makefile:1295: modules/gnutls/extra.go] Error 1
>
> Is it a regression or did we already have that problem?

A regression I think, the data service doesn't have recent data, but it
does know about builds that worked:

  https://data.guix.gnu.org/repository/1/branch/master/package/guile-gnutls/output-history?output=out&system=x86_64-linux&target=riscv64-linux-gnu

> That comes from this bit in (gnutls):
>
>   ;; Renaming.
>   (define protocol/ssl-3 protocol/ssl3)
>   (define protocol/tls-1.0 protocol/tls1-0)
>   (define protocol/tls-1.1 protocol/tls1-1)
>
> When cross-compiling, the .so cannot be loaded (understandably; see also
> GNUTLS_GUILE_CROSS_COMPILING) so ‘protocol/ssl3’ above is undefined.
> The problem is that when compiling (gnutls extra), we end up loading
> (gnutls) and thus evaluating the lines above, which fail.
>
> In Guile-Avahi I worked around it like so:
>
>   (define protocol/unspecified
>     (and (defined? 'protocol/unspec) protocol/unspec))
>
> I guess we could do that as well

That sort of makes sense, although I don't know why this wasn't failing
in the same way in the past. Build logs are available though, so maybe
this makes sense to someone.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#63331: Guile-GnuTLS/Git circular dependency

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> That sort of makes sense, although I don't know why this wasn't failing
> in the same way in the past. Build logs are available though, so maybe
> this makes sense to someone.

Turns out that’s because ‘gnutls-cross.patch’ was inadvertently
dismissed in 5e1e67442188ccca8db8c1dd092efbc6fc2c33dc.

I’ll re-apply it (probably upstream as well) unless someone has already
come up with a different solution (Josselin was looking into it).

Thanks,
Ludo’.

bug#63331: Guile-GnuTLS/Git circular dependency

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> The longer-term solution is to add a “builtin:git-download” derivation
> builder, just like we have “builtin:download”.  The implementation
> should be relatively easy, but we’ll have to be able to deal with
> daemons that lack this builtin possibly for several years.

Patch available!

  https://issues.guix.gnu.org/65866

Ludo’.

[bug#65866] bug#63331: Guile-GnuTLS/Git circular dependency and built-in git checkouts

[Vivien Kraus via Guix-patches via]

Hello!

Le lundi 11 septembre 2023 à 16:36 +0200, Ludovic Courtès a écrit :
> Eventually, when users are all running recent versions of
> ‘guix-daemon’ with support for “builtin:git-download” (2–4
> years from now?), we’ll be able to use “builtin:git-download”
> unconditionally and thus be sure there are no risks of
> derivation cycles.

Do foreign distros need to update their guix package as well? If that
is the case, the provided time frame might be optimistic.

> Note that the patch series adds a hard dependency on Git.
> This is because the existing ‘git-fetch’ code depends on Git

I applaud the switch to the regular git program from libgit2, as I
would then be able to pull from my cgit "dumb" server instead of having
to maintain a mirror.

Vivien

[bug#65866] [PATCH 0/8] Add built-in builder for Git checkouts

[Ludovic Courtès]

Hi,

Vivien Kraus <vivien@planete-kraus.eu> skribis:

> Le lundi 11 septembre 2023 à 16:36 +0200, Ludovic Courtès a écrit :
>> Eventually, when users are all running recent versions of
>> ‘guix-daemon’ with support for “builtin:git-download” (2–4
>> years from now?), we’ll be able to use “builtin:git-download”
>> unconditionally and thus be sure there are no risks of
>> derivation cycles.
>
> Do foreign distros need to update their guix package as well? If that
> is the case, the provided time frame might be optimistic.

At some point, we can change clients to print a warning saying that
their daemon is outdated if it lacks “builtin:git-download”.  That
should help speed things up.

>> Note that the patch series adds a hard dependency on Git.
>> This is because the existing ‘git-fetch’ code depends on Git
>
> I applaud the switch to the regular git program from libgit2, as I
> would then be able to pull from my cgit "dumb" server instead of having
> to maintain a mirror.

Nothing changes here: ‘git-fetch’ already uses Git.

Ludo’.

bug#63331: Guile-GnuTLS/Git circular dependency

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> The longer-term solution is to add a “builtin:git-download” derivation
>> builder, just like we have “builtin:download”.  The implementation
>> should be relatively easy, but we’ll have to be able to deal with
>> daemons that lack this builtin possibly for several years.
>
> Patch available!
>
>   https://issues.guix.gnu.org/65866

This was applied in the meantime.  Closing!