what to do about gnulib

what to do about gnulib

[Andy Wingo]

Hi,

I am not sure what to do about gnulib.

To recap, gnulib provides compile-time shims for "foreign" or
out-of-date target systems that allows Guile to program to a single
POSIX + GNU API and not worry too much about what the system actually
offers.  It is never installed; instead it includes the `gnulib-tool`
utility to update the in-tree copy of gnulib, for a selected set of
modules.

So far, so good.  However I was unable to update gnulib before the most
recent release.  The reason is essentially the problem described in this
issue:

  https://sourceware.org/bugzilla/show_bug.cgi?id=30051

To wit, running "autoreconf -vif" invokes the "autopoint" tool supplied
by installed gettext, which copies over .m4 files from installed
gettext, but these files are older than the ones that are already
"vendored" in-tree by gnulib.  Other parts of gnulib depend on the newer
gnulib-supplied macros which were stompled over; autoconf thus fails to
run.

So, I can't update gnulib right now.  Not a great situation.  The
options, as I understand them:

  1. Some projects (e.g. poke) seem to import the whole of gnulib as a
     git submodule, and then run "gnulib-tool --update" from that
     submodule as part of their bootstrap.  In this way the stompled
     files are restored.  However I do not want git submodules in Guile;
     they add additional steps for every time you change to a different
     HEAD, and I know from experience that I can't rely on myself to
     perform them all, much less any user with a bug report.  Do not
     want.

  2. Same as (1), but subtree merge.  This is what we do with lightening
     and what I would like to do with Whippet, if that project works
     out.  But gnulib is big; a checkout is 200 MB of data + 130 MB git
     repo.

  ** Note, for (1) and (2), if you wanted to preserve the ability to
     bootstrap from a tarball, you'd have to include gnulib in the
     tarball.  Of course you could argue that if you are not gnulib-tool
     --update'ing, are you really bootstrapping?  I don't know the
     answer.  ***

  3. Fix autopoint to not overwrite newer m4 files with its copies.  I
     don't know?

  4. Fix installed gettext to not define gl_ macros ?  That would make
     it so that it won't stomple on a local gnulib copy.  I don't know
     though.

  5. Update to whatever version of gnulib is the latest without this
     change?

  6. Something else?  Stop using gnulib?

I am not a gnulib wizard and would appreciate any thoughts :)  Cc'ing
Bruno as he is the expert.  Really I would like to avoid gnulib-tool
--update in Guile's bootstrap and just keep things like it is.  Of
course I could be being unreasonable.

Cheers,

Andy

RE: what to do about gnulib

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

>To wit, running "autoreconf -vif" invokes the "autopoint" tool supplied
by installed gettext, which copies over .m4 files from installed
gettext, but [problems]
> [...]
 > 6. Something else?  Stop using gnulib?

Maybe make a shell no-op shell script in “insert-directory-name-here/autopoint” and add “insert-directory-name-here” to path, so no replacing .m4 happens.

Best regards,
Maxime Devos


[-- Attachment #2: Type: text/html, Size: 3804 bytes --]

Re: what to do about gnulib

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Maxime Devos <maximedevos@telenet.be> writes:

>>To wit, running "autoreconf -vif" invokes the "autopoint" tool supplied
> by installed gettext, which copies over .m4 files from installed
> gettext, but [problems]
>> [...]
>  > 6. Something else?  Stop using gnulib?
>
> Maybe make a shell no-op shell script in “insert-directory-name-here/autopoint”
> and add “insert-directory-name-here” to path, so no replacing .m4 happens.

This can be simpler; I used the following to update Gnulib in the Guile
sources:

--8<---------------cut here---------------start------------->8---
  autopoint --force
  libtoolize --copy --install --force
  "${GNULIB_SRCDIR}${GNULIB_SRCDIR+/}"gnulib-tool --update
  AUTOPOINT=true LIBTOOLIZE=true autoreconf -fi
--8<---------------cut here---------------end--------------->8---
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

Re: what to do about gnulib

[Bruno Haible]

[-- Attachment #1: Type: text/plain, Size: 5050 bytes --]

Hi Andy,

Thanks for CCing me. You could equally well have CCed bug-gnulib; that
works too.

> To recap, gnulib provides compile-time shims for "foreign" or
> out-of-date target systems that allows Guile to program to a single
> POSIX + GNU API and not worry too much about what the system actually
> offers.

Gnulib also has other functionality than portability. There is also
  - build / maintenance infrastructure (which Guile makes use of),
  - other functionality [1].

> However I was unable to update gnulib before the most
> recent release.  The reason is essentially the problem described in this
> issue:
> 
>   https://sourceware.org/bugzilla/show_bug.cgi?id=30051
> 
> To wit, running "autoreconf -vif" invokes the "autopoint" tool supplied
> by installed gettext, which copies over .m4 files from installed
> gettext, but these files are older than the ones that are already
> "vendored" in-tree by gnulib.  Other parts of gnulib depend on the newer
> gnulib-supplied macros which were stompled over; autoconf thus fails to
> run.

I can reproduce a problem by
  0. using a git checkout of guile ('main' branch, not 'master' branch),
  1. using a GNU gettext version 0.22.5, 0.21, 0.20.2, or 0.19.8.1,
  2. removing the gnulib-local/m4/clock_time.m4.diff file, which no longer
     applies,
  3. running
     $ $GNULIB_SRCDIR/gnulib-tool --update
  4. running
     $ ./autogen.sh
     (which is what the HACKING file recommends).

It fails with

configure:12833: error: possibly undefined macro: gl_PTHREADLIB
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure:12943: error: possibly undefined macro: gl_WEAK_SYMBOLS
configure:27584: error: possibly undefined macro: gl_TYPE_WINT_T_PREREQ
autoreconf: error: /usr/bin/autoconf failed with exit status: 1

Following your analysis, I locally apply the workaround from the Gnulib
documentation section "3.11 Caveat: gettextize and autopoint users",
redo steps 3 and 4, and the './configure' succeeds, and 'make' succeeds
up to the point where it encounters a compilation error:

  CC       libguile_3.0_la-jit.lo
jit.c:373:30: error: macro "unreachable" passed 1 arguments, but takes just 0
  373 | unreachable (scm_jit_state *j)
      |                              ^

Explanation: 'unreachable' is, per ISO C 23 § 7.21.1, a macro that takes
0 arguments. The newer Gnulib makes it available even if the compiler does
not yet make it available.

Proposed patch attached.

>   1. Some projects (e.g. poke) seem to import the whole of gnulib as a
>      git submodule, and then run "gnulib-tool --update" from that
>      submodule as part of their bootstrap.  In this way the stompled
>      files are restored.  However I do not want git submodules in Guile;
>      they add additional steps for every time you change to a different
>      HEAD, and I know from experience that I can't rely on myself to
>      perform them all, much less any user with a bug report.  Do not
>      want.

The way Guile handles versioning of Gnulib-imported files is fine. It
corresponds to the "middle road" approach, described in the Gnulib manual [2],
section "3.13 Integration with Version Control Systems". It is thus
supported by Gnulib.

>   ** Note, for (1) and (2), if you wanted to preserve the ability to
>      bootstrap from a tarball, you'd have to include gnulib in the
>      tarball.  Of course you could argue that if you are not gnulib-tool
>      --update'ing, are you really bootstrapping?  I don't know the
>      answer.  ***

Simon Josefsson is working with the Debian people on this approach. [3]

>   3. Fix autopoint to not overwrite newer m4 files with its copies.  I
>      don't know?

This is done through the attached patch.

>   4. Fix installed gettext to not define gl_ macros ?  That would make
>      it so that it won't stomple on a local gnulib copy.  I don't know
>      though.
> 
>   5. Update to whatever version of gnulib is the latest without this
>      change?
> 
>   6. Something else?

Given that the attached patch works, you don't need to consider these
other approaches.

Note, though, that the imported po/ infrastructure will still be from 2010,
due to this line in configure.ac:
  AM_GNU_GETTEXT_VERSION([0.18.1])
It would be reasonable to bump this version specification to
  AM_GNU_GETTEXT_VERSION([0.19.8.1])
(from 2016) or
  AM_GNU_GETTEXT_VERSION([0.20.2])
(from 2020) or
  AM_GNU_GETTEXT_VERSION([0.21.1])
(from 2022).

Note also that when upgrading to a newer Gnulib, you now have another choice
than Gnulib's 'master' branch: the 'stable-202401' branch. See the Gnulib
documentation [2], section "1.6.1 Stable Branches".

      Bruno

[1] https://lists.gnu.org/archive/html/bug-standards/2024-05/msg00013.html
[2] https://www.gnu.org/software/gnulib/manual/html_node/index.html
[3] https://lists.gnu.org/archive/html/bug-gnulib/2024-05/msg00124.html

[-- Attachment #2: 0001-Fix-conflict-between-Gnulib-and-gettext-s-autopoint-.patch --]
[-- Type: text/x-patch, Size: 933 bytes --]

From f67b9b9953827dd23b1f71855d057323530bed98 Mon Sep 17 00:00:00 2001
From: Bruno Haible <bruno@clisp.org>
Date: Mon, 24 Jun 2024 18:53:06 +0200
Subject: [PATCH] Fix conflict between Gnulib and gettext's autopoint program.

* autogen.sh: Arrange to not invoke autopoint.
---
 autogen.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/autogen.sh b/autogen.sh
index 2e39fb5d8..f9ece0f24 100755
--- a/autogen.sh
+++ b/autogen.sh
@@ -33,6 +33,11 @@ echo ""
 ######################################################################
 ### update infrastructure
 
+# Avoid running GNU gettext's autopoint program at this stage, because
+# it would overwrite some *.m4 files from Gnulib. See the Gnulib manual
+# <https://www.gnu.org/software/gnulib/manual/html_node/index.html>
+# section "Caveat: gettextize and autopoint users".
+AUTOPOINT=true \
 autoreconf -i --force --verbose
 
 echo "Now run configure and make."
-- 
2.34.1

Re: what to do about gnulib

[Andy Wingo]

Hi Bruno!

Thanks for your kind answer and tips :)

On Mon 24 Jun 2024 19:09, Bruno Haible <bruno@clisp.org> writes:

> I can reproduce a problem by
>   0. using a git checkout of guile ('main' branch, not 'master' branch),
>   1. using a GNU gettext version 0.22.5, 0.21, 0.20.2, or 0.19.8.1,
>   2. removing the gnulib-local/m4/clock_time.m4.diff file, which no longer
>      applies,
>   3. running
>      $ $GNULIB_SRCDIR/gnulib-tool --update
>   4. running
>      $ ./autogen.sh
>      (which is what the HACKING file recommends).
>
> It fails with
>
> configure:12833: error: possibly undefined macro: gl_PTHREADLIB
>       If this token and others are legitimate, please use m4_pattern_allow.
>       See the Autoconf documentation.
> configure:12943: error: possibly undefined macro: gl_WEAK_SYMBOLS
> configure:27584: error: possibly undefined macro: gl_TYPE_WINT_T_PREREQ
> autoreconf: error: /usr/bin/autoconf failed with exit status: 1
>
> Following your analysis, I locally apply the workaround from the Gnulib
> documentation section "3.11 Caveat: gettextize and autopoint users",

Thank you!  It is embarrassing but I did not know this manual existed.
I will use this workaround.

Question, do you think it would be reasonable for autopoint to avoid
overwriting newer files?  It seems like the sort of problem we could
avoid, but who knows.

>   373 | unreachable (scm_jit_state *j)

Ah interesting, will fix.

> The way Guile handles versioning of Gnulib-imported files is fine.

That's great to hear.  In that case, no change planned to how we do
things.

> Note, though, that the imported po/ infrastructure will still be from 2010,
> due to this line in configure.ac:
>   AM_GNU_GETTEXT_VERSION([0.18.1])
> It would be reasonable to bump this version specification to
>   AM_GNU_GETTEXT_VERSION([0.19.8.1])
> (from 2016) or
>   AM_GNU_GETTEXT_VERSION([0.20.2])
> (from 2020) or
>   AM_GNU_GETTEXT_VERSION([0.21.1])
> (from 2022).
>
> Note also that when upgrading to a newer Gnulib, you now have another choice
> than Gnulib's 'master' branch: the 'stable-202401' branch. See the Gnulib
> documentation [2], section "1.6.1 Stable Branches".

Thanks!!

Best regards,

Andy

Re: what to do about gnulib

[Bruno Haible]

Hi Andy,

> Question, do you think it would be reasonable for autopoint to avoid
> overwriting newer files?  It seems like the sort of problem we could
> avoid, but who knows.

Such a solution would be
  - partly ineffective, because Gnulib has only recently started to use
    '# serial ...' annotations (with the correct syntax) in its *.m4 files,
  - hairy, because it could lead to one file being copied and another file
    being not copied. In other words, it's a heuristic, and heuristics are
    buggy.

Although it's not perfect, it's probably better to let the package maintainer
decide whether to run 'autopoint' and at which moment.

> > The way Guile handles versioning of Gnulib-imported files is fine.
> 
> That's great to hear.  In that case, no change planned to how we do
> things.

Well, through the new fast gnulib-tool [1] there is now the opportunity to
remove the copied gnulib files from your package's repository [2].
If a gnulib-tool invocation is added to autogen.sh, it no longer takes more
time to execute than the rest of autogen.sh.

Bruno

[1] https://lists.gnu.org/archive/html/info-gnu/2024-04/msg00003.html
[2] https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00320.html