Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Eli Zaretskii]

Paul,

Is there any chance to enhance admin/merge-gnulib so that if the
import renames some Gnulib modules, the renaming will be reflected in
nt/gnulib-cfg.mk?  Currently, such renaming needs to be caught in time
by someone who knows about nt/gnulib-cfg.mk, and then the
corresponding renames manually repeated there.  This is error-prone
and relies on approximately one person we have on board who knows
about these subtleties to pay attention.  Automation would go a long
way towards eliminating many issues due to such renaming.

There's also a similar problem with adding new modules.  Here, we need
a judgment call by an Emacs developer who is familiar with the w32
code, so I don't see how this could be automated.  However, sending a
heads-up email to this list by admin/merge-gnulib, if that is
feasible, could be a good-enough solution.

Can these enhancements be done, please?

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> There's also a similar problem with adding new modules.  Here, we need
> a judgment call by an Emacs developer who is familiar with the w32
> code, so I don't see how this could be automated.  However, sending a
> heads-up email to this list by admin/merge-gnulib, if that is
> feasible, could be a good-enough solution.

FWIW, the local email configuration is borken on some machines I work
on, so I'd find a "warning: new modules added, notify emacs-devel for
possible w32 adaption" message more useful than something that blindly
pipes things to mail(1).

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

On 2025-01-02 09:57, Stefan Kangas wrote:
> I'd find a "warning: new modules added, notify emacs-devel for
> possible w32 adaption" message more useful than something that blindly
> pipes things to mail(1).

Thanks for the suggestion; I installed the attached.

I don't see a nontrivial way to update nt/gnulib-cfg.mk automatically, 
even just for module renames as Eli suggested, as there are too many 
possibilities for screwups.

The "right" way to fix this would be to adjust Emacs and Gnulib so that 
builds would work the same way on MS-Windows that they do everywhere 
else (i.e., nt/gnulib-cfg.mk would become unnecessary). However, as I 
understand it this would be nontrivial and anyway it would require 
MS-Windows expertise that I lack.

In the meantime, module renames are rare enough that doing things by 
hand should take less work overall. Besides, the main problem here is 
changes to modules and/or adding modules, which I don't think 
admin/merge-gnulib can handle by itself.

[-- Attachment #2: 0001-admin-merge-gnulib-now-warns-if-module-lists-change.patch --]
[-- Type: text/x-patch, Size: 3317 bytes --]

From f65520cad46273967da2343feeb2201363c190e0 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 2 Jan 2025 11:29:26 -0800
Subject: [PATCH] admin/merge-gnulib now warns if module lists change
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

From suggestions by Eli Zaretski and Stefan Kangas in:
https://lists.gnu.org/r/emacs-devel/2025-01/msg00036.html
https://lists.gnu.org/r/emacs-devel/2025-01/msg00037.html
* admin/merge-gnulib: For style consistency,
prefer ‘[ EXPR ]’ to ‘test EXPR’ and prefer -e to -f.
Fix bug that forgot to clean lib directory if it’s ./lib.
Check for lib/gnulib.mk.in, which we used directly, not ‘configure’.
(LC_ALL): Set to C.
(autogen): New function, to make it clearer to user when
commands are operating in some other directory.
(get_module_list): New function.
(old_modules, new_modules): New vars.
If they differ, issue a warning.
---
 admin/merge-gnulib | 41 +++++++++++++++++++++++++++++++++--------
 1 file changed, 33 insertions(+), 8 deletions(-)

diff --git a/admin/merge-gnulib b/admin/merge-gnulib
index 458a2c325e3..2eab5ab8b9c 100755
--- a/admin/merge-gnulib
+++ b/admin/merge-gnulib
@@ -23,6 +23,9 @@
 
 # written by Paul Eggert
 
+# It is unclear whether module list sorting depends on locale; play it safe.
+export LC_ALL=C
+
 GNULIB_URL=https://git.savannah.gnu.org/git/gnulib.git
 
 GNULIB_MODULES='
@@ -92,22 +95,38 @@ gnulib_srcdir=
     exit 1 ;;
 esac
 
-test -x "$src"autogen.sh || {
+[ -x "$src"autogen.sh ] || {
   printf '%s\n' >&2 "$0: '${src:-.}' is not an Emacs source directory."
   exit 1
 }
 
-test -d "$gnulib_srcdir" ||
+[ -d "$gnulib_srcdir" ] ||
 git clone -- "$GNULIB_URL" "$gnulib_srcdir" ||
 exit
 
-test -x "$gnulib_srcdir"/gnulib-tool || {
+[ -x "$gnulib_srcdir"/gnulib-tool ] || {
   printf '%s\n' >&2 "$0: '$gnulib_srcdir' is not a Gnulib source directory."
   exit 1
 }
 
+autogen() {
+  if [ "$src" ]; then
+    printf "$0: entering $src ..." &&
+    (cd "$src" && ./autogen.sh) &&
+    printf "$0: leaving $src ..."
+  else
+    ./autogen.sh
+  fi
+}
+
 # gnulib-tool has problems with a bare checkout (Bug#32452#65).
-test -f configure || ./autogen.sh || exit
+# Also, we need gnulib.mk.in to get the old module list.
+[ -e "$src"lib/gnulib.mk.in ] || autogen || exit
+
+get_module_list() {
+  sed -n 's/## begin gnulib module //p' "$src"lib/gnulib.mk.in
+}
+old_modules=$(get_module_list) || exit
 
 avoided_flags=
 for module in $AVOIDED_MODULES; do
@@ -115,8 +134,8 @@ avoided_flags=
 done
 
 # Clean the lib directory as well.
-if [ -e "$src"/lib/Makefile ]; then
-   make -C "$src"/lib maintainer-clean
+if [ -e "$src"lib/Makefile ]; then
+   make -C "$src"lib maintainer-clean || exit
 fi
 
 "$gnulib_srcdir"/gnulib-tool --dir="$src" $GNULIB_TOOL_FLAGS \
@@ -140,5 +159,11 @@ avoided_flags=
    "$src"lib &&
 cp -- "$gnulib_srcdir"/m4/codeset.m4 \
    "$src"m4 &&
-{ test -z "$src" || cd "$src"; } &&
-./autogen.sh
+autogen &&
+new_modules=$(get_module_list) || exit
+
+test "$old_modules" = "$new_modules" ||
+  printf >&2 '%s\n' \
+    "$0: warning: module list changed; fix ../nt/gnulib-cfg.mk ..." \
+    "$0: warning: ... or notify emacs-devel for w32 adaption." \
+    "$0: warning: For more, run 'git diff ${src}lib/gnulib.mk.in'."
-- 
2.45.2

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Eli Zaretskii]

> Date: Thu, 2 Jan 2025 11:38:34 -0800
> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 2025-01-02 09:57, Stefan Kangas wrote:
> > I'd find a "warning: new modules added, notify emacs-devel for
> > possible w32 adaption" message more useful than something that blindly
> > pipes things to mail(1).
> 
> Thanks for the suggestion; I installed the attached.

Thanks, this is an improvement.

> I don't see a nontrivial way to update nt/gnulib-cfg.mk automatically, 
> even just for module renames as Eli suggested, as there are too many 
> possibilities for screwups.

Can you elaborate on that?  I only meant to edit the
OMIT_GNULIB_MODULE_* settings when a module gets renamed, nothing
else.  That is, if module FOO is renamed to FOO_BAR, the
OMIT_GNULIB_MODULE_FOO should be edited to OMIT_GNULIB_MODULE_FOO_BAR.

> The "right" way to fix this would be to adjust Emacs and Gnulib so that 
> builds would work the same way on MS-Windows that they do everywhere 
> else (i.e., nt/gnulib-cfg.mk would become unnecessary). However, as I 
> understand it this would be nontrivial and anyway it would require 
> MS-Windows expertise that I lack.

The problem here is that we omit modules because Emacs on Windows
implements or emulates many functionalities in a way that is specific
to Emacs and might not work well elsewhere.  One notable example is
that Emacs supports UTF-8 encoded file names, something Gnulib on
Windows does not, so we cannot use any Gnulib modules that handle file
names.  Gnulib is a general-purpose library, so it cannot use some of
the enhanced functionality we use in Emacs because we have intimate
knowledge of what Emacs needs and which functionalities it actually
uses.  For example, the emulation of pselect is tightly coupled to
what Emacs uses it for.

> In the meantime, module renames are rare enough that doing things by 
> hand should take less work overall. Besides, the main problem here is 
> changes to modules and/or adding modules, which I don't think 
> admin/merge-gnulib can handle by itself.

Yes, all the other changes cannot be automated.  But I thought at
least renames can be.

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Paul Eggert]

On 2025-01-02 12:30, Eli Zaretskii wrote:

>> I don't see a nontrivial way to update nt/gnulib-cfg.mk automatically,
>> even just for module renames as Eli suggested, as there are too many
>> possibilities for screwups.
> 
> Can you elaborate on that?  I only meant to edit the
> OMIT_GNULIB_MODULE_* settings when a module gets renamed, nothing
> else.

Oh, I meant only that module names are idiosyncratic, and some are 
substrings of others, and it's a bit of a pain to get the substitutions 
exactly right.

It could be done, but it's not worth the effort. As far as I can 
remember this is the first module renaming to affect Emacs since 
lib/gnulib.mk.in was introduced in 2017. Such a problem is in the noise 
compared, to ...


>> In the meantime, module renames are rare enough that doing things by
>> hand should take less work overall. Besides, the main problem here is
>> changes to modules and/or adding modules, which I don't think
>> admin/merge-gnulib can handle by itself.
> 
> Yes, all the other changes cannot be automated.

... this one.


> Gnulib is a general-purpose library, so it cannot use some of
> the enhanced functionality we use in Emacs because we have intimate
> knowledge of what Emacs needs and which functionalities it actually
> uses.  For example, the emulation of pselect is tightly coupled to
> what Emacs uses it for.

That's understandable, and I wasn't suggesting that we migrate all of 
Emacs's stuff into Gnulib. All I was thinking is that Emacs could use 
lib/config.mk.in unchanged, even when Emacs is being built on 
MS-Windows. (Admittedly this is not a project I could help much with.)

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Eli Zaretskii]

> Date: Thu, 2 Jan 2025 13:15:51 -0800
> Cc: stefankangas@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> > Gnulib is a general-purpose library, so it cannot use some of
> > the enhanced functionality we use in Emacs because we have intimate
> > knowledge of what Emacs needs and which functionalities it actually
> > uses.  For example, the emulation of pselect is tightly coupled to
> > what Emacs uses it for.
> 
> That's understandable, and I wasn't suggesting that we migrate all of 
> Emacs's stuff into Gnulib. All I was thinking is that Emacs could use 
> lib/config.mk.in unchanged, even when Emacs is being built on 
> MS-Windows. (Admittedly this is not a project I could help much with.)

Maybe I'm missing something, but I don't see how that could be done.
If we cannot use some Gnulib functionality on Windows, we cannot
possibly use and compile the corresponding module, can we?  Or what am
I missing?  Could you elaborate on what you had in mind?

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Paul Eggert]

On 2025-01-02 22:42, Eli Zaretskii wrote:
> Could you elaborate on what you had in mind?

I don't have anything special or genius or anything like that in mind. 
Just the feeling that it could be done without nt/gnulib-cfg.mk. Perhaps 
by goosing configure's cache variables, as nt/mingw-cfg.site already 
does. Or perhaps in some other way. Feel free to ignore my inkling.

Re: master bdb13a0e5c5 1/2: Adjust to Gnulib’s recent module renaming

[Eli Zaretskii]

> Date: Fri, 3 Jan 2025 11:06:38 -0800
> Cc: stefankangas@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 2025-01-02 22:42, Eli Zaretskii wrote:
> > Could you elaborate on what you had in mind?
> 
> I don't have anything special or genius or anything like that in mind. 
> Just the feeling that it could be done without nt/gnulib-cfg.mk. Perhaps 
> by goosing configure's cache variables, as nt/mingw-cfg.site already 
> does. Or perhaps in some other way. Feel free to ignore my inkling.

Setting configure's cache variables is a more troublesome method than
omitting modules.  First, Gnulib and Autoconf rename these variables
from time to time, which requires constant vigilance and followup
changes.  More importantly, understanding which Autoconf-determined
variable(s) need to be tweaked to force the configure script arrive at
the desired result is anything but simple.