Re: master bf7041a: Centralize subprocess creation in a single function.

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Either this patch or the next one broke the build on OpenBSD:

  CC       malloc/scratch_buffer_grow_preserve.o
malloc/scratch_buffer_grow_preserve.c:67:1: fatal error: opening dependency file deps/malloc/scratch_buffer_grow_preserve.d: No such file or directory
 libc_hidden_def (__libc_scratch_buffer_grow_preserve)
 ^~~~~~~~~~~~~~~
  CC       malloc/scratch_buffer_set_array_size.o
malloc/scratch_buffer_set_array_size.c:64:1: fatal error: opening dependency file deps/malloc/scratch_buffer_set_array_size.d: No such file or directory
 libc_hidden_def (__libc_scratch_buffer_set_array_size)
 ^~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
gmake[2]: *** [Makefile:95: malloc/scratch_buffer_grow_preserve.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
  CC       stat-time.o
malloc/scratch_buffer_grow.c:56:1: fatal error: opening dependency file deps/malloc/scratch_buffer_grow.d: No such file or directory
 libc_hidden_def (__libc_scratch_buffer_grow)
 ^~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [Makefile:95: malloc/scratch_buffer_grow.o] Error 1
gmake[2]: *** [Makefile:95: malloc/scratch_buffer_set_array_size.o] Error 1
gmake[2]: Leaving directory '/home/larsi/src/emacs/trunk/lib'
gmake[1]: *** [Makefile:411: lib] Error 2
gmake[1]: Leaving directory '/home/larsi/src/emacs/trunk'
gmake: *** [Makefile:1126: bootstrap] Error 2
*** Error 2 in /home/larsi/src/emacs/trunk (makefile:6 'bootstrap')

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Either this patch or the next one broke the build on OpenBSD:

And Macos, so perhaps on all BSD-derived systems?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 24 Dec 2020 17:09:15 +0100
> Cc: Philipp Stephani <phst@google.com>
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Either this patch or the next one broke the build on OpenBSD:
> 
> And Macos, so perhaps on all BSD-derived systems?

It also broke the Windows build (twice), but I hope I fixed this now.

Re: master bf7041a: Centralize subprocess creation in a single function.

[Eli Zaretskii]

> Date: Thu, 24 Dec 2020 18:21:36 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: phst@google.com, emacs-devel@gnu.org
> 
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Date: Thu, 24 Dec 2020 17:09:15 +0100
> > Cc: Philipp Stephani <phst@google.com>
> > 
> > Lars Ingebrigtsen <larsi@gnus.org> writes:
> > 
> > > Either this patch or the next one broke the build on OpenBSD:
> > 
> > And Macos, so perhaps on all BSD-derived systems?
> 
> It also broke the Windows build (twice), but I hope I fixed this now.

"This" == the MS-Windows build.  I don't have access to any *BSD
systems, so cannot try fixing those, sorry.

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> "This" == the MS-Windows build.  I don't have access to any *BSD
> systems, so cannot try fixing those, sorry.

I've never poked around in the gnulib stuff, so I'm not sure how it's
supposed to work.  But I did a "mkdir lib/deps/malloc", and that fixed
the build, so I guess something is ether not creating that directory, or
the deps files are supposed to be in libs/deps, and not in
lib/deps/malloc?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've never poked around in the gnulib stuff, so I'm not sure how it's
> supposed to work.  But I did a "mkdir lib/deps/malloc", and that fixed
> the build, so I guess something is ether not creating that directory, or
> the deps files are supposed to be in libs/deps, and not in
> lib/deps/malloc?

Or perhaps lib/malloc/* wasn't supposed to be in a sub-directory?  It's
the only sub-directory in lib that has .c files?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> I've never poked around in the gnulib stuff, so I'm not sure how it's
>> supposed to work.  But I did a "mkdir lib/deps/malloc", and that fixed
>> the build, so I guess something is ether not creating that directory, or
>> the deps files are supposed to be in libs/deps, and not in
>> lib/deps/malloc?
>
> Or perhaps lib/malloc/* wasn't supposed to be in a sub-directory?  It's
> the only sub-directory in lib that has .c files?

OK, the lib/deps dir is created here in "configure":

if test $AUTO_DEPEND = yes; then
   for dir in $AUTODEPEND_PARENTS; do
     as_dir=$dir/deps; as_fn_mkdir_p
   done
fi

So either this has to also mkdir lib/deps/malloc (which seems odd), or
the lib/malloc/* files should be moved up one step.  But they're
presumably in a sub-directory for a reason?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So either this has to also mkdir lib/deps/malloc (which seems odd), or
> the lib/malloc/* files should be moved up one step.  But they're
> presumably in a sub-directory for a reason?

Yup.  gnulib/modules/scratch_buffer has:

Files:
lib/scratch_buffer.h
lib/malloc/scratch_buffer.h
lib/malloc/scratch_buffer_grow.c
lib/malloc/scratch_buffer_grow_preserve.c
lib/malloc/scratch_buffer_set_array_size.c

So I guess we need to add something to admin/merge-gnulib to create the
sub-deps dirs for the modules that have them?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: phst@google.com,  emacs-devel@gnu.org
> Date: Thu, 24 Dec 2020 18:11:28 +0100
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > So either this has to also mkdir lib/deps/malloc (which seems odd), or
> > the lib/malloc/* files should be moved up one step.  But they're
> > presumably in a sub-directory for a reason?
> 
> Yup.  gnulib/modules/scratch_buffer has:
> 
> Files:
> lib/scratch_buffer.h
> lib/malloc/scratch_buffer.h
> lib/malloc/scratch_buffer_grow.c
> lib/malloc/scratch_buffer_grow_preserve.c
> lib/malloc/scratch_buffer_set_array_size.c
> 
> So I guess we need to add something to admin/merge-gnulib to create the
> sub-deps dirs for the modules that have them?

Could be, I'm not an expert on merge-gnulib.  Paul, can you help us
out, please?

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Could be, I'm not an expert on merge-gnulib.  Paul, can you help us
> out, please?

This patch fixes the build -- it puts the deps files in one directory,
even if there are sub-directories in lib.  I guess there could
theoretically be name clashes, but it seems very unlikely with the
gnulib naming scheme.


diff --git a/lib/Makefile.in b/lib/Makefile.in
index d68ec131dd..a280b7a331 100644
--- a/lib/Makefile.in
+++ b/lib/Makefile.in
@@ -67,7 +67,7 @@ SYSTEM_TYPE =
 
 DEPDIR = deps
 ifeq ($(AUTO_DEPEND),yes)
-  DEPFLAGS = -MMD -MF $(DEPDIR)/$*.d -MP
+  DEPFLAGS = -MMD -MF $(DEPDIR)/$(*F).d -MP
   -include $(ALLOBJS:%.o=$(DEPDIR)/%.d)
 else
   DEPFLAGS =


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> This patch fixes the build -- it puts the deps files in one directory,
> even if there are sub-directories in lib.  I guess there could
> theoretically be name clashes, but it seems very unlikely with the
> gnulib naming scheme.

I pushed the patch, since this unbroke the build.  But if other
solutions are better, please do go ahead and revert it after fixing it
in a different way.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Mattias Engdegård]

24 dec. 2020 kl. 18.43 skrev Lars Ingebrigtsen <larsi@gnus.org>:

> I pushed the patch, since this unbroke the build.  But if other
> solutions are better, please do go ahead and revert it after fixing it
> in a different way.

The build is still (ec8a17e938c) broken on macOS for out-of-tree builds:

  CC       malloc/scratch_buffer_grow.o
error: unable to open output file 'malloc/scratch_buffer_grow.o': 'No such file
      or directory'
1 error generated.
make[1]: *** [malloc/scratch_buffer_grow.o] Error 1
make: *** [lib] Error 2

As remedy I created the directory $BUILDROOT/lib/malloc after which it worked.

Re: master bf7041a: Centralize subprocess creation in a single function.

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 375 bytes --]

On 12/25/20 2:19 AM, Mattias Engdegård wrote:

>    CC       malloc/scratch_buffer_grow.o
> error: unable to open output file 'malloc/scratch_buffer_grow.o': 'No such file
>        or directory'
> 1 error generated.
> make[1]: *** [malloc/scratch_buffer_grow.o] Error 1
> make: *** [lib] Error 2

I installed the attached to try to fix this; please give it a try.

[-- Attachment #2: 0001-Fix-issues-with-lib-malloc-out-of-dir-builds.patch --]
[-- Type: text/x-patch, Size: 2322 bytes --]

From 597eef7b947a64ae8067058478f26a15c0e24a07 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Fri, 25 Dec 2020 19:16:27 -0800
Subject: [PATCH] Fix issues with lib/malloc out-of-dir builds
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Problem reported by Mattias Engdegård in:
https://lists.gnu.org/r/emacs-devel/2020-12/msg01626.html
* configure.ac: If the scratch_buffer module is enabled, create
lib/malloc and (if doing dependencies) lib/deps/malloc.
* lib/Makefile.in (DEPFLAGS) [AUTO_DEPEND]: No longer any need to
squash lib dependencies into a single directory.
(clean): Also remove */*.o and $(DEPDIR)/*/*.d, to clean out
malloc/*.o and deps/malloc/*.d.  Just remove *.d files in
dependencies so that rm does not complain about not being able to
remove deps/malloc.
(extraclean): Also remove malloc, if it is empty.
---
 configure.ac    | 6 ++++++
 lib/Makefile.in | 6 +++---
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/configure.ac b/configure.ac
index 888b415148..bf768441fe 100644
--- a/configure.ac
+++ b/configure.ac
@@ -5878,6 +5878,12 @@ m4_define
      AS_MKDIR_P([$dir/deps])
    done
 fi
+if $gl_gnulib_enabled_scratch_buffer; then
+  AS_MKDIR_P([lib/malloc])
+  if test $AUTO_DEPEND = yes; then
+    AS_MKDIR_P([lib/deps/malloc])
+  fi
+fi
 
 AC_OUTPUT
 
diff --git a/lib/Makefile.in b/lib/Makefile.in
index a280b7a331..e6b1ae6a9a 100644
--- a/lib/Makefile.in
+++ b/lib/Makefile.in
@@ -67,7 +67,7 @@ SYSTEM_TYPE =
 
 DEPDIR = deps
 ifeq ($(AUTO_DEPEND),yes)
-  DEPFLAGS = -MMD -MF $(DEPDIR)/$(*F).d -MP
+  DEPFLAGS = -MMD -MF $(DEPDIR)/$*.d -MP
   -include $(ALLOBJS:%.o=$(DEPDIR)/%.d)
 else
   DEPFLAGS =
@@ -118,7 +118,7 @@ TAGS:
 .PHONY: $(ETAGS) tags
 
 clean:
-	rm -f ./*.[ao] ./*-t \#* $(DEPDIR)/*
+	rm -f ./*.[ao] ./*/*.o ./*-t \#* $(DEPDIR)/*.d $(DEPDIR)/*/*.d
 mostlyclean: clean
 	rm -f $(filter-out %-t,$(MOSTLYCLEANFILES))
 distclean bootstrap-clean: mostlyclean
@@ -127,7 +127,7 @@ distclean bootstrap-clean:
 maintainer-clean: distclean
 	rm -f TAGS gnulib.mk
 extraclean: distclean
-	-rmdir sys 2>/dev/null
+	-rmdir malloc sys 2>/dev/null
 
 .PHONY: mostlyclean clean distclean bootstrap-clean maintainer-clean
 
-- 
2.27.0

Re: master bf7041a: Centralize subprocess creation in a single function.

[Mattias Engdegård]

26 dec. 2020 kl. 04.19 skrev Paul Eggert <eggert@cs.ucla.edu>:

> I installed the attached to try to fix this; please give it a try.

Building out-of-tree from scratch now works nicely. Thank you!

Re: master bf7041a: Centralize subprocess creation in a single function.

[Paul Eggert]

On 12/24/20 9:31 AM, Lars Ingebrigtsen wrote:
> This patch fixes the build -- it puts the deps files in one directory,
> even if there are sub-directories in lib.  I guess there could
> theoretically be name clashes, but it seems very unlikely with the
> gnulib naming scheme.

Thanks, that patch looks OK to me. If there are name clashes in the 
future, we could do something fancier.

Re: master bf7041a: Centralize subprocess creation in a single function.

[Paul Eggert]

By the way, after my recent cleanups for the Gnulib merge I'm still 
running into problems running parallel 'make check' on Fedora 33 and on 
Ubuntu 20.10. Plain 'make check' works, but 'make -j4 check' hangs. 
'top' reports:

     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ 
COMMAND
1858899 eggert    20   0  373536  33016  20968 R 100.0   0.4   2:18.83 
emacs
1858163 eggert    20   0  440044  99836  21144 R  99.7   1.2   2:21.01 
emacs
1860091 eggert    20   0  378844  38580  21200 R  99.7   0.5   2:11.89 
emacs
1862421 eggert    20   0  489052 142108  21568 R  99.0   1.7   0:17.66 
emacs

'ps -ejH' reports:

     PID    PGID     SID TTY          TIME CMD
1858162 1854607 1781976 pts/4    00:00:00                   sh
1858163 1854607 1781976 pts/4    00:02:48                     emacs
1859170 1859170 1859170 ?        00:00:00                       gpg2 
<defunct>
1858898 1854607 1781976 pts/4    00:00:00                   sh
1858899 1854607 1781976 pts/4    00:02:46                     emacs
1859249 1859249 1859249 ?        00:00:00                       gpg2 
<defunct>
1860090 1854607 1781976 pts/4    00:00:00                   sh
1860091 1854607 1781976 pts/4    00:02:39                     emacs
1860174 1860174 1860174 ?        00:00:00                       gpg2 
<defunct>
1862420 1854607 1781976 pts/4    00:00:00                   sh
1862421 1854607 1781976 pts/4    00:00:44                     emacs
1862429 1862429 1862429 ?        00:00:00                       bash 
<defunct>
...
1833188 1833188 1833188 ?        00:00:00   gpg-agent
1833497 1833497 1833497 ?        00:00:00   gpg-agent
1833690 1833690 1833690 ?        00:00:00   gpg-agent
1841465 1841465 1841465 ?        00:00:00   gpg-agent
1842132 1842132 1842132 ?        00:00:00   gpg-agent
1859145 1859145 1859145 ?        00:00:00   gpg-agent
1859215 1859215 1859215 ?        00:00:00   gpg-agent


I suspect the subprocess creation changes mentioned in this thread's 
subject line, though I haven't investigated further. Plus, I'm not sure 
why tests would be running GPG connected to my gpg-agent - isn't that 
going a bit far for 'make check'?

Re: master bf7041a: Centralize subprocess creation in a single function.

[Philipp Stephani]

Am Fr., 25. Dez. 2020 um 11:42 Uhr schrieb Paul Eggert <eggert@cs.ucla.edu>:
>
> By the way, after my recent cleanups for the Gnulib merge I'm still
> running into problems running parallel 'make check' on Fedora 33 and on
> Ubuntu 20.10. Plain 'make check' works, but 'make -j4 check' hangs.
> 'top' reports:
>
>      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
> 1858899 eggert    20   0  373536  33016  20968 R 100.0   0.4   2:18.83
> emacs
> 1858163 eggert    20   0  440044  99836  21144 R  99.7   1.2   2:21.01
> emacs
> 1860091 eggert    20   0  378844  38580  21200 R  99.7   0.5   2:11.89
> emacs
> 1862421 eggert    20   0  489052 142108  21568 R  99.0   1.7   0:17.66
> emacs
>
> 'ps -ejH' reports:
>
>      PID    PGID     SID TTY          TIME CMD
> 1858162 1854607 1781976 pts/4    00:00:00                   sh
> 1858163 1854607 1781976 pts/4    00:02:48                     emacs
> 1859170 1859170 1859170 ?        00:00:00                       gpg2
> <defunct>
> 1858898 1854607 1781976 pts/4    00:00:00                   sh
> 1858899 1854607 1781976 pts/4    00:02:46                     emacs
> 1859249 1859249 1859249 ?        00:00:00                       gpg2
> <defunct>
> 1860090 1854607 1781976 pts/4    00:00:00                   sh
> 1860091 1854607 1781976 pts/4    00:02:39                     emacs
> 1860174 1860174 1860174 ?        00:00:00                       gpg2
> <defunct>
> 1862420 1854607 1781976 pts/4    00:00:00                   sh
> 1862421 1854607 1781976 pts/4    00:00:44                     emacs
> 1862429 1862429 1862429 ?        00:00:00                       bash
> <defunct>
> ...
> 1833188 1833188 1833188 ?        00:00:00   gpg-agent
> 1833497 1833497 1833497 ?        00:00:00   gpg-agent
> 1833690 1833690 1833690 ?        00:00:00   gpg-agent
> 1841465 1841465 1841465 ?        00:00:00   gpg-agent
> 1842132 1842132 1842132 ?        00:00:00   gpg-agent
> 1859145 1859145 1859145 ?        00:00:00   gpg-agent
> 1859215 1859215 1859215 ?        00:00:00   gpg-agent
>
>
> I suspect the subprocess creation changes mentioned in this thread's
> subject line, though I haven't investigated further. Plus, I'm not sure
> why tests would be running GPG connected to my gpg-agent - isn't that
> going a bit far for 'make check'?
>

I can't reproduce these on either Debian testing or macOS Big Sur.

Re: master bf7041a: Centralize subprocess creation in a single function.

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> By the way, after my recent cleanups for the Gnulib merge I'm still
> running into problems running parallel 'make check' on Fedora 33 and
> on Ubuntu 20.10. Plain 'make check' works, but 'make -j4 check'
> hangs. 'top' reports:
>
>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
>     COMMAND
> 1858899 eggert    20   0  373536  33016  20968 R 100.0   0.4   2:18.83
> emacs
> 1858163 eggert    20   0  440044  99836  21144 R  99.7   1.2   2:21.01
> emacs
> 1860091 eggert    20   0  378844  38580  21200 R  99.7   0.5   2:11.89
> emacs
> 1862421 eggert    20   0  489052 142108  21568 R  99.0   1.7   0:17.66
> emacs

I'm not getting a hang, but I'm getting four failures on Fedora 32:

4 unexpected results:
   FAILED  mml-secure-en-decrypt-2
   FAILED  mml-secure-en-decrypt-3
   FAILED  mml-secure-select-preferred-keys-4
   FAILED  mml-secure-sign-verify-1

I'll have a look.

> 1859215 1859215 1859215 ?        00:00:00   gpg-agent
>
> I suspect the subprocess creation changes mentioned in this thread's
> subject line, though I haven't investigated further. Plus, I'm not
> sure why tests would be running GPG connected to my gpg-agent - isn't
> that going a bit far for 'make check'?

The tests start a local gpg-agent, but are supposed to kill them off
again.  I see that no longer works, but I've pushed a fix for that now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: master bf7041a: Centralize subprocess creation in a single function.

[Philipp Stephani]

Am Do., 24. Dez. 2020 um 17:38 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > "This" == the MS-Windows build.  I don't have access to any *BSD
> > systems, so cannot try fixing those, sorry.
>
> I've never poked around in the gnulib stuff, so I'm not sure how it's
> supposed to work.  But I did a "mkdir lib/deps/malloc", and that fixed
> the build, so I guess something is ether not creating that directory, or
> the deps files are supposed to be in libs/deps, and not in
> lib/deps/malloc?


IIUC Gnulib is intended to be used with Automake, but since we don't
use Automake any more, it seems we have to perform a few manual steps
(in the Makefile) like creating these subdirectories.

Re: master bf7041a: Centralize subprocess creation in a single function.

[Philipp Stephani]

Am Do., 24. Dez. 2020 um 17:08 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
>
> Either this patch or the next one broke the build on OpenBSD:
>
>   CC       malloc/scratch_buffer_grow_preserve.o
> malloc/scratch_buffer_grow_preserve.c:67:1: fatal error: opening dependency file deps/malloc/scratch_buffer_grow_preserve.d: No such file or directory
>  libc_hidden_def (__libc_scratch_buffer_grow_preserve)
>  ^~~~~~~~~~~~~~~
>   CC       malloc/scratch_buffer_set_array_size.o
> malloc/scratch_buffer_set_array_size.c:64:1: fatal error: opening dependency file deps/malloc/scratch_buffer_set_array_size.d: No such file or directory
>  libc_hidden_def (__libc_scratch_buffer_set_array_size)
>  ^~~~~~~~~~~~~~~
> compilation terminated.
> compilation terminated.


This is due to the updated Gnulib; I had to create the directory
lib/deps/malloc manually, after that it worked. Not sure whether
there's a better way. Probably these required directories should be
created automatically.