bug#56606: 29.0.50; recent master fails with "creating pipe: too many open files"

bug#56606: 29.0.50; recent master fails with "creating pipe: too many open files"

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 263 bytes --]


On Windows, load the attached file:

M-x load-file debug-process.el

It signals an error, with the message "Creating pipe: Too many open
files".

On Debian the error does not happen. In Emacs 28, and in emacs master
in December 2021, the error does not happen.


[-- Attachment #2: debug-process.el --]
[-- Type: application/emacs-lisp, Size: 1385 bytes --]

[-- Attachment #3: Type: text/plain, Size: 4169 bytes --]




In GNU Emacs 29.0.50 (build 2, x86_64-w64-mingw32)
 of 2022-07-16 built on TAKVER4
Repository revision: 35d0a2e0a767838c24d5853be798313aed7a42df
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 6.3.9600
System Description: Microsoft Windows 8.1 Pro (v6.3.0.9600.20478)

Configured using:
 'configure
PKG_CONFIG_PATH=/mingw64/lib/pkgconfig:/mingw64/share/pkgconfig'

Configured features:
ACL DBUS GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XPM ZLIB

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Summary

Minor modes in effect:
  debbugs-browse-mode: t
  bug-reference-mode: t
  display-time-mode: t
  delete-selection-mode: t
  icomplete-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug mailalias smerge-mode diff canlock bbdb-message
gnus-kill mule-util sort smiley gnus-cite flow-fill mm-archive mail-extr
textsec uni-scripts idna-mapping ucs-normalize uni-confusable
textsec-check gnus-async gnus-bcklg qp gnus-ml vc-mtn vc-git diff-mode
vc vc-dispatcher debbugs-browse bug-reference utf-7 imap rfc2104 nndraft
nnmh gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg nnml
gnus-cache bbdb-gnus network-stream nsm nntp gnus-art mm-uu mml2015
mm-view mml-smime smime gnutls dig gnus-sum shr pixel-fill kinsoku
url-file url-dired svg dom gnus-group gnus-undo gnus-start gnus-dbus
dbus xml gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time
iso8601 gnus-spec gnus-int gnus-range gnus-win gnus wid-edit nnheader
range message yank-media puny rfc822 mml mml-sec epa epg rfc6068
epg-config gnus-util time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 gmm-utils mailheader bbdb-mua bbdb-com pcase crm mailabbrev bbdb
derived bbdb-site timezone smtpmail sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils time delsel cus-load stephe-theme noutline
outline easy-mmode path-iterator cl-extra help-mode whitespace dired-x
dired-aux dired dired-loaddefs compile text-property-search comint
ansi-color uniquify-files icomplete xref project ring info package
browse-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie generate-lisp-file url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs password-cache
json subr-x map byte-opt gv bytecomp byte-compile cconv url-vars
cl-loaddefs cl-lib rmc iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
faces cus-face macroexp files window text-properties overlay sha1 md5
base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify dbusbind w32 lcms2
multi-tty make-network-process emacs)

Memory information:
((conses 16 366274 36249)
 (symbols 48 19973 3)
 (strings 32 79351 4442)
 (string-bytes 1 2356349)
 (vectors 16 59637)
 (vector-slots 8 1148125 74030)
 (floats 8 284 234)
 (intervals 56 2094 329)
 (buffers 992 28))

-- 
-- Stephe

bug#56606: Acknowledgement (29.0.50; recent master fails with "creating pipe: too many open files")

[Stephen Leake]

On emacs-devel, Eli said:

AFAICT, this loop in deactivate_process:

  /* Beware SIGCHLD hereabouts.  */

  for (i = 0; i < PROCESS_OPEN_FDS; i++)
    close_process_fd (&p->open_fd[i]);

doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
calls in make-pipe-process.  It calls 'close' with the right value,
and close returns zero, but the pipe stays open.  In Emacs 28, this
same loop successfully closes the descriptor.  I don't know why.

Perhaps bisecting could help.

-- 
-- Stephe

bug#56606: Acknowledgement (29.0.50; recent master fails with "creating pipe: too many open files")

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sun, 17 Jul 2022 02:42:04 -0700
> 
> On emacs-devel, Eli said:
> 
> AFAICT, this loop in deactivate_process:
> 
>   /* Beware SIGCHLD hereabouts.  */
> 
>   for (i = 0; i < PROCESS_OPEN_FDS; i++)
>     close_process_fd (&p->open_fd[i]);
> 
> doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
> calls in make-pipe-process.  It calls 'close' with the right value,
> and close returns zero, but the pipe stays open.  In Emacs 28, this
> same loop successfully closes the descriptor.  I don't know why.
> 
> Perhaps bisecting could help.

I think commit a81669c could be the culprit.

bug#56606: Acknowledgement (29.0.50; recent master fails with "creating pipe: too many open files")

[Eli Zaretskii]

> Cc: 56606@debbugs.gnu.org
> Date: Sun, 17 Jul 2022 13:20:12 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Stephen Leake <stephen_leake@stephe-leake.org>
> > Date: Sun, 17 Jul 2022 02:42:04 -0700
> > 
> > On emacs-devel, Eli said:
> > 
> > AFAICT, this loop in deactivate_process:
> > 
> >   /* Beware SIGCHLD hereabouts.  */
> > 
> >   for (i = 0; i < PROCESS_OPEN_FDS; i++)
> >     close_process_fd (&p->open_fd[i]);
> > 
> > doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
> > calls in make-pipe-process.  It calls 'close' with the right value,
> > and close returns zero, but the pipe stays open.  In Emacs 28, this
> > same loop successfully closes the descriptor.  I don't know why.
> > 
> > Perhaps bisecting could help.
> 
> I think commit a81669c could be the culprit.

I hope I fixed this now on master.

bug#56606: Acknowledgement (29.0.50; recent master fails with "creating pipe: too many open files")

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 56606@debbugs.gnu.org
>> Date: Sun, 17 Jul 2022 13:20:12 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> > From: Stephen Leake <stephen_leake@stephe-leake.org>
>> > Date: Sun, 17 Jul 2022 02:42:04 -0700
>> > 
>> > On emacs-devel, Eli said:
>> > 
>> > AFAICT, this loop in deactivate_process:
>> > 
>> >   /* Beware SIGCHLD hereabouts.  */
>> > 
>> >   for (i = 0; i < PROCESS_OPEN_FDS; i++)
>> >     close_process_fd (&p->open_fd[i]);
>> > 
>> > doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
>> > calls in make-pipe-process.  It calls 'close' with the right value,
>> > and close returns zero, but the pipe stays open.  In Emacs 28, this
>> > same loop successfully closes the descriptor.  I don't know why.
>> > 
>> > Perhaps bisecting could help.
>> 
>> I think commit a81669c could be the culprit.
>
> I hope I fixed this now on master.
>

Yes, it is fixed, by 637436970f34f860d50f73a514b3bafd0c5cace7.

-- 
-- Stephe

bug#56606: Acknowledgement (29.0.50; recent master fails with "creating pipe: too many open files")

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: 56606@debbugs.gnu.org
> Date: Mon, 18 Jul 2022 17:04:05 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Cc: 56606@debbugs.gnu.org
> >> Date: Sun, 17 Jul 2022 13:20:12 +0300
> >> From: Eli Zaretskii <eliz@gnu.org>
> >> 
> >> > From: Stephen Leake <stephen_leake@stephe-leake.org>
> >> > Date: Sun, 17 Jul 2022 02:42:04 -0700
> >> > 
> >> > On emacs-devel, Eli said:
> >> > 
> >> > AFAICT, this loop in deactivate_process:
> >> > 
> >> >   /* Beware SIGCHLD hereabouts.  */
> >> > 
> >> >   for (i = 0; i < PROCESS_OPEN_FDS; i++)
> >> >     close_process_fd (&p->open_fd[i]);
> >> > 
> >> > doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
> >> > calls in make-pipe-process.  It calls 'close' with the right value,
> >> > and close returns zero, but the pipe stays open.  In Emacs 28, this
> >> > same loop successfully closes the descriptor.  I don't know why.
> >> > 
> >> > Perhaps bisecting could help.
> >> 
> >> I think commit a81669c could be the culprit.
> >
> > I hope I fixed this now on master.
> >
> 
> Yes, it is fixed, by 637436970f34f860d50f73a514b3bafd0c5cace7.

Thanks, I'm therefore closing this bug.

tramp "too many open files" [Re: bug#56606

[Madhu]

The #56606 is archived and this may be unrelated and related to tramp,
but I wanted to note I encoutered something similar on recent master
(GNU Emacs 30.0.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo
version 1.16.0, Xaw3d scroll bars) of 2023-03-18) native compilation.

Running the image after it compiled, when trying to save a zip file
attachment in mew with y, I ran into the too many open files problem.

```
$ lsof -n|wc -l
59510
$ lsof -n |grep tramp|wc -l
4036
```

most of the entries in lsof were
```
COMMAND     PID   TID TASKCMD    USER   FD      TYPE             DEVICE    SIZE
/OFF       NODE NAME
emacs     30346 31097 gdbus     madhu  865r      REG               8,14       2
5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
emacs     30346 31097 gdbus     madhu  866r      REG               8,14       2
5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
emacs     30346 31097 gdbus     madhu  867r      REG               8,14       2
5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
emacs     30346 31097 gdbus     madhu  868r      REG               8,14       2
```

Of course I could do nothing but kill the process. I updated mew to 6.9
and tried the sequence again in a fresh emacs, but wasn't able to
reproduce this (but hit a vertico bug instead)

If this looks familiar, or if someone has a clue about it, please tell
me. Maybe I should a bug report under the `tramp' product if I hit it
again.

Re: tramp "too many open files" [Re: bug#56606

[Thierry Volpiatto]

Madhu <enometh@meer.net> writes:

> The #56606 is archived and this may be unrelated and related to tramp,
> but I wanted to note I encoutered something similar on recent master
> (GNU Emacs 30.0.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo
> version 1.16.0, Xaw3d scroll bars) of 2023-03-18) native compilation.
>
> Running the image after it compiled, when trying to save a zip file
> attachment in mew with y, I ran into the too many open files problem.
>
> ```
> $ lsof -n|wc -l
> 59510
> $ lsof -n |grep tramp|wc -l
> 4036
> ```
>
> most of the entries in lsof were
> ```
> COMMAND     PID   TID TASKCMD    USER   FD      TYPE             DEVICE    SIZE
> /OFF       NODE NAME
> emacs     30346 31097 gdbus     madhu  865r      REG               8,14       2
> 5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
> emacs     30346 31097 gdbus     madhu  866r      REG               8,14       2
> 5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
> emacs     30346 31097 gdbus     madhu  867r      REG               8,14       2
> 5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
> emacs     30346 31097 gdbus     madhu  868r      REG               8,14       2
> ```
>
> Of course I could do nothing but kill the process. I updated mew to 6.9
> and tried the sequence again in a fresh emacs, but wasn't able to
> reproduce this (but hit a vertico bug instead)
>
> If this looks familiar, or if someone has a clue about it, please tell
> me. Maybe I should a bug report under the `tramp' product if I hit it
> again.

From the Helm help:

As Tramp archive often crash Helm and Emacs, Helm does its best
to disable it, however it is hard to do so as Tramp Archive is
enabled inconditionally in Emacs.  Here I build my Emacs
without-dbus to ensure Tramp archive wont kickin unexpectedly.


-- 
Thierry

Re: tramp "too many open files" [Re: bug#56606

[Michael Albinus]

Thierry Volpiatto <thievol@posteo.net> writes:

Hi Thierry,

> From the Helm help:
>
> As Tramp archive often crash Helm and Emacs, Helm does its best
> to disable it, however it is hard to do so as Tramp Archive is
> enabled inconditionally in Emacs.  Here I build my Emacs
> without-dbus to ensure Tramp archive wont kickin unexpectedly.

There are two ways to disable tramp-archive.el:

- Set or bind tramp-archive-enabled to nil.

- Avoid file names like "/path/to/file.tar/". The combination of a known
  extension with a slash triggers tramp-archive.el. Usually, it is an
  error to specify such a file name, if not intended.

Best regards, Michael.

Re: tramp "too many open files" [Re: bug#56606

[Thierry Volpiatto]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]


Hi Michael,

Michael Albinus <michael.albinus@gmx.de> writes:

> Thierry Volpiatto <thievol@posteo.net> writes:
>
> Hi Thierry,
>
>> From the Helm help:
>>
>> As Tramp archive often crash Helm and Emacs, Helm does its best
>> to disable it, however it is hard to do so as Tramp Archive is
>> enabled inconditionally in Emacs.  Here I build my Emacs
>> without-dbus to ensure Tramp archive wont kickin unexpectedly.
>
> There are two ways to disable tramp-archive.el:
>
> - Set or bind tramp-archive-enabled to nil.

I know but it is not enough in some cases. I think it should be up to
the user to set this, however it is already set and autoloaded in tramp-archive.el:

    (defvar tramp-archive-enabled (featurep 'dbusbind)
      "Non-nil when file archive support is available.")


> - Avoid file names like "/path/to/file.tar/". The combination of a known
>   extension with a slash triggers tramp-archive.el. Usually, it is an
>   error to specify such a file name, if not intended.

Agree, but we sometimes get files from elsewhere with such filenames, of
course I never name my files/dirs like this.
Anyway I think tramp-archive should not kick in with such filenames or
any filenames unless user wants it.

Thanks.

-- 
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 686 bytes --]

Re: tramp "too many open files" [Re: bug#56606

[Michael Albinus]

Thierry Volpiatto <thievol@posteo.net> writes:

> Hi Michael,

Hi Thierry,

>> There are two ways to disable tramp-archive.el:
>>
>> - Set or bind tramp-archive-enabled to nil.
>
> I know but it is not enough in some cases. I think it should be up to
> the user to set this, however it is already set and autoloaded in tramp-archive.el:
>
>     (defvar tramp-archive-enabled (featurep 'dbusbind)
>       "Non-nil when file archive support is available.")

And this is intended. After loading tramp-archive, we have

(setq tramp-archive-enabled tramp-gvfs-enabled)

which is less invasive, I hope.

>> - Avoid file names like "/path/to/file.tar/". The combination of a known
>>   extension with a slash triggers tramp-archive.el. Usually, it is an
>>   error to specify such a file name, if not intended.
>
> Agree, but we sometimes get files from elsewhere with such filenames, of
> course I never name my files/dirs like this.
> Anyway I think tramp-archive should not kick in with such filenames or
> any filenames unless user wants it.

I do my best to bring external packages to proper coding.

And also, tramp-archive-file-name-handler checks in case of a file name
"/path/to/file.tar/", whether "/path/to/file.tar" is a directory. In
this case, tramp-archive also ceases to work.

There's always room for improvement. But disabling tramp-enabled by
default would mean we could get rid of the package, because it wouldn't
be visible any longer. It is a core package, after all.

But let's see what's the case with mew. That's what this bug report is
about.

> Thanks.

Best regards, Michael.

Re: tramp "too many open files" [Re: bug#56606

[Thierry Volpiatto]

[-- Attachment #1: Type: text/plain, Size: 1803 bytes --]


Hello Michael,

Michael Albinus <michael.albinus@gmx.de> writes:

> Thierry Volpiatto <thievol@posteo.net> writes:
>
>> Hi Michael,
>
> Hi Thierry,
>
>>> There are two ways to disable tramp-archive.el:
>>>
>>> - Set or bind tramp-archive-enabled to nil.
>>
>> I know but it is not enough in some cases. I think it should be up to
>> the user to set this, however it is already set and autoloaded in tramp-archive.el:
>>
>>     (defvar tramp-archive-enabled (featurep 'dbusbind)
>>       "Non-nil when file archive support is available.")
>
> And this is intended. After loading tramp-archive, we have
>
> (setq tramp-archive-enabled tramp-gvfs-enabled)
>
> which is less invasive, I hope.
>
>>> - Avoid file names like "/path/to/file.tar/". The combination of a known
>>>   extension with a slash triggers tramp-archive.el. Usually, it is an
>>>   error to specify such a file name, if not intended.
>>
>> Agree, but we sometimes get files from elsewhere with such filenames, of
>> course I never name my files/dirs like this.
>> Anyway I think tramp-archive should not kick in with such filenames or
>> any filenames unless user wants it.
>
> I do my best to bring external packages to proper coding.

I know, thanks.

> And also, tramp-archive-file-name-handler checks in case of a file name
> "/path/to/file.tar/", whether "/path/to/file.tar" is a directory.
> In this case, tramp-archive also ceases to work.

Yes, the problem we had in Helm was with while-no-input, when a check in
done under it (file-directory-p or whatever) a dbus event is sent and
while-no-input fails, we have to let bind while-no-input-ignore-events
with the new dbus-event added to fix the problem, perhaps tramp-archive
should take care of this?

Thanks.

-- 
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 686 bytes --]

Re: tramp "too many open files" [Re: bug#56606

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 744 bytes --]

Thierry Volpiatto <thievol@posteo.net> writes:

> Hello Michael,

Hi Thierry,

>> And also, tramp-archive-file-name-handler checks in case of a file name
>> "/path/to/file.tar/", whether "/path/to/file.tar" is a directory.
>> In this case, tramp-archive also ceases to work.
>
> Yes, the problem we had in Helm was with while-no-input, when a check in
> done under it (file-directory-p or whatever) a dbus event is sent and
> while-no-input fails, we have to let bind while-no-input-ignore-events
> with the new dbus-event added to fix the problem, perhaps tramp-archive
> should take care of this?

Indeed. dbus-event, file-notify and thread-event have been added to
while-no-input-ignore-events in Emacs 29.  What about the following
patch?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 713 bytes --]

diff --git a/lisp/tramp-gvfs.el b/lisp/tramp-gvfs.el
index 7323374c..0d23f5d8 100644
--- a/lisp/tramp-gvfs.el
+++ b/lisp/tramp-gvfs.el
@@ -872,6 +872,14 @@ arguments to pass to the OPERATION."
    (tramp-register-foreign-file-name-handler
     #'tramp-gvfs-file-name-p #'tramp-gvfs-file-name-handler)))

+;; Event type `dbus-event' is added to `while-no-input-ignore-events'
+;; in Emacs 29.1.  If it is missing, some packages like Helm report
+;; problems.  So we add it here.
+(when (and (featurep 'dbusbind)
+	   (not (memq 'dbus-event while-no-input-ignore-events)))
+  (setq while-no-input-ignore-events
+	(cons 'dbus-event while-no-input-ignore-events)))
+
 \f
 ;; D-Bus helper function.


[-- Attachment #3: Type: text/plain, Size: 35 bytes --]


> Thanks.

Best regards, Michael.

Re: tramp "too many open files" [Re: bug#56606

[Thierry Volpiatto]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]


Hi Michael,

Michael Albinus <michael.albinus@gmx.de> writes:

> Thierry Volpiatto <thievol@posteo.net> writes:
>
>> Hello Michael,
>
> Hi Thierry,
>
>>> And also, tramp-archive-file-name-handler checks in case of a file name
>>> "/path/to/file.tar/", whether "/path/to/file.tar" is a directory.
>>> In this case, tramp-archive also ceases to work.
>>
>> Yes, the problem we had in Helm was with while-no-input, when a check in
>> done under it (file-directory-p or whatever) a dbus event is sent and
>> while-no-input fails, we have to let bind while-no-input-ignore-events
>> with the new dbus-event added to fix the problem, perhaps tramp-archive
>> should take care of this?
>
> Indeed. dbus-event, file-notify and thread-event have been added to
> while-no-input-ignore-events in Emacs 29.  What about the following
> patch?

Yes looks good, I use about the same code in Helm actually.

Thanks.

> diff --git a/lisp/tramp-gvfs.el b/lisp/tramp-gvfs.el
> index 7323374c..0d23f5d8 100644
> --- a/lisp/tramp-gvfs.el
> +++ b/lisp/tramp-gvfs.el
> @@ -872,6 +872,14 @@ arguments to pass to the OPERATION."
>     (tramp-register-foreign-file-name-handler
>      #'tramp-gvfs-file-name-p #'tramp-gvfs-file-name-handler)))
>
> +;; Event type `dbus-event' is added to `while-no-input-ignore-events'
> +;; in Emacs 29.1.  If it is missing, some packages like Helm report
> +;; problems.  So we add it here.
> +(when (and (featurep 'dbusbind)
> +	   (not (memq 'dbus-event while-no-input-ignore-events)))
> +  (setq while-no-input-ignore-events
> +	(cons 'dbus-event while-no-input-ignore-events)))
> +
>  \f
>  ;; D-Bus helper function.
>
>
>> Thanks.
>
> Best regards, Michael.


-- 
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 686 bytes --]

Re: tramp "too many open files" [Re: bug#56606

[Michael Albinus]

Thierry Volpiatto <thievol@posteo.net> writes:

> Hi Michael,

Hi Thierry,

> Yes looks good, I use about the same code in Helm actually.

I've pushed this patch to master. It will also be contained in the
upcoming Tramp 2.6.0.3 on GNU ELPA.

> Thanks.

Best regards, Michael.

Re: tramp "too many open files" [Re: bug#56606

[Thierry Volpiatto]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]


Michael Albinus <michael.albinus@gmx.de> writes:

> Thierry Volpiatto <thievol@posteo.net> writes:
>
>> Hi Michael,
>
> Hi Thierry,
>
>> Yes looks good, I use about the same code in Helm actually.
>
> I've pushed this patch to master. It will also be contained in the
> upcoming Tramp 2.6.0.3 on GNU ELPA.

Great, thanks Michael.

>> Thanks.
>
> Best regards, Michael.


-- 
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 686 bytes --]

Re: tramp "too many open files" [Re: bug#56606

[Michael Albinus]

Madhu <enometh@meer.net> writes:

Hi,

> The #56606 is archived and this may be unrelated and related to tramp,
> but I wanted to note I encoutered something similar on recent master
> (GNU Emacs 30.0.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo
> version 1.16.0, Xaw3d scroll bars) of 2023-03-18) native compilation.

I might be wrong, but likely, bug#56606 is unrelated.

> Running the image after it compiled, when trying to save a zip file
> attachment in mew with y, I ran into the too many open files problem.
>
> ```
> $ lsof -n|wc -l
> 59510
> $ lsof -n |grep tramp|wc -l
> 4036
> ```
>
> most of the entries in lsof were
> ```
> COMMAND     PID   TID TASKCMD    USER   FD      TYPE             DEVICE    SIZE
> /OFF       NODE NAME
> emacs     30346 31097 gdbus     madhu  865r      REG               8,14       2
> 5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
> emacs     30346 31097 gdbus     madhu  866r      REG               8,14       2
> 5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
> emacs     30346 31097 gdbus     madhu  867r      REG               8,14       2
> 5318    1334064 /14/build/emacs/lisp/net/tramp-archive.elc
> emacs     30346 31097 gdbus     madhu  868r      REG               8,14       2
> ```

Hmm. tramp-archive.el uses th GVFS file system to read a file
archive. GVFS in this case uses libarchive(3).

> Of course I could do nothing but kill the process. I updated mew to 6.9
> and tried the sequence again in a fresh emacs, but wasn't able to
> reproduce this (but hit a vertico bug instead)
>
> If this looks familiar, or if someone has a clue about it, please tell
> me. Maybe I should a bug report under the `tramp' product if I hit it
> again.

What I could imagine is, that Emacs has many of the files open from the zip
file. However, mew shouldn't use tramp-archive.el ...

Well, I'm not familiar with mew. Could you please write a bug report,
which contains a recipe starting with "emacs -Q", and which tells how to
reach the point when the error appeared as reported above. The recipe is
also good when it doesn't show the error; I want to use it in order to
see how tramp-archive.el comes into play.

Best regards, Michael.

Re: tramp "too many open files" [Re: bug#56606

[Madhu]

*  Michael Albinus <87ttyfzcpf.fsf @gmx.de>
Wrote on Mon, 20 Mar 2023 16:04:12 +0100
> Madhu <enometh@meer.net> writes:
>> The #56606 is archived and this may be unrelated and related to tramp,
>> but I wanted to note I encoutered something similar on recent master
>> (GNU Emacs 30.0.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo
>> version 1.16.0, Xaw3d scroll bars) of 2023-03-18) native compilation.
>
> I might be wrong, but likely, bug#56606 is unrelated.

Thanks for your note, Sorry for the false alarm.

>
> Hmm. tramp-archive.el uses th GVFS file system to read a file
> archive. GVFS in this case uses libarchive(3).

I notice my gvfsd is not compiled with libarchive.

>> Of course I could do nothing but kill the process. I updated mew to 6.9
>> and tried the sequence again in a fresh emacs, but wasn't able to
>> reproduce this (but hit a vertico bug instead)
>>
>> If this looks familiar, or if someone has a clue about it, please tell
>> me. Maybe I should a bug report under the `tramp' product if I hit it
>> again.
>
> What I could imagine is, that Emacs has many of the files open from the zip
> file. However, mew shouldn't use tramp-archive.el ...

Ah, I might have made a mistake when entering the filename: by adding
"foo.zip/" when saving the file (it uses read-file-name)

> Well, I'm not familiar with mew. Could you please write a bug report,
> which contains a recipe starting with "emacs -Q", and which tells how to
> reach the point when the error appeared as reported above. The recipe is
> also good when it doesn't show the error; I want to use it in order to
> see how tramp-archive.el comes into play.

Usually I run without a system dbus (but with a session dbus for
emacs). tramp-archive disables itself when it finds it cannot connect
to a system dbus, so I was protected from it being triggered. (btw if
the situation changes, and a system dbus comes up later, it has to be
initialized explicitly with (dbus-init-bus :system))

I think I happened to have a system dbus when I triggered the
error. tramp found a gvfs and gvfs-fuse process, but it wouldn't have
worked because gvfds is not linked with libarchive.

So far I've not able to reproduce the error and am not sure. When I
try to trigger it with a filename with like /tmp/foo.zip/sle.txt, it
just says that there is no such directory.

[I'll try to compile gvfsd with libarchive and try again later, but it
will take some time to unscrew and fix the gentoo config: I have to
update libarchive, and that requires undoing some upstream e2fsprogs
decisions.]

Re: tramp "too many open files" [Re: bug#56606

[Michael Albinus]

Madhu <enometh@meer.net> writes:

Hi Madhu,

>> Hmm. tramp-archive.el uses th GVFS file system to read a file
>> archive. GVFS in this case uses libarchive(3).
>
> I notice my gvfsd is not compiled with libarchive.

No. gvfsd calls backends, /usr/libexec/gvfsd-archive in this case.

>> Well, I'm not familiar with mew. Could you please write a bug report,
>> which contains a recipe starting with "emacs -Q", and which tells how to
>> reach the point when the error appeared as reported above. The recipe is
>> also good when it doesn't show the error; I want to use it in order to
>> see how tramp-archive.el comes into play.
>
> Usually I run without a system dbus (but with a session dbus for
> emacs). tramp-archive disables itself when it finds it cannot connect
> to a system dbus, so I was protected from it being triggered. (btw if
> the situation changes, and a system dbus comes up later, it has to be
> initialized explicitly with (dbus-init-bus :system))

Right, although I'm not sure the system bus is needed. GVFS works over
the session bus if I'm not mistaken. The system bus is needed for
zeroconf only.

> I think I happened to have a system dbus when I triggered the
> error. tramp found a gvfs and gvfs-fuse process, but it wouldn't have
> worked because gvfds is not linked with libarchive.

Again, the existence of gvfsd-archive is what counts. I believe.

Best regards, Michael.

Re: {Spam?} Re: tramp "too many open files" [Re: bug#56606

[Madhu]

*  Michael Albinus <michael.albinus@gmx.de> <87bkkmsb7b.fsf@gmx.de>
Wrote on Tue, 21 Mar 2023 10:30:48 +0100
>>> Hmm. tramp-archive.el uses th GVFS file system to read a file
>>> archive. GVFS in this case uses libarchive(3).
>> I notice my gvfsd is not compiled with libarchive.
> No. gvfsd calls backends, /usr/libexec/gvfsd-archive in this case.

After I posted my message I had upgraded both libarchive and gvfsd
(1.45.3->1.50.4) and tried opening the zip attachment (which was now
saved via mew) with the / suffix and it worked as expected, without
blowing up, so nothing is actionable now.

My previous version of gvfsd did not have /usr/libexec/gvfsd-archive
when I reported the bug[1]

>> Usually I run without a system dbus (but with a session dbus for
>> emacs). tramp-archive disables itself when it finds it cannot connect
>> to a system dbus, so I was protected from it being triggered. (btw if
>> the situation changes, and a system dbus comes up later, it has to be
>> initialized explicitly with (dbus-init-bus :system))
> Right, although I'm not sure the system bus is needed. GVFS works over
> the session bus if I'm not mistaken. The system bus is needed for
> zeroconf only.

[tramp-gvfsd still seems to turn itself if it doesnt find the system
dbus enabled when it is first loaded. If the system bus comes up
later, emacs has to be made aware of it manually and tramp-gvfsd has
to be reloaded, and tramp-register-archive-autoload-file-name-handler
has to be invoked by hand before it can open archive files.]

---Best Regards, Madhu


>> I think I happened to have a system dbus when I triggered the
>> error. tramp found a gvfs and gvfs-fuse process, but it wouldn't have
>> worked because gvfds is not linked with libarchive.
> Again, the existence of gvfsd-archive is what counts. I believe

[1] A note to myself -- checked the ondisk zfs snapshot to make sure
it wasnt there when I triggered the blowup.  (Strangely I do seem to
have built a tarball of gvfs-1.46 in Nov 2020 to be installed and it
did have gvfsd-archive. If it had been installed I imagine a system
crash and zfs rollback had removed it)

Re: tramp "too many open files"

[Michael Albinus]

Madhu <enometh@meer.net> writes:

Hi Madhu,

> After I posted my message I had upgraded both libarchive and gvfsd
> (1.45.3->1.50.4) and tried opening the zip attachment (which was now
> saved via mew) with the / suffix and it worked as expected, without
> blowing up, so nothing is actionable now.

Good.

> My previous version of gvfsd did not have /usr/libexec/gvfsd-archive
> when I reported the bug[1]

I've added a sanity check to tramp-gvfs.el, in order to detect whether
the needed GVFS backend is applicable. Pushed to master.

Temporarily, I've removed the Fedora package gvfs-archive from my
laptop. And indeed, this sanity check did signal the problem. So I guess
we're done with this problem.

>>> Usually I run without a system dbus (but with a session dbus for
>>> emacs). tramp-archive disables itself when it finds it cannot connect
>>> to a system dbus, so I was protected from it being triggered. (btw if
>>> the situation changes, and a system dbus comes up later, it has to be
>>> initialized explicitly with (dbus-init-bus :system))
>> Right, although I'm not sure the system bus is needed. GVFS works over
>> the session bus if I'm not mistaken. The system bus is needed for
>> zeroconf only.
>
> [tramp-gvfsd still seems to turn itself if it doesnt find the system
> dbus enabled when it is first loaded. If the system bus comes up
> later, emacs has to be made aware of it manually and tramp-gvfsd has
> to be reloaded, and tramp-register-archive-autoload-file-name-handler
> has to be invoked by hand before it can open archive files.]

Hmm. I don't know whether this is a common case (running no system bus)
we need to handle in Tramp.

> ---Best Regards, Madhu

Best regards, Michael.

Build failure: Re: tramp "too many open files"

[Madhu]

Hello

*  Michael Albinus <michael.albinus@gmx.de> <87jzz3ibp0.fsf_-_@gmx.de>
Wrote on Sun, 26 Mar 2023 20:56:11 +0200
[snip]
>>>> Usually I run without a system dbus (but with a session dbus for
>>>> emacs). tramp-archive disables itself when it finds it cannot connect
>>>> to a system dbus, so I was protected from it being triggered. (btw if
>>>> the situation changes, and a system dbus comes up later, it has to be
>>>> initialized explicitly with (dbus-init-bus :system))
>>> Right, although I'm not sure the system bus is needed. GVFS works over
>>> the session bus if I'm not mistaken. The system bus is needed for
>>> zeroconf only.
>>
>> [tramp-gvfsd still seems to turn itself if it doesnt find the system
>> dbus enabled when it is first loaded. If the system bus comes up
>> later, emacs has to be made aware of it manually and tramp-gvfsd has
>> to be reloaded, and tramp-register-archive-autoload-file-name-handler
>> has to be invoked by hand before it can open archive files.]
>
> Hmm. I don't know whether this is a common case (running no system bus)
> we need to handle in Tramp.

This now results in a build failure -- on master a7dcc0d55c6

```
../../lisp/net/tramp-adb.el:231:22: Warning: the function ‘tramp-compat-string-replace’ might not be defined at runtime.
../../lisp/net/tramp-adb.el:215:6: Warning: the function ‘tramp-run-real-handler’ might not be defined at runtime.
  ELC      net/tramp-archive.elc

In toplevel form:
../../lisp/net/tramp-archive.el:113:2: Error: D-Bus error: "No connection to bus", :system
make[3]: *** [Makefile:328: net/tramp-archive.elc] Error 1
```

Re: Build failure: Re: tramp "too many open files"

[Michael Albinus]

Madhu <enometh@meer.net> writes:

> Hello

Hi,

> This now results in a build failure -- on master a7dcc0d55c6
>
> ```
> ../../lisp/net/tramp-adb.el:231:22: Warning: the function ‘tramp-compat-string-replace’ might not be defined at runtime.
> ../../lisp/net/tramp-adb.el:215:6: Warning: the function ‘tramp-run-real-handler’ might not be defined at runtime.
>   ELC      net/tramp-archive.elc
>
> In toplevel form:
> ../../lisp/net/tramp-archive.el:113:2: Error: D-Bus error: "No connection to bus", :system
> make[3]: *** [Makefile:328: net/tramp-archive.elc] Error 1
> ```

This looks like you have copied one (or more) Tramp files from its
repository into the Emacs tree. That doesn't work, there are subtle
dependencies between the files of a Tramp release.

If you want to use the new functionality, you must install Tramp from
its git repository in parallel, and adapt load-path accordingly. See
(info "(tramp) Obtaining TRAMP")

Or you use Emacs cloned from its git repository.

Best regards, Michael.

Re: {Spam?} Re: Build failure: Re: tramp "too many open files"

[Madhu]

* Michael Albinus <87353z75i7.fsf @gmx.de> :
Wrote on Sun, 14 May 2023 09:10:24 +0200:
> Madhu <enometh@meer.net> writes:
>> This now results in a build failure -- on master a7dcc0d55c6
>> ```
>> ../../lisp/net/tramp-adb.el:231:22: Warning: the function
>> ‘tramp-compat-string-replace’ might not be defined at runtime.
>> ../../lisp/net/tramp-adb.el:215:6: Warning: the function
>> ‘tramp-run-real-handler’ might not be defined at runtime.
>>   ELC      net/tramp-archive.elc
>>
>> In toplevel form:
>> ../../lisp/net/tramp-archive.el:113:2: Error: D-Bus error: "No
>> connection to bus", :system
>> make[3]: *** [Makefile:328: net/tramp-archive.elc] Error 1
>> ```
>
> This looks like you have copied one (or more) Tramp files from its
> repository into the Emacs tree. That doesn't work, there are subtle
> dependencies between the files of a Tramp release.
>
> If you want to use the new functionality, you must install Tramp from
> its git repository in parallel, and adapt load-path accordingly. See
> (info "(tramp) Obtaining TRAMP")
>
> Or you use Emacs cloned from its git repository.

Hmm, This was on regular* checkout of the emacs master.  I don't think
there are tramp files from the repository involved here.  (I did do a
git checkout of the tramp repo from a month ago, but I'm sure its not
in any paths related to this emacs)

(* local patches, none related to tramp)

To produce the error I built emacs without a system bus (as mentioned
upthread) I haven't looked too closely but I was able to work around it
with.

```
diff --git a/lisp/net/tramp-gvfs.el b/lisp/net/tramp-gvfs.el
index e3b42acfed5..31d563f0b89 100644
--- a/lisp/net/tramp-gvfs.el
+++ b/lisp/net/tramp-gvfs.el
@@ -2537,7 +2537,8 @@ tramp-gvfs-parse-device-names
   (let ((tramp-verbose 0)
        tramp-gvfs-dbus-event-vector fun)
     (when (and (autoload 'zeroconf-init "zeroconf")
-              (tramp-compat-funcall 'dbus-get-unique-name :system))
+              (ignore-errors
+              (tramp-compat-funcall 'dbus-get-unique-name :system)))
       ;; Add completion functions for services announced by DNS-SD.
       ;; See <http://www.dns-sd.org/ServiceTypes.html> for valid service types.
       (zeroconf-init tramp-gvfs-zeroconf-domain)
```

I haven't tried building without a session bus yet.

Re: Build failure: Re: tramp "too many open files"

[Michael Albinus]

Madhu <enometh@meer.net> writes:

Hi,

> To produce the error I built emacs without a system bus (as mentioned
> upthread) I haven't looked too closely but I was able to work around it
> with.
>
> ```
> diff --git a/lisp/net/tramp-gvfs.el b/lisp/net/tramp-gvfs.el
> index e3b42acfed5..31d563f0b89 100644
> --- a/lisp/net/tramp-gvfs.el
> +++ b/lisp/net/tramp-gvfs.el
> @@ -2537,7 +2537,8 @@ tramp-gvfs-parse-device-names
>    (let ((tramp-verbose 0)
>         tramp-gvfs-dbus-event-vector fun)
>      (when (and (autoload 'zeroconf-init "zeroconf")
> -              (tramp-compat-funcall 'dbus-get-unique-name :system))
> +              (ignore-errors
> +              (tramp-compat-funcall 'dbus-get-unique-name :system)))
>        ;; Add completion functions for services announced by DNS-SD.
>        ;; See <http://www.dns-sd.org/ServiceTypes.html> for valid service types.
>        (zeroconf-init tramp-gvfs-zeroconf-domain)
> ```

Thanks. I have pushed a fix to master, slightly different from your proposal.

Best regards, Michael.

bug#56606: Acknowledgement (29.0.50; recent master fails with "creating pipe: too many open files")

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sun, 17 Jul 2022 02:42:04 -0700
>> 
>> On emacs-devel, Eli said:
>> 
>> AFAICT, this loop in deactivate_process:
>> 
>>   /* Beware SIGCHLD hereabouts.  */
>> 
>>   for (i = 0; i < PROCESS_OPEN_FDS; i++)
>>     close_process_fd (&p->open_fd[i]);
>> 
>> doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
>> calls in make-pipe-process.  It calls 'close' with the right value,
>> and close returns zero, but the pipe stays open.  In Emacs 28, this
>> same loop successfully closes the descriptor.  I don't know why.
>> 
>> Perhaps bisecting could help.
>
> I think commit a81669c could be the culprit.

git bisect agrees.



-- 
-- Stephe

bug#56606: 29.0.50; recent master fails with "creating pipe: too many open files"

[Stephen Leake]

-- 
-- Stephe