From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: "Basil L. Contovounesios" Newsgroups: gmane.emacs.bugs Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output Date: Thu, 11 Oct 2018 15:57:50 +0100 Message-ID: <87r2gwbmma.fsf@tcd.ie> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: blaine.gmane.org 1539269836 22778 195.159.176.226 (11 Oct 2018 14:57:16 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Thu, 11 Oct 2018 14:57:16 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) To: 33018@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Thu Oct 11 16:57:12 2018 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gAcOl-0005oL-PL for geb-bug-gnu-emacs@m.gmane.org; Thu, 11 Oct 2018 16:57:12 +0200 Original-Received: from localhost ([::1]:34961 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gAcQs-0000S9-Cr for geb-bug-gnu-emacs@m.gmane.org; Thu, 11 Oct 2018 10:59:22 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:54542) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gAcQf-0000Qs-FF for bug-gnu-emacs@gnu.org; Thu, 11 Oct 2018 10:59:10 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gAcQY-00045u-Ht for bug-gnu-emacs@gnu.org; Thu, 11 Oct 2018 10:59:07 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:41126) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gAcQY-00044A-7G for bug-gnu-emacs@gnu.org; Thu, 11 Oct 2018 10:59:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gAcQY-0002VD-2K for bug-gnu-emacs@gnu.org; Thu, 11 Oct 2018 10:59:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: "Basil L. Contovounesios" Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 11 Oct 2018 14:59:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 33018 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.15392698969565 (code B ref -1); Thu, 11 Oct 2018 14:59:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 11 Oct 2018 14:58:16 +0000 Original-Received: from localhost ([127.0.0.1]:45384 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gAcPo-0002UC-4b for submit@debbugs.gnu.org; Thu, 11 Oct 2018 10:58:16 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:44674) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gAcPm-0002Tz-6l for submit@debbugs.gnu.org; Thu, 11 Oct 2018 10:58:14 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gAcPf-00036G-Fd for submit@debbugs.gnu.org; Thu, 11 Oct 2018 10:58:08 -0400 Original-Received: from lists.gnu.org ([2001:4830:134:3::11]:38962) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gAcPf-00036A-BT for submit@debbugs.gnu.org; Thu, 11 Oct 2018 10:58:07 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:54269) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gAcPe-0008MZ-8I for bug-gnu-emacs@gnu.org; Thu, 11 Oct 2018 10:58:07 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gAcPa-00033L-6H for bug-gnu-emacs@gnu.org; Thu, 11 Oct 2018 10:58:06 -0400 Original-Received: from mail-ed1-x530.google.com ([2a00:1450:4864:20::530]:33923) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gAcPZ-000325-Oz for bug-gnu-emacs@gnu.org; Thu, 11 Oct 2018 10:58:02 -0400 Original-Received: by mail-ed1-x530.google.com with SMTP id w19-v6so8564032eds.1 for ; Thu, 11 Oct 2018 07:58:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tcd-ie.20150623.gappssmtp.com; s=20150623; h=from:to:subject:message-id:user-agent:date:mime-version; bh=1fAPAhyfsrUs5ys8GTfE2w//twZLn/LwhXUXVmkrhME=; b=s75hc5ndvYO5rLDmY1xU6kJ/nxe3s/9nHTZDwtNwElZgOMkooJYeT+sHHeld9faqBA 15PZAHmUyfLOEKq2kFFx5d6ujHIYHxQVyozv7vwUcqPkbGMWGPlODhDFOyDjDk7yQEEj xTDVcuR87f/BajP5lQHNI2/fTLfW8bL9AL4ZDFve0tsZCSaQwUcw3w6PYTTweNBRwk5n QUOcN0ahSUNvwia2vVDLAjJeM+yMpFAVfO5AOy8ZObFcIC/fjlEAL1Q33VdvhwviQo14 g+MrrfK7ur0nnzUapuf9aqVFjVro7V79abVHpaeFNlBx+4XSEu1gJY7NCdLR0tLAMoOT NvTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:message-id:user-agent:date :mime-version; bh=1fAPAhyfsrUs5ys8GTfE2w//twZLn/LwhXUXVmkrhME=; b=QkxoebHKjuOL+Y/nvnqpcTbFuSj2HivvsaWmkL/xbQf9IcbDz+7tt9ViVaw414jhGI pEyi3shja+PHFQg6cdsMOSx7l2Ao1MxPeZOU0yfBiOA+DFoYvdVV8iJRXWB5Z1Q7nQUi lhJ6aa16apnWzR6GvzlTAJZaaF85LtwrVhOEVgGfdwap4md8jOTg07s5EBa6qQ/bXGfT RwGIErFuB/4FwCRe1uBC4Xuh/KRk63n7k+4JZcpq+EB+zoQE7J3t1ahxyzA2siGfIH04 pP3ek0KZFmouhB0T1pg6wNAVIyHPsAAQZwvYzDf+KxlmaElpf1HB05Q9VKSgfMSlZTc7 612g== X-Gm-Message-State: ABuFfogYs0GzgFpXYeT1OFSjbsrLAY88CoCdgEKULa60ctQ2QRbNRyqD +eFfsx+k2tRit8GEWHQWZP4f4lc1OWA= X-Google-Smtp-Source: ACcGV62L3Mwrs7M430dKlpa4OJ9vrxxbpyen6pIvSZvlhQGiWAisz1HCUvcBNaXXxBhDCdZyJ8WrTA== X-Received: by 2002:a17:906:5d10:: with SMTP id g16-v6mr2650055ejt.168.1539269878370; Thu, 11 Oct 2018 07:57:58 -0700 (PDT) Original-Received: from localhost ([213.233.148.8]) by smtp.gmail.com with ESMTPSA id q12-v6sm9823462edd.35.2018.10.11.07.57.57 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 11 Oct 2018 07:57:57 -0700 (PDT) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:151127 Archived-At: --=-=-= Content-Type: application/emacs-lisp Content-Disposition: attachment; filename=test.el Content-Transfer-Encoding: quoted-printable ;;; test.el --- test -*- lexical-binding: t -*- ;;; Code: (defun test-debug (&rest args) "Print `current-thread' and ARGS in timestamped `message'." (push (current-thread) args) (message "[%s] %s" (format-time-string "%T:%3N") (mapconcat #'prin1-to-string args " "))) (defun test-sentinel (proc msg) "Kill buffer of PROC when it exits." (test-debug proc 'sentinel (process-status proc) (substring msg 0 -1)) (and (not (eq (process-status proc) 'run)) (buffer-live-p (process-buffer proc)) (kill-buffer (process-buffer proc)))) (defun test-slave () "Create an asynchronous process and wait for it to exit." (let ((proc (make-process :name "test" :buffer (generate-new-buffer " *test*") :command '("wget" "-qO-" "https://en.wikipedia.org/wiki/Emac= s") :connection-type 'pipe :sentinel #'test-sentinel))) (while (eq (process-status proc) 'run) (test-debug proc 'accept-output (accept-process-output proc 5))) (test-debug proc 'exit (process-status proc) (process-exit-status proc)= ))) (defun test-master (async) "Successively run `test-slave'. With non-nil ASYNC, use a new thread for each invocation." (dotimes (i 5) (if async (thread-join (make-thread #'test-slave (format "test-%d" i))) (test-slave)))) (defun test-job (async) "Successively run `test-slave'. Like `test-master', but avoid blocking interactive sessions by running `test-master' in its own thread." (if noninteractive (test-master async) (make-thread (apply-partially #'test-master async) "test-master"))) (defun test-threads () "Run `test-master' with threads." (test-job t)) (defun test-no-threads () "Run `test-master' without threads." (test-job nil)) ;;; test.el ends here --=-=-= Content-Type: text/plain I attach a sample program test.el whose central function, test-slave, invokes wget asynchronously before waiting for the process to exit. The issue I'm facing is that running test-slave twice in succession, each time in a new thread, causes accept-process-output to hang with no output (unless a timeout argument is given, in which case the function returns nil) the second time around. When this happens, the process sentinel is never called, which is why I'm assuming accept-process-output is indeed "hanging" in some sense and it's not just that the process has already exited and so has no further output. I could very well be doing or assuming something incorrectly, but what baffles me is that the "hang" does not occur either when Emacs is run non-interactively, or when "https://en.wikipedia.org/wiki/Emacs" is replaced with "https://www.gnu.org/software/emacs/", or when test-slave is run in the current thread (and not in make-thread). Since I can reliably reproduce this on both an optimised build of master and a non-optimised build of emacs-26, I hope to be able to provide further insights using gdb as time allows. Please let me know if there are any specific details/output you would like me to provide. As a relatively inexperienced gdb user I welcome any tips and tricks for debugging threads and processes. Here are some ways test.el can be run to illustrate the issue: # All five processes exit successfully. emacs -batch -l test.el -f test-no-threads # All five processes (and threads) exit successfully. emacs -batch -l test.el -f test-threads # All five processes exit successfully. emacs -Q -l test.el -f test-no-threads # First process and thread exit successfully, # but accept-process-output starts timing out in second thread. # Warning: may leave empty wget-log files lying around. emacs -Q -l test.el -f test-threads Details of the two Emacs versions I'm using follow: In GNU Emacs 26.1.50 (build 2, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars) of 2018-10-11 built on thunk Repository revision: a7ebc6bf633bd3849ccab032dad6b1fd31b1ef43 Windowing system distributor 'The X.Org Foundation', version 11.0.12001000 System Description: Debian GNU/Linux testing (buster) Configured using: 'configure 'CC=ccache gcc' 'CFLAGS=-O0 -g3 -ggdb -gdwarf-4 -pipe' --config-cache --prefix=/home/blc/.local --program-suffix=26 --enable-checking=yes,glyphs --enable-check-lisp-object-type --with-mailutils --with-x-toolkit=lucid --with-modules --with-file-notification=yes --with-x' Configured features: XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD LCMS2 In GNU Emacs 27.0.50 (build 21, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars) of 2018-10-11 built on thunk Repository revision: 5bd8cfc14d4b0c78c07e65a583f42a10c4cbc06d Windowing system distributor 'The X.Org Foundation', version 11.0.12001000 System Description: Debian GNU/Linux buster/sid Configured using: 'configure --config-cache --prefix=/home/blc/.local --with-mailutils --with-x-toolkit=lucid --with-modules --with-file-notification=yes --with-x 'CC=ccache gcc' 'CFLAGS=-O2 -march=native -pipe'' Configured features: XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON LCMS2 GMP Thanks, -- Basil --=-=-=--