From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: npostavs@users.sourceforge.net Newsgroups: gmane.emacs.bugs Subject: bug#25549: 25.1; eshell grep gives inconsistent output Date: Thu, 02 Feb 2017 22:26:57 -0500 Message-ID: <87mve4osxq.fsf@users.sourceforge.net> References: <87tw8lzdaj.fsf@it-alanm.williamson-tn.org> <87mve8q7vl.fsf_-_@users.sourceforge.net> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: blaine.gmane.org 1486092375 28926 195.159.176.226 (3 Feb 2017 03:26:15 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 3 Feb 2017 03:26:15 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) Cc: me@alandmoore.com, 25549@debbugs.gnu.org To: Tijs Mallaerts Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Fri Feb 03 04:26:11 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cZUVk-0007BK-6d for geb-bug-gnu-emacs@m.gmane.org; Fri, 03 Feb 2017 04:26:08 +0100 Original-Received: from localhost ([::1]:60014 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cZUVn-0003ul-QF for geb-bug-gnu-emacs@m.gmane.org; Thu, 02 Feb 2017 22:26:11 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:47901) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cZUVh-0003tO-Iv for bug-gnu-emacs@gnu.org; Thu, 02 Feb 2017 22:26:06 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cZUVe-0000Q1-DZ for bug-gnu-emacs@gnu.org; Thu, 02 Feb 2017 22:26:05 -0500 Original-Received: from debbugs.gnu.org ([208.118.235.43]:56168) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cZUVe-0000Pl-7j for bug-gnu-emacs@gnu.org; Thu, 02 Feb 2017 22:26:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1cZUVd-0008RL-W7 for bug-gnu-emacs@gnu.org; Thu, 02 Feb 2017 22:26:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: npostavs@users.sourceforge.net Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 03 Feb 2017 03:26:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 25549 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: confirmed Original-Received: via spool by 25549-submit@debbugs.gnu.org id=B25549.148609235732433 (code B ref 25549); Fri, 03 Feb 2017 03:26:01 +0000 Original-Received: (at 25549) by debbugs.gnu.org; 3 Feb 2017 03:25:57 +0000 Original-Received: from localhost ([127.0.0.1]:54367 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cZUVZ-0008R0-5K for submit@debbugs.gnu.org; Thu, 02 Feb 2017 22:25:57 -0500 Original-Received: from mail-it0-f65.google.com ([209.85.214.65]:34745) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cZUVW-0008Qf-U2; Thu, 02 Feb 2017 22:25:55 -0500 Original-Received: by mail-it0-f65.google.com with SMTP id o185so547639itb.1; Thu, 02 Feb 2017 19:25:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=1vSf3Til1cRhNnQovrcgRceOrFtyI97L2HmzfkuR8j0=; b=YIfnDZVJzyv9jeOLco7dhOR6Y9NHBd6806M6jGFnT9SnNTs4r5wIl6g9A/bZySXzP+ 4xr/ZFWr9sQGh8lK30gCV9QOPUexTlw4JjyCy8yrXOGH7AOT7S+X+SXWgJv+acqpkG7z PWy18i3xn+KyYbCXVCBSfydcEDV1VNibJs2+Al6sPD6qGdQWSqdU8xehvn3Kf/CbW+mW TfXhEXb6QBN00naS7fn+J0pl6iS8Wezu5kPz4M4Z25ETYSEKwawwm1A+Up2S2oMmMHh0 BRxUcDgUZRcrwFShZLDbzmUnEALlpfM6EsQ5w+4BdvSQd81Yuib0pVoux80fSbHq9StN RsBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=1vSf3Til1cRhNnQovrcgRceOrFtyI97L2HmzfkuR8j0=; b=AulBaG/WNd5k9AJUiPuRxUCDY+j/M4TWpZpcnMZpFBXhODkeohriUM212oKUqmI5mq 1T+bacviqIUX8Sw6ozWzwrIV5O+nIsGxcBRTGPain2dnF5TFgjiEfwwKDIF9/kjpBJ+8 CKhg709nDA24yV9SBbPJ80PBnKj0te1NZj5bp+a+61g7wacJ4zpxazONFFYRn8+8aAgA hVhADjRG/nJXhTI1jrPl8Osxn507d9b8/k3356QCRTbV62VQkEd7lpcVOUujgVOh6odM JRKJpAwpMK1Uo6nArNRgsBu+9Jt8wgaWbzweI20GMuH5UFtMRZAzPKpZDEO5B+hNITJh 9isQ== X-Gm-Message-State: AIkVDXKelx3leIql3G7SE5L4uAugdMiZpOGWKGUCsEQ9cy4yzyuk8YOF/VxhuPPGxkJdbw== X-Received: by 10.36.112.137 with SMTP id f131mr628534itc.26.1486092349273; Thu, 02 Feb 2017 19:25:49 -0800 (PST) Original-Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id e24sm331513itd.6.2017.02.02.19.25.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 02 Feb 2017 19:25:48 -0800 (PST) In-Reply-To: <87mve8q7vl.fsf_-_@users.sourceforge.net> (npostavs@users.sourceforge.net's message of "Mon, 30 Jan 2017 21:29:50 -0500") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:128902 Archived-At: --=-=-= Content-Type: text/plain tags 25549 patch quit npostavs@users.sourceforge.net writes: > > Tijs Mallaerts writes: > >> On my machine following steps seem to be a reproducible test case: >> >> - create a file "test-file.txt" with 20.000 identical lines with content "This is a line." >> - open eshell and insert the command: cat test-file.txt | grep line | wc >> >> This seems to return a different result every time it's run. > > Thanks, I can reproduce with this. The problem can be more easily reproduced (i.e., more often, and only 3000 lines) by cat test-file.txt | sleepy-cat | wc where sleepy-cat is #!/bin/sh while read line ; do echo "$line" sleep 0.000001 done The problem happens when one of the commands in the pipeline sends its output to Emacs quickly and the next command in the pipeline is slower. On receiving data from the first command in eshell-insertion-filter we call eshell-output-object to send it to the next command, but since sending might block, Emacs can run other process filters and sentinels instead. In this case, while sending a data chunk from cmd1 to cmd2, we actually end up reading all the data from cmd1 until it terminates and we call its sentinel. The sentinel closes the pipes and sends EOF to cmd2, but we still haven't sent the data from cmd1 to cmd2 yet. Closing the pipes in a timer, as in the patch below, seems to fix it for me. --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=v1-0001-Make-sure-eshell-pipelines-don-t-drop-data.patch Content-Description: patch >From a1757a7114fbc20733554d8aa44cea0fa8a991e2 Mon Sep 17 00:00:00 2001 From: Noam Postavsky Date: Thu, 2 Feb 2017 09:19:43 -0500 Subject: [PATCH v1] Make sure eshell pipelines don't drop data * lisp/eshell/esh-proc.el (eshell-sentinel): If called while still handling output of the process, make sure to close the pipes only later, so that the next process in the pipeline recieves EOF only after getting all its input (Bug#25549). --- lisp/eshell/esh-proc.el | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/lisp/eshell/esh-proc.el b/lisp/eshell/esh-proc.el index b0dbb22..ba5cb5c 100644 --- a/lisp/eshell/esh-proc.el +++ b/lisp/eshell/esh-proc.el @@ -393,8 +393,20 @@ eshell-sentinel (unless (string= string "run") (unless (string-match "^\\(finished\\|exited\\)" string) (eshell-insertion-filter proc string)) - (eshell-close-handles (process-exit-status proc) 'nil - (cadr entry)))) + (let ((handles (nth 1 entry)) + (str (prog1 (nth 3 entry) + (setf (nth 3 entry) nil))) + (status (process-exit-status proc))) + ;; If we're in the middle of handling output + ;; from this process then schedule the EOF for + ;; later. + (letrec ((finish-io + (lambda () + (if (nth 4 entry) + (run-at-time 0 nil finish-io) + (when str (eshell-output-object str nil handles)) + (eshell-close-handles status 'nil handles))))) + (funcall finish-io))))) (eshell-remove-process-entry entry)))) (eshell-kill-process-function proc string))))) -- 2.9.3 --=-=-=--