From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Philipp Newsgroups: gmane.emacs.devel Subject: Re: master 8dcb19f 4/4: Add a unit test testing interaction between threads and processes. Date: Sun, 28 Feb 2021 19:30:46 +0100 Message-ID: <6C5E64C7-19A6-4D46-AC30-33C29FA5DEBB@gmail.com> References: <83k0s34eo1.fsf@gnu.org> <83h7n74cwe.fsf@gnu.org> <838s8i4ak8.fsf@gnu.org> Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="6370"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Philipp Stephani , emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Feb 28 19:32:05 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lGQrM-0001Z0-Qn for ged-emacs-devel@m.gmane-mx.org; Sun, 28 Feb 2021 19:32:04 +0100 Original-Received: from localhost ([::1]:36986 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lGQrL-0001V9-Qd for ged-emacs-devel@m.gmane-mx.org; Sun, 28 Feb 2021 13:32:03 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37048) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lGQqD-0000yG-L2 for emacs-devel@gnu.org; Sun, 28 Feb 2021 13:30:53 -0500 Original-Received: from mail-wr1-x431.google.com ([2a00:1450:4864:20::431]:40612) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lGQqA-0007ZN-Fp; Sun, 28 Feb 2021 13:30:53 -0500 Original-Received: by mail-wr1-x431.google.com with SMTP id d11so13735454wrj.7; Sun, 28 Feb 2021 10:30:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/qEUeQQ+VeL8OaJryIQAiuINJFZWwhqE46ovLV0KNAI=; b=ntTEOthtWpj8/QtDCH8Mxdojs4Jwv09427lGf3GLQ7/IvJR2398fRLAtW2wtphsqg4 41Owm1gsrSNc2xDscvAZhdML2obNxZQN2HyGJpXg23Pk+WN3QDZ9bcHZRyCYGWP2cKqW yvtYGvakNbyN02PeqnsmDR9b3pw7CRYoje1JfIJ8WEXtlGEHNNdXDDwya7SupC2tFw8R GA2GR0TR9LhiRLbqwhL6lamEUBMRN1YzIgU03uK3StfUNa3PrNfSjNf6QeHxa4eSDYR1 FNdLEcU4hi41GMe6gXqHPmrnOQq2do86LnbMQDWPbZxdfKe1aYDYINyuCEwfJCgvPDss RdHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/qEUeQQ+VeL8OaJryIQAiuINJFZWwhqE46ovLV0KNAI=; b=Mqu9gYyQdgSsJkPa7o7AxH8F6hTtl17ban1WMNHZHhuvJJCuooGJqo1UxUK7wpdYgP 8JFSTKGjBjNmIkSIwe3X6BxHGkBb2h6WxTpv5r4R7Ofb2xTDRHpZuzIlm8aTMKNVs6Aw vxHkqonE1VnHrnRQ8jPxX+Y25/BSCPuvOukJoN125YF/urZXiandfdsVpEi5vqPYlJ7y KwhJ4THkG8EmkQk7YTkFwPvbQV3Q2WPE8VDj7p9+ArzhxqMhlhLhfIc+agvBuayss5We ytS4Q4vqhDWolbv0KOf5GFyKeZ5E72ij2JmDAmcWTxHoJ59wBQ9V6E4ItE7zT5g/aeNq rv0w== X-Gm-Message-State: AOAM533T1TKvK87n184odhFtti2BTjmFjXCgOorkDBSQUvmQvHc6EGgj p/d8pA7/wJp4mA6kVDMDWNEV7zd3abv4JQ== X-Google-Smtp-Source: ABdhPJx00/Qnk29NBv+HwxW+TWsgpf/JR2C/P7a6cMHZs65y4lwkx3zR2ulqZww/1r6Khsksd24aRw== X-Received: by 2002:adf:8547:: with SMTP id 65mr13276323wrh.269.1614537047891; Sun, 28 Feb 2021 10:30:47 -0800 (PST) Original-Received: from philipps-mbp.fritz.box ([46.128.208.19]) by smtp.gmail.com with ESMTPSA id j125sm19721519wmb.44.2021.02.28.10.30.47 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 28 Feb 2021 10:30:47 -0800 (PST) In-Reply-To: <838s8i4ak8.fsf@gnu.org> X-Mailer: Apple Mail (2.3654.60.0.2.21) Received-SPF: pass client-ip=2a00:1450:4864:20::431; envelope-from=p.stephani2@gmail.com; helo=mail-wr1-x431.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:265756 Archived-At: > Am 24.01.2021 um 16:30 schrieb Eli Zaretskii : >=20 >> From: Philipp Stephani >> Date: Sun, 24 Jan 2021 14:22:57 +0100 >> Cc: Philipp Stephani , Emacs developers = >> Right now we don't have that many good tests that test the = interaction >> between threads and processes, so the intention of this test is to >> improve coverage in this area. >=20 > The general intent is clear and welcome; it's the details that I'm > interested in, because I need to know which, if any, parts of the > Windows emulation of the sub-process support need to be improved or > fixed. The specific scenario here is: Start a number of process and a number of = threads, each of which waits for one of the processes; then, cause the = processes to exit (since they run =E2=80=9Ecat=E2=80=9C, that works by = sending EOF); then, join all the threads. This should work without = hangs or errors. >=20 >>> There are some things the code does whose rationale I >>> don't understand. For example, why the call to set-process-thread: >>> what's the purpose? >>=20 >> The test wouldn't work without it, because it calls >> accept-process-output from a thread different from the thread in = which >> the process was created. >=20 > But set-process-thread accepts a thread argument, so you could bind > the process to a particular thread, couldn't you? Yes, but the threads that call `accept-process-output=E2=80=99 and = `process-send-eof=E2=80=99 are different threads; binding the process = object to one of the threads would cause the other call to fail. >=20 > More generally, my reading of the code is that the first thread which > calls accept-process-output will start listening on all the file > descriptors of all the sub-processes which were started until then, > and all the other threads will have no descriptors left to listen on. > Is this the situation you intended to create? No, but I=E2=80=99m not convinced that that=E2=80=99s actually how the = implementation behaves. Wouldn=E2=80=99t with the current = implementation each thread wait for the file descriptors of one process? > And if so, is this an > interesting practical use case, or are we just testing implementation > details? The goal here is definitely to test the observable behavior; the = implementation should follow from that. My interpretation of the Elisp = manual is that the test should succeed without errors (but unfortunately = the manual is still a bit vague on these matters). >=20 >>> Or why no-conversion in make-process? >>=20 >> Unit tests should typically test only one aspect, or a small number = of >> related aspects of the system under test. The goal of this test is = not >> to test the encoding procedures, so I turn off encoding here. >=20 > Thanks, so I now understand that the encoding stuff is not important > (since the subprocess doesn't produce any output, it cannot have any > effect, AFAIU). Yes. I=E2=80=99d be fine removing the :coding argument, but = `no-conversion=E2=80=99 seems to be the simplest choice. >=20 >>> More >>> generally, what is the basic idea of the test and the expectation >>> from each thread. >>=20 >> I don't understand this question. The idea is to have a test that >> calls accept-process-output from multiple threads to increase = coverage >> in this area. >=20 > Why is it important to test this situation, and what do we expect to > happen in it? Is it reasonable to have more than one thread wait for > output from a subprocess? The tests shouldn=E2=80=99t ask what=E2=80=99s reasonable, but what=E2=80=99= s documented or expected. Surely users can write code like this, and I = don=E2=80=99t see why that shouldn=E2=80=99t be supported. >=20 >>>> thread-join should yield. >>>=20 >>> Isn't that too late? The processes have exited already, so what = does >>> that test? >>=20 >> Again, I don't understand this question. Each unit test is a runnable >> example: by writing it, we document some aspect about how Emacs Lisp >> should behave, and by running it, we can find bugs (mismatches = between >> the intended and the actual behavior) and regressions. >=20 > The question I ask myself is what to do with the deviant behavior on > MS-Windows. If it behaves differently because what we test here are > the peculiarities of the semi-broken Unix mechanism known as "signals" > and/or some details of our implementation of subprocesses on Posix > platforms, then maybe there's no need to change anything in the > Windows code, and just either skip this test on MS-Windows or write a > different test which will work there. IOW, in such a case there's no > bug in the Windows port, it's just that it isn't bug-for-bug > compatible with Unix. I don=E2=80=99t think so. Again, I wrote the test based on my = understanding of the expected and promised behavior of the process and = thread functions, not the Posix implementation. >=20 > But if what happens in this test is important in practical use cases, > then perhaps our Windows code does need some changes in this area. >=20 > This is why I'm trying to understand the semantics of this test -- I'd > like to figure out which implementation details are important, and > thus should be emulated on Windows, and which are just details that > can be different (and portable Lisp programs will have to take those > differences into account). The way I see it, all aspects of the test are important =E2=80=94 their = behavior should be guaranteed by the implementation. >=20 > For example, the test verifies that each process exited with zero > status, but it doesn't care which thread detected that -- is this > important? IIUC threads don=E2=80=99t really =E2=80=9Edetect=E2=80=9C process exit. = What I=E2=80=99d assume is that the `thread-join=E2=80=99 calls are = synchronization points =E2=80=94 after calling `thread-join=E2=80=99 the = thread in question is guaranteed to have finished, and since the thread = has been waiting for a process to finish (using the = accept-process-output loop), the process is also guaranteed to have = finished, and this sequence of event is guaranteed exactly in that = order. At least that=E2=80=99s roughly the meaning of such = synchronization points in other languages. Since the process is = guaranteed to have finished, it will necessarily need to have the right = exit status stored in its process object, and since =E2=80=9Ecat=E2=80=9C = finishes with a zero exit status in case its stdin gets closed, that = must be zero. > Or what about delivery of SIGCHLD -- is this important > which thread gets delivered the signal, or how many threads should the > signal wake up? Not as long as the observable behavior stays the same (and isn=E2=80=99t = buggy). > Or is it important when and due to what reason(s) > does each thread exit? It=E2=80=99s important that each thread exits normally, i.e., due to its = thread function body exiting normally. It=E2=80=99s not important when = exactly the thread finishes, but the observable order of events matters: = first process-send-eof, then process exit, then thread exit, then = thread-join returns.