From mboxrd@z Thu Jan 1 00:00:00 1970
Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail
From: Philipp
Newsgroups: gmane.emacs.devel
Subject: Re: master 8dcb19f 4/4: Add a unit test testing interaction between
threads and processes.
Date: Sun, 28 Feb 2021 19:30:46 +0100
Message-ID: <6C5E64C7-19A6-4D46-AC30-33C29FA5DEBB@gmail.com>
References: <83k0s34eo1.fsf@gnu.org> <83h7n74cwe.fsf@gnu.org>
<838s8i4ak8.fsf@gnu.org>
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\))
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214";
logging-data="6370"; mail-complaints-to="usenet@ciao.gmane.io"
Cc: Philipp Stephani , emacs-devel@gnu.org
To: Eli Zaretskii
Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Feb 28 19:32:05 2021
Return-path:
Envelope-to: ged-emacs-devel@m.gmane-mx.org
Original-Received: from lists.gnu.org ([209.51.188.17])
by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.92)
(envelope-from )
id 1lGQrM-0001Z0-Qn
for ged-emacs-devel@m.gmane-mx.org; Sun, 28 Feb 2021 19:32:04 +0100
Original-Received: from localhost ([::1]:36986 helo=lists1p.gnu.org)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from )
id 1lGQrL-0001V9-Qd
for ged-emacs-devel@m.gmane-mx.org; Sun, 28 Feb 2021 13:32:03 -0500
Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37048)
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from )
id 1lGQqD-0000yG-L2
for emacs-devel@gnu.org; Sun, 28 Feb 2021 13:30:53 -0500
Original-Received: from mail-wr1-x431.google.com ([2a00:1450:4864:20::431]:40612)
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from )
id 1lGQqA-0007ZN-Fp; Sun, 28 Feb 2021 13:30:53 -0500
Original-Received: by mail-wr1-x431.google.com with SMTP id d11so13735454wrj.7;
Sun, 28 Feb 2021 10:30:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:subject:from:in-reply-to:date:cc
:content-transfer-encoding:message-id:references:to;
bh=/qEUeQQ+VeL8OaJryIQAiuINJFZWwhqE46ovLV0KNAI=;
b=ntTEOthtWpj8/QtDCH8Mxdojs4Jwv09427lGf3GLQ7/IvJR2398fRLAtW2wtphsqg4
41Owm1gsrSNc2xDscvAZhdML2obNxZQN2HyGJpXg23Pk+WN3QDZ9bcHZRyCYGWP2cKqW
yvtYGvakNbyN02PeqnsmDR9b3pw7CRYoje1JfIJ8WEXtlGEHNNdXDDwya7SupC2tFw8R
GA2GR0TR9LhiRLbqwhL6lamEUBMRN1YzIgU03uK3StfUNa3PrNfSjNf6QeHxa4eSDYR1
FNdLEcU4hi41GMe6gXqHPmrnOQq2do86LnbMQDWPbZxdfKe1aYDYINyuCEwfJCgvPDss
RdHw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
:content-transfer-encoding:message-id:references:to;
bh=/qEUeQQ+VeL8OaJryIQAiuINJFZWwhqE46ovLV0KNAI=;
b=Mqu9gYyQdgSsJkPa7o7AxH8F6hTtl17ban1WMNHZHhuvJJCuooGJqo1UxUK7wpdYgP
8JFSTKGjBjNmIkSIwe3X6BxHGkBb2h6WxTpv5r4R7Ofb2xTDRHpZuzIlm8aTMKNVs6Aw
vxHkqonE1VnHrnRQ8jPxX+Y25/BSCPuvOukJoN125YF/urZXiandfdsVpEi5vqPYlJ7y
KwhJ4THkG8EmkQk7YTkFwPvbQV3Q2WPE8VDj7p9+ArzhxqMhlhLhfIc+agvBuayss5We
ytS4Q4vqhDWolbv0KOf5GFyKeZ5E72ij2JmDAmcWTxHoJ59wBQ9V6E4ItE7zT5g/aeNq
rv0w==
X-Gm-Message-State: AOAM533T1TKvK87n184odhFtti2BTjmFjXCgOorkDBSQUvmQvHc6EGgj
p/d8pA7/wJp4mA6kVDMDWNEV7zd3abv4JQ==
X-Google-Smtp-Source: ABdhPJx00/Qnk29NBv+HwxW+TWsgpf/JR2C/P7a6cMHZs65y4lwkx3zR2ulqZww/1r6Khsksd24aRw==
X-Received: by 2002:adf:8547:: with SMTP id 65mr13276323wrh.269.1614537047891;
Sun, 28 Feb 2021 10:30:47 -0800 (PST)
Original-Received: from philipps-mbp.fritz.box ([46.128.208.19])
by smtp.gmail.com with ESMTPSA id j125sm19721519wmb.44.2021.02.28.10.30.47
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Sun, 28 Feb 2021 10:30:47 -0800 (PST)
In-Reply-To: <838s8i4ak8.fsf@gnu.org>
X-Mailer: Apple Mail (2.3654.60.0.2.21)
Received-SPF: pass client-ip=2a00:1450:4864:20::431;
envelope-from=p.stephani2@gmail.com; helo=mail-wr1-x431.google.com
X-Spam_score_int: -17
X-Spam_score: -1.8
X-Spam_bar: -
X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001,
RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: emacs-devel@gnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: "Emacs development discussions."
List-Unsubscribe: ,
List-Archive:
List-Post:
List-Help:
List-Subscribe: ,
Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org
Original-Sender: "Emacs-devel"
Xref: news.gmane.io gmane.emacs.devel:265756
Archived-At:
> Am 24.01.2021 um 16:30 schrieb Eli Zaretskii :
>=20
>> From: Philipp Stephani
>> Date: Sun, 24 Jan 2021 14:22:57 +0100
>> Cc: Philipp Stephani , Emacs developers =
>> Right now we don't have that many good tests that test the =
interaction
>> between threads and processes, so the intention of this test is to
>> improve coverage in this area.
>=20
> The general intent is clear and welcome; it's the details that I'm
> interested in, because I need to know which, if any, parts of the
> Windows emulation of the sub-process support need to be improved or
> fixed.
The specific scenario here is: Start a number of process and a number of =
threads, each of which waits for one of the processes; then, cause the =
processes to exit (since they run =E2=80=9Ecat=E2=80=9C, that works by =
sending EOF); then, join all the threads. This should work without =
hangs or errors.
>=20
>>> There are some things the code does whose rationale I
>>> don't understand. For example, why the call to set-process-thread:
>>> what's the purpose?
>>=20
>> The test wouldn't work without it, because it calls
>> accept-process-output from a thread different from the thread in =
which
>> the process was created.
>=20
> But set-process-thread accepts a thread argument, so you could bind
> the process to a particular thread, couldn't you?
Yes, but the threads that call `accept-process-output=E2=80=99 and =
`process-send-eof=E2=80=99 are different threads; binding the process =
object to one of the threads would cause the other call to fail.
>=20
> More generally, my reading of the code is that the first thread which
> calls accept-process-output will start listening on all the file
> descriptors of all the sub-processes which were started until then,
> and all the other threads will have no descriptors left to listen on.
> Is this the situation you intended to create?
No, but I=E2=80=99m not convinced that that=E2=80=99s actually how the =
implementation behaves. Wouldn=E2=80=99t with the current =
implementation each thread wait for the file descriptors of one process?
> And if so, is this an
> interesting practical use case, or are we just testing implementation
> details?
The goal here is definitely to test the observable behavior; the =
implementation should follow from that. My interpretation of the Elisp =
manual is that the test should succeed without errors (but unfortunately =
the manual is still a bit vague on these matters).
>=20
>>> Or why no-conversion in make-process?
>>=20
>> Unit tests should typically test only one aspect, or a small number =
of
>> related aspects of the system under test. The goal of this test is =
not
>> to test the encoding procedures, so I turn off encoding here.
>=20
> Thanks, so I now understand that the encoding stuff is not important
> (since the subprocess doesn't produce any output, it cannot have any
> effect, AFAIU).
Yes. I=E2=80=99d be fine removing the :coding argument, but =
`no-conversion=E2=80=99 seems to be the simplest choice.
>=20
>>> More
>>> generally, what is the basic idea of the test and the expectation
>>> from each thread.
>>=20
>> I don't understand this question. The idea is to have a test that
>> calls accept-process-output from multiple threads to increase =
coverage
>> in this area.
>=20
> Why is it important to test this situation, and what do we expect to
> happen in it? Is it reasonable to have more than one thread wait for
> output from a subprocess?
The tests shouldn=E2=80=99t ask what=E2=80=99s reasonable, but what=E2=80=99=
s documented or expected. Surely users can write code like this, and I =
don=E2=80=99t see why that shouldn=E2=80=99t be supported.
>=20
>>>> thread-join should yield.
>>>=20
>>> Isn't that too late? The processes have exited already, so what =
does
>>> that test?
>>=20
>> Again, I don't understand this question. Each unit test is a runnable
>> example: by writing it, we document some aspect about how Emacs Lisp
>> should behave, and by running it, we can find bugs (mismatches =
between
>> the intended and the actual behavior) and regressions.
>=20
> The question I ask myself is what to do with the deviant behavior on
> MS-Windows. If it behaves differently because what we test here are
> the peculiarities of the semi-broken Unix mechanism known as "signals"
> and/or some details of our implementation of subprocesses on Posix
> platforms, then maybe there's no need to change anything in the
> Windows code, and just either skip this test on MS-Windows or write a
> different test which will work there. IOW, in such a case there's no
> bug in the Windows port, it's just that it isn't bug-for-bug
> compatible with Unix.
I don=E2=80=99t think so. Again, I wrote the test based on my =
understanding of the expected and promised behavior of the process and =
thread functions, not the Posix implementation.
>=20
> But if what happens in this test is important in practical use cases,
> then perhaps our Windows code does need some changes in this area.
>=20
> This is why I'm trying to understand the semantics of this test -- I'd
> like to figure out which implementation details are important, and
> thus should be emulated on Windows, and which are just details that
> can be different (and portable Lisp programs will have to take those
> differences into account).
The way I see it, all aspects of the test are important =E2=80=94 their =
behavior should be guaranteed by the implementation.
>=20
> For example, the test verifies that each process exited with zero
> status, but it doesn't care which thread detected that -- is this
> important?
IIUC threads don=E2=80=99t really =E2=80=9Edetect=E2=80=9C process exit. =
What I=E2=80=99d assume is that the `thread-join=E2=80=99 calls are =
synchronization points =E2=80=94 after calling `thread-join=E2=80=99 the =
thread in question is guaranteed to have finished, and since the thread =
has been waiting for a process to finish (using the =
accept-process-output loop), the process is also guaranteed to have =
finished, and this sequence of event is guaranteed exactly in that =
order. At least that=E2=80=99s roughly the meaning of such =
synchronization points in other languages. Since the process is =
guaranteed to have finished, it will necessarily need to have the right =
exit status stored in its process object, and since =E2=80=9Ecat=E2=80=9C =
finishes with a zero exit status in case its stdin gets closed, that =
must be zero.
> Or what about delivery of SIGCHLD -- is this important
> which thread gets delivered the signal, or how many threads should the
> signal wake up?
Not as long as the observable behavior stays the same (and isn=E2=80=99t =
buggy).
> Or is it important when and due to what reason(s)
> does each thread exit?
It=E2=80=99s important that each thread exits normally, i.e., due to its =
thread function body exiting normally. It=E2=80=99s not important when =
exactly the thread finishes, but the observable order of events matters: =
first process-send-eof, then process exit, then thread exit, then =
thread-join returns.