Re: igc, macOS avoiding signals

[Pip Cet via "Emacs development discussions." <emacs-devel@gnu.org>]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Thu, 02 Jan 2025 12:34:58 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>> Cc: Stefan Kangas <stefankangas@gmail.com>, gerd.moellmann@gmail.com, eller.helmut@gmail.com, emacs-devel@gnu.org
>>
>> > We don't know the answer, AFAIU.  We could tell them that if they
>> > prefer a patch, we can send one.
>>
>> I see no requirement to modify MPS, so far.
>
> There's no requirement, but it would be silly, IMO, to try to solve
> these issues without ever talking to the MPS developers.  We have
> nothing to lose, but it's possible that they will point out other

(Or we'll point them to a discussion which accuses MPS of being
"unreasonable" and pretty much demands MPS is fundamentally changed to
match our alleged requirements, and they'll get the wrong idea and we'll
lose whatever goodwill we might have had.)

> solutions.  Why give up that up front?  It makes no sense to me.

I never said you shouldn't talk to the MPS developers!

I'm not going to unless I can present them with a patch that I could at
least imagine I'd consider if positions were reversed (unless I were
asked to do so, which seems unlikely :-) ).  I think it would be good to
treat their time as valuable enough not to simply point them to a very
long thread, but that's just my opinion.

>> >> If we do decide to contact them, I'm afraid that I don't have sufficient
>> >> context to accurately describe the proposed callback.  I would need to
>> >> ask someone to summarize the idea in sufficient detail so that we can
>> >> start a conversation.
>> >
>> > The problem is that evidently (at least on Posix platforms), if a
>> > program that uses MPS runs application code from a SIGPROF or a
>> > SIGALRM or a SIGCHLD signal handler can trigger a recursive access to
>> > the MPS arena,
>>
>> Let's be specific here: it's about accessing MPS memory, not about
>> allocating memory.
>
> I agree.  But then I didn't say anything about allocations.

(... that's what "being unspecific" means, isn't it?)

>> > which causes a fatal signal if that happens while MPS
>> > holds the arena lock.
>>
>> I don't know what "fatal signal" is supposed ot mean in this case, to
>> the MPS folks.
>
> It's an accepted terminology, but we could explain if needed.  My text
> was for Stefan (who does know what "fatal signal" means), not for
> quoting it verbatim in a message to MPS.

As I explained, it's also not the only problem.  A "fatal signal" is the
best of three possible undesirable outcomes.

>> > So we want to ask for a callback when MPS is
>> > about to lock the arena, and another callback immediately after it
>> > releases the lock.
>>
>> That's the first time I hear about the "about to lock the arena"
>> callback.  Wouldn't hurt, of course, but it's also a new idea.
>
> It was always the idea.

I'm sorry, but I really don't see how you can make that statement.  Your
most recent proposal said nothing about that part of your idea, see:

https://mail.gnu.org/archive/html/emacs-devel/2024-12/msg01537.html

If you expected me to be smart enough to read that email and conclude
that you want another callback which would block signals, and you meant
"unblock the signal" when you wrote "run the handler's body", I'm not.

>> "Immediately", of course, may be misleading: if another thread is
>> waiting for the lock, the lock will not be available until that thread
>> is done with it.
>
> Which other thread?  There can be only one Lisp thread running at any
> given time.

I thought we agreed we should trigger GC from a separate POSIX thread
for at least some builds (to debug things).  Anyway, I don't see why we
should present a solution for a locking problem that assumes things are
single-threaded anyway.

>> We could block the appropriate signals before (in some cases, quite a
>> while before) we take the arena lock and unblock them after we release
>> it.  That's not obviously the best solution, but it's the only one this
>> change would enable, AFAICS.
>
> The problem is, we don't know when the arena will be locked, thus the
> request for a callback.

The problem is that these precise callbacks would enable ONLY this
solution, while a different callback mechanism might enable others.

I also think it would be better to simply set a custom lock/unlock
function which is run INSTEAD of the lockix.c code, which would give us
options to fine-tune the behavior in case of lock contention (it's
perfectly okay to run signal handlers "while" trying to grab the lock,
which potentially takes a while.  They might finish, or they might need
the arena lock before we do.  Most importantly, that would give us a
timeout mechanism for interrupting lengthy scans for user interaction.

I think this is highly relevant; either we need to split long vectors,
or we need a way to interrupt scanning, and this is precisely the point
where we could do so.  If all we have is a pre-lock callback, we can't
interrupt scanning, and then we're forced to split long vectors...).

The main problem, to me, is that the single-lock design of MPS might
change to a multi-lock design, and what do we do then?  Do we need
per-thread per-lock storage, or does the per-thread signal mask suffice?

>> > Alternatively, if MPS already has a solution for such applications
>> > that use signals, we'd like to hear what they suggest.
>>
>> That's a useful question to ask, of course.  My understanding is that
>> MPS is configured mostly at build time, and your idea would amount to
>> creating a replacement for lockix.c and lockw3.c which allows blocking
>> signals.
>
> Once again, let's hear what the MPS developers have to say about that.

As I said, whoever wants to hit "send" can do so.  Not me, for now.

>> > As background, you can point them to this discussion:
>> >
>> >   https://lists.gnu.org/archive/html/emacs-devel/2024-06/msg00568.html
>>
>> I think the polite thing to do would be to agree on a short but accurate
>> summary of what it is we want, explaining why it would be helpful (it
>> may simplify things but isn't required for correctness).
>
> That discussion has several backtraces which might be useful for them
> to better understand the issue.

It also contains the "unreasonable" thing, IIRC, and it's quite long.
But, again, my consent is certainly not required :-)

All that said, while I don't think it's the best idea, if you insist on
this MPS change and do the signal-blocking thing, that would, at least,
finally settle the question.

Strictly speaking, we can merge without even splitting long vectors
(some people will decide to give it a try, create a billion-entry
vector, observe the totally unusable Emacs session that leaves them
with, and decide MPS isn't ready).

Pip