Re: Shrinking the C core

Re: Shrinking the C core

[Arthur Miller]

>> Date: Wed, 9 Aug 2023 21:19:11 -0400
>> From: "Eric S. Raymond" <esr@thyrsus.com>
>> Cc: emacs-devel@gnu.org
>> 
>> Po Lu <luangruo@yahoo.com>:
>> > "Eric S. Raymond" <esr@thyrsus.com> writes:
>> > 
>> > > When I first worked on Emacs code in the 1980s Lisp was already fast
>> > > enough, and machine speeds have gone up by something like 10^3 since.
>> > > I plain don't believe the "slower" part can be an issue on modern
>> > > hardware, not even on tiny SBCs.
>> > 
>> > Can you promise the same, if your changes are not restricted to one or
>> > two functions in fileio.c, but instead pervade throughout C source?
>> 
>> Yes, in fact, I can. Because if by some miracle we were able to
>> instantly rewrite the entirety of Emacs in Python (which I'm not
>> advocating, I chose it because it's the slowest of the major modern
>> scripting languages) basic considerations of clocks per second would
>> predict it to run a *dead minimum* of two orders of magnitude faster
>> than the Emacs of, say, 1990.
>> 
>> And 1990 Emacs was already way fast enough for the human eye and
>> brain, which can't even register interface lag of less than 0.17
>> seconds (look up the story of Jef Raskin and how he exploited this
>> psychophysical fact in the design of the Canon Cat sometime; it's very
>> instructive). The human auditory system can perceive finer timeslices,
>> down to about 0.02s in skilled musicians, but we're not using elisp
>> for audio signal processing.
> 
> This kind of argument is inherently flawed: it's true that today's
> machines are much faster than those in, say, 1990, but Emacs nowadays
> demands much more horsepower from the CPU than it did back then.
> What's more, Emacs is still a single-threaded Lisp machine, although
> in the last 10 years CPU power develops more and more in the direction
> of multiple cores and execution units, with single execution units
> being basically as fast (or as slow) today as they were a decade ago.
> 
> And if these theoretical arguments don't convince you, then there are
> facts: the Emacs display engine, for example, was completely rewritten
> since the 1990s, and is significantly more expensive than the old one
> (because it lifts several of the gravest limitations of the old
> redisplay).  Similarly with some other core parts and internals.
> 
> We are trying to make Lisp programs faster all the time, precisely
> because users do complain about annoying delays and slowness.  Various
> optimizations in the byte-compiler and the whole native-compilation
> feature are parts of this effort, and are another evidence that the
> performance concerns are not illusory in Emacs.  And we are still not
> there yet: people still do complain from time to time, and not always
> because someone selected a sub-optimal algorithm where better ones
> exist.
> 
> The slowdown caused by moving one primitive to Lisp might be
> insignificant, but these slowdowns add up and eventually do show in
> user-experience reports.  Rewriting code in Lisp also increases the GC
> pressure, and GC cycles are known as one of the significant causes of
> slow performance in quite a few cases.  We are currently tracking the
> GC performance (see the emacs-gc-stats@gnu.org mailing list) for that
> reason, in the hope that we can modify GC and/or its thresholds to
> improve performance.
> 
>> If you take away nothing else from this conversation, at least get it
>> through your head that "more Lisp might make Emacs too slow" is a
>> deeply, *deeply* silly idea. It's 2023 and the only ways you can make
>> a user-facing program slow enough for response lag to be noticeable
>> are disk latency on spinning rust, network round-trips, or operations
>> with a superlinear big-O in critical paths.  Mere interpretive overhead
>> won't do it.
> 
> We found this conclusion to be false in practice, at least in Emacs
> practice.
> 
>> > Finally, you haven't addressed the remainder of the reasons I itemized.
>> 
>> They were too obvious, describing problems that competent software
>> engineers know how to prevent or hedge against, and you addressed me
>> as though I were a n00b that just fell off a cabbage truck. My
>> earliest contributions to Emacs were done so long ago that they
>> predated the systematic Changelog convention; have you heard the
>> expression "teaching your grandmother to suck eggs"?  My patience for
>> that sort of thing is limited.
> 
> Please be more patient, and please consider what others here say to be
> mostly in good-faith and based on non-trivial experience.  If
> something in what others here say sounds like an offense to your
> intelligence, it is most probably a misunderstanding: for most people
> here English is not their first language, so don't expect them to
> always be able to find the best words to express what they want to
> say.

Very interesting discussion going on for a long time.

I think you are all correct, and wrong to an extent, but I believe that
nobody has touched the fundamental issue: emacs design is flawed beyond
repair for todays machines. Not necessarily in pejorative meaning, but
to repair Emacs you would have to significantly rework internals, to the
point of entire design rewrite. Emacs is a child of its time (like
everything else). It was designed for the time of single-core slow
machine, and its design makes sense in that perspective. However, for
todays multicore machines, the fact that a lisp machine is slapped on
top of an existing text editor (Gosslings I guess), and everything is
shared via global state, can't be addressed in any other way but to
rewrite Emacs core from ground up. No amount of patch slapping onto the
current design can compensate for the lack of appropriate desing.

RMS has one abandoned TECO design to adapt Emacs to new times; perhaps
the time has come to do it again.

Sure, you can rework the C core completely, and implement better GC,
threads, and what not, but it seems signficant less work to re-implement
Emacs in a language that already has proper infrastructure in the place,
such as Common Lisp on ccl/sbcl, instead of duplicating all the work.

I understand machines of the time were to slow for a language as big as
Common Lisp when Emacs switched from TECO to C, but today the situation
is different. I am quite sure that sbcl would offer enough speed for a
text editor implementation. As you all have already noticed, speed is
limited by a human factor anyway.

We also see many projects that are duplicating Emacs idea, and with all
right; the ideas of Emacs are indeed sound and very good. But it is a
shame to abandon 40 years of development and the community that Emacs
has, something other projects can't compete with. Emacs Lisp is de
facto, *the* best documented Lisp out there, and Emacs is one of the
best documented open source projects available, kudos for that to
everyone involved. It would be a pittiful waste to just throw away all
that documentation and work done, as well as the vivid community that
Emacs has. In my personal opinon, those are *the* qualities of Emacs as
a project.

I personally believe that Emacs as a project would benefit of complete
core rewrite, as well as unifying the extension and the implementation
language for several reasons. Yes, it is a lot of work, but even
rewriting C core to adapt Emacs to modern machines is a lot of work in
itself, if it is even possible. You can either let it be, and slowly but
surely see Emacs usage decline, or you can rewrite Emacs core (again) to
better suit modern multicore machines.

Re: Shrinking the C core

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Sun, 27 Aug 2023 17:14:41 +0200
> 
> I personally believe that Emacs as a project would benefit of complete
> core rewrite, as well as unifying the extension and the implementation
> language for several reasons. Yes, it is a lot of work, but even
> rewriting C core to adapt Emacs to modern machines is a lot of work in
> itself, if it is even possible.

Sure, this has been said here several times.

Two problems with this:

  . we need volunteer(s) to actually do the work, and the requirements
    for their talents are hard to meet; and
  . the rewrite needs to keep all or most of Lisp still working, which
    is IMNSHO almost impossible, because our Lisp is written for the
    current design

I leave the conclusions as an exercise ;-)

Re: Shrinking the C core

[Gerd Möllmann]

>  . we need volunteer(s) to actually do the work, and the requirements
>     for their talents are hard to meet; and

 From my POV, the gist of the problem is money. Firefox has it, Emacs 
doesn't.  With dozens of paid developers, which I think Firefox has, big 
projects like a rewrite or what else was mentioned, can be tackled. 
With volunteers, not so much.

Re: Shrinking the C core

[Arthur Miller]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>>  . we need volunteer(s) to actually do the work, and the requirements
>>     for their talents are hard to meet; and
>
> From my POV, the gist of the problem is money. Firefox has it, Emacs doesn't.
> With dozens of paid developers, which I think Firefox has, big projects like a
> rewrite or what else was mentioned, can be tackled. With volunteers, not so
> much.

Definitely. 

Re: Shrinking the C core

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Sun, 27 Aug 2023 17:14:41 +0200
>> 
>> I personally believe that Emacs as a project would benefit of complete
>> core rewrite, as well as unifying the extension and the implementation
>> language for several reasons. Yes, it is a lot of work, but even
>> rewriting C core to adapt Emacs to modern machines is a lot of work in
>> itself, if it is even possible.
>
> Sure, this has been said here several times.
>
> Two problems with this:
>
>   . we need volunteer(s) to actually do the work, and the requirements
>     for their talents are hard to meet; and
>   . the rewrite needs to keep all or most of Lisp still working, which
>     is IMNSHO almost impossible, because our Lisp is written for the
>     current design

If you implement your current design on the another platform, it would still
work the same, no?

By the way; I am sure you will have to implement quite feelable changes in the
current design regardless. Common Lisp implementation is just food for
thought. I think it was an interesting discussion, so I am just throwing out a
different idea I haven't seen around. I don't think is impossible, but I don't
say it is easy or trivial either. :)

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It was designed for the time of single-core slow
  > machine, and its design makes sense in that perspective. 

That is true.

                                                             However, for
  > todays multicore machines, the fact that a lisp machine is slapped on
  > top of an existing text editor (Gosslings I guess),

No, that's not how I developed it.

I first wrote Emacs Lisp as a standalone Lisp interpreter.
Then I wrote the code to handle buffers and editing the text in them.

Turning to the question of future development,
I think the design of Emacs is good enough,
To rewrite Emacs completely would be an enormous misdirected effort.

GNU Emacs is a part of the GNU opersating system,
whose goal is to provide, in the Free World, all the features
that people want.

With the same work it would take to rewrite Emacs from scratch,
we could write some other complex and powerful free program
which does some other job -- a job that there is no free software to do.

Or we could add features -- to Emacs or some other free program -- which
would greatly extend what people can do in the Free World.

Consider, for instance, the idea of extending a free spell checking
program so it could tell you the gender of each noun if you ask.
Going beyond that, maybe it could check for agreement of modifiers
with their nouns.  That would be a big help for writing in many
Indo-European languages and Semitic languages.  Perhaps also other
languages which have something grammatically like gender but not
based on gender of humans.

That would be a tiny fraction of the work that people are proposing
here.  And it would help a lot of users.

Would someone like to do it?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > It was designed for the time of single-core slow
>   > machine, and its design makes sense in that perspective. 
>
> That is true.
>
>                                                              However, for
>   > todays multicore machines, the fact that a lisp machine is slapped on
>   > top of an existing text editor (Gosslings I guess),
>
> No, that's not how I developed it.
>
> I first wrote Emacs Lisp as a standalone Lisp interpreter.
> Then I wrote the code to handle buffers and editing the text in them.

Ah; I thought I remembered either from some of your articles or some of
earlier disucssions that you obtained a copy of GE from a friend, and
realized it needed a proper Lisp interpreter. Cool to hear a bit of
history, thank you.

> Turning to the question of future development,
> I think the design of Emacs is good enough,
> To rewrite Emacs completely would be an enormous misdirected effort.

No, I don't suggest to rewrite entire Emacs. On contrary I am
conservative about keeping Elisp as-is, as I said, it would be a waste
of human effort to throw away 40 years of work. No, on the contrary, and
that is why I suggested what I did, otherwise I would go to some greener
editor and use that; if I didn't found all the old work important, so to
say.

I suggest to re-implement the Lisp API implemented in C, in Common
Lisp. I understand that you didn't like Common Lisp back in the time
because it was big and slow for the computers at that time; but nowadays
things have changed. SBCL and CCL are two quite good compilers and would
give access to more advanced Lisp Machines than what one in Emacs is,
better garbage collector, native threading, namespaces, and so on.

> GNU Emacs is a part of the GNU opersating system,
> whose goal is to provide, in the Free World, all the features
> that people want.

Well, yes, I totally understand and agree. What I see from the
interaction with the Emacs community on social media, or at least on
Reddit where most of non-dev community seems to be, people want features
that are not in Emacs. Those mentioned above come up quite
often.

But it is about more than just features. It is about the widening the
community. Free world is bigger than just Emacs, as you seem to imply
yourself. Lisp communities are small, I think it is a pitty that we
can't natively use Common Lisp libraries. If we had Emacs implemented in
Common Lisp, we would get access to lots more libraries and tools than
what is in Elisp packages only. More eyes would be looking at Emacs too.

If extension and implementation language were the same, than more people
would potentially get enabled to contribute to Emacs. Of course, just
having the core in a hackable Lisp language does not mean everyone
*will* contribute or be able to contribute, but potentially, there might
be one or another individual that would. It might encourage more
experimentation and make it easier to test new ideas.

The speed of Emacs development would potentially increase; developing in
Lisp is much faster compared to hack, recompile, start Emacs
cycle. Don't get me wrong, I don't mean hacking C is hard, impenetrable,
undoable or so, I am just saying it could be smoother and faster.

People also do ask for better tools that enable "programming at large"
in Emacs. We do use Emacs for more than just editing text at least some
of us; it has become an automation tool. People also seems to want to
use Emacs and Emacs Lisp as a more general programming tool to solve
other problems, not just to extend the text editor. I think it is a good
thing.

> With the same work it would take to rewrite Emacs from scratch,

I see I have expressed myself exaggerated; I wrote the mail in just 3-4
minutes of fast typing; I should have probably explained myself
better. I meant that certain parts of Emacs have to be re-designed from
the scrath, things that prohibit multithreading, the shared state. It
does not necessary mean, that I suggest to throw away entire Emacs and
rewrite everything from scratch.

> we could write some other complex and powerful free program
> which does some other job -- a job that there is no free software to
> do.

Or we could improve Emacs and make it a powerful tool in another areas
where it is not as used.

For example, I think Emacs would be great as a teaching tool, I think I
have said that before on this list. I see concept of buffers in Emacs
Lisp, or rather to say, concept of editor implemented in the programming
language as a really good and useful tool. I mean it. Common Lisp
(probably some before it) has given us a repl as a mandatory tool in the
language. Emacs Lisp shows us that an editor as part of the environment
or language is as important as repl. I do miss edebug and seing the
result in another buffer when stepping through when I program with
sbcl. I wish Common Lisp had buffers and editor built-in. Implementing
Emacs in Common Lisp, would in a way, merge those two, and perhaps Emacs
could be used as another Processing or as a teaching tool where kids can
learn programming by interactive development, have the environment
avaialable for introspection, step through and see the results as they
execute the code and similar.

> Or we could add features -- to Emacs or some other free program -- which
> would greatly extend what people can do in the Free World.
>
> Consider, for instance, the idea of extending a free spell checking
> program so it could tell you the gender of each noun if you ask.
> Going beyond that, maybe it could check for agreement of modifiers
> with their nouns.  That would be a big help for writing in many
> Indo-European languages and Semitic languages.  Perhaps also other
> languages which have something grammatically like gender but not
> based on gender of humans.

I have no familiarity with spell-checking so I can't reflect on that one. I
struggled to install Hunspell in my Emacs untill Zaretsky helped me some
10 years ago or so :). But what you suggest seems to me like you would
need to find an expert for each language in question, which suggests
rather social skills than programming skills. I am more for solving math
problems after eveyone else is at sleep, so I can't help you with that one.

> That would be a tiny fraction of the work that people are proposing
> here.

I think on contrary; I propose to switch Emacs to a different Lisp
implementation so that devs here don't have to re-implement and maintain
some important parts of the Lisp Machine if Emacs will get features lots
of people seem to ask for. While what I suggest is a volumous work as
well, it also save a lot of work elsewhere, and opens some other
possibilities.

By the way, a single threaded shared state does not thread itself just
because it is transplanted over a multithreaded implementation :). Of
course. But getting access to such platform enables people to use
threads, while the implementation itself is tranformed, as well as buys
other benefits as discussed above.

Thanks for the kind answer, and I am hope I am not take wrong; I don't
mean Emacs is bad, and certainly notthing wrong. Actually I am very fond
of Emacs, I find it very useful tool, and would just like to see it ever
better. 

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> By the way, a single threaded shared state does not thread itself just
> because it is transplanted over a multithreaded implementation :). Of
> course. But getting access to such platform enables people to use
> threads, while the implementation itself is tranformed, as well as buys
> other benefits as discussed above.

I'll reiterate this again: facilitating the execution of Lisp threads in
parallel -- a more or less complete job -- is 5% of the work required
for a concurrent Emacs.

The tedious, insipid task of interlocking all that ``single threaded
shared state'' follows and constitutes the bulk of that work, which
someone must sign up for irrespective of the language it is written in.

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> By the way, a single threaded shared state does not thread itself just
>> because it is transplanted over a multithreaded implementation :). Of
>> course. But getting access to such platform enables people to use
>> threads, while the implementation itself is tranformed, as well as buys
>> other benefits as discussed above.
>
> I'll reiterate this again: facilitating the execution of Lisp threads in
> parallel -- a more or less complete job -- is 5% of the work required
> for a concurrent Emacs.
>
> The tedious, insipid task of interlocking all that ``single threaded
> shared state'' follows and constitutes the bulk of that work, which
> someone must sign up for irrespective of the language it is written in.

Perhaps you are attacking the task from the wrong angle? I think that
sounds like a very tedius and error prone strategy. Why not refactor the
interpreter so you can instantiate it with an empty state in a separate
thread? I don't know; I am perhaps wrong, but isn't it more safe
strategy to isolate the state in their own threads instead of trying to
interloc bunch of stuff all over the place?

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> Perhaps you are attacking the task from the wrong angle? I think that
> sounds like a very tedius and error prone strategy. Why not refactor
> the interpreter so you can instantiate it with an empty state in a
> separate thread? I don't know; I am perhaps wrong, but isn't it more
> safe strategy to isolate the state in their own threads instead of
> trying to interloc bunch of stuff all over the place?

How is that distinct from running Emacs within a subprocess?

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Perhaps you are attacking the task from the wrong angle? I think that
>> sounds like a very tedius and error prone strategy. Why not refactor
>> the interpreter so you can instantiate it with an empty state in a
>> separate thread? I don't know; I am perhaps wrong, but isn't it more
>> safe strategy to isolate the state in their own threads instead of
>> trying to interloc bunch of stuff all over the place?
>
> How is that distinct from running Emacs within a subprocess?

It is not, but it is not a question :).

It is basically same as what async package does. I have actually said
that in defense of Emacs many times in disucssions when people ask for
"worker threads" on Reddit.

However, it is a bit more efficient if you could start an interpretter
in a thread than Emacs process; and it will lead to refactoring out Lisp
interpretter from the Emacs application state which will hopefully
lessen the shared state and make it easier to analyze the locking stuff
you talk about.

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> Very interesting discussion going on for a long time.
>
> I think you are all correct, and wrong to an extent, but I believe that
> nobody has touched the fundamental issue: emacs design is flawed beyond
> repair for todays machines. Not necessarily in pejorative meaning, but
> to repair Emacs you would have to significantly rework internals, to the
> point of entire design rewrite. Emacs is a child of its time (like
> everything else). It was designed for the time of single-core slow
> machine, and its design makes sense in that perspective. However, for
> todays multicore machines, the fact that a lisp machine is slapped on
> top of an existing text editor (Gosslings I guess), and everything is
> shared via global state, can't be addressed in any other way but to
> rewrite Emacs core from ground up. No amount of patch slapping onto the
> current design can compensate for the lack of appropriate desing.

Unix was designed for 16-bit uniprocessor machines, where the only form
of ``interlocking'' was:

   int x = spltty ();
   splx (x);

But today, both free BSD Unix and proprietary Unix scale to SMPs with
hundereds of processors, exploiting intricate interlocking around
individual kernel data structures.  The perfect antithesis to your
standpoint...

Re: Shrinking the C core

[chad]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Sun, Aug 27, 2023 at 9:42 PM Po Lu <luangruo@yahoo.com> wrote:

> But today, both free BSD Unix and proprietary Unix scale to SMPs with
> hundereds of processors, exploiting intricate interlocking around
> individual kernel data structures.  The perfect antithesis to your
> standpoint...
>

An old friend of mine did this work for NetBSD as his master's thesis at
MIT. Rebuilding an emacs would, at a very rough estimate, be no more work,
likely less. He had the benefit of being aware of other attempts to add SMP
support to other unixen, though.

One major problem with building a truly multi-core-optimized Emacs is that
so much of the core concepts revolve around shared state, especially
buffers. I *think* there's a workable design similar to a modern web
browser with moderate-to-high sandboxing and worker threads, but someone
would have to care enough about each of Emacs, modern browser
implementation (including javascript and sandboxing), and language
implementation to puzzle out the details. I don't know of such a person
(but it would be a bit surprising if I did!), and I don't have the same
sort of contact with grad students anymore. My hope is that there's still a
window for such a person, and they haven't all been swallowed up by VSCode
and co-pilot.

[-- Attachment #2: Type: text/html, Size: 1697 bytes --]

Re: Shrinking the C core

[Emanuel Berg]

Po Lu wrote:

> Unix was designed for 16-bit uniprocessor machines, where
> the only form of ``interlocking'' was:
>
>    int x = spltty ();
>    splx (x);
>
> But today, both free BSD Unix and proprietary Unix

BSD and proprietary Unix, what about Linux?

> scale to SMPs with hundereds of processors, exploiting
> intricate interlocking around individual kernel
> data structures.

I agree with those who say rewriting Emacs to get SMP would be
to burn down the house to kill the rats, but is that the only
way to get SMP?

Cannot the "single-thread Lisp machine" instead be modified
and extended from its current implementation, so that it would
support SMP?

If that is the future of computing, I think there is no
staying in the past, so it is rather a question of how to get
it in the best way possible.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Very interesting discussion going on for a long time.
>>
>> I think you are all correct, and wrong to an extent, but I believe that
>> nobody has touched the fundamental issue: emacs design is flawed beyond
>> repair for todays machines. Not necessarily in pejorative meaning, but
>> to repair Emacs you would have to significantly rework internals, to the
>> point of entire design rewrite. Emacs is a child of its time (like
>> everything else). It was designed for the time of single-core slow
>> machine, and its design makes sense in that perspective. However, for
>> todays multicore machines, the fact that a lisp machine is slapped on
>> top of an existing text editor (Gosslings I guess), and everything is
>> shared via global state, can't be addressed in any other way but to
>> rewrite Emacs core from ground up. No amount of patch slapping onto the
>> current design can compensate for the lack of appropriate desing.
>
> Unix was designed for 16-bit uniprocessor machines, where the only form
> of ``interlocking'' was:
>
>    int x = spltty ();
>    splx (x);
>
> But today, both free BSD Unix and proprietary Unix scale to SMPs with
> hundereds of processors, exploiting intricate interlocking around
> individual kernel data structures.  The perfect antithesis to your
> standpoint...

In which way is it "the perfect antithesis" to my standpoint, and what
is my standpoint?

If you wrote:

"where the only form of ``interlocking'' **is**",

than it would certainly be a "perfect anthithesis", but since you are
using the term *was* it means it *is no more*, a past tense, something
that has changed, and change is what I have suggested.

I think you are missunderstanding what I am saying: I am saying that
design needs to be changed; and the other thing is that I am suggesting
to rewrite the core API in CL instead of C, since you will get all those
things that constitute the Lisp Machine out for free; a better garbage
collector, a better threading, and better Lisp that are people are often
asking for. Of course you can implement all that stuff in C as well, it
just that it means redoing work that other people has done elsewhere.

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> In which way is it "the perfect antithesis" to my standpoint, and what
> is my standpoint?

That Emacs must be rewritten from the drawing board to leverage
multiprocessor operation.

> If you wrote:
>
> "where the only form of ``interlocking'' **is**",
>
> than it would certainly be a "perfect anthithesis", but since you are
> using the term *was* it means it *is no more*, a past tense, something
> that has changed, and change is what I have suggested.

Multiple organizations took Unix and independently transformed it into
what it is today: CMU, Novell, Sun, and of course the many volunteers
who orchestrate the development of the free BSD systems.

What fundamental issues (aside from manpower and time, of course) impede
Emacs from undergoing the same transformation?  The Unix experience
indicates that this is well-trodden ground.

> I think you are missunderstanding what I am saying: I am saying that
> design needs to be changed; and the other thing is that I am suggesting
> to rewrite the core API in CL instead of C, since you will get all those
> things that constitute the Lisp Machine out for free; a better garbage
> collector, a better threading, and better Lisp that are people are often
> asking for. Of course you can implement all that stuff in C as well, it
> just that it means redoing work that other people has done elsewhere.

Last I checked, none of the popular Common Lisp systems supported MS-DOS
or ran satisfactorily on Android, nor was Common Lisp an especially well
known language.  There is a reason Emacs is written in C, and why its
present design has endured for so long: it is portable, it is reasonably
fast, and it is a subject familiar to all programmers.

Other Lisp systems aren't panaceas either: rewritten in Common Lisp or
not, the present Emacs design will still require interlocking to operate
safely in a multiprocessor environment.  Ergo, such a rewrite will only
compound the effort needed to produce a hypothetical ``Lockmacs'' (so to
speak) with the additional pains required to rewrite Emacs in a new
language, and one very few of the present maintainers are well versed in
at that.

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> In which way is it "the perfect antithesis" to my standpoint, and what
>> is my standpoint?
>
> That Emacs must be rewritten from the drawing board to leverage
> multiprocessor operation.

I think you have missunderstand what I mean; I can try make it clear
that I mean parts that considers Emacs core, shared state and so on.

>> If you wrote:
>>
>> "where the only form of ``interlocking'' **is**",
>>
>> than it would certainly be a "perfect anthithesis", but since you are
>> using the term *was* it means it *is no more*, a past tense, something
>> that has changed, and change is what I have suggested.
>
> Multiple organizations took Unix and independently transformed it into
> what it is today: CMU, Novell, Sun, and of course the many volunteers
> who orchestrate the development of the free BSD systems.

You are missing the point: I wasn't talking about *who* changed the
Unix; the point here is that it got *changed*.

> What fundamental issues (aside from manpower and time, of course) impede
> Emacs from undergoing the same transformation?  The Unix experience
> indicates that this is well-trodden ground.

None. And nobody has suggested that such issues exists either, it is your
construction. On contrary, I am saying that Emacs should undergo such
transformation :).

>> I think you are missunderstanding what I am saying: I am saying that
>> design needs to be changed; and the other thing is that I am suggesting
>> to rewrite the core API in CL instead of C, since you will get all those
>> things that constitute the Lisp Machine out for free; a better garbage
>> collector, a better threading, and better Lisp that are people are often
>> asking for. Of course you can implement all that stuff in C as well, it
>> just that it means redoing work that other people has done elsewhere.
>
> Last I checked, none of the popular Common Lisp systems supported MS-DOS

True. MS-DOS and Windows 95 would have to go away. Perhaps some other
popular OS from 1980s-1990s? 

> or ran satisfactorily on Android, nor was Common Lisp an especially well
> known language.  There is a reason Emacs is written in C, and why its
> present design has endured for so long: it is portable, it is reasonably
> fast, and it is a subject familiar to all programmers.

Well yes, C is portable and fast, nobody denies that. So are some CL
compilers. 

> Other Lisp systems aren't panaceas either: rewritten in Common Lisp or

Nobody said it is "panaceas". But sbcl would provide a better lisp
machine, in terms of garbagecollector, threading etc; something you will
have to develop for Emacs eventually. You heard that one about a guy
with name Isaac Newton and standing on shoulders of others? :)

> not, the present Emacs design will still require interlocking to operate
> safely in a multiprocessor environment.  Ergo, such a rewrite will only
> compound the effort needed to produce a hypothetical ``Lockmacs'' (so to

Just reimplementing the C core would not, because you wouldn't use
multiple threads. But if you would like to thread Emacs command loop or
internals itself, than you would have to reimplement it that way of
course. Think of Java Swing; at least 20 years ago when I war
programming Java, it was not thread safe, but worker threads were OK.

> speak) with the additional pains required to rewrite Emacs in a new
> language, and one very few of the present maintainers are well versed in
> at that.

I am sure present maintainers are well versed with CL, just not very
interested to rewrite Emacs in CL; and I wouldn't expected them to be
either :-). 

Observe that I am presenting some food for thought; I don't expect Emacs
devs to jump and say "Yes! What a great idea!". But I wouldn't be
suprised if at least some of them have had the idea in their mind at
some point in time. 

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> I think you have missunderstand what I mean; I can try make it clear
> that I mean parts that considers Emacs core, shared state and so on.

That's what I was referring to, yes.

> You are missing the point: I wasn't talking about *who* changed the
> Unix; the point here is that it got *changed*.

My point is, Unix has demonstrated many times over that programs written
in C can be interlocked, and doing so is more productive than a rewrite.

> None. And nobody has suggested that such issues exists either, it is your
> construction. On contrary, I am saying that Emacs should undergo such
> transformation :).

That contradicts your assertions below, where you propose returning to
the drawing board to produce an Emacs written in Common Lisp.  Curiously
enough, I can't locate any precedent for large and complex Common Lisp
programs being modified for multi-processor operation.

> True. MS-DOS and Windows 95 would have to go away. Perhaps some other
> popular OS from 1980s-1990s? 

They wouldn't have to go away if Emacs remains written in C.

> Well yes, C is portable and fast, nobody denies that. So are some CL
> compilers. 

Which Common Lisp compiler is capable of linking with Java code through
the JNI?  (ABCL doesn't work, as the Android JVM can't interpret the
bytecode it generates.)

> Nobody said it is "panaceas". But sbcl would provide a better lisp
> machine, in terms of garbagecollector, threading etc; something you will
> have to develop for Emacs eventually. You heard that one about a guy
> with name Isaac Newton and standing on shoulders of others? :)

SBCL only supports kernel-based threads on a handful of systems, and its
garbage collection strategies vary between machine types.  I'm sure we
would be capable of supporting more while remaining portable, and Common
Lisp leaves much to be desired as a pair of shoulders to stand on.

> Just reimplementing the C core would not, because you wouldn't use
> multiple threads. But if you would like to thread Emacs command loop or
> internals itself, than you would have to reimplement it that way of
> course. Think of Java Swing; at least 20 years ago when I war
> programming Java, it was not thread safe, but worker threads were OK.

You're okay with Emacs crashing if threads, wielded incorrectly, trample
over internal state?  If so, the global lock could be lifted from Emacs
Lisp in under a week.

Instead of denigrating the C language, Emacs's C core, and so many other
components critical to Emacs that have been subject to unjustified
invective in recent times, why not direct your attention to a more
productive task, such as identifying the complex interdependencies
between different pieces of Emacs's global state to establish a detailed
plan for their interlocking?  For this task, the most important tool is
a control-flow visualizer such as cflow, and an ample supply of paper.

> I am sure present maintainers are well versed with CL, just not very
> interested to rewrite Emacs in CL; and I wouldn't expected them to be
> either :-). 

Two data points: I'm not an adept Common Lisp programmer, nor is Eli.
I'm not even an Emacs Lisp programmer by vocation.

Re: Shrinking the C core

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Instead of denigrating the C language, Emacs's C core, and so many other
> components critical to Emacs that have been subject to unjustified
> invective in recent times, why not direct your attention to a more
> productive task, such as identifying the complex interdependencies
> between different pieces of Emacs's global state to establish a detailed
> plan for their interlocking?  For this task, the most important tool is
> a control-flow visualizer such as cflow, and an ample supply of paper.

And we tried to dig into this question in
https://yhetil.org/emacs-devel/871qhnr4ty.fsf@localhost/
At least, the task does not look impossible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Arthur Miller]

Ihor Radchenko <yantar92@posteo.net> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Instead of denigrating the C language, Emacs's C core, and so many other
>> components critical to Emacs that have been subject to unjustified
>> invective in recent times, why not direct your attention to a more
>> productive task, such as identifying the complex interdependencies
>> between different pieces of Emacs's global state to establish a detailed
>> plan for their interlocking?  For this task, the most important tool is
>> a control-flow visualizer such as cflow, and an ample supply of paper.
>
> And we tried to dig into this question in
> https://yhetil.org/emacs-devel/871qhnr4ty.fsf@localhost/
> At least, the task does not look impossible.

That was a long thread to say mildly :) I don't have time to read all
that, but thanks for the pointer. To answer shortly.

Po has narrowed this to be a discussion about threading only, which
certainly wasn't my intention; I see it as an important part, but there
are other importnat aspects which would benefit Emacs than just
multithreading.

I generally agree with you, even with Po; I think it is possible in some
way, I never said it is impossible to rework Emacs.

For example if we could instantiate the interpretter on its own, with an
empty state, than you could perhaps run that in a separate thread, but I
am affraid everything is just one big state and would mean some
refactoring.

I am not sure if sitting and trying to puzzle pieces to interlock during
the execution as Po suggests is a very good idea. Again *anything* is
possible, but that sounds like a very tedious and error prone strategy
to me. I can't read the entire thread, there are tens if not 100s of
posts there, so I don't know what you have come up with.

However, my thesis was not that it is impossible to do in Emacs, but
that there is a lisp machine that already has figured out that work.

Re: Shrinking the C core

[Ihor Radchenko]

Arthur Miller <arthur.miller@live.com> writes:

> However, my thesis was not that it is impossible to do in Emacs, but
> that there is a lisp machine that already has figured out that work.

Unless that lisp machine supports all the Elisp concepts, it will not be
useful. So, from my point of view, what you suggest is: instead of
adding multi-threading to Elisp, add all the missing Elisp parts to
other lisp machine. I suspect that adding Elisp-unique parts in a way
that all the existing Elisp code continue working is harder compared to
modifying Elisp machine.

Feel free to prove me wrong.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Arthur Miller]

Ihor Radchenko <yantar92@posteo.net> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> However, my thesis was not that it is impossible to do in Emacs, but
>> that there is a lisp machine that already has figured out that work.
>
> Unless that lisp machine supports all the Elisp concepts, it will not be
> useful. So, from my point of view, what you suggest is: instead of
> adding multi-threading to Elisp, add all the missing Elisp parts to
> other lisp machine. I suspect that adding Elisp-unique parts in a way

Pretty much so yes.

>                     I suspect that adding Elisp-unique parts in a way
> that all the existing Elisp code continue working is harder compared to
> modifying Elisp machine.

No, I don't think so. Honestly. 

Or, as always, it depends :). Mostly on hat is considered as hard
parts. Adding missing stuff to CL is most volymous dull labour but
conceptually not very hard. There are some nasty spots like implementing
elisp reader for character handling. It was quite annoying since
character handlign is mostly just a wrapper over C and quite
inconsistent in Emacs, so it was lots of cases to cover and lots of
testing, but logically it was not a problem that needs some special
thinkering to solve it or so, if I am expressing myself clearly here

> Feel free to prove me wrong.

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> However, my thesis was not that it is impossible to do in Emacs, but
> that there is a lisp machine that already has figured out that work.

As is already customary from you, this conflates the job of running the
Lisp interpreter within different lwps with the task of making it safe
to do so, should code running in that interpreter attempt to leverage
Emacs primitives.

True, the first problem has already been solved by eminent Lisp
implementations.  But we have almost solved it as well.  And if two
threads modify a buffer simultaneously, leaving PT and PT_BYTE out of
accord, Emacs will crash regardless of whose threads it's using.

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> However, my thesis was not that it is impossible to do in Emacs, but
>> that there is a lisp machine that already has figured out that work.
>
> As is already customary from you

I'll take that one as english skill; and I really mean it in this case,
since the discussion further suggest that.

> Lisp interpreter within different lwps with the task of making it safe
> to do so, should code running in that interpreter attempt to leverage
> Emacs primitives.

I am not sure what you are trying to tell here.

> True, the first problem has already been solved by eminent Lisp
> implementations.  But we have almost solved it as well.  And if two

Well, I am very glad if you did. Honestly.

>                                                          And if two
> threads modify a buffer simultaneously, leaving PT and PT_BYTE out of
> accord, Emacs will crash regardless of whose threads it's using.

Is it possible to let one thread at a time modify the buffer (lock the
buffer) and use some notification system to tell other threads they
should update their view of buffer state when they resume, or perhaps
some "main thread" can update the buffer state a thread sees before it
is resumed?

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> Is it possible to let one thread at a time modify the buffer (lock the
                                                                ^^^^

You're describing interlocking.  Again, the bulk of the work someone
will have to perform, no matter what language we use for Emacs.

Re: Shrinking the C core

[Emanuel Berg]

Po Lu wrote:

> True, the first problem has already been solved by eminent
> Lisp implementations. But we have almost solved it as well.
> And if two threads modify a buffer simultaneously, leaving
> PT and PT_BYTE out of accord, Emacs will crash regardless of
> whose threads it's using.

Are we talking some POSIX PCB mutex/semaphore model for Emacs
so it can take advantage of modern-day multicore processors
and for example execute `let' clauses in parallel?

Or what exactly is the problem and proposed solution?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I think you have missunderstand what I mean; I can try make it clear
>> that I mean parts that considers Emacs core, shared state and so on.
>
> That's what I was referring to, yes.
>
>> You are missing the point: I wasn't talking about *who* changed the
>> Unix; the point here is that it got *changed*.
>
> My point is, Unix has demonstrated many times over that programs written
> in C can be interlocked, and doing so is more productive than a rewrite.

Unix which? Unix is not a single program; it is a family of operating
systems, and I am quite sure nobody runs the same kernel they did back in
1970. I don't know what you are talking about you have
"demonestrated". You said yourself *was* and not *is* which suggest that
those kernels are rewritten in order to support those C programs that
can be interlocked. Again you are constructing yourself things here: I
haven't said it is impossible to write mulththreaded C application that
can use locking.

There are many applicaitons that have been redesigned. Emacs has been
redesigned since its first incarnations as TECO editor if I have
understand the history. For the very last time: I haven't
said Emacs *can not be redesigned*, on the contrary, I said Emacs
*should be* redesigned. Emacs core so to say. You are fighting against
things I haven't said; I don't know why, if it is lack of English skill,
or something else, but we are talking constantly beside each other.

> in C can be interlocked, and doing so is more productive than a rewrite.

Are you saying that a non-multihtreaded program written in C can
suddenly become multithreaded without being re-written to support
threading? Nevermind, you are totally, on purpose or lack of
understanding, missing the point: you have to rewrite stuff to make it
multithreaded regardless of the language if you intention is to have
Emacs core multithreaded. Supporting multithreading is not same as
having entire Emacs internals be multithreaded.

>> None. And nobody has suggested that such issues exists either, it is your
>> construction. On contrary, I am saying that Emacs should undergo such
>> transformation :).
>
> That contradicts your assertions below, where you propose returning to

No, it does not because you have missunderstood that I mean something else.

> the drawing board to produce an Emacs written in Common Lisp.  Curiously

I propose porting core API to Common Lisp because that would save you
implementing all those things that you need to make Emacs
multithreaded, better GC etc, because it is not a trivial task.

Of course you can re-write all that stuff in C too, nobody is denying
that it is possible. I am saying there is no need to reduplicate the
work, and there are other benefits of having Emacs in Common Lisp.

> enough, I can't locate any precedent for large and complex Common Lisp
> programs being modified for multi-processor operation.

Look harder.

>> True. MS-DOS and Windows 95 would have to go away. Perhaps some other
>> popular OS from 1980s-1990s? 
>
> They wouldn't have to go away if Emacs remains written in C.

Tough argument indeed.

It is certainly very importnat to keep the development back to
support two ancient non-free OS:s.

>> Well yes, C is portable and fast, nobody denies that. So are some CL
>> compilers. 
>
> Which Common Lisp compiler is capable of linking with Java code through
> the JNI?  (ABCL doesn't work, as the Android JVM can't interpret the
> bytecode it generates.)

I don't know what you are trying to say; that sound like very uninformed
statement based on missunderstanding. Calling Java via native API and
generating bytecode that runs Lisp on top of Java as ABCL does are two
completely different and separate things. I am quite sure Emacs does not
generate Java byte code. But I don't see how is it even relevant, I
haven't suggested to run Emacs on JVM. Perhaps it is possible who know,
but I have suggested to use sbcl which is a native compiler specialized
for compiling Lisp (and runs on Android as well).

>> Nobody said it is "panaceas". But sbcl would provide a better lisp
>> machine, in terms of garbagecollector, threading etc; something you will
>> have to develop for Emacs eventually. You heard that one about a guy
>> with name Isaac Newton and standing on shoulders of others? :)
>
> SBCL only supports kernel-based threads on a handful of systems, and its

"only" ? :-) I can asure you that Lispers have access to both kernel and
user space threads. Search on Bordeaux threads.

> garbage collection strategies vary between machine types.  I'm sure we

Does it?

> would be capable of supporting more while remaining portable, and Common
> Lisp leaves much to be desired as a pair of shoulders to stand on.

I am also sure everything is possible, it is software, as a wise old
grey head once said. It is just how much effort and resource you are
pouring into it. SBCL runs on basically all important platforms on which
Emacs runs, minus as mentioned DOS and Windows95; I don't know if there is
 some other. But it is not the point. The point is: there is another
community working on the same problems you have or wish to solve. And
they have come far away than you. You could re-use their work, and if
you miss somethign add those missing parts so everyone would benefit, or
you can live in an isolated island and do everything yourself.

>> Just reimplementing the C core would not, because you wouldn't use
>> multiple threads. But if you would like to thread Emacs command loop or
>> internals itself, than you would have to reimplement it that way of
>> course. Think of Java Swing; at least 20 years ago when I war
>> programming Java, it was not thread safe, but worker threads were OK.
>
> You're okay with Emacs crashing if threads, wielded incorrectly, trample
> over internal state?  If so, the global lock could be lifted from Emacs
> Lisp in under a week.

I am telling you that sbcl/ccl has already solved those problems you
have to solve in order to have multithreading without having to lift
your lock.

> Instead of denigrating the C language, Emacs's C core, and so many other
> components critical to Emacs that have been subject to unjustified
> invective in recent times, why not direct your attention to a more

I don't think I understand how I "denigrate the C language" and what
"unjustify invective" in this case is, but I don't think it
matters. Once you also said I should be forbidden to talk about Emacs.

>                                    direct your attention to a more
> productive task, such as identifying the complex interdependencies
> between different pieces of Emacs's global state to establish a detailed
> plan for their interlocking?  For this task, the most important tool is
> a control-flow visualizer such as cflow, and an ample supply of paper.

Can we save nature and our selves?

>> I am sure present maintainers are well versed with CL, just not very
>> interested to rewrite Emacs in CL; and I wouldn't expected them to be
>> either :-). 
>
> Two data points: I'm not an adept Common Lisp programmer, nor is Eli.
> I'm not even an Emacs Lisp programmer by vocation.

Nor are you sole Emacs devs, aren't you? I don't think calling names is
appropriate, so I want, but I can come up with at least four people who
work on Emacs and who are experienced Lispers. How active they are in
development recently I don't know, I don't follow the list every day as
I am used to, but I certainly haven't targeted Eli nor you with that one.

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> Unix which? Unix is not a single program; it is a family of operating
> systems, and I am quite sure nobody runs the same kernel they did back
> in 1970. I don't know what you are talking about you have
> "demonestrated". You said yourself *was* and not *is* which suggest
> that those kernels are rewritten in order to support those C programs
> that can be interlocked. Again you are constructing yourself things
> here: I haven't said it is impossible to write mulththreaded C
> application that can use locking.

Each extant Unix system today contains gobs of code derived from the
Unix kernel developed at BTL in the 1970s.  None of their kernels were
ever rewritten -- rather, they were subject to iterative refinement that
eventually allowed them to operate on SMP systems.

> There are many applicaitons that have been redesigned. Emacs has been
> redesigned since its first incarnations as TECO editor if I have
> understand the history. For the very last time: I haven't said Emacs
> *can not be redesigned*, on the contrary, I said Emacs *should be*
> redesigned. Emacs core so to say. You are fighting against things I
> haven't said; I don't know why, if it is lack of English skill, or
> something else, but we are talking constantly beside each other.

Perhaps you define the word "rewrite" differently, but its customary
meaning implies a reimplementation from square zero.  My point is that
doing so is a gratuitous and unnecessary gesture, and we would be better
served by more realistic assessments and consequently more reasonable
proposals.

> Are you saying that a non-multihtreaded program written in C can
> suddenly become multithreaded without being re-written to support
> threading? Nevermind, you are totally, on purpose or lack of
> understanding, missing the point: you have to rewrite stuff to make it
> multithreaded regardless of the language if you intention is to have
> Emacs core multithreaded. Supporting multithreading is not same as
> having entire Emacs internals be multithreaded.

Since when does interlocking call for a rewrite?

> No, it does not because you have missunderstood that I mean something
> else.

Then clarify what it is, for the plans you present (rewriting Emacs in a
different language) sure resemble plans for a complete grounds-up
reimplementation.

> I propose porting core API to Common Lisp because that would save you
> implementing all those things that you need to make Emacs
> multithreaded, better GC etc, because it is not a trivial task.

At the cost of portability and control?  No, thank you.
Let me ask a question in turn: how many of the popular and often-cited
Common Lisp implementations are as old as Emacs?  What guarantee is
there of their continued existence 10 or 20 years into the future?

> Of course you can re-write all that stuff in C too, nobody is denying
> that it is possible. I am saying there is no need to reduplicate the
> work, and there are other benefits of having Emacs in Common Lisp.

The work to be duplicated is menial and more-or-less already complete.
The remaining bulk will need to be tackled irrespective of the language
used to implement Emacs.  The conclusion as to whether we should
overcome that bulk as-is, of if we should compound it with the challenge
of rewriting Emacs in a new language, is left as an exercise to the
reader.

> Look harder.

Any examples?

>>> Well yes, C is portable and fast, nobody denies that. So are some
>>> CL
>>> compilers. 
>>
>> Which Common Lisp compiler is capable of linking with Java code
>> through
>> the JNI?  (ABCL doesn't work, as the Android JVM can't interpret the
>> bytecode it generates.)
>
> I don't know what you are trying to say; that sound like very
> uninformed statement based on missunderstanding. Calling Java via
> native API and generating bytecode that runs Lisp on top of Java as
> ABCL does are two completely different and separate things. I am quite
> sure Emacs does not generate Java byte code. But I don't see how is it
> even relevant, I haven't suggested to run Emacs on JVM. Perhaps it is
> possible who know, but I have suggested to use sbcl which is a native
> compiler specialized for compiling Lisp (and runs on Android as well).

It's not possible to load a Common Lisp image into the JVM, because the
Java linker is only capable of linking compiled C objects that export C
symbols.  Therefore, SBCL cannot run within any GUI Android program,
given that they must be started from Java.

> "only" ? :-) I can asure you that Lispers have access to both kernel
> and user space threads. Search on Bordeaux threads.

That's only a wrapper library for the incoherent threading primitives
furnished by different Common Lisp implementations.

>> garbage collection strategies vary between machine types.  I'm sure
>> we
>
> Does it?

Yes it does.

> I am also sure everything is possible, it is software, as a wise old
> grey head once said. It is just how much effort and resource you are
> pouring into it. SBCL runs on basically all important platforms on
> which Emacs runs, minus as mentioned DOS and Windows95; I don't know
> if there is some other. But it is not the point. The point is: there
> is another community working on the same problems you have or wish to
> solve. And they have come far away than you. You could re-use their
> work, and if you miss somethign add those missing parts so everyone
> would benefit, or you can live in an isolated island and do everything
> yourself.

C is hardly an "isolated island".  Anyway, we need a portable, fast,
flexible and ubiquitous language, and Common Lisp doesn't fit the bill:
if a platform as anodyne as Android poses problems for it, what about
the next sandboxing contraption for Unix systems?  Or any future system
whose appearance down the line we cannot anticipate now?  What if our
requirements outgrow Common Lisp's capabilities?

> I am telling you that sbcl/ccl has already solved those problems you
> have to solve in order to have multithreading without having to lift
> your lock.

We have too.  What we have not, they cannot offer either.

>> Instead of denigrating the C language, Emacs's C core, and so many
>> other
>> components critical to Emacs that have been subject to unjustified
>> invective in recent times, why not direct your attention to a more
>
> I don't think I understand how I "denigrate the C language" and what
> "unjustify invective" in this case is, but I don't think it
> matters. Once you also said I should be forbidden to talk about Emacs.

It does, because such motives drive people to engage in many hours of
time wasting wild goose chases, all to ascertain which language
Providence meant for Emacs to be written in.

>>                                    direct your attention to a more
>> productive task, such as identifying the complex interdependencies
>> between different pieces of Emacs's global state to establish a
>> detailed
>> plan for their interlocking?  For this task, the most important tool
>> is
>> a control-flow visualizer such as cflow, and an ample supply of
>> paper.
>
> Can we save nature and our selves?

Parse error, sorry.

> Nor are you sole Emacs devs, aren't you? I don't think calling names
> is appropriate, so I want, but I can come up with at least four people
> who work on Emacs and who are experienced Lispers. How active they are
> in development recently I don't know, I don't follow the list every
> day as I am used to, but I certainly haven't targeted Eli nor you with
> that one.

Names?  Would they be willing to take responsibility for the areas which
we oversee, in the event a Common Lisp Emacs comes to pass?

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Unix which? Unix is not a single program; it is a family of operating
>> systems, and I am quite sure nobody runs the same kernel they did back
>> in 1970. I don't know what you are talking about you have
>> "demonestrated". You said yourself *was* and not *is* which suggest
>> that those kernels are rewritten in order to support those C programs
>> that can be interlocked. Again you are constructing yourself things
>> here: I haven't said it is impossible to write mulththreaded C
>> application that can use locking.
>
> Each extant Unix system today contains gobs of code derived from the
> Unix kernel developed at BTL in the 1970s.  None of their kernels were
> ever rewritten -- rather, they were subject to iterative refinement that
> eventually allowed them to operate on SMP systems.

The word "rewritten" does not imply any time frame.

>> There are many applicaitons that have been redesigned. Emacs has been
>> redesigned since its first incarnations as TECO editor if I have
>> understand the history. For the very last time: I haven't said Emacs
>> *can not be redesigned*, on the contrary, I said Emacs *should be*
>> redesigned. Emacs core so to say. You are fighting against things I
>> haven't said; I don't know why, if it is lack of English skill, or
>> something else, but we are talking constantly beside each other.
>
> Perhaps you define the word "rewrite" differently, but its customary
> meaning implies a reimplementation from square zero.  My point is that

Perhaps you should concentrate on C coding not on your work to define
English dictionary?

>                                                       My point is that
> doing so is a gratuitous and unnecessary gesture, and we would be better
> served by more realistic assessments and consequently more reasonable
> proposals.

Yes, you have expressed a year or more ago as well I should be forbidden
to talk about Emacs. :). Perhaps you shouldn't have answered; nobody has
forced you.

I discuss gladly any technical aspects; actually that was my
hope; but I am certainly neither amused nor very interested to discuss
neither my character as you implied in another answer in this thread,
nor meaning of words in English language. I have expressed an idea in a
thread I found interesting, if it bothers you for any reason, tough
shit, don't answer.

>> Are you saying that a non-multihtreaded program written in C can
>> suddenly become multithreaded without being re-written to support
>> threading? Nevermind, you are totally, on purpose or lack of
>> understanding, missing the point: you have to rewrite stuff to make it
>> multithreaded regardless of the language if you intention is to have
>> Emacs core multithreaded. Supporting multithreading is not same as
>> having entire Emacs internals be multithreaded.
>
> Since when does interlocking call for a rewrite?
>
>> No, it does not because you have missunderstood that I mean something
>> else.
>
> Then clarify what it is, for the plans you present (rewriting Emacs in a
> different language) sure resemble plans for a complete grounds-up
> reimplementation.
>
>> I propose porting core API to Common Lisp because that would save you
>> implementing all those things that you need to make Emacs
>> multithreaded, better GC etc, because it is not a trivial task.
>
> At the cost of portability and control?  No, thank you.
> Let me ask a question in turn: how many of the popular and often-cited
> Common Lisp implementations are as old as Emacs?  What guarantee is
> there of their continued existence 10 or 20 years into the future?
>
>> Of course you can re-write all that stuff in C too, nobody is denying
>> that it is possible. I am saying there is no need to reduplicate the
>> work, and there are other benefits of having Emacs in Common Lisp.
>
> The work to be duplicated is menial and more-or-less already complete.
> The remaining bulk will need to be tackled irrespective of the language
> used to implement Emacs.  The conclusion as to whether we should
> overcome that bulk as-is, of if we should compound it with the challenge
> of rewriting Emacs in a new language, is left as an exercise to the
> reader.
>
>> Look harder.
>
> Any examples?
>
>>>> Well yes, C is portable and fast, nobody denies that. So are some
>>>> CL
>>>> compilers. 
>>>
>>> Which Common Lisp compiler is capable of linking with Java code
>>> through
>>> the JNI?  (ABCL doesn't work, as the Android JVM can't interpret the
>>> bytecode it generates.)
>>
>> I don't know what you are trying to say; that sound like very
>> uninformed statement based on missunderstanding. Calling Java via
>> native API and generating bytecode that runs Lisp on top of Java as
>> ABCL does are two completely different and separate things. I am quite
>> sure Emacs does not generate Java byte code. But I don't see how is it
>> even relevant, I haven't suggested to run Emacs on JVM. Perhaps it is
>> possible who know, but I have suggested to use sbcl which is a native
>> compiler specialized for compiling Lisp (and runs on Android as well).
>
> It's not possible to load a Common Lisp image into the JVM, because the
> Java linker is only capable of linking compiled C objects that export C
> symbols.  Therefore, SBCL cannot run within any GUI Android program,
> given that they must be started from Java.
>
>> "only" ? :-) I can asure you that Lispers have access to both kernel
>> and user space threads. Search on Bordeaux threads.
>
> That's only a wrapper library for the incoherent threading primitives
> furnished by different Common Lisp implementations.
>
>>> garbage collection strategies vary between machine types.  I'm sure
>>> we
>>
>> Does it?
>
> Yes it does.
>
>> I am also sure everything is possible, it is software, as a wise old
>> grey head once said. It is just how much effort and resource you are
>> pouring into it. SBCL runs on basically all important platforms on
>> which Emacs runs, minus as mentioned DOS and Windows95; I don't know
>> if there is some other. But it is not the point. The point is: there
>> is another community working on the same problems you have or wish to
>> solve. And they have come far away than you. You could re-use their
>> work, and if you miss somethign add those missing parts so everyone
>> would benefit, or you can live in an isolated island and do everything
>> yourself.
>
> C is hardly an "isolated island".  Anyway, we need a portable, fast,
> flexible and ubiquitous language, and Common Lisp doesn't fit the bill:
> if a platform as anodyne as Android poses problems for it, what about
> the next sandboxing contraption for Unix systems?  Or any future system
> whose appearance down the line we cannot anticipate now?  What if our
> requirements outgrow Common Lisp's capabilities?
>
>> I am telling you that sbcl/ccl has already solved those problems you
>> have to solve in order to have multithreading without having to lift
>> your lock.
>
> We have too.  What we have not, they cannot offer either.
>
>>> Instead of denigrating the C language, Emacs's C core, and so many
>>> other
>>> components critical to Emacs that have been subject to unjustified
>>> invective in recent times, why not direct your attention to a more
>>
>> I don't think I understand how I "denigrate the C language" and what
>> "unjustify invective" in this case is, but I don't think it
>> matters. Once you also said I should be forbidden to talk about Emacs.
>
> It does, because such motives drive people to engage in many hours of
> time wasting wild goose chases, all to ascertain which language
> Providence meant for Emacs to be written in.
>
>>>                                    direct your attention to a more
>>> productive task, such as identifying the complex interdependencies
>>> between different pieces of Emacs's global state to establish a
>>> detailed
>>> plan for their interlocking?  For this task, the most important tool
>>> is
>>> a control-flow visualizer such as cflow, and an ample supply of
>>> paper.
>>
>> Can we save nature and our selves?
>
> Parse error, sorry.

You mean sorry but no sorry?

Look; you are always indulging into Twitch/Reddit style trolling which
in my opinion has nothing to do with your English skills as Eli suggests
and warns about.

I am fully aware that your aggressive tone and way you speak to me are
because of your assumptions that I am an idiot, and you were not the
only one on this list. You are completely free to have any opinion about
me, I can't care less.

Consider if it is good for Emacs and GNU projects to alienate people who
are willing to help, and if it is representative for the free world to
tell who should be allow to speak and what about.

>> Nor are you sole Emacs devs, aren't you? I don't think calling names
>> is appropriate, so I want, but I can come up with at least four people
>> who work on Emacs and who are experienced Lispers. How active they are
>> in development recently I don't know, I don't follow the list every
>> day as I am used to, but I certainly haven't targeted Eli nor you with
>> that one.
>
> Names?  Would they be willing to take responsibility for the areas which
> we oversee, in the event a Common Lisp Emacs comes to pass?

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> The word "rewritten" does not imply any time frame.

How is a time frame relevant?

> Yes, you have expressed a year or more ago as well I should be forbidden
> to talk about Emacs. :). Perhaps you shouldn't have answered; nobody has
> forced you.

Forbidden?  Are you sure you're not placing words in my mouth?

> I discuss gladly any technical aspects; actually that was my
> hope; but I am certainly neither amused nor very interested to discuss
> neither my character as you implied in another answer in this thread,
> nor meaning of words in English language. I have expressed an idea in a
> thread I found interesting, if it bothers you for any reason, tough
> shit, don't answer.

And I've been trying to articulate why such ideas invariably lead us
down purposeless wild goose chases.  I have never attempted to comment
on your character.

> You mean sorry but no sorry?
>
> Look; you are always indulging into Twitch/Reddit style trolling which
> in my opinion has nothing to do with your English skills as Eli suggests
> and warns about.

My haunts include neither Twitch nor Reddit.  Please watch your tone.

> I am fully aware that your aggressive tone and way you speak to me are
> because of your assumptions that I am an idiot, and you were not the
> only one on this list. You are completely free to have any opinion about
> me, I can't care less.

Aggressive how?

> Consider if it is good for Emacs and GNU projects to alienate people who
> are willing to help, and if it is representative for the free world to
> tell who should be allow to speak and what about.

If any criticism of your ideas from valid technical perspectives is
``alienation'', I suggest you revisit the expectations you apply to
others.

Re: Shrinking the C core

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 29 Aug 2023 12:15:12 +0800
> 
> > I am fully aware that your aggressive tone and way you speak to me are
> > because of your assumptions that I am an idiot, and you were not the
> > only one on this list. You are completely free to have any opinion about
> > me, I can't care less.
> 
> Aggressive how?
> 
> > Consider if it is good for Emacs and GNU projects to alienate people who
> > are willing to help, and if it is representative for the free world to
> > tell who should be allow to speak and what about.
> 
> If any criticism of your ideas from valid technical perspectives is
> ``alienation'', I suggest you revisit the expectations you apply to
> others.

IMNSHO, you should both cool down.  This discussion, which keeps
beating a long-dead horse, isn't worth of aggravation and hurt
feelings.  So I suggest to just agree to disagree and move on.

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Unix which? Unix is not a single program; it is a family of operating
>> systems, and I am quite sure nobody runs the same kernel they did back
>> in 1970. I don't know what you are talking about you have
>> "demonestrated". You said yourself *was* and not *is* which suggest
>> that those kernels are rewritten in order to support those C programs
>> that can be interlocked. Again you are constructing yourself things
>> here: I haven't said it is impossible to write mulththreaded C
>> application that can use locking.
>
> Each extant Unix system today contains gobs of code derived from the
> Unix kernel developed at BTL in the 1970s.  None of their kernels were
> ever rewritten -- rather, they were subject to iterative refinement that
> eventually allowed them to operate on SMP systems.
>
>> There are many applicaitons that have been redesigned. Emacs has been
>> redesigned since its first incarnations as TECO editor if I have
>> understand the history. For the very last time: I haven't said Emacs
>> *can not be redesigned*, on the contrary, I said Emacs *should be*
>> redesigned. Emacs core so to say. You are fighting against things I
>> haven't said; I don't know why, if it is lack of English skill, or
>> something else, but we are talking constantly beside each other.
>
> Perhaps you define the word "rewrite" differently, but its customary
> meaning implies a reimplementation from square zero.  My point is that
> doing so is a gratuitous and unnecessary gesture, and we would be better
> served by more realistic assessments and consequently more reasonable
> proposals.
>
>> Are you saying that a non-multihtreaded program written in C can
>> suddenly become multithreaded without being re-written to support
>> threading? Nevermind, you are totally, on purpose or lack of
>> understanding, missing the point: you have to rewrite stuff to make it
>> multithreaded regardless of the language if you intention is to have
>> Emacs core multithreaded. Supporting multithreading is not same as
>> having entire Emacs internals be multithreaded.
>
> Since when does interlocking call for a rewrite?
>
>> No, it does not because you have missunderstood that I mean something
>> else.
>
> Then clarify what it is, for the plans you present (rewriting Emacs in a
> different language) sure resemble plans for a complete grounds-up
> reimplementation.
>
>> I propose porting core API to Common Lisp because that would save you
>> implementing all those things that you need to make Emacs
>> multithreaded, better GC etc, because it is not a trivial task.
>
> At the cost of portability and control?  No, thank you.
> Let me ask a question in turn: how many of the popular and often-cited
> Common Lisp implementations are as old as Emacs?  What guarantee is
> there of their continued existence 10 or 20 years into the future?
>
>> Of course you can re-write all that stuff in C too, nobody is denying
>> that it is possible. I am saying there is no need to reduplicate the
>> work, and there are other benefits of having Emacs in Common Lisp.
>
> The work to be duplicated is menial and more-or-less already complete.
> The remaining bulk will need to be tackled irrespective of the language
> used to implement Emacs.  The conclusion as to whether we should
> overcome that bulk as-is, of if we should compound it with the challenge
> of rewriting Emacs in a new language, is left as an exercise to the
> reader.
>
>> Look harder.
>
> Any examples?
>
>>>> Well yes, C is portable and fast, nobody denies that. So are some
>>>> CL
>>>> compilers. 
>>>
>>> Which Common Lisp compiler is capable of linking with Java code
>>> through
>>> the JNI?  (ABCL doesn't work, as the Android JVM can't interpret the
>>> bytecode it generates.)
>>
>> I don't know what you are trying to say; that sound like very
>> uninformed statement based on missunderstanding. Calling Java via
>> native API and generating bytecode that runs Lisp on top of Java as
>> ABCL does are two completely different and separate things. I am quite
>> sure Emacs does not generate Java byte code. But I don't see how is it
>> even relevant, I haven't suggested to run Emacs on JVM. Perhaps it is
>> possible who know, but I have suggested to use sbcl which is a native
>> compiler specialized for compiling Lisp (and runs on Android as well).
>
> It's not possible to load a Common Lisp image into the JVM, because the
> Java linker is only capable of linking compiled C objects that export C
> symbols.  Therefore, SBCL cannot run within any GUI Android program,
> given that they must be started from Java.
>
>> "only" ? :-) I can asure you that Lispers have access to both kernel
>> and user space threads. Search on Bordeaux threads.
>
> That's only a wrapper library for the incoherent threading primitives
> furnished by different Common Lisp implementations.
>
>>> garbage collection strategies vary between machine types.  I'm sure
>>> we
>>
>> Does it?
>
> Yes it does.
>
>> I am also sure everything is possible, it is software, as a wise old
>> grey head once said. It is just how much effort and resource you are
>> pouring into it. SBCL runs on basically all important platforms on
>> which Emacs runs, minus as mentioned DOS and Windows95; I don't know
>> if there is some other. But it is not the point. The point is: there
>> is another community working on the same problems you have or wish to
>> solve. And they have come far away than you. You could re-use their
>> work, and if you miss somethign add those missing parts so everyone
>> would benefit, or you can live in an isolated island and do everything
>> yourself.
>
> C is hardly an "isolated island".  Anyway, we need a portable, fast,
> flexible and ubiquitous language, and Common Lisp doesn't fit the bill:
> if a platform as anodyne as Android poses problems for it, what about
> the next sandboxing contraption for Unix systems?  Or any future system
> whose appearance down the line we cannot anticipate now?  What if our
> requirements outgrow Common Lisp's capabilities?
>
>> I am telling you that sbcl/ccl has already solved those problems you
>> have to solve in order to have multithreading without having to lift
>> your lock.
>
> We have too.  What we have not, they cannot offer either.
>
>>> Instead of denigrating the C language, Emacs's C core, and so many
>>> other
>>> components critical to Emacs that have been subject to unjustified
>>> invective in recent times, why not direct your attention to a more
>>
>> I don't think I understand how I "denigrate the C language" and what
>> "unjustify invective" in this case is, but I don't think it
>> matters. Once you also said I should be forbidden to talk about Emacs.
>
> It does, because such motives drive people to engage in many hours of
> time wasting wild goose chases, all to ascertain which language
> Providence meant for Emacs to be written in.
>
>>>                                    direct your attention to a more
>>> productive task, such as identifying the complex interdependencies
>>> between different pieces of Emacs's global state to establish a
>>> detailed
>>> plan for their interlocking?  For this task, the most important tool
>>> is
>>> a control-flow visualizer such as cflow, and an ample supply of
>>> paper.
>>
>> Can we save nature and our selves?
>
> Parse error, sorry.
>
>> Nor are you sole Emacs devs, aren't you? I don't think calling names
>> is appropriate, so I want, but I can come up with at least four people
>> who work on Emacs and who are experienced Lispers. How active they are
>> in development recently I don't know, I don't follow the list every
>> day as I am used to, but I certainly haven't targeted Eli nor you with
>> that one.
>
> Names?  Would they be willing to take responsibility for the areas which
> we oversee, in the event a Common Lisp Emacs comes to pass?

Perhaps this recently released paper might be of interest to you:

https://applied-langua.ge/~hayley/swcl-gc.pdf

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I ask everyone in this discussion to try to disagree more kindly and
without rancor.  In this discussion we all want the same overall
goal, and if we disagree about the best way to achieve it, lets try to
work together to figure that out.

Please see https://gnu.org/philosophy/kind-communication.html.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Would you please stop arguing for rewriting Emacs in Common Lisp?  It
is a non-starter.

It would be an enormouse job -- including rewriting the Emacs Lisp
Referance Manual.  We can't be sure how much the benefit would be,
because Common Lisp has some drawbacks too.  But we know the cost is
prohibitive.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Would you please stop arguing for rewriting Emacs in Common Lisp?  It
> is a non-starter.

Can we not even talk about it?

Is it really bad to talk about an idea? If you believe that such a
simple idea can hurt you or your project, than I don't think "free
speach" and GNU goes together in the same text, more than possibly as a
sarcastic remark.

> It would be an enormouse job

Sure. I agree on that one.

>                                 including rewriting the Emacs Lisp
> Referance Manual.

Actually, I don't know if I was clear in my first post, but my raison
d'etre for converting Emacs to CL is to actually preserve emacs-lisp bug
by bug, feature by feature, for the explicit reason of not throwing away
40 years of work. As I said; Emacs Lisp is *the* best documented Lisp,
and that is extremely important. People complain a lot on social media
about CL being arcane (hyperspec) and 3rd party libraries undocumented
or badly documented. In my opinion the strong points of elisp are
documentation, manual, 3rd party packages and being relatively small.

Otherwise, if I didn't want to preserve the documentation and manual,
and don't want to run Emacs applications, I would just go over to some
of the text editors already implemented in common lisp. Their problem is
that they are "emacs like", but not Emacs. They basically have to
re-implement everything from scratch. Which is a shame. But if we
implement emacs lisp in cl than we can run 3rd party apps in Emacs in CL
and people can re-use their knowledge, docs and the manual in writing
3rd party applications.

>                    We can't be sure how much the benefit would be,

You can't know anything for sure. But you do know that you would got
work that yat has to be done in Emacs, and you know that you would get
language features that lots of people ask for. It seems that either
Common Lisp will come to Emacs or Emacs will come to Common Lisp,
judging by the attitudes that pop-up in social media from time to time.

You would also get faster iteration time. Testing Emacs core function
would be similar to working in Emacs Lisp: eval and run; instead of
recompile and re-start Emacs just to test.

You would also unify the extension language and the implementation
language, so more people would, at least in theory, be able to look and
fix and experiment with the core. Testing a different renderer or a
different buffer implementation could become much less tedious prospect
than in C core.

Of course, it is not "just", but I would like to hear would be technical
problems, or if it is not technically possible.

As I udnerstand Common Lisp was invented to unify all the dialects such
as Emacs Lisp, and seemed to work well. I see no special technical
problems in implementing elisp in cl either, but I am not an expert and
not so familiar with CL yet, so I have hoped for some constructive input
not trolling and sarcasms.

> because Common Lisp has some drawbacks too.

Sure. Everything has drawbacks. So is life. And in life we are
constantly navigating between cost and gain.

Tell me which are drawbacks; that is actually what I am interesting
in. CL is much bigger, EL is easier to learn, that is my personal
remark, but I think it is secondary compared to what we can get with CL.

>                                              But we know the cost is
> prohibitive.

Perhaps. The cost is certainly prohibitive for a single person to do it
all on their own.

TBH I hoped for some technical input like "it is possible because of" or
this can't be done in CL because of or whatever. Instead I have got a
sarcasm and a troll answer. I should have known though better than to
answer on that one; it takes two to troll, so I guess we are both
guilty there.

That would be my answer to remark about kind communication in the latter
mail.

Re: Shrinking the C core

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

On Fri, Sep 01, 2023 at 04:58:53PM +0200, Arthur Miller wrote:
> Richard Stallman <rms@gnu.org> writes:
> 
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> >
> > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
> > is a non-starter.
> 
> Can we not even talk about it?
> 
> [...] "free speach" [...]

Yikes. I call Godwin on this (half-kidding).

No, seriously. I think the least you could do would be to read
on the different CL re-implementations of Emacs (I only knew
about Climacs [1], but someone in this thread posted references
to a handful) and come back explaining to us why you think none
of those have gained traction and what to try next time.

Otherwise we're bound to turn around rehashing this thing time
and again without learning anything new.

If the idea is to ramble aimlessly for rambling's sake (which
is a very honourable idea indeed), perhaps emacs-tangents would
be a better place to enjoy that.

Quite a few very smart people have come up with this idea without
knowing what happened before. Here [2], for example is Tom Tromey
"hey, let's implement Emacs in Common Lisp" from 2012, whereas
Cliki (as pointed out by some of his readers) is from... 2008.

We're all dwarfs. If we don't stand on shoulders of giants, we
don't see zilch.

Cheers

[1] https://climacs.common-lisp.dev/
[2] https://tromey.com/blog/?p=709
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
  > > is a non-starter.

  > Can we not even talk about it?

There are people who seem to be pressuring to rewrite GNU Emacs in
Common Lisp, and they kee pushing for it.  They keeo talking about the
idea, and it feels like a demand.  I get the feeling that they think
that by continuing to discuss this as if it were a real option they
hope to _make_ us take the option seriously.  That is feels like pressure.

Would those would like to discuss that plseae take the discssion off
emacs-devel?  Then it will not be pressure.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

>>> Would you please stop arguing for rewriting Emacs in
>>> Common Lisp? It is a non-starter.
>
>> Can we not even talk about it?
>
> There are people who seem to be pressuring to rewrite GNU
> Emacs in Common Lisp, and they kee pushing for it. They keeo
> talking about the idea, and it feels like a demand. I get
> the feeling that they think that by continuing to discuss
> this as if it were a real option they hope to _make_ us take
> the option seriously. That is feels like pressure.
>
> Would those would like to discuss that plseae take the
> discssion off emacs-devel? Then it will not be pressure.

It sounds like you have been traumatized, almost, by such
discussion in the past, passibly?

To be fair I don't think anyone is pushing for that. It would
be to burn down the house to kill the rats.

We should rather identify advantages SBCL has over Elisp and
see if we can bridge that gap, possibly even using technology
straight from their toolbox.

As for speed I think we can conclude the native-compile
feature is the solution, but it hasn't been taken to its full
potential which is expected since it is quite new.

As for the multicore processing thing, I wonder how they
did that? And why can't we do that as well with Elisp?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Rudolf Schlatte]

Emanuel Berg <incal@dataswamp.org> writes:

> As for the multicore processing thing, I wonder how they
> did that? And why can't we do that as well with Elisp?

That is answered by the SBCL manual.  The first sentence of
http://sbcl.org/manual/index.html#Threading says: "SBCL supports a
fairly low-level threading interface that maps onto the host operating
system’s concept of threads or lightweight processes."

So race conditions resulting from multiple threads modifying Lisp data
structures concurrently are entirely the programmer's responsibility in
SBCL, to be handled by using the provided standard tools (mutexes,
locks, condition variables etc).  So if you create an application in CL,
you have to check that the libraries you use are thread-safe.  For
Emacs, where multiple applications and libraries are assembled into an
editor by the end user, instead of a programmer creating one application
to be run in isolation in an SBCL image, I believe this approach will
not work.

Re: Shrinking the C core

[Emanuel Berg]

Rudolf Schlatte wrote:

> So race conditions resulting from multiple threads modifying
> Lisp data structures concurrently are entirely the
> programmer's responsibility in SBCL, to be handled by using
> the provided standard tools (mutexes, locks, condition
> variables etc). So if you create an application in CL, you
> have to check that the libraries you use are thread-safe.
> For Emacs, where multiple applications and libraries are
> assembled into an editor by the end user, instead of
> a programmer creating one application to be run in isolation
> in an SBCL image, I believe this approach will not work.

Since Lisp is and should be the EVERYTHING language, it is all
well and good that the programmer can do that, however - we
should aim at the other end of everythingness as well -
namely, to have Elisp do that seamlessly and on the fly so
that multicored Elisp in execution would happen whenever
beneficial also (especially) when executing normal, including
all existing for that matter, Elisp code.

However, as a first step toward that, we would indeed welcome
specific constructs, e.g.

(let-para ((one (...))
           (two (...)) )
  (compute one two) )

Where 'para' is for "parallel" and would work as this, 'one'
would be computed on CPU core 1, 'two' on CPU core 2 -
starting simultaneously and executing in parallel as you might
have guessed - and the body, in this case "compute", would be
evaluated when both are done.

How would you do that?

At least we don't have to bother with let-para* LOL

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It sounds like you have been traumatized, almost, by such
  > discussion in the past, passibly?

If you have a clinical psychology degree, you might properly make
comments about such questions -- after a suitable examination a person
who agreed to be your patient

But if you are like the rest of us in not having such a degree, please
don't try to psychologize the people in our discussions.  It really is
not nice.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

>> It sounds like you have been traumatized, almost, by such
>> discussion in the past, passibly?
>
> If you have a clinical psychology degree, you might properly
> make comments about such questions -- after a suitable
> examination a person who agreed to be your patient
>
> But if you are like the rest of us in not having such
> a degree, please don't try to psychologize the people in our
> discussions. It really is not nice.

You are right, absolutely, but then I cannot see why people
can't push for a SBCL rewrite of Emacs?

I personally am not in favor of that idea anymore since speed
was the main issue and with native-compile that is solved to
a huge degree and is expected to be even better.

But what is wrong pushing for it, if some people feel that
way? It is on topic (Emacs development) and no one is forcing
anyone to read or take part in such discussions if they don't
like to. So it should be allowed, why not?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[tomas]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

On Thu, Sep 07, 2023 at 03:54:05AM +0200, Emanuel Berg wrote:

[setting mail-followup-to]

[...]

> You are right, absolutely, but then I cannot see why people
> can't push for a SBCL rewrite of Emacs?

But not in such an unconstructive way. Not less here, in this
monster thread, where each argument has been hashed out three
times. Not without having done some homework.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Shrinking the C core

[Alfred M. Szmidt]

   > But if you are like the rest of us in not having such
   > a degree, please don't try to psychologize the people in our
   > discussions. It really is not nice.

   You are right, absolutely, but then I cannot see why people
   can't push for a SBCL rewrite of Emacs?

Because the maintainers of GNU Emacs are not interested in such a push
and that it is a distracting topic.  If you wish to have a Emacs in
Common Lisp, there are plenty of other places to turn to -- for
example, Hemlock.

There is really not much more too it.

Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

>>> But if you are like the rest of us in not having such
>>> a degree, please don't try to psychologize the people in
>>> our discussions. It really is not nice.
>>
>> You are right, absolutely, but then I cannot see why people
>> can't push for a SBCL rewrite of Emacs?
>
> Because the maintainers of GNU Emacs are not interested in
> such a push and that it is a distracting topic.

Oh, you poor sweeties!

Is there anything else hundreds of Elisp programmers from all
over the world are unallowed to discuss out of concern for
a handful of dude's emotions about Common Lisp, SBCL, and the
GNU Emacs Elisp part cl-lib?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

> Oh, you poor sweeties!
> ...

Please keep in mind https://www.gnu.org/philosophy/kind-communication.html
Thanks!

Re: Shrinking the C core

[Emanuel Berg]

Ihor Radchenko wrote:

>> Oh, you poor sweeties!
>
> Please keep in mind
> https://www.gnu.org/philosophy/kind-communication.html

Please stop pushing for the disallowance of topics related to
SBCL and Emacs. It is a non-starter.

Also, please stop pushing for the disencouragement of using
the GNU Emacs Elisp features provided by cl-lib.
Also a non-starter.

One thing that can be discussed is how to get parallelism into
Emacs, to make use of modern CPU multicore architectures.

Emacs is a single-thread Lisp machine written in C, so can't
we have a multi-thread Lisp machine, using the same language?
And not by rewriting the whole thing, rather extending it to
allow for multithreading?

Didn't anyone try to do that already BTW? It feels like
a pretty basic idea and multicore CPUs have been around for
a while.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Alfred M. Szmidt]

You're quite free to discuss it, somewhere else -- for example
emacs-tangents.  emacs-devel is for the development of GNU Emacs, and
seeing that GNU Emacs has no intention of becoming Common Lisp, any
push towards that end are off-topic here.

And lets all please tone down.

Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

> You're quite free to discuss it, somewhere else -- for
> example emacs-tangents. emacs-devel is for the development
> of GNU Emacs, and seeing that GNU Emacs has no intention of
> becoming Common Lisp, any push towards that end are
> off-topic here.

People are "pushing" for a zillion of things here, that the
Emacs maintainers don't like. You gonna disallow that? I don't
think so!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> People are "pushing" for a zillion of things here, that the
> Emacs maintainers don't like. You gonna disallow that? I don't
> think so!

RMS is the Chief Gnusiance, and thus he has the authority to prohibit
certain topics from being discussed.  What are you?

Re: Shrinking the C core

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Date: Sat, 09 Sep 2023 14:25:12 +0800
> 
> Emanuel Berg <incal@dataswamp.org> writes:
> 
> > People are "pushing" for a zillion of things here, that the
> > Emacs maintainers don't like. You gonna disallow that? I don't
> > think so!
> 
> RMS is the Chief Gnusiance, and thus he has the authority to prohibit
> certain topics from being discussed.  What are you?

Please stop this useless and unkind line of arguing, both of you!

What is and isn't allowed here is not up to any of you.

Re: Shrinking the C core

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 07 Sep 2023 10:39:11 +0200
> 
> Ihor Radchenko wrote:
> 
> >> Oh, you poor sweeties!
> >
> > Please keep in mind
> > https://www.gnu.org/philosophy/kind-communication.html
> 
> Please stop pushing for the disallowance of topics related to
> SBCL and Emacs. It is a non-starter.

As long as you don't mock your opponents or are otherwise rude, it is
allowed.  However, please be more considerate when someone asks you
not to push a topic, since just repeating the push doesn't add
anything useful to the discussion.

> Also, please stop pushing for the disencouragement of using
> the GNU Emacs Elisp features provided by cl-lib.
> Also a non-starter.

People are entitled to expressing their opinions here; that is not
"pushing".  What is and isn't acceptable to go into the Emacs sources
is not decided by expressing opinions about stylistic preferences, and
not necessarily by people whose opinions you don't like.

> Emacs is a single-thread Lisp machine written in C, so can't
> we have a multi-thread Lisp machine, using the same language?
> And not by rewriting the whole thing, rather extending it to
> allow for multithreading?
> 
> Didn't anyone try to do that already BTW? It feels like
> a pretty basic idea and multicore CPUs have been around for
> a while.

This has been discussed several times, and the current conclusion (at
least mine) is that doing that for Emacs would be tantamount to
"rewriting the whole thing", yes.

Re: Shrinking the C core

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

> Please stop pushing for the disallowance of topics related to
> SBCL and Emacs. It is a non-starter.

Well. The main problem with "let's rewrite Emacs using X language" is
that it has been discussed and even attempted (for example:
https://github.com/remacs/remacs, https://github.com/emacs-ng/emacs-ng)
many times. Without success. So, I can see how raising the same topic
yet again does not look productive.

Yet, I do agree that there is no reason to _forbid_ such discussions.
But please clearly mark them under appropriate email subject, so that
people who do not want to follow them could filter out the subject in
their email client and not be bothered.

> Also, please stop pushing for the disencouragement of using
> the GNU Emacs Elisp features provided by cl-lib.
> Also a non-starter.

There is actually reasoning behind discouragement of using cl-lib in
_Emacs source code_. In particular, things like `cl-loop' that
constitute a mini-language of its own (or `pcase' for what its worth).

When such "mini-languages" are used too much, without a clear need,
people not familiar with CL conventions will have harder time to read
the Emacs source code - increased entry barrier for contribution.

Also, it is up to Emacs maintainers to decide on the preferred style
inside Emacs sources.

> One thing that can be discussed is how to get parallelism into
> Emacs, to make use of modern CPU multicore architectures.
>
> Emacs is a single-thread Lisp machine written in C, so can't
> we have a multi-thread Lisp machine, using the same language?
> And not by rewriting the whole thing, rather extending it to
> allow for multithreading?
>
> Didn't anyone try to do that already BTW? It feels like
> a pretty basic idea and multicore CPUs have been around for
> a while.

It is not as easy as you may think. Naive attempts will fail:
https://www.youtube.com/watch?v=7H1Pe9HkJ7I

You can see the gory details in
https://yhetil.org/emacs-devel/871qhnr4ty.fsf@localhost
I'd appreciate if we move the discussion about parallelism there to have
a single place with all the caveats detailed and discussed instead of
scattering things all over emacs-devel.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Emanuel Berg]

Ihor Radchenko wrote:

> When such "mini-languages" are used too much, without
> a clear need, people not familiar with CL conventions will
> have harder time to read the Emacs source code - increased
> entry barrier for contribution.

If it is used in a bad way, like everything else used in a bad
way, it is bad.

But used in a good way it adds tons of expressive power to
our Elisp!

Stefan Monnier, they don't belive me. Regardless of whatever
else has been said and done, you tell them how awesome Elisp
CL is. Maybe it will land that way?

But it is true anyway.

https://dataswamp.org/~incal/vidz/misc/gaia-tuvan-sliced.webm

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You are right, absolutely, but then I cannot see why people
  > can't push for a SBCL rewrite of Emacs?

Because that is not up for decision.  That decision is already made.

If the question were up for decision, arguing for a certain choice
would be normal participation.  When it isn't, arguing for a choice is
making life difficult.  I have too much work to do, and I can't keep
up.  So does Eli.  Eli can speak for himself, but if you make it necessary
for me to spend more time on this, that is making difficulties.

It is ok to discuss these questions on emacs-tangets, because
discussing it there won't add to my burden.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 10 Sep 2023 20:43:01 -0400
> 
>   > You are right, absolutely, but then I cannot see why people
>   > can't push for a SBCL rewrite of Emacs?
> 
> Because that is not up for decision.  That decision is already made.
> 
> If the question were up for decision, arguing for a certain choice
> would be normal participation.  When it isn't, arguing for a choice is
> making life difficult.  I have too much work to do, and I can't keep
> up.  So does Eli.  Eli can speak for himself, but if you make it necessary
> for me to spend more time on this, that is making difficulties.

IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
time and energy.  We've seen this many times (because people still
insist on bringing this up from time to time).  From where I stand,
the main reason is not even the fact that we decided not to do that,
but the fact that such a rewrite will never happen in practice.  Such
a rewrite is a massive job which requires very good knowledge of Emacs
internals and features, and a lot of time.  People who come close to
the required knowledge level are not interested in doing this job
(because they understand the futility), and those who think it should
be done simply don't know enough and/or don't have enough time on
their hands to pull it through.

If Emacs will ever be "rewritten", it will not be Emacs, but a
text-processing system with a very different architecture and design,
which will take from the Emacs experience the lessons we learned and
implement them differently, to produce a system whose starting point
is closer to the needs of today's users and whose main technologies
are more modern from the get-go.

Re: Shrinking the C core

[tomas]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

On Mon, Sep 11, 2023 at 03:24:43PM +0300, Eli Zaretskii wrote:

[...]

> IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
> time and energy.  We've seen this many times (because people still
> insist on bringing this up from time to time) [...]

What irks me most in such cases is not the idea itself, or the wish
to discuss it. It's rather that those people consistently seem to
want others to do the grunt work. Usually they seem to shy away from
looking at former attempts and trying to learn something from them.

Now this would be an interesting contribution: what's wrong with
Climacs? What's right? What went wrong with PortableHemlock and
what can we learn from it?

Heck, they even seem too lazy to take the discussion to emacs-tangents.

Huffing and puffing about freedom of expression is a bit...
strange in this context. But seems to be fashionable these days.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Shrinking the C core

[Emanuel Berg]

tomas wrote:

> Huffing and puffing about freedom of expression is a bit...
> strange in this context. But seems to be fashionable
> these days.

tomas, you can't say what you just said.

Because then I have to read it and I don't have time for that
- besides, I have already agreed with the people who share my
opinion to disagree with you. So the decision has been taken;
subsequent discussion has no meaning.

But it is okay for you to say it, just not here. Say it
somewhere else where I don't have to read it, that would be
acceptable. *generous*

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
> time and energy.  We've seen this many times (because people still
> insist on bringing this up from time to time).  From where I stand,
> the main reason is not even the fact that we decided not to do that,
> but the fact that such a rewrite will never happen in practice.  Such
> a rewrite is a massive job which requires very good knowledge of Emacs
> internals and features, and a lot of time.  People who come close to
> the required knowledge level are not interested in doing this job
> (because they understand the futility), and those who think it should
> be done simply don't know enough and/or don't have enough time on
> their hands to pull it through.
>
> If Emacs will ever be "rewritten", it will not be Emacs, but a
> text-processing system with a very different architecture and design,
> which will take from the Emacs experience the lessons we learned and
> implement them differently, to produce a system whose starting point
> is closer to the needs of today's users and whose main technologies
> are more modern from the get-go.

I couldn't agree more.

To me, a rewrite is quatsch, while adding CL facilities to Emacs makes a
lot of sense.

Re: Shrinking the C core

[João Távora]

On Tue, Sep 12, 2023 at 5:44 AM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> > If Emacs will ever be "rewritten", it will not be Emacs, but a
> > text-processing system with a very different architecture and design,
> > which will take from the Emacs experience the lessons we learned and
> > implement them differently, to produce a system whose starting point
> > is closer to the needs of today's users and whose main technologies
> > are more modern from the get-go.
>
> I couldn't agree more.
>
> To me, a rewrite is quatsch, while adding CL facilities to Emacs makes a
> lot of sense.

+1

A rewrite (or somehow the ability to embed a CL runtime to run all our
existing Elisp code) is not exactly 100% quatsch, but maybe 95%.

Re: Shrinking the C core

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2213 bytes --]

On Mon, Sep 11, 2023, 8:26 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Richard Stallman <rms@gnu.org>
> > Cc: emacs-devel@gnu.org
> > Date: Sun, 10 Sep 2023 20:43:01 -0400
> >
> >   > You are right, absolutely, but then I cannot see why people
> >   > can't push for a SBCL rewrite of Emacs?
> >
> > Because that is not up for decision.  That decision is already made.
> >
> > If the question were up for decision, arguing for a certain choice
> > would be normal participation.  When it isn't, arguing for a choice is
> > making life difficult.  I have too much work to do, and I can't keep
> > up.  So does Eli.  Eli can speak for himself, but if you make it
> necessary
> > for me to spend more time on this, that is making difficulties.
>
> IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
> time and energy.  We've seen this many times (because people still
> insist on bringing this up from time to time).  From where I stand,
> the main reason is not even the fact that we decided not to do that,
> but the fact that such a rewrite will never happen in practice.  Such
> a rewrite is a massive job which requires very good knowledge of Emacs
> internals and features, and a lot of time.  People who come close to
> the required knowledge level are not interested in doing this job
> (because they understand the futility), and those who think it should
> be done simply don't know enough and/or don't have enough time on
> their hands to pull it through.
>
> If Emacs will ever be "rewritten", it will not be Emacs, but a
> text-processing system with a very different architecture and design,
> which will take from the Emacs experience the lessons we learned and
> implement them differently, to produce a system whose starting point
> is closer to the needs of today's users and whose main technologies
> are more modern from the get-go.
>

It sounds like you have some specific ideas.  I wouldn't mind hearing them
at more length.

My understanding is the design is deliberately kept simple (or "simple") to
make it accessible to more programmers.

Instead of discussing porting emacs to CL,  why don't people work on
porting the compiler techniques used in CL to emacs?

Lynn

[-- Attachment #2: Type: text/html, Size: 3279 bytes --]

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Tue, 12 Sep 2023 07:31:35 -0400
> Cc: Richard Stallman <rms@gnu.org>, incal@dataswamp.org, emacs-devel <emacs-devel@gnu.org>
> 
>  If Emacs will ever be "rewritten", it will not be Emacs, but a
>  text-processing system with a very different architecture and design,
>  which will take from the Emacs experience the lessons we learned and
>  implement them differently, to produce a system whose starting point
>  is closer to the needs of today's users and whose main technologies
>  are more modern from the get-go.
> 
> It sounds like you have some specific ideas.  I wouldn't mind hearing them at more length.

They are all well known.  And they aren't ideas, just main design
features of Emacs which we found restrictive in some aspects:

  . "buffer with gap" for storing buffer text
  . "mark and sweep" GC
  . basic single-threaded MVC architecture
  . display engine design around the rectangular canvas model and
    on-the-fly layout decisions

> My understanding is the design is deliberately kept simple (or "simple") to make it accessible to
> more programmers.  

Richard will tell, but I don't think this goal was ever of high
priority, back when Emacs was still being designed.

> Instead of discussing porting emacs to CL,  why don't people work on porting the compiler
> techniques used in CL to emacs?  

Well, native-compilation is one step in that direction.  We've been
discussing something like that for many years, and we even tried a
couple of approaches (which didn't work).  Native compilation is the
first successful experiment in that direction.  Working on making the
native code more efficient is definitely encouraged.

Re: Emacs design and architecture (was: Shrinking the C core)

[Lynn Winebarger]

On Tue, Sep 12, 2023 at 9:01 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Tue, 12 Sep 2023 07:31:35 -0400
> > Cc: Richard Stallman <rms@gnu.org>, incal@dataswamp.org, emacs-devel <emacs-devel@gnu.org>
> >
> >  If Emacs will ever be "rewritten", it will not be Emacs, but a
> >  text-processing system with a very different architecture and design,
> >  which will take from the Emacs experience the lessons we learned and
> >  implement them differently, to produce a system whose starting point
> >  is closer to the needs of today's users and whose main technologies
> >  are more modern from the get-go.
> >
> > It sounds like you have some specific ideas.  I wouldn't mind hearing them at more length.
>
> They are all well known.  And they aren't ideas, just main design
> features of Emacs which we found restrictive in some aspects:

I think I was more interested in why you said it would be a
"text-processing system" rather than an editor or even IDE, what you
perceive as the unmet needs of today's users, and what modern
technologies you think are applicable to the problem.  I'm not an
expert in the display or text-encoding aspects of what Emacs does, so
I really don't know what you have in mind.  I do have some ideas in
terms of extension language implementation.  There are definitely some
techniques used in implementing V8 that could be brought over.

>   . "buffer with gap" for storing buffer text
>   . "mark and sweep" GC
>   . basic single-threaded MVC architecture

These three I have some ideas on, which I've outlined previously,
using git and other distributed VC techniques as inspiration.
However, those are longer term goals.  For my first effort, I'm
focusing on some low hanging fruit for creating tree-sitter parser
tables and lexers in elisp and being able to create them dynamically,
with some supporting technology that can also be applied to more
generic dumping and garbage collection.

>   . display engine design around the rectangular canvas model and
I don't know what the alternative is, exactly.  VS Code and other IDEs
I've used have more varied decorative elements and containers, but
text is still pretty much presented in rectangular containers with
sides parallel to the GUI window (frame in Emacs terms).

>     on-the-fly layout decisions
>
> > My understanding is the design is deliberately kept simple (or "simple") to make it accessible to
> > more programmers.
>
> Richard will tell, but I don't think this goal was ever of high
> priority, back when Emacs was still being designed.
>
> > Instead of discussing porting emacs to CL,  why don't people work on porting the compiler
> > techniques used in CL to emacs?
>
> Well, native-compilation is one step in that direction.  We've been
> discussing something like that for many years, and we even tried a
> couple of approaches (which didn't work).  Native compilation is the
> first successful experiment in that direction.  Working on making the
> native code more efficient is definitely encouraged.

It's on my agenda.  Not native code, per se, though.  I think there
are some changes to the VM design (that the native code has to be
consistent with) that are required.  The strict stack discipline is
inherently inefficient, and since not every function parameter is
dynamically scoped, there are plenty of opportunities to optimize away
unnecessary function call overhead.  That and more efficient GC design
go hand in hand.

Lynn

Re: Emacs design and architecture (was: Shrinking the C core)

[Christopher Dimech]

> Sent: Thursday, September 14, 2023 at 8:52 AM
> From: "Lynn Winebarger" <owinebar@gmail.com>
> To: "Eli Zaretskii" <eliz@gnu.org>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Subject: Re: Emacs design and architecture (was: Shrinking the C core)
>
> On Tue, Sep 12, 2023 at 9:01 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > > From: Lynn Winebarger <owinebar@gmail.com>
> > > Date: Tue, 12 Sep 2023 07:31:35 -0400
> > > Cc: Richard Stallman <rms@gnu.org>, incal@dataswamp.org, emacs-devel <emacs-devel@gnu.org>
> > >
> > >  If Emacs will ever be "rewritten", it will not be Emacs, but a
> > >  text-processing system with a very different architecture and design,
> > >  which will take from the Emacs experience the lessons we learned and
> > >  implement them differently, to produce a system whose starting point
> > >  is closer to the needs of today's users and whose main technologies
> > >  are more modern from the get-go.

One thing that needs rewriting is texinfo to use the latest functionalities 
that are provided by the LaTeX3 Project (https://www.latex-project.org/latex3/)

Today we continue to be restricted by what the TeX Core Engine can do.
We are continuing to work with the pre-1983 version before Leslie Lamport 
developed LaTeX, a higher-level markup language built on top of TeX. 

> > > It sounds like you have some specific ideas.  I wouldn't mind hearing them at more length.
> >
> > They are all well known.  And they aren't ideas, just main design
> > features of Emacs which we found restrictive in some aspects:
> 
> I think I was more interested in why you said it would be a
> "text-processing system" rather than an editor or even IDE, what you
> perceive as the unmet needs of today's users, and what modern
> technologies you think are applicable to the problem.  I'm not an
> expert in the display or text-encoding aspects of what Emacs does, so
> I really don't know what you have in mind.  I do have some ideas in
> terms of extension language implementation.  There are definitely some
> techniques used in implementing V8 that could be brought over.
> 
> >   . "buffer with gap" for storing buffer text
> >   . "mark and sweep" GC
> >   . basic single-threaded MVC architecture
> 
> These three I have some ideas on, which I've outlined previously,
> using git and other distributed VC techniques as inspiration.
> However, those are longer term goals.  For my first effort, I'm
> focusing on some low hanging fruit for creating tree-sitter parser
> tables and lexers in elisp and being able to create them dynamically,
> with some supporting technology that can also be applied to more
> generic dumping and garbage collection.
> 
> >   . display engine design around the rectangular canvas model and
> I don't know what the alternative is, exactly.  VS Code and other IDEs
> I've used have more varied decorative elements and containers, but
> text is still pretty much presented in rectangular containers with
> sides parallel to the GUI window (frame in Emacs terms).
> 
> >     on-the-fly layout decisions
> >
> > > My understanding is the design is deliberately kept simple (or "simple") to make it accessible to
> > > more programmers.
> >
> > Richard will tell, but I don't think this goal was ever of high
> > priority, back when Emacs was still being designed.
> >
> > > Instead of discussing porting emacs to CL,  why don't people work on porting the compiler
> > > techniques used in CL to emacs?
> >
> > Well, native-compilation is one step in that direction.  We've been
> > discussing something like that for many years, and we even tried a
> > couple of approaches (which didn't work).  Native compilation is the
> > first successful experiment in that direction.  Working on making the
> > native code more efficient is definitely encouraged.
> 
> It's on my agenda.  Not native code, per se, though.  I think there
> are some changes to the VM design (that the native code has to be
> consistent with) that are required.  The strict stack discipline is
> inherently inefficient, and since not every function parameter is
> dynamically scoped, there are plenty of opportunities to optimize away
> unnecessary function call overhead.  That and more efficient GC design
> go hand in hand.
> 
> Lynn
> 
>

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> From: Christopher Dimech <dimech@gmx.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 13 Sep 2023 23:19:03 +0200
> 
> One thing that needs rewriting is texinfo to use the latest functionalities 
> that are provided by the LaTeX3 Project (https://www.latex-project.org/latex3/)
> 
> Today we continue to be restricted by what the TeX Core Engine can do.
> We are continuing to work with the pre-1983 version before Leslie Lamport 
> developed LaTeX, a higher-level markup language built on top of TeX. 

Is this related to Emacs in any way?  I think you need to post this to
the Texinfo list (and I think they will tell you that the next Texinfo
will have LaTeX support).

Let's please remain focused on Emacs here, even in this discussion
that naturally touches some less-than-practical issues.

Re: Emacs design and architecture (was: Shrinking the C core)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > Today we continue to be restricted by what the TeX Core Engine can do.
  > We are continuing to work with the pre-1983 version before Leslie Lamport 
  > developed LaTeX, a higher-level markup language built on top of TeX. 

It is had to add new functionality in TeX.  At the same time, TeX great at
optimizing line and page breaks.  So it would be hard to replace TeX
without a big loss of quality in output.

I know little about LaTeX, but I thought it was nicer commands
built on the same engine as TeX.  What would we gain by using LaTeX
as the base instead of Plain TeX?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Werner LEMBERG]

> I know little about LaTeX, but I thought it was nicer commands built
> on the same engine as TeX.  What would we gain by using LaTeX as the
> base instead of Plain TeX?

There are a lot of severe drawbacks in the current `texinfo.tex`
implementation that cannot or will not be fixed.  The most serious are
the following.

* Texinfo exclusively uses the Computer Modern fonts with its 7-bit(!)
  OT1 font encoding.  This only supports a very limited set of
  (natural) languages with proper hyphenation and kerning.  To say it
  bluntly and slightly exaggerating, it delivers good typography for
  US English and nothing else.

* Many people don't like the appearance of the Computer Modern fonts.
  However, it is very hard to change to a different font family,
  especially because you have to create proper support for the OT1
  encoding, which is a non-trivial undertaking.  Additionally, you
  have to directly modify internal `texinfo.tex` macros since Texinfo
  doesn't provide a proper API to do so.

New versions of `texi2any` directly create LaTeX output files,
completely bypassing `texinfo.tex` and its limitations.  It is thus
possible to use the full typographical power of modern TeX engines,
which, for example, directly support Unicode and thus essentially all
languages of the world.


    Werner

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > * Texinfo exclusively uses the Computer Modern fonts with its 7-bit(!)
  >   OT1 font encoding.  This only supports a very limited set of
  >   (natural) languages with proper hyphenation and kerning.

That is a significant improvement.  Could you confirm that we already
support output via LaTeX (via texi2any)?

If so, is the reason people don't always use that due to the time cost
of running texi2any first?

How does the soeed if LeTeX compare with the speed of texinfo.tex?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Werner LEMBERG]

>   > * Texinfo exclusively uses the Computer Modern fonts with its
>   >   7-bit(!)  OT1 font encoding.  This only supports a very
>   >   limited set of (natural) languages with proper hyphenation and
>   >   kerning.
> 
> That is a significant improvement.  Could you confirm that we already
> support output via LaTeX (via texi2any)?

Who is 'we'?  If you mean the Emacs maintainers, I think that nobody
has tried yet to compile the Emacs documentation with the LaTeX
backend, but basically I don't expect any large issues (Eli certainly
knows more).  And sorry, I don't have time for testing a possible
LaTeX compilation by myself.

AFAICS, the whole Emacs documentation (except the tutorials) is
English only; in other words, there is no urgent need to change the
setup right now in any way.

> If so, is the reason people don't always use that due to the time
> cost of running texi2any first?

Compilation time is not an issue currently, I believe.  LaTeX support
in Texinfo is very new and still under heavy development AFAIK.  I
reckon that within the next few years many teams will try – and
possibly change to – the LaTeX front-end.

> How does the soeed if LeTeX compare with the speed of texinfo.tex?

Given that the recent version of `texi2any` is a Perl script, and that
LaTeX itself is slower than Texinfo, I assume that the combination of
the two are *far* slower than a direct call to plain TeX using
`texinfo.tex`.


    Werner

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Wed, 13 Sep 2023 16:52:15 -0400
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> I think I was more interested in why you said it would be a
> "text-processing system" rather than an editor or even IDE

Emacs is more than an editor and an IDE.  Gnus (and other MUAs we have
in Emacs) is neither, and neither is Org or ERC or any number of other
Emacs features.

> >   . display engine design around the rectangular canvas model and

> I don't know what the alternative is, exactly.

Finding that out is part of the job.  It's quite likely (I hope) that
the relevant technology already exists, and we don't need to invent
it.

> VS Code and other IDEs I've used have more varied decorative
> elements and containers, but text is still pretty much presented in
> rectangular containers with sides parallel to the GUI window (frame
> in Emacs terms).

You misunderstood what I meant by "rectangular canvas model", I think.
In Emacs, every screen line is represented as a "glyph row", a linear
array of glyphs, and the window's display is represented as a liner
array of glyph rows.  This is why we cannot (easily) have an image
that spans more than one screen line, and why we cannot have display
elements (like images or text boxes) overlaid on top of buffer text.

Re: Emacs design and architecture

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lynn Winebarger <owinebar@gmail.com>
>> Date: Wed, 13 Sep 2023 16:52:15 -0400
>> Cc: rms@gnu.org, emacs-devel@gnu.org
>> 
>> I think I was more interested in why you said it would be a
>> "text-processing system" rather than an editor or even IDE
>
> Emacs is more than an editor and an IDE.  Gnus (and other MUAs we have
> in Emacs) is neither, and neither is Org or ERC or any number of other
> Emacs features.
>
>> >   . display engine design around the rectangular canvas model and
>
>> I don't know what the alternative is, exactly.
>
> Finding that out is part of the job.  It's quite likely (I hope) that
> the relevant technology already exists, and we don't need to invent
> it.

Right.

This makes me think immediately of browser technology.  We all probably
know what can be done in web layouts, I guess.  As an example that it
can be used for an editor, take a look at the Monaco editor, which is
used in VSCode:

  https://github.com/microsoft/monaco-editor

(I'm not saying that should be done, or even that it could be done, or
anything :-)).

Re: Emacs design and architecture

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> This makes me think immediately of browser technology.  We all probably
> know what can be done in web layouts, I guess.  As an example that it
> can be used for an editor, take a look at the Monaco editor, which is
> used in VSCode:
>
>   https://github.com/microsoft/monaco-editor
>
> (I'm not saying that should be done, or even that it could be done, or
> anything :-)).

The screenshots in its description sure resemble a ``rectangular canvas
of rows'' to me, and I can't find its source code to establish if it is
implemented any differently.  Any pointers?

Thanks.

Re: Emacs design and architecture

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> This makes me think immediately of browser technology.  We all probably
>> know what can be done in web layouts, I guess.  As an example that it
>> can be used for an editor, take a look at the Monaco editor, which is
>> used in VSCode:
>>
>>   https://github.com/microsoft/monaco-editor
>>
>> (I'm not saying that should be done, or even that it could be done, or
>> anything :-)).
>
> The screenshots in its description sure resemble a ``rectangular canvas
> of rows'' to me, and I can't find its source code to establish if it is
> implemented any differently.  Any pointers?

Sorry, I didn't mean to imply that this editor is somehow
non-rectangular.

What I meant is that (a) this technology has advanced layout
capabilities, and (b) can obviously be used for interactive
applications, like an editor.

Re: Emacs design and architecture

[Helmut Eller]

On Thu, Sep 14 2023, Gerd Möllmann wrote:

> What I meant is that (a) this technology has advanced layout
> capabilities, and (b) can obviously be used for interactive
> applications, like an editor.

So has nobody yet written a front-end for Emacs that runs in a web
browser?

Helmut

Re: Emacs design and architecture

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Thu, Sep 14 2023, Gerd Möllmann wrote:
>
>> What I meant is that (a) this technology has advanced layout
>> capabilities, and (b) can obviously be used for interactive
>> applications, like an editor.
>
> So has nobody yet written a front-end for Emacs that runs in a web
> browser?

Not a front-end, but I found this quite funny (it's a few years old):

  http://brettcvz.github.io/ymacs/demo/

Re: Emacs design and architecture

[Helmut Eller]

On Thu, Sep 14 2023, Gerd Möllmann wrote:

> Helmut Eller <eller.helmut@gmail.com> writes:
>> So has nobody yet written a front-end for Emacs that runs in a web
>> browser?
>
> Not a front-end, but I found this quite funny (it's a few years old):
>
>   http://brettcvz.github.io/ymacs/demo/

Yes, that's pretty cool.  I wonder if it's possible to bind C-w or if
that is somehow disallowed by the browser.  C-r seems to work fine.

There exist also various terminal emulators like

  https://domterm.org/

that run partially in a web browser and that can actually run Emacs.  I
guess those could also, via protocol extension, be used to display
better graphics.

Helmut

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Helmut Eller <eller.helmut@gmail.com> writes:
  > >> So has nobody yet written a front-end for Emacs that runs in a web
  > >> browser?

In a strategic moral sense that would be a bad direction to go.  Most
users use a nonfree web browser, or a mostly free but nasty browser
such as Chromium.  The ones that are free are noncopylefted.  To
change Emacs into something people usually run under a browser would
be an enormous change for the worse in regard to giving users freedom.

The level of convenience is not th e most important one.  We must
judge significant design issues in terms of these questions: advancing
the territory of free software, of copyleft, and of separation from
unjust software.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Björn Bidar]

Richard Stallman <rms@gnu.org> writes:

>   > > Helmut Eller <eller.helmut@gmail.com> writes:
>   > >> So has nobody yet written a front-end for Emacs that runs in a web
>   > >> browser?
>
> In a strategic moral sense that would be a bad direction to go.  Most
> users use a nonfree web browser, or a mostly free but nasty browser
> such as Chromium.  The ones that are free are noncopylefted.  To
> change Emacs into something people usually run under a browser would
> be an enormous change for the worse in regard to giving users freedom.

The topic is already off from the original topic but because standards
are so open the user is free to choose which browser to use. One benefit
is that even if the user is forced to use non-free browsers that they
still can run free software running inside said browser.

In theory building with pure GTK already allows Emacs to run in a
browser (GTK broadway backend).

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > because standards
  > are so open the user is free to choose which browser to use.

That assertion would be true, supposing that Emacs communicates with
the browser using only standard protocols and some free browser
supports them.  I don't think that is what people are proposing though.

But if people implemented that method with standard protocols, the
problem I bring up would happen anyway.  If most users were to
communicate with Emacs using Chrome, which is non-free,
Emacs would become an incomplere editor that people usually
complete using a nonfree program.

That would be a terrible setback for the GNU Project and the
free software movement.  Any gains in convenience would be
a side issue compared with that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Philip Kaludercic]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Thu, Sep 14 2023, Gerd Möllmann wrote:
>
>> What I meant is that (a) this technology has advanced layout
>> capabilities, and (b) can obviously be used for interactive
>> applications, like an editor.
>
> So has nobody yet written a front-end for Emacs that runs in a web
> browser?

Using PGTK it is allegedly possible to have Emacs run in a browser[0],
but on my system the command fails with

$ GDK_BACKEND=broadway BROADWAY_DISPLAY=:6  lemacs -Q

(emacs:44517): Gtk-WARNING **: 18:23:14.025: cannot open display: :0


[0] https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg01442.html, 

Re: Emacs design and architecture

[joakim]

Philip Kaludercic <philipk@posteo.net> writes:

> Helmut Eller <eller.helmut@gmail.com> writes:
>
>> On Thu, Sep 14 2023, Gerd Möllmann wrote:
>>
>>> What I meant is that (a) this technology has advanced layout
>>> capabilities, and (b) can obviously be used for interactive
>>> applications, like an editor.
>>
>> So has nobody yet written a front-end for Emacs that runs in a web
>> browser?
>
> Using PGTK it is allegedly possible to have Emacs run in a browser[0],
> but on my system the command fails with
>
> $ GDK_BACKEND=broadway BROADWAY_DISPLAY=:6  lemacs -Q
>
> (emacs:44517): Gtk-WARNING **: 18:23:14.025: cannot open display: :0
>
>
> [0] https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg01442.html, 

This worked for me a couple of years ago. Havent tried recently.
>
-- 
Joakim Verona
joakim@verona.se

Re: Emacs design and architecture (was: Shrinking the C core)

[Lynn Winebarger]

On Thu, Sep 14, 2023 at 1:58 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Wed, 13 Sep 2023 16:52:15 -0400
> > Cc: rms@gnu.org, emacs-devel@gnu.org
> >
> > I think I was more interested in why you said it would be a
> > "text-processing system" rather than an editor or even IDE
>
> Emacs is more than an editor and an IDE.  Gnus (and other MUAs we have
> in Emacs) is neither, and neither is Org or ERC or any number of other
> Emacs features.

That makes sense.  The term "text-processing system" just evoked
something non-interactive to me, like sed, awk, tex, etc.


>
> > >   . display engine design around the rectangular canvas model and
>
> > I don't know what the alternative is, exactly.
>
> Finding that out is part of the job.  It's quite likely (I hope) that
> the relevant technology already exists, and we don't need to invent
> it.
>
> > VS Code and other IDEs I've used have more varied decorative
> > elements and containers, but text is still pretty much presented in
> > rectangular containers with sides parallel to the GUI window (frame
> > in Emacs terms).
>
> You misunderstood what I meant by "rectangular canvas model", I think.
> In Emacs, every screen line is represented as a "glyph row", a linear
> array of glyphs, and the window's display is represented as a liner
> array of glyph rows.  This is why we cannot (easily) have an image
> that spans more than one screen line, and why we cannot have display
> elements (like images or text boxes) overlaid on top of buffer text.

Could this be subdivided into two design issues:
* Decoupling the display (or view?) from the buffer being displayed
* Providing a more flexible canvas

So under the first would be doing something like interposing objects
representing syntactic entities in between the display and the buffer,
so the user could interact directly with those objects, as opposed to
having those objects attached to text intervals or overlays, and
having the interaction backed into.

For the other, it sounds like you'd like to have more GUI primitive
type operations available on the canvas directly, without having to
use explicit GUI objects like child frames.  But I have no idea how
far you would want to go in replicating GUI functions in the emacs
graphics display.  It's just not my area.  I'll go out on a limb and
say that to be interesting, the additional flexibility in the canvas
would have to be programmable from emacs lisp, and that would require
emacs lisp to be much faster than it is now, and essentially
parallelizable in a way that it is not now?  Those are things I can
work on, though they are not short term projects.



Lynn

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Thu, 14 Sep 2023 12:30:27 -0400
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> > You misunderstood what I meant by "rectangular canvas model", I think.
> > In Emacs, every screen line is represented as a "glyph row", a linear
> > array of glyphs, and the window's display is represented as a liner
> > array of glyph rows.  This is why we cannot (easily) have an image
> > that spans more than one screen line, and why we cannot have display
> > elements (like images or text boxes) overlaid on top of buffer text.
> 
> Could this be subdivided into two design issues:
> * Decoupling the display (or view?) from the buffer being displayed
> * Providing a more flexible canvas
> 
> So under the first would be doing something like interposing objects
> representing syntactic entities in between the display and the buffer,
> so the user could interact directly with those objects, as opposed to
> having those objects attached to text intervals or overlays, and
> having the interaction backed into.

How is this different from display properties, overlays and images,
which we already have?

The problem is not to display objects that don't come from buffer
text: we already have that.  The problem is that our layout engine
cannot superimpose one object on top of the other, and the geometry of
the layout is hard-coded as "glyph rows".  IOW, it's a pure display
layout problem, not a problem with decoupling display from buffers.

> For the other, it sounds like you'd like to have more GUI primitive
> type operations available on the canvas directly, without having to
> use explicit GUI objects like child frames.

We cannot draw on the canvas directly without making the display
engine aware of what we are drawing and where, because the display
engine is responsible for updating the display when something changes.
So what we need is to be able to place a glyph using arbitrary 3D
coordinates (the 3rd coordinate for overlaying stuff on top of what's
already displayed), instead of having to work with rows of glyphs.

Again, studying what others do in this area would be useful, I think.

Re: Emacs design and architecture (was: Shrinking the C core)

[Dmitry Gutov]

On 14/09/2023 19:52, Eli Zaretskii wrote:
>> For the other, it sounds like you'd like to have more GUI primitive
>> type operations available on the canvas directly, without having to
>> use explicit GUI objects like child frames.
> We cannot draw on the canvas directly without making the display
> engine aware of what we are drawing and where, because the display
> engine is responsible for updating the display when something changes.
> So what we need is to be able to place a glyph using arbitrary 3D
> coordinates (the 3rd coordinate for overlaying stuff on top of what's
> already displayed), instead of having to work with rows of glyphs.
> 
> Again, studying what others do in this area would be useful, I think.

Perhaps the classic case is the Mozilla browser, which has from the 
beginning implemented its browser UI (chrome) using an HTML-like 
technology called XUL which also uses CSS and JavaScript for styling 
(colors, size, alignment, positioning) and scripting. Except XUL 
provided access to the platform widgets, graphical elements, menus, 
dialogs, etc. Still, it's like a web page for displaying web pages, in a 
sense.

Not sure how relevant that is for Emacs: doing a wrapper for an OS 
toolkit is a lot of work.

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> Date: Fri, 15 Sep 2023 00:35:12 +0300
> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/09/2023 19:52, Eli Zaretskii wrote:
> >> For the other, it sounds like you'd like to have more GUI primitive
> >> type operations available on the canvas directly, without having to
> >> use explicit GUI objects like child frames.
> > We cannot draw on the canvas directly without making the display
> > engine aware of what we are drawing and where, because the display
> > engine is responsible for updating the display when something changes.
> > So what we need is to be able to place a glyph using arbitrary 3D
> > coordinates (the 3rd coordinate for overlaying stuff on top of what's
> > already displayed), instead of having to work with rows of glyphs.
> > 
> > Again, studying what others do in this area would be useful, I think.
> 
> Perhaps the classic case is the Mozilla browser, which has from the 
> beginning implemented its browser UI (chrome) using an HTML-like 
> technology called XUL which also uses CSS and JavaScript for styling 
> (colors, size, alignment, positioning) and scripting. Except XUL 
> provided access to the platform widgets, graphical elements, menus, 
> dialogs, etc. Still, it's like a web page for displaying web pages, in a 
> sense.
> 
> Not sure how relevant that is for Emacs: doing a wrapper for an OS 
> toolkit is a lot of work.

One of the important aspects to keep in mind in this regard is that
Emacs must give Lisp programs dynamic control of how stuff is
displayed, and should be able to exercise that control at high
frequency (a trivial example: pulse.el).  So I'm not sure CSS is the
right means for this job to begin with, although it perhaps could take
care of rarely-changing defaults.

Another thing to keep in mind is that there should be an efficient way
of telling the display back-end which portions of a window to redraw
given some changes in the text and display-related data structures.
(Of course, if the display back-end is so efficient that it can
redisplay everything fast enough, even if displaying across a network
on a different terminal, this aspect might be a non-issue.)

Re: Emacs design and architecture (was: Shrinking the C core)

[Yuri Khan]

On Fri, 15 Sept 2023 at 12:51, Eli Zaretskii <eliz@gnu.org> wrote:

> One of the important aspects to keep in mind in this regard is that
> Emacs must give Lisp programs dynamic control of how stuff is
> displayed, and should be able to exercise that control at high
> frequency (a trivial example: pulse.el).

In CSS, this is solved in core with transitions. A style specifies
that a certain property will change gradually, provides its target
value, the transition duration, and a transition curve. The styling
engine does all the work about calculating the intermediate values and
re-layouting each intermediate frame; the page author does not have to
script carefully timed property changes, and is in fact discouraged
from doing so because doing that in Javascript incurs CPU and battery
usage overhead.

The use case of pulse.el would be translated to a couple of styles
that say effectively “A pulsed span will instantly gain yellow
background” and “A non-pulsed span will linearly revert to whatever
background it had over the course of 200 milliseconds” and a small
function that sets the span to pulsed and then immediately to
non-pulsed.

(Implementing a CSS engine over a character terminal is a nontrivial
matter though.)

Re: Emacs design and architecture

[Max Brieiev]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Fri, 15 Sept 2023 at 12:51, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> One of the important aspects to keep in mind in this regard is that
>> Emacs must give Lisp programs dynamic control of how stuff is
>> displayed, and should be able to exercise that control at high
>> frequency (a trivial example: pulse.el).
>
> In CSS, this is solved in core with transitions. A style specifies
> that a certain property will change gradually, provides its target
> value, the transition duration, and a transition curve. The styling
> engine does all the work about calculating the intermediate values and
> re-layouting each intermediate frame; the page author does not have to
> script carefully timed property changes, and is in fact discouraged
> from doing so because doing that in Javascript incurs CPU and battery
> usage overhead.
>
> The use case of pulse.el would be translated to a couple of styles
> that say effectively “A pulsed span will instantly gain yellow
> background” and “A non-pulsed span will linearly revert to whatever
> background it had over the course of 200 milliseconds” and a small
> function that sets the span to pulsed and then immediately to
> non-pulsed.
>
> (Implementing a CSS engine over a character terminal is a nontrivial
> matter though.)

CSS is tricky and very hard to get right. This is the reason CSS
frameworks like Tailwind or Bootstrap exist.

I think building user interfaces with SVG is more plausible. It is the
same DOM model, it is reactive, and its display is very consistent
(unlike HTML/CSS combo).

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Fri, 15 Sep 2023 13:51:56 +0700
> Cc: Dmitry Gutov <dmitry@gutov.dev>, owinebar@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> On Fri, 15 Sept 2023 at 12:51, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > One of the important aspects to keep in mind in this regard is that
> > Emacs must give Lisp programs dynamic control of how stuff is
> > displayed, and should be able to exercise that control at high
> > frequency (a trivial example: pulse.el).
> 
> In CSS, this is solved in core with transitions. A style specifies
> that a certain property will change gradually, provides its target
> value, the transition duration, and a transition curve.

You have taken the pulse.el example too literally.  The fact that it
changes the color gradually is not relevant to the point I was trying
to make, but you made it the main point.

My point is that quite a few Lisp programs affect the display in
near-real time and at high frequency.  This is what makes Emacs so
powerful, and we don't want to lose this power when changing the
display engine.

> The use case of pulse.el would be translated to a couple of styles
> that say effectively “A pulsed span will instantly gain yellow
> background” and “A non-pulsed span will linearly revert to whatever
> background it had over the course of 200 milliseconds” and a small
> function that sets the span to pulsed and then immediately to
> non-pulsed.

This sounds like a lot of hair, when a Lisp program just wants to
change the color of some part of the display.

> (Implementing a CSS engine over a character terminal is a nontrivial
> matter though.)

Something else to keep in mind, I guess.  TTY colors are implemented
specially and separately in Emacs (under the hood; Lisp programs can
disregard the differences if they want), so it isn't a non-starter,
per se.

Re: Emacs design and architecture

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yuri Khan <yuri.v.khan@gmail.com>
>> Date: Fri, 15 Sep 2023 13:51:56 +0700
>> Cc: Dmitry Gutov <dmitry@gutov.dev>, owinebar@gmail.com, rms@gnu.org, emacs-devel@gnu.org
>> 
>> On Fri, 15 Sept 2023 at 12:51, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> > One of the important aspects to keep in mind in this regard is that
>> > Emacs must give Lisp programs dynamic control of how stuff is
>> > displayed, and should be able to exercise that control at high
>> > frequency (a trivial example: pulse.el).
>> 
>> In CSS, this is solved in core with transitions. A style specifies
>> that a certain property will change gradually, provides its target
>> value, the transition duration, and a transition curve.
>
> You have taken the pulse.el example too literally.  The fact that it
> changes the color gradually is not relevant to the point I was trying
> to make, but you made it the main point.
>
> My point is that quite a few Lisp programs affect the display in
> near-real time and at high frequency.  This is what makes Emacs so
> powerful, and we don't want to lose this power when changing the
> display engine.
>
>> The use case of pulse.el would be translated to a couple of styles
>> that say effectively “A pulsed span will instantly gain yellow
>> background” and “A non-pulsed span will linearly revert to whatever
>> background it had over the course of 200 milliseconds” and a small
>> function that sets the span to pulsed and then immediately to
>> non-pulsed.
>
> This sounds like a lot of hair, when a Lisp program just wants to
> change the color of some part of the display.
>
>> (Implementing a CSS engine over a character terminal is a nontrivial
>> matter though.)
>
> Something else to keep in mind, I guess.  TTY colors are implemented
> specially and separately in Emacs (under the hood; Lisp programs can
> disregard the differences if they want), so it isn't a non-starter,
> per se.

I think it would maybe be good to think about the following:

Random thoughts about a parallel redisplay, from a height of 10 km.

What currently happens to bring changes to the screen is that redisplay
is called quite frequently in the course of processing input for
example.  Redisplay determines what part of the a "model" (buffer) has
changed, if any.  It makes sure that all info it needs to proceed is
available; think jit-lock, i.e. it calls Lisp. Because redisplay is
called frequently, it must minimize what it does, which is the reason
for the complicated optimizations there.

Whatever is done in the end, I think it would first be necessary to
change this general principle, so that layout/drawing whatever can
happen in parallel.  Without that, I suspect redisplay would get too
slow, or would finally collapse to a black hole by its complexity.

This of course, would pose several problems.

- This form of parallel redisplay cannot call Lisp, so no jit-lock, no
  hooks, or whatever during redisplay.  And at the point where we call
  redisplay today we don't know what will be displayed...
- Parallel redisplay also needs either a copy of what it to be
  displayed, or the model must be some persistent data structure
  that makes immutable versions of buffer-text, for instance,
  available.

Re: Emacs design and architecture

[Dmitry Gutov]

On 15/09/2023 12:32, Gerd Möllmann wrote:
> I think it would maybe be good to think about the following:
> 
> Random thoughts about a parallel redisplay, from a height of 10 km.
> 
> What currently happens to bring changes to the screen is that redisplay
> is called quite frequently in the course of processing input for
> example.  Redisplay determines what part of the a "model" (buffer) has
> changed, if any.  It makes sure that all info it needs to proceed is
> available; think jit-lock, i.e. it calls Lisp. Because redisplay is
> called frequently, it must minimize what it does, which is the reason
> for the complicated optimizations there.
> 
> Whatever is done in the end, I think it would first be necessary to
> change this general principle, so that layout/drawing whatever can
> happen in parallel.  Without that, I suspect redisplay would get too
> slow, or would finally collapse to a black hole by its complexity.

I think it'd be an interesting project to study how an existing Emacs 
could output into a web page. Or a web driver, etc. How a "toolkit port" 
into HTML/JS could work.

That system is historically more complex than what you described, 
including lots of callbacks, e.g. it might be possible to implement 
jit-lock using the 'DOMContentLoaded' and 'scroll' events.

Consequently, though, parallel layout has been historically a hard 
problem for web browsers, but the current engines do that, at least to 
an extend (I think that was one of the selling features of Servo, later 
incorporated into Gecko). Complexity is definitely an issue, though.

Re: Emacs design and architecture

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 15/09/2023 12:32, Gerd Möllmann wrote:
>> I think it would maybe be good to think about the following:
>> Random thoughts about a parallel redisplay, from a height of 10 km.
>> What currently happens to bring changes to the screen is that
>> redisplay
>> is called quite frequently in the course of processing input for
>> example.  Redisplay determines what part of the a "model" (buffer) has
>> changed, if any.  It makes sure that all info it needs to proceed is
>> available; think jit-lock, i.e. it calls Lisp. Because redisplay is
>> called frequently, it must minimize what it does, which is the reason
>> for the complicated optimizations there.
>> Whatever is done in the end, I think it would first be necessary to
>> change this general principle, so that layout/drawing whatever can
>> happen in parallel.  Without that, I suspect redisplay would get too
>> slow, or would finally collapse to a black hole by its complexity.
>
> I think it'd be an interesting project to study how an existing Emacs
> could output into a web page. Or a web driver, etc. How a "toolkit
> port" into HTML/JS could work.
>
> That system is historically more complex than what you described,
> including lots of callbacks, e.g. it might be possible to implement
> jit-lock using the 'DOMContentLoaded' and 'scroll' events.
>
> Consequently, though, parallel layout has been historically a hard
> problem for web browsers, but the current engines do that, at least to
> an extend (I think that was one of the selling features of Servo,
> later incorporated into Gecko). Complexity is definitely an issue,
> though.

Maybe it would be worth looking at emacs-ng

  https://github.com/emacs-ng/emacs-ng

The README lists the feature

  Webrender

  WebRender is a GPU-based 2D rendering engine written in Rust from
  Mozilla. Firefox, the research web browser Servo, and other GUI
  frameworks draw with it. emacs-ng use it as a new experimental graphic
  backend to leverage GPU hardware.

Re: Emacs design and architecture

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Yuri Khan <yuri.v.khan@gmail.com>,
>   owinebar@gmail.com,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 15 Sep 2023 20:36:07 +0200
> 
> Maybe it would be worth looking at emacs-ng
> 
>   https://github.com/emacs-ng/emacs-ng
> 
> The README lists the feature
> 
>   Webrender
> 
>   WebRender is a GPU-based 2D rendering engine written in Rust from
>   Mozilla. Firefox, the research web browser Servo, and other GUI
>   frameworks draw with it. emacs-ng use it as a new experimental graphic
>   backend to leverage GPU hardware.

I didn't look at this, but if by "graphic backend" they mean a
replacement for xterm.c, then this is much less interesting, because
the basic limitations of the current display engine's layout and
iterator (which are all implemented in xdisp.c and dispnew.c) will
still be with us.

Re: Emacs design and architecture

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Yuri Khan <yuri.v.khan@gmail.com>,
>>   owinebar@gmail.com,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Fri, 15 Sep 2023 20:36:07 +0200
>> 
>> Maybe it would be worth looking at emacs-ng
>> 
>>   https://github.com/emacs-ng/emacs-ng
>> 
>> The README lists the feature
>> 
>>   Webrender
>> 
>>   WebRender is a GPU-based 2D rendering engine written in Rust from
>>   Mozilla. Firefox, the research web browser Servo, and other GUI
>>   frameworks draw with it. emacs-ng use it as a new experimental graphic
>>   backend to leverage GPU hardware.
>
> I didn't look at this, but if by "graphic backend" they mean a
> replacement for xterm.c, then this is much less interesting, because
> the basic limitations of the current display engine's layout and
> iterator (which are all implemented in xdisp.c and dispnew.c) will
> still be with us.

I've cloned the repo now, and it seems indeed to be a backend like xterm
or nsterm etc.  It's called wrterm, and is implemented in ca. 8.5 kloc
of Rust.  I can't read Rust fluently, but I'd say It implements the
usual functions for such a backend and not more.

So, I agree, that's not very interesting in this context.

Re: Emacs design and architecture

[Dmitry Gutov]

On 15/09/2023 22:19, Gerd Möllmann wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>>> From: Gerd Möllmann<gerd.moellmann@gmail.com>
>>> Cc: Eli Zaretskii<eliz@gnu.org>,  Yuri Khan<yuri.v.khan@gmail.com>,
>>>    owinebar@gmail.com,rms@gnu.org,emacs-devel@gnu.org
>>> Date: Fri, 15 Sep 2023 20:36:07 +0200
>>>
>>> Maybe it would be worth looking at emacs-ng
>>>
>>>    https://github.com/emacs-ng/emacs-ng
>>>
>>> The README lists the feature
>>>
>>>    Webrender
>>>
>>>    WebRender is a GPU-based 2D rendering engine written in Rust from
>>>    Mozilla. Firefox, the research web browser Servo, and other GUI
>>>    frameworks draw with it. emacs-ng use it as a new experimental graphic
>>>    backend to leverage GPU hardware.
>> I didn't look at this, but if by "graphic backend" they mean a
>> replacement for xterm.c, then this is much less interesting, because
>> the basic limitations of the current display engine's layout and
>> iterator (which are all implemented in xdisp.c and dispnew.c) will
>> still be with us.
> I've cloned the repo now, and it seems indeed to be a backend like xterm
> or nsterm etc.  It's called wrterm, and is implemented in ca. 8.5 kloc
> of Rust.  I can't read Rust fluently, but I'd say It implements the
> usual functions for such a backend and not more.
> 
> So, I agree, that's not very interesting in this context.

Yes, looks like it renders everything to a <canvas> element. Which is 
not so interesting from the layout POV - I had in mind a translation 
with more different HTML elements, I guess.

Practically speaking, it is a good choice. I recall some development 
blog post either for Atom or for VS Code where they described a decision 
to move from HTML elements to rendering the code file contents on a 
canvas as well. GitHub's online text editor uses <textarea> now (so it's 
probably the same in VS Code), although how the syntax highlighting gets 
applied anyway I don't quite understand.

Anyway, the previous approach was functional as well, if slower with 
larger files. But Atom or VS Code don't support free-form layout or 
embedding images in the buffer text, I think. The "newmacs" could as 
well have both kinds of buffers (one for large files/performance, 
another for advanced layout features).

Even if <canvas> is used for buffers, it doesn't have to be used for 
"chrome" (menus, buttons, window delimiters, fringes, mode lines). Using 
HTML (for example) just for that could bring the ability to render stuff 
on top of it all, such as alerts, popups, etc.

Microsoft also has a project that could be tried as a base (MIT 
Licensed): https://github.com/microsoft/vscode-webview-ui-toolkit. Or 
one could just use its internals for inspiration, because "Visual Studio 
Code design language" is probably not one of our goals.

Re: Emacs design and architecture

[Emanuel Berg]

When we speak of parallelism, i.e. the use of multicore CPUs,
that is, true concurrency, are we talking Lisp parallelism for
constructs that are parallel in nature, e.g.

(let ((a execute-on-core-one)
      (b execute-on-core-two) )
  (+ a b) )

or are we talking parallelism in the sense to make Emacs as
a program make use of multicores? One could then think of
several arrangement, for example, do display on one core, user
input on the next, and so on?

The former, Lisp parallelism, would be amazingly cool, but
maybe the latter would bring more actual benefit since one
could, if modules were skillfully separated, do lots of
optimization for their specific purposes and needs.

Maybe one could do both since they are not, uhm,
mutually exclusive?

Microkernel Emacs with concurrent Lisp anyone?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 16 Sep 2023 01:58:42 +0200
> 
> When we speak of parallelism, i.e. the use of multicore CPUs,
> that is, true concurrency, are we talking Lisp parallelism for
> constructs that are parallel in nature, e.g.
> 
> (let ((a execute-on-core-one)
>       (b execute-on-core-two) )
>   (+ a b) )
> 
> or are we talking parallelism in the sense to make Emacs as
> a program make use of multicores?

We are talking about both.  Except that true parallelism doesn't care
which core will do what job, but leaves that to the OS, asking only
that each job be done independently of the other. i.e. usually by
different cores.

> The former, Lisp parallelism, would be amazingly cool, but
> maybe the latter would bring more actual benefit since one
> could, if modules were skillfully separated, do lots of
> optimization for their specific purposes and needs.

If you know how to do one of them, you also know how to do the other,
because Emacs is a Lisp machine.

Re: Emacs design and architecture

[Emanuel Berg]

Eli Zaretskii wrote:

> If you know how to do one of them, you also know how to do
> the other, because Emacs is a Lisp machine.

How do other applications, that are multi-threaded, solve the
same obstacle, i.e. that of concurrent access to a big, shared
global state?

Because if the whole thing has to be locked for each thread to
access, it can be disputed if that is indeed any parallelism
at all. It will be multi-threaded and multi-core alright, but
not parallel execution unless one counts waiting for a shared
resource to become available as a sensible passivity.

So one would have to have a more fine-grained access to the
resource - by splitting it up in pieces. That way one thread
could access r_i while another accesses r_j and so on.

Don't know how such a division would look in practice tho?
Again it would be interesting to hear of how other
applications are doing it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sun, 17 Sep 2023 14:16:57 +0200
> 
> Eli Zaretskii wrote:
> 
> > If you know how to do one of them, you also know how to do
> > the other, because Emacs is a Lisp machine.
> 
> How do other applications, that are multi-threaded, solve the
> same obstacle, i.e. that of concurrent access to a big, shared
> global state?

Mult-threading and global state are two opposites: an application that
wants to be multi-threaded should have as small a global state as
possible, and where it cannot be avoided, use locking to access global
state from different threads.

> Because if the whole thing has to be locked for each thread to
> access, it can be disputed if that is indeed any parallelism
> at all. It will be multi-threaded and multi-core alright, but
> not parallel execution unless one counts waiting for a shared
> resource to become available as a sensible passivity.

That's what we have with Lisp threads now: while one thread runs, all
the others are stopped by a global lock.

> So one would have to have a more fine-grained access to the
> resource - by splitting it up in pieces. That way one thread
> could access r_i while another accesses r_j and so on.

This was discussed here earlier, and ion is that it's easier said than
done.

> Again it would be interesting to hear of how other
> applications are doing it.

There's a separate mutex for each global data structure.

Re: Emacs design and architecture. How about copy-on-write?

[Alan Mackenzie]

Hello, Eli.

On Sun, Sep 17, 2023 at 17:24:18 +0300, Eli Zaretskii wrote:
> > From: Emanuel Berg <incal@dataswamp.org>
> > Date: Sun, 17 Sep 2023 14:16:57 +0200

[ .... ]

> Mult-threading and global state are two opposites: an application that
> wants to be multi-threaded should have as small a global state as
> possible, and where it cannot be avoided, use locking to access global
> state from different threads.

> > Because if the whole thing has to be locked for each thread to
> > access, it can be disputed if that is indeed any parallelism
> > at all. It will be multi-threaded and multi-core alright, but
> > not parallel execution unless one counts waiting for a shared
> > resource to become available as a sensible passivity.

> That's what we have with Lisp threads now: while one thread runs, all
> the others are stopped by a global lock.

> > So one would have to have a more fine-grained access to the
> > resource - by splitting it up in pieces. That way one thread
> > could access r_i while another accesses r_j and so on.

> This was discussed here earlier, and ion is that it's easier said than
> done.

How about copy-on-write at a symbol
value-cell/function-cell/property-list level?

A value in a value-cell (etc.) would have an associated thread value, the
thread which "owns" it.  When thread1 gets cloned from thread0, it
continues to use thread0's data state until it writes foo (e.g. by
binding it).  This would create a new value/thread pair, foo/thread1.
Further writes of foo by thread1 would continue to access foo/thread1.

Possibly, we could use just one owning thread for all the cells of a
symbol.

When a thread terminates, all its owned variables would become garbage,
to be collected by our new garbage collector.  ;-)

Clearly, there would have to be some variables which would be
thread-global, i.e. there would be a single value cell shared by all
threads.  There would also doubltless be other complications.

The advantage of this approach, if it could be made to work, is that it
doesn't try to fight the massive global state which is Emacs Lisp,
instead accepting it and embracing it.

What do you think?

> > Again it would be interesting to hear of how other
> > applications are doing it.

> There's a separate mutex for each global data structure.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Sun, 17 Sep 2023 15:36:11 +0000
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> How about copy-on-write at a symbol
> value-cell/function-cell/property-list level?
> 
> A value in a value-cell (etc.) would have an associated thread value, the
> thread which "owns" it.  When thread1 gets cloned from thread0, it
> continues to use thread0's data state until it writes foo (e.g. by
> binding it).  This would create a new value/thread pair, foo/thread1.
> Further writes of foo by thread1 would continue to access foo/thread1.
> 
> Possibly, we could use just one owning thread for all the cells of a
> symbol.
> 
> When a thread terminates, all its owned variables would become garbage,
> to be collected by our new garbage collector.  ;-)

You assume that whatever the thread does can be discarded as garbage?
That's definitely not true in general: most things we do in Emacs
should leave some trace behind.  It is not clear when and how this
would happen under your proposal.  E.g., imagine that a thread runs
Gnus fetching articles, and try to describe how this will work.

> Clearly, there would have to be some variables which would be
> thread-global, i.e. there would be a single value cell shared by all
> threads.  There would also doubltless be other complications.

"Some variables"?  Our problem is that their name is a legion.  E.g.,
what do you propose to do with variables and changes to buffer text
that affect the display?  Or are you suggesting to have a separate
redisplay for each thread?  Or maybe you propose that each window has
its own thread, complete with its own display (and a separate thread
for each frame)?

These aspects need to be figured out if we want to discuss something
like this seriously.

> The advantage of this approach, if it could be made to work, is that it
> doesn't try to fight the massive global state which is Emacs Lisp,
> instead accepting it and embracing it.

I don't think you can avoid "fighting" it.  It's the elephant in the
room, and there's no way around that, because that global is the
backbone of the Emacs design, and all the Lisp programs and other
features implicitly assume it.

Re: Emacs design and architecture. How about copy-on-write?

[Alan Mackenzie]

Hello, Eli.

On Mon, Sep 18, 2023 at 13:30:15 +0300, Eli Zaretskii wrote:
> > Date: Sun, 17 Sep 2023 15:36:11 +0000
> > Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > How about copy-on-write at a symbol
> > value-cell/function-cell/property-list level?

> > A value in a value-cell (etc.) would have an associated thread value, the
> > thread which "owns" it.  When thread1 gets cloned from thread0, it
> > continues to use thread0's data state until it writes foo (e.g. by
> > binding it).  This would create a new value/thread pair, foo/thread1.
> > Further writes of foo by thread1 would continue to access foo/thread1.

> > Possibly, we could use just one owning thread for all the cells of a
> > symbol.

> > When a thread terminates, all its owned variables would become garbage,
> > to be collected by our new garbage collector.  ;-)

> You assume that whatever the thread does can be discarded as garbage?
> That's definitely not true in general: most things we do in Emacs
> should leave some trace behind.  It is not clear when and how this
> would happen under your proposal.  E.g., imagine that a thread runs
> Gnus fetching articles, and try to describe how this will work.

You mean that at the end of a thread, it will need to write results back
to variables "owned" by the calling thread.  Yes.  Either these variables
get marked as thread-global (not very attractive), or else we would need
to mark specific variables as not to be copied on write.

With a thread fetching Gnus articles, there is the additional
complication of having several such threads fetching from several servers
at once.  Then access to the result variables would need to be locked, to
prevent two threads overwriting eachother's results.  But this is so in
any multithreading system, no matter how it's done.

> > Clearly, there would have to be some variables which would be
> > thread-global, i.e. there would be a single value cell shared by all
> > threads.  There would also doubltless be other complications.

> "Some variables"?  Our problem is that their name is a legion.  E.g.,
> what do you propose to do with variables and changes to buffer text
> that affect the display?

I envisage each buffer being "owned" by a thread, possibly a special
thread just controlling access to the buffer.  Variables like
scroll-margin would be the thread's own binding of it.  There are a large
number of dynamic variables in redisplay which, in the current Emacs,
when bound by a thread would affect Emacs globally.  This c-o-w proposal
would fix this problem.

> Or are you suggesting to have a separate redisplay for each thread?  Or
> maybe you propose that each window has its own thread, complete with
> its own display (and a separate thread for each frame)?

I don't think several redisplay threads would be a good idea - usually,
there is just one screen Emacs is drawing on.  If we get down into
details, it might be worth having separate threads for each frame, or
even each window.

> These aspects need to be figured out if we want to discuss something
> like this seriously.

I'd like to emphasise that this copy-on-write idea is still just a vague
idea which might help, not a worked out firm proposal.

> > The advantage of this approach, if it could be made to work, is that it
> > doesn't try to fight the massive global state which is Emacs Lisp,
> > instead accepting it and embracing it.

> I don't think you can avoid "fighting" it.  It's the elephant in the
> room, and there's no way around that, because that global is the
> backbone of the Emacs design, and all the Lisp programs and other
> features implicitly assume it.

The c-o-w idea could steer around at least part of the globality.  I
think it would be relatively simple to implement (hah!) and wouldn't have
a large run-time cost, though of course there would be some.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Mon, 18 Sep 2023 11:38:56 +0000
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > When a thread terminates, all its owned variables would become garbage,
> > > to be collected by our new garbage collector.  ;-)
> 
> > You assume that whatever the thread does can be discarded as garbage?
> > That's definitely not true in general: most things we do in Emacs
> > should leave some trace behind.  It is not clear when and how this
> > would happen under your proposal.  E.g., imagine that a thread runs
> > Gnus fetching articles, and try to describe how this will work.
> 
> You mean that at the end of a thread, it will need to write results back
> to variables "owned" by the calling thread.  Yes.  Either these variables
> get marked as thread-global (not very attractive), or else we would need
> to mark specific variables as not to be copied on write.

After the thread terminates, and perhaps also while it still runs.
E.g., consider some kind of progress-reporting facility.

> With a thread fetching Gnus articles, there is the additional
> complication of having several such threads fetching from several servers
> at once.  Then access to the result variables would need to be locked, to
> prevent two threads overwriting eachother's results.  But this is so in
> any multithreading system, no matter how it's done.

The danger is indeed that most of the variables will need to be
protected by locks.  If we cannot find a way of avoiding that, we are
back at the current "only one thread at a time" model, and there's
nothing to gain.

> > > Clearly, there would have to be some variables which would be
> > > thread-global, i.e. there would be a single value cell shared by all
> > > threads.  There would also doubltless be other complications.
> 
> > "Some variables"?  Our problem is that their name is a legion.  E.g.,
> > what do you propose to do with variables and changes to buffer text
> > that affect the display?
> 
> I envisage each buffer being "owned" by a thread, possibly a special
> thread just controlling access to the buffer.  Variables like
> scroll-margin would be the thread's own binding of it.  There are a large
> number of dynamic variables in redisplay which, in the current Emacs,
> when bound by a thread would affect Emacs globally.  This c-o-w proposal
> would fix this problem.

I'm confused: suppose one thread modifies scroll-margin -- does that
affect the (global) redisplay?  If it does, how will this "solve" the
problem?  If it doesn't affect redisplay, how _can_ a thread change
scroll-margin in order to affect redisplay?

> > Or are you suggesting to have a separate redisplay for each thread?  Or
> > maybe you propose that each window has its own thread, complete with
> > its own display (and a separate thread for each frame)?
> 
> I don't think several redisplay threads would be a good idea - usually,
> there is just one screen Emacs is drawing on.

It's one screen, but each window is redrawn separately (on GUI
terminals).

> If we get down into details, it might be worth having separate
> threads for each frame, or even each window.

I think this aspect of multithreading is so crucial that it must be
considered from the get-go.  Redisplay is one of the few places in
Emacs where the assumption of global state is entrenched as deep as
possible, so without some new ideas it will basically preclude
multithreading.

> > I don't think you can avoid "fighting" it.  It's the elephant in the
> > room, and there's no way around that, because that global is the
> > backbone of the Emacs design, and all the Lisp programs and other
> > features implicitly assume it.
> 
> The c-o-w idea could steer around at least part of the globality.  I
> think it would be relatively simple to implement (hah!) and wouldn't have
> a large run-time cost, though of course there would be some.

I think the copy-on-write idea will work only for thread-local
variables, and we already have those in the form of let-bindings.  The
important (and the hard) part of this is elsewhere.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> You mean that at the end of a thread, it will need to write results back
>> to variables "owned" by the calling thread.  Yes.  Either these variables
>> get marked as thread-global (not very attractive), or else we would need
>> to mark specific variables as not to be copied on write.
>
> After the thread terminates, and perhaps also while it still runs.
> E.g., consider some kind of progress-reporting facility.

I think this can be approached similar to buffer-local values - all the
symbols will have default value (global), and per-thread values. When
threads set symbol value normally (for example, via setq), the
per-thread value is modified (and possibly written back to "global"
after the thread terminates). But thread might need to write "global"
directly - something similar to `set-default-toplevel-value' might be
used then.

>> With a thread fetching Gnus articles, there is the additional
>> complication of having several such threads fetching from several servers
>> at once.  Then access to the result variables would need to be locked, to
>> prevent two threads overwriting eachother's results.  But this is so in
>> any multithreading system, no matter how it's done.
>
> The danger is indeed that most of the variables will need to be
> protected by locks.  If we cannot find a way of avoiding that, we are
> back at the current "only one thread at a time" model, and there's
> nothing to gain.

I think that there is a limit to what can be done generically without
affecting the existing Elisp code. Modifying a global variable
asynchronously might need to be coded explicitly on Elisp level by more
granular locks (to be provided as a part of async thread API).

For example, if Gnus is collecting all the articles into a list of
(("server1" . articles) ("server2" . articles)), only Gnus will know
that it is enough to lock cdr of "server1" when fetching articles from
server 1.

In other scenarios, like modifying hash table, it might be possible to
provide thread-safe implementations on C level. Those might or might not
be good enough, depending on the use-case.

>> I envisage each buffer being "owned" by a thread, possibly a special
>> thread just controlling access to the buffer.  Variables like
>> scroll-margin would be the thread's own binding of it.  There are a large
>> number of dynamic variables in redisplay which, in the current Emacs,
>> when bound by a thread would affect Emacs globally.  This c-o-w proposal
>> would fix this problem.
>
> I'm confused: suppose one thread modifies scroll-margin -- does that
> affect the (global) redisplay?  If it does, how will this "solve" the
> problem?  If it doesn't affect redisplay, how _can_ a thread change
> scroll-margin in order to affect redisplay?

IMHO, the only sane way to utilize the existing redisplay is redisplay
lock - only one thread can request redisplay at a time, using its
thread-local state.

>> I don't think several redisplay threads would be a good idea - usually,
>> there is just one screen Emacs is drawing on.
>
> It's one screen, but each window is redrawn separately (on GUI
> terminals).

Technically yes, but AFAIU the code is written assuming single-threaded
execution. Decoupling redisplay of different windows would require
significant non-trivial changes in xdisp.c

>> The c-o-w idea could steer around at least part of the globality.  I
>> think it would be relatively simple to implement (hah!) and wouldn't have
>> a large run-time cost, though of course there would be some.
>
> I think the copy-on-write idea will work only for thread-local
> variables, and we already have those in the form of let-bindings.  The
> important (and the hard) part of this is elsewhere.

Yup. For example, buffer-local symbols are implemented using forward
pointers, overriding buffer object any time we alter (like let-bind) a
buffer-local value.

From implementation perspective, it might be easier to extend symbol
value slots of symbol object and buffer-local variable slots of buffer
objects to store values for multiple threads + a default value.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Alan Mackenzie <acm@muc.de>, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Mon, 18 Sep 2023 12:49:21 +0000
> 
> > I'm confused: suppose one thread modifies scroll-margin -- does that
> > affect the (global) redisplay?  If it does, how will this "solve" the
> > problem?  If it doesn't affect redisplay, how _can_ a thread change
> > scroll-margin in order to affect redisplay?
> 
> IMHO, the only sane way to utilize the existing redisplay is redisplay
> lock - only one thread can request redisplay at a time, using its
> thread-local state.

So if one thread changes scroll-margin and triggers redisplay, then
another thread triggers redisplay with its (different value of
scroll-margin), the display will scroll or move point back? and then
forward again? and then back again?

> >> I don't think several redisplay threads would be a good idea - usually,
> >> there is just one screen Emacs is drawing on.
> >
> > It's one screen, but each window is redrawn separately (on GUI
> > terminals).
> 
> Technically yes, but AFAIU the code is written assuming single-threaded
> execution.

In what way does it assume single-threaded execution?  We walk the
window tree of a frame and redisplay each leaf window in the tree,
that's all.  Maybe I don't understand what you mean by "assuming
single-threaded execution".

> Decoupling redisplay of different windows would require significant
> non-trivial changes in xdisp.c

In which part(s) of xdisp.c?  Most of xdisp.c handles redisplay of a
single window.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> IMHO, the only sane way to utilize the existing redisplay is redisplay
>> lock - only one thread can request redisplay at a time, using its
>> thread-local state.
>
> So if one thread changes scroll-margin and triggers redisplay, then
> another thread triggers redisplay with its (different value of
> scroll-margin), the display will scroll or move point back? and then
> forward again? and then back again?

Nope. I consider that redisplay is always synchronous (because of global
redisplay lock). If multiple threads trigger redisplay with different
scroll-margin values, it will be not different compared to the following
example:

(let ((scroll-margin 500)) (redisplay))
<...>
(redisplay)
<...>
(let ((scroll-margin 10)) (redisplay))
<...>

A better illustration is probably

(progn
  (let ((bidi-paragraph-direction 'right-to-left)) (redisplay))
  (sleep-for 1)
  (redisplay)
  (sleep-for 1)
  (let ((bidi-paragraph-direction 'right-to-left)) (redisplay))
  (sleep-for 1))

>> >> I don't think several redisplay threads would be a good idea - usually,
>> >> there is just one screen Emacs is drawing on.
>> >
>> > It's one screen, but each window is redrawn separately (on GUI
>> > terminals).
>> 
>> Technically yes, but AFAIU the code is written assuming single-threaded
>> execution.
>
> In what way does it assume single-threaded execution?  We walk the
> window tree of a frame and redisplay each leaf window in the tree,
> that's all.  Maybe I don't understand what you mean by "assuming
> single-threaded execution".

1. xdisp assumes at its core logic that current_buffer is always a
   single buffer. And it affects, for example mode-line faces.

2. `display_mode_line' uses global temporary override by calling
   `push_kboard'. AFAIU, it is also relying on single-threaded code.

3. xdisp is relying on a number of global-only variables like
   `mode-line-compact', `show-trailing-whitespace', and similar.
   AFAIR, things like `show-trailing-whitespace' affect how the
   optimizations are applied when deciding which windows should be
   redisplayed and which should not. I suspect that logic related to
   optimizations may be very fragile with async execution.

4. There are complex interactions between window redisplay, mode lines,
   echo area, and toolbars. AFAIR, if some Elisp (or maybe C) code,
   recursively called during window redisplay, affects mode-line/toolbar
   height, xdisp restarts (sometimes, partially) the redisplay process.
   I expect issues when this interacts with async redisplay of
   individual windows.

>> Decoupling redisplay of different windows would require significant
>> non-trivial changes in xdisp.c
>
> In which part(s) of xdisp.c?  Most of xdisp.c handles redisplay of a
> single window.

Yup, but see (4). I recall seeing a number of non-trivial `goto' calls
where faces or window geometry are checked for changes and redisplay has
to redo redisplay according to the changed values. When such geometry
changes can also happen asynchronously, a number of places in the code
that assumed fixed geometry and glyph matrix may be broken.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Mon, 18 Sep 2023 15:55:26 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> IMHO, the only sane way to utilize the existing redisplay is redisplay
> >> lock - only one thread can request redisplay at a time, using its
> >> thread-local state.
> >
> > So if one thread changes scroll-margin and triggers redisplay, then
> > another thread triggers redisplay with its (different value of
> > scroll-margin), the display will scroll or move point back? and then
> > forward again? and then back again?
> 
> Nope. I consider that redisplay is always synchronous

Synchronous to what?

> (because of global redisplay lock)

What is the global redisplay lock?

> If multiple threads trigger redisplay with different scroll-margin
> values, it will be not different compared to the following example:

I understand the problem, I'm asking what could or should be the
possible solutions.

> > In what way does it assume single-threaded execution?  We walk the
> > window tree of a frame and redisplay each leaf window in the tree,
> > that's all.  Maybe I don't understand what you mean by "assuming
> > single-threaded execution".
> 
> 1. xdisp assumes at its core logic that current_buffer is always a
>    single buffer. And it affects, for example mode-line faces.

No, it just assumes that when working on a window, its buffer is
temporarily made the current buffer.  But since we already made the
current buffer per-thread, this is not a problem.  From thread.h:

    /* This points to the current buffer.  */
    struct buffer *m_current_buffer;
  #define current_buffer (current_thread->m_current_buffer)

> 2. `display_mode_line' uses global temporary override by calling
>    `push_kboard'. AFAIU, it is also relying on single-threaded code.

This is a non-issue.  If we don't allow non-main threads interact with
the user, we can leave this code intact.  If we do allow interaction
from non-main threads, we just need to bind kboard-local variables to
their thread-specific values when we switch to the thread.

> 3. xdisp is relying on a number of global-only variables like
>    `mode-line-compact', `show-trailing-whitespace', and similar.
>    AFAIR, things like `show-trailing-whitespace' affect how the
>    optimizations are applied when deciding which windows should be
>    redisplayed and which should not. I suspect that logic related to
>    optimizations may be very fragile with async execution.

That's completely irrelevant to the issue at hand.  The fact that
Emacs has a huge global state, and all of its code relies on that is a
separate issue.  Here, I asked you in what sense is xdisp.c's code
single-threaded; if your answer is "because of its reliance on global
state", it means there's no separate problem of xdisp.c that is based
on single thread.

And those of these global variables that aren't changing while some
thread is redisplaying a window showing a particular buffer don't even
interfere with parallel redisplay.

As for redisplay optimizations, they are entirely based on buffer- and
window-local information, so I cannot imagine why you think they will
be very fragile.

> 4. There are complex interactions between window redisplay, mode lines,
>    echo area, and toolbars. AFAIR, if some Elisp (or maybe C) code,
>    recursively called during window redisplay, affects mode-line/toolbar
>    height, xdisp restarts (sometimes, partially) the redisplay process.
>    I expect issues when this interacts with async redisplay of
>    individual windows.

Why do you expect that?

> >> Decoupling redisplay of different windows would require significant
> >> non-trivial changes in xdisp.c
> >
> > In which part(s) of xdisp.c?  Most of xdisp.c handles redisplay of a
> > single window.
> 
> Yup, but see (4). I recall seeing a number of non-trivial `goto' calls
> where faces or window geometry are checked for changes and redisplay has
> to redo redisplay according to the changed values. When such geometry
> changes can also happen asynchronously, a number of places in the code
> that assumed fixed geometry and glyph matrix may be broken.

No, it just means any such changes need to be communicated to other
"redisplaying" threads, so that they also restart.

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Eli Zaretskii wrote:

> it just means any such changes need to be communicated to
> other "redisplaying" threads, so that they also restart

Instead of focusing on specific cases that one can imagine to
be problematic, why not look for a general policy what
should happen in the first place?

So, what does it mean for a thread to lock a global variable?
and set it? and use it? and then unlock it?

What does it mean to that thread for the duration of its
execution? And what does it mean to another thread that is, or
isn't, doing or wanting to do the same thing,
possibly simultaneously?

After that, one can think of how to setup a mechanism that
will safely uphold the model.

So first thing, define what a thread T can do to a global
variable V.

Second thing, what do we want to happen, when another
thread K does the same things in parallel, also to V?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 19 Sep 2023 00:48:15 +0200
> 
> Eli Zaretskii wrote:
> 
> > it just means any such changes need to be communicated to
> > other "redisplaying" threads, so that they also restart
> 
> Instead of focusing on specific cases that one can imagine to
> be problematic, why not look for a general policy what
> should happen in the first place?

The general policy is a solved problem, so there's no reason to
consider it.

> So, what does it mean for a thread to lock a global variable?
> and set it? and use it? and then unlock it?
> 
> What does it mean to that thread for the duration of its
> execution? And what does it mean to another thread that is, or
> isn't, doing or wanting to do the same thing,
> possibly simultaneously?
> 
> After that, one can think of how to setup a mechanism that
> will safely uphold the model.
> 
> So first thing, define what a thread T can do to a global
> variable V.
> 
> Second thing, what do we want to happen, when another
> thread K does the same things in parallel, also to V?

We don't need to discuss all this because solutions for thread
synchronization exist for a long time.  We even use quite a few of
them already: the Lisp threads we have in Emacs now provide some of
these synchronization primitives, which are built on top of existing
capabilities built into the OS and existing thread libraries.

So the problem is not how to lock and serialize access to a variable
in general, the problem is how to do this in Emacs so that we won't
need to lock everything.

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Eli Zaretskii wrote:

> We don't need to discuss all this because solutions for
> thread synchronization exist for a long time. We even use
> quite a few of them already: the Lisp threads we have in
> Emacs now provide some of these synchronization primitives,
> which are built on top of existing capabilities built into
> the OS and existing thread libraries.
>
> So the problem is not how to lock and serialize access to
> a variable in general, the problem is how to do this in
> Emacs so that we won't need to lock everything.

Okay, excellent, but then why isn't it enough to just maintain
a register of global variables and threads?

If a thread wants to use it, look in the register, is it
available? If not, get in line. And when it becomes available,
pop a thread in the line, if there is one, and start over?

Why do we have to lock everything just because we lock
a single variable?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

>> So the problem is not how to lock and serialize access to
>> a variable in general, the problem is how to do this in
>> Emacs so that we won't need to lock everything.
>
> Okay, excellent, but then why isn't it enough to just maintain
> a register of global variables and threads?

> If a thread wants to use it, look in the register, is it
> available? If not, get in line. And when it becomes available,
> pop a thread in the line, if there is one, and start over?
>
> Why do we have to lock everything just because we lock
> a single variable?

Because implementation details are tricky - a lot of Elisp internal
machinery is relying upon modifying global symbol objects, having
certain global C variables assigned to the "right" value, and buffer
objects having the right buffer-local values.
This particular issue has been discussed in details in
https://yhetil.org/emacs-devel/871qhnr4ty.fsf@localhost/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Ihor Radchenko wrote:

> Because implementation details are tricky - a lot of Elisp
> internal machinery is relying upon modifying global symbol
> objects, having certain global C variables assigned to the
> "right" value, and buffer objects having the right
> buffer-local values. This particular issue has been
> discussed in details in
> https://yhetil.org/emacs-devel/871qhnr4ty.fsf@localhost/

Yeah, but instead of adopting the lock mechanism to take into
account a possibly huge amount of such cases the lock
mechanism should be solid and work the same way for everyone
and everything.

If that breaks code that relies on the previous solution, that
will have to be fixed.

Because if the lock mechanism has to take into account tons of
weird cases and situations written for another solution, it
won't be good. And new code - what solution should it be
written for? The old or the new solution? Or the new
solution's exceptions not to break legacy code?

Oh, no. 2 wrongs don't make 1 right. Solid foundation
bottom-up approach, things incompatible with the sound
solution? No exceptions added to the sound solution, instead
fix them one at a time.

No shortcuts to the top!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

>> https://yhetil.org/emacs-devel/871qhnr4ty.fsf@localhost/
>
> Yeah, but instead of adopting the lock mechanism to take into
> account a possibly huge amount of such cases the lock
> mechanism should be solid and work the same way for everyone
> and everything.
> ...

Sorry, but I am completely lost. Cannot understand what you are trying
to propose.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Ihor Radchenko wrote:

>>> https://yhetil.org/emacs-devel/871qhnr4ty.fsf@localhost/
>>
>> Yeah, but instead of adopting the lock mechanism to take
>> into account a possibly huge amount of such cases the lock
>> mechanism should be solid and work the same way for
>> everyone and everything. [...]
>
> Sorry, but I am completely lost. Cannot understand what you
> are trying to propose.

We can't build a new solution filled with exceptions so it
won't break existing programs that were programmed for
another solution.

Instead we have to focus on the new solution and old programs
that break will have to be adopted to the new solution - or
discarded even, sometimes.

So one should create a minimal scenario that still includes
everything that can happen, and the worst case at that. If our
solution can solve that, then that's it.

So what is the base case? A global variable with two threads
that read, and two that write? And what is the worst case of
that base case? All of that happening at once?

When this "worst-case base case" is identified and solved in
a good way, whatever existing code that now breaks will have
to be modified.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

>> Sorry, but I am completely lost. Cannot understand what you
>> are trying to propose.
>
> We can't build a new solution filled with exceptions so it
> won't break existing programs that were programmed for
> another solution.

Eli's point is that the new concurrency framework must allow the
existing code to run without modifying it to fit concurrency. Not
necessarily fast (maybe with some interlocking), but at least without
breaking.

Otherwise, making actual use of concurrency will require rewriting a lot
of Elisp, in addition to C-level support, which is too much effort - we
will have to fight many non-obvious problems any time we want to call an
"old" Elisp function from purposely-written async thread.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Ihor Radchenko wrote:

>>> Sorry, but I am completely lost. Cannot understand what
>>> you are trying to propose.
>>
>> We can't build a new solution filled with exceptions so it
>> won't break existing programs that were programmed for
>> another solution.
>
> Eli's point is that the new concurrency framework must allow
> the existing code to run without modifying it to fit
> concurrency. Not necessarily fast (maybe with some
> interlocking), but at least without breaking.
>
> Otherwise, making actual use of concurrency will require
> rewriting a lot of Elisp, in addition to C-level support,
> which is too much effort - we will have to fight many
> non-obvious problems any time we want to call an "old" Elisp
> function from purposely-written async thread.

Multi-threaded concurrent access to global variables that will
not break programs that were written using global variables
with the then correct assumption no one else would touch them?

But it is still possible, just the meaning of `setq', `let'
etc will have to be changed so that they now are referred to
the lock mechanism and also actually create local variables
transparently which are subsequently used.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 19:15:27 +0300
> 
> Eli's point is that the new concurrency framework must allow the
> existing code to run without modifying it to fit concurrency. Not
> necessarily fast (maybe with some interlocking), but at least without
> breaking.
> 
> Otherwise, making actual use of concurrency will require rewriting a lot
> of Elisp

Starting from subr.el, simple.el, files.el, etc. -- stuff which every
Lisp program written for Emacs uses without thinking twice.

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Ihor Radchenko wrote:

> Because implementation details are tricky - a lot of Elisp
> internal machinery is relying upon modifying global symbol
> objects [...]

Yeah, but that should be fine, as long as they are locked and
unlocked safely, right?

But, if one aspire to reduce the number of global variables
used, a great way of doing that is lexical `let'-closures,
here is an example that allows for the same default value when
the function is called from Lisp and when used as an
interactive command, yet the data is only hardcoded once for
each variable.

With this method - and not just for that use case, also the
use case of 2 functions sharing a variable - one can reduce
global Lisp variable use a lot - I know since I've done that
do my own code, and it works great ever since.

;;; -*- lexical-binding: t -*-
;;
;; this file:
;;   https://dataswamp.org/~incal/emacs-init/math.el

(require 'cl-lib)

;; [...]

(let ((min-def 0)
      (max-def 9)
      (inc-def 1) )
  (cl-defun interval (&optional (min min-def)
                                (max max-def)
                                (inc inc-def) )
    (interactive `(,(read-number "min: " min-def)
                   ,(read-number "max: " max-def)
                   ,(read-number "inc: " inc-def)) )
    (unless (<= min max)
      (error "Bogus interval") )
    (unless (> inc 0)
      (error "Bogus increment") )
    (cl-loop
      for i from min to max by inc
      collect i) )
  (declare-function interval nil) )

;; (interval 10 5)        ; Bogus interval
;; (interval 1 3 -1)      ; Bogus increment
;; (interval 5 10)        ; (5 6 7 8 9 10)
;; (interval 1.8 2.0 0.1) ; (1.8 1.9 2.0)
;; (interval)             ; (0 1 2 3 4 5 6 7 8 9)
;; (interval 19 99)       ; (19 20 21 ... 97 98 99)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

> Ihor Radchenko wrote:
>
>> Because implementation details are tricky - a lot of Elisp
>> internal machinery is relying upon modifying global symbol
>> objects [...]
>
> Yeah, but that should be fine, as long as they are locked and
> unlocked safely, right?

No.

If we have something like (let ((case-fold-search t)) ...), we will lock
`case-fold-search' symbol for the whole duration of `let' call and block
any other threads trying to alter `case-fold-search' locally.

> But, if one aspire to reduce the number of global variables
> used, a great way of doing that is lexical `let'-closures,
> here is an example that allows for the same default value when
> the function is called from Lisp and when used as an
> interactive command, yet the data is only hardcoded once for
> each variable.

Not every variable can be used within lexical scope. In particular
special variables (see 12.10.4 Using Lexical Binding and
`special-variable-p') are always treated outside lexical binding.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Emanuel Berg <incal@dataswamp.org> writes:
>
>> Ihor Radchenko wrote:
>>
>>> Because implementation details are tricky - a lot of Elisp
>>> internal machinery is relying upon modifying global symbol
>>> objects [...]
>>
>> Yeah, but that should be fine, as long as they are locked and
>> unlocked safely, right?
>
> No.
>
> If we have something like (let ((case-fold-search t)) ...), we will lock
> `case-fold-search' symbol for the whole duration of `let' call and block
> any other threads trying to alter `case-fold-search' locally.

Emacs does not store the default value of each symbol in the variables
which they forward to.  All DEFVARs are thread local in this sense, for
each thread must be capable of simultaneously binding different values
to the same variable.

In my implementation, struct thread_state incorporates pointers to all
forwarded variables, each of which either points within thread_state
itself when its symbol is bound locally, or to `globals' otherwise.

Vcharset_map_path thus becomes:

#define Vcharset_map_path (*current_thread->f_Vcharset_map_path)

That being said, case-fold-search is a bad example.  I have not yet
established how buffer local variables will be efficiently represented
when multiple threads selecting the same buffer bind the same variables
simultaneously.  Once that hurdle is surmounted and the regex engine
itself is rendered thread-safe, threads will be capable of calling
regexp matching functions concurrently, with only their local bindings
of case_fold_search taking effect.  No symbol-based locking necessary.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> If we have something like (let ((case-fold-search t)) ...), we will lock
>> `case-fold-search' symbol for the whole duration of `let' call and block
>> any other threads trying to alter `case-fold-search' locally.
>
> ... for
> each thread must be capable of simultaneously binding different values
> to the same variable.

Sure. I did not object this. What I mean is that some symbol values are
tricky. In particular symbols with values being forwarded.

> In my implementation, struct thread_state incorporates pointers to all
> forwarded variables, each of which either points within thread_state
> itself when its symbol is bound locally, or to `globals' otherwise.

> Vcharset_map_path thus becomes:
>
> #define Vcharset_map_path (*current_thread->f_Vcharset_map_path)

Does it mean that each threads contains a copy of the whole `globals'? I
think I proposed this in the linked thread and it was objected as
something taking a lot of memory.

IMHO, copy-of-write would be better here - only store the thread-local
values that are actually altered by the thread.
Indeed, a simple #define will not be enough then, but we would not waste
memory copying the values that will never be changed anyway.

> That being said, case-fold-search is a bad example.

Which is why I used it :) It is innocent yet tricky.

> ... I have not yet
> established how buffer local variables will be efficiently represented
> when multiple threads selecting the same buffer bind the same variables
> simultaneously.

We discussed asynchronous access to buffers earlier and concluded that
modifying a buffer (and its buffer-local values) asynchronously is a
hard problem. In particular, handling gap asynchronously is simply not
possible. It might be more practical to allow simultaneous read-only
access to the same buffer, but interlock writes to buffer object
(including moving gap, changing buffer text, and changing buffer-local
values).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Does it mean that each threads contains a copy of the whole `globals'? I
> think I proposed this in the linked thread and it was objected as
> something taking a lot of memory.

It incorporates one pointer for each field within globals itself.  AFAIU
that was the initial impetus for introducing struct emacs_globals in the
first place.

> IMHO, copy-of-write would be better here - only store the thread-local
> values that are actually altered by the thread.

Of course only specbound values will be saved within each thread's
state, but each thread must maintain a pointer to the field that is
actually holding the value of the forwarded variables.

> Indeed, a simple #define will not be enough then, but we would not waste
> memory copying the values that will never be changed anyway.

On my machine, struct emacs_globals is 7490 bytes, just short of 7 KiB.
struct thread_state amounts to 12704 bytes, which does not come across
as excessive in this day and age.  Thread-local storage is then
dynamically allocated as bindings materialize.

> We discussed asynchronous access to buffers earlier and concluded that
> modifying a buffer (and its buffer-local values) asynchronously is a
> hard problem. In particular, handling gap asynchronously is simply not
> possible. It might be more practical to allow simultaneous read-only
> access to the same buffer, but interlock writes to buffer object
> (including moving gap, changing buffer text, and changing buffer-local
> values).

The gap and point are not buffer local values.  I'm referring to
variables saved within local_var_alist, which must be modified to
incorporate bindings for each thread.

The problem lies in how to perform this efficiently.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> Indeed, a simple #define will not be enough then, but we would not waste
>> memory copying the values that will never be changed anyway.
>
> On my machine, struct emacs_globals is 7490 bytes, just short of 7 KiB.
> struct thread_state amounts to 12704 bytes, which does not come across
> as excessive in this day and age.  Thread-local storage is then
> dynamically allocated as bindings materialize.

Sounds reasonable then. 7k does not look like a big deal to me.

>> We discussed asynchronous access to buffers earlier and concluded that
>> modifying a buffer (and its buffer-local values) asynchronously is a
>> hard problem. In particular, handling gap asynchronously is simply not
>> possible. It might be more practical to allow simultaneous read-only
>> access to the same buffer, but interlock writes to buffer object
>> (including moving gap, changing buffer text, and changing buffer-local
>> values).
>
> The gap and point are not buffer local values.  I'm referring to
> variables saved within local_var_alist, which must be modified to
> incorporate bindings for each thread.

As an aside: point should also be thread-local to make common Elisp code
work (like re-search-forward, for example).

> The problem lies in how to perform this efficiently.

The whole process of working with Lisp_Buffer_Local_Value is a bit
arcane with all its juggling of the values (`do_symval_forwarding',
`swap_in_symval_forwarding', etc) every time current_buffer changes.

I am wondering if you solved (seemingly) simpler cases with setting
Lisp_Fwd_Obj, Lisp_Fwd_Int, and Lisp_Fwd_Bool.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> I am wondering if you solved (seemingly) simpler cases with setting
> Lisp_Fwd_Obj, Lisp_Fwd_Int, and Lisp_Fwd_Bool.

They forward to thread local values.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> I am wondering if you solved (seemingly) simpler cases with setting
>> Lisp_Fwd_Obj, Lisp_Fwd_Int, and Lisp_Fwd_Bool.
>
> They forward to thread local values.

I see. Then, one way to approach buffer-local variables could be storing
an association of (buffer . thread-local-buffer-variables) under the
thread object and forward there.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Does it mean that each threads contains a copy of the whole `globals'? I
>> think I proposed this in the linked thread and it was objected as
>> something taking a lot of memory.
>
> It incorporates one pointer for each field within globals itself.  AFAIU
> that was the initial impetus for introducing struct emacs_globals in the
> first place.
>
>> IMHO, copy-of-write would be better here - only store the thread-local
>> values that are actually altered by the thread.
>
> Of course only specbound values will be saved within each thread's
> state, but each thread must maintain a pointer to the field that is
> actually holding the value of the forwarded variables.
>
>> Indeed, a simple #define will not be enough then, but we would not waste
>> memory copying the values that will never be changed anyway.
>
> On my machine, struct emacs_globals is 7490 bytes, just short of 7 KiB.

Correction: just over 7 KiB.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 21:35:49 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Ihor Radchenko <yantar92@posteo.net> writes:
> >
> >> Does it mean that each threads contains a copy of the whole `globals'? I
> >> think I proposed this in the linked thread and it was objected as
> >> something taking a lot of memory.
> >
> > It incorporates one pointer for each field within globals itself.  AFAIU
> > that was the initial impetus for introducing struct emacs_globals in the
> > first place.
> >
> >> IMHO, copy-of-write would be better here - only store the thread-local
> >> values that are actually altered by the thread.
> >
> > Of course only specbound values will be saved within each thread's
> > state, but each thread must maintain a pointer to the field that is
> > actually holding the value of the forwarded variables.
> >
> >> Indeed, a simple #define will not be enough then, but we would not waste
> >> memory copying the values that will never be changed anyway.
> >
> > On my machine, struct emacs_globals is 7490 bytes, just short of 7 KiB.
> 
> Correction: just over 7 KiB.

And don't forget that the globals defined in globals.h are just those
we define in C.  There are many globals defined in Lisp.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> And don't forget that the globals defined in globals.h are just those
> we define in C.  There are many globals defined in Lisp.

Those don't contribute to the size of struct thread_state; an alist
between threads and its local value is dynamically allocated when a
symbol is bound locally.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> And don't forget that the globals defined in globals.h are just those
>> we define in C.  There are many globals defined in Lisp.
>
> Those don't contribute to the size of struct thread_state; an alist
> between threads and its local value is dynamically allocated when a
                      ^^^^^^^^^^^^^^^

I intended to write ``and their local values.''

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: yantar92@posteo.net,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 08:55:49 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And don't forget that the globals defined in globals.h are just those
> > we define in C.  There are many globals defined in Lisp.
> 
> Those don't contribute to the size of struct thread_state; an alist
> between threads and its local value is dynamically allocated when a
> symbol is bound locally.

I'm not sure I understand: what is the difference between a global
variable defined in C and a global variable defined, say, in
simple.el?

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm not sure I understand: what is the difference between a global
> variable defined in C and a global variable defined, say, in
> simple.el?

The variable defined in C must be accessible by a C lvalue.  Namely,

  (*current_thread->f_Vname_of_the_variable)

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: yantar92@posteo.net,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 15:34:52 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm not sure I understand: what is the difference between a global
> > variable defined in C and a global variable defined, say, in
> > simple.el?
> 
> The variable defined in C must be accessible by a C lvalue.  Namely,
> 
>   (*current_thread->f_Vname_of_the_variable)

How is this different from

  (setq name-of-variable SOMETHING)

done by some thread to a global variable?

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> The variable defined in C must be accessible by a C lvalue.  Namely,
>> 
>>   (*current_thread->f_Vname_of_the_variable)
>
> How is this different from
>
>   (setq name-of-variable SOMETHING)
>
> done by some thread to a global variable?

For global variables defined in C sources, Emacs maintains an actual C
variable that can be used directly to set and store the value. Then,

(setq name-of-global-variable-defined-in-C SOMETHING) must arrange to
set Vname_of_global_variable_defined_in_C C variable as well.

This is not needed for global variables defined in Elisp.

Under the hood, Vname binding is stored via Lisp_Objfwd. See #define
DEFVAR_LISP in lisp.h. This forwarding must be accounted for in threads,
if we want to make them asynchronous. Po Lu solved this by adding
thread-local "globals" structs.

In contrast, global Elisp variables defined in Elisp for not store
Lisp_Objfwd and instead contain their symbol value directly in their
Elisp symbol objects. Setting and reading these symbol values involves
the common API: Fsymbol_value and Fset, which are much easier to change
to work with async threads compared to the having to handle all the

 Vname = val;

that are expected to work in C level.

To sum up: Elisp-only globals are handled differently from globals
defined in C code and should thus be approached differently when
implementing async threads.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 08:35:06 +0000
> 
> In contrast, global Elisp variables defined in Elisp for not store
> Lisp_Objfwd and instead contain their symbol value directly in their
> Elisp symbol objects. Setting and reading these symbol values involves
> the common API: Fsymbol_value and Fset, which are much easier to change
> to work with async threads compared to the having to handle all the
> 
>  Vname = val;
> 
> that are expected to work in C level.

Thank you for the (unnecessary) lecture, but I was actually asking
what needs to be changed and how to handle the global variables, both
those defined in C and those defined in Lisp.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Thank you for the (unnecessary) lecture,

Well... hope it was at least useful for others reading the thread :shrug:

> ... but I was actually asking
> what needs to be changed and how to handle the global variables, both
> those defined in C and those defined in Lisp.

Po Lu can probably answer this better.

As I understand his previous explanations, global variable values are
stored per-thread, if they were changed by that thread at some point.
For C-defined globals, the Vname instances are re-defined to
point to thread-local "globals" struct.

I do not recall Po Lu providing the details on how the thread-local
variables are stored.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> As I understand his previous explanations, global variable values are
> stored per-thread, if they were changed by that thread at some point.
> For C-defined globals, the Vname instances are re-defined to
> point to thread-local "globals" struct.
>
> I do not recall Po Lu providing the details on how the thread-local
> variables are stored.

Non-forwarded thread-local bindings are saved within a separate
association list in Lisp_Symbol, where each element ties a thread to its
local binding.  do_specbind and do_one_specbind manage this association
list by introducing new associations.  set_internal and
find_symbol_value search within this association list for a pair
matching the current thread, and set or return its cdr respectively if
it is present.

Within lispfwds, the pointer to the field itself is replaced with an
offset to a field within struct thread_state which is set to a pointer
to the matching field in `globals' or one of its local bindings.

Whenever a forwarded variable is specbound for the first time in a given
thread, the designated field within that thread's state is set to a
pointer into thread local storage, and once unbound, restored to its
initial value within globals.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 21/09/2023 14:49, Po Lu wrote:
> Ihor Radchenko<yantar92@posteo.net>  writes:
> 
>> As I understand his previous explanations, global variable values are
>> stored per-thread, if they were changed by that thread at some point.
>> For C-defined globals, the Vname instances are re-defined to
>> point to thread-local "globals" struct.
>>
>> I do not recall Po Lu providing the details on how the thread-local
>> variables are stored.
> Non-forwarded thread-local bindings are saved within a separate
> association list in Lisp_Symbol, where each element ties a thread to its
> local binding.  do_specbind and do_one_specbind manage this association
> list by introducing new associations.  set_internal and
> find_symbol_value search within this association list for a pair
> matching the current thread, and set or return its cdr respectively if
> it is present.
> 
> Within lispfwds, the pointer to the field itself is replaced with an
> offset to a field within struct thread_state which is set to a pointer
> to the matching field in `globals' or one of its local bindings.
> 
> Whenever a forwarded variable is specbound for the first time in a given
> thread, the designated field within that thread's state is set to a
> pointer into thread local storage, and once unbound, restored to its
> initial value within globals.

That reminds me of Clojure's threads and dynamic vars' semantics: 
https://clojure.org/about/concurrent_programming

   By default Vars are static, but per-thread bindings for Vars defined 
with metadata mark them as dynamic. Dynamic vars are also mutable 
references to objects. They have a (thread-shared) root binding which 
can be established by def, and can be set using *set!*, but only if they 
have been bound to a new storage location thread-locally using binding. 
Those bindings and any subsequent modifications to those bindings will 
only be seen within the thread by code in the dynamic scope of the 
binding block. Nested bindings obey a stack protocol and unwind as 
control exits the binding block.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: luangruo@yahoo.com, emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 10:13:31 +0000
> 
> As I understand his previous explanations, global variable values are
> stored per-thread, if they were changed by that thread at some point.

So if a thread changes a global variables, other threads will never
notice?  That's not what Emacs Lisp programs expect.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> So if a thread changes a global variables, other threads will never
> notice?  That's not what Emacs Lisp programs expect.

No, only if a thread specbinds a global variable.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Ihor Radchenko <yantar92@posteo.net>,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 21:12:20 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So if a thread changes a global variables, other threads will never
> > notice?  That's not what Emacs Lisp programs expect.
> 
> No, only if a thread specbinds a global variable.

Then my question wasn't really answered yet.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Then my question wasn't really answered yet.

Ah, I now understand what you were asking.

The variable's global value is modified and thereafter made available to
all other threads.  If multiple threads perform coinciding alterations
to the same symbol's value, the machine bus reconciles the conflict by
selecting a single value.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: yantar92@posteo.net,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 21:35:29 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then my question wasn't really answered yet.
> 
> Ah, I now understand what you were asking.
> 
> The variable's global value is modified and thereafter made available to
> all other threads.  If multiple threads perform coinciding alterations
> to the same symbol's value, the machine bus reconciles the conflict by
> selecting a single value.

And when this happens to globals defined in C, is it handled in the
same way? if not, why not?

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> And when this happens to globals defined in C, is it handled in the
> same way?

Yes, it is.  The underlying implementation is different (as it must
account for forwarding), but the effect is the same.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: yantar92@posteo.net,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 21:57:17 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And when this happens to globals defined in C, is it handled in the
> > same way?
> 
> Yes, it is.  The underlying implementation is different (as it must
> account for forwarding), but the effect is the same.

OK, that's what I was asking from the beginning.  I though there was
some significant difference.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> How is this different from
>
>   (setq name-of-variable SOMETHING)
>
> done by some thread to a global variable?

That's orthogonal to the subject at hand.  If name-of-variable is
forwarded, then SOMETHING must be recorded within either thread_state or
globals.  If not, it becomes the new value held by one of
name-of-variable's value cells.

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Ihor Radchenko wrote:

>>> Because implementation details are tricky - a lot of Elisp
>>> internal machinery is relying upon modifying global symbol
>>> objects [...]
>>
>> Yeah, but that should be fine, as long as they are locked
>> and unlocked safely, right?
>
> No.
>
> If we have something like (let ((case-fold-search t)) ...),
> we will lock `case-fold-search' symbol for the whole
> duration of `let' call and block any other threads trying to
> alter `case-fold-search' locally.

Yes, but so be it if that's the best we can do with `let' and
a multi-threaded, multi-core Emacs.

But one can also think of other ways to do that, for example
local copies for the duration of the form.

Overall, we will have to have a general solution, then examine
Lisp forms that do global variable setting on a form-by-form
basis and decide how they will relate to the general solution
in the best way possible, while still playing by its rules.

>> But, if one aspire to reduce the number of global variables
>> used, a great way of doing that is lexical `let'-closures,
>> here is an example that allows for the same default value
>> when the function is called from Lisp and when used as an
>> interactive command, yet the data is only hardcoded once
>> for each variable.
>
> Not every variable can be used within lexical scope.
> In particular special variables (see 12.10.4 Using Lexical
> Binding and `special-variable-p') are always treated outside
> lexical binding.

We are never gonna come up with a bottom-level solution that
works optimally for everything. Global/dynamic/special
variables are the exception and not the norm - yes, as we hear
from the name "special" BTW :) - and yes, they will have to be
locked one by one and for as long as it takes for the
execution to proceed safely.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Emacs design and architecture. How about copy-on-write?

[Drew Adams]

(Caveat: Not really following this thread,
and I'm no expert on concurrency etc.)

> Global/dynamic/special
> variables are the exception and not the norm - yes, as we hear
> from the name "special" BTW :) - and yes, they will have to be
> locked one by one and for as long as it takes for the
> execution to proceed safely.

I hate to say it, but one of the points (the
only point?) of global/dynamic/special in Lisp
is to be able to affect other code anywhere,
including code that isn't yet written.

https://www.gnu.org/software/emacs/emacs-paper.html#SEC17

There's no overall, top-level director that
manages where dynamic bindings should have
their effect and retain their values.

That's the point: the bindings are dynamic.

There's really no such thing as dynamic scope.

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000

  ... it is convenient to define dynamic scope
  to mean indefinite scope and dynamic extent.
  
  Thus we speak of "special" variables as
  having dynamic scope, or being dynamically
  scoped, because they have indefinite scope
  and dynamic extent: a special variable can
  be referred to anywhere as long as its
  binding is currently in effect.

  The term ``dynamic scope'' is a misnomer.
  Nevertheless it is both traditional and useful.

Dynamic bindings have _indefinite scope_:

  References may occur anywhere, in any program.

Dynamic bindings have _dynamic extent_:

  References may occur at any time in the interval
  between establishment of the entity and the
  explicit disestablishment of the entity.

  As a rule, the entity is disestablished when
  execution of the establishing construct completes
  or is otherwise terminated. Therefore entities
  with dynamic extent obey a stack-like discipline,
  paralleling the nested executions of their
  establishing constructs.

So if you want to control dynamic bindings wrt
concurrency, you really need to implement some
kind of _transactions_, which will create and
terminate the bindings as needed/declared.

Maybe an alternative could be to use a kind of
optimistic concurrency, using, e.g., a kind of
ETAG value (e.g., on special vars or sets of
them).  That might be sufficient for a single
updating operation.

But to be able to control multiple updates of
multiple variables over a given duration, I
think some kind of transactions will need to
be implemented.  You commit the series of
updates for a given transaction only if other
sessions haven't modified the same variables
concurrently (as determined by ETAG values).

Re: [External] : Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Drew Adams wrote:

>> Global/dynamic/special variables are the exception and not
>> the norm - yes, as we hear from the name "special" BTW :) -
>> and yes, they will have to be locked one by one and for as
>> long as it takes for the execution to proceed safely.
>
> I hate to say it, but one of the points (the only point?) of
> global/dynamic/special in Lisp is to be able to affect other
> code anywhere, including code that isn't yet written.

Yes, again one would have to agree on a general model first.
It would not be an abstraction only, but also a blueprint what
we want to achieve.

Then one would have to go thru all the cases where the global
state is interacted with and come up with ways how those cases
are to be changed so that they don't break existing code,
while also making sense in a multi-thread environment.

No solution or implementation are allowed to break existing
code, so that would be the minimum. And anything more than
that, they would have to make as much sense as possible in the
new, multi-threaded environment.

Global/dynamic/special variables would be one such case - or
rather, several cases, as you can interact with those by means
of several Lisp constructs, be it `setq' or `let' or whatever
else there is. It would all have to be covered one by one.

We now what `setq' does it a global/dynamic/special
variable now, in single-threaded Emacs. What would it do in
a multi-threaded Emacs? That does not break code written for
the old, single-threaded one? And so on.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 19 Sep 2023 13:14:15 +0200
> 
> Eli Zaretskii wrote:
> 
> > We don't need to discuss all this because solutions for
> > thread synchronization exist for a long time. We even use
> > quite a few of them already: the Lisp threads we have in
> > Emacs now provide some of these synchronization primitives,
> > which are built on top of existing capabilities built into
> > the OS and existing thread libraries.
> >
> > So the problem is not how to lock and serialize access to
> > a variable in general, the problem is how to do this in
> > Emacs so that we won't need to lock everything.
> 
> Okay, excellent, but then why isn't it enough to just maintain
> a register of global variables and threads?
> 
> If a thread wants to use it, look in the register, is it
> available? If not, get in line. And when it becomes available,
> pop a thread in the line, if there is one, and start over?

You are describing what we already do, what every multithreading
program does.  You just call it "register", whereas its real name is
"mutex".

> Why do we have to lock everything just because we lock
> a single variable?

If we need to lock 99.99% of Emacs "single" variables, it is easier to
lock everything.  Faster, too.

Once again, you are looking at the wrong aspects of the problem.
These aspects are easily solvable, and we have even solved some of
them in the current Emacs.  The difficulties are elsewhere.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> If we need to lock 99.99% of Emacs "single" variables, it is easier to
> lock everything.  Faster, too.

Interlocking variables is not necessary.  Extant buses guarantee that
word-sized (Lisp_Object) writes and reads from one CPU are always
coherently propagated to other processors.  And otherwise, interlocking
every global variables would incur an extreme performance penalty; two
extra load-locked / store-conditional pairs (or interlocked
instructions) for each read or write at the minimum.

Interlocking Lisp_Objects themselves is only mandatory under wide int
builds, where the size of each object is larger than the machine's word
size.  Supporting such builds in a multiprocessing Emacs will be an
extreme burden that I don't think we should shoulder.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
> Date: Tue, 19 Sep 2023 20:57:30 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If we need to lock 99.99% of Emacs "single" variables, it is easier to
> > lock everything.  Faster, too.
> 
> Interlocking variables is not necessary.  Extant buses guarantee that
> word-sized (Lisp_Object) writes and reads from one CPU are always
> coherently propagated to other processors.  And otherwise, interlocking
> every global variables would incur an extreme performance penalty; two
> extra load-locked / store-conditional pairs (or interlocked
> instructions) for each read or write at the minimum.

That avoids garbled values where part of a value is from one thread,
the other part from another thread.  But it does nothing to protect
the threads other than the one which changed the value from the
"surprise" of having stuff change under its feet.  Which is the main
problem to solve, since Emacs code is written under the assumption
that a variable in the global state doesn't change while some code
runs that doesn't change that variable.  That is why access to at
least some things will have to be serialized -- to allow threads that
access some part of the global state some level of control on what can
and cannot change while they are doing their job.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> That avoids garbled values where part of a value is from one thread,
> the other part from another thread.  But it does nothing to protect
> the threads other than the one which changed the value from the
> "surprise" of having stuff change under its feet.  Which is the main
> problem to solve, since Emacs code is written under the assumption
> that a variable in the global state doesn't change while some code
> runs that doesn't change that variable.  That is why access to at
> least some things will have to be serialized -- to allow threads that
> access some part of the global state some level of control on what can
> and cannot change while they are doing their job.

My solution (which I've put into practice in redisplay) is to save those
values before sensitive code is executed, and to refer to those saved
values within said code.

But right now I'm stymied by the representation of buffer-local
variables (or rather the lack thereof in a multiprocessing Emacs), so I
plan to give this subject a break for a week or two and revisit it
later.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 09:05:39 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That avoids garbled values where part of a value is from one thread,
> > the other part from another thread.  But it does nothing to protect
> > the threads other than the one which changed the value from the
> > "surprise" of having stuff change under its feet.  Which is the main
> > problem to solve, since Emacs code is written under the assumption
> > that a variable in the global state doesn't change while some code
> > runs that doesn't change that variable.  That is why access to at
> > least some things will have to be serialized -- to allow threads that
> > access some part of the global state some level of control on what can
> > and cannot change while they are doing their job.
> 
> My solution (which I've put into practice in redisplay) is to save those
> values before sensitive code is executed, and to refer to those saved
> values within said code.

That is _everyone's_ solution, not just yours.  But it is not as easy
in practice as it may sound.  E.g., imagine a subroutine that is
called by some higher-level functions, where both the callers and the
subroutine need to access the same variable.  When other threads are
running, there's no longer a guarantee that both the caller and the
callee will see the same value of that variable.  If they must use the
same value, you now need to pass that variable to the callee via its
API, and this is not scalable when you have more than a couple,
especially if the callee is not called directly, but via several
intermediate callers.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> My solution (which I've put into practice in redisplay) is to save those
>> values before sensitive code is executed, and to refer to those saved
>> values within said code.
>
> That is _everyone's_ solution, not just yours.  But it is not as easy
> in practice as it may sound.  E.g., imagine a subroutine that is
> called by some higher-level functions, where both the callers and the
> subroutine need to access the same variable.  When other threads are
> running, there's no longer a guarantee that both the caller and the
> callee will see the same value of that variable.  If they must use the
> same value, you now need to pass that variable to the callee via its
> API, and this is not scalable when you have more than a couple,
> especially if the callee is not called directly, but via several
> intermediate callers.

May the state be captured and passed to the callee under the hood?
Something like forking, but using thread-local state.

For lexical bindings, it is just a matter of passing over current
Vinternal_interpreter_environment; and for global bindings, it is a
matter of passing whatever is altered by the caller thread (with
copy-on-write, this info should be available).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> That is _everyone's_ solution, not just yours.  But it is not as easy
> in practice as it may sound.  E.g., imagine a subroutine that is
> called by some higher-level functions, where both the callers and the
> subroutine need to access the same variable.  When other threads are
> running, there's no longer a guarantee that both the caller and the
> callee will see the same value of that variable.  If they must use the
> same value, you now need to pass that variable to the callee via its
> API, and this is not scalable when you have more than a couple,
> especially if the callee is not called directly, but via several
> intermediate callers.

Then the other solution is to establish temporary thread-local bindings
for each of those variables.  I'm not looking forward to the drudgery of
ascertaining precisely which variables warrant such treatment...

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Eli Zaretskii wrote:

> If we need to lock 99.99% of Emacs "single" variables, it is
> easier to lock everything. Faster, too.

Why do we need to do that?

For one thread, one write operation, and one variable, only
that variable has to be locked?

(operate-on var1) ; here we need a lock on var1
(operate-on var2) ; etc

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 19 Sep 2023 21:38:39 +0200
> 
> Eli Zaretskii wrote:
> 
> > If we need to lock 99.99% of Emacs "single" variables, it is
> > easier to lock everything. Faster, too.
> 
> Why do we need to do that?
> 
> For one thread, one write operation, and one variable, only
> that variable has to be locked?
> 
> (operate-on var1) ; here we need a lock on var1
> (operate-on var2) ; etc

We don't have 'operate' in Emacs, but we do have setq, setf, and
others.  Where do you locak and where do you unlock in that case?

Just try to write a simplest Lisp program, and you will see the
problem.

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Eli Zaretskii wrote:

>>> If we need to lock 99.99% of Emacs "single" variables, it
>>> is easier to lock everything. Faster, too.
>> 
>> Why do we need to do that?
>> 
>> For one thread, one write operation, and one variable, only
>> that variable has to be locked?
>> 
>> (operate-on var1) ; here we need a lock on var1
>> (operate-on var2) ; etc
>
> We don't have 'operate' in Emacs, but we do have setq, setf,
> and others. Where do you locak and where do you unlock in
> that case?
>
> Just try to write a simplest Lisp program, and you will see
> the problem.

There is no way around it, in order to prevent a race
condition any global variable must be locked before it's value
can be altered.

So `setq', `setf' and any other setter of global variables
must first be refered to the locking mechanism where they
either acquire the lock, perform the write, and release the
lock; or, if the variable is already locked by some other
thread wanting to do the same thing, they must be queued so
they will get it in due time.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 22 Sep 2023 16:22:10 +0200
> 
> There is no way around it, in order to prevent a race
> condition any global variable must be locked before it's value
> can be altered.
> 
> So `setq', `setf' and any other setter of global variables
> must first be refered to the locking mechanism where they
> either acquire the lock, perform the write, and release the
> lock; or, if the variable is already locked by some other
> thread wanting to do the same thing, they must be queued so
> they will get it in due time.

So one thread uses setq, releases the lock, then another thread comes,
takes the lock and changes the value of that variable, and when the
first thread uses the variable after that, it will have a value
different from the one the thread used in its setq?  How can one write
a program under these conditions and know what it will do?

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Eli Zaretskii wrote:

>> There is no way around it, in order to prevent a race
>> condition any global variable must be locked before it's
>> value can be altered.
>> 
>> So `setq', `setf' and any other setter of global variables
>> must first be refered to the locking mechanism where they
>> either acquire the lock, perform the write, and release the
>> lock; or, if the variable is already locked by some other
>> thread wanting to do the same thing, they must be queued so
>> they will get it in due time.
>
> So one thread uses setq, releases the lock, then another
> thread comes, takes the lock and changes the value of that
> variable, and when the first thread uses the variable after
> that, it will have a value different from the one the thread
> used in its setq? How can one write a program under these
> conditions and know what it will do?

By relying not on global variables but on local variables.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 22 Sep 2023 18:00:42 +0200
> 
> Eli Zaretskii wrote:
> 
> >> There is no way around it, in order to prevent a race
> >> condition any global variable must be locked before it's
> >> value can be altered.
> >> 
> >> So `setq', `setf' and any other setter of global variables
> >> must first be refered to the locking mechanism where they
> >> either acquire the lock, perform the write, and release the
> >> lock; or, if the variable is already locked by some other
> >> thread wanting to do the same thing, they must be queued so
> >> they will get it in due time.
> >
> > So one thread uses setq, releases the lock, then another
> > thread comes, takes the lock and changes the value of that
> > variable, and when the first thread uses the variable after
> > that, it will have a value different from the one the thread
> > used in its setq? How can one write a program under these
> > conditions and know what it will do?
> 
> By relying not on global variables but on local variables.

It is impossible in Emacs to write a useful Lisp program that doesn't
rely on global variables and other parts of the global state.  You can
only write toy programs without that.

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Eli Zaretskii wrote:

> It is impossible in Emacs to write a useful Lisp program
> that doesn't rely on global variables and other parts of the
> global state. You can only write toy programs without that.

Lisp programs are not toy programs just because they don't use
global variables. One can reduce the reliance on global
variables to a huge degree with various methods. But that's
another discussion.

Because, if one is unallowed to break existing code
implementing an envisioned multi-threaded, multi-core model
one would have to deal with them, and it would then not matter
if they are few or if they are many in that regard.

But note that such a solution would favor any program that
would minimize their use, since those would have much less
overhead to deal with them, obviously. That, however, is
a good thing as it would favor a sound programming style that
minimize their use.

Again, first one would have to agree on a model how things
would work. Say that we go for the idea that each thread has
a superstructure of local variables, that would be accessible
as if they were globals. In a way, this would not be unlike
lexical `let'-closures which can also be used as if they were
globals, and that is one of the use cases of that method BTW.

Second, one would have to identify all Lisp constructs that
interact with the global state. Without changing how those are
used, one would have to think - what do they mean in the new,
thread model? Of many possibly interpretations, which ones do
not contradict how they were previously used?

This will have to be done on a case-by-case basis. One could
start with the most obvious cases, i.e. `setq', `setf', `let',
and so on.

It sounds like a lot but actually don't have to be that much.
Because first, there are a limited number of such constructs.
Second, even tho in terms of implementation it would have to
be done case-by-case, if one can identify a good way of
solving it for on such case, it is likely that method can be
brought over to the rest quite easily.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> So one thread uses setq, releases the lock, then another thread comes,
> takes the lock and changes the value of that variable, and when the
> first thread uses the variable after that, it will have a value
> different from the one the thread used in its setq?  How can one write
> a program under these conditions and know what it will do?

IMHO, there should be a new API to lock global variables by async
threads and/or set them.

I can imagine 3 possible scenarios:

1. Thread does not rely upon global variable value being constant, only
   upon being able to set it.

2. Thread sets global variable value and uses it later, expecting the
   value to remain constant.

3. Thread let-binds global variable.

(1) is easy - we just allow setting global value as we did in the past.
(2) is trickier. We might need to provide a new API, so that thread can
    put a lock onto the variables it works with.
    Or, alternatively, we might keep the global values thread-local
    until the thread terminates - it will most closely resemble what the
    existing Elisp code expects. Then, an API for explicitly setting
    top-level values will be needed for new code that will be written
    with concurrency in mind.
(3) is what Po Lu implemented in his current experimental code (AFAIU) -
    only set thread-local value with global variable is let-bound inside
    the thread. We know for sure that this value is neither reused by
    the concurrent threads nor needs to be kept outside the let-binding.

    BTW, Po Lu, what will happen when a thread spawns another async
    thread while some global variable is let-bound. Should the spawned
    thread somehow inherit the parent thread state?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 16:11:38 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So one thread uses setq, releases the lock, then another thread comes,
> > takes the lock and changes the value of that variable, and when the
> > first thread uses the variable after that, it will have a value
> > different from the one the thread used in its setq?  How can one write
> > a program under these conditions and know what it will do?
> 
> IMHO, there should be a new API to lock global variables by async
> threads and/or set them.

Which again means existing Lisp programs will be unable to run without
very serious changes.

> 1. Thread does not rely upon global variable value being constant, only
>    upon being able to set it.

This is an almost empty set in Emacs.

> 2. Thread sets global variable value and uses it later, expecting the
>    value to remain constant.

This is the problem to solve.

> 3. Thread let-binds global variable.

This is already solved in the current Lisp threadfs.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> 2. Thread sets global variable value and uses it later, expecting the
>>    value to remain constant.
>
> This is the problem to solve.

My suggestion is to (1) create a thread-local copy of a global variable
value at the moment a thread reads/writes that global variable; (2) If a
thread writes a global variable, write the thread-local value back to
global when the thread terminates.

IMHO, this approach will keep all the existing Elisp working.

And if some new Elisp wants to set global value which running
concurrently, introduce a new API.

>> 3. Thread let-binds global variable.
>
> This is already solved in the current Lisp threadfs.

Not really. Current Lisp threads (1) specbind the old value
thread-locally; (2) set symbol value globally; (3) unwind/rewind when
switching between threads.

The global symbol value is directly modified, which won't fly for
concurrent threads.

That said, AFAIU, Po Lu knows how to address this particular problem.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Emanuel Berg]

Ihor Radchenko wrote:

> My suggestion is to [...] create a thread-local copy of
> a global variable value at the moment a thread reads/writes
> that global variable [...]

Yes, if one isn't allowed to break existing code one would
have to do something like that.

Each thread would have a thread space of variables so that
while thinking it would access global variables, it would in
fact populate that thread space little by little and from then
on use those variables.

We could have OO-inspired interface to that thread space with
getters and setters, and all the Lisp that would have to be
modified to first look for a thread space instead of the
global variable, would use such getters and setters.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture. How about copy-on-write?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]

On Fri, Sep 22, 2023 at 06:51:43PM +0300, Eli Zaretskii wrote:
> > From: Emanuel Berg <incal@dataswamp.org>
> > Date: Fri, 22 Sep 2023 16:22:10 +0200
> > 
> > There is no way around it, in order to prevent a race
> > condition any global variable must be locked before it's value
> > can be altered.
> > 
> > So `setq', `setf' and any other setter of global variables
> > must first be refered to the locking mechanism where they
> > either acquire the lock, perform the write, and release the
> > lock; or, if the variable is already locked by some other
> > thread wanting to do the same thing, they must be queued so
> > they will get it in due time.
> 
> So one thread uses setq, releases the lock, then another thread comes,
> takes the lock and changes the value of that variable, and when the
> first thread uses the variable after that, it will have a value
> different from the one the thread used in its setq?  How can one write
> a program under these conditions and know what it will do?

Nicer even. Imagine the compiler saying "oh, we have the value
of this var already in this register!" (this is, after all, why
compiled code is significantly faster).

You can't do this anymore, because each global var is potentially
volatile.

Three quarter of your optimisation opportunities *poof* gone.

Another: suppose you have three globals which have to be
changed in sync. Do you lock around the whole change, although
each var is locked in itself?

Concurrency isn't easy. Most of the application has to "know",
in a way.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Nope. I consider that redisplay is always synchronous
>
> Synchronous to what?

Synchronous in a sense that no two `redisplay' calls can be executed in
parallel.

>> (because of global redisplay lock)
>
> What is the global redisplay lock?

I imagine that the implementation can involve global redisplay lock that
`redisplay' (or its internal functions) need to acquire before running.

>> If multiple threads trigger redisplay with different scroll-margin
>> values, it will be not different compared to the following example:
>
> I understand the problem, I'm asking what could or should be the
> possible solutions.

Sorry, I was not clear. My illustration was not to describe that there
is a problem. My illustration was to show that it is already possible to
run redisplay in various contexts (in the example - let-binding
contexts). Having multiple async threads call re-display with different
settings will be no different from what is already possible without
parallelism.

So, I claim that it is not a problem that should be solved (given that
we make sure that only a single redisplay call can run at a time).

>> 3. xdisp is relying on a number of global-only variables like
>>    `mode-line-compact', `show-trailing-whitespace', and similar.
>>    AFAIR, things like `show-trailing-whitespace' affect how the
>>    optimizations are applied when deciding which windows should be
>>    redisplayed and which should not. I suspect that logic related to
>>    optimizations may be very fragile with async execution.
>
> That's completely irrelevant to the issue at hand.  The fact that
> Emacs has a huge global state, and all of its code relies on that is a
> separate issue.  Here, I asked you in what sense is xdisp.c's code
> single-threaded; if your answer is "because of its reliance on global
> state", it means there's no separate problem of xdisp.c that is based
> on single thread.

Then, we had a misunderstanding. My point is exactly that xdisp.c relies
on global state _a lot_. And it will be non-trivial to change it.

In my mind, it is easier to first solve the problem with generic Elisp
async threads and only then try to look into async redisplay. Because
redisplay is already difficult to alter - untangling its global state
will be a huge task by itself.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Tue, 19 Sep 2023 11:36:24 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Nope. I consider that redisplay is always synchronous
> >
> > Synchronous to what?
> 
> Synchronous in a sense that no two `redisplay' calls can be executed in
> parallel.

I think they can, if they display different buffers in different
windows.

> >> (because of global redisplay lock)
> >
> > What is the global redisplay lock?
> 
> I imagine that the implementation can involve global redisplay lock that
> `redisplay' (or its internal functions) need to acquire before running.

I thought you were describing the current implementation (where
there's no lock).  If you are describing some hypothetical future
implementation, I don't see how we could usefully discuss this without
much more details of that hypothetical design.

> >> 3. xdisp is relying on a number of global-only variables like
> >>    `mode-line-compact', `show-trailing-whitespace', and similar.
> >>    AFAIR, things like `show-trailing-whitespace' affect how the
> >>    optimizations are applied when deciding which windows should be
> >>    redisplayed and which should not. I suspect that logic related to
> >>    optimizations may be very fragile with async execution.
> >
> > That's completely irrelevant to the issue at hand.  The fact that
> > Emacs has a huge global state, and all of its code relies on that is a
> > separate issue.  Here, I asked you in what sense is xdisp.c's code
> > single-threaded; if your answer is "because of its reliance on global
> > state", it means there's no separate problem of xdisp.c that is based
> > on single thread.
> 
> Then, we had a misunderstanding. My point is exactly that xdisp.c relies
> on global state _a lot_. And it will be non-trivial to change it.

It doesn't _rely_ on the state, not on the level of the code.  It
_accesses_ the state as every other Lisp program does, since that's
how Emacs accesses it everywhere.  But there are no hidden assumptions
that the variables used by xdisp.c are global.  If each one of them
will become buffer-local or thread-local, nothing significant will
change in the xdisp.c code.  All xdisp.c needs is to be able to access
the value when needed.  And since each window is processed separately,
there's no assumption that something observed when displaying window
W1 will necessarily hold when displaying window W2.

> In my mind, it is easier to first solve the problem with generic Elisp
> async threads and only then try to look into async redisplay.

We tried that already, with the existing Lisp threads.  One reason why
those are almost never used is that the display issue was left
unresolved.  Any real-life Lisp programs that tries to use threads
bumps into this issue sooner or later.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > What is the global redisplay lock?
>> 
>> I imagine that the implementation can involve global redisplay lock that
>> `redisplay' (or its internal functions) need to acquire before running.
>
> I thought you were describing the current implementation (where
> there's no lock).  If you are describing some hypothetical future
> implementation, I don't see how we could usefully discuss this without
> much more details of that hypothetical design.

This is very simple - I suggest to not touch xdisp.c as much as
possible, just make sure that it is interlocked. And focus on async
support in Elisp code. Once there is async support in Elisp code, we can
move further and look into xdisp.

(I suggest this because I feel that xdisp is a rabbit hole we may sink
in instead of doing something more productive)

>> Then, we had a misunderstanding. My point is exactly that xdisp.c relies
>> on global state _a lot_. And it will be non-trivial to change it.
>
> It doesn't _rely_ on the state, not on the level of the code.  It
> _accesses_ the state as every other Lisp program does, since that's
> how Emacs accesses it everywhere.  But there are no hidden assumptions
> that the variables used by xdisp.c are global.  If each one of them
> will become buffer-local or thread-local, nothing significant will
> change in the xdisp.c code.  All xdisp.c needs is to be able to access
> the value when needed.  And since each window is processed separately,
> there's no assumption that something observed when displaying window
> W1 will necessarily hold when displaying window W2.

I am particularly worried about scenarios when window geometry changes
by asynchronous threads. Or, say, face definitions. Imagine that it
happens at point when we are already drawing the now obsolete geometry
onto glass?

IMHO, it is similar to the problem with GC - for synchronous code, it is
enough to sprinkle maybe_gc in strategic places in the code; for
asynchronous code, we have to somehow make sure that GC is not running
in parallel with another thread trying to allocate memory at the very
same moment.

>> In my mind, it is easier to first solve the problem with generic Elisp
>> async threads and only then try to look into async redisplay.
>
> We tried that already, with the existing Lisp threads.  One reason why
> those are almost never used is that the display issue was left
> unresolved.

That might be one reason, but not the only reason. And certainly not the
most important reason for the use cases where I did try to use threads
myself.

> ... Any real-life Lisp programs that tries to use threads
> bumps into this issue sooner or later.

This is not true. Some Lisp programs? Yes. "Any"? No.
As we discussed previously, a number of Elisp programs can benefit from
async threads even when redisplay is not asynchronous - network queries
(gnus), text parsing (org-mode, slow LSP server communication, xml
processing).

To be clear, it would indeed be nice to have async redisplay. But before
having that we should have async Elisp code (otherwise we cannot run
Elisp from inside async redisplay). And I suggest to focus on this
first, necessary, step of getting async Elisp.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Tue, 19 Sep 2023 13:35:49 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > What is the global redisplay lock?
> >> 
> >> I imagine that the implementation can involve global redisplay lock that
> >> `redisplay' (or its internal functions) need to acquire before running.
> >
> > I thought you were describing the current implementation (where
> > there's no lock).  If you are describing some hypothetical future
> > implementation, I don't see how we could usefully discuss this without
> > much more details of that hypothetical design.
> 
> This is very simple - I suggest to not touch xdisp.c as much as
> possible, just make sure that it is interlocked. And focus on async
> support in Elisp code. Once there is async support in Elisp code, we can
> move further and look into xdisp.
> 
> (I suggest this because I feel that xdisp is a rabbit hole we may sink
> in instead of doing something more productive)

I'm actually of the opposite opinion.  I think trying to parallelize
redisplay is a lower-hanging fruit, and if successful, it could bring
non-trivial gains to Emacs even if the rest remains single-threaded.

> >> Then, we had a misunderstanding. My point is exactly that xdisp.c relies
> >> on global state _a lot_. And it will be non-trivial to change it.
> >
> > It doesn't _rely_ on the state, not on the level of the code.  It
> > _accesses_ the state as every other Lisp program does, since that's
> > how Emacs accesses it everywhere.  But there are no hidden assumptions
> > that the variables used by xdisp.c are global.  If each one of them
> > will become buffer-local or thread-local, nothing significant will
> > change in the xdisp.c code.  All xdisp.c needs is to be able to access
> > the value when needed.  And since each window is processed separately,
> > there's no assumption that something observed when displaying window
> > W1 will necessarily hold when displaying window W2.
> 
> I am particularly worried about scenarios when window geometry changes
> by asynchronous threads. Or, say, face definitions. Imagine that it
> happens at point when we are already drawing the now obsolete geometry
> onto glass?

xdisp.c has solutions for these two (and other similar) situations.
The problems have nothing to do with parallelism, they happen today
because we call Lisp at certain places in the display engine.

> >> In my mind, it is easier to first solve the problem with generic Elisp
> >> async threads and only then try to look into async redisplay.
> >
> > We tried that already, with the existing Lisp threads.  One reason why
> > those are almost never used is that the display issue was left
> > unresolved.
> 
> That might be one reason, but not the only reason. And certainly not the
> most important reason for the use cases where I did try to use threads
> myself.

There's more than one reason, that's true.  But it doesn't change the
fact that all those problems have to be resolved before we have
reasonably usable threads.

> > ... Any real-life Lisp programs that tries to use threads
> > bumps into this issue sooner or later.
> 
> This is not true. Some Lisp programs? Yes. "Any"? No.

Only toy Lisp programs can get away.

> As we discussed previously, a number of Elisp programs can benefit from
> async threads even when redisplay is not asynchronous - network queries
> (gnus), text parsing (org-mode, slow LSP server communication, xml
> processing).

You forget that almost every Lisp program in Emacs either displays
something or affects stuff that affects redisplay.  It's easy to
forget, I know, because in Emacs redisplay "just happens" by some
magic.

> To be clear, it would indeed be nice to have async redisplay.

I'm not talking about async redisplay, I'm talking about the ability
of non-main threads to display something or do something that would
cause redisplay change the stuff on the glass.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 19/09/2023 17:14, Eli Zaretskii wrote:
>> This is very simple - I suggest to not touch xdisp.c as much as
>> possible, just make sure that it is interlocked. And focus on async
>> support in Elisp code. Once there is async support in Elisp code, we can
>> move further and look into xdisp.
>>
>> (I suggest this because I feel that xdisp is a rabbit hole we may sink
>> in instead of doing something more productive)
> I'm actually of the opposite opinion.  I think trying to parallelize
> redisplay is a lower-hanging fruit, and if successful, it could bring
> non-trivial gains to Emacs even if the rest remains single-threaded.

Do we have evidence that the speed of redisplay is a limiting factor in 
general Emacs usage? I.e. with an optimized build and most of the 
popular modes (excepting certain display-heavy Org buffers, let's say).

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Tue, 19 Sep 2023 18:15:50 +0300
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 19/09/2023 17:14, Eli Zaretskii wrote:
> >> This is very simple - I suggest to not touch xdisp.c as much as
> >> possible, just make sure that it is interlocked. And focus on async
> >> support in Elisp code. Once there is async support in Elisp code, we can
> >> move further and look into xdisp.
> >>
> >> (I suggest this because I feel that xdisp is a rabbit hole we may sink
> >> in instead of doing something more productive)
> > I'm actually of the opposite opinion.  I think trying to parallelize
> > redisplay is a lower-hanging fruit, and if successful, it could bring
> > non-trivial gains to Emacs even if the rest remains single-threaded.
> 
> Do we have evidence that the speed of redisplay is a limiting factor in 
> general Emacs usage? I.e. with an optimized build and most of the 
> popular modes (excepting certain display-heavy Org buffers, let's say).

You already forgot the long-lines problems?

And there are other situations where we'd like faster redisplay, even
though they are not as bad.  For example, when a session has many
frames (I believe Stefan Monnier tends to have a lot of them in his
sessions).  To say nothing of the fact that displays become larger and
larger, and many people like to have their frames maximized.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 19/09/2023 18:37, Eli Zaretskii wrote:
>> Date: Tue, 19 Sep 2023 18:15:50 +0300
>> Cc:acm@muc.de,incal@dataswamp.org,emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 19/09/2023 17:14, Eli Zaretskii wrote:
>>>> This is very simple - I suggest to not touch xdisp.c as much as
>>>> possible, just make sure that it is interlocked. And focus on async
>>>> support in Elisp code. Once there is async support in Elisp code, we can
>>>> move further and look into xdisp.
>>>>
>>>> (I suggest this because I feel that xdisp is a rabbit hole we may sink
>>>> in instead of doing something more productive)
>>> I'm actually of the opposite opinion.  I think trying to parallelize
>>> redisplay is a lower-hanging fruit, and if successful, it could bring
>>> non-trivial gains to Emacs even if the rest remains single-threaded.
>> Do we have evidence that the speed of redisplay is a limiting factor in
>> general Emacs usage? I.e. with an optimized build and most of the
>> popular modes (excepting certain display-heavy Org buffers, let's say).
> You already forgot the long-lines problems?

The long-lines problem wasn't solvable with parallel redisplay either, 
it was a case of high algorithm complexity (a combination of them).

> And there are other situations where we'd like faster redisplay, even
> though they are not as bad.  For example, when a session has many
> frames (I believe Stefan Monnier tends to have a lot of them in his
> sessions).  To say nothing of the fact that displays become larger and
> larger, and many people like to have their frames maximized.

It would help to see some info on:

- Whether Stefan's build is optimized,
- How much time the many-frames case spends talking to the windowing 
system, compared to the time our redisplay takes internally. If there 
are many frames displayed, but the current buffer is shown in only one 
of them (or two, etc), then shouldn't the rest of the frames stay mostly 
still anyway?

I also like to have my frames maximized, and I have a 4K display -- even 
so, redisplay seems to be mostly fine (with 1-2 frames anyway). But 
rendering large windows at high resolution, while it can be taxing, 
generally bottlenecks at something else than the layout algorithms. 
Unless you're arguing that the layouts are going to become more complex 
exponentially as well.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Tue, 19 Sep 2023 19:01:31 +0300
> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> Do we have evidence that the speed of redisplay is a limiting factor in
> >> general Emacs usage? I.e. with an optimized build and most of the
> >> popular modes (excepting certain display-heavy Org buffers, let's say).
> > You already forgot the long-lines problems?
> 
> The long-lines problem wasn't solvable with parallel redisplay either, 
> it was a case of high algorithm complexity (a combination of them).

How do you know that parallel redisplay will not solve this?  This can
only be obvious when the detailed design of that is presented.  Until
then, it's anyone's guess.

Anyway, you asked for evidence that redisplay could benefit from
performance boost, and I gave you some.  Now you for some reason want
to claim that my evidence doesn't convince you, but it still is
evidence, right?

> It would help to see some info on:
> 
> - Whether Stefan's build is optimized,
> - How much time the many-frames case spends talking to the windowing 
> system, compared to the time our redisplay takes internally. If there 
> are many frames displayed, but the current buffer is shown in only one 
> of them (or two, etc), then shouldn't the rest of the frames stay mostly 
> still anyway?
> 
> I also like to have my frames maximized, and I have a 4K display -- even 
> so, redisplay seems to be mostly fine (with 1-2 frames anyway). But 
> rendering large windows at high resolution, while it can be taxing, 
> generally bottlenecks at something else than the layout algorithms. 
> Unless you're arguing that the layouts are going to become more complex 
> exponentially as well.

You seem to be opposed to attempts to explore ways of making Emacs
less single-threaded than it is today, unless someone provides a
proof, up front, that this will necessarily solve some particular
problem?  Why? what harm could that possibly do, if it succeeds?  And
if it doesn't succeed, at least we will have learned something
probably useful for the future.

The right time to ask the questions you are asking is when a more or
less detailed design is presented, and we can assess the complexity
and other possible downsides of the proposal against its gains, if
any.  We are not there yet.  Let's keep our minds open and see where
this leads us, okay?

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 19/09/2023 20:54, Eli Zaretskii wrote:
>> Date: Tue, 19 Sep 2023 19:01:31 +0300
>> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> Do we have evidence that the speed of redisplay is a limiting factor in
>>>> general Emacs usage? I.e. with an optimized build and most of the
>>>> popular modes (excepting certain display-heavy Org buffers, let's say).
>>> You already forgot the long-lines problems?
>>
>> The long-lines problem wasn't solvable with parallel redisplay either,
>> it was a case of high algorithm complexity (a combination of them).
> 
> How do you know that parallel redisplay will not solve this?  This can
> only be obvious when the detailed design of that is presented.  Until
> then, it's anyone's guess.

It's not 100%, but seems logical: the problem with long lines is that 
displaying a single line took non-linear amount of time proportional to 
its length. To be able to layout it in parallel, we would somehow have 
to be able to split it into independent pieces (perhaps using some 
caching mechanism?, IDK). If we are actually able to do that (which 
seems difficult), a non-parallel redisplay algorithm should happily use 
the same mid-line checkpoint information to work on smaller pieces of 
the line, likewise increasing the speed.

But I think we've much improved on the issue by reducing the complexity 
instead (correct me if I'm wrong here).

> Anyway, you asked for evidence that redisplay could benefit from
> performance boost, and I gave you some.  Now you for some reason want
> to claim that my evidence doesn't convince you, but it still is
> evidence, right?

I was looking for other examples, to be honest. Like, for example, 
evidence that redisplay is something that keeps an average Emacs session 
from running at 60fps. Or even at 30fps. That would make it a bottleneck.

When I asked the question, I was under the impression that people do 
have some data to point this way, and I just missed it.

> You seem to be opposed to attempts to explore ways of making Emacs
> less single-threaded than it is today, unless someone provides a
> proof, up front, that this will necessarily solve some particular
> problem?  Why? what harm could that possibly do, if it succeeds?  And
> if it doesn't succeed, at least we will have learned something
> probably useful for the future.

It is my experience that good benchmarking often helps with changing the 
design as well. Or coming up with a new one (there are a lot of options 
for how one could proceed).

I'm not saying it is necessary, but it is a good idea, I think.

> The right time to ask the questions you are asking is when a more or
> less detailed design is presented, and we can assess the complexity
> and other possible downsides of the proposal against its gains, if
> any.  We are not there yet.  Let's keep our minds open and see where
> this leads us, okay?

Sure.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Tue, 19 Sep 2023 23:21:46 +0300
> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 19/09/2023 20:54, Eli Zaretskii wrote:
> > 
> > How do you know that parallel redisplay will not solve this?  This can
> > only be obvious when the detailed design of that is presented.  Until
> > then, it's anyone's guess.
> 
> It's not 100%, but seems logical: the problem with long lines is that 
> displaying a single line took non-linear amount of time proportional to 
> its length. To be able to layout it in parallel, we would somehow have 
> to be able to split it into independent pieces (perhaps using some 
> caching mechanism?, IDK). If we are actually able to do that (which 
> seems difficult), a non-parallel redisplay algorithm should happily use 
> the same mid-line checkpoint information to work on smaller pieces of 
> the line, likewise increasing the speed.

You are considering only the redisplay of that single window which has
long lines.  But what about the other windows, which perhaps don't
have long lines?  And what about the main thread of Emacs itself,
which freezes for as long as redisplay didn't finish, thus preventing
user interaction and responsiveness in general?  These factors should
not be neglected.

> But I think we've much improved on the issue by reducing the complexity 
> instead (correct me if I'm wrong here).

If you mean the long-lines improvements in Emacs 29, then we traded
off correctness in at least some cases.  It wasn't for free.

> > Anyway, you asked for evidence that redisplay could benefit from
> > performance boost, and I gave you some.  Now you for some reason want
> > to claim that my evidence doesn't convince you, but it still is
> > evidence, right?
> 
> I was looking for other examples, to be honest. Like, for example, 
> evidence that redisplay is something that keeps an average Emacs session 
> from running at 60fps. Or even at 30fps. That would make it a bottleneck.

With enough text properties and overlays, you can definitely see a
tangible slowdown in frame rate.  E.g., Org developers don't want to
use text properties because they slow down redisplay, and thus they
consider them not scalable enough.

> > You seem to be opposed to attempts to explore ways of making Emacs
> > less single-threaded than it is today, unless someone provides a
> > proof, up front, that this will necessarily solve some particular
> > problem?  Why? what harm could that possibly do, if it succeeds?  And
> > if it doesn't succeed, at least we will have learned something
> > probably useful for the future.
> 
> It is my experience that good benchmarking often helps with changing the 
> design as well. Or coming up with a new one (there are a lot of options 
> for how one could proceed).

On the high-level design level, it should be clear without any need
for benchmarking that separating redisplay from the main thread could
bring benefits.  Isn't that what every GUI application out there does?
So trying to think about that is always a good thing, IMO.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> With enough text properties and overlays, you can definitely see a
> tangible slowdown in frame rate.  E.g., Org developers don't want to
> use text properties because they slow down redisplay, and thus they
> consider them not scalable enough.

I do not think that it needs be addressed by async threads.
The main problem with text properties, AFAIK, is that segment tree Emacs
uses makes searching for next single property change scale with the
total number of segments where _any_ arbitrary property changes, not
with actual number of that searched property segments.

I believe that it is more a question of rewriting text property handling
into multiple segment trees dedicated to individual properties.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 11:38:00 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > With enough text properties and overlays, you can definitely see a
> > tangible slowdown in frame rate.  E.g., Org developers don't want to
> > use text properties because they slow down redisplay, and thus they
> > consider them not scalable enough.
> 
> I do not think that it needs be addressed by async threads.
> The main problem with text properties, AFAIK, is that segment tree Emacs
> uses makes searching for next single property change scale with the
> total number of segments where _any_ arbitrary property changes, not
> with actual number of that searched property segments.

Regardless of all of the above, any expensive processing that can be
divided between several execution units will take less elapsed time,
right?  So, for example, processing text properties of two buffers
sequentially by the same thread (as this is done now) will take longer
than processing them concurrently in parallel, right?  So redisplaying
two windows by two independent threads running concurrently on two
execution units will be faster, right?

> I believe that it is more a question of rewriting text property handling
> into multiple segment trees dedicated to individual properties.

Not as far as redisplay is concerned, no.  The display engine _wants_
to look for the next change in any text property, because it processes
them all "together".  That is, it finds the buffer position where any
of the text properties change, then hands that position to each of the
known methods for handling property changes so that each one of them
does its thing.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Regardless of all of the above, any expensive processing that can be
> divided between several execution units will take less elapsed time,
> right?

Not necessarily. Only when the particular algorithm used can be
effectively converted into concurrent version. Sometimes, concurrent
version performs worse.

> ... So, for example, processing text properties of two buffers
> sequentially by the same thread (as this is done now) will take longer
> than processing them concurrently in parallel, right?
> ... So redisplaying
> two windows by two independent threads running concurrently on two
> execution units will be faster, right?

If the processing is completely independent, it is probably true.
Unless overheads to create new threads are comparable with processing
time (will matter for small buffers that can be process quickly).

>> I believe that it is more a question of rewriting text property handling
>> into multiple segment trees dedicated to individual properties.
>
> Not as far as redisplay is concerned, no.  The display engine _wants_
> to look for the next change in any text property, because it processes
> them all "together".  That is, it finds the buffer position where any
> of the text properties change, then hands that position to each of the
> known methods for handling property changes so that each one of them
> does its thing.

This is... not what I know. May you please point me where in the code it
is done?

From my previous experience, there separate processing of invisible
property and separate processing of composition property. But not
generic processing looking into all present text properties, including
the ones unrelated to redisplay.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 10:41:52 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I believe that it is more a question of rewriting text property handling
> >> into multiple segment trees dedicated to individual properties.
> >
> > Not as far as redisplay is concerned, no.  The display engine _wants_
> > to look for the next change in any text property, because it processes
> > them all "together".  That is, it finds the buffer position where any
> > of the text properties change, then hands that position to each of the
> > known methods for handling property changes so that each one of them
> > does its thing.
> 
> This is... not what I know. May you please point me where in the code it
> is done?

compute_stop_pos finds the next "stop position" where properties (or
something else of interest to redisplay) change, and when the
iteration gets to that position, handle_stop calls the available
handler methods to do their thing.  The intervals of the text
properties are examined by compute_stop_pos.

> >From my previous experience, there separate processing of invisible
> property and separate processing of composition property. But not
> generic processing looking into all present text properties, including
> the ones unrelated to redisplay.

All of the handlers are called one after the other on every stop
position we process.  So the processing is sequential, but not
"separate".  In particular, the higher-level iteration routine
(get_next_display_element), which detects when a stop position was
reached, doesn't know and doesn't care which text property triggered
the stop position; each called method checks if the current iteration
position needs it to do something, and if not, immediately returns.
That's how we support several properties that change at the same
buffer position.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> This is... not what I know. May you please point me where in the code it
>> is done?
>
> compute_stop_pos finds the next "stop position" where properties (or
> something else of interest to redisplay) change, and when the
> iteration gets to that position, handle_stop calls the available
> handler methods to do their thing.  The intervals of the text
> properties are examined by compute_stop_pos.

Thanks!
"stop position" is not triggered by _all_ the properties though.
`compute_stop_pos' calculates the largest region where the properties
listed in `it_props' ('fontified, 'face, 'invisible, and 'composition)
remain the same. If any other property not in the list changes, it is
ignored (except `char-property-alias-alist' which may link other
properties to 'fontified/face/invisible/composition).

Further, handle_stop handlers do search for their corresponding handled
property via Fnext_single_property_change, which again iterates over
every interval in the buffer until that single property value changes.

IMHO, one of the bottlenecks with the current implementation is that we
have to iterate over unrelated properties to find the position of
interest. So, redisplay performance is not just affected by the
properties that matter, but by _all_ properties in buffer (strictly
speaking, by all the intervals in buffer).

I believe that having a more efficient algorithm to detect where a
single property change should speed things up significantly. handle_stop
will benefit directly, while `compute_stop_pos' could query
min (mapcar (Fnext_single_property_change, it_props))
to avoid counting unrelated properties.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 10:05:23 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > compute_stop_pos finds the next "stop position" where properties (or
> > something else of interest to redisplay) change, and when the
> > iteration gets to that position, handle_stop calls the available
> > handler methods to do their thing.  The intervals of the text
> > properties are examined by compute_stop_pos.
> 
> Thanks!
> "stop position" is not triggered by _all_ the properties though.
> `compute_stop_pos' calculates the largest region where the properties
> listed in `it_props' ('fontified, 'face, 'invisible, and 'composition)
> remain the same. If any other property not in the list changes, it is
> ignored (except `char-property-alias-alist' which may link other
> properties to 'fontified/face/invisible/composition).

Does this contradict what I wrote? if so, how?

> Further, handle_stop handlers do search for their corresponding handled
> property via Fnext_single_property_change, which again iterates over
> every interval in the buffer until that single property value changes.

If they do this, it's because they need that to do their thing.
There's no requirement from the handlers to do that.  Moreover, we
could record the position of the next change of each property, when
the handler does determine that, and use that in compute_stop_pos, to
avoid repeating the same search.

> IMHO, one of the bottlenecks with the current implementation is that we
> have to iterate over unrelated properties to find the position of
> interest. So, redisplay performance is not just affected by the
> properties that matter, but by _all_ properties in buffer (strictly
> speaking, by all the intervals in buffer).
> 
> I believe that having a more efficient algorithm to detect where a
> single property change should speed things up significantly. handle_stop
> will benefit directly, while `compute_stop_pos' could query
> min (mapcar (Fnext_single_property_change, it_props))
> to avoid counting unrelated properties.

Feel free to present such a more efficient algorithm, and thanks.
And I'm not sure I understand why you think that

  min (mapcar (Fnext_single_property_change, it_props))

will be cheaper than the current algorithm in compute_stop_pos, since
Fnext_single_property_change basically examines the same intervals
again and again for each property in it_props.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > compute_stop_pos finds the next "stop position" where properties (or
>> > something else of interest to redisplay) change, and when the
>> > iteration gets to that position, handle_stop calls the available
>> > handler methods to do their thing.  The intervals of the text
>> > properties are examined by compute_stop_pos.
>> 
>> Thanks!
>> "stop position" is not triggered by _all_ the properties though.
>> `compute_stop_pos' calculates the largest region where the properties
>> listed in `it_props' ('fontified, 'face, 'invisible, and 'composition)
>> remain the same. If any other property not in the list changes, it is
>> ignored (except `char-property-alias-alist' which may link other
>> properties to 'fontified/face/invisible/composition).
>
> Does this contradict what I wrote? if so, how?

Does not contradict, but adds some details. IMHO, it is important that
not _a_ property should change to "stop". But one of a small set of
properties.

I did not write this to argue about your statement. Just to get a reply
if I missed something when reading the code.

>> Further, handle_stop handlers do search for their corresponding handled
>> property via Fnext_single_property_change, which again iterates over
>> every interval in the buffer until that single property value changes.
>
> If they do this, it's because they need that to do their thing.
> There's no requirement from the handlers to do that.  Moreover, we
> could record the position of the next change of each property, when
> the handler does determine that, and use that in compute_stop_pos, to
> avoid repeating the same search.

That might indeed be one possible optimization.

>> I believe that having a more efficient algorithm to detect where a
>> single property change should speed things up significantly. handle_stop
>> will benefit directly, while `compute_stop_pos' could query
>> min (mapcar (Fnext_single_property_change, it_props))
>> to avoid counting unrelated properties.
>
> Feel free to present such a more efficient algorithm, and thanks.
> And I'm not sure I understand why you think that
>
>   min (mapcar (Fnext_single_property_change, it_props))
>
> will be cheaper than the current algorithm in compute_stop_pos, since
> Fnext_single_property_change basically examines the same intervals
> again and again for each property in it_props.

The algorithm I have in mind is to change interval tree to store several
trees together - a common tree storing segments delimited by _a_
property change + a tree for individual properties. That way, looking up
for next single property change will be a simple call to next_interval for
the property itself + property aliases.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 15:49:21 +0300
> 
> > Feel free to present such a more efficient algorithm, and thanks.
> > And I'm not sure I understand why you think that
> >
> >   min (mapcar (Fnext_single_property_change, it_props))
> >
> > will be cheaper than the current algorithm in compute_stop_pos, since
> > Fnext_single_property_change basically examines the same intervals
> > again and again for each property in it_props.
> 
> The algorithm I have in mind is to change interval tree to store several
> trees together - a common tree storing segments delimited by _a_
> property change + a tree for individual properties. That way, looking up
> for next single property change will be a simple call to next_interval for
> the property itself + property aliases.

Feel free to implement that and benchmark it against the current
implementation.  If it's significantly faster, I';m sure we will
accept the changes.

Thanks.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> The algorithm I have in mind is to change interval tree to store several
>> trees together - a common tree storing segments delimited by _a_
>> property change + a tree for individual properties. That way, looking up
>> for next single property change will be a simple call to next_interval for
>> the property itself + property aliases.
>
> Feel free to implement that and benchmark it against the current
> implementation.  If it's significantly faster, I';m sure we will
> accept the changes.

It is on my todo list.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> On the high-level design level, it should be clear without any need
> for benchmarking that separating redisplay from the main thread could
> bring benefits.  Isn't that what every GUI application out there does?
> So trying to think about that is always a good thing, IMO.

Au contraire, almost all GUI program limit redisplay to their main
threads.

Emacs is not a video game, for which the touchstone of performance is a
measurement taken in ``frames-per-second''.  It is a text editor that
only updates the display subsequent to an editing operation.  That is,
unless you perform more than 60 edits per second and expect an
intervening redisplay after each edit.

Benchmarking redisplay with such measurements is an exercise in fatuity.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Dmitry Gutov <dmitry@gutov.dev>,  yantar92@posteo.net,  acm@muc.de,
>   incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 20:21:39 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > On the high-level design level, it should be clear without any need
> > for benchmarking that separating redisplay from the main thread could
> > bring benefits.  Isn't that what every GUI application out there does?
> > So trying to think about that is always a good thing, IMO.
> 
> Au contraire, almost all GUI program limit redisplay to their main
> threads.

??? On Windows at least, the UI thread is a separate thread, not
necessarily the main one.  We have that in Emacs on Windows.

> Emacs is not a video game, for which the touchstone of performance is a
> measurement taken in ``frames-per-second''.  It is a text editor that
> only updates the display subsequent to an editing operation.  That is,
> unless you perform more than 60 edits per second and expect an
> intervening redisplay after each edit.

This sounds as if you are talking about something other than Emacs,
really.  Try running some day with all the redisplay optimizations
turned off and with enough windows/frames visible, and you will see
how far we are from 60 fps goal.  There's a lot of room for
improvement even without being a video game, just an editor.

Or visit Reddit and see how many users complain that Emacs is slower
in displaying stuff than, say, VSCode.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 20/09/2023 17:51, Eli Zaretskii wrote:
>> Emacs is not a video game, for which the touchstone of performance is a
>> measurement taken in ``frames-per-second''.  It is a text editor that
>> only updates the display subsequent to an editing operation.  That is,
>> unless you perform more than 60 edits per second and expect an
>> intervening redisplay after each edit.
> This sounds as if you are talking about something other than Emacs,
> really.  Try running some day with all the redisplay optimizations
> turned off and with enough windows/frames visible, and you will see
> how far we are from 60 fps goal.  There's a lot of room for
> improvement even without being a video game, just an editor.
> 
> Or visit Reddit and see how many users complain that Emacs is slower
> in displaying stuff than, say, VSCode.

I see those voices, and see the couter-opinions as well. But note that 
those that do complain so are likely comparing the simple case of 
one-buffer-one-frame (so redisplaying multiple windows in parallel 
wouldn't quite fix that).

So their complaints can be compounded from different things:

- Anybody using CC Mode with large files might notice latency during 
editing from running before/after-change-functions (because its design 
involves keeping the syntax info up-to-date in the whole buffer).

- People using LSP likely see better performance because of VC Code 
being able to parse the server's output in a different thread (or web 
worker? I haven't seen the exact design). LSP-Mode folks wrote a POC 
branch of Emacs which parallelizes the JSON parsing as well: 
https://www.reddit.com/r/emacs/comments/ymrkyn/async_nonblocking_jsonrpc_or_lsp_performance/ 
Some work toward mainlining this feature (perhaps after generalizing) 
could help. Or maybe the "multiple concurrent interpreters" approach 
could yield similar gains.

- The "snazzy" mode-line packages sometimes slow things down more than 
expected.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Wed, 20 Sep 2023 22:39:47 +0300
> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> - The "snazzy" mode-line packages sometimes slow things down more than 
> expected.

Why shouldn't this be considered part of our problem with redisplay?
Emacs isn't supposed to be fast enough only in "emacs -Q".

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 21/09/2023 07:31, Eli Zaretskii wrote:
>> Date: Wed, 20 Sep 2023 22:39:47 +0300
>> Cc:yantar92@posteo.net,acm@muc.de,incal@dataswamp.org,emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> - The "snazzy" mode-line packages sometimes slow things down more than
>> expected.
> Why shouldn't this be considered part of our problem with redisplay?
> Emacs isn't supposed to be fast enough only in "emacs -Q".

Oh, I never said we don't have problems.

This one should be solved at the source, though (fixing the package so 
that the mode-line redisplays faster). Or perhaps by disallowing :eval - 
if that is at all feasible.

But I suppose throttling mode-line display should help with wayward 
customizations (which is good, of course, all other things equal).

Re: Emacs design and architecture. How about copy-on-write?

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> - People using LSP likely see better performance because of VC Code
> being able to parse the server's output in a different thread (or web
> worker? I haven't seen the exact design). LSP-Mode folks wrote a POC
> branch of Emacs which parallelizes the JSON parsing as well:
> https://www.reddit.com/r/emacs/comments/ymrkyn/async_nonblocking_jsonrpc_or_lsp_performance/
> Some work toward mainlining this feature (perhaps after generalizing)
> could help. Or maybe the "multiple concurrent interpreters" approach
> could yield similar gains.

Could someone please contact them and ask if they plan to upstream it?

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 22/09/2023 01:21, Stefan Kangas wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> - People using LSP likely see better performance because of VC Code
>> being able to parse the server's output in a different thread (or web
>> worker? I haven't seen the exact design). LSP-Mode folks wrote a POC
>> branch of Emacs which parallelizes the JSON parsing as well:
>> https://www.reddit.com/r/emacs/comments/ymrkyn/async_nonblocking_jsonrpc_or_lsp_performance/
>> Some work toward mainlining this feature (perhaps after generalizing)
>> could help. Or maybe the "multiple concurrent interpreters" approach
>> could yield similar gains.
> Could someone please contact them and ask if they plan to upstream it?

I think they're too busy and not motivated enough at the moment: 
https://github.com/emacs-lsp/emacs/issues/7

Further, it seems that at least for some users the current solution is 
not working very well: https://github.com/emacs-lsp/emacs/issues/9, but 
that seems macOS specific.

Anyway, I think it might need more work to get into upstreamable shape, 
but it's up for discussion how much it's going to be. Overall, the diff 
is not huge, and the first question would be is how generic we'd want 
this feature to be, or whether a separate JSON-parsing thread is 
more-or-less fine.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> This sounds as if you are talking about something other than Emacs,
> really.  Try running some day with all the redisplay optimizations
> turned off and with enough windows/frames visible, and you will see
> how far we are from 60 fps goal.  There's a lot of room for
> improvement even without being a video game, just an editor.

We don't have a "60FPS goal."  "Frames per second" is a metric that
applies to programs which are perpetually redrawing their windows,
either as fast as the monitor refreshes or as fast as the CPU permits.

I don't type 60 characters per second -- so there is no reason for Emacs
to display at 60 frames per second.

If you open the GUI inspector tool of any Gtk+ program, then enable its
"frames per second" counter, you will quickly recognize that a GUI
program such as a text editor seldom, if ever, redraws itself at such a
rate.

> Or visit Reddit and see how many users complain that Emacs is slower
> in displaying stuff than, say, VSCode.

I seriously doubt redisplay performance is the cause of their plights.
It's much more likely to be GC, slow fontification code, or post command
hooks, all three of which present with near identical symptoms.

There's also the old experiment where Eric Naggum disabled garbage
collection messages, yielding a visible speedup.

Re: Emacs design and architecture. How about copy-on-write?

[Adam Porter]

If I may interject:

> We don't have a "60FPS goal."  "Frames per second" is a metric that
> applies to programs which are perpetually redrawing their windows,
> either as fast as the monitor refreshes or as fast as the CPU permits.
> 
> I don't type 60 characters per second -- so there is no reason for Emacs
> to display at 60 frames per second.
> 
> If you open the GUI inspector tool of any Gtk+ program, then enable its
> "frames per second" counter, you will quickly recognize that a GUI
> program such as a text editor seldom, if ever, redraws itself at such a
> rate.

There would seem to be numerous scenarios in which a 60 FPS (or higher) 
redraw rate would be desired, such as smooth scrolling (i.e. 
pixel-by-pixel at a high speed), whether text or images are in the 
buffer (especially if Emacs one day gains improved support for images, 
i.e. displaying an image across lines of text without splitting it into 
many smaller images).

As well, as we all know, Emacs is much more than merely a text editor; 
it's a platform upon which a variety of different applications are run.

Finally, attaining high redraw rates means leaving a large margin, 
helping to maintain acceptable performance under heavier loads, ones we 
may not anticipate at this time.

So I think it's very much a worthwhile goal to be able to redraw at 60+ FPS.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Adam Porter <adam@alphapapa.net> writes:

> There would seem to be numerous scenarios in which a 60 FPS (or
> higher) redraw rate would be desired, such as smooth scrolling
> (i.e. pixel-by-pixel at a high speed), whether text or images are in
> the buffer (especially if Emacs one day gains improved support for
> images, i.e. displaying an image across lines of text without
> splitting it into many smaller images).

Pixel-by-pixel scrolling triggers redisplay optimizations, and Emacs is
capable of equaling a 60 or 30 Hz display refresh rate when these
optimizations are applied.  This was measured when
pixel-scroll-precision-mode was first written.

> As well, as we all know, Emacs is much more than merely a text editor;
> it's a platform upon which a variety of different applications are
> run.

All of which edit or present text.

> Finally, attaining high redraw rates means leaving a large margin,
> helping to maintain acceptable performance under heavier loads, ones
> we may not anticipate at this time.
>
> So I think it's very much a worthwhile goal to be able to redraw at
> 60+ FPS.

I disagree.  It is by far more important to run Lisp (such as Gnus, or
JSON RPC decoding) in the background.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: dmitry@gutov.dev,  yantar92@posteo.net,  acm@muc.de,
>   incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 08:41:29 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This sounds as if you are talking about something other than Emacs,
> > really.  Try running some day with all the redisplay optimizations
> > turned off and with enough windows/frames visible, and you will see
> > how far we are from 60 fps goal.  There's a lot of room for
> > improvement even without being a video game, just an editor.
> 
> We don't have a "60FPS goal."  "Frames per second" is a metric that
> applies to programs which are perpetually redrawing their windows,
> either as fast as the monitor refreshes or as fast as the CPU permits.

We don't have to interpret "60 fps" too literally.  A possible
Emacs-friendly interpretation is "60 display updates per second", for
example.  Here's an experiment to try:

  (global-set-key "\C-z" (function (lambda () (interactive) (scroll-up 1))))

Then visit a large file and lean on C-z.

Another possibility is to lean of C-v.

> I don't type 60 characters per second -- so there is no reason for Emacs
> to display at 60 frames per second.

Leaning on a key, as in the examples above, will produce input events
at a relatively high rate (albeit usually lower than 60 per sec), each
one of them requiring some kind of redisplay.  Even leaning on C-f
triggers redisplay at high rate.  I suggest to try this in various
non-dash-Q configurations and see how we fare in these situations.
There's nothing outlandish in expecting an editor to keep up with
high-rate scrolling commands, I hope you agree with that at least.

> > Or visit Reddit and see how many users complain that Emacs is slower
> > in displaying stuff than, say, VSCode.
> 
> I seriously doubt redisplay performance is the cause of their plights.

You can doubt all you want, but some of what they say is definitely
related to redisplay, IMNSHO.

> There's also the old experiment where Eric Naggum disabled garbage
> collection messages, yielding a visible speedup.

That's orthogonal to the issue at hand.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> We don't have to interpret "60 fps" too literally.  A possible
> Emacs-friendly interpretation is "60 display updates per second", for
> example.  Here's an experiment to try:
>
>   (global-set-key "\C-z" (function (lambda () (interactive) (scroll-up 1))))
>
> Then visit a large file and lean on C-z.
>
> Another possibility is to lean of C-v.
>
>> I don't type 60 characters per second -- so there is no reason for Emacs
>> to display at 60 frames per second.
>
> Leaning on a key, as in the examples above, will produce input events
> at a relatively high rate (albeit usually lower than 60 per sec), each
> one of them requiring some kind of redisplay.  Even leaning on C-f
> triggers redisplay at high rate.  I suggest to try this in various
> non-dash-Q configurations and see how we fare in these situations.
> There's nothing outlandish in expecting an editor to keep up with
> high-rate scrolling commands, I hope you agree with that at least.

I agree, but slow scrolling is not redisplay's fault.  If I type:

  M-: (jit-lock-fontify-all) RET

or enable c-ts-mode prior to leaning on C-v, within an Emacs session
with a 790 MiB resident set that has been executing continuously for 32
days, then Emacs scrolls (xterm.c and xdisp.c) as fast as any other
editor on my system.  Conceivably even faster than some others, that is
to say the VS Code demonstration site at vscode.dev.

> You can doubt all you want, but some of what they say is definitely
> related to redisplay, IMNSHO.

Any examples?  Because my experience with p-s-p-m demonstrates that
redisplay is absolutely capable of matching the display refresh rate
while scrolling, the very cornerstone of
`pixel-scroll-precision-use-momentum'.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: dmitry@gutov.dev,  yantar92@posteo.net,  acm@muc.de,
>   incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 15:48:58 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Leaning on a key, as in the examples above, will produce input events
> > at a relatively high rate (albeit usually lower than 60 per sec), each
> > one of them requiring some kind of redisplay.  Even leaning on C-f
> > triggers redisplay at high rate.  I suggest to try this in various
> > non-dash-Q configurations and see how we fare in these situations.
> > There's nothing outlandish in expecting an editor to keep up with
> > high-rate scrolling commands, I hope you agree with that at least.
> 
> I agree, but slow scrolling is not redisplay's fault.  If I type:
> 
>   M-: (jit-lock-fontify-all) RET
> 
> or enable c-ts-mode prior to leaning on C-v, within an Emacs session
> with a 790 MiB resident set that has been executing continuously for 32
> days, then Emacs scrolls (xterm.c and xdisp.c) as fast as any other
> editor on my system.

Then try leaning on C-n or C-p _after_ everything is already
fontified.  You will still see that Emacs sometimes cannot keep up,
especially if lines are not too short.

You will never succeed in convincing me that our redisplay is as fast
as I'd like it to be.  I've seen too many proofs to the contrary.

> > You can doubt all you want, but some of what they say is definitely
> > related to redisplay, IMNSHO.
> 
> Any examples?

Sorry, you will have to find them yourself.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Then try leaning on C-n or C-p _after_ everything is already
> fontified.  You will still see that Emacs sometimes cannot keep up,
> especially if lines are not too short.

Long lines are, at worst, infrequently encountered in source code,
aren't they?  And our troubles with long lines are an algorithmic
impediment, which cannot be ameliorated merely by redisplaying each
window on a different CPU.

> You will never succeed in convincing me that our redisplay is as fast
> as I'd like it to be.  I've seen too many proofs to the contrary.

From my POV, we frequently encounter circumstances where GC, Font Lock,
process filters or post-command-hooks precipitate difficulties keeping
Emacs responsive to user input.  Seldom are such problems unequivocally
attributed to redisplay itself.

I guess we can only agree to disagree.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> Then try leaning on C-n or C-p _after_ everything is already
>> fontified.  You will still see that Emacs sometimes cannot keep up,
>> especially if lines are not too short.
>
> Long lines are, at worst, infrequently encountered in source code,
> aren't they?  And our troubles with long lines are an algorithmic
> impediment, which cannot be ameliorated merely by redisplaying each
> window on a different CPU.

I think that there is at least one way to address long lines using
asynchronous redisplay - put a placeholder on the problematic line and
continue calculating the actual rendering in the background instead.
That will not force us to compromise between rendering time and
accuracy, as we do now, with long-line-threshold.

Similar approach might be used to render mode lines - render a
placeholder until it is fully calculated, keeping Emacs responsive.

This idea is also relevant to a situation when asynchronous threads
"locks" a buffer whose window is visible. If a user tries to switch to
such buffer, it would be nice if Emacs does not hang until the lock is
released.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 21/09/2023 12:39, Ihor Radchenko wrote:
> Po Lu<luangruo@yahoo.com>  writes:
> 
>>> Then try leaning on C-n or C-p_after_  everything is already
>>> fontified.  You will still see that Emacs sometimes cannot keep up,
>>> especially if lines are not too short.
>> Long lines are, at worst, infrequently encountered in source code,
>> aren't they?  And our troubles with long lines are an algorithmic
>> impediment, which cannot be ameliorated merely by redisplaying each
>> window on a different CPU.
> I think that there is at least one way to address long lines using
> asynchronous redisplay - put a placeholder on the problematic line and
> continue calculating the actual rendering in the background instead.
> That will not force us to compromise between rendering time and
> accuracy, as we do now, with long-line-threshold.

Until you're laid out the long line, you don't know which screen line it 
will finish at, or at which height specifically (it might have images, 
or taller text due to faces, etc). So you can't render the remainder of 
the buffer either.

> Similar approach might be used to render mode lines - render a
> placeholder until it is fully calculated, keeping Emacs responsive.

I hope by "placeholder" you mean the previous rendered image of it. But 
then the mode-line won't be refreshed at 60fps still, which some said is 
a good goal? (I tentatively agree).

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Dmitry Gutov <dmitry@gutov.dev> writes:

>> I think that there is at least one way to address long lines using
>> asynchronous redisplay - put a placeholder on the problematic line and
>> continue calculating the actual rendering in the background instead.
>> That will not force us to compromise between rendering time and
>> accuracy, as we do now, with long-line-threshold.
>
> Until you're laid out the long line, you don't know which screen line it 
> will finish at, or at which height specifically (it might have images, 
> or taller text due to faces, etc). So you can't render the remainder of 
> the buffer either.

What I had in mind is the processing to happen in two stages: (1)
calculate "quick-and-dirty" layout; (2) start calculating "accurate"
layout. If calculating "accurate" layout takes long time, go ahead and
draw the "quick-and-dirty" version on the glass, while "accurate"
version continues being computed in the background.

Of course, this threshold should be configurable.

>> Similar approach might be used to render mode lines - render a
>> placeholder until it is fully calculated, keeping Emacs responsive.
>
> I hope by "placeholder" you mean the previous rendered image of it.

More precisely, what I had in mind is the old glyph matrix + some
indication that the rendering is not up-to-date (like classic sand clock
icon)

> ... But then the mode-line won't be refreshed at 60fps still, which
> some said is a good goal? (I tentatively agree).

I also agree that it is a good goal.

The scenario I had in mind is poorly written third-party mode line
packages that, for example, query git on every redisplay. There is
nothing we can do about poor Elisp code in third-party packages, but the
proposed approach will at least prevent Emacs being laggy or hanging in
the eyes of the user.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 21/09/2023 13:48, Ihor Radchenko wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>>> I think that there is at least one way to address long lines using
>>> asynchronous redisplay - put a placeholder on the problematic line and
>>> continue calculating the actual rendering in the background instead.
>>> That will not force us to compromise between rendering time and
>>> accuracy, as we do now, with long-line-threshold.
>> Until you're laid out the long line, you don't know which screen line it
>> will finish at, or at which height specifically (it might have images,
>> or taller text due to faces, etc). So you can't render the remainder of
>> the buffer either.
> What I had in mind is the processing to happen in two stages: (1)
> calculate "quick-and-dirty" layout; (2) start calculating "accurate"
> layout. If calculating "accurate" layout takes long time, go ahead and
> draw the "quick-and-dirty" version on the glass, while "accurate"
> version continues being computed in the background.
> 
> Of course, this threshold should be configurable.
> 
>>> Similar approach might be used to render mode lines - render a
>>> placeholder until it is fully calculated, keeping Emacs responsive.
>> I hope by "placeholder" you mean the previous rendered image of it.
> More precisely, what I had in mind is the old glyph matrix + some
> indication that the rendering is not up-to-date (like classic sand clock
> icon)

Any visible indicators of incomplete layouts or "not yet updated" icons 
are going to be a visual nuisance if the refresh speed is still above 
5fps. So we should be careful with those.

And even the "slow" mode-line packages still manage to maintain at least 
5fps, I believe.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Dmitry Gutov <dmitry@gutov.dev> writes:

>>> I hope by "placeholder" you mean the previous rendered image of it.
>> More precisely, what I had in mind is the old glyph matrix + some
>> indication that the rendering is not up-to-date (like classic sand clock
>> icon)
>
> Any visible indicators of incomplete layouts or "not yet updated" icons 
> are going to be a visual nuisance if the refresh speed is still above 
> 5fps. So we should be careful with those.

Sure. This is a minor detail that can be customizeable, for example.

> And even the "slow" mode-line packages still manage to maintain at least 
> 5fps, I believe.

Nope. 0.2 sec is the delay that is barely noticeable. When "slow"
mode-line packages affect users it is usually much worse than 5fps.
(Though sometimes it is due to redisplay being fired frequently, not the
individual refresh times being too large).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 21/09/2023 14:17, Ihor Radchenko wrote:
>> And even the "slow" mode-line packages still manage to maintain at least
>> 5fps, I believe.
> Nope. 0.2 sec is the delay that is barely noticeable. When "slow"
> mode-line packages affect users it is usually much worse than 5fps.
> (Though sometimes it is due to redisplay being fired frequently, not the
> individual refresh times being too large).

My own experience is finding certain mode-lines that look okay on the 
first glance, but then after measuring, finding them to be the chief 
contributor to the lower refresh rate. Those might refresh well above 
10fps, but spending 100ms (or even 50ms) on a mode-line is ridiculous.

Delays which are noticeable by humans start at around 100ms for random 
ones. But if the goal is 60fps, redisplay should finish at below 2ms.

Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 21/09/2023 14:17, Ihor Radchenko wrote:
>>> And even the "slow" mode-line packages still manage to maintain at least
>>> 5fps, I believe.
>> Nope. 0.2 sec is the delay that is barely noticeable. When "slow"
>> mode-line packages affect users it is usually much worse than 5fps.
>> (Though sometimes it is due to redisplay being fired frequently, not the
>> individual refresh times being too large).
>
> My own experience is finding certain mode-lines that look okay on the 
> first glance, but then after measuring, finding them to be the chief 
> contributor to the lower refresh rate. Those might refresh well above 
> 10fps, but spending 100ms (or even 50ms) on a mode-line is ridiculous.
>
> Delays which are noticeable by humans start at around 100ms for random 
> ones. But if the goal is 60fps, redisplay should finish at below 2ms.

Which just make the point stronger - even slight performance
inefficiencies for mode line constructs are affecting user experience a
lot.

It might be a good idea to address this problem even within the current
single-threaded redisplay.

What about automatically debouncing slow :eval constructs in the mode
line? If we make sure that :eval constructs running longer than a
threshold do not run too frequently, it can certainly improve user
experience.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Dmitry Gutov]

On 21/09/2023 14:36, Ihor Radchenko wrote:
> What about automatically debouncing slow :eval constructs in the mode
> line? If we make sure that :eval constructs running longer than a
> threshold do not run too frequently, it can certainly improve user
> experience.

Sounds worth looking into.

Though if we just do this silently, it can hide performance problems, 
both discouraging the mode-line authors from fixing them, and creating 
odd behaviors (from the user's POV) when something which should change, 
doesn't.

BTW, most advanced mode-lines (including smart-mode-line, which I use, 
and is reasonably fast) use the :eval constructs pretty much everywhere.

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 21/09/2023 14:36, Ihor Radchenko wrote:
>> What about automatically debouncing slow :eval constructs in the mode
>> line? If we make sure that :eval constructs running longer than a
>> threshold do not run too frequently, it can certainly improve user
>> experience.
>
> Sounds worth looking into.
>
> Though if we just do this silently, it can hide performance problems, 
> both discouraging the mode-line authors from fixing them, and creating 
> odd behaviors (from the user's POV) when something which should change, 
> doesn't.

A warning may be displayed when this "debouncing" is activated.
This will notify the users yet not degrading performance.

> BTW, most advanced mode-lines (including smart-mode-line, which I use, 
> and is reasonably fast) use the :eval constructs pretty much everywhere.

Yup. And users often have no idea how much performance may be affected
by these "fancy" mode-lines. Leading to absence of bug reports in some
edge cases and general complaints that "Emacs is slow".

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, Eli Zaretskii <eliz@gnu.org>, acm@muc.de,
>  incal@dataswamp.org, emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 10:07:38 +0000
> 
> > Though if we just do this silently, it can hide performance problems, 
> > both discouraging the mode-line authors from fixing them, and creating 
> > odd behaviors (from the user's POV) when something which should change, 
> > doesn't.
> 
> A warning may be displayed when this "debouncing" is activated.

You cannot easily display any warnings from redisplay.

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Though if we just do this silently, it can hide performance problems, 
>> > both discouraging the mode-line authors from fixing them, and creating 
>> > odd behaviors (from the user's POV) when something which should change, 
>> > doesn't.
>> 
>> A warning may be displayed when this "debouncing" is activated.
>
> You cannot easily display any warnings from redisplay.

Can't the code inside :eval display warnings?

In any case, it is not a big problem to arrange the warning to be
displayed after redisplay finishes.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 12:50:39 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Though if we just do this silently, it can hide performance problems, 
> >> > both discouraging the mode-line authors from fixing them, and creating 
> >> > odd behaviors (from the user's POV) when something which should change, 
> >> > doesn't.
> >> 
> >> A warning may be displayed when this "debouncing" is activated.
> >
> > You cannot easily display any warnings from redisplay.
> 
> Can't the code inside :eval display warnings?

No, because displaying a warning requires redisplay, and we don't
support recursive redisplays.

> In any case, it is not a big problem to arrange the warning to be
> displayed after redisplay finishes.

I don't see a reason to do that with this particular kind of problems.
We currently don't display anything by default even if Lisp called by
redisplay signals an error, we just log that in *Messages*.  Why
should these :eval problems we treated differently?

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> In any case, it is not a big problem to arrange the warning to be
>> displayed after redisplay finishes.
>
> I don't see a reason to do that with this particular kind of problems.
> We currently don't display anything by default even if Lisp called by
> redisplay signals an error, we just log that in *Messages*.  Why
> should these :eval problems we treated differently?

The aim here is to make users aware that the proposed optimization is
activated. It does not have to be warning specifically. Might be
something else visible enough. Warning was just the first thing that
came to my mind.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 13:10:40 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't see a reason to do that with this particular kind of problems.
> > We currently don't display anything by default even if Lisp called by
> > redisplay signals an error, we just log that in *Messages*.  Why
> > should these :eval problems we treated differently?
> 
> The aim here is to make users aware that the proposed optimization is
> activated. It does not have to be warning specifically. Might be
> something else visible enough. Warning was just the first thing that
> came to my mind.

They can look at *Messages* and see the warning there, like they do
with any other redisplay problem.

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> The aim here is to make users aware that the proposed optimization is
>> activated. It does not have to be warning specifically. Might be
>> something else visible enough. Warning was just the first thing that
>> came to my mind.
>
> They can look at *Messages* and see the warning there, like they do
> with any other redisplay problem.

I personally rarely reach to *Messages* and many messages are simply
lost when displayed in quick succession. But that's my IMHO - I do not
like how messages about redisplay problems are displayed either.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, Eli Zaretskii <eliz@gnu.org>, acm@muc.de,
>  incal@dataswamp.org, emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 11:36:34 +0000
> 
> What about automatically debouncing slow :eval constructs in the mode
> line? If we make sure that :eval constructs running longer than a
> threshold do not run too frequently, it can certainly improve user
> experience.

How do you propose to evaluate just portions of the mode line?  What
will the :eval whose processing is bypassed yield, and how will that
affect processing of the rest of mode-line-format, given its
highly-recursive processing structure and the fact that the text
produced by this processing is laid out as it is produced?

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> What about automatically debouncing slow :eval constructs in the mode
>> line? If we make sure that :eval constructs running longer than a
>> threshold do not run too frequently, it can certainly improve user
>> experience.
>
> How do you propose to evaluate just portions of the mode line?  What
> will the :eval whose processing is bypassed yield, and how will that
> affect processing of the rest of mode-line-format, given its
> highly-recursive processing structure and the fact that the text
> produced by this processing is laid out as it is produced?

What I have in mind is the following:

1. Every time Emacs processes :eval construct it (a) measures the time
   taken; (b) caches return value.
   The total mode line render time is also recorded.

2. If the total render time exceeds configurable threshold, processing
   the most time-consuming :eval constructs will be suspended until
   "debounce" time since the last full processing.

   By "suspended", I mean that cached :eval value is reused instead of
   invoking :eval again. Possibly, not reused verbatim, but by replacing
   the cached value with some kind of placeholder that has the same
   total height/width as the cached value.

   For example, consider :eval segment invoking external git command and
   producing branch name: "⛬ master".
   If that command takes too long, Emacs will display
   "⏳       " instead of
   "⛬ master", until the mode-line redisplay that is some fixed time
   ahead.

3. After "debounce" time since the last "suspend", all the :eval are
   processed in full again, generating up-to-date mode-line.

In terms of values, I am thinking something like 0.2 sec for threshold
and 1.0 sec for debounce. This way, mode line will be outdated no longer
than 1 second.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 10:23:47 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How do you propose to evaluate just portions of the mode line?  What
> > will the :eval whose processing is bypassed yield, and how will that
> > affect processing of the rest of mode-line-format, given its
> > highly-recursive processing structure and the fact that the text
> > produced by this processing is laid out as it is produced?
> 
> What I have in mind is the following:
> 
> 1. Every time Emacs processes :eval construct it (a) measures the time
>    taken; (b) caches return value.

This needs to somehow identify each :eval and record its execution
time.

> 2. If the total render time exceeds configurable threshold, processing
>    the most time-consuming :eval constructs will be suspended until
>    "debounce" time since the last full processing.
> 
>    By "suspended", I mean that cached :eval value is reused instead of
>    invoking :eval again. Possibly, not reused verbatim, but by replacing
>    the cached value with some kind of placeholder that has the same
>    total height/width as the cached value.

This needs also to cache the last value of each :eval.

It also assumes that these elements are quasi-static, for long enough
time to decide which one(s) are slow and reuse their cached values.
But mode-line-format is just a variable, and a Lisp program can
legitimately change it at a high frequency, which will defeat any
useful caching (because the :eval elements change and no longer match
their cached entries).  Moreover, a Lisp program can decide to
deliberately defeat this caching by relatively simple measures.  Last,
but not least, all this additional processing will make mode-line
display slower even if all the :eval forms are completely benign.

Also, Emacs has a tradition of leaving Lisp programs enough rope to
hang themselves, as long as they don't cause crashes and other
catastrophic outcomes.  This proposal will prevent a Lisp program in
:eval from doing its thing simply because we don't like its timing
(which, btw, depends on the CPU power, so a threshold that is an
absolute number of seconds is not necessarily the best idea).  In some
quarters, this could be seen as breaking the contract.

The example with an external Git command is very relevant: using a
cached value could present to the user completely misleading
indications.

So I think this is a lot of trouble for little gain, and could be also
against our long-standing policies.

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> 1. Every time Emacs processes :eval construct it (a) measures the time
>>    taken; (b) caches return value.
>
> This needs to somehow identify each :eval and record its execution
> time.

Just sxhash, for example.

>> 2. If the total render time exceeds configurable threshold, processing
>>    the most time-consuming :eval constructs will be suspended until
>>    "debounce" time since the last full processing.
>> 
>>    By "suspended", I mean that cached :eval value is reused instead of
>>    invoking :eval again. Possibly, not reused verbatim, but by replacing
>>    the cached value with some kind of placeholder that has the same
>>    total height/width as the cached value.
>
> This needs also to cache the last value of each :eval.

Yes.

> It also assumes that these elements are quasi-static, for long enough
> time to decide which one(s) are slow and reuse their cached values.

You are right. Such assumption may or may not hold. Though nothing
prevents us from deciding based on running average of several previous
:eval processing times instead of just one previous time.

> But mode-line-format is just a variable, and a Lisp program can
> legitimately change it at a high frequency, which will defeat any
> useful caching (because the :eval elements change and no longer match
> their cached entries).

Sure. Though I am not aiming to solve such scenarios. IMHO, they are not
very common.

> ... Moreover, a Lisp program can decide to
> deliberately defeat this caching by relatively simple measures.

I see no reason to prevent that if it is deliberate. The point is that
Lisp program will declare that "yes, I know that mode line may be slow"
by doing such thing. Which should be totally fine as long as it is the
decision made by the Lisp program author.

> ... Last, but not least, all this additional processing will make
> mode-line display slower even if all the :eval forms are completely
> benign.

Maybe. But we can try and see if it is actually going to be the case.

> Also, Emacs has a tradition of leaving Lisp programs enough rope to
> hang themselves, as long as they don't cause crashes and other
> catastrophic outcomes.  This proposal will prevent a Lisp program in
> :eval from doing its thing simply because we don't like its timing
> (which, btw, depends on the CPU power, so a threshold that is an
> absolute number of seconds is not necessarily the best idea).  In some
> quarters, this could be seen as breaking the contract.

IMHO, long-line-threshold, for example, is also breaking the contract
you are referring to. So, if we leave legitimate ways for a program to
bypass this optimization when necessary, it should not be a problem.

> The example with an external Git command is very relevant: using a
> cached value could present to the user completely misleading
> indications.
>
> So I think this is a lot of trouble for little gain, and could be also
> against our long-standing policies.

Well. Strictly speaking, current Git branch may not be up-to-date even
without my proposal. Simply because Emacs gives no guarantees about how
frequently the mode line is refreshed. redisplay optimizations may
decide to not refresh the mode-line when typing.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 16:03:05 +0300
> 
> redisplay optimizations may decide to not refresh the mode-line when
> typing.

If they do, it's a bug.

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> redisplay optimizations may decide to not refresh the mode-line when
>> typing.
>
> If they do, it's a bug.

You are right. Typing appears to trigger mode-line redisplay.
What I thought is a redisplay optimization is different - `vc-mode'
variable used to display VC status is not updated on every redisplay.
Instead, it is modified when visiting a file and when saving a file.
The end result is the same - VC status is not always up-to-date.

Similar approach is often used by other well-optimized mode-line
constructs, leading to sometimes-outdated mode-line.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 13:19:42 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> redisplay optimizations may decide to not refresh the mode-line when
> >> typing.
> >
> > If they do, it's a bug.
> 
> You are right. Typing appears to trigger mode-line redisplay.
> What I thought is a redisplay optimization is different - `vc-mode'
> variable used to display VC status is not updated on every redisplay.
> Instead, it is modified when visiting a file and when saving a file.
> The end result is the same - VC status is not always up-to-date.
> 
> Similar approach is often used by other well-optimized mode-line
> constructs, leading to sometimes-outdated mode-line.

These are not problems with display, these are problems with the modes
which define and calculate the respective mode-line variables.

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> What I thought is a redisplay optimization is different - `vc-mode'
>> variable used to display VC status is not updated on every redisplay.
>> Instead, it is modified when visiting a file and when saving a file.
>> The end result is the same - VC status is not always up-to-date.
>> 
>> Similar approach is often used by other well-optimized mode-line
>> constructs, leading to sometimes-outdated mode-line.
>
> These are not problems with display, these are problems with the modes
> which define and calculate the respective mode-line variables.

I would not call my example a "problem with the mode". There is simply
no easy way to make git process call faster. So, a compromise is chosen
to not affect Emacs performance.

I am proposing a similar compromise to improve default experience.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Dmitry Gutov]

On 22/09/2023 13:23, Ihor Radchenko wrote:
> What I have in mind is the following:
> 
> 1. Every time Emacs processes :eval construct it (a) measures the time
>     taken; (b) caches return value.
>     The total mode line render time is also recorded.
> 
> 2. If the total render time exceeds configurable threshold, processing
>     the most time-consuming :eval constructs will be suspended until
>     "debounce" time since the last full processing.
> 
>     By "suspended", I mean that cached :eval value is reused instead of
>     invoking :eval again. Possibly, not reused verbatim, but by replacing
>     the cached value with some kind of placeholder that has the same
>     total height/width as the cached value.
> 
>     For example, consider :eval segment invoking external git command and
>     producing branch name: "⛬ master".
>     If that command takes too long, Emacs will display
>     "⏳       " instead of
>     "⛬ master", until the mode-line redisplay that is some fixed time
>     ahead.
> 
> 3. After "debounce" time since the last "suspend", all the :eval are
>     processed in full again, generating up-to-date mode-line.

I think it can be worth a try, although this general thing sounds easier 
to do (in a different way) with parallel rendering of 
buffers/mode-lines/etc. If we ever get to that, of course.

> In terms of values, I am thinking something like 0.2 sec for threshold
> and 1.0 sec for debounce. This way, mode line will be outdated no longer
> than 1 second.

FWIW, the "slow" mode-lines I had tried in the past were still much 
faster than the above.

RE: [External] : Re: Debouncing slow mode line constructs (was: Emacs design and architecture. How about copy-on-write?)

[Drew Adams]

(Caveat: Not really following this thread.)

> What I have in mind is the following:
> 
> 1. Every time Emacs processes :eval construct it (a) measures the time
>    taken; (b) caches return value.
>    The total mode line render time is also recorded.
> 
> 2. If the total render time exceeds configurable threshold, processing
>    the most time-consuming :eval constructs will be suspended until
>    "debounce" time since the last full processing....
> 
> 3. After "debounce" time since the last "suspend", all the :eval are
>    processed in full again, generating up-to-date mode-line.

Assuming that's feasible, and might be done
(e.g. Eli's mail suggests it's not so easy):

Instead of just doing that automatically,
or just letting users turn it on/off, why
not give users more of a heads-up and more
control over how to handle the problem?

When you've detected that some mode-line
processing (e.g. with :eval, but maybe even
otherwise) takes "too long" (based on a
user-configurable threshold), take whatever
action the user has previously chosen, e.g.:

1. What you describe: automatic debouncing.
2. Highlight the mode-line in some obvious
   way, as long as the problem exists.
3. #1 & #2: highlight and debounce.
4. Prompt user for what to do.
5. Invoke some user-defined or 3rd-party
   library-defined function.  (One example
   could be to report the problem to the
   library maintainer.)
6. ...

User choice could be by config (option).
Or it could be by user code or 3rd-party
library code (e.g. mode-line libraries
could provide alternative behavior).

That's off the top of my head, obviously.
The idea is to, in some way:

 * Let users know what's happening.
 * Give users some control over what to do.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, Eli Zaretskii <eliz@gnu.org>, acm@muc.de,
>  incal@dataswamp.org, emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 10:48:25 +0000
> 
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
> >> I think that there is at least one way to address long lines using
> >> asynchronous redisplay - put a placeholder on the problematic line and
> >> continue calculating the actual rendering in the background instead.
> >> That will not force us to compromise between rendering time and
> >> accuracy, as we do now, with long-line-threshold.
> >
> > Until you're laid out the long line, you don't know which screen line it 
> > will finish at, or at which height specifically (it might have images, 
> > or taller text due to faces, etc). So you can't render the remainder of 
> > the buffer either.
> 
> What I had in mind is the processing to happen in two stages: (1)
> calculate "quick-and-dirty" layout; (2) start calculating "accurate"
> layout. If calculating "accurate" layout takes long time, go ahead and
> draw the "quick-and-dirty" version on the glass, while "accurate"
> version continues being computed in the background.
> 
> Of course, this threshold should be configurable.
> 
> >> Similar approach might be used to render mode lines - render a
> >> placeholder until it is fully calculated, keeping Emacs responsive.
> >
> > I hope by "placeholder" you mean the previous rendered image of it.
> 
> More precisely, what I had in mind is the old glyph matrix + some
> indication that the rendering is not up-to-date (like classic sand clock
> icon)

You are basically proposing us to provide broken, incorrect, and
outdated display in some cases.  That won't fly, so it's a
non-starter.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 21/09/2023 16:00, Eli Zaretskii wrote:
> You are basically proposing us to provide broken, incorrect, and
> outdated display in some cases.  That won't fly, so it's a
> non-starter.

If might prove an okay experience, if the "outdated" part is within 
certain bounds. Like if the current buffer refreshes at 30/60fps, and 
the mode-line at 10/20fps, for example.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Thu, 21 Sep 2023 16:30:00 +0300
> Cc: luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 21/09/2023 16:00, Eli Zaretskii wrote:
> > You are basically proposing us to provide broken, incorrect, and
> > outdated display in some cases.  That won't fly, so it's a
> > non-starter.
> 
> If might prove an okay experience, if the "outdated" part is within 
> certain bounds. Like if the current buffer refreshes at 30/60fps, and 
> the mode-line at 10/20fps, for example.

That's not what was proposed.  The proposal was to do a
"quick-and-dirty" layout first, and use the results if the "accurate"
layout takes too long.  The quick-and-dirty layout can only be based
on ignoring changes in fonts and disregarding images and stuff like
that, so the screen lines will be wrong, and you will see them jump
and/or rearrange after some delay, which could be some seconds.  Some
browsers do that when they need to download images, and it always jars
me when that happens.  Usually, I also lose the place where I was
reading.

I hope Emacs will never behave like that.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 21/09/2023 16:44, Eli Zaretskii wrote:
>> Date: Thu, 21 Sep 2023 16:30:00 +0300
>> Cc: luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 21/09/2023 16:00, Eli Zaretskii wrote:
>>> You are basically proposing us to provide broken, incorrect, and
>>> outdated display in some cases.  That won't fly, so it's a
>>> non-starter.
>>
>> If might prove an okay experience, if the "outdated" part is within
>> certain bounds. Like if the current buffer refreshes at 30/60fps, and
>> the mode-line at 10/20fps, for example.
> 
> That's not what was proposed.  The proposal was to do a
> "quick-and-dirty" layout first, and use the results if the "accurate"
> layout takes too long.

I think I've also commented on the potential problems in my first reply 
to that. But since it's just an abstract suggestion at this point, we 
can try and see the constructive aspects of it.

> The quick-and-dirty layout can only be based
> on ignoring changes in fonts and disregarding images and stuff like
> that, so the screen lines will be wrong, and you will see them jump
> and/or rearrange after some delay, which could be some seconds.  Some
> browsers do that when they need to download images, and it always jars
> me when that happens.  Usually, I also lose the place where I was
> reading.

Some details on what the browsers really do:

- When the images are not downloaded yet, it's possible that the browser 
doesn't know how to lay out the page yet, and the time to download sad 
images is not guaranteed. So what you're describing is really the 
best-effort approach, which has been most noticeable in the past, over 
slow network connections (or in countries/locations with poor network 
coverage even nowadays). Either way, it's a best practice for the web 
page authors to include the dimensions of the images in the <img> tags 
-- when they are present, the layout can be done correctly even while 
the images are not downloaded yet.

- If a web page is complex enough, and I scroll it down after opening 
very quickly, it can happen so Firefox doesn't manage to render the page 
quickly enough, perhaps due to other load. When that happens, it shows 
the bottom of the page in a sort of blurry haze (which is replaced as 
soon as it's ready). Overall, it feels good enough usability-wise, 
because it doesn't look jerky or "blinky" or etc. And of course it's a 
rare occasion overall.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Dmitry Gutov <dmitry@gutov.dev> writes:

> - If a web page is complex enough, and I scroll it down after opening 
> very quickly, it can happen so Firefox doesn't manage to render the page 
> quickly enough, perhaps due to other load. When that happens, it shows 
> the bottom of the page in a sort of blurry haze (which is replaced as 
> soon as it's ready). Overall, it feels good enough usability-wise, 
> because it doesn't look jerky or "blinky" or etc. And of course it's a 
> rare occasion overall.

I think that partial rendering of a buffer might be actually tricky.
In particular, when user tries to "move into" the not-yet-rendered part
of the buffer. Many point motion commands re-use parts of the display
code to determine where to move the point to. I am not sure how to
address such situation without blocking Emacs.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> If might prove an okay experience, if the "outdated" part is within 
>> certain bounds. Like if the current buffer refreshes at 30/60fps, and 
>> the mode-line at 10/20fps, for example.
>
> That's not what was proposed.  The proposal was to do a
> "quick-and-dirty" layout first, and use the results if the "accurate"
> layout takes too long.  The quick-and-dirty layout can only be based
> on ignoring changes in fonts and disregarding images and stuff like
> that, so the screen lines will be wrong, and you will see them jump
> and/or rearrange after some delay, which could be some seconds.  Some
> browsers do that when they need to download images, and it always jars
> me when that happens.  Usually, I also lose the place where I was
> reading.
>
> I hope Emacs will never behave like that.

I can see how this can be annoying.
What about "quick-and-dirty" layout being "empty" window that is also
blocked for user interaction. (I am assuming that redisplay is
asynchronous here). Then, the user will be blocked from interacting
with a single problematic window being redisplayed, but could still work
with other windows/buffers without waiting for Emacs to "unhang".

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>, luangruo@yahoo.com, acm@muc.de,
>  incal@dataswamp.org, emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 10:28:23 +0000
> 
> What about "quick-and-dirty" layout being "empty" window that is also
> blocked for user interaction. (I am assuming that redisplay is
> asynchronous here). Then, the user will be blocked from interacting
> with a single problematic window being redisplayed, but could still work
> with other windows/buffers without waiting for Emacs to "unhang".

You will need to describe in more detail what you mean by "user
interaction with a window".  In Emacs, users don't interact with
windows, they interact with Emacs.  Did you mean that such a window
should not be the selected one?  If so, does it mean Emacs should make
another window to be the selected one if the selected one takes too
long to display?  If that is what you mean, I'm not sure users will
like that.  For example, touch typists could be tricked into modifying
a completely different buffer in this way.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> What about "quick-and-dirty" layout being "empty" window that is also
>> blocked for user interaction. (I am assuming that redisplay is
>> asynchronous here). Then, the user will be blocked from interacting
>> with a single problematic window being redisplayed, but could still work
>> with other windows/buffers without waiting for Emacs to "unhang".
>
> You will need to describe in more detail what you mean by "user
> interaction with a window".  In Emacs, users don't interact with
> windows, they interact with Emacs.  Did you mean that such a window
> should not be the selected one?

I meant that such a window can be selected, but attempting to do
anything that interacts with the window should cause user-error while
the window is being processed by redisplay.

I imagine that users will be allowed to, at least, C-x o away from the
window or switch buffer. But not enter text or run arbitrary commands.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 13:06:00 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > You will need to describe in more detail what you mean by "user
> > interaction with a window".  In Emacs, users don't interact with
> > windows, they interact with Emacs.  Did you mean that such a window
> > should not be the selected one?
> 
> I meant that such a window can be selected, but attempting to do
> anything that interacts with the window should cause user-error while
> the window is being processed by redisplay.
> 
> I imagine that users will be allowed to, at least, C-x o away from the
> window or switch buffer. But not enter text or run arbitrary commands.

That is still far from complete.  But in any case, this seems to
require addition of tests to many Emacs primitives, which would check
if the selected window is "kosher", and if so, produce a user-error.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: dmitry@gutov.dev,  yantar92@posteo.net,  acm@muc.de,
>   incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 17:25:07 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then try leaning on C-n or C-p _after_ everything is already
> > fontified.  You will still see that Emacs sometimes cannot keep up,
> > especially if lines are not too short.
> 
> Long lines are, at worst, infrequently encountered in source code,
> aren't they?

_Very_ long (as in: many megabytes) lines are indeed rare.  But what I
said was "not too short", meaning that don't try this with 5-character
or 20-character lines and conclude that all is well.  Source code with
several dozens of characters are still frequent enough.

> And our troubles with long lines are an algorithmic impediment,
> which cannot be ameliorated merely by redisplaying each window on a
> different CPU.

Someone already said that, and I explained why this is false.

> I guess we can only agree to disagree.

It's about time.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 20/09/2023 14:28, Eli Zaretskii wrote:
>> Date: Tue, 19 Sep 2023 23:21:46 +0300
>> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 19/09/2023 20:54, Eli Zaretskii wrote:
>>>
>>> How do you know that parallel redisplay will not solve this?  This can
>>> only be obvious when the detailed design of that is presented.  Until
>>> then, it's anyone's guess.
>>
>> It's not 100%, but seems logical: the problem with long lines is that
>> displaying a single line took non-linear amount of time proportional to
>> its length. To be able to layout it in parallel, we would somehow have
>> to be able to split it into independent pieces (perhaps using some
>> caching mechanism?, IDK). If we are actually able to do that (which
>> seems difficult), a non-parallel redisplay algorithm should happily use
>> the same mid-line checkpoint information to work on smaller pieces of
>> the line, likewise increasing the speed.
> 
> You are considering only the redisplay of that single window which has
> long lines.  But what about the other windows, which perhaps don't
> have long lines?  And what about the main thread of Emacs itself,
> which freezes for as long as redisplay didn't finish, thus preventing
> user interaction and responsiveness in general?  These factors should
> not be neglected.

Splitting an N^3 program in two to run in half the time won't make it 
have adequate performance. When the problem is algorithmic, we'll have 
to fix it as such.

>> But I think we've much improved on the issue by reducing the complexity
>> instead (correct me if I'm wrong here).
> 
> If you mean the long-lines improvements in Emacs 29, then we traded
> off correctness in at least some cases.  It wasn't for free.

I see.

Well, we definitely did that WRT to font-lock (the part which we might 
tweak or roll back in the future). I wasn't sure if that's also what 
happened in the deep of the display engine as well.

>>> Anyway, you asked for evidence that redisplay could benefit from
>>> performance boost, and I gave you some.  Now you for some reason want
>>> to claim that my evidence doesn't convince you, but it still is
>>> evidence, right?
>>
>> I was looking for other examples, to be honest. Like, for example,
>> evidence that redisplay is something that keeps an average Emacs session
>> from running at 60fps. Or even at 30fps. That would make it a bottleneck.
> 
> With enough text properties and overlays, you can definitely see a
> tangible slowdown in frame rate.  E.g., Org developers don't want to
> use text properties because they slow down redisplay, and thus they
> consider them not scalable enough.
> 
>>> You seem to be opposed to attempts to explore ways of making Emacs
>>> less single-threaded than it is today, unless someone provides a
>>> proof, up front, that this will necessarily solve some particular
>>> problem?  Why? what harm could that possibly do, if it succeeds?  And
>>> if it doesn't succeed, at least we will have learned something
>>> probably useful for the future.
>>
>> It is my experience that good benchmarking often helps with changing the
>> design as well. Or coming up with a new one (there are a lot of options
>> for how one could proceed).
> 
> On the high-level design level, it should be clear without any need
> for benchmarking that separating redisplay from the main thread could
> bring benefits.  Isn't that what every GUI application out there does?
> So trying to think about that is always a good thing, IMO.

Okay, here's a question that I should have started with: what kind of 
parallelism are we discussing?

A) Initially I figured it's the ability to render a single window using 
multiple threads of execution. It's the one my mind immediately went to, 
but might the hardest of the bunch because it would require us to 
parallelize the display algorithm for a single buffer.

B) Being able to render different windows/buffers fully in parallel. I 
think that requires us to implement concurrent Lisp threads, at the very 
least. That's likely to drag down the single-thread performance (though 
it's hard to predict by how much).

C) Having only a separate (not Lisp) thread for GUI might help with 
responsiveness somewhat (by throttling certain expensive windows down, 
for example), but overall it might not improve by much. Though I suppose 
it could render in parallel those buffers which don't need to run Lisp 
to display? Anyway it would be synchronized to the one Lisp interpreter.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Wed, 20 Sep 2023 22:22:53 +0300
> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> Okay, here's a question that I should have started with: what kind of 
> parallelism are we discussing?

Not clear yet.

> A) Initially I figured it's the ability to render a single window using 
> multiple threads of execution. It's the one my mind immediately went to, 
> but might the hardest of the bunch because it would require us to 
> parallelize the display algorithm for a single buffer.
> 
> B) Being able to render different windows/buffers fully in parallel. I 
> think that requires us to implement concurrent Lisp threads, at the very 
> least. That's likely to drag down the single-thread performance (though 
> it's hard to predict by how much).
> 
> C) Having only a separate (not Lisp) thread for GUI might help with 
> responsiveness somewhat (by throttling certain expensive windows down, 
> for example), but overall it might not improve by much. Though I suppose 
> it could render in parallel those buffers which don't need to run Lisp 
> to display? Anyway it would be synchronized to the one Lisp interpreter.

All of the above need some fundamental issues to be resolved to enable
any of them.  I think at this point it is best to try to look for
solutions for those fundamental issues, and only after that see which
of the above are possible and worth our while.

Re: Emacs design and architecture. How about copy-on-write?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's not 100%, but seems logical: the problem with long lines is that 
  > displaying a single line took non-linear amount of time proportional to 
  > its length. To be able to layout it in parallel, we would somehow have 
  > to be able to split it into independent pieces (perhaps using some 
  > caching mechanism?, IDK).

The usual source of very long lines, in my experience, is when people
write text and let their lines run on and on.  Many people's email
user agents encourage that.

I usually use fill-region to fix the problem, if therr are blank lines
betwee these single-line paragaphs.

Peraps we can designheeuristics for finding tre paragaph bondaries and
insert blak ines between them.  Then we could full the text
autoimatically in more cases.  This would not only fix the slowness,'it
would als make the text easier to read.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> (I suggest this because I feel that xdisp is a rabbit hole we may sink
>> in instead of doing something more productive)
>
> I'm actually of the opposite opinion.  I think trying to parallelize
> redisplay is a lower-hanging fruit, and if successful, it could bring
> non-trivial gains to Emacs even if the rest remains single-threaded.

Without having Elisp async threads, what you suggest is only possible
for the non-Elisp parts of the redisplay.

May you list which C-level parts of the redisplay are known to be slow?

AFAIU, long-lines performance is largely problematic when running Elisp.
(Also, isn't it solved already?)

Another one you mentioned is displaying many frames. But isn't it
optimized by the part of the code that delays redisplay of frames that
are not visible? Or does Stefan (or anyone) tend to have so many
_visible_ frames?

>> I am particularly worried about scenarios when window geometry changes
>> by asynchronous threads. Or, say, face definitions. Imagine that it
>> happens at point when we are already drawing the now obsolete geometry
>> onto glass?
>
> xdisp.c has solutions for these two (and other similar) situations.
> The problems have nothing to do with parallelism, they happen today
> because we call Lisp at certain places in the display engine.

Yes, but now we know exactly in which segments of the code the geometry
might change. And check for it in strategic places.

In the case of asynchronous threads, the geometry may change any time
(when another threads happens to run something that affects geometry),
which is not accounted for by the current code.

>> > We tried that already, with the existing Lisp threads.  One reason why
>> > those are almost never used is that the display issue was left
>> > unresolved.
>> 
>> That might be one reason, but not the only reason. And certainly not the
>> most important reason for the use cases where I did try to use threads
>> myself.
>
> There's more than one reason, that's true.  But it doesn't change the
> fact that all those problems have to be resolved before we have
> reasonably usable threads.

I disagree. Yes, _all_ the problems are to be solved eventually. But
solving _some_ of the problems will already be an improvement. IMHO, it
is not all-or-nothing; we can split the problem into several
sub-problems and solve them one by one.

>> As we discussed previously, a number of Elisp programs can benefit from
>> async threads even when redisplay is not asynchronous - network queries
>> (gnus), text parsing (org-mode, slow LSP server communication, xml
>> processing).
>
> You forget that almost every Lisp program in Emacs either displays
> something or affects stuff that affects redisplay.  It's easy to
> forget, I know, because in Emacs redisplay "just happens" by some
> magic.

I don't forget that. However, we do not have to make each and every
thread _fully_ asynchronous. It is often enough to process the
performance bottlenecks asynchronously. After that processing, we can go
ahead and present the results synchronously to user.

Let me provide a more concrete example.
Consider something as simple as grepping across project files.
The process of grepping involves: (1) opening and searching many files;
(2) presenting results to the user. Stage (1) does not really involve
user interaction or redisplay and can greatly benefit from asynchronous
threads - we can simply search multiple files at the same time. Then,
even if stage (2) has to be synchronous, it does not matter - stage (1)
is what takes most of the time and makes the user wait.

>> To be clear, it would indeed be nice to have async redisplay.
>
> I'm not talking about async redisplay, I'm talking about the ability
> of non-main threads to display something or do something that would
> cause redisplay change the stuff on the glass.

Then, how will it be possible without first having async Elisp threads?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 09:47:46 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> (I suggest this because I feel that xdisp is a rabbit hole we may sink
> >> in instead of doing something more productive)
> >
> > I'm actually of the opposite opinion.  I think trying to parallelize
> > redisplay is a lower-hanging fruit, and if successful, it could bring
> > non-trivial gains to Emacs even if the rest remains single-threaded.
> 
> Without having Elisp async threads, what you suggest is only possible
> for the non-Elisp parts of the redisplay.

The display engine is written in C, not in Lisp.  What to do with the
few calls into Lisp we do in redisplay is an open question, but being
able to call Lisp from redisplay and being able to run several async
Lisp threads concurrently are two very different problems (although if
someone comes up with a solution for the latter, that would probably
solve the former as well).

> May you list which C-level parts of the redisplay are known to be slow?

I already did, in response to Dmitry's identical question.

> AFAIU, long-lines performance is largely problematic when running Elisp.

No, the slow parts are pure C.

> (Also, isn't it solved already?)

Not solved, but "bypassed".  The solution does have its disadvantages,
although we found them to be a much lesser evil than the original
problem.

> Another one you mentioned is displaying many frames. But isn't it
> optimized by the part of the code that delays redisplay of frames that
> are not visible? Or does Stefan (or anyone) tend to have so many
> _visible_ frames?

The latter.

> >> I am particularly worried about scenarios when window geometry changes
> >> by asynchronous threads. Or, say, face definitions. Imagine that it
> >> happens at point when we are already drawing the now obsolete geometry
> >> onto glass?
> >
> > xdisp.c has solutions for these two (and other similar) situations.
> > The problems have nothing to do with parallelism, they happen today
> > because we call Lisp at certain places in the display engine.
> 
> Yes, but now we know exactly in which segments of the code the geometry
> might change. And check for it in strategic places.

No, we don't.  We check the indications of this when a window or a
frame has been processed, that's all.

> In the case of asynchronous threads, the geometry may change any time
> (when another threads happens to run something that affects geometry),
> which is not accounted for by the current code.

The geometry cannot change by itself, because redisplay doesn't change
it.  It just detects the situation where the geometry must change, and
computes the variables that would determine the new geometry.  Then it
triggers resizing and recomputation of the geometry before retrying
redisplay.  The trigger to resize can be delivered to all redisplaying
threads, which will make them stop what they are doing, and then retry
after resize.

> >> > We tried that already, with the existing Lisp threads.  One reason why
> >> > those are almost never used is that the display issue was left
> >> > unresolved.
> >> 
> >> That might be one reason, but not the only reason. And certainly not the
> >> most important reason for the use cases where I did try to use threads
> >> myself.
> >
> > There's more than one reason, that's true.  But it doesn't change the
> > fact that all those problems have to be resolved before we have
> > reasonably usable threads.
> 
> I disagree. Yes, _all_ the problems are to be solved eventually. But
> solving _some_ of the problems will already be an improvement. IMHO, it
> is not all-or-nothing; we can split the problem into several
> sub-problems and solve them one by one.

You've lost context, and effectively changed the subject, so now this
sub-thread is not useful anymore.  To recap, I was trying to explain
why the display aspects of any concurrency cannot be disregarded if we
want to have a workable solution that can be used in practice.

> >> As we discussed previously, a number of Elisp programs can benefit from
> >> async threads even when redisplay is not asynchronous - network queries
> >> (gnus), text parsing (org-mode, slow LSP server communication, xml
> >> processing).
> >
> > You forget that almost every Lisp program in Emacs either displays
> > something or affects stuff that affects redisplay.  It's easy to
> > forget, I know, because in Emacs redisplay "just happens" by some
> > magic.
> 
> I don't forget that. However, we do not have to make each and every
> thread _fully_ asynchronous. It is often enough to process the
> performance bottlenecks asynchronously. After that processing, we can go
> ahead and present the results synchronously to user.

This assumes the display is always in the end.  But that is a false
assumption in Emacs: there are progress messages, there are prompts
and requests for confirmation, there are display updates to show the
stuff processed so far, etc.

> Let me provide a more concrete example.
> Consider something as simple as grepping across project files.
> The process of grepping involves: (1) opening and searching many files;
> (2) presenting results to the user. Stage (1) does not really involve
> user interaction or redisplay and can greatly benefit from asynchronous
> threads - we can simply search multiple files at the same time. Then,
> even if stage (2) has to be synchronous, it does not matter - stage (1)
> is what takes most of the time and makes the user wait.

You describe an Emacs without async subprocesses -- this is how it
works on MSDOS, for example.  On more capable systems we display in
parallel with running Grep.

> > I'm not talking about async redisplay, I'm talking about the ability
> > of non-main threads to display something or do something that would
> > cause redisplay change the stuff on the glass.
> 
> Then, how will it be possible without first having async Elisp threads?

Async Lisp threads is a much harder problem to solve.  If we solve it,
this one will also be solved; but the opposite is not true.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> AFAIU, long-lines performance is largely problematic when running Elisp.
>
> No, the slow parts are pure C.
>
>> (Also, isn't it solved already?)
>
> Not solved, but "bypassed".  The solution does have its disadvantages,
> although we found them to be a much lesser evil than the original
> problem.

May you point to the xdisp.c function that is being slow?

>> Another one you mentioned is displaying many frames. But isn't it
>> optimized by the part of the code that delays redisplay of frames that
>> are not visible? Or does Stefan (or anyone) tend to have so many
>> _visible_ frames?
>
> The latter.

Ok. But is it possible to redisplay the whole frame without ever calling Elisp?

>> > There's more than one reason, that's true.  But it doesn't change the
>> > fact that all those problems have to be resolved before we have
>> > reasonably usable threads.
>> 
>> I disagree. Yes, _all_ the problems are to be solved eventually. But
>> solving _some_ of the problems will already be an improvement. IMHO, it
>> is not all-or-nothing; we can split the problem into several
>> sub-problems and solve them one by one.
>
> You've lost context, and effectively changed the subject, so now this
> sub-thread is not useful anymore.  To recap, I was trying to explain
> why the display aspects of any concurrency cannot be disregarded if we
> want to have a workable solution that can be used in practice.

I believe that I did not lose the context. Unless I misunderstand
something.

What I suggested is to defer the redisplay concurrency and leave it
synchronous (interlocked) for now. Concurrency of Elisp that does not
trigger redisplay can also be useful.

It looks to me that we just disagree about the relative importance of
redisplay concurrency.

>> I don't forget that. However, we do not have to make each and every
>> thread _fully_ asynchronous. It is often enough to process the
>> performance bottlenecks asynchronously. After that processing, we can go
>> ahead and present the results synchronously to user.
>
> This assumes the display is always in the end.  But that is a false
> assumption in Emacs: there are progress messages, there are prompts
> and requests for confirmation, there are display updates to show the
> stuff processed so far, etc.

Again, not always. I do not think that the _initial_ goal should be
making every aspect of Elisp working asynchronously without
interlocking. We certainly want all Elisp to work in threads, possibly
with global lock. But if we get at least some part of useful Elisp to
run truly concurrently, it will already be a win.

>> Let me provide a more concrete example.
>> Consider something as simple as grepping across project files.
>> The process of grepping involves: (1) opening and searching many files;
>> (2) presenting results to the user. Stage (1) does not really involve
>> user interaction or redisplay and can greatly benefit from asynchronous
>> threads - we can simply search multiple files at the same time. Then,
>> even if stage (2) has to be synchronous, it does not matter - stage (1)
>> is what takes most of the time and makes the user wait.
>
> You describe an Emacs without async subprocesses -- this is how it
> works on MSDOS, for example.  On more capable systems we display in
> parallel with running Grep.

I think you misunderstood. I did not mean calling GNU grep. I meant
calling M-x grep - Elisp code searching across many Emacs buffers. It
would be useful if such code could run concurrently.

Another example is org-agenda that is running a large number of regexp
search across many buffers. Some real-world users search across as many
as 500 buffers, with results accumulated into agenda. It would certainly
be useful to run that regexp search concurrently, in multiple buffers at
once. Or maybe even on several regions of a single, large buffer.

>> > I'm not talking about async redisplay, I'm talking about the ability
>> > of non-main threads to display something or do something that would
>> > cause redisplay change the stuff on the glass.
>> 
>> Then, how will it be possible without first having async Elisp threads?
>
> Async Lisp threads is a much harder problem to solve.  If we solve it,
> this one will also be solved; but the opposite is not true.

I agree here. But do note that the interest (and some trial code by Po
Lu) so far has been focused on Elisp threads.

The problem with display code, even if it is easier to transform it to
concurrent, is that understanding display code is by itself is a
time-consuming task. Making changes in it (even small changes) is no
less trivial (I am referring to bug#64596).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 10:29:48 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> AFAIU, long-lines performance is largely problematic when running Elisp.
> >
> > No, the slow parts are pure C.
> >
> >> (Also, isn't it solved already?)
> >
> > Not solved, but "bypassed".  The solution does have its disadvantages,
> > although we found them to be a much lesser evil than the original
> > problem.
> 
> May you point to the xdisp.c function that is being slow?

The basic iteration and layout calculations are slow because (a) they
must examine every character between point A and point B to figure out
the relative position of these points on the screen, and (b) because
this examination can only go forward, not back (so to go back, we must
"jump" very far back, and then go forward).

> >> Another one you mentioned is displaying many frames. But isn't it
> >> optimized by the part of the code that delays redisplay of frames that
> >> are not visible? Or does Stefan (or anyone) tend to have so many
> >> _visible_ frames?
> >
> > The latter.
> 
> Ok. But is it possible to redisplay the whole frame without ever calling Elisp?

Probably not "without ever", but why is it important in this context?
Most of the processing is in C, not in Lisp.

> >> > There's more than one reason, that's true.  But it doesn't change the
> >> > fact that all those problems have to be resolved before we have
> >> > reasonably usable threads.
> >> 
> >> I disagree. Yes, _all_ the problems are to be solved eventually. But
> >> solving _some_ of the problems will already be an improvement. IMHO, it
> >> is not all-or-nothing; we can split the problem into several
> >> sub-problems and solve them one by one.
> >
> > You've lost context, and effectively changed the subject, so now this
> > sub-thread is not useful anymore.  To recap, I was trying to explain
> > why the display aspects of any concurrency cannot be disregarded if we
> > want to have a workable solution that can be used in practice.
> 
> I believe that I did not lose the context. Unless I misunderstand
> something.
> 
> What I suggested is to defer the redisplay concurrency and leave it
> synchronous (interlocked) for now. Concurrency of Elisp that does not
> trigger redisplay can also be useful.
> 
> It looks to me that we just disagree about the relative importance of
> redisplay concurrency.

Yes, exactly.  My point is that without having a reasonable solution
for display, any concurrency will be dead in the water from the
get-go.

> >> I don't forget that. However, we do not have to make each and every
> >> thread _fully_ asynchronous. It is often enough to process the
> >> performance bottlenecks asynchronously. After that processing, we can go
> >> ahead and present the results synchronously to user.
> >
> > This assumes the display is always in the end.  But that is a false
> > assumption in Emacs: there are progress messages, there are prompts
> > and requests for confirmation, there are display updates to show the
> > stuff processed so far, etc.
> 
> Again, not always. I do not think that the _initial_ goal should be
> making every aspect of Elisp working asynchronously without
> interlocking. We certainly want all Elisp to work in threads, possibly
> with global lock. But if we get at least some part of useful Elisp to
> run truly concurrently, it will already be a win.

Not if we want to run the existing Lisp programs in threads with
minimal or no changes.  Because Lisp programs we have now require
display and user interaction all the time, whether by themselves or by
lower-level subroutines they call.

> >> Let me provide a more concrete example.
> >> Consider something as simple as grepping across project files.
> >> The process of grepping involves: (1) opening and searching many files;
> >> (2) presenting results to the user. Stage (1) does not really involve
> >> user interaction or redisplay and can greatly benefit from asynchronous
> >> threads - we can simply search multiple files at the same time. Then,
> >> even if stage (2) has to be synchronous, it does not matter - stage (1)
> >> is what takes most of the time and makes the user wait.
> >
> > You describe an Emacs without async subprocesses -- this is how it
> > works on MSDOS, for example.  On more capable systems we display in
> > parallel with running Grep.
> 
> I think you misunderstood. I did not mean calling GNU grep. I meant
> calling M-x grep - Elisp code searching across many Emacs buffers. It
> would be useful if such code could run concurrently.

It will be much more useful if it could also show the hits as it runs,
instead of requiring the user to wait till it finishes.

> Another example is org-agenda that is running a large number of regexp
> search across many buffers. Some real-world users search across as many
> as 500 buffers, with results accumulated into agenda. It would certainly
> be useful to run that regexp search concurrently, in multiple buffers at
> once. Or maybe even on several regions of a single, large buffer.

And you never want to show the accumulated agenda as it is
accumulated?

> >> > I'm not talking about async redisplay, I'm talking about the ability
> >> > of non-main threads to display something or do something that would
> >> > cause redisplay change the stuff on the glass.
> >> 
> >> Then, how will it be possible without first having async Elisp threads?
> >
> > Async Lisp threads is a much harder problem to solve.  If we solve it,
> > this one will also be solved; but the opposite is not true.
> 
> I agree here. But do note that the interest (and some trial code by Po
> Lu) so far has been focused on Elisp threads.

I'm quite sure people who disregard the display when they are talking
about concurrent Lisp threads are going to repeat the same mistake we
made with the existing Lisp threads.  We must learn from past
experience if we want to succeed.

> The problem with display code, even if it is easier to transform it to
> concurrent, is that understanding display code is by itself is a
> time-consuming task. Making changes in it (even small changes) is no
> less trivial (I am referring to bug#64596).

That is true about any non-trivial part of Emacs internals.  At least
with the display code we have some expertise and experience on board,
something that is not always true elsewhere in Emacs.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> May you point to the xdisp.c function that is being slow?
>
> The basic iteration and layout calculations are slow because (a) they
> must examine every character between point A and point B to figure out
> the relative position of these points on the screen, and (b) because
> this examination can only go forward, not back (so to go back, we must
> "jump" very far back, and then go forward).

In other words, processing a single window is being slow.
The question now is whether processing the whole window can be split
into multiple independent chunks. If it can, such processing can run in
parallel.

>> Ok. But is it possible to redisplay the whole frame without ever calling Elisp?
>
> Probably not "without ever", but why is it important in this context?
> Most of the processing is in C, not in Lisp.

My point is that any time we have to run Lisp during redisplay, without
async Elisp threads, we have to acquire global lock first, which will
degrade concurrency. I am not sure how bad the degradation will be.

>> Again, not always. I do not think that the _initial_ goal should be
>> making every aspect of Elisp working asynchronously without
>> interlocking. We certainly want all Elisp to work in threads, possibly
>> with global lock. But if we get at least some part of useful Elisp to
>> run truly concurrently, it will already be a win.
>
> Not if we want to run the existing Lisp programs in threads with
> minimal or no changes.  Because Lisp programs we have now require
> display and user interaction all the time, whether by themselves or by
> lower-level subroutines they call.

Sure. _existing_. What I have in mind is programs specifically written
to take advantage of concurrency. Just to address common performance
bottlenecks.

>> I think you misunderstood. I did not mean calling GNU grep. I meant
>> calling M-x grep - Elisp code searching across many Emacs buffers. It
>> would be useful if such code could run concurrently.
>
> It will be much more useful if it could also show the hits as it runs,
> instead of requiring the user to wait till it finishes.

Maybe. But reducing the overall waiting time at the cost of not seeing
the progress is an OK compromise, IMHO.

>> Another example is org-agenda that is running a large number of regexp
>> search across many buffers. Some real-world users search across as many
>> as 500 buffers, with results accumulated into agenda. It would certainly
>> be useful to run that regexp search concurrently, in multiple buffers at
>> once. Or maybe even on several regions of a single, large buffer.
>
> And you never want to show the accumulated agenda as it is
> accumulated?

Yes. Because part of the fontification is performed at the very end,
when everything is accumulated.

>> I agree here. But do note that the interest (and some trial code by Po
>> Lu) so far has been focused on Elisp threads.
>
> I'm quite sure people who disregard the display when they are talking
> about concurrent Lisp threads are going to repeat the same mistake we
> made with the existing Lisp threads.  We must learn from past
> experience if we want to succeed.

What about addressing the existing problems with cooperating Lisp
threads then?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Fri, 22 Sep 2023 10:48:46 +0000
> 
> >> Again, not always. I do not think that the _initial_ goal should be
> >> making every aspect of Elisp working asynchronously without
> >> interlocking. We certainly want all Elisp to work in threads, possibly
> >> with global lock. But if we get at least some part of useful Elisp to
> >> run truly concurrently, it will already be a win.
> >
> > Not if we want to run the existing Lisp programs in threads with
> > minimal or no changes.  Because Lisp programs we have now require
> > display and user interaction all the time, whether by themselves or by
> > lower-level subroutines they call.
> 
> Sure. _existing_. What I have in mind is programs specifically written
> to take advantage of concurrency. Just to address common performance
> bottlenecks.

If we are talking about an Emacs where programs meant for threads will
need to be written from scratch using special protocols and
primitives, then all bets are off, and I'm not sure everything we
discussed at such a great length lately is at all useful or even
relevant.  The idea was to allow existing Lisp programs run from
threads with little or no changes, by just starting a thread which
runs a function that is already written and debugged when running in
the (single) main thread.  If this is not what you have in mind, try
first to see if users will be likely to switch to such an Emacs or use
such threads, when they know they will need to drop everything and
start from scratch.  Who will want to write a "multithreaded Gnus"
starting from scratch?

> >> I think you misunderstood. I did not mean calling GNU grep. I meant
> >> calling M-x grep - Elisp code searching across many Emacs buffers. It
> >> would be useful if such code could run concurrently.
> >
> > It will be much more useful if it could also show the hits as it runs,
> > instead of requiring the user to wait till it finishes.
> 
> Maybe. But reducing the overall waiting time at the cost of not seeing
> the progress is an OK compromise, IMHO.

If the time is more than, say, a second or two, then no, such a
compromise will not be liked.  At least for Grep-style searches and
other compile-like commands.

> > I'm quite sure people who disregard the display when they are talking
> > about concurrent Lisp threads are going to repeat the same mistake we
> > made with the existing Lisp threads.  We must learn from past
> > experience if we want to succeed.
> 
> What about addressing the existing problems with cooperating Lisp
> threads then?

What about it?  Patches are welcome, of course.  Last time we
discussed these issues, we were unable to find good ideas for solving
them.  Maybe we should try discussing them again.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> If we are talking about an Emacs where programs meant for threads will
> need to be written from scratch using special protocols and
> primitives, then all bets are off, and I'm not sure everything we
> discussed at such a great length lately is at all useful or even
> relevant.  The idea was to allow existing Lisp programs run from
> threads with little or no changes, by just starting a thread which
> runs a function that is already written and debugged when running in
> the (single) main thread.  If this is not what you have in mind, try
> first to see if users will be likely to switch to such an Emacs or use
> such threads, when they know they will need to drop everything and
> start from scratch.  Who will want to write a "multithreaded Gnus"
> starting from scratch?

Let me clarify.
I am not saying that existing Elisp code should not be allowed to run
from threads. It should.

However, I think that it can be acceptable to leave certain things
interlocked - if async thread is querying, for example, user input or
redisplay, acquire a global (or input/redisplay) lock, and run that
portion of the thread synchronously.

That way, we can still allow useful Elisp to run concurrently, when
running CPU-intensive computation. These CPU-intensive parts will need
to be rewritten with concurrency in mind. But only parts - the rest of
the code could be left unchanged without breaking things.

Later, we may also figure out more tricky parts related to async
input/redisplay. 

>> Maybe. But reducing the overall waiting time at the cost of not seeing
>> the progress is an OK compromise, IMHO.
>
> If the time is more than, say, a second or two, then no, such a
> compromise will not be liked.  At least for Grep-style searches and
> other compile-like commands.

Sure. I am talking about M-x grep on large projects, when it takes tens
of seconds to finish. Same for org-agenda - it is not uncommon when
org-agenda searches last for a minute when search criteria is complex.

>> What about addressing the existing problems with cooperating Lisp
>> threads then?
>
> What about it?  Patches are welcome, of course.  Last time we
> discussed these issues, we were unable to find good ideas for solving
> them.  Maybe we should try discussing them again.

Are you referring to input discussion? Something else? I think that it
could be useful to document problems to be solved in etc/TODO.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Sat, 23 Sep 2023 11:07:54 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If we are talking about an Emacs where programs meant for threads will
> > need to be written from scratch using special protocols and
> > primitives, then all bets are off, and I'm not sure everything we
> > discussed at such a great length lately is at all useful or even
> > relevant.  The idea was to allow existing Lisp programs run from
> > threads with little or no changes, by just starting a thread which
> > runs a function that is already written and debugged when running in
> > the (single) main thread.  If this is not what you have in mind, try
> > first to see if users will be likely to switch to such an Emacs or use
> > such threads, when they know they will need to drop everything and
> > start from scratch.  Who will want to write a "multithreaded Gnus"
> > starting from scratch?
> 
> Let me clarify.
> I am not saying that existing Elisp code should not be allowed to run
> from threads. It should.
> 
> However, I think that it can be acceptable to leave certain things
> interlocked - if async thread is querying, for example, user input or
> redisplay, acquire a global (or input/redisplay) lock, and run that
> portion of the thread synchronously.

If this is done under the hood by the infrastructure, and the Lisp
code doesn't have to be modified, this is, of course, fine.

If the Lisp program that wants to interact with the user will need to
do something special when it runs from a non-main thread, that is
worse, but maybe still feasible.

But what can we do about programs that call general-purpose
subroutines, and those subroutines decide to prompt the user?  And
this is the problem which bothers me, because we are used to call many
low-level subroutines without thinking about what that subroutine
will do and whether it will want to prompt the user.

As a trivial example, any function that modifies a file-visiting
buffer could prompt the user if the file was meanwhile modified on
disk.  This prompt is completely out of control of any Lisp program
which does something that modifies buffer text.  How do we handle
these cases? their name is a legion.

> >> What about addressing the existing problems with cooperating Lisp
> >> threads then?
> >
> > What about it?  Patches are welcome, of course.  Last time we
> > discussed these issues, we were unable to find good ideas for solving
> > them.  Maybe we should try discussing them again.
> 
> Are you referring to input discussion? Something else?

I don't know what you mean by "input discussion", but I did already
point you to the relevant discussion, so maybe that is it.

> I think that it could be useful to document problems to be solved in
> etc/TODO.

Feel free.  I don't think I can write something useful for that file,
so I didn't.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 23/09/2023 14:23, Eli Zaretskii wrote:
> As a trivial example, any function that modifies a file-visiting
> buffer could prompt the user if the file was meanwhile modified on
> disk.  This prompt is completely out of control of any Lisp program
> which does something that modifies buffer text.  How do we handle
> these cases? their name is a legion.

Any function that prompts the user is not supposed to be fast. So it 
might as well acquire any number of global locks to do that.

So it's not an issue for concurrency - though it might become a 
stumbling block for the "concurrent redisplay" sub-project.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Sat, 23 Sep 2023 15:53:11 +0300
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 23/09/2023 14:23, Eli Zaretskii wrote:
> > As a trivial example, any function that modifies a file-visiting
> > buffer could prompt the user if the file was meanwhile modified on
> > disk.  This prompt is completely out of control of any Lisp program
> > which does something that modifies buffer text.  How do we handle
> > these cases? their name is a legion.
> 
> Any function that prompts the user is not supposed to be fast. So it 
> might as well acquire any number of global locks to do that.

That's not my point.  My point is that if we say that changes to adapt
existing code to threads are acceptable, we will have to make those
changes all over our infrastructure, otherwise programs written for
threads will not ready for threads 100%.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 23/09/2023 16:01, Eli Zaretskii wrote:
>> Date: Sat, 23 Sep 2023 15:53:11 +0300
>> Cc:acm@muc.de,incal@dataswamp.org,emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 23/09/2023 14:23, Eli Zaretskii wrote:
>>> As a trivial example, any function that modifies a file-visiting
>>> buffer could prompt the user if the file was meanwhile modified on
>>> disk.  This prompt is completely out of control of any Lisp program
>>> which does something that modifies buffer text.  How do we handle
>>> these cases? their name is a legion.
>> Any function that prompts the user is not supposed to be fast. So it
>> might as well acquire any number of global locks to do that.
> That's not my point.  My point is that if we say that changes to adapt
> existing code to threads are acceptable, we will have to make those
> changes all over our infrastructure, otherwise programs written for
> threads will not ready for threads 100%.

I agree: functions like yes-or-no-p will have to, internally in their 
implementation, acquire the "redisplay lock" or whatever it'll be 
called, and do other things to ensure that they work from non-default 
threads too.

This will likely make them a little slower compared to the single-thread 
model, but hopefully not to a degree that's humanly noticeable.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Sat, 23 Sep 2023 16:08:23 +0300
> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 23/09/2023 16:01, Eli Zaretskii wrote:
> >> Date: Sat, 23 Sep 2023 15:53:11 +0300
> >> Cc:acm@muc.de,incal@dataswamp.org,emacs-devel@gnu.org
> >> From: Dmitry Gutov<dmitry@gutov.dev>
> >>
> >> On 23/09/2023 14:23, Eli Zaretskii wrote:
> >>> As a trivial example, any function that modifies a file-visiting
> >>> buffer could prompt the user if the file was meanwhile modified on
> >>> disk.  This prompt is completely out of control of any Lisp program
> >>> which does something that modifies buffer text.  How do we handle
> >>> these cases? their name is a legion.
> >> Any function that prompts the user is not supposed to be fast. So it
> >> might as well acquire any number of global locks to do that.
> > That's not my point.  My point is that if we say that changes to adapt
> > existing code to threads are acceptable, we will have to make those
> > changes all over our infrastructure, otherwise programs written for
> > threads will not ready for threads 100%.
> 
> I agree: functions like yes-or-no-p will have to, internally in their 
> implementation, acquire the "redisplay lock" or whatever it'll be 
> called, and do other things to ensure that they work from non-default 
> threads too.
> 
> This will likely make them a little slower compared to the single-thread 
> model, but hopefully not to a degree that's humanly noticeable.

I'm not bothered by slowness, I'm much more bothered by the magnitude
of the changes this will incur.  I don't even know how to identify all
the places which would need such changes.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm not bothered by slowness, I'm much more bothered by the magnitude
> of the changes this will incur.  I don't even know how to identify all
> the places which would need such changes.

I _hope_ that it will be enough to acquire global lock once redisplay or
user input is requested.

If my optimism does not hold, a safe approach that can be employed is to
mark the functions that are thread-safe explicitly and acquire global
lock by default.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 23/09/2023 16:15, Eli Zaretskii wrote:
>> Date: Sat, 23 Sep 2023 16:08:23 +0300
>> Cc: yantar92@posteo.net, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 23/09/2023 16:01, Eli Zaretskii wrote:
>>>> Date: Sat, 23 Sep 2023 15:53:11 +0300
>>>> Cc:acm@muc.de,incal@dataswamp.org,emacs-devel@gnu.org
>>>> From: Dmitry Gutov<dmitry@gutov.dev>
>>>>
>>>> On 23/09/2023 14:23, Eli Zaretskii wrote:
>>>>> As a trivial example, any function that modifies a file-visiting
>>>>> buffer could prompt the user if the file was meanwhile modified on
>>>>> disk.  This prompt is completely out of control of any Lisp program
>>>>> which does something that modifies buffer text.  How do we handle
>>>>> these cases? their name is a legion.
>>>> Any function that prompts the user is not supposed to be fast. So it
>>>> might as well acquire any number of global locks to do that.
>>> That's not my point.  My point is that if we say that changes to adapt
>>> existing code to threads are acceptable, we will have to make those
>>> changes all over our infrastructure, otherwise programs written for
>>> threads will not ready for threads 100%.
>>
>> I agree: functions like yes-or-no-p will have to, internally in their
>> implementation, acquire the "redisplay lock" or whatever it'll be
>> called, and do other things to ensure that they work from non-default
>> threads too.
>>
>> This will likely make them a little slower compared to the single-thread
>> model, but hopefully not to a degree that's humanly noticeable.
> 
> I'm not bothered by slowness, I'm much more bothered by the magnitude
> of the changes this will incur.  I don't even know how to identify all
> the places which would need such changes.

That would be up to the implementor: do a patch, run with existing code, 
notice problems, patch some more. Etc. The issue is creating a 
reasonably complete (and useable) implementation that at least some 
developers start testing the waters.

Anyway, some concern about performance might be warranted. For example, 
the recent "Python without GIL" project describes a 9% difference in 
single-thread performance, which the author of course tried to make up 
for with new optimizations 
(https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit?pli=1).

I recall that even a 2% (?) difference caused by the 
symbols-with-positions feature was considered a problem not too long ago.

Re: Emacs design and architecture. How about copy-on-write?

[Yuri Khan]

On Sat, 23 Sept 2023 at 20:09, Dmitry Gutov <dmitry@gutov.dev> wrote:

> I agree: functions like yes-or-no-p will have to, internally in their
> implementation, acquire the "redisplay lock" or whatever it'll be
> called, and do other things to ensure that they work from non-default
> threads too.

That’s not the only possible implementation. A function that wants a
confirmation could package up the prompt with a continuation function,
and post that as a message to the main thread. The main thread would
pump its message queue, display the prompt, and schedule the
continuation (with the prompt result) to a worker thread.

That would require the code using prompt functions to be transformed
to continuation-passing style though, either explicitly in code, or
automatically by the lisp reader.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 23/09/2023 17:23, Yuri Khan wrote:
> On Sat, 23 Sept 2023 at 20:09, Dmitry Gutov<dmitry@gutov.dev>  wrote:
> 
>> I agree: functions like yes-or-no-p will have to, internally in their
>> implementation, acquire the "redisplay lock" or whatever it'll be
>> called, and do other things to ensure that they work from non-default
>> threads too.
> That’s not the only possible implementation. A function that wants a
> confirmation could package up the prompt with a continuation function,
> and post that as a message to the main thread. The main thread would
> pump its message queue, display the prompt, and schedule the
> continuation (with the prompt result) to a worker thread.
> 
> That would require the code using prompt functions to be transformed
> to continuation-passing style though, either explicitly in code, or
> automatically by the lisp reader.

Yes, that's the other option which would make a lot of code require 
changes to be usable in threaded environment. As I understand, people 
would like to avoid that.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm confused: suppose one thread modifies scroll-margin -- does that
> affect the (global) redisplay?  If it does, how will this "solve" the
> problem?  If it doesn't affect redisplay, how _can_ a thread change
> scroll-margin in order to affect redisplay?

It can't.  Such adjustments must be performed from the main thread, as
in all other modern GUI systems.

In a prototype that I've been entertaining, this problem is simply left
unsolved.  Redisplay et all can only be called from the main thread;
attempting to call functions that are not thread safe (presently all
aside from those defined within alloc.c, that have been properly
interlocked) signal an error.  Objfwd variables consulted by redisplay
are saved around redisplay_internal (and other similar functions), so
that their values as perceived by redisplay can never change while it is
in progress.

This is no different from other GUI systems, where calling functions or
setting variables that affect the UI outside the main thread is strictly
forbidden, and either gives rise to a prompt crash or aborts with a
failure indication.  That multiple threads are incapable of sharing
control over a functioning GUI system is a lesson taught with sweat and
blood, and for it to be lost on us would be an awful shame.

I suggest that people interested in a multi-threaded Emacs answer these
two questions instead:

  - Is there a portable (in POSIX) method for reliably stopping a POSIX
    thread, with all callee-saved registers and register variables
    within leaf functions saved to the stack or some other area in
    memory?

  - How will finalizers, buffer modification hooks, symbol value
    watchers and the like be executed in response to garbage collection,
    buffer modification, or setting symbols in non-main threads?

Instead of belaboring the subject of how different threads will somehow
get away with coinciding calls to redisplay, or asynchronous
modifications to its state.  Because decades of research and experience
from an innumerable number of individuals and organizations have
produced an unequivocal answer: they won't.

Just two cents from someone who actually _HAS_ a multi-processing Emacs
in a quasi-functional state.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> In a prototype that I've been entertaining, this problem is simply left
> unsolved.  Redisplay et all can only be called from the main thread;
> attempting to call functions that are not thread safe (presently all
> aside from those defined within alloc.c

Also, several special forms in eval.c, data.c, fixnum arithmetic, and
the bytecode interpreter of course.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> I suggest that people interested in a multi-threaded Emacs answer these
> two questions instead:
>
>   - Is there a portable (in POSIX) method for reliably stopping a POSIX
>     thread, with all callee-saved registers and register variables
>     within leaf functions saved to the stack or some other area in
>     memory?

This question sounds like a part of a specific implementation detail you
have in mind. May you share more context?

>   - How will finalizers, buffer modification hooks, symbol value
>     watchers and the like be executed in response to garbage collection,
>     buffer modification, or setting symbols in non-main threads?

1. finalizers
   AFAIU, you are referring to GC finalizers.
   But is GC thread-safe?

2. symbol value watchers are triggered by let-bindings among other
   things. At least for let-binding triggers, it makes sense to keep the
   thread-local context for value watchers. I see not why other kind of
   triggers should not keep the modifier thread context either.

   Of course, one might want to process triggers in a separate thread,
   but that may be done by explicitly spawning one.

3. modification hooks are always triggered within a single buffer.
   According to previous discussions, we concluded that it would be
   difficult to allow multiple threads modify the same buffer, so,
   similar to watchers, thread context may be important, and the hooks
   should probably run from within the trigger tread.

> Just two cents from someone who actually _HAS_ a multi-processing Emacs
> in a quasi-functional state.

Is it available anywhere in public repository?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Alan Mackenzie <acm@muc.de>,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Mon, 18 Sep 2023 21:30:17 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm confused: suppose one thread modifies scroll-margin -- does that
> > affect the (global) redisplay?  If it does, how will this "solve" the
> > problem?  If it doesn't affect redisplay, how _can_ a thread change
> > scroll-margin in order to affect redisplay?
> 
> It can't.  Such adjustments must be performed from the main thread, as
> in all other modern GUI systems.

So a non-main thread cannot do anything that affects the display?
Like move point in a buffer that is shown in some window?

> This is no different from other GUI systems, where calling functions or
> setting variables that affect the UI outside the main thread is strictly
> forbidden, and either gives rise to a prompt crash or aborts with a
> failure indication.  That multiple threads are incapable of sharing
> control over a functioning GUI system is a lesson taught with sweat and
> blood, and for it to be lost on us would be an awful shame.

If non-main threads cannot change stuff that affects the display, what
can those threads do? compute the 10ⁿ-th digit of π?

Useful stuff in Emacs almost always affects the display, so if a
non-main threads cannot do that, they will be useless.  Or what am I
missing?

> Just two cents from someone who actually _HAS_ a multi-processing Emacs
> in a quasi-functional state.

IMNSHO, if you left the display issues unsolved, you are no closer to
the solution than we are in Emacs 30.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> So a non-main thread cannot do anything that affects the display?
> Like move point in a buffer that is shown in some window?

This is possible, but moving point won't induce a redisplay of that
window.

> If non-main threads cannot change stuff that affects the display, what
> can those threads do? compute the 10ⁿ-th digit of π?

And other tasks like this, responsible for blocking Emacs.  Since most
of the time, Emacs is not blocked in redisplay, but reading process
output or performing difficult computations.

> Useful stuff in Emacs almost always affects the display, so if a
> non-main threads cannot do that, they will be useless.  Or what am I
> missing?

Consider the case of Gnus or Eglot: it will enable either of them to
fetch news or receive LSP output from a second thread, decode it, and
subsequently transfer it to the main thread for display.  Such expensive
processing is the reason people desire a multi-processing Emacs, because
it will facilitate efficiently running these operations in the
background and possibly on a different CPU.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Tue, 19 Sep 2023 07:41:41 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So a non-main thread cannot do anything that affects the display?
> > Like move point in a buffer that is shown in some window?
> 
> This is possible, but moving point won't induce a redisplay of that
> window.

So if point moves off the window, we will have a window that doesn't
show point?

> > If non-main threads cannot change stuff that affects the display, what
> > can those threads do? compute the 10ⁿ-th digit of π?
> 
> And other tasks like this, responsible for blocking Emacs.  Since most
> of the time, Emacs is not blocked in redisplay, but reading process
> output or performing difficult computations.

Yes, but almost everything we do in Emacs has its purpose of affecting
display, eventually.  Including process output we read and whatever
computations we do.  Emacs is a real-time display editor, first and
foremost, so this should not be a surprise.  The only notable
exception from this rule is batch-style execution in a script.

> > Useful stuff in Emacs almost always affects the display, so if a
> > non-main threads cannot do that, they will be useless.  Or what am I
> > missing?
> 
> Consider the case of Gnus or Eglot: it will enable either of them to
> fetch news or receive LSP output from a second thread, decode it, and
> subsequently transfer it to the main thread for display.

How do you "transfer it to the main thread for display", exactly?  And
won't you want to display some kind of progress indicator while
fetching? or show an error message if things fail?  Every Lisp program
invokes gazillion low-level subroutines and primitives, and some of
those feel free to ask the user questions, show messages, etc.  Even
process.c shows messages,l and modifying a file-visiting buffer might
ask about supersession locks.  We cannot just forbid display from
non-main threads, unless we are willing to rewrite most of the
application code in Emacs, and many of the primitives as well.  The
only solution that avoids the massive rewrite is to invent mechanisms
that still allow non-main threads to communicate with users.

> Such expensive processing is the reason people desire a
> multi-processing Emacs, because it will facilitate efficiently
> running these operations in the background and possibly on a
> different CPU.

I know the theory.  I'm talking about the details -- the devil is
usually there.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> So if point moves off the window, we will have a window that doesn't
> show point?

We will have a window whose point does not reflect its contents on the
display.  A supervening redisplay within the main thread will give due
consideration to its new value when it does transpire.

> Yes, but almost everything we do in Emacs has its purpose of affecting
> display, eventually.  Including process output we read and whatever
> computations we do.  Emacs is a real-time display editor, first and
> foremost, so this should not be a surprise.  The only notable
> exception from this rule is batch-style execution in a script.

These eventualities must take place within the main thread, that's all.

> How do you "transfer it to the main thread for display", exactly?

By setting a global variable that is subsequently read by the main
thread, modifying unread-command-events, and signaling a condition
variable that induces read_char to return the new value of
unread-command-events.

> And won't you want to display some kind of progress indicator while
> fetching? or show an error message if things fail?  Every Lisp program
> invokes gazillion low-level subroutines and primitives, and some of
> those feel free to ask the user questions, show messages, etc.  Even
> process.c shows messages, and modifying a file-visiting buffer might
> ask about supersession locks.  We cannot just forbid display from
> non-main threads, unless we are willing to rewrite most of the
> application code in Emacs, and many of the primitives as well.  The
> only solution that avoids the massive rewrite is to invent mechanisms
> that still allow non-main threads to communicate with users.

process.c can't be employed from non-main threads, as they are forbidden
from entering wait_reading_process_output.  Likewise for the file-lock
stuff, as it calls read_char.

My idea is that non-main threads will communicate with subprocesses and
read or write files through a different set of primitives, but they have
not been implemented.  The principle difficulty with the existing file
primitives is that they can call thread-unsafe Lisp; file name handlers
provided by TRAMP, for example.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 09:01:56 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So if point moves off the window, we will have a window that doesn't
> > show point?
> 
> We will have a window whose point does not reflect its contents on the
> display.  A supervening redisplay within the main thread will give due
> consideration to its new value when it does transpire.

I don't understand why only the main thread is allowed to do that.
What is special in the main thread wrt redisplay that other threads
are forbidden from doing that?  Would it be okay to have a separate
non-main redisplay thread, for example? if not, why not?

> > Yes, but almost everything we do in Emacs has its purpose of affecting
> > display, eventually.  Including process output we read and whatever
> > computations we do.  Emacs is a real-time display editor, first and
> > foremost, so this should not be a surprise.  The only notable
> > exception from this rule is batch-style execution in a script.
> 
> These eventualities must take place within the main thread, that's all.

They many times happen in the middle of processing, not just
"eventually".

> > How do you "transfer it to the main thread for display", exactly?
> 
> By setting a global variable that is subsequently read by the main
> thread, modifying unread-command-events, and signaling a condition
> variable that induces read_char to return the new value of
> unread-command-events.

How is this different from telling the main thread to stop, and then
doing the display from the thread which triggered that?  IOW, why do
we need to ask the main thread to actually display (and perform
input), as opposed to just get out of the way for a short while?

> > And won't you want to display some kind of progress indicator while
> > fetching? or show an error message if things fail?  Every Lisp program
> > invokes gazillion low-level subroutines and primitives, and some of
> > those feel free to ask the user questions, show messages, etc.  Even
> > process.c shows messages, and modifying a file-visiting buffer might
> > ask about supersession locks.  We cannot just forbid display from
> > non-main threads, unless we are willing to rewrite most of the
> > application code in Emacs, and many of the primitives as well.  The
> > only solution that avoids the massive rewrite is to invent mechanisms
> > that still allow non-main threads to communicate with users.
> 
> process.c can't be employed from non-main threads, as they are forbidden
> from entering wait_reading_process_output.  Likewise for the file-lock
> stuff, as it calls read_char.
> 
> My idea is that non-main threads will communicate with subprocesses and
> read or write files through a different set of primitives, but they have
> not been implemented.

Why do they have to be a different set, not the same set which is made
thread-aware?  E.g., if you don't want a primitive to do something
when it runs in a non-main thread, it is easy to write a condition for
that, and leave the rest of the code intact.

> The principle difficulty with the existing file primitives is that
> they can call thread-unsafe Lisp; file name handlers provided by
> TRAMP, for example.

Tramp should be the least of our problems.  The main problem with
refusing to run Lisp from non-main threads is that Emacs will cease to
be Emacs.  The various Lisp hooks, calls into Lisp from C, and control
on the internals exposed to Lisp are what makes Emacs the environment
that is flexible and powerful to unprecedented degree.  Take that
away, and programs written for such an "Emacs" will be much less
powerful and interesting, and certainly much less extensible.

Re: Emacs design and architecture. How about copy-on-write?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't understand why only the main thread is allowed to do that.
> What is special in the main thread wrt redisplay that other threads
> are forbidden from doing that?  Would it be okay to have a separate
> non-main redisplay thread, for example? if not, why not?

Because toolkits forbid that.  One thread is customarily designated the
``main'' thread when the toolkit is initialized, after which any attempt
to invoke display routines from another thread induces a prompt abort.
This manifests itself most severely on NS and both GTK builds.

> They many times happen in the middle of processing, not just
> "eventually".

Then the text written to the glass will be inconsistent as long as
redisplay transpires while processing is still taking place.  Averting
such situations is not within our purview, but that of authors writing
Lisp code that exploits threads.

> How is this different from telling the main thread to stop, and then
> doing the display from the thread which triggered that?  IOW, why do
> we need to ask the main thread to actually display (and perform
> input), as opposed to just get out of the way for a short while?

Even with the toolkit problem notwithstanding, we would still have to
wait for the main thread to enter a section of code where it becomes
safe to yield to a second thread.  The main thread would then be hung
until any number of other threads complete redisplay, which IMNSHO
eliminates the raison d'etre for running multiple threads concurrently.

And there must be some other reason no other extant programs have
adopted such an approach.

> Why do they have to be a different set, not the same set which is made
> thread-aware?  E.g., if you don't want a primitive to do something
> when it runs in a non-main thread, it is easy to write a condition for
> that, and leave the rest of the code intact.

I'm not completely certain, given that the time I have dedicated to this
has mostly been committed to solving the buffer-local variable problem.

Let's bury this particular hatchet for the time being and revisit it
later, when I have the Lisp interpreter itself in a better shape.  OK?

Thanks.

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 20:13:49 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't understand why only the main thread is allowed to do that.
> > What is special in the main thread wrt redisplay that other threads
> > are forbidden from doing that?  Would it be okay to have a separate
> > non-main redisplay thread, for example? if not, why not?
> 
> Because toolkits forbid that.  One thread is customarily designated the
> ``main'' thread when the toolkit is initialized, after which any attempt
> to invoke display routines from another thread induces a prompt abort.
> This manifests itself most severely on NS and both GTK builds.

You are talking about the last stage of redisplay: delivering stuff to
the glass.  But xdisp.c is not about that, most of it has no
interaction with the toolkit.

> > They many times happen in the middle of processing, not just
> > "eventually".
> 
> Then the text written to the glass will be inconsistent as long as
> redisplay transpires while processing is still taking place.  Averting
> such situations is not within our purview, but that of authors writing
> Lisp code that exploits threads.

So we drop on the floor the gobs of Lisp code written during the last
40 years, and tell the authors to rewrite everything from scratch
under the new regime?

> > How is this different from telling the main thread to stop, and then
> > doing the display from the thread which triggered that?  IOW, why do
> > we need to ask the main thread to actually display (and perform
> > input), as opposed to just get out of the way for a short while?
> 
> Even with the toolkit problem notwithstanding, we would still have to
> wait for the main thread to enter a section of code where it becomes
> safe to yield to a second thread.  The main thread would then be hung
> until any number of other threads complete redisplay, which IMNSHO
> eliminates the raison d'etre for running multiple threads concurrently.

The same will happen if only the main thread can display.  If a
non-main thread wants to prompt the user, it will have to wait until
the main thread becomes available, prompts the user, and returns the
response.

Someone always has to wait when threads need to synchronize.  There's
no way around that.  Doing this in the thread that needs to
display/prompt is easier because the context and relevant variables
don't need to be communicated to another thread.

Re: Emacs design and architecture. How about copy-on-write?

[Dmitry Gutov]

On 20/09/2023 15:13, Po Lu wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>> I don't understand why only the main thread is allowed to do that.
>> What is special in the main thread wrt redisplay that other threads
>> are forbidden from doing that?  Would it be okay to have a separate
>> non-main redisplay thread, for example? if not, why not?
> Because toolkits forbid that.  One thread is customarily designated the
> ``main'' thread when the toolkit is initialized, after which any attempt
> to invoke display routines from another thread induces a prompt abort.
> This manifests itself most severely on NS and both GTK builds.

Could the "main thread" which works on the level of toolkits be/become a 
distinct notion from the "main Lisp thread"?

>> They many times happen in the middle of processing, not just
>> "eventually".
> Then the text written to the glass will be inconsistent as long as
> redisplay transpires while processing is still taking place.  Averting
> such situations is not within our purview, but that of authors writing
> Lisp code that exploits threads.
> 
>> How is this different from telling the main thread to stop, and then
>> doing the display from the thread which triggered that?  IOW, why do
>> we need to ask the main thread to actually display (and perform
>> input), as opposed to just get out of the way for a short while?
> Even with the toolkit problem notwithstanding, we would still have to
> wait for the main thread to enter a section of code where it becomes
> safe to yield to a second thread.  The main thread would then be hung
> until any number of other threads complete redisplay, which IMNSHO
> eliminates the raison d'etre for running multiple threads concurrently.
> 
> And there must be some other reason no other extant programs have
> adopted such an approach.

I would imagine (if we do get concurrent threads) that the UI thread 
might work like this: enumerate the visible buffers, "lock" each of the 
visible ones and then run redisplay on them. What's not-obvious is how 
to fit in the Lisp callbacks that are required in each of those by 
redisplay, especially, if some of the buffers are locked by non-default 
threads.

At the moment (IIUC) this just looks like recursive calls because any 
redisplay is triggered by the main thread. And the above scheme would 
require running Lisp in a buffer which is locked by the thread different 
from the one that triggered redisplay.

Though perhaps redisplay wouldn't be triggered by threads at all - not 
often, anyway. A thread could send a "request for redisplay" (if it 
needs to call something like 'redisplay' or 'posn-at-point'), at which 
point it might get suspended until the "display thread" gets to it. 
Then, if required, the thread runs any additional Lisp which is 
necessary to redisplay the current buffer. This could make those calls 
slower, but only experience would show by how much. If a Lisp thread 
doesn't really ask for redisplay, it could still be suspended by the 
"display thread" to do its thing. Could it run the necessary Lisp code 
on the suspended Lisp thread, though?

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> Date: Wed, 20 Sep 2023 21:50:59 +0300
> Cc: acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> Though perhaps redisplay wouldn't be triggered by threads at all - not 
> often, anyway. A thread could send a "request for redisplay" (if it 
> needs to call something like 'redisplay' or 'posn-at-point'), at which 
> point it might get suspended until the "display thread" gets to it. 

Nitpicking: posn-at-point and similar functions are not "redisplay",
they don't redraw any windows.  They just execute some code which is
also used by redisplay.  So there should be no problem calling
posn-at-point regardless of any UI thread and other threads, as long
as access to buffer text is possible.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> We will have a window whose point does not reflect its contents on the
>> display.  A supervening redisplay within the main thread will give due
>> consideration to its new value when it does transpire.
>
> I don't understand why only the main thread is allowed to do that.
> What is special in the main thread wrt redisplay that other threads
> are forbidden from doing that?  Would it be okay to have a separate
> non-main redisplay thread, for example? if not, why not?

Apart from toolkit considerations, consider two asynchronous threads
requesting to display something in the echo area. We cannot allow one
thread "cancel" displaying its message by another thread - this will
lose one of the messages that should be displayed to user. So, there at
least should be some kind of queue for threads trying to display
something in the same window.

Another scenario is one thread displaying multiline message while
another thread displaying a window adjacent to echo area. Let's say that
the thread redisplaying window started running first, considered its
window geometry, and started calculating the glyph matrix according to
the known number of lines fitting that window height. Then, "message"
thread wants to resize echo area to fit its multiline message. What
should the "window" thread do? Should it cancel its redisplay? Restart
it? Wait for the "message" thread to finish? It is not very clear and
likely depend on the specific scenario at hand.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, acm@muc.de, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 10:08:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> We will have a window whose point does not reflect its contents on the
> >> display.  A supervening redisplay within the main thread will give due
> >> consideration to its new value when it does transpire.
> >
> > I don't understand why only the main thread is allowed to do that.
> > What is special in the main thread wrt redisplay that other threads
> > are forbidden from doing that?  Would it be okay to have a separate
> > non-main redisplay thread, for example? if not, why not?
> 
> Apart from toolkit considerations, consider two asynchronous threads
> requesting to display something in the echo area. We cannot allow one
> thread "cancel" displaying its message by another thread - this will
> lose one of the messages that should be displayed to user. So, there at
> least should be some kind of queue for threads trying to display
> something in the same window.
> 
> Another scenario is one thread displaying multiline message while
> another thread displaying a window adjacent to echo area. Let's say that
> the thread redisplaying window started running first, considered its
> window geometry, and started calculating the glyph matrix according to
> the known number of lines fitting that window height. Then, "message"
> thread wants to resize echo area to fit its multiline message. What
> should the "window" thread do? Should it cancel its redisplay? Restart
> it? Wait for the "message" thread to finish? It is not very clear and
> likely depend on the specific scenario at hand.

We have these situations already: arrange for some timer to display an
echo-area message after several seconds, then type M-x or something
else to enter the minibuffer, and watch how the message is displayed.
We solved this, more-or-less, in Emacs 29.

In any case, having to deal with multiple simultaneous messages is a
far cry from disallowing any non-main thread to display anything.

Re: Emacs design and architecture. How about copy-on-write?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> We have these situations already: arrange for some timer to display an
> echo-area message after several seconds, then type M-x or something
> else to enter the minibuffer, and watch how the message is displayed.
> We solved this, more-or-less, in Emacs 29.

In other words, it is some kind of primitive async queue, right?

> In any case, having to deal with multiple simultaneous messages is a
> far cry from disallowing any non-main thread to display anything.

I agree. I also think that non-main threads should not be disallowed to
trigger redisplay.

However,

(1) this redisplay should better be synchronous (first waiting for all
    other threads to finish their redisplay) when using the existing
    redisplay primitives.

(2) we may want to introduce asynchronous API to not block during
    redisplay, when threads do not care about display-related call
    returning a value. For example, there is no reason to wait until a
    message if displayed for async thread reporting its progress. It
    might be better to have something like `async-message' that will
    submit a request to display when Emacs decides to. Of course, such
    async API should not rely on thread-local context.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Emacs design and architecture. How about copy-on-write?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: luangruo@yahoo.com, acm@muc.de, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Thu, 21 Sep 2023 10:35:14 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We have these situations already: arrange for some timer to display an
> > echo-area message after several seconds, then type M-x or something
> > else to enter the minibuffer, and watch how the message is displayed.
> > We solved this, more-or-less, in Emacs 29.
> 
> In other words, it is some kind of primitive async queue, right?

No, it's a display trick.

Re: Emacs design and architecture. How about copy-on-write?

[Björn Bidar]

Po Lu <luangruo@yahoo.com> writes:

>> Useful stuff in Emacs almost always affects the display, so if a
>> non-main threads cannot do that, they will be useless.  Or what am I
>> missing?
>
> Consider the case of Gnus or Eglot: it will enable either of them to
> fetch news or receive LSP output from a second thread, decode it, and
> subsequently transfer it to the main thread for display.  Such expensive
> processing is the reason people desire a multi-processing Emacs, because
> it will facilitate efficiently running these operations in the
> background and possibly on a different CPU.

Magit is another example: A lot of modes are busy gathering data or
search through data. If the operations can run in the background not
blocking the main thread that runs the ui it would be to kill for (so to
speak).

Rendering the ui can take some time sometimes but not as often.

Re: Emacs design and architecture. How about copy-on-write?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > A value in a value-cell (etc.) would have an associated thread value, the
  > thread which "owns" it.  When thread1 gets cloned from thread0, it
  > continues to use thread0's data state until it writes foo (e.g. by
  > binding it).  This would create a new value/thread pair, foo/thread1.
  > Further writes of foo by thread1 would continue to access foo/thread1.

It sounds reasonable to me.  It might be feasible to implement
as part of the mechanism that handles local values.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Emanuel Berg]

Eli Zaretskii wrote:

>> Again it would be interesting to hear of how other
>> applications are doing it.
>
> There's a separate mutex for each global data structure.

There should be a lot of experience doing that in C system
programming not the least. Maybe we can bring something like
that straight from some FOSS project?

>> Because if the whole thing has to be locked for each thread
>> to access, it can be disputed if that is indeed any
>> parallelism at all. It will be multi-threaded and
>> multi-core alright, but not parallel execution unless one
>> counts waiting for a shared resource to become available as
>> a sensible passivity.
>
> That's what we have with Lisp threads now: while one thread
> runs, all the others are stopped by a global lock.

But do we also have Lisp threads that execute on separate
CPU cores in parallel, even tho they can't do anything
sensible both simultaneously, for said reasons?

We don't have that, right, because there is no reason to, as
the threads are unable to lock individual resources anyway?

So before we have a good reason to do that, we need to solve
how a single thread locks a single, unlocked global data
structure, access it's data, and then unlocks it?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Microsoft also has a project that could be tried as a base (MIT
> Licensed): https://github.com/microsoft/vscode-webview-ui-toolkit. Or
> one could just use its internals for inspiration, because "Visual
> Studio Code design language" is probably not one of our goals.

Looks like Javascript/Typescript to me?

Re: Emacs design and architecture

[Dmitry Gutov]

On 16/09/2023 11:41, Gerd Möllmann wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> Microsoft also has a project that could be tried as a base (MIT
>> Licensed):https://github.com/microsoft/vscode-webview-ui-toolkit. Or
>> one could just use its internals for inspiration, because "Visual
>> Studio Code design language" is probably not one of our goals.
> Looks like Javascript/Typescript to me?

Yep. But any attempt to reuse an existing browser engine would likely 
have to deal with JavaScript on some level, I think.

Re: Emacs design and architecture

[Sebastian Miele]

> From: Dmitry Gutov <dmitry@gutov.dev>
> Date: Sat, 2023-09-16 14:02 +0300
>
> On 16/09/2023 11:41, Gerd Möllmann wrote:
>> Dmitry Gutov<dmitry@gutov.dev>  writes:
>> 
>>> Microsoft also has a project that could be tried as a base (MIT
>>> Licensed):https://github.com/microsoft/vscode-webview-ui-toolkit. Or
>>> one could just use its internals for inspiration, because "Visual
>>> Studio Code design language" is probably not one of our goals.
>> Looks like Javascript/Typescript to me?
>
> Yep. But any attempt to reuse an existing browser engine would likely
> have to deal with JavaScript on some level, I think.

In the foreseeable future, probably not.  I do not know the details.
But there is WebAssembly.  In order to access the DOM and possibly other
browser API, at least a few months ago, it was still necessary to
somehow go through JS.  But it is very unlikely that that will not
change in a not too distant future.  There are many developements going
on in that area that (will) make implementing further languages on top
of WebAssembly easier and the languages and APIs interoperable with less
and less overhead, and more and more common management (including GC).
I have only a very superficial view.  But in the last months I gained
the impression, that WebAssembly and standards and stuff around it
probably will become a very versatile and interoperable VM
infrastracture, including "WebAssamble-native" APIs to almost anything.

What may be interesting in that direction, too, are experimental browser
engines like Servo.  In the last months I read somewhere (and by people
contributing to it) that Servo more or less explicitly has the aim to
allow to take/use only parts of it, and to have as clear and
approachable source code as possible.  A next generation Emacs probably
would not need or even want all that a web browser has to support.  It
could concentrate on a subset of web-stuff that already is known to work
very well and efficiently.

Disclaimer: I really do not know the details.  This is only a
superficial view gained during the last months.

Re: Emacs design and architecture

[Po Lu]

Sebastian Miele <iota@whxvd.name> writes:

> In the foreseeable future, probably not.  I do not know the details.
> But there is WebAssembly.  In order to access the DOM and possibly other
> browser API, at least a few months ago, it was still necessary to
> somehow go through JS.  But it is very unlikely that that will not
> change in a not too distant future.  There are many developements going
> on in that area that (will) make implementing further languages on top
> of WebAssembly easier and the languages and APIs interoperable with less
> and less overhead, and more and more common management (including GC).
> I have only a very superficial view.  But in the last months I gained
> the impression, that WebAssembly and standards and stuff around it
> probably will become a very versatile and interoperable VM
> infrastracture, including "WebAssamble-native" APIs to almost anything.
>
> What may be interesting in that direction, too, are experimental browser
> engines like Servo.  In the last months I read somewhere (and by people
> contributing to it) that Servo more or less explicitly has the aim to
> allow to take/use only parts of it, and to have as clear and
> approachable source code as possible.  A next generation Emacs probably
> would not need or even want all that a web browser has to support.  It
> could concentrate on a subset of web-stuff that already is known to work
> very well and efficiently.
>
> Disclaimer: I really do not know the details.  This is only a
> superficial view gained during the last months.

ISTM that it would be more productive to examine LibreOffice.  We do
wish to turn Emacs into a word processor, after all.  And LibreOffice
provides layout capabilities that are in no way inferior to those
supplies by the editors found within web browsers, all while being
designed to edit rather than display documents.

And as a second data point, none of the web browsers cited in this
thread still function on the computers I use five days a week, at my
workplace.  Nor do they support Windows XP, which I believe Eli uses.

Re: Emacs design and architecture

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> ISTM that it would be more productive to examine LibreOffice.  We do
> wish to turn Emacs into a word processor, after all.

Out of interest, it that a project goal today?

Re: Emacs design and architecture

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> ISTM that it would be more productive to examine LibreOffice.  We do
>> wish to turn Emacs into a word processor, after all.
>
> Out of interest, it that a project goal today?

Yes, see ``Emacs as a word processor'' under etc/TODO.

Re: Emacs design and architecture

[Björn Bidar]

Po Lu <luangruo@yahoo.com> writes:

> Sebastian Miele <iota@whxvd.name> writes:
>
>> In the foreseeable future, probably not.  I do not know the details.
>> But there is WebAssembly.  In order to access the DOM and possibly other
>> browser API, at least a few months ago, it was still necessary to
>> somehow go through JS.  But it is very unlikely that that will not
>> change in a not too distant future.  There are many developements going
>> on in that area that (will) make implementing further languages on top
>> of WebAssembly easier and the languages and APIs interoperable with less
>> and less overhead, and more and more common management (including GC).
>> I have only a very superficial view.  But in the last months I gained
>> the impression, that WebAssembly and standards and stuff around it
>> probably will become a very versatile and interoperable VM
>> infrastracture, including "WebAssamble-native" APIs to almost anything.
>>
>> What may be interesting in that direction, too, are experimental browser
>> engines like Servo.  In the last months I read somewhere (and by people
>> contributing to it) that Servo more or less explicitly has the aim to
>> allow to take/use only parts of it, and to have as clear and
>> approachable source code as possible.  A next generation Emacs probably
>> would not need or even want all that a web browser has to support.  It
>> could concentrate on a subset of web-stuff that already is known to work
>> very well and efficiently.
>>
>> Disclaimer: I really do not know the details.  This is only a
>> superficial view gained during the last months.
>
> ISTM that it would be more productive to examine LibreOffice.  We do
> wish to turn Emacs into a word processor, after all.  And LibreOffice
> provides layout capabilities that are in no way inferior to those
> supplies by the editors found within web browsers, all while being
> designed to edit rather than display documents.

From my point of view Emacs slowly turns into an environment that runs
modes that greatly become to behave more like applications rather than
extensions to a program. Emacs is in someway like a web browser or a
jvm.

These more extensive modes require more advanced features similar as
when turning Emacs into a "word processor".
In my opinion Emacs being single threaded is the biggest hurdle in that,
gui lockup is the biggest no no in regular gui apps.

Re: Emacs design and architecture

[Dmitry Gutov]

On 16/09/2023 17:20, Björn Bidar wrote:
> These more extensive modes require more advanced features similar as
> when turning Emacs into a "word processor".
> In my opinion Emacs being single threaded is the biggest hurdle in that,
> gui lockup is the biggest no no in regular gui apps.

JavaScript is single-threaded.

They have a certain solution for parallelism in specific workloads, 
though (called Web Workers).

Re: Emacs design and architecture

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 16/09/2023 17:20, Björn Bidar wrote:
>> These more extensive modes require more advanced features similar as
>> when turning Emacs into a "word processor".
>> In my opinion Emacs being single threaded is the biggest hurdle in that,
>> gui lockup is the biggest no no in regular gui apps.
>
> JavaScript is single-threaded.
>
> They have a certain solution for parallelism in specific workloads,
> though (called Web Workers).

In Emacs, GC can run Lisp I'm told (finalizers), regexp matching can
definitely run Lisp (bug#58042), and the devil knows what else :-(.  I
personally have no idea how it would be possible to untangle this for a
multi-threaded Lisp.

Re: Emacs design and architecture

[Dmitry Gutov]

On 16/09/2023 20:07, Gerd Möllmann wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> On 16/09/2023 17:20, Björn Bidar wrote:
>>> These more extensive modes require more advanced features similar as
>>> when turning Emacs into a "word processor".
>>> In my opinion Emacs being single threaded is the biggest hurdle in that,
>>> gui lockup is the biggest no no in regular gui apps.
>> JavaScript is single-threaded.
>>
>> They have a certain solution for parallelism in specific workloads,
>> though (called Web Workers).
> In Emacs, GC can run Lisp I'm told (finalizers), regexp matching can
> definitely run Lisp (bug#58042), and the devil knows what else 🙁.  I
> personally have no idea how it would be possible to untangle this for a
> multi-threaded Lisp.

Each Web Worker (in JS) uses a separate address space, without sharing 
references, exchanging information through "messages" (which are copied, 
not shared).

So this would be like parallel Lisp interpreters with separate obarrays, 
memory spaces, etc.

Re: Emacs design and architecture

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 16/09/2023 20:07, Gerd Möllmann wrote:
>> Dmitry Gutov<dmitry@gutov.dev>  writes:
>> 
>>> On 16/09/2023 17:20, Björn Bidar wrote:
>>>> These more extensive modes require more advanced features similar as
>>>> when turning Emacs into a "word processor".
>>>> In my opinion Emacs being single threaded is the biggest hurdle in that,
>>>> gui lockup is the biggest no no in regular gui apps.
>>> JavaScript is single-threaded.
>>>
>>> They have a certain solution for parallelism in specific workloads,
>>> though (called Web Workers).
>> In Emacs, GC can run Lisp I'm told (finalizers), regexp matching can
>> definitely run Lisp (bug#58042), and the devil knows what else 🙁.  I
>> personally have no idea how it would be possible to untangle this for a
>> multi-threaded Lisp.
>
> Each Web Worker (in JS) uses a separate address space, without sharing
> references, exchanging information through "messages" (which are
> copied, not shared).

Yes, that's what I gathered from reading about web workers on the web.
Do you know what, if anything, makes JS single-threaded?

> So this would be like parallel Lisp interpreters with separate
> obarrays, memory spaces, etc.

Which means someone has to "modularize" the current C code, so that, for
example, more than one GC module, Interpreter module, etc. could exist.

Not easy, but in my eyes, it would definitely be an improvement.  Is
there someone already working in this direction?

(Not that anyone gets thei mpression I'd work on this...  Just saying
:-).

Re: Emacs design and architecture

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1997 bytes --]

On Sat, Sep 16, 2023, 2:33 PM Gerd Möllmann <gerd.moellmann@gmail.com>
wrote:

> Dmitry Gutov <dmitry@gutov.dev> writes:
>
> > On 16/09/2023 20:07, Gerd Möllmann wrote:
> >> Dmitry Gutov<dmitry@gutov.dev>  writes:
> >>
> >>> On 16/09/2023 17:20, Björn Bidar wrote:
> >>>> These more extensive modes require more advanced features similar as
> >>>> when turning Emacs into a "word processor".
> >>>> In my opinion Emacs being single threaded is the biggest hurdle in
> that,
> >>>> gui lockup is the biggest no no in regular gui apps.
> >>> JavaScript is single-threaded.
> >>>
> >>> They have a certain solution for parallelism in specific workloads,
> >>> though (called Web Workers).
> >> In Emacs, GC can run Lisp I'm told (finalizers), regexp matching can
> >> definitely run Lisp (bug#58042), and the devil knows what else 🙁.  I
> >> personally have no idea how it would be possible to untangle this for a
> >> multi-threaded Lisp.
> >
> > Each Web Worker (in JS) uses a separate address space, without sharing
> > references, exchanging information through "messages" (which are
> > copied, not shared).
>
> Yes, that's what I gathered from reading about web workers on the web.
> Do you know what, if anything, makes JS single-threaded?
>
> > So this would be like parallel Lisp interpreters with separate
> > obarrays, memory spaces, etc.
>
> Which means someone has to "modularize" the current C code, so that, for
> example, more than one GC module, Interpreter module, etc. could exist.
>
> Not easy, but in my eyes, it would definitely be an improvement.  Is
> there someone already working in this direction?
>
> (Not that anyone gets thei mpression I'd work on this...  Just saying
> :-).
>

I've mentioned before that multithreading should be split into (same
process) multi-lisp-cpu multiprocessing and same lisp CPU multithreading.
It's on my agenda, but there's a lot of lower hanging fruit/prerequisites
to do first.

Lynn

[-- Attachment #2: Type: text/html, Size: 2987 bytes --]

Re: Emacs design and architecture

[Gerd Möllmann]

Lynn Winebarger <owinebar@gmail.com> writes:

>  Which means someone has to "modularize" the current C code, so that, for
>  example, more than one GC module, Interpreter module, etc. could exist.
>
>  Not easy, but in my eyes, it would definitely be an improvement.  Is
>  there someone already working in this direction?
>
>  (Not that anyone gets thei mpression I'd work on this...  Just saying
>  :-).
>
> I've mentioned before that multithreading should be split into (same
> process) multi-lisp-cpu multiprocessing and same lisp CPU
> multithreading.  It's on my agenda, but there's a lot of lower hanging
> fruit/prerequisites to do first.

Ok, good to know.  Thanks.

Re: Emacs design and architecture

[Dmitry Gutov]

On 16/09/2023 21:33, Gerd Möllmann wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> On 16/09/2023 20:07, Gerd Möllmann wrote:
>>> Dmitry Gutov<dmitry@gutov.dev>  writes:
>>>
>>>> On 16/09/2023 17:20, Björn Bidar wrote:
>>>>> These more extensive modes require more advanced features similar as
>>>>> when turning Emacs into a "word processor".
>>>>> In my opinion Emacs being single threaded is the biggest hurdle in that,
>>>>> gui lockup is the biggest no no in regular gui apps.
>>>> JavaScript is single-threaded.
>>>>
>>>> They have a certain solution for parallelism in specific workloads,
>>>> though (called Web Workers).
>>> In Emacs, GC can run Lisp I'm told (finalizers), regexp matching can
>>> definitely run Lisp (bug#58042), and the devil knows what else 🙁.  I
>>> personally have no idea how it would be possible to untangle this for a
>>> multi-threaded Lisp.
>>
>> Each Web Worker (in JS) uses a separate address space, without sharing
>> references, exchanging information through "messages" (which are
>> copied, not shared).
> 
> Yes, that's what I gathered from reading about web workers on the web.
> Do you know what, if anything, makes JS single-threaded?

I'm not sure if I understand the question, but to my knowledge, it uses 
the same kind of "interpreter lock" as is common in other "interpreted" 
languages too (Ruby, Python, PHP, Lua).

That's not the whole story, though, because it's a lock within the 
"interpreter context", and every open tab has its own. Most tab do not 
interact in any way, cannot access each other's data, so it's fine to 
run their JS code in parallel. Likewise, JS on the web doesn't have any 
access to the browser chrome, meaning they're fine to run in parallel 
too. In fact, modern browsers run several processes: one for the UI and 
several for the browser tabs (last info I saw on this, link below, is 
that Chrome used a process-per-tab, and Firefox combined several tabs in 
one process using parallel threads).

https://medium.com/mozilla-tech/the-search-for-the-goldilocks-browser-and-why-firefox-may-be-just-right-for-you-1f520506aa35

This is already more parallelism than we have (each piece of our code 
can access everything), but OTOH they have more spinning plates at the 
same time.

>> So this would be like parallel Lisp interpreters with separate
>> obarrays, memory spaces, etc.
> 
> Which means someone has to "modularize" the current C code, so that, for
> example, more than one GC module, Interpreter module, etc. could exist.
> 
> Not easy, but in my eyes, it would definitely be an improvement.  Is
> there someone already working in this direction?
> 
> (Not that anyone gets thei mpression I'd work on this...  Just saying
> :-).

The best I can point to are comments on Reddit around when emacs-ng 
launched: https://www.reddit.com/r/emacs/comments/kmh9k1/comment/ghm68j2/

It included the ability to run code in Deno's Web Workers. It feels like 
something that would help with some existing problems (e.g. parting JSON 
in a worker, extracting a subpath from it and sending the result to the 
parent process), but I don't remember reading any serious code done with 
it. Possibly just because all big packages have to keep compatibility 
with the GNU version anyway, so I'm hopeful about such prospects if we 
ever even get a prototype in GNU Emacs. At the very least, it can be 
like a faster async.el, with all of its existing use cases.

Re: Emacs design and architecture

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I'm not sure if I understand the question, but to my knowledge, it
> uses the same kind of "interpreter lock" as is common in other
> "interpreted" languages too (Ruby, Python, PHP, Lua).

Yes, thanks, that was what I meant.

> That's not the whole story, though, because it's a lock within the
> "interpreter context", and every open tab has its own. Most tab do not
> interact in any way, cannot access each other's data, so it's fine to
> run their JS code in parallel. Likewise, JS on the web doesn't have
> any access to the browser chrome, meaning they're fine to run in
> parallel too. In fact, modern browsers run several processes: one for
> the UI and several for the browser tabs (last info I saw on this, link
> below, is that Chrome used a process-per-tab, and Firefox combined
> several tabs in one process using parallel threads).
>
> https://medium.com/mozilla-tech/the-search-for-the-goldilocks-browser-and-why-firefox-may-be-just-right-for-you-1f520506aa35
>
> This is already more parallelism than we have (each piece of our code
> can access everything), but OTOH they have more spinning plates at the
> same time.
>
>>> So this would be like parallel Lisp interpreters with separate
>>> obarrays, memory spaces, etc.
>> Which means someone has to "modularize" the current C code, so that,
>> for
>> example, more than one GC module, Interpreter module, etc. could exist.
>> Not easy, but in my eyes, it would definitely be an improvement.  Is
>> there someone already working in this direction?
>> (Not that anyone gets thei mpression I'd work on this...  Just
>> saying
>> :-).
>
> The best I can point to are comments on Reddit around when emacs-ng
> launched:
> https://www.reddit.com/r/emacs/comments/kmh9k1/comment/ghm68j2/
>
> It included the ability to run code in Deno's Web Workers. 

Yes, that's of course also a "solution" - add a second
language/VM/runtime or runtime to Lisp.  It at least avoids untangling
Emacs' C code :-).

> It feels like something that would help with some existing problems
> (e.g. parting JSON in a worker, extracting a subpath from it and
> sending the result to the parent process), but I don't remember
> reading any serious code done with it. Possibly just because all big
> packages have to keep compatibility with the GNU version anyway, so
> I'm hopeful about such prospects if we ever even get a prototype in
> GNU Emacs. At the very least, it can be like a faster async.el, with
> all of its existing use cases.

Hm.

Re: Emacs design and architecture

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Dmitry Gutov <dmitry@gutov.dev> writes:
>
>> On 16/09/2023 17:20, Björn Bidar wrote:
>>> These more extensive modes require more advanced features similar
>>> as
>>> when turning Emacs into a "word processor".
>>> In my opinion Emacs being single threaded is the biggest hurdle in
>>> that,
>>> gui lockup is the biggest no no in regular gui apps.
>>
>> JavaScript is single-threaded.
>>
>> They have a certain solution for parallelism in specific workloads,
>> though (called Web Workers).
>
> In Emacs, GC can run Lisp I'm told (finalizers), regexp matching can
> definitely run Lisp (bug#58042), and the devil knows what else :-(.  I
> personally have no idea how it would be possible to untangle this for
> a multi-threaded Lisp.

Presumably, finalizers must become thread safe, and Lisp run during
regexp matching will run within whichever thread calls string-match.  I
don't see any unsurmountable impediments here.

Re: Emacs design and architecture

[Dmitry Gutov]

On 16/09/2023 16:00, Po Lu wrote:

> ISTM that it would be more productive to examine LibreOffice.  We do
> wish to turn Emacs into a word processor, after all.  And LibreOffice
> provides layout capabilities that are in no way inferior to those
> supplies by the editors found within web browsers, all while being
> designed to edit rather than display documents.

A non-programmable specialized application vs. an extensible platform.

> And as a second data point, none of the web browsers cited in this
> thread still function on the computers I use five days a week, at my
> workplace.

They do at mine.

> Nor do they support Windows XP, which I believe Eli uses.

Not sure how that's relevant in a discussion about architecture.

Re: Emacs design and architecture

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 16/09/2023 16:00, Po Lu wrote:
>
>> ISTM that it would be more productive to examine LibreOffice.  We do
>> wish to turn Emacs into a word processor, after all.  And
>> LibreOffice
>> provides layout capabilities that are in no way inferior to those
>> supplies by the editors found within web browsers, all while being
>> designed to edit rather than display documents.
>
> A non-programmable specialized application vs. an extensible platform.

LibreOffice is programmable, in almost the same language as that used to
implement Emacs.  And given that it's designed to edit documents, it's
certainly a better match than a web browser.

Re: Emacs design and architecture

[Immanuel Litzroth]

On Sun, Sep 17, 2023 at 3:03 AM Po Lu <luangruo@yahoo.com> wrote:

> LibreOffice is programmable, in almost the same language as that used to
> implement Emacs.  And given that it's designed to edit documents, it's
> certainly a better match than a web browser.

I don't understand this remark. When you talk about "the language used
to implement
emacs" are you talking about C or elisp? What is the LibreOffice
extension language that
is "almost the same"? Can you provide a link?
Thanks in advance,
Immanuel


-- 
-- A man must either resolve to point out nothing new or to become a
slave to defend it. -- Sir Isaac Newton

Re: Emacs design and architecture

[Po Lu]

Immanuel Litzroth <immanuel.litzroth@gmail.com> writes:

> I don't understand this remark. When you talk about "the language used
> to implement emacs" are you talking about C or elisp? 

C, of course.

> What is the LibreOffice extension language that is "almost the same"?
> Can you provide a link?

LibreOffice is written in C++.  And it is programmable, by virtue of its
nature as a computer program.

Re: Emacs design and architecture

[Dmitry Gutov]

On 17/09/2023 09:43, Po Lu wrote:
> Immanuel Litzroth<immanuel.litzroth@gmail.com>  writes:
> 
>> I don't understand this remark. When you talk about "the language used
>> to implement emacs" are you talking about C or elisp?
> C, of course.
> 
>> What is the LibreOffice extension language that is "almost the same"?
>> Can you provide a link?
> LibreOffice is written in C++.  And it is programmable, by virtue of its
> nature as a computer program.

Scriptable programs have their specific challenges, so it's helpful to 
look at architectures suited to that.

Anyway, I stand corrected, LibreOffice is actually quite scriptable (in 
Java, Python and others): 
https://wiki.documentfoundation.org/Documentation/DevGuide/Scripting_Framework

Though probably not to the extent that Emacs is.

Re: Emacs design and architecture

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 17/09/2023 09:43, Po Lu wrote:
>> Immanuel Litzroth<immanuel.litzroth@gmail.com>  writes:
>> 
>>> I don't understand this remark. When you talk about "the language
>>> used
>>> to implement emacs" are you talking about C or elisp?
>> C, of course.
>> 
>>> What is the LibreOffice extension language that is "almost the
>>> same"?
>>> Can you provide a link?
>> LibreOffice is written in C++.  And it is programmable, by virtue of
>> its
>> nature as a computer program.
>
> Scriptable programs have their specific challenges, so it's helpful to
> look at architectures suited to that.

Given that we don't "script" the Emacs display process itself, there is
truly no distinction between a "scriptable" display engine and others.

The MO of Emacs extension code is to modify the contents of a buffer,
and let redisplay do the heavy lifting.  Such an approach will fare
equally irrespective of the representation used for buffer contents.

Re: Emacs design and architecture

[Dmitry Gutov]

On 18/09/2023 03:11, Po Lu wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> On 17/09/2023 09:43, Po Lu wrote:
>>> Immanuel Litzroth<immanuel.litzroth@gmail.com>   writes:
>>>
>>>> I don't understand this remark. When you talk about "the language
>>>> used
>>>> to implement emacs" are you talking about C or elisp?
>>> C, of course.
>>>
>>>> What is the LibreOffice extension language that is "almost the
>>>> same"?
>>>> Can you provide a link?
>>> LibreOffice is written in C++.  And it is programmable, by virtue of
>>> its
>>> nature as a computer program.
>> Scriptable programs have their specific challenges, so it's helpful to
>> look at architectures suited to that.
> Given that we don't "script" the Emacs display process itself, there is
> truly no distinction between a "scriptable" display engine and others.

We do: just as others have noted, redisplay calls back into Lisp. Which 
allows us to implement jit-lock with language-specific logic written in 
Lisp.

It has for a long time been the method by which we manage to do syntax 
highlighting with adequate performance even in large files.

Re: Emacs design and architecture

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> We do: just as others have noted, redisplay calls back into
> Lisp. Which allows us to implement jit-lock with language-specific
> logic written in Lisp.
>
> It has for a long time been the method by which we manage to do syntax
> highlighting with adequate performance even in large files.

Jit-Lock doesn't control the display process itself.  It's only a
facility which enables buffer text to be selectively modified _before_
it is actually displayed.

Re: Emacs design and architecture

[Dmitry Gutov]

On 18/09/2023 14:12, Po Lu wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> We do: just as others have noted, redisplay calls back into
>> Lisp. Which allows us to implement jit-lock with language-specific
>> logic written in Lisp.
>>
>> It has for a long time been the method by which we manage to do syntax
>> highlighting with adequate performance even in large files.
> Jit-Lock doesn't control the display process itself.  It's only a
> facility which enables buffer text to be selectively modified_before_
> it is actually displayed.

Naturally.

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We do: just as others have noted, redisplay calls back into Lisp. Which 
  > allows us to implement jit-lock with language-specific logic written in 
  > Lisp.

To say that this happens in "redisplay" is too general to indicate
whether there is anything tricky or dangerous about it.
Calls from outer parts of redisplay are not problematical.
They are no more dangerous than calling before or after redisplay.

What would be problematical is calling from within the parts
of redisplay that alter the display matrices.  An error there
would be hard to recover.  It could crash.

It is many years since I looked at that code, but ISTR that the
calls for font lock are in the outer parts, specifically to
avoid this danger.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 20 Sep 2023 14:35:41 -0400
> 
> What would be problematical is calling from within the parts
> of redisplay that alter the display matrices.  An error there
> would be hard to recover.  It could crash.
> 
> It is many years since I looked at that code, but ISTR that the
> calls for font lock are in the outer parts, specifically to
> avoid this danger.

No, they are called from where the glyph matrices are constructed.
But we catch Lisp errors when we call jit-lock (and any other Lisp)
from the display code, so any error just causes a message in
*Messages* (and might leave the text unfontified).  So it is not
dangerous.

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In the foreseeable future, probably not.  I do not know the details.
  > But there is WebAssembly.  In order to access the DOM and possibly other
  > browser API, at least a few months ago, it was still necessary to
  > somehow go through JS.

Javascript and webassembly are both paths by which nonfree programs is
fed to users without their ever asking to install them.  Yes we have
free web browsers, but a free web browser won't give you freedom if it
often spontaneously downloads a nonfree program and runs that without
warning you.

This is why we developed LibreJS for Firefox -- to detect and block
nonfree Javascript code (or webassembly code).

I think that some of Emacs's web browsing facilities include
Javascript execution.  (Is this correct?)  I think it was a mistake to
add that to Emacs -- it led us away from the goal of the GNU system.

To correct that mistake, we should change those features so as to
reduce the tendency to let nonfree software into Emacs users'
activity.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[chad]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

On Sun, Sep 17, 2023 at 7:03 PM Richard Stallman <rms@gnu.org> wrote:

> [...]
> I think that some of Emacs's web browsing facilities include
> Javascript execution.  (Is this correct?)
>

None of the web browsers I've seen for emacs support javascript execution,
and it would be a difficult thing to add. At a guess, the easiest method
would involve emacs running (or communicating with) an existing browser and
interrogating the results, very roughly along the same lines as CIDER or
SLIME bring clojure or common lisp execution into emacs.

Note that it's _much_ easier to bring simple javascript execution into
emacs -- there are cli/repl-style javascript interpreters available for
such tasks. They don't have access to the other core parts of the _web
browsing_ facilities, in particular the DOM (Document Object Model) that
contains the structure of the material to be browsed, nor the CSS
facilities for styling same.

In short: the path the "modern web" took to get to interactive "rich"
documents on the web ended up very strongly tied to a particular underlying
representation of the content, including how it is loaded, displayed,
styled, and updated. Those pieces are far more theoretically distinct than
actually composable.

~Chad

[-- Attachment #2: Type: text/html, Size: 1686 bytes --]

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > None of the web browsers I've seen for emacs support javascript execution,

That is a relief.

  > and it would be a difficult thing to add. At a guess, the easiest method
  > would involve emacs running (or communicating with) an existing browser and
  > interrogating the results, very roughly along the same lines as CIDER or
  > SLIME bring clojure or common lisp execution into emacs.

I first saw CIDER mentioned in a recent discussion, and no one who
mentioned it explained what it does.  I don't know what SLIME does
either.  Would you please tell me briefly what they do, in 20 lines
for each?  I don't want details, at least not to start.

  > Note that it's _much_ easier to bring simple javascript execution into
  > emacs -- there are cli/repl-style javascript interpreters available for
  > such tasks.

I think that this development would make EMacs less coherent and more
work to maintain and document.  It is better not to do this.

  > In short: the path the "modern web" took to get to interactive "rich"
  > documents on the web ended up very strongly tied to a particular underlying
  > representation of the content, including how it is loaded, displayed,
  > styled, and updated. Those pieces are far more theoretically distinct than
  > actually composable.

That might seem unfortunate in a technical sense, but it turns out
beneficial because it saves us from temptation.  Since the "modern
web" works by trampling users' fredeom by sending programs they do not
control to run on their machines, GNU Emacs should not get on board
with that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Bob Rogers]

   From: Richard Stallman <rms@gnu.org>
   Date: Thu, 21 Sep 2023 16:26:24 -0400

   I first saw CIDER mentioned in a recent discussion, and no one who
   mentioned it explained what it does.  I don't know what SLIME does
   either.  Would you please tell me briefly what they do, in 20 lines
   for each?  I don't want details, at least not to start.

It's even easier than that:  SLIME is "The Superior Lisp Interaction
Mode for Emacs" and it essentially replaced ILISP among Emacs users in
the CL development community 20-odd years ago.  SLIME provides REPL,
debugging interface, inspector, and editing hacks for Emacs.  CIDER is
"the Clojure(Script) Interactive Development Environment that Rocks," so
it's similar for Clojure; I've never used CIDER, but it says it was
inspired by SLIME.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks for explaining.

That seems like the right way for Emacs to interface to those languages,
the same way it interfaces to various other languages.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Dmitry Gutov]

On 16/09/2023 14:59, Sebastian Miele wrote:
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> Date: Sat, 2023-09-16 14:02 +0300
>>
>> On 16/09/2023 11:41, Gerd Möllmann wrote:
>>> Dmitry Gutov<dmitry@gutov.dev>  writes:
>>>
>>>> Microsoft also has a project that could be tried as a base (MIT
>>>> Licensed):https://github.com/microsoft/vscode-webview-ui-toolkit. Or
>>>> one could just use its internals for inspiration, because "Visual
>>>> Studio Code design language" is probably not one of our goals.
>>> Looks like Javascript/Typescript to me?
>>
>> Yep. But any attempt to reuse an existing browser engine would likely
>> have to deal with JavaScript on some level, I think.
> 
> In the foreseeable future, probably not.  I do not know the details.
> But there is WebAssembly.  In order to access the DOM and possibly other
> browser API, at least a few months ago, it was still necessary to
> somehow go through JS.  But it is very unlikely that that will not
> change in a not too distant future.  There are many developements going
> on in that area that (will) make implementing further languages on top
> of WebAssembly easier and the languages and APIs interoperable with less
> and less overhead, and more and more common management (including GC).
> I have only a very superficial view.  But in the last months I gained
> the impression, that WebAssembly and standards and stuff around it
> probably will become a very versatile and interoperable VM
> infrastracture, including "WebAssamble-native" APIs to almost anything.

Wasm or JS, with this approach it seems like we'd need to compile Lisp 
to JS/WA, or use FFI, or (probably) both.

None of this is a deal-breaker for an experimental port/GUI, but it 
definitely doesn't make it a weekend project.

> What may be interesting in that direction, too, are experimental browser
> engines like Servo.  In the last months I read somewhere (and by people
> contributing to it) that Servo more or less explicitly has the aim to
> allow to take/use only parts of it, and to have as clear and
> approachable source code as possible.  A next generation Emacs probably
> would not need or even want all that a web browser has to support.  It
> could concentrate on a subset of web-stuff that already is known to work
> very well and efficiently.

Looking at these, it doesn't sound like it's easy to replace the current 
JS engine (Firefox's SpiderMonkey) with something else. Although to make 
it run Lisp would be pretty cool.

https://github.com/servo/servo/discussions/29100
https://github.com/servo/servo/discussions/25419

The comments in the latter led me to this, though:

https://github.com/fschutt/azul (Rust desktop GUI framework based on 
WebRender with no JavaScript involved, looks like).

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Wasm or JS, with this approach it seems like we'd need to compile Lisp 
  > to JS/WA, or use FFI, or (probably) both.

You're proposing to evaluate issues by technical criteria alone.
This decision has implications about freedom -- and danger to it.
That has to be the principal basis for deciding.

In this question, we have to compare JavaScript with Emavcs Lisp in
regards to the likely seepage of nonfree software into use of Emacs.

It is possible to release a nonfree program in Emacs Lisp, especially
if it uses only the documented Emacs Lisp interfaces.  But there is no
widespread practice of doing so, so in practice that is not a
significanmt problem as far as I have heard.

By contrast, the web is crawling with nonfree JavaScript programs and
most internet users run them unaware of the issue they raise, or even
that they are downloading them.  Anything that GNU software does with
JavaScript is a potential problem for achieving the GNU system's goal:
giving users freedom.  GNU won't achieve that if normal use of GNU
programs invites use of nonfree JavaScript.

Therefore, any plan to make GNU software relate more often to
JavaScript needs to be scrutinized above all in terms of the possible
danger.  In GNU, freedom for the users (which includes us developers)
is the first priority.

There is also a technical reason why effectively combining Emacs with
the LibreOffice code is not a good idea.  It would make a large,
confusing hybrid mess.  To debug it would require knowing more
languages.

Some of the methods implemented in LibreOffice may be useful for
Emacs, so we should choose the useful ones and adapt them to the
context of Emacs.  And keep it customizable in Emacs Lisp.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >>   WebRender is a GPU-based 2D rendering engine written in Rust from
  > >>   Mozilla. Firefox, the research web browser Servo, and other GUI
  > >>   frameworks draw with it. emacs-ng use it as a new experimental graphic
  > >>   backend to leverage GPU hardware.

If that means it is designed to require or encourage using a GPU to
display through, that is a moral problem.  Most GPUs require the user
to install nonfree firmware.  GNU software must not suggest users do
that.  So we cannot make GNU Emacs depend on that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture

[Emanuel Berg]

Gerd Möllmann wrote:

> Whatever is done in the end, I think it would first be
> necessary to change this general principle, so that
> layout/drawing whatever can happen in parallel.

Maybe one could have just a huge state computed, a whole bunch
of digits, and then one could feed it to any display module
that can make sense of it, so it would draw it according to
its own methods, then we could simple try what would work?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs design and architecture

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> This of course, would pose several problems.
>
> - This form of parallel redisplay cannot call Lisp, so no jit-lock, no
>   hooks, or whatever during redisplay.  And at the point where we call
>   redisplay today we don't know what will be displayed...
> - Parallel redisplay also needs either a copy of what it to be
>   displayed, or the model must be some persistent data structure
>   that makes immutable versions of buffer-text, for instance,
>   available.

Some more idle musings:

Let's say persistent data structures for buffer-text, overlays, text
properties and maybe other things are out of question because of the
effort involved.

Another idea would be to produce, where we currently run redisplay, a
"model" that can be processed concurrently.  One could just guess how
much text will be displayed, run Lisp code as necessary (jit-lock), skip
invisble text, and so on, so that the parallel redisplay doesn't have
the need to call Lisp.  For simplicity, think of that new form of model
as HTML.

That could be so slow that parallelism is not a win.

What about the move_.* functions used by Lisp (vertical-motion, ...)?
To what degree does backward compatibily restrict what can be done in
the first place?  I'm beginning to think it sets quite harsh limits.

Doesn't look good to me :-(.

Re: Emacs design and architecture

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> This of course, would pose several problems.
>>
>> - This form of parallel redisplay cannot call Lisp, so no jit-lock, no
>>   hooks, or whatever during redisplay.  And at the point where we call
>>   redisplay today we don't know what will be displayed...
>> - Parallel redisplay also needs either a copy of what it to be
>>   displayed, or the model must be some persistent data structure
>>   that makes immutable versions of buffer-text, for instance,
>>   available.
>
> Some more idle musings:
>
> Let's say persistent data structures for buffer-text, overlays, text
> properties and maybe other things are out of question because of the
> effort involved.
>
> Another idea would be to produce, where we currently run redisplay, a
> "model" that can be processed concurrently.  One could just guess how
> much text will be displayed, run Lisp code as necessary (jit-lock), skip
> invisble text, and so on, so that the parallel redisplay doesn't have
> the need to call Lisp.  For simplicity, think of that new form of model
> as HTML.
>
> That could be so slow that parallelism is not a win.
>
> What about the move_.* functions used by Lisp (vertical-motion, ...)?
> To what degree does backward compatibily restrict what can be done in
> the first place?  I'm beginning to think it sets quite harsh limits.
>
> Doesn't look good to me :-(.

Here are some things I found wrt to buffer-text implementation.

A ninteresting paper about text representations in general:
https://www.cs.unm.edu/~crowley/papers/sds.pdf

Piece tables:
https://www.averylaird.com/programming/the%20text%20editor/2017/09/30/the-piece-table

An implementation:
https://github.com/cdacamar/fredbuf

Something like that could be an inspiration for a persisten buffer-text
implementation, which I think would be a prerequisite for a parallel
redisplay.  (I don't believe in the idea of a snapshort model for parallel
redisplay \aanymore).  This would not solve overlays, text properties
and what not, of course.

Re: Emacs design and architecture

[Helmut Eller]

On Sun, Sep 17 2023, Gerd Möllmann wrote:

> Something like that could be an inspiration for a persisten buffer-text
> implementation, which I think would be a prerequisite for a parallel
> redisplay.  (I don't believe in the idea of a snapshort model for parallel
> redisplay \aanymore).  This would not solve overlays, text properties
> and what not, of course.

Out of curiosity: has somebody tried to vectorize the display code?

I don't know the details, but very abstractly speaking there seems to be
an iterator that loops over the characters in a buffer and computes
various things as it goes along.  Could some of those computations be
done on vectors, of say 8, characters?

Helmut

Re: Emacs design and architecture

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Yuri Khan <yuri.v.khan@gmail.com>,
>   dmitry@gutov.dev,  owinebar@gmail.com,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 17 Sep 2023 17:44:35 +0200
> 
> Out of curiosity: has somebody tried to vectorize the display code?

Not that I know of.

> I don't know the details, but very abstractly speaking there seems to be
> an iterator that loops over the characters in a buffer and computes
> various things as it goes along.  Could some of those computations be
> done on vectors, of say 8, characters?

The decisions are made after each processed buffer/string position,
and frequently processing of position N depends on results of
processing of position N-1.  If these caveats don't get in the way of
vectorizing, I guess we could try that.

Re: Emacs design and architecture

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>> This of course, would pose several problems.
>>>
>>> - This form of parallel redisplay cannot call Lisp, so no jit-lock, no
>>>   hooks, or whatever during redisplay.  And at the point where we call
>>>   redisplay today we don't know what will be displayed...
>>> - Parallel redisplay also needs either a copy of what it to be
>>>   displayed, or the model must be some persistent data structure
>>>   that makes immutable versions of buffer-text, for instance,
>>>   available.
>>
>> Some more idle musings:
>>
>> Let's say persistent data structures for buffer-text, overlays, text
>> properties and maybe other things are out of question because of the
>> effort involved.
>>
>> Another idea would be to produce, where we currently run redisplay, a
>> "model" that can be processed concurrently.  One could just guess how
>> much text will be displayed, run Lisp code as necessary (jit-lock), skip
>> invisble text, and so on, so that the parallel redisplay doesn't have
>> the need to call Lisp.  For simplicity, think of that new form of model
>> as HTML.
>>
>> That could be so slow that parallelism is not a win.
>>
>> What about the move_.* functions used by Lisp (vertical-motion, ...)?
>> To what degree does backward compatibily restrict what can be done in
>> the first place?  I'm beginning to think it sets quite harsh limits.
>>
>> Doesn't look good to me :-(.
>
> Here are some things I found wrt to buffer-text implementation.
>
> A ninteresting paper about text representations in general:
> https://www.cs.unm.edu/~crowley/papers/sds.pdf
>
> Piece tables:
> https://www.averylaird.com/programming/the%20text%20editor/2017/09/30/the-piece-table
>
> An implementation:
> https://github.com/cdacamar/fredbuf
>
> Something like that could be an inspiration for a persisten buffer-text
> implementation, which I think would be a prerequisite for a parallel
> redisplay.  (I don't believe in the idea of a snapshort model for parallel
> redisplay \aanymore).  This would not solve overlays, text properties
> and what not, of course.

And a bit more:

This is an interesting article about VSCode's buffer implementation,
which was changed from a line-based representation to a piece tree
(don't know if persistent or not):

https://code.visualstudio.com/blogs/2018/03/23/text-buffer-reimplementation

In summary, I think that, for a parallel redisplay in Emacs, a persistent
piece tree would probably a good choice, and it is doable.  From a
performance aspect, the Monaco editor (see link above) seems to show
that it's not too bad.

Overlays, text properties, calls to Lisp etc. remain open aspects.

Re: Emacs design and architecture (was: Shrinking the C core)

[Dmitry Gutov]

On 15/09/2023 09:51, Yuri Khan wrote:
> On Fri, 15 Sept 2023 at 12:51, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> One of the important aspects to keep in mind in this regard is that
>> Emacs must give Lisp programs dynamic control of how stuff is
>> displayed, and should be able to exercise that control at high
>> frequency (a trivial example: pulse.el).
> 
> In CSS, this is solved in core with transitions. A style specifies
> that a certain property will change gradually, provides its target
> value, the transition duration, and a transition curve. The styling
> engine does all the work about calculating the intermediate values and
> re-layouting each intermediate frame; the page author does not have to
> script carefully timed property changes, and is in fact discouraged
> from doing so because doing that in Javascript incurs CPU and battery
> usage overhead.

There are different ways to do it, but the classical correspondence to 
what pulse.el does is to use JavaScript, which would trigger on a timer 
and modify the DOM tree, exactly how pulse is triggered on a timer and 
updates the overlays in a buffer.

It's fast enough, and the JavaScript-based animations had reached more 
complexity than what we do with Emacs. It's just that the web sites have 
been getting more complex and complex over the years, that doing many of 
those animations is costly for battery and can bring the frame rate down 
(along with many other tabs being loaded at the same time). TBH, I can't 
think of many existing animations in Emacs that would be well-served by 
CSS animation instructions. Maybe just extra eye candy like exploding 
cursors.

> The use case of pulse.el would be translated to a couple of styles
> that say effectively “A pulsed span will instantly gain yellow
> background” and “A non-pulsed span will linearly revert to whatever
> background it had over the course of 200 milliseconds” and a small
> function that sets the span to pulsed and then immediately to
> non-pulsed.

That's also an option, but that seems to depart from making a good 
comparison between systems. It's not like anybody is thinking of 
implementing animation instructions in the display engine, are they?

The closest stuff to CSS that we have are stuff like 'display' and 
'invisible' properties, as well as display specs like :width, :align-to, 
:height and (slice ...).

> (Implementing a CSS engine over a character terminal is a nontrivial
> matter though.)

I don't know, if such engine is limited to a rectangular canvas of rows 
of characters, that would work more or less the same as it does not.

Re: Emacs design and architecture (was: Shrinking the C core)

[Dmitry Gutov]

On 15/09/2023 08:50, Eli Zaretskii wrote:
> Another thing to keep in mind is that there should be an efficient way
> of telling the display back-end which portions of a window to redraw
> given some changes in the text and display-related data structures.
> (Of course, if the display back-end is so efficient that it can
> redisplay everything fast enough, even if displaying across a network
> on a different terminal, this aspect might be a non-issue.)

I think in the case of HTML it's the layout engine which decides which 
areas of the window might change after some part of the DOM tree, or the 
display rules, have been modified.

Re: Emacs design and architecture (was: Shrinking the C core)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > They are all well known.  And they aren't ideas, just main design
  > features of Emacs which we found restrictive in some aspects:

  >   . "buffer with gap" for storing buffer text
  >   . "mark and sweep" GC
  >   . basic single-threaded MVC architecture
  >   . display engine design around the rectangular canvas model and
  >     on-the-fly layout decisions

These are internal design decisions.
The last one does relate to some user-level features
such as data in the buffer.  The first three don't.

So I don't see these as being related to simplicity
that benefits the users.

Changes in those design aspects could make development and maintenance
harder and/or more likely to go wrong.  For example, some other kinds
of GC could impose constraints on development that could make writing
C code for Emacs error-prone.  But they won't complicate _use_ of
Emacs as long as we avoid those errors.

A more complex and powerful data structure in the buffer is something
I would like to see.  I would like it to enable TeX-style formatting
of text for display.

However, finding a good design for this will be a hard job, I think.
We would want it to facilitate redisplay and also be easy for editing.
It needs to support editing primitives of the traditional Emacs kind,
which treat text as a sequence somehow (even if not the same as now),
and editing primitives that operate on the structure and make them
convenient.

All four of those directions are basically independent.
Any one could be done with or without the others.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: owinebar@gmail.com, emacs-devel@gnu.org
> Date: Sat, 16 Sep 2023 20:45:00 -0400
> 
>   >   . "buffer with gap" for storing buffer text
>   >   . "mark and sweep" GC
>   >   . basic single-threaded MVC architecture
>   >   . display engine design around the rectangular canvas model and
>   >     on-the-fly layout decisions
> 
> These are internal design decisions.
> The last one does relate to some user-level features
> such as data in the buffer.  The first three don't.
> 
> So I don't see these as being related to simplicity
> that benefits the users.

That is true, by and large, but the simplicity that benefits users is
not the only issue at stake.  There's also the issue of being capable
of performing complex computations efficiently, i.e. quickly and
without locking up Emacs; multi-threading and a more modern GC could
give us at least some of that.  Being able to use multiple execution
units in parallel is also the main (perhaps the only) path towards
letting Emacs make good use of modern system architectures.

> A more complex and powerful data structure in the buffer is something
> I would like to see.  I would like it to enable TeX-style formatting
> of text for display.

In off-line discussions with Stefan Monnier, we both arrived at the
conclusion that some basic limitations of the current display engine
cannot be lifted without redesigning how buffer text is stored and
accessed.  So yes, buffer's data structures are an important factor in
what features we could introduce in the future, in particular related
to sophisticated and efficient redisplay.

> However, finding a good design for this will be a hard job, I think.
> We would want it to facilitate redisplay and also be easy for editing.
> It needs to support editing primitives of the traditional Emacs kind,
> which treat text as a sequence somehow (even if not the same as now),
> and editing primitives that operate on the structure and make them
> convenient.

Right.

> All four of those directions are basically independent.
> Any one could be done with or without the others.

They are not completely independent, no.  They _could_ be independent
if the design of each direction took into consideration the possible
designs of the other directions up front, so as not to preclude
developments of those other directions or make them too hard.  And
doing some changes in only one direction could mean unnecessary work.
For example, any significant changes in the display engine without
simultaneous changes in buffer text access would mean either writing
code that will be tossed in the future, or coming up with some
abstraction that could be easily changed when buffer text access
changes; both mean non-trivial extra efforts.  Likewise, redesigning
buffer text storage and access must have the display engine needs
effect the main design decisions, even though redisplay is not being
reimplemented at the same time.

Re: Emacs design and architecture (was: Shrinking the C core)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > These are internal design decisions.
  > > The last one does relate to some user-level features
  > > such as data in the buffer.  The first three don't.
  > > 
  > > So I don't see these as being related to simplicity
  > > that benefits the users.

  > That is true, by and large, but the simplicity that benefits users is
  > not the only issue at stake.  There's also the issue of being capable
  > of performing complex computations efficiently, i.e. quickly and
  > without locking up Emacs; multi-threading and a more modern GC could
  > give us at least some of that.

I agree, these kinds of changes

>   >   . "buffer with gap" for storing buffer text
>   >   . "mark and sweep" GC
>   >   . basic single-threaded MVC architecture
>   >   . display engine design around the rectangular canvas model and
>   >     on-the-fly layout decisions

could possibly be improvements.  I was arguing only against one
putative benefit, "simplicity".

  > In off-line discussions with Stefan Monnier, we both arrived at the
  > conclusion that some basic limitations of the current display engine
  > cannot be lifted without redesigning how buffer text is stored and
  > accessed.

I agree they would require something more powerful than text
properties and overlays as they are now.  Text properties were
designed to work consistently with copying text between buffers and
strings, and that imposed a limit.  What made this conceptually simple was
the idea that each character individually has its own properties.

If we add a more powerful method of attaching non-text data to
buffers, the design phase should include specifying precisely how they
would behave when copying text between buffers and strings.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs design and architecture (was: Shrinking the C core)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: owinebar@gmail.com, emacs-devel@gnu.org
> Date: Tue, 19 Sep 2023 06:22:36 -0400
> 
>   > In off-line discussions with Stefan Monnier, we both arrived at the
>   > conclusion that some basic limitations of the current display engine
>   > cannot be lifted without redesigning how buffer text is stored and
>   > accessed.
> 
> I agree they would require something more powerful than text
> properties and overlays as they are now.  Text properties were
> designed to work consistently with copying text between buffers and
> strings, and that imposed a limit.  What made this conceptually simple was
> the idea that each character individually has its own properties.
> 
> If we add a more powerful method of attaching non-text data to
> buffers, the design phase should include specifying precisely how they
> would behave when copying text between buffers and strings.

It isn't just about text properties and overlays.  Even simple C-n
requires us to march all the way between the position of point to the
position that's below it on the screen.  We must process each and
every buffer position in-between, and perform a large portion of what
redisplay does (specifically, its layout calculations, which require
the metrics of every character we traverse) to find the character that
is directly below point, so that we could move point there.  All this
because buffer text is a single unstructured string of bytes.  If we
want to speed up redisplay in such situations, we must get some help
from the buffer text itself, so as not to have to redo all those
layout calculations each and every time we need to convert buffer
positions to screen coordinates or back.

Re: Emacs design and architecture (was: Shrinking the C core)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If we add a more powerful method of attaching non-text data to
  > > buffers, the design phase should include specifying precisely how they
  > > would behave when copying text between buffers and strings.

  >   All this
  > because buffer text is a single unstructured string of bytes.  If we
  > want to speed up redisplay in such situations, we must get some help
  > from the buffer text itself, so as not to have to redo all those
  > layout calculations each and every time we need to convert buffer
  > positions to screen coordinates or back.

I think we are looking at the same sort of idea
from two different angles.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Emanuel Berg]

Lynn Winebarger wrote:

> Instead of discussing porting emacs to CL, why don't people
> work on porting the compiler techniques used in CL to emacs?

Absolutely, that is what one should do!

And with cl-lib and SLIME and other tools, Emacs-CL
integration has reached impressive results already.

What I understood from the previous discussion the speed
advantage for SBCL was to a large degree because of explicit
optimizations the programmer would put in the CL source.
Those are perhaps not so acute to bring over to Elisp, since
such use in an editor-environment probably would be limited.
However I think it would still be interesting and useful,
since we could use them in libraries and vanilla Emacs Elisp
source. In time, even Joe Elisp hackers would use them
directly, in their .emacs files.

Also the speed advantage (and whole issue of Elisp being slow)
is likely to diminish because of the very impressive native
compilation feature, which has showed great results while
still being at an early stage. Native compilation will also
benefit from improvements in compiler design and future
computer architecture progress.

So IIUC the only big thing SBCL has, and we don't, is
multi-threading. Bummer, but it is what it is.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
>   > > is a non-starter.
>
>   > Can we not even talk about it?
>
> There are people who seem to be pressuring to rewrite GNU Emacs in
> Common Lisp, and they kee pushing for it.  They keeo talking about the
> idea, and it feels like a demand.  I get the feeling that they think
> that by continuing to discuss this as if it were a real option they
> hope to _make_ us take the option seriously.  That is feels like pressure.
>
> Would those would like to discuss that plseae take the discssion off
> emacs-devel?  Then it will not be pressure.

Nobody is pressuring you or demanding anything. And those "they" are
real human beings you can address directly :).

Never mind; it is OK; I'll leave so I don't disturb the order in the
Church :-).

I wish you all best.

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
  > is a non-starter.

  > It would be an enormouse job -- including rewriting the Emacs Lisp
  > Referance Manual.

Also, there are aspects of Common Lisp which i rejected as clumsy and
best avoided.  All those keyword arguments!  I intentionally excluded
tham from Emacs and I am not going to let them in.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

> Also, there are aspects of Common Lisp which i rejected as
> clumsy and best avoided. All those keyword arguments!
> I intentionally excluded tham from Emacs and I am not going
> to let them in.

Certainly not indispensible but they are in, in `cl-defun'.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
>   > is a non-starter.
>
>   > It would be an enormouse job -- including rewriting the Emacs Lisp
>   > Referance Manual.
>
> Also, there are aspects of Common Lisp which i rejected as clumsy and

With all respect to you, but sound to me like an emotional argument, not
a rational one.

I don't know why you reject them as clumsy, but why does it matter to us
if a tool includes features we don't use? I don't know if I understand
it correctly, but CL was made so other Lisps can be abstracted on top of
it; so we use those features to abstract stuff on top of those lengthy
verbose functions? In other words we can hide those keyword params
behind elispy abstractions if we don't like them?

> best avoided.

Why are they best avoided? Is there some technical reason or is it
psychological?

> best avoided.  All those keyword arguments!  I intentionally excluded
> tham from Emacs and I am not going to let them in.

I don't want to be impolite, but they are already in; via cl-lib.el.

Re: Shrinking the C core

[Alan Mackenzie]

Hello, Arthur.

On Wed, Sep 06, 2023 at 07:04:43 +0200, Arthur Miller wrote:
> Richard Stallman <rms@gnu.org> writes:

> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> >   > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
> >   > is a non-starter.

> >   > It would be an enormouse job -- including rewriting the Emacs Lisp
> >   > Referance Manual.

> > Also, there are aspects of Common Lisp which i rejected as clumsy and

> With all respect to you, but sound to me like an emotional argument, not
> a rational one.

It's a rational argument expressed in emotional terms for simplicity.

> I don't know why you reject them as clumsy, but why does it matter to us
> if a tool includes features we don't use?

It matters a very great deal.  In practice you cannot avoid "using" these
features if you have to understand or debug somebody else's code.

> I don't know if I understand it correctly, but CL was made so other
> Lisps can be abstracted on top of it; so we use those features to
> abstract stuff on top of those lengthy verbose functions? In other
> words we can hide those keyword params behind elispy abstractions if we
> don't like them?

That's complexity and bloat.  Far better not to have them in the first
place.

> > best avoided.

> Why are they best avoided? Is there some technical reason or is it
> psychological?

It's because they are not needed.  Bear in mind, Common Lisp is a massive
language, much like C++ or PL/1 or Algol-68.  With such a language,
nobody uses all of it (it's just too big), but everybody has her own
personal subset.  This creates difficulty when somebody else has to
understand that first hacker's code.

CL is a niche language; it has not captured the hacker mindset.  I think
it is just too big and too unwieldy.

> > best avoided.  All those keyword arguments!  I intentionally excluded
> > tham from Emacs and I am not going to let them in.

> I don't want to be impolite, but they are already in; via cl-lib.el.

Yes.  There was a time not so long ago when cl.el was banned from use in
our Lisp code, except for at compile time.  Our Emacs Lisp was small,
simple to understand, and easy to learn.  Now things in cl-lib.el get
used as if they are just a normal part of Emacs Lisp.  Our language is
thus MUCH more difficult to understand, perhaps by a factor of somewhere
between 3 and 10.  When perusing even established parts of Emacs I groan
inwardly every time I encounter one of these needless cl-lib features.
It stops me dead, forcing me to consult doc strings (which are often
missing and often inadequate even when they are present) or even manuals.

Were we to rewrite Emacs in Common Lisp, these things would get worse
fairly quickly.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Shrinking the C core

[Emanuel Berg]

Alan Mackenzie wrote:

>> I don't know why you reject them as clumsy, but why does it
>> matter to us if a tool includes features we don't use?
>
> It matters a very great deal. In practice you cannot avoid
> "using" these features if you have to understand or debug
> somebody else's code.

But it is up to him/her how [s]he writes his/her code.

>> I don't know if I understand it correctly, but CL was made
>> so other Lisps can be abstracted on top of it; so we use
>> those features to abstract stuff on top of those lengthy
>> verbose functions? In other words we can hide those keyword
>> params behind elispy abstractions if we don't like them?
>
> That's complexity and bloat. Far better not to have them in
> the first place.

Not everyone is a minimalist and certainly Emacs and Elisp
allow for maximalists as well.

>>> best avoided. All those keyword arguments! I intentionally
>>> excluded tham from Emacs and I am not going to let
>>> them in.
>>
>> I don't want to be impolite, but they are already in; via
>> cl-lib.el.
>
> Yes. There was a time not so long ago when cl.el was banned
> from use in our Lisp code, except for at compile time.
> Our Emacs Lisp was small, simple to understand, and easy to
> learn. Now things in cl-lib.el get used as if they are just
> a normal part of Emacs Lisp.

They are a normal part of Emacs Lisp by definition. Check out
cl-lib.el and see if you can find any CL there. This file is
part of GNU Emacs!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Thu, Sep 07, 2023 at 01:03:28AM +0200, Emanuel Berg wrote:

[Setting mail-followup-to, in the hopes to redirect this
thread to a more appropriate place]

> Alan Mackenzie wrote:
> 
> >> I don't know why you reject them as clumsy, but why does it
> >> matter to us if a tool includes features we don't use?
> >
> > It matters a very great deal. In practice you cannot avoid
> > "using" these features if you have to understand or debug
> > somebody else's code.
> 
> But it is up to him/her how [s]he writes his/her code.

This is absolutely naïve. A language (a computer language,
too) is a communication device, and therefore inherently
a social construct.

There's always a tension (and different languages solve this
in different ways) between allowing too much width (and thus
creating different, possibly disjoint subcultures (cf. C++)
and unifying too much, thus suffocating possible creativity.

There's no (technically) "right solution" to this social
question.

In the case of Emacs Lisp, we'll have to accept that people
like Richard and Eli carry more weight in those questions
than you and me, be it because they've put orders of magnitude
more of work in there than us.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Shrinking the C core

[Emanuel Berg]

tomas wrote:

>>> It matters a very great deal. In practice you cannot avoid
>>> "using" these features if you have to understand or debug
>>> somebody else's code.
>> 
>> But it is up to him/her how [s]he writes his/her code.
>
> This is absolutely naive. A language (a computer language,
> too) is a communication device, and therefore inherently
> a social construct.

What are you talking about, people are allowed to use parts of
Elisp, but not other parts like cl-lib, that makes them naive?

At the same time they are advanced enough to use the
supposedly super-complex features of cl-lib?

> In the case of Emacs Lisp, we'll have to accept that people
> like Richard and Eli carry more weight in those questions
> than you and me, be it because they've put orders of
> magnitude more of work in there than us.

Absolutely not, everyone is allowed to use Elisp in any way
they want including parts of it that Richard and Eli or anyone
else for that matter don't like, if they or someone else at
some later stage choose to maintain that same code, that has
absolutely nothing to do with it and that hypothetical
situation is something they have chosen to enter and, if so,
it is their problem if they don't like that code's reliance on
certain parts of Elisp.

#@%$&!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Po Lu]

Alan Mackenzie <acm@muc.de> writes:

> Yes.  There was a time not so long ago when cl.el was banned from use in
> our Lisp code, except for at compile time.  Our Emacs Lisp was small,
> simple to understand, and easy to learn.  Now things in cl-lib.el get
> used as if they are just a normal part of Emacs Lisp.  Our language is
> thus MUCH more difficult to understand, perhaps by a factor of somewhere
> between 3 and 10.  When perusing even established parts of Emacs I groan
> inwardly every time I encounter one of these needless cl-lib features.
> It stops me dead, forcing me to consult doc strings (which are often
> missing and often inadequate even when they are present) or even manuals.

I agree.  The use of generic functions for window system initialization
irks me to no end.

Re: Shrinking the C core

[Emanuel Berg]

Po Lu wrote:

>> Yes. There was a time not so long ago when cl.el was banned
>> from use in our Lisp code, except for at compile time.
>> Our Emacs Lisp was small, simple to understand, and easy to
>> learn. Now things in cl-lib.el get used as if they are just
>> a normal part of Emacs Lisp. Our language is thus MUCH more
>> difficult to understand, perhaps by a factor of somewhere
>> between 3 and 10. When perusing even established parts of
>> Emacs I groan inwardly every time I encounter one of these
>> needless cl-lib features. It stops me dead, forcing me to
>> consult doc strings (which are often missing and often
>> inadequate even when they are present) or even manuals.
>
> I agree. The use of generic functions for window system
> initialization irks me to no end.

The cl-lib features are awesome, especially `cl-loop' but
other come to mind as well. It isn't much more complicated
than anything else and provides much more expressive power
to Elisp.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The cl libraries of Emacs Lisp define functions that are not standard
parts of Emacs Lisp.  They are not documented in the Emacs Lisp
Reference Manual -- for good reason (we don't want to write that
documentation or have a commitment to maintain ever after later).  As
a consequence, code which uses those constructs is extra work to
understand and to maintain.

Therefore, it is better to avoid using them, when that is possible.
Especially in important or central parts of Emacs.

If you want to use them in your own private code, that's no problem
for anyone else -- do what you feel like, there.  Using them in a
not-terribly-vital package that you maintain and some others use is
not a big drawback.  However, using them in code others need to
maintain makes maintaining Emacs harder.

I do not say "impossible", but it is worth avoiding.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 10 Sep 2023 20:43:01 -0400
> 
> The cl libraries of Emacs Lisp define functions that are not standard
> parts of Emacs Lisp.  They are not documented in the Emacs Lisp
> Reference Manual -- for good reason (we don't want to write that
> documentation or have a commitment to maintain ever after later).  As
> a consequence, code which uses those constructs is extra work to
> understand and to maintain.

This particular aspect -- documentation -- is somewhat blurred these
days: you will find some "cl-*" functions in the ELisp manual.
(Typing "i cl- TAB" in the ELisp manual produces 5 completion
candidates.)  So we do document some cl functions when we find that
useful.

> If you want to use them in your own private code, that's no problem
> for anyone else -- do what you feel like, there.  Using them in a
> not-terribly-vital package that you maintain and some others use is
> not a big drawback.  However, using them in code others need to
> maintain makes maintaining Emacs harder.
> 
> I do not say "impossible", but it is worth avoiding.

This message is in many cases met with opposition and thus is not easy
to send in practice.

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This particular aspect -- documentation -- is somewhat blurred these
  > days: you will find some "cl-*" functions in the ELisp manual.
  > (Typing "i cl- TAB" in the ELisp manual produces 5 completion
  > candidates.)

That's few exceptions out of many cl- functions.  It seems that what I
said is still nearly 100% correct.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

> The cl libraries of Emacs Lisp define functions that are not
> standard parts of Emacs Lisp.

What is the definition of standard Emacs Lisp, stuff that
don't have to be `require'd?

Because cl-lib is part of vanilla Emacs, you don't have to
install it, even from GNU ELPA or any such other
official source. It is there.

> They are not documented in the Emacs Lisp Reference Manual
> -- for good reason (we don't want to write that
> documentation or have a commitment to maintain ever after
> later).

cl-lib should of course be documented like everything else,
and it is.

But just as an experiment, I run (checkdoc-current-buffer t)
in the cl-lib.el buffer and it had 14 remarks. One could do
that for the other cl-lib files as well.

cl-lib.el:133: Argument ‘keys’ should appear (as KEYS) in the doc string
cl-lib.el:160: All variables and subroutines might as well have a documentation string
cl-lib.el:166: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:226: Arguments occur in the doc string out of order
cl-lib.el:258: Argument ‘specs’ should appear (as SPECS) in the doc string
cl-lib.el:357: Argument ‘cl-func’ should appear (as CL-FUNC) in the doc string
cl-lib.el:450: Argument ‘rest’ should appear (as REST) in the doc string
cl-lib.el:488: Argument ‘cl-item’ should appear (as CL-ITEM) in the doc string
cl-lib.el:501: Argument ‘cl-new’ should appear (as CL-NEW) in the doc string
cl-lib.el:510: All variables and subroutines might as well have a documentation string
cl-lib.el:539: All variables and subroutines might as well have a documentation string
cl-lib.el:577: Probably "returns" should be imperative "return"

GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0)
of 2023-09-08 [commit a2f977d94e0356c7414876e988adedd2ab7b52f2]

> code which uses those constructs is extra work to understand
> and to maintain.

But then what are we gonna use instead of `cl-decf',
`cl-incf', `cl-labels', `cl-map' and so on?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Manuel Giraud via Emacs development discussions.]

Po Lu <luangruo@yahoo.com> writes:

> Alan Mackenzie <acm@muc.de> writes:
>
>> Yes.  There was a time not so long ago when cl.el was banned from use in
>> our Lisp code, except for at compile time.  Our Emacs Lisp was small,
>> simple to understand, and easy to learn.  Now things in cl-lib.el get
>> used as if they are just a normal part of Emacs Lisp.  Our language is
>> thus MUCH more difficult to understand, perhaps by a factor of somewhere
>> between 3 and 10.  When perusing even established parts of Emacs I groan
>> inwardly every time I encounter one of these needless cl-lib features.
>> It stops me dead, forcing me to consult doc strings (which are often
>> missing and often inadequate even when they are present) or even manuals.
>
> I agree.  The use of generic functions for window system initialization
> irks me to no end.

Not Emacs related anymore, but I'd like to hear your point of view on
this matter.  It seems to me that a window system is a good candidate
for genericity.
-- 
Manuel Giraud

Re: Shrinking the C core

[Po Lu]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

> Not Emacs related anymore, but I'd like to hear your point of view on
> this matter.  It seems to me that a window system is a good candidate
> for genericity.

I was citing the use of CL generics for functions such as
`window-system-initialization', which should be replaced with a cond
switch on the value of `window-system'.

Re: Shrinking the C core

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Alan Mackenzie <acm@muc.de>,  Arthur Miller <arthur.miller@live.com>,
>  Richard Stallman <rms@gnu.org>,  emacs-devel@gnu.org
> Date: Thu, 07 Sep 2023 17:20:45 +0800
> 
> I was citing the use of CL generics for functions such as
> `window-system-initialization', which should be replaced with a cond
> switch on the value of `window-system'.

To be fair, that would make it harder to have each specific
implementation of the initialization functions on its own file, which
is specific for that window-system.

Re: Shrinking the C core

[Arthur Miller]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Arthur.
>
> On Wed, Sep 06, 2023 at 07:04:43 +0200, Arthur Miller wrote:
>> Richard Stallman <rms@gnu.org> writes:
>
>> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> > [[[ whether defending the US Constitution against all enemies,     ]]]
>> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>> >   > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
>> >   > is a non-starter.
>
>> >   > It would be an enormouse job -- including rewriting the Emacs Lisp
>> >   > Referance Manual.
>
>> > Also, there are aspects of Common Lisp which i rejected as clumsy and
>
>> With all respect to you, but sound to me like an emotional argument, not
>> a rational one.
>
> It's a rational argument expressed in emotional terms for simplicity.

What you basicaly say: it somehow is a summarization of some more
advanced/complicated reasons which can't be explained, but have to be
expressed in the term "I don't like it" for the simplicity?

I don't think it is a rational argument, but I explicitly didn't want to
speculate why RMS feels so, but asked him to clarify for pure respect
for him as a person. You can try to teach me why is it a rational
argument and why it has to be expressed in emotional terms for
simplicity, and I promise I'll try to understand it as best as I can,
but I have hard time to see such an explanation, because the argument is
truly just an opinion.

CL for sure has things that in retrospect can be debated about, but I
don't think keyword arguments are one of those.

Also note that keyword arguments, or at least similar principle is used
in other parts of Emacs, not just cl-lib; for example look at
define-minor-mode. While define-minor-mode macro does not use "&key" as
an explicit keyword, the arguments to it are a plist which basicaly
gives you the very same effect. The "new" defvar-keymap is another
example.

Keyword arguments are undeniably useful because they let us omit
arguments we are not interested in, and type only the ones we are
interested in. I don't see how they complicate a Lisp by much; we have
&rest and &optional so what is problem having &key?

In my personal opinion it is by far more clumsy to have two defun
macros, and to prefix one with cl- for no good reason at all; and than
have to explain for users why they are two different versions, how they
differ, when they should use each, why would they like to use one or
another etc. If ordinary "defun" was simply upgraded to act as cl-defun,
potential user dilemas and explanations would be avoided.

>> I don't know why you reject them as clumsy, but why does it matter to us
>> if a tool includes features we don't use?
>
> It matters a very great deal.  In practice you cannot avoid "using" these
> features if you have to understand or debug somebody else's code.

In practice most of code will use some common part of the language, and
relatively little code will use some specialized parts. In a big
language that targets many different kind of developers, there will
always be features that are needed by some developers but not by
all. Falacy in your, by now very tired and wornout argument, is the
assumption that everyone will use everything in the language all the
time, which certainly isn't the case.

Interestingly, yesterday why commuting home after the work, I have heard
a talk by Stroustrup, quite recent just a few week old:

https://www.youtube.com/watch?v=eo-4ZSLn3jc

The talk is more or less about that very argument: why is C++ so big and
why are there so many features in c++. (If the link gets removed search
on Stroustrup C++ 2023). I think that was one of better Stroustrup's
talks, definitely worth watching.

There is also an incredibly inspirational talk by Steel (search on
"Growing the Language"); I have seen it just a few weeks ago myself:

https://www.youtube.com/watch?v=lw6TaiXzHAE

Observe that Steel gave us the "most minimal" and "elegant" Lisp as
Schemer's like to think of Scheme, which is probably important to
remember in the context of this discussion. If you don't watch it, the
essence is in this summary, and in my opinion is really a clever
insight:

"Over the last quarter-century Guy Steele has been convinced that trying
to design a complete and perfect programming language is the worst thing
you can do. A programming language (including its associated libraries)
must grow over time as its user community and its development community
grow. This is a different situation from 25 years ago, when all such
communities were relatively small. The difference is a problem of
scale. As a result, programming language design now and in the future is
necessarily as much a matter of social engineering as technical
engineering and must rely more on a set of general principles than on a
set of specific technical decisions."

>> I don't know if I understand it correctly, but CL was made so other
>> Lisps can be abstracted on top of it; so we use those features to
>> abstract stuff on top of those lengthy verbose functions? In other
>> words we can hide those keyword params behind elispy abstractions if we
>> don't like them?
>
> That's complexity and bloat.  Far better not to have them in the first
> place.

Complexity and bloat by which metric?

Emacs has long time ago exceeded CL in numbers of "core functions". By
the time I have extracted all Lisp defuns from the C core, a couple of
months ago or so, it was ~1800 functions in C core. That is by far many
more than what CL standard defines. I know that some of Emacs devs are
lurking on Reddit behind pseudonims, I don't know if you are one of
them, but there are also quite many users who consider Emacs to be quite
bloated and would like many of some older parts to be expunded out of
the core.

Anyway, by the nature of Lisp, there are very few actual language
concepts, and there are quite few additional concepts that CL offers
that need support of the system that are not found in Emacs; but most of
additions to the language are actually looking exactly like the language
itself. The nice thing is that, when you write a defun in Lisp, it
becomes part of the language itself, unlike languages like Java, C, C++
etc where users can not extend the language itself since the syntax has
to be encoded in the compiler. In other words, it means, that even if
you don't want it, people can extend it the way they like it, and cl.el
was a proof.

>> > best avoided.
>
>> Why are they best avoided? Is there some technical reason or is it
>> psychological?
>
> It's because they are not needed.  Bear in mind, Common Lisp is a massive

I don't know if you agree with me or not, but in order to drive from
Nuremberg to Berlin you certainly don't need AC in a car; but on a hot
summer day, with 40 degrees (celsius) it is very nice to have AC in the
car, isn't it? Sure, keyword arguments are not needed; but they are
handy from time to time. Optional or rest arguments are not needed
either, but they too are nice to have sometimes, aren't they? What
makes keyword arguments more "clumsy" then? 

To be honest to you, I really didn't want to answer this mail, because I
see just very same arguments I have seen many, many times before. I had
very much, very same sentiment about C++ as you, and I didn't even know
much of CL at all; I just thought it was big and bloated and didn't want
to use it, untill not so long time ago. Untill perhaps a year or two
ago, when I invested some time in learning more about CL and tried to
understand why things are as they are, not just in CL, but in general;
C++ very much included.

> It's because they are not needed.  Bear in mind, Common Lisp is a massiveI
> language, much like C++ or PL/1 or Algol-68.  With such a language,
> nobody uses all of it (it's just too big), but everybody has her own
> personal subset.  This creates difficulty when somebody else has to
> understand that first hacker's code.

I totally understand you; but I believe this is a bit misguided. For the
first, for the same argument as above, nobody uses the entire language
because they don't need it; not because it is too big. Some parts target
certain specialized domains which are needed by some groups and not by
everyone. I think that is more true for C++ and less for CL, because CL
is not that massive as you are trying to make it. CL is also a child of its
time, just like Emacs, and some parts are probably in need of a
revision, but it certainly stands the proof of time when it comes to
core language principles.

If we look at this:

(defun directory-files (directory &optional full match nosort count)
...)

When you call this function in the code; how easy is to remember the
order and number of arguments? Was it 5 or 6 arguments? I had fortune to
contribute the idea for the 5th argument, and yet I typed by a misstake
the wrong number:

CL-USER> (directory-files "./" nil nil nil nil 5)
; Debugger entered on #<SB-INT:SIMPLE-PROGRAM-ERROR "invalid number of arguments: ~S" {1001E92C63}>
[1] CL-USER> 
; Evaluation aborted on #<SB-INT:SIMPLE-PROGRAM-ERROR "invalid number of arguments: ~S" {1001E92C63}>
CL-USER> (directory-files "./" nil nil nil 5)
("alloc.lisp" "buffer.lisp" "callint.lisp" "callproc.lisp" "casefiddle.lisp")
CL-USER>

If I only change &optional to &key:

(defun directory-files (directory &key full match nosort count) ... )

I can now type:

CL-USER> (directory-files "./" :count 5)
("#dired.lisp#" ".#dired.lisp" "alloc.lisp" "buffer.lisp" "callint.lisp")
CL-USER> 

How "clumsy" is that? Honestly, I am not trying to be PITA or devil's
advocate or anything like that; but I think that everyone will agree
that the second one is both nicer, less error prone to type and puts
less cognitive load on users to remember or lookup the information.

Just as a remark: by just having it in CL I was able to just go to the
code, and change "optional" for "key", C-x C-e in Sly and I changed the
implementation of "directory-files", nice? For the record if you want to
try it (just a fast hack for the demonstration purpose, needs cl-ppcre lib):

(defun directory-files (directory &key full match nosort count)
  (let ((files
          (if full
              (mapcar #'namestring (uiop:directory-files directory))
              (mapcar #'file-namestring (uiop:directory-files directory)))))
    (when match
      (let ((scanner (ppcre:create-scanner match))
            matches)
        (dolist (filename files)
          (when (ppcre:scan scanner filename)
            (push filename matches)))
        (setf files matches)))
    (unless nosort
      (setq files (sort files #'string-lessp)))
    (when (and (numberp count) (> count 0))
      (when (> count (length files))
        (setf count (length files)))
         (setf files (subseq files 0 count)))
    files))

Observe also that I agree with you; keyword arguments are not needed,
but are very nice to have.

> CL is a niche language; it has not captured the hacker mindset.  I think
> it is just too big and too unwieldy.
>
>> > best avoided.  All those keyword arguments!  I intentionally excluded
>> > tham from Emacs and I am not going to let them in.
>
>> I don't want to be impolite, but they are already in; via cl-lib.el.
>
> Yes.  There was a time not so long ago when cl.el was banned from use in
> our Lisp code, except for at compile time.  Our Emacs Lisp was small,
> simple to understand, and easy to learn.  Now things in cl-lib.el get
> used as if they are just a normal part of Emacs Lisp.  Our language is
> thus MUCH more difficult to understand, perhaps by a factor of somewhere
> between 3 and 10.  When perusing even established parts of Emacs I groan
> inwardly every time I encounter one of these needless cl-lib features.
> It stops me dead, forcing me to consult doc strings (which are often
> missing and often inadequate even when they are present) or even manuals.

"MUCH more difficult" for whom in which sense?

What you are telling is that "good old times were so much better" which
is just an emotional argument. You are actually just reinforcing the
same sentiment that RMS expressed. I shared that sentiment myself, but I
think it is misguided.

We can certainly speak about "old ways", let us take the or-idiom or how
should I call it: initialization of the default value for optional arguments:

(defun foo (&optional who-am-I)
  (let ((who-am-I (or who-am-I "foo")))
    (message "I am %s." who-am-I)))

Is that really better than typing:

(cl-defun foo (&optional (who-am-I "foo"))
  (message "I am %s." who-am-I))

The user has to learn the idiom, which uses operator "or" to perform
something that visually has nothing to do with the intention of the
code, and also has to type the additional let-form each and every
time. Than the users who are not familiar with the idiom will perhaps
come up with their own version, using setq or some other thing, and you
will really have to think what the user wanted to say with their code if
you had to debug it. Is it better than seing an initialiser and knowing
directly what is the default value and everyone using uniform syntax?

There was a discussion, perhaps a couple of years ago, I think on Emacs
help list, I don't remember the detials, but I think something about
car, cdr, cdar, cddr & co, and using "nth" instead of them. Monnier said a
clever thing there: using those old functions is like writing in
assembler, but without the benefit of additional speed (something in
that sense). I think he was more than correct there. It is much more
clear to say (nth N list), than to chain cars and cdrs together.

A bit further, that really is about abstractions. We can for sure do
lots of stuff manually, without higher-level abstractions; but higher
level abstractions, like using initializer in a defun arguments, help us
convey the meaning more clearly and concize, we can rationalize about
abstractions, and they are probably easier to remember than teaching
people how to use some common pattern or idiom.

I also happened to read the CMUCL manual just yesterday. They have a
nice writing about using structures instead of lists in CL:

"Even if structures weren't more efficient than other representations,
structure use would still be attractive because programs that use
structures in appropriate ways are much more maintainable and robust
than programs written using only lists. For example:
     

    (rplaca (caddr (cadddr x)) (caddr y))

could have been written using structures in this way:
     

    (setf (beverage-flavor (astronaut-beverage x)) (beverage-flavor y))

The second version is more maintainable because it is easier to
understand what it is doing."

https://cmucl.org/downloads/doc/cmu-user-2010-05-03/compiler-hint.html#toc189

> Were we to rewrite Emacs in Common Lisp, these things would get worse
> fairly quickly.

I think you are very, very wrong about that one; on the contrary we
would get some tools that Emacs is lacking to manage the code
complexity, but I don't think they are actually applicable on the
existing code, so they don't matter much. But things certainly wouldn't
get worse. That is just my opinion of course, you can think I am an
idiot and I can be wrong, but at least give me a reason to change my
mind that isn't based on subjective opinion.

Re: Shrinking the C core

[Gerd Möllmann]

Arthur Miller <arthur.miller@live.com> writes:

> you can think I am an idiot and I can be wrong,

I guess it won't surprise anyone that I'm such an idiot as well :-).

Re: Shrinking the C core

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 297 bytes --]

On Fri, Sep 8, 2023, 08:35 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

> Arthur Miller <arthur.miller@live.com> writes:
>
> > you can think I am an idiot and I can be wrong,
>
> I guess it won't surprise anyone that I'm such an idiot as well :-).
>

Count another idiot here!

>

[-- Attachment #2: Type: text/html, Size: 880 bytes --]

RE: [External] : Re: Shrinking the C core

[Drew Adams]

FWIW, +1 for your mail, Arthur -
pretty much each of the points you made.
___

As one who's used CL (long ago) but is
no expert about it or Elisp or Lisp
generally, I happen to agree about the
usefulness of keyword args _for users_.

I can't really speak to implementation,
compilation, maintenance, etc., all of
which are of course also important.

AFAIK there has never been a real, open,
serious discussion about keyword args
for Elisp.  And (I think) I've always
respected the decision to not bring up
the question.  But I do appreciate it
being at least _presented_, if not put
on the table for outright discussion.

In general, I like that Richard speaks
up and decides, and I generally agree
with his judgments as helmsman.  But on
this one my own experience tells me
something different.

Not that I have any interesting opposing
arguments.  Nor do I want to argue about
this.  But my experience with CL has led
me to appreciate the ease and handiness
of using keyword args.

Argument order, and the need to provide
all args up through the last optional
one you really want to provide, are an
unnecessary burden on human readers of
code.  Writers too, of course, but it's
more important that code be clear for
readers (including writers as readers).

Unnecessary burden.  So Occam asks "Why
then?".  In his role as _implementer_ of
a language Occam might ask "Why bother
with keywords args?"  But in his role as
user I think his question would be "Why
always need to know & respect arg order?"

I also haven't noticed that having named
arguments is detrimental to code that
analyses or generates code.  I have some,
but less, experience with that - maybe
it's something to consider; dunno.
___

That's likely all I'll say about this.
Just one opinion.  Keyword args can be
incredibly useful _for users_.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Drew Adams <drew.adams@oracle.com> writes:

> FWIW, +1 for your mail, Arthur -
> pretty much each of the points you made.
> ___
>
> As one who's used CL (long ago) but is
> no expert about it or Elisp or Lisp
> generally, I happen to agree about the
> usefulness of keyword args _for users_.

Interestingly, how things can be percieved wrongly over the wire.
I always percieved you as an CL veteran and expert :).

> I can't really speak to implementation,
> compilation, maintenance, etc., all of
> which are of course also important.
>
> AFAIK there has never been a real, open,
> serious discussion about keyword args
> for Elisp.  And (I think) I've always
> respected the decision to not bring up
> the question.  But I do appreciate it
> being at least _presented_, if not put
> on the table for outright discussion.
>
> In general, I like that Richard speaks
> up and decides, and I generally agree
> with his judgments as helmsman.  But on
> this one my own experience tells me
> something different.
>
> Not that I have any interesting opposing
> arguments.  Nor do I want to argue about
> this.  But my experience with CL has led
> me to appreciate the ease and handiness
> of using keyword args.

I really didn't want to argue about keyword arguments either, but
Richard brought them up himself. Like you, and probably everyone else
who is familiar with Richards writing through the years, through his
articles and this very mailing list, I am aware of the elephant in the
room, but honestly, I think his friends, Eli and others, are too kind and
would do Richard more favor if they told him when he is dead wrong and
made him to challenge his own view on some questions from time to
time. This reminded me of John Cleese and Faulty Towers, no idea if you
had that in States, but I think every European here knows which episode
I am thinking of at the moment.

> Argument order, and the need to provide
> all args up through the last optional
> one you really want to provide, are an
> unnecessary burden on human readers of
> code.  Writers too, of course, but it's
> more important that code be clear for
> readers (including writers as readers).
>
> Unnecessary burden.  So Occam asks "Why
> then?".  In his role as _implementer_ of
> a language Occam might ask "Why bother
> with keywords args?"  But in his role as
> user I think his question would be "Why
> always need to know & respect arg order?"
>
> I also haven't noticed that having named
> arguments is detrimental to code that
> analyses or generates code.  I have some,
> but less, experience with that - maybe
> it's something to consider; dunno.
> ___
>
> That's likely all I'll say about this.
> Just one opinion.  Keyword args can be
> incredibly useful _for users_.

Thank you, for wording this so concisely.  You are much better with
words than me. I am not a good writer, not even in my own language, so
mine explanations and arguments are always long and round-about for some
reason.

Best regards

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Alan Mackenzie <acm@muc.de>,  Richard Stallman <rms@gnu.org>,
>  "luangruo@yahoo.com" <luangruo@yahoo.com>,  "emacs-devel@gnu.org"
>  <emacs-devel@gnu.org>
> Date: Sat, 09 Sep 2023 13:55:46 +0200
> 
> I really didn't want to argue about keyword arguments either, but
> Richard brought them up himself. Like you, and probably everyone else
> who is familiar with Richards writing through the years, through his
> articles and this very mailing list, I am aware of the elephant in the
> room, but honestly, I think his friends, Eli and others, are too kind and
> would do Richard more favor if they told him when he is dead wrong and
> made him to challenge his own view on some questions from time to
> time.

This comes out quite condescending, FYI.  You probably didn't mean
that, but the result is not nice anyway.  One more reason to stop this
dispute about personal stylistic preferences, which was futile from
the get-go.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Alan Mackenzie <acm@muc.de>,  Richard Stallman <rms@gnu.org>,
>>  "luangruo@yahoo.com" <luangruo@yahoo.com>,  "emacs-devel@gnu.org"
>>  <emacs-devel@gnu.org>
>> Date: Sat, 09 Sep 2023 13:55:46 +0200
>> 
>> I really didn't want to argue about keyword arguments either, but
>> Richard brought them up himself. Like you, and probably everyone else
>> who is familiar with Richards writing through the years, through his
>> articles and this very mailing list, I am aware of the elephant in the
>> room, but honestly, I think his friends, Eli and others, are too kind and
>> would do Richard more favor if they told him when he is dead wrong and
>> made him to challenge his own view on some questions from time to
>> time.
>
> This comes out quite condescending, FYI.  You probably didn't mean
> that, but the result is not nice anyway.

No, I certainly didn't mean as patronizing if that is what "condecending".

> that, but the result is not nice anyway.  One more reason to stop this
> dispute about personal stylistic preferences, which was futile from
> the get-go.

I agree it is futile.

No problems; thanks for telling me.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> > FWIW, +1 for your mail, Arthur -
> > pretty much each of the points you made.
> > ___
> >
> > As one who's used CL (long ago) but is
> > no expert about it or Elisp or Lisp
> > generally, I happen to agree about the
> > usefulness of keyword args _for users_.
> 
> Interestingly, how things can be percieved wrongly over the wire.
> I always percieved you as an CL veteran and expert :).

Nope.  Just a long-ago user.  Being around a
long time can sometimes give the impression
of being a veteran.

> I think his friends, Eli and others, are too kind and
> would do Richard more favor if they told him when he
> is dead wrong 

I have no reason to think Richard is dead
wrong about whether Elisp should have
keyword args.  I just related my (good)
experience with them in CL, as one user,
long ago.

Elisp could of course go in many directions.
That it hasn't taken certain directions I'm
generally thankful for, FWIW.  It's not a
bad language, and not every proposal or
imagined change represents real progress. ;-)

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I implemented Common Lisp for the Lisp Machine in 1983.
That included the generic sequence functions with their
many keyword arguments.

In 1984 I wrote Emacs Lisp and decided to omit all that.
I have no regrets.  Their absence makes Emacs Lisp easier to use.

Keyword arguments are fine for an unusual, heavyweight operation such
as creating a frame.  The sort of function whose doc you need to look
up again each time you use it.  When something is as cumbersome as
that, the question of how to pass the arguments is a pile of minor
details that you'll see when you look it up.

But it's a pain in the neck to make simple everyday functions such as
`append' and `length' that cumbersome just for the sake of following a
general system.  Part of this system is that the default argument
values follow the system, rather than following the traditional Lisp
functions -- so getting the traditional behavior requires actually
specifying the keyword arguments to override defaults, almost every
time.

Switching to Common Lisp would cause those painful incompatibilities.
It is unacceptable.

Defining the CL functions with a name prefix is ok, because the
traditional Lisp functions are still available compatibly and we don't
have to use the CL functions.  This is what we have now.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[João Távora]

On Mon, Sep 11, 2023 at 1:40 AM Richard Stallman <rms@gnu.org> wrote:

> Keyword arguments are fine for an unusual, heavyweight operation such
> as creating a frame.  The sort of function whose doc you need to look
> up again each time you use it.  When something is as cumbersome as
> that, the question of how to pass the arguments is a pile of minor
> details that you'll see when you look it up.

It's possible that in 1984 there weren't any good at-point-documentation
systems that immediately show the available keyword arguments of a given
call being typed.  I was very young at the time, so I can't attest to it.

But nowadays such systems exist, including built-in, in Emacs.

But even in the absence of those conveniences, each family of functions
in CL, (say, the sequence manipulating functions) take standard sets
of keyword arguments (like :START, :END, :FROM-END, :KEY, :TEST).

This makes them very easy to remember and makes using e.g. CL's SORT
much more practical than Emacs Lisp 'sort'.

> But it's a pain in the neck to make simple everyday functions such as
> `append' and `length' that cumbersome just for the sake of following a
> general system.

APPEND and LENGTH in Common Lisp do not take keyword arguments,
(nor have they ever?) so I believe your argument is weakened by
criticizing a suggestion that no-one is offering.

Let's look at a traditional Elisp macro define-minor-mode.

According to the CVS history, you created this function in 1997, with
three positional optional arguments.  Then more and more arguments
came along.  Possibly due to the realization by Emacs developers
around 2000, already more than 20 years ago,  that positional arguments
are maintenance hazards, the macro now accepts keyword arguments
(and the positional arguments are actively discouraged).

Unfortunately, because cl-lib.el wasn't used for this particular change,
the aforementioned at-point documentation facilities aren't available,
which, to some of us, would appear just another manifestation of
Greenspun's 10th rule (which I assume everyone in this niche topic
is familar with).

Many other functions with a large amount of optional arguments
(completing-read comes to mind) would be much, much easier
to use with keyword arguments.  Without them, we find ourselves
wondering about how many nils to sprinkle before the argument
we want to pass.

João

RE: [External] : Re: Shrinking the C core

[Drew Adams]

I thought my previous post in this thread would
likely be my only such, but I'd like now to add a
point, FWIW.

Richard, you spoke about implementing support for
keyword args in ~CL (Lisp-machine Lisp code that
you wrote), and how you found their presence to be
a bother for implementers and (I think) users.

I mentioned that I appreciated _using_ keyword args
with CL, long ago.  I'll add that it was especially
in code that I wrote that I found them useful (as
opposed to their presence in standard CL code).
That is, for _users_ of my code (but including me,
while developing and testing).  And especially for
functions that allowed for more than a few args.

In a nutshell, with keyword args a single function
can replace multiple related/similar functions.

And when you have multiple things to juggle, it
helps to name them, and it helps to not need to
address them in a particular order.

Here's an analogy:

Imagine Lisp without &optional args.  You'd end by
defining more functions of the same "family".

IME, the same difference applies to the absence of
keyword args.  You end up defining more functions,
for the convenience of not having to specifying a
zillion nil args.  Plus you have to pay attention
to arg order.

Of course sometimes it makes sense to define a
separate function or two, to cover particular
common use cases.  But in general optional args
are handy, and so are keyword args, for the same
reason: many functions in one - a single name to
rule them all. ;-)

Now, if you always _had to_ use keyword args, so
you could _never_ just provide args without names
(in the proper order), then that would definitely
be an unnecessary bother.  We probably all agree
about that.

In Lisp all args to functions are evaluated, and
there really is no user-level dependence on the
order of their evaluation.  Given that, Occam says
that arg order doesn't matter, and there's no real
_need_ to specify args in an order, filling in nil
args as needed to get to the last optional arg you
need.

From a user point of view, being able to have the
option of specifying an arg by its position OR by
its name is just a plus - I don't see any downside.

From the point of view of a language implementer
things might be different, of course.

Re: [External] : Re: Shrinking the C core

[Tomas Hlavaty]

On Mon 11 Sep 2023 at 16:10, João Távora <joaotavora@gmail.com> wrote:
> This makes them very easy to remember and makes using e.g. CL's SORT
> much more practical than Emacs Lisp 'sort'.

not really, it seems that CL:SORT is a bad example

the CL spec defines sort like this:

   sort sequence predicate &key key => sorted-sequence
   stable-sort sequence predicate &key key => sorted-sequence

(btw why two functions and not extra stablep keyword argument?)

it could have been defined as:

   sort sequence predicate &optional key => sorted-sequence
   stable-sort sequence predicate &optional key => sorted-sequence

and the same could be done in elisp which would be backwards compatible

> Let's look at a traditional Elisp macro define-minor-mode.

the usual CL argument list does not seem to be able to express arguments
of such shape

it looks like whoever extended the original argument list did it
"weirdly" using custom ad-hoc single-use argument list parser.

> are maintenance hazards, the macro now accepts keyword arguments

in CL, the arguments would normally be in a list before body, something
like

   (define-minor-mode MODE ([KEYWORD VAL ... ]) [DOC] &rest BODY)

Re: [External] : Re: Shrinking the C core

[João Távora]

On Mon, Sep 11, 2023 at 9:37 PM Tomas Hlavaty <tom@logand.com> wrote:

> not really, it seems that CL:SORT is a bad example

The example was for comparing to Emacs's lisp 'sort', where
IMO it's much easier to do

  (cl-sort things-having-foo #'< :key #'foo)

than

  (sort things-having-foo (lambda (a b) (< (foo a) (foo b))))

As to the advantage of @key key vs &optional key it is -- in my
eyes,  at least -- that you already know its meaning from many other
functions.

> (btw why two functions and not extra stablep keyword argument?)

Why not? Maybe that makes passing sort and stable-sort to higher
order functions more practical?

> it could have been defined as:
>
>    sort sequence predicate &optional key => sorted-sequence
>    stable-sort sequence predicate &optional key => sorted-sequence
>
> and the same could be done in elisp which would be backwards compatible

Sure, but if according to your conjecture Elisp's grew a &optional
arg, why couldn't it grow a &key arg and be backwards compatible?

> > Let's look at a traditional Elisp macro define-minor-mode.
>
> the usual CL argument list does not seem to be able to express arguments
> of such shape
>
> it looks like whoever extended the original argument list did it
> "weirdly" using custom ad-hoc single-use argument list parser.

Quite likely the cl machinery wasn't available at the time...  But yes,
Greenspun's 10th.

> > are maintenance hazards, the macro now accepts keyword arguments
>
> in CL, the arguments would normally be in a list before body, something
> like
>
>    (define-minor-mode MODE ([KEYWORD VAL ... ]) [DOC] &rest BODY)

OK, but this is irrelevant.  This is a macro, not a function.
For all practical purposes it was extended with keyword arguments.
In fact you could have used cl-lib's cl-destructuring-bind.

João

Re: [External] : Re: Shrinking the C core

[Tomas Hlavaty]

On Mon 11 Sep 2023 at 22:10, João Távora <joaotavora@gmail.com> wrote:
> On Mon, Sep 11, 2023 at 9:37 PM Tomas Hlavaty <tom@logand.com> wrote:
>> not really, it seems that CL:SORT is a bad example
>
> The example was for comparing to Emacs's lisp 'sort', where
> IMO it's much easier to do
>
>   (cl-sort things-having-foo #'< :key #'foo)
>
> than
>
>   (sort things-having-foo (lambda (a b) (< (foo a) (foo b))))

That has nothing to do with &key.
It has everything to do with the lack of the KEY argument
and the KEY argument could as well be &optional instead of &key.


If you find lack of the key argument "much not easier", you can define
your own sort, something like:

   (defun sort3 (seq pred key)
     (sort seq (lambda (a b)
                 (funcall pred
                   (funcall key a)
                   (funcall key b)))))
where

   (sort3 things-having-foo #'< #'foo)

is "much easier to do" than

   (cl-sort things-having-foo #'< :key #'foo)

or

   (sort things-having-foo (lambda (a b) (< (foo a) (foo b))))

Additionally, you'll get much nicer tool support automatically,
e.g. autodoc tells me:

   sort3: (SEQ PRED KEY)
   cl-sort: (SEQ PREDICATE [KEYWORD VALUE]...)

One can see that autodoc for cl-sort is severely crippled.

> As to the advantage of @key key vs &optional key it is -- in my
> eyes,  at least -- that you already know its meaning from many other
> functions.

The point Richard raised was that your "many other functions" with &key
are bad way of doing it.

Simply naming the argument KEY would achieve the same goal of already
knowing its meaning from many other functions.  Making it &key does not
change that.

>> (btw why two functions and not extra stablep keyword argument?)
>
> Why not? Maybe that makes passing sort and stable-sort to higher
> order functions more practical?

That speculation does not sound plausible.

By that logic, CL would probably define 4 functions:

   sort sequence predicate => sorted-sequence
   sort3 sequence predicate key => sorted-sequence
   stable-sort sequence predicate => sorted-sequence
   stable-sort3 sequence predicate key => sorted-sequence

>> > Let's look at a traditional Elisp macro define-minor-mode.
>>
>> the usual CL argument list does not seem to be able to express arguments
>> of such shape
>>
>> it looks like whoever extended the original argument list did it
>> "weirdly" using custom ad-hoc single-use argument list parser.
>
> Quite likely the cl machinery wasn't available at the time...  But yes,
> Greenspun's 10th.
>
>> > are maintenance hazards, the macro now accepts keyword arguments
>>
>> in CL, the arguments would normally be in a list before body, something
>> like
>>
>>    (define-minor-mode MODE ([KEYWORD VAL ... ]) [DOC] &rest BODY)
>
> OK, but this is irrelevant.  This is a macro, not a function.
> For all practical purposes it was extended with keyword arguments.
> In fact you could have used cl-lib's cl-destructuring-bind.

No, that's missing the point.  CL itself is not able to express the
argument list structure in the shape it is implemented in
define-minor-mode.  You need custom argument parser anyway.

autodoc for define-minor-mode is also useless:

   define-minor-mode: (MODE DOC [KEYWORD VAL ... &rest BODY])

The way define-minor-mode sneaks in keyword arguments is pretty bad.  It
implements custom argument list parser and makes tools that understand
and work with argument lists useless, which causes extra manual work in
other areas.  If there was a unified argument list parser implemented in
one place and used consistently, it would avoid lots of manual work.

Back to the [KEYWORD VAL ... ] example, putting the argument list of the
macro into an extra list would allow one to use unified argument list
parser, but the way define-minor-mode arguments are specified at the
moment, CL:DESTRUCTURING-BIND cannot destructure such structure (&key
before &rest/&body is malformed).

Another example, take &body as opposed to &rest.  It automatically
declares the intent and how the indentation is meant to be computed.
With &rest indentation needs to be specified individually and manually
per case.  Lack of &body in elisp is annoying inconvenience.  Example:
(with-help-window BUFFER-OR-NAME &rest BODY) has (declare (indent 1))
which would not be needed if it was (with-help-window BUFFER-OR-NAME
&body BODY).

I think that rather than arguing about keyword arguments only, it would
be better to push for a unified argument list format which would be
followed, understood and supported by relevant tools.  So far, argument
list structure in elisp grew into ad-hoc mess which one needs to decode
manually from docstring on case by case basis; and no amount of CL would
help.

Re: [External] : Re: Shrinking the C core

[João Távora]

On Tue, Sep 12, 2023 at 8:52 PM Tomas Hlavaty <tom@logand.com> wrote:

> It has everything to do with the lack of the KEY argument
> and the KEY argument could as well be &optional instead of &key.

Yes, it could, but should it?  I think not, because consistency.
You think yes, because whatever, so let's just disagree.

> If you find lack of the key argument "much not easier", you can define
> your own sort, something like:

The point with having good library functions is that the user doesn't
need to define her ad-hoc versions.

> One can see that autodoc for cl-sort is severely crippled.

That's a problem with autodoc and cl-sort's definition, not its protocol.
Functions defined with cl-func are much better and autodoc functionality
for SLIME/SLY is spot on and detects the correct argument.  This is
no technical hurdle.

João

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

João Távora wrote:

> Many other functions with a large amount of optional
> arguments (completing-read comes to mind) would be much,
> much easier to use with keyword arguments. Without them, we
> find ourselves wondering about how many nils to sprinkle
> before the argument we want to pass.

Agreed, it is better with keyword arguments than a long list
of nils because of optional arguments.

But it is better yet to not have the functions take so many
arguments in the first place, but to split them up and have
the function name be more specific what is going to happen.

Sometimes this just doesn't happen, it's life and I have
a hard time seeing any situation (long list of nils for
optional arguments vs. keywords) being a real problem
to anyone?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Eric S. Raymond]

Richard Stallman <rms@gnu.org>:
> Switching to Common Lisp would cause those painful incompatibilities.
> It is unacceptable.

I did some checking into this a few weeks ago and even wrote a bit of Lisp to do
an audit.  There are other reasons I decided to get up to speed on Common Lisp
recently, and because I have Emacs Lisp engraved on my forebrain this interested
me in getting a good grasp on how incompatible with SBCL it actally is.

Count of elisp core functions: 1476 
Total SBCL/elisp function name collisions: 145 
Primitives identical in elisp and SBCL: 116
Primitives not known identical: 28

The first two lines are crunched from querying SBCL's symbol list and
groveling through the Emacs sourcefiles; I can supply the SBCL code I
write to do this if anybody cares.

The second two lines are from me reading the SBCL documentation a lot;
I needed immediate motivation to do that and this was a good one.

I was able to write trivial implementations for 15 of those 28 collisions.
The 14 I didn't implement are these:

(require read-char read random process-status process-plist print
princ prin1-to-string prin1 load documentation defvar aref)

What this means is that it would, in principle, be possible to write
an SBCL core that implements all of the Emacs primitives, with only a rather
small amount of shim code required to make existing elisp code execute
in an environment where it thinks it sees *all* elisp primitives.

The code would look something like this, where I have deleted all but one emulation
to showcase setting up the emulation environment.

	;; Implement functions shared between CL and elisp in an elisp-like way

	(defun elisp-append (&rest sequences)
	  "(append &rest SEQUENCES)

	  Probably introduced at or before Emacs version 18.
	  This function does not change global state, including the match data.

	Concatenate all the arguments and make the result a list.
	The result is a list whose elements are the elements of all the arguments.
	Each argument may be a list, vector or string.
	The last argument is not copied, just used as the tail of the new list."
	  (apply 'append (mapcar
			  (lambda (it)
			    (if (listp it) it (listp it)))
			  sequences))
	  )

	(defmacro within-elisp (form)
	  "Evaluate FORM in an environment with elisp behavior."
	  `(flet ((append #'emacs-append)
		 (delete-file-internal #'elisp-delete-file-internal)
		 (eval #'elisp-eval)
		 (format #'elisp-format)
		 (intern #'elisp-intern)
		 (make-hash-table #'elisp-make-hash-table)
		 (make-string #'elisp-make-string)
		 (rename-file #'elisp-rename-file)
		 (sort #'elisp-sort)
		 (string #'elisp-string)
		 (string-equal #'elisp-string-equal)
		 (string-lessp #'elisp-string-lessp)
		 (symbol-value #'elisp-symbol-value)
		 (type-of #'elisp-type-of)
		 (unintern #'elisp-unintern)
		 (yes-or-no-p #'elisp-yes-or-no-p)
		 )
	    ,form)
	  )

With this encapsulation, "painful incompatiblities" between elisp and
a core written in SBCL would never need be visible to anyone who put
an .el extension on their code to tell the core it should be executed
using within-elisp.

I'm not minimizing the amount of work an SBCL core would take,
there's a lot of devil in the details of 1476 + 14 primitive
functions.  Nor am I interested in joining a political argument
about whether it should be done; I have too much else on my
plate for that.

But it could be done. There is a technical path forward to it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Richard Stallman <rms@gnu.org>:
>> Switching to Common Lisp would cause those painful incompatibilities.
>> It is unacceptable.
>
> I did some checking into this a few weeks ago and even wrote a bit of Lisp to do
> an audit.  There are other reasons I decided to get up to speed on Common Lisp
> recently, and because I have Emacs Lisp engraved on my forebrain this interested
> me in getting a good grasp on how incompatible with SBCL it actally is.
>
> Count of elisp core functions: 1476 
> Total SBCL/elisp function name collisions: 145 
> Primitives identical in elisp and SBCL: 116
> Primitives not known identical: 28
>
> The first two lines are crunched from querying SBCL's symbol list and
> groveling through the Emacs sourcefiles; I can supply the SBCL code I
> write to do this if anybody cares.
>
> The second two lines are from me reading the SBCL documentation a lot;
> I needed immediate motivation to do that and this was a good one.
>
> I was able to write trivial implementations for 15 of those 28 collisions.
> The 14 I didn't implement are these:
>
> (require read-char read random process-status process-plist print
> princ prin1-to-string prin1 load documentation defvar aref)
>
> What this means is that it would, in principle, be possible to write
> an SBCL core that implements all of the Emacs primitives, with only a rather
> small amount of shim code required to make existing elisp code execute
> in an environment where it thinks it sees *all* elisp primitives.
>
> The code would look something like this, where I have deleted all but one emulation
> to showcase setting up the emulation environment.
>
> 	;; Implement functions shared between CL and elisp in an elisp-like way
>
> 	(defun elisp-append (&rest sequences)
> 	  "(append &rest SEQUENCES)
>
> 	  Probably introduced at or before Emacs version 18.
> 	  This function does not change global state, including the match data.
>
> 	Concatenate all the arguments and make the result a list.
> 	The result is a list whose elements are the elements of all the arguments.
> 	Each argument may be a list, vector or string.
> 	The last argument is not copied, just used as the tail of the new list."
> 	  (apply 'append (mapcar
> 			  (lambda (it)
> 			    (if (listp it) it (listp it)))
> 			  sequences))
> 	  )
>
> 	(defmacro within-elisp (form)
> 	  "Evaluate FORM in an environment with elisp behavior."
> 	  `(flet ((append #'emacs-append)
> 		 (delete-file-internal #'elisp-delete-file-internal)
> 		 (eval #'elisp-eval)
> 		 (format #'elisp-format)
> 		 (intern #'elisp-intern)
> 		 (make-hash-table #'elisp-make-hash-table)
> 		 (make-string #'elisp-make-string)
> 		 (rename-file #'elisp-rename-file)
> 		 (sort #'elisp-sort)
> 		 (string #'elisp-string)
> 		 (string-equal #'elisp-string-equal)
> 		 (string-lessp #'elisp-string-lessp)
> 		 (symbol-value #'elisp-symbol-value)
> 		 (type-of #'elisp-type-of)
> 		 (unintern #'elisp-unintern)
> 		 (yes-or-no-p #'elisp-yes-or-no-p)
> 		 )
> 	    ,form)
> 	  )
>
> With this encapsulation, "painful incompatiblities" between elisp and
> a core written in SBCL would never need be visible to anyone who put
> an .el extension on their code to tell the core it should be executed
> using within-elisp.

There is a little bit more than just this; but that one was very clever
to smooth out basic differences. It all depends on the ambition of
course. I do wrap cl funcitons and re-implement some of those that are
missing in CL just to be able to attach the doc strings to symbols, and
counting on sbcl inlining to remove the function call. I am not sure yet
if it is a good strategy, but I can do it relatively automated. Looks
like this:

(defun take (n list)
  "Return the first N elements of LIST.
If N is zero or negative, return nil.
If N is greater or equal to the length of LIST, return LIST (or a copy)."
  ;; todo - this can probably be done without taking the length of the list
  (cl:when (cl:> n 0)
    (cl:let ((l (cl:length list)))
      (when (cl:> n l) (setf n l))
      (cl:subseq list 0 n))))

(defun ntake (n list)
  "Modify LIST to keep only the first N elements.
If N is zero or negative, return nil.
If N is greater or equal to the length of LIST, return LIST unmodified.
Otherwise, return LIST after truncating it."
  (cl:when (cl:> n 0)
    (cl:let ((l list))
      (eli:while (cl:and (cl:> n 1) (cl:cdr l))
             (cl:setf l (cl:cdr l))
             (cl:decf n))
      (cl:rplacd l nil)
      list)))

(declaim (inline nthcdr))
(defun nthcdr (n list)
  "Take cdr N times on LIST, return the result."
  (cl:nthcdr n list))

(declaim (inline nth))
(defun nth (n list)
  "Return the Nth element of LIST.
N counts from zero.  If LIST is not that long, nil is returned."
  (cl:nth n list))

But tere is more than just functions and macros.

The reader needs some help; Emacs uses different syntax for characters
and escapes for example; ? vs #\, but the escape syntax is were a real
hear pulling is. Also Emacs does not have character type but basically
uses whatever int comes out of C and thus Emacs API can do arithmetic
operaions and comparisons on characters whereas in CL they have to use
character specific operators. I have implemented the reader to
understand ? and escapes and to spit out integer codes instead of cl
characters so I don't need to dispatch on each mathematical operation.

There is still more, "markers are numbers" in Emacs, so mathematical
operations are applicable to those as well. I haven't decided if I will
do same as they do in C sources or I'll try to implement static dispatch
for generic functions and have mathematical operations as generic
functions, I have to test how it works, but there are other things like
those that have to be taken care of. And there is more, that is
just the tip of the iceberg :).

I am also talking only about lisp implemented in C core. There weould be
need to do some work on the elisp side too; eieo, cl-lib, defun/defmacro,
stuff like that, but that is probably by far lesser and easier to fix.

> I'm not minimizing the amount of work an SBCL core would take,
> there's a lot of devil in the details of 1476 + 14 primitive
> functions.  Nor am I interested in joining a political argument
> about whether it should be done; I have too much else on my
> plate for that.

Yes, something like that; it is volumous and lots of work, for one
person probably impossible, but it is not impossible to do. 

Re: [External] : Re: Shrinking the C core

[Gerd Möllmann]

"Eric S. Raymond" <esr@thyrsus.com> writes:

 > But it could be done. There is a technical path forward to it.

Which would have to cope with buffer-local bindings.
Just saying :-).

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> "Eric S. Raymond" <esr@thyrsus.com> writes:
>
>  > But it could be done. There is a technical path forward to it.
>
> Which would have to cope with buffer-local bindings.
> Just saying :-).

I think they can be dealt with in at least two different ways; but I haven't got
so far yet, so I might be wrong.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I implemented Common Lisp for the Lisp Machine in 1983.
> That included the generic sequence functions with their
> many keyword arguments.
>
> In 1984 I wrote Emacs Lisp and decided to omit all that.
> I have no regrets.  Their absence makes Emacs Lisp easier to use.

Thank you for explanation and clarficiation. Now at least I understand
why you don't like them.

> Keyword arguments are fine for an unusual, heavyweight operation such
> as creating a frame.  The sort of function whose doc you need to look
> up again each time you use it.  When something is as cumbersome as
> that, the question of how to pass the arguments is a pile of minor
> details that you'll see when you look it up.
>
> But it's a pain in the neck to make simple everyday functions such as
> `append' and `length' that cumbersome just for the sake of following a
> general system.  Part of this system is that the default argument
> values follow the system, rather than following the traditional Lisp
> functions -- so getting the traditional behavior requires actually
> specifying the keyword arguments to override defaults, almost every
> time.

Of course, but nobody suggests it is all-in. `length' is not a keyworded
in CL either. It would be madness to do something like naming all
arguments in all functions. But it is convenient to have keywords in
some places, like for example in define-minor-mode or make-process, or
even "new" define-keymap.

> Switching to Common Lisp would cause those painful incompatibilities.
> It is unacceptable.
>
> Defining the CL functions with a name prefix is ok, because the
> traditional Lisp functions are still available compatibly and we don't
> have to use the CL functions.  This is what we have now.

If cl-lib was loaded into the lisp image, we could even rename cl-defun
to defun, after we have initialized all the other stuff that goes into
the dump, and have "upgraded" vesion of defun and would be fully
compatible with traditional Lisp functions. We could even go further and
remove cl- prefix from bunch of symbols to make them less cumbersome to
use, and everything would stil be compatible with traditional Lisp
functions.

I think there is a thing that makes Lisp a bit special compared to other
programming languages, and Steel mentions it briefly in his legendary
talk: anyone can add a new feature to the language and it becomes part
of the vocabulary without distinction from the built-in stuff. Keyword
arguments does not require anything special from the C runtime, and we
see them pop-up in different forms in elisp. Perhaps elisp would be
better off to accept them and make cl-defun the norm, instead of seing
macros like define-minor-mode re-implement the technique and some other
macro possibly do something similar which leads to more less maintanable
code.

But that is just loud thinking, since my point wasn't to suggest any
changes, I just wanted to understand why you didn't like keyword
arguments per se, so thank you very much for the answer.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

We should extend the current design so that Emacs is
multi-threaded, for this to happen we should agree on a model
how this is supposed to be done, and after that it is just
a matter what needs to be done and what parts need to be
adopted to realize that model.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Arthur Miller wrote:

> Of course, but nobody suggests it is all-in. `length' is not
> a keyworded in CL either. It would be madness to do
> something like naming all arguments in all functions. But it
> is convenient to have keywords in some places [...]

Keywords typically make for shorter interfaces and function
calls that communicate more, however the shorter interface
advantage and the need to have function calls to communicate
more should be put against the typical downside with this
style, which is very long and complicated functions that do
a lot more than they should, often.

And, because very long functions isn't good style anyway,
those should probably be split up - and if they are split up,
the interfaces will be shorter and the style clear that way
instead, so the need for keywords to do that will also be
reduced or disappear, really.

>> Defining the CL functions with a name prefix is ok, because
>> the traditional Lisp functions are still available
>> compatibly and we don't have to use the CL functions.
>> This is what we have now.
>
> If cl-lib was loaded into the lisp image, we could even
> rename cl-defun to defun, after we have initialized all the
> other stuff that goes into the dump, and have "upgraded"
> vesion of defun and would be fully compatible with
> traditional Lisp functions. We could even go further and
> remove cl- prefix from bunch of symbols to make them less
> cumbersome to use, and everything would stil be compatible
> with traditional Lisp functions.

Indeed, that could be done for every cl-X case where X is
a non-cl-lib Elisp function and cl-X already does everything
X does with the same interface. Those could be cancelled out.

For the cases where there is no such X outside of cl-lib, e.g.
`cl-incf' and `cl-decf' (no "incf" or "decf" anymore?) the
cl-lib prefix could also be dropped. Another example is
`cl-loop'. One could solve this with aliases, possibly.

Because yes, it looks a bit strange that such rudimentary
stuff has to be prefixed to note their belonging to some
particular implementation or library, and even more so when
there isn't any competition or alternative, even?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Tomas Hlavaty]

On Tue 12 Sep 2023 at 04:30, Arthur Miller <arthur.miller@live.com> wrote:
> Perhaps elisp would be better off to accept them and make cl-defun the
> norm, instead of seing macros like define-minor-mode re-implement the
> technique and some other macro possibly do something similar which
> leads to more less maintanable code.

cl-defmacro would not help with define-minor-mode because it cannot
describe the special argument list structure in this case.

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course, but nobody suggests it is all-in. `length' is not a keyworded
  > in CL either.

Oops, I thought it was -- but the time when I implemented and used
Common Lisp was 40 years ago.

I am pretty sure `member' used keyword arguments, and I think that
getting behavior equivalent to traditional Lisp `member' required
specifying a keyword argument.  I never forgot that, during that
period, because I got reminded of it almost every day.

  >  But it is convenient to have keywords in
  > some places, like for example in define-minor-mode or make-process, or
  > even "new" define-keymap.

I would not object to using keyword arguments for functions like that
-- complex and cumbersome to use, and not used often.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Bob Rogers]

   From: Arthur Miller <arthur.miller@live.com>
   Date: Fri, 08 Sep 2023 04:00:56 +0200

   Alan Mackenzie <acm@muc.de> writes:

   > Hello, Arthur.
   >
   > On Wed, Sep 06, 2023 at 07:04:43 +0200, Arthur Miller wrote:
   >> Richard Stallman <rms@gnu.org> writes:
   > . . .
   >> > Also, there are aspects of Common Lisp which i rejected as clumsy and
   >
   >> With all respect to you, but sound to me like an emotional argument, not
   >> a rational one.
   >
   > It's a rational argument expressed in emotional terms for simplicity.

   What you basicaly say: it somehow is a summarization of some more
   advanced/complicated reasons which can't be explained, but have to be
   expressed in the term "I don't like it" for the simplicity?

It's an aesthetic argument, and I am sure we can all agree that language
design has a large and important aesthetic component.  I believe Richard
can justify leaving out (e.g.) keywords from the core language on that
basis alone.

   Mind you, I do agree with your practical arguments, and I appreciate
the references concerning the growing complexity of modern languages and
their communities.  What I think nobody has yet mentioned (and at the
risk of adding to an overlong and possibily off-topic discussion) is the
fact that the learning-curve issue should be less of a problem to the
embedded language of the "self-documenting" editor.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Shrinking the C core

[Emanuel Berg]

Arthur Miller wrote:

> We can certainly speak about "old ways", let us take the
> or-idiom or how should I call it: initialization of the
> default value for optional arguments:
>
> (defun foo (&optional who-am-I)
>   (let ((who-am-I (or who-am-I "foo")))
>     (message "I am %s." who-am-I)))
>
> Is that really better than typing:
>
> (cl-defun foo (&optional (who-am-I "foo"))
>   (message "I am %s." who-am-I))
>
> The user has to learn the idiom, which uses operator "or" to
> perform something that visually has nothing to do with the
> intention of the code, and also has to type the additional
> let-form each and every time. Than the users who are not
> familiar with the idiom will perhaps come up with their own
> version, using setq or some other thing, and you will really
> have to think what the user wanted to say with their code if
> you had to debug it. Is it better than seing an initialiser
> and knowing directly what is the default value and everyone
> using uniform syntax?

This is a very good example where the CL way is simply
superior. I don't know how many ways I've seen people,
including myself, setting the default value: `or', 'unless',
`setq', `let' - or forgetting about it, for that matter.

But we do have `cl-defun' - in Elisp - so it isn't like we
don't have it for anyone to use if and when desired. I learned
about `cl-defun' much later than I did &optional but if I want
optional arguments now `cl-defun' is a much better choise than
`defun', because of the much more clear syntax to provide the
default value.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > (defun foo (&optional who-am-I)
  > >   (let ((who-am-I (or who-am-I "foo")))
  > >     (message "I am %s." who-am-I)))
  > >
  > > Is that really better than typing:
  > >
  > > (cl-defun foo (&optional (who-am-I "foo"))
  > >   (message "I am %s." who-am-I))

In Emavs Lisp you can write

(defun foo  (&optional who-am-I)
  (message "I am %s." (or who-am-I "foo")))

which is just as simple as the cl-defun method.

I have no objection to the feature of default values for optional
arguments.  It does not make code harder to understand.

The reason I did not implement that in 1984 was to keep Emacs Lisp's
implementation smaller and finish it sooner.  The latter is no longer
pertinent.  The former reason may still be valid, because implementing
this will add some coimplexity.  Is it worth its cost?  I don't know.


  
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

You are being personally offensive now.
Please stop that, and take this discussion to emacs-tangents.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> You are being personally offensive now.
> Please stop that, and take this discussion to emacs-tangents.

That is unfortuante and sad if you feel that way; my intention is not to
be personally offensive; I am attacking a (bad) argument, certainly not
you as a person, nor Alan; I appologize if it comes out that way.

best regards

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I accept your apology, but please move that topic to emacs-tangents.
The idea of rewriting Emacs in Common Lisp is not on the table for
serious consideration.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Andrea Monaco]

Turning a program into multi-threaded _may_ make it faster and more
responsive, but that is not necessary nor sufficient.

As an example on my laptop, Emacs (a big single-threaded program) is
faster and more responsive than web browsers and mail clients (big
multi-threaded programs).

Sometimes emacs freezes briefly, but so do the other programs.



Andrea Monaco

Re: Shrinking the C core

[Arthur Miller]

Andrea Monaco <andrea.monaco@autistici.org> writes:

> Turning a program into multi-threaded _may_ make it faster and more
> responsive, but that is not necessary nor sufficient.
>
> As an example on my laptop, Emacs (a big single-threaded program) is
> faster and more responsive than web browsers and mail clients (big
> multi-threaded programs).

I am affraid that would be very missleading comparison. Your browsers do
a __lot__ more work than Emacs does, and I mean __lot__. It is like
saying Notepad is so very much faster than Gimp. Trying opening an image
of size say 1200z1024 in Emacs and zoom in/out, compare to the same
operation in  your browser.

> Sometimes emacs freezes briefly, but so do the other programs.
>
>
>
> Andrea Monaco

Re: Emacs design and architecture. How about copy-on-write?

[zhanghj]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

    The same will happen if only the main thread can display.  If a
    non-main thread wants to prompt the user, it will have to wait until
    the main thread becomes available, prompts the user, and returns the
    response.

    Someone always has to wait when threads need to synchronize.  There's
    no way around that.  Doing this in the thread that needs to
    display/prompt is easier because the context and relevant variables
    don't need to be communicated to another thread.

Sorry for my poor English.

Message queue is a good solution for parallel and concurrence.

I think it is not a good idea to let all threads to do display things directly.
It will be better to let non-main threads to do display things via message queue,
like many GUI toolkit does. It is also like the message loop in Emacs.
This will make many things easier.

For GUI toolkit on windows, there are two basic API: Sendmessage and PostMessage.
Non-gui thread can use theme to do display and UI related things.

For example in Emacs, when a non-main thread calls `message', a small display task is built
and put into the message queue of the main thread, like `PostMessage` does.
If it calls `read-file-name`, an UI task is built and put into the message queue of
the main thread like `SendMessage` does, while the caller thread will be blocked until result comes.

When all display jobs are all in one thread (decoupled with other threads), display jobs may be scheduled
more efficiently and may be concurrent internally.

[-- Attachment #2: Type: text/html, Size: 1685 bytes --]

Re: Emacs design and architecture. How about copy-on-write?

[Payas Relekar]

Po Lu <luangruo@yahoo.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Then try leaning on C-n or C-p _after_ everything is already
>> fontified.  You will still see that Emacs sometimes cannot keep up,
>> especially if lines are not too short.
>
> Long lines are, at worst, infrequently encountered in source code,

Long lines are frequently encountered in prose like say Org-mode, that
might require font lock extensively, depending on the document.