Re: Shrinking the C core

Re: Shrinking the C core

[tomas]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

On Thu, Sep 07, 2023 at 09:23:49AM +0200, Emanuel Berg wrote:
> tomas wrote:

[...]

> > In the case of Emacs Lisp, we'll have to accept that people
> > like Richard and Eli carry more weight [...]

> Absolutely not, everyone is allowed to use Elisp in any way
> they want [...]

I'm not discussing this with you here anymore. Glad to continue
in emacs-tangents@, but I won't answer anything in emacs-devel@

Followup set.
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> And there is always mobsters and lynchmob that like to give shit to
> other people to feel themselves better. You should be ashamed of
> yourself.
>
> The toxic atmosphere like this is why I keep myself away from the
> Emacs mailing lists in general. I posted once in about a year or more
> and offered a patch and regretted it. If you think I came now just to
> ask you to write something for me, think twice.

Watch your language, and take this off emacs-devel@gnu.org.  Since you
aren't even subscribed to that, this ought to be a simple matter of
editing one address within the carbon copy list.  This I have now done
on your behalf.

Re: Shrinking the C core

[Eli Zaretskii]

[Redirected to emacs-tangents.]

> From: Arthur Miller <arthur.miller@live.com>
> CC: emacs-devel@gnu.org
> Date: Tue, 12 Sep 2023 05:46:37 +0200
> 
> Mnjah; you know as well as I, and I have written it in the very first
> mail why I want Emacs in Lisp. There are already other applications and
> editors inpsired by Emacs, that is not the question. Problem with them
> is they can't run Emacs applications and they don't have Emacs manual
> and the well written documentation. I think it would be waste of the
> effort of many people to throw it away. You may disagree and that is OK.

It will not be a waste of effort if the "neo-Emacs" will take into
consideration the main lessons we learned during those 40 years, and
will have a different architectural design to avoid the pitfalls and
allow extensions we cannot currently support.  Reimplementing the same
design in a different language, be it SBCL, Python, Go, Rust, or
whatever -- that _is_ a waste of effort, because you will have the
same Emacs with the same basic restrictions and limitations, which are
really not expected from a modern GUI program.  We tolerate them in
Emacs because in return we get so much and because the performance is
still reasonable as long as we can live with the limitations, but
starting a radically new implementation from the same point with the
same basic design means that we will have to tolerate those same
limitations for decades in the future, and that's simply silly.  The
net result will be a huge effort, lots of work to stabilize the result
to be anywhere close to what we have now, and all that for what?

Which is why anyone who understands the internals well enough to do
the job will never do it.

Yes, this can be done in principle.  But it shouldn't.

P.S.  And don't be afraid of losing some of the applications and the
documentation -- these should be the least of your worries.  Sticking
to the same design for these reasons is ... I don't even have
civilized words to describe this kind of thinking.

Re: Shrinking the C core

[chad]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]

Now that we're in -tangets (thanks for doing that, btw)...

On Tue, Sep 12, 2023 at 7:57 AM Eli Zaretskii <eliz@gnu.org> wrote:

> [...] And don't be afraid of losing some of the applications and the
> documentation -- these should be the least of your worries. [...]
>

My own opinion has come around to match Eli's (far more relevant/expert) on
this topic, but in case it helps anyone else: I think this reflexive desire
to keep a strong hold on all the existing elisp comes from the long line of
emacs clones like Edwin, Hemlock, climacs, Alpha, and a long list of less
well-known alternatives. I "lived through" several of these, and the sense
I got from every one I actually tried was that people eventually fell back
to the "real thing" emacs, largely because it worked (well enough?) and
already had some facilities that were missing in the putative replacement.

summary: I think it's hard to unlearn this lesson from history

It's entirely possible that the shift of "typical computing" towards
massively multi-core distributed etc. is the final straw, or at least the
high bar that a massively-shared-state-lisp-machine can't vault -- while
still being good enough to cross most of the moats we see in everyday
usage. It's also possible that there's some adaptation that will arise,
perhaps along the lines of how web workers/service worker threads interact
with the DOM in the modern browser, that keeps Emacs going even longer. I
remember the days when "Yeah, that will happen shortly after the release of
Emacs 21" was the in-joke for porcine aeronautics, and we just saw Emacs 29
released, so: it could happen.

I do think that a very early step needs to be "figure out how to handle
concurrent analysis and editing across multiple cores and perhaps
machines", but that probably just reflects a bunch of my personal interests.

~Chad

[-- Attachment #2: Type: text/html, Size: 2366 bytes --]

Re: Shrinking the C core

[Yuri Khan]

On Wed, 13 Sept 2023 at 01:23, chad <yandros@gmail.com> wrote:

> It's also possible that there's some adaptation that will arise, perhaps along the lines of how web workers/service worker threads interact with the DOM in the modern browser, that keeps Emacs going even longer.

Actually, as far as I understand, service workers don’t interact with
the DOM. They have access to offline storage that lets them keep
state, and they can intercept requests ordinarily destined to their
origin servers and respond to them in lieu of the server.

Web workers don’t have access to the DOM either; instead, they listen
for and respond to JSON messages sent by page scripts.

So nothing new here: concurrency gets much easier when you disallow
(or vastly minimize) shared state.

Re: Shrinking the C core

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Tue, 12 Sep 2023 14:22:57 -0400
> Cc: Arthur Miller <arthur.miller@live.com>, emacs-tangents@gnu.org
> 
> I do think that a very early step needs to be "figure out how to handle concurrent analysis and editing
> across multiple cores and perhaps machines", but that probably just reflects a bunch of my personal
> interests.

One of the main advantages of multithreading in Emacs is to have a
separate UI thread, so that we wouldn't have Emacs appear frozen when
it does some computation.  To have such a separate thread in Emacs is
tricky, because we allow to run Lisp both on input and on output
(redisplay).  This is one of the main reasons why Emacs is so
powerful, so giving this up would be a huge loss.  So someone would
need to figure out how to keep this feature and still run I/O in a
separate thread...

Re: Shrinking the C core

[joakim]

chad <yandros@gmail.com> writes:

> Now that we're in -tangets (thanks for doing that, btw)...
>
> On Tue, Sep 12, 2023 at 7:57 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
>  [...] And don't be afraid of losing some of the applications and the
>  documentation -- these should be the least of your worries. [...]
>
> My own opinion has come around to match Eli's (far more relevant/expert) on this topic, but in case it helps anyone else:
> I think this reflexive desire to keep a strong hold on all the existing elisp comes from the long line of emacs clones
> like Edwin, Hemlock, climacs, Alpha, and a long list of less well-known alternatives. I "lived through" several of these,
> and the sense I got from every one I actually tried was that people eventually fell back to the "real thing" emacs,
> largely because it worked (well enough?) and already had some facilities that were missing in the putative replacement. 
>
> summary: I think it's hard to unlearn this lesson from history
>
> It's entirely possible that the shift of "typical computing" towards massively multi-core distributed etc. is the final
> straw, or at least the high bar that a massively-shared-state-lisp-machine can't vault -- while still being good enough
> to cross most of the moats we see in everyday usage. It's also possible that there's some adaptation that will arise,
> perhaps along the lines of how web workers/service worker threads interact with the DOM in the modern browser, that keeps
> Emacs going even longer. I remember the days when "Yeah, that will happen shortly after the release of Emacs 21" was the
> in-joke for porcine aeronautics, and we just saw Emacs 29 released, so: it could happen.
>
> I do think that a very early step needs to be "figure out how to handle concurrent analysis and editing across multiple
> cores and perhaps machines", but that probably just reflects a bunch of my personal interests.

Since we are in "tangents" now, I suppose it wont harm if I add some
opinions as well.

I also tried many different emacsen, and different editors, and always
came back to the mother-ship.


The thread seemed to focus mostly on things that Emacs doesnt do so
well, and not on what Emacs does remarkably well.

For instance, the multi-tty feature is fantastic, I use it all the
time. You can use emacs on a tiny raspberry, or on a super large
machine. There is tramp, and well, the kitchen sink. And emacs is
infinitely tweakable, which is very useful. The buffer/window paradigm
is really useful.

The "new" batch of applications dont do much of the above things, so
while of course more visually apealing, they dont add much else(well of
course, LSP, and stuff, but emacs has that now as well)

So while the thread was about multi-threading, if I had a magic wand to
fix emacs things, I would fix that bother me daily.

- if I'm connected to an emacs session by ssh, and mistakenly make a
  cli command dump tens of megabytes of spewage to the shell buffer, I'm in trouble
  and cant easily get out of it. I need to open a new ssh session and
  kill the rampaging cli. This is quite tedious. Would concurrency fix this?

- Same for long-lines, this is still not a solved problem.

- Gnus refreshes slowly, maybe that could be helped with concurrency,
  but it could also be helped with more async work in gnus.

Anyway, thats my 0.02€, please ignore and carry on...


>
> ~Chad
>
-- 
Joakim Verona
joakim@verona.se

Re: Shrinking the C core

[Eli Zaretskii]

> X-Spam-Status: No, score=-1.01 tagged_above=-999 required=6.2
> 	tests=[ALL_TRUSTED=-1, T_SCC_BODY_TEXT_LINE=-0.01] autolearn=disabled
> From: joakim@verona.se
> Cc: Eli Zaretskii <eliz@gnu.org>,  Arthur Miller <arthur.miller@live.com>,
>   emacs-tangents@gnu.org
> Date: Tue, 12 Sep 2023 21:07:41 +0200
> 
> - if I'm connected to an emacs session by ssh, and mistakenly make a
>   cli command dump tens of megabytes of spewage to the shell buffer, I'm in trouble
>   and cant easily get out of it. I need to open a new ssh session and
>   kill the rampaging cli. This is quite tedious. Would concurrency fix this?

This should be doable, and doesn't come anywhere near the "rewrite"
job.  You just need a way of blocking the output from the shell, and
then use Emacs commands to kill it.

> - Same for long-lines, this is still not a solved problem.

You didn't try Emacs 29 yet, did you?

> - Gnus refreshes slowly, maybe that could be helped with concurrency,
>   but it could also be helped with more async work in gnus.

Concurrency can help you keep reading messages while Gnus refreshes in
parallel, but it won't easily help you refresh faster, unless someone
comes up with a way of collecting the update in parallel chunks (in
which case they should be able to do that today with the emacs-async
package, I think).

Re: Shrinking the C core

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> X-Spam-Status: No, score=-1.01 tagged_above=-999 required=6.2
>> 	tests=[ALL_TRUSTED=-1, T_SCC_BODY_TEXT_LINE=-0.01] autolearn=disabled
>> From: joakim@verona.se
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Arthur Miller <arthur.miller@live.com>,
>>   emacs-tangents@gnu.org
>> Date: Tue, 12 Sep 2023 21:07:41 +0200
>> 
>> - if I'm connected to an emacs session by ssh, and mistakenly make a
>>   cli command dump tens of megabytes of spewage to the shell buffer, I'm in trouble
>>   and cant easily get out of it. I need to open a new ssh session and
>>   kill the rampaging cli. This is quite tedious. Would concurrency fix this?
>
> This should be doable, and doesn't come anywhere near the "rewrite"
> job.  You just need a way of blocking the output from the shell, and
> then use Emacs commands to kill it.

That would be really great!

>> - Same for long-lines, this is still not a solved problem.
>
> You didn't try Emacs 29 yet, did you?

Well, I rebuild from master like at least once a month, so I should be
good right? But I see your point, I should come up with a more tangible
measure than claiming long lines to be unsolved. Sorry for that.

>
>> - Gnus refreshes slowly, maybe that could be helped with concurrency,
>>   but it could also be helped with more async work in gnus.
>
> Concurrency can help you keep reading messages while Gnus refreshes in
> parallel, but it won't easily help you refresh faster, unless someone
> comes up with a way of collecting the update in parallel chunks (in
> which case they should be able to do that today with the emacs-async
> package, I think).

Yes, one of these days I should really try one of the gnus-async hacks.

-- 
Joakim Verona
joakim@verona.se

Re: Shrinking the C core

[Emanuel Berg]

joakim wrote:

>>> Gnus refreshes slowly, maybe that could be helped with
>>> concurrency, but it could also be helped with more async
>>> work in gnus.
>>
>> Concurrency can help you keep reading messages while Gnus
>> refreshes in parallel, but it won't easily help you refresh
>> faster, unless someone comes up with a way of collecting
>> the update in parallel chunks (in which case they should be
>> able to do that today with the emacs-async package,
>> I think).
>
> Yes, one of these days I should really try one of the
> gnus-async hacks.

I've heard many times that Gnus is slow, but never
experienced it. I have 6 servers and 26 groups.

How many do people have, for it to get slow?

Or is it slow for some other reason?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Arthur Miller]

>> IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
>> time and energy.  We've seen this many times (because people still
>> insist on bringing this up from time to time).  From where I stand,
>> the main reason is not even the fact that we decided not to do that,
>> but the fact that such a rewrite will never happen in practice.  Such
>> a rewrite is a massive job which requires very good knowledge of Emacs
>> internals and features, and a lot of time.  People who come close to
>> the required knowledge level are not interested in doing this job
>> (because they understand the futility), and those who think it should
>> be done simply don't know enough and/or don't have enough time on
>> their hands to pull it through.
>>
>> If Emacs will ever be "rewritten", it will not be Emacs, but a
>> text-processing system with a very different architecture and design,
>> which will take from the Emacs experience the lessons we learned and
>> implement them differently, to produce a system whose starting point
>> is closer to the needs of today's users and whose main technologies
>> are more modern from the get-go.
>
>I couldn't agree more.
>
>To me, a rewrite is quatsch, while adding CL facilities to Emacs makes a
>lot of sense.

I use to say often: either CL will come to Emacs or Emacs to CL, whichever
way around. We need some of features available on CL platforms, sbcl
notably: built-in concurrency and better garbage collectors from the get-go; and
some of the CL language features, namespaces notably, would be very nice to
have. I am not sure which one is easier to achieve, porting elisp to cl, or
rewriting core to have all those features. CFFI would also be nice to have so
that users can extend Emacs themselves with other libraries and not have to wait
for the core devs to do it for them. That would also lessen the burden on
maintaining that stuff in the core.

Re: Shrinking the C core

[Gerd Möllmann]

Arthur Miller <arthur.miller@live.com> writes:

>>> IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
>>> time and energy.  We've seen this many times (because people still
>>> insist on bringing this up from time to time).  From where I stand,
>>> the main reason is not even the fact that we decided not to do that,
>>> but the fact that such a rewrite will never happen in practice.  Such
>>> a rewrite is a massive job which requires very good knowledge of Emacs
>>> internals and features, and a lot of time.  People who come close to
>>> the required knowledge level are not interested in doing this job
>>> (because they understand the futility), and those who think it should
>>> be done simply don't know enough and/or don't have enough time on
>>> their hands to pull it through.
>>>
>>> If Emacs will ever be "rewritten", it will not be Emacs, but a
>>> text-processing system with a very different architecture and design,
>>> which will take from the Emacs experience the lessons we learned and
>>> implement them differently, to produce a system whose starting point
>>> is closer to the needs of today's users and whose main technologies
>>> are more modern from the get-go.
>>
>>I couldn't agree more.
>>
>>To me, a rewrite is quatsch, while adding CL facilities to Emacs makes a
>>lot of sense.
>
> I use to say often: either CL will come to Emacs or Emacs to CL, whichever
> way around. We need some of features available on CL platforms, sbcl
> notably: built-in concurrency and better garbage collectors from the get-go; and
> some of the CL language features, namespaces notably, would be very nice to
> have.

I agree.  Alas, others, who haven't seem the light yet, don't :-).

> I am not sure which one is easier to achieve, porting elisp to cl, or
> rewriting core to have all those features.

I don't know either, of course. I guess it depends on the feature.  Some
random thoughts:

I'm pretty sure that CL packages could be added to Emacs as it is, if
some people would work on it.

I'm also pretty sure that an incremental + generational GC could be
added, at least as an option, because I would have almost done it some
20+ years ago.  It was torpedoed by a patent issue concerning
mostly-copying GC.  The patent has since expired. A lot of work, of
course.  I think some people do or have done something in this area, but
I don't know details.

I'm not at all sure that non-cooperative multi-threading could be added
to Emacs.  But I'm also not sure how a CL core would help here.

On the other hand, I'm pretty convinced that an Emacs core written in CL
would have to be close to 100% compatible with the existing C core to be
accepted by users.  That includes a CL rewrite of the C Elisp, including
byte code interpreter.

That's a massive endeavor.  My hair stands up when I remember the
compatibility problems I faced with the new redisplay ages ago.
Multiply that by some factor > 1.  But maybe that's a burnt child
dreading the fire :-).

I'm also not sure how a CL (not Elisp) program would look like using a
CL Emacs core.  Is it nice enough, so to speak?  Think of Emacs strings,
which couldn't be CL strings because of text properties, buffer-local
variables...

(Another ansatz might be to make Emacs C core a lib.  I haven't given
that much thought, but it could be more promising than rewriting the
whole shit in CL :-).)

> CFFI would also be nice to have so
> that users can extend Emacs themselves with other libraries and not have to wait
> for the core devs to do it for them. That would also lessen the burden on
> maintaining that stuff in the core.

FFI for Emacs once existed, I think Dave Love wrote one, for instance.
Don't know what became of that.  Might be an issue with interfacing to
non-free libs.

Re: Shrinking the C core

[joakim]

Emanuel Berg <incal@dataswamp.org> writes:

> joakim wrote:
>
>>>> Gnus refreshes slowly, maybe that could be helped with
>>>> concurrency, but it could also be helped with more async
>>>> work in gnus.
>>>
>>> Concurrency can help you keep reading messages while Gnus
>>> refreshes in parallel, but it won't easily help you refresh
>>> faster, unless someone comes up with a way of collecting
>>> the update in parallel chunks (in which case they should be
>>> able to do that today with the emacs-async package,
>>> I think).
>>
>> Yes, one of these days I should really try one of the
>> gnus-async hacks.
>
> I've heard many times that Gnus is slow, but never
> experienced it. I have 6 servers and 26 groups.

About 700 groups, a mix of nnimap, and nntp.

nnimap is slowest. 

But its not all that slow, and Thunderbird would be equally slow I
think, and thunderbird doesnt come close organizing all thouse groups,
thats why i stay with gnus.

>
> How many do people have, for it to get slow?
>
> Or is it slow for some other reason?
-- 
Joakim Verona
joakim@verona.se

Re: Shrinking the C core

[Fraga, Eric]

On Wednesday, 13 Sep 2023 at 07:05, Emanuel Berg wrote:
> How many do people have, for it to get slow?

It's not just the number of groups etc. but also what servers you need
to access.  The latter is not specifically an issue with gnus in that
any other MUA would suffer as well; the problem is that if gnus is
taking a long time retrieving emails, I cannot do anything else in the
meantime.  Emacs is my window manager... and single threaded.

An aside: gnus' nnmaildir implementation is *very* slow under some
circumstances, to do with having to resequence all the articles in a
group.  But nnml is very good generally.

-- 
Eric S Fraga via gnus (Emacs 30.0.50 2023-08-14) on Debian 12.1

Re: Shrinking the C core

[Arthur Miller]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>>>> IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
>>>> time and energy.  We've seen this many times (because people still
>>>> insist on bringing this up from time to time).  From where I stand,
>>>> the main reason is not even the fact that we decided not to do that,
>>>> but the fact that such a rewrite will never happen in practice.  Such
>>>> a rewrite is a massive job which requires very good knowledge of Emacs
>>>> internals and features, and a lot of time.  People who come close to
>>>> the required knowledge level are not interested in doing this job
>>>> (because they understand the futility), and those who think it should
>>>> be done simply don't know enough and/or don't have enough time on
>>>> their hands to pull it through.
>>>>
>>>> If Emacs will ever be "rewritten", it will not be Emacs, but a
>>>> text-processing system with a very different architecture and design,
>>>> which will take from the Emacs experience the lessons we learned and
>>>> implement them differently, to produce a system whose starting point
>>>> is closer to the needs of today's users and whose main technologies
>>>> are more modern from the get-go.
>>>
>>>I couldn't agree more.
>>>
>>>To me, a rewrite is quatsch, while adding CL facilities to Emacs makes a
>>>lot of sense.
>>
>> I use to say often: either CL will come to Emacs or Emacs to CL, whichever
>> way around. We need some of features available on CL platforms, sbcl
>> notably: built-in concurrency and better garbage collectors from the get-go; and
>> some of the CL language features, namespaces notably, would be very nice to
>> have.
>
> I agree.  Alas, others, who haven't seem the light yet, don't :-).
>
>> I am not sure which one is easier to achieve, porting elisp to cl, or
>> rewriting core to have all those features.
>
> I don't know either, of course. I guess it depends on the feature.  Some
> random thoughts:
>
> I'm pretty sure that CL packages could be added to Emacs as it is, if
> some people would work on it.

With "CL packages" you mean namespaces? There seem to be already a
branch that implements them, but I don't know how well it works, I
haven't tried it. I think the reasoning is "with don't want them because
we have done 40 years without". But IDK for sure. If there is some
technical reason, then they could help the author to implement it the
way they like it technically.

> I'm also pretty sure that an incremental + generational GC could be
> added, at least as an option, because I would have almost done it some
> 20+ years ago.  It was torpedoed by a patent issue concerning
> mostly-copying GC.  The patent has since expired. A lot of work, of
> course.  I think some people do or have done something in this area, but
> I don't know details.

I am not very familiar wth GC:s implementation more then just some
bired-eye overview. SBCL recently started to move towards non-moving
GC to help with the speed, notably when calling native code which does
not like it's pointers moved underneath, but I am not expert on details
there, these what I have got from the paper:

https://applied-langua.ge/~hayley/swcl-gc.pdf

> I'm not at all sure that non-cooperative multi-threading could be added
> to Emacs.  But I'm also not sure how a CL core would help here.

They are exposing posix threads and have done some work to make at least
parts of the Lisp system work well with threads, and it seems it is
working well for many applicaitons.

http://www.lichteblau.com/sbcl/doc/manual/sbcl/Implementation-_0028Linux-x86_0029.html#Implementation-_0028Linux-x86_0029

I think there is also a missconception in Emacs community that Emacs
loop itself has to be parallelized; I am not sure it is needed; I think
for many people it would be enough to expose threading in form of "js
workers" or something like that. It can be done with processes of
course, but people seem to constantly scream about it in disucssions. CL
has things like lparallel and green threads built on top of hardware
threads, so even there is a bit of job already done. I am sure all that
can be done in Emacs too, but I think, both communities would be more
helped if we perhaps used sbcl and interested individuals helped make
sbcl runtime better intead of reduplication the entire effort.

> On the other hand, I'm pretty convinced that an Emacs core written in CL
> would have to be close to 100% compatible with the existing C core to be
> accepted by users.  That includes a CL rewrite of the C Elisp, including
> byte code interpreter.

Yes, my conclusion too.

> accepted by users.  That includes a CL rewrite of the C Elisp, including
> byte code interpreter.

I am not sure how much of byte-interpretter is needed; I was thinking
how byte interpretter and native compiler fitt there. Oviously since
sbcl is a compiler, with don't need all that stuff, but I am not sure
how much of byte code intepretter is needed. I am sure we need to
understand all of the syntax, since byte code is a valid elisp,
according to the manual; so the reader have to be able to read the
syntax I guess, as printed representation, and has to print same stuff
back to feed into elisp functions.

> That's a massive endeavor.  My hair stands up when I remember the
> compatibility problems I faced with the new redisplay ages ago.
> Multiply that by some factor > 1.  But maybe that's a burnt child
> dreading the fire :-).

Yes, I know. I am fully aware that it is an impossibility for someone
alone, even for a very few. I don't think it is a burnt child, since
yes, the character renderer of Emacs has to be implemented if Emacs
applications will run unchanged.

Hopefully it will be possible to implement Emacs stuff as a special kind
of terminal/character renderer over some sort of tree/graph structure. I
think CLOS and CL have much better tools to refactor that stuff than C,
but what do I know, I haven't tried that and I am not sure if I will
tbh. I am bolling with the ideas. I have seen what they do in other
similar CL software (Hemlock, McClIM, Lem), perhaps there is something
that can be reused there, but I don't know how much and what yet. And,
yes text properties are a special chapter on its own :).

> I'm also not sure how a CL (not Elisp) program would look like using a
> CL Emacs core.

It would look exactly the same, that is the point :). If I wan't to be
able to run elisp applications from elpa/melpa the elisp compatibility
has to be 1:1 so to say. I think it is OK to have some "special places",
like an empty eieio.el, and bunch of aliases or compiler macros for cl-*
stuff, but in general it should be able to read an elisp file without
mofications. 

> CL Emacs core.  Is it nice enough, so to speak?  Think of Emacs strings,
> which couldn't be CL strings because of text properties, buffer-local
> variables...

Yes, buffer locals have to be implemented in CL, thanks to namespaces we
can hav a cl:string and el:string, of which el:string can be a struct
with interval and a list of properties or some other I don't know. The
hairy stuff with string properties is splitting of intervals and
properties and so on. There is much more than just those. I don't know
yet how efficient it would, I plan to test with generic functions. There
are some people testing with static dispatch for generic functions, but
I am not sure if that would work or not yet (works only for compile time
strings), I haven't had time to experiment yet.

> (Another ansatz might be to make Emacs C core a lib.  I haven't given
> that much thought, but it could be more promising than rewriting the
> whole shit in CL :-).)

Perhaps, but I don't think I personally care longer. Once you test and
learn how nice it is to be able to just eval a function and test the
change, I am not going back to hackig C again, at least not in my spare
time. Life is too short for that.

>> CFFI would also be nice to have so
>> that users can extend Emacs themselves with other libraries and not have to wait
>> for the core devs to do it for them. That would also lessen the burden on
>> maintaining that stuff in the core.
>
> FFI for Emacs once existed, I think Dave Love wrote one, for instance.
> Don't know what became of that.  Might be an issue with interfacing to
> non-free libs.

I didn't know it ever existed in Emacs. 

I though it was a political decision to not have it in Emacs. Which I
understand completely, but after seeing the recent software development,
I believe is hurting "us" much more than "them". As I said in some mail
in the context of keyword args, entire life is a struggle between
choices to maximize the gain. In this case I believe our net loss is
much bigger than gain. Interesting to note is that other GNU projects
are not have political dilemmas about CFFI, for example GCL (Gnu Common
Lisp) or GNU Guile.

I am also aware of emacs-ffi via modules:

https://github.com/tromey/emacs-ffi

But nobody seems to be using it since modules are not big thing in Emacs
community so people barey know about it. That can probably change if
someone wrote a "kille app", or in other words an interesting enough
package that uses it.

Yet another hand, that I haven't seen anywhere, would to load sbcl via
compiled module. Sbcl got ability to save itself as a loadable library,
so it would be possible to basically expose Emacs runtime to sbcl and
export some interface to work with it from within Emacs. I am not sure
if that would be a big win over current way of working via external
process and sockets, but an idea perhaps? I don't have time to try it
myself.

Re: Shrinking the C core

[Gerd Möllmann]

Arthur Miller <arthur.miller@live.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>> I'm pretty sure that CL packages could be added to Emacs as it is, if
>> some people would work on it.
>
> With "CL packages" you mean namespaces? There seem to be already a
> branch that implements them, but I don't know how well it works, I
> haven't tried it.

Yup, that's mine, as a pastime :-).  I have no plans with that.

>> I'm also pretty sure that an incremental + generational GC could be
>> added, at least as an option, because I would have almost done it some
>> 20+ years ago.  It was torpedoed by a patent issue concerning
>> mostly-copying GC.  The patent has since expired. A lot of work, of
>> course.  I think some people do or have done something in this area, but
>> I don't know details.
>
> I am not very familiar wth GC:s implementation more then just some
> bired-eye overview. SBCL recently started to move towards non-moving
> GC to help with the speed, notably when calling native code which does
> not like it's pointers moved underneath, but I am not expert on details
> there, these what I have got from the paper:
>
> https://applied-langua.ge/~hayley/swcl-gc.pdf

Interesting, thanks for the pointer!

>> I'm not at all sure that non-cooperative multi-threading could be added
>> to Emacs.  But I'm also not sure how a CL core would help here.
>
> They are exposing posix threads and have done some work to make at least
> parts of the Lisp system work well with threads, and it seems it is
> working well for many applicaitons.
>
> http://www.lichteblau.com/sbcl/doc/manual/sbcl/Implementation-_0028Linux-x86_0029.html#Implementation-_0028Linux-x86_0029

Yes, I remember to some degree.  I think Daniel Barlow started adding
thread support to SBCL at the time I constributed to CMUCL.  ISTR some
communiaction with him about the implemenattion of dynamic bindings in
the presence of threads.

> I think there is also a missconception in Emacs community that Emacs
> loop itself has to be parallelized; I am not sure it is needed;

Don't know what the "loop" refers to.

I think the biggest problem with uncooperative multi-threading in Emacs
is that Emacs has so much global state.  As a consequence, it's unsafe
to let two threads use anything in the C code in parallel.  Maybe one
could think of using some Python-like GIL, but that's kind of pointless,
isn't it?

> I think
> for many people it would be enough to expose threading in form of "js
> workers" or something like that. It can be done with processes of
> course, but people seem to constantly scream about it in disucssions.

Parallel worker threads have the same problem as above.  And cooperative
threads are there already, I think.

> CL has things like lparallel and green threads built on top of
> hardware threads, so even there is a bit of job already done. I am
> sure all that can be done in Emacs too, but I think, both communities
> would be more helped if we perhaps used sbcl and interested
> individuals helped make sbcl runtime better intead of reduplication
> the entire effort.

Is this realistic?  I mean how many people would be interested to do that?

>> On the other hand, I'm pretty convinced that an Emacs core written in CL
>> would have to be close to 100% compatible with the existing C core to be
>> accepted by users.  That includes a CL rewrite of the C Elisp, including
>> byte code interpreter.
>
> Yes, my conclusion too.
>
>> accepted by users.  That includes a CL rewrite of the C Elisp, including
>> byte code interpreter.
>
> I am not sure how much of byte-interpretter is needed; I was thinking
> how byte interpretter and native compiler fitt there. Oviously since
> sbcl is a compiler, with don't need all that stuff, but I am not sure
> how much of byte code intepretter is needed. I am sure we need to
> understand all of the syntax, since byte code is a valid elisp,
> according to the manual; so the reader have to be able to read the
> syntax I guess, as printed representation, and has to print same stuff
> back to feed into elisp functions.

Not sure what you are saying.  Something has to execute the bytecode,
or not?

>> That's a massive endeavor.  My hair stands up when I remember the
>> compatibility problems I faced with the new redisplay ages ago.
>> Multiply that by some factor > 1.  But maybe that's a burnt child
>> dreading the fire :-).
>
> Yes, I know. I am fully aware that it is an impossibility for someone
> alone, even for a very few. I don't think it is a burnt child, since
> yes, the character renderer of Emacs has to be implemented if Emacs
> applications will run unchanged.

(I think you misunderstoof the phrase.  I'm the burnt child...).

> Hopefully it will be possible to implement Emacs stuff as a special kind
> of terminal/character renderer over some sort of tree/graph structure. I
> think CLOS and CL have much better tools to refactor that stuff than C,
> but what do I know, I haven't tried that and I am not sure if I will
> tbh. I am bolling with the ideas. I have seen what they do in other
> similar CL software (Hemlock, McClIM, Lem), perhaps there is something
> that can be reused there, but I don't know how much and what yet. And,
> yes text properties are a special chapter on its own :).

So, your plan would be to re-implement redisplay in CL.  Good look with
that.  And a ton of other stuff, like the stuff Eli mentioned.  And the
result of the whole massive effort is then 100% compatible with current
Emacs.

That's what I call quatsch, sorry, but honestly.

Work on Lem if you like CL that much :-)

Re: Shrinking the C core

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-tangents@gnu.org
> Date: Thu, 14 Sep 2023 15:03:00 +0200
> 
> I think the biggest problem with uncooperative multi-threading in Emacs
> is that Emacs has so much global state.  As a consequence, it's unsafe
> to let two threads use anything in the C code in parallel.

Indeed, that's the main problem.  Just adding threads to Lisp is easy.

> Maybe one could think of using some Python-like GIL, but that's kind
> of pointless, isn't it?

We already have that with the Lisp threads that we have in Emacs.

Re: Shrinking the C core

[Emanuel Berg]

joakim wrote:

>> I've heard many times that Gnus is slow, but never
>> experienced it. I have 6 servers and 26 groups.
>
> About 700 groups, a mix of nnimap, and nntp.

700 groups! Here we see the elite .se hackers at work.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Emanuel Berg]

Fraga, Eric wrote:

>> How many do people have, for it to get slow?
>
> It's not just the number of groups etc. but also what
> servers you need to access.

"What servers", does that mean what servers in particular or
what server types, i.e. the protocols employed?

Here are my 5 protocols and 6 servers.

nnfolder: archive                    (opened)
 nndraft:                            (opened)
    nnml:                            (opened)
  nnimap: Mail                       (opened)
    nntp: news.eternal-september.org (opened)
    nntp: news.gmane.io              (opened)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Fraga, Eric]

On Friday, 15 Sep 2023 at 07:51, Emanuel Berg wrote:
> "What servers", does that mean what servers in particular or
> what server types, i.e. the protocols employed?

Particular servers.  Some have greater latency than others and having
Emacs hang while it waits for servers to respond can be frustrating.

-- 
Eric S Fraga via gnus (Emacs 30.0.50 2023-09-14) on Debian 12.1

Re: Shrinking the C core

[Emanuel Berg]

Eli Zaretskii wrote:

>> I think the biggest problem with uncooperative
>> multi-threading in Emacs is that Emacs has so much global
>> state. As a consequence, it's unsafe to let two threads use
>> anything in the C code in parallel.
>
> Indeed, that's the main problem. Just adding threads to Lisp
> is easy.

Is there a part that is safe to access in parallel?

If not, or for the part that is unsafe, where is
synchronization of access to be done, in Lisp or in C?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 15 Sep 2023 19:11:22 +0200
> 
> Eli Zaretskii wrote:
> 
> >> I think the biggest problem with uncooperative
> >> multi-threading in Emacs is that Emacs has so much global
> >> state. As a consequence, it's unsafe to let two threads use
> >> anything in the C code in parallel.
> >
> > Indeed, that's the main problem. Just adding threads to Lisp
> > is easy.
> 
> Is there a part that is safe to access in parallel?

No.  The global state cannot be possibly accessed in parallel from two
or more threads.

> If not, or for the part that is unsafe, where is
> synchronization of access to be done, in Lisp or in C?

Both.