redisplay system of emacs

redisplay system of emacs

[alin.s]

I was wondering if it is possible to change the system of redisplay of emacs
to a less obfuscated one.

The current system , based on redisplay internal, for me it is very
obfuscated.

It could be possible to adopt a more clear system and keep all the other
functionalism ?

Could it be possible to take off all the redisplay and create a standardized
system of redisplay, that everybody can understand quickly? Everybody can
write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
but as time as new add ons appear every day, that means that the system is
very standardized and easy to learn.

What are your opinions concerning this problem?



-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27349166.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Eli Zaretskii]

> Date: Wed, 27 Jan 2010 16:19:09 -0800 (PST)
> From: "alin.s" <alinsoar@voila.fr>
> Cc: 
> 
> 
> I was wondering if it is possible to change the system of redisplay of emacs
> to a less obfuscated one.
> 
> The current system , based on redisplay internal, for me it is very
> obfuscated.

It's true that the code is quite arcane, and in a couple of places
barely maintainable, even though it was almost completely overhauled
just 10 years ago.

The high level of the display engine is very simple and can be
explained in a few simple sentences, but the details...

> It could be possible to adopt a more clear system and keep all the other
> functionalism ?

I'm not sure.  Somebody should propose a design, and then we could
discuss it.  But designing and implementing a completely new display
engine that supports everything Emacs supports now is not an easy
task, so it will take a highly motivated and able individual to make
that happen.

> Could it be possible to take off all the redisplay and create a standardized
> system of redisplay, that everybody can understand quickly? Everybody can
> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
> but as time as new add ons appear every day, that means that the system is
> very standardized and easy to learn.

Last time I looked, Mozilla wasn't anywhere close to supporting the
features Emacs has in its display engine.

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 5:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> Could it be possible to take off all the redisplay and create a standardized
>> system of redisplay, that everybody can understand quickly? Everybody can
>> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
>> but as time as new add ons appear every day, that means that the system is
>> very standardized and easy to learn.
>
> Last time I looked, Mozilla wasn't anywhere close to supporting the
> features Emacs has in its display engine.


What features?

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Thu, 28 Jan 2010 10:07:57 +0100
> Cc: "alin.s" <alinsoar@voila.fr>, Emacs-devel@gnu.org
> 
> On Thu, Jan 28, 2010 at 5:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >> Could it be possible to take off all the redisplay and create a standardized
> >> system of redisplay, that everybody can understand quickly? Everybody can
> >> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
> >> but as time as new add ons appear every day, that means that the system is
> >> very standardized and easy to learn.
> >
> > Last time I looked, Mozilla wasn't anywhere close to supporting the
> > features Emacs has in its display engine.
> 
> 
> What features?

The ones described in the nodes "Display Property" and "Special
Properties" in the ELisp manual, for example.

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 12:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Thu, 28 Jan 2010 10:07:57 +0100
>> Cc: "alin.s" <alinsoar@voila.fr>, Emacs-devel@gnu.org
>>
>> On Thu, Jan 28, 2010 at 5:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >
>> >> Could it be possible to take off all the redisplay and create a standardized
>> >> system of redisplay, that everybody can understand quickly? Everybody can
>> >> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
>> >> but as time as new add ons appear every day, that means that the system is
>> >> very standardized and easy to learn.
>> >
>> > Last time I looked, Mozilla wasn't anywhere close to supporting the
>> > features Emacs has in its display engine.
>>
>>
>> What features?
>
> The ones described in the nodes "Display Property" and "Special
> Properties" in the ELisp manual, for example.


Thanks Eli, maybe I understand what you mean but I am a bit surprised.

The Mozilla display engine displays images and text with different
properties very well. And it is very flexible in its way to do that
(since CSS requires that).

I guess it does not directly has something that reminds of "special
properties", but would that be hard to add?

Another point is of course searching for the properties. But Emacs has
its own difficulties there (overlays).

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Thu, 28 Jan 2010 12:47:26 +0100
> Cc: alinsoar@voila.fr, Emacs-devel@gnu.org
> 
> The Mozilla display engine displays images and text with different
> properties very well. And it is very flexible in its way to do that
> (since CSS requires that).

Even images support in Emacs (which is admittedly young and unevolved)
already supports advanced features like :pointer and :map (see the
node "Image Descriptors").  And I wasn't talking about simple text
properties that just change faces or the size of the characters, of
course.  What about properties that need to evaluate Lisp forms or
depend on Lisp-level variables?

> I guess it does not directly has something that reminds of "special
> properties", but would that be hard to add?

I have no idea.

> Another point is of course searching for the properties. But Emacs has
> its own difficulties there (overlays).

The difficulties is not with searching, the difficulties are with
maintaining many overlays.

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 1:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Thu, 28 Jan 2010 12:47:26 +0100
>> Cc: alinsoar@voila.fr, Emacs-devel@gnu.org
>>
>> The Mozilla display engine displays images and text with different
>> properties very well. And it is very flexible in its way to do that
>> (since CSS requires that).
>
> Even images support in Emacs (which is admittedly young and unevolved)
> already supports advanced features like :pointer and :map (see the
> node "Image Descriptors").


?? Of course Mozilla supports these kind of things too.


> And I wasn't talking about simple text
> properties that just change faces or the size of the characters, of
> course.  What about properties that need to evaluate Lisp forms or
> depend on Lisp-level variables?


Yes, I know you are talking about this. But AFAICS Mozilla has the
framework to support similar things. Otherwise it could not support
all the things needed to display certain web pages.


>> I guess it does not directly has something that reminds of "special
>> properties", but would that be hard to add?
>
> I have no idea.
>
>> Another point is of course searching for the properties. But Emacs has
>> its own difficulties there (overlays).
>
> The difficulties is not with searching, the difficulties are with
> maintaining many overlays.


I thought the problem was that it takes long time to search for the
overlays at point for example. Is that not the case?

Re: redisplay system of emacs

[Miles Bader]

Lennart Borgman <lennart.borgman@gmail.com> writes:
> ?? Of course Mozilla supports these kind of things too.

Note that mozilla's display engine makes some dramatically different
assumptions about what is important for the display engine to do.

Most importantly, it does the layout and display calculation for the
_entire page_ at once.  Emacs, by contrast does it on the fly for the
small amount being displayed at the moment.

Mozilla's method allows some nice things -- for instance it makes much
more complicated layout tractable -- but it really really sucks for huge
files, and in general probably isn't such a good idea if the document
tends to change a lot in real time.  Emacs' method, by contrast works
really well for those cases.

I think this difference in approach makes sense given the different
goals of the two applications, and it's not at all clear that Mozilla's
display engine would work well for Emacs too.

Of course, you could try it and see.... :|

-miles

-- 
Cat, n. A soft, indestructible automaton provided by nature to be kicked when
things go wrong in the domestic circle.

Re: redisplay system of emacs

[alin.s]

OK,

Let us keep as good Emacs' method. But let us clarify somehow in the least
details ! 




Miles Bader-2 wrote:
> 
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>> ?? Of course Mozilla supports these kind of things too.
> 
> Note that mozilla's display engine makes some dramatically different
> assumptions about what is important for the display engine to do.
> 
> Most importantly, it does the layout and display calculation for the
> _entire page_ at once.  Emacs, by contrast does it on the fly for the
> small amount being displayed at the moment.
> 
> Mozilla's method allows some nice things -- for instance it makes much
> more complicated layout tractable -- but it really really sucks for huge
> files, and in general probably isn't such a good idea if the document
> tends to change a lot in real time.  Emacs' method, by contrast works
> really well for those cases.
> 
> I think this difference in approach makes sense given the different
> goals of the two applications, and it's not at all clear that Mozilla's
> display engine would work well for Emacs too.
> 
> Of course, you could try it and see.... :|
> 
> -miles
> 
> -- 
> Cat, n. A soft, indestructible automaton provided by nature to be kicked
> when
> things go wrong in the domestic circle.
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27357399.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 3:10 PM, Miles Bader <miles@gnu.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>> ?? Of course Mozilla supports these kind of things too.
>
> Note that mozilla's display engine makes some dramatically different
> assumptions about what is important for the display engine to do.
>
> Most importantly, it does the layout and display calculation for the
> _entire page_ at once.  Emacs, by contrast does it on the fly for the
> small amount being displayed at the moment.
>
> Mozilla's method allows some nice things -- for instance it makes much
> more complicated layout tractable --


Yes, it is necessary for web files.


> but it really really sucks for huge
> files, and in general probably isn't such a good idea if the document
> tends to change a lot in real time.  Emacs' method, by contrast works
> really well for those cases.


True. I have no idea if Mozilla allows this kind a display handling
too. Maybe it does.


> I think this difference in approach makes sense given the different
> goals of the two applications, and it's not at all clear that Mozilla's
> display engine would work well for Emacs too.
>
> Of course, you could try it and see.... :|


Some rainy day, perhaps...


> -miles
>
> --
> Cat, n. A soft, indestructible automaton provided by nature to be kicked when
> things go wrong in the domestic circle.
>

Re: redisplay system of emacs

[Paul R]

Hello,

> Note that mozilla's display engine makes some dramatically different
> assumptions about what is important for the display engine to do.


I found a related discussion in a previous thread, mid-2009 :

  http://lists.gnu.org/archive/html/emacs-devel/2009-07/threads.html#00344


As many emacs youngsters, I feel that emacs really uses too much
home-made code. Surprisingly, Emacs does not benefit that much from the
free software ecosystem. This is certainly connected to the fact that it
has been ahead of its time in many aspects, so it had to build its own
bricks without waiting for others to provide them. But in the meantime
some things has changed, high quality generic free software fragments
has entered the game. Entrusting them rather than home-made code would
certainly be a big job, but would instantly enlarge emacs (indirect)
contributors list.


> Most importantly, it does the layout and display calculation for the
> _entire page_ at once. Emacs, by contrast does it on the fly for the
> small amount being displayed at the moment.

> Mozilla's method allows some nice things -- for instance it makes much
> more complicated layout tractable -- but it really really sucks for
> huge files, and in general probably isn't such a good idea if the
> document tends to change a lot in real time. Emacs' method, by
> contrast works really well for those cases.

The performance of those engines has gotten surprisingly good, and is
still improving at a very fast rate. I would be interested to have some
real-case benchmarks. In the meantime I pasted the whole wikipedia
article into http://ckeditor.com/demo then edited some text in the
middle of the page. It was not blazing fast to be honnest, but still
usable on my 10 years old hardware.

-- 
  Paul

Re: redisplay system of emacs

[David Kastrup]

Paul R <paul.r.ml@gmail.com> writes:

> The performance of those engines has gotten surprisingly good, and is
> still improving at a very fast rate.

It is nowhere near workable for large documents.  Not web pages,
documents.

> I would be interested to have some real-case benchmarks. In the
> meantime I pasted the whole wikipedia article into
> http://ckeditor.com/demo then edited some text in the middle of the
> page. It was not blazing fast to be honnest, but still usable on my 10
> years old hardware.

Uh, Emacs needs to work without noticeable delays and with syntax
highlighting for documents of _book_ size, easily containing thousands
of pages.  In fact, the size of the total document should not affect
editing speed.  And we are not just talking "editing" or fixed documents
here: run-off compilation/log output needs to be fast.

-- 
David Kastrup

Re: redisplay system of emacs

[Lennart Borgman]

On Fri, Jan 29, 2010 at 11:17 AM, David Kastrup <dak@gnu.org> wrote:
>
> Uh, Emacs needs to work without noticeable delays and with syntax
> highlighting for documents of _book_ size, easily containing thousands
> of pages.


Is this necessarily related to the display engine?

Re: redisplay system of emacs

[David Kastrup]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Fri, Jan 29, 2010 at 11:17 AM, David Kastrup <dak@gnu.org> wrote:
>>
>> Uh, Emacs needs to work without noticeable delays and with syntax
>> highlighting for documents of _book_ size, easily containing thousands
>> of pages.
>
>
> Is this necessarily related to the display engine?

Since the Mozilla model renders the whole document (admittedly starting
before it is completely available), I don't see how it should be
otherwise.

-- 
David Kastrup

Re: redisplay system of emacs

[Lennart Borgman]

On Fri, Jan 29, 2010 at 11:30 AM, David Kastrup <dak@gnu.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Fri, Jan 29, 2010 at 11:17 AM, David Kastrup <dak@gnu.org> wrote:
>>>
>>> Uh, Emacs needs to work without noticeable delays and with syntax
>>> highlighting for documents of _book_ size, easily containing thousands
>>> of pages.
>>
>>
>> Is this necessarily related to the display engine?
>
> Since the Mozilla model renders the whole document (admittedly starting
> before it is completely available), I don't see how it should be
> otherwise.


I am not sure that is the Mozilla model. It is just necessary for web pages.

Re: redisplay system of emacs

[Miles Bader]

Lennart Borgman <lennart.borgman@gmail.com> writes:
>> Uh, Emacs needs to work without noticeable delays and with syntax
>> highlighting for documents of _book_ size, easily containing thousands
>> of pages.
>
> Is this necessarily related to the display engine?

Different display engines models are affected differently by changes in
document size.

Mozilla's model, where the _entire document_ is converted into an
internal "layed-out-document" data structure, and then that is rendered,
and which is mainly targeted at documents of "reasonable" length but
with complex layout requirements (web pages), is pretty likely to have
an unacceptably high overhead (time and memory used by the
layed-out-document) for extremely large documents.

Emacs, which doesn't maintain a separate "layed-out-document" data
structure at all, and mainly just worries about rendering what's on the
screen from the raw document, has a very different set of tradeoffs.

Now, it's certainly _possible_ that mozilla actually contains special
code to deal with very-large-but-very-simple document structures, and
use a different method with different tradeoffs to render them, but...
it seems unlikely , and AFAIK, it doesn't.  [but of course, this
"special" code would more or less just be an implementation of what
Emacs does already!]

It's also _possible_ that mozilla's algorithms and data-structures are
so incredibly fast and efficient that this overhead is acceptable even
for extremely large documents on modern machines.  Judging from the
observed performance of e.g. firefox on large but simple web pages,
though, this also seems pretty unlikely.

[Emacs of course also does some processing that is proportional to
document length, e.g., coding/decoding, and at the most basic,
I/O... however such processing is likely to be much simpler/faster (and
memory efficient) than what Mozilla has to do layout the document for
display.]

-Miles

-- 
Friendless, adj. Having no favors to bestow. Destitute of fortune. Addicted to
utterance of truth and common sense.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Date: Fri, 29 Jan 2010 20:03:47 +0900
> Cc: David Kastrup <dak@gnu.org>, emacs-devel@gnu.org
> 
> Emacs, which doesn't maintain a separate "layed-out-document" data
> structure at all

Actually, I think it does: that's the "glyph matrices", which are a
structured description of what should be drawn.  The display-specific
back end then actually draws the glyphs according to what the glyph
matrix tells it.

Not sure if it's relevant or important for this discussion, though.

Re: redisplay system of emacs

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
>> Emacs, which doesn't maintain a separate "layed-out-document" data
>> structure at all
>
> Actually, I think it does: that's the "glyph matrices", which are a
> structured description of what should be drawn.  The display-specific
> back end then actually draws the glyphs according to what the glyph
> matrix tells it.

That is only used for the portion of the document actually being
displayed on the screen at the moment, though, so it's more of a
temporary data structure representing the current screen than a
"layed-out" copy of the entire document like Mozilla uses.

-Miles

-- 
Brain, n. An apparatus with which we think we think.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: dak@gnu.org,  lennart.borgman@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 30 Jan 2010 00:10:16 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Emacs, which doesn't maintain a separate "layed-out-document" data
> >> structure at all
> >
> > Actually, I think it does: that's the "glyph matrices", which are a
> > structured description of what should be drawn.  The display-specific
> > back end then actually draws the glyphs according to what the glyph
> > matrix tells it.
> 
> That is only used for the portion of the document actually being
> displayed on the screen at the moment

But of course: that's all Emacs display engine cares about.

Re: redisplay system of emacs

[Paul R]

Hi David,

> Uh, Emacs needs to work without noticeable delays and with syntax
> highlighting for documents of _book_ size, easily containing thousands
> of pages. In fact, the size of the total document should not affect
> editing speed.

Is it so common to write books in a single monolithic file ? The two
typesetting systems I use from emacs allow (and encourage) to split the
documents in many files. And emacs corresponding modes are designed for
easy navigation between them.

That said, I totally agree that this is a real concern : what happens if
someone wants to _open_ a huge document. As Lennart said, it probably
isn't the job of the display system to restrict the data to render to
a subset, but rather the job of an intermediate layer, between the
document and the rendering engine.


-- 
  Paul

Re: redisplay system of emacs

[David Kastrup]

Paul R <paul.r.ml@gmail.com> writes:

> Hi David,
>
>> Uh, Emacs needs to work without noticeable delays and with syntax
>> highlighting for documents of _book_ size, easily containing
>> thousands of pages. In fact, the size of the total document should
>> not affect editing speed.
>
> Is it so common to write books in a single monolithic file ?

Yes.  Because otherwise search&replace and other stuff become
increasingly painful to do.  In particular if not everybody working with
the same documents is using Emacs.

> The two typesetting systems I use from emacs allow (and encourage) to
> split the documents in many files. And emacs corresponding modes are
> designed for easy navigation between them.

It is fine if Emacs supports more than one way of organizing your work.
It is not fine if it falls apart if you don't do it in a particular way.

-- 
David Kastrup

Re: redisplay system of emacs

[Stefan Monnier]

> In fact, the size of the total document should not affect editing
> speed.  And we are not just talking "editing" or fixed documents here:
> run-off compilation/log output needs to be fast.

That's the ideal goal, yes.  And Emacs is hopefully closer to it than
web browsers.  But Emacs is definitely not there in general: in some
cases (fundamental-mode, no line-number-mode, ...), it's mostly
there, but once you add enough font-locking and stuff, document size
does matter.  So we end up only caring about "fast enough on oldish
hardware when editing fairly large documents", and give up on "still
fast even on C header files larger than your memory" (sorry, Alan ;-)


        Stefan

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Paul R <paul.r.ml@gmail.com>
> Cc: Lennart Borgman <lennart.borgman@gmail.com>,  alinsoar@voila.fr,  Eli Zaretskii <eliz@gnu.org>,  Emacs-devel@gnu.org
> Date: Fri, 29 Jan 2010 11:04:42 +0100
> 
> Hello,
> 
> > Note that mozilla's display engine makes some dramatically different
> > assumptions about what is important for the display engine to do.
> 
> 
> I found a related discussion in a previous thread, mid-2009 :
> 
>   http://lists.gnu.org/archive/html/emacs-devel/2009-07/threads.html#00344

Yeah, this pops up from time to time.

> As many emacs youngsters, I feel that emacs really uses too much
> home-made code. Surprisingly, Emacs does not benefit that much from the
> free software ecosystem.

That's a pretty general assessment.  Any data points other than the
display engine?

> This is certainly connected to the fact that it
> has been ahead of its time in many aspects, so it had to build its own
> bricks without waiting for others to provide them. But in the meantime
> some things has changed, high quality generic free software fragments
> has entered the game. Entrusting them rather than home-made code would
> certainly be a big job, but would instantly enlarge emacs (indirect)
> contributors list.

"Big job" is an understatement of the century.  Most of these "generic
free software fragments" are not written with Emacs peculiar
requirements in mind, and don't have a Lisp API.

Where these issues don't matter, Emacs does use existing software,
like with the font support and display-specific back ends that
actually render the characters on glass.

> The performance of those engines has gotten surprisingly good, and is
> still improving at a very fast rate. I would be interested to have some
> real-case benchmarks.

One of the most important benchmarks would be inserting or deleting a
small amount of text in a very large document.  As an interactive
editor, Emacs has to deal with this most of the time.  Not sure if
those other display engines are optimized to this modus operandi.

Re: redisplay system of emacs

[Paul R]

Hi Eli,

>> As many emacs youngsters, I feel that emacs really uses too much
>> home-made code. Surprisingly, Emacs does not benefit that much from
>> the free software ecosystem.

Eli> That's a pretty general assessment. Any data points other than the
Eli> display engine?

Emacs Lisp and all the librairies that emacs hackers had to implement on
top of it, IOW the 'emacs lisp standard library'.

Since 80ies, many languages appeared, and many of them meet very well
the requirements to extend a text editor. And because they are
general-purpose language, they do much more. Designing a language,
implementing it, maintaining it, providing a large and up to date
standard library, is a project on its own. Scheme, Ruby or Python come
to mind. The latters, at least, come with extensive support to parsing,
file operations, networking and so on.

Eli> "Big job" is an understatement of the century. Most of these
Eli> "generic free software fragments" are not written with Emacs
Eli> peculiar requirements in mind, and don't have a Lisp API.

Yes you are right. I'm mostly thinking loud how could be designed emacs
in today landscape.


-- 
  Paul

Re: redisplay system of emacs

[David Kastrup]

Paul R <paul.r.ml@gmail.com> writes:

> Hi Eli,
>
>>> As many emacs youngsters, I feel that emacs really uses too much
>>> home-made code. Surprisingly, Emacs does not benefit that much from
>>> the free software ecosystem.
>
> Eli> That's a pretty general assessment. Any data points other than the
> Eli> display engine?
>
> Emacs Lisp and all the librairies that emacs hackers had to implement on
> top of it, IOW the 'emacs lisp standard library'.
>
> Since 80ies, many languages appeared, and many of them meet very well
> the requirements to extend a text editor. And because they are
> general-purpose language, they do much more.

So where are all those powerful extensible text editors?

> Designing a language, implementing it, maintaining it, providing a
> large and up to date standard library, is a project on its
> own. Scheme, Ruby or Python come to mind. The latters, at least, come
> with extensive support to parsing, file operations, networking and so
> on.

So where are all those powerful extensible text editors?

-- 
David Kastrup

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Paul R <paul.r.ml@gmail.com>
> Date: Fri, 29 Jan 2010 14:06:20 +0100
> Cc: Emacs-devel@gnu.org
> 
> >> As many emacs youngsters, I feel that emacs really uses too much
> >> home-made code. Surprisingly, Emacs does not benefit that much from
> >> the free software ecosystem.
> 
> Eli> That's a pretty general assessment. Any data points other than the
> Eli> display engine?
> 
> Emacs Lisp and all the librairies that emacs hackers had to implement on
> top of it, IOW the 'emacs lisp standard library'.

And what replacements do you see for that in the ``free software
ecosystem''?  Apart of replacing Lisp with another language, an
endeavor that no one succeeded in to this day (although there were
attempts, and one of them is even still alive), what else could be
done to replace all that?

Re: redisplay system of emacs

[Chong Yidong]

Paul R <paul.r.ml@gmail.com> writes:

> Since 80ies, many languages appeared, and many of them meet very well
> the requirements to extend a text editor. And because they are
> general-purpose language, they do much more. Designing a language,
> implementing it, maintaining it, providing a large and up to date
> standard library, is a project on its own. Scheme, Ruby or Python come
> to mind. The latters, at least, come with extensive support to parsing,
> file operations, networking and so on.

This has been discussed before ;-)

http://lists.gnu.org/archive/html/emacs-devel/2006-04/msg00030.html

Re: redisplay system of emacs

[Stefan Monnier]

Eli> That's a pretty general assessment. Any data points other than the
Eli> display engine?

Most of what Emacs's code offers is available elsewhere one way or
another in *apparently* better shape and with an active community.
That's true of the redisplay as we've been discussing, it's true of the
extension language, the GC, the coding-conversion libraries, the
wiget/menu code, and probably a lot more.

So, yes, it's probably fairly easy to do an Emacs-like editor using
another extension language etc... and the result is likely to be
cleaner, more efficient, better maintained code in many ways.
Actually such Emacs-like editors exist, using extension languages like
CommonLisp, OCaml, Haskell, Python, younameit.

The problem is that they're not Emacs, so they start with a small user
community and it's hard for them to grow.  At some point, I've
considered the possibility to switch to one of those (where the
extension language is statically typed ;-), but ... they didn't have
PCL-CVS, diff-mode, Gnus, ... so it was really an uphill battle.
Maybe later.


        Stefan

Re: redisplay system of emacs

[Óscar Fuentes]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

[snip]

> At some point, I've considered the possibility to switch to one of
> those (where the extension language is statically typed ;-), but
> ... they didn't have PCL-CVS, diff-mode, Gnus, ... so it was really an
> uphill battle.  Maybe later.

Now that you are on the mod of discussing far-fetched possibilities,
think on a language that...

* Statically typed ;-) No fancy type system, though. Dynamic typing can
  be emulated.

* Can easily interface with C/C++. Registering a C function, type or
  variable is a one-liner. In principle, it is possible to automatize
  the task.

* Optionally, can compile to efficient native code.

* Lisp-like syntax.

* Has defmacro (yay!).

* Has eval.

* Has instrospection facilities.

Whenever I look at the Emacs C codebase, I think that a language like
the described above would do wonders replacing a good chunk of it.

I have such language with the corresponding compiler. It has its warts,
no doubt. But, as a maintainer of the Emacs C code, how it looks the
perspective of using a language like that instead of the Emacs C
pseudo-dialect? Is it painful enough to maintain the C codebase to make
you dream about switching to something else?

Re: redisplay system of emacs

[Richard Stallman]

    Whenever I look at the Emacs C codebase, I think that a language like
    the described above would do wonders replacing a good chunk of it.

The code written in C in Emacs is mostly in C so it can be as fast as
possible.  To move it into such a language would be a big practical
loss.

There is some code that was originally in C but doesn't need to be
fast.  We have rewritten some of that code in Lisp in recent years.
If more such C code remains, we could move that to Lisp too.

Re: redisplay system of emacs

[Óscar Fuentes]

Richard Stallman <rms@gnu.org> writes:

>     Whenever I look at the Emacs C codebase, I think that a language like
>     the described above would do wonders replacing a good chunk of it.
>
> The code written in C in Emacs is mostly in C so it can be as fast as
> possible.  To move it into such a language would be a big practical
> loss.

Just a clarification:

You missed the part that says "Optionally, can compile to efficient
native code." Here, efficient means "at least as good as C".

Nowadays, thanks to projects like LLVM, it is easy to roll an
industrial-grade optimizing compiler if the language runtime
requirements are not too far from C. It is straightforward to design a
language that equals C in speed of execution and that is way more
expressive.

Actually, some of those languages often compile applications to code
that is faster than their C counterpart, as they are more
optimizer-friendly.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 30 Jan 2010 13:51:41 +0100
> 
> Nowadays, thanks to projects like LLVM, it is easy to roll an
> industrial-grade optimizing compiler if the language runtime
> requirements are not too far from C. It is straightforward to design a
> language that equals C in speed of execution and that is way more
> expressive.
> 
> Actually, some of those languages often compile applications to code
> that is faster than their C counterpart, as they are more
> optimizer-friendly.

But switching to a language that is significantly less widespread than
C would mean significantly less potential contributors.

We could learn from Texinfo experience: the next release replaces the
venerable makeinfo implementation in C with a modified texi2html,
which is written in Perl.  I suspect that, once that happens, the only
one who will hack the translator will be its author and no one else.
We shall see.

Re: redisplay system of emacs

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Nowadays, thanks to projects like LLVM, it is easy to roll an
>> industrial-grade optimizing compiler if the language runtime
>> requirements are not too far from C. It is straightforward to design a
>> language that equals C in speed of execution and that is way more
>> expressive.
>> 
>> Actually, some of those languages often compile applications to code
>> that is faster than their C counterpart, as they are more
>> optimizer-friendly.
>
> But switching to a language that is significantly less widespread than
> C would mean significantly less potential contributors.

The Emacs C codebase is, to most effects, written on a dialect of C that
requires learning by its own.

In principle, it is possible to create a language which is similar
enough to Lisp that the amount of learning it requires is similar in
magnitude to the Emacs C dialect for someone who already knows Lisp, but
being as optimizable as C and much more expressive and
maintainable. (Some people would retort that this was already tried:
that Emacs variant written on Common Lisp)

I know this is just wild dreaming given the current status of Emacs.

[snip]

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 30 Jan 2010 20:21:18 +0100
> 
> > But switching to a language that is significantly less widespread than
> > C would mean significantly less potential contributors.
> 
> The Emacs C codebase is, to most effects, written on a dialect of C that
> requires learning by its own.

Yes, but that's hardly the same as learning an entirely new language.

Re: redisplay system of emacs

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> 
>> Actually, some of those languages often compile applications to code
>> that is faster than their C counterpart, as they are more
>> optimizer-friendly.
>
> But switching to a language that is significantly less widespread than
> C would mean significantly less potential contributors.

The majority of Emacs is written in Elisp.  Which is significantly less
widespread than pretty much anything else.

I don't think that in the last decade or so people contributed
non-trivial C code to Emacs without knowledge of Elisp.

-- 
David Kastrup

Re: redisplay system of emacs

[Richard Stallman]

    Nowadays, thanks to projects like LLVM, it is easy to roll an
    industrial-grade optimizing compiler if the language runtime
    requirements are not too far from C. It is straightforward to design a
    language that equals C in speed of execution and that is way more
    expressive.

I am skeptical.  In any case, if we use a compiled language we will
compile it with GCC.  We must support other GNU packages whenever
possible.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-devel@gnu.org
> Date: Fri, 29 Jan 2010 13:35:28 -0500
> 
> Eli> That's a pretty general assessment. Any data points other than the
> Eli> display engine?
> 
> Most of what Emacs's code offers is available elsewhere one way or
> another in *apparently* better shape and with an active community.
> That's true of the redisplay as we've been discussing, it's true of the
> extension language, the GC, the coding-conversion libraries, the
> wiget/menu code, and probably a lot more.
> 
> So, yes, it's probably fairly easy to do an Emacs-like editor using
> another extension language etc...

I thought the issue was not creation of a new editor, but using in
Emacs code written elsewhere for some tasks Emacs now does with code
written specifically for it.

> At some point, I've considered the possibility to switch to one of
> those (where the extension language is statically typed ;-), but
> ... they didn't have PCL-CVS, diff-mode, Gnus, ... so it was really
> an uphill battle.

Heh, that's how I came to Emacs 15 years ago: I got tired
reimplementing in a workalike all the features I missed and in Emacs
were there to be used...

Re: redisplay system of emacs

[Stefan Monnier]

>> So, yes, it's probably fairly easy to do an Emacs-like editor using
>> another extension language etc...
> I thought the issue was not creation of a new editor, but using in
> Emacs code written elsewhere for some tasks Emacs now does with code
> written specifically for it.

Given that there's no other Elisp implementation that I know of, if you
want to use one of the existing language implementations to replace
Emacs's own code...


        Stefan

Re: redisplay system of emacs

[Stephen J. Turnbull]

Stefan Monnier writes:

 > Given that there's no other Elisp implementation that I know of, if you
 > want to use one of the existing language implementations to replace
 > Emacs's own code...

Eh?<wink>  I think Miles might also <wink>, too ....  And Mike Sperber
and students, for that matter.

There's also librepl, used in the sawfish window manager for X11,
which was deliberately modeled on Elisp.  Obviously the domain-
specific functions will differ, but the Lisp environment was indeed
similar.

Re: redisplay system of emacs

[Fabian Ezequiel Gallina]

2010/1/29 Stefan Monnier <monnier@iro.umontreal.ca>:
>
> So, yes, it's probably fairly easy to do an Emacs-like editor using
> another extension language etc... and the result is likely to be
> cleaner, more efficient, better maintained code in many ways.
> Actually such Emacs-like editors exist, using extension languages like
> CommonLisp, OCaml, Haskell, Python, younameit.
>
> The problem is that they're not Emacs, so they start with a small user
> community and it's hard for them to grow.  At some point, I've
> considered the possibility to switch to one of those (where the
> extension language is statically typed ;-), but ... they didn't have
> PCL-CVS, diff-mode, Gnus, ... so it was really an uphill battle.
> Maybe later.
>

I think Emacs Lisp is a great extension language. To be honest, when I
was changing my current programing editor and was testing VIM and
Emacs, I've choosen Emacs mainly because of Lisp (...and because I
hated switching modes in VIM).

I have to say also, that beign (mainly) a Python programmer I feel
Emacs Lisp very familiar in some places, and I guess it's true what
people say that is an enlightning language.

There is a really nice project which allows you to extend GNU/Emacs
using Python called Pymacs[0]. I guess that instead of changing Emacs
Lisp we should develop and encourage this kind of interfaces. This
will keep everyone happy, people who actually wants to learn Emacs
Lisp and people who don't want quit using their preferred language.

An example of an extension using Pymacs would be ropemacs[1] which is
a Python refactoring tool (that saves my life everyday).

[0] http://pymacs.progiciels-bpi.ca/pymacs.html
[1] http://rope.sourceforge.net/ropemacs.html


Regards,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: redisplay system of emacs

[David Kastrup]

Fabian Ezequiel Gallina <galli.87@gmail.com> writes:

> 2010/1/29 Stefan Monnier <monnier@iro.umontreal.ca>:
>>
>> So, yes, it's probably fairly easy to do an Emacs-like editor using
>> another extension language etc... and the result is likely to be
>> cleaner, more efficient, better maintained code in many ways.
>> Actually such Emacs-like editors exist, using extension languages
>> like CommonLisp, OCaml, Haskell, Python, younameit.
>>
>> The problem is that they're not Emacs, so they start with a small
>> user community and it's hard for them to grow.  At some point, I've
>> considered the possibility to switch to one of those (where the
>> extension language is statically typed ;-), but ... they didn't have
>> PCL-CVS, diff-mode, Gnus, ... so it was really an uphill battle.
>> Maybe later.
>
> I think Emacs Lisp is a great extension language.

I don't think so.  It is a functional language used in a procedural
style, causing permanent changes as its ultimate purpose.  Not even with
lexical bindings.

It does automatic allocation and garbage collection, is concise and with
data structures catered to the task at hand.  Mostly.

If it were a great extension language, it would be in use outside of
Emacs.

Things like Lua would meet the bill quite better, but they are too late
in the game now.

Emacs is not a program, it is an ecosystem.  And ecosystems play by
different rules than individuals.  Emacs has not yet reached
equilibrium, the state where bit rot destroys functionality at the same
pace as new stuff gets added.

Moving that ecosystem to a different language is about as ambitious as
moving a carbon-based ecosystem to silicon-based.

You don't know where to start.

-- 
David Kastrup

Re: redisplay system of emacs

[Stefan Monnier]

> I think Emacs Lisp is a great extension language.

It's *very* difficult to make it fast, so it leaves a lot of work on
C part of the code.


        Stefan

Re: redisplay system of emacs

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Paul R <paul.r.ml@gmail.com>
>
>> As many emacs youngsters, I feel that emacs really uses too much
>> home-made code. Surprisingly, Emacs does not benefit that much from
>> the free software ecosystem.
>
> That's a pretty general assessment.  Any data points other than the
> display engine?

One main data point is that its copyright assignment policies don't
allow taking XEmacs code.

Since only explicitly FSF copyright-assigned contributions can be used
in Emacs, Emacs' involvement in the "free software ecosystem" as such is
purely contributing, not taking.  As far as generally available material
such as toolkit libraries are concerned, Emacs does not benefit more or
less than proprietary applications from the "free software ecosystem".

It is a consciously made tradeoff.  It has been expensive at times.  Not
everything comes cheap.

-- 
David Kastrup

Re: redisplay system of emacs

[Ken Hori]

+1 vote.

On Wed, Jan 27, 2010 at 4:19 PM, alin.s <alinsoar@voila.fr> wrote:
>
> I was wondering if it is possible to change the system of redisplay of emacs
> to a less obfuscated one.
>
> The current system , based on redisplay internal, for me it is very
> obfuscated.
>
> It could be possible to adopt a more clear system and keep all the other
> functionalism ?
>
> Could it be possible to take off all the redisplay and create a standardized
> system of redisplay, that everybody can understand quickly? Everybody can
> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
> but as time as new add ons appear every day, that means that the system is
> very standardized and easy to learn.
>
> What are your opinions concerning this problem?
>
>
>
> --
> View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27349166.html
> Sent from the Emacs - Dev mailing list archive at Nabble.com.
>
>
>
>

redisplay system of emacs

[Stephen J. Turnbull]

alin.s writes:

 > Could it be possible to take off all the redisplay and create a
 > standardized system of redisplay, that everybody can understand
 > quickly? Everybody can write an add-on for Mozilla.

But nobody (except a few experts) messes with redisplay.  Redisplay is
*hard*.  There are very few redisplays capable of doing what Emacs can
do.  Emacs, XEmacs.  *Maybe* Gecko (Mozilla) or Webkit (Safari).  On
the other hand, the way you write your post, I doubt it requires
changes to redisplay, but I could be wrong.  So I think you need to
explain what it is you want to do.

Note that Mozilla has a high-level language (XUL) for managing its
display (and other aspects of user interface).  Add-ons are written in
that language.  Well, so does Emacs: Emacs Lisp.

 > I do not know how redisplay of Mozilla works, but as time as new
 > add ons appear every day, that means that the system is very
 > standardized and easy to learn.

Emacs is also standardized and easy to learn.  There are just many
fewer people interested in writing Lisp libraries than there are
writing Mozilla add-ons.

Re: redisplay system of emacs

[alin.s]

When I first started to write the tabs, I started from redisplay_internal.
Fortunatelly, I found the solution such that I did not touch that code.

But trying to read it, it was impossible for me to make a general coherence
of the code. Only in parts I could understand it.

Yes, in genral the redisplay is superior with its overlays, but many
overlays like in linum-mode make it works bad.

I did not invest much effort in trying to read it, because I am afraid that
apart a few persons who were motivated to write some code in that part,
nobody knows it.

I wanted to say that it should be adopted a system that everybody can
understand, in order for everybody interested to be able to gain a coherence
in thought about that part of emacs.





-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27356155.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Stephen J. Turnbull]

alin.s writes:

 > 
 > When I first started to write the tabs, I started from redisplay_internal.
 > Fortunatelly, I found the solution such that I did not touch that code.

It's not "fortunate", it's designed that way.  Redisplay is the lowest
level of the display code, and tightly bound to the event loop.  If
there's an X display involved, life is very difficult, because X
protocol is totally asynchronous.

But most things (like adding a tab widget) should be able to work with
higher level display features.

Re: redisplay system of emacs

[alin.s]

An improvement in redisplay for X can be done by defining in the edit area of
every window a subwindow for every character. For a window of geometry
200x70 of characters, it would be 1400 windows registered in X-server.

The advantage is that redisplay would work automatically; every character
has associated its own expose event; the event PointerMotionMask would
simply signify LeaveWindow and EnterWindow; so it will always be able to be
captured without resource consuming.

To say more, in order to clear a character it would require no computation,
but only a simply call of XClearWindow().

Every window could have its own font.

And to say more, an image of high dimension will be divided in many
subwindows, and emacs will be able to display images normally, not as a huge
glyph.

And finally, because this structure is identical to the geometry of the
console, the code for X and console can be unified in many places.

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27560255.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Juanma Barranquero]

On Fri, Feb 12, 2010 at 09:31, alin.s <alinsoar@voila.fr> wrote:

> An improvement in redisplay for X can be done by defining in the edit area of
> every window a subwindow for every character. For a window of geometry
> 200x70 of characters, it would be 1400 windows registered in X-server.

Wouldn't that be 14,000?

    Juanma

Re: redisplay system of emacs

[alin.s]

Juanma Barranquero wrote:
> 
>> An improvement in redisplay for X can be done by defining in the edit
>> area of
>> every window a subwindow for every character. For a window of geometry
>> 200x70 of characters, it would be 1400 windows registered in X-server.
> 
> Wouldn't that be 14,000?
> 

I do not pretend the solution to be realistic, I just asked if it is good or
not.

The only structure that would be allocated is a lot of little windows. Apart
from this, the computations on the client side are simpler.



-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27563728.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Jan Djärv]

alin.s skrev:
> An improvement in redisplay for X can be done by defining in the edit area of
> every window a subwindow for every character. For a window of geometry
> 200x70 of characters, it would be 1400 windows registered in X-server.

You are mad.  Everything would be 1400 times more.  1400 times more calls to 
the X server, 1400 times more GC:s created, 1400 times more events from the X 
server, 1400 times more Xft structures (XftDraws for example) created.
Emacs would be at least 1400 times slower.

> 
> The advantage is that redisplay would work automatically; every character
> has associated its own expose event; the event PointerMotionMask would
> simply signify LeaveWindow and EnterWindow; so it will always be able to be
> captured without resource consuming.

But Emacs still needs to figure out what character that is in that window in 
order to redraw it.

> 
> To say more, in order to clear a character it would require no computation,
> but only a simply call of XClearWindow().
> 
> Every window could have its own font.

So say we have 1400 windows each with a different font with a different size? 
  How do you purpose we lay this out?  Instead of laying out characters we are 
now laying out windows.  Same problem, but with an enormous overhead.

> 
> And to say more, an image of high dimension will be divided in many
> subwindows, and emacs will be able to display images normally, not as a huge
> glyph.

Again, making the display of an image to be so much slower, because instead of 
one (ideally), call to the X server, we now have one per window.  And what a 
nightmare to figure out what part of an image that needs to be redrawn and 
moved if the user changes the size of the Emacs frame...

> 
> And finally, because this structure is identical to the geometry of the
> console, the code for X and console can be unified in many places.
> 

Not very likely.

Did 1:st of April arrive early?

	Jan D.

Re: redisplay system of emacs

[alin.s]

alin.s skrev:
> An improvement in redisplay for X can be done by defining in the edit area
> of
> every window a subwindow for every character. For a window of geometry
> 200x70 of characters, it would be 1400 windows registered in X-server.

You are mad.  Everything would be 1400 times more.  1400 times more calls to 
the X server, 1400 times more GC:s created, 1400 times more events from the
X 
server, 1400 times more Xft structures (XftDraws for example) created.
Emacs would be at least 1400 times slower.

The computations are not done as you say. Please read the X windows manual.

If you make 1000 identical calls of Xlib functions, Xlib put them into its
queue, and send them once.

On the other side, taking into account that I forgot a ZERO, we could have
many times calls to Xlib, but anyways, not too much, taking into account the
queue of Xlib.

I do not know what is bad to keep in the server 100.000 of little
structures. Apart from memory consuming, no problem of speed.

You do not need 14.000 Graphics Contextes, only 1 for window is enough!

It is not 14000 times slower, it is a few times faster.

It is many times more consuming of memory of the server side. Please learn
Xlib programming to understand it.

> 
> To say more, in order to clear a character it would require no
> computation,
> but only a simply call of XClearWindow().
> 
> Every window could have its own font.

So say we have 1400 windows each with a different font with a different
size? 
  How do you purpose we lay this out?  Instead of laying out characters we
are 
now laying out windows.  Same problem, but with an enormous overhead.

Same problem, only 1 GC for every window.

> 
> And to say more, an image of high dimension will be divided in many
> subwindows, and emacs will be able to display images normally, not as a
> huge
> glyph.

Again, making the display of an image to be so much slower, because instead
of 
one (ideally), call to the X server, we now have one per window.  And what a 
nightmare to figure out what part of an image that needs to be redrawn and 
moved if the user changes the size of the Emacs frame...

We do have 1 call for window, and 1 network message to the server for every
update.

The only problem in this implementation would be to compute the size of
every window, and to compute its coordinates.


> 
> And finally, because this structure is identical to the geometry of the
> console, the code for X and console can be unified in many places.
> 

Not very likely.

I do not know. Maybe.


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27563610.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Jan Djärv]

alin.s skrev:
> 
> 
> alin.s skrev:
>> An improvement in redisplay for X can be done by defining in the edit area
>> of
>> every window a subwindow for every character. For a window of geometry
>> 200x70 of characters, it would be 1400 windows registered in X-server.
> 
> You are mad.  Everything would be 1400 times more.  1400 times more calls to 
> the X server, 1400 times more GC:s created, 1400 times more events from the
> X 
> server, 1400 times more Xft structures (XftDraws for example) created.
> Emacs would be at least 1400 times slower.
> 
> The computations are not done as you say. Please read the X windows manual.

I have done so for every edition since X10.

> 
> If you make 1000 identical calls of Xlib functions, Xlib put them into its
> queue, and send them once.
> 

Yes it does put calls in a queue.  All 1400 of them instead of one.  But given 
how IPC works, in reality it will be many writes to the socket anyway, even if 
the queue tries to minimize that.  And the amount of data is still 1400 (or 
14000) times more.  X does not squeeze several calls into one, there are still 
several calls.  And they are not identical anyway, they are for different windows.

> 
> I do not know what is bad to keep in the server 100.000 of little
> structures. Apart from memory consuming, no problem of speed.

But you have to keep track of all handles to these data as well in Emacs so 
both have to have 14000 times more data.

> 
> You do not need 14.000 Graphics Contextes, only 1 for window is enough!

Ok, so GC:s wasn't a good example.

> It is many times more consuming of memory of the server side. Please learn
> Xlib programming to understand it.

No need to be condescending, it doesn't help your cause.  I've done X 
programming since the late 80:s.

> 
>> To say more, in order to clear a character it would require no
>> computation,
>> but only a simply call of XClearWindow().
>>
>> Every window could have its own font.
> 
> So say we have 1400 windows each with a different font with a different
> size? 
>   How do you purpose we lay this out?  Instead of laying out characters we
> are 
> now laying out windows.  Same problem, but with an enormous overhead.
> 
> Same problem, only 1 GC for every window.

No it is not the same problem.  For every window you will have to tell X where 
it will be positioned, i,e. x and y and width and height.  If every window 
have its own font with its own size you have to calculate all this for every 
window, and the recalculate it for all windows when a font changes in one 
window.  Sounds like what redisplay does today.

> 
>> And to say more, an image of high dimension will be divided in many
>> subwindows, and emacs will be able to display images normally, not as a
>> huge
>> glyph.
> 
> Again, making the display of an image to be so much slower, because instead
> of 
> one (ideally), call to the X server, we now have one per window.  And what a 
> nightmare to figure out what part of an image that needs to be redrawn and 
> moved if the user changes the size of the Emacs frame...
> 
> We do have 1 call for window, and 1 network message to the server for every
> update.

... times 14000 windows...

> 
> The only problem in this implementation would be to compute the size of
> every window, and to compute its coordinates.

So you have gained nothing really, redisplay is the hard part.

	Jan D.

Re: redisplay system of emacs

[alin.s]

I agree that it is not a good plan . I thought that the X server notifies
you exactly the characters that need redisplaying, and so it would be very
clear the redisplay for everybody !





Jan Djärv wrote:
> 
> alin.s skrev:
>> 
>> 
>> alin.s skrev:
>>> An improvement in redisplay for X can be done by defining in the edit
>>> area
>>> of
>>> every window a subwindow for every character. For a window of geometry
>>> 200x70 of characters, it would be 1400 windows registered in X-server.
>> 
>> You are mad.  Everything would be 1400 times more.  1400 times more calls
>> to 
>> the X server, 1400 times more GC:s created, 1400 times more events from
>> the
>> X 
>> server, 1400 times more Xft structures (XftDraws for example) created.
>> Emacs would be at least 1400 times slower.
>> 
>> The computations are not done as you say. Please read the X windows
>> manual.
> 
> I have done so for every edition since X10.
> 
>> 
>> If you make 1000 identical calls of Xlib functions, Xlib put them into
>> its
>> queue, and send them once.
>> 
> 
> Yes it does put calls in a queue.  All 1400 of them instead of one.  But
> given 
> how IPC works, in reality it will be many writes to the socket anyway,
> even if 
> the queue tries to minimize that.  And the amount of data is still 1400
> (or 
> 14000) times more.  X does not squeeze several calls into one, there are
> still 
> several calls.  And they are not identical anyway, they are for different
> windows.
> 
>> 
>> I do not know what is bad to keep in the server 100.000 of little
>> structures. Apart from memory consuming, no problem of speed.
> 
> But you have to keep track of all handles to these data as well in Emacs
> so 
> both have to have 14000 times more data.
> 
>> 
>> You do not need 14.000 Graphics Contextes, only 1 for window is enough!
> 
> Ok, so GC:s wasn't a good example.
> 
>> It is many times more consuming of memory of the server side. Please
>> learn
>> Xlib programming to understand it.
> 
> No need to be condescending, it doesn't help your cause.  I've done X 
> programming since the late 80:s.
> 
>> 
>>> To say more, in order to clear a character it would require no
>>> computation,
>>> but only a simply call of XClearWindow().
>>>
>>> Every window could have its own font.
>> 
>> So say we have 1400 windows each with a different font with a different
>> size? 
>>   How do you purpose we lay this out?  Instead of laying out characters
>> we
>> are 
>> now laying out windows.  Same problem, but with an enormous overhead.
>> 
>> Same problem, only 1 GC for every window.
> 
> No it is not the same problem.  For every window you will have to tell X
> where 
> it will be positioned, i,e. x and y and width and height.  If every window 
> have its own font with its own size you have to calculate all this for
> every 
> window, and the recalculate it for all windows when a font changes in one 
> window.  Sounds like what redisplay does today.
> 
>> 
>>> And to say more, an image of high dimension will be divided in many
>>> subwindows, and emacs will be able to display images normally, not as a
>>> huge
>>> glyph.
>> 
>> Again, making the display of an image to be so much slower, because
>> instead
>> of 
>> one (ideally), call to the X server, we now have one per window.  And
>> what a 
>> nightmare to figure out what part of an image that needs to be redrawn
>> and 
>> moved if the user changes the size of the Emacs frame...
>> 
>> We do have 1 call for window, and 1 network message to the server for
>> every
>> update.
> 
> ... times 14000 windows...
> 
>> 
>> The only problem in this implementation would be to compute the size of
>> every window, and to compute its coordinates.
> 
> So you have gained nothing really, redisplay is the hard part.
> 
> 	Jan D.
> 
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27564505.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[alin.s]

Jan Djärv wrote:
> 
>> If you make 1000 identical calls of Xlib functions, Xlib put them into
>> its
>> queue, and send them once.
>> 
> 
> Yes it does put calls in a queue.  All 1400 of them instead of one.  But
> given 
> how IPC works, in reality it will be many writes to the socket anyway,
> even if 
> the queue tries to minimize that.  And the amount of data is still 1400
> (or 
> 14000) times more.  X does not squeeze several calls into one, there are
> still 
> several calls.  And they are not identical anyway, they are for different
> windows.
> 
> 

You can keep on server side many little windows, and on the client side you
can keep the image of the screen, and the little windows notifies you what
part of the image was changed, you update the local bitmap, and when the
bitmap is created, you send it to the server with only 1 X lib call.

The idea is that you ask the server to tell you what must be refreshed on
the scree, and you need no more obfuscated internal redisplay.


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27564728.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Jan Djärv]

alin.s skrev:
> 
> 
> You can keep on server side many little windows, and on the client side you
> can keep the image of the screen, and the little windows notifies you what
> part of the image was changed, you update the local bitmap, and when the
> bitmap is created, you send it to the server with only 1 X lib call.
> 
> The idea is that you ask the server to tell you what must be refreshed on
> the scree, and you need no more obfuscated internal redisplay.
> 

This is done already, the expose event contains the rectangle that needs to be 
updated.  And if you need more exact information you can use the XDamage 
extension.

What you are describing is basically double buffering.  Since Gtk+ does this 
already, it makes sense for Emacs to go that way too, either by using more 
Gtk+ (we now turn double buffering off), or develop its own.

	Jan D.

Re: redisplay system of emacs

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> What you are describing is basically double buffering.  Since Gtk+
> does this already, it makes sense for Emacs to go that way too,

I don't see this dependency.

> either by using more Gtk+ (we now turn double buffering off),

In particular if one can tell Gtk+ not to double buffer.

> or develop its own.

I don't think that the performance in partly obscured windows is really
that important.

Or maybe I don't understand what this is about.

-- 
David Kastrup

Re: redisplay system of emacs

[Jan Djärv]

David Kastrup skrev:

> I don't think that the performance in partly obscured windows is really
> that important.
> 

No, it isn't.  But if you can get info about exactly what parts needs to be 
updated, maybe redisplay could do less work.  But the main point is that if 
you use double buffering (i.e. first render in an off-screen pixmap, and then 
copy the pixmap to the X window in one go), you can use a very crude 
redisplay, since you always copy from an up-to-date copy.  No need to worry 
about flickering.

	Jan D.

Re: redisplay system of emacs

[alin.s]

I did not want to state categorically that such or such solution is the good
one.

For example, if every emacs-window would be composed of lots of subwindows
on server side, that lots of sub-windows could be completely hidden. Never
use them to write something in them. They just report exactly the part of
the buffer that should be redisplayed. They only catch and report events.
They are invisible.

Redisplay_internal does lots and lots of computations to detect which part
of the buffer should be redisplayed at a given moment. All these
computations can be avoided using a system of subwindows (1 for every char).
That was my idea: to replace redisplay_internal with something simple and
efficient. Something simple and efficient is to keep in the emacs a vector
of little windows, and to register in the server all these windows.

The structure of windows is changed only when configure event is caught or
when one changes the font, or split windows, etc. So very rarely.

95% of the operations of redisplay can be covered with the system of little
windows registered in the server. No more need of redisplay_internal for X.





David Kastrup wrote:
> 
> Jan Djärv <jan.h.d@swipnet.se> writes:
> 
>> What you are describing is basically double buffering.  Since Gtk+
>> does this already, it makes sense for Emacs to go that way too,
> 
> I don't see this dependency.
> 
>> either by using more Gtk+ (we now turn double buffering off),
> 
> In particular if one can tell Gtk+ not to double buffer.
> 
>> or develop its own.
> 
> I don't think that the performance in partly obscured windows is really
> that important.
> 
> Or maybe I don't understand what this is about.
> 
> -- 
> David Kastrup
> 
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27566385.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[David Kastrup]

"alin.s" <alinsoar@voila.fr> writes:

> I did not want to state categorically that such or such solution is the good
> one.
>
> For example, if every emacs-window would be composed of lots of subwindows
> on server side, that lots of sub-windows could be completely hidden. Never
> use them to write something in them. They just report exactly the part of
> the buffer that should be redisplayed. They only catch and report events.
> They are invisible.
>
> Redisplay_internal does lots and lots of computations to detect which
> part of the buffer should be redisplayed at a given moment.

By checking which on-screen parts have changed how.

> All these computations can be avoided using a system of subwindows (1
> for every char).

That is nonsense.  One then has to check which subwindows need to be
moved/replaced as a result of buffer changes.  The X server has no clue
about buffers and the display matrix and its relative arrangement.

> That was my idea: to replace redisplay_internal with something simple
> and efficient. Something simple and efficient is to keep in the emacs
> a vector of little windows, and to register in the server all these
> windows.

And the server works by magic?  Unlikely.  Even if the scheme were not
unfeasible.

> The structure of windows is changed only when configure event is
> caught or when one changes the font, or split windows, etc. So very
> rarely.

Every insertion of a single character of a proportional font will change
the "windows" layout.

> 95% of the operations of redisplay can be covered with the system of
> little windows registered in the server. No more need of
> redisplay_internal for X.

I think you have the wrong idea about what redisplay_internal does, and
what an X server can do on its own.

-- 
David Kastrup

Re: redisplay system of emacs

[alin.s]

> The structure of windows is changed only when configure event is
> caught or when one changes the font, or split windows, etc. So very
> rarely.

Every insertion of a single character of a proportional font will change
the "windows" layout.


This is true, because not all characters have the same width. Anyways, the
idea of many subwindows was not good.


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27585994.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Chong Yidong]

I'm not one to discourage people from hacking on whatever they feel
like, but IMO discussing a revamp of the Emacs redisplay system is a
waste of time.  Gerd Moellmann sweated blood getting the present system
up and running for Emacs 21, and it basically works well enough.  There
are lots and lots of areas in Emacs that (IMO) are in much more urgent
need of improvement.

If you have a hankering for making core code changes, I'd suggest
looking at the concurrency project or the GTK widget embedding project,
and not a radical overhaul of redisplay itself.  (Stefan and I will make
our goals for Emacs 24 more concrete when we open the trunk for Emacs 24
development.)

Re: redisplay system of emacs

[Giuseppe Scrivano]

Chong Yidong <cyd@stupidchicken.com> writes:

> If you have a hankering for making core code changes, I'd suggest
> looking at the concurrency project or the GTK widget embedding project,
> and not a radical overhaul of redisplay itself.  (Stefan and I will make
> our goals for Emacs 24 more concrete when we open the trunk for Emacs 24
> development.)

The emacs concurrency project, despite recently a not very active
development, is in a quite advanced development status.

Since last time we discussed about emacs-mt, many problems were fixed,
buffer-locking was removed and we have finally implemented per thread
let-bound buffer local variables.

I am a bit stuck now; would you mind to try emacs-mt and let us know
what we are missing before think (hopefully) about a merge?  It is not
well tested, and surely there are bugs waiting to be discovered.

The git repository is accessible here (g-exp branch):

  git://gitorious.org/emacs-mt/emacs-mt.git

Thanks,
Giuseppe

Re: redisplay system of emacs

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]

On Feb 22, 2010, at 7:45 PM, Giuseppe Scrivano wrote:
> 
> I am a bit stuck now; would you mind to try emacs-mt and let us know
> what we are missing before think (hopefully) about a merge?  It is not
> well tested, and surely there are bugs waiting to be discovered.
> 
> The git repository is accessible here (g-exp branch):
> 
>  git://gitorious.org/emacs-mt/emacs-mt.git


It doesn't compile on my machine (Mac OS X 10.6):

gcc -c  -Demacs -DHAVE_CONFIG_H  -I. -I/Users/dr/em23/src -Dtemacs     -g -O2 -Wdeclaration-after-statement -Wno-pointer-sign   -MMD -MF deps/dispnew.d dispnew.c
In file included from lisp.h:3690,
                 from dispnew.c:31:
thread.h:128: error: thread-local storage not supported for this target
thread.h:142: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'global_lock'
make[1]: *** [dispnew.o] Error 1
make: *** [src] Error 2

See also: http://lists.apple.com/archives/xcode-users/2006/Jun/msg00551.html
http://lists.apple.com/archives/xcode-users/2007/jun/msg00120.html

That link suggests that the feature you're using is related to ELF and thus not portable.

Thanks for your work on this - looking forward to seeing the results!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: redisplay system of emacs

[Tom Tromey]

>>>>> "David" == David Reitter <david.reitter@gmail.com> writes:

David> That link suggests that the feature you're using is related to
David> ELF and thus not portable.

Yeah, we currently use __thread and the POSIX thread API.  These can be
made portable by someone who has the know-how.

Tom

Re: redisplay system of emacs

[Richard Stallman]

I am impressed and glad.  Concurrency would enable us to solve the
problem of multi-terminal Emacs, by enabling each terminal to enter
the minibuffer in parallel.

Concurrency (was: redisplay system of emacs)

[Stefan Monnier]

> The emacs concurrency project, despite recently a not very active
> development, is in a quite advanced development status.
[...]
> The git repository is accessible here (g-exp branch):
>   git://gitorious.org/emacs-mt/emacs-mt.git

Could you install your code (assuming it does not have contributions
from people who haven't signed the relevant paperwork, of course) into
a new `concurrent' branch in the Bzr repository?

That would be very helpful,


        Stefan


PS: Obviously, you'll first have to convert it to Bzr and then you'll
probably need to rebase it on top of Emacs's Bzr trunk.

Re: Concurrency (was: redisplay system of emacs)

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> PS: Obviously, you'll first have to convert it to Bzr and then you'll
> probably need to rebase it on top of Emacs's Bzr trunk.

Most likely it's much easier the other way round.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Concurrency (was: redisplay system of emacs)

[Giuseppe Scrivano]

I have rebased the emacs-mt patches on the last bazaar revision; I had
to rebase the git repository as well to get a linear source history that
I could use easier with bazaar.

I uploaded the `concurrent' branch on fencepost in
~gscrivano/emacs/concurrent, can somebody please install it?  I will
also need write permissions to continue working on it.

As we moved to bazaar, the emacs git mirror is not used anymore by the
concurrent emacs project.

Cheers,
Giuseppe



Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The emacs concurrency project, despite recently a not very active
>> development, is in a quite advanced development status.
> [...]
>> The git repository is accessible here (g-exp branch):
>>   git://gitorious.org/emacs-mt/emacs-mt.git
>
> Could you install your code (assuming it does not have contributions
> from people who haven't signed the relevant paperwork, of course) into
> a new `concurrent' branch in the Bzr repository?
>
> That would be very helpful,
>
>
>         Stefan
>
>
> PS: Obviously, you'll first have to convert it to Bzr and then you'll
> probably need to rebase it on top of Emacs's Bzr trunk.

Re: Concurrency (was: redisplay system of emacs)

[Giuseppe Scrivano]

Hello,

Finally I have created a new branch for the concurrent emacs project, it
is accessible here:

  http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

Cheers,
Giuseppe

Re: Concurrency

[Ted Zlatanov]

On Thu, 25 Mar 2010 17:49:25 +0100 Giuseppe Scrivano <gscrivano@gnu.org> wrote: 

GS> Finally I have created a new branch for the concurrent emacs project, it
GS> is accessible here:

GS>   http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

I'd like to look at it.  Is there a quick-start guide that explains the
changes and how to get started?

Thanks
Ted

Re: Concurrency

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

GS> http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

Ted> I'd like to look at it.  Is there a quick-start guide that explains the
Ted> changes and how to get started?

Nope.  However, there aren't really many new Emacs primitives.

On my branch I had run-in-thread (and a convenience wrapper macro,
with-new-thread), and yield.  Giuseppe's branch was slightly different,
and I think he added some mutex support as well; I still haven't looked
at these.

I can explain the bulk of the C changes if you want to understand those.

Tom

Re: Concurrency

[Ted Zlatanov]

On Fri, 26 Mar 2010 13:37:51 -0600 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
GS> http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

Ted> I'd like to look at it.  Is there a quick-start guide that explains the
Ted> changes and how to get started?

Tom> Nope.  However, there aren't really many new Emacs primitives.

Tom> On my branch I had run-in-thread (and a convenience wrapper macro,
Tom> with-new-thread), and yield.  Giuseppe's branch was slightly different,
Tom> and I think he added some mutex support as well; I still haven't looked
Tom> at these.

Tom> I can explain the bulk of the C changes if you want to understand those.

I'd love to know what's in Giuseppe's branch from the perspective of a
user.  Specifically, I want to know how a rewrite of the Gnus summary
buffer entry and threading might work in the new system.  But that's a
big undertaking so some simple demos would be better:

- a demo that increments a shared variable up to 10 with two threads
  that fight over the lock

- a map/reduce demo of totaling a list of numbers with N threads

- retrieve some network data asynchronously using threads, not async
  processes

Those demos would necessarily show the new primitives in a practical
way.  You or Giuseppe can explain the internals as well, that would be
helpful, but demos are really useful.

Thanks
Ted

Re: Concurrency

[Stefan Monnier]

> I'd love to know what's in Giuseppe's branch from the perspective of a
> user.  Specifically, I want to know how a rewrite of the Gnus summary

Actually, we need both: we need to document the changes, as seen by an
Elisp programmer (fits in etc/NEWS), and we need to document the way it
works internally, the invariants upon which it relies and things
like that.


        Stefan

Re: Concurrency

[Tom Tromey]

Stefan> Actually, we need both: we need to document the changes, as seen
Stefan> by an Elisp programmer (fits in etc/NEWS), and we need to
Stefan> document the way it works internally, the invariants upon which
Stefan> it relies and things like that.

I can supply some of the raw data here, if not the commits.

We added a few new lisp objects:

* thread_state.  This represents a single thread.  Before dumping, only
  one thread may exist.

  Some C variables which were formerly global were moved into this
  struct, and replaced with a redirecting #define.  E.g.:

  /* Pointer to beginning of specpdl.  */
  struct specbinding *m_specpdl;
#define specpdl (current_thread->m_specpdl)

  Exactly which things needed to be in this structure was determined in
  an ad hoc way; mostly by looking at `nm' to see what globals were
  interesting.

* Lisp_Mutex.  A mutex, should be obvious.

* Lisp_ThreadLocal.  This is used to represent a thread-local binding.
  It has a slot for the global value, and an alist mapping threads to
  a thread-local slot.  If a thread is not in the alist, the global
  binding is used.

  A Lisp_ThreadLocal can appear in a number of places, e.g. in a
  variable binding, in a Lisp_Object global variable, etc.

  We rewrote a lot of the C code to automatically indirect through a
  Lisp_ThreadLocal, when found.  There is a new header, globals.h, which
  has many lines of the form:

#define Vafter_change_functions *find_variable_location (&impl_Vafter_change_functions)

  A couple things about this line:

  * The expansion is chosen so that both rvalue and lvalue uses work.
  * The underlying global is renamed to catch any errors.

There are other changes (in the GC, in the binding code) but I think
those are relatively self-explanatory in the code... at least, hopefully
as much as the code was before.

Tom

Re: Concurrency

[Tom Tromey]

Tom> Nope.  However, there aren't really many new Emacs primitives.

Here's a quick rundown of the new primitives.

(run-in-thread FUNC)
  Take a no-argument function as an argument.
  Makes a new thread and calls the function.

(with-new-thread FORM)
  A convenience macro that takes a form, wraps it in a lambda,
  and calls run-in-thread.

(yield)
  Yield control.
  Emacs has semi-cooperative threading.  Thread switches happen during
  I/O or by explicit yield.

(make-mutex)
  Make a new lock and return it

(mutex-lock MUTEX)
  Acquire a mutex.  If already held by this thread, returns.
  If the mutex is held by some other thread, blocks until it is
  available.

(mutex-unlock MUTEX)
  Release the lock.

There is also a new variable, minibuffer-mutex.  I think it is pretty
self-explanatory, but Giuseppe should probably explain it.

I don't have time to write demos or even proper documentation, I'm
afraid.

Giuseppe also has a patch of some kind to Gnus, but I didn't see it in
the tree, and I don't know exactly what it does.

Tom

Re: Concurrency

[Stefan Monnier]

> Here's a quick rundown of the new primitives.

> (run-in-thread FUNC)
>   Take a no-argument function as an argument.
>   Makes a new thread and calls the function.

> (with-new-thread FORM)
>   A convenience macro that takes a form, wraps it in a lambda,
>   and calls run-in-thread.

> (yield)
>   Yield control.
>   Emacs has semi-cooperative threading.  Thread switches happen during
>   I/O or by explicit yield.

> (make-mutex)
>   Make a new lock and return it

> (mutex-lock MUTEX)
>   Acquire a mutex.  If already held by this thread, returns.

I.e. it's a "recursive/reentrant mutex".

>   If the mutex is held by some other thread, blocks until it is
>   available.

> (mutex-unlock MUTEX)
>   Release the lock.

Sounds good.  Could someone complete the above with a description of how
let-bindings work (and/or how they interact with buffer-local bindings)?
Also, a list of bugs/problem/issues would be handy,


        Stefan


PS: And place this info in a file in the `concurrent' branch.

Re: Concurrency

[Davis Herring]

>> (mutex-lock MUTEX)
>>   Acquire a mutex.  If already held by this thread, returns.
>
> I.e. it's a "recursive/reentrant mutex".

Hmm -- is it fully recursive, where you must unlock it as many times as
you locked it?  (I don't much care for the semi-recursive kind where one
unlock is sufficient regardless of the number of lock operations...)

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Concurrency

[Giuseppe Scrivano]

"Davis Herring" <herring@lanl.gov> writes:

>>> (mutex-lock MUTEX)
>>>   Acquire a mutex.  If already held by this thread, returns.
>>
>> I.e. it's a "recursive/reentrant mutex".
>
> Hmm -- is it fully recursive, where you must unlock it as many times as
> you locked it?  (I don't much care for the semi-recursive kind where one
> unlock is sufficient regardless of the number of lock operations...)

No, it is sufficient only one unlock to release it.

Cheers,
Giuseppe

Re: Concurrency

[Stefan Monnier]

>>>> (mutex-lock MUTEX)
>>>> Acquire a mutex.  If already held by this thread, returns.
>>> I.e. it's a "recursive/reentrant mutex".
>> Hmm -- is it fully recursive, where you must unlock it as many times as
>> you locked it?  (I don't much care for the semi-recursive kind where one
>> unlock is sufficient regardless of the number of lock operations...)
> No, it is sufficient only one unlock to release it.

Huh?  That sounds like a bug,


        Stefan

Re: Concurrency

[Giuseppe Scrivano]

I have changed how `make-mutex' works.  Now it accepts an optional
argument; if the argument is non nil then a recursive mutex is created,
by default it is nil.  By default a normal mutex (non recursive) is
created.

I have also changed mutex-unlock on a recursive mutex to match the
number of times it was mutex-lock'ed.  To release a recursive mutex it
must be unlocked as many times as it was locked.

I have added a mutexp function that I forgot to add before.

Giuseppe



Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>> (mutex-lock MUTEX)
>>>>> Acquire a mutex.  If already held by this thread, returns.
>>>> I.e. it's a "recursive/reentrant mutex".
>>> Hmm -- is it fully recursive, where you must unlock it as many times as
>>> you locked it?  (I don't much care for the semi-recursive kind where one
>>> unlock is sufficient regardless of the number of lock operations...)
>> No, it is sufficient only one unlock to release it.
>
> Huh?  That sounds like a bug,
>
>
>         Stefan

Re: Concurrency

[Tom Tromey]

Giuseppe> I have changed how `make-mutex' works.  Now it accepts an
Giuseppe> optional argument; if the argument is non nil then a recursive
Giuseppe> mutex is created, by default it is nil.  By default a normal
Giuseppe> mutex (non recursive) is created.

I think it would be better to have all mutexes be recursive, because it
is safer.

Also, I think mutex-unlock should throw some kind of error if the mutex
is owned by a different thread.  What do you think of that?

Tom

Re: Concurrency

[Stefan Monnier]

> Also, I think mutex-unlock should throw some kind of error if the mutex
> is owned by a different thread.  What do you think of that?

That doesn't sound useful.  There are perfectly valid ways to use mutexes
where the locker and the unlocker are not the same thread.


        Stefan

Re: Concurrency

[Ken Raeburn]

On Mar 29, 2010, at 12:16, Stefan Monnier wrote:
>> Also, I think mutex-unlock should throw some kind of error if the mutex
>> is owned by a different thread.  What do you think of that?
> 
> That doesn't sound useful.  There are perfectly valid ways to use mutexes
> where the locker and the unlocker are not the same thread.

True... but there are models where it would be a bug, plain and simple.
Maybe it should be an option to mutex-unlock?

Ken

Re: Concurrency

[Stefan Monnier]

>>> Also, I think mutex-unlock should throw some kind of error if the mutex
>>> is owned by a different thread.  What do you think of that?
>> That doesn't sound useful.  There are perfectly valid ways to use mutexes
>> where the locker and the unlocker are not the same thread.
> True... but there are models where it would be a bug, plain and simple.

Bugs happen.  It's only a problem if they happen often enough to warrant
the pain of imposing additional constraints.  I'd expect most uses of
"same-thread unlock" to be covered by the scoped with-mutex
construct anyway.

> Maybe it should be an option to mutex-unlock?

Let's not add complexity until it's clear that it's needed.


        Stefan

Re: Concurrency

[Ken Raeburn]

On Mar 29, 2010, at 11:37, Tom Tromey wrote:
> I think it would be better to have all mutexes be recursive, because it
> is safer.

I can see some desire for recursive mutexes, but I lean towards Butenhof's view myself.  If all mutexes are recursive, I'd like a way to find out if I've already got a mutex locked, so I can manually detect when I've screwed up my locking model, even if someone else wants to use recursive mutexes in their code.

BTW, one extension I'd suggest is an optional timeout for mutex-lock: nil would mean block until the lock is acquired, a positive number would mean wait up to that long (e.g., pthread_mutex_timedlock), and a non-positive number would mean try to acquire the lock but don't wait.  I'm not sure offhand if failure to acquire the lock should be indicated by throwing an error or a return value; I'm leaning a little towards throwing an error.

If this really catches on, read/write locks will probably be wanted too, but they can be built on mutexes and condition variables initially, and moved into C code later if they're really helpful and performance-critical.

For debugging, I think I'd want a way to check whether a particular thread has locked a mutex.  Not a predicate someone might be tempted to use for non-debugging purposes, more like an assertion that throws an error when false.

> Also, I think mutex-unlock should throw some kind of error if the mutex
> is owned by a different thread.  What do you think of that?

Probably the best thing -- and likewise if the mutex isn't locked at all.

Ken

Re: Concurrency

[Tom Tromey]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

Tom> I think it would be better to have all mutexes be recursive, because it
Tom> is safer.

Ken> I can see some desire for recursive mutexes, but I lean towards
Ken> Butenhof's view myself.

I realized later I should probably expand on this.

The issue I have with non-recursive mutexes has to do with the nature of
Emacs itself.  First, there is a large body of existing elisp that we
don't want to break.  And, second, Emacs changes over time and it is
very common to write backward compatibility hacks and the like to make
elisp work with multiple versions.  Finally, Emacs is not and will never
be a high-performance environment.

I think these issues imply that we should pick the safe option always.

As a concrete example, consider the one existing user-visible mutex in
Emacs right now: the minibuffer mutex.  Lots of code, like yes-or-no-p,
acquires and releases this mutex.

But suppose you want to write a function that uses the minibuffer for a
multi-step operation, and you want to use yes-or-no-p (or something
else) as a subroutine.

If you have recursive mutexes, no problem, just acquire the lock.

If you don't have recursive mutexes, then all the primitives must be
duplicated into -lock and -nolock variants; or, the primitives should do
no locking and must be called with the lock held.

I'm not a fan of the first option, since it means an explosion of
functions.  And the second option is also bad, because it breaks
existing elisp.

(FWIW I am not sure a minibuffer mutex is the right thing.  It seems
like this should be at least per-terminal.)

Ken> If all mutexes are recursive, I'd like a
Ken> way to find out if I've already got a mutex locked, so I can
Ken> manually detect when I've screwed up my locking model, even if
Ken> someone else wants to use recursive mutexes in their code.

Ken> BTW, one extension I'd suggest is an optional timeout for
Ken> mutex-lock

These make sense to me.

Tom

Re: Concurrency

[Stefan Monnier]

> (FWIW I am not sure a minibuffer mutex is the right thing.  It seems
> like this should be at least per-terminal.)

I think the minibuffer-mutex variable makes a lot of sense, but yes, it
will have to be terminal-local, but that can be changed later once the
rest of the code is ready for it.


        Stefan

Re: Concurrency

[Giuseppe Scrivano]

Tom Tromey <tromey@redhat.com> writes:

> Also, I think mutex-unlock should throw some kind of error if the mutex
> is owned by a different thread.  What do you think of that?

I think we should consider what will happen when the current cooperative
threads model will be replaced and threads will run freely.  In that
case probably we will want to use pthread mutexes, not our
implementation.  Any invariant we introduce now, must be followed later
as well or we will end to break code.  Pthread specifications say for
default mutex'es that the behaviour is undefined when you attempt to
unlock a mutex owned by another thread, instead an error is returned
when a recursive mutex is used.  So I think we can rely on this error
and introduce it only if we are not going to use default mutex'es,
unless later we want to implement them on the top of recursive ones.

What do you think?

Cheers,
Giuseppe

Re: Concurrency

[Stefan Monnier]

>> Also, I think mutex-unlock should throw some kind of error if the mutex
>> is owned by a different thread.  What do you think of that?
> I think we should consider what will happen when the current cooperative
> threads model will be replaced and threads will run freely.  In that
> case probably we will want to use pthread mutexes, not our
> implementation.

That's a good point.


        Stefan "who really wishes we used higher-level abstractions
                than locks, such as atomic-regions"

Re: Concurrency

[Tom Tromey]

>>>>> "Davis" == Davis Herring <herring@lanl.gov> writes:

>>> (mutex-lock MUTEX)
>>> Acquire a mutex.  If already held by this thread, returns.
>> 
>> I.e. it's a "recursive/reentrant mutex".

Davis> Hmm -- is it fully recursive, where you must unlock it as many times as
Davis> you locked it?  (I don't much care for the semi-recursive kind where one
Davis> unlock is sufficient regardless of the number of lock operations...)

Yeah, it doesn't keep a count.  And in addition to this problem,
mutex-unlock has a bug where it can unlock any mutex, not just ones own
by the current thread.

Tom

Re: Concurrency

[Daniel Colascione]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/28/10 4:25 PM, Davis Herring wrote:
>>> (mutex-lock MUTEX)
>>>   Acquire a mutex.  If already held by this thread, returns.
>>
>> I.e. it's a "recursive/reentrant mutex".
> 
> Hmm -- is it fully recursive, where you must unlock it as many times as
> you locked it?  (I don't much care for the semi-recursive kind where one
> unlock is sufficient regardless of the number of lock operations...)
> 
> Davis
> 

It really shouldn't be recursive at all:

http://www.zaval.org/resources/library/butenhof1.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAkuvyJYACgkQ17c2LVA10VsB3QCdHyXXPK9JMkbnDYqslY/TnBBY
6rcAoMr8ZlIpRGutZKa9mDGVzABJJZ3G
=0TRS
-----END PGP SIGNATURE-----

Re: Concurrency

[Stefan Monnier]

> It really shouldn't be recursive at all:
> http://www.zaval.org/resources/library/butenhof1.html

I'd tend to agree, although I doubt it has much impact on the actual
problem of getting concurrency working reliably and usefully.  I.e. it
should be easy to change it later.


        Stefan

Re: Concurrency

[Tom Tromey]

Daniel> It really shouldn't be recursive at all:
Daniel> http://www.zaval.org/resources/library/butenhof1.html

My experience with Java is that recursive mutexes are pretty convenient,
and in practice don't cause any actual problems.  I think most of this
paper is cutely worded but basically wrong.

I'm sure in the end it won't matter though.  If you really want
non-recursive mutexes, somebody else can reinvent them in elisp.

Tom

Re: Concurrency

[Tom Tromey]

Stefan> Could someone complete the above with a description of how
Stefan> let-bindings work (and/or how they interact with buffer-local
Stefan> bindings)?

A let binding is always thread-local.  This is true whether the variable
is buffer-local or not.

The view within a single thread is basically the same as how elisp works
today.

The interaction of let-bindings and buffer-local bindings is
complicated, but I think really no different from before.  At least,
that was our goal, of course there could be bugs.

Stefan> Also, a list of bugs/problem/issues would be handy,

* Documentation; NEWS, as you said, but also the lisp reference manual.

* The current code makes no attempt to portable, it relies on POSIX
  threads and the GNU __thread extension.  This is not too hard to fix.
  I think it is reasonably important to keep __thread on systems that
  support it, for performance.

* Suppose you have existing elisp that creates a process with a filter,
  and the filter changes a let-bound variable, and the "outer" elisp
  blocks in sit-for waiting for the filter to do its thing.  Nothing in
  the current code guarantees that the process filter will be run in the
  "correct" thread.

* There may be oddities with redisplay running in the "wrong" thread.

* Keyboard-local variables and let-bindings may not work together as
  expected.

I'm sure there are more, this is all I remember offhand.

Stefan> PS: And place this info in a file in the `concurrent' branch.

I hope someone else can do that.

Tom

Re: Concurrency

[Ken Raeburn]

On Mar 28, 2010, at 17:17, Tom Tromey wrote:
> * The current code makes no attempt to portable, it relies on POSIX
>  threads and the GNU __thread extension.  This is not too hard to fix.
>  I think it is reasonably important to keep __thread on systems that
>  support it, for performance.

From what I understand, porting to Windows should be easy enough.  "A simple matter of coding."

Dealing with __thread shouldn't be hard if it's not used pervasively, so pthread_{get,set}specific calls can be used in its place.

> * Suppose you have existing elisp that creates a process with a filter,
>  and the filter changes a let-bound variable, and the "outer" elisp
>  blocks in sit-for waiting for the filter to do its thing.  Nothing in
>  the current code guarantees that the process filter will be run in the
>  "correct" thread.

That may be a reason to force the filter to run in the same thread that created it.  Perhaps process filters could be run at thread-switch opportunities like 'yield', too?  (And vice versa -- perhaps calls that permit the running of process filters should also permit thread switching?)

Sounds like this is coming along nicely!

Ken

Re: Concurrency

[Tom Tromey]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

Tom> * The current code makes no attempt to portable, it relies on POSIX
Tom> threads and the GNU __thread extension.  This is not too hard to
Tom> fix.  I think it is reasonably important to keep __thread on
Tom> systems that support it, for performance.

Ken> Dealing with __thread shouldn't be hard if it's not used
Ken> pervasively, so pthread_{get,set}specific calls can be used in its
Ken> place.

There is exactly one __thread variable.

Tom> * Suppose you have existing elisp that creates a process with a filter,
Tom> and the filter changes a let-bound variable, and the "outer" elisp
Tom> blocks in sit-for waiting for the filter to do its thing.  Nothing in
Tom> the current code guarantees that the process filter will be run in the
Tom> "correct" thread.

Ken> That may be a reason to force the filter to run in the same thread
Ken> that created it.

Yeah.  I was thinking perhaps process filters could have a thread
attribute, which would default to their creating thread.  Setting this
to nil could mean that the process filter could run in any thread.

Ken> Perhaps process filters could be run at thread-switch opportunities
Ken> like 'yield', too?  (And vice versa -- perhaps calls that permit
Ken> the running of process filters should also permit thread
Ken> switching?)

I think (but am not 100% sure) that this already happens -- IIUC, things
like sleep-for and sit-for end up in wait_reading_process_output, which
has a possible yield in it.

Tom

Re: Concurrency

[Stefan Monnier]

Tom> * Suppose you have existing elisp that creates a process with a filter,
Tom> and the filter changes a let-bound variable, and the "outer" elisp
Tom> blocks in sit-for waiting for the filter to do its thing.  Nothing in
Tom> the current code guarantees that the process filter will be run in the
Tom> "correct" thread.
Ken> That may be a reason to force the filter to run in the same thread
Ken> that created it.
> Yeah.  I was thinking perhaps process filters could have a thread
> attribute, which would default to their creating thread.  Setting this
> to nil could mean that the process filter could run in any thread.

Or maybe accept-process-output should cause the filters to be run in the
waiting thread (especially when the `proc' argument is provided).


        Stefan

gsoc for concurrent Emacs? (was: Concurrency)

[Ted Zlatanov]

On Sun, 28 Mar 2010 15:17:49 -0600 Tom Tromey <tromey@redhat.com> wrote: 

Tom> * Documentation; NEWS, as you said, but also the lisp reference manual.
...
Stefan> PS: And place this info in a file in the `concurrent' branch.

Tom> I hope someone else can do that.

Just a suggestion:

I think the concurrent Emacs project is nicely standalone and would make
a good GSoC project (see
http://www.gnu.org/software/soc-projects/ideas-2010.html, where Guile
integration is already mentioned).  The concurrent Emacs branch
obviously needs some work, especially with documentation, but is
probably at a level suitable for an interested CS/CE student.

I'm not the one to propose this but if Giuseppe or Tom want, they can
e-mail summer-of-code@gnu.org as the URL above suggests.  GSoC requires
a mentor so it's not completely hands-off...

Ted

Re: gsoc for concurrent Emacs? (was: Concurrency)

[Giuseppe Scrivano]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think the concurrent Emacs project is nicely standalone and would make
> a good GSoC project (see
> http://www.gnu.org/software/soc-projects/ideas-2010.html, where Guile
> integration is already mentioned).  The concurrent Emacs branch
> obviously needs some work, especially with documentation, but is
> probably at a level suitable for an interested CS/CE student.

I am not sure there are enough tasks at this point that a GSoC project
is needed for the cooperative model we are using and I am afraid it is
too early yet to think about a new one.  AFAICS, the main activity to
get the concurrent model working is bug-fixing.



> I'm not the one to propose this but if Giuseppe or Tom want, they can
> e-mail summer-of-code@gnu.org as the URL above suggests.  GSoC requires
> a mentor so it's not completely hands-off...

If the Emacs maintainers think it is worthwhile, I am available for
mentoring.


Cheers,
Giuseppe

Re: Concurrency

[Giuseppe Scrivano]

Hello, sorry my late reply:

Tom Tromey <tromey@redhat.com> writes:

> There is also a new variable, minibuffer-mutex.  I think it is pretty
> self-explanatory, but Giuseppe should probably explain it.

It is a mutex that protects the mini-buffer from concurrent accesses.
Only one thread can use the minibuffer at a time.



> Giuseppe also has a patch of some kind to Gnus, but I didn't see it in
> the tree, and I don't know exactly what it does.

No, I don't have a patch, I have just experimented with "(run-in-thread
gnus)" to verify it loads correctly in a separate thread.


Also we have to mention the two new macros:

(defmacro with-new-thread (&rest body)) -- BODY is executed in a new
thread.

(defmacro with-critical-section (mutex &rest body)) BODY is executed
after MUTEX is locked and automatically released on exit.

Giuseppe

Re: Concurrency

[Daniel Colascione]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/28/10 3:40 PM, Tom Tromey wrote:
> Tom> Nope.  However, there aren't really many new Emacs primitives.
> 
> Here's a quick rundown of the new primitives.

Thanks for summing these up.

> (run-in-thread FUNC)
> (with-new-thread FORM)
> (yield)
> (make-mutex)
> (mutex-lock MUTEX)
> (mutex-unlock MUTEX)

In writing a different cooperatively-multitasked system, I've found that
I needed a kind of wait queue ("condition variable" if you will). While
you can build one out of the above primitives, it'd be better to provide
it as a part of the standard library.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAkuvyTsACgkQ17c2LVA10VurjQCgj5oE4YOg7oEgf68pDxNoHD1k
siMAni1B7DZHzmcKlki81x1D0m9cSIlk
=d5/q
-----END PGP SIGNATURE-----

Re: Concurrency

[Tom Tromey]

Daniel> In writing a different cooperatively-multitasked system, I've found that
Daniel> I needed a kind of wait queue ("condition variable" if you will). While
Daniel> you can build one out of the above primitives, it'd be better to provide
Daniel> it as a part of the standard library.

We purposely kept the API as minimal as possible.
E.g., there isn't even a user-visible thread object.

It is no trouble to add things, it just must be done thoughtfully, and
in particular with an eye toward not breaking the possibility of
preemptive threading.

On this particular point, I agree, condition variables will be required
sooner or later.

Tom

Re: redisplay system of emacs

[alin.s]

David Kastrup wrote:
> 
> 
>> 95% of the operations of redisplay can be covered with the system of
>> little windows registered in the server. No more need of
>> redisplay_internal for X.
> 
> I think you have the wrong idea about what redisplay_internal does, and
> what an X server can do on its own.
> 
> 

I\m not interested at all about X and graphical programming. I ' ve read the
Xlib manual just to be able to finish the task of tabs for
GTK/Athena/LessTiff


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27586106.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Davis Herring]

[Alin, I'm fully aware that you're not advocating this idea now.  I don't
mean to attack, but to explain something that wasn't said explicitly.]

> Redisplay_internal does lots and lots of computations to detect which part
> of the buffer should be redisplayed at a given moment. All these
> computations can be avoided using a system of subwindows (1 for every
> char).

Those computations are complicated because the buffer contents (or window
width, or scroll amount, or...) might have changed, not just because the
window might have been damaged in some complicated fashion.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: redisplay system of emacs

[grischka]

>> Redisplay_internal does lots and lots of computations to detect which part
>> of the buffer should be redisplayed at a given moment. All these
>> computations can be avoided using a system of subwindows (1 for every
>> char).
>
> Those computations are complicated because the buffer contents (or window
> width, or scroll amount, or...) might have changed, not just because the
> window might have been damaged in some complicated fashion.

It's not _that_complicated.  Basically it just draws the new screen to
memory, then compares it with the current screen and then updates the
differences.

And actually it works nicely and fast, just maybe slightly over-designed
in cases.  (For example changing "bazfoobar" to "baz,foobar" would output
",f" and "obar" and thus reuse the originally second "o" for the new first
one.  One might think that drawing one additional "o" could be faster than
two XDrawString, but so what).

Whether the code could be written to be easier to understand is a different
question but who has time for cleanups anyway.

--- grischka

Re: redisplay system of emacs

[Thien-Thi Nguyen]

() grischka <grishka@gmx.de>
() Tue, 16 Feb 2010 20:20:06 +0100

   but who has time for cleanups anyway.

Sometimes janitors and archeologists find the time.

thi

Re: redisplay system of emacs

[alin.s]

Thien-Thi Nguyen-6 wrote:
> 
> 
>    but who has time for cleanups anyway.
> 
> Sometimes janitors and archeologists find the time.
> 
> 

I hope some archaeologist who has good will to read my post.




-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27623998.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Eli Zaretskii]

> Date: Tue, 16 Feb 2010 20:20:06 +0100
> From: grischka <grishka@gmx.de>
> Cc: emacs-devel@gnu.org
> 
> >> Redisplay_internal does lots and lots of computations to detect which part
> >> of the buffer should be redisplayed at a given moment. All these
> >> computations can be avoided using a system of subwindows (1 for every
> >> char).
> >
> > Those computations are complicated because the buffer contents (or window
> > width, or scroll amount, or...) might have changed, not just because the
> > window might have been damaged in some complicated fashion.
> 
> It's not _that_complicated.  Basically it just draws the new screen to
> memory, then compares it with the current screen and then updates the
> differences.

That's only a small part of the story.  It draws only part of the
screen, not all of it (unless optimizations fail), and it goes to
great lengths to minimize that part of the screen.

The reason is that delivering to the screen is not the heaviest part
of the redisplay; it's that ``drawing to memory'' that is.

Re: redisplay system of emacs

[grischka]

Eli Zaretskii wrote:
>> It's not _that_complicated.  Basically it just draws the new screen to
>> memory, then compares it with the current screen and then updates the
>> differences.
> 
> That's only a small part of the story.  It draws only part of the
> screen, not all of it (unless optimizations fail), and it goes to
> great lengths to minimize that part of the screen.

Sure, thats what I wrote: It updates only the differences.

> The reason is that delivering to the screen is not the heaviest part
> of the redisplay; it's that ``drawing to memory'' that is.

That is probably true nowadays with fast graphical screens (say for PCs
after ~1995).  However the design of emacs' "redisplay" is based on the
contrary assumption that sending the updates to the terminal more expensive
than their calculation.  If you want to it is optimized for _very_ slow
terminals to the expense of code simplicity.  I assume it was working great
with 14400 baud connections, but then again emacs didn't have colors and
var-pitch fonts at that times, either.

--- grischka

Re: redisplay system of emacs

[Eli Zaretskii]

> Date: Tue, 16 Feb 2010 21:56:03 +0100
> From: grischka <grishka@gmx.de>
> CC: herring@lanl.gov, emacs-devel@gnu.org
> 
> Eli Zaretskii wrote:
> >> It's not _that_complicated.  Basically it just draws the new screen to
> >> memory, then compares it with the current screen and then updates the
> >> differences.
> > 
> > That's only a small part of the story.  It draws only part of the
> > screen, not all of it (unless optimizations fail), and it goes to
> > great lengths to minimize that part of the screen.
> 
> Sure, thats what I wrote: It updates only the differences.

The point is, it tries to limit the region where it looks for
differences to as small portion of the screen as possible, sometimes a
single line.  Therefore ``draws the new screen to memory'' is an
exception; the rule is ``draws a small portion of the new screen to
memory''.

> > The reason is that delivering to the screen is not the heaviest part
> > of the redisplay; it's that ``drawing to memory'' that is.
> 
> That is probably true nowadays with fast graphical screens (say for PCs
> after ~1995).  However the design of emacs' "redisplay" is based on the
> contrary assumption that sending the updates to the terminal more expensive
> than their calculation.

The current display code was designed in 1999.