redisplay system of emacs

redisplay system of emacs

[alin.s]

I was wondering if it is possible to change the system of redisplay of emacs
to a less obfuscated one.

The current system , based on redisplay internal, for me it is very
obfuscated.

It could be possible to adopt a more clear system and keep all the other
functionalism ?

Could it be possible to take off all the redisplay and create a standardized
system of redisplay, that everybody can understand quickly? Everybody can
write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
but as time as new add ons appear every day, that means that the system is
very standardized and easy to learn.

What are your opinions concerning this problem?



-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27349166.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Eli Zaretskii]

> Date: Wed, 27 Jan 2010 16:19:09 -0800 (PST)
> From: "alin.s" <alinsoar@voila.fr>
> Cc: 
> 
> 
> I was wondering if it is possible to change the system of redisplay of emacs
> to a less obfuscated one.
> 
> The current system , based on redisplay internal, for me it is very
> obfuscated.

It's true that the code is quite arcane, and in a couple of places
barely maintainable, even though it was almost completely overhauled
just 10 years ago.

The high level of the display engine is very simple and can be
explained in a few simple sentences, but the details...

> It could be possible to adopt a more clear system and keep all the other
> functionalism ?

I'm not sure.  Somebody should propose a design, and then we could
discuss it.  But designing and implementing a completely new display
engine that supports everything Emacs supports now is not an easy
task, so it will take a highly motivated and able individual to make
that happen.

> Could it be possible to take off all the redisplay and create a standardized
> system of redisplay, that everybody can understand quickly? Everybody can
> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
> but as time as new add ons appear every day, that means that the system is
> very standardized and easy to learn.

Last time I looked, Mozilla wasn't anywhere close to supporting the
features Emacs has in its display engine.

Re: redisplay system of emacs

[Ken Hori]

+1 vote.

On Wed, Jan 27, 2010 at 4:19 PM, alin.s <alinsoar@voila.fr> wrote:
>
> I was wondering if it is possible to change the system of redisplay of emacs
> to a less obfuscated one.
>
> The current system , based on redisplay internal, for me it is very
> obfuscated.
>
> It could be possible to adopt a more clear system and keep all the other
> functionalism ?
>
> Could it be possible to take off all the redisplay and create a standardized
> system of redisplay, that everybody can understand quickly? Everybody can
> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
> but as time as new add ons appear every day, that means that the system is
> very standardized and easy to learn.
>
> What are your opinions concerning this problem?
>
>
>
> --
> View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27349166.html
> Sent from the Emacs - Dev mailing list archive at Nabble.com.
>
>
>
>

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 5:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> Could it be possible to take off all the redisplay and create a standardized
>> system of redisplay, that everybody can understand quickly? Everybody can
>> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
>> but as time as new add ons appear every day, that means that the system is
>> very standardized and easy to learn.
>
> Last time I looked, Mozilla wasn't anywhere close to supporting the
> features Emacs has in its display engine.


What features?

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Thu, 28 Jan 2010 10:07:57 +0100
> Cc: "alin.s" <alinsoar@voila.fr>, Emacs-devel@gnu.org
> 
> On Thu, Jan 28, 2010 at 5:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >> Could it be possible to take off all the redisplay and create a standardized
> >> system of redisplay, that everybody can understand quickly? Everybody can
> >> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
> >> but as time as new add ons appear every day, that means that the system is
> >> very standardized and easy to learn.
> >
> > Last time I looked, Mozilla wasn't anywhere close to supporting the
> > features Emacs has in its display engine.
> 
> 
> What features?

The ones described in the nodes "Display Property" and "Special
Properties" in the ELisp manual, for example.

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 12:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Thu, 28 Jan 2010 10:07:57 +0100
>> Cc: "alin.s" <alinsoar@voila.fr>, Emacs-devel@gnu.org
>>
>> On Thu, Jan 28, 2010 at 5:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >
>> >> Could it be possible to take off all the redisplay and create a standardized
>> >> system of redisplay, that everybody can understand quickly? Everybody can
>> >> write an add-on for Mozilla. I do not know how redisplay of Mozilla works,
>> >> but as time as new add ons appear every day, that means that the system is
>> >> very standardized and easy to learn.
>> >
>> > Last time I looked, Mozilla wasn't anywhere close to supporting the
>> > features Emacs has in its display engine.
>>
>>
>> What features?
>
> The ones described in the nodes "Display Property" and "Special
> Properties" in the ELisp manual, for example.


Thanks Eli, maybe I understand what you mean but I am a bit surprised.

The Mozilla display engine displays images and text with different
properties very well. And it is very flexible in its way to do that
(since CSS requires that).

I guess it does not directly has something that reminds of "special
properties", but would that be hard to add?

Another point is of course searching for the properties. But Emacs has
its own difficulties there (overlays).

redisplay system of emacs

[Stephen J. Turnbull]

alin.s writes:

 > Could it be possible to take off all the redisplay and create a
 > standardized system of redisplay, that everybody can understand
 > quickly? Everybody can write an add-on for Mozilla.

But nobody (except a few experts) messes with redisplay.  Redisplay is
*hard*.  There are very few redisplays capable of doing what Emacs can
do.  Emacs, XEmacs.  *Maybe* Gecko (Mozilla) or Webkit (Safari).  On
the other hand, the way you write your post, I doubt it requires
changes to redisplay, but I could be wrong.  So I think you need to
explain what it is you want to do.

Note that Mozilla has a high-level language (XUL) for managing its
display (and other aspects of user interface).  Add-ons are written in
that language.  Well, so does Emacs: Emacs Lisp.

 > I do not know how redisplay of Mozilla works, but as time as new
 > add ons appear every day, that means that the system is very
 > standardized and easy to learn.

Emacs is also standardized and easy to learn.  There are just many
fewer people interested in writing Lisp libraries than there are
writing Mozilla add-ons.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Thu, 28 Jan 2010 12:47:26 +0100
> Cc: alinsoar@voila.fr, Emacs-devel@gnu.org
> 
> The Mozilla display engine displays images and text with different
> properties very well. And it is very flexible in its way to do that
> (since CSS requires that).

Even images support in Emacs (which is admittedly young and unevolved)
already supports advanced features like :pointer and :map (see the
node "Image Descriptors").  And I wasn't talking about simple text
properties that just change faces or the size of the characters, of
course.  What about properties that need to evaluate Lisp forms or
depend on Lisp-level variables?

> I guess it does not directly has something that reminds of "special
> properties", but would that be hard to add?

I have no idea.

> Another point is of course searching for the properties. But Emacs has
> its own difficulties there (overlays).

The difficulties is not with searching, the difficulties are with
maintaining many overlays.

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 1:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Thu, 28 Jan 2010 12:47:26 +0100
>> Cc: alinsoar@voila.fr, Emacs-devel@gnu.org
>>
>> The Mozilla display engine displays images and text with different
>> properties very well. And it is very flexible in its way to do that
>> (since CSS requires that).
>
> Even images support in Emacs (which is admittedly young and unevolved)
> already supports advanced features like :pointer and :map (see the
> node "Image Descriptors").


?? Of course Mozilla supports these kind of things too.


> And I wasn't talking about simple text
> properties that just change faces or the size of the characters, of
> course.  What about properties that need to evaluate Lisp forms or
> depend on Lisp-level variables?


Yes, I know you are talking about this. But AFAICS Mozilla has the
framework to support similar things. Otherwise it could not support
all the things needed to display certain web pages.


>> I guess it does not directly has something that reminds of "special
>> properties", but would that be hard to add?
>
> I have no idea.
>
>> Another point is of course searching for the properties. But Emacs has
>> its own difficulties there (overlays).
>
> The difficulties is not with searching, the difficulties are with
> maintaining many overlays.


I thought the problem was that it takes long time to search for the
overlays at point for example. Is that not the case?

Re: redisplay system of emacs

[alin.s]

When I first started to write the tabs, I started from redisplay_internal.
Fortunatelly, I found the solution such that I did not touch that code.

But trying to read it, it was impossible for me to make a general coherence
of the code. Only in parts I could understand it.

Yes, in genral the redisplay is superior with its overlays, but many
overlays like in linum-mode make it works bad.

I did not invest much effort in trying to read it, because I am afraid that
apart a few persons who were motivated to write some code in that part,
nobody knows it.

I wanted to say that it should be adopted a system that everybody can
understand, in order for everybody interested to be able to gain a coherence
in thought about that part of emacs.





-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27356155.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Miles Bader]

Lennart Borgman <lennart.borgman@gmail.com> writes:
> ?? Of course Mozilla supports these kind of things too.

Note that mozilla's display engine makes some dramatically different
assumptions about what is important for the display engine to do.

Most importantly, it does the layout and display calculation for the
_entire page_ at once.  Emacs, by contrast does it on the fly for the
small amount being displayed at the moment.

Mozilla's method allows some nice things -- for instance it makes much
more complicated layout tractable -- but it really really sucks for huge
files, and in general probably isn't such a good idea if the document
tends to change a lot in real time.  Emacs' method, by contrast works
really well for those cases.

I think this difference in approach makes sense given the different
goals of the two applications, and it's not at all clear that Mozilla's
display engine would work well for Emacs too.

Of course, you could try it and see.... :|

-miles

-- 
Cat, n. A soft, indestructible automaton provided by nature to be kicked when
things go wrong in the domestic circle.

Re: redisplay system of emacs

[Stephen J. Turnbull]

alin.s writes:

 > 
 > When I first started to write the tabs, I started from redisplay_internal.
 > Fortunatelly, I found the solution such that I did not touch that code.

It's not "fortunate", it's designed that way.  Redisplay is the lowest
level of the display code, and tightly bound to the event loop.  If
there's an X display involved, life is very difficult, because X
protocol is totally asynchronous.

But most things (like adding a tab widget) should be able to work with
higher level display features.

Re: redisplay system of emacs

[alin.s]

OK,

Let us keep as good Emacs' method. But let us clarify somehow in the least
details ! 




Miles Bader-2 wrote:
> 
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>> ?? Of course Mozilla supports these kind of things too.
> 
> Note that mozilla's display engine makes some dramatically different
> assumptions about what is important for the display engine to do.
> 
> Most importantly, it does the layout and display calculation for the
> _entire page_ at once.  Emacs, by contrast does it on the fly for the
> small amount being displayed at the moment.
> 
> Mozilla's method allows some nice things -- for instance it makes much
> more complicated layout tractable -- but it really really sucks for huge
> files, and in general probably isn't such a good idea if the document
> tends to change a lot in real time.  Emacs' method, by contrast works
> really well for those cases.
> 
> I think this difference in approach makes sense given the different
> goals of the two applications, and it's not at all clear that Mozilla's
> display engine would work well for Emacs too.
> 
> Of course, you could try it and see.... :|
> 
> -miles
> 
> -- 
> Cat, n. A soft, indestructible automaton provided by nature to be kicked
> when
> things go wrong in the domestic circle.
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27357399.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Lennart Borgman]

On Thu, Jan 28, 2010 at 3:10 PM, Miles Bader <miles@gnu.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>> ?? Of course Mozilla supports these kind of things too.
>
> Note that mozilla's display engine makes some dramatically different
> assumptions about what is important for the display engine to do.
>
> Most importantly, it does the layout and display calculation for the
> _entire page_ at once.  Emacs, by contrast does it on the fly for the
> small amount being displayed at the moment.
>
> Mozilla's method allows some nice things -- for instance it makes much
> more complicated layout tractable --


Yes, it is necessary for web files.


> but it really really sucks for huge
> files, and in general probably isn't such a good idea if the document
> tends to change a lot in real time.  Emacs' method, by contrast works
> really well for those cases.


True. I have no idea if Mozilla allows this kind a display handling
too. Maybe it does.


> I think this difference in approach makes sense given the different
> goals of the two applications, and it's not at all clear that Mozilla's
> display engine would work well for Emacs too.
>
> Of course, you could try it and see.... :|


Some rainy day, perhaps...


> -miles
>
> --
> Cat, n. A soft, indestructible automaton provided by nature to be kicked when
> things go wrong in the domestic circle.
>

Re: redisplay system of emacs

[Paul R]

Hello,

> Note that mozilla's display engine makes some dramatically different
> assumptions about what is important for the display engine to do.


I found a related discussion in a previous thread, mid-2009 :

  http://lists.gnu.org/archive/html/emacs-devel/2009-07/threads.html#00344


As many emacs youngsters, I feel that emacs really uses too much
home-made code. Surprisingly, Emacs does not benefit that much from the
free software ecosystem. This is certainly connected to the fact that it
has been ahead of its time in many aspects, so it had to build its own
bricks without waiting for others to provide them. But in the meantime
some things has changed, high quality generic free software fragments
has entered the game. Entrusting them rather than home-made code would
certainly be a big job, but would instantly enlarge emacs (indirect)
contributors list.


> Most importantly, it does the layout and display calculation for the
> _entire page_ at once. Emacs, by contrast does it on the fly for the
> small amount being displayed at the moment.

> Mozilla's method allows some nice things -- for instance it makes much
> more complicated layout tractable -- but it really really sucks for
> huge files, and in general probably isn't such a good idea if the
> document tends to change a lot in real time. Emacs' method, by
> contrast works really well for those cases.

The performance of those engines has gotten surprisingly good, and is
still improving at a very fast rate. I would be interested to have some
real-case benchmarks. In the meantime I pasted the whole wikipedia
article into http://ckeditor.com/demo then edited some text in the
middle of the page. It was not blazing fast to be honnest, but still
usable on my 10 years old hardware.

-- 
  Paul

Re: redisplay system of emacs

[David Kastrup]

Paul R <paul.r.ml@gmail.com> writes:

> The performance of those engines has gotten surprisingly good, and is
> still improving at a very fast rate.

It is nowhere near workable for large documents.  Not web pages,
documents.

> I would be interested to have some real-case benchmarks. In the
> meantime I pasted the whole wikipedia article into
> http://ckeditor.com/demo then edited some text in the middle of the
> page. It was not blazing fast to be honnest, but still usable on my 10
> years old hardware.

Uh, Emacs needs to work without noticeable delays and with syntax
highlighting for documents of _book_ size, easily containing thousands
of pages.  In fact, the size of the total document should not affect
editing speed.  And we are not just talking "editing" or fixed documents
here: run-off compilation/log output needs to be fast.

-- 
David Kastrup

Re: redisplay system of emacs

[Lennart Borgman]

On Fri, Jan 29, 2010 at 11:17 AM, David Kastrup <dak@gnu.org> wrote:
>
> Uh, Emacs needs to work without noticeable delays and with syntax
> highlighting for documents of _book_ size, easily containing thousands
> of pages.


Is this necessarily related to the display engine?

Re: redisplay system of emacs

[David Kastrup]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Fri, Jan 29, 2010 at 11:17 AM, David Kastrup <dak@gnu.org> wrote:
>>
>> Uh, Emacs needs to work without noticeable delays and with syntax
>> highlighting for documents of _book_ size, easily containing thousands
>> of pages.
>
>
> Is this necessarily related to the display engine?

Since the Mozilla model renders the whole document (admittedly starting
before it is completely available), I don't see how it should be
otherwise.

-- 
David Kastrup

Re: redisplay system of emacs

[Paul R]

Hi David,

> Uh, Emacs needs to work without noticeable delays and with syntax
> highlighting for documents of _book_ size, easily containing thousands
> of pages. In fact, the size of the total document should not affect
> editing speed.

Is it so common to write books in a single monolithic file ? The two
typesetting systems I use from emacs allow (and encourage) to split the
documents in many files. And emacs corresponding modes are designed for
easy navigation between them.

That said, I totally agree that this is a real concern : what happens if
someone wants to _open_ a huge document. As Lennart said, it probably
isn't the job of the display system to restrict the data to render to
a subset, but rather the job of an intermediate layer, between the
document and the rendering engine.


-- 
  Paul

Re: redisplay system of emacs

[David Kastrup]

Paul R <paul.r.ml@gmail.com> writes:

> Hi David,
>
>> Uh, Emacs needs to work without noticeable delays and with syntax
>> highlighting for documents of _book_ size, easily containing
>> thousands of pages. In fact, the size of the total document should
>> not affect editing speed.
>
> Is it so common to write books in a single monolithic file ?

Yes.  Because otherwise search&replace and other stuff become
increasingly painful to do.  In particular if not everybody working with
the same documents is using Emacs.

> The two typesetting systems I use from emacs allow (and encourage) to
> split the documents in many files. And emacs corresponding modes are
> designed for easy navigation between them.

It is fine if Emacs supports more than one way of organizing your work.
It is not fine if it falls apart if you don't do it in a particular way.

-- 
David Kastrup

Re: redisplay system of emacs

[Miles Bader]

Lennart Borgman <lennart.borgman@gmail.com> writes:
>> Uh, Emacs needs to work without noticeable delays and with syntax
>> highlighting for documents of _book_ size, easily containing thousands
>> of pages.
>
> Is this necessarily related to the display engine?

Different display engines models are affected differently by changes in
document size.

Mozilla's model, where the _entire document_ is converted into an
internal "layed-out-document" data structure, and then that is rendered,
and which is mainly targeted at documents of "reasonable" length but
with complex layout requirements (web pages), is pretty likely to have
an unacceptably high overhead (time and memory used by the
layed-out-document) for extremely large documents.

Emacs, which doesn't maintain a separate "layed-out-document" data
structure at all, and mainly just worries about rendering what's on the
screen from the raw document, has a very different set of tradeoffs.

Now, it's certainly _possible_ that mozilla actually contains special
code to deal with very-large-but-very-simple document structures, and
use a different method with different tradeoffs to render them, but...
it seems unlikely , and AFAIK, it doesn't.  [but of course, this
"special" code would more or less just be an implementation of what
Emacs does already!]

It's also _possible_ that mozilla's algorithms and data-structures are
so incredibly fast and efficient that this overhead is acceptable even
for extremely large documents on modern machines.  Judging from the
observed performance of e.g. firefox on large but simple web pages,
though, this also seems pretty unlikely.

[Emacs of course also does some processing that is proportional to
document length, e.g., coding/decoding, and at the most basic,
I/O... however such processing is likely to be much simpler/faster (and
memory efficient) than what Mozilla has to do layout the document for
display.]

-Miles

-- 
Friendless, adj. Having no favors to bestow. Destitute of fortune. Addicted to
utterance of truth and common sense.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Paul R <paul.r.ml@gmail.com>
> Cc: Lennart Borgman <lennart.borgman@gmail.com>,  alinsoar@voila.fr,  Eli Zaretskii <eliz@gnu.org>,  Emacs-devel@gnu.org
> Date: Fri, 29 Jan 2010 11:04:42 +0100
> 
> Hello,
> 
> > Note that mozilla's display engine makes some dramatically different
> > assumptions about what is important for the display engine to do.
> 
> 
> I found a related discussion in a previous thread, mid-2009 :
> 
>   http://lists.gnu.org/archive/html/emacs-devel/2009-07/threads.html#00344

Yeah, this pops up from time to time.

> As many emacs youngsters, I feel that emacs really uses too much
> home-made code. Surprisingly, Emacs does not benefit that much from the
> free software ecosystem.

That's a pretty general assessment.  Any data points other than the
display engine?

> This is certainly connected to the fact that it
> has been ahead of its time in many aspects, so it had to build its own
> bricks without waiting for others to provide them. But in the meantime
> some things has changed, high quality generic free software fragments
> has entered the game. Entrusting them rather than home-made code would
> certainly be a big job, but would instantly enlarge emacs (indirect)
> contributors list.

"Big job" is an understatement of the century.  Most of these "generic
free software fragments" are not written with Emacs peculiar
requirements in mind, and don't have a Lisp API.

Where these issues don't matter, Emacs does use existing software,
like with the font support and display-specific back ends that
actually render the characters on glass.

> The performance of those engines has gotten surprisingly good, and is
> still improving at a very fast rate. I would be interested to have some
> real-case benchmarks.

One of the most important benchmarks would be inserting or deleting a
small amount of text in a very large document.  As an interactive
editor, Emacs has to deal with this most of the time.  Not sure if
those other display engines are optimized to this modus operandi.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Date: Fri, 29 Jan 2010 20:03:47 +0900
> Cc: David Kastrup <dak@gnu.org>, emacs-devel@gnu.org
> 
> Emacs, which doesn't maintain a separate "layed-out-document" data
> structure at all

Actually, I think it does: that's the "glyph matrices", which are a
structured description of what should be drawn.  The display-specific
back end then actually draws the glyphs according to what the glyph
matrix tells it.

Not sure if it's relevant or important for this discussion, though.

Re: redisplay system of emacs

[Paul R]

Hi Eli,

>> As many emacs youngsters, I feel that emacs really uses too much
>> home-made code. Surprisingly, Emacs does not benefit that much from
>> the free software ecosystem.

Eli> That's a pretty general assessment. Any data points other than the
Eli> display engine?

Emacs Lisp and all the librairies that emacs hackers had to implement on
top of it, IOW the 'emacs lisp standard library'.

Since 80ies, many languages appeared, and many of them meet very well
the requirements to extend a text editor. And because they are
general-purpose language, they do much more. Designing a language,
implementing it, maintaining it, providing a large and up to date
standard library, is a project on its own. Scheme, Ruby or Python come
to mind. The latters, at least, come with extensive support to parsing,
file operations, networking and so on.

Eli> "Big job" is an understatement of the century. Most of these
Eli> "generic free software fragments" are not written with Emacs
Eli> peculiar requirements in mind, and don't have a Lisp API.

Yes you are right. I'm mostly thinking loud how could be designed emacs
in today landscape.


-- 
  Paul

Re: redisplay system of emacs

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Paul R <paul.r.ml@gmail.com>
>
>> As many emacs youngsters, I feel that emacs really uses too much
>> home-made code. Surprisingly, Emacs does not benefit that much from
>> the free software ecosystem.
>
> That's a pretty general assessment.  Any data points other than the
> display engine?

One main data point is that its copyright assignment policies don't
allow taking XEmacs code.

Since only explicitly FSF copyright-assigned contributions can be used
in Emacs, Emacs' involvement in the "free software ecosystem" as such is
purely contributing, not taking.  As far as generally available material
such as toolkit libraries are concerned, Emacs does not benefit more or
less than proprietary applications from the "free software ecosystem".

It is a consciously made tradeoff.  It has been expensive at times.  Not
everything comes cheap.

-- 
David Kastrup

Re: redisplay system of emacs

[David Kastrup]

Paul R <paul.r.ml@gmail.com> writes:

> Hi Eli,
>
>>> As many emacs youngsters, I feel that emacs really uses too much
>>> home-made code. Surprisingly, Emacs does not benefit that much from
>>> the free software ecosystem.
>
> Eli> That's a pretty general assessment. Any data points other than the
> Eli> display engine?
>
> Emacs Lisp and all the librairies that emacs hackers had to implement on
> top of it, IOW the 'emacs lisp standard library'.
>
> Since 80ies, many languages appeared, and many of them meet very well
> the requirements to extend a text editor. And because they are
> general-purpose language, they do much more.

So where are all those powerful extensible text editors?

> Designing a language, implementing it, maintaining it, providing a
> large and up to date standard library, is a project on its
> own. Scheme, Ruby or Python come to mind. The latters, at least, come
> with extensive support to parsing, file operations, networking and so
> on.

So where are all those powerful extensible text editors?

-- 
David Kastrup

Re: redisplay system of emacs

[Lennart Borgman]

On Fri, Jan 29, 2010 at 11:30 AM, David Kastrup <dak@gnu.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Fri, Jan 29, 2010 at 11:17 AM, David Kastrup <dak@gnu.org> wrote:
>>>
>>> Uh, Emacs needs to work without noticeable delays and with syntax
>>> highlighting for documents of _book_ size, easily containing thousands
>>> of pages.
>>
>>
>> Is this necessarily related to the display engine?
>
> Since the Mozilla model renders the whole document (admittedly starting
> before it is completely available), I don't see how it should be
> otherwise.


I am not sure that is the Mozilla model. It is just necessary for web pages.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Paul R <paul.r.ml@gmail.com>
> Date: Fri, 29 Jan 2010 14:06:20 +0100
> Cc: Emacs-devel@gnu.org
> 
> >> As many emacs youngsters, I feel that emacs really uses too much
> >> home-made code. Surprisingly, Emacs does not benefit that much from
> >> the free software ecosystem.
> 
> Eli> That's a pretty general assessment. Any data points other than the
> Eli> display engine?
> 
> Emacs Lisp and all the librairies that emacs hackers had to implement on
> top of it, IOW the 'emacs lisp standard library'.

And what replacements do you see for that in the ``free software
ecosystem''?  Apart of replacing Lisp with another language, an
endeavor that no one succeeded in to this day (although there were
attempts, and one of them is even still alive), what else could be
done to replace all that?

Re: redisplay system of emacs

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
>> Emacs, which doesn't maintain a separate "layed-out-document" data
>> structure at all
>
> Actually, I think it does: that's the "glyph matrices", which are a
> structured description of what should be drawn.  The display-specific
> back end then actually draws the glyphs according to what the glyph
> matrix tells it.

That is only used for the portion of the document actually being
displayed on the screen at the moment, though, so it's more of a
temporary data structure representing the current screen than a
"layed-out" copy of the entire document like Mozilla uses.

-Miles

-- 
Brain, n. An apparatus with which we think we think.

Re: redisplay system of emacs

[Chong Yidong]

Paul R <paul.r.ml@gmail.com> writes:

> Since 80ies, many languages appeared, and many of them meet very well
> the requirements to extend a text editor. And because they are
> general-purpose language, they do much more. Designing a language,
> implementing it, maintaining it, providing a large and up to date
> standard library, is a project on its own. Scheme, Ruby or Python come
> to mind. The latters, at least, come with extensive support to parsing,
> file operations, networking and so on.

This has been discussed before ;-)

http://lists.gnu.org/archive/html/emacs-devel/2006-04/msg00030.html

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: dak@gnu.org,  lennart.borgman@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 30 Jan 2010 00:10:16 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Emacs, which doesn't maintain a separate "layed-out-document" data
> >> structure at all
> >
> > Actually, I think it does: that's the "glyph matrices", which are a
> > structured description of what should be drawn.  The display-specific
> > back end then actually draws the glyphs according to what the glyph
> > matrix tells it.
> 
> That is only used for the portion of the document actually being
> displayed on the screen at the moment

But of course: that's all Emacs display engine cares about.

Re: redisplay system of emacs

[Stefan Monnier]

> In fact, the size of the total document should not affect editing
> speed.  And we are not just talking "editing" or fixed documents here:
> run-off compilation/log output needs to be fast.

That's the ideal goal, yes.  And Emacs is hopefully closer to it than
web browsers.  But Emacs is definitely not there in general: in some
cases (fundamental-mode, no line-number-mode, ...), it's mostly
there, but once you add enough font-locking and stuff, document size
does matter.  So we end up only caring about "fast enough on oldish
hardware when editing fairly large documents", and give up on "still
fast even on C header files larger than your memory" (sorry, Alan ;-)


        Stefan

Re: redisplay system of emacs

[Stefan Monnier]

Eli> That's a pretty general assessment. Any data points other than the
Eli> display engine?

Most of what Emacs's code offers is available elsewhere one way or
another in *apparently* better shape and with an active community.
That's true of the redisplay as we've been discussing, it's true of the
extension language, the GC, the coding-conversion libraries, the
wiget/menu code, and probably a lot more.

So, yes, it's probably fairly easy to do an Emacs-like editor using
another extension language etc... and the result is likely to be
cleaner, more efficient, better maintained code in many ways.
Actually such Emacs-like editors exist, using extension languages like
CommonLisp, OCaml, Haskell, Python, younameit.

The problem is that they're not Emacs, so they start with a small user
community and it's hard for them to grow.  At some point, I've
considered the possibility to switch to one of those (where the
extension language is statically typed ;-), but ... they didn't have
PCL-CVS, diff-mode, Gnus, ... so it was really an uphill battle.
Maybe later.


        Stefan

Re: redisplay system of emacs

[Óscar Fuentes]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

[snip]

> At some point, I've considered the possibility to switch to one of
> those (where the extension language is statically typed ;-), but
> ... they didn't have PCL-CVS, diff-mode, Gnus, ... so it was really an
> uphill battle.  Maybe later.

Now that you are on the mod of discussing far-fetched possibilities,
think on a language that...

* Statically typed ;-) No fancy type system, though. Dynamic typing can
  be emulated.

* Can easily interface with C/C++. Registering a C function, type or
  variable is a one-liner. In principle, it is possible to automatize
  the task.

* Optionally, can compile to efficient native code.

* Lisp-like syntax.

* Has defmacro (yay!).

* Has eval.

* Has instrospection facilities.

Whenever I look at the Emacs C codebase, I think that a language like
the described above would do wonders replacing a good chunk of it.

I have such language with the corresponding compiler. It has its warts,
no doubt. But, as a maintainer of the Emacs C code, how it looks the
perspective of using a language like that instead of the Emacs C
pseudo-dialect? Is it painful enough to maintain the C codebase to make
you dream about switching to something else?

Re: redisplay system of emacs

[grischka]

David Kastrup wrote:
 > Since only explicitly FSF copyright-assigned contributions can be used
 > in Emacs, Emacs' involvement in the "free software ecosystem" as such is
 > purely contributing, not taking.

Emacs is contributing in the sense of providing a swiss knife to
produce free software, but not in the sense of being free software.
For that it would need to be modular.

One could imagine a modular emacs consisting of a lisp engine that
could be used without the editor, a display module that can be used
without windows/frames, an UI module that would allow to build
standalone applications.  That is applications with emacs under the
hood, as opposed to the one emacs with many applications under the
hood.

But as is, emacs comes with an implicit structural clause to its
license, as in "[You may convey a work based on the Program, ...]
BUT WE DO OUR BEST TO PREVENT THAT."

Well, that is how I see it.  Technical and structural reasons are
real, and if in practice such reasons prevent distribution of
modified versions, then the license becomes pointless.

Now, of course the problem exists elsewhere too.  People may or may
not be aware, and may or may not address it, in this or that way.

--- grischka

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-devel@gnu.org
> Date: Fri, 29 Jan 2010 13:35:28 -0500
> 
> Eli> That's a pretty general assessment. Any data points other than the
> Eli> display engine?
> 
> Most of what Emacs's code offers is available elsewhere one way or
> another in *apparently* better shape and with an active community.
> That's true of the redisplay as we've been discussing, it's true of the
> extension language, the GC, the coding-conversion libraries, the
> wiget/menu code, and probably a lot more.
> 
> So, yes, it's probably fairly easy to do an Emacs-like editor using
> another extension language etc...

I thought the issue was not creation of a new editor, but using in
Emacs code written elsewhere for some tasks Emacs now does with code
written specifically for it.

> At some point, I've considered the possibility to switch to one of
> those (where the extension language is statically typed ;-), but
> ... they didn't have PCL-CVS, diff-mode, Gnus, ... so it was really
> an uphill battle.

Heh, that's how I came to Emacs 15 years ago: I got tired
reimplementing in a workalike all the features I missed and in Emacs
were there to be used...

Re: redisplay system of emacs

[Stephen J. Turnbull]

grischka writes:

 > But as is, emacs comes with an implicit structural clause to its
 > license, as in "[You may convey a work based on the Program, ...]
 > BUT WE DO OUR BEST TO PREVENT THAT."

That's simply not true.  There have historically been a large number
of editors based on Emacs code, some of which are in active
development (of course XEmacs is an example, and many wilder
alternatives have existed: pymacs, perlmacs).  And every long time
user has more or less substantial app-specific code in their init
file, while many have private branches, sometimes shared with friends.
Corporate IT departments often have quite substantial applications
built on Emacs (a friend of mine feeds his family by maintaining a bug
database query frontend in Emacs Lisp for a leading technology company
-- that's his fulltime job).

I can say to those who say "we can rebuild Emacs using third-party
libraries and it would be more maintainable and flexible" that XEmacs
has tried that several times for different areas of functionality, and
for one reason or another the code has always come back out again.  In
fact, those third party libraries have always either turned out to
lack the flexibility demanded by an editor application, or to be less
stable than XEmacs itself, and since they didn't offer any high-level
capabilities that couldn't be coded in three lines of Lisp, they went
back on the shelf.

Of course low-level facilities like displaying glyphs from fonts are
best done with specialized libraries.  Guess what?  Emacs uses native
X for legacy fonts, but it doesn't go to Xrender for scalable fonts,
it uses freetype and/or Xft.  But even at the level of displaying
images which you would think would be eminently suitable for a third-
party library like libmagick or maybe netpbm, XEmacs was forced to
abandon its attemtp to use libmagick as a common interface to all
image formats, and go back to direct support of the various underlying
libraries such as libjpeg and libungif -- which itself turned out to
be too unstable, so instead of using it, we had until recently a
locally hacked code based on an older version.  Today, we've gone back
to using giflib directly and libmagick is probably stable enough to
use as a replacement for our custom high-level code, but if you were
to try to use all of the various libraries proposed to replace core
Emacs functionality, some would be lemons and you'd take two steps
back for every step forward.

You can complain that "plugins" should be written in a more popular
language like Python or Visual BASIC<snort />, of course, but as
several others have pointed out, you would not be able to take
advantage of 3 decades worth of Lisp libraries in doing that, and
often the libraries available in Python or Ruby are more buggy because
much younger.

And if you look at what people who want a non-Emacs advanced editor
for their Python or Ruby projects do, they don't write it in Python or
Ruby.  (Yes, I know about IDLE, and it's not an advanced editor, nor
is it as easy to extend as Emacs.)  No, they go hack on vim!

As for browsing 1GB (well, for Emacs it would be 256MB, I guess?) log
files, there's nothing like (X)Emacs!

Re: redisplay system of emacs

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> grischka writes:
>
>  > But as is, emacs comes with an implicit structural clause to its
>  > license, as in "[You may convey a work based on the Program, ...]
>  > BUT WE DO OUR BEST TO PREVENT THAT."
>
> That's simply not true.  There have historically been a large number
> of editors based on Emacs code, some of which are in active
> development (of course XEmacs is an example, and many wilder
> alternatives have existed: pymacs, perlmacs).

Maybe the point was that it is not trivial to get home-brewn extensions
back into Emacs upstream.  In particular, if they have been brewed in
somebody else's home...

> I can say to those who say "we can rebuild Emacs using third-party
> libraries and it would be more maintainable and flexible" that XEmacs
> has tried that several times for different areas of functionality, and
> for one reason or another the code has always come back out again.

Thanks for that data point.

> As for browsing 1GB (well, for Emacs it would be 256MB, I guess?)

most-positive-fixnum => 536870911

> log files, there's nothing like (X)Emacs!

less is better, actually.  I need my virtual memory for other things
than decorated log files.

-- 
David Kastrup

Re: redisplay system of emacs

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Sat, 30 Jan 2010 14:39:48 +0900
> Cc: emacs-devel@gnu.org
> 
> As for browsing 1GB (well, for Emacs it would be 256MB, I guess?) log
> files, there's nothing like (X)Emacs!            ^^^^^

512MB (on 32bit machines).

Re: redisplay system of emacs

[Fabian Ezequiel Gallina]

2010/1/29 Stefan Monnier <monnier@iro.umontreal.ca>:
>
> So, yes, it's probably fairly easy to do an Emacs-like editor using
> another extension language etc... and the result is likely to be
> cleaner, more efficient, better maintained code in many ways.
> Actually such Emacs-like editors exist, using extension languages like
> CommonLisp, OCaml, Haskell, Python, younameit.
>
> The problem is that they're not Emacs, so they start with a small user
> community and it's hard for them to grow.  At some point, I've
> considered the possibility to switch to one of those (where the
> extension language is statically typed ;-), but ... they didn't have
> PCL-CVS, diff-mode, Gnus, ... so it was really an uphill battle.
> Maybe later.
>

I think Emacs Lisp is a great extension language. To be honest, when I
was changing my current programing editor and was testing VIM and
Emacs, I've choosen Emacs mainly because of Lisp (...and because I
hated switching modes in VIM).

I have to say also, that beign (mainly) a Python programmer I feel
Emacs Lisp very familiar in some places, and I guess it's true what
people say that is an enlightning language.

There is a really nice project which allows you to extend GNU/Emacs
using Python called Pymacs[0]. I guess that instead of changing Emacs
Lisp we should develop and encourage this kind of interfaces. This
will keep everyone happy, people who actually wants to learn Emacs
Lisp and people who don't want quit using their preferred language.

An example of an extension using Pymacs would be ropemacs[1] which is
a Python refactoring tool (that saves my life everyday).

[0] http://pymacs.progiciels-bpi.ca/pymacs.html
[1] http://rope.sourceforge.net/ropemacs.html


Regards,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: redisplay system of emacs

[David Kastrup]

Fabian Ezequiel Gallina <galli.87@gmail.com> writes:

> 2010/1/29 Stefan Monnier <monnier@iro.umontreal.ca>:
>>
>> So, yes, it's probably fairly easy to do an Emacs-like editor using
>> another extension language etc... and the result is likely to be
>> cleaner, more efficient, better maintained code in many ways.
>> Actually such Emacs-like editors exist, using extension languages
>> like CommonLisp, OCaml, Haskell, Python, younameit.
>>
>> The problem is that they're not Emacs, so they start with a small
>> user community and it's hard for them to grow.  At some point, I've
>> considered the possibility to switch to one of those (where the
>> extension language is statically typed ;-), but ... they didn't have
>> PCL-CVS, diff-mode, Gnus, ... so it was really an uphill battle.
>> Maybe later.
>
> I think Emacs Lisp is a great extension language.

I don't think so.  It is a functional language used in a procedural
style, causing permanent changes as its ultimate purpose.  Not even with
lexical bindings.

It does automatic allocation and garbage collection, is concise and with
data structures catered to the task at hand.  Mostly.

If it were a great extension language, it would be in use outside of
Emacs.

Things like Lua would meet the bill quite better, but they are too late
in the game now.

Emacs is not a program, it is an ecosystem.  And ecosystems play by
different rules than individuals.  Emacs has not yet reached
equilibrium, the state where bit rot destroys functionality at the same
pace as new stuff gets added.

Moving that ecosystem to a different language is about as ambitious as
moving a carbon-based ecosystem to silicon-based.

You don't know where to start.

-- 
David Kastrup

Re: redisplay system of emacs

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > grischka writes:
 > >
 > >  > But as is, emacs comes with an implicit structural clause to its
 > >  > license, as in "[You may convey a work based on the Program, ...]
 > >  > BUT WE DO OUR BEST TO PREVENT THAT."
 > >
 > > That's simply not true.  There have historically been a large number
 > > of editors based on Emacs code,
 > 
 > Maybe the point was that it is not trivial to get home-brewn extensions
 > back into Emacs upstream.  In particular, if they have been brewed in
 > somebody else's home...

That's not how I read it.  It seems to me that the OP (and others in
this thread, as well is in the package manager threads) are looking
for a more open ecology, something like the various C*AN networks on
the "Emacs is a platform for application development" side, and FFI on
the the "Emacs interacts with various communication protocols and
storage formats" side.

In other words, less, not more, centralization.  A smaller core Emacs
doing less better, and delegating more to 3rd-party libraries.

 > > As for browsing 1GB (well, for Emacs it would be 256MB, I guess?)
 > 
 > most-positive-fixnum => 536870911

Oops.  I wish you had posted 30 minutes earlier, I just misquoted
myself in another channel.  Thank you (and Eli) for the correction,
anyway.

 > > log files, there's nothing like (X)Emacs!

 > less is better, actually.  I need my virtual memory for other things
 > than decorated log files.

I don't decorate the log files.  I use things like M-x occur or M-x
delete-non-matching lines on them, though.  Does less support such
features now?

Re: redisplay system of emacs

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > most-positive-fixnum => 536870911
>
> Oops.  I wish you had posted 30 minutes earlier, I just misquoted
> myself in another channel.  Thank you (and Eli) for the correction,
> anyway.

This value has been in flux, anyway.  I suppose it might at one point of
time reach the 1GB value of XEmacs (which is pretty much the maximum
power of 2 possible for tagged words).

>  > > log files, there's nothing like (X)Emacs!
>
>  > less is better, actually.  I need my virtual memory for other things
>  > than decorated log files.
>
> I don't decorate the log files.  I use things like M-x occur or M-x
> delete-non-matching lines on them, though.  Does less support such
> features now?

Well, you can search for non-matches by starting the regexp with !.  And
you can limit the display to matching/non-matching lines using &.  And
you can search across multiple files with specific search pattern
characters as well.

It's not really all too bad.

-- 
David Kastrup

Re: redisplay system of emacs

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Sat, 30 Jan 2010 20:01:19 +0900
> Cc: emacs-devel@gnu.org
> 
>  > > log files, there's nothing like (X)Emacs!
> 
>  > less is better, actually.  I need my virtual memory for other things
>  > than decorated log files.
> 
> I don't decorate the log files.  I use things like M-x occur or M-x
> delete-non-matching lines on them, though.  Does less support such
> features now?

Since version 424, Less has this:

  * New "&" command allows filtering of lines based on a pattern.

But you need that version installed (it was released less than a year
ago) to take advantage of this, and it doesn't seem to support the
equivalent of delete-matching-lines; instead, you will need to invert
the regexp manually.

In my daytime job, I also need to browse huge log files.  I love Less
and use it for simple browsing and searching (the fact that it doesn't
have Emacs limitations of size is sometimes very important).  But as
soon as I need to do something complex, like compare two large log
files or jump between several places, I always fire up Emacs (if it's
not up already), because it's just too painful with Less, even though
features to do that do exist.  If the file is too large, I just cut
the interesting part with Sed.

Re: redisplay system of emacs

[Richard Stallman]

    Whenever I look at the Emacs C codebase, I think that a language like
    the described above would do wonders replacing a good chunk of it.

The code written in C in Emacs is mostly in C so it can be as fast as
possible.  To move it into such a language would be a big practical
loss.

There is some code that was originally in C but doesn't need to be
fast.  We have rewritten some of that code in Lisp in recent years.
If more such C code remains, we could move that to Lisp too.

Re: redisplay system of emacs

[Richard Stallman]

    Emacs is contributing in the sense of providing a swiss knife to
    produce free software, but not in the sense of being free software.

You are factually mistaken.  Emacs is free software, and people take
advantage of that freedom with their modified versions.

The term "ecosystem" is best avoided because it supposes an
amoral stance.  See http://www.gnu.org/philosophy/words-to-avoid.html
for the explanation.

Re: redisplay system of emacs

[Paul R]

David, Eli, Stephen, Stefan, and others, thank you very much for your
insights. That was well argumented, and very constructive, which I guess
is not an easy task when speaking from the inside reality of the system,
to an enthusiastic outsider. As Stefan pointed out, decades of
developers horsepower has so far provided better results than
alternative systems, although better designed and backed-up by robust
third-party components.

Is there an official "emacs 2" page (or project) where emacs people can
elaborate on how they would design emacs, if they had to start it from
scratch today ? If not, I would be happy to start one and try to
sumarize your points, from this thread. It would also be interesting, in
this project, to monitor various other attempts to create extensible
systems, and retrieve feedback. For exemple, I am amazed by the success
of the Xmonad window manager in this field : dynamic community, good
documentation, robust and correct behaviour. Few people have expected it
3 years ago, when this Haskell extensible system was first announced.
A lot of people (including me) considered static strictness to be a poor
design choice for extensibility. This is just an example of interesting
feedback from other's experiences.



-- 
  Paul

Re: redisplay system of emacs

[Paul R]

Richard,

> The term "ecosystem" is best avoided because it supposes an amoral
> stance. See http://www.gnu.org/philosophy/words-to-avoid.html for the
> explanation.

I don't think the word ecosystem "(...) implies the absence of intention
and ethics", as stated in this page. It does not implie the presence of
them either. I think they are independant concepts, and that a free
software ecosystem is an acceptable metaphore because it shows that
their is interdependency (in fact free software licence favours this
interdependency).

Can you suggest an alternative word that expresses this simple, yet
fundamental, concept ?



-- 
  Paul

Re: redisplay system of emacs

[Óscar Fuentes]

Richard Stallman <rms@gnu.org> writes:

>     Whenever I look at the Emacs C codebase, I think that a language like
>     the described above would do wonders replacing a good chunk of it.
>
> The code written in C in Emacs is mostly in C so it can be as fast as
> possible.  To move it into such a language would be a big practical
> loss.

Just a clarification:

You missed the part that says "Optionally, can compile to efficient
native code." Here, efficient means "at least as good as C".

Nowadays, thanks to projects like LLVM, it is easy to roll an
industrial-grade optimizing compiler if the language runtime
requirements are not too far from C. It is straightforward to design a
language that equals C in speed of execution and that is way more
expressive.

Actually, some of those languages often compile applications to code
that is faster than their C counterpart, as they are more
optimizer-friendly.

Re: redisplay system of emacs

[Alan Mackenzie]

Hi, Stephen,

On Sat, Jan 30, 2010 at 08:01:19PM +0900, Stephen J. Turnbull wrote:

> In other words, less, not more, centralization.  A smaller core Emacs
> doing less better, and delegating more to 3rd-party libraries.

You mean, with a project team based around a micro-colonel?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: redisplay system of emacs

[Alan Mackenzie]

Hi, Paul,

On Sat, Jan 30, 2010 at 01:11:57PM +0100, Paul R wrote:
> Richard,

> > The term "ecosystem" is best avoided because it supposes an amoral
> > stance. See http://www.gnu.org/philosophy/words-to-avoid.html for the
> > explanation.

> I don't think the word ecosystem "(...) implies the absence of intention
> and ethics", as stated in this page.

Are you a native English speaker?  "Ecosystem" is a system of ecology,
which is the study of how organisms react with eachother and their
shared environment.  Implicit in ecology is its participants'
obliviousness to ecology.

> It does not imply the presence of them either. I think they are
> independant concepts, and that a free software ecosystem is an
> acceptable metaphor, because it shows that there is interdependency
> (in fact free software licence favours this interdependency).

There are other words which also imply interdependency yet which are
less laden with loaded meanings.  "Ecosystem" implies its participants
(hackers etc.) are on the level of bugs, beetles and bacteria.  It
denigrates hackers, suggesting they are simply swept along helplessly by
outrageous fortune, rather than being the agents of it.  Some of these
other words would be better, much better, such as ....

> Can you suggest an alternative word that expresses this simple, yet
> fundamental, concept ?

A "community" for example, which expresses all the tenets of
interdependency and tension.  If you want to emphasise the ideas of
competition between bits of free software (say, between perl, python and
ruby), the best word is perhaps "market", or "marketplace of ideas".

> -- 
>   Paul

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: redisplay system of emacs

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Paul,
>
> On Sat, Jan 30, 2010 at 01:11:57PM +0100, Paul R wrote:
>> Richard,
>
>> > The term "ecosystem" is best avoided because it supposes an amoral
>> > stance. See http://www.gnu.org/philosophy/words-to-avoid.html for the
>> > explanation.
>
>> I don't think the word ecosystem "(...) implies the absence of intention
>> and ethics", as stated in this page.
>
> Are you a native English speaker?  "Ecosystem" is a system of ecology,
> which is the study of how organisms react with eachother and their
> shared environment.  Implicit in ecology is its participants'

Implicit in "ecosystem" you likely mean.

> obliviousness to ecology.
>
> There are other words which also imply interdependency yet which are
> less laden with loaded meanings.  "Ecosystem" implies its participants
> (hackers etc.) are on the level of bugs, beetles and bacteria.

With regard to their function, yes.  "Ecosystem" is used exactly when
talking about an emergent rather than controlled system.

>> Can you suggest an alternative word that expresses this simple, yet
>> fundamental, concept ?
>
> A "community" for example, which expresses all the tenets of
> interdependency and tension.

But it's simply the wrong choice of words if you want to include the
effects on entities that profit without active participation.

The FSF is a _charity_, and the whole point of a charity is that it
differs from a community by benefitting people on the receiving end.

> If you want to emphasise the ideas of competition between bits of free
> software (say, between perl, python and ruby), the best word is
> perhaps "market", or "marketplace of ideas".

Again, different connotations.

By far the largest ratio of profiting people don't act as an active part
in some "community".  Choosing terms excluding them from consideration
is not doing the concept full justice.

Since free software changes the world far beyond the scope of its active
communities, I don't find "ecosystem" a bad choice of words when
describing the _effects_ of free software, even though it may prosper
mostly from within communities.

-- 
David Kastrup

Re: redisplay system of emacs

[Juanma Barranquero]

On Sat, Jan 30, 2010 at 14:26, Alan Mackenzie <acm@muc.de> wrote:

> "Ecosystem" implies its participants
> (hackers etc.) are on the level of bugs, beetles and bacteria.

Yes, of course, because apes, whales, dolphins and other quite
intelligent animals are not part of any ecosystem.

    Juanma

Re: redisplay system of emacs

[Stephen J. Turnbull]

Paul R writes:

 > Is there an official "emacs 2" page (or project) where emacs people can
 > elaborate on how they would design emacs, if they had to start it from
 > scratch today ?

Dunno.  Some references for you, though:

The Emacswiki (http://www.emacswiki.org/), especially the
ExtensionLanguage, WishList, EmacsImplementations, and History pages
may be of interest, and a good place for such a resource.  Ben Wing's
Architecting XEmacs is "official" (as much as anything can be in a
free software project) but for the wrong Emacs
(http://www.xemacs.org/Architecting-XEmacs/).  The XEmacs Internals
manual is probably the most detailed description of current Emacs
implementation, although again it's the wrong Emacs (and partial).
(http://www.xemacs.org/Documentation/21.5/html/internals.html).

 > For exemple, I am amazed by the success of the Xmonad window
 > manager in this field : dynamic community, good documentation,
 > robust and correct behaviour.

I wasn't.  They knew what they wanted, the problem was well-defined,
and that community is oriented to correctness-by-design.

Note that Emacs is none of those (google for xwem.el and Emacsspeak
and tell me again anybody knows what Emacs users want and the problem
is well-defined :-), and the first two "is nots" are inherent in the
problem domain.  The third could be changed, but seems unlikely.  The
"kaizen" style (continuous improvement and extension of a simple idea
and initial implementation by dynamic hacking) has served the Emacs
community well so far.

Re: redisplay system of emacs

[Paul R]

Hello Alan,

> Are you a native English speaker? "Ecosystem" is a system of ecology,
> which is the study of how organisms react with eachother and their
> shared environment. Implicit in ecology is its participants'
> obliviousness to ecology.

No, I am not a native English speaker, so I cared to read the definition
I had at hand, as well as the wikipedia article before giving my point
of view. I could not relate what I read there to what I read in the GNU
words-to-avoid page.

> There are other words which also imply interdependency yet which are
> less laden with loaded meanings. "Ecosystem" implies its participants
> (hackers etc.) are on the level of bugs, beetles and bacteria. It
> denigrates hackers, suggesting they are simply swept along helplessly
> by outrageous fortune, rather than being the agents of it. Some of
> these other words would be better, much better, such as ....

A free software ecosystem is composed of free software, isn't it ?

Also, it is probably just cultural, but around me the word 'ecosystem'
connotes very respected ideas, of equilibrium, sustainability, fairness.

-- 
  Paul

Re: redisplay system of emacs

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > Are you a native English speaker?  "Ecosystem" is a system of ecology,
 > which is the study of how organisms react with eachother and their
 > shared environment.  Implicit in ecology is its participants'
 > obliviousness to ecology.

I am a native speaker of English, and while *in science* "ecosystem"
*normally* is used to refer to natural systems where the participants
are "oblivious to ecology", in *policy discussions* (like this thread)
it almost always signifies that that speaker takes a moral stance
which (1) values the ecosystem as a whole and (2) holds that conscious
participants in an ecosystem have a responsibility to help preserve
that ecosystem.

Both principles are somewhat opposed to the individualistic ethos of
the free software movement.  I am not at all surprised that (some)
diehard free software advocates dislike the word "ecosystem", and most
especially the suggestion that they are members of one.

 > There are other words which also imply interdependency yet which are
 > less laden with loaded meanings.  "Ecosystem" implies its participants
 > (hackers etc.) are on the level of bugs, beetles and bacteria.

Ah, if only we could aspire to such moral heights!  Unfortunately,
"just call me Lucifer, 'cause I'm in need of some restraint."  (It's
spelled differently in the Preamble to the GNU General Public License,
but you can find that statement there if you look. :-)

 > It denigrates hackers, suggesting they are simply swept along
 > helplessly by outrageous fortune, rather than being the agents of
 > it. 

Hackers certainly do behave outrageously on occasion.  However, while
they are not helpless, to call them "agents of fortune" is hubris.

 > A "community" for example, which expresses all the tenets of
 > interdependency and tension.

No good, sorry.  "Community" derives from the word "common".  As soon
as we speak of multiple communities, we're clearly lost some important
commonality, and the need for a term such as "ecosystem" becomes
urgent.  To borrow a word from David (+1 to everything he wrote, by
the way), "ecosystem" stands for the *emergent* properties of a
network of more or less different communities.

Any casual observer of the Japanese or US political systems is
immediately aware that communities only with extreme difficulty behave
as rationally as humans, let alone beetles.  I see no reason why a
network of communities shouldn't be treated as an ecosystem, even if
you object to the human members being treated as part of an ecosystem.

 > If you want to emphasise the ideas of competition between bits of
 > free software (say, between perl, python and ruby), the best word
 > is perhaps "market", or "marketplace of ideas".

But that is precisely *not* the desired connotation!  The idea is to
emphasize and encourage cooperation and sharing among those bits.  For
example, though Bazaar, git, Subversion, and Mercurial compete "in the
marketplace of ideas", they are currently engaged in hammering out the
"fastimport format", a common dump format for VCS data.  When they're
done, you'll be able to dump a bzr database and read it into git, and
get sane behavior.  Bazaar and Mercurial will probably even be able to
share code for dumping and undumping.

How'd that happen?  I'd sure like to know, because I'd like to apply
it to the Emacs-XEmacs-SXEmacs etc community.  But "community" doesn't
tell me anything about where to look.  Nor does "market".  "Ecosystem"
does....

Re: redisplay system of emacs

[Stephen J. Turnbull]

Paul R writes:

 > A free software ecosystem is composed of free software, isn't it ?

No, to be precise, it's composed of *users* of free software.

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 30 Jan 2010 13:51:41 +0100
> 
> Nowadays, thanks to projects like LLVM, it is easy to roll an
> industrial-grade optimizing compiler if the language runtime
> requirements are not too far from C. It is straightforward to design a
> language that equals C in speed of execution and that is way more
> expressive.
> 
> Actually, some of those languages often compile applications to code
> that is faster than their C counterpart, as they are more
> optimizer-friendly.

But switching to a language that is significantly less widespread than
C would mean significantly less potential contributors.

We could learn from Texinfo experience: the next release replaces the
venerable makeinfo implementation in C with a modified texi2html,
which is written in Perl.  I suspect that, once that happens, the only
one who will hack the translator will be its author and no one else.
We shall see.

Re: redisplay system of emacs

[Stefan Monnier]

>> So, yes, it's probably fairly easy to do an Emacs-like editor using
>> another extension language etc...
> I thought the issue was not creation of a new editor, but using in
> Emacs code written elsewhere for some tasks Emacs now does with code
> written specifically for it.

Given that there's no other Elisp implementation that I know of, if you
want to use one of the existing language implementations to replace
Emacs's own code...


        Stefan

Re: redisplay system of emacs

[Stephen J. Turnbull]

Stefan Monnier writes:

 > Given that there's no other Elisp implementation that I know of, if you
 > want to use one of the existing language implementations to replace
 > Emacs's own code...

Eh?<wink>  I think Miles might also <wink>, too ....  And Mike Sperber
and students, for that matter.

There's also librepl, used in the sawfish window manager for X11,
which was deliberately modeled on Elisp.  Obviously the domain-
specific functions will differ, but the Lisp environment was indeed
similar.

Re: redisplay system of emacs

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Nowadays, thanks to projects like LLVM, it is easy to roll an
>> industrial-grade optimizing compiler if the language runtime
>> requirements are not too far from C. It is straightforward to design a
>> language that equals C in speed of execution and that is way more
>> expressive.
>> 
>> Actually, some of those languages often compile applications to code
>> that is faster than their C counterpart, as they are more
>> optimizer-friendly.
>
> But switching to a language that is significantly less widespread than
> C would mean significantly less potential contributors.

The Emacs C codebase is, to most effects, written on a dialect of C that
requires learning by its own.

In principle, it is possible to create a language which is similar
enough to Lisp that the amount of learning it requires is similar in
magnitude to the Emacs C dialect for someone who already knows Lisp, but
being as optimizable as C and much more expressive and
maintainable. (Some people would retort that this was already tried:
that Emacs variant written on Common Lisp)

I know this is just wild dreaming given the current status of Emacs.

[snip]

Re: redisplay system of emacs

[Stefan Monnier]

> I think Emacs Lisp is a great extension language.

It's *very* difficult to make it fast, so it leaves a lot of work on
C part of the code.


        Stefan

Re: redisplay system of emacs

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 30 Jan 2010 20:21:18 +0100
> 
> > But switching to a language that is significantly less widespread than
> > C would mean significantly less potential contributors.
> 
> The Emacs C codebase is, to most effects, written on a dialect of C that
> requires learning by its own.

Yes, but that's hardly the same as learning an entirely new language.

Re: redisplay system of emacs

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> 
>> Actually, some of those languages often compile applications to code
>> that is faster than their C counterpart, as they are more
>> optimizer-friendly.
>
> But switching to a language that is significantly less widespread than
> C would mean significantly less potential contributors.

The majority of Emacs is written in Elisp.  Which is significantly less
widespread than pretty much anything else.

I don't think that in the last decade or so people contributed
non-trivial C code to Emacs without knowledge of Elisp.

-- 
David Kastrup

Re: redisplay system of emacs

[Richard Stallman]

    I don't think the word ecosystem "(...) implies the absence of intention
    and ethics", as stated in this page. It does not implie the presence of
    them either.

The term implies a stance in which intention and ethics do not enter.
Whoever takes that stance probably has some idea of ethics too, but
that stance does not treat it as pertinent.  That's the problem in the
term.

    I think they are independant concepts,

Treating them as independent is a step on the wrong path.  The first
question we should ask about a program is "Is this program ethical?"
For instance, is it free software?  If it is not free, it's unethical.

    Can you suggest an alternative word that expresses this simple, yet
    fundamental, concept ?

I think it's a secondary issue.  The reason software should be free is
because users deserve freedom.  Whether programs depend on each other
is merely a technical issue.  At most it affects the rate of some
development, but it doesn't play a role in the more basic question of
whether the development good or evil.

    Also, it is probably just cultural, but around me the word 'ecosystem'
    connotes very respected ideas, of equilibrium, sustainability, fairness.

The concept of fairness plays no role in the study of ecosystems.  We
don't ask whether it is fair for an owl to eat a mouse, or for a mouse
to eat a plant.  We just note that these interactions are part of the
system.

Real ecosystems are often not in equilibrium.  The populations of some
species can vary widely from year to year.  Some ecosystems cannot
continue without big changes.

Re: redisplay system of emacs

[Richard Stallman]

    Nowadays, thanks to projects like LLVM, it is easy to roll an
    industrial-grade optimizing compiler if the language runtime
    requirements are not too far from C. It is straightforward to design a
    language that equals C in speed of execution and that is way more
    expressive.

I am skeptical.  In any case, if we use a compiled language we will
compile it with GCC.  We must support other GNU packages whenever
possible.

Re: redisplay system of emacs

[grischka]

Richard Stallman wrote:
>     Also, it is probably just cultural, but around me the word 'ecosystem'
>     connotes very respected ideas, of equilibrium, sustainability, fairness.
> 
> The concept of fairness plays no role in the study of ecosystems.  We
> don't ask whether it is fair for an owl to eat a mouse, or for a mouse
> to eat a plant.  We just note that these interactions are part of the
> system.

But we are not owls or mice.  Humans in between have become able
to ask whether it is fair towards other humans or future generations
or even nature, if we burn oil for energy, as an example.  We also
recognize that this is not a secondary issue but may directly impact
our freedom.

Back to software, see that COPYING is only one of many files in the
emacs-x.x.tar.bz2 that people will receive.  This means that the
freedom of this software is defined by the legal clauses as well as
by the algorithms and design in the code.

So, in my opinion it is necessary to offer the potential in the software
in a shape that allows efficient reuse for other purposes.

--- grischka

Re: redisplay system of emacs

[Richard Stallman]

    > The concept of fairness plays no role in the study of ecosystems.  We
    > don't ask whether it is fair for an owl to eat a mouse, or for a mouse
    > to eat a plant.  We just note that these interactions are part of the
    > system.

    But we are not owls or mice.  Humans in between have become able
    to ask whether it is fair towards other humans or future generations
    or even nature, if we burn oil for energy, as an example.

Exactly.  This aspect of things is what the term "ecosystem" does not
recognize, and that's why it is better not to use that term here.

Re: redisplay system of emacs

[Stephen J. Turnbull]

Richard Stallman writes:
 >     > The concept of fairness plays no role in the study of ecosystems.  We
 >     > don't ask whether it is fair for an owl to eat a mouse, or for a mouse
 >     > to eat a plant.  We just note that these interactions are part of the
 >     > system.
 > 
 >     But we are not owls or mice.  Humans in between have become able
 >     to ask whether it is fair towards other humans or future generations
 >     or even nature, if we burn oil for energy, as an example.
 > 
 > Exactly.  This aspect of things is what the term "ecosystem" does not
 > recognize, and that's why it is better not to use that term here.

I really am amused by this turn of discussion, because advocates of
copyleft are in precisely the same position.  Their *amoral*,
objective analysis of human behavior is that some humans will try to
restrict the software freedom of others, and the vast majority of
those others will see no profit in resisting.  Therefore, a free
software license that *deliberately restricts* licensees' freedom is
carefully designed, a profoundly moral act.

In fact, even in the presence of copyright or patent, an individual
always has the right to choose only software under a free license, at
the cost of giving up proprietary software, thus preserving his own
freedom.  Therefore copyleft, restricting the freedom to bargain away
the "four freedoms" in return for something else of value, can be
justified morally *only* by reference to the software "ecosystem", ie,
emergent effects of your use of non-free software on my freedom.[1]

As far as I can see, opposing the use of the *word* "ecosystem" to
clarify how the analysis is conducted is simply an attempt to restrict
the field of discussion of your position and policies to ground you're
comfortable defending.

Footnotes: 
[1]  Strictly speaking, "but I *like* to apply copyleft licenses to
software I write" is an unanswerable moral justification -- but that
doesn't justify asking others to use copyleft for their work.

Re: redisplay system of emacs

[Richard Stallman]

     > Exactly.  This aspect of things is what the term "ecosystem" does not
     > recognize, and that's why it is better not to use that term here.

    I really am amused by this turn of discussion, because advocates of
    copyleft are in precisely the same position.  Their *amoral*,
    objective analysis of human behavior

This is a paradox -- an appearance of contradiction that comes from
a misunderstanding.

The argument for copyleft comes from taking a moral stance towards the
situation in which many people do not follow our moral ideals.  It is
a fact that many people in our field take an amoral stance towards
this issue, and it is important to recognize the facts, but that is
not the same as taking an amoral stance ourselves.

By contrast, if we call our software an "ecosystem", then we take
an amoral stance.  That's what we shouldn't do.

Thus, the difference between _our stance_ and our recognition of
_others' stances_ dispels the paradox.

Re: redisplay system of emacs

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>      > Exactly.  This aspect of things is what the term "ecosystem" does not
>      > recognize, and that's why it is better not to use that term here.
>
>     I really am amused by this turn of discussion, because advocates of
>     copyleft are in precisely the same position.  Their *amoral*,
>     objective analysis of human behavior
>
> This is a paradox -- an appearance of contradiction that comes from
> a misunderstanding.
>
> The argument for copyleft comes from taking a moral stance towards the
> situation in which many people do not follow our moral ideals.  It is
> a fact that many people in our field take an amoral stance towards
> this issue, and it is important to recognize the facts, but that is
> not the same as taking an amoral stance ourselves.
>
> By contrast, if we call our software an "ecosystem", then we take
> an amoral stance.  That's what we shouldn't do.

The whole point is that most people can't be bothered.  You can call
that good or bad, but their use and distribution of free software is not
governed by a moral stance.  And not that of the software authors
either.  And you'll find that most contributors can't be bothered about
licensing, either.  They'll sign papers and are glad that's it.

Whether or not you take a moral stance does not imply that everybody
else in the system does.  There is enough free software that nobody
bothers anymore about morals.  People contribute to free software
because it hardly makes a difference and is what others do.  There is
lots of free software by now, and little morals.

In a similar way, the stock market trades lots of money, but most money
is traded on money rather than on goods.  Lots of money, little payout.

An emergent system.  An ecosystem with rules of its own, rules that its
originators did not plan in that manner.

> Thus, the difference between _our stance_ and our recognition of
> _others' stances_ dispels the paradox.

"our stance" is lost in the noise.  But it still gets the world
somewhere, because the noise has no direction and will get things
backwards just as often as forwards.

-- 
David Kastrup

Re: redisplay system of emacs

[Lennart Borgman]

On Tue, Feb 2, 2010 at 10:42 PM, David Kastrup <dak@gnu.org> wrote:
>
> Whether or not you take a moral stance does not imply that everybody
> else in the system does.  There is enough free software that nobody
> bothers anymore about morals.  People contribute to free software
> because it hardly makes a difference and is what others do.  There is
> lots of free software by now, and little morals.


Maybe you are right, but there is perhaps a couple of things to notice:

- Quality: This is in my opinion a critical thing for free software.
If the quality is not good enough it will be a burdon to use free
software. This is a moral question then, especially since there are
still people that needs free software because of the cost.

- Future constraints: I think we should not forget about the
possibility of future restraints. All the evil attachs on the internet
gives arguments to restrict free software. I believe they will be
used. The way to meat that threat and protect integrity and freedom is
(again) quality.

Re: redisplay system of emacs

[Stephen J. Turnbull]

Richard Stallman writes:
 >      > Exactly.  This aspect of things is what the term "ecosystem" does not
 >      > recognize, and that's why it is better not to use that term here.
 > 
 >     I really am amused by this turn of discussion, because advocates of
 >     copyleft are in precisely the same position.  Their *amoral*,
 >     objective analysis of human behavior
 > 
 > This is a paradox -- an appearance of contradiction that comes from
 > a misunderstanding.

But the misunderstanding is yours.

 > The argument for copyleft comes from taking a moral stance towards the
 > situation in which many people do not follow our moral ideals.  It is
 > a fact that many people in our field take an amoral stance towards
 > this issue, and it is important to recognize the facts, but that is
 > not the same as taking an amoral stance ourselves.

Of course it's not.  But taking a moral stance does not imply taking
*your* moral stance.  There are other moral stances, and those who
hold them are often as fervent about them as you are about yours.  The
choice is not between your moral stance and an amoral stance; it is
among many moral stances (including the extreme case of an amoral
stance).

In particular, use of the word "ecosystem" is typically a signal that
the person is taking a moral stance that values relationships and
stability thereof.  Depending on the person, it is sometimes more,
sometimes less than they value software freedom.  (And sometimes more,
sometimes less than they value freedom of any kind.)

 > By contrast, if we call our software an "ecosystem", then we take
 > an amoral stance.  That's what we shouldn't do.

Of course we *should* take an amoral stance in explaining "how things
work".  It is madness to try to apply "should" to the facts.

And I don't know what you mean by "our software" (use of possessives
by a free software advocate? tut-tut!), but if there exists a separate
body of "their software", then it is technically incorrect to call
"our software" an "ecosystem".  Ecosystems are *closed* systems, but
software has a strong tendency to become related to other software.
Unless you are speaking of all software, it's not big enough to be an
ecosystem.

 > Thus, the difference between _our stance_ and our recognition of
 > _others' stances_ dispels the paradox.

No, it doesn't, because it doesn't explain why you refuse to use a
single word, "ecosystem", that emphasizes the existence of variety
(including but not restricted to variety of moral stances) and the
behavioral interactions that entails.  See also David's response.

Re: redisplay system of emacs

[David Kastrup]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Tue, Feb 2, 2010 at 10:42 PM, David Kastrup <dak@gnu.org> wrote:
>>
>> Whether or not you take a moral stance does not imply that everybody
>> else in the system does.  There is enough free software that nobody
>> bothers anymore about morals.  People contribute to free software
>> because it hardly makes a difference and is what others do.  There is
>> lots of free software by now, and little morals.
>
>
> Maybe you are right, but there is perhaps a couple of things to notice:
>
> - Quality: This is in my opinion a critical thing for free software.

It is because most people can't be bothered about freedom.

> If the quality is not good enough it will be a burdon to use free
> software. This is a moral question then, especially since there are
> still people that needs free software because of the cost.

Free software is not about cost.

-- 
David Kastrup

Re: redisplay system of emacs

[Juanma Barranquero]

On Wed, Feb 3, 2010 at 03:48, Stephen J. Turnbull <stephen@xemacs.org> wrote:

> Of course we *should* take an amoral stance in explaining "how things
> work".  It is madness to try to apply "should" to the facts.

Glad to hear someone say this. "ecosystem" is a description, or a
label, of a kind of process; what their participants think or want is
irrelevant *to the description*. Perhaps wolves have long
philosophical discussions among them about the need and ethics of
eating rabbits; that will affect *how* the ecosystem evolves, not
whether it *is* an ecosystem or not. Those that study the software
ecosystem can, and likely do, write long dissertations about the
effect of free software on the system, but surely not about the
labeling of the system per se.

    Juanma

Re: redisplay system of emacs

[Richard Stallman]

    > By contrast, if we call our software an "ecosystem", then we take
    > an amoral stance.  That's what we shouldn't do.

    The whole point is that most people can't be bothered.  You can call
    that good or bad, but their use and distribution of free software is not
    governed by a moral stance.

That is how things are.  The point is, we are trying to change how
things are, not just observe.

    Whether or not you take a moral stance does not imply that everybody
    else in the system does.

Our mission is to lead them, not imitate them.

Re: redisplay system of emacs

[Richard Stallman]

     > This is a paradox -- an appearance of contradiction that comes from
     > a misunderstanding.

    But the misunderstanding is yours.

I chose not to cast blame for it; I tried to point out the shift of
topic in a way that didn't criticize anyone.

It is clear that the shift came from you.  Your message was a response
to mine, and it shifted the topic subject in the way I described --
from whether we are amoral, to whether others are.  However, I am not
trying to castigate you for that.  Your message claimed to show a
contradiction in the free software philosophy, but my aim is not to
counterattack, only to show there is no contradiction.

    Of course it's not.  But taking a moral stance does not imply taking
    *your* moral stance.

This particular moral stance is that of the free software movement and
the GNU Project.  This project is based on that stance.

You are entitled to your own views, of course.

    No, it doesn't, because it doesn't explain why you refuse to use a
    single word, "ecosystem", that emphasizes the existence of variety
    (including but not restricted to variety of moral stances) and the
    behavioral interactions that entails.

People distribute and use software in a variety of ways; that is the
present situation.  We say some are good and some are bad.  Our goal
is to put an end to the one that are bad.

The term "ecosystem" is unhelpful for this goal because it implies a
nonjudgmental approach to that variety, and our whole purpose is based
on judging them.

Re: redisplay system of emacs

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     > By contrast, if we call our software an "ecosystem", then we take
>     > an amoral stance.  That's what we shouldn't do.
>
>     The whole point is that most people can't be bothered.  You can call
>     that good or bad, but their use and distribution of free software is not
>     governed by a moral stance.
>
> That is how things are.  The point is, we are trying to change how
> things are, not just observe.

We won't change the way how things are by calling them what we'd want
them to be instead.

> Our mission is to lead them, not imitate them.

You can't lead anybody by telling him that he already is where you want
him to be.  In particular if he isn't.

-- 
David Kastrup

Re: redisplay system of emacs

[Daniel Colascione]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/3/10 9:15 AM, David Kastrup wrote:
> We won't change the way how things are by calling them what we'd want
> them to be instead.

Recent political history would suggest that you are incorrect here.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAktphbwACgkQ17c2LVA10VtZ5ACgsvFSDk9bpViJ+ATD354Ufez2
hyQAnjIyySzwJNHlmWnBdrf4IJcZKPuJ
=lmyx
-----END PGP SIGNATURE-----

Re: redisplay system of emacs

[Stephen J. Turnbull]

Richard Stallman writes:

 >      > This is a paradox -- an appearance of contradiction that
 >      > comes from a misunderstanding.
 > 
 >     But the misunderstanding is yours.
 > 
 > I chose not to cast blame for it; I tried to point out the shift of
 > topic in a way that didn't criticize anyone.

Indeed you did.  But that was not possible for me, since I needed to
contradict your explicit claim that my position is due to a
misunderstanding.  That being so, I might as well make my criticism
explicit.

 > It is clear that the shift came from you.

There was no shift.

You see?  Even though I used passive phrasing this time, the criticism
is unmistakable.

 > Your message was a response to mine, and it shifted the topic
 > subject in the way I described -- from whether we are amoral, to
 > whether others are.

No, it did not.  Some others are amoral (and occasionally even
immoral), but that is a fact.  We are moral (if occasionally we slip),
and that is also a fact.  Who is amoral is simply not in question as
far as I can see.  My claim is that use of the word "ecosystem"
does not imply an amoral stance, and therefore we may use it.

The point is that "ecosystem" describes only one aspect of the
phenomenon of software development.  That aspect is not very
self-conscious and therefore amoral.  Nevertheless, it is powerful and
we should include it in our reckoning when we consider what kinds of
actions will advance our moral goals.  The word "ecosystem" is an aid
to that reckoning, and indeed used properly can be very persuasive, to
certain people, against copyright and patent.

 > Your message claimed to show a contradiction in the free software
 > philosophy, but my aim is not to counterattack, only to show there
 > is no contradiction.

Excuse me?  I did not mention "contradiction".  There is some kind of
misunderstanding here.  I described a similarity of the argument used
to justify copyleft to the kind of argument that uses the word
"ecosystem".  But that does not imply a philosophical contradiction in
your deprecation of the word "ecosystem".

 >     But taking a moral stance does not imply taking *your* moral
 >     stance.
 > 
 > This particular moral stance is that of the free software movement
 > and the GNU Project.  This project is based on that stance.

Certainly, and equally certainly I made no claim to the contrary.  But
I do claim that having a particular moral stance does not entitle you
to call other moral stances "amoral".

 > The term "ecosystem" is unhelpful for this goal because it implies a
 > nonjudgmental approach to that variety, and our whole purpose is based
 > on judging them.

The last part of that sentence doesn't make sense to me.

I disagree that it's non-judgmental.  Preserving ecosystems is a
positive value.  Interfering with their natural flows, as copyright
and patent clearly do in the realm of software, is bad, just as bad as
any pollution of our biological environment.

As for being unhelpful, understanding the existing modes of the
software distribution system as an ecosystem cannot replace advocacy
of software freedom for its own sake.  But it can be complementary.
It is important to the movement that people (inside and outside the
movement) understand that successes like the Linux kernel, the Apache
webserver, and the Mozilla web browser are not flukes.  These programs
were not written by "two genius programmers in a garage".  These
programs were written by a cast of thousands, including contributions
from some of the most ruthless users ("abusers", if you prefer) of
"intellectual property rights" in the world.

And they weren't written by radical reformers with a social agenda,
either.  The contribution of the GNU project is essential, as history
and the current social environment make it all too likely that the
cause of freedom will be ignored if nobody makes a point of freedom.
But the success of freely-licensed software without a social agenda is
our hope for a victory within the current term of copyright :-/, and
a proof of righteousness, if you like.

Sharing software freely *is* the natural state and equilibrium of the
"software ecosystem".  Distorting that equilibrium is wrong, morally
wrong.  Not in the same way nor to the same degree as violating
fundamental human rights, but it's still wrong.

You will probably respond that this is a variation on the Raymondist
heresy.  True, it's not an argument that software freedom is a basic
human right.  But it's different from Raymondism.  "Open source" is
based on an economic argument that software freedom is profitable.
The argument from the ecosystem is that software freedom is natural,
and that disturbing its equilibrium is *wrong*.  This is more closely
related to, and complementary to, the argument from freedom as a right
than it is to economism.

Regards,

Re: redisplay system of emacs

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> I disagree that it's non-judgmental.  Preserving ecosystems is a
> positive value.

No, it isn't.  Underdeveloped countries have a stable ecosystem of
enough people dying early due to scarcity of resources that population
is maintained.  Or at least they had until their land became "owned" and
the flow of money meant that it was more effective to starve them
completely and use the land for producing lifestock fodder for the
countries delivering the guns to the "land owners".

Preserving the preexisting ecosystem would not have been a positive
value.  A positive value would have been giving (rather than taking)
them the means to improve their manner of living.

There are complex and stable ecosystems with parasites passing through a
multiple number of hosts in their development.

Preserving those is not a positive value.

Proprietary software has its own ecosystems.  Preserving those is not
the aim of the FSF and others.

And free software also has its ecosystems.  Not all of that which
happens in those systems is desirable.

But that does not mean that the term is inappropriate to describe
phenomena which are not planned, but make for dynamics beyond the
individual control of the software creators.

-- 
David Kastrup

Re: redisplay system of emacs

[grischka]

Stephen J. Turnbull wrote:
> Nevertheless, it is powerful and
> we should include it in our reckoning when we consider what kinds of
> actions will advance our moral goals.  The word "ecosystem" is an aid
> to that reckoning, and indeed used properly can be very persuasive, to
> certain people, against copyright and patent.

Yep.  Just as a matter of lingual ecology such powerful term must be
reused and not be thrown away easily.

--- grischka

Re: redisplay system of emacs

[Óscar Fuentes]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

[snip]

> "Open source" is
> based on an economic argument that software freedom is profitable.

About this specific topic, is there any serious economic analysis of
Free Software? Because it is not clear to me that some people would get
the software they need on the world envisioned by the FSF.

[snip]

Re: redisplay system of emacs

[Lennart Borgman]

On Wed, Feb 3, 2010 at 7:36 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>
> [snip]
>
>> "Open source" is
>> based on an economic argument that software freedom is profitable.
>
> About this specific topic, is there any serious economic analysis of
> Free Software? Because it is not clear to me that some people would get
> the software they need on the world envisioned by the FSF.


I do not know of any, but it is an interesting subject.

Doing a reasonable economic analysis is difficult and it seems like it
often fails. Compare for example todays news: 37 miljon US citizens
need economic help to get the most basic food they need. Something
must be utterly wrong in an economic analysis that leads to such a
situation.

To me the important factor to analys today is redistribution because
that is where the market economy fails. Redistribution means
redistribution of power. Economic power. Political power. Power to
tell about experiences.

I think for some of us there is an ethical problem that when trying to
help with free software we can't avoid also helping those we think is
fighting against freedom. But it is the net effect that counts in the
end. An economical analysis should look upon that.

I could go on writing about this, but I guess it would be
uninteresting here ... ;-)

Re: redisplay system of emacs

[Ted Zlatanov]

On Wed, 3 Feb 2010 20:03:47 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> Doing a reasonable economic analysis is difficult and it seems like it
LB> often fails. Compare for example todays news: 37 miljon US citizens
LB> need economic help to get the most basic food they need. Something
LB> must be utterly wrong in an economic analysis that leads to such a
LB> situation.

The right way to state it:

"37 million US citizens are losing weight.  Want to know the secret?"

Only half-joking...

Ted

Re: redisplay system of emacs

[Lennart Borgman]

2010/2/3 Ted Zlatanov <tzz@lifelogs.com>:
> On Wed, 3 Feb 2010 20:03:47 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote:
>
> LB> Doing a reasonable economic analysis is difficult and it seems like it
> LB> often fails. Compare for example todays news: 37 miljon US citizens
> LB> need economic help to get the most basic food they need. Something
> LB> must be utterly wrong in an economic analysis that leads to such a
> LB> situation.
>
> The right way to state it:
>
> "37 million US citizens are losing weight.  Want to know the secret?"
>
> Only half-joking...


Those who want to loose weight might want to try my little pause.el
instead... ;-)

   http://bazaar.launchpad.net/~nxhtml/nxhtml/main/annotate/head%3A/util/pause.el

Re: redisplay system of emacs

[Stephen J. Turnbull]

Óscar Fuentes writes:

 > About this specific topic, is there any serious economic analysis of
 > Free Software?

Yes.  There is list on the FSF's web site, but it was chosen for
political correctness and some of it is trash, while the rest should
be considered extremely controversial.  There isn't much that's
reliably known, even 15 years or so into the research program.

If you're interested in following up, write me off-list; this thread
has gotten way out of hand.

Re: redisplay system of emacs

[Richard Stallman]

    Glad to hear someone say this. "ecosystem" is a description, or a
    label, of a kind of process; what their participants think or want is
    irrelevant *to the description*.

If we only wanted to study and describe them, "ecosystem" would be a
suitable term.  However, describing them is just secondary issue for
our main concern which is to make them more ethical.

Re: redisplay system of emacs

[Richard Stallman]

    The point is that "ecosystem" describes only one aspect of the
    phenomenon of software development.  That aspect is not very
    self-conscious and therefore amoral.  Nevertheless, it is powerful and
    we should include it in our reckoning when we consider what kinds of
    actions will advance our moral goals.

We already do take account of others' tendencies (whatever their
basis) when they are relevant to our decisions.

However, using the term "ecosystem" to refer to them would shift the
way WE frame the issues, from a moral/political one to an amoral
nonjudgmental one.  That is what we should not do.

    Excuse me?  I did not mention "contradiction".  There is some kind of
    misunderstanding here.  I described a similarity of the argument used
    to justify copyleft to the kind of argument that uses the word
    "ecosystem".

You didn't use the word "contradiction" but that was implicit
in the argument you made.  You attacked, I refuted, and now you
deny the attack.


We do not want to "preserve" the software "ecosystem".  On the contrary,
our aim is to eliminate part of it.  If you want a biological analogy
for what we do, the eradication of smallpox is a good example.

Re: redisplay system of emacs

[Richard Stallman]

You have noticed how words affect people's thoughts.

I recommend people read Lakoff's articles about how the words used
to frame issues affect politics.

Re: redisplay system of emacs

[Juanma Barranquero]

> If we only wanted to study and describe them, "ecosystem" would be a
> suitable term.  However, describing them is just secondary issue for
> our main concern which is to make them more ethical.

Then, it would be more reasonable to spend time and energy discussing
why or how do we try to make it more ethical, instead of alienating
people by confusing the terminology, which is both already in place,
and correct.

    Juanma

Re: redisplay system of emacs

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     The point is that "ecosystem" describes only one aspect of the
>     phenomenon of software development.  That aspect is not very
>     self-conscious and therefore amoral.  Nevertheless, it is powerful and
>     we should include it in our reckoning when we consider what kinds of
>     actions will advance our moral goals.
>
> We already do take account of others' tendencies (whatever their
> basis) when they are relevant to our decisions.
>
> However, using the term "ecosystem" to refer to them would shift the
> way WE frame the issues, from a moral/political one to an amoral
> nonjudgmental one.  That is what we should not do.

I don't see that.  The main point of freedom is that it benefits
everyone, including those who don't fight for it.

Do we really want to spread the impression that we don't want to have
anybody benefit from free software that does not agree with our own
values and moral stances?

The GPL creates a pool of free software shielded from proprietarization
of essential freedoms.  Do we really want to spread the idea that it is
immoral to prefer using free software unless you share our morals, and
that you should rather go elsewhere than benefit from it?

Freedom is not just for fighters.

-- 
David Kastrup

Re: redisplay system of emacs

[Stephen J. Turnbull]

Richard Stallman writes:

 >     Excuse me?  I did not mention "contradiction".
 > 
 > You didn't use the word "contradiction" but that was implicit
 > in the argument you made.  You attacked, I refuted, and now you
 > deny the attack.

I did not then and do not now deny that it was an "attack" of sorts.
Attacking is a fault of mine, I admit that.  But I have already told
you that as far as I'm concerned the "contradiction" interpretation is
untenable.  You are defending against an attack that never came.

As for the rest of your post, it's clear that my points have long
since been made: they have been constructively addressed by others.  I
see no point in continuing this thread, as nobody will profit from
doing so.

Re: redisplay system of emacs

[Richard Stallman]

    Yes.  There is list on the FSF's web site, but it was chosen for
    political correctness and some of it is trash, 

Name calling like this says more about your hostility than about us.

If you want to argue against the GNU Project's philosophy, please
use gnu.misc.discuss.  If you want to just call names, please do it
outside our lists.

Re: redisplay system of emacs

[Stephen J. Turnbull]

Richard Stallman writes:

 >     Yes.  There is list on the FSF's web site, but it was chosen for
 >     political correctness and some of it is trash, 
 > 
 > Name calling like this says more about your hostility than about us.

Ad hominem attacks say more about you than they do about me.  There
are people here searching for truth, and such usage is only going to
offend them and make them distrust you on this subject.  I imagine my
many "fans" will be quite amused at you taking me down this way, but
is it really worth it?

Having said that, I am indeed in error.  Going back for confirmation,
I could not find the list I referred to.  The article, indirectly
linked from www.fsf.org, at

          http://endsoftpatents.org/resources-for-economists

is in fact balanced, citing well-done research on both sides of the
aisle.  The page itself is well-written and presents a very difficult
but crucial issue (the "null-result bias" of statistical practice)
quite clearly, though with a slight amount of exaggeration (most
harmless).  The fact that it comes to the conclusion that patents are
bad is neither surprising nor evidence of bias; it's quite reasonable
based on the works cited, which are well-known in the field.

However, it is quite limited, though it would be a reasonable starting
point for a study of patent economics.  It is very much focused on
patents, refers mostly to resources which are either in print or
probably electronically available only with a subscription, and
doesn't really present a starting point for the study the OP seems
interested in.

The list of third-party resources at

         http://www.gnu.org/philosophy/third-party-ideas.html

is less balanced, but describes itself as a list of opinions.  This is
not problematic, either, but the OP probably will find nothing that
really addresses his question.

I do not know where the much more extensive list I recall went.  It
linked or cited a number of articles based on anecdotes of people
whose works were suppressed by the copyright or patent laws which
lacked generalizable analysis, as well as the extremely controversial
and flawed theoretical analysis by Boldrin and Levine (a
representative selection is _Perfectly Competitive Innovation_ <URL
http://levine.sscnet.ucla.edu/papers/pci_august06.pdf>, although I'm
not sure that was the work cited in the FSF's list).  Maybe it was an
earlier version of resources-for-economists that was hosted on
www.fsf.org.

In any case, as far as I can tell that list no longer exists on an
FSF-related site.  I apologize for the misinformation, and thank you
(as a representative of the FSF) for "taking out the trash".

Sincerely yours,

Re: redisplay system of emacs

[Richard Stallman]

    Then, it would be more reasonable to spend time and energy discussing
    why or how do we try to make it more ethical, instead of alienating
    people by confusing the terminology, which is both already in place,
    and correct.

To influence people we need to present arguments, and we do.  We also
need to choose our words to frame the issues in the right way, and we
do that too.

"Ecosystem" frames the issue according to an outlook that disagrees
with ours, so we don't use it.  The fact that other people use it is
no reason we should.  Perhaps they disagree with our basic outlook.
They have a right to their views, but it makes no sense for us to use
terms which promote their outlook.

Re: redisplay system of emacs

[grischka]

Richard Stallman wrote:
>     Then, it would be more reasonable to spend time and energy discussing
>     why or how do we try to make it more ethical, instead of alienating
>     people by confusing the terminology, which is both already in place,
>     and correct.
> 
> To influence people we need to present arguments, and we do.  We also
> need to choose our words to frame the issues in the right way, and we
> do that too.
> 
> "Ecosystem" frames the issue according to an outlook that disagrees
> with ours, so we don't use it.  The fact that other people use it is
> no reason we should.  Perhaps they disagree with our basic outlook.
> They have a right to their views, but it makes no sense for us to use
> terms which promote their outlook.
> 

You may have noticed that nowadays the term "ecosystem" covers
very well the "ideas of ethical responsibility" as enumerated at
    http://www.gnu.org/philosophy/words-to-avoid.html#Ecosystem
It is true however that we need to carefully avoid cheap biological
and evolutionary analogies.

In any case, it is a total mystery to me how you wanted to stop
your fellow software people to extend their native thinking in
terms of systems onto non-technical areas, eventually.

If that is not your outlook, what is your outlook then?  How do
you suppose to promote freedom with software if not in terms of
its nature, that is defining systems?

--- grischka

Re: redisplay system of emacs

[Richard Stallman]

    I don't see that.  The main point of freedom is that it benefits
    everyone, including those who don't fight for it.

Freedom is a benefit for everyone who has freedom, including those who
don't fight for it.  But we have a fight on our hands, and if we want
to win it, we need to encourange more people to fight for freedom.
We need to present a clear example of judging the right and
wrong of the things people do in the software field.

      Do we really want to spread the idea that it is
    immoral to prefer using free software unless you share our morals,

We never say that.

Re: redisplay system of emacs

[alin.s]

An improvement in redisplay for X can be done by defining in the edit area of
every window a subwindow for every character. For a window of geometry
200x70 of characters, it would be 1400 windows registered in X-server.

The advantage is that redisplay would work automatically; every character
has associated its own expose event; the event PointerMotionMask would
simply signify LeaveWindow and EnterWindow; so it will always be able to be
captured without resource consuming.

To say more, in order to clear a character it would require no computation,
but only a simply call of XClearWindow().

Every window could have its own font.

And to say more, an image of high dimension will be divided in many
subwindows, and emacs will be able to display images normally, not as a huge
glyph.

And finally, because this structure is identical to the geometry of the
console, the code for X and console can be unified in many places.

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27560255.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Juanma Barranquero]

On Fri, Feb 12, 2010 at 09:31, alin.s <alinsoar@voila.fr> wrote:

> An improvement in redisplay for X can be done by defining in the edit area of
> every window a subwindow for every character. For a window of geometry
> 200x70 of characters, it would be 1400 windows registered in X-server.

Wouldn't that be 14,000?

    Juanma

Re: redisplay system of emacs

[Jan Djärv]

alin.s skrev:
> An improvement in redisplay for X can be done by defining in the edit area of
> every window a subwindow for every character. For a window of geometry
> 200x70 of characters, it would be 1400 windows registered in X-server.

You are mad.  Everything would be 1400 times more.  1400 times more calls to 
the X server, 1400 times more GC:s created, 1400 times more events from the X 
server, 1400 times more Xft structures (XftDraws for example) created.
Emacs would be at least 1400 times slower.

> 
> The advantage is that redisplay would work automatically; every character
> has associated its own expose event; the event PointerMotionMask would
> simply signify LeaveWindow and EnterWindow; so it will always be able to be
> captured without resource consuming.

But Emacs still needs to figure out what character that is in that window in 
order to redraw it.

> 
> To say more, in order to clear a character it would require no computation,
> but only a simply call of XClearWindow().
> 
> Every window could have its own font.

So say we have 1400 windows each with a different font with a different size? 
  How do you purpose we lay this out?  Instead of laying out characters we are 
now laying out windows.  Same problem, but with an enormous overhead.

> 
> And to say more, an image of high dimension will be divided in many
> subwindows, and emacs will be able to display images normally, not as a huge
> glyph.

Again, making the display of an image to be so much slower, because instead of 
one (ideally), call to the X server, we now have one per window.  And what a 
nightmare to figure out what part of an image that needs to be redrawn and 
moved if the user changes the size of the Emacs frame...

> 
> And finally, because this structure is identical to the geometry of the
> console, the code for X and console can be unified in many places.
> 

Not very likely.

Did 1:st of April arrive early?

	Jan D.

Re: redisplay system of emacs

[alin.s]

alin.s skrev:
> An improvement in redisplay for X can be done by defining in the edit area
> of
> every window a subwindow for every character. For a window of geometry
> 200x70 of characters, it would be 1400 windows registered in X-server.

You are mad.  Everything would be 1400 times more.  1400 times more calls to 
the X server, 1400 times more GC:s created, 1400 times more events from the
X 
server, 1400 times more Xft structures (XftDraws for example) created.
Emacs would be at least 1400 times slower.

The computations are not done as you say. Please read the X windows manual.

If you make 1000 identical calls of Xlib functions, Xlib put them into its
queue, and send them once.

On the other side, taking into account that I forgot a ZERO, we could have
many times calls to Xlib, but anyways, not too much, taking into account the
queue of Xlib.

I do not know what is bad to keep in the server 100.000 of little
structures. Apart from memory consuming, no problem of speed.

You do not need 14.000 Graphics Contextes, only 1 for window is enough!

It is not 14000 times slower, it is a few times faster.

It is many times more consuming of memory of the server side. Please learn
Xlib programming to understand it.

> 
> To say more, in order to clear a character it would require no
> computation,
> but only a simply call of XClearWindow().
> 
> Every window could have its own font.

So say we have 1400 windows each with a different font with a different
size? 
  How do you purpose we lay this out?  Instead of laying out characters we
are 
now laying out windows.  Same problem, but with an enormous overhead.

Same problem, only 1 GC for every window.

> 
> And to say more, an image of high dimension will be divided in many
> subwindows, and emacs will be able to display images normally, not as a
> huge
> glyph.

Again, making the display of an image to be so much slower, because instead
of 
one (ideally), call to the X server, we now have one per window.  And what a 
nightmare to figure out what part of an image that needs to be redrawn and 
moved if the user changes the size of the Emacs frame...

We do have 1 call for window, and 1 network message to the server for every
update.

The only problem in this implementation would be to compute the size of
every window, and to compute its coordinates.


> 
> And finally, because this structure is identical to the geometry of the
> console, the code for X and console can be unified in many places.
> 

Not very likely.

I do not know. Maybe.


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27563610.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[alin.s]

Juanma Barranquero wrote:
> 
>> An improvement in redisplay for X can be done by defining in the edit
>> area of
>> every window a subwindow for every character. For a window of geometry
>> 200x70 of characters, it would be 1400 windows registered in X-server.
> 
> Wouldn't that be 14,000?
> 

I do not pretend the solution to be realistic, I just asked if it is good or
not.

The only structure that would be allocated is a lot of little windows. Apart
from this, the computations on the client side are simpler.



-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27563728.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Jan Djärv]

alin.s skrev:
> 
> 
> alin.s skrev:
>> An improvement in redisplay for X can be done by defining in the edit area
>> of
>> every window a subwindow for every character. For a window of geometry
>> 200x70 of characters, it would be 1400 windows registered in X-server.
> 
> You are mad.  Everything would be 1400 times more.  1400 times more calls to 
> the X server, 1400 times more GC:s created, 1400 times more events from the
> X 
> server, 1400 times more Xft structures (XftDraws for example) created.
> Emacs would be at least 1400 times slower.
> 
> The computations are not done as you say. Please read the X windows manual.

I have done so for every edition since X10.

> 
> If you make 1000 identical calls of Xlib functions, Xlib put them into its
> queue, and send them once.
> 

Yes it does put calls in a queue.  All 1400 of them instead of one.  But given 
how IPC works, in reality it will be many writes to the socket anyway, even if 
the queue tries to minimize that.  And the amount of data is still 1400 (or 
14000) times more.  X does not squeeze several calls into one, there are still 
several calls.  And they are not identical anyway, they are for different windows.

> 
> I do not know what is bad to keep in the server 100.000 of little
> structures. Apart from memory consuming, no problem of speed.

But you have to keep track of all handles to these data as well in Emacs so 
both have to have 14000 times more data.

> 
> You do not need 14.000 Graphics Contextes, only 1 for window is enough!

Ok, so GC:s wasn't a good example.

> It is many times more consuming of memory of the server side. Please learn
> Xlib programming to understand it.

No need to be condescending, it doesn't help your cause.  I've done X 
programming since the late 80:s.

> 
>> To say more, in order to clear a character it would require no
>> computation,
>> but only a simply call of XClearWindow().
>>
>> Every window could have its own font.
> 
> So say we have 1400 windows each with a different font with a different
> size? 
>   How do you purpose we lay this out?  Instead of laying out characters we
> are 
> now laying out windows.  Same problem, but with an enormous overhead.
> 
> Same problem, only 1 GC for every window.

No it is not the same problem.  For every window you will have to tell X where 
it will be positioned, i,e. x and y and width and height.  If every window 
have its own font with its own size you have to calculate all this for every 
window, and the recalculate it for all windows when a font changes in one 
window.  Sounds like what redisplay does today.

> 
>> And to say more, an image of high dimension will be divided in many
>> subwindows, and emacs will be able to display images normally, not as a
>> huge
>> glyph.
> 
> Again, making the display of an image to be so much slower, because instead
> of 
> one (ideally), call to the X server, we now have one per window.  And what a 
> nightmare to figure out what part of an image that needs to be redrawn and 
> moved if the user changes the size of the Emacs frame...
> 
> We do have 1 call for window, and 1 network message to the server for every
> update.

... times 14000 windows...

> 
> The only problem in this implementation would be to compute the size of
> every window, and to compute its coordinates.

So you have gained nothing really, redisplay is the hard part.

	Jan D.

Re: redisplay system of emacs

[alin.s]

I agree that it is not a good plan . I thought that the X server notifies
you exactly the characters that need redisplaying, and so it would be very
clear the redisplay for everybody !





Jan Djärv wrote:
> 
> alin.s skrev:
>> 
>> 
>> alin.s skrev:
>>> An improvement in redisplay for X can be done by defining in the edit
>>> area
>>> of
>>> every window a subwindow for every character. For a window of geometry
>>> 200x70 of characters, it would be 1400 windows registered in X-server.
>> 
>> You are mad.  Everything would be 1400 times more.  1400 times more calls
>> to 
>> the X server, 1400 times more GC:s created, 1400 times more events from
>> the
>> X 
>> server, 1400 times more Xft structures (XftDraws for example) created.
>> Emacs would be at least 1400 times slower.
>> 
>> The computations are not done as you say. Please read the X windows
>> manual.
> 
> I have done so for every edition since X10.
> 
>> 
>> If you make 1000 identical calls of Xlib functions, Xlib put them into
>> its
>> queue, and send them once.
>> 
> 
> Yes it does put calls in a queue.  All 1400 of them instead of one.  But
> given 
> how IPC works, in reality it will be many writes to the socket anyway,
> even if 
> the queue tries to minimize that.  And the amount of data is still 1400
> (or 
> 14000) times more.  X does not squeeze several calls into one, there are
> still 
> several calls.  And they are not identical anyway, they are for different
> windows.
> 
>> 
>> I do not know what is bad to keep in the server 100.000 of little
>> structures. Apart from memory consuming, no problem of speed.
> 
> But you have to keep track of all handles to these data as well in Emacs
> so 
> both have to have 14000 times more data.
> 
>> 
>> You do not need 14.000 Graphics Contextes, only 1 for window is enough!
> 
> Ok, so GC:s wasn't a good example.
> 
>> It is many times more consuming of memory of the server side. Please
>> learn
>> Xlib programming to understand it.
> 
> No need to be condescending, it doesn't help your cause.  I've done X 
> programming since the late 80:s.
> 
>> 
>>> To say more, in order to clear a character it would require no
>>> computation,
>>> but only a simply call of XClearWindow().
>>>
>>> Every window could have its own font.
>> 
>> So say we have 1400 windows each with a different font with a different
>> size? 
>>   How do you purpose we lay this out?  Instead of laying out characters
>> we
>> are 
>> now laying out windows.  Same problem, but with an enormous overhead.
>> 
>> Same problem, only 1 GC for every window.
> 
> No it is not the same problem.  For every window you will have to tell X
> where 
> it will be positioned, i,e. x and y and width and height.  If every window 
> have its own font with its own size you have to calculate all this for
> every 
> window, and the recalculate it for all windows when a font changes in one 
> window.  Sounds like what redisplay does today.
> 
>> 
>>> And to say more, an image of high dimension will be divided in many
>>> subwindows, and emacs will be able to display images normally, not as a
>>> huge
>>> glyph.
>> 
>> Again, making the display of an image to be so much slower, because
>> instead
>> of 
>> one (ideally), call to the X server, we now have one per window.  And
>> what a 
>> nightmare to figure out what part of an image that needs to be redrawn
>> and 
>> moved if the user changes the size of the Emacs frame...
>> 
>> We do have 1 call for window, and 1 network message to the server for
>> every
>> update.
> 
> ... times 14000 windows...
> 
>> 
>> The only problem in this implementation would be to compute the size of
>> every window, and to compute its coordinates.
> 
> So you have gained nothing really, redisplay is the hard part.
> 
> 	Jan D.
> 
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27564505.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[alin.s]

Jan Djärv wrote:
> 
>> If you make 1000 identical calls of Xlib functions, Xlib put them into
>> its
>> queue, and send them once.
>> 
> 
> Yes it does put calls in a queue.  All 1400 of them instead of one.  But
> given 
> how IPC works, in reality it will be many writes to the socket anyway,
> even if 
> the queue tries to minimize that.  And the amount of data is still 1400
> (or 
> 14000) times more.  X does not squeeze several calls into one, there are
> still 
> several calls.  And they are not identical anyway, they are for different
> windows.
> 
> 

You can keep on server side many little windows, and on the client side you
can keep the image of the screen, and the little windows notifies you what
part of the image was changed, you update the local bitmap, and when the
bitmap is created, you send it to the server with only 1 X lib call.

The idea is that you ask the server to tell you what must be refreshed on
the scree, and you need no more obfuscated internal redisplay.


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27564728.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Jan Djärv]

alin.s skrev:
> 
> 
> You can keep on server side many little windows, and on the client side you
> can keep the image of the screen, and the little windows notifies you what
> part of the image was changed, you update the local bitmap, and when the
> bitmap is created, you send it to the server with only 1 X lib call.
> 
> The idea is that you ask the server to tell you what must be refreshed on
> the scree, and you need no more obfuscated internal redisplay.
> 

This is done already, the expose event contains the rectangle that needs to be 
updated.  And if you need more exact information you can use the XDamage 
extension.

What you are describing is basically double buffering.  Since Gtk+ does this 
already, it makes sense for Emacs to go that way too, either by using more 
Gtk+ (we now turn double buffering off), or develop its own.

	Jan D.

Re: redisplay system of emacs

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> What you are describing is basically double buffering.  Since Gtk+
> does this already, it makes sense for Emacs to go that way too,

I don't see this dependency.

> either by using more Gtk+ (we now turn double buffering off),

In particular if one can tell Gtk+ not to double buffer.

> or develop its own.

I don't think that the performance in partly obscured windows is really
that important.

Or maybe I don't understand what this is about.

-- 
David Kastrup

Re: redisplay system of emacs

[Jan Djärv]

David Kastrup skrev:

> I don't think that the performance in partly obscured windows is really
> that important.
> 

No, it isn't.  But if you can get info about exactly what parts needs to be 
updated, maybe redisplay could do less work.  But the main point is that if 
you use double buffering (i.e. first render in an off-screen pixmap, and then 
copy the pixmap to the X window in one go), you can use a very crude 
redisplay, since you always copy from an up-to-date copy.  No need to worry 
about flickering.

	Jan D.

Re: redisplay system of emacs

[alin.s]

I did not want to state categorically that such or such solution is the good
one.

For example, if every emacs-window would be composed of lots of subwindows
on server side, that lots of sub-windows could be completely hidden. Never
use them to write something in them. They just report exactly the part of
the buffer that should be redisplayed. They only catch and report events.
They are invisible.

Redisplay_internal does lots and lots of computations to detect which part
of the buffer should be redisplayed at a given moment. All these
computations can be avoided using a system of subwindows (1 for every char).
That was my idea: to replace redisplay_internal with something simple and
efficient. Something simple and efficient is to keep in the emacs a vector
of little windows, and to register in the server all these windows.

The structure of windows is changed only when configure event is caught or
when one changes the font, or split windows, etc. So very rarely.

95% of the operations of redisplay can be covered with the system of little
windows registered in the server. No more need of redisplay_internal for X.





David Kastrup wrote:
> 
> Jan Djärv <jan.h.d@swipnet.se> writes:
> 
>> What you are describing is basically double buffering.  Since Gtk+
>> does this already, it makes sense for Emacs to go that way too,
> 
> I don't see this dependency.
> 
>> either by using more Gtk+ (we now turn double buffering off),
> 
> In particular if one can tell Gtk+ not to double buffer.
> 
>> or develop its own.
> 
> I don't think that the performance in partly obscured windows is really
> that important.
> 
> Or maybe I don't understand what this is about.
> 
> -- 
> David Kastrup
> 
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27566385.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[David Kastrup]

"alin.s" <alinsoar@voila.fr> writes:

> I did not want to state categorically that such or such solution is the good
> one.
>
> For example, if every emacs-window would be composed of lots of subwindows
> on server side, that lots of sub-windows could be completely hidden. Never
> use them to write something in them. They just report exactly the part of
> the buffer that should be redisplayed. They only catch and report events.
> They are invisible.
>
> Redisplay_internal does lots and lots of computations to detect which
> part of the buffer should be redisplayed at a given moment.

By checking which on-screen parts have changed how.

> All these computations can be avoided using a system of subwindows (1
> for every char).

That is nonsense.  One then has to check which subwindows need to be
moved/replaced as a result of buffer changes.  The X server has no clue
about buffers and the display matrix and its relative arrangement.

> That was my idea: to replace redisplay_internal with something simple
> and efficient. Something simple and efficient is to keep in the emacs
> a vector of little windows, and to register in the server all these
> windows.

And the server works by magic?  Unlikely.  Even if the scheme were not
unfeasible.

> The structure of windows is changed only when configure event is
> caught or when one changes the font, or split windows, etc. So very
> rarely.

Every insertion of a single character of a proportional font will change
the "windows" layout.

> 95% of the operations of redisplay can be covered with the system of
> little windows registered in the server. No more need of
> redisplay_internal for X.

I think you have the wrong idea about what redisplay_internal does, and
what an X server can do on its own.

-- 
David Kastrup

Re: redisplay system of emacs

[alin.s]

> The structure of windows is changed only when configure event is
> caught or when one changes the font, or split windows, etc. So very
> rarely.

Every insertion of a single character of a proportional font will change
the "windows" layout.


This is true, because not all characters have the same width. Anyways, the
idea of many subwindows was not good.


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27585994.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[alin.s]

David Kastrup wrote:
> 
> 
>> 95% of the operations of redisplay can be covered with the system of
>> little windows registered in the server. No more need of
>> redisplay_internal for X.
> 
> I think you have the wrong idea about what redisplay_internal does, and
> what an X server can do on its own.
> 
> 

I\m not interested at all about X and graphical programming. I ' ve read the
Xlib manual just to be able to finish the task of tabs for
GTK/Athena/LessTiff


-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27586106.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Davis Herring]

[Alin, I'm fully aware that you're not advocating this idea now.  I don't
mean to attack, but to explain something that wasn't said explicitly.]

> Redisplay_internal does lots and lots of computations to detect which part
> of the buffer should be redisplayed at a given moment. All these
> computations can be avoided using a system of subwindows (1 for every
> char).

Those computations are complicated because the buffer contents (or window
width, or scroll amount, or...) might have changed, not just because the
window might have been damaged in some complicated fashion.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: redisplay system of emacs

[grischka]

>> Redisplay_internal does lots and lots of computations to detect which part
>> of the buffer should be redisplayed at a given moment. All these
>> computations can be avoided using a system of subwindows (1 for every
>> char).
>
> Those computations are complicated because the buffer contents (or window
> width, or scroll amount, or...) might have changed, not just because the
> window might have been damaged in some complicated fashion.

It's not _that_complicated.  Basically it just draws the new screen to
memory, then compares it with the current screen and then updates the
differences.

And actually it works nicely and fast, just maybe slightly over-designed
in cases.  (For example changing "bazfoobar" to "baz,foobar" would output
",f" and "obar" and thus reuse the originally second "o" for the new first
one.  One might think that drawing one additional "o" could be faster than
two XDrawString, but so what).

Whether the code could be written to be easier to understand is a different
question but who has time for cleanups anyway.

--- grischka

Re: redisplay system of emacs

[Thien-Thi Nguyen]

() grischka <grishka@gmx.de>
() Tue, 16 Feb 2010 20:20:06 +0100

   but who has time for cleanups anyway.

Sometimes janitors and archeologists find the time.

thi

Re: redisplay system of emacs

[Eli Zaretskii]

> Date: Tue, 16 Feb 2010 20:20:06 +0100
> From: grischka <grishka@gmx.de>
> Cc: emacs-devel@gnu.org
> 
> >> Redisplay_internal does lots and lots of computations to detect which part
> >> of the buffer should be redisplayed at a given moment. All these
> >> computations can be avoided using a system of subwindows (1 for every
> >> char).
> >
> > Those computations are complicated because the buffer contents (or window
> > width, or scroll amount, or...) might have changed, not just because the
> > window might have been damaged in some complicated fashion.
> 
> It's not _that_complicated.  Basically it just draws the new screen to
> memory, then compares it with the current screen and then updates the
> differences.

That's only a small part of the story.  It draws only part of the
screen, not all of it (unless optimizations fail), and it goes to
great lengths to minimize that part of the screen.

The reason is that delivering to the screen is not the heaviest part
of the redisplay; it's that ``drawing to memory'' that is.

Re: redisplay system of emacs

[grischka]

Eli Zaretskii wrote:
>> It's not _that_complicated.  Basically it just draws the new screen to
>> memory, then compares it with the current screen and then updates the
>> differences.
> 
> That's only a small part of the story.  It draws only part of the
> screen, not all of it (unless optimizations fail), and it goes to
> great lengths to minimize that part of the screen.

Sure, thats what I wrote: It updates only the differences.

> The reason is that delivering to the screen is not the heaviest part
> of the redisplay; it's that ``drawing to memory'' that is.

That is probably true nowadays with fast graphical screens (say for PCs
after ~1995).  However the design of emacs' "redisplay" is based on the
contrary assumption that sending the updates to the terminal more expensive
than their calculation.  If you want to it is optimized for _very_ slow
terminals to the expense of code simplicity.  I assume it was working great
with 14400 baud connections, but then again emacs didn't have colors and
var-pitch fonts at that times, either.

--- grischka

Re: redisplay system of emacs

[Eli Zaretskii]

> Date: Tue, 16 Feb 2010 21:56:03 +0100
> From: grischka <grishka@gmx.de>
> CC: herring@lanl.gov, emacs-devel@gnu.org
> 
> Eli Zaretskii wrote:
> >> It's not _that_complicated.  Basically it just draws the new screen to
> >> memory, then compares it with the current screen and then updates the
> >> differences.
> > 
> > That's only a small part of the story.  It draws only part of the
> > screen, not all of it (unless optimizations fail), and it goes to
> > great lengths to minimize that part of the screen.
> 
> Sure, thats what I wrote: It updates only the differences.

The point is, it tries to limit the region where it looks for
differences to as small portion of the screen as possible, sometimes a
single line.  Therefore ``draws the new screen to memory'' is an
exception; the rule is ``draws a small portion of the new screen to
memory''.

> > The reason is that delivering to the screen is not the heaviest part
> > of the redisplay; it's that ``drawing to memory'' that is.
> 
> That is probably true nowadays with fast graphical screens (say for PCs
> after ~1995).  However the design of emacs' "redisplay" is based on the
> contrary assumption that sending the updates to the terminal more expensive
> than their calculation.

The current display code was designed in 1999.

Re: redisplay system of emacs

[Chong Yidong]

I'm not one to discourage people from hacking on whatever they feel
like, but IMO discussing a revamp of the Emacs redisplay system is a
waste of time.  Gerd Moellmann sweated blood getting the present system
up and running for Emacs 21, and it basically works well enough.  There
are lots and lots of areas in Emacs that (IMO) are in much more urgent
need of improvement.

If you have a hankering for making core code changes, I'd suggest
looking at the concurrency project or the GTK widget embedding project,
and not a radical overhaul of redisplay itself.  (Stefan and I will make
our goals for Emacs 24 more concrete when we open the trunk for Emacs 24
development.)

Re: redisplay system of emacs

[alin.s]

Thien-Thi Nguyen-6 wrote:
> 
> 
>    but who has time for cleanups anyway.
> 
> Sometimes janitors and archeologists find the time.
> 
> 

I hope some archaeologist who has good will to read my post.




-- 
View this message in context: http://old.nabble.com/redisplay-system-of-emacs-tp27349166p27623998.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

Re: redisplay system of emacs

[Giuseppe Scrivano]

Chong Yidong <cyd@stupidchicken.com> writes:

> If you have a hankering for making core code changes, I'd suggest
> looking at the concurrency project or the GTK widget embedding project,
> and not a radical overhaul of redisplay itself.  (Stefan and I will make
> our goals for Emacs 24 more concrete when we open the trunk for Emacs 24
> development.)

The emacs concurrency project, despite recently a not very active
development, is in a quite advanced development status.

Since last time we discussed about emacs-mt, many problems were fixed,
buffer-locking was removed and we have finally implemented per thread
let-bound buffer local variables.

I am a bit stuck now; would you mind to try emacs-mt and let us know
what we are missing before think (hopefully) about a merge?  It is not
well tested, and surely there are bugs waiting to be discovered.

The git repository is accessible here (g-exp branch):

  git://gitorious.org/emacs-mt/emacs-mt.git

Thanks,
Giuseppe

Re: redisplay system of emacs

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]

On Feb 22, 2010, at 7:45 PM, Giuseppe Scrivano wrote:
> 
> I am a bit stuck now; would you mind to try emacs-mt and let us know
> what we are missing before think (hopefully) about a merge?  It is not
> well tested, and surely there are bugs waiting to be discovered.
> 
> The git repository is accessible here (g-exp branch):
> 
>  git://gitorious.org/emacs-mt/emacs-mt.git


It doesn't compile on my machine (Mac OS X 10.6):

gcc -c  -Demacs -DHAVE_CONFIG_H  -I. -I/Users/dr/em23/src -Dtemacs     -g -O2 -Wdeclaration-after-statement -Wno-pointer-sign   -MMD -MF deps/dispnew.d dispnew.c
In file included from lisp.h:3690,
                 from dispnew.c:31:
thread.h:128: error: thread-local storage not supported for this target
thread.h:142: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'global_lock'
make[1]: *** [dispnew.o] Error 1
make: *** [src] Error 2

See also: http://lists.apple.com/archives/xcode-users/2006/Jun/msg00551.html
http://lists.apple.com/archives/xcode-users/2007/jun/msg00120.html

That link suggests that the feature you're using is related to ELF and thus not portable.

Thanks for your work on this - looking forward to seeing the results!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: redisplay system of emacs

[Tom Tromey]

>>>>> "David" == David Reitter <david.reitter@gmail.com> writes:

David> That link suggests that the feature you're using is related to
David> ELF and thus not portable.

Yeah, we currently use __thread and the POSIX thread API.  These can be
made portable by someone who has the know-how.

Tom

Re: redisplay system of emacs

[Richard Stallman]

I am impressed and glad.  Concurrency would enable us to solve the
problem of multi-terminal Emacs, by enabling each terminal to enter
the minibuffer in parallel.

Concurrency (was: redisplay system of emacs)

[Stefan Monnier]

> The emacs concurrency project, despite recently a not very active
> development, is in a quite advanced development status.
[...]
> The git repository is accessible here (g-exp branch):
>   git://gitorious.org/emacs-mt/emacs-mt.git

Could you install your code (assuming it does not have contributions
from people who haven't signed the relevant paperwork, of course) into
a new `concurrent' branch in the Bzr repository?

That would be very helpful,


        Stefan


PS: Obviously, you'll first have to convert it to Bzr and then you'll
probably need to rebase it on top of Emacs's Bzr trunk.

Re: Concurrency (was: redisplay system of emacs)

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> PS: Obviously, you'll first have to convert it to Bzr and then you'll
> probably need to rebase it on top of Emacs's Bzr trunk.

Most likely it's much easier the other way round.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Concurrency (was: redisplay system of emacs)

[Giuseppe Scrivano]

I have rebased the emacs-mt patches on the last bazaar revision; I had
to rebase the git repository as well to get a linear source history that
I could use easier with bazaar.

I uploaded the `concurrent' branch on fencepost in
~gscrivano/emacs/concurrent, can somebody please install it?  I will
also need write permissions to continue working on it.

As we moved to bazaar, the emacs git mirror is not used anymore by the
concurrent emacs project.

Cheers,
Giuseppe



Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The emacs concurrency project, despite recently a not very active
>> development, is in a quite advanced development status.
> [...]
>> The git repository is accessible here (g-exp branch):
>>   git://gitorious.org/emacs-mt/emacs-mt.git
>
> Could you install your code (assuming it does not have contributions
> from people who haven't signed the relevant paperwork, of course) into
> a new `concurrent' branch in the Bzr repository?
>
> That would be very helpful,
>
>
>         Stefan
>
>
> PS: Obviously, you'll first have to convert it to Bzr and then you'll
> probably need to rebase it on top of Emacs's Bzr trunk.

Re: Concurrency (was: redisplay system of emacs)

[Giuseppe Scrivano]

Hello,

Finally I have created a new branch for the concurrent emacs project, it
is accessible here:

  http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

Cheers,
Giuseppe

Re: Concurrency

[Ted Zlatanov]

On Thu, 25 Mar 2010 17:49:25 +0100 Giuseppe Scrivano <gscrivano@gnu.org> wrote: 

GS> Finally I have created a new branch for the concurrent emacs project, it
GS> is accessible here:

GS>   http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

I'd like to look at it.  Is there a quick-start guide that explains the
changes and how to get started?

Thanks
Ted

Re: Concurrency

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

GS> http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

Ted> I'd like to look at it.  Is there a quick-start guide that explains the
Ted> changes and how to get started?

Nope.  However, there aren't really many new Emacs primitives.

On my branch I had run-in-thread (and a convenience wrapper macro,
with-new-thread), and yield.  Giuseppe's branch was slightly different,
and I think he added some mutex support as well; I still haven't looked
at these.

I can explain the bulk of the C changes if you want to understand those.

Tom

Re: Concurrency

[Ted Zlatanov]

On Fri, 26 Mar 2010 13:37:51 -0600 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
GS> http://bzr.savannah.gnu.org/r/emacs/other-branches/concurrency/

Ted> I'd like to look at it.  Is there a quick-start guide that explains the
Ted> changes and how to get started?

Tom> Nope.  However, there aren't really many new Emacs primitives.

Tom> On my branch I had run-in-thread (and a convenience wrapper macro,
Tom> with-new-thread), and yield.  Giuseppe's branch was slightly different,
Tom> and I think he added some mutex support as well; I still haven't looked
Tom> at these.

Tom> I can explain the bulk of the C changes if you want to understand those.

I'd love to know what's in Giuseppe's branch from the perspective of a
user.  Specifically, I want to know how a rewrite of the Gnus summary
buffer entry and threading might work in the new system.  But that's a
big undertaking so some simple demos would be better:

- a demo that increments a shared variable up to 10 with two threads
  that fight over the lock

- a map/reduce demo of totaling a list of numbers with N threads

- retrieve some network data asynchronously using threads, not async
  processes

Those demos would necessarily show the new primitives in a practical
way.  You or Giuseppe can explain the internals as well, that would be
helpful, but demos are really useful.

Thanks
Ted

Re: Concurrency

[Stefan Monnier]

> I'd love to know what's in Giuseppe's branch from the perspective of a
> user.  Specifically, I want to know how a rewrite of the Gnus summary

Actually, we need both: we need to document the changes, as seen by an
Elisp programmer (fits in etc/NEWS), and we need to document the way it
works internally, the invariants upon which it relies and things
like that.


        Stefan

Re: Concurrency

[Tom Tromey]

Tom> Nope.  However, there aren't really many new Emacs primitives.

Here's a quick rundown of the new primitives.

(run-in-thread FUNC)
  Take a no-argument function as an argument.
  Makes a new thread and calls the function.

(with-new-thread FORM)
  A convenience macro that takes a form, wraps it in a lambda,
  and calls run-in-thread.

(yield)
  Yield control.
  Emacs has semi-cooperative threading.  Thread switches happen during
  I/O or by explicit yield.

(make-mutex)
  Make a new lock and return it

(mutex-lock MUTEX)
  Acquire a mutex.  If already held by this thread, returns.
  If the mutex is held by some other thread, blocks until it is
  available.

(mutex-unlock MUTEX)
  Release the lock.

There is also a new variable, minibuffer-mutex.  I think it is pretty
self-explanatory, but Giuseppe should probably explain it.

I don't have time to write demos or even proper documentation, I'm
afraid.

Giuseppe also has a patch of some kind to Gnus, but I didn't see it in
the tree, and I don't know exactly what it does.

Tom

Re: Concurrency

[Stefan Monnier]

> Here's a quick rundown of the new primitives.

> (run-in-thread FUNC)
>   Take a no-argument function as an argument.
>   Makes a new thread and calls the function.

> (with-new-thread FORM)
>   A convenience macro that takes a form, wraps it in a lambda,
>   and calls run-in-thread.

> (yield)
>   Yield control.
>   Emacs has semi-cooperative threading.  Thread switches happen during
>   I/O or by explicit yield.

> (make-mutex)
>   Make a new lock and return it

> (mutex-lock MUTEX)
>   Acquire a mutex.  If already held by this thread, returns.

I.e. it's a "recursive/reentrant mutex".

>   If the mutex is held by some other thread, blocks until it is
>   available.

> (mutex-unlock MUTEX)
>   Release the lock.

Sounds good.  Could someone complete the above with a description of how
let-bindings work (and/or how they interact with buffer-local bindings)?
Also, a list of bugs/problem/issues would be handy,


        Stefan


PS: And place this info in a file in the `concurrent' branch.

Re: Concurrency

[Davis Herring]

>> (mutex-lock MUTEX)
>>   Acquire a mutex.  If already held by this thread, returns.
>
> I.e. it's a "recursive/reentrant mutex".

Hmm -- is it fully recursive, where you must unlock it as many times as
you locked it?  (I don't much care for the semi-recursive kind where one
unlock is sufficient regardless of the number of lock operations...)

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Concurrency

[Giuseppe Scrivano]

"Davis Herring" <herring@lanl.gov> writes:

>>> (mutex-lock MUTEX)
>>>   Acquire a mutex.  If already held by this thread, returns.
>>
>> I.e. it's a "recursive/reentrant mutex".
>
> Hmm -- is it fully recursive, where you must unlock it as many times as
> you locked it?  (I don't much care for the semi-recursive kind where one
> unlock is sufficient regardless of the number of lock operations...)

No, it is sufficient only one unlock to release it.

Cheers,
Giuseppe

Re: Concurrency

[Giuseppe Scrivano]

Hello, sorry my late reply:

Tom Tromey <tromey@redhat.com> writes:

> There is also a new variable, minibuffer-mutex.  I think it is pretty
> self-explanatory, but Giuseppe should probably explain it.

It is a mutex that protects the mini-buffer from concurrent accesses.
Only one thread can use the minibuffer at a time.



> Giuseppe also has a patch of some kind to Gnus, but I didn't see it in
> the tree, and I don't know exactly what it does.

No, I don't have a patch, I have just experimented with "(run-in-thread
gnus)" to verify it loads correctly in a separate thread.


Also we have to mention the two new macros:

(defmacro with-new-thread (&rest body)) -- BODY is executed in a new
thread.

(defmacro with-critical-section (mutex &rest body)) BODY is executed
after MUTEX is locked and automatically released on exit.

Giuseppe

Re: Concurrency

[Tom Tromey]

Stefan> Could someone complete the above with a description of how
Stefan> let-bindings work (and/or how they interact with buffer-local
Stefan> bindings)?

A let binding is always thread-local.  This is true whether the variable
is buffer-local or not.

The view within a single thread is basically the same as how elisp works
today.

The interaction of let-bindings and buffer-local bindings is
complicated, but I think really no different from before.  At least,
that was our goal, of course there could be bugs.

Stefan> Also, a list of bugs/problem/issues would be handy,

* Documentation; NEWS, as you said, but also the lisp reference manual.

* The current code makes no attempt to portable, it relies on POSIX
  threads and the GNU __thread extension.  This is not too hard to fix.
  I think it is reasonably important to keep __thread on systems that
  support it, for performance.

* Suppose you have existing elisp that creates a process with a filter,
  and the filter changes a let-bound variable, and the "outer" elisp
  blocks in sit-for waiting for the filter to do its thing.  Nothing in
  the current code guarantees that the process filter will be run in the
  "correct" thread.

* There may be oddities with redisplay running in the "wrong" thread.

* Keyboard-local variables and let-bindings may not work together as
  expected.

I'm sure there are more, this is all I remember offhand.

Stefan> PS: And place this info in a file in the `concurrent' branch.

I hope someone else can do that.

Tom

Re: Concurrency

[Tom Tromey]

>>>>> "Davis" == Davis Herring <herring@lanl.gov> writes:

>>> (mutex-lock MUTEX)
>>> Acquire a mutex.  If already held by this thread, returns.
>> 
>> I.e. it's a "recursive/reentrant mutex".

Davis> Hmm -- is it fully recursive, where you must unlock it as many times as
Davis> you locked it?  (I don't much care for the semi-recursive kind where one
Davis> unlock is sufficient regardless of the number of lock operations...)

Yeah, it doesn't keep a count.  And in addition to this problem,
mutex-unlock has a bug where it can unlock any mutex, not just ones own
by the current thread.

Tom

Re: Concurrency

[Daniel Colascione]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/28/10 4:25 PM, Davis Herring wrote:
>>> (mutex-lock MUTEX)
>>>   Acquire a mutex.  If already held by this thread, returns.
>>
>> I.e. it's a "recursive/reentrant mutex".
> 
> Hmm -- is it fully recursive, where you must unlock it as many times as
> you locked it?  (I don't much care for the semi-recursive kind where one
> unlock is sufficient regardless of the number of lock operations...)
> 
> Davis
> 

It really shouldn't be recursive at all:

http://www.zaval.org/resources/library/butenhof1.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAkuvyJYACgkQ17c2LVA10VsB3QCdHyXXPK9JMkbnDYqslY/TnBBY
6rcAoMr8ZlIpRGutZKa9mDGVzABJJZ3G
=0TRS
-----END PGP SIGNATURE-----

Re: Concurrency

[Daniel Colascione]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/28/10 3:40 PM, Tom Tromey wrote:
> Tom> Nope.  However, there aren't really many new Emacs primitives.
> 
> Here's a quick rundown of the new primitives.

Thanks for summing these up.

> (run-in-thread FUNC)
> (with-new-thread FORM)
> (yield)
> (make-mutex)
> (mutex-lock MUTEX)
> (mutex-unlock MUTEX)

In writing a different cooperatively-multitasked system, I've found that
I needed a kind of wait queue ("condition variable" if you will). While
you can build one out of the above primitives, it'd be better to provide
it as a part of the standard library.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAkuvyTsACgkQ17c2LVA10VurjQCgj5oE4YOg7oEgf68pDxNoHD1k
siMAni1B7DZHzmcKlki81x1D0m9cSIlk
=d5/q
-----END PGP SIGNATURE-----

Re: Concurrency

[Stefan Monnier]

>>>> (mutex-lock MUTEX)
>>>> Acquire a mutex.  If already held by this thread, returns.
>>> I.e. it's a "recursive/reentrant mutex".
>> Hmm -- is it fully recursive, where you must unlock it as many times as
>> you locked it?  (I don't much care for the semi-recursive kind where one
>> unlock is sufficient regardless of the number of lock operations...)
> No, it is sufficient only one unlock to release it.

Huh?  That sounds like a bug,


        Stefan

Re: Concurrency

[Stefan Monnier]

> It really shouldn't be recursive at all:
> http://www.zaval.org/resources/library/butenhof1.html

I'd tend to agree, although I doubt it has much impact on the actual
problem of getting concurrency working reliably and usefully.  I.e. it
should be easy to change it later.


        Stefan

Re: Concurrency

[Tom Tromey]

Daniel> It really shouldn't be recursive at all:
Daniel> http://www.zaval.org/resources/library/butenhof1.html

My experience with Java is that recursive mutexes are pretty convenient,
and in practice don't cause any actual problems.  I think most of this
paper is cutely worded but basically wrong.

I'm sure in the end it won't matter though.  If you really want
non-recursive mutexes, somebody else can reinvent them in elisp.

Tom

Re: Concurrency

[Tom Tromey]

Daniel> In writing a different cooperatively-multitasked system, I've found that
Daniel> I needed a kind of wait queue ("condition variable" if you will). While
Daniel> you can build one out of the above primitives, it'd be better to provide
Daniel> it as a part of the standard library.

We purposely kept the API as minimal as possible.
E.g., there isn't even a user-visible thread object.

It is no trouble to add things, it just must be done thoughtfully, and
in particular with an eye toward not breaking the possibility of
preemptive threading.

On this particular point, I agree, condition variables will be required
sooner or later.

Tom

Re: Concurrency

[Giuseppe Scrivano]

I have changed how `make-mutex' works.  Now it accepts an optional
argument; if the argument is non nil then a recursive mutex is created,
by default it is nil.  By default a normal mutex (non recursive) is
created.

I have also changed mutex-unlock on a recursive mutex to match the
number of times it was mutex-lock'ed.  To release a recursive mutex it
must be unlocked as many times as it was locked.

I have added a mutexp function that I forgot to add before.

Giuseppe



Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>> (mutex-lock MUTEX)
>>>>> Acquire a mutex.  If already held by this thread, returns.
>>>> I.e. it's a "recursive/reentrant mutex".
>>> Hmm -- is it fully recursive, where you must unlock it as many times as
>>> you locked it?  (I don't much care for the semi-recursive kind where one
>>> unlock is sufficient regardless of the number of lock operations...)
>> No, it is sufficient only one unlock to release it.
>
> Huh?  That sounds like a bug,
>
>
>         Stefan

Re: Concurrency

[Tom Tromey]

Giuseppe> I have changed how `make-mutex' works.  Now it accepts an
Giuseppe> optional argument; if the argument is non nil then a recursive
Giuseppe> mutex is created, by default it is nil.  By default a normal
Giuseppe> mutex (non recursive) is created.

I think it would be better to have all mutexes be recursive, because it
is safer.

Also, I think mutex-unlock should throw some kind of error if the mutex
is owned by a different thread.  What do you think of that?

Tom

Re: Concurrency

[Stefan Monnier]

> Also, I think mutex-unlock should throw some kind of error if the mutex
> is owned by a different thread.  What do you think of that?

That doesn't sound useful.  There are perfectly valid ways to use mutexes
where the locker and the unlocker are not the same thread.


        Stefan

Re: Concurrency

[Ken Raeburn]

On Mar 28, 2010, at 17:17, Tom Tromey wrote:
> * The current code makes no attempt to portable, it relies on POSIX
>  threads and the GNU __thread extension.  This is not too hard to fix.
>  I think it is reasonably important to keep __thread on systems that
>  support it, for performance.

From what I understand, porting to Windows should be easy enough.  "A simple matter of coding."

Dealing with __thread shouldn't be hard if it's not used pervasively, so pthread_{get,set}specific calls can be used in its place.

> * Suppose you have existing elisp that creates a process with a filter,
>  and the filter changes a let-bound variable, and the "outer" elisp
>  blocks in sit-for waiting for the filter to do its thing.  Nothing in
>  the current code guarantees that the process filter will be run in the
>  "correct" thread.

That may be a reason to force the filter to run in the same thread that created it.  Perhaps process filters could be run at thread-switch opportunities like 'yield', too?  (And vice versa -- perhaps calls that permit the running of process filters should also permit thread switching?)

Sounds like this is coming along nicely!

Ken

Re: Concurrency

[Ken Raeburn]

On Mar 29, 2010, at 11:37, Tom Tromey wrote:
> I think it would be better to have all mutexes be recursive, because it
> is safer.

I can see some desire for recursive mutexes, but I lean towards Butenhof's view myself.  If all mutexes are recursive, I'd like a way to find out if I've already got a mutex locked, so I can manually detect when I've screwed up my locking model, even if someone else wants to use recursive mutexes in their code.

BTW, one extension I'd suggest is an optional timeout for mutex-lock: nil would mean block until the lock is acquired, a positive number would mean wait up to that long (e.g., pthread_mutex_timedlock), and a non-positive number would mean try to acquire the lock but don't wait.  I'm not sure offhand if failure to acquire the lock should be indicated by throwing an error or a return value; I'm leaning a little towards throwing an error.

If this really catches on, read/write locks will probably be wanted too, but they can be built on mutexes and condition variables initially, and moved into C code later if they're really helpful and performance-critical.

For debugging, I think I'd want a way to check whether a particular thread has locked a mutex.  Not a predicate someone might be tempted to use for non-debugging purposes, more like an assertion that throws an error when false.

> Also, I think mutex-unlock should throw some kind of error if the mutex
> is owned by a different thread.  What do you think of that?

Probably the best thing -- and likewise if the mutex isn't locked at all.

Ken

Re: Concurrency

[Ken Raeburn]

On Mar 29, 2010, at 12:16, Stefan Monnier wrote:
>> Also, I think mutex-unlock should throw some kind of error if the mutex
>> is owned by a different thread.  What do you think of that?
> 
> That doesn't sound useful.  There are perfectly valid ways to use mutexes
> where the locker and the unlocker are not the same thread.

True... but there are models where it would be a bug, plain and simple.
Maybe it should be an option to mutex-unlock?

Ken

Re: Concurrency

[Tom Tromey]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

Tom> * The current code makes no attempt to portable, it relies on POSIX
Tom> threads and the GNU __thread extension.  This is not too hard to
Tom> fix.  I think it is reasonably important to keep __thread on
Tom> systems that support it, for performance.

Ken> Dealing with __thread shouldn't be hard if it's not used
Ken> pervasively, so pthread_{get,set}specific calls can be used in its
Ken> place.

There is exactly one __thread variable.

Tom> * Suppose you have existing elisp that creates a process with a filter,
Tom> and the filter changes a let-bound variable, and the "outer" elisp
Tom> blocks in sit-for waiting for the filter to do its thing.  Nothing in
Tom> the current code guarantees that the process filter will be run in the
Tom> "correct" thread.

Ken> That may be a reason to force the filter to run in the same thread
Ken> that created it.

Yeah.  I was thinking perhaps process filters could have a thread
attribute, which would default to their creating thread.  Setting this
to nil could mean that the process filter could run in any thread.

Ken> Perhaps process filters could be run at thread-switch opportunities
Ken> like 'yield', too?  (And vice versa -- perhaps calls that permit
Ken> the running of process filters should also permit thread
Ken> switching?)

I think (but am not 100% sure) that this already happens -- IIUC, things
like sleep-for and sit-for end up in wait_reading_process_output, which
has a possible yield in it.

Tom

Re: Concurrency

[Tom Tromey]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

Tom> I think it would be better to have all mutexes be recursive, because it
Tom> is safer.

Ken> I can see some desire for recursive mutexes, but I lean towards
Ken> Butenhof's view myself.

I realized later I should probably expand on this.

The issue I have with non-recursive mutexes has to do with the nature of
Emacs itself.  First, there is a large body of existing elisp that we
don't want to break.  And, second, Emacs changes over time and it is
very common to write backward compatibility hacks and the like to make
elisp work with multiple versions.  Finally, Emacs is not and will never
be a high-performance environment.

I think these issues imply that we should pick the safe option always.

As a concrete example, consider the one existing user-visible mutex in
Emacs right now: the minibuffer mutex.  Lots of code, like yes-or-no-p,
acquires and releases this mutex.

But suppose you want to write a function that uses the minibuffer for a
multi-step operation, and you want to use yes-or-no-p (or something
else) as a subroutine.

If you have recursive mutexes, no problem, just acquire the lock.

If you don't have recursive mutexes, then all the primitives must be
duplicated into -lock and -nolock variants; or, the primitives should do
no locking and must be called with the lock held.

I'm not a fan of the first option, since it means an explosion of
functions.  And the second option is also bad, because it breaks
existing elisp.

(FWIW I am not sure a minibuffer mutex is the right thing.  It seems
like this should be at least per-terminal.)

Ken> If all mutexes are recursive, I'd like a
Ken> way to find out if I've already got a mutex locked, so I can
Ken> manually detect when I've screwed up my locking model, even if
Ken> someone else wants to use recursive mutexes in their code.

Ken> BTW, one extension I'd suggest is an optional timeout for
Ken> mutex-lock

These make sense to me.

Tom

Re: Concurrency

[Giuseppe Scrivano]

Tom Tromey <tromey@redhat.com> writes:

> Also, I think mutex-unlock should throw some kind of error if the mutex
> is owned by a different thread.  What do you think of that?

I think we should consider what will happen when the current cooperative
threads model will be replaced and threads will run freely.  In that
case probably we will want to use pthread mutexes, not our
implementation.  Any invariant we introduce now, must be followed later
as well or we will end to break code.  Pthread specifications say for
default mutex'es that the behaviour is undefined when you attempt to
unlock a mutex owned by another thread, instead an error is returned
when a recursive mutex is used.  So I think we can rely on this error
and introduce it only if we are not going to use default mutex'es,
unless later we want to implement them on the top of recursive ones.

What do you think?

Cheers,
Giuseppe

Re: Concurrency

[Stefan Monnier]

Tom> * Suppose you have existing elisp that creates a process with a filter,
Tom> and the filter changes a let-bound variable, and the "outer" elisp
Tom> blocks in sit-for waiting for the filter to do its thing.  Nothing in
Tom> the current code guarantees that the process filter will be run in the
Tom> "correct" thread.
Ken> That may be a reason to force the filter to run in the same thread
Ken> that created it.
> Yeah.  I was thinking perhaps process filters could have a thread
> attribute, which would default to their creating thread.  Setting this
> to nil could mean that the process filter could run in any thread.

Or maybe accept-process-output should cause the filters to be run in the
waiting thread (especially when the `proc' argument is provided).


        Stefan

Re: Concurrency

[Stefan Monnier]

>>> Also, I think mutex-unlock should throw some kind of error if the mutex
>>> is owned by a different thread.  What do you think of that?
>> That doesn't sound useful.  There are perfectly valid ways to use mutexes
>> where the locker and the unlocker are not the same thread.
> True... but there are models where it would be a bug, plain and simple.

Bugs happen.  It's only a problem if they happen often enough to warrant
the pain of imposing additional constraints.  I'd expect most uses of
"same-thread unlock" to be covered by the scoped with-mutex
construct anyway.

> Maybe it should be an option to mutex-unlock?

Let's not add complexity until it's clear that it's needed.


        Stefan

Re: Concurrency

[Stefan Monnier]

> (FWIW I am not sure a minibuffer mutex is the right thing.  It seems
> like this should be at least per-terminal.)

I think the minibuffer-mutex variable makes a lot of sense, but yes, it
will have to be terminal-local, but that can be changed later once the
rest of the code is ready for it.


        Stefan

Re: Concurrency

[Tom Tromey]

Stefan> Actually, we need both: we need to document the changes, as seen
Stefan> by an Elisp programmer (fits in etc/NEWS), and we need to
Stefan> document the way it works internally, the invariants upon which
Stefan> it relies and things like that.

I can supply some of the raw data here, if not the commits.

We added a few new lisp objects:

* thread_state.  This represents a single thread.  Before dumping, only
  one thread may exist.

  Some C variables which were formerly global were moved into this
  struct, and replaced with a redirecting #define.  E.g.:

  /* Pointer to beginning of specpdl.  */
  struct specbinding *m_specpdl;
#define specpdl (current_thread->m_specpdl)

  Exactly which things needed to be in this structure was determined in
  an ad hoc way; mostly by looking at `nm' to see what globals were
  interesting.

* Lisp_Mutex.  A mutex, should be obvious.

* Lisp_ThreadLocal.  This is used to represent a thread-local binding.
  It has a slot for the global value, and an alist mapping threads to
  a thread-local slot.  If a thread is not in the alist, the global
  binding is used.

  A Lisp_ThreadLocal can appear in a number of places, e.g. in a
  variable binding, in a Lisp_Object global variable, etc.

  We rewrote a lot of the C code to automatically indirect through a
  Lisp_ThreadLocal, when found.  There is a new header, globals.h, which
  has many lines of the form:

#define Vafter_change_functions *find_variable_location (&impl_Vafter_change_functions)

  A couple things about this line:

  * The expansion is chosen so that both rvalue and lvalue uses work.
  * The underlying global is renamed to catch any errors.

There are other changes (in the GC, in the binding code) but I think
those are relatively self-explanatory in the code... at least, hopefully
as much as the code was before.

Tom

Re: Concurrency

[Stefan Monnier]

>> Also, I think mutex-unlock should throw some kind of error if the mutex
>> is owned by a different thread.  What do you think of that?
> I think we should consider what will happen when the current cooperative
> threads model will be replaced and threads will run freely.  In that
> case probably we will want to use pthread mutexes, not our
> implementation.

That's a good point.


        Stefan "who really wishes we used higher-level abstractions
                than locks, such as atomic-regions"

gsoc for concurrent Emacs? (was: Concurrency)

[Ted Zlatanov]

On Sun, 28 Mar 2010 15:17:49 -0600 Tom Tromey <tromey@redhat.com> wrote: 

Tom> * Documentation; NEWS, as you said, but also the lisp reference manual.
...
Stefan> PS: And place this info in a file in the `concurrent' branch.

Tom> I hope someone else can do that.

Just a suggestion:

I think the concurrent Emacs project is nicely standalone and would make
a good GSoC project (see
http://www.gnu.org/software/soc-projects/ideas-2010.html, where Guile
integration is already mentioned).  The concurrent Emacs branch
obviously needs some work, especially with documentation, but is
probably at a level suitable for an interested CS/CE student.

I'm not the one to propose this but if Giuseppe or Tom want, they can
e-mail summer-of-code@gnu.org as the URL above suggests.  GSoC requires
a mentor so it's not completely hands-off...

Ted

Re: gsoc for concurrent Emacs? (was: Concurrency)

[Giuseppe Scrivano]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think the concurrent Emacs project is nicely standalone and would make
> a good GSoC project (see
> http://www.gnu.org/software/soc-projects/ideas-2010.html, where Guile
> integration is already mentioned).  The concurrent Emacs branch
> obviously needs some work, especially with documentation, but is
> probably at a level suitable for an interested CS/CE student.

I am not sure there are enough tasks at this point that a GSoC project
is needed for the cooperative model we are using and I am afraid it is
too early yet to think about a new one.  AFAICS, the main activity to
get the concurrent model working is bug-fixing.



> I'm not the one to propose this but if Giuseppe or Tom want, they can
> e-mail summer-of-code@gnu.org as the URL above suggests.  GSoC requires
> a mentor so it's not completely hands-off...

If the Emacs maintainers think it is worthwhile, I am available for
mentoring.


Cheers,
Giuseppe