Can we go GTK-only?

Can we go GTK-only?

[Daniel Colascione]

I've been doing some work on xterm.c and friends lately --- it's a very 
complex subsystem made even more complex by supporting five different 
toolkits in the same piece of code. GTK support adds a special twist: 
GTK is nothing like an old-fashioned Xt-style toolkit, and trying to 
force it into the mold requires going behind GTK's back in ways that are 
fragile, inefficient, and unsupported. New versions of GTK regularly 
break Emacs. Emacs cannot take advantage of GTK's support for non-X 
targets, like Wayland.

Do we really need all the legacy infrastructure in xterm? Do many people 
really build Emacs as a Motif application? The X window subsystem would 
be made much simpler if we made it GTK3-only.

If we did, much of the complexity of the existing code would evaporate, 
since we'd be using GTK's infrastructure instead of our own. We'd turn 
Emacs frames into well-behaved GTK widgets --- just run redisplay inside 
the Emacs-widget's draw method, draw onto the Cairo surface GTK 
provides. Instead of flushing frames and drawing to the screen directly 
from input-waiting routines, we'd rely exclusively on dirtying the Emacs 
widget and letting the GTK's normal Expose-event processing take care of 
invoking redisplay.

We'd have to get rid of the SIGIO input subsystem too, but I think 
that's a good thing.

The size of the xterm codebase would shrink by at least two thirds, I 
think. We already _have_ Cairo support: sure, it's broken, but the part 
that's broken is integrating with the horrible Xt hacks, not the drawing 
itself.

As a stepping stone, we should at least remove support for the 
no-toolkit, Motif, and Athena configurations. (Lucid I can understand.)

Re: Can we go GTK-only?

[Frank Haun]

On Thu, 27 Oct 2016 12:54:39 -0700, Daniel Colascione wrote:

> Do many people really build Emacs as a Motif application?

I knew a lot of people using:

  --with-x-toolkit=lucid

Me too. Better performance on slow hardware and no problems with Emacs
as daemon.

Frank

Re: Can we go GTK-only?

[Daniel Colascione]

On October 27, 2016 1:05:50 PM PDT, Frank Haun <fh@fhaun.de> wrote:
>On Thu, 27 Oct 2016 12:54:39 -0700, Daniel Colascione wrote:
>
>> Do many people really build Emacs as a Motif application?
>
>I knew a lot of people using:
>
>  --with-x-toolkit=lucid
>
>Me too. Better performance on slow hardware and no problems with Emacs
>as daemon.

I'm not proposing that we get rid of the lucid X support, but I agree with Paul that we should rip GTK out of the X subsystem and make it a peer of NS and Windows.

>
>Frank

Re: Can we go GTK-only?

[Frank Haun]

On Thu, 27 Oct 2016 13:45:23 -0700, Daniel Colascione wrote:

> On October 27, 2016 1:05:50 PM PDT, Frank Haun <fh@fhaun.de> wrote:
>>On Thu, 27 Oct 2016 12:54:39 -0700, Daniel Colascione wrote:
>>
>>> Do many people really build Emacs as a Motif application?
>>
>>I knew a lot of people using:
>>
>>  --with-x-toolkit=lucid
>>
>>Me too. Better performance on slow hardware and no problems with Emacs
>>as daemon.
>
> I'm not proposing that we get rid of the lucid X support, but I agree
> with Paul that we should rip GTK out of the X subsystem and make it a
> peer of NS and Windows.

Ah ok, thanks for clarification.

Frank

Re: Can we go GTK-only?

[Paul Eggert]

On 10/27/2016 12:54 PM, Daniel Colascione wrote:
> As a stepping stone, we should at least remove support for the 
> no-toolkit, Motif, and Athena configurations. (Lucid I can understand.)

How much would this stepping stone save us? Few people need or use those 
three configurations, so if deleting them simplifies Emacs maintenance 
considerably, it might be a win.

Alternatively, how about if we change the GTK port to be more like the 
NS port? That is, we leave xterm.o, xfns.c, xselect.c etc. alone (except 
for removing the GTK-related parts), and have new source files 
gtkterm.c, gtkfns.c, gtkselect.c etc. as needed to support GTK. The GTK 
port would no longer define HAVE_X_WINDOWS. That way, the no-toolkit, 
Motif, and Athena configurations would still work, without getting in 
the way of GTK.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Thu, 27 Oct 2016 13:32:14 -0700 Paul Eggert <eggert@cs.ucla.edu>
wrote:
> Alternatively, how about if we change the GTK port to be more like
> the NS port? That is, we leave xterm.o, xfns.c, xselect.c etc.
> alone (except for removing the GTK-related parts), and have new
> source files gtkterm.c, gtkfns.c, gtkselect.c etc. as needed to
> support GTK. The GTK port would no longer define HAVE_X_WINDOWS.
> That way, the no-toolkit, Motif, and Athena configurations would
> still work, without getting in the way of GTK.

This seems like a good idea. Emacs already has multiple front ends.
Having a distinct GTK front end seems like a good, clean sort of
segregation. The legacy code can be left in and alone for now for the
benefit of those who use it.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Thu, 27 Oct 2016 19:15:02 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Daniel Colascione <dancol@dancol.org>,
> 	Emacs developers <emacs-devel@gnu.org>
> 
> On Thu, 27 Oct 2016 13:32:14 -0700 Paul Eggert <eggert@cs.ucla.edu>
> wrote:
> > Alternatively, how about if we change the GTK port to be more like
> > the NS port? That is, we leave xterm.o, xfns.c, xselect.c etc.
> > alone (except for removing the GTK-related parts), and have new
> > source files gtkterm.c, gtkfns.c, gtkselect.c etc. as needed to
> > support GTK. The GTK port would no longer define HAVE_X_WINDOWS.
> > That way, the no-toolkit, Motif, and Athena configurations would
> > still work, without getting in the way of GTK.
> 
> This seems like a good idea. Emacs already has multiple front ends.
> Having a distinct GTK front end seems like a good, clean sort of
> segregation. The legacy code can be left in and alone for now for the
> benefit of those who use it.

It's not as trivial as it sounds, because our experience shows that
such a separation causes duplication of code.  Minimizing code
duplication makes the job harder.

Re: Can we go GTK-only?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Do we really need all the legacy infrastructure in xterm? Do many people 
  > really build Emacs as a Motif application? The X window subsystem would 
  > be made much simpler if we made it GTK3-only.

Some of those alternatives can be deleted, but we shouldn't rush to
delete them all just because that would simplify things.

How about writing GTK support separate from the X support?
That would achieve the same goal.  Maybe someday we would delete the
other platforms.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 27 Oct 2016 12:54:39 -0700
> 
> As a stepping stone, we should at least remove support for the 
> no-toolkit, Motif, and Athena configurations. (Lucid I can understand.)

I would object to removing the no-toolkit code, because it can be used
to quickly and simply adapt Emacs to any new toolkit.

I don't know how wide Motif and Athena are used; IMO we shouldn't
delete them if they are still in use.  E.g., are the major Linux
distros still offering such packages?

It's okay to make GTK a separate back-end, especially if that makes
maintenance easier (and I agree that it most probably will).  But I'd
like to warn people here to put too much trust into GTK as our only
terminal back-end, because, like any Free Software package, it can
easily stagnate and die at some point.  Having Emacs build-able with
alternative toolkits makes us more future-proof, and also prevents us
from the danger of writing low-level code that assumes only GTK is
used.

Re: Can we go GTK-only?

[Ulrich Mueller]

>>>>> On Fri, 28 Oct 2016, Eli Zaretskii wrote:

> I don't know how wide Motif and Athena are used; IMO we shouldn't
> delete them if they are still in use.  E.g., are the major Linux
> distros still offering such packages?

Gentoo offers both, and looking at metadata in bugs reported to us, it
appears that both of them are used by a significant fraction of users
(in spite of GTK+ being the build option preferred by our ebuilds).

Also, GTK+ (in version 2 or 3) still crashes for multi-tty Emacs,
due to bug 85715 <https://bugzilla.gnome.org/show_bug.cgi?id=85715>.
Given that this has been reported 14 years ago, my guess is that it
will never be fixed.

Has anyone tried porting Emacs to Qt? :)

Ulrich

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Fri, 28 Oct 2016 09:27:01 +0200
> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org
> From: Ulrich Mueller <ulm@gentoo.org>
> 
> >>>>> On Fri, 28 Oct 2016, Eli Zaretskii wrote:
> 
> > I don't know how wide Motif and Athena are used; IMO we shouldn't
> > delete them if they are still in use.  E.g., are the major Linux
> > distros still offering such packages?
> 
> Gentoo offers both, and looking at metadata in bugs reported to us, it
> appears that both of them are used by a significant fraction of users
> (in spite of GTK+ being the build option preferred by our ebuilds).

Thanks for the data points.

> Also, GTK+ (in version 2 or 3) still crashes for multi-tty Emacs,
> due to bug 85715 <https://bugzilla.gnome.org/show_bug.cgi?id=85715>.
> Given that this has been reported 14 years ago, my guess is that it
> will never be fixed.

Yes, that's one reason why I don't think we should put all of our eggs
in the GTK+ basket.

> Has anyone tried porting Emacs to Qt? :)

We talked about it ;-)

Re: Can we go GTK-only?

[Frank Haun]

On Fri, 28 Oct 2016 09:22:04 +0300, Eli Zaretskii wrote:

> I don't know how wide Motif and Athena are used; IMO we shouldn't
> delete them if they are still in use.  E.g., are the major Linux
> distros still offering such packages?

Debian does.

Frank

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Frank Haun <fh@fhaun.de>
> Date: 28 Oct 2016 10:48:03 -0000
> 
> On Fri, 28 Oct 2016 09:22:04 +0300, Eli Zaretskii wrote:
> 
> > I don't know how wide Motif and Athena are used; IMO we shouldn't
> > delete them if they are still in use.  E.g., are the major Linux
> > distros still offering such packages?
> 
> Debian does.

Thanks.

Re: Can we go GTK-only?

[Stefan Monnier]

> As a stepping stone, we should at least remove support for the no-toolkit,
> Motif, and Athena configurations. (Lucid I can understand.)

One of the X11 non-Gtk build options needs to be kept, because of Gtk's
failure to support multiple X servers
(https://bugzilla.gnome.org/show_bug.cgi?id=85715).

I think of all the non-Gtk X11 builds, the Lucid build is by far the
most popular, so I'd vote to keep that one.

As for splitting Gtk support into its own backend separate from the
x*.c ones.  It sounds like a good idea, but I wish it were an occasion
to do at least one of two things:
- *start* adding support for "multi backend" support (i.e. being able to
  build Emacs such that it can have simultaneously an ns-frame,
  a Gtk-frame, and a Lucid-frame (and maybe even a w32-frame via Wine ;-)).
- reduce the amount of code duplication between the various backends
  (ns/w32/gtk/x/...).


        Stefan

Re: Can we go GTK-only?

[Ken Raeburn]

On Oct 28, 2016, at 09:35, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> As for splitting Gtk support into its own backend separate from the
> x*.c ones.  It sounds like a good idea, but I wish it were an occasion
> to do at least one of two things:
> - *start* adding support for "multi backend" support (i.e. being able to
>  build Emacs such that it can have simultaneously an ns-frame,
>  a Gtk-frame, and a Lucid-frame (and maybe even a w32-frame via Wine ;-)).

Yes, please!  I’ve often wanted to have NS and X11 windows open in one process.  Though where at least some of these libraries want to own the top-level event processing loop, it could get interesting.

I wonder if we should let them run their event loops, in their own threads, and separate that from the non-GUI events like subprocesses and timers, and from Lisp evaluation, as much as possible.  For that matter, some of the subprocess handling could probably use helper threads, like for TLS encryption and decryption work.

> - reduce the amount of code duplication between the various backends
>  (ns/w32/gtk/x/…).

…and get away from the “everything emulates an X toolkit” approach.

Ken

Re: Can we go GTK-only?

[Stefan Monnier]

> I wonder if we should let them run their event loops, in their own threads,
> and separate that from the non-GUI events like subprocesses and timers, and
> from Lisp evaluation, as much as possible.  For that matter, some of the
> subprocess handling could probably use helper threads, like for TLS
> encryption and decryption work.

Ideally, yes, I definitely think we should split the Emacs process into
various threads:
- some threads for the GUI.
- one thread for the Elisp engine (which receives events from the GUI
  threads among others).

So the GUI thread would handle the expose events, read-locking the
matrices, while the redisplay would run in another thread, write-locking
those same matrices.


        Stefan

Re: Can we go GTK-only?

[Daniel Colascione]

On October 30, 2016 2:42:18 PM PDT, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> I wonder if we should let them run their event loops, in their own
>threads,
>> and separate that from the non-GUI events like subprocesses and
>timers, and
>> from Lisp evaluation, as much as possible.  For that matter, some of
>the
>> subprocess handling could probably use helper threads, like for TLS
>> encryption and decryption work.
>
>Ideally, yes, I definitely think we should split the Emacs process into
>various threads:
>- some threads for the GUI.
>- one thread for the Elisp engine (which receives events from the GUI
>  threads among others).
>
>So the GUI thread would handle the expose events, read-locking the
>matrices, while the redisplay would run in another thread,
>write-locking
>those same matrices.

Agreed. That's basically how IntelliJ works too. We can do even better too: there's no reason these parts need to run in the same process it even the same machine. 

But this scheme won't really eliminate the coupling between redisplay and lisp though, since the former calls into the latter. The redisplay thread has to block waiting on lisp anyway.

Anyway, I strongly encourage you to look at the React Native rendering model. It's the most elegant way I've seen of constructing GUIs in general.

The key insight there is that we shouldn't have redisplay *lock* the display matrix and render it. The lisp universe should send a *copy* of the matrix set, then go about its business. This way, redisplay can go display that copy and everything is decoupled. You turn the system into an Erlang like message passing environment.

.

Re: Can we go GTK-only?

[Stefan Monnier]

> But this scheme won't really eliminate the coupling between redisplay and
> lisp though, since the former calls into the latter.  The redisplay thread
> has to block waiting on lisp anyway.

Indeed.  Moving redisplay to a separate thread would be neat, but it'd
take a much more significant amount of work, I think.

> The key insight there is that we shouldn't have redisplay *lock* the display
> matrix and render it.  The lisp universe should send a *copy* of the matrix
> set, then go about its business.

Good, point we can decouple the two a bit more this way.  Actually, we
already have this copy since we have 2 sets of matrices.  So we just
need to say that the "current matrix" belongs to the GUI thread/process,
while the "desired matrix" belongs to the redisplay.  And the "send
a copy" is done by comparing the two, so only the part that changed is
sent (and so the GUI thread gets to know which part needs to be redrawn).

Of course, comparing the two sets of matrices requires access to both,
so either we need the two threads to belong to the same process and we
temporarily use locking (but this locking is not used when responding
to expose events, it's only used to update the current matrices), or we
need a 3rd copy.


        Stefan

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 30 Oct 2016 19:57:31 -0400
> Cc: Ken Raeburn <raeburn@raeburn.org>, emacs-devel@gnu.org
> 
> > The key insight there is that we shouldn't have redisplay *lock* the display
> > matrix and render it.  The lisp universe should send a *copy* of the matrix
> > set, then go about its business.
> 
> Good, point we can decouple the two a bit more this way.  Actually, we
> already have this copy since we have 2 sets of matrices.  So we just
> need to say that the "current matrix" belongs to the GUI thread/process,
> while the "desired matrix" belongs to the redisplay.  And the "send
> a copy" is done by comparing the two, so only the part that changed is
> sent (and so the GUI thread gets to know which part needs to be redrawn).
> 
> Of course, comparing the two sets of matrices requires access to both,

What's more, some redisplay optimizations use the current matrices
without constructing the desired ones.

But since the matrices are not exposed to Lisp, I don't understand why
you think there's any problem to be solved here.

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 30 Oct 2016 19:57:31 -0400
> Cc: Ken Raeburn <raeburn@raeburn.org>, emacs-devel@gnu.org
> 
> Good, point we can decouple the two a bit more this way.  Actually, we
> already have this copy since we have 2 sets of matrices.  So we just
> need to say that the "current matrix" belongs to the GUI thread/process,
> while the "desired matrix" belongs to the redisplay.  And the "send
> a copy" is done by comparing the two, so only the part that changed is
> sent (and so the GUI thread gets to know which part needs to be redrawn).

This will need some very careful design, because currently the data
structures we pass the terminal-specific backend is intimately tied up
to information known only to the display engine, like the faces
(identified just by their numbers), the font and font driver to use,
the window and the frame where to draw the glyphs, etc.  Dividing this
between 2 threads, let alone processes, is not easy.

Re: Can we go GTK-only?

[YAMAMOTO Mitsuharu]

>>>>> On Sun, 30 Oct 2016 15:49:02 -0700, Daniel Colascione <dancol@dancol.org> said:

>> Ideally, yes, I definitely think we should split the Emacs process into
>> various threads:
>> - some threads for the GUI.
>> - one thread for the Elisp engine (which receives events from the GUI
>> threads among others).
>> 
>> So the GUI thread would handle the expose events, read-locking the
>> matrices, while the redisplay would run in another thread,
>> write-locking
>> those same matrices.

> Agreed. That's basically how IntelliJ works too. We can do even better too: there's no reason these parts need to run in the same process it even the same machine. 

Cocoa (maybe also for GNUstep?) has a restriction that GUI events must
be processed in the main thread.  Probably the Lisp thread also wants
to be main, especially when we start it by a tty-only session and then
add a GUI session via multi-tty.  So we have to separate processes for
GUI front-end and Lisp back-end to support such situations, anyway.

If we can separate them cleanly, then it might be possible to create a
front-end for iPad that communicates with a Lisp back-end running in
another machine.  It would also work around the GTK+ problem about
multi-tty mentioned in this "thread".

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Can we go GTK-only?

[Ken Raeburn]

> Cocoa (maybe also for GNUstep?) has a restriction that GUI events must
> be processed in the main thread.  Probably the Lisp thread also wants
> to be main, especially when we start it by a tty-only session and then
> add a GUI session via multi-tty.

Nothing comes to my mind that requires that the Lisp thread be the initial thread; I would think we could start a new thread for the main Lisp interpreter.  (At least in the X11 build, that should be fairly easily testable, too — just move *everything* we do from the initial thread to a new thread, and have the initial thread just wait around doing nothing.)  But if there are toolkits or libraries besides Cocoa that also insist on running in the main thread, they may not be able to coexist in use in one process, unless we manage the main event loop ourselves, or tell one GUI event handler how to recognize when another has activity that we need to mediate some callbacks for.

>  So we have to separate processes for
> GUI front-end and Lisp back-end to support such situations, anyway.

It might not be a bad way to go, if we can keep the interprocess communication efficient enough, but I don’t think it’ll be required.  Even if we do that, some part of the main Emacs process needs to communicate with these multiple GUI processes, so do we wind up with multiple UI threads in the main process anyway?

Ken

Re: Can we go GTK-only?

[Perry E. Metzger]

On Mon, 31 Oct 2016 04:24:31 -0400 Ken Raeburn <raeburn@raeburn.org>
wrote:
> >  So we have to separate processes for
> > GUI front-end and Lisp back-end to support such situations,
> > anyway.  
> 
> It might not be a bad way to go, if we can keep the interprocess
> communication efficient enough, but I don’t think it’ll be
> required.  Even if we do that, some part of the main Emacs process
> needs to communicate with these multiple GUI processes, so do we
> wind up with multiple UI threads in the main process anyway?

Or keep everything event driven. Or both.

BTW, if this gets done, documenting the protocol between the
front and back ends would be really nice for future porters.

One could also imagine adding quite different front ends, say one
based on Electron, which would allow very nice web integration.

A good place to prototype this would be a tty front end, since
the tty has a comparatively simple job.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[YAMAMOTO Mitsuharu]

>>>>> On Mon, 31 Oct 2016 04:24:31 -0400, Ken Raeburn <raeburn@raeburn.org> said:

>> Cocoa (maybe also for GNUstep?) has a restriction that GUI events
>> must be processed in the main thread.  Probably the Lisp thread
>> also wants to be main, especially when we start it by a tty-only
>> session and then add a GUI session via multi-tty.

> Nothing comes to my mind that requires that the Lisp thread be the
> initial thread; I would think we could start a new thread for the
> main Lisp interpreter.  (At least in the X11 build, that should be
> fairly easily testable, too — just move *everything* we do from the
> initial thread to a new thread, and have the initial thread just
> wait around doing nothing.)

Indeed.  Then this is not a serious restriction for attaching a Cocoa
GUI terminal with multi-tty.  Detaching is easier if the GUI process
is separated (as far as I know, the only way for a Cocoa GUI
application to disconnect from the window server or Dock on macOS is
to terminate the application process), but this has nothing to do with
the "main thread restriction".

>> So we have to separate processes for GUI front-end and Lisp
>> back-end to support such situations, anyway.

> It might not be a bad way to go, if we can keep the interprocess
> communication efficient enough, but I don’t think it’ll be required.
> Even if we do that, some part of the main Emacs process needs to
> communicate with these multiple GUI processes, so do we wind up with
> multiple UI threads in the main process anyway?

You don't need multiple UI threads.  The overall shape wouldn't be
much different from the current one for communication with multiple
X11 servers.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Sun, 30 Oct 2016 15:49:02 -0700
> Cc: emacs-devel@gnu.org
> 
> The key insight there is that we shouldn't have redisplay *lock* the display matrix and render it. The lisp universe should send a *copy* of the matrix set, then go about its business. This way, redisplay can go display that copy and everything is decoupled. You turn the system into an Erlang like message passing environment.

The glyph matrices are not exposed to Lisp, so there's no issue here
to begin with.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Mon, 31 Oct 2016 05:33:23 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Sun, 30 Oct 2016 15:49:02 -0700
> > Cc: emacs-devel@gnu.org
> > 
> > The key insight there is that we shouldn't have redisplay *lock*
> > the display matrix and render it. The lisp universe should send a
> > *copy* of the matrix set, then go about its business. This way,
> > redisplay can go display that copy and everything is decoupled.
> > You turn the system into an Erlang like message passing
> > environment.  
> 
> The glyph matrices are not exposed to Lisp, so there's no issue here
> to begin with.

BTW, a +1 on moving concurrency in general towards an Erlang style
model. It's much easier to program against. Yes, I know this is only
being discussed for display management but it could be used for much
more.

(And that reminds me: on top of the rest of the list of things that
lock up all of emacs unpleasantly which would be nice to have
concurrency for: checking for package updates in a package manager.)

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Sun, 30 Oct 2016 15:49:02 -0700
> Cc: emacs-devel@gnu.org
> 
> Agreed. That's basically how IntelliJ works too. We can do even better too: 
> there's no reason these parts need to run in the same process it even the same 
> machine. 

I think we should be careful when comparing Emacs with systems that
have a different basic design.  In Emacs, almost everything is driven
by changes in buffer text and the data structures that accompany
buffer text, and the Lisp interpreter is inherent in many
display-related features.

For example, this:

> But this scheme won't really eliminate the coupling between redisplay and lisp 
> though, since the former calls into the latter. The redisplay thread has to 
> block waiting on lisp anyway.

is a major issue that those other systems might not have.

> Anyway, I strongly encourage you to look at the React Native rendering model. 
> It's the most elegant way I've seen of constructing GUIs in general.

Where's that model described in enough technical detail to be able to
compare it with what we do in Emacs?

> The key insight there is that we shouldn't have redisplay *lock* the display 
> matrix and render it. The lisp universe should send a *copy* of the matrix set, 
> then go about its business. This way, redisplay can go display that copy and 
> everything is decoupled. You turn the system into an Erlang like message 
> passing environment.

There's probably some confusion or misunderstanding here: the Lisp
code in Emacs has no access to the glyph matrices, so it does not (and
cannot) "send the display matrix" to the display engine.  Thus, the
fact that the display engine can lock the glyph matrices is due to its
being their sole writer and reader.  The communication between the
Lisp universe and the display engine is solely via the buffer text and
the auxiliary data structures (text properties, overlays, etc.).

Re: Can we go GTK-only?

[Daniel Colascione]

On 10/31/2016 08:56 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Sun, 30 Oct 2016 15:49:02 -0700
>> Cc: emacs-devel@gnu.org
>>
>> Agreed. That's basically how IntelliJ works too. We can do even better too:
>> there's no reason these parts need to run in the same process it even the same
>> machine.
>
> I think we should be careful when comparing Emacs with systems that
> have a different basic design.  In Emacs, almost everything is driven
> by changes in buffer text and the data structures that accompany
> buffer text, and the Lisp interpreter is inherent in many
> display-related features.
>
> For example, this:
>
>> But this scheme won't really eliminate the coupling between redisplay and lisp
>> though, since the former calls into the latter. The redisplay thread has to
>> block waiting on lisp anyway.
>
> is a major issue that those other systems might not have.
>
>> Anyway, I strongly encourage you to look at the React Native rendering model.
>> It's the most elegant way I've seen of constructing GUIs in general.
>
> Where's that model described in enough technical detail to be able to
> compare it with what we do in Emacs?
>
>> The key insight there is that we shouldn't have redisplay *lock* the display
>> matrix and render it. The lisp universe should send a *copy* of the matrix set,
>> then go about its business. This way, redisplay can go display that copy and
>> everything is decoupled. You turn the system into an Erlang like message
>> passing environment.
>
> There's probably some confusion or misunderstanding here: the Lisp
> code in Emacs has no access to the glyph matrices, so it does not (and
> cannot) "send the display matrix" to the display engine.  Thus, the
> fact that the display engine can lock the glyph matrices is due to its
> being their sole writer and reader.  The communication between the
> Lisp universe and the display engine is solely via the buffer text and
> the auxiliary data structures (text properties, overlays, etc.).

The "lisp universe" is a terrible name for what I had in mind. Sorry for 
the confusion. Stefan is proposing splitting Emacs into at least two 
parts: one part that responds to OS-level events: repainting, input, and 
so on; and one part that runs lisp code, changes buffer text, and so on. 
It's this latter part I've been calling the "lisp universe", but it's 
more than that.

Of course there's no direct access to glyph matrices, but you can 
imagine a scheme where both of the two parts above each have an idea of 
what desired layout should be.

Re: Can we go GTK-only?

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, raeburn@raeburn.org, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 31 Oct 2016 08:59:11 -0700
> 
> Of course there's no direct access to glyph matrices, but you can 
> imagine a scheme where both of the two parts above each have an idea of 
> what desired layout should be.

That's exactly the part that needs to be carefully designed.  It is
nowhere near what we have now, because the current expression of "the
idea of what the desired layout should be" is the glyph matrices,
which in their present shape cannot be shared by several processes.

I'm not even sure we could come up with a useful scheme that would
allow such a division in a way that will let the communications
between the two work efficiently.  It remains to be shown that this is
feasible.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Mon, 31 Oct 2016 18:47:23 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Cc: monnier@iro.umontreal.ca, raeburn@raeburn.org,
> > emacs-devel@gnu.org From: Daniel Colascione <dancol@dancol.org>
> > Date: Mon, 31 Oct 2016 08:59:11 -0700
> > 
> > Of course there's no direct access to glyph matrices, but you can 
> > imagine a scheme where both of the two parts above each have an
> > idea of what desired layout should be.  
> 
> That's exactly the part that needs to be carefully designed.  It is
> nowhere near what we have now, because the current expression of
> "the idea of what the desired layout should be" is the glyph
> matrices, which in their present shape cannot be shared by several
> processes.
> 
> I'm not even sure we could come up with a useful scheme that would
> allow such a division in a way that will let the communications
> between the two work efficiently.  It remains to be shown that this
> is feasible.

So there would seem to be two obvious ways to keep the two
matrices in sync. One would be to record all the changes made on the
one side as they are made and serialize that. The other way would be
to use a dynamic programming algorithm to find the minimum set of
differences and transmit them. Given that redisplay is likely nearly
continuous, I'd say the former makes more sense.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Mon, 31 Oct 2016 13:54:38 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Daniel Colascione <dancol@dancol.org>, raeburn@raeburn.org,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > That's exactly the part that needs to be carefully designed.  It is
> > nowhere near what we have now, because the current expression of
> > "the idea of what the desired layout should be" is the glyph
> > matrices, which in their present shape cannot be shared by several
> > processes.
> > 
> > I'm not even sure we could come up with a useful scheme that would
> > allow such a division in a way that will let the communications
> > between the two work efficiently.  It remains to be shown that this
> > is feasible.
> 
> So there would seem to be two obvious ways to keep the two
> matrices in sync. One would be to record all the changes made on the
> one side as they are made and serialize that. The other way would be
> to use a dynamic programming algorithm to find the minimum set of
> differences and transmit them. Given that redisplay is likely nearly
> continuous, I'd say the former makes more sense.

I think you are missing the point.  The hard part is to figure out how
to convey the information across the divide, without forcing the
receiving side to repeat a large part of the layout calculations and
face resolution that the sending side already did.

The rest is much easier, either way.

I suggest to look at the implementation of the write_glyphs method
that is invoked when a screen line or its part needs to be delivered
to the glass.  Specifically, look at 'struct glyph_string', which is
the object used to describe a series of glyphs the terminal-specific
back-end needs to draw.  You will see there what information is being
communicated to the back-end; dividing Emacs in two would need to find
a way to communicate the equivalent of that, without relying on common
memory.

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 30 Oct 2016 17:42:18 -0400
> Cc: emacs-devel@gnu.org
> 
> Ideally, yes, I definitely think we should split the Emacs process into
> various threads:
> - some threads for the GUI.
> - one thread for the Elisp engine (which receives events from the GUI
>   threads among others).

What do "the GUI threads" do?  "GUI" can mean many things, so the
above is too vague, IMO.

> So the GUI thread would handle the expose events, read-locking the
> matrices, while the redisplay would run in another thread, write-locking
> those same matrices.

Handling an expose event entails consulting buffer contents and
overlays, e.g. when the exposed region has mouse highlight, so doing
that asynchronously might show incorrect display, if the main thread
modifies the buffer.

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Sun, 30 Oct 2016 10:43:37 -0400
> Cc: emacs-devel@gnu.org
> 
> I wonder if we should let them run their event loops, in their own threads, and 
> separate that from the non-GUI events like subprocesses and timers, and from 
> Lisp evaluation, as much as possible.  For that matter, some of the subprocess 
> handling could probably use helper threads, like for TLS encryption and 
> decryption work.

Beware of asking for too much, because that would mean a complete
redesign, something that I think is impractical.  One problem with
having too much code in separate threads is that only the main thread
can call malloc/free, i.e. you cannot create/destroy objects in other
threads.  Another problem is calling QUIT: you need to switch to the
main thread before you do that, or else you will crash.  And there are
probably other limitations.

The w32 build uses a separate thread for receiving the window-system
events, so you might look there for some ideas that won't blow up the
basic MVC design of Emacs.  In a nutshell, the events received by the
w32 input thread are pushed onto a queue which is serviced by an
emulation of a socket-read hook, called from keyboard.c, like all the
other such hooks.

Of course, this doesn't help when the main thread is busy doing
something prolonged, because it then doesn't get to processing the
input events.

Re: Can we go GTK-only?

[Ken Raeburn]

On Oct 31, 2016, at 11:54, Eli Zaretskii <eliz@gnu.org> wrote:

>   One problem with
> having too much code in separate threads is that only the main thread
> can call malloc/free, i.e. you cannot create/destroy objects in other
> threads.

Wow.  Is that a Windows limitation?  It’s certainly not true under POSIX threads.

Creating Lisp objects, that’d be another matter, unless locking is introduced to the allocator.

>  Another problem is calling QUIT: you need to switch to the
> main thread before you do that, or else you will crash.

Yes, quit checking, generation, and inhibition would all need to be reviewed.

>  And there are
> probably other limitations.

Probably.  I doubt any are insurmountable.

> The w32 build uses a separate thread for receiving the window-system
> events, so you might look there for some ideas that won't blow up the
> basic MVC design of Emacs.  In a nutshell, the events received by the
> w32 input thread are pushed onto a queue which is serviced by an
> emulation of a socket-read hook, called from keyboard.c, like all the
> other such hooks.

Seems like a good place to start.

> Of course, this doesn't help when the main thread is busy doing
> something prolonged, because it then doesn't get to processing the
> input events.

Yeah, in cases like that it’d be no better than today.

Ken

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 31 Oct 2016 14:22:50 -0400
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org
> 
> 
> On Oct 31, 2016, at 11:54, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >   One problem with
> > having too much code in separate threads is that only the main thread
> > can call malloc/free, i.e. you cannot create/destroy objects in other
> > threads.
> 
> Wow.  Is that a Windows limitation?  It’s certainly not true under POSIX threads.

No, it's a general limitation: malloc is non-reentrant.

> Creating Lisp objects, that’d be another matter, unless locking is introduced to the allocator.

If you cannot allocate Lisp objects, the scope of what you can do in
the non-main threads is greatly diminished.  E.g., no computation
intensive jobs that operate on buffer text can be off-loaded to other
threads.

Re: Can we go GTK-only?

[Daniel Colascione]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Mon, 31 Oct 2016 14:22:50 -0400
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>>  emacs-devel@gnu.org
>> 
>> 
>> On Oct 31, 2016, at 11:54, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> >   One problem with
>> > having too much code in separate threads is that only the main thread
>> > can call malloc/free, i.e. you cannot create/destroy objects in other
>> > threads.
>> 
>> Wow.  Is that a Windows limitation?  It’s certainly not true under POSIX threads.
>
> No, it's a general limitation: malloc is non-reentrant.

Reentrancy is not the same as thread safety.

Of course you can call malloc from multiple threads. Otherwise, projects
like jemalloc would be pointless. You can freely allocate and deallocate
from different threads on both POSIX and Windows systems, and there is
no need to free an object on the thread that allocated it.

>> Creating Lisp objects, that’d be another matter, unless locking is introduced to the allocator.
>
> If you cannot allocate Lisp objects, the scope of what you can do in
> the non-main threads is greatly diminished.  E.g., no computation
> intensive jobs that operate on buffer text can be off-loaded to other
> threads.

Allocation of lisp objects is different. _That_ isn't thread safe
right now. The easiest way to address this problem is a GIL.

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 31 Oct 2016 14:04:54 -0700
> Cc: Ken Raeburn <raeburn@raeburn.org>, monnier@iro.umontreal.ca,
> 	emacs-devel@gnu.org
> 
> >> >   One problem with
> >> > having too much code in separate threads is that only the main thread
> >> > can call malloc/free, i.e. you cannot create/destroy objects in other
> >> > threads.
> >> 
> [...]
> Of course you can call malloc from multiple threads. Otherwise, projects
> like jemalloc would be pointless. You can freely allocate and deallocate
> from different threads on both POSIX and Windows systems, and there is
> no need to free an object on the thread that allocated it.

IMO, this is not a safe assumption, even though in practice more and
more systems out there provide thread-safe native malloc.  Only C11
mandates that malloc/realloc/free shall be thread-safe, and we don't
yet require C11.  gmalloc is only thread-safe if Emacs is built with
pthreads.  ralloc is not thread-safe at all.  xmalloc calls
memory_full, which manipulates global state and calls xsignal, so that
is not thread-safe, either.

IOW, we are barely out of the woods with thread-safety of memory
allocation, so IMO it's too early for us to build basic infrastructure
on the thread-safety assumption.  For experimental and exotic
features, yes, but not for something that must work well on all
supported systems.

> >> Creating Lisp objects, that’d be another matter, unless locking is 
> >> introduced to the allocator.
> >
> > If you cannot allocate Lisp objects, the scope of what you can do in
> > the non-main threads is greatly diminished.  E.g., no computation
> > intensive jobs that operate on buffer text can be off-loaded to other
> > threads.
> 
> Allocation of lisp objects is different. _That_ isn't thread safe
> right now. The easiest way to address this problem is a GIL.

GIL hurts performance so much that I'd question any GIL-based design
that attempts to support off-loading CPU-intensive tasks to worker
threads.

Re: Can we go GTK-only?

[Paul Eggert]

On 11/01/2016 08:11 AM, Eli Zaretskii wrote:
 > Only C11 mandates that malloc/realloc/free shall be thread-safe, and 
we don't yet require C11.

This is too pessimistic. C11 was the first C standard to talk about 
threads, which is why it's the first C standard to specify whether 
malloc is thread-safe. In practice it should be safe to assume that 
malloc is thread-safe on multithreaded platforms, as C programmers would 
have revolted en masse otherwise.

 > gmalloc is only thread-safe if Emacs is built with pthreads.

Yes, and that's what one would expect. If you build Emacs in 
single-threaded mode, malloc won't be thread-safe. But in the normal 
case nowadays, malloc should be thread-safe.

 > ralloc is not thread-safe at all.

Yes, and ralloc as it stands should not be used on modern platforms. We 
clearly need to move in that direction anyway.

 > xmalloc calls memory_full, which manipulates global state and calls 
xsignal, so that is not thread-safe, either.

That's fine, so long as xmalloc is called only in the Emacs Lisp thread. 
xmalloc is like xsignal, Fcons, and lots of other functions that should 
be called only in the Emacs Lisp thread.

Re: Can we go GTK-only?

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 1 Nov 2016 09:28:34 -0700
> 
> On 11/01/2016 08:11 AM, Eli Zaretskii wrote:
>  > Only C11 mandates that malloc/realloc/free shall be thread-safe, and 
> we don't yet require C11.
> 
> This is too pessimistic. C11 was the first C standard to talk about 
> threads, which is why it's the first C standard to specify whether 
> malloc is thread-safe. In practice it should be safe to assume that 
> malloc is thread-safe on multithreaded platforms, as C programmers would 
> have revolted en masse otherwise.

"It should be safe" and "it's safe" are 2 different things.

>  > gmalloc is only thread-safe if Emacs is built with pthreads.
> 
> Yes, and that's what one would expect. If you build Emacs in 
> single-threaded mode, malloc won't be thread-safe. But in the normal 
> case nowadays, malloc should be thread-safe.

pthreads is not the only way to have threads.

>  > ralloc is not thread-safe at all.
> 
> Yes, and ralloc as it stands should not be used on modern platforms. We 
> clearly need to move in that direction anyway.

We do move in that direction, but we aren't there yet.

>  > xmalloc calls memory_full, which manipulates global state and calls 
> xsignal, so that is not thread-safe, either.
> 
> That's fine, so long as xmalloc is called only in the Emacs Lisp thread. 

I'd imagine any code that wants to allocate from the heap will call
xmalloc, as we never call malloc directly in Emacs AFAIK.

Like I said: we are barely out of the woods, so allocations from the
heap in non-main threads should be avoided.

Re: Can we go GTK-only?

[Daniel Colascione]

On 11/01/2016 09:49 AM, Eli Zaretskii wrote:
>> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Tue, 1 Nov 2016 09:28:34 -0700
>>
>> On 11/01/2016 08:11 AM, Eli Zaretskii wrote:
>>  > Only C11 mandates that malloc/realloc/free shall be thread-safe, and
>> we don't yet require C11.
>>
>> This is too pessimistic. C11 was the first C standard to talk about
>> threads, which is why it's the first C standard to specify whether
>> malloc is thread-safe. In practice it should be safe to assume that
>> malloc is thread-safe on multithreaded platforms, as C programmers would
>> have revolted en masse otherwise.
>
> "It should be safe" and "it's safe" are 2 different things.
>
>>  > gmalloc is only thread-safe if Emacs is built with pthreads.
>>
>> Yes, and that's what one would expect. If you build Emacs in
>> single-threaded mode, malloc won't be thread-safe. But in the normal
>> case nowadays, malloc should be thread-safe.
>
> pthreads is not the only way to have threads.

On any modern system POSIX system it is. Counterexample, please.

>>  > ralloc is not thread-safe at all.
>>
>> Yes, and ralloc as it stands should not be used on modern platforms. We
>> clearly need to move in that direction anyway.
>
> We do move in that direction, but we aren't there yet.
>
>>  > xmalloc calls memory_full, which manipulates global state and calls
>> xsignal, so that is not thread-safe, either.
>>
>> That's fine, so long as xmalloc is called only in the Emacs Lisp thread.
>
> I'd imagine any code that wants to allocate from the heap will call
> xmalloc, as we never call malloc directly in Emacs AFAIK.

That's because we want xmalloc to handle memory exhaustion in a sane 
way. A thread calling system malloc on its own can handle memory 
exhaustion a different way. Memory exhaustion being reported in some way 
is what really matters.

> Like I said: we are barely out of the woods, so allocations from the
> heap in non-main threads should be avoided.

This restriction makes a big class of programs very difficult to write. 
Besides, in my Emacs right now, I have in addition to the main thread a 
generic glib worker thread, a dconf worker thread, and a gdbus thread. 
I'd be amazed if none of these threads allocated memory.

Re: Can we go GTK-only?

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 1 Nov 2016 09:54:51 -0700
> 
> >>  > gmalloc is only thread-safe if Emacs is built with pthreads.
> >>
> >> Yes, and that's what one would expect. If you build Emacs in
> >> single-threaded mode, malloc won't be thread-safe. But in the normal
> >> case nowadays, malloc should be thread-safe.
> >
> > pthreads is not the only way to have threads.
> 
> On any modern system POSIX system it is. Counterexample, please.

You can find them yourself if you are interested.  I have more
important things to do with my time.

> >>  > xmalloc calls memory_full, which manipulates global state and calls
> >> xsignal, so that is not thread-safe, either.
> >>
> >> That's fine, so long as xmalloc is called only in the Emacs Lisp thread.
> >
> > I'd imagine any code that wants to allocate from the heap will call
> > xmalloc, as we never call malloc directly in Emacs AFAIK.
> 
> That's because we want xmalloc to handle memory exhaustion in a sane 
> way. A thread calling system malloc on its own can handle memory 
> exhaustion a different way. Memory exhaustion being reported in some way 
> is what really matters.

We'd have to write that stuff before this issue can be regarded as
solved.

> > Like I said: we are barely out of the woods, so allocations from the
> > heap in non-main threads should be avoided.
> 
> This restriction makes a big class of programs very difficult to write. 

That's what I'm saying: that class of programs is not yet feasible for
Emacs.  In a couple of years, maybe.

> Besides, in my Emacs right now, I have in addition to the main thread a 
> generic glib worker thread, a dconf worker thread, and a gdbus thread. 

We are not talking about your Emacs or mine.  (In my Emacs, those
problems cannot happen at all, because as you know on Windows each DLL
calls the memory allocator it was linked against, so the tricks Emacs
plays with its private malloc are harmless, because no external
library will ever call it.)

But I was talking about the mainstream Emacs on J.R.Hacker's random
machine out there.  That is the platform about whose stability I
worry.

Re: Can we go GTK-only?

[Daniel Colascione]

On 11/01/2016 10:08 AM, Eli Zaretskii wrote:
>> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 1 Nov 2016 09:54:51 -0700
>>
>>>>  > gmalloc is only thread-safe if Emacs is built with pthreads.
>>>>
>>>> Yes, and that's what one would expect. If you build Emacs in
>>>> single-threaded mode, malloc won't be thread-safe. But in the normal
>>>> case nowadays, malloc should be thread-safe.
>>>
>>> pthreads is not the only way to have threads.
>>
>> On any modern system POSIX system it is. Counterexample, please.
>
> You can find them yourself if you are interested.  I have more
> important things to do with my time.

What other thread mechanism? Windows threads? Malloc is thread-safe. GNU 
Pth? Not really multi-threaded, so malloc is safe. Old-fashioned 
pre-pthreads Sun threads? We don't support that platform, and I strongly 
suspect malloc was safe there too.

You're the one making the unusual claim, so you're the one who needs to 
provide evidence.


>>>>  > xmalloc calls memory_full, which manipulates global state and calls
>>>> xsignal, so that is not thread-safe, either.
>>>>
>>>> That's fine, so long as xmalloc is called only in the Emacs Lisp thread.
>>>
>>> I'd imagine any code that wants to allocate from the heap will call
>>> xmalloc, as we never call malloc directly in Emacs AFAIK.
>>
>> That's because we want xmalloc to handle memory exhaustion in a sane
>> way. A thread calling system malloc on its own can handle memory
>> exhaustion a different way. Memory exhaustion being reported in some way
>> is what really matters.
>
> We'd have to write that stuff before this issue can be regarded as
> solved.

It'll be solved in whatever code does the memory allocation. Nobody 
knows yet what that looks like, because it isn't been written, but it's 
obvious in broad strokes that you can check whether malloc returns NULL 
and stop what you're doing or set an error flag _somehow_. There's no 
technical problem to be solved: just code to be written.

>
>>> Like I said: we are barely out of the woods, so allocations from the
>>> heap in non-main threads should be avoided.
>>
>> This restriction makes a big class of programs very difficult to write.
>
> That's what I'm saying: that class of programs is not yet feasible for
> Emacs.  In a couple of years, maybe.

What will have changed in a few years? If you have evidence that we have 
a problem now, please provide it. If your presupposition of 
evidence-free now, it'll be evidence-free in a few years too.

>> Besides, in my Emacs right now, I have in addition to the main thread a
>> generic glib worker thread, a dconf worker thread, and a gdbus thread.
>
> We are not talking about your Emacs or mine.  (In my Emacs, those
> problems cannot happen at all, because as you know on Windows each DLL
> calls the memory allocator it was linked against, so the tricks Emacs
> plays with its private malloc are harmless, because no external
> library will ever call it.)
>
> But I was talking about the mainstream Emacs on J.R.Hacker's random
> machine out there.  That is the platform about whose stability I
> worry.

"R.Hacker's random machine" is not a supported platform. Name a 
supported platform that concerns you.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 19:08:51 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca,
> > emacs-devel@gnu.org From: Daniel Colascione <dancol@dancol.org>
> > Date: Tue, 1 Nov 2016 09:54:51 -0700
> >   
> > >>  > gmalloc is only thread-safe if Emacs is built with
> > >>  > pthreads.  
> > >>
> > >> Yes, and that's what one would expect. If you build Emacs in
> > >> single-threaded mode, malloc won't be thread-safe. But in the
> > >> normal case nowadays, malloc should be thread-safe.  
> > >
> > > pthreads is not the only way to have threads.  
> > 
> > On any modern system POSIX system it is. Counterexample, please.  
> 
> You can find them yourself if you are interested.  I have more
> important things to do with my time.

You do realize that this means that most of the rest of us are not
going to take your claim seriously.

I cannot name

a) a POSIX system that uses a threading system other than pthreads
(which makes sense since the POSIX standard mandates pthreads)

b) any system that implements both C and threads where malloc is not
thread safe and has not always been thread safe since the threading
implementation was shipped (which makes sense because otherwise no
threaded software would be stable).

It is possible such systems existed long ago. They do not now.

If you claim you can find such things but that you do not choose to,
I think the rest of us are simply going to believe your claim as
false.

As for non-POSIX systems Emacs runs on, the only significant one is
Windows, and in Windows, malloc is thread safe in threaded code, just
as one would expect.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Lars Ingebrigtsen]

I've tried to google for various combinations of
malloc/thread/unsafe/bug, and the only thing I've found is this:

https://sourceware.org/bugzilla/show_bug.cgi?id=6952

In 2008, there was a malloc bug in glibc that, in some situations, made
using malloc from the non-main thread problematic.  (And typically
enough, Ulrich Drepper responds "Worthless report.", and more emails
happen, and then some time passes, and then he fixes the bug without
responding to the report.  Ah, glibc.  :-))

I may be using the wrong search terms, but that's the only thing I found
of interest...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Tue, 1 Nov 2016 15:15:49 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org,
> 	raeburn@raeburn.org, monnier@iro.umontreal.ca, eggert@cs.ucla.edu
> 
> On Tue, 01 Nov 2016 19:08:51 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > > Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca,
> > > emacs-devel@gnu.org From: Daniel Colascione <dancol@dancol.org>
> > > Date: Tue, 1 Nov 2016 09:54:51 -0700
> > >   
> > > >>  > gmalloc is only thread-safe if Emacs is built with
> > > >>  > pthreads.  
> > > >>
> > > >> Yes, and that's what one would expect. If you build Emacs in
> > > >> single-threaded mode, malloc won't be thread-safe. But in the
> > > >> normal case nowadays, malloc should be thread-safe.  
> > > >
> > > > pthreads is not the only way to have threads.  
> > > 
> > > On any modern system POSIX system it is. Counterexample, please.  
> > 
> > You can find them yourself if you are interested.  I have more
> > important things to do with my time.
> 
> You do realize that this means that most of the rest of us are not
> going to take your claim seriously.

And until now you did?  "I am having trouble believing" -- it's your
words.  That's a terrific incentive to continue this discussion.

Re: Can we go GTK-only?

[Paul Eggert]

On 11/01/2016 09:49 AM, Eli Zaretskii wrote:
> pthreads is not the only way to have threads. 

Whether one uses pthreads or not, a thread-safe malloc is the only sane 
way malloc in a multithreaded application. We shouldn't have to worry 
about multithreaded environments where malloc does not work. Such 
environments are too buggy to be a reasonable porting target for Emacs.

> I'd imagine any code that wants to allocate from the heap will call 
> xmalloc

No, code that allocates from the heap in non-Emacs-Lisp threads should 
call malloc directly. It shouldn't call Emacs xmalloc. Emacs xmalloc is 
intended for the Emacs Lisp thread. That's how it works now, anyway, and 
we should simply keep it that way.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 1 Nov 2016 09:28:34 -0700 Paul Eggert <eggert@cs.ucla.edu>
wrote:
> On 11/01/2016 08:11 AM, Eli Zaretskii wrote:
>  > Only C11 mandates that malloc/realloc/free shall be thread-safe,
>  > and   
> we don't yet require C11.
> 
> This is too pessimistic. C11 was the first C standard to talk about 
> threads, which is why it's the first C standard to specify whether 
> malloc is thread-safe.

Precisely. Before then, the thread safety of malloc was discussed in
standards that talked about threads, like the pthreads standards in
POSIX.

> In practice it should be safe to assume that 
> malloc is thread-safe on multithreaded platforms, as C programmers
> would have revolted en masse otherwise.

Again, the Single Unix Specification and its predecessors always
said, very explicitly, that malloc was thread safe, as does Windows.
I am unaware of the existence of a platform that supports both C and
threads in which malloc is not documented to be thread safe.
Certainly if such a platform once existed, it is long gone and does
not support a modern Emacs.

>  > gmalloc is only thread-safe if Emacs is built with pthreads.  
> 
> Yes, and that's what one would expect. If you build Emacs in 
> single-threaded mode, malloc won't be thread-safe. But in the
> normal case nowadays, malloc should be thread-safe.

Precisely. Generally, the library calls are only thread safe if you
link with the pthreads library or the like, but we're presuming that
is what would happen here.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 17:11:57 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Of course you can call malloc from multiple threads. Otherwise,
> > projects like jemalloc would be pointless. You can freely
> > allocate and deallocate from different threads on both POSIX and
> > Windows systems, and there is no need to free an object on the
> > thread that allocated it.  
> 
> IMO, this is not a safe assumption, even though in practice more and
> more systems out there provide thread-safe native malloc.  Only C11
> mandates that malloc/realloc/free shall be thread-safe, and we don't
> yet require C11.

I was under the impression the requirement that malloc be thread safe
was before now a POSIX/pthreads thing, not a C standard thing, and
that this had been the case for a very long time.

SUS says this explicitly here: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09

   2.9.1 Thread-Safety

   All functions defined by this volume of POSIX.1-2008 shall be
   thread-safe, except that the following functions need not be
   thread-safe.

It then goes on to not mention malloc, so malloc is required to be
thread safe.

I can look up old versions of the standard but I believe it was the
case as long as pthreads has been around.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Tue, 1 Nov 2016 12:41:12 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Daniel Colascione <dancol@dancol.org>, raeburn@raeburn.org,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On Tue, 01 Nov 2016 17:11:57 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > > Of course you can call malloc from multiple threads. Otherwise,
> > > projects like jemalloc would be pointless. You can freely
> > > allocate and deallocate from different threads on both POSIX and
> > > Windows systems, and there is no need to free an object on the
> > > thread that allocated it.  
> > 
> > IMO, this is not a safe assumption, even though in practice more and
> > more systems out there provide thread-safe native malloc.  Only C11
> > mandates that malloc/realloc/free shall be thread-safe, and we don't
> > yet require C11.
> 
> I was under the impression the requirement that malloc be thread safe
> was before now a POSIX/pthreads thing, not a C standard thing, and
> that this had been the case for a very long time.

We don't only support POSIX platforms.  And even for POSIX platforms,
you can find on the net reports about thread-unsafe malloc up to 2013
and 2014.  That's not "very long time".

> I can look up old versions of the standard but I believe it was the
> case as long as pthreads has been around.

My concern is not with the standards, but with the actual situation
out there.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 18:54:35 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > I was under the impression the requirement that malloc be thread
> > safe was before now a POSIX/pthreads thing, not a C standard
> > thing, and that this had been the case for a very long time.  
> 
> We don't only support POSIX platforms.

It is also the case on Windows and on all non-POSIX platforms I'm
aware of that are of significance. That said, I don't believe that
Emacs supports much other than POSIX and Windows, so we could easily
check the others.

> And even for POSIX
> platforms, you can find on the net reports about thread-unsafe
> malloc up to 2013 and 2014.  That's not "very long time".

Are you sure this wasn't just a report from some people who linked
with the wrong library?

The standard has required that malloc be thread safe as long as
pthreads has been around IIRC. I'm unaware of any such reports of
lack of support, but if they exist, it would indicate a desperately
broken platform that couldn't run any thread safe code to speak of.
One of the first things one needs to do in order to support pthreads
is to produce a libc that knows about pthread safety. If you don't
have that, you can port essentially 0 thread safe programs and users
can write essentially nothing. malloc is a very basic call.

I could be wrong here, of course, but I'd need to see much more
concrete evidence of it.

> > I can look up old versions of the standard but I believe it was
> > the case as long as pthreads has been around.  
> 
> My concern is not with the standards, but with the actual situation
> out there.

The actual implementations *have* to handle thread safety of the basic
library calls or basically no software will run. malloc is such a
basic call that if you didn't handle it correctly you couldn't expect
pthreads to be of any use on your platform.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Tue, 1 Nov 2016 13:22:02 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: dancol@dancol.org, raeburn@raeburn.org, monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> 
> > And even for POSIX
> > platforms, you can find on the net reports about thread-unsafe
> > malloc up to 2013 and 2014.  That's not "very long time".
> 
> Are you sure this wasn't just a report from some people who linked
> with the wrong library?

Yes.

> The standard has required that malloc be thread safe as long as
> pthreads has been around IIRC.

Requiring is one thing; being able to implement it and weed out the
bugs is quite another.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 19:46:30 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Tue, 1 Nov 2016 13:22:02 -0400
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: dancol@dancol.org, raeburn@raeburn.org,
> > monnier@iro.umontreal.ca, emacs-devel@gnu.org
> >   
> > > And even for POSIX
> > > platforms, you can find on the net reports about thread-unsafe
> > > malloc up to 2013 and 2014.  That's not "very long time".  
> > 
> > Are you sure this wasn't just a report from some people who linked
> > with the wrong library?  
> 
> Yes.
> 
> > The standard has required that malloc be thread safe as long as
> > pthreads has been around IIRC.  
> 
> Requiring is one thing; being able to implement it and weed out the
> bugs is quite another.
> 

I am having trouble believing this is really the case. Essentially
all threaded software would break if required libc library calls
weren't thread safe. Could you please point at a specific
example of this having happened on a platform built after the mid
1990s?

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 19:46:30 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > The standard has required that malloc be thread safe as long as
> > pthreads has been around IIRC.  
> 
> Requiring is one thing; being able to implement it and weed out the
> bugs is quite another.

A bug in something as basic as malloc/free thread handling would
start causing ported C programs to fail almost immediately. The
operating system code would never get out of testing in such a state
let alone be released to users. malloc is called far too often.

I would not be shocked if some rare latent bugs existed in the average
threaded libc, after all, bugs are common in software, and
multithreading is particularly hard.

However, I would be shocked if a single threaded libc did not handle
this particular case in as bug free a manner as the rest of malloc
itself. If one is willing to accept the use of the system malloc in
the non-threaded case on the premise that it is reasonably bug free,
then one should likely accept that it handles thread safety as well.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Daniel Colascione]

On 11/01/2016 08:11 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 31 Oct 2016 14:04:54 -0700
>> Cc: Ken Raeburn <raeburn@raeburn.org>, monnier@iro.umontreal.ca,
>> 	emacs-devel@gnu.org
>>
>>>>>   One problem with
>>>>> having too much code in separate threads is that only the main thread
>>>>> can call malloc/free, i.e. you cannot create/destroy objects in other
>>>>> threads.
>>>>
>> [...]
>> Of course you can call malloc from multiple threads. Otherwise, projects
>> like jemalloc would be pointless. You can freely allocate and deallocate
>> from different threads on both POSIX and Windows systems, and there is
>> no need to free an object on the thread that allocated it.
>
> IMO, this is not a safe assumption, even though in practice more and
> more systems out there provide thread-safe native malloc.  Only C11
> mandates that malloc/realloc/free shall be thread-safe, and we don't
> yet require C11.

The C99 standard is silent on threads, so of course it doesn't mandate 
thread-safe malloc and free.

   gmalloc is only thread-safe if Emacs is built with
> pthreads.  ralloc is not thread-safe at all.  xmalloc calls
> memory_full, which manipulates global state and calls xsignal, so that
> is not thread-safe, either.

Name one system we support that both _has_ threads and that doesn't have 
a thread-safe system malloc. If we're using our own malloc and _that_ 
isn't thread-safe, that doesn't count. I insist that on modern systems, 
the malloc and free that come with libc are thread safe.

> IOW, we are barely out of the woods with thread-safety of memory
> allocation,

No we're not. When it comes to the thread safety of memory allocation, 
we're well out of the woods, down the mountains, across the plains, in 
the city, and siting at a coffee shop sipping a our-over latte while 
reading an article on our fancy e-readers about how nobody goes outside 
anymore. That's how well-established thread-safe malloc and free are on 
systems that support threads at all.

> so IMO it's too early for us to build basic infrastructure
> on the thread-safety assumption.  For experimental and exotic
> features, yes, but not for something that must work well on all
> supported systems.

No it isn't.

>>>> Creating Lisp objects, that’d be another matter, unless locking is
>>>> introduced to the allocator.
>>>
>>> If you cannot allocate Lisp objects, the scope of what you can do in
>>> the non-main threads is greatly diminished.  E.g., no computation
>>> intensive jobs that operate on buffer text can be off-loaded to other
>>> threads.
>>
>> Allocation of lisp objects is different. _That_ isn't thread safe
>> right now. The easiest way to address this problem is a GIL.
>
> GIL hurts performance so much that I'd question any GIL-based design
> that attempts to support off-loading CPU-intensive tasks to worker
> threads.

On what basis do you make this claim? As someone mentioned previously, 
that Python paper isn't really relevant, as we're not doing CPU 
preemption. A GIL would let multiple threads run Lisp code and, while 
blocked, call into native code that did computation-intensive work. The 
main effect of the GIL is to prevent Lisp taking advantage of hardware 
parallelism, but since we're already in the position of being unable to 
take advantage of hardware parallelism, I don't see it really hurting us.

And at least we get concurrency and the potential to release the GIL 
while we do other work (like redisplay).

Re: Can we go GTK-only?

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 1 Nov 2016 09:45:41 -0700
> 
> Name one system we support that both _has_ threads and that doesn't have 
> a thread-safe system malloc. If we're using our own malloc and _that_ 
> isn't thread-safe, that doesn't count. I insist that on modern systems, 
> the malloc and free that come with libc are thread safe.

You can insist all you like, it won't change my mind: thread-safety in
malloc is only now becoming widespread and reliable enough, and older
systems where there are various bugs in that regard are still with us
in significant numbers.  Just google the keywords, and you will see
the bug reports and their dates.

> >> Allocation of lisp objects is different. _That_ isn't thread safe
> >> right now. The easiest way to address this problem is a GIL.
> >
> > GIL hurts performance so much that I'd question any GIL-based design
> > that attempts to support off-loading CPU-intensive tasks to worker
> > threads.
> 
> On what basis do you make this claim? As someone mentioned previously, 
> that Python paper isn't really relevant, as we're not doing CPU 
> preemption.

I think we've lost context: this thread is not about the concurrency
branch, where only one thread runs at a time, for which that Python
paper is irrelevant.  This thread (or at least what I wrote above) is
about the proposal to have more than one thread that performs
CPU-intensive tasks, so that the main thread could go about its
business.  For that, you will definitely want CPU preemption, because
those tasks don't have to run Lisp.

Re: Can we go GTK-only?

[Daniel Colascione]

On 11/01/2016 10:01 AM, Eli Zaretskii wrote:
>> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 1 Nov 2016 09:45:41 -0700
>>
>> Name one system we support that both _has_ threads and that doesn't have
>> a thread-safe system malloc. If we're using our own malloc and _that_
>> isn't thread-safe, that doesn't count. I insist that on modern systems,
>> the malloc and free that come with libc are thread safe.
>
> You can insist all you like, it won't change my mind: thread-safety in
> malloc is only now becoming widespread and reliable enough, and older
> systems where there are various bugs in that regard are still with us
> in significant numbers.  Just google the keywords, and you will see
> the bug reports and their dates.

Extraordinary claims require extraordinary evidence. Your claim is 
extraordinary: it's been common practice for _decades_ to make memory 
allocations from multiple threads in multithreaded programming. Programs 
that do this, many of which I use every day, run reliably. You need to 
provide evidence that what you say is true.

I did conduct search you suggested. The examples I found all involves 
people not compiling with the compile flags for enabling thread safety. 
What did these people expect to happen?

>>>> Allocation of lisp objects is different. _That_ isn't thread safe
>>>> right now. The easiest way to address this problem is a GIL.
>>>
>>> GIL hurts performance so much that I'd question any GIL-based design
>>> that attempts to support off-loading CPU-intensive tasks to worker
>>> threads.
>>
>> On what basis do you make this claim? As someone mentioned previously,
>> that Python paper isn't really relevant, as we're not doing CPU
>> preemption.
>
> I think we've lost context: this thread is not about the concurrency
> branch, where only one thread runs at a time, for which that Python
> paper is irrelevant.  This thread (or at least what I wrote above) is
> about the proposal to have more than one thread that performs
> CPU-intensive tasks, so that the main thread could go about its
> business.  For that, you will definitely want CPU preemption, because
> those tasks don't have to run Lisp.

If those CPU-intensive tasks are not written in Lisp, there is no need 
to hold the GIL while running them, so other threads can run Lisp in 
parallel.

Re: Can we go GTK-only?

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 1 Nov 2016 10:06:14 -0700
> 
> On 11/01/2016 10:01 AM, Eli Zaretskii wrote:
> >> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> >> From: Daniel Colascione <dancol@dancol.org>
> >> Date: Tue, 1 Nov 2016 09:45:41 -0700
> >>
> >> Name one system we support that both _has_ threads and that doesn't have
> >> a thread-safe system malloc. If we're using our own malloc and _that_
> >> isn't thread-safe, that doesn't count. I insist that on modern systems,
> >> the malloc and free that come with libc are thread safe.
> >
> > You can insist all you like, it won't change my mind: thread-safety in
> > malloc is only now becoming widespread and reliable enough, and older
> > systems where there are various bugs in that regard are still with us
> > in significant numbers.  Just google the keywords, and you will see
> > the bug reports and their dates.
> 
> Extraordinary claims require extraordinary evidence. Your claim is 
> extraordinary: it's been common practice for _decades_ to make memory 
> allocations from multiple threads in multithreaded programming.

This is simply incorrect.  On _some_ platforms, that is true.  But not
on all, not anywhere near that.

> > I think we've lost context: this thread is not about the concurrency
> > branch, where only one thread runs at a time, for which that Python
> > paper is irrelevant.  This thread (or at least what I wrote above) is
> > about the proposal to have more than one thread that performs
> > CPU-intensive tasks, so that the main thread could go about its
> > business.  For that, you will definitely want CPU preemption, because
> > those tasks don't have to run Lisp.
> 
> If those CPU-intensive tasks are not written in Lisp, there is no need 
> to hold the GIL while running them, so other threads can run Lisp in 
> parallel.

CPU-intensive threads that cannot manipulate Lisp objects (not run
Lisp, but create and destroy Lisp objects) are not very useful in
Emacs.

Re: Can we go GTK-only?

[Daniel Colascione]

On 11/01/2016 10:15 AM, Eli Zaretskii wrote:
>> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 1 Nov 2016 10:06:14 -0700
>>
>> On 11/01/2016 10:01 AM, Eli Zaretskii wrote:
>>>> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>>>> From: Daniel Colascione <dancol@dancol.org>
>>>> Date: Tue, 1 Nov 2016 09:45:41 -0700
>>>>
>>>> Name one system we support that both _has_ threads and that doesn't have
>>>> a thread-safe system malloc. If we're using our own malloc and _that_
>>>> isn't thread-safe, that doesn't count. I insist that on modern systems,
>>>> the malloc and free that come with libc are thread safe.
>>>
>>> You can insist all you like, it won't change my mind: thread-safety in
>>> malloc is only now becoming widespread and reliable enough, and older
>>> systems where there are various bugs in that regard are still with us
>>> in significant numbers.  Just google the keywords, and you will see
>>> the bug reports and their dates.
>>
>> Extraordinary claims require extraordinary evidence. Your claim is
>> extraordinary: it's been common practice for _decades_ to make memory
>> allocations from multiple threads in multithreaded programming.
>
> This is simply incorrect.  On _some_ platforms, that is true.  But not
> on all, not anywhere near that.

I'm not asking you to enumerate all platforms with both threads and 
thread-unsafe malloc. I'm just asking for _one_ platform that a) has 
threads, b) has thread-unsafe malloc, and that c) Emacs supports today. 
If you're researched this topic, you should be able to name one.

>
>>> I think we've lost context: this thread is not about the concurrency
>>> branch, where only one thread runs at a time, for which that Python
>>> paper is irrelevant.  This thread (or at least what I wrote above) is
>>> about the proposal to have more than one thread that performs
>>> CPU-intensive tasks, so that the main thread could go about its
>>> business.  For that, you will definitely want CPU preemption, because
>>> those tasks don't have to run Lisp.
>>
>> If those CPU-intensive tasks are not written in Lisp, there is no need
>> to hold the GIL while running them, so other threads can run Lisp in
>> parallel.
>
> CPU-intensive threads that cannot manipulate Lisp objects (not run
> Lisp, but create and destroy Lisp objects) are not very useful in
> Emacs.
>

Redisplay itself can run without invoking Lisp _most_ of the time, and 
it can take the GIL when it does. Other tasks, like image decoding and 
calls into native modules, can also run without invoking Lisp.

Re: Can we go GTK-only?

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 1 Nov 2016 10:18:58 -0700
> 
> > CPU-intensive threads that cannot manipulate Lisp objects (not run
> > Lisp, but create and destroy Lisp objects) are not very useful in
> > Emacs.
> 
> Redisplay itself can run without invoking Lisp _most_ of the time

No, it can't.  It calls Lisp almost all the time.  The two main places
where that usually happens is JIT font-lock and mode-line display.

Maybe you are thinking about "emacs -Q" which is idle, but that's not
a very interesting use case, when multi-threading is being discussed.

I don't want to argue about your other arguments because I don't see a
point.

Re: Can we go GTK-only?

[Daniel Colascione]

On 11/01/2016 10:44 AM, Eli Zaretskii wrote:
>> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 1 Nov 2016 10:18:58 -0700
>>
>>> CPU-intensive threads that cannot manipulate Lisp objects (not run
>>> Lisp, but create and destroy Lisp objects) are not very useful in
>>> Emacs.
>>
>> Redisplay itself can run without invoking Lisp _most_ of the time
>
> No, it can't.  It calls Lisp almost all the time.  The two main places
> where that usually happens is JIT font-lock and mode-line display.
>
> Maybe you are thinking about "emacs -Q" which is idle, but that's not
> a very interesting use case, when multi-threading is being discussed.
>
> I don't want to argue about your other arguments because I don't see a
> point.

Why don't you see a point in buttressing your claims with facts?

Re: Can we go GTK-only?

[Stefan Monnier]

> This is simply incorrect.  On _some_ platforms, that is true.  But not
> on all, not anywhere near that.

Give us a hint what hides behind this "not all".  Clearly, I'm not alone
here who really has no idea what you're afraid of, here.  Multi-threaded
programming is nasty, yes, but not nasty enough that you can't
use malloc.

BTW, w.r.t a GIL, AFAIK this discussion started mentioning threads
mostly as a way to *structure* the code to ease up interaction with libs
that want to have their own event loop (i.e. a matter of concurrency),
not as a way to improve performance.  So the use of a GIL wouldn't be
particularly scary.


        Stefan

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Daniel Colascione <dancol@dancol.org>,  raeburn@raeburn.org,  emacs-devel@gnu.org
> Date: Tue, 01 Nov 2016 15:14:22 -0400
> 
> > This is simply incorrect.  On _some_ platforms, that is true.  But not
> > on all, not anywhere near that.
> 
> Give us a hint what hides behind this "not all".  Clearly, I'm not alone
> here who really has no idea what you're afraid of, here.  Multi-threaded
> programming is nasty, yes, but not nasty enough that you can't
> use malloc.

What's the use?  You don't want to hear.  So let's leave it at that.

> BTW, w.r.t a GIL, AFAIK this discussion started mentioning threads
> mostly as a way to *structure* the code to ease up interaction with libs
> that want to have their own event loop (i.e. a matter of concurrency),
> not as a way to improve performance.

That's only part of what was said.  The other parts said nothing about
structuring.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 21:22:20 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: Daniel Colascione <dancol@dancol.org>,  raeburn@raeburn.org,
> > emacs-devel@gnu.org Date: Tue, 01 Nov 2016 15:14:22 -0400
> >   
> > > This is simply incorrect.  On _some_ platforms, that is true.
> > > But not on all, not anywhere near that.  
> > 
> > Give us a hint what hides behind this "not all".  Clearly, I'm
> > not alone here who really has no idea what you're afraid of,
> > here.  Multi-threaded programming is nasty, yes, but not nasty
> > enough that you can't use malloc.  
> 
> What's the use?  You don't want to hear.  So let's leave it at that.

In addition to not being able to name an OS that has (a) emacs
runs on (b) has threads and (c) supports C where (d) malloc is not
thread safe, I cannot name a person in this discussion who thinks you
are saying something correct here.

This is not a question of one person not wanting to hear a well
established claim in spite of evidence. There is a large group, in
fact, that thinks you are making a weird claim without evidence, and I
do not believe anyone so far thinks you are correct.

Again, this is easily resolved. Give us an example of such an OS. Or
perhaps say "I cannot give such an example" and let go of it with
dignity.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 19:15:31 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Extraordinary claims require extraordinary evidence. Your claim
> > is extraordinary: it's been common practice for _decades_ to make
> > memory allocations from multiple threads in multithreaded
> > programming.
> 
> This is simply incorrect.  On _some_ platforms, that is true.  But
> not on all, not anywhere near that.

You've explicitly refused to name an exception, and no one else is
aware of one, so how can we give credence to your claim?

Again, were your claim truly correct, no multithreaded C or C++
software would be stable on such a platform, so it seems like a very
unlikely statement. This is the sort of bug that would be found in the
first week that threading package shipped. The relevant standards have
also required it as long as threads have existed. I am disinclined to
believe it is true without evidence, and you refuse to present
evidence.

By the way: no one reasonable will think less of you if you later say
you made a mistake. People make mistakes.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Tue, 1 Nov 2016 15:20:27 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Daniel Colascione <dancol@dancol.org>, raeburn@raeburn.org,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On Tue, 01 Nov 2016 19:15:31 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > > Extraordinary claims require extraordinary evidence. Your claim
> > > is extraordinary: it's been common practice for _decades_ to make
> > > memory allocations from multiple threads in multithreaded
> > > programming.
> > 
> > This is simply incorrect.  On _some_ platforms, that is true.  But
> > not on all, not anywhere near that.
> 
> You've explicitly refused to name an exception, and no one else is
> aware of one, so how can we give credence to your claim?
> 
> Again, were your claim truly correct, no multithreaded C or C++
> software would be stable on such a platform, so it seems like a very
> unlikely statement. This is the sort of bug that would be found in the
> first week that threading package shipped. The relevant standards have
> also required it as long as threads have existed. I am disinclined to
> believe it is true without evidence, and you refuse to present
> evidence.

I was not talking about multithreading in general.  I was talking
specifically about Emacs, its coding practices, and its particular
design and needs wrt memory allocation.  I named several factors that
together lead me to the conclusion that we are not yet ready to allow
arbitrary multithreading in Emacs, although we and the supported
platforms are moving in the right direction.  The problems and issues
with thread-safe malloc in C libraries is just one of these factors,
perhaps not even the most important one, since at least in Emacs 25
many platforms we support don't use their native malloc.  (We switched
most of them to native malloc in Emacs 26, but we don't yet know
whether the results will be good enough, although we hope so.)

If you want to make this discussion a constructive one, please argue
about these aspects: about Emacs and Emacs alone, and how it can or
cannot tolerate arbitrary memory allocations, both for C and Lisp
objects, in multiple threads.  IOW, the arguments in such a
constructive discussion should be about specific aspects of Emacs
design and implementation, and about Emacs programming, that are
related to memory management.

As for "claims": this is more about gut feelings, based on the factors
I mentioned, than about anything else.  It is OK to disagree with gut
feelings, even if you agree with the facts.  It is NOT okay to make
this a discussion about my credibility.  If my credibility is being
questioned, I will simply step down.

> By the way: no one reasonable will think less of you if you later say
> you made a mistake. People make mistakes.

This goes both ways, of course.

Re: Can we go GTK-only?

[Daniel Colascione]

On 11/01/2016 01:05 PM, Eli Zaretskii wrote:
>> Date: Tue, 1 Nov 2016 15:20:27 -0400
>> From: "Perry E. Metzger" <perry@piermont.com>
>> Cc: Daniel Colascione <dancol@dancol.org>, raeburn@raeburn.org,
>>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
>>
>> On Tue, 01 Nov 2016 19:15:31 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
>>>> Extraordinary claims require extraordinary evidence. Your claim
>>>> is extraordinary: it's been common practice for _decades_ to make
>>>> memory allocations from multiple threads in multithreaded
>>>> programming.
>>>
>>> This is simply incorrect.  On _some_ platforms, that is true.  But
>>> not on all, not anywhere near that.
>>
>> You've explicitly refused to name an exception, and no one else is
>> aware of one, so how can we give credence to your claim?
>>
>> Again, were your claim truly correct, no multithreaded C or C++
>> software would be stable on such a platform, so it seems like a very
>> unlikely statement. This is the sort of bug that would be found in the
>> first week that threading package shipped. The relevant standards have
>> also required it as long as threads have existed. I am disinclined to
>> believe it is true without evidence, and you refuse to present
>> evidence.
>
> I was not talking about multithreading in general.  I was talking
> specifically about Emacs, its coding practices, and its particular
> design and needs wrt memory allocation.

You categorically stated that memory allocation off the main thread is 
unsafe. Once you admit that this statement is incorrect, we can move on 
to the Emacs-specific discussion, since we cannot talk about Emacs 
specifics without understanding the platforms on which Emacs runs.

The distinction is important: Lisp allocation is unsafe because of the 
layers we put on top of the system heap, not because the system heap is 
itself unusable. For example, we can create a scheme where we have 
multiple Lisp universes, each with its own heap and GC, and each bound 
to a single thread. (Like Web Workers.) If system malloc were unsafe, 
this scheme would not be thread-safe,  But since system malloc _is_ 
thread safe, this Emacs-specific approach to multithreading becomes 
feasible.


> I named several factors that
> together lead me to the conclusion that we are not yet ready to allow
> arbitrary multithreading in Emacs, although we and the supported
> platforms are moving in the right direction.  The problems and issues
> with thread-safe malloc in C libraries is just one of these factors,
> perhaps not even the most important one, since at least in Emacs 25
> many platforms we support don't use their native malloc.  (We switched
> most of them to native malloc in Emacs 26, but we don't yet know
> whether the results will be good enough, although we hope so.)
>
> If you want to make this discussion a constructive one, please argue
> about these aspects: about Emacs and Emacs alone, and how it can or
> cannot tolerate arbitrary memory allocations, both for C and Lisp
> objects, in multiple threads.

Emacs today can tolerate arbitrary memory allocations off the main 
thread after dumping using the system heap. I know this to be case 
because Emacs today, on the most common platforms --- OS X, GNU/Linux, 
and Windows, all perform allocations off the main thread in very common 
configurations _already_.

> IOW, the arguments in such a
> constructive discussion should be about specific aspects of Emacs
> design and implementation, and about Emacs programming, that are
> related to memory management.
>
> As for "claims": this is more about gut feelings, based on the factors
> I mentioned, than about anything else.  It is OK to disagree with gut
> feelings, even if you agree with the facts.  It is NOT okay to make
> this a discussion about my credibility.  If my credibility is being
> questioned, I will simply step down.

When prototype aircraft were designed according to gut feelings about 
how aerodynamics worked, they were unable to take off under their own 
power. When the Wright brothers designed their aircraft using data from 
a wind tunnel, their aircraft flew.

Re: Can we go GTK-only?

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 1 Nov 2016 13:17:01 -0700
> 
> > I was not talking about multithreading in general.  I was talking
> > specifically about Emacs, its coding practices, and its particular
> > design and needs wrt memory allocation.
> 
> You categorically stated that memory allocation off the main thread is 
> unsafe.

No, I didn't.  I said something else: that it wasn't safe to assume
memory allocations can be freely done in Emacs in non-main threads.

> Once you admit that this statement is incorrect

I don't have to admit anything, because I never said that.

> we can move on to the Emacs-specific discussion, since we cannot
> talk about Emacs specifics without understanding the platforms on
> which Emacs runs.

More to the point, we cannot talk about Emacs without a good
understanding of Emacs internals and coding practices.

> The distinction is important: Lisp allocation is unsafe because of the 
> layers we put on top of the system heap, not because the system heap is 
> itself unusable. For example, we can create a scheme where we have 
> multiple Lisp universes, each with its own heap and GC, and each bound 
> to a single thread. (Like Web Workers.) If system malloc were unsafe, 
> this scheme would not be thread-safe,  But since system malloc _is_ 
> thread safe, this Emacs-specific approach to multithreading becomes 
> feasible.

I never said it was infeasible.  I said that we are not yet ready for
that in Emacs.  Granted, given enough development and changes in our
practices, and enough experience with native malloc on most platforms,
some limited multithreading will be possible in Emacs.  Where exactly
to put those limits and how much development it will take remains to
be seen.

Once again, this is about the current situation and the current
infrastructure.  It is not about some future, nor about the
feasibility of this in general.  It was my impression that some of the
people in this thread thought they can start creating application
threads in Emacs tomorrow morning, so I wrote a few words of caution,
reminding them that there are currently several obstacles on that road
that need to be negotiated first.

> Emacs today can tolerate arbitrary memory allocations off the main 
> thread after dumping using the system heap. I know this to be case 
> because Emacs today, on the most common platforms --- OS X, GNU/Linux, 
> and Windows, all perform allocations off the main thread in very common 
> configurations _already_.

The only platform I know about that starts non-main threads which run
Emacs application code is MS-Windows, and there we don't allocate
anything off the heap, at least not off the same heap that is used by
the main (Lisp) thread.

What other threads on which platforms are you talking about?

> > As for "claims": this is more about gut feelings, based on the factors
> > I mentioned, than about anything else.  It is OK to disagree with gut
> > feelings, even if you agree with the facts.  It is NOT okay to make
> > this a discussion about my credibility.  If my credibility is being
> > questioned, I will simply step down.
> 
> When prototype aircraft were designed according to gut feelings about 
> how aerodynamics worked, they were unable to take off under their own 
> power. When the Wright brothers designed their aircraft using data from 
> a wind tunnel, their aircraft flew.

Irrelevant example.  This is about architecture and design, not about
implementation.  Intuition, creativity, and gut feelings are always
involved in this.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Tue, 01 Nov 2016 22:42:27 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca,
> > emacs-devel@gnu.org From: Daniel Colascione <dancol@dancol.org>
> > Date: Tue, 1 Nov 2016 13:17:01 -0700
> >
> > > I was not talking about multithreading in general.  I was
> > > talking specifically about Emacs, its coding practices, and its
> > > particular design and needs wrt memory allocation.
> >
> > You categorically stated that memory allocation off the main
> > thread is unsafe.
>
> No, I didn't.

You were saying platforms exist where Emacs runs and malloc() is not
thread safe mere hours ago. I am happy to provide you with your
comments from the archive if you don't recall that.

However, might we take this as meaning that you now agree that
malloc() is indeed thread safe?


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Tue, 1 Nov 2016 22:26:06 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Daniel Colascione <dancol@dancol.org>, raeburn@raeburn.org,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > > You categorically stated that memory allocation off the main
> > > thread is unsafe.
> >
> > No, I didn't.
> 
> You were saying platforms exist where Emacs runs and malloc() is not
> thread safe mere hours ago.

No, I said some of them had thread-related bugs reported as recently
as few years ago.  Which means they only recently became or are
becoming mature enough.

> However, might we take this as meaning that you now agree that
> malloc() is indeed thread safe?

Theoretically, yes.  In practice, see above.

Re: Can we go GTK-only?

[Daniel Colascione]

On November 2, 2016 8:49:30 AM PDT, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Tue, 1 Nov 2016 22:26:06 -0400
>> From: "Perry E. Metzger" <perry@piermont.com>
>> Cc: Daniel Colascione <dancol@dancol.org>, raeburn@raeburn.org,
>>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> 
>> > > You categorically stated that memory allocation off the main
>> > > thread is unsafe.
>> >
>> > No, I didn't.
>> 
>> You were saying platforms exist where Emacs runs and malloc() is not
>> thread safe mere hours ago.
>
>No, I said some of them had thread-related bugs reported as recently
>as few years ago.  Which means they only recently became or are
>becoming mature enough.
>
>> However, might we take this as meaning that you now agree that
>> malloc() is indeed thread safe?
>
>Theoretically, yes.  In practice, see above.

In practice it's safe as well. Nobody else is developing software under the assumption that libc malloc is not thread safe.

Re: Can we go GTK-only?

[YAMAMOTO Mitsuharu]

>>>>> On Tue, 01 Nov 2016 22:42:27 +0200, Eli Zaretskii <eliz@gnu.org> said:

>> Emacs today can tolerate arbitrary memory allocations off the main
>> thread after dumping using the system heap. I know this to be case
>> because Emacs today, on the most common platforms --- OS X,
>> GNU/Linux, and Windows, all perform allocations off the main thread
>> in very common configurations _already_.

> The only platform I know about that starts non-main threads which
> run Emacs application code is MS-Windows, and there we don't
> allocate anything off the heap, at least not off the same heap that
> is used by the main (Lisp) thread.

The font backend driver macfont.m imported from the Mac port does
deallocation off the main thread for some data that were allocated in
the main thread.

  1489	              dispatch_group_async (group, queue, ^{
  1490	                  int nkeys;
  1491	                  uintptr_t key;
  1492	                  nkeys = nkeys_or_perm;
  1493	                  for (key = row * (256 / NGLYPHS_IN_VALUE); ; key++)
  1494	                    if (CFDictionaryContainsKey (dictionary,
  1495	                                                 (const void *) key))
  1496	                      {
  1497	                        CFDictionaryRemoveValue (dictionary,
  1498	                                                 (const void *) key);
  1499	                        if (--nkeys == 0)
  1500	                          break;
  1501	                      }
  1502	                });

I can disable this for NS if it does not fit with the current policy
of GNU Emacs mainline source code.
(You can find more use cases of threads via libdispatch in the Mac
port.  Even basic drawings are done off the main thread.)

Of course, external libraries would do memory allocation/deallocation
across the threads behind the scenes.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Wed, 02 Nov 2016 14:00:49 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: Daniel Colascione <dancol@dancol.org>,
> 	raeburn@raeburn.org,
> 	emacs-devel@gnu.org,
> 	monnier@iro.umontreal.ca,
> 	perry@piermont.com
> 
> > The only platform I know about that starts non-main threads which
> > run Emacs application code is MS-Windows, and there we don't
> > allocate anything off the heap, at least not off the same heap that
> > is used by the main (Lisp) thread.
> 
> The font backend driver macfont.m imported from the Mac port does
> deallocation off the main thread for some data that were allocated in
> the main thread.
> 
>   1489	              dispatch_group_async (group, queue, ^{
>   1490	                  int nkeys;
>   1491	                  uintptr_t key;
>   1492	                  nkeys = nkeys_or_perm;
>   1493	                  for (key = row * (256 / NGLYPHS_IN_VALUE); ; key++)
>   1494	                    if (CFDictionaryContainsKey (dictionary,
>   1495	                                                 (const void *) key))
>   1496	                      {
>   1497	                        CFDictionaryRemoveValue (dictionary,
>   1498	                                                 (const void *) key);
>   1499	                        if (--nkeys == 0)
>   1500	                          break;
>   1501	                      }
>   1502	                });
> 
> I can disable this for NS if it does not fit with the current policy
> of GNU Emacs mainline source code.

I know next to nothing about NS.  These are system APIs, right?  If
so, calling them could be okay, and I defer to NS and OS X people here
to make that call.

One things bothers me, though: can't this arrangement, where data is
allocated in one thread and deallocated in another, cause races?  Or
do these threads have means to synchronize between them?

> (You can find more use cases of threads via libdispatch in the Mac
> port.  Even basic drawings are done off the main thread.)

That's okay, the non-main threads in the MS-Windows build also do some
non-trivial stuff.  As long as there's no QUIT, directly or
indirectly, in those other threads, and no calls that can cons Lisp
objects or cause GC etc., threads are safe.

> Of course, external libraries would do memory allocation/deallocation
> across the threads behind the scenes.

Sure, that's not what bothers me.

Thanks.

Re: Can we go GTK-only?

[YAMAMOTO Mitsuharu]

>>>>> On Wed, 02 Nov 2016 17:46:26 +0200, Eli Zaretskii <eliz@gnu.org> said:

>> 1489	              dispatch_group_async (group, queue, ^{
>> 1490	                  int nkeys;
>> 1491	                  uintptr_t key;
>> 1492	                  nkeys = nkeys_or_perm;
>> 1493	                  for (key = row * (256 / NGLYPHS_IN_VALUE); ; key++)
>> 1494	                    if (CFDictionaryContainsKey (dictionary,
>> 1495	                                                 (const void *) key))
>> 1496	                      {
>> 1497	                        CFDictionaryRemoveValue (dictionary,
>> 1498	                                                 (const void *) key);
>> 1499	                        if (--nkeys == 0)
>> 1500	                          break;
>> 1501	                      }
>> 1502	                });
>> 
>> I can disable this for NS if it does not fit with the current policy
>> of GNU Emacs mainline source code.

> I know next to nothing about NS.  These are system APIs, right?  If
> so, calling them could be okay, and I defer to NS and OS X people here
> to make that call.

They internally use malloc/free.  What if the code like above were
using malloc/free instead of CF functions?

BTW, I just remember the current implementation of xfree should not be
used from a non-main thread if XMALLOC_BLOCK_INPUT_CHECK is defined.
The Mac port contains a code that calls "free" off the main thread for
some data that were allocated in the main thread.

void
mac_free_gc (GC gc)
{
  CGColorRelease (gc->cg_fore_color);
  CGColorRelease (gc->cg_back_color);
  if (gc->clip_rects_data)
    CFRelease (gc->clip_rects_data);
#if defined (XMALLOC_BLOCK_INPUT_CHECK) && DRAWING_USE_GCD
  /* Don't use xfree here, because this might be called in a non-main
     thread.  */
  free (gc);
#else
  xfree (gc);
#endif
}

> One things bothers me, though: can't this arrangement, where data is
> allocated in one thread and deallocated in another, cause races?  Or
> do these threads have means to synchronize between them?

For CFDictionary itself, its fundamental operations are thread-safe,
as well as malloc/free.  For the font data structure that use
CFDictionary, the main thread waits for the other thread after doing
some independent tasks that do not use CFDictionary.

  1543	          if (group)
  1544	            {
  1545	              dispatch_group_wait (group, DISPATCH_TIME_FOREVER);
  1546	              dispatch_release (group);
  1547	            }
  1548	        }

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Can we go GTK-only?

[Eli Zaretskii]

> Date: Thu, 03 Nov 2016 12:43:18 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: dancol@dancol.org,
> 	raeburn@raeburn.org,
> 	emacs-devel@gnu.org,
> 	monnier@iro.umontreal.ca,
> 	perry@piermont.com
> 
> > I know next to nothing about NS.  These are system APIs, right?  If
> > so, calling them could be okay, and I defer to NS and OS X people here
> > to make that call.
> 
> They internally use malloc/free.  What if the code like above were
> using malloc/free instead of CF functions?
> 
> BTW, I just remember the current implementation of xfree should not be
> used from a non-main thread if XMALLOC_BLOCK_INPUT_CHECK is defined.
> The Mac port contains a code that calls "free" off the main thread for
> some data that were allocated in the main thread.

Yes, xmalloc/xfree should only be called from the main thread.  The
problem with calling malloc directly in Emacs is that you need to deal
with failures, which is not trivial in Emacs, whereas xmalloc and
friends has all that figured out already.

> For CFDictionary itself, its fundamental operations are thread-safe,
> as well as malloc/free.  For the font data structure that use
> CFDictionary, the main thread waits for the other thread after doing
> some independent tasks that do not use CFDictionary.

Then I guess this is okay.

Thanks.

Re: Can we go GTK-only?

[Stefan Monnier]

> I was not talking about multithreading in general.  I was talking
> specifically about Emacs, its coding practices, and its particular
> design and needs wrt memory allocation.  I named several factors that

OK, so we agree that Emacs can assume that on any system where we can
use multithreading we can assume that the system's malloc will work just
right.  So the only potential difficulties come from Emacs itself,
i.e. things over which *we* have control.


        Stefan

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: "Perry E. Metzger" <perry@piermont.com>,  dancol@dancol.org,  raeburn@raeburn.org,  emacs-devel@gnu.org
> Date: Tue, 01 Nov 2016 20:27:49 -0400
> 
> > I was not talking about multithreading in general.  I was talking
> > specifically about Emacs, its coding practices, and its particular
> > design and needs wrt memory allocation.  I named several factors that
> 
> OK, so we agree that Emacs can assume that on any system where we can
> use multithreading we can assume that the system's malloc will work just
> right.

"Should", not "will".  And on some systems, only with very recent
library versions.

> So the only potential difficulties come from Emacs itself,
> i.e. things over which *we* have control.

Most of it, yes.  As I was trying to tell in my OP.

Re: Can we go GTK-only?

[Stefan Monnier]

> "Should", not "will".  And on some systems, only with very recent
> library versions.

Again: which system are you think of that's had a buggy
malloc-with-threads in the last, say, ten years?


        Stefan

Re: Can we go GTK-only?

[Nikolaus Rath]

On Nov 02 2016, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> "Should", not "will".  And on some systems, only with very recent
>> library versions.
>
> Again: which system are you think of that's had a buggy
> malloc-with-threads in the last, say, ten years?

GNU libc on Linux in 2008, it seems:

https://sourceware.org/bugzilla/show_bug.cgi?id=6952

(Just providing the link, personally I think that even with such bugs it
it safe to assume a threadsafe malloc both then and now. Otherwise you
may just as well assume that nothing works anywhere at all).

Best,
Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Can we go GTK-only?

[Nikolaus Rath]

On Nov 02 2016, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> "Should", not "will".  And on some systems, only with very recent
>> library versions.
>
> Again: which system are you think of that's had a buggy
> malloc-with-threads in the last, say, ten years?

GNU libc on Linux in 2008, it seems:

https://sourceware.org/bugzilla/show_bug.cgi?id=6952

Best,
Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Can we go GTK-only?

[Paul Eggert]

On 11/02/2016 12:25 PM, Nikolaus Rath wrote:
> GNU libc on Linux in 2008, it seems:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=6952

It appears that bug was fixed in glibc 2.8 (2008). I don't think this 
old bug should affect decisions of how Emacs should evolve from here on. 
If anybody still wants to build on a pre-2008 glibc system, they can 
configure without GTK.

Re: Can we go GTK-only?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > GNU libc on Linux in 2008, it seems:

Since Linux is just the kernel, not the whole system,
this is not "on Linux".  In this case, GNU libc is running
on (and as part of) the GNU/Linux system.

Even when you're talking about a past bug, or a present bug, we still
deserve to get some of the credit when you talk about the GNU system.
So would you please call it "GNU/Linux", never just "Linux"?

See http://gnu.org/gnu/linux-and-gnu.html and
http://gnu.org/gnu/gnu-linux-faq.html, plus the history in
http://gnu.org/gnu/the-gnu-project.html.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Can we go GTK-only?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: perry@piermont.com,  dancol@dancol.org,  raeburn@raeburn.org,  emacs-devel@gnu.org
> Date: Wed, 02 Nov 2016 12:04:48 -0400
> 
> > "Should", not "will".  And on some systems, only with very recent
> > library versions.
> 
> Again: which system are you think of that's had a buggy
> malloc-with-threads in the last, say, ten years?

Even glibc had a bug.  And one or 2 of *BSDs.

Re: Can we go GTK-only?

[Perry E. Metzger]

On Wed, 02 Nov 2016 22:13:13 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: perry@piermont.com,  dancol@dancol.org,
> > raeburn@raeburn.org,  emacs-devel@gnu.org Date: Wed, 02 Nov 2016
> > 12:04:48 -0400 
> > > "Should", not "will".  And on some systems, only with very
> > > recent library versions.  
> > 
> > Again: which system are you think of that's had a buggy
> > malloc-with-threads in the last, say, ten years?  
> 
> Even glibc had a bug.  And one or 2 of *BSDs.
> 

Which BSD? Name. Date. And yes, glibc had a bug in free. It caused a
memory leak, not a crash, and it was fixed.

-- 
Perry E. Metzger		perry@piermont.com

Re: Can we go GTK-only?

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

PM> You've explicitly refused to name an exception, and no one else is
PM> aware of one, so how can we give credence to your claim?

EZ> As for "claims": this is more about gut feelings, based on the factors I
EZ> mentioned, than about anything else. It is OK to disagree with gut
EZ> feelings, even if you agree with the facts. It is NOT okay to make this a
EZ> discussion about my credibility. If my credibility is being questioned, I
EZ> will simply step down.

Please do not discourage our co-maintainer by speaking disrespectfully on this
mailing list!!! Those who do so will be moderated out of the discussion. I'd
much rather have Eli stick around, than see a minor technical point championed
at the expense of one of our most stalwart contributors.

To everyone: If you can't phrase your disagreements and observations in calm,
rational language, then please refrain. We should be speaking to each other as
though we were what we are: computer professionals who know what we're doing,
and who all want Emacs to succeed.

Claims backed up by data argue well here; attacking the position or
credibility of others not only fails to win the point, it loses big on morale,
which is much more precious commodity. We're a small community; there is no
need for this.

Eli, please do not step down. You are my ray of light against a dark sea of
never-ending bug reports.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Can we go GTK-only?

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

>>>>> John Wiegley <johnw@gnu.org> writes:

> To everyone: If you can't phrase your disagreements and observations in
> calm, rational language, then please refrain. We should be speaking to each
> other as though we were what we are: computer professionals who know what
> we're doing, and who all want Emacs to succeed.

Hi everyone,

It was brought up to me by a kind soul on IRC that my response to this thread
was very one-sided, as if I were defending only Eli and, by implication,
saying that others in this thread (Paul, Daniel, Perry, Stefan, primarily)
were the ones being defensive and disrespectful.

I did not come here to say that Eli was right. I came to say that discouraging
a contributor is *always wrong*. I love all of you, because you give your time
and energy freely to make my favorite piece of software in the world work
well. I don't want to see anyone become discouraged, ever. And when it
happens, I will fight back.

Each side has valid points in this discussion, but we are not making progress.
Eli has concerns, which automatically means that I share those concerns,
because this is how we work together. Others want evidence, and I agree that
we should find this evidence. If malloc is, or is not, thread-safe enough for
Emacs, is a great question that I'd like to see resolved.

I apologize if I came across as saying that everyone else should just pack up
and go home because the Great Maintainer Hath Spoken. My argument is with the
damaging nature of tone. We can too easily damage enthusiasm on this mailing
list, where tempers flare and words come cheap. Yet enthusiasm is the most
valuable commodity we have, since volunteerism is our only path to progress.

So to Paul, Daniel, Perry, Stefan and the rest: I hear you, you don't want us
to reject the use of malloc out of hand. To Eli: I hear you, experience has
given you pause, and while you don't have all the URLs ready at hand, you
don't want us to ignore your gut just yet.

Perhaps we can return to the original thread, asking if we can go GTK-only,
since it was going so well for 34 messages, before these next 61 took us down
a darker road.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 625 bytes --]