Should this package be included into the NS port?

Should this package be included into the NS port?

[George Plymale II]

Hi all,

I recently submitted a couple of patches to the Emacs Mac Port by
Mitsuharu Yamamoto. The pull request can be seen here:
https://bitbucket.org/mituharu/emacs-mac/pull-requests/2/add-multi-tty-support-to-be-on-par-with/diff

These patches were basically the continuation of this thread that I
started back in January:
https://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00373.html

Yet, there is a problem with all this which is mentioned in Alan Third's
message on that thread:

> [...] if you do this on the NS port:

>      Open GUI instance
>      start server
>      run ‘emacsclient -t’ in the terminal
>      close GUI frame

> You can’t open a new GUI frame, even though the GUI instance with the
> server is still running. You can see it in the dock, and access the
> menus, but I can’t find any way to create a new frame.

( https://lists.gnu.org/archive/html/emacs-devel/2018-01/msg00430.html )

The same issue occurs with my patches to the Mac Port and it seems to be
a problem with macOS's dock. Fortunately, there is an Elisp package by
Ryan C. Thompson which addresses this problem:
https://github.com/DarwinAwardWinner/osx-pseudo-daemon

This package implements a "pseudo-daemon" which is described as such:

> If you've ever tried to use Emacs in daemon mode on Mac OS, you might
> have noticed that after you close the last graphical Emacs client
> frame, the Emacs dock icon and menu bar become non-functional until
> you create a new graphical frame. This package implements nearly
> identical behavior to daemon mode using a simple hack: whenever the
> last graphical frame is closed, a new hidden frame is created. The
> next time Emacs is activated, the hidden frame is revealed. The result
> is essentially the same as using daemon mode, but without the
> drawbacks.

I actually have been using this package constantly for a couple of years
now and it is a very reliable hack which basically solves this issue. I
asked Yamamoto in my above pull request whether he would want to include
this package in the Mac Port in order to solve the issue (re)introduced
by my patches, and he suggested that I present it to the Emacs upstream
first. In his words:

> How about proposing inclusion of osx-pseudo-daemon upstream (for the
> NS port, of course)? I'd like to avoid enabling premature/unstable
> multi-tty with GUI as well as diverging from upstream for easier
> maintenance.

So here I am :)

Ryan C. Thompson's package is licensed under the GPLv3 so I suppose that
it could be included into the NS port if he signs the copyright
paperwork. It may also need a bit more firming-up before heading into
the mainline, but I think it's pretty solid as-is.

Thoughts?

Thanks,
- George Plymale II

Re: Should this package be included into the NS port?

[Stefan Monnier]

> Ryan C. Thompson's package is licensed under the GPLv3 so I suppose that
> it could be included into the NS port if he signs the copyright
> paperwork. It may also need a bit more firming-up before heading into
> the mainline, but I think it's pretty solid as-is.

The Gtk port has a somewhat similar hack in the C code to avoid closing
the display (to try and delay the infamous multi-display Gtk bug).

So I think it might make sense to add such a workaround for this OSX bug
much more directly in the core (rather than via defadvice).


        Stefan

Re: Should this package be included into the NS port?

[Alan Third]

On Tue, May 15, 2018 at 02:25:30PM -0400, Stefan Monnier wrote:
> > Ryan C. Thompson's package is licensed under the GPLv3 so I suppose that
> > it could be included into the NS port if he signs the copyright
> > paperwork. It may also need a bit more firming-up before heading into
> > the mainline, but I think it's pretty solid as-is.
> 
> The Gtk port has a somewhat similar hack in the C code to avoid closing
> the display (to try and delay the infamous multi-display Gtk bug).
> 
> So I think it might make sense to add such a workaround for this OSX bug
> much more directly in the core (rather than via defadvice).

It might be worth investigating exactly why the NS port can’t open a
new frame in this situation, but I’ve not got round to it yet. It
seems rather strange to me.
-- 
Alan Third

Re: Should this package be included into the NS port?

[George Plymale II]

Alan Third <alan@idiocy.org> writes:

> It might be worth investigating exactly why the NS port can’t open a
> new frame in this situation, but I’ve not got round to it yet. It
> seems rather strange to me.

Yes, I agree. I would also very much like to know the cause of this bug,
but alas I'm a bit strapped for time as well. I've done some casual,
fruitless experimentation with it in the past yet I've gotten no closer
to the cause. If anyone would be willing to take up the torch for this
and do some detective work, that would be awesome.

Thanks,
- George Plymale II

Re: Should this package be included into the NS port?

[Alan Third]

On Tue, May 15, 2018 at 10:48:20PM -0400, George Plymale II wrote:
> Alan Third <alan@idiocy.org> writes:
> 
> > It might be worth investigating exactly why the NS port can’t open a
> > new frame in this situation, but I’ve not got round to it yet. It
> > seems rather strange to me.
> 
> Yes, I agree. I would also very much like to know the cause of this bug,
> but alas I'm a bit strapped for time as well. I've done some casual,
> fruitless experimentation with it in the past yet I've gotten no closer
> to the cause. If anyone would be willing to take up the torch for this
> and do some detective work, that would be awesome.

I’ve had a dig around and it looks as though emacsclient is able to
open a new GUI frame, so it’s not quite as dire as I’d previously
thought.

There are two problems, though.

* Application Menu

When the last NS frame is deleted the menus aren’t updated, so I think
they’ll be the menus as defined for the last frame. They should
probably be cut right back to just the ‘Emacs’ menu, and perhaps
‘help’.

(NOTE: The menus are a bit of a mess, Nick Helm and I have been
discussing them in bug#31371.)

* Dock Menu

This has a ‘new frame’ option but just plain doesn’t work. I think the
problem is the way it’s trying to create a new frame:

    - (void)newFrame: (id)sender
    {
      NSTRACE ("[EmacsApp newFrame:]");
    
      struct frame *emacsframe = SELECTED_FRAME ();
      NSEvent *theEvent = [NSApp currentEvent];
    
      if (!emacs_event)
        return;
      emacs_event->kind = NS_NONKEY_EVENT;
      emacs_event->code = KEY_NS_NEW_FRAME;
      emacs_event->modifiers = 0;
      EV_TRAILER (theEvent);
    }

This seems to rely on there being an existing NS frame, but we’ve
deleted the last one. I assume if we were clever we’d be able to use
the, still open, terminal frame.

Is there some other way of creating a new GUI frame? I assume there
must be since emacsclient can do it.

* ALSO!

Closing the terminal frame when there are no GUI frames left results
in a crash. I guess Emacs doesn’t think it needs to clean up the NS
stuff or something? I don’t really know what’s going on there.
-- 
Alan Third

Re: Should this package be included into the NS port?

[George Plymale II]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

> I’ve had a dig around and it looks as though emacsclient is able to
> open a new GUI frame, so it’s not quite as dire as I’d previously
> thought.

Oh, neat. I hadn't tested that, so that's good to know.

> There are two problems, though.

> * Application Menu

> When the last NS frame is deleted the menus aren’t updated, so I think
> they’ll be the menus as defined for the last frame. They should
> probably be cut right back to just the ‘Emacs’ menu, and perhaps
> ‘help’.

Won't those menus just get updated once a new frame is created? I'm not
exactly sure what the implications of this are.

> (NOTE: The menus are a bit of a mess, Nick Helm and I have been
> discussing them in bug#31371.)

I read your discussion at bug#31371 ( here's link for others' reference:
https://lists.gnu.org/archive/html/bug-gnu-emacs/2018-05/msg00149.html ).
Strangely, this bug doesn't occur at all on the Emacs Mac Port. The Help
menu search actually works fantastically on the Mac Port. It can show
info nodes, menu items, and it actually works when you click on
them. Here's a screenshot: 

[-- Attachment #2: Help menu on the Emacs Mac Port --]
[-- Type: image/png, Size: 48332 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1712 bytes --]



There's no freezing, no focus issues, nothing liked what you guys
described in bug#31371. It works completely as expected.

> * Dock Menu

> This has a ‘new frame’ option but just plain doesn’t work. I think the
> problem is the way it’s trying to create a new frame:

>     - (void)newFrame: (id)sender
>     {
>       NSTRACE ("[EmacsApp newFrame:]");
    
>       struct frame *emacsframe = SELECTED_FRAME ();
>       NSEvent *theEvent = [NSApp currentEvent];
    
>       if (!emacs_event)
>         return;
>       emacs_event->kind = NS_NONKEY_EVENT;
>       emacs_event->code = KEY_NS_NEW_FRAME;
>       emacs_event->modifiers = 0;
>       EV_TRAILER (theEvent);
>     }

> This seems to rely on there being an existing NS frame, but we’ve
> deleted the last one. I assume if we were clever we’d be able to use
> the, still open, terminal frame.

> Is there some other way of creating a new GUI frame? I assume there
> must be since emacsclient can do it.

I suppose you must be right that there's another way to do it, but I
don't know what that would be. I could try and dig around in the code if
you'd like.

> * ALSO!

> Closing the terminal frame when there are no GUI frames left results
> in a crash. I guess Emacs doesn’t think it needs to clean up the NS
> stuff or something? I don’t really know what’s going on there.

Strange, I don't see a "crash" in the Emacs Mac Port. I don't notice any
abnormal exit status, error, or system popup, if that's what you mean. I
don't know what could be going on there either, but I'm tending to think
that we need to follow the Emacs Mac Port's example in at least some of
these instances.

Re: Should this package be included into the NS port?

[Nick Helm]

On Sat, 19 May 2018 at 09:21:38 +1200, George Plymale II wrote:

> I read your discussion at bug#31371 ( here's link for others' reference:
> https://lists.gnu.org/archive/html/bug-gnu-emacs/2018-05/msg00149.html ).
> Strangely, this bug doesn't occur at all on the Emacs Mac Port. The Help
> menu search actually works fantastically on the Mac Port. It can show
> info nodes, menu items, and it actually works when you click on
> them. Here's a screenshot: 
>
> There's no freezing, no focus issues, nothing liked what you guys
> described in bug#31371. It works completely as expected.

You're right, the mac-port doesn't exhibit this particular bug. However,
it's not perfect in this area either, due to it using many of the same
techniques. (Your screenshot actually illustrates my point – the
Lisp-derived menu items shouldn't be visible while the Spotlight for
Help search field is active. You'll also likely find that this issue
doesn't happen every time you open Help).

IMO, the mac-port is a good option, but NS is catching up and has even
leapt ahead in some keys ways in the last year or so.

Re: Should this package be included into the NS port?

[George Plymale II]

Nick Helm <nick@tenpoint.co.nz> writes:

> You're right, the mac-port doesn't exhibit this particular bug. However,
> it's not perfect in this area either, due to it using many of the same
> techniques. (Your screenshot actually illustrates my point – the
> Lisp-derived menu items shouldn't be visible while the Spotlight for
> Help search field is active.

I'm a little confused by this -- you're saying that the info nodes
(which I suppose are the Lisp-derived menu items) should _not_ be
visible?

> You'll also likely find that this issue doesn't happen every time you
> open Help).

Which issue are you referring to?

> IMO, the mac-port is a good option, but NS is catching up and has even
> leapt ahead in some keys ways in the last year or so.

Just out of curiosity, in what ways has NS leapt ahead?

Re: Should this package be included into the NS port?

[Nick Helm]

On Sun, 20 May 2018 at 03:49:01 +1200, George Plymale II wrote:

> Nick Helm <nick@tenpoint.co.nz> writes:
>
>> You're right, the mac-port doesn't exhibit this particular bug. However,
>> it's not perfect in this area either, due to it using many of the same
>> techniques. (Your screenshot actually illustrates my point – the
>> Lisp-derived menu items shouldn't be visible while the Spotlight for
>> Help search field is active.
>
> I'm a little confused by this -- you're saying that the info nodes
> (which I suppose are the Lisp-derived menu items) should _not_ be
> visible?

Sorry, I didn't explain it very well. No, the help book topics should be
there. I'm talking about the menu items – everything from 'Send Emacs
Feedback to Apple' and below.  

As I understand it, macOS tries to hide these menu items when the help
search field is populated and only shows them again when the field
becomes empty. But on Emacs they can get added back in when the menu-bar
updates.

Re: Should this package be included into the NS port?

[George Plymale II]

[I had to resend this message because my attached screencast was too big to be
accepted onto the mailing list. I apologize if this creates duplicate entries in
your inboxes.]


Nick Helm <nick@tenpoint.co.nz> writes:

> Sorry, I didn't explain it very well. No, the help book topics should be
> there. I'm talking about the menu items – everything from 'Send Emacs
> Feedback to Apple' and below.  
>
> As I understand it, macOS tries to hide these menu items when the help
> search field is populated and only shows them again when the field
> becomes empty. But on Emacs they can get added back in when the menu-bar
> updates.

Oh, okay. I see what you're saying. It's funny that you mention all of this
because I actually noticed right after posting my screenshot, that my screenshot
is kind of misleading. And I didn't even know it was misleading until playing
around with Help a bit more. Now I understand what was going on, thanks to your
remarks. But I think it behooves me to post what the actual precise behavior was
since you've discussed what caused it.

The menu items from 'Send Emacs Feedback to Apple' and below actually only
appear after removing focus from Help and when the search field is empty, as you
say. I.e., the Mac Port seems to actually kind of get it right but not 100% of
the time (contrary to what is suggested by my earlier screenshot). Here's a
screencast to show precisely what happens with the Help menu on the Mac Port
(since a screenshot seems insufficient for this):
https://cl.ly/2L2p0J0k053n/Help%20menu%20weirdness.mp4

As a bit of an aside, the Mac Port seems to get a bit screwed up when you try to
do certain things while the Help search field is not empty. For example:

- While trying to record this screencast, I got Emacs to hang when I clicked
Help (and the search field was not empty).
- I also got it to hang in another session by having something in the search
field and typing "/scp:" in `find-file' (doing a completely unrelated task).
- Furthermore, I got it to crash by repeatedly clicking Help when an item was in
the search field (you can see that mentioned in the screencast itself).

Strange stuff, but I guess you're right -- the Mac Port is not perfect in this
area.

I guess that the takeaway here is that Emacs is just all-around weird on
macOS. If we really want to get to the root of some of these problems, it would
probably be ideal to combine work from both the NS port and the Mac Port and see
if we can't somehow get Emacs to behave more like a regular Mac app
internally. I know that Alan and I already discussed earlier in the thread why
this may not be possible (and may not even be accepted by RMS since he has
previously expressed rejection of any Emacs features on OS X which are not also
present on Linux [and there may even be licensing concerns, but I'm not sure
about this]), but this really does seem to be the ideal solution. Having a Mac
Port and an NS port seems kind of silly esp. when we're dealing with issues like
this where we would greatly benefit from unity, not fragmentation and dealing
with the problem(s) separately. Once again, just my 2¢

Thanks,
- George Plymale II

Re: Should this package be included into the NS port?

[Alan Third]

On Fri, May 18, 2018 at 05:21:38PM -0400, George Plymale II wrote:
> > * Application Menu
> 
> > When the last NS frame is deleted the menus aren’t updated, so I think
> > they’ll be the menus as defined for the last frame. They should
> > probably be cut right back to just the ‘Emacs’ menu, and perhaps
> > ‘help’.
> 
> Won't those menus just get updated once a new frame is created? I'm not
> exactly sure what the implications of this are.

Yes, they will, but it’s still messy. You end up with menus that do
absolutely nothing.

> > Is there some other way of creating a new GUI frame? I assume there
> > must be since emacsclient can do it.
> 
> I suppose you must be right that there's another way to do it, but I
> don't know what that would be. I could try and dig around in the code if
> you'd like.

I was hoping someone who knew would read this and give me a pointer in
the right direction. If not I’ll get round to it at some point. :)

> > * ALSO!
> 
> > Closing the terminal frame when there are no GUI frames left results
> > in a crash. I guess Emacs doesn’t think it needs to clean up the NS
> > stuff or something? I don’t really know what’s going on there.
> 
> Strange, I don't see a "crash" in the Emacs Mac Port. I don't notice any
> abnormal exit status, error, or system popup, if that's what you mean. I
> don't know what could be going on there either, but I'm tending to think
> that we need to follow the Emacs Mac Port's example in at least some of
> these instances.

One thing to bear in mind is that the Mac port is a completely
different code base than the NS port. The Mac port was originally
written for the Carbon API, and the NS port for NeXTSTEP (now Cocoa
and GNUstep).

They do share some things (most notably macfont.m), but it’s not a
given that something broken in one will be broken in the other, or
that the code can be copied directly between them.
-- 
Alan Third

Re: Should this package be included into the NS port?

[George Plymale II]

> Yes, they will, but it’s still messy. You end up with menus that do
> absolutely nothing.

Oh, okay.

> I was hoping someone who knew would read this and give me a pointer in
> the right direction. If not I’ll get round to it at some point. :)

Oh, LOL, sorry.

> One thing to bear in mind is that the Mac port is a completely
> different code base than the NS port. The Mac port was originally
> written for the Carbon API, and the NS port for NeXTSTEP (now Cocoa
> and GNUstep).
>
> They do share some things (most notably macfont.m), but it’s not a
> given that something broken in one will be broken in the other, or
> that the code can be copied directly between them.

Right, I am aware of such differences. I just thought that perhaps some
of the Mac Port's code could be somehow imitated in the NS port to solve
this particular problem. But I can completely understand why that may
not be viable without changing a lot of parts in the NS port.

If it's not doable, that's fine. I'm just throwing suggestions out to
see what sticks.

- George Plymale II

Re: Should this package be included into the NS port?

[Stefan Monnier]

>> > Is there some other way of creating a new GUI frame? I assume there
>> > must be since emacsclient can do it.
>> I suppose you must be right that there's another way to do it, but I
>> don't know what that would be.  I could try and dig around in the code if
>> you'd like.

AFAIK server.el uses make-frame-on-display.  Don't know if that has been
made to do something useful in the NS port.


        Stefan

Re: Should this package be included into the NS port?

[George Plymale II]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> AFAIK server.el uses make-frame-on-display.  Don't know if that has been
> made to do something useful in the NS port.

Thanks for the info.

Alan, does that answer your question? I'm not sure if we are using
make-frame-on-display in the NS port.

- George Plymale II

Re: Should this package be included into the NS port?

[Van L]

> George Plymale II writes:
> 
>  in the NS port

Can the Emacs 26.1 release document readme steps for likely best results building your own NS port? and mention the Mac port.

Re: Should this package be included into the NS port?

[Alan Third]

On Tue, May 22, 2018 at 11:48:46AM +1000, Van L wrote:
> 
> > George Plymale II writes:
> > 
> >  in the NS port
> 
> Can the Emacs 26.1 release document readme steps for likely best
> results building your own NS port?

What sort of thing do you have in mind?

> and mention the Mac port.

Seems unlikely given it’s not part of GNU Emacs.
-- 
Alan Third

Off Topic (was: Should this package be included into the NS port?)

[Van L]

> Alan Third writes:
> 
> Van L writes:
>> 
>>> George Plymale II writes:
>>> 
>>> in the NS port
>> 
>> Can the Emacs 26.1 release document readme steps for likely best
>> results building your own NS port?
> 
> What sort of thing do you have in mind?

Something to improve and explain the configure options specific to a great experience on the platform. I use the following by ritural trial and error and don’t know why. I couldn’t put the sicp.info file in /usr/share/info despite su-ing and don’t know why.

  #+BEGIN_EXAMPLE
    CC=clang ./configure \
	--disable-acl \
	--prefix="$HOME/2018/Applications" \
	--with-xpm \
	--with-jpeg \
	--with-tiff \
	--with-gif \
	--with-png \
	--with-xml2 \
	--with-imagemagick \
	--with-xft \
	--with-libotf \
	--with-ns \
	--with-zlib \
	--without-selinux \
	--without-gpm \
	--without-dbus \
	--without-gsettings \
	--without-libsystemd \
	--with-mailutils \
	--with-x \
	--with-x-toolkit="lucid" \
    #    --without-compress-install \
    #    --without-x \
    #    --disable-ns-self-contained \
    #    --without-gnutls \
    #    --with-modules \

  #+END_EXAMPLE

>> and mention the Mac port.
> 
> Seems unlikely given it’s not part of GNU Emacs.

Detailing where and why Mac port does better than the GNU Emacs sets a target to better.

Re: Off Topic (was: Should this package be included into the NS port?)

[Alan Third]

On Wed, May 23, 2018 at 12:30:08PM +1000, Van L wrote:
> Something to improve and explain the configure options specific to a
> great experience on the platform. I use the following by ritural
> trial and error and don’t know why. I couldn’t put the sicp.info
> file in /usr/share/info despite su-ing and don’t know why.
> 
>   #+BEGIN_EXAMPLE
>     CC=clang ./configure \
> 	--disable-acl \
> 	--prefix="$HOME/2018/Applications" \
> 	--with-xpm \
> 	--with-jpeg \
> 	--with-tiff \
> 	--with-gif \
> 	--with-png \
> 	--with-xml2 \
> 	--with-imagemagick \
> 	--with-xft \
> 	--with-libotf \
> 	--with-ns \
> 	--with-zlib \
> 	--without-selinux \
> 	--without-gpm \
> 	--without-dbus \
> 	--without-gsettings \
> 	--without-libsystemd \
> 	--with-mailutils \
> 	--with-x \
> 	--with-x-toolkit="lucid" \
>     #    --without-compress-install \
>     #    --without-x \
>     #    --disable-ns-self-contained \
>     #    --without-gnutls \
>     #    --with-modules \
> 
>   #+END_EXAMPLE

I’ve seen this kind of incantation used before and I have to be
honest, I’ve no idea why. I just use:

    ./configure

and it works fine. I notice you’re building with both --with-ns and
--with-x, and I’ve no idea what actually happens in that case...

> >> and mention the Mac port.
> > 
> > Seems unlikely given it’s not part of GNU Emacs.
> 
> Detailing where and why Mac port does better than the GNU Emacs sets a target to better.

IMO it’s better to raise bug reports about things that don’t work so
well in the NS port with concrete examples.

BTW you might find the TODO file interesting, it discusses a number of
issues in the NS port.
-- 
Alan Third

emacs-26.1-rc1: ./configure (was: Off Topic)

[Van L]

> Alan Third writes:
> 
> I have to be
> honest, I’ve no idea why. I just use:
> 
>    ./configure
> 
> and it works fine.

Er, not really for me, I get at the end of tag release emacs-26.1-rc1’s

  ./configure

  configure: WARNING: This configuration installs a 'movemail' program
  that does not retrieve POP3 email.  By default, Emacs 25 and earlier
  installed a 'movemail' program that retrieved POP3 email via only
  insecure channels, a practice that is no longer recommended but that
  you can continue to support by using './configure --with-pop'.
  configure: You might want to install GNU Mailutils
  <https://mailutils.org> and use './configure --with-mailutils’.

and that might’ve pushed me to explore configure’s switches, by trial and error.

Re: emacs-26.1-rc1: ./configure

[Robert Pluim]

Van L <van@scratch.space> writes:

>> Alan Third writes:
>> 
>> I have to be
>> honest, I’ve no idea why. I just use:
>> 
>>    ./configure
>> 
>> and it works fine.
>
> Er, not really for me, I get at the end of tag release emacs-26.1-rc1’s
>
>   ./configure
>
>   configure: WARNING: This configuration installs a 'movemail' program
>   that does not retrieve POP3 email.  By default, Emacs 25 and earlier
>   installed a 'movemail' program that retrieved POP3 email via only
>   insecure channels, a practice that is no longer recommended but that
>   you can continue to support by using './configure --with-pop'.
>   configure: You might want to install GNU Mailutils
>   <https://mailutils.org> and use './configure --with-mailutils’.
>
> and that might’ve pushed me to explore configure’s switches, by trial and error.

The intent was to push you to install GNU Mailutils, since thatʼs a
better replacement for Emacs's built-in movemail. Can we improve the message?

Robert

Re: emacs-26.1-rc1: ./configure

[Van L]

> Robert Pluim writes:
> 
> The intent was to push you to install GNU Mailutils, since thatʼs a
> better replacement for Emacs's built-in movemail. Can we improve the message?

The message could point to the place where the TODO file is for NS Port and leave instructions for how to get email going inside of Emacs.

I have the following details to fill in the gaps:

https://wiki.gandi.net/en/mail/imap

I couldn’t make up my mind which to pick Rmail or Gnus. 

Gnus is newer? and I wanted to try newsserver but it wasn’t obvious where my Internet provider put that.

Re: emacs-26.1-rc1: ./configure

[Robert Pluim]

Van L <van@scratch.space> writes:

>> Robert Pluim writes:
>> 
>> The intent was to push you to install GNU Mailutils, since thatʼs a
>> better replacement for Emacs's built-in movemail. Can we improve the message?
>
> The message could point to the place where the TODO file is for NS Port and leave instructions for how to get email going inside of Emacs.
>

I meant specifically the message about Mailutils.

> I have the following details to fill in the gaps:
>
> https://wiki.gandi.net/en/mail/imap
>
> I couldn’t make up my mind which to pick Rmail or Gnus. 
>
> Gnus is newer? and I wanted to try newsserver but it wasn’t obvious where my Internet provider put that.

Those are all valid questions, but the answers should not be in the
output of ./configure

Robert

Re: emacs-26.1-rc1: ./configure

[Van L]

> Robert Pluim writes:
> 
> I meant specifically the message about Mailutils.
> 
> … the answers should not be in the
> output of ./configure

I figured Mailutils is too deep for me and about privacy paranoia.

The output of ./configure should simply have a one line hint pointing to the next-step file for mail related fine tuning.

Re: emacs-26.1-rc1: ./configure

[Van L]

>> Can we improve the message?

Provide the mechanism:

  ./info

Re: Should this package be included into the NS port?

[Alan Third]

On Mon, May 21, 2018 at 09:42:32PM -0400, George Plymale II wrote:
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> 
> > AFAIK server.el uses make-frame-on-display.  Don't know if that has been
> > made to do something useful in the NS port.
> 
> Thanks for the info.
> 
> Alan, does that answer your question? I'm not sure if we are using
> make-frame-on-display in the NS port.

Sorry, yes. I’ve only got limited time for Emacs at the moment, so I’m
being a bit slow in responding.

This does help, though.

It seems that if you end up in the situation where you’ve only got the
terminal left you can use

    (make-frame-on-display (car (x-display-list)))

to create a new GUI frame, but the problem now is that the Emacs event
the dock menu fires (ns-new-frame) is only available in NS frames (or
something, I’m unclear), so if the current frame is a terminal frame,
and you click on ‘new frame’ in the dock menu, the terminal displays

    <268435468> is undefined

I assume what we need to do is define the ns-new-frame event somewhere
other than nsterm.h, which is where it is now.

This all sounds wrong to me, tbh. But I don’t know of any other way we
can run lisp from within the NSApp run loop than by firing off events
or setting a flag and catching it later.

-- 
Alan Third

Re: Should this package be included into the NS port?

[George Plymale II]

Alan Third <alan@idiocy.org> writes:

> Sorry, yes. I’ve only got limited time for Emacs at the moment, so I’m
> being a bit slow in responding.

No problem, I was just unsure if you saw Stefan's message.

> This does help, though.
>
> It seems that if you end up in the situation where you’ve only got the
> terminal left you can use
>
>     (make-frame-on-display (car (x-display-list)))
>
> to create a new GUI frame, but the problem now is that the Emacs event
> the dock menu fires (ns-new-frame) is only available in NS frames (or
> something, I’m unclear), so if the current frame is a terminal frame,
> and you click on ‘new frame’ in the dock menu, the terminal displays
>
>     <268435468> is undefined
>
> I assume what we need to do is define the ns-new-frame event somewhere
> other than nsterm.h, which is where it is now.
>
> This all sounds wrong to me, tbh. But I don’t know of any other way we
> can run lisp from within the NSApp run loop than by firing off events
> or setting a flag and catching it later.

Hmm, very interesting. Yes, all of what you're saying makes sense
although it does perhaps seem a bit "wrong" as you say. Nonetheless, a
"wrong" solution may be better than no solution. Just my 2¢

- George Plymale II

Re: Should this package be included into the NS port?

[Nick Helm]

On Sat, 19 May 2018 at 07:36:32 +1200, Alan Third wrote:

> There are two problems, though.
>
> * Application Menu
>
> When the last NS frame is deleted the menus aren’t updated, so I think
> they’ll be the menus as defined for the last frame. They should
> probably be cut right back to just the ‘Emacs’ menu, and perhaps
> ‘help’.

This is interesting. I think we could tweak the EmacsMenu side of things
to do this, even with the current code. We'd want to make sure to
include some kind of new frame command somewhere.

These menus are derived dynamically from Lisp though, and it might be
difficult to convince it to provide valid entries when no frame is
selected. 

One way around that might be to create a new menu containing
static NSMenu versions of File and Help, much like we do for the appMenu
now. When no frame is selected and there are no Lisp menu entries, we
switch mainMenu to this new menu. When a frame is created, we switch
back. 

A side benefit would be that we can set File and Help to anything
we want, e.g. add a new frame command and fix up the quit duplication.

> * Dock Menu
>
> This has a ‘new frame’ option but just plain doesn’t work. I think the
> problem is the way it’s trying to create a new frame:...
>
> This seems to rely on there being an existing NS frame, but we’ve
> deleted the last one. I assume if we were clever we’d be able to use
> the, still open, terminal frame.

With all of this discussion, I feel like I'm missing something really
fundamental. Putting emacsclient aside for a moment, if we were dealing
with a standard Cocoa app, the window server and NSApp loop would keep
running in the background and we could close everything, then create a
new frame (as a view on an NSWindow instance) as needed.

How is the architecture of Emacs on NS so different from standard Cocoa
that these kinds of workarounds are necessary? 

Re: Should this package be included into the NS port?

[Alan Third]

On Sat, May 19, 2018 at 04:42:54PM +1200, Nick Helm wrote:
> On Sat, 19 May 2018 at 07:36:32 +1200, Alan Third wrote:
> 
> > There are two problems, though.
> >
> > * Application Menu
> >
> > When the last NS frame is deleted the menus aren’t updated, so I think
> > they’ll be the menus as defined for the last frame. They should
> > probably be cut right back to just the ‘Emacs’ menu, and perhaps
> > ‘help’.
> 
> This is interesting. I think we could tweak the EmacsMenu side of things
> to do this, even with the current code. We'd want to make sure to
> include some kind of new frame command somewhere.
> 
> These menus are derived dynamically from Lisp though, and it might be
> difficult to convince it to provide valid entries when no frame is
> selected. 

I’m not sure if the ‘Emacs’ menu is derived from lisp. If you turn off
menus completely it still exists.

> One way around that might be to create a new menu containing
> static NSMenu versions of File and Help, much like we do for the appMenu
> now. When no frame is selected and there are no Lisp menu entries, we
> switch mainMenu to this new menu. When a frame is created, we switch
> back. 

Perhaps we could modify ns_update_menubar to handle the case where
there’s no frame?

> > * Dock Menu
> >
> > This has a ‘new frame’ option but just plain doesn’t work. I think the
> > problem is the way it’s trying to create a new frame:...
> >
> > This seems to rely on there being an existing NS frame, but we’ve
> > deleted the last one. I assume if we were clever we’d be able to use
> > the, still open, terminal frame.
> 
> With all of this discussion, I feel like I'm missing something really
> fundamental. Putting emacsclient aside for a moment, if we were dealing
> with a standard Cocoa app, the window server and NSApp loop would keep
> running in the background and we could close everything, then create a
> new frame (as a view on an NSWindow instance) as needed.

FWIW, the NSApp loop does keep running, but it’s embedded in ns_select
and ns_read_socket. We can, in theory, create a new frame directly,
but it needs to run through Emacs first.

> How is the architecture of Emacs on NS so different from standard Cocoa
> that these kinds of workarounds are necessary? 

The fundamental issue in the NS port is that we have two bosses: the
NSApp run loop, and Emacs lisp, and we need to mediate between them.

A standard Cocoa app responds to events from the NSApp loop, and
that’s pretty much it. As I understand it you can make a complete app
without very much work this way. The NSApp loop is the boss.

Emacs’ architecture expects everything to go through lisp and it makes
decisions on what we do with our GUI. Emacs lisp is the boss.

We can’t react directly to NS events like a normal NS app. We get
input from the NSApp loop and send it to lisp. Lisp might tell us to
do something any time. We also need to know how Emacs expects to do
things (the GUI code was originally written for a completely different
platform after all) and then try and make Cocoa/GNUstep handle it
correctly.

In an ideal world (and I believe the Mac port has gone this way) we
would put the NSApp run loop in one thread, and Emacs lisp in another
and let them communicate with each other asynchronously. This wouldn’t
solve everything, but it would make some of our problems easier.

We can’t easily do that, though, as the inter‐thread communication
systems provided in Objective‐C are either a pain to implement with
complex types like Lisp_Object, or aren’t compatible with GCC and/or
GNUstep (Grand Central Dispatch).

Additionally I think we sometimes have workarounds fixing workarounds
fixing hacks, all written by different people, and the simplest
solution is to strip it right back to the basics. I have a feeling the
menus fall somewhat into this category, and the drag‐n‐drop stuff
definitely does.


In an ideal world (and I believe the Mac port has gone this way) we
TL;DR: Emacs internal architecture doesn’t fit well into the standard
NS application architecture.
-- 
Alan Third

Re: Should this package be included into the NS port?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 4655 bytes --]

Alan Third <alan@idiocy.org> schrieb am Sa., 19. Mai 2018 um 12:34 Uhr:

> On Sat, May 19, 2018 at 04:42:54PM +1200, Nick Helm wrote:
> > On Sat, 19 May 2018 at 07:36:32 +1200, Alan Third wrote:
> >
> > > There are two problems, though.
> > >
> > > * Application Menu
> > >
> > > When the last NS frame is deleted the menus aren’t updated, so I think
> > > they’ll be the menus as defined for the last frame. They should
> > > probably be cut right back to just the ‘Emacs’ menu, and perhaps
> > > ‘help’.
> >
> > This is interesting. I think we could tweak the EmacsMenu side of things
> > to do this, even with the current code. We'd want to make sure to
> > include some kind of new frame command somewhere.
> >
> > These menus are derived dynamically from Lisp though, and it might be
> > difficult to convince it to provide valid entries when no frame is
> > selected.
>
> I’m not sure if the ‘Emacs’ menu is derived from lisp. If you turn off
> menus completely it still exists.
>
> > One way around that might be to create a new menu containing
> > static NSMenu versions of File and Help, much like we do for the appMenu
> > now. When no frame is selected and there are no Lisp menu entries, we
> > switch mainMenu to this new menu. When a frame is created, we switch
> > back.
>
> Perhaps we could modify ns_update_menubar to handle the case where
> there’s no frame?
>
> > > * Dock Menu
> > >
> > > This has a ‘new frame’ option but just plain doesn’t work. I think the
> > > problem is the way it’s trying to create a new frame:...
> > >
> > > This seems to rely on there being an existing NS frame, but we’ve
> > > deleted the last one. I assume if we were clever we’d be able to use
> > > the, still open, terminal frame.
> >
> > With all of this discussion, I feel like I'm missing something really
> > fundamental. Putting emacsclient aside for a moment, if we were dealing
> > with a standard Cocoa app, the window server and NSApp loop would keep
> > running in the background and we could close everything, then create a
> > new frame (as a view on an NSWindow instance) as needed.
>
> FWIW, the NSApp loop does keep running, but it’s embedded in ns_select
> and ns_read_socket. We can, in theory, create a new frame directly,
> but it needs to run through Emacs first.
>
> > How is the architecture of Emacs on NS so different from standard Cocoa
> > that these kinds of workarounds are necessary?
>
> The fundamental issue in the NS port is that we have two bosses: the
> NSApp run loop, and Emacs lisp, and we need to mediate between them.
>
> A standard Cocoa app responds to events from the NSApp loop, and
> that’s pretty much it. As I understand it you can make a complete app
> without very much work this way. The NSApp loop is the boss.
>
> Emacs’ architecture expects everything to go through lisp and it makes
> decisions on what we do with our GUI. Emacs lisp is the boss.
>
> We can’t react directly to NS events like a normal NS app. We get
> input from the NSApp loop and send it to lisp. Lisp might tell us to
> do something any time. We also need to know how Emacs expects to do
> things (the GUI code was originally written for a completely different
> platform after all) and then try and make Cocoa/GNUstep handle it
> correctly.
>
> In an ideal world (and I believe the Mac port has gone this way) we
> would put the NSApp run loop in one thread, and Emacs lisp in another
> and let them communicate with each other asynchronously. This wouldn’t
> solve everything, but it would make some of our problems easier.
>
> We can’t easily do that, though, as the inter‐thread communication
> systems provided in Objective‐C are either a pain to implement with
> complex types like Lisp_Object, or aren’t compatible with GCC and/or
> GNUstep (Grand Central Dispatch).
>
> Additionally I think we sometimes have workarounds fixing workarounds
> fixing hacks, all written by different people, and the simplest
> solution is to strip it right back to the basics. I have a feeling the
> menus fall somewhat into this category, and the drag‐n‐drop stuff
> definitely does.
>
>
> In an ideal world (and I believe the Mac port has gone this way) we
> TL;DR: Emacs internal architecture doesn’t fit well into the standard
> NS application architecture.
>

FWIW, this problem description and the solution seem the same on every
OS/toolkit: the Lisp interpreter should always run in a background thread
and communicate with the UI thread asynchronously through message passing.

[-- Attachment #2: Type: text/html, Size: 5293 bytes --]

Re: Should this package be included into the NS port?

[George Plymale II]

Thanks a lot for your elucidation of the fundamental problem, Alan. Your post
reminds me of an article with a somewhat more detailed description of this same
sort of problem on X11. The author of said article was trying to remove
flickering in Emacs when frames are resized and he managed to accomplish it by
adding support for double-buffering. I think it may be informative to Nick &
others, since your post echoes many of the same points. Here is the link:
https://www.facebook.com/notes/daniel-colascione/buttery-smooth-emacs/10155313440066102/
(Warning: the article has multiple instances of profanity so read it at your own
discretion)

Thanks,
- George Plymale II

Re: Should this package be included into the NS port?

[Nick Helm]

On Sat, 19 May 2018 at 22:33:29 +1200, Alan Third wrote:

> On Sat, May 19, 2018 at 04:42:54PM +1200, Nick Helm wrote:
>> 
>> This is interesting. I think we could tweak the EmacsMenu side of things
>> to do this, even with the current code. We'd want to make sure to
>> include some kind of new frame command somewhere.
>> 
>> These menus are derived dynamically from Lisp though, and it might be
>> difficult to convince it to provide valid entries when no frame is
>> selected. 
>
> I’m not sure if the ‘Emacs’ menu is derived from lisp. If you turn off
> menus completely it still exists.

Sorry, I meant to say the mainMenu, which holds all the Lisp menu stuff.
Yes, the menu named 'Emacs' (the appMenu) is defined in nsterm.m and
doesn't change as far as I know.

>> One way around that might be to create a new menu containing
>> static NSMenu versions of File and Help, much like we do for the appMenu
>> now. When no frame is selected and there are no Lisp menu entries, we
>> switch mainMenu to this new menu. When a frame is created, we switch
>> back. 
>
> Perhaps we could modify ns_update_menubar to handle the case where
> there’s no frame?

Maybe, I'm not sure. At the moment, as soon as ns_update_menubar sees
that it's not acting on the selected frame it gives up. I tried a quick
hack that sets f (the frame) to nil, bypasses the Lisp menu update code
and switches in a different static menu, but I couldn't get it to work
at all.

BTW, if you toggle the menu display off with (menu-bar-mode -1), you can
switch between frames/windows without the Lisp menus updating. This
prevents set_frame_menubar (and ns_update_menubar) from being called at
all.

As soon as you click in the menu-bar though, x_activate_menubar is
called, which does an ns_update_menubar, which inserts the menus for the
selected frame/window. Not sure why this happens, maybe Lisp assumes the
menu-bar is un-clickable if it's not supposed to be displayed? (mac
always shows a menu-bar, however)

I only mention this because I wonder if there might be a way to handle
the no-frame case with set_frame_menubar and x_activate_menubar, and
avoid messing with ns_update_menubar at all?

Re: Should this package be included into the NS port?

[Nick Helm]

On Sat, 19 May 2018 at 22:33:29 +1200, Alan Third wrote:

> FWIW, the NSApp loop does keep running, but it’s embedded in ns_select
> and ns_read_socket. We can, in theory, create a new frame directly,
> but it needs to run through Emacs first.
>
> The fundamental issue in the NS port is that we have two bosses: the
> NSApp run loop, and Emacs lisp, and we need to mediate between them.
>
> A standard Cocoa app responds to events from the NSApp loop, and
> that’s pretty much it. As I understand it you can make a complete app
> without very much work this way. The NSApp loop is the boss.
>
> Emacs’ architecture expects everything to go through lisp and it makes
> decisions on what we do with our GUI. Emacs lisp is the boss.
>
> We can’t react directly to NS events like a normal NS app. We get
> input from the NSApp loop and send it to lisp. Lisp might tell us to
> do something any time. We also need to know how Emacs expects to do
> things (the GUI code was originally written for a completely different
> platform after all) and then try and make Cocoa/GNUstep handle it
> correctly.
>
> In an ideal world (and I believe the Mac port has gone this way) we
> would put the NSApp run loop in one thread, and Emacs lisp in another
> and let them communicate with each other asynchronously. This wouldn’t
> solve everything, but it would make some of our problems easier.
>
> We can’t easily do that, though, as the inter‐thread communication
> systems provided in Objective‐C are either a pain to implement with
> complex types like Lisp_Object, or aren’t compatible with GCC and/or
> GNUstep (Grand Central Dispatch).
>
> Additionally I think we sometimes have workarounds fixing workarounds
> fixing hacks, all written by different people, and the simplest
> solution is to strip it right back to the basics. I have a feeling the
> menus fall somewhat into this category, and the drag‐n‐drop stuff
> definitely does.

Thanks for the great explanations. I think we should add something like
this to the head of nsterm.m.

I have more questions though (apologies in advance if they're
ill-informed or I've missed something obvious).

With all the to-and-fro of control between the Lisp and NSApp loops, why
don't we see more event collisions (or race conditions or whatever
competing events are called) like we saw recently with the Help menu
bug? Redisplay, for example, must be diabolical.

Putting emacsclient aside again, Emacs as a process (the Lisp loop) is
intrinsically linked to the concept of a Lisp frame, right? In other
words, no Lisp frame, no Emacs. So why does it matter whether that last
frame is on a tty or GUI? 

And, I guess, how does emacsclient beat this one-frame-must-exist
limitation – does it maintain some kind of fake frame that it hides from
the windowing system?

Thanks again.

Re: Should this package be included into the NS port?

[Eli Zaretskii]

> From: Nick Helm <nick@tenpoint.co.nz>
> Date: Wed, 23 May 2018 17:11:35 +1200
> 
> > In an ideal world (and I believe the Mac port has gone this way) we
> > would put the NSApp run loop in one thread, and Emacs lisp in another
> > and let them communicate with each other asynchronously. This wouldn’t
> > solve everything, but it would make some of our problems easier.

FWIW, that's what the w32 port does.

> > We can’t easily do that, though, as the inter‐thread communication
> > systems provided in Objective‐C are either a pain to implement with
> > complex types like Lisp_Object, or aren’t compatible with GCC and/or
> > GNUstep (Grand Central Dispatch).

Why do you need to have the communications between the threads be in
terms of Lisp objects?  In the w32 port, they communicate in C
objects, and the communications pipe is the read_socket_hook.  So in a
nutshell, the GUI event thread sends input to the Lisp interpreter as
if it was the other end of a socket through which window-system
events, including keyboard, mouse, and everything else, are coming.

> Putting emacsclient aside again, Emacs as a process (the Lisp loop) is
> intrinsically linked to the concept of a Lisp frame, right? In other
> words, no Lisp frame, no Emacs.

No, that's not true.  Why did you think that?

> And, I guess, how does emacsclient beat this one-frame-must-exist
> limitation – does it maintain some kind of fake frame that it hides from
> the windowing system?

That's exactly why your notion of frame-centric Emacs is inaccurate.

Re: Should this package be included into the NS port?

[Stefan Monnier]

>> And, I guess, how does emacsclient beat this one-frame-must-exist
>> limitation – does it maintain some kind of fake frame that it hides from
>> the windowing system?
> That's exactly why your notion of frame-centric Emacs is inaccurate.

IIRC, the emacs daemon solves this by using the "initial-frame" which is
associated to the "stdio terminal" (the special frame and terminal also
used in batch mode).


        Stefan

Re: Should this package be included into the NS port?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Wed, 23 May 2018 12:37:39 -0400
> Cc: alan@idiocy.org, Nick Helm <nick@tenpoint.co.nz>,
> 	georgedp@orbitalimpact.com, emacs-devel@gnu.org
> 
> >> And, I guess, how does emacsclient beat this one-frame-must-exist
> >> limitation – does it maintain some kind of fake frame that it hides from
> >> the windowing system?
> > That's exactly why your notion of frame-centric Emacs is inaccurate.
> 
> IIRC, the emacs daemon solves this by using the "initial-frame" which is
> associated to the "stdio terminal" (the special frame and terminal also
> used in batch mode).

Yes, but if you count that as a frame, then the one-frame-must-exist
limitation is meaningless, because that frame always exists, it comes
from the time Emacs was dumped.

Re: Should this package be included into the NS port?

[Alan Third]

On Wed, May 23, 2018 at 06:26:14PM +0300, Eli Zaretskii wrote:
> > From: Nick Helm <nick@tenpoint.co.nz>
> > Date: Wed, 23 May 2018 17:11:35 +1200
> > 
> > > In an ideal world (and I believe the Mac port has gone this way) we
> > > would put the NSApp run loop in one thread, and Emacs lisp in another
> > > and let them communicate with each other asynchronously. This wouldn’t
> > > solve everything, but it would make some of our problems easier.
> 
> FWIW, that's what the w32 port does.
> 
> > > We can’t easily do that, though, as the inter‐thread communication
> > > systems provided in Objective‐C are either a pain to implement with
> > > complex types like Lisp_Object, or aren’t compatible with GCC and/or
> > > GNUstep (Grand Central Dispatch).
> 
> Why do you need to have the communications between the threads be in
> terms of Lisp objects?  In the w32 port, they communicate in C
> objects, and the communications pipe is the read_socket_hook.  So in a
> nutshell, the GUI event thread sends input to the Lisp interpreter as
> if it was the other end of a socket through which window-system
> events, including keyboard, mouse, and everything else, are coming.

The NS port is similar, it executes the NS run loop within
ns_read_socket, and it sends events to the Lisp interpreter.

The main problem I have is things coming in the opposite direction. I
don’t know if W32 is the same, but we can only run GUI related code in
the main thread. If we separate the Lisp thread and the NS run loop
thread, then code called from Lisp can’t directly interact with the NS
GUI code, it has to be dispatched to the NS run loop thread.

If we do this we’ll have to use performSelectorOnMainThread, which
calls a method or function on the main thread, but only allows you to
pass a single parameter. I believe it’s possible to serialise multiple
arguments into a single one, but I’ve no idea if that’s practical.

I think this is possible, but will probably require a fair amount of
work.

It’s also possible I’m over‐complicating this. I feel rather out of my
depth.

-- 
Alan Third

Re: Should this package be included into the NS port?

[Eli Zaretskii]

> Date: Wed, 23 May 2018 22:21:29 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: Nick Helm <nick@tenpoint.co.nz>, emacs-devel@gnu.org,
> 	georgedp@orbitalimpact.com, monnier@iro.umontreal.ca
> 
> The main problem I have is things coming in the opposite direction. I
> don’t know if W32 is the same, but we can only run GUI related code in
> the main thread.

What do you mean by "GUI code" here?

> If we separate the Lisp thread and the NS run loop thread, then code
> called from Lisp can’t directly interact with the NS GUI code, it
> has to be dispatched to the NS run loop thread.

The w32 build has a similar problem, and it solves it by setting up
message queue between the main and the input threads.  Messages are
sent between the two threads when needed, and the Windows native
messages are used in the opposite direction.  So when one thread needs
something done by the other thread, it sends the appropriate message,
waits for a response, then continues (and sometimes it doesn't have to
wait for any response, it depends on the case).

Can NS use some similar machinery?

Re: Should this package be included into the NS port?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2095 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Do., 24. Mai 2018 um 18:38 Uhr:

> > Date: Wed, 23 May 2018 22:21:29 +0100
> > From: Alan Third <alan@idiocy.org>
> > Cc: Nick Helm <nick@tenpoint.co.nz>, emacs-devel@gnu.org,
> >       georgedp@orbitalimpact.com, monnier@iro.umontreal.ca
> >
> > The main problem I have is things coming in the opposite direction. I
> > don’t know if W32 is the same, but we can only run GUI related code in
> > the main thread.
>
> What do you mean by "GUI code" here?
>

Code that interacts with the windowing system.


>
> > If we separate the Lisp thread and the NS run loop thread, then code
> > called from Lisp can’t directly interact with the NS GUI code, it
> > has to be dispatched to the NS run loop thread.
>
> The w32 build has a similar problem, and it solves it by setting up
> message queue between the main and the input threads.  Messages are
> sent between the two threads when needed, and the Windows native
> messages are used in the opposite direction.  So when one thread needs
> something done by the other thread, it sends the appropriate message,
> waits for a response, then continues (and sometimes it doesn't have to
> wait for any response, it depends on the case).
>
> Can NS use some similar machinery?
>
>
Most likely yes (because UI toolkits always work that way). The
to-be-written GTK+ terminal should use the same mechanism.

See
https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/Multithreading/ThreadSafetySummary/ThreadSafetySummary.html#//apple_ref/doc/uid/10000057i-CH12-123351-BBCFIIEB
,
https://developer.apple.com/documentation/appkit/nsapplication/1428710-postevent?language=objc,
and
https://developer.apple.com/documentation/objectivec/nsobject/1414900-performselectoronmainthread?language=objc
.

What's the "input thread" in the Windows implementation? At least on macOS
the UI thread (i.e. the thread that creates windows and handles events)
must be the main thread, which means the Lisp interpreter thread(s) should
become background threads.

[-- Attachment #2: Type: text/html, Size: 3396 bytes --]

Re: Should this package be included into the NS port?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1909 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Do., 24. Mai 2018 um
19:46 Uhr:

>
>
>
>> > If we separate the Lisp thread and the NS run loop thread, then code
>> > called from Lisp can’t directly interact with the NS GUI code, it
>> > has to be dispatched to the NS run loop thread.
>>
>> The w32 build has a similar problem, and it solves it by setting up
>> message queue between the main and the input threads.  Messages are
>> sent between the two threads when needed, and the Windows native
>> messages are used in the opposite direction.  So when one thread needs
>> something done by the other thread, it sends the appropriate message,
>> waits for a response, then continues (and sometimes it doesn't have to
>> wait for any response, it depends on the case).
>>
>> Can NS use some similar machinery?
>>
>>
> Most likely yes (because UI toolkits always work that way). The
> to-be-written GTK+ terminal should use the same mechanism.
>
> See
> https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/Multithreading/ThreadSafetySummary/ThreadSafetySummary.html#//apple_ref/doc/uid/10000057i-CH12-123351-BBCFIIEB
> ,
> https://developer.apple.com/documentation/appkit/nsapplication/1428710-postevent?language=objc,
> and
> https://developer.apple.com/documentation/objectivec/nsobject/1414900-performselectoronmainthread?language=objc
> .
>
> What's the "input thread" in the Windows implementation? At least on macOS
> the UI thread (i.e. the thread that creates windows and handles events)
> must be the main thread, which means the Lisp interpreter thread(s) should
> become background threads.
>

However, the documentation also says "The main thread is the one blocked in
the run method of NSApplication", which seems to indicate that the "main"
thread doesn't actually have to be the same thread that the main function
runs in.

[-- Attachment #2: Type: text/html, Size: 2928 bytes --]

Re: Should this package be included into the NS port?

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 24 May 2018 19:46:19 +0200
> Cc: Alan Third <alan@idiocy.org>, nick@tenpoint.co.nz, georgedp@orbitalimpact.com, 
> 	monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> What's the "input thread" in the Windows implementation?

Not sure what kind of answer did you want to that.  You can find an
overview of how this stuff works in the w32 port around line 3250 of
w32fns.c; the "input thread" I alluded to is the one that runs the
message pump and the related window proc function.

> At least on macOS the UI thread (i.e. the thread
> that creates windows and handles events) must be the main thread, which means the Lisp interpreter thread
> (s) should become background threads. 

There's no such limitation on Windows, so the main thread runs Lisp.

Re: Should this package be included into the NS port?

[George Plymale II]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The Gtk port has a somewhat similar hack in the C code to avoid closing
> the display (to try and delay the infamous multi-display Gtk bug).
>
> So I think it might make sense to add such a workaround for this OSX bug
> much more directly in the core (rather than via defadvice).

That sounds great to me. I wasn't aware of the Gtk bug that you mention
(I've never used a Linux desktop with multiple monitors, so that's
probably why) but it's good to know that this macOS issue isn't entirely
unique. And yes, I agree that it would make sense to add this workaround
into the core. That would probably be a better choice, all things considered.

I was afraid that the developers here would be unwilling to include this
package or anything like it so I'm glad that I stand corrected.

Thanks,
- George Plymale II

Re: Should this package be included into the NS port?

[George Plymale II]

Is there anyone else in particular that I should Cc for further
discussion on this topic? I'm not sure who's presently in charge of the
NS port. No rush or anything... I just would prefer for this not to go
stale.

Thanks.

Re: Should this package be included into the NS port?

[Alan Third]

On Thu, May 17, 2018 at 06:13:30PM -0400, George Plymale II wrote:
> Is there anyone else in particular that I should Cc for further
> discussion on this topic? I'm not sure who's presently in charge of the
> NS port. No rush or anything... I just would prefer for this not to go
> stale.

No, nobody else. I don’t think there’s officially anyone in charge of
the NS port, but in practice it’s probably me.
-- 
Alan Third

Re: Should this package be included into the NS port?

[George Plymale II]

Alan Third <alan@idiocy.org> writes:

> No, nobody else. I don’t think there’s officially anyone in charge of
> the NS port, but in practice it’s probably me.

Gotcha. Thanks for letting me know.

Re: Should this package be included into the NS port?

[Michael Albinus]

Alan Third <alan@idiocy.org> writes:

Hi Alan,

>> Is there anyone else in particular that I should Cc for further
>> discussion on this topic? I'm not sure who's presently in charge of the
>> NS port. No rush or anything... I just would prefer for this not to go
>> stale.
>
> No, nobody else. I don’t think there’s officially anyone in charge of
> the NS port, but in practice it’s probably me.

Maybe you add yourself to admin/MAINTAINERS?

Best regards, Michael.

Re: Should this package be included into the NS port?

[Nick Helm]

On Tue, 15 May 2018 at 17:19:50 +1200, George Plymale II wrote:

> This package implements a "pseudo-daemon"...
>
> I actually have been using this package constantly for a couple of years
> now and it is a very reliable hack which basically solves this issue. 
>
> Thoughts?

I've been using a similar workaround on macOS for a while too. It's
pretty simple - I just advise delete-frame to hide the last frame
instead of closing it. It has the same effect as osx-pseudo-daemon.

IMO, these aren't good solutions though, as they just mask NS's bad
behaviour.

Re: Should this package be included into the NS port?

[George Plymale II]

Nick Helm <nick@tenpoint.co.nz> writes:

> IMO, these aren't good solutions though, as they just mask NS's bad
> behaviour.

Sure, it would be ideal to actually fix the issue. I'm all for
that. However, I don't think we should completely discount the usage of
osx-pseudo-daemon in case we can't fix the issue for whatever reason.

- George Plymale II

Re: Should this package be included into the NS port?

[George Plymale II]

Hi again folks,

The discourse so far regarding this issue has been very interesting. It
seems there's been a lot of back-and-forth about how to solve this issue
in the core. I won't reiterate or re-hash all of it here, but suffice it
to say that it seems this brings some significant difficulties due to
the way Emacs and the NS machinery cooperate. I'm still wondering if
perhaps we shouldn't just put the band-aid of osx-pseudo-daemon on the
problem for now.

I don't want anyone to feel over-burdened with the task of taking on
this issue or to feel out of their depth, so I'm just putting this on
the table once more. I talked to Ryan C. Thompson on Github the other
day; he said he's already signed the FSF paperwork so his package should
be acceptable into the core with little to no issues on that front.

I think this package will be a significantly easier solution for the
time being (if incomplete), and it will give us a little more time to
mull over how to solve the real problem for good.

Thanks,
- George Plymale II

Re: Should this package be included into the NS port?

[Alan Third]

On Tue, May 29, 2018 at 05:29:09PM -0400, George Plymale II wrote:
> Hi again folks,
> 
> The discourse so far regarding this issue has been very interesting. It
> seems there's been a lot of back-and-forth about how to solve this issue
> in the core. I won't reiterate or re-hash all of it here, but suffice it
> to say that it seems this brings some significant difficulties due to
> the way Emacs and the NS machinery cooperate. I'm still wondering if
> perhaps we shouldn't just put the band-aid of osx-pseudo-daemon on the
> problem for now.
> 
> I don't want anyone to feel over-burdened with the task of taking on
> this issue or to feel out of their depth, so I'm just putting this on
> the table once more. I talked to Ryan C. Thompson on Github the other
> day; he said he's already signed the FSF paperwork so his package should
> be acceptable into the core with little to no issues on that front.
> 
> I think this package will be a significantly easier solution for the
> time being (if incomplete), and it will give us a little more time to
> mull over how to solve the real problem for good.

FWIW I think we can solve this by defining:

(defun ns-make-frame ()
  (interactive)
  (make-frame-on-display (car (x-display-list))))

in ns-win.el, binding it to ns-new-frame, and then making that event
available everywhere. I’m not sure how to do the last bit. At the
moment it’s defined in nsterm.h; presumably it needs to be defined
somewhere else.

As described elsewhere the menus will be pretty much useless, but the
dock icon menu should be able to create a new frame.

It might be better to use the package until we can sort out the menu
issue. Thoughts?
-- 
Alan Third

Re: Should this package be included into the NS port?

[George Plymale II]

First off, thanks for your concrete feedback, Alan. I think we all
appreciate it.

> FWIW I think we can solve this by defining:

> (defun ns-make-frame ()
>   (interactive)
>   (make-frame-on-display (car (x-display-list))))

> in ns-win.el, binding it to ns-new-frame, and then making that event
> available everywhere.

Sounds reasonable... I could try my hand at this at some point, I
suppose, unless anyone else is more inclined or capable.

> I’m not sure how to do the last bit. At the moment it’s defined in
> nsterm.h; presumably it needs to be defined somewhere else.

What are you specifically referring to that is defined in nsterm.h? I'm
not sure I'm following you at this part 100%.

> As described elsewhere the menus will be pretty much useless, but the
> dock icon menu should be able to create a new frame.

> It might be better to use the package until we can sort out the menu
> issue. Thoughts?

Yes, I think the package would be an acceptable option till we can sort
out the menu issue. But other than that, what you're saying does seem
reasonable. I'm Cc'ing Ryan C. Thompson as well on this message in case
he'd like to chime in about his package.

Thanks,
- George Plymale II

Re: Should this package be included into the NS port?

[Alan Third]

On Tue, May 29, 2018 at 07:40:33PM -0400, George Plymale II wrote:
> > I’m not sure how to do the last bit. At the moment it’s defined in
> > nsterm.h; presumably it needs to be defined somewhere else.
> 
> What are you specifically referring to that is defined in nsterm.h? I'm
> not sure I'm following you at this part 100%.

Sorry, the event is defined in nsterm.h.

Open a GUI session then connect a -nw emacsclient to it. Make sure the
terminal frame is the current frame (type something in it), then go to
the dock menu and click ‘New Frame’, you’ll see a nonsense keybinding
error reported in the terminal frame’s minibuffer.

If you then select the GUI frame and do the same thing it works as
expected. I believe this means that because the event type is defined
in nsterm.h the terminal frame doesn’t understand it.

-- 
Alan Third

Re: Should this package be included into the NS port?

[Stefan Monnier]

> Open a GUI session then connect a -nw emacsclient to it. Make sure the
> terminal frame is the current frame (type something in it), then go to
> the dock menu and click ‘New Frame’, you’ll see a nonsense keybinding
> error reported in the terminal frame’s minibuffer.
>
> If you then select the GUI frame and do the same thing it works as
> expected. I believe this means that because the event type is defined
> in nsterm.h the terminal frame doesn’t understand it.

The description above doesn't match my understanding, but without seeing
more details (e.g. a backtrace of the "nonsense keybinding error"),
I can't tell whether I'm just misunderstanding something.


        Stefan

Re: Should this package be included into the NS port?

[George Plymale II]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Open a GUI session then connect a -nw emacsclient to it. Make sure the
>> terminal frame is the current frame (type something in it), then go to
>> the dock menu and click ‘New Frame’, you’ll see a nonsense keybinding
>> error reported in the terminal frame’s minibuffer.
>>
>> If you then select the GUI frame and do the same thing it works as
>> expected. I believe this means that because the event type is defined
>> in nsterm.h the terminal frame doesn’t understand it.
>
> The description above doesn't match my understanding, but without seeing
> more details (e.g. a backtrace of the "nonsense keybinding error"),
> I can't tell whether I'm just misunderstanding something.

I tried getting a backtrace of the error for you to see, but I see no
such backtrace after enabling both `toggle-debug-on-error' and
`toggle-debug-on-quit'. All I see is this in *Messages*:

<268435468> is undefined

Re: Should this package be included into the NS port?

[Stefan Monnier]

> <268435468> is undefined

Hmm... that is a weird event, indeed.
[ No surprise you can't "catch the error", because it's not an `error`:
  you just end up running the `undefined` command which emits this
  message and beeps.  ]

Someone will need to try and figure out where that comes from.


        Stefan

Re: Should this package be included into the NS port?

[Alan Third]

On Sat, Jun 02, 2018 at 03:39:19PM -0400, Stefan Monnier wrote:
> > <268435468> is undefined
> 
> Hmm... that is a weird event, indeed.
> [ No surprise you can't "catch the error", because it's not an `error`:
>   you just end up running the `undefined` command which emits this
>   message and beeps.  ]

I’m pretty sure that number is the one defined in nsterm.h line 751:

    #define KEY_NS_NEW_FRAME               ((1<<28)|(0<<16)|12)

and it looks like it’s defined for lisp in x-setup-function-keys in
common-win.el, but it looks to me like it should be set up in all
frames where (featurep 'ns) is true.

If I run emacsclient -nw (featurep 'ns) returns true, but the event
appears to be undefined (just shows the number).

Oh, I just read the docstring:

    "Set up `function-key-map' on the graphical frame FRAME."

I guess that function doesn’t run on terminal frames, then. That might
explain it.

Would we be able to define this (or maybe all of the NS events?) in
ns-win.el instead of in this function? Would that be a bad idea?
-- 
Alan Third

Re: Should this package be included into the NS port?

[Stefan Monnier]

> I’m pretty sure that number is the one defined in nsterm.h line 751:
>
>     #define KEY_NS_NEW_FRAME               ((1<<28)|(0<<16)|12)

Hmm... I wonder why the code uses a number instead of a symbol?
Apparently it's not just specific to NS since system-key-alist is
apparently also used under X11.  Here it says:

    system-key-alist is a variable defined in ‘keyboard.c’.
    Its value is ((65280 . remove))
    It is a terminal-local variable; global value is the same.

But AFAIK, this 65280 is an X11 keysym code.  I.e. should never be
turned into a Lisp numeric event (since there are only used for
characters).

I guess someone who understands why we have modify_event_symbol might be
able to help us figure out how to best fix this problem.

> and it looks like it’s defined for lisp in x-setup-function-keys in
> common-win.el, but it looks to me like it should be set up in all
> frames where (featurep 'ns) is true.

This code is not run for non-GUI frames/terminals, I think.

> I guess that function doesn’t run on terminal frames, then. That might
> explain it.

Exactly.

> Would we be able to define this (or maybe all of the NS events?) in
> ns-win.el instead of in this function?

Yes.

> Would that be a bad idea?

Seems no worse than having it in common-win.el and
wrapped in (featurep 'ns)

But it might not help, since ns-win.el is used for NS GUI frames,
whereas we'd want it for NS non-GUI frames.


        Stefan

Re: Should this package be included into the NS port?

[George Plymale II]

Alan Third <alan@idiocy.org> writes:

> Sorry, the event is defined in nsterm.h.
>
> Open a GUI session then connect a -nw emacsclient to it. Make sure the
> terminal frame is the current frame (type something in it), then go to
> the dock menu and click ‘New Frame’, you’ll see a nonsense keybinding
> error reported in the terminal frame’s minibuffer.
>
> If you then select the GUI frame and do the same thing it works as
> expected. I believe this means that because the event type is defined
> in nsterm.h the terminal frame doesn’t understand it.

Oh, okay, now I see what you mean. Yes, you're right; that needs to be
reworked to get the event to work in terminal frames. I believe that you
or perhaps someone else will know best how to do that. If not, I can try
to poke around at some point and see what I can do.

Re: Should this package be included into the NS port?

[Ryan Thompson]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Tue, May 29, 2018 at 4:40 PM George Plymale II <
georgedp@orbitalimpact.com> wrote:

> I'm Cc'ing Ryan C. Thompson as well on this message in case
> he'd like to chime in about his package.
>
> Thanks,
> - George Plymale II
>

Hello,

As before, I don't have much to add, as I'm not familiar with the internals
of Emacs responsible for the underlying issue. But as I've said before, I'm
perfectly happy to have my "pseudo-daemon" code incorporated into Emacs
itself, and I have already signed the copyright assignment papers for a few
small prior contributions to Emacs. If there is interest, I can rewrite my
mode as a patch to Emacs rather than advice. One thing we should think
about is what happens when my package is installed and enabled on an Emacs
that also includes this behavior as a core feature. I believe that it
shouldn't cause any problems, since only one or the other will ever
activate, not both.

In practice, the main drawback to using this mode is that when the last
existing frame is closed the new frame that it creates is briefly visible
before being hidden. This isn't a major issue in general, but it could
easily confuse people who don't know what's going on and make them think
that a glitch has occurred. If anyone knows how to create a hidden frame
without ever making it visible, I'd be happy to fix that.

Regards,

Ryan Thompson

[-- Attachment #2: Type: text/html, Size: 1803 bytes --]

Re: Should this package be included into the NS port?

[Stefan Monnier]

> that a glitch has occurred. If anyone knows how to create a hidden frame
> without ever making it visible, I'd be happy to fix that.

server.el supposedly manages to do that.  At least, it woprked for me
when I wrote the code, but I don't know if the NS port handles it
correctly:

(defun server-select-display (display)
  ;; If the current frame is on `display' we're all set.
  ;; Similarly if we are unable to open frames on other displays, there's
  ;; nothing more we can do.
  (unless (or (not (fboundp 'make-frame-on-display))
              (server--on-display-p (selected-frame) display))
    ;; Otherwise, look for an existing frame there and select it.
    (dolist (frame (frame-list))
      (when (server--on-display-p frame display)
	(select-frame frame)))
    ;; If there's no frame on that display yet, create and select one.
    (unless (server--on-display-p (selected-frame) display)
      (let* ((buffer (generate-new-buffer " *server-dummy*"))
             (frame (make-frame-on-display
                     display
                     ;; Make it display (and remember) some dummy buffer, so
                     ;; we can detect later if the frame is in use or not.
                     `((server-dummy-buffer . ,buffer)
                       ;; This frame may be deleted later (see
                       ;; server-unselect-display) so we want it to be as
                       ;; unobtrusive as possible.
                       (visibility . nil)))))
        (select-frame frame)
        (set-window-buffer (selected-window) buffer)
        frame))))


-- Stefan

Re: Should this package be included into the NS port?

[George Plymale II]

Ryan Thompson <rct@thompsonclan.org> writes:

> Hello,
>
> As before, I don't have much to add, as I'm not familiar with the internals
> of Emacs responsible for the underlying issue. But as I've said before, I'm
> perfectly happy to have my "pseudo-daemon" code incorporated into Emacs
> itself, and I have already signed the copyright assignment papers for a few
> small prior contributions to Emacs. If there is interest, I can rewrite my
> mode as a patch to Emacs rather than advice.

Thanks a lot Ryan! I think you're probably the most qualified person to
help out with this. Importing your package as a patch is also probably a
good idea; an advice seems like more of a 3rd-party thing to do.

> One thing we should think about is what happens when my package is
> installed and enabled on an Emacs that also includes this behavior as
> a core feature. I believe that it shouldn't cause any problems, since
> only one or the other will ever activate, not both.

If it does cause any problems, perhaps you could add an ad-hoc variable
or something and include a check for it in your package. Perhaps it
could issue a warning if it detects that it's running on an Emacs which
has the fix in the core. E.g.,

"Warning: Your Emacs already has this fix in the core, so this package
is actually not needed"

> In practice, the main drawback to using this mode is that when the last
> existing frame is closed the new frame that it creates is briefly visible
> before being hidden. This isn't a major issue in general, but it could
> easily confuse people who don't know what's going on and make them think
> that a glitch has occurred. If anyone knows how to create a hidden frame
> without ever making it visible, I'd be happy to fix that.

It seems that Stefan has mentioned something on how to deal with this in
an earlier message. I think I've seen this glitch a few times in the
past, but it never bothered me so I didn't really care. I don't know
much else about this particular issue so I can't comment further on it.