with-editor seems like a bug fix

with-editor seems like a bug fix

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What with-editor does seems like a bug fix to me, not like a feature.
It really ought to be standardly enabled in Emacs whenever the Emacs
client is.

Is there any chance we can convince the author to sign papers,
or perhaps have someone else write different code to do this job?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: with-editor seems like a bug fix

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Wed, 26 Jun 2024 23:07:14 -0400
> 
> What with-editor does seems like a bug fix to me, not like a feature.
> It really ought to be standardly enabled in Emacs whenever the Emacs
> client is.

If someone can explain what is/are the bug(s) and show a recipe for
reproducing it/them, we will fix it, yes.

We had a long discussion of this back in Sep 2023, and the conclusion
was that some of what with-editor does could justify one or two
utility functions in Emacs, and the other parts are due to semi-broken
Emacs installations (which I'm not sure we want to support, but maybe
if someone describes the details, we'll decide to support some of
them).

> Is there any chance we can convince the author to sign papers,
> or perhaps have someone else write different code to do this job?

That's not the issue here.  The issue is to understand what are the
problems with-editor tries to solve, and then decide which ones of
them we want to solve in Emacs and how.  This was the conclusion from
that Sep 2023 discussion, at least from my POV.

Re: with-editor seems like a bug fix

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> Is there any chance we can convince the author to sign papers,
>> or perhaps have someone else write different code to do this job?
>
> That's not the issue here.  The issue is to understand what are the
> problems with-editor tries to solve, and then decide which ones of
> them we want to solve in Emacs and how.  This was the conclusion from
> that Sep 2023 discussion, at least from my POV.

None the less, it would be prudent to get papers from its developers, I
think, it being intimately connected with Magit.  Which calls to mind
the question, whatever became of our plans to import Magit into core?

Re: with-editor seems like a bug fix

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Date: Wed, 26 Jun 2024 23:07:14 -0400
>> 
>> What with-editor does seems like a bug fix to me, not like a feature.
>> It really ought to be standardly enabled in Emacs whenever the Emacs
>> client is.
>
> If someone can explain what is/are the bug(s) and show a recipe for
> reproducing it/them, we will fix it, yes.

I said it in september 2023 and I don't know if it's a bug that needs a
fix but here is recipe of what 'with-editor' does well and could be
useful:

   - C-x C-f /ssh:remote-server|su::
   - M-x eshell
   - vipw  --> and here it works, opening an Emacs buffer through
               emacsclient so I could edit and C-c C-c when done

Note that:
   - EDITOR is set to "emacsclient" on the localhost
   - Emacs/emacsclient is not installed on remote-server

It seems that 'with-editor' does that with the following shell hackery
into $EDITOR on the remote server:

sh -c 'printf "\nWITH-EDITOR: $$ OPEN $0\037$1\037 IN $(pwd)\n"; sleep
604800 & sleep=$!; trap "kill $sleep; exit 0" USR1; trap "kill $sleep;
exit 1" USR2; wait $sleep'

So maybe that is what we need in Emacs if we want this feature.
-- 
Manuel Giraud

Re: with-editor seems like a bug fix

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: rms@gnu.org,  emacs-devel@gnu.org
> Date: Thu, 27 Jun 2024 11:44:54 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Richard Stallman <rms@gnu.org>
> >> Date: Wed, 26 Jun 2024 23:07:14 -0400
> >> 
> >> What with-editor does seems like a bug fix to me, not like a feature.
> >> It really ought to be standardly enabled in Emacs whenever the Emacs
> >> client is.
> >
> > If someone can explain what is/are the bug(s) and show a recipe for
> > reproducing it/them, we will fix it, yes.
> 
> I said it in september 2023 and I don't know if it's a bug that needs a
> fix but here is recipe of what 'with-editor' does well and could be
> useful:
> 
>    - C-x C-f /ssh:remote-server|su::
>    - M-x eshell
>    - vipw  --> and here it works, opening an Emacs buffer through
>                emacsclient so I could edit and C-c C-c when done
> 
> Note that:
>    - EDITOR is set to "emacsclient" on the localhost
>    - Emacs/emacsclient is not installed on remote-server
> 
> It seems that 'with-editor' does that with the following shell hackery
> into $EDITOR on the remote server:
> 
> sh -c 'printf "\nWITH-EDITOR: $$ OPEN $0\037$1\037 IN $(pwd)\n"; sleep
> 604800 & sleep=$!; trap "kill $sleep; exit 0" USR1; trap "kill $sleep;
> exit 1" USR2; wait $sleep'

Ouch!

> So maybe that is what we need in Emacs if we want this feature.

I hope not.

Michael, can you please look into this and see what can we do in such
cases?

Re: with-editor seems like a bug fix

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> Michael, can you please look into this and see what can we do in such
> cases?

Will do. It might take some few days. I'm in the final phase of Tramp
regression tests; over the weekend I plan to release Tramp 2.7.1. After
that, I should have more time.

Best regards, Michael.

Re: with-editor seems like a bug fix

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> >> What with-editor does seems like a bug fix to me, not like a feature.
>> >> It really ought to be standardly enabled in Emacs whenever the Emacs
>> >> client is.
>> >
>> > If someone can explain what is/are the bug(s) and show a recipe for
>> > reproducing it/them, we will fix it, yes.
>>
>> I said it in september 2023 and I don't know if it's a bug that needs a
>> fix but here is recipe of what 'with-editor' does well and could be
>> useful:
>>
>>    - C-x C-f /ssh:remote-server|su::
>>    - M-x eshell
>>    - vipw  --> and here it works, opening an Emacs buffer through
>>                emacsclient so I could edit and C-c C-c when done
>>
>> Note that:
>>    - EDITOR is set to "emacsclient" on the localhost
>>    - Emacs/emacsclient is not installed on remote-server
>>
>> It seems that 'with-editor' does that with the following shell hackery
>> into $EDITOR on the remote server:
>>
>> sh -c 'printf "\nWITH-EDITOR: $$ OPEN $0\037$1\037 IN $(pwd)\n"; sleep
>> 604800 & sleep=$!; trap "kill $sleep; exit 0" USR1; trap "kill $sleep;
>> exit 1" USR2; wait $sleep'
>
> Ouch!
>
>> So maybe that is what we need in Emacs if we want this feature.
>
> I hope not.
>
> Michael, can you please look into this and see what can we do in such
> cases?

As Manual says, we've discussed this last autumn.

In the local case, with-editor tries to set a proper environment for
calling emacsclient from child processes started in Emacs. This could
be 'git commit', 'vipw', 'crontab -e', whatever. Mainly it solves two
tasks: searching for a proper emacsclient program (which isn't trivial,
as Jonas did explain), and setting the $EDITOR environment variable when
calling the child process.

In the remote case, when a program is called on another host,
with-editor doesn't use emacsclient at all. Finding a proper emacsclient
remotely is more complicate, and setting the $EDITOR environment
variable needs more logic, because it must use emacsclient's -T
argument. Jonas has said that he doesn't use Tramp himself, so using a
remote emacsclient wasn't an option for him. The emacsclient -T argument
exists since Emacs 26; the with-editor package requires Emacs 25.1, it
wasn't available when the package started to exist.

And there are even more problems using a remote emacsclient program. It
simply coouldn't exist remotely. It simply couldn't be possible to use a
socket from the remote host back to the local host, for example if the
local host is indide a firewall, and the remote host isn't. It simply
couldn't be possible to establish a socket at all from the remote host
to the local host, when the remote host is reachable via a (Tramp)
multi-hop path only, like "/ssh:user1@proxyhost|ssh:user2@targethost:".

TL;DR: I don't believe with-editor is just a bug fix. There are reasons
       for the implementation as-it-is. It might profit from using
       emacsclient also for the remote case (when possible), but nobody
       has taken the stab.

Best regards, Michael.

Re: with-editor seems like a bug fix

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Manuel Giraud <manuel@ledu-giraud.fr>,  rms@gnu.org,
>   emacs-devel@gnu.org, Jonas Bernoulli <jonas@bernoul.li>
> Date: Sat, 29 Jun 2024 13:06:54 +0200
> 
> >> I said it in september 2023 and I don't know if it's a bug that needs a
> >> fix but here is recipe of what 'with-editor' does well and could be
> >> useful:
> >>
> >>    - C-x C-f /ssh:remote-server|su::
> >>    - M-x eshell
> >>    - vipw  --> and here it works, opening an Emacs buffer through
> >>                emacsclient so I could edit and C-c C-c when done
> >>
> >> Note that:
> >>    - EDITOR is set to "emacsclient" on the localhost
> >>    - Emacs/emacsclient is not installed on remote-server
> >>
> >> It seems that 'with-editor' does that with the following shell hackery
> >> into $EDITOR on the remote server:
> >>
> >> sh -c 'printf "\nWITH-EDITOR: $$ OPEN $0\037$1\037 IN $(pwd)\n"; sleep
> >> 604800 & sleep=$!; trap "kill $sleep; exit 0" USR1; trap "kill $sleep;
> >> exit 1" USR2; wait $sleep'
> >
> > Ouch!
> >
> >> So maybe that is what we need in Emacs if we want this feature.
> >
> > I hope not.
> >
> > Michael, can you please look into this and see what can we do in such
> > cases?
> 
> As Manual says, we've discussed this last autumn.
> 
> In the local case, with-editor tries to set a proper environment for
> calling emacsclient from child processes started in Emacs. This could
> be 'git commit', 'vipw', 'crontab -e', whatever. Mainly it solves two
> tasks: searching for a proper emacsclient program (which isn't trivial,
> as Jonas did explain), and setting the $EDITOR environment variable when
> calling the child process.

My conclusion from discussing the local case was that problems with
finding the right version of emacsclient, such as they are, are
limited to broken Emacs installations.  There was also a possibility
that these problems exist in valid installations, but no one was able
to describe such situations.  Until someone does, I don't see why we
should consider what we have buggy.

What I was asking was limited to the remote case.

> In the remote case, when a program is called on another host,
> with-editor doesn't use emacsclient at all. Finding a proper emacsclient
> remotely is more complicate, and setting the $EDITOR environment
> variable needs more logic, because it must use emacsclient's -T
> argument. Jonas has said that he doesn't use Tramp himself, so using a
> remote emacsclient wasn't an option for him. The emacsclient -T argument
> exists since Emacs 26; the with-editor package requires Emacs 25.1, it
> wasn't available when the package started to exist.
> 
> And there are even more problems using a remote emacsclient program. It
> simply coouldn't exist remotely. It simply couldn't be possible to use a
> socket from the remote host back to the local host, for example if the
> local host is indide a firewall, and the remote host isn't. It simply
> couldn't be possible to establish a socket at all from the remote host
> to the local host, when the remote host is reachable via a (Tramp)
> multi-hop path only, like "/ssh:user1@proxyhost|ssh:user2@targethost:".

What you say here seems to imply that the remote case cannot be solved
in principle?

> TL;DR: I don't believe with-editor is just a bug fix. There are reasons
>        for the implementation as-it-is. It might profit from using
>        emacsclient also for the remote case (when possible), but nobody
>        has taken the stab.

I don't see how we can discuss a package which solves problems we
don't understand.  Until and unless someone describes the problems
which with-editor solves in enough detail for us to understand what
goes wrong and where, I don't think we can discuss the addition of the
package to core.

Thanks.

Re: with-editor seems like a bug fix

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> What you say here seems to imply that the remote case cannot be solved
> in principle?

"in general" might describe it better.

>> TL;DR: I don't believe with-editor is just a bug fix. There are reasons
>>        for the implementation as-it-is. It might profit from using
>>        emacsclient also for the remote case (when possible), but nobody
>>        has taken the stab.
>
> I don't see how we can discuss a package which solves problems we
> don't understand.

Oh, we understand. When there is no emacsclient on the remote host, we
cannot use it. with-editor solves this problems.

> Until and unless someone describes the problems which with-editor
> solves in enough detail for us to understand what goes wrong and
> where, I don't think we can discuss the addition of the package to
> core.

Yep, there are more use cases to investigate. But there doesn't seem
enough pressure (from users) to change the status quo.

I believe it is OK to keep with-editor in ELPA.

> Thanks.

Best regards, Michael.

Re: with-editor seems like a bug fix

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: manuel@ledu-giraud.fr,  rms@gnu.org,  emacs-devel@gnu.org,
>   jonas@bernoul.li
> Date: Sat, 29 Jun 2024 14:47:07 +0200
> 
> >> TL;DR: I don't believe with-editor is just a bug fix. There are reasons
> >>        for the implementation as-it-is. It might profit from using
> >>        emacsclient also for the remote case (when possible), but nobody
> >>        has taken the stab.
> >
> > I don't see how we can discuss a package which solves problems we
> > don't understand.
> 
> Oh, we understand. When there is no emacsclient on the remote host, we
> cannot use it. with-editor solves this problems.

I meant in the local case.

For the remote case, if you think the solution in with-editor is the
best we can do, let's install something like that in our own code, or
maybe provide a user option for doing that.

> > Until and unless someone describes the problems which with-editor
> > solves in enough detail for us to understand what goes wrong and
> > where, I don't think we can discuss the addition of the package to
> > core.
> 
> Yep, there are more use cases to investigate. But there doesn't seem
> enough pressure (from users) to change the status quo.
> 
> I believe it is OK to keep with-editor in ELPA.

Fine by me.  Removal from ELPA was never on the table.

Re: with-editor seems like a bug fix

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> I meant in the local case.
>
> For the remote case, if you think the solution in with-editor is the
> best we can do, let's install something like that in our own code, or
> maybe provide a user option for doing that.

What if we install a simplified version in vanilla Emacs? That is

- Use always emacsclient, locally or remote
- Check for emacsclient only at the well know places
- Recommend to use with-editor from ELPA if this fails

Jonas, WDYT?

>> I believe it is OK to keep with-editor in ELPA.
>
> Fine by me.  Removal from ELPA was never on the table.

Good. I was a little bit nervous because of the subject "with-editor
seems like a bug fix".

Best regards, Michael.

Re: with-editor seems like a bug fix

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: manuel@ledu-giraud.fr,  rms@gnu.org,  emacs-devel@gnu.org,
>   jonas@bernoul.li
> Date: Sat, 29 Jun 2024 16:39:11 +0200
> 
> What if we install a simplified version in vanilla Emacs? That is
> 
> - Use always emacsclient, locally or remote
> - Check for emacsclient only at the well know places
> - Recommend to use with-editor from ELPA if this fails

I'd need to see the patch to understand what that means in practice.

Re: with-editor seems like a bug fix

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> What if we install a simplified version in vanilla Emacs? That is
>>
>> - Use always emacsclient, locally or remote
>> - Check for emacsclient only at the well know places
>> - Recommend to use with-editor from ELPA if this fails
>
> I'd need to see the patch to understand what that means in practice.

If Jonas agrees this approach, I could try to puzzle something together.

Best regards, Michael.

Re: with-editor seems like a bug fix

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael Albinus <michael.albinus@gmx.de>
>> Cc: manuel@ledu-giraud.fr,  rms@gnu.org,  emacs-devel@gnu.org,
>>   jonas@bernoul.li
>> Date: Sat, 29 Jun 2024 14:47:07 +0200
>> 
>> >> TL;DR: I don't believe with-editor is just a bug fix. There are reasons
>> >>        for the implementation as-it-is. It might profit from using
>> >>        emacsclient also for the remote case (when possible), but nobody
>> >>        has taken the stab.
>> >
>> > I don't see how we can discuss a package which solves problems we
>> > don't understand.
>> 
>> Oh, we understand. When there is no emacsclient on the remote host, we
>> cannot use it. with-editor solves this problems.
>
> I meant in the local case.

FWIW, I have never yet encountered a case, where having the emacs server
started and EDITOR set to "emacsclient" failed in the local case.

That said, skimming through the with-editor's code, it seems to try real
hard to find a correct emacsclient (even "remacsclient" for remacs!)  It
even seems to start the emacs server if it is not already the case.  So
I agree that it could be declared "broken" installations.
-- 
Manuel Giraud

Re: with-editor seems like a bug fix

[Stefan Kangas]

Manuel Giraud via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> FWIW, I have never yet encountered a case, where having the emacs server
> started and EDITOR set to "emacsclient" failed in the local case.
>
> That said, skimming through the with-editor's code, it seems to try real
> hard to find a correct emacsclient (even "remacsclient" for remacs!)  It
> even seems to start the emacs server if it is not already the case.  So
> I agree that it could be declared "broken" installations.

See also the previous discussion, starting here:

    https://lists.gnu.org/r/emacs-devel/2023-09/msg00113.html

Re: with-editor seems like a bug fix

[Jonas Bernoulli]

Michael Albinus <michael.albinus@gmx.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> Hi Eli,
>
>>> What if we install a simplified version in vanilla Emacs? That is
>>>
>>> - Use always emacsclient, locally or remote
>>> - Check for emacsclient only at the well know places
>>> - Recommend to use with-editor from ELPA if this fails
>>
>> I'd need to see the patch to understand what that means in practice.
>
> If Jonas agrees this approach, I could try to puzzle something together.

Sounds good to me in principal.

I am unsure whether you plan to add a stripped-down with-editor.el to
Emacs or implement the mentioned functionality in existing libraries.
The latter seems more reasonable to me (and I'm guessing that's what
you had in mind).

Please also consider these features:

- Add `server-window-alist'.  This would allow both packages and users
  to specify that certain buffers should be displayed differently from
  the default specified by `server-window'.

- Allow running `server-start' in multiple emacs instances, without
  the user/package who calls that, having figure out if another instance
  is already using the same name.  I.e., something to replace:

    (unless (process-live-p server-process)
      (when (server-running-p server-name)
        (setq server-name (format "server%s" (emacs-pid)))
        (when (server-running-p server-name)
          (server-force-delete server-name)))
      (server-start))

By the way, I really liked you summary of what with-editor.el does and
why.

Cheers,
Jonas

Re: with-editor seems like a bug fix

[Michael Albinus]

Jonas Bernoulli <jonas@bernoul.li> writes:

Hi Jonas,

>>>> - Use always emacsclient, locally or remote
>>>> - Check for emacsclient only at the well know places
>>>> - Recommend to use with-editor from ELPA if this fails
>>>
>>> I'd need to see the patch to understand what that means in practice.
>>
>> If Jonas agrees this approach, I could try to puzzle something together.
>
> Sounds good to me in principal.

Sorry for the late reply. I'm very slow these days :-(

> I am unsure whether you plan to add a stripped-down with-editor.el to
> Emacs or implement the mentioned functionality in existing libraries.
> The latter seems more reasonable to me (and I'm guessing that's what
> you had in mind).

Yes, the latter.

I will implement this in several steps. For every step I'll write a bug
report (severity wishlist) first, so we have a place to discuss. And not
to poison the emacs-devel ML with our jabber :-)

> Please also consider these features:
>
> - Add `server-window-alist'.  This would allow both packages and users
>   to specify that certain buffers should be displayed differently from
>   the default specified by `server-window'.

That's the first step. Bug#71971.

> - Allow running `server-start' in multiple emacs instances, without
>   the user/package who calls that, having figure out if another instance
>   is already using the same name.  I.e., something to replace:
>
>     (unless (process-live-p server-process)
>       (when (server-running-p server-name)
>         (setq server-name (format "server%s" (emacs-pid)))
>         (when (server-running-p server-name)
>           (server-force-delete server-name)))
>       (server-start))

I'll check.

> Cheers,
> Jonas

Best regards, Michael.