Adding with-editor to Emacs?

Adding with-editor to Emacs?

[Jonas Bernoulli]

Richard Stallman <rms@gnu.org> writes:

> Are you involved in developing this?

Yes, I am the primary author.  Contributions by other people are pretty
minor, or by people who have already done the copyright paperwork, so
that should not be a problem.

> It seems to me that this would fit best in emacsclient and its
> associated Lisp code -- that having it as a separate package
> is extra baggage.
>
> WDYT?

Adding this to Emacs would make sense.  We would have to determine to
what extend existing code should be updated to do what this package does
and what parts should remain in a package/library in its own right.

For example, it would make sense to deprecate variable `server-window'
with a new `server-window-alist', even if the rest of with-editor.el
remained a separate library, or even outside of Emacs.  (The new
variable would give packages more control over how their windows should
be displayed, if they have special needs, while still allowing users
to customize the default, or even to override the packages' wishes.)

I think the next step is to ask Eli and others, how they would want to
integrate the library / the functionality it provides.

An obstacle in getting this done quickly, is that I am getting requests
to do work in areas that I wasn't planning on touching any time soon,
with a higher than usual frequency.  So the process of integrating this
into Emacs could be somewhat slow, depending on how much we diverge from
just merging this as-is in the form of a separate library/package.

> having it as a separate package is extra baggage.

Maybe.  As far as adding `server-window-alist' goes, that would
certainly be a win.

But other than that, this package is basically "done" in its current
form.  Work is only really required when someone found yet another way
to package Emacs, without thinking too much about emacsclient, if at
all.  So I must point out that the primary effect for my personally
will be extra work, and not just in the short term.

     Cheers,
     Jonas

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Jonas Bernoulli <jonas@bernoul.li>
> Cc: rms@gnu.org
> Date: Fri, 01 Sep 2023 16:38:02 +0200
> 
> I think the next step is to ask Eli and others, how they would want to
> integrate the library / the functionality it provides.

I'm probably missing something because if all we want is to allow
child processes to use the current Emacs session as their editor, we
just need to inject some environment variables into
process-environment when running those child processes, and start the
server.  Emacs knows very well where to find its corresponding
emacsclient.  Why is there a need for a separate library?

> As far as adding `server-window-alist' goes, that would certainly be
> a win.

If we want to extend server-window in some ways, that could be a
separate change.  It sounds like its utility is not necessarily
specific to with-editor or to its main feature of allowing
sub-processes use the parent Emacs as their editor.

(I couldn't find the beginning of this discussion, so maybe I missed
some of the relevant context, in which case I apologize.)

Re: Adding with-editor to Emacs?

[Jim Porter]

On 9/1/2023 9:12 AM, Eli Zaretskii wrote:
>> From: Jonas Bernoulli <jonas@bernoul.li>
>> Cc: rms@gnu.org
>> Date: Fri, 01 Sep 2023 16:38:02 +0200
>>
>> I think the next step is to ask Eli and others, how they would want to
>> integrate the library / the functionality it provides.
> 
> I'm probably missing something because if all we want is to allow
> child processes to use the current Emacs session as their editor, we
> just need to inject some environment variables into
> process-environment when running those child processes, and start the
> server.  Emacs knows very well where to find its corresponding
> emacsclient.  Why is there a need for a separate library?

I'm not totally sure *all* of what with-editor does, but it does let you 
do this over Tramp too (still using your local emacsclient). That 
requires a bit more magic to get it work.

Re: Adding with-editor to Emacs?

[Jonas Bernoulli]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jonas Bernoulli <jonas@bernoul.li>
>> Cc: rms@gnu.org
>> Date: Fri, 01 Sep 2023 16:38:02 +0200
>> 
>> I think the next step is to ask Eli and others, how they would want to
>> integrate the library / the functionality it provides.
>
> I'm probably missing something because if all we want is to allow
> child processes to use the current Emacs session as their editor, we
> just need to inject some environment variables into
> process-environment when running those child processes, and start the
> server.

That's the core of what with-editor does.  Additionally

- It tries hard to find the correct emacsclient to use.

- It implements a "sleeping editor".  This is a shell script, which
  outputs a request on stdout and then waits to be told to return.
  With-editor use a process filter too look for that output and when
  it sees it, it responds in a similar fashion to server.el.  This
  is useful because makes it possible to do this over Tramp.  (I
  believe this could also be done using regular emacsclient+server.el,
  but that is difficult to setup and a security risk if not done
  correctly.

- It provides some convenience functionality to use this from various
  shells running inside Emacs.

> Emacs knows very well where to find its corresponding emacsclient.

I wasn't aware of that.  How can I make use of that knowledge?  I.e.,
is there a function that answers the question "what is the path of the
emacsclient that was installed with this version of emacs"?

> Why is there a need for a separate library?

I agree that there theoretically isn't a need for this library, but it
turned out that just setting EDITOR=emacsclient in the environment of
the sub process also doesn't cut it, because for many users (who never
use emacsclient directly), would have to add some configuration to make
it work.  The core and original functionality provided by with-editor,
is making the configuration unnecessary by using heuristics.

Maybe I miss something, but I concluded that emacs does *not* know
its corresponding emacsclient.  Once I know how to make use of that
knowledge, I can simplify with-editor quite a bit, with will benefit
the packages that already use it.  And if that really is fail-safe,
then Emacs doesn't have much to gain from integrating with-editor.

>> As far as adding `server-window-alist' goes, that would certainly be
>> a win.
>
> If we want to extend server-window in some ways, that could be a
> separate change.  It sounds like its utility is not necessarily
> specific to with-editor or to its main feature of allowing
> sub-processes use the parent Emacs as their editor.

Agreed and that is something I was planning to suggest eventually,
while I had no plan to suggest merging all of with-editor.el.

> (I couldn't find the beginning of this discussion, so maybe I missed
> some of the relevant context, in which case I apologize.)

The text I quoted, was the very beginning of this discussion.

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Jonas Bernoulli <jonas@bernoul.li>
> Cc: emacs-devel@gnu.org, rms@gnu.org
> Date: Fri, 01 Sep 2023 19:44:53 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm probably missing something because if all we want is to allow
> > child processes to use the current Emacs session as their editor, we
> > just need to inject some environment variables into
> > process-environment when running those child processes, and start the
> > server.
> 
> That's the core of what with-editor does.  Additionally
> 
> - It tries hard to find the correct emacsclient to use.

See below: I don't think I understand why this has to be "hard".

> - It implements a "sleeping editor".  This is a shell script, which
>   outputs a request on stdout and then waits to be told to return.
>   With-editor use a process filter too look for that output and when
>   it sees it, it responds in a similar fashion to server.el.  This
>   is useful because makes it possible to do this over Tramp.  (I
>   believe this could also be done using regular emacsclient+server.el,
>   but that is difficult to setup and a security risk if not done
>   correctly.

If we want a better/safer client-server connections for remote hosts,
it should be handled in Tramp, I think.

> - It provides some convenience functionality to use this from various
>   shells running inside Emacs.

Can you provide details?  The above is too terse for me to understand
the functionality.

> > Emacs knows very well where to find its corresponding emacsclient.
> 
> I wasn't aware of that.  How can I make use of that knowledge?  I.e.,
> is there a function that answers the question "what is the path of the
> emacsclient that was installed with this version of emacs"?

When Emacs runs installed, emacsclient is in the same directory where
we install the Emacs binary, so emacsclient should be in
invocation-directory.  If Emacs runs uninstalled, emacsclient is in
../lib-src/ relative to invocation-directory.  Whether Emacs runs
installed or not is determined by the value of installation-directory:
if it's nil, Emacs runs installed.

There could be a complication if the programs were renamed when
installed (see TRANSFORM in the top-level Makefile).  When that
happens, emacsclient's name is also TRANSFORMed in the same way.  You
can find out what was the actual name under which Emacs was invoked
from the value of invocation-name, and then apply the same TRANSFORM
to the name of emacsclient.

> > Why is there a need for a separate library?
> 
> I agree that there theoretically isn't a need for this library, but it
> turned out that just setting EDITOR=emacsclient in the environment of
> the sub process also doesn't cut it, because for many users (who never
> use emacsclient directly), would have to add some configuration to make
> it work.  The core and original functionality provided by with-editor,
> is making the configuration unnecessary by using heuristics.

Can you elaborate on the configuration that is needed?  I always
thought that emacsclient worked for everybody OOTB.

> > (I couldn't find the beginning of this discussion, so maybe I missed
> > some of the relevant context, in which case I apologize.)
> 
> The text I quoted, was the very beginning of this discussion.

Yes, but did RMS write anything that you haven't quoted?

Re: Adding with-editor to Emacs?

[Jonas Bernoulli]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jonas Bernoulli <jonas@bernoul.li>
>> Cc: emacs-devel@gnu.org, rms@gnu.org
>> Date: Fri, 01 Sep 2023 19:44:53 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:

>> - It implements a "sleeping editor".  This is a shell script, which
>>   outputs a request on stdout and then waits to be told to return.
>>   With-editor use a process filter too look for that output and when
>>   it sees it, it responds in a similar fashion to server.el.  This
>>   is useful because makes it possible to do this over Tramp.  (I
>>   believe this could also be done using regular emacsclient+server.el,
>>   but that is difficult to setup and a security risk if not done
>>   correctly.
>
> If we want a better/safer client-server connections for remote hosts,
> it should be handled in Tramp, I think.

Probably and it would be great if Tramp did handle that, but I don't
even use Tramp except when users report that there is an issue when
using Tramp with one of my packages.  So I am not the right person to
implement it there, but if Michael were to tackle this, then maybe I
would have some insights that could be useful.  Or not.

>> - It provides some convenience functionality to use this from various
>>   shells running inside Emacs.
>
> Can you provide details?  The above is too terse for me to understand
> the functionality.

Essentially what it does is "export EDITOR=emacsclient" without as if
that line existed in the shells init file.  When summarized like that,
this is of course a bit of a silly feature -- users can just put that
line in their init file instead.  (Also I should point out that I did
not actually want to add support for this, but got talked into it.)

The reason why people wanted me to this is that simply using
"emacsclient" as the value turned out not to be good enough.

>> > Emacs knows very well where to find its corresponding emacsclient.
>> 
>> I wasn't aware of that.  How can I make use of that knowledge?  I.e.,
>> is there a function that answers the question "what is the path of the
>> emacsclient that was installed with this version of emacs"?
>
> When Emacs runs installed, emacsclient is in the same directory where
> we install the Emacs binary, so emacsclient should be in
> invocation-directory.

"I have seen things you wouldn't believe..."

I wish this was true, but unfortunately keep coming up with new and
innovative ways of not doing that.  The worst platform is macOS, but I
also had to add a special case for Debian.  Users or distributions may
install emacs in a weird location, and then symlink emacs onto $PATH
(but without doing the same for emacsclient).  It is also possible to
install multiple versions of emacs in the same directory and append
version strings to the installed binaries.  To select one of these
versions a user/distribution may add a symlink named symlink in the
same directory.  One may or may not do the same for emacsclient.  If
emacsclient isn't a symlink, then an ancient emacsclient may still be
leftover from an earlier manual installed that was not properly
uninstalled. ...

> If Emacs runs uninstalled, emacsclient is in ../lib-src/ relative to
> invocation-directory.  Whether Emacs runs installed or not is
> determined by the value of installation-directory: if it's nil, Emacs
> runs installed.

... and that too.

> There could be a complication if the programs were renamed when
> installed (see TRANSFORM in the top-level Makefile).  When that
> happens, emacsclient's name is also TRANSFORMed in the same way.  You
> can find out what was the actual name under which Emacs was invoked
> from the value of invocation-name, and then apply the same TRANSFORM
> to the name of emacsclient.
>
>> > Why is there a need for a separate library?
>> 
>> I agree that there theoretically isn't a need for this library, but it
>> turned out that just setting EDITOR=emacsclient in the environment of
>> the sub process also doesn't cut it, because for many users (who never
>> use emacsclient directly), would have to add some configuration to make
>> it work.  The core and original functionality provided by with-editor,
>> is making the configuration unnecessary by using heuristics.
>
> Can you elaborate on the configuration that is needed?  I always
> thought that emacsclient worked for everybody OOTB.

Sadly that isn't the case, see above.

Normally these complications are the user's problem.  They may not even
use emacsclient.  If they don't actively choose to use emacsclient, they
will never know it is actually broken for them.  If they try to use it
and it doesn't work, they can decide whether it is worth trying to
figure out to work around those mistakes.  If they want to use it but
cannot fix it for themselves, they can contact the people who packaged
Emacs for them.

For me the situation was different, I started with something like this
in one of my packages, effectively forcing users to use "emacsclient":

  (let ((process-environment process-environment))
    (setenv "EDITOR"
            (expand-file-name "emacsclient" invocation-directory))
    (call-process "git" nil nil nil "commit"))

Like you I assumed this was a solved problem.

And then the "I cannot commit anymore" reports started pouring in.

I decided to implement with-editor, and to once in a while add a new
heuristic to detect the correct emacsclient, instead of having to
support users 1on1 for evermore, to figure out how exactly emacsclient
isn't located and/or named what it is supposed to be, in their
particular case.

>> > (I couldn't find the beginning of this discussion, so maybe I missed
>> > some of the relevant context, in which case I apologize.)
>> 
>> The text I quoted, was the very beginning of this discussion.
>
> Yes, but did RMS write anything that you haven't quoted?

Except

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

and the signature?

No, the text I quoted is all of it.

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Jonas Bernoulli <jonas@bernoul.li>
> Cc: stefankangas@gmail.com, emacs-devel@gnu.org, rms@gnu.org
> Date: Fri, 01 Sep 2023 22:23:43 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Emacs knows very well where to find its corresponding emacsclient.
> >> 
> >> I wasn't aware of that.  How can I make use of that knowledge?  I.e.,
> >> is there a function that answers the question "what is the path of the
> >> emacsclient that was installed with this version of emacs"?
> >
> > When Emacs runs installed, emacsclient is in the same directory where
> > we install the Emacs binary, so emacsclient should be in
> > invocation-directory.
> 
> "I have seen things you wouldn't believe..."
> 
> I wish this was true, but unfortunately keep coming up with new and
> innovative ways of not doing that.  The worst platform is macOS, but I
> also had to add a special case for Debian.  Users or distributions may
> install emacs in a weird location, and then symlink emacs onto $PATH
> (but without doing the same for emacsclient).  It is also possible to
> install multiple versions of emacs in the same directory and append
> version strings to the installed binaries.  To select one of these
> versions a user/distribution may add a symlink named symlink in the
> same directory.  One may or may not do the same for emacsclient.  If
> emacsclient isn't a symlink, then an ancient emacsclient may still be
> leftover from an earlier manual installed that was not properly
> uninstalled. ...

Then I guess you should describe all those atrocities in detail, so
that we could perhaps devise ways of handling it.

Also, since the client-server protocol doesn't change much, there's
usually no need to insist on invoking the emacsclient that came with
this particular Emacs, you can use any other.

> > If Emacs runs uninstalled, emacsclient is in ../lib-src/ relative to
> > invocation-directory.  Whether Emacs runs installed or not is
> > determined by the value of installation-directory: if it's nil, Emacs
> > runs installed.
> 
> ... and that too.

What do you mean? the value of installation-directory is calculated
internally by Emacs.

> >> I agree that there theoretically isn't a need for this library, but it
> >> turned out that just setting EDITOR=emacsclient in the environment of
> >> the sub process also doesn't cut it, because for many users (who never
> >> use emacsclient directly), would have to add some configuration to make
> >> it work.  The core and original functionality provided by with-editor,
> >> is making the configuration unnecessary by using heuristics.
> >
> > Can you elaborate on the configuration that is needed?  I always
> > thought that emacsclient worked for everybody OOTB.
> 
> Sadly that isn't the case, see above.

In what ways does "the above" mean that emacsclient doesn't work OOTB?
Do you mean that emacsclient is installed in a place that just typing
"emacsclient RET" at the shell prompt fails to run it?  If so, that's
a broken installation, and Emacs shouldn't really try to fix that.  Or
do you mean something else?

> Normally these complications are the user's problem.  They may not even
> use emacsclient.  If they don't actively choose to use emacsclient, they
> will never know it is actually broken for them.  If they try to use it
> and it doesn't work, they can decide whether it is worth trying to
> figure out to work around those mistakes.  If they want to use it but
> cannot fix it for themselves, they can contact the people who packaged
> Emacs for them.
> 
> For me the situation was different, I started with something like this
> in one of my packages, effectively forcing users to use "emacsclient":
> 
>   (let ((process-environment process-environment))
>     (setenv "EDITOR"
>             (expand-file-name "emacsclient" invocation-directory))
>     (call-process "git" nil nil nil "commit"))
> 
> Like you I assumed this was a solved problem.
> 
> And then the "I cannot commit anymore" reports started pouring in.

We need to understand better the problems which caused this.  The
usual fallback for

  (expand-file-name "emacsclient" invocation-directory)

is

  (expand-file-name "emacsclient"
                    (expand-file-name "../lib-src/" invocation-directory))

and if that also fails, just use "emacsclient", i.e. find some version
of the client on PATH.  If you are saying this doesn't work, either,
then we need to understand why and in what cases, and then decide
what, if anything, to do about that.

Re: Adding with-editor to Emacs?

[Michael Albinus]

Jonas Bernoulli <jonas@bernoul.li> writes:

Hi Jonas,

> Probably and it would be great if Tramp did handle that, but I don't
> even use Tramp except when users report that there is an issue when
> using Tramp with one of my packages.  So I am not the right person to
> implement it there, but if Michael were to tackle this, then maybe I
> would have some insights that could be useful.  Or not.

I will check what's possible, but not the next week (I'm occupied
otherwise). I'll come back with questions to you ...

Best regards, Michael.

Re: Adding with-editor to Emacs?

[Jonas Bernoulli]

Michael Albinus <michael.albinus@gmx.de> writes:

> Jonas Bernoulli <jonas@bernoul.li> writes:
>
> Hi Jonas,
>
>> Probably and it would be great if Tramp did handle that, but I don't
>> even use Tramp except when users report that there is an issue when
>> using Tramp with one of my packages.  So I am not the right person to
>> implement it there, but if Michael were to tackle this, then maybe I
>> would have some insights that could be useful.  Or not.
>
> I will check what's possible, but not the next week (I'm occupied
> otherwise). I'll come back with questions to you ...
>
> Best regards, Michael.

There is absolutely no hurry.  I am also very much occupied and don't
really have the bandwidth to also deal with this right now.

Re: Adding with-editor to Emacs?

[Jonas Bernoulli]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jonas Bernoulli <jonas@bernoul.li>
>> Cc: stefankangas@gmail.com, emacs-devel@gnu.org, rms@gnu.org
>> Date: Fri, 01 Sep 2023 22:23:43 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> > Emacs knows very well where to find its corresponding emacsclient.
>> >> 
>> >> I wasn't aware of that.  How can I make use of that knowledge?  I.e.,
>> >> is there a function that answers the question "what is the path of the
>> >> emacsclient that was installed with this version of emacs"?
>> >
>> > When Emacs runs installed, emacsclient is in the same directory where
>> > we install the Emacs binary, so emacsclient should be in
>> > invocation-directory.
>> 
>> "I have seen things you wouldn't believe..."
>> 
>> I wish this was true, but unfortunately keep coming up with new and
>> innovative ways of not doing that.  The worst platform is macOS, but I
>> also had to add a special case for Debian.  Users or distributions may
>> install emacs in a weird location, and then symlink emacs onto $PATH
>> (but without doing the same for emacsclient).  It is also possible to
>> install multiple versions of emacs in the same directory and append
>> version strings to the installed binaries.  To select one of these
>> versions a user/distribution may add a symlink named symlink in the
>> same directory.  One may or may not do the same for emacsclient.  If
>> emacsclient isn't a symlink, then an ancient emacsclient may still be
>> leftover from an earlier manual installed that was not properly
>> uninstalled. ...
>
> Then I guess you should describe all those atrocities in detail, so
> that we could perhaps devise ways of handling it.

I do not have a list of those atrocities and I do not have the bandwidth
and motivation to compile that in the next few months.  I have code that
deals with it though (with-editor-locate-emacsclient and the functions
it uses).  Using git to trace the history of that code, would give you
commits with explanations and/or links to places were the issues that
are being addressed were described.

> Also, since the client-server protocol doesn't change much, there's
> usually no need to insist on invoking the emacsclient that came with
> this particular Emacs, you can use any other.

True, but because breaking changes have happened, this doesn't *always*
work.

By the way, my code does not insist on an exact match.  It first tries
to find an emacsclient with matching x.y.z, if that fails x.y, and if
that fails too, just x is also accepted.

Maybe, if that too failed, it would work in 99% of cases to just use
whatever emacsclient was found, completely ignoring what its version is.

I have decided to not do that because I have not enough experience
debugging particular client/server incompatibilities, to do that
efficiently, when confronted with a potentially bad user report.  I do
however have a lot of experience asking the right questions when a user
shows up saying "I cannot commit, the emacsclient cannot be found".

I am not saying this is the right way.  It is the approach I arrived at,
because it allows me to waste less time remote debugging other people's
messed up Emacs installations.

>> > If Emacs runs uninstalled, emacsclient is in ../lib-src/ relative to
>> > invocation-directory.  Whether Emacs runs installed or not is
>> > determined by the value of installation-directory: if it's nil, Emacs
>> > runs installed.
>> 
>> ... and that too.
>
> What do you mean? the value of installation-directory is calculated
> internally by Emacs.

All I meant to say, is that, even if nobody messed up their Emacs
installations, we would still have to try two different approaches to
find the "correct" emacsclient.  Having to handle two cases is not a
biggie, the messed up installations are.

This was supposed to be my fun way of saying that.  It wasn't important.
I also considered leaving that part unanswered.

>> >> I agree that there theoretically isn't a need for this library, but it
>> >> turned out that just setting EDITOR=emacsclient in the environment of
>> >> the sub process also doesn't cut it, because for many users (who never
>> >> use emacsclient directly), would have to add some configuration to make
>> >> it work.  The core and original functionality provided by with-editor,
>> >> is making the configuration unnecessary by using heuristics.
>> >
>> > Can you elaborate on the configuration that is needed?  I always
>> > thought that emacsclient worked for everybody OOTB.
>> 
>> Sadly that isn't the case, see above.
>
> In what ways does "the above" mean that emacsclient doesn't work OOTB?

Yes.

> Do you mean that emacsclient is installed in a place that just typing
> "emacsclient RET" at the shell prompt fails to run it?

Yes.

> If so, that's
> a broken installation, and Emacs shouldn't really try to fix that.  Or
> do you mean something else?

That's a reasonable decision, and I tend to agree.

However, I personally had no choice but to find a workaround.  I
switched my package from "git commit -m 'message that has already been
written from scratch'" to using "EDITOR=emacsclient git commit" as
intended by Git.  That made it possible to use messages (resp. message
templates) generated by Git and/or user/project git-hooks.  That's a
very useful feature, one could even say that not allowing the user to
do that, was a bug in my package.

I also was under the impression that it would be safe to do that, after
all EDITOR=emacsclient is all that is need, and surely we can rely on
emacsclient being located somewhere on PATH.  Little did I know how many
messed up Emacs installations there are in the wild.

And then the bug reports started coming in.  The workflows of a lot of
people were disrupted, and to some them it seemed like it was all my
fault.  Most people were respectful but there were also two or three
complete !@$!@#$!@#.

I had to come up with a solution, fast.  I chose to implement kludges.

Identifying the authors of the broken Emacs installations, contacting
them and explaining the issue to them, and then waiting for months/years
until the updates trickle down to users, was not an option.  I need a
solution now.  And this was such an exhausting experience, I did not
have the energy to *also* contact everyone who had messed up their Emacs
package.  And it is such a bad memory (it was the first time I got
massively attacked for publishing free software), that I am also not
volunteering to do that work now.

I am quite happy with with-editor in its current form.  It solves my
problem; users can commit using my code, even when their Emacs
installations are messed up.  Occasionally, maybe two or three times a
year, someone comes up with a new strange way to package Emacs; and when
I deal with it, adding yet another special case.  That's good enough for
me.

I never claimed with-editor should be added to Emacs and don't really
have any desire to do so.  Richard wants to add it, and I decided to
not stand in the way.  If you decide that this isn't the right fit for
Emacs, I am perfectly fine with that, and actually agree.

If you decide that with-editor isn't the right fit, but Emacs needs
something like it, that is also not completely unreasonable.  But I
am not volunteering to do that work.

But let's repeat what you said above:

> Do you mean that emacsclient is installed in a place that just typing
> "emacsclient RET" at the shell prompt fails to run it?  If so, that's
> a broken installation, and Emacs shouldn't really try to fix that.

I think this is a very reasonable for Emacs.  In other words, the best
course of action is to just forget the suggestion that with-editor is
added to Emacs.  There is no real need and nobody volunteering to do the
work anyway.

>> Normally these complications are the user's problem.  They may not even
>> use emacsclient.  If they don't actively choose to use emacsclient, they
>> will never know it is actually broken for them.  If they try to use it
>> and it doesn't work, they can decide whether it is worth trying to
>> figure out to work around those mistakes.  If they want to use it but
>> cannot fix it for themselves, they can contact the people who packaged
>> Emacs for them.
>> 
>> For me the situation was different, I started with something like this
>> in one of my packages, effectively forcing users to use "emacsclient":
>> 
>>   (let ((process-environment process-environment))
>>     (setenv "EDITOR"
>>             (expand-file-name "emacsclient" invocation-directory))
>>     (call-process "git" nil nil nil "commit"))
>> 
>> Like you I assumed this was a solved problem.
>> 
>> And then the "I cannot commit anymore" reports started pouring in.
>
> We need to understand better the problems which caused this.  The
> usual fallback for
>
>   (expand-file-name "emacsclient" invocation-directory)
>
> is
>
>   (expand-file-name "emacsclient"
>                     (expand-file-name "../lib-src/" invocation-directory))
>
> and if that also fails, just use "emacsclient", i.e. find some version
> of the client on PATH.  If you are saying this doesn't work, either,
> then we need to understand why and in what cases, and then decide
> what, if anything, to do about that.

The worst offenders are various "Emacs for macOS" packages, that
do "package Emacs the way we are supposed to do that on macOS".

I think it works something like this: All files that make up an Emacs
installation are put in a single tarball (or probably some other archive
format).  The tarball contains everything starting at "usr", though some
things get added and moved around (but not everyone does that the same
way).  Additionally the tarball contains some xml file that says which
of the contained binaries should be executed when the user clicks on the
tarball (aka "application") in the Finder.  There appears to be a
convention to relocate the primary binary to the top-level of the
tarball, and other binaries to a "bin" directory, which itself is
located at the top-level.

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Jonas Bernoulli <jonas@bernoul.li>
> Cc: stefankangas@gmail.com, emacs-devel@gnu.org, rms@gnu.org
> Date: Sat, 02 Sep 2023 20:12:40 +0200
> 
> > Then I guess you should describe all those atrocities in detail, so
> > that we could perhaps devise ways of handling it.
> 
> I do not have a list of those atrocities and I do not have the bandwidth
> and motivation to compile that in the next few months.  I have code that
> deals with it though (with-editor-locate-emacsclient and the functions
> it uses).  Using git to trace the history of that code, would give you
> commits with explanations and/or links to places were the issues that
> are being addressed were described.

It's not that easy: AFAIU, with-editor was part of Magit, and was
separated into a repository of its own not very long ago.  And the
answers to my questions seem to be before the split.

I also looked at the present code and found it to have quite a few
non-trivial parts whose purpose I couldn't easily explain or guess,
and which are not explained in the comments, either.

So, if you have no time or motivation to describe the problems you
tried to solve with that code, I guess someone else will need to find
out and describe the problems in a way that we could then consider for
inclusion.  I cannot myself afford digging through the Magit's Git
repository to find the description of these problems, sorry.

Or maybe Richard already knows the answers, since he thought this
should be added to Emacs.

Re: Adding with-editor to Emacs?

[Stefan Kangas]

Jonas Bernoulli <jonas@bernoul.li> writes:

> Identifying the authors of the broken Emacs installations, contacting
> them and explaining the issue to them, and then waiting for months/years
> until the updates trickle down to users, was not an option.  I need a
> solution now.  And this was such an exhausting experience, I did not
> have the energy to *also* contact everyone who had messed up their Emacs
> package.  And it is such a bad memory (it was the first time I got
> massively attacked for publishing free software), that I am also not
> volunteering to do that work now.

Wow, what a ride.  I admire your patience, is all I can say.

>> Do you mean that emacsclient is installed in a place that just typing
>> "emacsclient RET" at the shell prompt fails to run it?  If so, that's
>> a broken installation, and Emacs shouldn't really try to fix that.
>
> I think this is a very reasonable for Emacs.  In other words, the best
> course of action is to just forget the suggestion that with-editor is
> added to Emacs.  There is no real need and nobody volunteering to do the
> work anyway.

It sounds like with-editor for the most part contains workarounds for
broken Emacs installations?  Is there anything in use-package that does
not belong to that category, and that you therefore think *should*
really be fixed in Emacs?

Perhaps it would be worth focusing on just that part.

Re: Adding with-editor to Emacs?

[Jonas Bernoulli]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jonas Bernoulli <jonas@bernoul.li>
>> Cc: stefankangas@gmail.com, emacs-devel@gnu.org, rms@gnu.org
>> Date: Sat, 02 Sep 2023 20:12:40 +0200
>> 
>> > Then I guess you should describe all those atrocities in detail, so
>> > that we could perhaps devise ways of handling it.
>> 
>> I do not have a list of those atrocities and I do not have the bandwidth
>> and motivation to compile that in the next few months.  I have code that
>> deals with it though (with-editor-locate-emacsclient and the functions
>> it uses).  Using git to trace the history of that code, would give you
>> commits with explanations and/or links to places were the issues that
>> are being addressed were described.
>
> It's not that easy: AFAIU, with-editor was part of Magit, and was
> separated into a repository of its own not very long ago.  And the
> answers to my questions seem to be before the split.

That doesn't really complicate the archaeology, we would just have to
run "git log ..." twice instead of just once.  But yes, the history
is more ancient as the first casual "git log ..." in the with-editor
repository suggests.

> I also looked at the present code and found it to have quite a few
> non-trivial parts whose purpose I couldn't easily explain or guess,
> and which are not explained in the comments, either.

Yes and IMO it would even be fair to say the code is a bit weird, and
we likely could do a better job by first listing all the cases we have
to address and then start over.  Doing that would also risk breaking
something that was previously fixed.

> So, if you have no time or motivation to describe the problems you
> tried to solve with that code, I guess someone else will need to find
> out and describe the problems in a way that we could then consider for
> inclusion.  I cannot myself afford digging through the Magit's Git
> repository to find the description of these problems, sorry.

I don't expect you to do that.  And you don't expect me to do it either.
If someone else decides to do it, that would be great, but if not,
that's what it is.

That would change once a (future) package that is part of Emacs, needs
to something like with-editor's emacsclient kludge or the additional
features it provides.

Until then, I think we can shelve this.  Hell, I might even one day
decide to produce the list myself after all.  But for the last few
months "just one more thing" has just kept popping up, and I really want
to finally get around to work on the things that I actually want to work
on again.  It has been a while.

The server-window-alist thing is a different story, and as you said,
should be handled separately.  But that too is "just one more thing"
in a long sequence of "just one more"...

(Hm, in February I got a bit burnouty and took a break, which did
wonders.  It seem I am again approaching a point where that is the
logical course of action.  Sorry, if some of my "don't assign me any
more work, I did not volunteer" was a bit rude.)

> Or maybe Richard already knows the answers, since he thought this
> should be added to Emacs.

Re: Adding with-editor to Emacs?

[Jonas Bernoulli]

Stefan Kangas <stefankangas@gmail.com> writes:

> Jonas Bernoulli <jonas@bernoul.li> writes:
>
>> Identifying the authors of the broken Emacs installations, contacting
>> them and explaining the issue to them, and then waiting for months/years
>> until the updates trickle down to users, was not an option.  I need a
>> solution now.  And this was such an exhausting experience, I did not
>> have the energy to *also* contact everyone who had messed up their Emacs
>> package.  And it is such a bad memory (it was the first time I got
>> massively attacked for publishing free software), that I am also not
>> volunteering to do that work now.
>
> Wow, what a ride.  I admire your patience, is all I can say.

Thanks.  Sometimes I have to vent a bit, even if I usually end up
regretting to have done so in public.  It helps to hear some
understanding words.  We've all been there, sometimes things just
get to stressful.

>>> Do you mean that emacsclient is installed in a place that just typing
>>> "emacsclient RET" at the shell prompt fails to run it?  If so, that's
>>> a broken installation, and Emacs shouldn't really try to fix that.
>>
>> I think this is a very reasonable for Emacs.  In other words, the best
>> course of action is to just forget the suggestion that with-editor is
>> added to Emacs.  There is no real need and nobody volunteering to do the
>> work anyway.
>
> It sounds like with-editor for the most part contains workarounds for
> broken Emacs installations?

That was the original feature but now it also contains a poorman's
substitute for emacsclient/server that works processes started from
Emacs (but also including processes running on remote machines).

As far as I am concerned, the package was done then.  Then requests to
support various emacs shells came in, and of course this could also be
useful for async-shell-command, and it became complex enough to warrant
a manual.  I wrote the in org and export it to texi, and of course once
(if) with-editor is added to Emacs, I will be informed that the
generated texi is not up to snuff....

This all started with a rather reasonable feature that just depended on
things not being broken, and then spiraled completely out of control,
with people asking me to add just add one more feature, again and again,
because after all with-editor would be the logical place to implement it.

Oh no!  Some memory is coming back.  When it was originally suggested
that we switch from "git commit -m 'done'" to "EDITOR=emacsclient git
commit", I agreed that this was obviously the right thing to do, but I
actually also realized that doing so would be risky and wanted to do it
slowly as an opt-in feature to avoid breakage, but everyone was "no no,
that is totally safe" and talked me into just pulling the plug.

So I hope you all understand now why I get a bit touchy when being
asked to work on this just a little more.

> Is there anything in use-package that does
> not belong to that category, and that you therefore think *should*
> really be fixed in Emacs?
>
> Perhaps it would be worth focusing on just that part.

(use-package?  I'll assume you meant with-editor.)

Replacing server-window with server-window-alist or something like that.

Re: Adding with-editor to Emacs?

[Stefan Kangas]

Jonas Bernoulli <jonas@bernoul.li> writes:

> Replacing server-window with server-window-alist or something like that.

If you could file a feature request with the details, that'd be
appreciated.  No rush, of course.

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 2 Sep 2023 12:56:32 -0700
> Cc: emacs-devel@gnu.org, rms@gnu.org
> 
> Jonas Bernoulli <jonas@bernoul.li> writes:
> 
> > Identifying the authors of the broken Emacs installations, contacting
> > them and explaining the issue to them, and then waiting for months/years
> > until the updates trickle down to users, was not an option.  I need a
> > solution now.  And this was such an exhausting experience, I did not
> > have the energy to *also* contact everyone who had messed up their Emacs
> > package.  And it is such a bad memory (it was the first time I got
> > massively attacked for publishing free software), that I am also not
> > volunteering to do that work now.
> 
> Wow, what a ride.  I admire your patience, is all I can say.
> 
> >> Do you mean that emacsclient is installed in a place that just typing
> >> "emacsclient RET" at the shell prompt fails to run it?  If so, that's
> >> a broken installation, and Emacs shouldn't really try to fix that.
> >
> > I think this is a very reasonable for Emacs.  In other words, the best
> > course of action is to just forget the suggestion that with-editor is
> > added to Emacs.  There is no real need and nobody volunteering to do the
> > work anyway.
> 
> It sounds like with-editor for the most part contains workarounds for
> broken Emacs installations?  Is there anything in use-package that does
> not belong to that category, and that you therefore think *should*
> really be fixed in Emacs?

I was thinking about a function that would attempt to produce the best
guess for how to invoke the version of emacsclient which corresponds
to the running Emacs binary.  It is probably only important for
features like Magit, which invoke programs known to need $EDITOR.
That is quite a special case, which I think explains why no one
requested this before.  Still, it could be a useful addition.

To support the "usual" cases, Emacs running installed and uninstalled,
is easy enough.  Supporting program-name transformations specified by
the --program-prefix/suffix and --program-transform-name
configure-time option is a bit trickier, but still reasonably
straightforward, at least for some values of transformations.  The
final fallback should be just "emacsclient", to be found by the
system's program loader using its rules (which could include more than
just searching PATH).

However, to cover all the cases that with-editor supports now, we need
to understand them, because I'm not sure we want to support all of
them, or if we do, do it in the same way as with-editor does.  I
couldn't find that explanation in with-editor.el, and the code there
doesn't always explain itself, in particular where it alludes to macOS
and Debian quirks.

Re: Adding with-editor to Emacs?

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jonas Bernoulli <jonas@bernoul.li>
>> Cc: emacs-devel@gnu.org, rms@gnu.org
>> Date: Fri, 01 Sep 2023 19:44:53 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I'm probably missing something because if all we want is to allow
>> > child processes to use the current Emacs session as their editor, we
>> > just need to inject some environment variables into
>> > process-environment when running those child processes, and start the
>> > server.
>> 
>> That's the core of what with-editor does.  Additionally
>> 
>> - It tries hard to find the correct emacsclient to use.
>
> See below: I don't think I understand why this has to be "hard".
>
>> - It implements a "sleeping editor".  This is a shell script, which
>>   outputs a request on stdout and then waits to be told to return.
>>   With-editor use a process filter too look for that output and when
>>   it sees it, it responds in a similar fashion to server.el.  This
>>   is useful because makes it possible to do this over Tramp.  (I
>>   believe this could also be done using regular emacsclient+server.el,
>>   but that is difficult to setup and a security risk if not done
>>   correctly.
>
> If we want a better/safer client-server connections for remote hosts,
> it should be handled in Tramp, I think.
>
>> - It provides some convenience functionality to use this from various
>>   shells running inside Emacs.
>
> Can you provide details?  The above is too terse for me to understand
> the functionality.

Just my one 2 cents datapoint:

I have this line in my init.el
(add-hook 'eshell-mode-hook 'with-editor-export-editor)

and now, I can do a standard Unix "crontab -e" or "vipw" to edit those
special files from a remote eshell (be it via sudo, ssh whatever).
-- 
Manuel Giraud

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: Jonas Bernoulli <jonas@bernoul.li>,  stefankangas@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org
> Date: Sun, 03 Sep 2023 16:36:17 +0200
> 
> >> - It provides some convenience functionality to use this from various
> >>   shells running inside Emacs.
> >
> > Can you provide details?  The above is too terse for me to understand
> > the functionality.
> 
> Just my one 2 cents datapoint:
> 
> I have this line in my init.el
> (add-hook 'eshell-mode-hook 'with-editor-export-editor)
> 
> and now, I can do a standard Unix "crontab -e" or "vipw" to edit those
> special files from a remote eshell (be it via sudo, ssh whatever).

So we are talking about EDITOR settings missing from the shell init
files?

(I believe "M-x shell" does read the shell's init file, doesn't it?)

Re: Adding with-editor to Emacs?

[Lynn Winebarger]

On Sat, Sep 2, 2023 at 2:58 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Jonas Bernoulli <jonas@bernoul.li>
> > Cc: stefankangas@gmail.com, emacs-devel@gnu.org, rms@gnu.org
> > Date: Sat, 02 Sep 2023 20:12:40 +0200
> >
> > > Then I guess you should describe all those atrocities in detail, so
> > > that we could perhaps devise ways of handling it.
> >
> > I do not have a list of those atrocities and I do not have the bandwidth
> > and motivation to compile that in the next few months.  I have code that
> > deals with it though (with-editor-locate-emacsclient and the functions
> > it uses).  Using git to trace the history of that code, would give you
> > commits with explanations and/or links to places were the issues that
> > are being addressed were described.
>
> It's not that easy: AFAIU, with-editor was part of Magit, and was
> separated into a repository of its own not very long ago.  And the
> answers to my questions seem to be before the split.
>
> I also looked at the present code and found it to have quite a few
> non-trivial parts whose purpose I couldn't easily explain or guess,
> and which are not explained in the comments, either.
>
> So, if you have no time or motivation to describe the problems you
> tried to solve with that code, I guess someone else will need to find
> out and describe the problems in a way that we could then consider for
> inclusion.  I cannot myself afford digging through the Magit's Git
> repository to find the description of these problems, sorry.
>
> Or maybe Richard already knows the answers, since he thought this
> should be added to Emacs.
>

Just to put it out there, wouldn't putting the basic function in core
emacs, without all the workarounds for packaging fails, put the onus
on the packager to either adhere to the standard layout or patch their
distribution to make the function work accordingly?  That seems like
one of the benefits of a package being included in the core.

Lynn

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sun, 3 Sep 2023 13:02:49 -0400
> Cc: Jonas Bernoulli <jonas@bernoul.li>, stefankangas@gmail.com, emacs-devel@gnu.org, 
> 	rms@gnu.org
> 
> Just to put it out there, wouldn't putting the basic function in core
> emacs, without all the workarounds for packaging fails, put the onus
> on the packager to either adhere to the standard layout or patch their
> distribution to make the function work accordingly?  That seems like
> one of the benefits of a package being included in the core.

I don't think I understand what you are saying here.  At least one
sentence failed to parse.

Re: Adding with-editor to Emacs?

[Lynn Winebarger]

On Sun, Sep 3, 2023 at 1:22 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Lynn Winebarger <owinebar@gmail.com>
> >
> > Just to put it out there, wouldn't putting the basic function in core
> > emacs, without all the workarounds for packaging fails, put the onus
> > on the packager to either adhere to the standard layout or patch their
> > distribution to make the function work accordingly?  That seems like
> > one of the benefits of a package being included in the core.
>
> I don't think I understand what you are saying here.  At least one
> sentence failed to parse.

It seemed to me the problems Jonas described with finding the correct
emacsclient binary resulted from the library being in an external
package rather than part of the GNU Emacs distribution.
That is, if the "with-editor" function were
* implemented in a core emacs library following the simple approach
you suggested, based on the built-in expected location of emacsclient
* had a test to verify this function works when installed
then failures due to odd packaging conventions or plain oversight
would be the responsibility of the emacs packager/distribution.  The
failures could be fixed either by adopting the standard layout or by
patching the emacs distribution.  Either way, it would not be the
concern of the core emacs maintainer of the library containing the
with-editor function.

That is one of the benefits of being upstream  from emacs
distributions (as a GNU emacs contributor) instead of "downstream" of
the distribution as a package author.  At least, it would seem that
way.

Lynn

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sun, 3 Sep 2023 14:21:46 -0400
> Cc: jonas@bernoul.li, stefankangas@gmail.com, emacs-devel@gnu.org, rms@gnu.org
> 
> On Sun, Sep 3, 2023 at 1:22 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > > From: Lynn Winebarger <owinebar@gmail.com>
> > >
> > > Just to put it out there, wouldn't putting the basic function in core
> > > emacs, without all the workarounds for packaging fails, put the onus
> > > on the packager to either adhere to the standard layout or patch their
> > > distribution to make the function work accordingly?  That seems like
> > > one of the benefits of a package being included in the core.
> >
> > I don't think I understand what you are saying here.  At least one
> > sentence failed to parse.
> 
> It seemed to me the problems Jonas described with finding the correct
> emacsclient binary resulted from the library being in an external
> package rather than part of the GNU Emacs distribution.

That's not how I understood what Jonas was saying.  He said he started
with the simple approach I suggested, but it wasn't enough, it still
failed in some cases.

Re: Adding with-editor to Emacs?

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Manuel Giraud <manuel@ledu-giraud.fr>
>> Cc: Jonas Bernoulli <jonas@bernoul.li>,  stefankangas@gmail.com,
>>   emacs-devel@gnu.org,  rms@gnu.org
>> Date: Sun, 03 Sep 2023 16:36:17 +0200
>> 
>> >> - It provides some convenience functionality to use this from various
>> >>   shells running inside Emacs.
>> >
>> > Can you provide details?  The above is too terse for me to understand
>> > the functionality.
>> 
>> Just my one 2 cents datapoint:
>> 
>> I have this line in my init.el
>> (add-hook 'eshell-mode-hook 'with-editor-export-editor)
>> 
>> and now, I can do a standard Unix "crontab -e" or "vipw" to edit those
>> special files from a remote eshell (be it via sudo, ssh whatever).
>
> So we are talking about EDITOR settings missing from the shell init
> files?
>
> (I believe "M-x shell" does read the shell's init file, doesn't it?)

Not really I guess.  I don't know how to set EDITOR correctly to do the
following:

   - M-x shell
   - ssh remote-server
   - su -l
   - vipw  --> open a new buffer in *this* Emacs

But, I could do this with with-editor and eshell (maybe it could work
for M-x shell also, I don't know) as follow:

   - C-x C-f /ssh:remote-server|su::
   - M-x eshell
   - vipw  --> and here it works, opening an Emacs buffer through
               emacsclient so I could edit and C-c C-c when done

I am not an expert of with-editor but it is one that it does well.
-- 
Manuel Giraud

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: jonas@bernoul.li,  stefankangas@gmail.com,  emacs-devel@gnu.org,
>   rms@gnu.org
> Date: Sun, 03 Sep 2023 20:54:47 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Manuel Giraud <manuel@ledu-giraud.fr>
> >> Cc: Jonas Bernoulli <jonas@bernoul.li>,  stefankangas@gmail.com,
> >>   emacs-devel@gnu.org,  rms@gnu.org
> >> Date: Sun, 03 Sep 2023 16:36:17 +0200
> >> 
> >> >> - It provides some convenience functionality to use this from various
> >> >>   shells running inside Emacs.
> >> >
> >> > Can you provide details?  The above is too terse for me to understand
> >> > the functionality.
> >> 
> >> Just my one 2 cents datapoint:
> >> 
> >> I have this line in my init.el
> >> (add-hook 'eshell-mode-hook 'with-editor-export-editor)
> >> 
> >> and now, I can do a standard Unix "crontab -e" or "vipw" to edit those
> >> special files from a remote eshell (be it via sudo, ssh whatever).
> >
> > So we are talking about EDITOR settings missing from the shell init
> > files?
> >
> > (I believe "M-x shell" does read the shell's init file, doesn't it?)
> 
> Not really I guess.  I don't know how to set EDITOR correctly to do the
> following:
> 
>    - M-x shell
>    - ssh remote-server
>    - su -l
>    - vipw  --> open a new buffer in *this* Emacs
> 
> But, I could do this with with-editor and eshell (maybe it could work
> for M-x shell also, I don't know) as follow:
> 
>    - C-x C-f /ssh:remote-server|su::
>    - M-x eshell
>    - vipw  --> and here it works, opening an Emacs buffer through
>                emacsclient so I could edit and C-c C-c when done

You've lost me here.  You assume I know what with-editor does?

Re: Adding with-editor to Emacs?

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> Not really I guess.  I don't know how to set EDITOR correctly to do the
>> following:
>> 
>>    - M-x shell
>>    - ssh remote-server
>>    - su -l
>>    - vipw  --> open a new buffer in *this* Emacs
>> 
>> But, I could do this with with-editor and eshell (maybe it could work
>> for M-x shell also, I don't know) as follow:
>> 
>>    - C-x C-f /ssh:remote-server|su::
>>    - M-x eshell
>>    - vipw  --> and here it works, opening an Emacs buffer through
>>                emacsclient so I could edit and C-c C-c when done
>
> You've lost me here.  You assume I know what with-editor does?

Ok, I might be lost also (as I said I'm not an expert of everything
with-editor does).  I'll try to rephrase.  with-editor is able to
correctly set EDITOR to have any shell command that need editing appear
into an Emacs buffer locally.  Such command might be over a ssh session
or into a sudo for instance.

So I guess you're right it is just a matter of setting EDITOR after all.
But I think that this is the hard part that with-editor does for you.
-- 
Manuel Giraud

Re: Adding with-editor to Emacs?

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: jonas@bernoul.li,  stefankangas@gmail.com,  emacs-devel@gnu.org,
>   rms@gnu.org
> Date: Mon, 04 Sep 2023 10:21:15 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> [...]
> 
> >> Not really I guess.  I don't know how to set EDITOR correctly to do the
> >> following:
> >> 
> >>    - M-x shell
> >>    - ssh remote-server
> >>    - su -l
> >>    - vipw  --> open a new buffer in *this* Emacs
> >> 
> >> But, I could do this with with-editor and eshell (maybe it could work
> >> for M-x shell also, I don't know) as follow:
> >> 
> >>    - C-x C-f /ssh:remote-server|su::
> >>    - M-x eshell
> >>    - vipw  --> and here it works, opening an Emacs buffer through
> >>                emacsclient so I could edit and C-c C-c when done
> >
> > You've lost me here.  You assume I know what with-editor does?
> 
> Ok, I might be lost also (as I said I'm not an expert of everything
> with-editor does).  I'll try to rephrase.  with-editor is able to
> correctly set EDITOR to have any shell command that need editing appear
> into an Emacs buffer locally.  Such command might be over a ssh session
> or into a sudo for instance.

And it provides a separate value for EDITOR to each one of these
cases?

> So I guess you're right it is just a matter of setting EDITOR after all.
> But I think that this is the hard part that with-editor does for you.

Re: Adding with-editor to Emacs?

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Manuel Giraud <manuel@ledu-giraud.fr>
>> Cc: jonas@bernoul.li,  stefankangas@gmail.com,  emacs-devel@gnu.org,
>>   rms@gnu.org
>> Date: Mon, 04 Sep 2023 10:21:15 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> [...]
>> 
>> >> Not really I guess.  I don't know how to set EDITOR correctly to do the
>> >> following:
>> >> 
>> >>    - M-x shell
>> >>    - ssh remote-server
>> >>    - su -l
>> >>    - vipw  --> open a new buffer in *this* Emacs
>> >> 
>> >> But, I could do this with with-editor and eshell (maybe it could work
>> >> for M-x shell also, I don't know) as follow:
>> >> 
>> >>    - C-x C-f /ssh:remote-server|su::
>> >>    - M-x eshell
>> >>    - vipw  --> and here it works, opening an Emacs buffer through
>> >>                emacsclient so I could edit and C-c C-c when done
>> >
>> > You've lost me here.  You assume I know what with-editor does?
>> 
>> Ok, I might be lost also (as I said I'm not an expert of everything
>> with-editor does).  I'll try to rephrase.  with-editor is able to
>> correctly set EDITOR to have any shell command that need editing appear
>> into an Emacs buffer locally.  Such command might be over a ssh session
>> or into a sudo for instance.
>
> And it provides a separate value for EDITOR to each one of these
> cases?

This I'm not sure.  I have tested on two different machines and on a
local "doas" and echo $EDITOR always returns this:

sh -c 'printf "\nWITH-EDITOR: $$ OPEN $0\037$1\037 IN $(pwd)\n"; sleep 604800 & sleep=$!; trap "kill $sleep; exit 0" USR1; trap "kill $sleep; exit 1" USR2; wait $sleep'

So it seems there is some shell hackery going on.
-- 
Manuel Giraud

Re: Adding with-editor to Emacs?

[Manuel Giraud via Emacs development discussions.]

Hi,

FWIW, I've just made the following test with the bundle VC support
of Emacs:

        ⋅ 'C-x C-f' /ssh:remote-server:/tmp/
        ⋅ '+' new-project
        ⋅ 'C-x C-f' new-project/new-file
            … enter some text … 
        ⋅ 'C-x v v' choose the Git backend
        ⋅ 'C-x v v' enter the first log message
        ⋅ 'C-c C-c'

AFAIK, there is no use of with-editor here.  So as with-editor seems to
be used to do such remote editing into Magit maybe there is difference
in philosophy at play.
-- 
Manuel Giraud

Re: Adding with-editor to Emacs?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You've lost me here.  You assume I know what with-editor does?

I've been puzzled because I thought that the editor in Emacs was Emacs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Adding with-editor to Emacs?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Ok, I might be lost also (as I said I'm not an expert of everything
  > with-editor does).  I'll try to rephrase.  with-editor is able to
  > correctly set EDITOR to have any shell command that need editing appear
  > into an Emacs buffer locally.  Such command might be over a ssh session
  > or into a sudo for instance.

Now I understand the idea -- but that _name_ does not explain it well.
How about renaming it to something easier to understand.

Maybe with-editor-envvar-to-edit-cmd-in-emacs?

Perhaps it could be a little shorter tan that, but at least
that one will be clear.  If I haven't misunderstood the meaning myself;-{
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Adding with-editor to Emacs?

[Björn Bidar]

Richard Stallman <rms@gnu.org> writes:

>   > You've lost me here.  You assume I know what with-editor does?
>
> I've been puzzled because I thought that the editor in Emacs was Emacs.

It's about telling the external programs to do THING /with-editor/,
editor because the $EDITOR environment that is used.
Externally be it remote or locally. I think doing the guess work to set
$EDITOR correctly in each instance is to difficult to be done in the
individual shells configuration, especially since they don't know about and
can't assume tramp.

Re: Adding with-editor to Emacs?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's about telling the external programs to do THING /with-editor/,
  > editor because the $EDITOR environment that is used.

Now I see how it is useful, but I suggest we change its name before
installing it in Emacs or either ELPA.  If we call it
`with-EDITOR-envvar' it purpose will be much clearer.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Adding with-editor to Emacs?

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > It's about telling the external programs to do THING /with-editor/,
>   > editor because the $EDITOR environment that is used.
>
> Now I see how it is useful, but I suggest we change its name before
> installing it in Emacs or either ELPA.  If we call it
> `with-EDITOR-envvar' it purpose will be much clearer.

As someone who also frequently has difficulties understanding how people
come up with package names, I don't see how your suggestion is clearer
than the current name.

-- 
Philip Kaludercic

Re: Adding with-editor to Emacs?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Now I see how it is useful, but I suggest we change its name before
  > > installing it in Emacs or either ELPA.  If we call it
  > > `with-EDITOR-envvar' it purpose will be much clearer.

  > As someone who also frequently has difficulties understanding how people
  > come up with package names, I don't see how your suggestion is clearer
  > than the current name.

I'm suggesting a new name for the macro, primarily, but it could
be used for the package too.

The purpose of the construct is to bind the envvar EDITOR for an
invocation of another process.  `with-editor' is unclear -- when I saw
it, I could not begin to imagine what that could mean.
`with-EDITOR-envvar' says that it has to do with binding the envvar
named EDITOR.

If someone presents a better suggestion, that is good.
I'm only saying that this name is better than `with-editor'.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Adding with-editor to Emacs?

[Michael Albinus]

Jonas Bernoulli <jonas@bernoul.li> writes:

Hi Jonas,

>>> - It implements a "sleeping editor".  This is a shell script, which
>>>   outputs a request on stdout and then waits to be told to return.
>>>   With-editor use a process filter too look for that output and when
>>>   it sees it, it responds in a similar fashion to server.el.  This
>>>   is useful because makes it possible to do this over Tramp.  (I
>>>   believe this could also be done using regular emacsclient+server.el,
>>>   but that is difficult to setup and a security risk if not done
>>>   correctly.
>>
>> If we want a better/safer client-server connections for remote hosts,
>> it should be handled in Tramp, I think.
>
> Probably and it would be great if Tramp did handle that, but I don't
> even use Tramp except when users report that there is an issue when
> using Tramp with one of my packages.  So I am not the right person to
> implement it there, but if Michael were to tackle this, then maybe I
> would have some insights that could be useful.  Or not.

Finally, I've found a free time slot to check with-editor. In fact,
emacsclient can also be called on a remote machine in order to reach the
local Emacs server instance. What is needed is, that emacsclient must
tell the Emacs server where it is located.

Since Emacs 26, emacsclient is prepared for this. If you call
"emacsclient --tramp=<PREFIX>", all file names on the server side,
emacsclient sends as "/path/to/file", are "<PREFIX>/path/to/file".

If you detect a remote directory in with-editor, you can add the option
(concat "--tramp=" (file-remote-p default-directory)) to the emacsclient
call, because file-remote-p returns exactly the needed prefix.

Best regards, Michael.

Re: Adding with-editor to Emacs?

[Manuel Giraud via Emacs development discussions.]

Michael Albinus <michael.albinus@gmx.de> writes:

> Jonas Bernoulli <jonas@bernoul.li> writes:
>
> Hi Jonas,
>
>>>> - It implements a "sleeping editor".  This is a shell script, which
>>>>   outputs a request on stdout and then waits to be told to return.
>>>>   With-editor use a process filter too look for that output and when
>>>>   it sees it, it responds in a similar fashion to server.el.  This
>>>>   is useful because makes it possible to do this over Tramp.  (I
>>>>   believe this could also be done using regular emacsclient+server.el,
>>>>   but that is difficult to setup and a security risk if not done
>>>>   correctly.
>>>
>>> If we want a better/safer client-server connections for remote hosts,
>>> it should be handled in Tramp, I think.
>>
>> Probably and it would be great if Tramp did handle that, but I don't
>> even use Tramp except when users report that there is an issue when
>> using Tramp with one of my packages.  So I am not the right person to
>> implement it there, but if Michael were to tackle this, then maybe I
>> would have some insights that could be useful.  Or not.
>
> Finally, I've found a free time slot to check with-editor. In fact,
> emacsclient can also be called on a remote machine in order to reach the
> local Emacs server instance. What is needed is, that emacsclient must
> tell the Emacs server where it is located.

Do you an emacsclient on the server to do this?  AFAIU, with-editor does
not need this.
-- 
Manuel Giraud

Re: Adding with-editor to Emacs?

[Michael Albinus]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

Hi Manuel,

>>>> If we want a better/safer client-server connections for remote hosts,
>>>> it should be handled in Tramp, I think.
>>>
>>> Probably and it would be great if Tramp did handle that, but I don't
>>> even use Tramp except when users report that there is an issue when
>>> using Tramp with one of my packages.  So I am not the right person to
>>> implement it there, but if Michael were to tackle this, then maybe I
>>> would have some insights that could be useful.  Or not.
>>
>> Finally, I've found a free time slot to check with-editor. In fact,
>> emacsclient can also be called on a remote machine in order to reach the
>> local Emacs server instance. What is needed is, that emacsclient must
>> tell the Emacs server where it is located.
>
> Do you an emacsclient on the server to do this?  AFAIU, with-editor does
> not need this.

with-editor has its own implementation (a shell script) for communication
between the local Emacs server and a remote host. I have been asked how
this could be replaced by using a remote emacsclient and
Tramp. Especially, how the file names should look like. IIUC the
with-editor code, emacsclient is not used in the remote case.

Best regards, Michael.

Re: Adding with-editor to Emacs?

[Manuel Giraud via Emacs development discussions.]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Michael,

>>>>> If we want a better/safer client-server connections for remote hosts,
>>>>> it should be handled in Tramp, I think.
>>>>
>>>> Probably and it would be great if Tramp did handle that, but I don't
>>>> even use Tramp except when users report that there is an issue when
>>>> using Tramp with one of my packages.  So I am not the right person to
>>>> implement it there, but if Michael were to tackle this, then maybe I
>>>> would have some insights that could be useful.  Or not.
>>>
>>> Finally, I've found a free time slot to check with-editor. In fact,
>>> emacsclient can also be called on a remote machine in order to reach the
>>> local Emacs server instance. What is needed is, that emacsclient must
>>> tell the Emacs server where it is located.
>>
>> Do you an emacsclient on the server to do this?  AFAIU, with-editor does
>> not need this.
>
> with-editor has its own implementation (a shell script) for communication
> between the local Emacs server and a remote host. I have been asked how
> this could be replaced by using a remote emacsclient and
> Tramp. Especially, how the file names should look like.

Ok, thanks for this explanation.  But I don't think that requiring
emacsclient on the server will be very convenient.

> IIUC the with-editor code, emacsclient is not used in the remote case.

I understood the same: in the remote case, EDITOR is set to a shell
script hack.  BTW, I don't know what is the use case of with-editor if
it is not remote: in the local case, "export EDITOR=emacsclient" should
be sufficient.
-- 
Manuel Giraud