Re: Adding with-editor to Emacs?

[Jonas Bernoulli <jonas@bernoul.li>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jonas Bernoulli <jonas@bernoul.li>
>> Cc: stefankangas@gmail.com, emacs-devel@gnu.org, rms@gnu.org
>> Date: Fri, 01 Sep 2023 22:23:43 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> > Emacs knows very well where to find its corresponding emacsclient.
>> >> 
>> >> I wasn't aware of that.  How can I make use of that knowledge?  I.e.,
>> >> is there a function that answers the question "what is the path of the
>> >> emacsclient that was installed with this version of emacs"?
>> >
>> > When Emacs runs installed, emacsclient is in the same directory where
>> > we install the Emacs binary, so emacsclient should be in
>> > invocation-directory.
>> 
>> "I have seen things you wouldn't believe..."
>> 
>> I wish this was true, but unfortunately keep coming up with new and
>> innovative ways of not doing that.  The worst platform is macOS, but I
>> also had to add a special case for Debian.  Users or distributions may
>> install emacs in a weird location, and then symlink emacs onto $PATH
>> (but without doing the same for emacsclient).  It is also possible to
>> install multiple versions of emacs in the same directory and append
>> version strings to the installed binaries.  To select one of these
>> versions a user/distribution may add a symlink named symlink in the
>> same directory.  One may or may not do the same for emacsclient.  If
>> emacsclient isn't a symlink, then an ancient emacsclient may still be
>> leftover from an earlier manual installed that was not properly
>> uninstalled. ...
>
> Then I guess you should describe all those atrocities in detail, so
> that we could perhaps devise ways of handling it.

I do not have a list of those atrocities and I do not have the bandwidth
and motivation to compile that in the next few months.  I have code that
deals with it though (with-editor-locate-emacsclient and the functions
it uses).  Using git to trace the history of that code, would give you
commits with explanations and/or links to places were the issues that
are being addressed were described.

> Also, since the client-server protocol doesn't change much, there's
> usually no need to insist on invoking the emacsclient that came with
> this particular Emacs, you can use any other.

True, but because breaking changes have happened, this doesn't *always*
work.

By the way, my code does not insist on an exact match.  It first tries
to find an emacsclient with matching x.y.z, if that fails x.y, and if
that fails too, just x is also accepted.

Maybe, if that too failed, it would work in 99% of cases to just use
whatever emacsclient was found, completely ignoring what its version is.

I have decided to not do that because I have not enough experience
debugging particular client/server incompatibilities, to do that
efficiently, when confronted with a potentially bad user report.  I do
however have a lot of experience asking the right questions when a user
shows up saying "I cannot commit, the emacsclient cannot be found".

I am not saying this is the right way.  It is the approach I arrived at,
because it allows me to waste less time remote debugging other people's
messed up Emacs installations.

>> > If Emacs runs uninstalled, emacsclient is in ../lib-src/ relative to
>> > invocation-directory.  Whether Emacs runs installed or not is
>> > determined by the value of installation-directory: if it's nil, Emacs
>> > runs installed.
>> 
>> ... and that too.
>
> What do you mean? the value of installation-directory is calculated
> internally by Emacs.

All I meant to say, is that, even if nobody messed up their Emacs
installations, we would still have to try two different approaches to
find the "correct" emacsclient.  Having to handle two cases is not a
biggie, the messed up installations are.

This was supposed to be my fun way of saying that.  It wasn't important.
I also considered leaving that part unanswered.

>> >> I agree that there theoretically isn't a need for this library, but it
>> >> turned out that just setting EDITOR=emacsclient in the environment of
>> >> the sub process also doesn't cut it, because for many users (who never
>> >> use emacsclient directly), would have to add some configuration to make
>> >> it work.  The core and original functionality provided by with-editor,
>> >> is making the configuration unnecessary by using heuristics.
>> >
>> > Can you elaborate on the configuration that is needed?  I always
>> > thought that emacsclient worked for everybody OOTB.
>> 
>> Sadly that isn't the case, see above.
>
> In what ways does "the above" mean that emacsclient doesn't work OOTB?

Yes.

> Do you mean that emacsclient is installed in a place that just typing
> "emacsclient RET" at the shell prompt fails to run it?

Yes.

> If so, that's
> a broken installation, and Emacs shouldn't really try to fix that.  Or
> do you mean something else?

That's a reasonable decision, and I tend to agree.

However, I personally had no choice but to find a workaround.  I
switched my package from "git commit -m 'message that has already been
written from scratch'" to using "EDITOR=emacsclient git commit" as
intended by Git.  That made it possible to use messages (resp. message
templates) generated by Git and/or user/project git-hooks.  That's a
very useful feature, one could even say that not allowing the user to
do that, was a bug in my package.

I also was under the impression that it would be safe to do that, after
all EDITOR=emacsclient is all that is need, and surely we can rely on
emacsclient being located somewhere on PATH.  Little did I know how many
messed up Emacs installations there are in the wild.

And then the bug reports started coming in.  The workflows of a lot of
people were disrupted, and to some them it seemed like it was all my
fault.  Most people were respectful but there were also two or three
complete !@$!@#$!@#.

I had to come up with a solution, fast.  I chose to implement kludges.

Identifying the authors of the broken Emacs installations, contacting
them and explaining the issue to them, and then waiting for months/years
until the updates trickle down to users, was not an option.  I need a
solution now.  And this was such an exhausting experience, I did not
have the energy to *also* contact everyone who had messed up their Emacs
package.  And it is such a bad memory (it was the first time I got
massively attacked for publishing free software), that I am also not
volunteering to do that work now.

I am quite happy with with-editor in its current form.  It solves my
problem; users can commit using my code, even when their Emacs
installations are messed up.  Occasionally, maybe two or three times a
year, someone comes up with a new strange way to package Emacs; and when
I deal with it, adding yet another special case.  That's good enough for
me.

I never claimed with-editor should be added to Emacs and don't really
have any desire to do so.  Richard wants to add it, and I decided to
not stand in the way.  If you decide that this isn't the right fit for
Emacs, I am perfectly fine with that, and actually agree.

If you decide that with-editor isn't the right fit, but Emacs needs
something like it, that is also not completely unreasonable.  But I
am not volunteering to do that work.

But let's repeat what you said above:

> Do you mean that emacsclient is installed in a place that just typing
> "emacsclient RET" at the shell prompt fails to run it?  If so, that's
> a broken installation, and Emacs shouldn't really try to fix that.

I think this is a very reasonable for Emacs.  In other words, the best
course of action is to just forget the suggestion that with-editor is
added to Emacs.  There is no real need and nobody volunteering to do the
work anyway.

>> Normally these complications are the user's problem.  They may not even
>> use emacsclient.  If they don't actively choose to use emacsclient, they
>> will never know it is actually broken for them.  If they try to use it
>> and it doesn't work, they can decide whether it is worth trying to
>> figure out to work around those mistakes.  If they want to use it but
>> cannot fix it for themselves, they can contact the people who packaged
>> Emacs for them.
>> 
>> For me the situation was different, I started with something like this
>> in one of my packages, effectively forcing users to use "emacsclient":
>> 
>>   (let ((process-environment process-environment))
>>     (setenv "EDITOR"
>>             (expand-file-name "emacsclient" invocation-directory))
>>     (call-process "git" nil nil nil "commit"))
>> 
>> Like you I assumed this was a solved problem.
>> 
>> And then the "I cannot commit anymore" reports started pouring in.
>
> We need to understand better the problems which caused this.  The
> usual fallback for
>
>   (expand-file-name "emacsclient" invocation-directory)
>
> is
>
>   (expand-file-name "emacsclient"
>                     (expand-file-name "../lib-src/" invocation-directory))
>
> and if that also fails, just use "emacsclient", i.e. find some version
> of the client on PATH.  If you are saying this doesn't work, either,
> then we need to understand why and in what cases, and then decide
> what, if anything, to do about that.

The worst offenders are various "Emacs for macOS" packages, that
do "package Emacs the way we are supposed to do that on macOS".

I think it works something like this: All files that make up an Emacs
installation are put in a single tarball (or probably some other archive
format).  The tarball contains everything starting at "usr", though some
things get added and moved around (but not everyone does that the same
way).  Additionally the tarball contains some xml file that says which
of the contained binaries should be executed when the user clicks on the
tarball (aka "application") in the Finder.  There appears to be a
convention to relocate the primary binary to the top-level of the
tarball, and other binaries to a "bin" directory, which itself is
located at the top-level.