problems with emacsclient started by cygwin programs (mswindows)

problems with emacsclient started by cygwin programs (mswindows)

[layer]

I have a general problem with emacsclient on mswindows, when the
program calling emacsclient is a cygwin program (e.g., git).  The
cygwin program passes command line arguments to emacsclient using
cygwin path conventions (e.g., /home/layer/...) and emacs doesn't know
what to do with them.

So, I propose and will implement the agreed upon changes to server.el
which add a hook that people can define that will take care of this
situation.

I'm mailing the list because I want to declare I'm interested in
solving this problem and give people a chance to comment.

I think the changes are pretty straightforward.  The section of the
code in question (from emacs 23.2) is:

(defun* server-process-filter (proc string)
  ...
		 ;; -file FILENAME:  Load the given file.
		 ((and (equal "-file" arg)
		       command-line-args-left)
		  (let ((file (pop command-line-args-left)))
		    (if coding-system
			(setq file (decode-coding-string file coding-system)))
                    (setq file (expand-file-name file dir))
		    (push (cons file filepos) files)
		    (server-log (format "New file: %s %s"
                                        file (or filepos "")) proc))
		  (setq filepos nil))

The place for the hook, IMO, is just before the expand-file-name.

I've read 
  http://www.gnu.org/software/emacs/elisp/html_node/Hooks.html
and regarding the name for the hook... I can't think of a really good
name, but something like

  server-filename-frob-function

is all that comes to mind.  `frob' does appear a few other times in
23.2, and my use does appear to be in line with what's there.  I'm
certainly receptive to other names.  Being an "abnormal" hook function
(according to the definition at the URL above), I suffixed it with
-function, as suggested.  The hook will take a single argument, the
filename given to -file by emacsclient, and return the new filename to
use.

Comments?  Questions?

Thanks.

Kevin

Re: problems with emacsclient started by cygwin programs (mswindows)

[layer]

OK, changes below for my proposed hook.  Comments welcome.

If I've not submitted this correctly and there's a FAQ about
submitting, please point me to it and I'll gladly do it some other
way.

Thanks.

Kevin

$ diff -u /usr/share/emacs/23.2/lisp/server.el server.el
--- /usr/share/emacs/23.2/lisp/server.el	2010-04-10 12:31:19.000000000 -0700
+++ server.el	2011-06-27 13:36:14.000000000 -0700
@@ -145,6 +145,14 @@
   :group 'server
   :type 'hook)
 
+(defcustom server-filename-frob-function nil
+  "Hook run on filename passed by client (using the -file in the protocol)
+used to possibly transform the filename in a form better suited for use
+by the Emacs server.  This might include transforming Cygwin paths on
+Windows into something the Emacs server can use."
+  :group 'server
+  :type 'hook)
+
 (defvar server-process nil
   "The current server process.")
 
@@ -988,6 +996,8 @@
 		  (let ((file (pop command-line-args-left)))
 		    (if coding-system
 			(setq file (decode-coding-string file coding-system)))
+		    (when server-filename-frob-function
+		      (setq file (funcall server-filename-frob-function file)))
                     (setq file (expand-file-name file dir))
 		    (push (cons file filepos) files)
 		    (server-log (format "New file: %s %s"
$ 

Re: problems with emacsclient started by cygwin programs (mswindows)

[Stefan Monnier]

> OK, changes below for my proposed hook.  Comments welcome.
> If I've not submitted this correctly and there's a FAQ about
> submitting, please point me to it and I'll gladly do it some other
> way.

I don't know Windows well enough to know exactly what should happen, but
here are some thoughts:
- you can solve your problem with something like cywin-mount.el.
  I know Eli doesn't like cywin-mount.el, but I think we should at least
  add such a feature to GNU ELPA.
- It might still make sense to want to understand cygwin-style file
  names from emacsclient without wanting cygwin-mount (which tries to
  handle cywin file names *everywhere*).  But then it should probably
  work not only for emacsclient but also for file name args to emacs
  itself (as a general rule, we should strive to make `emacs' and
  `emacsclient' command line handling more similar rather than more
  different).


        Stefan

Re: problems with emacsclient started by cygwin programs (mswindows)

[layer]

Stefan Monnier wrote:

>> > OK, changes below for my proposed hook.  Comments welcome.
>> > If I've not submitted this correctly and there's a FAQ about
>> > submitting, please point me to it and I'll gladly do it some other
>> > way.
>> 
>> I don't know Windows well enough to know exactly what should happen, but
>> here are some thoughts:
>> - you can solve your problem with something like cywin-mount.el.
>>   I know Eli doesn't like cywin-mount.el, but I think we should at least
>>   add such a feature to GNU ELPA.

I use cygwin-mount.el and it doesn't solve the problem.  The reason:
expand-file-name was creating invalid pathnames (c:/home/... rather
than c:/cygwin/home/...).  cygwin-mount.el is for rewriting *valid*
pathnames.

>> - It might still make sense to want to understand cygwin-style file
>>   names from emacsclient without wanting cygwin-mount (which tries to
>>   handle cywin file names *everywhere*).  But then it should probably
>>   work not only for emacsclient but also for file name args to emacs
>>   itself (as a general rule, we should strive to make `emacs' and
>>   `emacsclient' command line handling more similar rather than more
>>   different).

The problem I'm trying to solve is unique to a cygwin program
calling emacsclient.  I'm not sure we want to generalize it beyond
that.


I use Windows a lot, and cygwin and (non-cygwin) emacs make life
bearable.  The server.el hack is the last piece of the puzzle for me
that makes it all work together.


Go ahead and google "windows emacsclient" and see how many people are
struggling with this.  In my Nth push to get emacsclient to work on
windows a few weeks ago, I found cygwin-mount.el, and it works for
most situations.  Just not when I'm in a subdir of ~ where bash
reports my directory as /home/...  and not /c/cygwin/home/...  the
latter would be handled 100% properly *without modification* by
cygwin-mount.


For me, I'm set.  I've got a 100% working system.  I'm trying to help
all those out there in emacs land that are hitting the wall with
emacsclient on windows.

Kevin

Re: problems with emacsclient started by cygwin programs (mswindows)

[Stefan Monnier]

>>> > OK, changes below for my proposed hook.  Comments welcome.
>>> > If I've not submitted this correctly and there's a FAQ about
>>> > submitting, please point me to it and I'll gladly do it some other
>>> > way.
>>> I don't know Windows well enough to know exactly what should happen, but
>>> here are some thoughts:
>>> - you can solve your problem with something like cywin-mount.el.
>>> I know Eli doesn't like cywin-mount.el, but I think we should at least
>>> add such a feature to GNU ELPA.
> I use cygwin-mount.el and it doesn't solve the problem.  The reason:
> expand-file-name was creating invalid pathnames (c:/home/... rather
> than c:/cygwin/home/...).  cygwin-mount.el is for rewriting *valid*
> pathnames.

I'm not sure I understand exactly what you're saying: is it because of
limitations of cygwin-mount or is it because something else makes it
impossible for cygwin-mount to do the right thing?

>>> - It might still make sense to want to understand cygwin-style file
>>> names from emacsclient without wanting cygwin-mount (which tries to
>>> handle cywin file names *everywhere*).  But then it should probably
>>> work not only for emacsclient but also for file name args to emacs
>>> itself (as a general rule, we should strive to make `emacs' and
>>> `emacsclient' command line handling more similar rather than more
>>> different).
> The problem I'm trying to solve is unique to a cygwin program
> calling emacsclient.  I'm not sure we want to generalize it beyond
> that.

I can't imagine why the same problem wouldn't show up if you try to run
`emacs' from cygwin's bash.
And please understand that I don't mean to say that your suggestion is
wrong, just that the same should be done in startup.el to match your
change in server.el.

> most situations.  Just not when I'm in a subdir of ~ where bash
> reports my directory as /home/...  and not /c/cygwin/home/...  the
> latter would be handled 100% properly *without modification* by
> cygwin-mount.

IIRC /home is "mounted" from /c/cygwin/home, and cywin-mount has/had
support for such rewriting, tho it may not be enabled by default, or may
not have been updated to work with recent Emacsen/Cygwin.


        Stefan

Re: problems with emacsclient started by cygwin programs (mswindows)

[layer]

Stefan Monnier wrote:

>> >>> > OK, changes below for my proposed hook.  Comments welcome.
>> >>> > If I've not submitted this correctly and there's a FAQ about
>> >>> > submitting, please point me to it and I'll gladly do it some other
>> >>> > way.
>> >>> I don't know Windows well enough to know exactly what should happen, but
>> >>> here are some thoughts:
>> >>> - you can solve your problem with something like cywin-mount.el.
>> >>> I know Eli doesn't like cywin-mount.el, but I think we should at least
>> >>> add such a feature to GNU ELPA.
>> > I use cygwin-mount.el and it doesn't solve the problem.  The reason:
>> > expand-file-name was creating invalid pathnames (c:/home/... rather
>> > than c:/cygwin/home/...).  cygwin-mount.el is for rewriting *valid*
>> > pathnames.
>> 
>> I'm not sure I understand exactly what you're saying: is it because of
>> limitations of cygwin-mount or is it because something else makes it
>> impossible for cygwin-mount to do the right thing?

The latter, I think.  cygwin-mount is for translating valid cygwin
pathnames into something that the native (non-cygwin) emacs can use.
/c/foo/bar.el becomes c:/foo/bar.el.

The problem I'm trying to solve is this: a cygwin program (e.g. git)
calling emacsclient in a subdir of ~ will pass something like
/home/user/foo.el to emacsclient.  When emacs gets it, it calls
expand-file-name on it like this:

  (expand-file-name "/home/user/foo.el" nil) =>
  "c:/home/user/foo.el"

That's an invalid pathname.  More importantly, it is outside the realm
of cygwin-mount because this is not a cygwin-style path.  It's a
native windows path that is invalid.

>> >>> - It might still make sense to want to understand cygwin-style file
>> >>> names from emacsclient without wanting cygwin-mount (which tries to
>> >>> handle cywin file names *everywhere*).  But then it should probably
>> >>> work not only for emacsclient but also for file name args to emacs
>> >>> itself (as a general rule, we should strive to make `emacs' and
>> >>> `emacsclient' command line handling more similar rather than more
>> >>> different).
>> > The problem I'm trying to solve is unique to a cygwin program
>> > calling emacsclient.  I'm not sure we want to generalize it beyond
>> > that.
>> 
>> I can't imagine why the same problem wouldn't show up if you try to run
>> `emacs' from cygwin's bash.
>> And please understand that I don't mean to say that your suggestion is
>> wrong, just that the same should be done in startup.el to match your
>> change in server.el.

You've completely lost me here, sorry.  (I don't mean this in a
critical way, I just don't grok what you're saying.)

>> > most situations.  Just not when I'm in a subdir of ~ where bash
>> > reports my directory as /home/...  and not /c/cygwin/home/...  the
>> > latter would be handled 100% properly *without modification* by
>> > cygwin-mount.
>> 
>> IIRC /home is "mounted" from /c/cygwin/home, and cywin-mount has/had
>> support for such rewriting, tho it may not be enabled by default, or may
>> not have been updated to work with recent Emacsen/Cygwin.

Not correct:

oob2$ mount
C:/cygwin/bin on /usr/bin type ntfs (binary,noacl)
C:/cygwin/lib on /usr/lib type ntfs (binary,noacl)
C:/cygwin on / type ntfs (binary,noacl)
C: on /c type ntfs (binary,noacl,posix=0,user)
D: on /d type ntfs (binary,noacl,posix=0,user)
E: on /e type vfat (binary,noacl,posix=0,user)
Y: on /cygdrive/y type smbfs (binary,noacl,posix=0,user,noumount,auto)
Z: on /cygdrive/z type ntfs (binary,noacl,posix=0,user,noumount,auto)
oob2$ 

There is no mount for /home.

I'm not wedded to my solution.  If there's a cygwin-mount-based
solution, I'm OK with it.  However, I just don't see it.

cygwin-mount seems to work fine with 23.2, btw.

Kevin

Re: problems with emacsclient started by cygwin programs (mswindows)

[Stefan Monnier]

>>> I'm not sure I understand exactly what you're saying: is it because of
>>> limitations of cygwin-mount or is it because something else makes it
>>> impossible for cygwin-mount to do the right thing?

> The latter, I think.  cygwin-mount is for translating valid cygwin
> pathnames into something that the native (non-cygwin) emacs can use.
> /c/foo/bar.el becomes c:/foo/bar.el.

> The problem I'm trying to solve is this: a cygwin program (e.g. git)
> calling emacsclient in a subdir of ~ will pass something like
> /home/user/foo.el to emacsclient.  When emacs gets it, it calls
> expand-file-name on it like this:

>   (expand-file-name "/home/user/foo.el" nil) =>
>   "c:/home/user/foo.el"

> That's an invalid pathname.  More importantly, it is outside the realm
> of cygwin-mount because this is not a cygwin-style path.  It's a
> native windows path that is invalid.

Then I don't understand: what should be the right return value of
(expand-file-name "/home/user/foo.el") and how could Emacs figure it out?

In any case, this is unrelated to my comment "you can solve your problem
with something like cywin-mount.el", where I didn't mean "use
cygwin-mount.el" but just use something that works in a similar way,
i.e. by changing the behavior of things like expand-file-name, hence the
behavior throughout Emacs rather than only at the boundary between Emacs
and the calling program.

>>> I can't imagine why the same problem wouldn't show up if you try to run
>>> `emacs' from cygwin's bash.
>>> And please understand that I don't mean to say that your suggestion is
>>> wrong, just that the same should be done in startup.el to match your
>>> change in server.el.
> You've completely lost me here, sorry.  (I don't mean this in a
> critical way, I just don't grok what you're saying.)

IIUC your problem shows up when you run "emacsclient ~/foo.el" from
Cygwin bash and I'm saying that it would show up just the same if you
run "emacs ~/foo.el" from that same bash.

>>> > most situations.  Just not when I'm in a subdir of ~ where bash
>>> > reports my directory as /home/...  and not /c/cygwin/home/...  the
>>> > latter would be handled 100% properly *without modification* by
>>> > cygwin-mount.
>>> 
>>> IIRC /home is "mounted" from /c/cygwin/home, and cywin-mount has/had
>>> support for such rewriting, tho it may not be enabled by default, or may
>>> not have been updated to work with recent Emacsen/Cygwin.

> Not correct:

> oob2$ mount
> C:/cygwin/bin on /usr/bin type ntfs (binary,noacl)
> C:/cygwin/lib on /usr/lib type ntfs (binary,noacl)
> C:/cygwin on / type ntfs (binary,noacl)
> C: on /c type ntfs (binary,noacl,posix=0,user)
> D: on /d type ntfs (binary,noacl,posix=0,user)
> E: on /e type vfat (binary,noacl,posix=0,user)
> Y: on /cygdrive/y type smbfs (binary,noacl,posix=0,user,noumount,auto)
> Z: on /cygdrive/z type ntfs (binary,noacl,posix=0,user,noumount,auto)
> oob2$ 

> There is no mount for /home.

But there a mount from c:/cygwin to / so cygwin-mount should be able
to translate /home/user/foo.el to c:/cygwin/home/user/foo.el.  At least
in theory.


        Stefan

Re: problems with emacsclient started by cygwin programs (mswindows)

[layer]

Stefan, I appreciate you hashing this out with me.  More below.

Stefan Monnier wrote:

>> >>> I'm not sure I understand exactly what you're saying: is it because of
>> >>> limitations of cygwin-mount or is it because something else makes it
>> >>> impossible for cygwin-mount to do the right thing?
>> 
>> > The latter, I think.  cygwin-mount is for translating valid cygwin
>> > pathnames into something that the native (non-cygwin) emacs can use.
>> > /c/foo/bar.el becomes c:/foo/bar.el.
>> 
>> > The problem I'm trying to solve is this: a cygwin program (e.g. git)
>> > calling emacsclient in a subdir of ~ will pass something like
>> > /home/user/foo.el to emacsclient.  When emacs gets it, it calls
>> > expand-file-name on it like this:
>> 
>> >   (expand-file-name "/home/user/foo.el" nil) =>
>> >   "c:/home/user/foo.el"
>> 
>> > That's an invalid pathname.  More importantly, it is outside the realm
>> > of cygwin-mount because this is not a cygwin-style path.  It's a
>> > native windows path that is invalid.
>> 
>> Then I don't understand: what should be the right return value of
>> (expand-file-name "/home/user/foo.el") and how could Emacs figure it out?

I think the current return value is the only sane one.  My point was
only that the pathname was invalid outside of cygwin.

>> In any case, this is unrelated to my comment "you can solve your problem
>> with something like cywin-mount.el", where I didn't mean "use
>> cygwin-mount.el" but just use something that works in a similar way,
>> i.e. by changing the behavior of things like expand-file-name, hence the
>> behavior throughout Emacs rather than only at the boundary between Emacs
>> and the calling program.

I mostly agree.  The problem only happens when external pathnames get
introduced into emacs.  The ways this can happen are:

 - command line
 - server (used by emacsclient)

I'm not sure they can get into emacs in other meaningful ways.

Continuing my thoughts at the end...

>> >>> I can't imagine why the same problem wouldn't show up if you try to run
>> >>> `emacs' from cygwin's bash.
>> >>> And please understand that I don't mean to say that your suggestion is
>> >>> wrong, just that the same should be done in startup.el to match your
>> >>> change in server.el.
>> > You've completely lost me here, sorry.  (I don't mean this in a
>> > critical way, I just don't grok what you're saying.)
>> 
>> IIUC your problem shows up when you run "emacsclient ~/foo.el" from
>> Cygwin bash and I'm saying that it would show up just the same if you
>> run "emacs ~/foo.el" from that same bash.

I see now what you mean, and I agree it would be the same problem.

>> 
>> >>> > most situations.  Just not when I'm in a subdir of ~ where bash
>> >>> > reports my directory as /home/...  and not /c/cygwin/home/...  the
>> >>> > latter would be handled 100% properly *without modification* by
>> >>> > cygwin-mount.
>> >>> 
>> >>> IIRC /home is "mounted" from /c/cygwin/home, and cywin-mount has/had
>> >>> support for such rewriting, tho it may not be enabled by default, or may
>> >>> not have been updated to work with recent Emacsen/Cygwin.
>> 
>> > Not correct:
>> 
>> > oob2$ mount
>> > C:/cygwin/bin on /usr/bin type ntfs (binary,noacl)
>> > C:/cygwin/lib on /usr/lib type ntfs (binary,noacl)
>> > C:/cygwin on / type ntfs (binary,noacl)
>> > C: on /c type ntfs (binary,noacl,posix=0,user)
>> > D: on /d type ntfs (binary,noacl,posix=0,user)
>> > E: on /e type vfat (binary,noacl,posix=0,user)
>> > Y: on /cygdrive/y type smbfs (binary,noacl,posix=0,user,noumount,auto)
>> > Z: on /cygdrive/z type ntfs (binary,noacl,posix=0,user,noumount,auto)
>> > oob2$ 
>> 
>> > There is no mount for /home.
>> 
>> But there a mount from c:/cygwin to / so cygwin-mount should be able
>> to translate /home/user/foo.el to c:/cygwin/home/user/foo.el.  At least
>> in theory.

But when it's OK to do this?  What if there was a c:/home/ and
c:/cygwin/home, then what would /home refer to?  Could be c:/home/ and
not c:/cygwin/home/.

To me, this global interpretation of pathnames (if I correctly
interpret what you are suggesting for expand-file-name or
cygwin-mount) to fix them is asking for trouble.  On the other hand, I
know the emacs command line and emacs server inputs will for me always
have cygwin paths on them.

If, however, I was using the cygwin-built emacs, none of what we've
been discussing would be needed.  That's why I think it's best in a
hook.

I do (now) agree that hook should be applied to the command line
processing.  If we come to agreement on all this I can look into doing
that, too.

Kevin

Re: problems with emacsclient started by cygwin programs (mswindows)

[Stefan Monnier]

> I mostly agree.  The problem only happens when external pathnames get
> introduced into Emacs.  The ways this can happen are:
>  - command line
>  - server (used by emacsclient)
> I'm not sure they can get into Emacs in other meaningful ways.

They can show up in several other ways, e.g. in *compile*/*grep* if your
compiler/grep is using cygwin.

Enumerating them all and making them all do the right thing is rather
tedious/difficult, which is why cygwin-mount is fairly popular for those
people who mix cygwin and the native Windows port of Emacs: by handling
cygwin filenames everywhere, it's sure to handle them where needed.

>>> But there a mount from c:/cygwin to / so cygwin-mount should be able
>>> to translate /home/user/foo.el to c:/cygwin/home/user/foo.el.  At least
>>> in theory.
> But when it's OK to do this?  What if there was a c:/home/ and
> c:/cygwin/home, then what would /home refer to?  Could be c:/home/ and
> not c:/cygwin/home/.

Indeed, cygwin-mount isn't told when a file is using cygwin syntax and
when it uses some other syntax, so it has to guess.  That's probably why
it did not handle your /home (too ambiguous).

> To me, this global interpretation of pathnames (if I correctly
> interpret what you are suggesting for expand-file-name or
> cygwin-mount) to fix them is asking for trouble.  On the other hand, I
> know the Emacs command line and Emacs server inputs will for me always
> have cygwin paths on them.

OK.

> I do (now) agree that hook should be applied to the command line
> processing.  If we come to agreement on all this I can look into doing
> that, too.

That sounds great.  I think we should simply add a new function
file-name-import-function (feel free to come up with a better name)
which by default is set to `identity' and whose intention is to
translate an external file name (coming from another process) into one
usable inside Emacs.  And then call this function where appropriate.


        Stefan