Re: M-x mystery

Re: M-x mystery

[Silvio Levy]

Hi,

Sorry to bother you all, but the info in 
http://www.gnu.org/software/emacs/manual/html_node/elisp/Functions-for-Key-Lookup.html
doesn't seem to correspond to my experience.  I wonder if it
corresponds to yours.

First, let me tell you what does work as described. My ALT key does
seem to be interpreted by Emacs as Meta, in principle. The page above says

        M-<F1>, a function key, is not converted into <ESC> <F1>. 

and indeed, if I type ALT-F5 the response is "<M-f5> undefined", as
expected. (I'm using F5 because my F1 is intercepted by the window
manager - can't figure out how.)

And, ALT-x is interpreted as ESC x, also as described above. 

The problem is the variable meta-prefix-char, which is supposed to
alter that relationship.

If I load a file containing only the command (setq meta-prefix-char [f5])
and check the value of the variable, the answer is [f5].  

******** Yet, after that, ALT-x is still interpreted as ESC x. ********  

You might say that [f5] is not a valid prefix key. But [f2] is (it's
bound to 2C-mode-map in two-column.el). After I set meta-prefix-char
to [f2], typing ALT-x is still interpreted as ESC x, not [f2] x.

In fact, I can't even duplicate the example near the bottom of the
"Functions for Key Lookup" page (link above). Namely, setting
meta-prefix-char to 24 **should** make M-x behave like C-x, according
to that page. But that does not happen; even with meta-prefix-char=24,
ALT-x is interpreted as ESC x.

What have I misunderstood?

This is all in emacs 23.1.50.1 called with -q.

Silvio

RE: M-x mystery

[Drew Adams]

> The problem is the variable meta-prefix-char, which is supposed to
> alter that relationship.
> 
> If I load a file containing only the command (setq 
> meta-prefix-char [f5]) and check the value of the variable,
> the answer is [f5].  
> 
> ******** Yet, after that, ALT-x is still interpreted as ESC 
> x. ********  
> 
> You might say that [f5] is not a valid prefix key. But [f2] is (it's
> bound to 2C-mode-map in two-column.el). After I set meta-prefix-char
> to [f2], typing ALT-x is still interpreted as ESC x, not [f2] x.

I'm really no expert on these things.  No doubt someone will clarify better.
But [f2] is not a _character_.  (It is a vector.)  A character is a non-negative
integer (in a certain range) - see the Elisp manual, node `Character Codes'.

`meta-prefix-char' is a user option. Try using Customize to change its value.
Customize does type-checking, and it will not even allow you to set the variable
to [f2].  It raises this error: 

"This field should contain a single character"

(It should say "must", not "should".)

> In fact, I can't even duplicate the example near the bottom of the
> "Functions for Key Lookup" page (link above). Namely, setting
> meta-prefix-char to 24 **should** make M-x behave like C-x, according
> to that page. But that does not happen; 

I think it does.  Try `C-h k M-s' or `C-h k C-M-s'.  They correspond to `C-x s'
and `C-x C-x', respectively.

> even with meta-prefix-char=24,
> ALT-x is interpreted as ESC x.

I think you're doing things backwards.  Setting `meta-prefix-char' to the `C-x'
character (24) makes what used to be treated as `C-x SOMETHING' be treated as
`M-SOMETHING'.  If `C-x x' were defined as a command, then `M-x' would then
invoke that command (and not `execute-extended-command').  Since `C-x x' is
undefined by default, after setting `meta-prefix-char' to 24, `C-h k M-x' says
it is undefined.

In any case, doing that in no way affects the treatment of Meta as ESC, AFAIK.

> What have I misunderstood?

1. The value of `meta-prefix-char' needs to be a character.

2. I think the treatment of Meta as ESC happens at a different level.  Node
`Format of Keymaps' says this:

"Keymaps do not directly record bindings for the meta characters.
Instead, meta characters are regarded for purposes of key lookup as
sequences of two characters, the first of which is <ESC> (or whatever
is currently the value of `meta-prefix-char').  Thus, the key `M-a' is
internally represented as `<ESC> a', and its global binding is found at
the slot for `a' in `esc-map' (*note Prefix Keys::).

This conversion applies only to characters, not to function keys or
other input events; thus, `M-<end>' has nothing to do with `<ESC>
<end>'."

It seems that a Meta character is treated as a sequence of two chars, the first
of which is the value of `meta-prefix-char'.  AFAIK, that's just the way it is.
Someone will correct me if I misunderstand.

shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

Hi everyone,

I'm having trouble when generating a TAGS file on windows. I should say that
everything works fine on Linux (almost, see bottom of the message), so it
really seems to be a windows specific problem.

Note that I'm using GNU Emacs 23.3.1, with Exuberant CTags 5.8

So here is the thing. Say I have a "test" folder, with 2 source files, e.g.
test1.c and test2.c
I also have a "filelist.txt", e.g. generated with "GnuWin32 find", which
contains :

./test1.v
./test2.v

Now, I generate my TAGS file from a plain windows shell, with the following
command:

more filelist.txt | ctags -e -L -

This works fine, and, as expected, the paths to the source files in the TAGS
file are relative, so the entire "test" folder and the TAGS file can be
moved to a different location.

I'm developing a small emacs package to provide some sort of IDE. So now I'm
trying to do the same from within emacs. If I do:

M-x shell-command
more filelist.txt | ctags -e -L -

...then the result is different, and now paths in the TAGS file become
absolute. Even using the explicit option --tag-relative=yes does not change
anything.

I am not sure whether Exuberant CTags or Emacs shell-command is to blame. I
believe it's got something to do with "shell-command", but it's just a guess
really. Can anyone reproduce the problem and/or explain what's happening ?

PS, on a slightly different note:

1/ on linux, shell-command also does something funny. Try "more
filelist.txt" and check the output buffer. It contains some sort of "header"
with "::::::::::::::" so the same command obviously fails. Use "less"
instead of "more" and then everything works

2/ I have also found that "GnuWin32 find" is painfully slow on network
(unix) drives, and that using the built-in dos command "dir /b /s" is much
faster (but less powerful of course). Has anyone experienced the same ? Also
"find" miserably fails if any directory it traverses contains (unix)
symbolic links (which can't be resolved from windows, of course, but this
shouldn't be a problem).

Re: shell-command causes problems with absolute/relative paths in TAGS

[Vladimir Murzin]

Hi David 

AFAIK "less"/"more" are pagers and suitable for reading files by human. Have you tried using "cat"?
Best wishes, Vladimir Murzin

-----Original Message-----
From: "David Chappaz" <david.chappaz@free.fr>
Sender: help-gnu-emacs-bounces+murzin.v=gmail.com@gnu.orgDate: Wed, 4 Jan 2012 16:05:13 
To: <help-gnu-emacs@gnu.org>
Subject: shell-command causes problems with absolute/relative paths in TAGS

Hi everyone,

I'm having trouble when generating a TAGS file on windows. I should say that
everything works fine on Linux (almost, see bottom of the message), so it
really seems to be a windows specific problem.

Note that I'm using GNU Emacs 23.3.1, with Exuberant CTags 5.8

So here is the thing. Say I have a "test" folder, with 2 source files, e.g.
test1.c and test2.c
I also have a "filelist.txt", e.g. generated with "GnuWin32 find", which
contains :

./test1.v
./test2.v

Now, I generate my TAGS file from a plain windows shell, with the following
command:

more filelist.txt | ctags -e -L -

This works fine, and, as expected, the paths to the source files in the TAGS
file are relative, so the entire "test" folder and the TAGS file can be
moved to a different location.

I'm developing a small emacs package to provide some sort of IDE. So now I'm
trying to do the same from within emacs. If I do:

M-x shell-command
more filelist.txt | ctags -e -L -

...then the result is different, and now paths in the TAGS file become
absolute. Even using the explicit option --tag-relative=yes does not change
anything.

I am not sure whether Exuberant CTags or Emacs shell-command is to blame. I
believe it's got something to do with "shell-command", but it's just a guess
really. Can anyone reproduce the problem and/or explain what's happening ?

PS, on a slightly different note:

1/ on linux, shell-command also does something funny. Try "more
filelist.txt" and check the output buffer. It contains some sort of "header"
with "::::::::::::::" so the same command obviously fails. Use "less"
instead of "more" and then everything works

2/ I have also found that "GnuWin32 find" is painfully slow on network
(unix) drives, and that using the built-in dos command "dir /b /s" is much
faster (but less powerful of course). Has anyone experienced the same ? Also
"find" miserably fails if any directory it traverses contains (unix)
symbolic links (which can't be resolved from windows, of course, but this
shouldn't be a problem).

RE: shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

Vladimir, thanks for the very prompt reply !
Unfortunately, "cat" does not change the behaviour...

I understand that "more" was perhaps not the best choice, but, I can't help
thinking that shell-command somehow does something more than just sending a
command to the shell....


-----Original Message-----
From: Vladimir Murzin [mailto:murzin.v@gmail.com] 
Sent: 04 January 2012 16:15
To: David Chappaz; help-gnu-emacs-bounces+murzin.v=gmail.com@gnu.org;
help-gnu-emacs@gnu.org
Subject: Re: shell-command causes problems with absolute/relative paths in
TAGS

Hi David 

AFAIK "less"/"more" are pagers and suitable for reading files by human. Have
you tried using "cat"?
Best wishes, Vladimir Murzin

-----Original Message-----
From: "David Chappaz" <david.chappaz@free.fr>
Sender: help-gnu-emacs-bounces+murzin.v=gmail.com@gnu.orgDate: Wed, 4 Jan
2012 16:05:13 
To: <help-gnu-emacs@gnu.org>
Subject: shell-command causes problems with absolute/relative paths in TAGS

Hi everyone,

I'm having trouble when generating a TAGS file on windows. I should say that
everything works fine on Linux (almost, see bottom of the message), so it
really seems to be a windows specific problem.

Note that I'm using GNU Emacs 23.3.1, with Exuberant CTags 5.8

So here is the thing. Say I have a "test" folder, with 2 source files, e.g.
test1.c and test2.c
I also have a "filelist.txt", e.g. generated with "GnuWin32 find", which
contains :

./test1.v
./test2.v

Now, I generate my TAGS file from a plain windows shell, with the following
command:

more filelist.txt | ctags -e -L -

This works fine, and, as expected, the paths to the source files in the TAGS
file are relative, so the entire "test" folder and the TAGS file can be
moved to a different location.

I'm developing a small emacs package to provide some sort of IDE. So now I'm
trying to do the same from within emacs. If I do:

M-x shell-command
more filelist.txt | ctags -e -L -

...then the result is different, and now paths in the TAGS file become
absolute. Even using the explicit option --tag-relative=yes does not change
anything.

I am not sure whether Exuberant CTags or Emacs shell-command is to blame. I
believe it's got something to do with "shell-command", but it's just a guess
really. Can anyone reproduce the problem and/or explain what's happening ?

PS, on a slightly different note:

1/ on linux, shell-command also does something funny. Try "more
filelist.txt" and check the output buffer. It contains some sort of "header"
with "::::::::::::::" so the same command obviously fails. Use "less"
instead of "more" and then everything works

2/ I have also found that "GnuWin32 find" is painfully slow on network
(unix) drives, and that using the built-in dos command "dir /b /s" is much
faster (but less powerful of course). Has anyone experienced the same ? Also
"find" miserably fails if any directory it traverses contains (unix)
symbolic links (which can't be resolved from windows, of course, but this
shouldn't be a problem).

Re: shell-command causes problems with absolute/relative paths in TAGS

[Bob Proulx]

Please don't hijack discussion threads.  You replied to Drew Adams
message with the subject "M-x mystery" and changed the subject.  If
you wish to create a new topic please create a new message.  Changing
the subject is used for topic drift within a discussion thread and
does not create a new thread of discussion.  This is to your advantage
since people who are skipping the thread will also skip your message.

See your message in context here:

  http://lists.gnu.org/archive/html/help-gnu-emacs/2012-01/threads.html#00035

David Chappaz wrote:
> Note that I'm using GNU Emacs 23.3.1, with Exuberant CTags 5.8

I am not using that version of tags and so I can't check the options
but to me using "-" (indicating to read standard input) looks to me
like a strange way to do things.  And piping from more to it is
strange times strange creating a lot of it.

> more filelist.txt | ctags -e -L -

The -L means to read from a file list.  So this is running more, a
screen pager for human consumption, and then piping that to ctags and
telling ctags to read the standard input.  That is a lot of work to do
to accomplish that task.

Also, search the web for "useless use of cat".  This is the same thing
but just using a different program in the same way.  In other words,
don't try to substitute cat for more in the above.

Instead have you tried simply running ctags on the file?  Try this instead:

  ctags -e -L filelist.txt

That should work more reliably the same on all operating systems.
Expecially since pipes on MS-Windows are not speedy.

> PS, on a slightly different note:
> 
> 1/ on linux, shell-command also does something funny. Try "more
> filelist.txt" and check the output buffer. It contains some sort of "header"
> with "::::::::::::::" so the same command obviously fails. Use "less"
> instead of "more" and then everything works

If there are multiple files then more can page through all of them and
therefore more shows a visual indicator between files.  Also 'head'
and 'tail' do similar things.  If you are seeing this then more is
trying to page through multiple files to the human.  It isn't intended
to be piped like that.

Seeing the ":::::::::::::: FILENAME ::::::::::::::" header is a clue
to your problem.  It tells me that more has been told to read multiple
files.  But the command you list does not have multiple files.  I am
sure that when you run it from the command line and saying that it is
working okay there that it isn't getting multiple files.  So somehow
when calling that pipeline from shell-command there are multiple files
being offered to the command.  That is probably somehow related to
your difference in behavior of relative paths versus absolute paths.
In the simple command line case I am sure it is what you see is what
you get.  But in the shell-command case I think a completely different
set of files is being stuffed into the ctags standard input and
because those are different files they only appear to be absolute
versions

I don't have any comments on the other questions about MS since I do
not have that available for testing.

Bob

RE: shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

> Bob Proulx wrote:
> Please don't hijack discussion threads.  You replied to Drew Adams
> message with the subject "M-x mystery" and changed the subject.  If
> you wish to create a new topic please create a new message.  Changing
> the subject is used for topic drift within a discussion thread and
> does not create a new thread of discussion.  This is to your advantage
> since people who are skipping the thread will also skip your message.

I can't argue with that. You're 100% correct, and I'm just hoping Drew Adams
will forgive me. I replied to an existing message and changed the subject
out of laziness I suppose, and didn't realise it would have unwanted
consequences. I'll learn from this mistake !
Out of curiosity, is there any way to move this discussion in a new thread,
or is it too late ?

> I am not using that version of tags and so I can't check the options
> but to me using "-" (indicating to read standard input) looks to me
> like a strange way to do things.  And piping from more to it is
> strange times strange creating a lot of it.
> 
> > more filelist.txt | ctags -e -L -

As you have guessed, I'm not a shell expert, but I now understand the
example I gave was a bit strange indeed.
In fact I ended up with this command, because I was trying to "debug" a more
complex command, where the result of "find" is piped to ctags:

find -type f -name "*.c" | ctags -e -L -

Presumably this way of doing is more "conventional" but again I'm not an
expert, and if you think this is a bit awkward for any reason, please let me
know !

Because I was getting the absolute/relative path problem I mentioned, I
simply split the command into two fragments, explicitly creating an
intermediate file:

find -type f -name "*.c" > filelist.txt
cat filelist.txt | ctags -e -L -
 
> Instead have you tried simply running ctags on the file?  Try this
> instead:
> 
>   ctags -e -L filelist.txt

Doh, I should have thought of doing that from the start. This is by far the
simplest and most natural way of doing things...
 
> That should work more reliably the same on all operating systems.
> Expecially since pipes on MS-Windows are not speedy.

And yet I still get the same behaviour...
Running "ctags -e -L filelist.txt" directly from the shell and doing the
same in shell-command still produces two different results... How can that
be ?? This is really beyond me !

> > PS, on a slightly different note:
> >
> > 1/ on linux, shell-command also does something funny. 
> 
> Seeing the ":::::::::::::: FILENAME ::::::::::::::" header is a clue
> to your problem.  It tells me that more has been told to read multiple
> files.  But the command you list does not have multiple files.  I am
> sure that when you run it from the command line and saying that it is
> working okay there that it isn't getting multiple files.  So somehow
> when calling that pipeline from shell-command there are multiple files
> being offered to the command.  

Yes that makes perfect sense. I tried something like "more test1.c test2.c"
directly from the command line, and I did get the separator with the
filename. So when doing "more filetest.txt" from shell-command, "more" is
clearly getting multiple files.

> That is probably somehow related to
> your difference in behavior of relative paths versus absolute paths.

Yes I suppose so. 
In fact I've just discovered that "ctags -e -L filelist.txt" works fine when
directly used in a proper windows shell, but does not when used in an
emacs's shell buffer (i.e. that produces the same result, with absolute
paths, as with shell-command)
It sounds to me that the problem is related to an extra termination
character added to the command line by emacs before it's passed to the
shell... But I'm not skilled enough to figure out the details... Any ideas ?

RE: shell-command causes problems with absolute/relative paths in TAGS

[Unknown]

Hi David,
On Thu, 5 Jan 2012 12:20:30 -0000, "David Chappaz" <david.chappaz@free.fr> wrote:
> > Bob Proulx wrote:
> > Please don't hijack discussion threads.
[...]


> Out of curiosity, is there any way to move this discussion in a new thread,
> or is it too late ?

It's too late: Threading of emails is done with respect to their
header (References:, In-Reply-To:) information.  This happens on
ones computer.  In order to move part of a thread one would have
to change header information of all local copies of relevant
emails on all computers of all recipents...

For individual purposes the mail user agent "mutt" for instance
lets users decide to break threads.  After breaking thei are
shown as two distinct threads -- but only on this users local
copy of emails. 

Ciao, Gregor

Re: shell-command causes problems with absolute/relative paths in TAGS

[Bob Proulx]

David Chappaz wrote:
> Bob Proulx wrote:
> > > more filelist.txt | ctags -e -L -
> 
> As you have guessed, I'm not a shell expert, but I now understand the
> example I gave was a bit strange indeed.

Oh good.  We all have to start somewhere.

> In fact I ended up with this command, because I was trying to "debug" a more
> complex command, where the result of "find" is piped to ctags:
> 
> find -type f -name "*.c" | ctags -e -L -
> 
> Presumably this way of doing is more "conventional" but again I'm not an
> expert, and if you think this is a bit awkward for any reason, please let me
> know !

Yes.  That way is much more normal.  Because the find is doing
something useful and contributing positively to the overall task.
That way is a very typical way of doing things.  Good!

Unfortunately it suffers problems when people embed newlines inside of
filenames.  As long as you don't do that, and it is unusual to do so,
then I would keep on doing it exactly that way.  Not using newlines in
a C source code file name is good conventional practice.  If I found
anyone putting newlines in a C source code filename I would convince
them otherwise.  :-)  So you are fine.  But it is useful to understand
where the convention breaks down.

> Because I was getting the absolute/relative path problem I mentioned, I
> simply split the command into two fragments, explicitly creating an
> intermediate file:
> 
> find -type f -name "*.c" > filelist.txt
> cat filelist.txt | ctags -e -L -

I can now see how you got there.  Like most things knowing the
background story fills in all of the details and makes everything make
sense.  (But in case anyone jumped into the thread here we are all
agreed that the improvement below is better.)

> > Instead have you tried simply running ctags on the file?  Try this
> > instead:
> > 
> >   ctags -e -L filelist.txt
> 
> Doh, I should have thought of doing that from the start. This is by far the
> simplest and most natural way of doing things...

Yay!  Much simpler and more direct.  Simple is good.

> > That should work more reliably the same on all operating systems.
> > Expecially since pipes on MS-Windows are not speedy.
> 
> And yet I still get the same behaviour...
> Running "ctags -e -L filelist.txt" directly from the shell and doing the
> same in shell-command still produces two different results... How can that
> be ?? This is really beyond me !

Drat!  I was hoping that the direct filename method would clean things
up and solve your problem.  I don't really have any other hints.
Because I can't recreate your problem.  It works fine for me.

> > Seeing the ":::::::::::::: FILENAME ::::::::::::::" header is a clue
> > to your problem.  It tells me that more has been told to read multiple
> > files.  But the command you list does not have multiple files.  I am
> > sure that when you run it from the command line and saying that it is
> > working okay there that it isn't getting multiple files.  So somehow
> > when calling that pipeline from shell-command there are multiple files
> > being offered to the command.  
> 
> Yes that makes perfect sense. I tried something like "more test1.c test2.c"
> directly from the command line, and I did get the separator with the
> filename. So when doing "more filetest.txt" from shell-command, "more" is
> clearly getting multiple files.

Can you make out the filenames that it is getting from the output?
Can you do some testing and try to figure out what filenames are being
added to the command?  I think if you could deduce what the extra file
names are then that would be a huge big clue to determining where they
are coming from and then fixing the problem.  Perhaps using 'echo' to
echo the args out would be useful.  But the problem is that you are
using an operating system that I don't know the right debug techniques
and so do not know what to suggest.  I would suggest echo but I fear
that the echo I would see and the echo you would see would be
completely different and that will just confound things.  For me a
program such as echo that just echo printed all of its arguments would
allow me to see the extra arguments supplied.  But I don't know what
it will do on your system.  It may or may not be useful.

  M-! echo

> > That is probably somehow related to
> > your difference in behavior of relative paths versus absolute paths.
> 
> Yes I suppose so. 
> In fact I've just discovered that "ctags -e -L filelist.txt" works fine when
> directly used in a proper windows shell,

That makes sense so far.

>                                           but does not when used in an
> emacs's shell buffer (i.e. that produces the same result, with absolute
> paths, as with shell-command)

It sounds like a bug in your port of emacs in the area of
shell-command.  Is there a different port that you could try?  Maybe
it would behave differently.

> It sounds to me that the problem is related to an extra termination
> character added to the command line by emacs before it's passed to the
> shell... But I'm not skilled enough to figure out the details... Any ideas ?

I am one of those people what has been using Unix and GNU systems all
of my life and have virtually bypassed ever using an MS machine.  When
I try to use one I am lost and frustrated.  Which is why I have
sympathy for people going the other direction and try to help out when
I can.  But it means that I really have no advice available for you.

Since it looks like an operating system interface bug in shell-command
I think I would try to work around it by using a helper script.  I
would create a helper script which did the actions I wanted (find
piped to ctags in this case) and then call the script instead of the
raw commands.  That could help debug things too since then once you
are inside the helper script you can add more debugging there.

Bob

RE: shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

> > So when doing "more filetest.txt" from shell-command, "more" is 
> > clearly getting multiple files.
> 
> Can you make out the filenames that it is getting from the output?

I tried, and tried... but didn't succeed. From a plain shell, "more" does
not insert any header, unless there are multiple files Now, when I do "more
test1.c" from shell-command, a header is inserted (as if there were multiple
files), but no second or third header appears (as if there was only one...),
would it be with a blank file name.

But I'm only seeing this behaviour on linux, and although it might be
related to my problem (on windows), it doesn't really affect me. Perhaps
this is something you can reproduce and investigate.

Now, on Windows, 

> > I've just discovered that "ctags -e -L filelist.txt" does not [work] 
> > when used in an emacs's shell buffer (i.e. that produces the same 
> > result, with absolute paths, as with shell-command)

After more detailed investigations, I've found out the following.
If, before doing M-x shell, I evaluate

(setq explicit-cmdproxy.exe-args  '("/q"))

to prevent shell commands from being echoed, then suddenly the TAGS file is
generated properly, with relative filenames. 

In other words, the same command has two different behaviours when issued in
an emacs shell buffer, depending on if explicit-cmdproxy.exe-args is set to
"" or "/q". That does not make any sense, or does it ? 

Something looks really odd. Also, shell-command is not affected by the value
of explicit-cmdproxy.exe-args, and always behaves the same.

> Which is why I have
> sympathy for people going the other direction and try to help out when 
> I can.

I appreciate your support, and even if you're not a MS-Windows expert, your
ideas are very welcome !
To be honest, most of the time I use emacs on linux, so the lisp package I'm
developing works fine. But occasionally I use it on Windows, which is why
I'm trying to get it to work on this platform too... 
 
> Since it looks like an operating system interface bug in shell-command 
> I think I would try to work around it by using a helper script.

Yeah, in the short term, that's a very good plan indeed to work around the
problem!

Re: shell-command causes problems with absolute/relative paths in TAGS

[Eli Zaretskii]

> From: "David Chappaz" <david.chappaz@free.fr>
> Date: Fri, 6 Jan 2012 11:12:12 -0000
> 
> If, before doing M-x shell, I evaluate
> 
> (setq explicit-cmdproxy.exe-args  '("/q"))
> 
> to prevent shell commands from being echoed, then suddenly the TAGS file is
> generated properly, with relative filenames. 

I see no such variable in Emacs.  Does it come from some third-party
package you have loaded?  If so, perhaps that package is causing this
problem.

Also, originally you talked about "M-x shell-command", not "M-x shell".  
Which one is it?

RE: shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

Eli Zaretskii wrote
> > If, before doing M-x shell, I evaluate
> > (setq explicit-cmdproxy.exe-args  '("/q"))
> > to prevent shell commands from being echoed, then suddenly the TAGS file
> > is generated properly, with relative filenames.
> 
> I see no such variable in Emacs. 

The variable name depends on which shell you use. By default, only
explicit-bash-args and explicit-sh-args are defined in emacs.
On windows, cmdproxy.exe is the default shell, hence the variable name.

> Also, originally you talked about "M-x shell-command", not "M-x shell".
> Which one is it?

Sorry about the confusion.I'll try to summarise.

With M-x shell-command, the behaviour is always incorrect.
With M-x shell, the behaviour is sometimes correct, sometimes not (more on
that below), depending on cosmetic changes (like explicit-cmdproxy.exe-args)
which should not have any influence (I think).
 
> FWIW, I cannot reproduce this on MS-Windows with Emacs 23.3.  I don't
> have Exuberant CTags, but using the etags program provided with
> Emacs.  I get relative file names with either method.

I need to parse languages which etags doesn't know, which is why I'm using
exuberant ctags.

But I did notice that the behaviour is fine with the vanilla etags provided
with emacs. Therefore, you will have to use exuberant ctags to reproduce the
problem. You can download binaries from
http://prdownloads.sourceforge.net/ctags/ctags58.zip. No install is
required, just copy the .exe in the test directory.

Also, we can't exclude the fact that the problem is in exuberant ctags
itself, but for now I just don't understand at all what's happening.

> Can you reproduce the problem in "emacs -Q"?  If not, there's some
> customization of yours that causes this, or maybe it is a problem with
> Exuberant on Windows.

Yes, same problem with --no-init-file

In fact, after a little debugging, I've devised the following experiment for
you to reproduce (with --no-init-file)

1/ From a freshly opened emacs if do M-x shell followed by:

ctags -e -L filelist.txt

or even

"C:/Program Files/Emacs/emacs-23.3/bin/cmdproxy.exe" /C "ctags -e -L
filelist.txt"

then everything is fine.

2/ Now from e.g. a scratch buffer, I evaluate

(progn
   (cd "C:/test/")
   (call-process-region (point) (point) "C:/Program
Files/Emacs/emacs-23.3/bin/cmdproxy.exe" nil (current-buffer) nil "-c"
"ctags -e -L filelist.txt"))

which is more or less what M-x shell-command would do... then the result is
incorrect.

3/ Worse, if you kill the original shell buffer created in 1/, and repeat
the same operation as in 1/... then the result is incorrect.

So it really looks like something is happening in call-process-region...

Re: shell-command causes problems with absolute/relative paths in TAGS

[Eli Zaretskii]

> From: "David Chappaz" <david.chappaz@free.fr>
> Date: Fri, 6 Jan 2012 13:04:57 -0000
> 
> Eli Zaretskii wrote
> > > If, before doing M-x shell, I evaluate
> > > (setq explicit-cmdproxy.exe-args  '("/q"))
> > > to prevent shell commands from being echoed, then suddenly the TAGS file
> > > is generated properly, with relative filenames.
> > 
> > I see no such variable in Emacs. 
> 
> The variable name depends on which shell you use. By default, only
> explicit-bash-args and explicit-sh-args are defined in emacs.
> On windows, cmdproxy.exe is the default shell, hence the variable name.

Well, I tried that on Windows with cmdproxy as the shell, and I still
don't see this variable.

> 2/ Now from e.g. a scratch buffer, I evaluate
> 
> (progn
>    (cd "C:/test/")
>    (call-process-region (point) (point) "C:/Program
> Files/Emacs/emacs-23.3/bin/cmdproxy.exe" nil (current-buffer) nil "-c"
> "ctags -e -L filelist.txt"))
> 
> which is more or less what M-x shell-command would do... then the result is
> incorrect.

Yes, I see the problem.

> 3/ Worse, if you kill the original shell buffer created in 1/, and repeat
> the same operation as in 1/... then the result is incorrect.
> 
> So it really looks like something is happening in call-process-region...

It must be a bug in ctags, no matter what call-process-region does.  I
suspect that it doesn't correctly handle the backslash in file names,
and fails to realize that C:\foo\bar\baz.c and C:/foo/bar share the
same directory.  That's because the "absolute" file names it produces
are of the form "C:\foo\bar\./file", note the mixture of forward- and
back-slashes.

RE: shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

> > The variable name depends on which shell you use. By default, only
> > explicit-bash-args and explicit-sh-args are defined in emacs.
> > On windows, cmdproxy.exe is the default shell, hence the variable name.
> 
> Well, I tried that on Windows with cmdproxy as the shell, and I still
> don't see this variable.

Ah well I haven't been accurate enough.
It's for the user to create this variable if they want to. It will only be
used if its name perfectly matches that of the shell.
See details in section 7.10 of
http://www.gnu.org/software/emacs/windows/Sub_002dprocesses.html

 
> > 2/ Now from e.g. a scratch buffer, I evaluate
> >
> > (progn
> >    (cd "C:/test/")
> >    (call-process-region (point) (point) "C:/Program
> > Files/Emacs/emacs-23.3/bin/cmdproxy.exe" nil (current-buffer) nil "-c"
> > "ctags -e -L filelist.txt"))
> >
> > which is more or less what M-x shell-command would do... then the result
> is
> > incorrect.
> 
> Yes, I see the problem.

That's a start, I'm not the only one any more :-)

 
> > 3/ Worse, if you kill the original shell buffer created in 1/, and
> repeat
> > the same operation as in 1/... then the result is incorrect.
> >
> > So it really looks like something is happening in call-process-region...
> 
> It must be a bug in ctags, no matter what call-process-region does.  I
> suspect that it doesn't correctly handle the backslash in file names,
> and fails to realize that C:\foo\bar\baz.c and C:/foo/bar share the
> same directory.  That's because the "absolute" file names it produces
> are of the form "C:\foo\bar\./file", note the mixture of forward- and
> back-slashes.

Yes, what you're saying makes perfect sense. The mixture of slashes is
indeed an indication, but what I can't explain, is why ctags would produce a
different result before and after call-process-region is called (steps 1/
and 3/ respectively)

Surely emacs must be passing slightly different arguments to ctags (perhaps
slashes in a different orientation), before and after the first call to
call-process-region ?

RE: shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

> Surely emacs must be passing slightly different arguments to ctags
> (perhaps
> slashes in a different orientation), before and after the first call to
> call-process-region ?
 
Anyway, when you use something as simple as "ctags -e -L filelist.txt" in a
shell buffer, there aren't any slashes that can change direction....

So how can this command line work before the first call to
call-process-region, and not after that ?

Re: shell-command causes problems with absolute/relative paths in TAGS

[Eli Zaretskii]

> From: "David Chappaz" <david.chappaz@free.fr>
> Date: Fri, 6 Jan 2012 16:03:47 -0000
> 
> 
> > > The variable name depends on which shell you use. By default, only
> > > explicit-bash-args and explicit-sh-args are defined in emacs.
> > > On windows, cmdproxy.exe is the default shell, hence the variable name.
> > 
> > Well, I tried that on Windows with cmdproxy as the shell, and I still
> > don't see this variable.
> 
> Ah well I haven't been accurate enough.
> It's for the user to create this variable if they want to. It will only be
> used if its name perfectly matches that of the shell.
> See details in section 7.10 of
> http://www.gnu.org/software/emacs/windows/Sub_002dprocesses.html

Never mind, that variable is not needed for reproducing the problem.

> Surely emacs must be passing slightly different arguments to ctags (perhaps
> slashes in a different orientation), before and after the first call to
> call-process-region ?

Probably some environment variable rather than command-line argument.

Re: shell-command causes problems with absolute/relative paths in TAGS

[Eli Zaretskii]

> From: "David Chappaz" <david.chappaz@free.fr>
> Date: Wed, 4 Jan 2012 16:05:13 -0000
> 
> 2/ I have also found that "GnuWin32 find" is painfully slow on network
> (unix) drives

Try the port of Findutils from here:

  http://sourceforge.net/projects/ezwinports/files/

I did that port because the one on GnuWin32 is abysmally slow, on any
drive, not just network ones.

RE: shell-command causes problems with absolute/relative paths in TAGS

[David Chappaz]

> Try the port of Findutils from here:
> 
>   http://sourceforge.net/projects/ezwinports/files/
> 
> I did that port because the one on GnuWin32 is abysmally slow, on any
> drive, not just network ones.

Wow that really makes a massive difference !!

I am not asking for the details, as I'm unlikely to understand, but... how
come the GnuWin32 port hasn't been updated or fixed ?

Not only is your port much faster, it also doesn't completely fall apart
when encountering symbolic links on a unix network drive (which of course
can't be resolved)

You've made my day !
Cheers !
David.

Re: shell-command causes problems with absolute/relative paths in TAGS

[Eli Zaretskii]

> From: "David Chappaz" <david.chappaz@free.fr>
> Date: Fri, 6 Jan 2012 14:04:57 -0000
> 
> >   http://sourceforge.net/projects/ezwinports/files/
> > 
> > I did that port because the one on GnuWin32 is abysmally slow, on any
> > drive, not just network ones.
> 
> Wow that really makes a massive difference !!
> 
> I am not asking for the details, as I'm unlikely to understand, but... how
> come the GnuWin32 port hasn't been updated or fixed ?

I don't know; I cannot speak for the GnuWin32 project.  The fact is,
many ports there are outdated and some are outright broken.

I did offer them my once port of GNU ID Utils (another package whose
GnuWin32 is hopelessly broken), and even got an agreement and uploaded
the zip archives where they told me to.  But months passed and nothing
happened, so I decided to start my own project on SourceForge, and you
can find ID Utils there as well (and much more).

> Not only is your port much faster, it also doesn't completely fall apart
> when encountering symbolic links on a unix network drive (which of course
> can't be resolved)

Yes, and xargs doesn't fail, and locate actually works, etc. etc.

Whoever did the GnuWin32 port probably didn't test the result well
enough.

> You've made my day !

I'm glad I could help.

Re: shell-command causes problems with absolute/relative paths in TAGS

[Juanma Barranquero]

On Fri, Jan 6, 2012 at 16:12, Eli Zaretskii <eliz@gnu.org> wrote:

> I don't know; I cannot speak for the GnuWin32 project.  The fact is,
> many ports there are outdated and some are outright broken.

The News / Announcements heading in the main page lists three updated
packages for 2010, none for 2011. I'd say the project doesn't exactly
seem vibrant with energy.

    Juanma

Re: shell-command causes problems with absolute/relative paths in TAGS

[Eli Zaretskii]

> From: "David Chappaz" <david.chappaz@free.fr>
> Date: Wed, 4 Jan 2012 16:05:13 -0000
> 
> Now, I generate my TAGS file from a plain windows shell, with the following
> command:
> 
> more filelist.txt | ctags -e -L -
> 
> This works fine, and, as expected, the paths to the source files in the TAGS
> file are relative, so the entire "test" folder and the TAGS file can be
> moved to a different location.
> 
> I'm developing a small emacs package to provide some sort of IDE. So now I'm
> trying to do the same from within emacs. If I do:
> 
> M-x shell-command
> more filelist.txt | ctags -e -L -
> 
> ...then the result is different, and now paths in the TAGS file become
> absolute. Even using the explicit option --tag-relative=yes does not change
> anything.

FWIW, I cannot reproduce this on MS-Windows with Emacs 23.3.  I don't
have Exuberant CTags, but using the etags program provided with
Emacs.  I get relative file names with either method.

Can you reproduce the problem in "emacs -Q"?  If not, there's some
customization of yours that causes this, or maybe it is a problem with
Exuberant on Windows.