* What does Emacs on w32 know that grep can't figure out?
@ 2010-09-30 23:13 Lennart Borgman
2010-09-30 23:58 ` Juanma Barranquero
2010-10-01 4:00 ` Stephen J. Turnbull
0 siblings, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-09-30 23:13 UTC (permalink / raw
To: Emacs-Devel devel
I saved to a file with CCleaner the list of installed programs on my
pc. This file is maybe a text file, maybe not. Emacs can read it and
think it is a text file (I believe) with
buffer-file-coding-system is a variable defined in `C source code'.
Its value is
utf-16le-with-signature-unix
Local in buffer install.txt;
However trying to search this file from a cmd prompt with (gnuwin32)
grep does not work. And it does not work with cygwin grep either. They
think it is a binary file (even though I changed the line delimiter to
unix style).
What is going on? Is grep sometimes useless on w32 now, or? (How do we
handle that in Emacs?)
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-09-30 23:13 What does Emacs on w32 know that grep can't figure out? Lennart Borgman
@ 2010-09-30 23:58 ` Juanma Barranquero
2010-10-01 0:29 ` Lennart Borgman
2010-10-01 7:09 ` Eli Zaretskii
2010-10-01 4:00 ` Stephen J. Turnbull
1 sibling, 2 replies; 62+ messages in thread
From: Juanma Barranquero @ 2010-09-30 23:58 UTC (permalink / raw
To: Lennart Borgman; +Cc: Emacs-Devel devel
> This file is maybe a text file, maybe not.
It's a text file, but not ASCII. It's encoded as Emacs is telling you:
utf-16, with signature, in little-endian order (it starts with FF FE).
> Its value is
> utf-16le-with-signature-unix
Curious. I get utf-16le-with-signature-dos, consistent with the file
having CRLF (which it does).
> However trying to search this file from a cmd prompt with (gnuwin32)
> grep does not work. And it does not work with cygwin grep either. They
> think it is a binary file
It is. This is unrelated to Emacs. The grep tools you're using do not
support utf-16 text.
> (even though I changed the line delimiter to unix style).
Ah, that explains the -unix above.
> What is going on?
I hope this is clearer now.
> Is grep sometimes useless on w32 now, or?
It will be useless whenever you try to grep a file in a coding system
the grep tool does not support...
> (How do we
> handle that in Emacs?)
How do we handle what? If you mean, how can Emacs read the text just
fine... Emacs has supported UTF-16 and many other coding systems for
years.
As for what can you do,
C-x <RET> f iso-latin-1-unix <RET>
then save.
Juanma
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-09-30 23:58 ` Juanma Barranquero
@ 2010-10-01 0:29 ` Lennart Borgman
2010-10-01 0:37 ` Juanma Barranquero
` (2 more replies)
2010-10-01 7:09 ` Eli Zaretskii
1 sibling, 3 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-01 0:29 UTC (permalink / raw
To: Juanma Barranquero; +Cc: Emacs-Devel devel
On Fri, Oct 1, 2010 at 1:58 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>
>> However trying to search this file from a cmd prompt with (gnuwin32)
>> grep does not work. And it does not work with cygwin grep either. They
>> think it is a binary file
>
> It is. This is unrelated to Emacs. The grep tools you're using do not
> support utf-16 text.
>
>> Is grep sometimes useless on w32 now, or?
>
> It will be useless whenever you try to grep a file in a coding system
> the grep tool does not support...
Yes, of course, that is actually what I am saying...
I actually think that there is no grep etc that handles this kind of
files on w32. Is there on *nix?
>> (How do we
>> handle that in Emacs?)
>
> How do we handle what? If you mean, how can Emacs read the text just
> fine... Emacs has supported UTF-16 and many other coding systems for
> years.
Using grep is an integral part of Emacs. If grep does not support the
coding then that part of Emacs will fail.
> As for what can you do,
>
> C-x <RET> f iso-latin-1-unix <RET>
>
> then save.
Yes, but that does not help if you have a lot of files you want to search.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 0:29 ` Lennart Borgman
@ 2010-10-01 0:37 ` Juanma Barranquero
2010-10-01 1:06 ` Lennart Borgman
2010-10-01 5:37 ` Jan D.
2010-10-01 7:13 ` Eli Zaretskii
2010-10-01 22:40 ` Stefan Monnier
2 siblings, 2 replies; 62+ messages in thread
From: Juanma Barranquero @ 2010-10-01 0:37 UTC (permalink / raw
To: Lennart Borgman; +Cc: Emacs-Devel devel
On Fri, Oct 1, 2010 at 02:29, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> I actually think that there is no grep etc that handles this kind of
> files on w32. Is there on *nix?
I don't know. I don't use *nix (though I sometimes use GNU/*ux, but I
haven't needed to grep utf-16 files).
> Using grep is an integral part of Emacs.
Relatively speaking. In thirteen years of Emacs use I've grepped from
inside Emacs perhaps twice.
> If grep does not support the
> coding then that part of Emacs will fail.
Absurd. It's up to you to look for a grep that will grok the files you
want to search. grep is not included with Emacs.
> Yes, but that does not help if you have a lot of files you want to search.
Bad luck. You can always use an elisp script to convert them all.
Juanma
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 0:37 ` Juanma Barranquero
@ 2010-10-01 1:06 ` Lennart Borgman
2010-10-01 1:20 ` Juanma Barranquero
2010-10-01 5:37 ` Jan D.
1 sibling, 1 reply; 62+ messages in thread
From: Lennart Borgman @ 2010-10-01 1:06 UTC (permalink / raw
To: Juanma Barranquero; +Cc: Emacs-Devel devel
On Fri, Oct 1, 2010 at 2:37 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 02:29, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>
>> I actually think that there is no grep etc that handles this kind of
>> files on w32. Is there on *nix?
>
> I don't know. I don't use *nix (though I sometimes use GNU/*ux, but I
> haven't needed to grep utf-16 files).
>
>> Using grep is an integral part of Emacs.
>
> Relatively speaking. In thirteen years of Emacs use I've grepped from
> inside Emacs perhaps twice.
In those a bit less years I have used Emacs regularly now I guess I
might have used Emacs two times without using grep from within Emacs
;-)
>> If grep does not support the
>> coding then that part of Emacs will fail.
>
> Absurd. It's up to you to look for a grep that will grok the files you
> want to search. grep is not included with Emacs.
If the tool chain is broken then the tools that depends on it are also
broken. Aren't they?
(And commands two use Emacs are actually included in the Emacs manual!)
>> Yes, but that does not help if you have a lot of files you want to search.
>
> Bad luck. You can always use an elisp script to convert them all.
Yes, but that is a quite unfortunate state IMO. New users will just
say "ah, this tool is broken, let us try something else".
And I think they are right if we can not cure a problem such as this.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 1:06 ` Lennart Borgman
@ 2010-10-01 1:20 ` Juanma Barranquero
2010-10-01 1:32 ` Lennart Borgman
0 siblings, 1 reply; 62+ messages in thread
From: Juanma Barranquero @ 2010-10-01 1:20 UTC (permalink / raw
To: Lennart Borgman; +Cc: Emacs-Devel devel
On Fri, Oct 1, 2010 at 03:06, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> If the tool chain is broken then the tools that depends on it are also
> broken. Aren't they?
Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
from inside it. YMMV.
> Yes, but that is a quite unfortunate state IMO. New users will just
> say "ah, this tool is broken, let us try something else".
And what will they use?
Juanma
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 1:20 ` Juanma Barranquero
@ 2010-10-01 1:32 ` Lennart Borgman
2010-10-01 1:49 ` Juanma Barranquero
2010-10-01 2:41 ` PJ Weisberg
0 siblings, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-01 1:32 UTC (permalink / raw
To: Juanma Barranquero; +Cc: Emacs-Devel devel
On Fri, Oct 1, 2010 at 3:20 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 03:06, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>
>> If the tool chain is broken then the tools that depends on it are also
>> broken. Aren't they?
>
> Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
> from inside it. YMMV.
I am a bit surprised by that.
>> Yes, but that is a quite unfortunate state IMO. New users will just
>> say "ah, this tool is broken, let us try something else".
>
> And what will they use?
Some tool with an internal search capable of handling utf-16 files?
And are not most of the things to handle that from within Emacs
actually in place? Though of course putting files into buffers to
search them are out of the question for performance reasons.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 1:32 ` Lennart Borgman
@ 2010-10-01 1:49 ` Juanma Barranquero
2010-10-01 2:41 ` PJ Weisberg
1 sibling, 0 replies; 62+ messages in thread
From: Juanma Barranquero @ 2010-10-01 1:49 UTC (permalink / raw
To: Lennart Borgman; +Cc: Emacs-Devel devel
On Fri, Oct 1, 2010 at 03:32, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> I am a bit surprised by that.
Don't know why. As I said, I don't use grep from Emacs.
> Some tool with an internal search capable of handling utf-16 files?
Such as?
> And are not most of the things to handle that from within Emacs
> actually in place?
What do you mean, exactly? If you're trying to grep an external file,
with an external tool, it's hardly an Emacs problem. You could work
around it by loading the files into buffers and using Emacs
functionality (with you reject), or using Emacs to convert the files
and search with the external tool (which you reject), but other than
that, how do you expect Emacs to help?
Juanma
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 1:32 ` Lennart Borgman
2010-10-01 1:49 ` Juanma Barranquero
@ 2010-10-01 2:41 ` PJ Weisberg
2010-10-01 5:43 ` PJ Weisberg
1 sibling, 1 reply; 62+ messages in thread
From: PJ Weisberg @ 2010-10-01 2:41 UTC (permalink / raw
To: Lennart Borgman; +Cc: Juanma Barranquero, Emacs-Devel devel
On 9/30/10, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 3:20 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>> Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
>> from inside it. YMMV.
>
> I am a bit surprised by that.
Well, the "grep through files" function is semi-broken, but then
again, the "send email" function is broken if your mail server is
down. ::shrug::
> And are not most of the things to handle that from within Emacs
> actually in place? Though of course putting files into buffers to
> search them are out of the question for performance reasons.
If you take a look at the grep function in Emacs
you'll see that all it really does is call a shell command and
collect the output. So all the stuff that's "already in place"
probably would require loading the text into a buffer.
^ permalink raw reply [flat|nested] 62+ messages in thread
* What does Emacs on w32 know that grep can't figure out?
2010-09-30 23:13 What does Emacs on w32 know that grep can't figure out? Lennart Borgman
2010-09-30 23:58 ` Juanma Barranquero
@ 2010-10-01 4:00 ` Stephen J. Turnbull
2010-10-01 7:03 ` Eli Zaretskii
2010-10-01 7:29 ` Eli Zaretskii
1 sibling, 2 replies; 62+ messages in thread
From: Stephen J. Turnbull @ 2010-10-01 4:00 UTC (permalink / raw
To: Lennart Borgman; +Cc: Emacs-Devel devel
Lennart Borgman writes:
> However trying to search this file from a cmd prompt with (gnuwin32)
> grep does not work.
No, it almost certainly won't. grep is byte-oriented and doesn't know
anything about coding systems. On Unix with a UTF-8-capable terminal
you would do something like
iconv --from=UTF-16 --to=UTF-8 $FILE | grep some-string
I would think that either Cygwin or Windows provides a version of
iconv. If not, changing the file to UTF-8 (instead of UTF-16) using
Emacs should make it grep'able. In some cases grep may think this is
a binary file anyway; if so, use the --text switch to force grep to
treat the file as text.
> And it does not work with cygwin grep either. They think it is a
> binary file (even though I changed the line delimiter to unix
> style).
The EOL delimiter is not a problem. grep should ignore the presence
or absence of CR when checking for binary files. The only time it is
likely to matter is if you are searching for a word at the end of the
line, in which case instead of "word$" you can use "word\015?$" or
something like that (if it matters, grep may be EOL-agnostic these
days).
Now, of course they think a UTF-16-encoded file is a binary file. It
almost certainly contains NUL bytes (because an ASCII or Latin-1
character will always have a trailing NUL in UTF-16LE).
> What is going on? Is grep sometimes useless on w32 now, or? (How do we
> handle that in Emacs?)
Emacs tries to guess what the encoding is if you don't specify it. It
may guess wrong in certain cases, but it should be extremely accurate
in case of any Unicode format.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 0:37 ` Juanma Barranquero
2010-10-01 1:06 ` Lennart Borgman
@ 2010-10-01 5:37 ` Jan D.
2010-10-01 7:34 ` Eli Zaretskii
1 sibling, 1 reply; 62+ messages in thread
From: Jan D. @ 2010-10-01 5:37 UTC (permalink / raw
To: Juanma Barranquero; +Cc: Lennart Borgman, Emacs-Devel devel
On 2010-10-01 02:37, Juanma Barranquero wrote:
> On Fri, Oct 1, 2010 at 02:29, Lennart Borgman<lennart.borgman@gmail.com> wrote:
>
>> I actually think that there is no grep etc that handles this kind of
>> files on w32. Is there on *nix?
>
> I don't know. I don't use *nix (though I sometimes use GNU/*ux, but I
> haven't needed to grep utf-16 files).
>
This is one of many reasons why *nix-like OSes uses UTF-8. Tools that
only understand ASCII can still work on ASCII. GNU grep understands
UTF-8 I think.
Jan D.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 2:41 ` PJ Weisberg
@ 2010-10-01 5:43 ` PJ Weisberg
0 siblings, 0 replies; 62+ messages in thread
From: PJ Weisberg @ 2010-10-01 5:43 UTC (permalink / raw
To: Lennart Borgman; +Cc: Juanma Barranquero, Emacs-Devel devel
On Thu, Sep 30, 2010 at 7:41 PM, PJ Weisberg
<pj@irregularexpressions.net> wrote:
> On 9/30/10, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>> On Fri, Oct 1, 2010 at 3:20 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>
>>> Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
>>> from inside it. YMMV.
>>
>> I am a bit surprised by that.
>
> Well, the "grep through files" function is semi-broken, but then
> again, the "send email" function is broken if your mail server is
> down. ::shrug::
This issue deserves more respect than it's been getting, though. It's
perfectly reasonable to expect a utility for searching through text to
find text in a file containing text.
Emacs delegates that task to grep, and if it doesn't work properly
that should be addressed in grep. The percentage of grep users who
use it through Emacs is tiny, so assuming for the moment that everyone
agrees that grep should support UTF-16, addressing it in emacs instead
of grep would mean the problem is still mostly unfixed.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 4:00 ` Stephen J. Turnbull
@ 2010-10-01 7:03 ` Eli Zaretskii
2010-10-01 7:29 ` Eli Zaretskii
1 sibling, 0 replies; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-01 7:03 UTC (permalink / raw
To: Stephen J. Turnbull; +Cc: lennart.borgman, emacs-devel
> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 01 Oct 2010 13:00:02 +0900
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>
> > What is going on? Is grep sometimes useless on w32 now, or? (How do we
> > handle that in Emacs?)
>
> Emacs tries to guess what the encoding is if you don't specify it. It
> may guess wrong in certain cases, but it should be extremely accurate
> in case of any Unicode format.
Especially with UTF-16 that always has a BOM signature.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-09-30 23:58 ` Juanma Barranquero
2010-10-01 0:29 ` Lennart Borgman
@ 2010-10-01 7:09 ` Eli Zaretskii
1 sibling, 0 replies; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-01 7:09 UTC (permalink / raw
To: Juanma Barranquero; +Cc: lennart.borgman, emacs-devel
> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 1 Oct 2010 01:58:38 +0200
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>
> > Is grep sometimes useless on w32 now, or?
>
> It will be useless whenever you try to grep a file in a coding system
> the grep tool does not support...
Grep treats its input as a stream of bytes. So it will not be able to
support any encoding where each character, including ASCII, takes 2 or
more bytes. Unless you also encode the search string in the same
encoding, that is.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 0:29 ` Lennart Borgman
2010-10-01 0:37 ` Juanma Barranquero
@ 2010-10-01 7:13 ` Eli Zaretskii
2010-10-01 9:28 ` Lennart Borgman
2010-10-01 22:40 ` Stefan Monnier
2 siblings, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-01 7:13 UTC (permalink / raw
To: Lennart Borgman; +Cc: lekktu, emacs-devel
> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Fri, 1 Oct 2010 02:29:35 +0200
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>
> > How do we handle what? If you mean, how can Emacs read the text just
> > fine... Emacs has supported UTF-16 and many other coding systems for
> > years.
>
> Using grep is an integral part of Emacs. If grep does not support the
> coding then that part of Emacs will fail.
What's wrong with "M-x occur" and/or "M-x multi-occur"?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 4:00 ` Stephen J. Turnbull
2010-10-01 7:03 ` Eli Zaretskii
@ 2010-10-01 7:29 ` Eli Zaretskii
1 sibling, 0 replies; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-01 7:29 UTC (permalink / raw
To: Stephen J. Turnbull; +Cc: lennart.borgman, emacs-devel
> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 01 Oct 2010 13:00:02 +0900
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>
> > And it does not work with cygwin grep either. They think it is a
> > binary file (even though I changed the line delimiter to unix
> > style).
>
> The EOL delimiter is not a problem.
Certainly not. Grep says a file is binary only if it contains null
bytes.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 5:37 ` Jan D.
@ 2010-10-01 7:34 ` Eli Zaretskii
2010-10-01 8:12 ` Andreas Schwab
0 siblings, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-01 7:34 UTC (permalink / raw
To: Jan D.; +Cc: lekktu, lennart.borgman, emacs-devel
> Date: Fri, 01 Oct 2010 07:37:59 +0200
> From: "Jan D." <jan.h.d@swipnet.se>
> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
> Emacs-Devel devel <emacs-devel@gnu.org>
>
> GNU grep understands UTF-8 I think.
Not really. Grep just compares bytes one by one, it knows nothing
about multibyte characters.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 7:34 ` Eli Zaretskii
@ 2010-10-01 8:12 ` Andreas Schwab
2010-10-01 12:00 ` Eli Zaretskii
0 siblings, 1 reply; 62+ messages in thread
From: Andreas Schwab @ 2010-10-01 8:12 UTC (permalink / raw
To: Eli Zaretskii; +Cc: lekktu, Jan D., lennart.borgman, emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
>> Date: Fri, 01 Oct 2010 07:37:59 +0200
>> From: "Jan D." <jan.h.d@swipnet.se>
>> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
>> Emacs-Devel devel <emacs-devel@gnu.org>
>>
>> GNU grep understands UTF-8 I think.
>
> Not really. Grep just compares bytes one by one, it knows nothing
> about multibyte characters.
grep uses exactly the character encoding as specified by the locale.
But it does not support wide character encodings, only multibyte ones.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 7:13 ` Eli Zaretskii
@ 2010-10-01 9:28 ` Lennart Borgman
2010-10-01 10:35 ` Mathias Dahl
2010-10-01 12:02 ` Eli Zaretskii
0 siblings, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-01 9:28 UTC (permalink / raw
To: Eli Zaretskii; +Cc: lekktu, emacs-devel
On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>
>> > How do we handle what? If you mean, how can Emacs read the text just
>> > fine... Emacs has supported UTF-16 and many other coding systems for
>> > years.
>>
>> Using grep is an integral part of Emacs. If grep does not support the
>> coding then that part of Emacs will fail.
>
> What's wrong with "M-x occur" and/or "M-x multi-occur"?
They does not do what grep does.
And as I said opening the files in a buffer will be bad because of performance.
However using the internal regexp engine for searching the files would
be useful, but I have no idea how complicated it is to implement
something like that.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 9:28 ` Lennart Borgman
@ 2010-10-01 10:35 ` Mathias Dahl
2010-10-01 11:24 ` Thierry Volpiatto
2010-10-01 11:26 ` Lennart Borgman
2010-10-01 12:02 ` Eli Zaretskii
1 sibling, 2 replies; 62+ messages in thread
From: Mathias Dahl @ 2010-10-01 10:35 UTC (permalink / raw
To: Lennart Borgman; +Cc: lekktu, Eli Zaretskii, emacs-devel
I often use `% g' in Dired for finding files with a certain text in
them. Together with recursive Dired listings this often solves my
needs (although it is slower than using grep.)
/Mathias
On Fri, Oct 1, 2010 at 11:28 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>>
>>> > How do we handle what? If you mean, how can Emacs read the text just
>>> > fine... Emacs has supported UTF-16 and many other coding systems for
>>> > years.
>>>
>>> Using grep is an integral part of Emacs. If grep does not support the
>>> coding then that part of Emacs will fail.
>>
>> What's wrong with "M-x occur" and/or "M-x multi-occur"?
>
> They does not do what grep does.
>
> And as I said opening the files in a buffer will be bad because of performance.
>
> However using the internal regexp engine for searching the files would
> be useful, but I have no idea how complicated it is to implement
> something like that.
>
>
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 10:35 ` Mathias Dahl
@ 2010-10-01 11:24 ` Thierry Volpiatto
2010-10-01 11:26 ` Lennart Borgman
1 sibling, 0 replies; 62+ messages in thread
From: Thierry Volpiatto @ 2010-10-01 11:24 UTC (permalink / raw
To: emacs-devel
Have a look also at ioccur.el:
http://mercurial.intuxication.org/hg/ioccur
M-x ioccur
M-x ioccur-dired
M-x ioccur-find-buffer-matching
Mathias Dahl <mathias.dahl@gmail.com> writes:
> I often use `% g' in Dired for finding files with a certain text in
> them. Together with recursive Dired listings this often solves my
> needs (although it is slower than using grep.)
>
> /Mathias
>
> On Fri, Oct 1, 2010 at 11:28 AM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>>>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>>>
>>>> > How do we handle what? If you mean, how can Emacs read the text just
>>>> > fine... Emacs has supported UTF-16 and many other coding systems for
>>>> > years.
>>>>
>>>> Using grep is an integral part of Emacs. If grep does not support the
>>>> coding then that part of Emacs will fail.
>>>
>>> What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>>
>> And as I said opening the files in a buffer will be bad because of performance.
>>
>> However using the internal regexp engine for searching the files would
>> be useful, but I have no idea how complicated it is to implement
>> something like that.
>>
>>
>
>
--
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 10:35 ` Mathias Dahl
2010-10-01 11:24 ` Thierry Volpiatto
@ 2010-10-01 11:26 ` Lennart Borgman
1 sibling, 0 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-01 11:26 UTC (permalink / raw
To: Mathias Dahl; +Cc: lekktu, Eli Zaretskii, emacs-devel
Thanks Mathias, that kind of worked, but I get no *grep* buffer.
Beside that it opened Wink for me. Looking at the code I can't see
that it could do that - but it does. Anyone understand what
happened??? (Wink is a w32 program for making screencasts. Very good,
but unfortunately not GPL. I have used it for nXhtml screencasts.)
On Fri, Oct 1, 2010 at 12:35 PM, Mathias Dahl <mathias.dahl@gmail.com> wrote:
> I often use `% g' in Dired for finding files with a certain text in
> them. Together with recursive Dired listings this often solves my
> needs (although it is slower than using grep.)
>
> /Mathias
>
> On Fri, Oct 1, 2010 at 11:28 AM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>>>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>>>
>>>> > How do we handle what? If you mean, how can Emacs read the text just
>>>> > fine... Emacs has supported UTF-16 and many other coding systems for
>>>> > years.
>>>>
>>>> Using grep is an integral part of Emacs. If grep does not support the
>>>> coding then that part of Emacs will fail.
>>>
>>> What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>>
>> And as I said opening the files in a buffer will be bad because of performance.
>>
>> However using the internal regexp engine for searching the files would
>> be useful, but I have no idea how complicated it is to implement
>> something like that.
>>
>>
>
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 8:12 ` Andreas Schwab
@ 2010-10-01 12:00 ` Eli Zaretskii
2010-10-01 12:40 ` Andreas Schwab
` (2 more replies)
0 siblings, 3 replies; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-01 12:00 UTC (permalink / raw
To: Andreas Schwab; +Cc: lekktu, jan.h.d, lennart.borgman, emacs-devel
> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: "Jan D." <jan.h.d@swipnet.se>, lekktu@gmail.com, lennart.borgman@gmail.com, emacs-devel@gnu.org
> Date: Fri, 01 Oct 2010 10:12:08 +0200
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> Date: Fri, 01 Oct 2010 07:37:59 +0200
> >> From: "Jan D." <jan.h.d@swipnet.se>
> >> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
> >> Emacs-Devel devel <emacs-devel@gnu.org>
> >>
> >> GNU grep understands UTF-8 I think.
> >
> > Not really. Grep just compares bytes one by one, it knows nothing
> > about multibyte characters.
>
> grep uses exactly the character encoding as specified by the locale.
What do you mean by "support"? It compares text by comparing its
constituent bytes, without really understanding that a given sequence
of bytes is a single character.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 9:28 ` Lennart Borgman
2010-10-01 10:35 ` Mathias Dahl
@ 2010-10-01 12:02 ` Eli Zaretskii
2010-10-01 15:03 ` Lennart Borgman
2010-10-01 15:11 ` PJ Weisberg
1 sibling, 2 replies; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-01 12:02 UTC (permalink / raw
To: Lennart Borgman; +Cc: lekktu, emacs-devel
> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Fri, 1 Oct 2010 11:28:49 +0200
> Cc: lekktu@gmail.com, emacs-devel@gnu.org
>
> > What's wrong with "M-x occur" and/or "M-x multi-occur"?
>
> They does not do what grep does.
Like what?
> And as I said opening the files in a buffer will be bad because of performance.
What do you think Grep does? Doesn't it open the files and read them?
> However using the internal regexp engine for searching the files would
> be useful, but I have no idea how complicated it is to implement
> something like that.
Emacs does similar things in quite a few places already.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 12:00 ` Eli Zaretskii
@ 2010-10-01 12:40 ` Andreas Schwab
2010-10-01 17:19 ` Jan Djärv
2010-10-01 22:45 ` Stefan Monnier
2 siblings, 0 replies; 62+ messages in thread
From: Andreas Schwab @ 2010-10-01 12:40 UTC (permalink / raw
To: Eli Zaretskii; +Cc: lekktu, jan.h.d, lennart.borgman, emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: "Jan D." <jan.h.d@swipnet.se>, lekktu@gmail.com, lennart.borgman@gmail.com, emacs-devel@gnu.org
>> Date: Fri, 01 Oct 2010 10:12:08 +0200
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> >> Date: Fri, 01 Oct 2010 07:37:59 +0200
>> >> From: "Jan D." <jan.h.d@swipnet.se>
>> >> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
>> >> Emacs-Devel devel <emacs-devel@gnu.org>
>> >>
>> >> GNU grep understands UTF-8 I think.
>> >
>> > Not really. Grep just compares bytes one by one, it knows nothing
>> > about multibyte characters.
>>
>> grep uses exactly the character encoding as specified by the locale.
>
> What do you mean by "support"?
There is no "support" in that sentence?
> It compares text by comparing its constituent bytes, without really
> understanding that a given sequence of bytes is a single character.
You are wrong.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 12:02 ` Eli Zaretskii
@ 2010-10-01 15:03 ` Lennart Borgman
2010-10-01 15:11 ` PJ Weisberg
1 sibling, 0 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-01 15:03 UTC (permalink / raw
To: Eli Zaretskii; +Cc: lekktu, emacs-devel
On Fri, Oct 1, 2010 at 2:02 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Fri, 1 Oct 2010 11:28:49 +0200
>> Cc: lekktu@gmail.com, emacs-devel@gnu.org
>>
>> > What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>
> Like what?
Look at all the files in a directory.
>> And as I said opening the files in a buffer will be bad because of performance.
>
> What do you think Grep does? Doesn't it open the files and read them?
I think we have discussed before that putting the files in a buffer
would be a performance problem in a case like this, especially if the
file is big.
So I think something inside Emacs that could determine file coding but
did not put the file in a buffer is what I would want for this. Is
that hard to implement from what we have?
>> However using the internal regexp engine for searching the files would
>> be useful, but I have no idea how complicated it is to implement
>> something like that.
>
> Emacs does similar things in quite a few places already.
You mean similar to what I suggested above? (Or do you mean something
quite different, something on the elisp level?)
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 12:02 ` Eli Zaretskii
2010-10-01 15:03 ` Lennart Borgman
@ 2010-10-01 15:11 ` PJ Weisberg
1 sibling, 0 replies; 62+ messages in thread
From: PJ Weisberg @ 2010-10-01 15:11 UTC (permalink / raw
To: Eli Zaretskii; +Cc: lekktu, Lennart Borgman, emacs-devel
On Fri, Oct 1, 2010 at 12:09 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> Grep treats its input as a stream of bytes. So it will not be able to
> support any encoding where each character, including ASCII, takes 2 or
> more bytes. Unless you also encode the search string in the same
> encoding, that is.
In 5 minutes of testing I see that grep (GNU grep 2.6.3) doesn't
handle null bytes in the pattern. (String ends at the first null, as
c-strings are wont to do.)
On Fri, Oct 1, 2010 at 5:02 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Fri, 1 Oct 2010 11:28:49 +0200
>> Cc: lekktu@gmail.com, emacs-devel@gnu.org
>>
>> > What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>
> Like what?
Recursively act on all files in a directory, for one.
>> And as I said opening the files in a buffer will be bad because of performance.
>
> What do you think Grep does? Doesn't it open the files and read them?
One line at a time. That'll make a difference for large files.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 12:00 ` Eli Zaretskii
2010-10-01 12:40 ` Andreas Schwab
@ 2010-10-01 17:19 ` Jan Djärv
2010-10-01 22:45 ` Stefan Monnier
2 siblings, 0 replies; 62+ messages in thread
From: Jan Djärv @ 2010-10-01 17:19 UTC (permalink / raw
To: Eli Zaretskii
Cc: lekktu@gmail.com, Andreas Schwab, lennart.borgman@gmail.com,
emacs-devel@gnu.org
1 okt 2010 kl. 14:00 skrev Eli Zaretskii <eliz@gnu.org>:
>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: "Jan D." <jan.h.d@swipnet.se>, lekktu@gmail.com, lennart.borgman@gmail.com, emacs-devel@gnu.org
>> Date: Fri, 01 Oct 2010 10:12:08 +0200
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> Date: Fri, 01 Oct 2010 07:37:59 +0200
>>>> From: "Jan D." <jan.h.d@swipnet.se>
>>>> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
>>>> Emacs-Devel devel <emacs-devel@gnu.org>
>>>>
>>>> GNU grep understands UTF-8 I think.
>>>
>>> Not really. Grep just compares bytes one by one, it knows nothing
>>> about multibyte characters.
>>
>> grep uses exactly the character encoding as specified by the locale.
>
> What do you mean by "support"? It compares text by comparing its
> constituent bytes, without really understanding that a given sequence
> of bytes is a single character.
Grep knows about character classes, like [:alpha:] and [:lower:]. But I guess most of that comes from the locale support in libc.
Jan D.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 0:29 ` Lennart Borgman
2010-10-01 0:37 ` Juanma Barranquero
2010-10-01 7:13 ` Eli Zaretskii
@ 2010-10-01 22:40 ` Stefan Monnier
2010-10-01 23:35 ` Miles Bader
2 siblings, 1 reply; 62+ messages in thread
From: Stefan Monnier @ 2010-10-01 22:40 UTC (permalink / raw
To: Lennart Borgman; +Cc: Juanma Barranquero, Emacs-Devel devel
> I actually think that there is no grep etc that handles this kind of
> files on w32. Is there on *nix?
UTF-16 is the problem, and it (basically) doesn't exist on non-w32
platforms.
Stefan
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 12:00 ` Eli Zaretskii
2010-10-01 12:40 ` Andreas Schwab
2010-10-01 17:19 ` Jan Djärv
@ 2010-10-01 22:45 ` Stefan Monnier
2 siblings, 0 replies; 62+ messages in thread
From: Stefan Monnier @ 2010-10-01 22:45 UTC (permalink / raw
To: Eli Zaretskii
Cc: lekktu, jan.h.d, Andreas Schwab, lennart.borgman, emacs-devel
> What do you mean by "support"? It compares text by comparing its
> constituent bytes, without really understanding that a given sequence
> of bytes is a single character.
A little test shows that "grep foo.bar ~/tmp/foobar"
finds in foobar the line "fooébar" even though that é uses
several bytes (I'm using a utf-8 locale).
Similarly "grep 'é*' toto" and "grep '[éa]' toto" will correctly treat
the several bytes in é as a single element.
Stefan
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 22:40 ` Stefan Monnier
@ 2010-10-01 23:35 ` Miles Bader
2010-10-02 0:33 ` Lennart Borgman
0 siblings, 1 reply; 62+ messages in thread
From: Miles Bader @ 2010-10-01 23:35 UTC (permalink / raw
To: emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I actually think that there is no grep etc that handles this kind of
>> files on w32. Is there on *nix?
>
> UTF-16 is the problem, and it (basically) doesn't exist on non-w32
> platforms.
Yup.
Though it seems MS is trying to fool/coerce the rest of the world into
greater adoption of that abortion... (e.g. the C++0x standard, where
u"xxx" now means UTF-16 text...)
-Miles
--
The trouble with most people is that they think with their hopes or
fears or wishes rather than with their minds. -- Will Durant
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-01 23:35 ` Miles Bader
@ 2010-10-02 0:33 ` Lennart Borgman
2010-10-02 7:29 ` Eli Zaretskii
0 siblings, 1 reply; 62+ messages in thread
From: Lennart Borgman @ 2010-10-02 0:33 UTC (permalink / raw
To: Miles Bader; +Cc: emacs-devel
On Sat, Oct 2, 2010 at 1:35 AM, Miles Bader <miles@gnu.org> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I actually think that there is no grep etc that handles this kind of
>>> files on w32. Is there on *nix?
>>
>> UTF-16 is the problem, and it (basically) doesn't exist on non-w32
>> platforms.
>
> Yup.
>
> Though it seems MS is trying to fool/coerce the rest of the world into
> greater adoption of that abortion... (e.g. the C++0x standard, where
> u"xxx" now means UTF-16 text...)
So where is the problem? Is utf-16 actually used on w32 in many apps?
Or is this a left-over in CCleaner?
If utf-16 is actually used is this there then a policy to not support
it in grep or is it a lack of resources? (Whatever the case it makes
grep a pretty unreliable tool.)
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 0:33 ` Lennart Borgman
@ 2010-10-02 7:29 ` Eli Zaretskii
2010-10-02 10:12 ` Lennart Borgman
0 siblings, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-02 7:29 UTC (permalink / raw
To: Lennart Borgman; +Cc: emacs-devel, miles
> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 2 Oct 2010 02:33:32 +0200
> Cc: emacs-devel@gnu.org
>
> So where is the problem? Is utf-16 actually used on w32 in many apps?
> Or is this a left-over in CCleaner?
UTF-16 is used for all the Unicode APIs on Windows. That's the
implementation of the wchar_t type on Windows. In particular, all the
file names in the NTFS filesystem are actually stored as UTF-16, and
all the OS databases in memory are stored in UTF-16 format.
So this is not a left-over, this is what Windows uses internally.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 7:29 ` Eli Zaretskii
@ 2010-10-02 10:12 ` Lennart Borgman
2010-10-02 10:32 ` Thierry Volpiatto
2010-10-02 11:59 ` Eli Zaretskii
0 siblings, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-02 10:12 UTC (permalink / raw
To: Eli Zaretskii; +Cc: miles, emacs-devel
On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>> Cc: emacs-devel@gnu.org
>>
>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>> Or is this a left-over in CCleaner?
>
> UTF-16 is used for all the Unicode APIs on Windows. That's the
> implementation of the wchar_t type on Windows. In particular, all the
> file names in the NTFS filesystem are actually stored as UTF-16, and
> all the OS databases in memory are stored in UTF-16 format.
>
> So this is not a left-over, this is what Windows uses internally.
Thanks. So then the question is why the grep program does not support
it. Is this a policy question, a lack of resources, or something else?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 10:12 ` Lennart Borgman
@ 2010-10-02 10:32 ` Thierry Volpiatto
2010-10-02 10:50 ` Lennart Borgman
` (2 more replies)
2010-10-02 11:59 ` Eli Zaretskii
1 sibling, 3 replies; 62+ messages in thread
From: Thierry Volpiatto @ 2010-10-02 10:32 UTC (permalink / raw
To: emacs-devel
Lennart Borgman <lennart.borgman@gmail.com> writes:
> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>> Cc: emacs-devel@gnu.org
>>>
>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>> Or is this a left-over in CCleaner?
>>
>> UTF-16 is used for all the Unicode APIs on Windows. That's the
>> implementation of the wchar_t type on Windows. In particular, all the
>> file names in the NTFS filesystem are actually stored as UTF-16, and
>> all the OS databases in memory are stored in UTF-16 format.
>>
>> So this is not a left-over, this is what Windows uses internally.
>
>
> Thanks. So then the question is why the grep program does not support
> it. Is this a policy question, a lack of resources, or something else?
>
>
It seem most modern windows versions come with powershell.
Why windows users don't use it as backend for Emacs instead of using
external Gnuwin32 programs?
--
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 10:32 ` Thierry Volpiatto
@ 2010-10-02 10:50 ` Lennart Borgman
2010-10-02 12:01 ` Eli Zaretskii
2010-10-02 14:51 ` Lennart Borgman
2 siblings, 0 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-02 10:50 UTC (permalink / raw
To: Thierry Volpiatto; +Cc: emacs-devel
On Sat, Oct 2, 2010 at 12:32 PM, Thierry Volpiatto
<thierry.volpiatto@gmail.com> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>>> Or is this a left-over in CCleaner?
>>>
>>> UTF-16 is used for all the Unicode APIs on Windows. That's the
>>> implementation of the wchar_t type on Windows. In particular, all the
>>> file names in the NTFS filesystem are actually stored as UTF-16, and
>>> all the OS databases in memory are stored in UTF-16 format.
>>>
>>> So this is not a left-over, this is what Windows uses internally.
>>
>>
>> Thanks. So then the question is why the grep program does not support
>> it. Is this a policy question, a lack of resources, or something else?
>>
>>
> It seem most modern windows versions come with powershell.
> Why windows users don't use it as backend for Emacs instead of using
> external Gnuwin32 programs?
Are you effectively not saying that because of the lack of utf-16
support in gnu grep it would be better for Emacs to depend on non-gnu
and even ms tools on w32?
I would say: perhaps. What do others think?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 10:12 ` Lennart Borgman
2010-10-02 10:32 ` Thierry Volpiatto
@ 2010-10-02 11:59 ` Eli Zaretskii
2010-10-02 12:44 ` Lennart Borgman
1 sibling, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-02 11:59 UTC (permalink / raw
To: Lennart Borgman; +Cc: miles, emacs-devel
> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 2 Oct 2010 12:12:08 +0200
> Cc: emacs-devel@gnu.org, miles@gnu.org
>
> Thanks. So then the question is why the grep program does not support
> it.
Because no one has coded that in Grep, I'd guess.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 10:32 ` Thierry Volpiatto
2010-10-02 10:50 ` Lennart Borgman
@ 2010-10-02 12:01 ` Eli Zaretskii
2010-10-02 14:51 ` Lennart Borgman
2 siblings, 0 replies; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-02 12:01 UTC (permalink / raw
To: Thierry Volpiatto; +Cc: emacs-devel
> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Sat, 02 Oct 2010 12:32:16 +0200
>
> It seem most modern windows versions come with powershell.
> Why windows users don't use it as backend for Emacs instead of using
> external Gnuwin32 programs?
How would that help, exactly?
When PowerShell came out, I tried it, and found that it had its own
quirks, not surprisingly. So we would be replacing one bunch of
problems with another. At least the current tools are widely known
and well understood, and we have much higher chances getting changes
into tools like Grep than into PowerShell.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 11:59 ` Eli Zaretskii
@ 2010-10-02 12:44 ` Lennart Borgman
2010-10-02 13:15 ` Eli Zaretskii
0 siblings, 1 reply; 62+ messages in thread
From: Lennart Borgman @ 2010-10-02 12:44 UTC (permalink / raw
To: Eli Zaretskii; +Cc: emacs-devel, miles
On Sat, Oct 2, 2010 at 1:59 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 2 Oct 2010 12:12:08 +0200
>> Cc: emacs-devel@gnu.org, miles@gnu.org
>>
>> Thanks. So then the question is why the grep program does not support
>> it.
>
> Because no one has coded that in Grep, I'd guess.
Is that really the whole answer? If utf-16 is so widely used on w32
then is it not surprising that it is not understood by Grep?
Maybe you are right, but it would be interesting to get some more
insight in this.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 12:44 ` Lennart Borgman
@ 2010-10-02 13:15 ` Eli Zaretskii
0 siblings, 0 replies; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-02 13:15 UTC (permalink / raw
To: Lennart Borgman; +Cc: emacs-devel, miles
> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 2 Oct 2010 14:44:31 +0200
> Cc: miles@gnu.org, emacs-devel@gnu.org
>
> On Sat, Oct 2, 2010 at 1:59 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Lennart Borgman <lennart.borgman@gmail.com>
> >> Date: Sat, 2 Oct 2010 12:12:08 +0200
> >> Cc: emacs-devel@gnu.org, miles@gnu.org
> >>
> >> Thanks. So then the question is why the grep program does not support
> >> it.
> >
> > Because no one has coded that in Grep, I'd guess.
>
>
> Is that really the whole answer? If utf-16 is so widely used on w32
> then is it not surprising that it is not understood by Grep?
Look at the ChangeLog in Grep distribution: there are exactly zero
contributors of Windows related code there. About the only entry that
mentions Windows is a minor bug fixed 6 months ago. And that's it.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 10:32 ` Thierry Volpiatto
2010-10-02 10:50 ` Lennart Borgman
2010-10-02 12:01 ` Eli Zaretskii
@ 2010-10-02 14:51 ` Lennart Borgman
2010-10-02 15:05 ` Thierry Volpiatto
2 siblings, 1 reply; 62+ messages in thread
From: Lennart Borgman @ 2010-10-02 14:51 UTC (permalink / raw
To: Thierry Volpiatto; +Cc: emacs-devel
On Sat, Oct 2, 2010 at 12:32 PM, Thierry Volpiatto
<thierry.volpiatto@gmail.com> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>>> Or is this a left-over in CCleaner?
>>>
>>> UTF-16 is used for all the Unicode APIs on Windows. That's the
>>> implementation of the wchar_t type on Windows. In particular, all the
>>> file names in the NTFS filesystem are actually stored as UTF-16, and
>>> all the OS databases in memory are stored in UTF-16 format.
>>>
>>> So this is not a left-over, this is what Windows uses internally.
>>
>>
>> Thanks. So then the question is why the grep program does not support
>> it. Is this a policy question, a lack of resources, or something else?
>>
>>
> It seem most modern windows versions come with powershell.
> Why windows users don't use it as backend for Emacs instead of using
> external Gnuwin32 programs?
Based on your advice I just took the trouble to install powershell. I
can not see that it addresses the problem with utf-16, but maybe I am
missing something. Can you please tell me how to search also utf-16
files with it?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 14:51 ` Lennart Borgman
@ 2010-10-02 15:05 ` Thierry Volpiatto
2010-10-02 15:23 ` Eli Zaretskii
0 siblings, 1 reply; 62+ messages in thread
From: Thierry Volpiatto @ 2010-10-02 15:05 UTC (permalink / raw
To: Lennart Borgman; +Cc: emacs-devel
Lennart Borgman <lennart.borgman@gmail.com> writes:
> On Sat, Oct 2, 2010 at 12:32 PM, Thierry Volpiatto
> <thierry.volpiatto@gmail.com> wrote:
>> Lennart Borgman <lennart.borgman@gmail.com> writes:
>>
>>> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>>>> Cc: emacs-devel@gnu.org
>>>>>
>>>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>>>> Or is this a left-over in CCleaner?
>>>>
>>>> UTF-16 is used for all the Unicode APIs on Windows. That's the
>>>> implementation of the wchar_t type on Windows. In particular, all the
>>>> file names in the NTFS filesystem are actually stored as UTF-16, and
>>>> all the OS databases in memory are stored in UTF-16 format.
>>>>
>>>> So this is not a left-over, this is what Windows uses internally.
>>>
>>>
>>> Thanks. So then the question is why the grep program does not support
>>> it. Is this a policy question, a lack of resources, or something else?
>>>
>>>
>> It seem most modern windows versions come with powershell.
>> Why windows users don't use it as backend for Emacs instead of using
>> external Gnuwin32 programs?
>
>
> Based on your advice I just took the trouble to install powershell. I
> can not see that it addresses the problem with utf-16, but maybe I am
> missing something. Can you please tell me how to search also utf-16
> files with it?
That's just an idea i sent, i personally don't use (or really few
sometimes on other machines) windows systems and i have just a basic
knowledge of all that is related to it and microsoft in general (e.g Powershell).
So i can't help you much here.
Anyway, Eli seem to have tried it and is not convinced.
--
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 15:05 ` Thierry Volpiatto
@ 2010-10-02 15:23 ` Eli Zaretskii
2010-10-02 15:56 ` Lennart Borgman
0 siblings, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-02 15:23 UTC (permalink / raw
To: Thierry Volpiatto; +Cc: lennart.borgman, emacs-devel
> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Sat, 02 Oct 2010 17:05:18 +0200
> Cc: emacs-devel@gnu.org
>
> > Based on your advice I just took the trouble to install powershell. I
> > can not see that it addresses the problem with utf-16, but maybe I am
> > missing something. Can you please tell me how to search also utf-16
> > files with it?
>
> That's just an idea i sent, i personally don't use (or really few
> sometimes on other machines) windows systems and i have just a basic
> knowledge of all that is related to it and microsoft in general (e.g Powershell).
> So i can't help you much here.
> Anyway, Eli seem to have tried it and is not convinced.
Looks like the PowerShell cmdlet that's supposed to be equivalent of
Grep is Select-String.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 15:23 ` Eli Zaretskii
@ 2010-10-02 15:56 ` Lennart Borgman
2010-10-02 20:40 ` Mathias Dahl
2010-10-03 4:10 ` Lennart Borgman
0 siblings, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-02 15:56 UTC (permalink / raw
To: Eli Zaretskii; +Cc: emacs-devel, Thierry Volpiatto
On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>> Cc: emacs-devel@gnu.org
>>
>> > Based on your advice I just took the trouble to install powershell. I
>> > can not see that it addresses the problem with utf-16, but maybe I am
>> > missing something. Can you please tell me how to search also utf-16
>> > files with it?
>>
>> That's just an idea i sent, i personally don't use (or really few
>> sometimes on other machines) windows systems and i have just a basic
>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>> So i can't help you much here.
>> Anyway, Eli seem to have tried it and is not convinced.
>
> Looks like the PowerShell cmdlet that's supposed to be equivalent of
> Grep is Select-String.
Yes, thanks. And that actually works with utf-16.
So now we have a strange situation.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 15:56 ` Lennart Borgman
@ 2010-10-02 20:40 ` Mathias Dahl
2010-10-02 22:20 ` Eli Zaretskii
2010-10-03 4:10 ` Lennart Borgman
1 sibling, 1 reply; 62+ messages in thread
From: Mathias Dahl @ 2010-10-02 20:40 UTC (permalink / raw
To: Lennart Borgman; +Cc: Eli Zaretskii, Thierry Volpiatto, emacs-devel
If Windows' built-in find command supports UTF-16, then that might be
an option (maybe better since I think find will be installed on more
Windows PCs) too:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true
It sounds like it does at least parts of what grep does. Try it out and see.
/Mathias
On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>> Cc: emacs-devel@gnu.org
>>>
>>> > Based on your advice I just took the trouble to install powershell. I
>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>> > missing something. Can you please tell me how to search also utf-16
>>> > files with it?
>>>
>>> That's just an idea i sent, i personally don't use (or really few
>>> sometimes on other machines) windows systems and i have just a basic
>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>> So i can't help you much here.
>>> Anyway, Eli seem to have tried it and is not convinced.
>>
>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>> Grep is Select-String.
>
> Yes, thanks. And that actually works with utf-16.
>
> So now we have a strange situation.
>
>
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 20:40 ` Mathias Dahl
@ 2010-10-02 22:20 ` Eli Zaretskii
2010-10-03 0:35 ` David Robinow
0 siblings, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-02 22:20 UTC (permalink / raw
To: Mathias Dahl; +Cc: thierry.volpiatto, lennart.borgman, emacs-devel
> From: Mathias Dahl <mathias.dahl@gmail.com>
> Date: Sat, 2 Oct 2010 22:40:15 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
> Thierry Volpiatto <thierry.volpiatto@gmail.com>
>
> If Windows' built-in find command supports UTF-16, then that might be
> an option (maybe better since I think find will be installed on more
> Windows PCs) too:
>
> http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true
Somebody should first make sure FIND supports Unicode strings. Most
of the console programs don't.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 22:20 ` Eli Zaretskii
@ 2010-10-03 0:35 ` David Robinow
2010-10-03 4:05 ` Eli Zaretskii
0 siblings, 1 reply; 62+ messages in thread
From: David Robinow @ 2010-10-03 0:35 UTC (permalink / raw
To: Eli Zaretskii
Cc: emacs-devel, thierry.volpiatto, lennart.borgman, Mathias Dahl
On Sat, Oct 2, 2010 at 6:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Mathias Dahl <mathias.dahl@gmail.com>
>> Date: Sat, 2 Oct 2010 22:40:15 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>> Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>
>> If Windows' built-in find command supports UTF-16, then that might be
>> an option (maybe better since I think find will be installed on more
>> Windows PCs) too:
>>
>> http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true
>
> Somebody should first make sure FIND supports Unicode strings. Most
> of the console programs don't.
>
It appears to (Vista Home Premium), as do TYPE and MORE. Which ones don''t?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 0:35 ` David Robinow
@ 2010-10-03 4:05 ` Eli Zaretskii
2010-10-03 4:29 ` David Robinow
0 siblings, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-03 4:05 UTC (permalink / raw
To: David Robinow
Cc: emacs-devel, thierry.volpiatto, lennart.borgman, mathias.dahl
> Date: Sat, 2 Oct 2010 20:35:30 -0400
> From: David Robinow <drobinow@gmail.com>
> Cc: Mathias Dahl <mathias.dahl@gmail.com>, thierry.volpiatto@gmail.com,
> lennart.borgman@gmail.com, emacs-devel@gnu.org
>
> On Sat, Oct 2, 2010 at 6:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Mathias Dahl <mathias.dahl@gmail.com>
> >> Date: Sat, 2 Oct 2010 22:40:15 +0200
> >> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
> >> Thierry Volpiatto <thierry.volpiatto@gmail.com>
> >>
> >> If Windows' built-in find command supports UTF-16, then that might be
> >> an option (maybe better since I think find will be installed on more
> >> Windows PCs) too:
> >>
> >> http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true
> >
> > Somebody should first make sure FIND supports Unicode strings. Most
> > of the console programs don't.
> >
> It appears to (Vista Home Premium)
How did you test that?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-02 15:56 ` Lennart Borgman
2010-10-02 20:40 ` Mathias Dahl
@ 2010-10-03 4:10 ` Lennart Borgman
2010-10-03 10:39 ` Thierry Volpiatto
2010-10-03 19:09 ` Eli Zaretskii
1 sibling, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-03 4:10 UTC (permalink / raw
To: Eli Zaretskii; +Cc: emacs-devel, Thierry Volpiatto
On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>> Cc: emacs-devel@gnu.org
>>>
>>> > Based on your advice I just took the trouble to install powershell. I
>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>> > missing something. Can you please tell me how to search also utf-16
>>> > files with it?
>>>
>>> That's just an idea i sent, i personally don't use (or really few
>>> sometimes on other machines) windows systems and i have just a basic
>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>> So i can't help you much here.
>>> Anyway, Eli seem to have tried it and is not convinced.
>>
>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>> Grep is Select-String.
>
> Yes, thanks. And that actually works with utf-16.
>
> So now we have a strange situation.
Or maybe not. Looking at the documentation for Select-String here
http://technet.microsoft.com/en-us/library/dd315403.aspx
it does not look as it autodetects the coding system in the file.
It does not even say it can handle utf-16, or at least I can't find
it. Maybe someone who understands this better can see it there?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 4:05 ` Eli Zaretskii
@ 2010-10-03 4:29 ` David Robinow
2010-10-03 5:39 ` Eli Zaretskii
0 siblings, 1 reply; 62+ messages in thread
From: David Robinow @ 2010-10-03 4:29 UTC (permalink / raw
To: Eli Zaretskii
Cc: emacs-devel, thierry.volpiatto, lennart.borgman, mathias.dahl
On Sun, Oct 3, 2010 at 12:05 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sat, 2 Oct 2010 20:35:30 -0400
>> From: David Robinow <drobinow@gmail.com>
>> Cc: Mathias Dahl <mathias.dahl@gmail.com>, thierry.volpiatto@gmail.com,
>> lennart.borgman@gmail.com, emacs-devel@gnu.org
>>
>> On Sat, Oct 2, 2010 at 6:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> > Somebody should first make sure FIND supports Unicode strings. Most
>> > of the console programs don't.
>> >
>> It appears to (Vista Home Premium)
>
> How did you test that?
I found a UTF-16 file on my computer. Used 'TYPE' from a cmd window
to see the data.
Used cygwin 'file' to verify that it was UTF-16.
Then cygwin 'grep' which failed (I don't have a Windows version but
nobody claims it works anyway). Then 'FIND' from a cmd window (using
the same string which failed for grep) and it worked.
I will try Windows XP tomorrow.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 4:29 ` David Robinow
@ 2010-10-03 5:39 ` Eli Zaretskii
2010-10-03 7:25 ` Thierry Volpiatto
0 siblings, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-03 5:39 UTC (permalink / raw
To: David Robinow
Cc: emacs-devel, thierry.volpiatto, lennart.borgman, mathias.dahl
> Date: Sun, 3 Oct 2010 00:29:41 -0400
> From: David Robinow <drobinow@gmail.com>
> Cc: mathias.dahl@gmail.com, thierry.volpiatto@gmail.com,
> lennart.borgman@gmail.com, emacs-devel@gnu.org
>
> > How did you test that?
> I found a UTF-16 file on my computer. Used 'TYPE' from a cmd window
> to see the data.
I'm not interested in TYPE, it's an internal command of CMD, not a
program.
> Used cygwin 'file' to verify that it was UTF-16.
> Then cygwin 'grep' which failed (I don't have a Windows version but
> nobody claims it works anyway). Then 'FIND' from a cmd window (using
> the same string which failed for grep) and it worked.
What string did you use with FIND, and (assuming it included non-ASCII
characters) how did you type it into the CMD window?
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 5:39 ` Eli Zaretskii
@ 2010-10-03 7:25 ` Thierry Volpiatto
0 siblings, 0 replies; 62+ messages in thread
From: Thierry Volpiatto @ 2010-10-03 7:25 UTC (permalink / raw
To: emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
>> Date: Sun, 3 Oct 2010 00:29:41 -0400
>> From: David Robinow <drobinow@gmail.com>
>> Cc: mathias.dahl@gmail.com, thierry.volpiatto@gmail.com,
>> lennart.borgman@gmail.com, emacs-devel@gnu.org
>>
>> > How did you test that?
>> I found a UTF-16 file on my computer. Used 'TYPE' from a cmd window
>> to see the data.
>
> I'm not interested in TYPE, it's an internal command of CMD, not a
> program.
>
>> Used cygwin 'file' to verify that it was UTF-16.
>> Then cygwin 'grep' which failed (I don't have a Windows version but
>> nobody claims it works anyway). Then 'FIND' from a cmd window (using
>> the same string which failed for grep) and it worked.
>
> What string did you use with FIND, and (assuming it included non-ASCII
> characters) how did you type it into the CMD window?
>
I have tried FINDSTR.exe from powershell, and it seem to work fine.
http://tinyurl.com/36vez2n
--
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 4:10 ` Lennart Borgman
@ 2010-10-03 10:39 ` Thierry Volpiatto
2010-10-03 13:23 ` Thierry Volpiatto
2010-10-03 19:09 ` Eli Zaretskii
1 sibling, 1 reply; 62+ messages in thread
From: Thierry Volpiatto @ 2010-10-03 10:39 UTC (permalink / raw
To: emacs-devel
Lennart Borgman <lennart.borgman@gmail.com> writes:
> On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>> > Based on your advice I just took the trouble to install powershell. I
>>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>>> > missing something. Can you please tell me how to search also utf-16
>>>> > files with it?
>>>>
>>>> That's just an idea i sent, i personally don't use (or really few
>>>> sometimes on other machines) windows systems and i have just a basic
>>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>>> So i can't help you much here.
>>>> Anyway, Eli seem to have tried it and is not convinced.
>>>
>>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>>> Grep is Select-String.
>>
>> Yes, thanks. And that actually works with utf-16.
>>
>> So now we have a strange situation.
>
>
> Or maybe not. Looking at the documentation for Select-String here
>
> http://technet.microsoft.com/en-us/library/dd315403.aspx
>
> it does not look as it autodetects the coding system in the file.
>
> It does not even say it can handle utf-16, or at least I can't find
> it. Maybe someone who understands this better can see it there?
Can you try to use FINDSTR from command line (it work from eshell) and
tell me if that work on your utf-16 files?
(It seem this command is not part of powershell)
I wrote some code that use this command with success on windows (with
anything).
That would allow to have a decent grep that work on windows without
having to install external programs.
Even if that seem not powerfull as grep for regexps, that provide a
default grep command better than "grep command not found".
--
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 10:39 ` Thierry Volpiatto
@ 2010-10-03 13:23 ` Thierry Volpiatto
0 siblings, 0 replies; 62+ messages in thread
From: Thierry Volpiatto @ 2010-10-03 13:23 UTC (permalink / raw
To: emacs-devel
Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
>> <lennart.borgman@gmail.com> wrote:
>>> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>>>> Cc: emacs-devel@gnu.org
>>>>>
>>>>> > Based on your advice I just took the trouble to install powershell. I
>>>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>>>> > missing something. Can you please tell me how to search also utf-16
>>>>> > files with it?
>>>>>
>>>>> That's just an idea i sent, i personally don't use (or really few
>>>>> sometimes on other machines) windows systems and i have just a basic
>>>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>>>> So i can't help you much here.
>>>>> Anyway, Eli seem to have tried it and is not convinced.
>>>>
>>>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>>>> Grep is Select-String.
>>>
>>> Yes, thanks. And that actually works with utf-16.
>>>
>>> So now we have a strange situation.
>>
>>
>> Or maybe not. Looking at the documentation for Select-String here
>>
>> http://technet.microsoft.com/en-us/library/dd315403.aspx
>>
>> it does not look as it autodetects the coding system in the file.
>>
>> It does not even say it can handle utf-16, or at least I can't find
>> it. Maybe someone who understands this better can see it there?
>
> Can you try to use FINDSTR from command line (it work from eshell) and
> tell me if that work on your utf-16 files?
> (It seem this command is not part of powershell)
>
> I wrote some code that use this command with success on windows (with
> anything).
It is now available on anything-config.el
Use M-x anything-do-grep
No tweak, work out of the box on GNU/Linux and windows as it is.
That's a first shot, feedback welcome.
> That would allow to have a decent grep that work on windows without
> having to install external programs.
>
> Even if that seem not powerfull as grep for regexps, that provide a
> default grep command better than "grep command not found".
--
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 4:10 ` Lennart Borgman
2010-10-03 10:39 ` Thierry Volpiatto
@ 2010-10-03 19:09 ` Eli Zaretskii
2010-10-03 22:50 ` Lennart Borgman
1 sibling, 1 reply; 62+ messages in thread
From: Eli Zaretskii @ 2010-10-03 19:09 UTC (permalink / raw
To: Lennart Borgman; +Cc: emacs-devel, thierry.volpiatto
> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 3 Oct 2010 06:10:14 +0200
> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
>
> http://technet.microsoft.com/en-us/library/dd315403.aspx
>
> it does not look as it autodetects the coding system in the file.
Why should it? What tool does, besides Emacs? You are supposed to
tell it the encoding using the -Encoding option.
> It does not even say it can handle utf-16, or at least I can't find
> it.
It does support UTF-16. That's the "Unicode" part of the values you
can submit to the -Encoding option (you need to "think MS" to get it).
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 19:09 ` Eli Zaretskii
@ 2010-10-03 22:50 ` Lennart Borgman
2010-10-05 0:46 ` Lennart Borgman
0 siblings, 1 reply; 62+ messages in thread
From: Lennart Borgman @ 2010-10-03 22:50 UTC (permalink / raw
To: Eli Zaretskii; +Cc: thierry.volpiatto, emacs-devel
On Sun, Oct 3, 2010 at 9:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sun, 3 Oct 2010 06:10:14 +0200
>> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
>>
>> http://technet.microsoft.com/en-us/library/dd315403.aspx
>>
>> it does not look as it autodetects the coding system in the file.
>
> Why should it?
I don't understand why you are asking. The answer is of course:
Because it would be useful when searching for words in many files (and
you do not know their coding system).
Or, don't you think so?
> What tool does, besides Emacs?
I am not sure, but I would expect at least some better search
engines/indexers to do this. I can't see how they can avoid it.
>> It does not even say it can handle utf-16, or at least I can't find
>> it.
>
> It does support UTF-16. That's the "Unicode" part of the values you
> can submit to the -Encoding option (you need to "think MS" to get it).
Thanks.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-03 22:50 ` Lennart Borgman
@ 2010-10-05 0:46 ` Lennart Borgman
2010-10-05 0:51 ` Lennart Borgman
2010-10-05 0:56 ` Juanma Barranquero
0 siblings, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-05 0:46 UTC (permalink / raw
To: Eli Zaretskii; +Cc: thierry.volpiatto, emacs-devel
On Mon, Oct 4, 2010 at 12:50 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sun, Oct 3, 2010 at 9:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sun, 3 Oct 2010 06:10:14 +0200
>>> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
>>>
>>> http://technet.microsoft.com/en-us/library/dd315403.aspx
>>>
>>> it does not look as it autodetects the coding system in the file.
>>
>> Why should it?
..
>> What tool does, besides Emacs?
..
>> It does support UTF-16. That's the "Unicode" part of the values you
>> can submit to the -Encoding option (you need to "think MS" to get it).
This discussion seems to have stalled. So here is a summary of what we
have found:
- utf-16 is a problem since the grep program does not handle it and
utf-16 is common on w32.
- There seems to be no program we can use that autodetects file coding
the way Emacs does.
- I would expect Emacs users to believe thata search from within Emacs
would autodetect coding system since Emacs does it. (Eli disagrees on
this.)
What can we do? The options I can see are:
- Use powershell + select-string (+ another cmdlet for dir tree
searching) on w32 by default. That has two benefits:
1. Users on w32 does not have to install grep+find. But on older
systems they have to install powershell. Anyway it is easy to test for
powershell and use it if it is installed. (A problem is the hardcoding
of parameters to grep, like -i, but that can be resolved).
2. Make an internal grep command in Emacs. A naive version in elisp
is quick to write but will be inefficient especially for large files.
A C version will require some restructuring of insert_file_contents,
but seems otherwise not hard to code. (Except perhaps for buffering
for effiency, but I have no idea if that is needed.)
I think 2 is the best choice, but I do not expect anyone to write it
now. Maybe add it to our to-do list?
For the moment I suggest that we implement 1.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-05 0:46 ` Lennart Borgman
@ 2010-10-05 0:51 ` Lennart Borgman
2010-10-05 0:56 ` Juanma Barranquero
1 sibling, 0 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-05 0:51 UTC (permalink / raw
To: Eli Zaretskii; +Cc: thierry.volpiatto, emacs-devel
On Tue, Oct 5, 2010 at 2:46 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Mon, Oct 4, 2010 at 12:50 AM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Sun, Oct 3, 2010 at 9:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Sun, 3 Oct 2010 06:10:14 +0200
>>>> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
>>>>
>>>> http://technet.microsoft.com/en-us/library/dd315403.aspx
>>>>
>>>> it does not look as it autodetects the coding system in the file.
>>>
>>> Why should it?
> ..
>>> What tool does, besides Emacs?
> ..
>>> It does support UTF-16. That's the "Unicode" part of the values you
>>> can submit to the -Encoding option (you need to "think MS" to get it).
>
> This discussion seems to have stalled. So here is a summary of what we
> have found:
>
> - utf-16 is a problem since the grep program does not handle it and
> utf-16 is common on w32.
>
> - There seems to be no program we can use that autodetects file coding
> the way Emacs does.
>
> - I would expect Emacs users to believe thata search from within Emacs
> would autodetect coding system since Emacs does it. (Eli disagrees on
> this.)
A shit. Doing two things simultaneously does not improve you
writing... It should have been like this:
What can we do? The options I can see are:
A) Use powershell + select-string (+ another cmdlet for dir tree
searching) on w32 by default. That has two benefits:
1. Users on w32 does not have to install grep+find. But on older
systems they have to install powershell. Anyway it is easy to test for
powershell and use it if it is installed. (A problem is the hardcoding
of parameters to grep, like -i, but that can be resolved).
2. utf-16 files can also be searched on w32.
B) Make an internal grep command in Emacs. A naive version in elisp
is quick to write but will be inefficient especially for large files.
A C version will require some restructuring of insert_file_contents,
but seems otherwise not hard to code. (Except perhaps for buffering
for effiency, but I have no idea if that is needed.)
I think B is the best choice, but I do not expect anyone to write it
now. Maybe add it to our to-do list?
For the moment I suggest that we implement A.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-05 0:46 ` Lennart Borgman
2010-10-05 0:51 ` Lennart Borgman
@ 2010-10-05 0:56 ` Juanma Barranquero
2010-10-05 1:36 ` Lennart Borgman
1 sibling, 1 reply; 62+ messages in thread
From: Juanma Barranquero @ 2010-10-05 0:56 UTC (permalink / raw
To: Lennart Borgman; +Cc: Eli Zaretskii, emacs-devel, thierry.volpiatto
On Tue, Oct 5, 2010 at 02:46, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> - I would expect Emacs users to believe thata search from within Emacs
> would autodetect coding system since Emacs does it. (Eli disagrees on
> this.)
I agree with him, BTW.
> What can we do? The options I can see are:
C.- Do nothing, and leave to the user the task of finding a tool that
can grep utf-16.
D.- Do nothing, and leave to the user the task of converting the
utf-16 files to another coding, as utf-16 is the internal format of
32-bit Windows APIs, but it is rarely found in files, even on Windows.
Juanma
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-05 0:56 ` Juanma Barranquero
@ 2010-10-05 1:36 ` Lennart Borgman
2010-10-05 1:37 ` Lennart Borgman
2010-10-05 1:54 ` Juanma Barranquero
0 siblings, 2 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-05 1:36 UTC (permalink / raw
To: Juanma Barranquero; +Cc: Eli Zaretskii, emacs-devel, thierry.volpiatto
On Tue, Oct 5, 2010 at 2:56 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>
>> What can we do? The options I can see are:
>
> C.- Do nothing, and leave to the user the task of finding a tool that
> can grep utf-16.
>
> D.- Do nothing, and leave to the user the task of converting the
> utf-16 files to another coding, as utf-16 is the internal format of
> 32-bit Windows APIs, but it is rarely found in files, even on Windows.
How would the user no which of C and D we have choosen?
If we decide on this (which seems likely now), then should we not
change default file coding system to utf-8? We kind of expect that,
but I do not understand what consequences changing the default file
coding system will have.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-05 1:36 ` Lennart Borgman
@ 2010-10-05 1:37 ` Lennart Borgman
2010-10-05 1:54 ` Juanma Barranquero
1 sibling, 0 replies; 62+ messages in thread
From: Lennart Borgman @ 2010-10-05 1:37 UTC (permalink / raw
To: Juanma Barranquero; +Cc: Eli Zaretskii, emacs-devel, thierry.volpiatto
On Tue, Oct 5, 2010 at 3:36 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Tue, Oct 5, 2010 at 2:56 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>>
>>> What can we do? The options I can see are:
>>
>> C.- Do nothing, and leave to the user the task of finding a tool that
>> can grep utf-16.
>>
>> D.- Do nothing, and leave to the user the task of converting the
>> utf-16 files to another coding, as utf-16 is the internal format of
>> 32-bit Windows APIs, but it is rarely found in files, even on Windows.
>
> How would the user no which of C and D we have choosen?
no => know ...
> If we decide on this (which seems likely now), then should we not
> change default file coding system to utf-8? We kind of expect that,
> but I do not understand what consequences changing the default file
> coding system will have.
>
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: What does Emacs on w32 know that grep can't figure out?
2010-10-05 1:36 ` Lennart Borgman
2010-10-05 1:37 ` Lennart Borgman
@ 2010-10-05 1:54 ` Juanma Barranquero
1 sibling, 0 replies; 62+ messages in thread
From: Juanma Barranquero @ 2010-10-05 1:54 UTC (permalink / raw
To: Lennart Borgman; +Cc: Eli Zaretskii, emacs-devel, thierry.volpiatto
On Tue, Oct 5, 2010 at 03:36, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> If we decide on this (which seems likely now), then should we not
> change default file coding system to utf-8?
Which coding-system variable are you referring to? buffer-file-coding-system?
The language environment (and coding system variables) should reflex,
IMO, what the user is most likely to encounter.
I personally do
(set-language-environment "UTF-8")
(set-coding-system-priority 'utf-16le-with-signature 'utf-16be-with-signature)
(setq file-name-coding-system 'windows-1252)
in my .emacs, but I want to generate UTF-8 files. I think most Windows
users will be happy with the current defaults.
Juanma
^ permalink raw reply [flat|nested] 62+ messages in thread
end of thread, other threads:[~2010-10-05 1:54 UTC | newest]
Thread overview: 62+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-09-30 23:13 What does Emacs on w32 know that grep can't figure out? Lennart Borgman
2010-09-30 23:58 ` Juanma Barranquero
2010-10-01 0:29 ` Lennart Borgman
2010-10-01 0:37 ` Juanma Barranquero
2010-10-01 1:06 ` Lennart Borgman
2010-10-01 1:20 ` Juanma Barranquero
2010-10-01 1:32 ` Lennart Borgman
2010-10-01 1:49 ` Juanma Barranquero
2010-10-01 2:41 ` PJ Weisberg
2010-10-01 5:43 ` PJ Weisberg
2010-10-01 5:37 ` Jan D.
2010-10-01 7:34 ` Eli Zaretskii
2010-10-01 8:12 ` Andreas Schwab
2010-10-01 12:00 ` Eli Zaretskii
2010-10-01 12:40 ` Andreas Schwab
2010-10-01 17:19 ` Jan Djärv
2010-10-01 22:45 ` Stefan Monnier
2010-10-01 7:13 ` Eli Zaretskii
2010-10-01 9:28 ` Lennart Borgman
2010-10-01 10:35 ` Mathias Dahl
2010-10-01 11:24 ` Thierry Volpiatto
2010-10-01 11:26 ` Lennart Borgman
2010-10-01 12:02 ` Eli Zaretskii
2010-10-01 15:03 ` Lennart Borgman
2010-10-01 15:11 ` PJ Weisberg
2010-10-01 22:40 ` Stefan Monnier
2010-10-01 23:35 ` Miles Bader
2010-10-02 0:33 ` Lennart Borgman
2010-10-02 7:29 ` Eli Zaretskii
2010-10-02 10:12 ` Lennart Borgman
2010-10-02 10:32 ` Thierry Volpiatto
2010-10-02 10:50 ` Lennart Borgman
2010-10-02 12:01 ` Eli Zaretskii
2010-10-02 14:51 ` Lennart Borgman
2010-10-02 15:05 ` Thierry Volpiatto
2010-10-02 15:23 ` Eli Zaretskii
2010-10-02 15:56 ` Lennart Borgman
2010-10-02 20:40 ` Mathias Dahl
2010-10-02 22:20 ` Eli Zaretskii
2010-10-03 0:35 ` David Robinow
2010-10-03 4:05 ` Eli Zaretskii
2010-10-03 4:29 ` David Robinow
2010-10-03 5:39 ` Eli Zaretskii
2010-10-03 7:25 ` Thierry Volpiatto
2010-10-03 4:10 ` Lennart Borgman
2010-10-03 10:39 ` Thierry Volpiatto
2010-10-03 13:23 ` Thierry Volpiatto
2010-10-03 19:09 ` Eli Zaretskii
2010-10-03 22:50 ` Lennart Borgman
2010-10-05 0:46 ` Lennart Borgman
2010-10-05 0:51 ` Lennart Borgman
2010-10-05 0:56 ` Juanma Barranquero
2010-10-05 1:36 ` Lennart Borgman
2010-10-05 1:37 ` Lennart Borgman
2010-10-05 1:54 ` Juanma Barranquero
2010-10-02 11:59 ` Eli Zaretskii
2010-10-02 12:44 ` Lennart Borgman
2010-10-02 13:15 ` Eli Zaretskii
2010-10-01 7:09 ` Eli Zaretskii
2010-10-01 4:00 ` Stephen J. Turnbull
2010-10-01 7:03 ` Eli Zaretskii
2010-10-01 7:29 ` Eli Zaretskii
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).