What does Emacs on w32 know that grep can't figure out?

What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

I saved to a file with CCleaner the list of installed programs on my
pc. This file is maybe a text file, maybe not. Emacs can read it and
think it is a text file (I believe) with

  buffer-file-coding-system is a variable defined in `C source code'.
  Its value is
  utf-16le-with-signature-unix
  Local in buffer install.txt;

However trying to search this file from a cmd prompt with (gnuwin32)
grep does not work. And it does not work with cygwin grep either. They
think it is a binary file (even though I changed the line delimiter to
unix style).

What is going on? Is grep sometimes useless on w32 now, or? (How do we
handle that in Emacs?)

Re: What does Emacs on w32 know that grep can't figure out?

[Juanma Barranquero]

> This file is maybe a text file, maybe not.

It's a text file, but not ASCII. It's encoded as Emacs is telling you:
utf-16, with signature, in little-endian order (it starts with FF FE).

>  Its value is
>  utf-16le-with-signature-unix

Curious. I get utf-16le-with-signature-dos, consistent with the file
having CRLF (which it does).

> However trying to search this file from a cmd prompt with (gnuwin32)
> grep does not work. And it does not work with cygwin grep either. They
> think it is a binary file

It is. This is unrelated to Emacs. The grep tools you're using do not
support utf-16 text.

> (even though I changed the line delimiter to unix style).

Ah, that explains the -unix above.

> What is going on?

I hope this is clearer now.

> Is grep sometimes useless on w32 now, or?

It will be useless whenever you try to grep a file in a coding system
the grep tool does not support...

> (How do we
> handle that in Emacs?)

How do we handle what? If you mean, how can Emacs read the text just
fine... Emacs has supported UTF-16 and many other coding systems for
years.

As for what can you do,

C-x <RET> f iso-latin-1-unix <RET>

then save.

    Juanma

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Fri, Oct 1, 2010 at 1:58 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>
>> However trying to search this file from a cmd prompt with (gnuwin32)
>> grep does not work. And it does not work with cygwin grep either. They
>> think it is a binary file
>
> It is. This is unrelated to Emacs. The grep tools you're using do not
> support utf-16 text.
>
>> Is grep sometimes useless on w32 now, or?
>
> It will be useless whenever you try to grep a file in a coding system
> the grep tool does not support...


Yes, of course, that is actually what I am saying...

I actually think that there is no grep etc that handles this kind of
files on w32. Is there on *nix?


>> (How do we
>> handle that in Emacs?)
>
> How do we handle what? If you mean, how can Emacs read the text just
> fine... Emacs has supported UTF-16 and many other coding systems for
> years.

Using grep is an integral part of Emacs. If grep does not support the
coding then that part of Emacs will fail.


> As for what can you do,
>
> C-x <RET> f iso-latin-1-unix <RET>
>
> then save.

Yes, but that does not help if you have a lot of files you want to search.

Re: What does Emacs on w32 know that grep can't figure out?

[Juanma Barranquero]

On Fri, Oct 1, 2010 at 02:29, Lennart Borgman <lennart.borgman@gmail.com> wrote:

> I actually think that there is no grep etc that handles this kind of
> files on w32. Is there on *nix?

I don't know. I don't use *nix (though I sometimes use GNU/*ux, but I
haven't needed to grep utf-16 files).

> Using grep is an integral part of Emacs.

Relatively speaking. In thirteen years of Emacs use I've grepped from
inside Emacs perhaps twice.

> If grep does not support the
> coding then that part of Emacs will fail.

Absurd. It's up to you to look for a grep that will grok the files you
want to search. grep is not included with Emacs.

> Yes, but that does not help if you have a lot of files you want to search.

Bad luck. You can always use an elisp script to convert them all.

    Juanma

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Fri, Oct 1, 2010 at 2:37 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 02:29, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>
>> I actually think that there is no grep etc that handles this kind of
>> files on w32. Is there on *nix?
>
> I don't know. I don't use *nix (though I sometimes use GNU/*ux, but I
> haven't needed to grep utf-16 files).
>
>> Using grep is an integral part of Emacs.
>
> Relatively speaking. In thirteen years of Emacs use I've grepped from
> inside Emacs perhaps twice.


In those a bit less years I have used Emacs regularly now I guess I
might have used Emacs two times without using grep from within Emacs
;-)


>> If grep does not support the
>> coding then that part of Emacs will fail.
>
> Absurd. It's up to you to look for a grep that will grok the files you
> want to search. grep is not included with Emacs.


If the tool chain is broken then the tools that depends on it are also
broken. Aren't they?

(And commands two use Emacs are actually included in the Emacs manual!)


>> Yes, but that does not help if you have a lot of files you want to search.
>
> Bad luck. You can always use an elisp script to convert them all.


Yes, but that is a quite unfortunate state IMO. New users will just
say "ah, this tool is broken, let us try something else".

And I think they are right if we can not cure a problem such as this.

Re: What does Emacs on w32 know that grep can't figure out?

[Juanma Barranquero]

On Fri, Oct 1, 2010 at 03:06, Lennart Borgman <lennart.borgman@gmail.com> wrote:

> If the tool chain is broken then the tools that depends on it are also
> broken. Aren't they?

Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
from inside it. YMMV.

> Yes, but that is a quite unfortunate state IMO. New users will just
> say "ah, this tool is broken, let us try something else".

And what will they use?

    Juanma

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Fri, Oct 1, 2010 at 3:20 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 03:06, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>
>> If the tool chain is broken then the tools that depends on it are also
>> broken. Aren't they?
>
> Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
> from inside it. YMMV.

I am a bit surprised by that.

>> Yes, but that is a quite unfortunate state IMO. New users will just
>> say "ah, this tool is broken, let us try something else".
>
> And what will they use?

Some tool with an internal search capable of handling utf-16 files?

And are not most of the things to handle that from within Emacs
actually in place? Though of course putting files into buffers to
search them are out of the question for performance reasons.

Re: What does Emacs on w32 know that grep can't figure out?

[Juanma Barranquero]

On Fri, Oct 1, 2010 at 03:32, Lennart Borgman <lennart.borgman@gmail.com> wrote:

> I am a bit surprised by that.

Don't know why. As I said, I don't use grep from Emacs.

> Some tool with an internal search capable of handling utf-16 files?

Such as?

> And are not most of the things to handle that from within Emacs
> actually in place?

What do you mean, exactly? If you're trying to grep an external file,
with an external tool, it's hardly an Emacs problem. You could work
around it by loading the files into buffers and using Emacs
functionality (with you reject), or using Emacs to convert the files
and search with the external tool (which you reject), but other than
that, how do you expect Emacs to help?

    Juanma

Re: What does Emacs on w32 know that grep can't figure out?

[PJ Weisberg]

On 9/30/10, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 3:20 AM, Juanma Barranquero <lekktu@gmail.com> wrote:

>> Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
>> from inside it. YMMV.
>
> I am a bit surprised by that.

Well, the "grep through files" function is semi-broken, but then
again, the "send email" function is broken if your mail server is
down.  ::shrug::

> And are not most of the things to handle that from within Emacs
> actually in place? Though of course putting files into buffers to
> search them are out of the question for performance reasons.

If you take a look at the grep function in Emacs
 you'll see that all it really does is call a shell command and
collect the output.  So all the stuff that's "already in place"
probably would require loading the text into a buffer.

What does Emacs on w32 know that grep can't figure out?

[Stephen J. Turnbull]

Lennart Borgman writes:

 > However trying to search this file from a cmd prompt with (gnuwin32)
 > grep does not work.

No, it almost certainly won't.  grep is byte-oriented and doesn't know
anything about coding systems.  On Unix with a UTF-8-capable terminal
you would do something like

   iconv --from=UTF-16 --to=UTF-8 $FILE | grep some-string

I would think that either Cygwin or Windows provides a version of
iconv.  If not, changing the file to UTF-8 (instead of UTF-16) using
Emacs should make it grep'able.  In some cases grep may think this is
a binary file anyway; if so, use the --text switch to force grep to
treat the file as text.

 > And it does not work with cygwin grep either. They think it is a
 > binary file (even though I changed the line delimiter to unix
 > style).

The EOL delimiter is not a problem.  grep should ignore the presence
or absence of CR when checking for binary files.  The only time it is
likely to matter is if you are searching for a word at the end of the
line, in which case instead of "word$" you can use "word\015?$" or
something like that (if it matters, grep may be EOL-agnostic these
days).

Now, of course they think a UTF-16-encoded file is a binary file.  It
almost certainly contains NUL bytes (because an ASCII or Latin-1
character will always have a trailing NUL in UTF-16LE).

 > What is going on? Is grep sometimes useless on w32 now, or? (How do we
 > handle that in Emacs?)

Emacs tries to guess what the encoding is if you don't specify it.  It
may guess wrong in certain cases, but it should be extremely accurate
in case of any Unicode format.

Re: What does Emacs on w32 know that grep can't figure out?

[Jan D.]

On 2010-10-01 02:37, Juanma Barranquero wrote:
> On Fri, Oct 1, 2010 at 02:29, Lennart Borgman<lennart.borgman@gmail.com>  wrote:
>
>> I actually think that there is no grep etc that handles this kind of
>> files on w32. Is there on *nix?
>
> I don't know. I don't use *nix (though I sometimes use GNU/*ux, but I
> haven't needed to grep utf-16 files).
>

This is one of many reasons why *nix-like OSes uses UTF-8.  Tools that 
only understand ASCII can still work on ASCII.  GNU grep understands 
UTF-8 I think.

	Jan D.

Re: What does Emacs on w32 know that grep can't figure out?

[PJ Weisberg]

On Thu, Sep 30, 2010 at 7:41 PM, PJ Weisberg
<pj@irregularexpressions.net> wrote:
> On 9/30/10, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>> On Fri, Oct 1, 2010 at 3:20 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>
>>> Oh, I wouldn't consider Emacs broken if I cannot grep utf-16 files
>>> from inside it. YMMV.
>>
>> I am a bit surprised by that.
>
> Well, the "grep through files" function is semi-broken, but then
> again, the "send email" function is broken if your mail server is
> down.  ::shrug::

This issue deserves more respect than it's been getting, though.  It's
perfectly reasonable to expect a utility for searching through text to
find text in a file containing text.

Emacs delegates that task to grep, and if it doesn't work properly
that should be addressed in grep.  The percentage of grep users who
use it through Emacs is tiny, so assuming for the moment that everyone
agrees that grep should support UTF-16, addressing it in emacs instead
of grep would mean the problem is still mostly unfixed.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 01 Oct 2010 13:00:02 +0900
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
> 
>  > What is going on? Is grep sometimes useless on w32 now, or? (How do we
>  > handle that in Emacs?)
> 
> Emacs tries to guess what the encoding is if you don't specify it.  It
> may guess wrong in certain cases, but it should be extremely accurate
> in case of any Unicode format.

Especially with UTF-16 that always has a BOM signature.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 1 Oct 2010 01:58:38 +0200
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
> 
> > Is grep sometimes useless on w32 now, or?
> 
> It will be useless whenever you try to grep a file in a coding system
> the grep tool does not support...

Grep treats its input as a stream of bytes.  So it will not be able to
support any encoding where each character, including ASCII, takes 2 or
more bytes.  Unless you also encode the search string in the same
encoding, that is.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Fri, 1 Oct 2010 02:29:35 +0200
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
> 
> > How do we handle what? If you mean, how can Emacs read the text just
> > fine... Emacs has supported UTF-16 and many other coding systems for
> > years.
> 
> Using grep is an integral part of Emacs. If grep does not support the
> coding then that part of Emacs will fail.

What's wrong with "M-x occur" and/or "M-x multi-occur"?

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 01 Oct 2010 13:00:02 +0900
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
> 
>  > And it does not work with cygwin grep either. They think it is a
>  > binary file (even though I changed the line delimiter to unix
>  > style).
> 
> The EOL delimiter is not a problem.

Certainly not.  Grep says a file is binary only if it contains null
bytes.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> Date: Fri, 01 Oct 2010 07:37:59 +0200
> From: "Jan D." <jan.h.d@swipnet.se>
> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
> 	Emacs-Devel devel <emacs-devel@gnu.org>
> 
> GNU grep understands UTF-8 I think.

Not really.  Grep just compares bytes one by one, it knows nothing
about multibyte characters.

Re: What does Emacs on w32 know that grep can't figure out?

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 01 Oct 2010 07:37:59 +0200
>> From: "Jan D." <jan.h.d@swipnet.se>
>> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
>> 	Emacs-Devel devel <emacs-devel@gnu.org>
>> 
>> GNU grep understands UTF-8 I think.
>
> Not really.  Grep just compares bytes one by one, it knows nothing
> about multibyte characters.

grep uses exactly the character encoding as specified by the locale.
But it does not support wide character encodings, only multibyte ones.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>
>> > How do we handle what? If you mean, how can Emacs read the text just
>> > fine... Emacs has supported UTF-16 and many other coding systems for
>> > years.
>>
>> Using grep is an integral part of Emacs. If grep does not support the
>> coding then that part of Emacs will fail.
>
> What's wrong with "M-x occur" and/or "M-x multi-occur"?

They does not do what grep does.

And as I said opening the files in a buffer will be bad because of performance.

However using the internal regexp engine for searching the files would
be useful, but I have no idea how complicated it is to implement
something like that.

Re: What does Emacs on w32 know that grep can't figure out?

[Mathias Dahl]

I often use `% g' in Dired for finding files with a certain text in
them. Together with recursive Dired listings this often solves my
needs (although it is slower than using grep.)

/Mathias

On Fri, Oct 1, 2010 at 11:28 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>>
>>> > How do we handle what? If you mean, how can Emacs read the text just
>>> > fine... Emacs has supported UTF-16 and many other coding systems for
>>> > years.
>>>
>>> Using grep is an integral part of Emacs. If grep does not support the
>>> coding then that part of Emacs will fail.
>>
>> What's wrong with "M-x occur" and/or "M-x multi-occur"?
>
> They does not do what grep does.
>
> And as I said opening the files in a buffer will be bad because of performance.
>
> However using the internal regexp engine for searching the files would
> be useful, but I have no idea how complicated it is to implement
> something like that.
>
>

Re: What does Emacs on w32 know that grep can't figure out?

[Thierry Volpiatto]

Have a look also at ioccur.el:
http://mercurial.intuxication.org/hg/ioccur

M-x ioccur
M-x ioccur-dired
M-x ioccur-find-buffer-matching

Mathias Dahl <mathias.dahl@gmail.com> writes:

> I often use `% g' in Dired for finding files with a certain text in
> them. Together with recursive Dired listings this often solves my
> needs (although it is slower than using grep.)
>
> /Mathias
>
> On Fri, Oct 1, 2010 at 11:28 AM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>>>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>>>
>>>> > How do we handle what? If you mean, how can Emacs read the text just
>>>> > fine... Emacs has supported UTF-16 and many other coding systems for
>>>> > years.
>>>>
>>>> Using grep is an integral part of Emacs. If grep does not support the
>>>> coding then that part of Emacs will fail.
>>>
>>> What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>>
>> And as I said opening the files in a buffer will be bad because of performance.
>>
>> However using the internal regexp engine for searching the files would
>> be useful, but I have no idea how complicated it is to implement
>> something like that.
>>
>>
>
>

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

Thanks Mathias, that kind of worked, but I get no *grep* buffer.

Beside that it opened Wink for me. Looking at the code I can't see
that it could do that - but it does. Anyone understand what
happened??? (Wink is a w32 program for making screencasts. Very good,
but unfortunately not GPL. I have used it for nXhtml screencasts.)


On Fri, Oct 1, 2010 at 12:35 PM, Mathias Dahl <mathias.dahl@gmail.com> wrote:
> I often use `% g' in Dired for finding files with a certain text in
> them. Together with recursive Dired listings this often solves my
> needs (although it is slower than using grep.)
>
> /Mathias
>
> On Fri, Oct 1, 2010 at 11:28 AM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Fri, Oct 1, 2010 at 9:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Fri, 1 Oct 2010 02:29:35 +0200
>>>> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
>>>>
>>>> > How do we handle what? If you mean, how can Emacs read the text just
>>>> > fine... Emacs has supported UTF-16 and many other coding systems for
>>>> > years.
>>>>
>>>> Using grep is an integral part of Emacs. If grep does not support the
>>>> coding then that part of Emacs will fail.
>>>
>>> What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>>
>> And as I said opening the files in a buffer will be bad because of performance.
>>
>> However using the internal regexp engine for searching the files would
>> be useful, but I have no idea how complicated it is to implement
>> something like that.
>>
>>
>

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: "Jan D." <jan.h.d@swipnet.se>,  lekktu@gmail.com,  lennart.borgman@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 01 Oct 2010 10:12:08 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Date: Fri, 01 Oct 2010 07:37:59 +0200
> >> From: "Jan D." <jan.h.d@swipnet.se>
> >> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
> >> 	Emacs-Devel devel <emacs-devel@gnu.org>
> >> 
> >> GNU grep understands UTF-8 I think.
> >
> > Not really.  Grep just compares bytes one by one, it knows nothing
> > about multibyte characters.
> 
> grep uses exactly the character encoding as specified by the locale.

What do you mean by "support"?  It compares text by comparing its
constituent bytes, without really understanding that a given sequence
of bytes is a single character.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Fri, 1 Oct 2010 11:28:49 +0200
> Cc: lekktu@gmail.com, emacs-devel@gnu.org
> 
> > What's wrong with "M-x occur" and/or "M-x multi-occur"?
> 
> They does not do what grep does.

Like what?

> And as I said opening the files in a buffer will be bad because of performance.

What do you think Grep does?  Doesn't it open the files and read them?

> However using the internal regexp engine for searching the files would
> be useful, but I have no idea how complicated it is to implement
> something like that.

Emacs does similar things in quite a few places already.

Re: What does Emacs on w32 know that grep can't figure out?

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: "Jan D." <jan.h.d@swipnet.se>,  lekktu@gmail.com,  lennart.borgman@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 01 Oct 2010 10:12:08 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Date: Fri, 01 Oct 2010 07:37:59 +0200
>> >> From: "Jan D." <jan.h.d@swipnet.se>
>> >> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
>> >> 	Emacs-Devel devel <emacs-devel@gnu.org>
>> >> 
>> >> GNU grep understands UTF-8 I think.
>> >
>> > Not really.  Grep just compares bytes one by one, it knows nothing
>> > about multibyte characters.
>> 
>> grep uses exactly the character encoding as specified by the locale.
>
> What do you mean by "support"?

There is no "support" in that sentence?

> It compares text by comparing its constituent bytes, without really
> understanding that a given sequence of bytes is a single character.

You are wrong.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Fri, Oct 1, 2010 at 2:02 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Fri, 1 Oct 2010 11:28:49 +0200
>> Cc: lekktu@gmail.com, emacs-devel@gnu.org
>>
>> > What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>
> Like what?


Look at all the files in a directory.


>> And as I said opening the files in a buffer will be bad because of performance.
>
> What do you think Grep does?  Doesn't it open the files and read them?


I think we have discussed before that putting the files in a buffer
would be a performance problem in a case like this, especially if the
file is big.

So I think something inside Emacs that could determine file coding but
did not put the file in a buffer is what I would want for this. Is
that hard to implement from what we have?


>> However using the internal regexp engine for searching the files would
>> be useful, but I have no idea how complicated it is to implement
>> something like that.
>
> Emacs does similar things in quite a few places already.


You mean similar to what I suggested above? (Or do you mean something
quite different, something on the elisp level?)

Re: What does Emacs on w32 know that grep can't figure out?

[PJ Weisberg]

On Fri, Oct 1, 2010 at 12:09 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> Grep treats its input as a stream of bytes.  So it will not be able to
> support any encoding where each character, including ASCII, takes 2 or
> more bytes.  Unless you also encode the search string in the same
> encoding, that is.

In 5 minutes of testing I see that grep (GNU grep 2.6.3) doesn't
handle null bytes in the pattern.  (String ends at the first null, as
c-strings are wont to do.)

On Fri, Oct 1, 2010 at 5:02 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Fri, 1 Oct 2010 11:28:49 +0200
>> Cc: lekktu@gmail.com, emacs-devel@gnu.org
>>
>> > What's wrong with "M-x occur" and/or "M-x multi-occur"?
>>
>> They does not do what grep does.
>
> Like what?

Recursively act on all files in a directory, for one.

>> And as I said opening the files in a buffer will be bad because of performance.
>
> What do you think Grep does?  Doesn't it open the files and read them?

One line at a time.  That'll make a difference for large files.

Re: What does Emacs on w32 know that grep can't figure out?

[Jan Djärv]

1 okt 2010 kl. 14:00 skrev Eli Zaretskii <eliz@gnu.org>:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: "Jan D." <jan.h.d@swipnet.se>,  lekktu@gmail.com,  lennart.borgman@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 01 Oct 2010 10:12:08 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>>> Date: Fri, 01 Oct 2010 07:37:59 +0200
>>>> From: "Jan D." <jan.h.d@swipnet.se>
>>>> Cc: Lennart Borgman <lennart.borgman@gmail.com>,
>>>>    Emacs-Devel devel <emacs-devel@gnu.org>
>>>> 
>>>> GNU grep understands UTF-8 I think.
>>> 
>>> Not really.  Grep just compares bytes one by one, it knows nothing
>>> about multibyte characters.
>> 
>> grep uses exactly the character encoding as specified by the locale.
> 
> What do you mean by "support"?  It compares text by comparing its
> constituent bytes, without really understanding that a given sequence
> of bytes is a single character.

Grep knows about character classes, like [:alpha:] and [:lower:]. But I guess most of that comes from the locale support in libc.

    Jan D.

Re: What does Emacs on w32 know that grep can't figure out?

[Stefan Monnier]

> I actually think that there is no grep etc that handles this kind of
> files on w32. Is there on *nix?

UTF-16 is the problem, and it (basically) doesn't exist on non-w32
platforms.


        Stefan

Re: What does Emacs on w32 know that grep can't figure out?

[Stefan Monnier]

> What do you mean by "support"?  It compares text by comparing its
> constituent bytes, without really understanding that a given sequence
> of bytes is a single character.

A little test shows that "grep foo.bar ~/tmp/foobar"
finds in foobar the line "fooébar" even though that é uses
several bytes (I'm using a utf-8 locale).

Similarly "grep 'é*' toto" and "grep '[éa]' toto" will correctly treat
the several bytes in é as a single element.


        Stefan

Re: What does Emacs on w32 know that grep can't figure out?

[Miles Bader]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I actually think that there is no grep etc that handles this kind of
>> files on w32. Is there on *nix?
>
> UTF-16 is the problem, and it (basically) doesn't exist on non-w32
> platforms.

Yup.

Though it seems MS is trying to fool/coerce the rest of the world into
greater adoption of that abortion... (e.g. the C++0x standard, where
u"xxx" now means UTF-16 text...)

-Miles

-- 
The trouble with most people is that they think with their hopes or
fears or wishes rather than with their minds.  -- Will Durant

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sat, Oct 2, 2010 at 1:35 AM, Miles Bader <miles@gnu.org> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I actually think that there is no grep etc that handles this kind of
>>> files on w32. Is there on *nix?
>>
>> UTF-16 is the problem, and it (basically) doesn't exist on non-w32
>> platforms.
>
> Yup.
>
> Though it seems MS is trying to fool/coerce the rest of the world into
> greater adoption of that abortion... (e.g. the C++0x standard, where
> u"xxx" now means UTF-16 text...)


So where is the problem? Is utf-16 actually used on w32 in many apps?
Or is this a left-over in CCleaner?

If utf-16 is actually used is this there then a policy to not support
it in grep or is it a lack of resources? (Whatever the case it makes
grep a pretty unreliable tool.)

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 2 Oct 2010 02:33:32 +0200
> Cc: emacs-devel@gnu.org
> 
> So where is the problem? Is utf-16 actually used on w32 in many apps?
> Or is this a left-over in CCleaner?

UTF-16 is used for all the Unicode APIs on Windows.  That's the
implementation of the wchar_t type on Windows.  In particular, all the
file names in the NTFS filesystem are actually stored as UTF-16, and
all the OS databases in memory are stored in UTF-16 format.

So this is not a left-over, this is what Windows uses internally.

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>> Cc: emacs-devel@gnu.org
>>
>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>> Or is this a left-over in CCleaner?
>
> UTF-16 is used for all the Unicode APIs on Windows.  That's the
> implementation of the wchar_t type on Windows.  In particular, all the
> file names in the NTFS filesystem are actually stored as UTF-16, and
> all the OS databases in memory are stored in UTF-16 format.
>
> So this is not a left-over, this is what Windows uses internally.


Thanks. So then the question is why the grep program does not support
it. Is this a policy question, a lack of resources, or something else?

Re: What does Emacs on w32 know that grep can't figure out?

[Thierry Volpiatto]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>> Cc: emacs-devel@gnu.org
>>>
>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>> Or is this a left-over in CCleaner?
>>
>> UTF-16 is used for all the Unicode APIs on Windows.  That's the
>> implementation of the wchar_t type on Windows.  In particular, all the
>> file names in the NTFS filesystem are actually stored as UTF-16, and
>> all the OS databases in memory are stored in UTF-16 format.
>>
>> So this is not a left-over, this is what Windows uses internally.
>
>
> Thanks. So then the question is why the grep program does not support
> it. Is this a policy question, a lack of resources, or something else?
>
>
It seem most modern windows versions come with powershell.
Why windows users don't use it as backend for Emacs instead of using
external Gnuwin32 programs?

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sat, Oct 2, 2010 at 12:32 PM, Thierry Volpiatto
<thierry.volpiatto@gmail.com> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>>> Or is this a left-over in CCleaner?
>>>
>>> UTF-16 is used for all the Unicode APIs on Windows.  That's the
>>> implementation of the wchar_t type on Windows.  In particular, all the
>>> file names in the NTFS filesystem are actually stored as UTF-16, and
>>> all the OS databases in memory are stored in UTF-16 format.
>>>
>>> So this is not a left-over, this is what Windows uses internally.
>>
>>
>> Thanks. So then the question is why the grep program does not support
>> it. Is this a policy question, a lack of resources, or something else?
>>
>>
> It seem most modern windows versions come with powershell.
> Why windows users don't use it as backend for Emacs instead of using
> external Gnuwin32 programs?


Are you effectively not saying that because of the lack of utf-16
support in gnu grep it would be better for Emacs to depend on non-gnu
and even ms tools on w32?

I would say: perhaps. What do others think?

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 2 Oct 2010 12:12:08 +0200
> Cc: emacs-devel@gnu.org, miles@gnu.org
> 
> Thanks. So then the question is why the grep program does not support
> it.

Because no one has coded that in Grep, I'd guess.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Sat, 02 Oct 2010 12:32:16 +0200
> 
> It seem most modern windows versions come with powershell.
> Why windows users don't use it as backend for Emacs instead of using
> external Gnuwin32 programs?

How would that help, exactly?

When PowerShell came out, I tried it, and found that it had its own
quirks, not surprisingly.  So we would be replacing one bunch of
problems with another.  At least the current tools are widely known
and well understood, and we have much higher chances getting changes
into tools like Grep than into PowerShell.

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sat, Oct 2, 2010 at 1:59 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 2 Oct 2010 12:12:08 +0200
>> Cc: emacs-devel@gnu.org, miles@gnu.org
>>
>> Thanks. So then the question is why the grep program does not support
>> it.
>
> Because no one has coded that in Grep, I'd guess.


Is that really the whole answer? If utf-16 is so widely used on w32
then is it not surprising that it is not understood by Grep?

Maybe you are right, but it would be interesting to get some more
insight in this.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 2 Oct 2010 14:44:31 +0200
> Cc: miles@gnu.org, emacs-devel@gnu.org
> 
> On Sat, Oct 2, 2010 at 1:59 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Lennart Borgman <lennart.borgman@gmail.com>
> >> Date: Sat, 2 Oct 2010 12:12:08 +0200
> >> Cc: emacs-devel@gnu.org, miles@gnu.org
> >>
> >> Thanks. So then the question is why the grep program does not support
> >> it.
> >
> > Because no one has coded that in Grep, I'd guess.
> 
> 
> Is that really the whole answer? If utf-16 is so widely used on w32
> then is it not surprising that it is not understood by Grep?

Look at the ChangeLog in Grep distribution: there are exactly zero
contributors of Windows related code there.  About the only entry that
mentions Windows is a minor bug fixed 6 months ago.  And that's it.

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sat, Oct 2, 2010 at 12:32 PM, Thierry Volpiatto
<thierry.volpiatto@gmail.com> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>>> Or is this a left-over in CCleaner?
>>>
>>> UTF-16 is used for all the Unicode APIs on Windows.  That's the
>>> implementation of the wchar_t type on Windows.  In particular, all the
>>> file names in the NTFS filesystem are actually stored as UTF-16, and
>>> all the OS databases in memory are stored in UTF-16 format.
>>>
>>> So this is not a left-over, this is what Windows uses internally.
>>
>>
>> Thanks. So then the question is why the grep program does not support
>> it. Is this a policy question, a lack of resources, or something else?
>>
>>
> It seem most modern windows versions come with powershell.
> Why windows users don't use it as backend for Emacs instead of using
> external Gnuwin32 programs?


Based on your advice I just took the trouble to install powershell. I
can not see that it addresses the problem with utf-16, but maybe I am
missing something. Can you please tell me how to search also utf-16
files with it?

Re: What does Emacs on w32 know that grep can't figure out?

[Thierry Volpiatto]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Sat, Oct 2, 2010 at 12:32 PM, Thierry Volpiatto
> <thierry.volpiatto@gmail.com> wrote:
>> Lennart Borgman <lennart.borgman@gmail.com> writes:
>>
>>> On Sat, Oct 2, 2010 at 9:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>>> Date: Sat, 2 Oct 2010 02:33:32 +0200
>>>>> Cc: emacs-devel@gnu.org
>>>>>
>>>>> So where is the problem? Is utf-16 actually used on w32 in many apps?
>>>>> Or is this a left-over in CCleaner?
>>>>
>>>> UTF-16 is used for all the Unicode APIs on Windows.  That's the
>>>> implementation of the wchar_t type on Windows.  In particular, all the
>>>> file names in the NTFS filesystem are actually stored as UTF-16, and
>>>> all the OS databases in memory are stored in UTF-16 format.
>>>>
>>>> So this is not a left-over, this is what Windows uses internally.
>>>
>>>
>>> Thanks. So then the question is why the grep program does not support
>>> it. Is this a policy question, a lack of resources, or something else?
>>>
>>>
>> It seem most modern windows versions come with powershell.
>> Why windows users don't use it as backend for Emacs instead of using
>> external Gnuwin32 programs?
>
>
> Based on your advice I just took the trouble to install powershell. I
> can not see that it addresses the problem with utf-16, but maybe I am
> missing something. Can you please tell me how to search also utf-16
> files with it?

That's just an idea i sent, i personally don't use (or really few
sometimes on other machines) windows systems and i have just a basic
knowledge of all that is related to it and microsoft in general (e.g Powershell).
So i can't help you much here.
Anyway, Eli seem to have tried it and is not convinced.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Sat, 02 Oct 2010 17:05:18 +0200
> Cc: emacs-devel@gnu.org
> 
> > Based on your advice I just took the trouble to install powershell. I
> > can not see that it addresses the problem with utf-16, but maybe I am
> > missing something. Can you please tell me how to search also utf-16
> > files with it?
> 
> That's just an idea i sent, i personally don't use (or really few
> sometimes on other machines) windows systems and i have just a basic
> knowledge of all that is related to it and microsoft in general (e.g Powershell).
> So i can't help you much here.
> Anyway, Eli seem to have tried it and is not convinced.

Looks like the PowerShell cmdlet that's supposed to be equivalent of
Grep is Select-String.

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>> Cc: emacs-devel@gnu.org
>>
>> > Based on your advice I just took the trouble to install powershell. I
>> > can not see that it addresses the problem with utf-16, but maybe I am
>> > missing something. Can you please tell me how to search also utf-16
>> > files with it?
>>
>> That's just an idea i sent, i personally don't use (or really few
>> sometimes on other machines) windows systems and i have just a basic
>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>> So i can't help you much here.
>> Anyway, Eli seem to have tried it and is not convinced.
>
> Looks like the PowerShell cmdlet that's supposed to be equivalent of
> Grep is Select-String.

Yes, thanks. And that actually works with utf-16.

So now we have a strange situation.

Re: What does Emacs on w32 know that grep can't figure out?

[Mathias Dahl]

If Windows' built-in find command supports UTF-16, then that might be
an option (maybe better since I think find will be installed on more
Windows PCs) too:

http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true

It sounds like it does at least parts of what grep does. Try it out and see.

/Mathias

On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>> Cc: emacs-devel@gnu.org
>>>
>>> > Based on your advice I just took the trouble to install powershell. I
>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>> > missing something. Can you please tell me how to search also utf-16
>>> > files with it?
>>>
>>> That's just an idea i sent, i personally don't use (or really few
>>> sometimes on other machines) windows systems and i have just a basic
>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>> So i can't help you much here.
>>> Anyway, Eli seem to have tried it and is not convinced.
>>
>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>> Grep is Select-String.
>
> Yes, thanks. And that actually works with utf-16.
>
> So now we have a strange situation.
>
>

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Mathias Dahl <mathias.dahl@gmail.com>
> Date: Sat, 2 Oct 2010 22:40:15 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org, 
> 	Thierry Volpiatto <thierry.volpiatto@gmail.com>
> 
> If Windows' built-in find command supports UTF-16, then that might be
> an option (maybe better since I think find will be installed on more
> Windows PCs) too:
> 
> http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true

Somebody should first make sure FIND supports Unicode strings.  Most
of the console programs don't.

Re: What does Emacs on w32 know that grep can't figure out?

[David Robinow]

On Sat, Oct 2, 2010 at 6:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Mathias Dahl <mathias.dahl@gmail.com>
>> Date: Sat, 2 Oct 2010 22:40:15 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>>       Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>
>> If Windows' built-in find command supports UTF-16, then that might be
>> an option (maybe better since I think find will be installed on more
>> Windows PCs) too:
>>
>> http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true
>
> Somebody should first make sure FIND supports Unicode strings.  Most
> of the console programs don't.
>
It appears to (Vista Home Premium), as do TYPE and MORE. Which ones don''t?

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> Date: Sat, 2 Oct 2010 20:35:30 -0400
> From: David Robinow <drobinow@gmail.com>
> Cc: Mathias Dahl <mathias.dahl@gmail.com>, thierry.volpiatto@gmail.com, 
> 	lennart.borgman@gmail.com, emacs-devel@gnu.org
> 
> On Sat, Oct 2, 2010 at 6:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Mathias Dahl <mathias.dahl@gmail.com>
> >> Date: Sat, 2 Oct 2010 22:40:15 +0200
> >> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
> >>       Thierry Volpiatto <thierry.volpiatto@gmail.com>
> >>
> >> If Windows' built-in find command supports UTF-16, then that might be
> >> an option (maybe better since I think find will be installed on more
> >> Windows PCs) too:
> >>
> >> http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/find.mspx?mfr=true
> >
> > Somebody should first make sure FIND supports Unicode strings.  Most
> > of the console programs don't.
> >
> It appears to (Vista Home Premium)

How did you test that?

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>> Cc: emacs-devel@gnu.org
>>>
>>> > Based on your advice I just took the trouble to install powershell. I
>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>> > missing something. Can you please tell me how to search also utf-16
>>> > files with it?
>>>
>>> That's just an idea i sent, i personally don't use (or really few
>>> sometimes on other machines) windows systems and i have just a basic
>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>> So i can't help you much here.
>>> Anyway, Eli seem to have tried it and is not convinced.
>>
>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>> Grep is Select-String.
>
> Yes, thanks. And that actually works with utf-16.
>
> So now we have a strange situation.


Or maybe not. Looking at the documentation for Select-String here

  http://technet.microsoft.com/en-us/library/dd315403.aspx

it does not look as it autodetects the coding system in the file.

It does not even say it can handle utf-16, or at least I can't find
it. Maybe someone who understands this better can see it there?

Re: What does Emacs on w32 know that grep can't figure out?

[David Robinow]

On Sun, Oct 3, 2010 at 12:05 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sat, 2 Oct 2010 20:35:30 -0400
>> From: David Robinow <drobinow@gmail.com>
>> Cc: Mathias Dahl <mathias.dahl@gmail.com>, thierry.volpiatto@gmail.com,
>>       lennart.borgman@gmail.com, emacs-devel@gnu.org
>>
>> On Sat, Oct 2, 2010 at 6:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> > Somebody should first make sure FIND supports Unicode strings.  Most
>> > of the console programs don't.
>> >
>> It appears to (Vista Home Premium)
>
> How did you test that?
 I found a UTF-16 file on my computer. Used 'TYPE' from a cmd window
to see the data.
Used cygwin 'file' to verify that it was UTF-16.
Then cygwin 'grep' which failed (I don't have a Windows version but
nobody claims it works anyway). Then 'FIND' from a cmd window (using
the same string which failed for grep) and it worked.
I will try Windows XP tomorrow.

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> Date: Sun, 3 Oct 2010 00:29:41 -0400
> From: David Robinow <drobinow@gmail.com>
> Cc: mathias.dahl@gmail.com, thierry.volpiatto@gmail.com, 
> 	lennart.borgman@gmail.com, emacs-devel@gnu.org
> 
> > How did you test that?
>  I found a UTF-16 file on my computer. Used 'TYPE' from a cmd window
> to see the data.

I'm not interested in TYPE, it's an internal command of CMD, not a
program.

> Used cygwin 'file' to verify that it was UTF-16.
> Then cygwin 'grep' which failed (I don't have a Windows version but
> nobody claims it works anyway). Then 'FIND' from a cmd window (using
> the same string which failed for grep) and it worked.

What string did you use with FIND, and (assuming it included non-ASCII
characters) how did you type it into the CMD window?

Re: What does Emacs on w32 know that grep can't figure out?

[Thierry Volpiatto]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 3 Oct 2010 00:29:41 -0400
>> From: David Robinow <drobinow@gmail.com>
>> Cc: mathias.dahl@gmail.com, thierry.volpiatto@gmail.com, 
>> 	lennart.borgman@gmail.com, emacs-devel@gnu.org
>> 
>> > How did you test that?
>>  I found a UTF-16 file on my computer. Used 'TYPE' from a cmd window
>> to see the data.
>
> I'm not interested in TYPE, it's an internal command of CMD, not a
> program.
>
>> Used cygwin 'file' to verify that it was UTF-16.
>> Then cygwin 'grep' which failed (I don't have a Windows version but
>> nobody claims it works anyway). Then 'FIND' from a cmd window (using
>> the same string which failed for grep) and it worked.
>
> What string did you use with FIND, and (assuming it included non-ASCII
> characters) how did you type it into the CMD window?
>
I have tried FINDSTR.exe from powershell, and it seem to work fine.
http://tinyurl.com/36vez2n

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: What does Emacs on w32 know that grep can't figure out?

[Thierry Volpiatto]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>> > Based on your advice I just took the trouble to install powershell. I
>>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>>> > missing something. Can you please tell me how to search also utf-16
>>>> > files with it?
>>>>
>>>> That's just an idea i sent, i personally don't use (or really few
>>>> sometimes on other machines) windows systems and i have just a basic
>>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>>> So i can't help you much here.
>>>> Anyway, Eli seem to have tried it and is not convinced.
>>>
>>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>>> Grep is Select-String.
>>
>> Yes, thanks. And that actually works with utf-16.
>>
>> So now we have a strange situation.
>
>
> Or maybe not. Looking at the documentation for Select-String here
>
>   http://technet.microsoft.com/en-us/library/dd315403.aspx
>
> it does not look as it autodetects the coding system in the file.
>
> It does not even say it can handle utf-16, or at least I can't find
> it. Maybe someone who understands this better can see it there?

Can you try to use FINDSTR from command line (it work from eshell) and
tell me if that work on your utf-16 files?
(It seem this command is not part of powershell)

I wrote some code that use this command with success on windows (with
anything).

That would allow to have a decent grep that work on windows without
having to install external programs.

Even if that seem not powerfull as grep for regexps, that provide a
default grep command better than "grep command not found". 

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: What does Emacs on w32 know that grep can't figure out?

[Thierry Volpiatto]

Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:

> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Sat, Oct 2, 2010 at 5:56 PM, Lennart Borgman
>> <lennart.borgman@gmail.com> wrote:
>>> On Sat, Oct 2, 2010 at 5:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>>>>> Date: Sat, 02 Oct 2010 17:05:18 +0200
>>>>> Cc: emacs-devel@gnu.org
>>>>>
>>>>> > Based on your advice I just took the trouble to install powershell. I
>>>>> > can not see that it addresses the problem with utf-16, but maybe I am
>>>>> > missing something. Can you please tell me how to search also utf-16
>>>>> > files with it?
>>>>>
>>>>> That's just an idea i sent, i personally don't use (or really few
>>>>> sometimes on other machines) windows systems and i have just a basic
>>>>> knowledge of all that is related to it and microsoft in general (e.g Powershell).
>>>>> So i can't help you much here.
>>>>> Anyway, Eli seem to have tried it and is not convinced.
>>>>
>>>> Looks like the PowerShell cmdlet that's supposed to be equivalent of
>>>> Grep is Select-String.
>>>
>>> Yes, thanks. And that actually works with utf-16.
>>>
>>> So now we have a strange situation.
>>
>>
>> Or maybe not. Looking at the documentation for Select-String here
>>
>>   http://technet.microsoft.com/en-us/library/dd315403.aspx
>>
>> it does not look as it autodetects the coding system in the file.
>>
>> It does not even say it can handle utf-16, or at least I can't find
>> it. Maybe someone who understands this better can see it there?
>
> Can you try to use FINDSTR from command line (it work from eshell) and
> tell me if that work on your utf-16 files?
> (It seem this command is not part of powershell)
>
> I wrote some code that use this command with success on windows (with
> anything).
It is now available on anything-config.el
Use M-x anything-do-grep

No tweak, work out of the box on GNU/Linux and windows as it is.
That's a first shot, feedback welcome.

> That would allow to have a decent grep that work on windows without
> having to install external programs.
>
> Even if that seem not powerfull as grep for regexps, that provide a
> default grep command better than "grep command not found". 

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: What does Emacs on w32 know that grep can't figure out?

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 3 Oct 2010 06:10:14 +0200
> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
> 
>   http://technet.microsoft.com/en-us/library/dd315403.aspx
> 
> it does not look as it autodetects the coding system in the file.

Why should it?  What tool does, besides Emacs?  You are supposed to
tell it the encoding using the -Encoding option.

> It does not even say it can handle utf-16, or at least I can't find
> it.

It does support UTF-16.  That's the "Unicode" part of the values you
can submit to the -Encoding option (you need to "think MS" to get it).

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Sun, Oct 3, 2010 at 9:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sun, 3 Oct 2010 06:10:14 +0200
>> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
>>
>>   http://technet.microsoft.com/en-us/library/dd315403.aspx
>>
>> it does not look as it autodetects the coding system in the file.
>
> Why should it?

I don't understand why you are asking. The answer is of course:
Because it would be useful when searching for words in many files (and
you do not know their coding system).

Or, don't you think so?

> What tool does, besides Emacs?

I am not sure, but I would expect at least some better search
engines/indexers to do this. I can't see how they can avoid it.

>> It does not even say it can handle utf-16, or at least I can't find
>> it.
>
> It does support UTF-16.  That's the "Unicode" part of the values you
> can submit to the -Encoding option (you need to "think MS" to get it).

Thanks.

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Mon, Oct 4, 2010 at 12:50 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sun, Oct 3, 2010 at 9:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sun, 3 Oct 2010 06:10:14 +0200
>>> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
>>>
>>>   http://technet.microsoft.com/en-us/library/dd315403.aspx
>>>
>>> it does not look as it autodetects the coding system in the file.
>>
>> Why should it?
..
>> What tool does, besides Emacs?
..
>> It does support UTF-16.  That's the "Unicode" part of the values you
>> can submit to the -Encoding option (you need to "think MS" to get it).

This discussion seems to have stalled. So here is a summary of what we
have found:

- utf-16 is a problem since the grep program does not handle it and
utf-16 is common on w32.

- There seems to be no program we can use that autodetects file coding
the way Emacs does.

- I would expect Emacs users to believe thata search from within Emacs
would autodetect coding system since Emacs does it. (Eli disagrees on
this.)

What can we do? The options I can see are:

- Use powershell + select-string (+ another cmdlet for dir tree
searching) on w32 by default. That has two benefits:

  1. Users on w32 does not have to install grep+find. But on older
systems they have to install powershell. Anyway it is easy to test for
powershell and use it if it is installed. (A problem is the hardcoding
of parameters to grep, like -i, but that can be resolved).

  2. Make an internal grep command in Emacs. A naive version in elisp
is quick to write but will be inefficient especially for large files.
A C version will require some restructuring of insert_file_contents,
but seems otherwise not hard to code. (Except perhaps for buffering
for effiency, but I have no idea if that is needed.)


I think 2 is the best choice, but I do not expect anyone to write it
now. Maybe add it to our to-do list?

For the moment I suggest that we implement 1.

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Tue, Oct 5, 2010 at 2:46 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Mon, Oct 4, 2010 at 12:50 AM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Sun, Oct 3, 2010 at 9:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>>> Date: Sun, 3 Oct 2010 06:10:14 +0200
>>>> Cc: Thierry Volpiatto <thierry.volpiatto@gmail.com>, emacs-devel@gnu.org
>>>>
>>>>   http://technet.microsoft.com/en-us/library/dd315403.aspx
>>>>
>>>> it does not look as it autodetects the coding system in the file.
>>>
>>> Why should it?
> ..
>>> What tool does, besides Emacs?
> ..
>>> It does support UTF-16.  That's the "Unicode" part of the values you
>>> can submit to the -Encoding option (you need to "think MS" to get it).
>
> This discussion seems to have stalled. So here is a summary of what we
> have found:
>
> - utf-16 is a problem since the grep program does not handle it and
> utf-16 is common on w32.
>
> - There seems to be no program we can use that autodetects file coding
> the way Emacs does.
>
> - I would expect Emacs users to believe thata search from within Emacs
> would autodetect coding system since Emacs does it. (Eli disagrees on
> this.)

A shit. Doing two things simultaneously does not improve you
writing... It should have been like this:

What can we do? The options I can see are:

A) Use powershell + select-string (+ another cmdlet for dir tree
searching) on w32 by default. That has two benefits:

  1. Users on w32 does not have to install grep+find. But on older
systems they have to install powershell. Anyway it is easy to test for
powershell and use it if it is installed. (A problem is the hardcoding
of parameters to grep, like -i, but that can be resolved).

  2. utf-16 files can also be searched on w32.

B) Make an internal grep command in Emacs. A naive version in elisp
is quick to write but will be inefficient especially for large files.
A C version will require some restructuring of insert_file_contents,
but seems otherwise not hard to code. (Except perhaps for buffering
for effiency, but I have no idea if that is needed.)


I think B is the best choice, but I do not expect anyone to write it
now. Maybe add it to our to-do list?

For the moment I suggest that we implement A.

Re: What does Emacs on w32 know that grep can't figure out?

[Juanma Barranquero]

On Tue, Oct 5, 2010 at 02:46, Lennart Borgman <lennart.borgman@gmail.com> wrote:

> - I would expect Emacs users to believe thata search from within Emacs
> would autodetect coding system since Emacs does it. (Eli disagrees on
> this.)

I agree with him, BTW.

> What can we do? The options I can see are:

C.- Do nothing, and leave to the user the task of finding a tool that
can grep utf-16.

D.- Do nothing, and leave to the user the task of converting the
utf-16 files to another coding, as utf-16 is the internal format of
32-bit Windows APIs, but it is rarely found in files, even on Windows.

    Juanma

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Tue, Oct 5, 2010 at 2:56 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>
>> What can we do? The options I can see are:
>
> C.- Do nothing, and leave to the user the task of finding a tool that
> can grep utf-16.
>
> D.- Do nothing, and leave to the user the task of converting the
> utf-16 files to another coding, as utf-16 is the internal format of
> 32-bit Windows APIs, but it is rarely found in files, even on Windows.

How would the user no which of C and D we have choosen?

If we decide on this (which seems likely now), then should we not
change default file coding system to utf-8? We kind of expect that,
but I do not understand what consequences changing the default file
coding system will have.

Re: What does Emacs on w32 know that grep can't figure out?

[Lennart Borgman]

On Tue, Oct 5, 2010 at 3:36 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Tue, Oct 5, 2010 at 2:56 AM, Juanma Barranquero <lekktu@gmail.com> wrote:
>>
>>> What can we do? The options I can see are:
>>
>> C.- Do nothing, and leave to the user the task of finding a tool that
>> can grep utf-16.
>>
>> D.- Do nothing, and leave to the user the task of converting the
>> utf-16 files to another coding, as utf-16 is the internal format of
>> 32-bit Windows APIs, but it is rarely found in files, even on Windows.
>
> How would the user no which of C and D we have choosen?

no => know ...

> If we decide on this (which seems likely now), then should we not
> change default file coding system to utf-8? We kind of expect that,
> but I do not understand what consequences changing the default file
> coding system will have.
>

Re: What does Emacs on w32 know that grep can't figure out?

[Juanma Barranquero]

On Tue, Oct 5, 2010 at 03:36, Lennart Borgman <lennart.borgman@gmail.com> wrote:

> If we decide on this (which seems likely now), then should we not
> change default file coding system to utf-8?

Which coding-system variable are you referring to? buffer-file-coding-system?

The language environment (and coding system variables) should reflex,
IMO, what the user is most likely to encounter.

I personally do

  (set-language-environment "UTF-8")
  (set-coding-system-priority 'utf-16le-with-signature 'utf-16be-with-signature)
  (setq file-name-coding-system 'windows-1252)

in my .emacs, but I want to generate UTF-8 files. I think most Windows
users will be happy with the current defaults.

    Juanma