auto-recognizing utf-16le ?

auto-recognizing utf-16le ?

[Miles Bader]

[-- Attachment #1: Type: text/plain, Size: 1358 bytes --]

Someone on #emacs noticed that emacs doesn't seem to auto-recognize
files encoding using utf-16le.  Visiting a file which uses such an
encoding results in the buffer having coding-system "no-conversion
(alias: binary)", and lots of ^@ (NUL) characters in the buffer.

Forcing the encoding with "C-x C-m r utf-16le RET" results in the
correct thing happening.

[He was on windows where this coding system is common, so it's kind of
annoying for him.]

I noticed that the same happens on debian.

I thought maybe he could just do:

   (prefer-coding-system 'utf-16le-dos)

but it seems to have no effect.

To reproduce:

   1. Save this message's attachment to a file "/tmp/oink"

   2.　Start emacs with:  HOME=/tmp emacs -Q

   3. Visit the file you saved:  C-x C-f /tmp/oink RET

   4. ** Notice that the buffer contains ^@ (NUL) characters, and that
      the buffer coding-system is "no-conversion (binary)"

   5. Re-visit the file, forcing the coding-system:

         C-x C-m r utf-16le RET yes RET

   6. ** Notice that the file contents are now correct

   7. Kill the current buffer:  C-x k RET

   8. Evaluate:  M-: (prefer-coding-system 'utf-16le) RET

   9. Visit the file again:  C-x C-f /tmp/oink RET

  10. ** Notice that prefer-coding-system didn't seem to have any effect


Thanks,

-Miles


[-- Attachment #2: test file encoded using utf-16le --]
[-- Type: application/octet-stream, Size: 30 bytes --]

[-- Attachment #3: Type: text/plain, Size: 167 bytes --]



-- 
Justice, n. A commodity which in a more or less adulterated condition the
State sells to the citizen as a reward for his allegiance, taxes and personal
service.

Re: auto-recognizing utf-16le ?

[Andreas Schwab]

Miles Bader <miles.bader@necel.com> writes:

> Someone on #emacs noticed that emacs doesn't seem to auto-recognize
> files encoding using utf-16le.

UTF-16 detection never tries to auto detect files without a signature.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: auto-recognizing utf-16le ?

[Miles Bader]

Andreas Schwab <schwab@linux-m68k.org> writes:
>> Someone on #emacs noticed that emacs doesn't seem to auto-recognize
>> files encoding using utf-16le.
>
> UTF-16 detection never tries to auto detect files without a signature.

So are UTF-16 files without a signature an anomaly?

-Miles

-- 
Philosophy, n. A route of many roads leading from nowhere to nothing.

Re: auto-recognizing utf-16le ?

[Kenichi Handa]

In article <m2r5xlw4qp.fsf@igel.home>, Andreas Schwab <schwab@linux-m68k.org> writes:

> Miles Bader <miles.bader@necel.com> writes:
> > Someone on #emacs noticed that emacs doesn't seem to auto-recognize
> > files encoding using utf-16le.

> UTF-16 detection never tries to auto detect files without a signature.

No.  detect_coding_utf_16 tries to check if the file is
UTF-16 or not by checking the dispersion of Eth and Oth
bytes where E is even and O is odd.  But, there were two
bugs in the code.  One was already fixed by this change.

2009-06-15  Andreas Schwab  <schwab@linux-m68k.org>

	* coding.c (detect_coding_utf_16): Fix typo counting odd bytes.

And, I've just installed a fix of another bug.

So, with the latest code, if you set
inhibit-null-byte-detection to t, and prefer utf-16be and/or
utf-16le, Emacs will detect UTF-16 files without BOM in most
cases.

---
Kenichi Handa
handa@m17n.org

Re: auto-recognizing utf-16le ?

[Andreas Schwab]

Kenichi Handa <handa@m17n.org> writes:

> And, I've just installed a fix of another bug.

I think instead of

      while (detect_info->rejected != CATEGORY_MASK_UTF_16)

you probably want to check this:

      while ((detect_info->rejected & CATEGORY_MASK_UTF_16) != CATEGORY_MASK_UTF_16)

since there may be bits for other categories already set in rejected.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: auto-recognizing utf-16le ?

[Kenichi Handa]

In article <m24oug1av0.fsf@linux-m68k.org>, Andreas Schwab <schwab@linux-m68k.org> writes:

> Kenichi Handa <handa@m17n.org> writes:
> > And, I've just installed a fix of another bug.

> I think instead of

>       while (detect_info->rejected != CATEGORY_MASK_UTF_16)

> you probably want to check this:

>       while ((detect_info->rejected & CATEGORY_MASK_UTF_16) != CATEGORY_MASK_UTF_16)

> since there may be bits for other categories already set in rejected.

Ah!  You are right, thank you.  I installed that fix.

---
Kenichi Handa
handa@m17n.org