* Single unrecognized character wrecks entire display
@ 2012-08-18 16:35 Alexandre Oberlin
2012-08-18 22:33 ` Peter Dyballa
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Alexandre Oberlin @ 2012-08-18 16:35 UTC (permalink / raw
To: help-gnu-emacs
Hi all,
I have a file created by a translation software. The file initially
displayed well in emacs and after some edit in the translation software it
didn’t anymore and reverted to raw text. I got this message when I tried
to save in utf-8:
utf-8-mac cannot encode these: \351 \350 \351 \342 \351 \234 \350 \350
\311 \240
In short all the non ASCI had become unrecognized. However once I removed
a \234 character,that should display as œ, I could save it and recover a
correct display in emacs.
Is there a way to quickly spot the offending character within emacs in
such cases ?
Alexandre
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-18 16:35 Single unrecognized character wrecks entire display Alexandre Oberlin
@ 2012-08-18 22:33 ` Peter Dyballa
[not found] ` <mailman.7220.1345329241.855.help-gnu-emacs@gnu.org>
2012-08-22 15:01 ` Stefan Monnier
2 siblings, 0 replies; 11+ messages in thread
From: Peter Dyballa @ 2012-08-18 22:33 UTC (permalink / raw
To: Alexandre Oberlin; +Cc: help-gnu-emacs
Am 18.08.2012 um 18:35 schrieb Alexandre Oberlin:
> Is there a way to quickly spot the offending character within emacs in
> such cases ?
Yes! When you try save a faulty "text" in UTF-8 then GNU Emacs will name the faulty codes. \234 is such a faulty code. In HEX it's (U+00)9C – obviously an 8-bit control character. With \240 or U+00A0 you have again a text character.
œ is U+0153 or \523. You need to update your translator.
--
Greetings
Pete
Windows, c'est un peu comme le beaujolais nouveau: à chaque nouvelle cuvée on sait que ce sera dégueulasse, mais on en prend quand même, par masochisme.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
[not found] ` <mailman.7220.1345329241.855.help-gnu-emacs@gnu.org>
@ 2012-08-22 9:36 ` Alexandre Oberlin
2012-08-22 15:18 ` Peter Dyballa
[not found] ` <mailman.7419.1345648764.855.help-gnu-emacs@gnu.org>
0 siblings, 2 replies; 11+ messages in thread
From: Alexandre Oberlin @ 2012-08-22 9:36 UTC (permalink / raw
To: help-gnu-emacs
Thank you Peter for your answer.
> Yes! When you try save a faulty "text" in UTF-8 then GNU Emacs will name
> the faulty codes.
The problem is that it names a full list as bad characters, when only one.
utf-8-mac cannot encode these: \351 \350 \351 \342 \351 \234 \350 \350
\311 \240
How can I spot the true non utf-8 without trying them all?
Windows, c'est un peu comme le beaujolais nouveau: à chaque nouvelle
cuvée on sait que ce sera dégueulasse, mais on en prend quand même, par
masochisme.
Masochisme ou obligation ?
Alexandre
On Sun, 19 Aug 2012 00:33:25 +0200, Peter Dyballa <Peter_Dyballa@web.de>
wrote:
>
> Am 18.08.2012 um 18:35 schrieb Alexandre Oberlin:
>
>> Is there a way to quickly spot the offending character within emacs in
>> such cases ?
>
> Yes! When you try save a faulty "text" in UTF-8 then GNU Emacs will name
> the faulty codes. \234 is such a faulty code. In HEX it's (U+00)9C –
> obviously an 8-bit control character. With \240 or U+00A0 you have again
> a text character.
>
> œ is U+0153 or \523. You need to update your translator.
>
> --
> Greetings
>
> Pete
>
> Windows, c'est un peu comme le beaujolais nouveau: à chaque nouvelle
> cuvée on sait que ce sera dégueulasse, mais on en prend quand même, par
> masochisme.
>
>
--
Using Opera's revolutionary email client: http://www.opera.com/mail/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-18 16:35 Single unrecognized character wrecks entire display Alexandre Oberlin
2012-08-18 22:33 ` Peter Dyballa
[not found] ` <mailman.7220.1345329241.855.help-gnu-emacs@gnu.org>
@ 2012-08-22 15:01 ` Stefan Monnier
2012-08-24 13:13 ` Alexandre Oberlin
2 siblings, 1 reply; 11+ messages in thread
From: Stefan Monnier @ 2012-08-22 15:01 UTC (permalink / raw
To: help-gnu-emacs
> Is there a way to quickly spot the offending character within Emacs in
> such cases?
Good question. Please send it as a feature request via
M-x report-emacs-bug.
As for an answer, it might be that C-x RET r utf-8 RET might do the
trick (basically, the single offending byte caused Emacs to decide that
the file is not using utf-8 and read it as a binary file; so by
forcing the use of utf-8 you should get all the utf-8 encoded chars to
appear correctly and the invalid byte to appear as \234).
Stefan
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-22 9:36 ` Alexandre Oberlin
@ 2012-08-22 15:18 ` Peter Dyballa
[not found] ` <mailman.7419.1345648764.855.help-gnu-emacs@gnu.org>
1 sibling, 0 replies; 11+ messages in thread
From: Peter Dyballa @ 2012-08-22 15:18 UTC (permalink / raw
To: Alexandre Oberlin; +Cc: help-gnu-emacs
Am 22.08.2012 um 11:36 schrieb Alexandre Oberlin:
> The problem is that it names a full list as bad characters, when only one.
> utf-8-mac cannot encode these: \351 \350 \351 \342 \351 \234 \350 \350
> \311 \240
> How can I spot the true non utf-8 without trying them all?
When you set read-quoted-char-radix to 8 you can search for these "characters" in the text by:
C-s C-q 3 5 1 RET
Hopefully! I think the problem is that your convertor (can't you use something reliable like iconv or recode?) makes mistakes. \240 or A0 in hex exists as partner of another byte (with C2 it constructs NO-BREAK SPACE, with C3 it's LATIN SMALL LETTER A WITH GRAVE, …), \234 or 9C builds with C3 LATIN CAPITAL LETTER U WITH DIAERESIS etc. I think what GNU Emacs wants to tell you and what I did not understand the first time is, that some characters obviously are not encoded correctly so that these "isolated" *bytes* are left over, they don't fit into regular 2- or 3- or even 4-byte codes of the UTF-8 encoding – and of course none of them is an ASCII character encoded by one byte (i.e., itself).
The utf-8-mac encoding in GNU Emacs is UTF-8 that uses ^M or CR as end of line character (UNIX uses ^J or Line Feed).
Can you give us some more details of the original source and the convertor, and its working principle (command line options)? How do you open it in GNU Emacs? How does it behave when you had launched GNU Emacs with -Q, i.e., with none of your possibly faulty customisation? By using for example on the command line:
env LC_CTYPE=UTF-8 LANG=fr_FR.UTF-8 emacs -Q &
or
env LC_CTYPE=UTF-8 LANG=fr_FR.UTF-8 /Applications/Emacs.app/Contents/MacOS/Emacs -Q &
GNU Emacs should then automatically switch to some UTF-8 encoding – whether it's Apple or UNIX or MS line endings should not play such a role. You should see, if the input is faulty, searchable octal codes.
--
Greetings
Pete
A lot of us are working harder than we want, at things we don't like to do. Why? ...In order to afford the sort of existence we don't care to live.
– Bradford Angier
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-22 15:01 ` Stefan Monnier
@ 2012-08-24 13:13 ` Alexandre Oberlin
2012-08-25 3:07 ` Stefan Monnier
0 siblings, 1 reply; 11+ messages in thread
From: Alexandre Oberlin @ 2012-08-24 13:13 UTC (permalink / raw
To: help-gnu-emacs
> it might be that C-x RET r utf-8 RET might do the
> trick
Thanks Stefan, but this does not work.
Alexandre
>> Is there a way to quickly spot the offending character within Emacs in
>> such cases?
>
> Good question. Please send it as a feature request via
> M-x report-emacs-bug.
>
> As for an answer, it might be that C-x RET r utf-8 RET might do the
> trick (basically, the single offending byte caused Emacs to decide that
> the file is not using utf-8 and read it as a binary file; so by
> forcing the use of utf-8 you should get all the utf-8 encoded chars to
> appear correctly and the invalid byte to appear as \234).
>
>
> Stefan
--
Using Opera's revolutionary email client: http://www.opera.com/mail/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
[not found] ` <mailman.7419.1345648764.855.help-gnu-emacs@gnu.org>
@ 2012-08-24 13:46 ` Alexandre Oberlin
2012-08-24 14:54 ` Alexandre Oberlin
2012-08-24 15:01 ` Peter Dyballa
0 siblings, 2 replies; 11+ messages in thread
From: Alexandre Oberlin @ 2012-08-24 13:46 UTC (permalink / raw
To: help-gnu-emacs
Thank you Peter for your detailed reply.
On Wed, 22 Aug 2012 17:18:36 +0200, Peter Dyballa <Peter_Dyballa@web.de>
wrote:
> When you set read-quoted-char-radix to 8 you can search for these
> "characters" in the text by:
>
> C-s C-q 3 5 1 RET
>
Nice command to find eventual other occurrences once you’ve found the
culprit!
> Hopefully! I think the problem is that your convertor (can't you use
> something reliable like iconv or recode?) makes mistakes.
iconv acts just the same. It tells me the 13th character is faulty (\351),
while only the 40th is (\234)
> \240 or A0 in hex exists as partner of another byte (with C2 it
> constructs NO-BREAK SPACE, with C3 it's LATIN SMALL LETTER A WITH GRAVE,
> …), \234 or 9C builds with C3 LATIN CAPITAL LETTER U WITH DIAERESIS etc.
> I think what GNU Emacs wants to tell you and what I did not understand
> the first time is, that some characters obviously are not encoded
> correctly so that these "isolated" *bytes* are left over, they don't fit
> into regular 2- or 3- or even 4-byte codes of the UTF-8 encoding – and
> of course none of them is an ASCII character encoded by one byte (i.e.,
> itself).
Clear.
> Can you give us some more details of the original source and the
> convertor, and its working principle (command line options)?
It is the target document output of Trados Studio 2009 (well known
translation software running on Windows only). The original is in English
and has no such problems, but opens in emacs as raw text Mac.
> How do you open it in GNU Emacs?
C-x C-f
How does it behave when you had launched GNU Emacs
> env LC_CTYPE=UTF-8 LANG=fr_FR.UTF-8 emacs -Q &
Same.
I’m using GNU emacs on Linux, on Cygwin when I use Trados. The behavior is
the same on that regard.
Cheers,
Alexandre
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-24 13:46 ` Alexandre Oberlin
@ 2012-08-24 14:54 ` Alexandre Oberlin
2012-08-24 15:08 ` Peter Dyballa
2012-08-24 15:01 ` Peter Dyballa
1 sibling, 1 reply; 11+ messages in thread
From: Alexandre Oberlin @ 2012-08-24 14:54 UTC (permalink / raw
To: help-gnu-emacs
On Fri, 24 Aug 2012 15:46:47 +0200, Alexandre Oberlin <please@nospam.com>
wrote:
> The original is in English and has no such problems, but opens in emacs
> as raw text Mac.
Wrong! Actually the original also has offending characters in geographic
names:
\207 and \222.
This is getting very weird.
There are 2 different issues :
1. will the file display correctly?
2. will it save in utf-8?
I could save in utf-8 only by manually specifying utf-8-mac when saving
(changing the buffer’s mode to utf-8-mac is not enough). However the
display remains garbled.
I could display correctly the other accented characters only by erasing
\207 and \234, though \222 would not hurt in that regard.
Alexandre
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-24 13:46 ` Alexandre Oberlin
2012-08-24 14:54 ` Alexandre Oberlin
@ 2012-08-24 15:01 ` Peter Dyballa
1 sibling, 0 replies; 11+ messages in thread
From: Peter Dyballa @ 2012-08-24 15:01 UTC (permalink / raw
To: Alexandre Oberlin; +Cc: help-gnu-emacs
Am 24.08.2012 um 15:46 schrieb Alexandre Oberlin:
> iconv acts just the same. It tells me the 13th character is faulty (\351), while only the 40th is (\234)
Alexandre,
I think you're making here the same mistake as I did before! \351 is not the number of the character in the Unicode encoding but an UTF-8 byte. The UTF encodings are multi-byte encodings and therefore there cannot be that byte \351 stands for character \351 (or 233 decimal or E9 hexadecimal). Iconv and GNU Emacs obviously find some single isolated bytes are spread into the text. This could also explain the different counting: characters vs. bytes (13th vs. 40th).
Could you try a native MS Losedos GNU Emacs?
Could you send me privately such a translation output before GNU Emacs or iconv have changed anything? Can it be that this output is not plain text but some structured format containing these odd bytes you mentioned initially which might switch font or emphasising or tell where a paragraph ends or a footnote starts?
--
Greetings
Pete
"By filing this bug report you have challenged the honor of my family. Prepare to die!"
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-24 14:54 ` Alexandre Oberlin
@ 2012-08-24 15:08 ` Peter Dyballa
0 siblings, 0 replies; 11+ messages in thread
From: Peter Dyballa @ 2012-08-24 15:08 UTC (permalink / raw
To: Alexandre Oberlin; +Cc: help-gnu-emacs
Am 24.08.2012 um 16:54 schrieb Alexandre Oberlin:
> On Fri, 24 Aug 2012 15:46:47 +0200, Alexandre Oberlin <please@nospam.com> wrote:
>> The original is in English and has no such problems, but opens in emacs as raw text Mac.
> Wrong! Actually the original also has offending characters in geographic names:
> \207 and \222.
This could be easily cured: pass it to a converter which can remove inappropriate bytes. Iconv has the -c option. You could use it to convert from UTF-8 to UTF-8-MAC and from one the untouched original file to a working copy. Then translate the working copy.
--
Greetings
Pete
Build a man a fire and he'll be warm for a night, but set a man on fire and he'll be warm for the rest of his life.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Single unrecognized character wrecks entire display
2012-08-24 13:13 ` Alexandre Oberlin
@ 2012-08-25 3:07 ` Stefan Monnier
0 siblings, 0 replies; 11+ messages in thread
From: Stefan Monnier @ 2012-08-25 3:07 UTC (permalink / raw
To: help-gnu-emacs
>> it might be that C-x RET r utf-8 RET might do the trick
> Thanks Stefan, but this does not work.
This operation should work, so if it "does not work" you probably found
a bug. If it did do something, but just not what you wanted, then it
would help to spell out what it did and in which way this didn't satisfy
your needs.
>> Good question. Please send it as a feature request via
>> M-x report-emacs-bug.
And this suggestion was really the most important one: IIUC you have
solved your immediate problem and are looking for a way to avoid such
pain in the future, so M-x report-emacs-bug is one of the best ways to
do that.
Stefan
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-08-25 3:07 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-18 16:35 Single unrecognized character wrecks entire display Alexandre Oberlin
2012-08-18 22:33 ` Peter Dyballa
[not found] ` <mailman.7220.1345329241.855.help-gnu-emacs@gnu.org>
2012-08-22 9:36 ` Alexandre Oberlin
2012-08-22 15:18 ` Peter Dyballa
[not found] ` <mailman.7419.1345648764.855.help-gnu-emacs@gnu.org>
2012-08-24 13:46 ` Alexandre Oberlin
2012-08-24 14:54 ` Alexandre Oberlin
2012-08-24 15:08 ` Peter Dyballa
2012-08-24 15:01 ` Peter Dyballa
2012-08-22 15:01 ` Stefan Monnier
2012-08-24 13:13 ` Alexandre Oberlin
2012-08-25 3:07 ` Stefan Monnier
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.