`decode-coding-string' question

`decode-coding-string' question

[Paul Pogonyshev]

Say I have a string with various text properties set.  If I then
apply `decode-coding-string' to it, all the properties are lost.
Is there a way to transfer properties from ``character beginning''
(i.e. first character of a number being combined during decoding)
to the decoded character?

I.e. UTF-8 representation of copyright sign is "\xc2\xa9".  Can I
transfer properties of '\xc2' character to the copyright sign
character and discard properties of '\xa9' character?

Paul

Re: `decode-coding-string' question

[Kenichi Handa]

In article <200607040035.01379.pogonyshev@gmx.net>, Paul Pogonyshev <pogonyshev@gmx.net> writes:

> Say I have a string with various text properties set.  If I then
> apply `decode-coding-string' to it, all the properties are lost.
> Is there a way to transfer properties from ``character beginning''
> (i.e. first character of a number being combined during decoding)
> to the decoded character?

In the current implementation, it's impossible.  But, first
of all, why do you have text properties on unibyte string?
I think all text processing should be done after the string
is decoded.

---
Kenichi Handa
handa@m17n.org

Re: `decode-coding-string' question

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Tue, 04 Jul 2006 09:50:16 +0900
> Cc: emacs-devel@gnu.org
> 
> But, first of all, why do you have text properties on unibyte
> string?  I think all text processing should be done after the string
> is decoded.

Seconded.  Undecoded string is not text, strictly speaking, it's a
stream of bytes with no clear notion of a character.

Re: `decode-coding-string' question

[Richard Stallman]

    Say I have a string with various text properties set.  If I then
    apply `decode-coding-string' to it, all the properties are lost.
    Is there a way to transfer properties from ``character beginning''
    (i.e. first character of a number being combined during decoding)
    to the decoded character?

It would be a nice improvement to make, but not trivial.

Re: `decode-coding-string' question

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     Say I have a string with various text properties set.  If I then
>     apply `decode-coding-string' to it, all the properties are lost.
>     Is there a way to transfer properties from ``character beginning''
>     (i.e. first character of a number being combined during decoding)
>     to the decoded character?
>
> It would be a nice improvement to make, but not trivial.

I think it would be more important that recoding/encoding/decoding
preserved markers.  Again, likely not trivial.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: `decode-coding-string' question

[Johan Bockgård]

While we're at it: I would like that `format-time-string' preserves
text properties (like `format' does).

Re: `decode-coding-string' question

[Paul Pogonyshev]

Kenichi Handa wrote:
> In article <200607040035.01379.pogonyshev@gmx.net>, Paul Pogonyshev <pogonyshev@gmx.net> writes:
> 
> > Say I have a string with various text properties set.  If I then
> > apply `decode-coding-string' to it, all the properties are lost.
> > Is there a way to transfer properties from ``character beginning''
> > (i.e. first character of a number being combined during decoding)
> > to the decoded character?
> 
> In the current implementation, it's impossible.  But, first
> of all, why do you have text properties on unibyte string?
> I think all text processing should be done after the string
> is decoded.

Bad.  OK, here is my task: I have a C string in the sources, possibly
containing encoded characters, like

	"foo bla \xc2\xa9",

the last thing being the UTF-8 copyright characters.  I want to
decode the string (can do that) _and_ know where particular
characters begin. Currently I set text property `point' on character
beginnings, but `decode-coding-string' eats them :(  Can anyone see a
different solution, maybe ugly if nothing else?  (Except that custom
implementation of `decode-coding-string' doesn't count as a solution
;)

Paul

Re: `decode-coding-string' question

[Richard Stallman]

    Seconded.  Undecoded string is not text, strictly speaking, it's a
    stream of bytes with no clear notion of a character.

Since text properties apply to portions of a string or buffer,
they can make sense on encoded text.  So the idea is not nonsense.
There might be uses for it.

However, I am not sure it would be worth the work to implement
this, even if someone competent wants to do it.

Re: `decode-coding-string' question

[Kenichi Handa]

In article <200607041831.18435.pogonyshev@gmx.net>, Paul Pogonyshev <pogonyshev@gmx.net> writes:

> Bad.  OK, here is my task: I have a C string in the sources, possibly
> containing encoded characters, like

> 	"foo bla \xc2\xa9",

> the last thing being the UTF-8 copyright characters.  I want to
> decode the string (can do that) _and_ know where particular
> characters begin. Currently I set text property `point' on character
> beginnings, but `decode-coding-string' eats them :(  Can anyone see a
> different solution, maybe ugly if nothing else?  (Except that custom
> implementation of `decode-coding-string' doesn't count as a solution
> ;)

Why don't you find paticular characters in the decoded
string?

---
Kenichi Handa
handa@m17n.org

Re: `decode-coding-string' question

[Paul Pogonyshev]

Kenichi Handa wrote:
> In article <200607041831.18435.pogonyshev@gmx.net>, Paul Pogonyshev <pogonyshev@gmx.net> writes:
> 
> > Bad.  OK, here is my task: I have a C string in the sources, possibly
> > containing encoded characters, like
> 
> > 	"foo bla \xc2\xa9",
> 
> > the last thing being the UTF-8 copyright characters.  I want to
> > decode the string (can do that) _and_ know where particular
> > characters begin. Currently I set text property `point' on character
> > beginnings, but `decode-coding-string' eats them :(  Can anyone see a
> > different solution, maybe ugly if nothing else?  (Except that custom
> > implementation of `decode-coding-string' doesn't count as a solution
> > ;)
> 
> Why don't you find paticular characters in the decoded
> string?

I do.  But I need to know where they begin in the buffer (containing
the encoded C string.)  I don't see a way to keep this information at
present... :(

For instance, if the buffer only contains "\xc2\xa9foo", I'd like
to receive a string with the following text properties:

  #("©foo" 0 1 (point 0) 1 2 (point 8) 2 3 (point 9) 3 4 (point 10))

The first characters actually takes 8 characters in the buffer!

Paul

Re: `decode-coding-string' question

[Stuart D. Herring]

> I do.  But I need to know where they begin in the buffer (containing
> the encoded C string.)  I don't see a way to keep this information at
> present... :(
>
> For instance, if the buffer only contains "\xc2\xa9foo", I'd like
> to receive a string with the following text properties:
>
>   #("©foo" 0 1 (point 0) 1 2 (point 8) 2 3 (point 9) 3 4 (point 10))
>
> The first characters actually takes 8 characters in the buffer!

This is a horrible hack, but could you take the "©foo" that you get
(without properties), turn it into "©-f-o-o-", then reencode it and look
for the '-'s you added?  It might run into trouble if there were -s in the
string already, but you could always compare the original and -ed strings
to resolve that.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: `decode-coding-string' question

[Paul Pogonyshev]

Stuart D. Herring wrote:
> > I do.  But I need to know where they begin in the buffer (containing
> > the encoded C string.)  I don't see a way to keep this information at
> > present... :(
> >
> > For instance, if the buffer only contains "\xc2\xa9foo", I'd like
> > to receive a string with the following text properties:
> >
> >   #("©foo" 0 1 (point 0) 1 2 (point 8) 2 3 (point 9) 3 4 (point 10))
> >
> > The first characters actually takes 8 characters in the buffer!
> 
> This is a horrible hack, but could you take the "©foo" that you get
> (without properties), turn it into "©-f-o-o-", then reencode it and look
> for the '-'s you added?  It might run into trouble if there were -s in the
> string already, but you could always compare the original and -ed strings
> to resolve that.

Probably I don't have any other option, I have to try something like
this...  Or maybe reencode character by one...

Thanks!

Paul

Re: `decode-coding-string' question

[Kenichi Handa]

In article <200607051911.45299.pogonyshev@gmx.net>, Paul Pogonyshev <pogonyshev@gmx.net> writes:

>> Why don't you find paticular characters in the decoded
>> string?

> I do.  But I need to know where they begin in the buffer (containing
> the encoded C string.)  I don't see a way to keep this information at
> present... :(

How did you make that buffer?  Why don't you have an
already-decoded text in that buffer?

> For instance, if the buffer only contains "\xc2\xa9foo", I'd like
> to receive a string with the following text properties:

>   #("©foo" 0 1 (point 0) 1 2 (point 8) 2 3 (point 9) 3 4 (point 10))

> The first characters actually takes 8 characters in the buffer!

They are just displayed by 8 characters, as well as, a
control character, say Formfeed (C-l), is displayed by 2
characters "^L".

---
Kenichi Handa
handa@m17n.org

Re: `decode-coding-string' question

[Paul Pogonyshev]

Kenichi Handa wrote:
> In article <200607051911.45299.pogonyshev@gmx.net>, Paul Pogonyshev <pogonyshev@gmx.net> writes:
> 
> >> Why don't you find paticular characters in the decoded
> >> string?
> 
> > I do.  But I need to know where they begin in the buffer (containing
> > the encoded C string.)  I don't see a way to keep this information at
> > present... :(
> 
> How did you make that buffer?  Why don't you have an
> already-decoded text in that buffer?

Because it's a C source file.  Strings have to be encoded there.

> > For instance, if the buffer only contains "\xc2\xa9foo", I'd like
> > to receive a string with the following text properties:
> 
> >   #("©foo" 0 1 (point 0) 1 2 (point 8) 2 3 (point 9) 3 4 (point 10))
> 
> > The first characters actually takes 8 characters in the buffer!
> 
> They are just displayed by 8 characters, as well as, a
> control character, say Formfeed (C-l), is displayed by 2
> characters "^L".

They are 8 different characters in the buffer.

Paul

Re: `decode-coding-string' question

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmx.net>
> Date: Thu, 6 Jul 2006 18:52:28 +0300
> Cc: Kenichi Handa <handa@m17n.org>
> 
> > > I do.  But I need to know where they begin in the buffer (containing
> > > the encoded C string.)  I don't see a way to keep this information at
> > > present... :(
> > 
> > How did you make that buffer?  Why don't you have an
> > already-decoded text in that buffer?
> 
> Because it's a C source file.  Strings have to be encoded there.

Paul, there's some misunderstanding here, so please bear with us.
Handa-san cannot understand how come you have undecoded characters in
the buffer, and neither can I.

The fact that it's a C file does not matter: Emacs _always_ decodes
characters when it visits the file, no matter if it's a C file or
something else.  In the text you get in your buffer the characters
should be decoded.  The question is, how come it didn't decode these
characters in your case?  Are there other non-ASCII characters in the
same file, perhaps? if so, what characters are those?  For that
matter, can you post a small sample file that, when visited in Emacs,
leaves the UTF-8 encoded characters undecoded in the buffer?  Please
post that file as a binary attachment, to avoid munging it by email
software en- and de-coding.

Re: `decode-coding-string' question

[Paul Pogonyshev]

Eli Zaretskii wrote:
> > From: Paul Pogonyshev <pogonyshev@gmx.net>
> > Date: Thu, 6 Jul 2006 18:52:28 +0300
> > Cc: Kenichi Handa <handa@m17n.org>
> > 
> > > > I do.  But I need to know where they begin in the buffer (containing
> > > > the encoded C string.)  I don't see a way to keep this information at
> > > > present... :(
> > > 
> > > How did you make that buffer?  Why don't you have an
> > > already-decoded text in that buffer?
> > 
> > Because it's a C source file.  Strings have to be encoded there.
> 
> Paul, there's some misunderstanding here, so please bear with us.
> Handa-san cannot understand how come you have undecoded characters in
> the buffer, and neither can I.
> 
> The fact that it's a C file does not matter: Emacs _always_ decodes
> characters when it visits the file, no matter if it's a C file or
> something else.  In the text you get in your buffer the characters
> should be decoded.  The question is, how come it didn't decode these
> characters in your case?  Are there other non-ASCII characters in the
> same file, perhaps? if so, what characters are those?  For that
> matter, can you post a small sample file that, when visited in Emacs,
> leaves the UTF-8 encoded characters undecoded in the buffer?  Please
> post that file as a binary attachment, to avoid munging it by email
> software en- and de-coding.

There is indeed a misunderstanding.  The characters in the buffer _are_
decoded.  However the characters form C escape sequence, like "\xc2\xa9".
To know what character is encoded by this C sequence, I first translate
strings "\xc2" and "\xa9" to the appropriate (undecoded!) characters.
The resulting string of length 2 is encoded in UTF-8 and I decode it
to receive the copyright character or whatever.

Phew.  Hope it is clearer now.  Anyway, it is not so important for me
anymore, since gettext doesn't support non-ASCII characters in
untranslated strings with fairly recent GNU libc.  (And yes, I tried
inserting non-ASCII characters in the untranslated strings.)

Paul

Re: `decode-coding-string' question

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmx.net>
> Date: Thu, 6 Jul 2006 23:34:21 +0300
> Cc: handa@m17n.org
> 
> There is indeed a misunderstanding.  The characters in the buffer _are_
> decoded.  However the characters form C escape sequence, like "\xc2\xa9"

Right, I see the problem now.

> To know what character is encoded by this C sequence, I first translate
> strings "\xc2" and "\xa9" to the appropriate (undecoded!) characters.
> The resulting string of length 2 is encoded in UTF-8 and I decode it
> to receive the copyright character or whatever.

Why not use `(decode-coding-string "\xc2\xa9" 'utf-8)' right away?  It
gives me the right character directly.

Btw, why don't we have a feature in cc-mode to transparently decode
and encode such strings when the source file is read/written?  If
detecting the encoding is an issue, we could for starters ask that
users state that in some file-local variable.

Re: `decode-coding-string' question

[Paul Pogonyshev]

Eli Zaretskii wrote:
> > To know what character is encoded by this C sequence, I first translate
> > strings "\xc2" and "\xa9" to the appropriate (undecoded!) characters.
> > The resulting string of length 2 is encoded in UTF-8 and I decode it
> > to receive the copyright character or whatever.
> 
> Why not use `(decode-coding-string "\xc2\xa9" 'utf-8)' right away?  It
> gives me the right character directly.

Because you underquoted the string.  It is actually `(decode-coding-string
"\\xc2\\xa9" 'utf-8)' and does nothing...

> Btw, why don't we have a feature in cc-mode to transparently decode
> and encode such strings when the source file is read/written?  If
> detecting the encoding is an issue, we could for starters ask that
> users state that in some file-local variable.

Given that it is not easy to decode strings (and quite slow)...

Paul

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: `decode-coding-string' question

[David Kastrup]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> Eli Zaretskii wrote:
>> > To know what character is encoded by this C sequence, I first translate
>> > strings "\xc2" and "\xa9" to the appropriate (undecoded!) characters.
>> > The resulting string of length 2 is encoded in UTF-8 and I decode it
>> > to receive the copyright character or whatever.
>> 
>> Why not use `(decode-coding-string "\xc2\xa9" 'utf-8)' right away?  It
>> gives me the right character directly.
>
> Because you underquoted the string.  It is actually `(decode-coding-string
> "\\xc2\\xa9" 'utf-8)' and does nothing...

Because you overquoted the string...

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: `decode-coding-string' question

[Paul Pogonyshev]

David Kastrup wrote:
> Paul Pogonyshev <pogonyshev@gmx.net> writes:
> 
> > Eli Zaretskii wrote:
> >> > To know what character is encoded by this C sequence, I first translate
> >> > strings "\xc2" and "\xa9" to the appropriate (undecoded!) characters.
> >> > The resulting string of length 2 is encoded in UTF-8 and I decode it
> >> > to receive the copyright character or whatever.
> >> 
> >> Why not use `(decode-coding-string "\xc2\xa9" 'utf-8)' right away?  It
> >> gives me the right character directly.
> >
> > Because you underquoted the string.  It is actually `(decode-coding-string
> > "\\xc2\\xa9" 'utf-8)' and does nothing...
> 
> Because you overquoted the string...

...

The buffer contains 8 (eight) characters.  ?\\ ?x ?c ?2 ?\\ ?x ?a ?9.

Paul

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: `decode-coding-string' question

[David Kastrup]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> David Kastrup wrote:
>> Paul Pogonyshev <pogonyshev@gmx.net> writes:
>> 
>> > Eli Zaretskii wrote:
>> >> > To know what character is encoded by this C sequence, I first translate
>> >> > strings "\xc2" and "\xa9" to the appropriate (undecoded!) characters.
>> >> > The resulting string of length 2 is encoded in UTF-8 and I decode it
>> >> > to receive the copyright character or whatever.
>> >> 
>> >> Why not use `(decode-coding-string "\xc2\xa9" 'utf-8)' right away?  It
>> >> gives me the right character directly.
>> >
>> > Because you underquoted the string.  It is actually `(decode-coding-string
>> > "\\xc2\\xa9" 'utf-8)' and does nothing...
>> 
>> Because you overquoted the string...
>
> ...
>
> The buffer contains 8 (eight) characters.  ?\\ ?x ?c ?2 ?\\ ?x ?a ?9.

Yes, I was aware of that.

Let us assume that you fetched the string including the double quotes
into the variable `string'.

Then take a look at (read string).  That should get you the right
amount of backslashes.

Alternatively, assume that point is on the first quote character of
the string.  Then you can use
(read current-buffer)
for reading the string.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642