ncr (numeric character reference) to unicode

ncr (numeric character reference) to unicode

[B. T. Raven]

Does any of you know whether nxhtml has the capability to convert 
sequences like this:

שַׁלוֹם.
(shalom in Hebrew)

to the equivalent Unicode string. N. Walsh had a couple of .el files 
that implemented this I think, but they required cl to be loaded also.
Will Emacs 23.1 also support bidi when it is released?

Thanks,

Ed

Re: ncr (numeric character reference) to unicode

[Eli Zaretskii]

> Date: Mon, 13 Apr 2009 15:17:47 -0500
> From: "B. T. Raven" <nihil@nihilo.net>
> 
> Will Emacs 23.1 also support bidi when it is released?

Sadly, no.  Bidirectional editing needs quite a bit of supporting
code and major changes to some fundamental Emacs features, such as
text fill, and no one succeeded to write the code for that yet.

Re: ncr (numeric character reference) to unicode

[Miles Bader]

"B. T. Raven" <nihil@nihilo.net> writes:
> Does any of you know whether nxhtml has the capability to convert
> sequences like this:
>
> &#1513;&#1473;&#1463;&#1500;&#1493;&#1465;&#1501;.
> (shalom in Hebrew)

The following should work:

   (defun expand-html-encoded-chars (start end)
     (interactive "r")
     (save-excursion
       (goto-char start)
       (while (re-search-forward "&#\\([0-9]+\\);" end t)
         (replace-match 
          (char-to-string
           (decode-char 'ucs (string-to-number (match-string 1))) )
          t t))))

-Miles

-- 
Guilt, n. The condition of one who is known to have committed an indiscretion,
as distinguished from the state of him who has covered his tracks.

Re: ncr (numeric character reference) to unicode

[B. T. Raven]

Miles Bader wrote:
> "B. T. Raven" <nihil@nihilo.net> writes:
>> Does any of you know whether nxhtml has the capability to convert
>> sequences like this:
>>
>> &#1513;&#1473;&#1463;&#1500;&#1493;&#1465;&#1501;.
>> (shalom in Hebrew)
> 
> The following should work:
> 
>    (defun expand-html-encoded-chars (start end)
>      (interactive "r")
>      (save-excursion
>        (goto-char start)
>        (while (re-search-forward "&#\\([0-9]+\\);" end t)
>          (replace-match 
>           (char-to-string
>            (decode-char 'ucs (string-to-number (match-string 1))) )
>           t t))))
> 
> -Miles
> 

Thanks, Eli and Miles. The conversion works fine (with uncomposed 
glyphs, that is, points as separate characters, same as in the html 
codes). I referenced the command in an alias:

(defalias 'xhc 'expand-html-encoded-chars)

and then tried to do the same with this function:

(defun reverse-string (beg end)
   (interactive "r")
   (setq str (buffer-substring beg end))
   (apply #'string (nreverse (string-to-list str))))

but it doesn't seem to work, although it doesn't produce errors in a 
traceback buffer. What am I missing?

Thanks,

Ed

Re: ncr (numeric character reference) to unicode

[Stephen Berman]

On Tue, 14 Apr 2009 11:42:09 -0500 "B. T. Raven" <nihil@nihilo.net> wrote:

> Miles Bader wrote:
>> "B. T. Raven" <nihil@nihilo.net> writes:
>>> Does any of you know whether nxhtml has the capability to convert
>>> sequences like this:
>>>
>>> &#1513;&#1473;&#1463;&#1500;&#1493;&#1465;&#1501;.
>>> (shalom in Hebrew)
>>
>> The following should work:
>>
>>    (defun expand-html-encoded-chars (start end)
>>      (interactive "r")
>>      (save-excursion
>>        (goto-char start)
>>        (while (re-search-forward "&#\\([0-9]+\\);" end t)
>>          (replace-match           (char-to-string
>>            (decode-char 'ucs (string-to-number (match-string 1))) )
>>           t t))))
>>
>> -Miles
>>
>
> Thanks, Eli and Miles. The conversion works fine (with uncomposed glyphs, that
> is, points as separate characters, same as in the html codes). I referenced
> the command in an alias:
>
> (defalias 'xhc 'expand-html-encoded-chars)
>
> and then tried to do the same with this function:
>
> (defun reverse-string (beg end)
>   (interactive "r")
>   (setq str (buffer-substring beg end))
>   (apply #'string (nreverse (string-to-list str))))
>
> but it doesn't seem to work, although it doesn't produce errors in a traceback
> buffer. What am I missing?
>
> Thanks,
>
> Ed

Does this do what you want?

(defun reverse-string (beg end)
  (interactive "r")
  (xhc beg end)
  (let* ((beg (region-beginning))
	 (end (region-end))
	 (str1 (buffer-substring beg end))
	 (str2 (apply #'string (nreverse (string-to-list str1)))))
    (replace-string str1 str2 nil beg end)))

Steve Berman

Re: ncr (numeric character reference) to unicode

[B. T. Raven]

Stephen Berman wrote:
> On Tue, 14 Apr 2009 11:42:09 -0500 "B. T. Raven" <nihil@nihilo.net> wrote:
> 
>> Miles Bader wrote:
>>> "B. T. Raven" <nihil@nihilo.net> writes:
>>>> Does any of you know whether nxhtml has the capability to convert
>>>> sequences like this:
>>>>
>>>> &#1513;&#1473;&#1463;&#1500;&#1493;&#1465;&#1501;.
>>>> (shalom in Hebrew)
>>> The following should work:
>>>
>>>    (defun expand-html-encoded-chars (start end)
>>>      (interactive "r")
>>>      (save-excursion
>>>        (goto-char start)
>>>        (while (re-search-forward "&#\\([0-9]+\\);" end t)
>>>          (replace-match           (char-to-string
>>>            (decode-char 'ucs (string-to-number (match-string 1))) )
>>>           t t))))
>>>
>>> -Miles
>>>
>> Thanks, Eli and Miles. The conversion works fine (with uncomposed glyphs, that
>> is, points as separate characters, same as in the html codes). I referenced
>> the command in an alias:
>>
>> (defalias 'xhc 'expand-html-encoded-chars)
>>
>> and then tried to do the same with this function:
>>
>> (defun reverse-string (beg end)
>>   (interactive "r")
>>   (setq str (buffer-substring beg end))
>>   (apply #'string (nreverse (string-to-list str))))
>>
>> but it doesn't seem to work, although it doesn't produce errors in a traceback
>> buffer. What am I missing?
>>
>> Thanks,
>>
>> Ed
> 
> Does this do what you want?
> 
> (defun reverse-string (beg end)
>   (interactive "r")
>   (xhc beg end)
>   (let* ((beg (region-beginning))
> 	 (end (region-end))
> 	 (str1 (buffer-substring beg end))
> 	 (str2 (apply #'string (nreverse (string-to-list str1)))))
>     (replace-string str1 str2 nil beg end)))
> 
> Steve Berman
> 
> 
> 

That would probably do a little more than I want. Miles' expand html 
function is only needed if someone sends these ncr sequences in email. 
Btw, why are beg and end calculated in the function if they are passed 
to it? This almost does what I want:

(defun reverse-bufsubstring (beg end)
   (interactive "r")
   (let* (
	 (str1 (buffer-substring beg end))
	 (str2 (apply #'string (nreverse (string-to-list str1)))))
     (replace-string str1 str2 nil beg end)))


except that it converts

same
one
as
before

into this:

erofeb
sa
eno
emas

so now that has to be reversed line by line rather than character by 
character. Anyway, all of this is just a kludge until the gurus come up 
with a real bidi functionality.

Thanks again,

Ed

Re: ncr (numeric character reference) to unicode

[Kevin Rodgers]

B. T. Raven wrote:
> (defalias 'xhc 'expand-html-encoded-chars)
> 
> and then tried to do the same with this function:
> 
> (defun reverse-string (beg end)
>   (interactive "r")
>   (setq str (buffer-substring beg end))
>   (apply #'string (nreverse (string-to-list str))))
> 
> but it doesn't seem to work, although it doesn't produce errors in a 
> traceback buffer. What am I missing?

Miles' expand-html-encoded-chars function modifies the buffer, with 
replace-match.  Your reverse-string function generates a value, but
does not modify the buffer.

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: ncr (numeric character reference) to unicode

[Kevin Rodgers]

B. T. Raven wrote:
> That would probably do a little more than I want. Miles' expand html 
> function is only needed if someone sends these ncr sequences in email. 
> Btw, why are beg and end calculated in the function if they are passed 
> to it? This almost does what I want:
> 
> (defun reverse-bufsubstring (beg end)
>   (interactive "r")
>   (let* (
>      (str1 (buffer-substring beg end))
>      (str2 (apply #'string (nreverse (string-to-list str1)))))
>     (replace-string str1 str2 nil beg end)))
> 
> 
> except that it converts
> 
> same
> one
> as
> before
> 
> into this:
> 
> erofeb
> sa
> eno
> emas
> 
> so now that has to be reversed line by line rather than character by 
> character. Anyway, all of this is just a kludge until the gurus come up 
> with a real bidi functionality.

(defun reverse-region-by-line (beg end)
   (interactive "r")
   (save-excursion
     (goto-char beg)
     (while (and (< (point) end) (re-search-forward "\\=.*$" end t))
       (replace-match (apply #'string
			    (nreverse (string-to-list (match-string 0)))))
       (forward-line))))


-- 
Kevin Rodgers
Denver, Colorado, USA

Re: ncr (numeric character reference) to unicode

[Stephen Berman]

On Wed, 15 Apr 2009 20:42:49 -0500 "B. T. Raven" <nihil@nihilo.net> wrote:

> Stephen Berman wrote:
[...]
>> Does this do what you want?
>>
>> (defun reverse-string (beg end)
>>   (interactive "r")
>>   (xhc beg end)
>>   (let* ((beg (region-beginning))
>> 	 (end (region-end))
>> 	 (str1 (buffer-substring beg end))
>> 	 (str2 (apply #'string (nreverse (string-to-list str1)))))
>>     (replace-string str1 str2 nil beg end)))
>>
>> Steve Berman
>>
>>
>>
>
> That would probably do a little more than I want. Miles' expand html function
> is only needed if someone sends these ncr sequences in email. Btw, why are beg
> and end calculated in the function if they are passed to it? 

When xhc is called on beg and end (since I mistakenly thought you wanted
to convert the HTML entities and reverse the result in one blow) the
region is changed, so it has to be recalculated for the arguments of
buffer-substring (actually, only region-end changes, so beg really
shouldn't be recalculated).  Of course, new variables could have been
used in the let* clause.

Steve Berman

Re: ncr (numeric character reference) to unicode

[B. T. Raven]

Kevin Rodgers wrote:
> B. T. Raven wrote:
>> (defalias 'xhc 'expand-html-encoded-chars)
>>
>> and then tried to do the same with this function:
>>
>> (defun reverse-string (beg end)
>>   (interactive "r")
>>   (setq str (buffer-substring beg end))
>>   (apply #'string (nreverse (string-to-list str))))
>>
>> but it doesn't seem to work, although it doesn't produce errors in a 
>> traceback buffer. What am I missing?
> 
> Miles' expand-html-encoded-chars function modifies the buffer, with 
> replace-match.  Your reverse-string function generates a value, but
> does not modify the buffer.
> 

Yes, of course. That finally dawned on me. Thanks Kevin and Steve. This 
is finally what I want:

(defun reverse-string (str)
   (apply #'string (nreverse (string-to-list str))))


(defun reverse-region-by-line (beg end)
   (interactive "r")
   (save-excursion
     (goto-char beg)
     (while (and (< (point) end) (re-search-forward "\\=.*$" end t))
       (replace-match (reverse-string (match-string 0)))
       (forward-line))))

But now I find that if I copy-paste from Emacs 23.0.90.1, the Greek letters


αβγδ

appear in Mozilla Tbird (here) in the original order but


בִּּרֵאשׁיתבָּרָּאא לֹהִים אלתשָּׁמַיִם וְ אלת הָ ּאָרֶ ׃


is automatically reversed without running the above command on its 
region. ??? Is there invisible bidi info in the string or is it just the 
fact that the characters are Hebrew that causes this?

Re: ncr (numeric character reference) to unicode

[Stephen Berman]

On Thu, 16 Apr 2009 22:39:43 -0500 "B. T. Raven" <nihil@nihilo.net> wrote:

> But now I find that if I copy-paste from Emacs 23.0.90.1, the Greek letters
>
>
> αβγδ
>
> appear in Mozilla Tbird (here) in the original order but
>
>
> בִּּרֵאשׁיתבָּרָּאא לֹהִים אלתשָּׁמַיִם וְ אלת הָ ּאָרֶ ׃
>
>
> is automatically reversed without running the above command on its region. ???
> Is there invisible bidi info in the string or is it just the fact that the
> characters are Hebrew that causes this?

Presumably the latter.  I guess Thunderbird works like OpenOffice.org,
which also automatically reverses the Hebrew text, and whose Help entry
for "bi-directional writing" says: 
,----
| Currently, OpenOffice.org supports Hindi, Thai, Hebrew, and Arabic as
| CTL [Complex Text Layout] languages.  If you select the text flow from
| right to left, embedded Western text still runs from left to
| right. The cursor responds to the arrow keys in that Right Arrow moves
| it "to the text end" and Left Arrow "to the text start".
`----

Steve Berman

Re: ncr (numeric character reference) to unicode

[B. T. Raven]

Stephen Berman wrote:
> On Thu, 16 Apr 2009 22:39:43 -0500 "B. T. Raven" <nihil@nihilo.net> wrote:
> 
>> But now I find that if I copy-paste from Emacs 23.0.90.1, the Greek letters
>>
>>
>> αβγδ
>>
>> appear in Mozilla Tbird (here) in the original order but
>>
>>
>>

Saw this right to left order in Emacs 23:

בִּּרֵאשׁיתבָּרָּאא לֹהִים אלתשָּׁמַיִם וְ אלת הָ ּאָרֶ ׃

Copy-pasted here in Tbird:


׃ ֶרָאּ ָה תלא ְו םִיַמָּׁשתלא םיִהֹל אאָּרָּבתיׁשאֵרִּּב

First line C-c C-v (CUA) here in Tbird:

בִּּרֵאשׁיתבָּרָּאא לֹהִים אלתשָּׁמַיִם וְ אלת הָ ּאָרֶ ׃

And it doesn't matter which direction the Hebrew text is selected in. In 
fact S-arrow won't move over the Hebrew, character-by-character but it 
selects the whole line, whether characters are in forward or reverse 
order and whether cursor starts at left or right.

Ed




>>
>>
>> is automatically reversed without running the above command on its region. ???
>> Is there invisible bidi info in the string or is it just the fact that the
>> characters are Hebrew that causes this?
> 
> Presumably the latter.  I guess Thunderbird works like OpenOffice.org,
> which also automatically reverses the Hebrew text, and whose Help entry
> for "bi-directional writing" says: 
> ,----
> | Currently, OpenOffice.org supports Hindi, Thai, Hebrew, and Arabic as
> | CTL [Complex Text Layout] languages.  If you select the text flow from
> | right to left, embedded Western text still runs from left to
> | right. The cursor responds to the arrow keys in that Right Arrow moves
> | it "to the text end" and Left Arrow "to the text start".
> `----
> 
> Steve Berman


> 
> 
>