Re: command fill-paragraph deletes leading Umlauts if line begins with space

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Richard Stallman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

    A function in AUCTeX for doing indentation looks at whitespace syntax
    for finding the first non-whitespace character (and so does
    `back-to-indentation' in CVS Emacs).  That means it will skip the "Ü"
    and delete everything from the beginning of the line to and including
    the "Ü".

    I removed this code in CVS AUCTeX which now only uses
    `back-to-indentation'.  In Emacs 21.3 this function does not look at
    character syntax but simply skips spaces and tab characters at the
    beginning of a line.

If AucTeX specifically wants to skip just space and tab, it could do
that explicitly, rather than calling back-to-indentation.  That would
work in the latest Emacs.

We could look at trying to fix this by giving different syntax
values to the unibyte non-ASCII codes.  The problem is that
(I think) different non-ASCII coding systems would want different
syntax values.  However, making codes 200-377 all "word constituent"
might be better than making them all "whitespace".

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Ralf Angeli]

* Richard Stallman (2004-12-24) writes:

>     A function in AUCTeX for doing indentation looks at whitespace syntax
>     for finding the first non-whitespace character (and so does
>     `back-to-indentation' in CVS Emacs).  That means it will skip the "Ü"
>     and delete everything from the beginning of the line to and including
>     the "Ü".
>
>     I removed this code in CVS AUCTeX which now only uses
>     `back-to-indentation'.  In Emacs 21.3 this function does not look at
>     character syntax but simply skips spaces and tab characters at the
>     beginning of a line.
>
> If AucTeX specifically wants to skip just space and tab, it could do
> that explicitly, rather than calling back-to-indentation.  That would
> work in the latest Emacs.

Yes, this would be a possibility.  Normally I tend to use functions
provided by Emacs if the do what I need.  In case of
`back-to-indentation' (now looking for whitespace syntax) I thought it
would be good because there might be other whitespace characters in a
LaTeX file besides space and tab we want to skip.

> We could look at trying to fix this by giving different syntax
> values to the unibyte non-ASCII codes.  The problem is that
> (I think) different non-ASCII coding systems would want different
> syntax values.

Couldn't the information about the character syntax which is used in
multibyte mode also be used in unibyte mode?  (I hope this question is
not too naive.)

> However, making codes 200-377 all "word constituent"
> might be better than making them all "whitespace".

>From my uninformed point of view this sounds reasonable.

-- 
Ralf

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Eli Zaretskii]

> From: Ralf Angeli <angeli@iwi.uni-sb.de>
> Date: Sun, 26 Dec 2004 23:39:51 +0100
> 
> Couldn't the information about the character syntax which is used in
> multibyte mode also be used in unibyte mode?

The problem is, as Richard pointed out, that unibyte characters don't
tell what character set they belong to.  Different unibyte character
sets might well assign different meanings, and thus different syntax
properties, to the same 8-bit code.

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Ralf Angeli]

* Eli Zaretskii (2004-12-27) writes:

>> From: Ralf Angeli <angeli@iwi.uni-sb.de>
>> Date: Sun, 26 Dec 2004 23:39:51 +0100
>> 
>> Couldn't the information about the character syntax which is used in
>> multibyte mode also be used in unibyte mode?
>
> The problem is, as Richard pointed out, that unibyte characters don't
> tell what character set they belong to.  Different unibyte character
> sets might well assign different meanings, and thus different syntax
> properties, to the same 8-bit code.

Sure, but IIUC Emacs (in unibyte mode) recognizes the character set by
looking at the language environment.  It could then look up which
syntax a certain 8-bit code has in the respective character set.  I
thought there might exist some mapping for multibyte mode which could
be used for these lookups.

-- 
Ralf

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Richard Stallman]

    Couldn't the information about the character syntax which is used in
    multibyte mode also be used in unibyte mode?  (I hope this question is
    not too naive.)

It is naive, but it might make sense in a way.  We could imagine
making the case-conversion commands convert each character to
multibyte and check its syntax.  That way, the syntax of these unibyte
characters could be determined from the current equivalency.

What do people think of that idea?
In particular, Handa, what do you think?

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Stefan Monnier]

> It is naive, but it might make sense in a way.  We could imagine
> making the case-conversion commands convert each character to
> multibyte and check its syntax.  That way, the syntax of these unibyte
> characters could be determined from the current equivalency.

Maybe I'm lazy, but I think that it's better to tell those users to not use
unibyte mode.  That will not only save us work, but will also help them
avoid the various other quirks that can show up with unibyte mode.


        Stefan

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <E1CizG7-0006Q0-74@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     Couldn't the information about the character syntax which is used in
>     multibyte mode also be used in unibyte mode?  (I hope this question is
>     not too naive.)

> It is naive, but it might make sense in a way.  We could imagine
> making the case-conversion commands convert each character to
> multibyte and check its syntax.  That way, the syntax of these unibyte
> characters could be determined from the current equivalency.

> What do people think of that idea?
> In particular, Handa, what do you think?

I've already done it in emacs-unicode (in regex.c and
casefiddle.c).

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <jwvoegfnzrd.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  It is naive, but it might make sense in a way.  We could imagine
>>  making the case-conversion commands convert each character to
>>  multibyte and check its syntax.  That way, the syntax of these unibyte
>>  characters could be determined from the current equivalency.

> Maybe I'm lazy, but I think that it's better to tell those users to not use
> unibyte mode.  That will not only save us work, but will also help them
> avoid the various other quirks that can show up with unibyte mode.

I fully agree.

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Richard Stallman]

    > It is naive, but it might make sense in a way.  We could imagine
    > making the case-conversion commands convert each character to
    > multibyte and check its syntax.  That way, the syntax of these unibyte
    > characters could be determined from the current equivalency.

    > What do people think of that idea?
    > In particular, Handa, what do you think?

    I've already done it in emacs-unicode (in regex.c and
    casefiddle.c).

Would you please install it in the current Emacs?

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <E1CjL5l-0000wi-Qv@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>>  It is naive, but it might make sense in a way.  We could imagine
>>  making the case-conversion commands convert each character to
>>  multibyte and check its syntax.  That way, the syntax of these unibyte
>>  characters could be determined from the current equivalency.

>>  What do people think of that idea?
>>  In particular, Handa, what do you think?

>     I've already done it in emacs-unicode (in regex.c and
>     casefiddle.c).

> Would you please install it in the current Emacs?

It seems that I misunderstood the original problem.

I've done that (convert unibyte to multibyte and then check
the syntax) in emacs-unicode because emacs-unicode doesn't
keep syntaxes of unibyte characters in syntax-table
(char-table).

But the current Emacs keeps them in syntax table and updates
them when a language environment is changed in unibyte-mode.
I've just confirmed that 0334 (U-umlaut in Latin-1) has
syntax word-constituent in unibyte-mode in Latin-1
lang. env.  So, I don't understand what is the problem.

And,
>>  making the case-conversion commands convert each character to
>>  multibyte and check its syntax.

Why does case-conversion have to check syntax?

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Ralf Angeli]

* Kenichi Handa (2004-12-29) writes:

> I've done that (convert unibyte to multibyte and then check
> the syntax) in emacs-unicode because emacs-unicode doesn't
> keep syntaxes of unibyte characters in syntax-table
> (char-table).
>
> But the current Emacs keeps them in syntax table and updates
> them when a language environment is changed in unibyte-mode.
> I've just confirmed that 0334 (U-umlaut in Latin-1) has
> syntax word-constituent in unibyte-mode in Latin-1
> lang. env.  So, I don't understand what is the problem.

I've got a latin-1 language environment as well (`C-h v
current-language-environment RET' returns
`current-language-environment's value is "Latin-1"').

Executing the following example code

(with-temp-buffer
  (set-buffer-multibyte nil)
  (insert (string 220))
  (syntax-after (point-min)))

returns (0), i.e. whitespace syntax.  Tested with a freshly checked
out CVS Emacs from the trunk.

-- 
Ralf

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Richard Stallman]

    But the current Emacs keeps them in syntax table and updates
    them when a language environment is changed in unibyte-mode.
    I've just confirmed that 0334 (U-umlaut in Latin-1) has
    syntax word-constituent in unibyte-mode in Latin-1
    lang. env.

Maybe he didn't set the language environment.
What is the situation in the CVS Emacs if you never set the
language environment?  Conversion to multibyte uses Latin-1
by default.

    >>  making the case-conversion commands convert each character to
    >>  multibyte and check its syntax.

    Why does case-conversion have to check syntax?

M-c detects word boundaries with syntax checking.

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <E1Cjkib-0004k5-0L@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     But the current Emacs keeps them in syntax table and updates
>     them when a language environment is changed in unibyte-mode.
>     I've just confirmed that 0334 (U-umlaut in Latin-1) has
>     syntax word-constituent in unibyte-mode in Latin-1
>     lang. env.

> Maybe he didn't set the language environment.
> What is the situation in the CVS Emacs if you never set the
> language environment?

If LANG is not set or is "C", Emacs starts in English
lang. env., and in that case, all 8-bit characters has
whitespace syntax.  In this situation, 0334 is displayed as
\334 (not as U-umlaut).  So, I think it shouldn't have
wordconstituent syntax.

> Conversion to multibyte uses Latin-1 by default.

Yes.  But that conversion is mainly for a user using
multibyte mode.  In unibyte mode
(i.e. default-enable-multibyte-characters is nil),
to-multibyte conversion won't happen usually.

>>>   making the case-conversion commands convert each character to
>>>   multibyte and check its syntax.

>     Why does case-conversion have to check syntax?

> M-c detects word boundaries with syntax checking.

Ah, I see.  By the way, in unibyte English lang. env.,
case-table is also reset to the default, i.e., not set for
latin-1.

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <E1Cja36-0001gs-NV@neutrino.iwi.uni-sb.de>, Ralf Angeli <angeli@iwi.uni-sb.de> writes:
> I've got a latin-1 language environment as well (`C-h v
> current-language-environment RET' returns
> `current-language-environment's value is "Latin-1"').

> Executing the following example code

> (with-temp-buffer
>   (set-buffer-multibyte nil)
>   (insert (string 220))
>   (syntax-after (point-min)))

> returns (0), i.e. whitespace syntax.  Tested with a freshly checked
> out CVS Emacs from the trunk.

I wrote "unibyte-mode" and it's different from a unibyte
buffer in multibyte-mode.  Please try the same thing while
starting Emacs with "--unibyte" arg.

I've guessed that the original problem happened in
unibyte-mode (I didn't get your orignal mail), but it seems
not.  Then, why do you have to use a unibyte buffer in
multibyte mode?

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Ralf Angeli]

* Kenichi Handa (2005-01-06) writes:

> In article <E1Cja36-0001gs-NV@neutrino.iwi.uni-sb.de>, Ralf Angeli <angeli@iwi.uni-sb.de> writes:
>
>> Executing the following example code
>
>> (with-temp-buffer
>>   (set-buffer-multibyte nil)
>>   (insert (string 220))
>>   (syntax-after (point-min)))
>
>> returns (0), i.e. whitespace syntax.  Tested with a freshly checked
>> out CVS Emacs from the trunk.
>
> I wrote "unibyte-mode" and it's different from a unibyte
> buffer in multibyte-mode.  Please try the same thing while
> starting Emacs with "--unibyte" arg.

Hm, then I get word syntax.

> I've guessed that the original problem happened in
> unibyte-mode (I didn't get your orignal mail), but it seems
> not.  Then, why do you have to use a unibyte buffer in
> multibyte mode?

Personally I neither use unibyte mode nor unibyte buffers.  The
original problem was reported by somebody else on bug-gnu-emacs, see
<URL:http://lists.gnu.org/archive/html/bug-gnu-emacs/2004-12/msg00282.html>.

You can see in his report that `default-enable-multibyte-characters'
is nil.  And the user agent string mentions "(with unibyte mode)".  So
if he started Emacs with "--unibyte" and latin-1 characters should
have word syntax in this case, I don't know why `skip-syntax-forward'
which was used in AUCTeX's (and is used in CVS Emacs')
back-to-indentation function skips these characters.

In my follow-up to his original question I suggested to use Emacs in
multibyte mode but I don't know if he got the answer because the email
address in his report is defunct.

-- 
Ralf

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <E1CmSrz-00041u-CR@neutrino.iwi.uni-sb.de>, Ralf Angeli <angeli@iwi.uni-sb.de> writes:

> Personally I neither use unibyte mode nor unibyte buffers.  The
> original problem was reported by somebody else on bug-gnu-emacs, see
> <URL:http://lists.gnu.org/archive/html/bug-gnu-emacs/2004-12/msg00282.html>.

Ah, I now see the whole story.

> You can see in his report that `default-enable-multibyte-characters'
> is nil.  And the user agent string mentions "(with unibyte mode)".  So
> if he started Emacs with "--unibyte" and latin-1 characters should
> have word syntax in this case,

As his LANG is en_US.ISO-8859-15, he should be in latin-9
lang. env.  But...

> I don't know why `skip-syntax-forward'
> which was used in AUCTeX's (and is used in CVS Emacs')
> back-to-indentation function skips these characters.

surely it's strange.  The other strange thing is that his
locale-coding-system is nil.  So, perhaps he is in English
lang. env. somehow.  And, in that case, \334 has whitespace
syntax.

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Stefan Monnier]

> If LANG is not set or is "C", Emacs starts in English
> lang. env., and in that case, all 8-bit characters has
> whitespace syntax.

Isn't that a bug.  Shouldn't it be something safer like punctuation syntax?


        Stefan

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <87r7kyhdgo.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  If LANG is not set or is "C", Emacs starts in English
>>  lang. env., and in that case, all 8-bit characters has
>>  whitespace syntax.

> Isn't that a bug.  Shouldn't it be something safer like punctuation syntax?

Perhaps.  But, C-a, etc. are also whitespace.

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Stefan Monnier]

>>> If LANG is not set or is "C", Emacs starts in English
>>> lang. env., and in that case, all 8-bit characters has
>>> whitespace syntax.

>> Isn't that a bug.  Shouldn't it be something safer like punctuation syntax?

> Perhaps.  But, C-a, etc. are also whitespace.

Huh!!  You're right!!
That sounds very dangerous to me, but it seems like I'm just paranoid,


        Stefan

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> If LANG is not set or is "C", Emacs starts in English
>>>> lang. env., and in that case, all 8-bit characters has
>>>> whitespace syntax.
>
>>> Isn't that a bug.  Shouldn't it be something safer like
>>> punctuation syntax?
>
>> Perhaps.  But, C-a, etc. are also whitespace.
>
> Huh!!  You're right!!  That sounds very dangerous to me, but it
> seems like I'm just paranoid,

If we are being paranoid about licensing issues, we might as well be
paranoid about declaring characters whitespace.  I find it absurd to
assume that in texts any control character should be considered
equivalent to a space when formatting.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Kenichi Handa]

In article <87sm5dfgi8.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>  If LANG is not set or is "C", Emacs starts in English
>>>>  lang. env., and in that case, all 8-bit characters has
>>>>  whitespace syntax.

>>>  Isn't that a bug.  Shouldn't it be something safer like punctuation syntax?

>>  Perhaps.  But, C-a, etc. are also whitespace.

> Huh!!  You're right!!
> That sounds very dangerous to me, but it seems like I'm just paranoid,

I, too, think it's a little bit dangerous.  But, changing
them now will also cause unnecessary danger.  It may be good
to test it in emacs-unicode.

---
Ken'ichi HANDA
handa@m17n.org

Re: command fill-paragraph deletes leading Umlauts if line begins with space

[Stefan]

> I, too, think it's a little bit dangerous.  But, changing
> them now will also cause unnecessary danger.  It may be good
> to test it in emacs-unicode.

100% agreement,


        Stefan