getpwent, user-full-name and utf-8

getpwent, user-full-name and utf-8

[David Kastrup]

Hi,

user-full-name is set using getpwentry without decoding the resulting
byte string at all.

The manual page of getpwent does not mention any encoding of
/etc/passwd, neither does that of /etc/passwd.

It is a safe bet, however, that /etc/passwd is not encoded in
emacs-mule.

Since different users may use different language environments, I
propose that we decode the results from getpwent according to utf-8.

There will likely be similar problems with other system functions
(name server lookup?).  emacs-mule certainly is not the right answer
to the encoding problem.  And the problem will persist with
emacs-unicode2 as well since there is a difference between illegal
byte sequences and decoded illegal byte sequences.

I propose that we bite the bullet, assume a fixed external system
encoding of utf-8 for such strings, and recode accordingly.

-- 
David Kastrup

Re: getpwent, user-full-name and utf-8

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Wed, 21 Mar 2007 10:58:08 +0100
> 
> I propose that we bite the bullet, assume a fixed external system
> encoding of utf-8 for such strings, and recode accordingly.

I'd rather assume that usernames are encoded in the locale's encoding,
not necessarily in UTF-8.

Re: getpwent, user-full-name and utf-8

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Wed, 21 Mar 2007 10:58:08 +0100
>> 
>> I propose that we bite the bullet, assume a fixed external system
>> encoding of utf-8 for such strings, and recode accordingly.
>
> I'd rather assume that usernames are encoded in the locale's
> encoding, not necessarily in UTF-8.

That assumes that every user operates under the same locale, and that
this locale agrees with the locale of the system files.  In particular
on multi-user machines, that is not realistic.

It might be reasonable to add a new variable to hold the system locale
which should not depend on the user locale.  However, it is somewhat
late for this.  Clearly, assuming emacs-mule encoding for the system,
as it now appears the case, is always wrong.

For current systems, assuming utf-8 will likely be correct most of the
time, at least.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: getpwent, user-full-name and utf-8

[Miles Bader]

David Kastrup <dak@gnu.org> writes:
> Eli Zaretskii <eliz@gnu.org> writes:
>> I'd rather assume that usernames are encoded in the locale's
>> encoding, not necessarily in UTF-8.
...
> That assumes that every user operates under the same locale, and that
> this locale agrees with the locale of the system files.  In particular
> on multi-user machines, that is not realistic.
...
> For current systems, assuming utf-8 will likely be correct most of the
> time, at least.

Do you have any data to back that up?

If you think of multiuser systems versus single-user systems, I'd think:

  * On a single-user systems, the user's locale would often match
    /etc/passwd.

  * Multi-user systems tend to be much longer-lived (I think much of the
    data on the servers at my work dates back 15 years or more -- often
    the hardware gets upgraded, but the user-related data is just kept
    verbatim from the old system), and in many cases probably have user
    databases that predate widespread use of utf-8.  In europe I guess
    that would mean they use latin-XX.

There's really no way you can always get it right, but my intuition is
that the safest thing to do is use the locale as Eli suggests.

Of course you're right that emacs-mule is basically never
correct... (maybe there are some crazies out there though :-)

-Miles

-- 
Is it true that nothing can be known?  If so how do we know this?  -Woody Allen

Re: getpwent, user-full-name and utf-8

[Jan Djärv]

David Kastrup skrev:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> From: David Kastrup <dak@gnu.org>
>>> Date: Wed, 21 Mar 2007 10:58:08 +0100
>>>
>>> I propose that we bite the bullet, assume a fixed external system
>>> encoding of utf-8 for such strings, and recode accordingly.
>> I'd rather assume that usernames are encoded in the locale's
>> encoding, not necessarily in UTF-8.
> 
> That assumes that every user operates under the same locale, and that
> this locale agrees with the locale of the system files.  In particular
> on multi-user machines, that is not realistic.

Since users themselves can set their full name, I'd think the user locale 
would be a good choice.

> 
> It might be reasonable to add a new variable to hold the system locale
> which should not depend on the user locale.  However, it is somewhat
> late for this.  Clearly, assuming emacs-mule encoding for the system,
> as it now appears the case, is always wrong.
> 
> For current systems, assuming utf-8 will likely be correct most of the
> time, at least.

UTF-8 is much better than emacs-mule.  If it is not too much work, I'd suggest 
checking if the name is valid UTF-8, and if it isn't, use the user locale.

	Jan D.

Re: getpwent, user-full-name and utf-8

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> David Kastrup skrev:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: David Kastrup <dak@gnu.org>
>>>> Date: Wed, 21 Mar 2007 10:58:08 +0100
>>>>
>>>> I propose that we bite the bullet, assume a fixed external system
>>>> encoding of utf-8 for such strings, and recode accordingly.
>>> I'd rather assume that usernames are encoded in the locale's
>>> encoding, not necessarily in UTF-8.
>>
>> That assumes that every user operates under the same locale, and that
>> this locale agrees with the locale of the system files.  In particular
>> on multi-user machines, that is not realistic.
>
> Since users themselves can set their full name,

Since when?

Well, I took a look at the manual page of passwd(1) on my GNU/Linux
system, and the description indeed says:

	passwd also changes account information, such as the full
	name of the user, the user's login shell, or his/her
	password expiry date and interval.

Amusingly, however, there is no option for doing any of that except
the password related stuff.

There is, however,

CHFN(1) -- 06/06/2006 -- User Commands

NAME
	chfn - change real user name and information

SYNOPSIS

	chfn [-f full_name] [-r room_no] [-w work_ph] [-h home_ph]
		[-o other] [user]

DESCRIPTION

	chfn changes user fullname, office number, office extension,
	and home phone number information for a user's account. This
	information is typically printed by finger(1) and similar
	programs. A normal user may only change the fields for her
	own account, subject to the restrictions in
	/etc/login.defs. (The default configuration is to prevent
	users from changing their fullname.) The super user may
	change any field for any account. Additionally, only the
	super user may use the -o option to change the undefined
	portions of the GECOS field.

	The only restriction placed on the contents of the fields is
	that no control characters may be present, nor any of comma,
	colon, or equal sign. The other field does not have this
	restriction, and is used to store accounting information
	used by other applications.

So with the default settings, a user can't change his settings.

> I'd think the user locale would be a good choice.

It is not the worst choice, but I consider it likely that a better way
would be something like a separate "system locale".  For lack of
better information, one could let it default to the user locale, but
it should be at least configurable separately.

Does anybody have access to the X/Open or Posix specs?  Maybe
something is said about this in there.

-- 
David Kastrup

Re: getpwent, user-full-name and utf-8

[Jan Djärv]

David Kastrup skrev:
> Jan Djärv <jan.h.d@swipnet.se> writes:
> 
>> David Kastrup skrev:
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: David Kastrup <dak@gnu.org>
>>>>> Date: Wed, 21 Mar 2007 10:58:08 +0100
>>>>>
>>>>> I propose that we bite the bullet, assume a fixed external system
>>>>> encoding of utf-8 for such strings, and recode accordingly.
>>>> I'd rather assume that usernames are encoded in the locale's
>>>> encoding, not necessarily in UTF-8.
>>> That assumes that every user operates under the same locale, and that
>>> this locale agrees with the locale of the system files.  In particular
>>> on multi-user machines, that is not realistic.
>> Since users themselves can set their full name,
> 
> Since when?
> 
> Well, I took a look at the manual page of passwd(1) on my GNU/Linux
> system, and the description indeed says:
> 
> 	passwd also changes account information, such as the full
> 	name of the user, the user's login shell, or his/her
> 	password expiry date and interval.
> 
> Amusingly, however, there is no option for doing any of that except
> the password related stuff.
> 
> There is, however,
> 
> CHFN(1) -- 06/06/2006 -- User Commands

Yes that is what I meant.

> 
...
> 	/etc/login.defs. (The default configuration is to prevent
> 	users from changing their fullname.)



> 
> So with the default settings, a user can't change his settings.

Ok, I haven't seen that restriction before.

> 
>> I'd think the user locale would be a good choice.
> 
> It is not the worst choice, but I consider it likely that a better way
> would be something like a separate "system locale".  For lack of
> better information, one could let it default to the user locale, but
> it should be at least configurable separately.
> 
> Does anybody have access to the X/Open or Posix specs?  Maybe
> something is said about this in there.
> 

They don't say anything, they don't specify any passwd or chfn program.  The 
description of <pwd.h> just says:

   The <pwd.h> header shall provide a definition for struct passwd, which shall
   include at least the following members:

   char    *pw_name   User's login name.
...

and that is about it.

I checked some of the machines I have accounts on, but no admin access to, and 
they all use Latin-1 (which BTW is not my locale, UTF-8 is).  So some system 
locale may not be a bad idea, but default UTF-8 is OK.

	Jan D.

Re: getpwent, user-full-name and utf-8

[Jan Djärv]

Jan Djärv skrev:

> 
> They don't say anything, they don't specify any passwd or chfn program.  
> The description of <pwd.h> just says:
> 
>   The <pwd.h> header shall provide a definition for struct passwd, which 
> shall
>   include at least the following members:
> 
>   char    *pw_name   User's login name.
> ...
> 
> and that is about it.
> 

I.e. they don't specify a GECOS field at all.

	Jan D.

Re: getpwent, user-full-name and utf-8

[Richard Stallman]

    I propose that we bite the bullet, assume a fixed external system
    encoding of utf-8 for such strings, and recode accordingly.

Since this is a matter of interacting with the rest of the system, the
pertinent question is not how people OUGHT to encode /etc/passwd, but
rather how they DO encode it.

Is there a common practice of encoding parts of /etc/passwd in utf-8?