bug#57531: 28.1; Character encoding missing for "eo"

[Jonathan Reeve <jonathan@jonreeve.com>]

> I believe this is due to the fact that the text was saved in UTF-8,
> and Emacs was trying to decode it as if it were in Latin-3.

That’s the problem. Emacs should decode UTF-8 as UTF-8, not Latin-3. The fact that it assumes a UTF-8 locale is in fact a Latin-3 locale, without any reasoning for that, is a problem.

> Using the prefer-coding-system customization should fix that.

The user shouldn’t have to customize an encoding system to have a UTF-8 locale be interpreted as UTF-8. A UTF-8 locale should be encoded as such, without needing to be told otherwise.

> I disagree.  I think your system doesn’t tell Emacs enough to guess
> correctly.

It does, though. The locale data is already there, which is why I only have this problem in Emacs, and nowhere else on my whole system. The problem is in this line from `locale-language-names'. Here’s what it says:

`("eo" . "Esperanto")'

Here’s what it should say:

`("eo" "Esperanto" utf-8)'

> There’s no evidence of “eo” being a UTF-8 locale, except what we see
> in glibc.  Which is just one library on just one OS.

The evidence is everywhere, in fact, across my whole system, and every other system I’ve used. And glibc is not just one library on one OS: it’s the reference data for locales across any UNIX-like or POSIX system.

Show me any other library on any other OS that has locale data that suggests that “eo” is anything other than UTF-8. In particular, show me a library that shows that the eo locale should be encoded as latin-3.

> Emacs cannot know the system character set unless the system tells
> that.  The way to tell that is via the locale’s codeset.  If that is
> impossible, the next best is for you to tell that to Emacs in your
> init file.  I don’t understand why you insist on not using the
> solution I proposed.

The system says that it’s a UTF-8 locale. It’s interpreted as a UTF-8 locale by every other program except for emacs. It’s only emacs that incorrectly assumes latin-3, and for no reason, as far as I can tell. That’s because it’s getting its locale/encoding information from `locale-language-names', which is incorrect, or at least incomplete.

> Please try the solution I proposed, and if it doesn’t work, let’s see
> what else is needed.  If you keep insisting on defaulting Esperanto to
> UTF-8, I see now way to make any progress here.

You’re not proposing a solution, you’re proposing a workaround. Any other user with the “eo” locale will have this same problem, and they shouldn’t be expected to find this email thread, in order to find a special hack to have their system work as expected.

There’s no reason at all why Esperanto should be encoded in latin-3. It never has been, as far as I can tell, and it never well be, in latin-3, with the eo locale. If you can find any good reason why it should be in latin-3, I’m all ears, but as far as I can tell, this is a mistake.

Please keep in mind that /I’m trying to help improve emacs,/ by submitting a bug report about behavior in emacs that is incorrect and potentially causing problems for many users. Language like “I see now way to make any progress here” doesn’t extend any courtesy, any effort towards understanding this problem, or even any effort towards giving it the benefit of the doubt. It makes the bug reporting process unnecessarily adversarial, and, quite frankly, feels unprofessional.