Re: Improving the handling of system data (env, users, paths, ...)

[Eli Zaretskii <eliz@gnu.org>]

> From: Jean Abou Samra <jean@abou-samra.fr>
> Cc: rlb@defaultvalue.org, guile-devel@gnu.org
> Date: Sun, 07 Jul 2024 17:16:31 +0200
> 
> There are non-trivial backwards compatibility implications. To give
> a concrete case: LilyPond definitely has code that would break if
> passed a string whose "conversion to UTF-8" gave something not valid
> UTF-8. (An example off the top: passing strings to the Pango API and to
> GLib's PCRE-based regex API. By the way, running "emacs $'\xb5'"
> gives a Pango warning on the terminal, I assume because of trying
> to display the file name as the window title.)

Probably.  Do you consider it a problem in Emacs or in Pango?

> From the implementation point of view: conversion from an encoding to
> another could no longer use libiconv, because it stops on invalid
> multibyte sequences. Likewise, Guile could probably not use libiconv
> anymore. This means a large implementation cost to reimplement all
> of this in Guile.

Or relatively small additions to libiconv, should their developers
agree with such an extension.

> I don't think it's worth it. If anybody's going to work on this problem,
> I'd recommend simply adding APIs like program-arguments-bytevector,
> getenv-bytevector and the like, returning raw bytevectors instead of strings,
> and letting programs which need to be reliable against invalid UTF-8
> in the environment use these.
> 
> That is also the approach taken in, e.g., Rust (except that due to the
> static typing, you are forced to handle the "invalid UTF-8" error case
> when you use, e.g., std::env::args as opposed to std::env::args_os).

The Emacs experience shows that (rare) raw bytes as part of otherwise
completely valid text are a fact of life.  They happen all the time,
for whatever reasons.  Granted, those reasons are most probably
something misconfigured somewhere, but as long as that happens in a
program other than the one you are developing, or even on another
computer, the ability of the user, let alone the programmer, to fix
the whole world is, how shall I put it, somewhat limited.  The
question is what do you when this stuff happens, and how you prepare
your package for dealing with it as well as reasonably possible?

Here's an example just from today: I've received an email from RMS, no
less, with obviously garbled address:

  To: Björn Bidar <bjorn.bidar@thaodan.de>

Now, this is a typical case of misinterpreting UTF-8 as Latin-1 (on
RMS's machine, not on mine); the correct name is Björn Bidar.  But
when you get such mojibake from your MTA, what do you do? signal an
error and refuse to show the message?  Good luck explaining to your
users that you are right behaving like that!  We in Emacs decided
differently, but that's us.

Once again, I described what we do in Emacs in the hope that it will
help you find your own solution.  If it doesn't help, that's fine by
me; there's no need to argue as long as what we do is understood.