Re: [PATCH 2/5] [mingw]: Have compiled-file-name produce valid names.

[Mark H Weaver <mhw@netris.org>]

Andy Wingo <wingo@pobox.com> writes:
> That's the crazy thing: file names on GNU aren't in any encoding!  They
> are byte strings that may or may not decode to a string, given some
> encoding.  Granted, they're mostly UTF-8 these days, but users have the
> darndest files...
[...]
> On Tue 03 May 2011 00:18, ludo@gnu.org (Ludovic Courtès) writes:
>> I think GLib and the like expect UTF-8 as the file name encoding and
>> complain otherwise, so UTF-8 might be a better default than locale
>> encoding (and it’s certainly wiser to be locale-independent.)
>
> It's more complicated than that.  Here's the old interface that they
> used, which attempted to treat paths as utf-8:
>
>   http://developer.gnome.org/glib/unstable/glib-Character-Set-Conversion.html
>   (search for "file name encoding")
>
> The new API is abstract, so it allows operations like "get-display-name"
> and "get-bytes":
>
>   http://developer.gnome.org/gio/2.28/GFile.html  (search for "encoding"
>   in that page)
>
>   "All GFiles have a basename (get with g_file_get_basename()). These
>   names are byte strings that are used to identify the file on the
>   filesystem (relative to its parent directory) and there is no
>   guarantees that they have any particular charset encoding or even make
>   any sense at all. If you want to use filenames in a user interface you
>   should use the display name that you can get by requesting the
>   G_FILE_ATTRIBUTE_STANDARD_DISPLAY_NAME attribute with
>   g_file_query_info(). This is guaranteed to be in utf8 and can be used
>   in a user interface. But always store the real basename or the GFile
>   to use to actually access the file, because there is no way to go from
>   a display name to the actual name."

In my opinion, this is a bad approach to take in Guile.  When developers
are careful to robustly handle filenames with invalid encoding, it will
lead to overly complex code.  More often, when developers write more
straightforward code, it will lead to code that works most of the time
but fails badly when confronted with weird filenames.  This is the same
type of problem that plagues Bourne shell scripts.  Let's please not go
down that road.

There is a better way.  We can do a variant of what Python 3 does,
described in PEP 383 <http://www.python.org/dev/peps/pep-0383/>.

Basically, the idea is to provide alternative versions of
scm_{to,from}_stringn that allow arbitrary bytevectors to be turned into
strings and back again without any lossage.  These alternative versions
would be used for operations involving filenames et al, and should
probably also be made available to users.

Basically the idea is that "invalid bytes" are mapped to code points
that will never appear in any valid encoding.  PEP 383 maps such bytes
to a range of surrogate code points that are reserved for use in UTF-16
surrogate pairs, and are otherwise considered invalid by Unicode.  There
are other possible mapping schemes as well.  See section 3.7 of Unicode
Technical Report #36 <http://www.unicode.org/reports/tr36/> for more
discussion on this.

I can understand why some say that filenames in GNU are not really
strings but rather bytevectors.  I respectfully disagree.  Filenames,
environment variables, command-line arguments, etc, are _conceptually_
strings.  Let's not muddle that concept just because the transition to
Unicode has not yet been completed in the GNU world.

Hopefully in the future, these old-style POSIX byte strings will once
again become true strings in concept.  All that's required for this to
happen is for popular software to agree to standardize on the use of
UTF-8 for all of these things.  This is reasonably likely to happen at
some point.

In practice, I see no advantage to calling them bytevectors other than
to allow lossless storage of oddball filenames.  It's not as if any sane
user interface is going to display them in hex.  Think about it.  What
are you really going to do with the bytevector version, other than to
store it in case you want to convert it back into a filename,
environment variable, or command-line argument?  Think about the mess
that this will make to otherwise simple code.  Also think about the
obscure bugs that will arise from programmers who balk at this and
simply pass around the strings instead.

Let's keep things simple.  Let's use plain strings for everything that
is _conceptually_ a string.  Let's instead deal with the occasional
ill-encoded-filename by allowing strings to represent these oddballs.

     Best,
      Mark