From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Mark H Weaver Newsgroups: gmane.lisp.guile.devel Subject: Re: [PATCH 2/5] [mingw]: Have compiled-file-name produce valid names. Date: Tue, 03 May 2011 23:59:59 -0400 Message-ID: <87wri7ru80.fsf@netris.org> References: <1297784103-18322-1-git-send-email-janneke-list@xs4all.nl> <1297784103-18322-3-git-send-email-janneke-list@xs4all.nl> <87r58gzuoy.fsf@gnu.org> <87vcxsycds.fsf@gnu.org> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1304481642 31333 80.91.229.12 (4 May 2011 04:00:42 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Wed, 4 May 2011 04:00:42 +0000 (UTC) Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , guile-devel@gnu.org To: Andy Wingo Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Wed May 04 06:00:36 2011 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1QHTGN-0001J7-Pw for guile-devel@m.gmane.org; Wed, 04 May 2011 06:00:36 +0200 Original-Received: from localhost ([::1]:45040 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QHTGN-0002Oc-2V for guile-devel@m.gmane.org; Wed, 04 May 2011 00:00:35 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]:44953) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QHTGK-0002OM-6X for guile-devel@gnu.org; Wed, 04 May 2011 00:00:33 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QHTGJ-00039n-3N for guile-devel@gnu.org; Wed, 04 May 2011 00:00:32 -0400 Original-Received: from world.peace.net ([96.39.62.75]:42480) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QHTGH-00038o-3i; Wed, 04 May 2011 00:00:29 -0400 Original-Received: from 209-6-39-128.c3-0.smr-ubr1.sbo-smr.ma.cable.rcn.com ([209.6.39.128] helo=freedomincluded) by world.peace.net with esmtpa (Exim 4.69) (envelope-from ) id 1QHTFz-0000y0-Kz; Wed, 04 May 2011 00:00:11 -0400 Original-Received: from mhw by freedomincluded with local (Exim 4.69) (envelope-from ) id 1QHTFn-0002WQ-K5; Tue, 03 May 2011 23:59:59 -0400 In-Reply-To: (Andy Wingo's message of "Tue, 03 May 2011 09:44:10 +0200") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 96.39.62.75 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:12424 Archived-At: Andy Wingo writes: > That's the crazy thing: file names on GNU aren't in any encoding! They > are byte strings that may or may not decode to a string, given some > encoding. Granted, they're mostly UTF-8 these days, but users have the > darndest files... [...] > On Tue 03 May 2011 00:18, ludo@gnu.org (Ludovic Court=C3=A8s) writes: >> I think GLib and the like expect UTF-8 as the file name encoding and >> complain otherwise, so UTF-8 might be a better default than locale >> encoding (and it=E2=80=99s certainly wiser to be locale-independent.) > > It's more complicated than that. Here's the old interface that they > used, which attempted to treat paths as utf-8: > > http://developer.gnome.org/glib/unstable/glib-Character-Set-Conversion.= html > (search for "file name encoding") > > The new API is abstract, so it allows operations like "get-display-name" > and "get-bytes": > > http://developer.gnome.org/gio/2.28/GFile.html (search for "encoding" > in that page) > > "All GFiles have a basename (get with g_file_get_basename()). These > names are byte strings that are used to identify the file on the > filesystem (relative to its parent directory) and there is no > guarantees that they have any particular charset encoding or even make > any sense at all. If you want to use filenames in a user interface you > should use the display name that you can get by requesting the > G_FILE_ATTRIBUTE_STANDARD_DISPLAY_NAME attribute with > g_file_query_info(). This is guaranteed to be in utf8 and can be used > in a user interface. But always store the real basename or the GFile > to use to actually access the file, because there is no way to go from > a display name to the actual name." In my opinion, this is a bad approach to take in Guile. When developers are careful to robustly handle filenames with invalid encoding, it will lead to overly complex code. More often, when developers write more straightforward code, it will lead to code that works most of the time but fails badly when confronted with weird filenames. This is the same type of problem that plagues Bourne shell scripts. Let's please not go down that road. There is a better way. We can do a variant of what Python 3 does, described in PEP 383 . Basically, the idea is to provide alternative versions of scm_{to,from}_stringn that allow arbitrary bytevectors to be turned into strings and back again without any lossage. These alternative versions would be used for operations involving filenames et al, and should probably also be made available to users. Basically the idea is that "invalid bytes" are mapped to code points that will never appear in any valid encoding. PEP 383 maps such bytes to a range of surrogate code points that are reserved for use in UTF-16 surrogate pairs, and are otherwise considered invalid by Unicode. There are other possible mapping schemes as well. See section 3.7 of Unicode Technical Report #36 for more discussion on this. I can understand why some say that filenames in GNU are not really strings but rather bytevectors. I respectfully disagree. Filenames, environment variables, command-line arguments, etc, are _conceptually_ strings. Let's not muddle that concept just because the transition to Unicode has not yet been completed in the GNU world. Hopefully in the future, these old-style POSIX byte strings will once again become true strings in concept. All that's required for this to happen is for popular software to agree to standardize on the use of UTF-8 for all of these things. This is reasonably likely to happen at some point. In practice, I see no advantage to calling them bytevectors other than to allow lossless storage of oddball filenames. It's not as if any sane user interface is going to display them in hex. Think about it. What are you really going to do with the bytevector version, other than to store it in case you want to convert it back into a filename, environment variable, or command-line argument? Think about the mess that this will make to otherwise simple code. Also think about the obscure bugs that will arise from programmers who balk at this and simply pass around the strings instead. Let's keep things simple. Let's use plain strings for everything that is _conceptually_ a string. Let's instead deal with the occasional ill-encoded-filename by allowing strings to represent these oddballs. Best, Mark