bug#26353: VFS name encoding

[Danny Milosavljevic <dannym@scratchpost.org>]

Hi Ludo,

> The problem of how to deal with file name encoding has been discussed on
> the Guile side so hopefully the next release in the 2.2 series will have
> a solution for this.

For what it's worth, I think the sane solution is the Plan 9 solution:  Just represent file names as bytevectors.  Programs which don't care about the actual name - for example programs that just want to do (for-each unlink (scandir (string->utf8 "."))) or something - have no reason to care about the encoding at all.  And then use UTF-8 encoding everywhere (for the file names, also for everything else) throughout the operating system for the tools that do care.

There are also utf8 mount options in the Linux kernel to be able to present UTF-8 names to userspace even when the actual names on disk are something else - and we should use them.  (I think we should even modify <file-system> flags to default to "utf8" or "iocharset=utf8" where possible)

This conversion of UTF-8 to UCS-4 especially is really just busywork.  My opinion changed over the years - earlier I was all for UCS-4.  But actually, most tools don't care about the actual content of the file names - it's just an opaque ID to them (similar to an UUID).  Representing them as something else in userspace again (inviting another conversion failure) is just ... unnecessary. 

In any case, it would be different if we had a non-UNIX kernel underneath.  But as long as we do have UNIX the kernel VFS interface expects bytevectors, preferrably interpreted as UTF-8 (if interpreted at all).

I think this is also the consensus among the major Linux distributions and also among lowlevel libraries like glib: They assume one is using UTF-8 filenames and default to it whereever possible.