From mboxrd@z Thu Jan 1 00:00:00 1970 From: Danny Milosavljevic Subject: bug#33848: Store references in SBCL-compiled code are "invisible" Date: Thu, 27 Dec 2018 14:52:58 +0100 Message-ID: <20181227145258.0c420eac@scratchpost.org> References: <87r2e8jpfx.fsf@gnu.org> <877eg0i43j.fsf@netris.org> <87d0psi1xo.fsf@gnu.org> <874lb3kin6.fsf@ambrevar.xyz> <87sgynezha.fsf@gnu.org> <87tvj2yesd.fsf@netris.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/psKMAB3Tik_KCO1Se691Qt1"; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([208.118.235.92]:40652) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gcW6x-00052A-4k for bug-guix@gnu.org; Thu, 27 Dec 2018 08:54:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gcW6s-0001PU-64 for bug-guix@gnu.org; Thu, 27 Dec 2018 08:54:07 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:37475) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gcW6s-0001Nx-0i for bug-guix@gnu.org; Thu, 27 Dec 2018 08:54:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gcW6r-0000Tm-QV for bug-guix@gnu.org; Thu, 27 Dec 2018 08:54:01 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87tvj2yesd.fsf@netris.org> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Mark H Weaver Cc: Pierre Neidhardt , 33848@debbugs.gnu.org --Sig_/psKMAB3Tik_KCO1Se691Qt1 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi Mark, On Mon, 24 Dec 2018 13:12:23 -0500 Mark H Weaver wrote: > Of course, the usual reason to choose UTF-32 is to support non-ASCII > characters while retaining fixed-width code points, so that string > lookups are straightforward and efficient. This kind of lookup is almost never what is necessary. There are many people who assume character is the same as codepoint and to those people UTF-32 brings something to the table, but it's really not useful if people do text processing correctly, see below. (Of course whether packages actually do this remains to be seen) > Using UTF-8 improves space efficiency, but at the cost of extra code >complexity. I agree. > That extra > complexity is what I guess we would need to add to each program that > currently uses UTF-32. Yes, but they usually have to do stream processing even with UTF-32 (because a character can be composed of possibly infinite number of codepoints), so the infrastructure should be already there and the effort should be minimal. > Alternatively, we could extend the on-disk > format to support UTF-8 and then add some kind of "load hook" that > converts the string to UTF-32 at load time. Either way, it's likely to > be a can of worms. If it ever came to that, a pluggable reference scanner would be=20 preferrable. But really, it would irk me to have so much complexity in something so basic (the reference scanner) for no end-user gain (as a distribution we could just mandate UTF-8 for references and the problem would be gone for the user with no loss of functionality). It's always easy to add special cases - but more code means more bugs and I think if possible it's best to have only the simple case implemented in the core - because it's less complicated which means more likely to be correct (for the case it does handle). In the end it depends on what would be more code, and more widely used. Also, if we wanted to debug reference errors, we couldn't use grep anymore because it can't handle utf-32 either (neither can any of the other UNIX to= ols). Also, I really don't want to return to the time where I had to call iconv once every three commands to be able to do anything useful on UNIX. Also, the build daemon is written in C++ and C++ strings are widely known to have very very bad codepoint awareness (to say nothing about the horrible conversion facilities). Also, if both UTF-32 and UTF-8 are used on disk, care needs to not misdetect an UTF-8 sequence as an UTF-32 sequence of different text - or the other way around -, but that's unlikely for ASCII strings. > I really think it would be a mistake to try to force every program and > language implementation to use our preferred string representation. I > suspect it would be vastly easier to compromise and support a few other > popular string representations in Guix, namely UTF-16 and UTF-32. In 1992, UTF-8 was invented. Subsequently, most of the Internet, all new GNU Linux distributions etc, all UNIX GUI frameworks, Subversion etc standardized on UTF-8, with the eventual goal of standardizing all network transfer and storage to UTF-8. I think that by now the outliers are the ones who need to change, otherwise these senseless encoding conversions will never cease. It's not like different encodings allow for better expression of writings or anything useful to the end user. As a distribution we can't force upstream to change, but just filing bug reports upstream would make us see where they stand on this. > If you don't want to change the daemon, it could be worked around in our > build-side code as follows: we could add a new phase to certain build > systems (or possibly gnu-build-system) that scans each output for > UTF-16/32 encoded store references that are never referenced in UTF-8. > If such references exist, a file with an unobtrusive name would be added > to that output containing those references encoded in UTF-8. This would > enable our daemon's existing reference scanner to find all of the > references. I agree that that would be nice. As a first step, even just detecting problems like that and erroring out would be okay - in order to find them in the first place. Right now, it's difficult to detect and so also diffic= ult to say how wide-spread the problem is. If the problem is wide-spread enough my tune could change very quickly. What you propose is similar to what I did in Java in Guix, only it gives us even more advantages in the Java case (faster class loading and eventual non-propagated inputs). --Sig_/psKMAB3Tik_KCO1Se691Qt1 Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlwk2ToACgkQ5xo1VCww uqUUzAgApbxUHv/XlbjYMXvV4cOY0maxbx92ndZlJiukCN+bIiMqhuCd7PdEoL7Z 1d9ABxe+2oXO4Nkjpez71nhK8ym8KwRYNDuTkCSZbzUJwNEee2pF/OlU2Y+Jugz5 ICSlYGCFfwx6Buf9bZReYq1e5qjO//QSytgYC061gYURw/abtGSEyvllHWv4qrl6 DFfQuQilycHAOqrT/ACBtMgFFnsV7miHs6CrKTSPPBWKKuA3BM4STNUfHlMeb8un gNUH3ijbDviqBgRiqDy50dZ0kbFv8zSm1LytoySX0qZ7j5oidDJGHATGbEXp4sDU 1dPmBSYeasLAVfJn4RQSQFAKba6TuA== =gkV6 -----END PGP SIGNATURE----- --Sig_/psKMAB3Tik_KCO1Se691Qt1--