From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: store reference detection (was Re: JARs and reference scanning) Date: Thu, 27 Apr 2017 15:46:59 +0200 Message-ID: <87fugu6jzg.fsf@gnu.org> References: <87a876pwaq.fsf@gmail.com> <8760hr7mwl.fsf@gmail.com> <20170426.135333.1620868924745053745.post@thomasdanckaert.be> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:52923) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d3jlh-0005uC-8D for guix-devel@gnu.org; Thu, 27 Apr 2017 09:47:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d3jld-0004in-6O for guix-devel@gnu.org; Thu, 27 Apr 2017 09:47:37 -0400 In-Reply-To: <20170426.135333.1620868924745053745.post@thomasdanckaert.be> (Thomas Danckaert's message of "Wed, 26 Apr 2017 13:53:33 +0200 (CEST)") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Thomas Danckaert Cc: guix-devel@gnu.org Hello, Thomas Danckaert skribis: > From: Chris Marusich > Subject: Re: JARs and reference scanning > Date: Tue, 25 Apr 2017 22:34:02 -0700 > >>> I have to admit that I do not know at all how the reference >>> scanning and >>> dependency-tracking in the store works. >> >> As I understand it, the mechanism used by the Guix daemon (and the >> Nix >> daemon) for scanning references doesn't work when the output of a >> derivation is scrambled in some way (e.g. compressed). Therefore, >> if we >> use JAR files, they should not be compressed. > > The code scanning for reference is in nix/libstore/references.cc . It > looks for base32 hashes encoded as character strings in the binaries. > > Could/should this be generalized somehow? Apart from compression, > store filenames encoded with 16-bit character encodings also cause > problems (can happen with Qt or WxWidgets). And the are probably more > cases where it fails. Really? Qt/WxWidgets =E2=80=9Chide=E2=80=9D store references by default? > Does it make sense to expand the reference detecting code (perhaps > this would lead to too many different special cases?), or maybe > provide a mechanism to force references when the daemon cannot detect > them. I suppose you can always add a text file with a list of store > items to the output, but maybe there's a more elegant way? =E2=80=98propagated-inputs=E2=80=99 is one way to manually specify run-time= references. It works at the package level and not at the store level=E2=80=94that is, t= he store item=E2=80=99s references are unaffected by what =E2=80=98propagated-= inputs=E2=80=99 contains. It=E2=80=99s usually enough for our purposes though. In that sense, I think jars are comparable to Python/Perl libraries, which do not carry reference information by themselves and thus need a manually-provided =E2=80=98propagated-inputs=E2=80=99. WDYT? Ludo=E2=80=99.