From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#22990: Grafts leads to inefficient substitute info retrieval Date: Fri, 11 Mar 2016 17:52:59 +0100 Message-ID: <8737rxx8gk.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:58069) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeQKD-0003GW-3v for bug-guix@gnu.org; Fri, 11 Mar 2016 11:54:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aeQKA-0002Nv-E1 for bug-guix@gnu.org; Fri, 11 Mar 2016 11:54:05 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:48221) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeQKA-0002Np-AK for bug-guix@gnu.org; Fri, 11 Mar 2016 11:54:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84) (envelope-from ) id 1aeQKA-0001hF-2r for bug-guix@gnu.org; Fri, 11 Mar 2016 11:54:02 -0500 Sender: "Debbugs-submit" Resent-Message-ID: Received: from eggs.gnu.org ([2001:4830:134:3::10]:57899) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeQJE-0003BY-PZ for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aeQJC-0001sH-3V for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:04 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:48889) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeQJC-0001sD-0d for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:02 -0500 Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:49892 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1aeQJB-0005Gm-Eo for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:01 -0500 List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org To: 22990@debbugs.gnu.org As of right now (v0.9.0-2007-g66a30a3), =E2=80=98graft-derivation=E2=80=99 = works either by: 1. Fetching substitute info about the things being built so that it can determine its references, which in turns allows it to determine whether they need to be grafted. 2. Building stuff, as a last resort, so that it can determine its references. Case #1 is hopefully going to be the most common. The problem with #1 is that when building a profile, we do one =E2=80=98package-derivation=E2=80=99 call for each package in the profile, = which translates in one =E2=80=98graft-derivation=E2=80=99 call for each relevant= package=C2=B9, which translates into one =E2=80=98references/substitutes=E2=80=99 call for= each. Concretely, what this means is this: $ guix package -u substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% [=E2=80=A6] The following files would be downloaded: Each of the initial =E2=80=9Cupdating list=E2=80=9D message corresponds to = an HTTP request for a single narinfo file, which can take around 1 second. Instead, the ideal thing would be to fetch the narinfo files for all the relevant packages at once; that way, we=E2=80=99d spawn =E2=80=98guix subst= itute=E2=80=99 only once, and it would benefit from HTTP pipelining (one round-trip instead of N.) To achieve this, I=E2=80=99m thinking of extending gexp code such that gexp compilers can return a list of applicable grafts. The =E2=80=98package=E2= =80=99 compiler would do #:graft? #f and instead let =E2=80=98gexp->derivation=E2= =80=99 call =E2=80=98graft-derivation=E2=80=99. I=E2=80=99ll give it a try and report back. Ludo=E2=80=99. =C2=B9 A package is =E2=80=9Crelevant=E2=80=9D if =E2=80=98package-grafts= =E2=80=99 returns a non-empty list.