From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:700:3204::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id IPKvAXM2mGW8mQAAkFu2QA (envelope-from ) for ; Fri, 05 Jan 2024 18:03:47 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id 0AZONXI2mGXVBwEAe85BDQ (envelope-from ) for ; Fri, 05 Jan 2024 18:03:47 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1704474226; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=AbQiC/nNEnkxtc+quQ03oHeGFcLTgg6VcqoGCkGDJKY=; b=JsAYHg8zc+wuoZttbeCNloDotHPIrqE9z4gAZ6kYPmYmCLZhEp2+1H0DT6yN/AwdHT8oxw 0VvKQY1EKGpPjQoKRoCn7+CnP1F/HySv/rfcolSj4aiVFH6VTZhW/9ahy7mwCPgKmoCv+f MxhAGdN7yq64fX3Pt2KbVuXr/jfrF7HaFUlq7M1xe5BNqu9bA8tb58CAC36bbzo0R2wkq/ 6oUYHcPkmGMQ73OGYigsCtkqbzY2x3moCV1AD1GOFfi9iGW4CNPu5HeyCx7Vkd/7aaxchU TT/isstE1WvKWVA3WuGCaherWEuPXESmn8zBDacJkIMb2uOGuKscAFD3ZGppSA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1704474226; a=rsa-sha256; cv=none; b=QyYWiDL5NsbaAd2w5TVKHF6ThoGFNSrYz0/8H4la+a1WP7fKgtqfiQGjDQH6sexE/hq2YD 41KXyeMDlsIlVkp0Es6nSG/NfGfmAP0I8W2pGpZ4trZgzaKtODhA6WporUorDSLeSCSh+H LvU+f8y8YvtR/7F+KJTFO1GHw403juetcK813Zfhne50KwYm3ITPaZcqEtnXwdNI3+hYt5 SFHoGIqLPpCDenTY7CPnj+V7K5pCinhwKPbhJcy+1Gc1p19ihRjrL61RM88D+LJlwb/Zck GHjtr/UJ7w8WVB67AIhMNovl1hXycZm4cho3reYP7fR//LMAwAUz2u0r+bSFYQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 94D932069F for ; Fri, 5 Jan 2024 18:03:46 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rLnbQ-0006SC-3X; Fri, 05 Jan 2024 12:03:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rLnbM-0006LI-GZ for guix-devel@gnu.org; Fri, 05 Jan 2024 12:03:21 -0500 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rLnbK-0002IW-Ia; Fri, 05 Jan 2024 12:03:19 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id D1C9027BBE2; Fri, 5 Jan 2024 17:03:14 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id ba3d1559; Fri, 5 Jan 2024 17:03:14 +0000 (UTC) References: <87zg00xvuv.fsf@cbaines.net> <87y1exvj2n.fsf@gnu.org> User-agent: mu4e 1.10.7; emacs 29.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Subject: Re: Performance of computing cross derivations Date: Fri, 05 Jan 2024 16:41:14 +0000 In-reply-to: <87y1exvj2n.fsf@gnu.org> Message-ID: <87le93r9v3.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Queue-Id: 94D932069F X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -8.39 X-Spam-Score: -8.39 X-TUID: JS3fEepVf5TB --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi, > > Christopher Baines skribis: > >> When asked by the data service, it seems to take Guix around 3 minutes >> to compute cross derivations for all packages (to a single >> target). Here's a simple script that replicates this: ... > One idiom that defeats caching is: > > (define (make-me-a-package x y z) > (package > =E2=80=A6)) > > Such a procedure returns a fresh package every time it=E2=80=99s called, > preventing caching from happening (because cache entries are compared > with =E2=80=98eq?=E2=80=99). That typically leads to lower hit rates. > > Anyway, lots of words to say that I don=E2=80=99t see anything immediately > obvious with cross-compilation, yet I wouldn=E2=80=99t be surprised if so= me of > these cache-defeating idioms were used because we=E2=80=99ve payed less > attention to this. I've got a feeling that performance has got worse since I looked at this originally, I've finally got around to having a further look. I spent some time looking at various metrics, but it was most useful to just write the cache keys of various types to files and have a read. The cross-base module was causing many issues, as all but one of the procedures there produced new package records each time. There is also make-rust-sysroot which showed up. I've sent some patches as #68266 to add memoization to avoid this, and that seems to speed things up. Looking at other things in the cache, I think there are some issues with file-append and local-file. The use of file-append in svn-fetch and local-file in the lower procedure in the python build system both bloat the cache for example, although I'm less sure about how to address these cases. One thing I am sure about though, is that these problems will come back. Maybe we could add some reporting in to Guix to look through the cache at the keys, lower them all and check for equivalence. That way it should be possible to automate saying that having [1] in the cache several thousand times is unhelpful. The data service could then run this reporting and store it. 1: # "/bin/svn"> --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmWYNlBfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XcUHg//SQ+sd6IaoiwdkoHQtKX5Xv8n8eskCZy9 RwKsuXMKCY1aoKHwy/MBpEYdVYiI1sHpn72EtELprT3AtXU0gxm1mnh/Fu2Wim7c dL/PIgf2PdSxmjIOjm3+n40VQ29oAEa3eRI+kNUavnPgpGTFZUvoeESY6SxoUITD yh2qhZC5gzQMuVqNNjOzXgSVCvj8UQ0B4P2EMxMCQczk7oMv/bQruw1iyYeqfP06 tDOdMZh5DZjWZ7UaH+YA01LT++ZgD1yNMoGQxaSZ/AzOG0MvKwlGwn6AZI8h/A4C 8WFZTYVafBHdDTTfgfdFXxNoAPOp6UukWkFHbuMQG5l6zj9mcFRTnkm1rWBA7/zW eX0xeSc9FxcnD38PZu5I9yJcJeGsgmrgOfyqKwkcc5XFtZ5r0WZq37e2XxXapBI+ fnGWRUG2d0O5cwIqGIleE80SEO2qqF1ys/olgN+7OdqVo71HyfDG50zNy60GhM4r gqVsAAp5nf3AzHk8Fi4dpjhl+1Q22z8/2waHmR1YrJyA9/bz+NtoyYwxd7J1UFLE 2VmT1AfCaPEy/J47sYyYmZLLBHus5qmaMR5DjCnabp+p7Kw3ERkipjRL8Rnb2Tt+ a++7S885I9cBYPMiwIpHN6FBFb2NRWCoigm9ZIh+NGT7dVLWDhYPoaCgwH1yI84v rmUDM3ySV7o= =i7sb -----END PGP SIGNATURE----- --=-=-=--