From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id 0AN3D5+PgGQtUwEASxT56A (envelope-from ) for ; Wed, 07 Jun 2023 16:09:35 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id YCsuD5+PgGTpdwEAauVa8A (envelope-from ) for ; Wed, 07 Jun 2023 16:09:35 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 817193B7FE for ; Wed, 7 Jun 2023 16:09:34 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6tpx-0006rz-60; Wed, 07 Jun 2023 10:08:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6tpr-0006r6-MU for guix-devel@gnu.org; Wed, 07 Jun 2023 10:08:27 -0400 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6tpp-0001Wa-RR for guix-devel@gnu.org; Wed, 07 Jun 2023 10:08:27 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:3a91:a0a4:ecee:f157]) by mira.cbaines.net (Postfix) with ESMTPSA id 2062C27BBE2; Wed, 7 Jun 2023 15:08:24 +0100 (BST) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id d1501177; Wed, 7 Jun 2023 14:08:19 +0000 (UTC) References: <168560801502.2735.15747711991420829133@vcs2.savannah.gnu.org> <20230601082655.B4449C23EC5@vcs2.savannah.gnu.org> <87wn0l5xkk.fsf@cbaines.net> <87ttvo49wh.fsf@cbaines.net> User-agent: mu4e 1.10.2; emacs 28.2 From: Christopher Baines To: Efraim Flashner Cc: guix-devel@gnu.org Subject: Re: 02/03: gnu: openblas: Update architectures we provide substitutes for. Date: Wed, 07 Jun 2023 14:54:38 +0100 In-reply-to: Message-ID: <87bkhr1i26.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1686146975; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=rQJkVLxEgCdWXemVgSY4/7QugcTy30juXjJNy24d0VM=; b=NqXYMEAdjV8LQg+tp1L3TWt60bVxQ2RQfl7px1j3Ws2N0XaICXuFtvuo968Spz3hiYv7sD vJb6wA+bWz+CJ8e8DneAPrmZ4msQ+2SslR+KPS85ytScbn67XzclsZRUO+AQ9IHCZM0SlB nljQ4WKV2joYAhO0EIukKKFmkPr9e6THQPlyRHpVyVGLBjPuw39Py3cp3W5M5tzXcjlbNm hbewy1dXrsNwQ+ZQ05gZoby6o593Uvj8IqEJeuoGoHWrOzmi9WX03X9ZrgEOqKcr4qmw/Y w48AeYVW9JFOuZsnNdbrFP7crGaKqKGmk80r0xuUgaqlhy/M02Ud8P6KfUuiMQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1686146975; a=rsa-sha256; cv=none; b=M6BqzsBQUFgYOFKxQpRoyUuiIq9mCy5iE1AZDQogsnX2W22H42t4j2TNG5VMM2E9cDWWxh 1rd61WoWRIHYqBZrx7IekGB7DDfl7mTSJKsRHsLDBiA6Mhgiz4T5mxbIeWzcWVsHI6cRrC P5m3k/12Id0c6wgnU7vuuXR7uSlyMEPIJqTEdvR09hb2M7wC1tIXacZHRi8ObE83RicBx2 mfikzGQFwmlkY/EiUUb58/5Ac8FIzpTWNVc+pSH8/yORAvttJjiB5GtVVNymjyEjkqk6FE Yu7rhiuzFGNA7woOOHv8+gnk5ButrhLe8DQGBjI9yKlPezo5iK8Fn1F3V3Uv2g== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -5.10 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 817193B7FE X-Spam-Score: -5.10 X-TUID: Q67bHoQacy5E --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Efraim Flashner writes: > On Sat, Jun 03, 2023 at 08:12:48PM +0100, Christopher Baines wrote: >>=20 >> Efraim Flashner writes: >>=20 >> > On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote: >> >>=20 >> >> guix-commits@gnu.org writes: >> >>=20 >> >> > efraim pushed a commit to branch master >> >> > in repository guix. >> >> > >> >> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482 >> >> > Author: Efraim Flashner >> >> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300 >> >> > >> >> > gnu: openblas: Update architectures we provide substitutes for. >> >> > >> >> > * gnu/packages/maths.scm (openblas)[arguments]: Adjust the subs= titutable? >> >> > flag to only not provide substitutes when building for powerpc-= linux. >> >> > Adjust the comment accordingly. >> >> > --- >> >> > gnu/packages/maths.scm | 11 ++--------- >> >> > 1 file changed, 2 insertions(+), 9 deletions(-) >> >>=20 >> >> I've been looking at why armhf-linux substitute availability has been >> >> dropping recently, and I think this change triggered a lot of >> >> rebuilds. Could this have gone to core-updates? >> >>=20 >> >> =E2=86=92 guix refresh -l openblas >> >> Building the following 2282 packages would ensure 5596 dependent pack= ages are rebuilt: ... >> > >> > It's not that it's triggered rebuilds, but that it's triggered builds. >> > It's also triggered builds on powerpc64le and riscv64. Before any >> > package which had openblas as a transitive dependency wasn't built by >> > the CI because it wasn't substitutable=C2=B9. People still have the op= tion of >> > using package transformations to use openblas tuned for the cortex a7 = or >> > a15 on armhf, but in reality this just unlocks substitutes for those >> > ~5600 packages which wasn't available before. >> > >> > =C2=B9 We saw this in the past briefly in the past when openzfs made i= ts way >> > as a dependency to qemu and through that to Gnome. >>=20 >> Ok, so the documentation does mention "rebuilding", and I do see that >> indeed ci.guix.gnu.org doesn't build not substitutable things. >>=20 >> Although I think it doesn't apply recursively. Take qjson, guix refresh >> -l tells me it's dependent on openblas, and looking back at say this >> output [1] for powerpc64le-linux, that's available from both >> ci.guix.gnu.org. Which makes sense, as that derivation is substitutable, >> even though one of it's inputs isn't. >>=20 >> 1: https://data.guix.gnu.org/gnu/store/fibiwzyz8s899ccpix5zs6r2pcdpxk5b-= qjson-0.9.0 >>=20 >> Maybe on the client side this works differently, and guix won't >> substitute things which have a non substitutable input? >>=20 >> Assuming ci.guix.gnu.org was building things for armhf-linux, I think >> this would have still caused ~5596 rebuilds, and as I say, I think for >> systems like powerpc64le-linux, I think it did cause ~5596 >> rebuilds. > > I looked into it more. First I ran on master: > './pre-inst-env guix build --no-grafts --system=3Darmhf-linux openblas -d' > /gnu/store/whi4yhiw2b0c0i3n6l8s0qfcphkvbzg4-openblas-0.3.20.drv > > Then I locally reverted the patch expanding the architectures where we > provided substitutes: > /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv > > Then I locally reverted the patch adjusting the make-flags: > /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv > > It seems I was wrong, changing the #:substitutable? flag _does_ change > the derivation of the package. I also checked the substitutes and saw > that bordeaux did (and does) have substitutes for the non-substitutable > version, showing that it was built before. I also checked for > powerpc64le, to see if perhaps cuirass worked a different way and > honored the #:substitutable flag by not building it there, and it too > has substitutes for both versions of openblas. > > I didn't check for riscv64 but I assume the case is the same with a > changing derivation. > > So obviously if I had realized this would cause ~5596 rebuilds per > affected arch I wouldn't have pushed the patch. I should've checked the > derivation before and after to make sure it didn't change. Thanks for investigating further. There's maybe a secondary issue here about why changing the substitutibility of a package affects it's outputs. That doesn't make much sense to me, but maybe there's a reason. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmSAj1FfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xekhg//Wx1gKdieGYRMTQ6rx/BoBoUybiM/PBAi HRd5BbiP2HGVrWZWYy3QjAIocAoqA70dQu1ClTLLQ9jMP3JT2i3aL5m8yCb83PBQ 5T1kDUZsgU8+m+Ud5l+b/rjfJT3o/v5FlKFomtRMYvUWdczD0Els2AvMlwQ8ETNx Yb2vn4qFNb3MUUvVOhrsrvcERGJAB55MdFxkuRzayAt0cBagMXCHmuk0H1E12fKL sXfugXoW33GE1OW3r+VYENNsNSml5tpTnARaR0lITSiPiBojzOc+81poyQ78BTiQ O1ZOlNqI2WXiL7/LcLzCPacAfQoEtWU8zZNT2rVfv6X2o5Yd7WXgltAaNDlNBeU0 uEJkjRdxxI6yixNHftxWmsE4Ds6OjZ0XNn+sGKt2rfN/mxUvlmDnSrCIMRGOgCkh P2n4+NhvOmbAX2sxll+vsv+waWHzM4G8972S7ZY+JeQ57XwXJqgBFLHp0AGTGqNe p300RnNYCvQ6UwphOgYWZD+si7zuWO8FnrBF7gP3bKpJXcZcNr6UYoKVzzpO2Ye9 Q6Wo/gahA+WEpeoiivTQC/SvRVr6AYOxEuI4pUWCIWS/p+H5xuHdiUA9QIxuDuQ6 idmoKFaM1ub0US33wHvcq+AMAoslDJQzqdUlDTIS/NSZbszchHEHkvQ3RgZ6nRGw GGPPrMrp2VM= =iHpo -----END PGP SIGNATURE----- --=-=-=--