From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id ECsuFHcEjmBlgQEAgWs5BA (envelope-from ) for ; Sun, 02 May 2021 03:46:31 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id +HXSD3cEjmCYawAAB5/wlQ (envelope-from ) for ; Sun, 02 May 2021 01:46:31 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6342C1CDBF for ; Sun, 2 May 2021 03:46:30 +0200 (CEST) Received: from localhost ([::1]:33086 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ld1Bl-0003BI-Ac for larch@yhetil.org; Sat, 01 May 2021 21:46:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42554) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ld1B3-0003B5-8h for guix-devel@gnu.org; Sat, 01 May 2021 21:45:45 -0400 Received: from cascadia.aikidev.net ([2600:3c01:e000:267:0:a171:de7:c]:49560) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ld1Ay-0001jc-Qj for guix-devel@gnu.org; Sat, 01 May 2021 21:45:44 -0400 Received: from localhost (97-120-1-76.ptld.qwest.net [97.120.1.76]) (Authenticated sender: vagrant@cascadia.debian.net) by cascadia.aikidev.net (Postfix) with ESMTPSA id 7231F1AB0A; Sat, 1 May 2021 18:45:37 -0700 (PDT) From: Vagrant Cascadian To: guix-devel Subject: linux-libre source tarballs Date: Sat, 01 May 2021 18:45:32 -0700 Message-ID: <87lf8xvrgj.fsf@yucca> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: none client-ip=2600:3c01:e000:267:0:a171:de7:c; envelope-from=vagrant@debian.org; helo=cascadia.aikidev.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619919990; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=tZAybjjMu+WEoYTUh2dVqYJ1Tttrnmq9HTbTSc11/Gg=; b=NY9GF4+9uv19OKH3/V4D9WHm0y1U6WZZFCCLaLgsChV3tEIAqFoJbGjvoo4tv4WkDSDqo9 gDU69lxmElZFGFKp3VbGQvzffpAOYKvYgJuHasfUpLDg2jiU4ZdWxnYmHwvBt4n273q0YD ROslW3b0vGUkhf6k5+cxtD43is5p3KL0WrIafKq/vBsBcdWfMXk2JmWTD48pZfx5RYfdp3 +Vad1bD7tYFErtXTHGH6QoJOYS5RZA/gW2JsypJcXF7BAFQUKzMlhlOJfQIC4h8ZYg5vrf O4lDthjQtMbWwpltYIXWUUJtosh+T5j2XOH+RSLnZgB+6Kj88LqGqHLFKP/ESQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619919990; a=rsa-sha256; cv=none; b=LB411We3aZqyLpXgDelAfClgVh51EBtfb5fookOo9SQgUT/4wJYet+V4edtctSNrXiTotO AYZxLQ6d4t0J93eDLz4XLssTop5BNzZLicXoG5/QaVb4BOd+rAIKSyhMINfTpSKL6esV5d my798zdncEs1PE880dPbQT+KgkoIM9Aabcct1P7BbnT3UuqpRMMbaVTsEEjnqmzkdVktHI lKKk79GJqBueo9e6FiEGZurPtr4FWMPmrXEMXfpj2+XGz0SMhkBXNL8WxwE7mF3z9tamw5 sk5b29nTxGdhmRE0ha+YOJbZa9neAlbvDGeVw69rE2kJW+VPcYUCAY+G5/4oNA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -1.56 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 6342C1CDBF X-Spam-Score: -1.56 X-Migadu-Scanner: scn0.migadu.com X-TUID: ysMWhh0PP5R3 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable So, for better or worse, I do a lot with guix on aarch64 platforms and sometimes even armhf ones. And these platforms are... often... quite... slow... The way guix does updates of linux-libre sources by downloading upstream linux tarballs and applying the linux-libre deblobbing and related scripts to them is really technically excellent. Pragmatically speaking, on slower platforms this is a huge resource overhead. So much so that ci.guix.gnu.org *usually* times out when generating the linux-libre aarch64 tarballs: https://ci.guix.gnu.org/search?query=3Dsystem%3Aaarch64-linux+linux-libre= -arm64-generic Same story for "linux-libre". This means that platforms that are already fairly slow have to build one or more source tarballs every linux-libre update, which can take several hours to build the source for a kernel that takes only an hour or less of compilation time. I explored some of the alternatives last year: https://lists.gnu.org/archive/html/guix-devel/2020-08/msg00089.html Some of the surrounding threads also go into various issues around this topic, but I'll summarize what I tried: * Using linux-libre's git repository. A *bit* slow to download and consumed ~1GB of space in /gnu/store, though much faster than running the deblobbing scripts. Probably would deduplicate fairly well between different versions, but would require re-downloading a lot of content. Maybe a more savvy method to make use of this would be possible. If a method somewhat like the mirror:// syntax for git repositories allowed for a local git cache or a local git proxy, this could save a lot of downloading and still get some benefits of git. Needs someone to actually do the work, of course... =20 * Using the linux-libre tarballs. May have some issues with long-term availability, but fast to download and the deblobbing scripts have effectively already been run, saving a lot of local work. There is basically support for this in guix already, and was the method used in guix before july 2019 1ad9c105c208caa9059924cbfbe4759c8101f6c9. Recently I've been thinking more about this for aarch64 and armhf; there are a few things that might help considerably: * Bump the timeouts on ci.guix.gnu.org for linux-libre so that the deblob scripts actually get a chance to finish. Doesn't require much change in the guix infrastructure to significantly improve the status quo for slower architectures. Might take some tweaking over time to find the right timeouts. * Default to using the fsfla linux-libre tarballs on aarch64 and armhf, while doing the full local deblobbing process for other faster architectures. This keeps the infrastructure of both methods up-to-date and tested, and can be used to do interesting things like comparing to make sure the linux-libre tarballs guix generates and the fsfla linux-libre tarballs do not contain significant differences. This may very well require maintenance overhead by generating two different hashes (one for the guix deblobbed tarball, and one for the linux-libre deblobbed tarball), and doing things a little differently by architecture is a little awkward. * Produce the guix tarballs in a way that architectures could share the resulting tarball. I'm not sure what hoops this would require jumping through, something like generate the tarball, upload somewhere, update hash. The tarball produced by running the deblob scripts really shouldn't be architecture-dependent; re-doing this work on each architecture seems suboptimal (though I get is also somewhat a result of the functional package management model). I like the idea of keeping all the infrastructure there to handle what is currently done now, but would really like to figure out alternatives at least for aarch64 and armhf. Thanks for reading so far, please let me know what you think! live well, vagrant --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCYI4EPAAKCRDcUY/If5cW qtllAP0bzRYJyA6nbnLyZKu68daGl5dSjBMVvVUdksxE8AG1QgD7BcumhD+Dxrjc jZRC3LUYG1s12FIS71Cd4kHZd4v5pgE= =nNkb -----END PGP SIGNATURE----- --=-=-=--