From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id AHJoKthuNV8iDQAA0tVLHw (envelope-from ) for ; Thu, 13 Aug 2020 16:48:24 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id oLQ5JthuNV/JfQAAB5/wlQ (envelope-from ) for ; Thu, 13 Aug 2020 16:48:24 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 11928940215 for ; Thu, 13 Aug 2020 16:48:24 +0000 (UTC) Received: from localhost ([::1]:56282 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k6GOs-0005ny-Tv for larch@yhetil.org; Thu, 13 Aug 2020 12:48:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60750) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k6GO9-0005OP-KB for guix-devel@gnu.org; Thu, 13 Aug 2020 12:47:37 -0400 Received: from cascadia.aikidev.net ([2600:3c01:e000:267:0:a171:de7:c]:44806) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k6GO7-0003aj-Ib for guix-devel@gnu.org; Thu, 13 Aug 2020 12:47:37 -0400 Received: from localhost (unknown [IPv6:2600:3c01:e000:21:21:21:0:100e]) (Authenticated sender: vagrant@aikidev.net) by cascadia.aikidev.net (Postfix) with ESMTPSA id EDED21AA54; Thu, 13 Aug 2020 09:47:28 -0700 (PDT) From: Vagrant Cascadian To: Mark H Weaver , Jason Self Subject: Re: Linux-libre git repository In-Reply-To: <87d03vv0nm.fsf@netris.org> References: <87d03vv0nm.fsf@netris.org> Date: Thu, 13 Aug 2020 09:47:21 -0700 Message-ID: <87wo22bihy.fsf@ponder> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: none client-ip=2600:3c01:e000:267:0:a171:de7:c; envelope-from=vagrant@reproducible-builds.org; helo=cascadia.aikidev.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/13 12:47:30 X-ACL-Warn: Detected OS = ??? X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: -2.61 X-TUID: hAz05rkhLWC7 --=-=-= Content-Type: text/plain On 2020-08-12, Mark H Weaver wrote: > Mark H Weaver wrote: >>> the linux-libre project periodically deletes most of its older >>> tarballs, even if there are no accidents. > > Jason Self responded: >> Just FYI that git://linux-libre.fsfla.org/releases.git was created >> mainly to solve that problem. Versions are now pretty much permanent. > > That's helpful, thanks. I didn't know about this. Out of curiosity, is > this git repository advertised anywhere? I wasn't able to easily find > it on , but I didn't > look carefully, perhaps I missed it. News item for 2020-05-31 mentions it, but clearly it should be more prominently displayed or documented. > One question: Would it solve the problem that I mentioned in my earlier > email, namely the problem of how to determine which precise commit > introduced a regression between two stable kernel releases? If not, I > think that justifies the machinery that Guix includes to do the > deblobbing itself. The granularity appears to be at the level of released tags. I see tags with one commit per release, with an independent history from previous versions; I *think* git bisect wouldn't work without some manual fiddling and you'd have to manually bisect based on version. I tried a quick experiment using the linux-libre git repository to build a package for arm64 in gnu/packages/linux.scm: (define-public linux-libre-fsfla-git-arm64-generic (let* ((version "5.8.1-gnu") (source (origin (method git-fetch) (uri (git-reference (url "git://linux-libre.fsfla.org/releases.git") (commit (string-append "sources/v" version)))) (file-name (git-file-name "linux-libre-fsfla-git" version)) (sha256 (base32 "05v2l4r34nbkv6wpgrzydlb0fkpswpvzdya9vx30wap3n9a9wp6n")) (patches (list %boot-logo-patch %linux-libre-arm-export-__sync_icache_dcache-patch))))) (make-linux-libre* version source '("aarch64-linux") #:defconfig "defconfig" #:extra-version "fsfla-git-arm64-generic"))) The source checkout was quite slow to download, and took up ~1GB in the store once completed. I'm not sure how guix's git origin works exactly; if it downloads the entire git history even to perform a shallow checkout of a single commit, and then throws out the git history? It did appear to be calling git with flags to perform a shallow checkout. It certainly was slower than downloading a compressed tarball. The de-duplication of /gnu/store might still be beneficial if you have significantly more than ~10 versions in /gnu/store, as not every file changes with every release, but overall using compressed tarballs seems to be faster to download and extract even on a slow machine. This partly points to challenges with guix's handling of git repositories, exacerbated by larger git repositories. It would be more viable if there was some way to cache git results such as running "git clone --bare ~/.cache/guix/..." if not present, and "git fetch origin" if present and then populating the store from cached git repository, much like done with "guix pull" ... Surely this has been brought up before? Maybe this breaks the purity of guix's functional paradigm, but arguably no more than a caching http proxy really. It is also possible to retrieve tarballs directly from linux-libre git tags, though I know at least projects hosted on github this does occasionally result in non-identical tarballs. Not sure what factors might trigger this, other than changing tags, but possibly different git versions, tar versions and flags, and compression tool versions and optimizations could be a factor. Reproducible builds has documented some potential causes: https://reproducible-builds.org/docs/archives/ There are also the released linux-libre tarballs, though that may have the persistence issue previously mentioned. The code to do so is still present in guix, I made a package using: (define-public linux-libre-fsfla-arm64-generic (make-linux-libre "5.8.1" "1v7glmvz3laj1awh5zrqclp2pzfs0cjf6y3n6v97j7z901s1vlxd" '("aarch64-linux") #:defconfig "defconfig" #:extra-version "fsfla-arm64-generic")) After patching the make-linux-libre call to also include a patch needed for newer versions: - (patches (list %boot-logo-patch))) + (patches (list %boot-logo-patch + %linux-libre-arm-export-__sync_icache_dcache-patch))) Not sure why that patch isn't upstream; Debian has been carrying it for some years now... and my guix build failed to build without it on aarch64/arm64. live well, vagrant --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCXzVumgAKCRDcUY/If5cW qptVAP0UehInBoUMXeUdI4hDV34XuLJSsu+/tQZxA7+vje3JYQEAokoLflKM5ona ZRkrntveEmoSQ15lqKl2T+uwh6H5UAo= =Sr0R -----END PGP SIGNATURE----- --=-=-=--