[-- Attachment #1: Type: text/plain, Size: 3984 bytes --] So, for better or worse, I do a lot with guix on aarch64 platforms and sometimes even armhf ones. And these platforms are... often... quite... slow... The way guix does updates of linux-libre sources by downloading upstream linux tarballs and applying the linux-libre deblobbing and related scripts to them is really technically excellent. Pragmatically speaking, on slower platforms this is a huge resource overhead. So much so that ci.guix.gnu.org *usually* times out when generating the linux-libre aarch64 tarballs: https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic Same story for "linux-libre". This means that platforms that are already fairly slow have to build one or more source tarballs every linux-libre update, which can take several hours to build the source for a kernel that takes only an hour or less of compilation time. I explored some of the alternatives last year: https://lists.gnu.org/archive/html/guix-devel/2020-08/msg00089.html Some of the surrounding threads also go into various issues around this topic, but I'll summarize what I tried: * Using linux-libre's git repository. A *bit* slow to download and consumed ~1GB of space in /gnu/store, though much faster than running the deblobbing scripts. Probably would deduplicate fairly well between different versions, but would require re-downloading a lot of content. Maybe a more savvy method to make use of this would be possible. If a method somewhat like the mirror:// syntax for git repositories allowed for a local git cache or a local git proxy, this could save a lot of downloading and still get some benefits of git. Needs someone to actually do the work, of course... * Using the linux-libre tarballs. May have some issues with long-term availability, but fast to download and the deblobbing scripts have effectively already been run, saving a lot of local work. There is basically support for this in guix already, and was the method used in guix before july 2019 1ad9c105c208caa9059924cbfbe4759c8101f6c9. Recently I've been thinking more about this for aarch64 and armhf; there are a few things that might help considerably: * Bump the timeouts on ci.guix.gnu.org for linux-libre so that the deblob scripts actually get a chance to finish. Doesn't require much change in the guix infrastructure to significantly improve the status quo for slower architectures. Might take some tweaking over time to find the right timeouts. * Default to using the fsfla linux-libre tarballs on aarch64 and armhf, while doing the full local deblobbing process for other faster architectures. This keeps the infrastructure of both methods up-to-date and tested, and can be used to do interesting things like comparing to make sure the linux-libre tarballs guix generates and the fsfla linux-libre tarballs do not contain significant differences. This may very well require maintenance overhead by generating two different hashes (one for the guix deblobbed tarball, and one for the linux-libre deblobbed tarball), and doing things a little differently by architecture is a little awkward. * Produce the guix tarballs in a way that architectures could share the resulting tarball. I'm not sure what hoops this would require jumping through, something like generate the tarball, upload somewhere, update hash. The tarball produced by running the deblob scripts really shouldn't be architecture-dependent; re-doing this work on each architecture seems suboptimal (though I get is also somewhat a result of the functional package management model). I like the idea of keeping all the infrastructure there to handle what is currently done now, but would really like to figure out alternatives at least for aarch64 and armhf. Thanks for reading so far, please let me know what you think! live well, vagrant [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 227 bytes --]
On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote: > Pragmatically speaking, on slower platforms this is a huge resource > overhead. So much so that ci.guix.gnu.org *usually* times out when > generating the linux-libre aarch64 tarballs: > > https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic Thanks for letting me know. I didn't know this was happening. The immediate solution is for me to make sure the tarballs have built before committing the updates. I already do this for x86_64 and I can start doing it for aarch64 too. > * Using linux-libre's git repository. We could do this. The decision is up to the maintainers, ultimately. I'd rather we did not because linux-libre is not the "upstream" of the kernel in a meaningful way (they do not develop or fix bugs), and it's not "source code" in the GNU sense, which is the "preferred form of the work for making changes in it": http://www.gnu.org/licenses/gpl-faq.en.html#GPLOtherThanSoftware Not to mention that cloning a 1 GB Git repo is rather expensive compared to downloading a tarball, and thus also suboptimal for the ARM boards that can't easily deblob. Although if people are building the linux-libre tarballs, they are also compiling, which is of course way more expensive. > * Using the linux-libre tarballs. May have some issues with long-term > availability, but fast to download and the deblobbing scripts have > effectively already been run, saving a lot of local work. There is > basically support for this in guix already, and was the method used in > guix before july 2019 1ad9c105c208caa9059924cbfbe4759c8101f6c9. As you say, these tarballs are not available for very long. I think this option is not practical for that reason. > * Bump the timeouts on ci.guix.gnu.org for linux-libre so that the > deblob scripts actually get a chance to finish. Doesn't require much > change in the guix infrastructure to significantly improve the status > quo for slower architectures. Might take some tweaking over time to > find the right timeouts. Will do.
On Sat, May 01, 2021 at 10:45:22PM -0400, Leo Famulari wrote:
> The immediate solution is for me to make sure the tarballs have built
> before committing the updates. I already do this for x86_64 and I can
> start doing it for aarch64 too.
I started building the current derivations "by hand" on the build farm
with a longer max-silent-time.
I'll check in a day or so if it succeeded and how long it took. We can
adjust the packages based on that to avoid "silent time" timeouts even
when I don't pre-build these tarballs.
Leo Famulari <leo@famulari.name> skribis:
> On Sat, May 01, 2021 at 10:45:22PM -0400, Leo Famulari wrote:
>> The immediate solution is for me to make sure the tarballs have built
>> before committing the updates. I already do this for x86_64 and I can
>> start doing it for aarch64 too.
>
> I started building the current derivations "by hand" on the build farm
> with a longer max-silent-time.
For packages we can add a ‘max-silent-time’ property, but there’s
nothing like this for origins.
I wonder if there’s a way we could address it in (gnu ci).
Thoughts?
Ludo’.
On Sun, May 02, 2021 at 11:08:49PM +0200, Ludovic Courtès wrote: > For packages we can add a ‘max-silent-time’ property, but there’s > nothing like this for origins. Oh, right. > I wonder if there’s a way we could address it in (gnu ci). > > Thoughts? In the meantime, can we make a 'kernel-updates' Cuirass job again, that would build a kernel-updates branch on Savannah? We had one recently for a brief period but I stopped using it, because I could work more quickly building the tarballs "by hand" (although I've only been doing it for x86_64). However, it would be helpful to let Cuirass at least try to build these derivations, with me checking the results and retrying failures before pushing kernel-updates to master.
On Sun, May 02, 2021 at 06:26:15PM -0400, Leo Famulari wrote: > In the meantime, can we make a 'kernel-updates' Cuirass job again, that > would build a kernel-updates branch on Savannah? Mistake: I'd want to call the branch wip-kernel-updates, so that it's rebaseable. > We had one recently for a brief period but I stopped using it, because I > could work more quickly building the tarballs "by hand" (although I've > only been doing it for x86_64). > > However, it would be helpful to let Cuirass at least try to build these > derivations, with me checking the results and retrying failures before > pushing kernel-updates to master.
[-- Attachment #1: Type: text/plain, Size: 1323 bytes --] On Sat, May 01, 2021 at 10:45:22PM -0400, Leo Famulari wrote: > The immediate solution is for me to make sure the tarballs have built > before committing the updates. I already do this for x86_64 and I can > start doing it for aarch64 too. Well, this is easier said than done, currently. I was able to build the "arm64-generic" tarball for 5.11.18, but we actually don't have the capability to build these things reliably at the moment. We only have 1 real aarch64 machine (Overdrive 1000) [0] in the build farm, and it's never idle. The emulated builders are super slow. We could change (gnu ci) to increase the timeout for these jobs, like Ludo suggested, but I'm not sure it's worth it? I estimate the Overdrive would spend at least one day a week just building tarballs, which is unreasonable IMO. The problem is that the linux-libre team told us that we shouldn't trust the linux-libre deblobbing scripts to work and that we have to "deblob-check" them, which takes a very long time. [1] I suggest we disable these checks for now, at least for aarch64; have they ever actually found a problem? At least for aarch64, the choice is not between "blobby tarball or non-blobby tarball", but rather between "no tarball or tarball". [0] I can add another 4 cores (Macchiatobin) to the build farm, maybe next weekend. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --]
Hello, Leo, On May 1, 2021, Leo Famulari <leo@famulari.name> wrote: > it's not "source code" in the GNU sense, which is the "preferred form > of the work for making changes in it": That's not accurate. Linux-libre sources, whether obtained from a git repository or from a tarball, are just as suitable for kernel development as any other snapshot. There is an alternate understanding of source code form taking shape in some communities that suggests that a version control repository is somehow more "source"ish than any of the versions held in it. No doubt the additional versions in it may be useful in some cases, but it does not follow that other versions and their relationship are part of "source code" in the GNU sense. In the GNU sense, any individual snapshot or release meets the definition of source code. > Not to mention that cloning a 1 GB Git repo is rather expensive compared > to downloading a tarball You don't have to clone anything. If all you want is a tarball, use 'git archive --remote'. > As you say, these tarballs are not available for very long. *nod*, that's why we switched to a git repo, and extract the tarballs from it. Back when we made the switch, I wrote extensive documentation and recipes as to the possibilities of getting to the GNU Linux-libre sources from it, how to verify signatures, etc. It was to be assessed by then maintainers of Guix recipes to build GNU Linux-libre before publishing, but it doesn't look like I ever got feedback on them, and they hvae been unpublished so far. Is it correct to assume that they have never reached you? Would you like to have a look and let me know whether they make sense to you? I'd be glad to dig them up and share them with you, if so, and to take your feedback towards making them available to the public at large. Thanks, -- Alexandre Oliva, happy hacker https://FSFLA.org/blogs/lxo/ Free Software Activist GNU Toolchain Engineer Disinformation flourishes because many people care deeply about injustice but very few check the facts. Ask me about <https://stallmansupport.org>
On Mon, May 03, 2021 at 01:39:54PM -0300, Alexandre Oliva wrote: > You don't have to clone anything. If all you want is a tarball, use > 'git archive --remote'. Thanks, I didn't know about this feature. Is the tarball generation "stable" across Git releases? Or is it subject to change? We had this issue with Git archive creation in the past. > Is it correct to assume that they have never reached you? Would you > like to have a look and let me know whether they make sense to you? I'd > be glad to dig them up and share them with you, if so, and to take your > feedback towards making them available to the public at large. Personally, as a non-maintainer, I'd prefer to preserve the status quo of our linux-libre packaging. If the maintainers decide that it should be changed, I'll follow their decision.
On May 3, 2021, Leo Famulari <leo@famulari.name> wrote: > Is the tarball generation "stable" across Git releases? Or is it > subject to change? We had this issue with Git archive creation in the > past. I have heard about early format changes, but since Linux itself has used git archive to generate tarballs on the fly, and published digital signatures based on archives generated this way, I presume an expectation that changes to the format won't be made without a very compelling reason. -- Alexandre Oliva, happy hacker https://FSFLA.org/blogs/lxo/ Free Software Activist GNU Toolchain Engineer Disinformation flourishes because many people care deeply about injustice but very few check the facts. Ask me about <https://stallmansupport.org>
Hello,
Alexandre Oliva <lxoliva@fsfla.org> writes:
[...]
> Back when we made the switch, I wrote extensive documentation and
> recipes as to the possibilities of getting to the GNU Linux-libre
> sources from it, how to verify signatures, etc. It was to be assessed
> by then maintainers of Guix recipes to build GNU Linux-libre before
> publishing, but it doesn't look like I ever got feedback on them, and
> they hvae been unpublished so far.
>
> Is it correct to assume that they have never reached you? Would you
> like to have a look and let me know whether they make sense to you? I'd
> be glad to dig them up and share them with you, if so, and to take your
> feedback towards making them available to the public at large.
I for one have not seen them. Would you mind making them available
somewhere? I'd be interested in having a look.
Thanks,
Maxim
On Thu, May 06, 2021 at 01:10:45AM -0300, Alexandre Oliva wrote:
> I have heard about early format changes, but since Linux itself has used
> git archive to generate tarballs on the fly, and published digital
> signatures based on archives generated this way, I presume an
> expectation that changes to the format won't be made without a very
> compelling reason.
I asked because, historically, the linux-libre project has chosen not to
store tarballs of its generated linux-libre source trees.
Linux, on the other hand, does store their tarballs in perpetuity, even
if they are created with `git archive`.
Do you know for sure that Linux generates tarballs "on the fly" and does
not simply preserve the signed tarballs forever?
Otherwise, the PGP signatures could go "stale" at some point, and that
would be unprecedented and unacceptable.
On May 6, 2021, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote: > Alexandre Oliva <lxoliva@fsfla.org> writes: >> recipes as to the possibilities of getting to the GNU Linux-libre >> sources from it, how to verify signatures, etc > I for one have not seen them. Would you mind making them available > somewhere? I'd be interested in having a look. Not as extensive as I thought ;-) unless I found only an early version, but so here it goes: * Download one release as git checkout: git clone -b sources/v$version --single-branch \ git://linux-libre.fsfla.org/releases.git linux-libre-$version/ * Example substitutions: version=5.7-gnu upversion=5.7 prefix=linux-$upversion/ * If you plan on verifying any releases: git archive --remote=git://linux-libre.fsfla.org/releases.git \ --format tar master | tar -x -O keyring.gpg | gpg --import - * If you plan on verifying release $version: git archive --remote=git://linux-libre.fsfla.org/releases.git \ --format tar logs/v$version | tar -x && if test -f linux-libre-$version.git.tar.sign; then prefix=linux-libre-$version/ targit=.git else prefix=linux-$upversion/ targit= fi * Download/archive one release as git tarball: # Use the conditional prefix above to be able to verify. git archive --remote=git://linux-libre.fsfla.org/releases.git \ --format tar.gz --prefix=$prefix sources/v$version \ -o linux-libre-$version$targit.tar.gz * Verify one release in git repo: git archive --format tar --prefix=$prefix sources/v$version | gpg --verify logs/v$version/linux-libre-$version$targit.tar.sign - * Verify one release in a (gzip-compressed) tarball: # Must have used the conditional prefix above to verify. zcat linux-libre-$version$targit.tar.gz | gpg --verify linux-libre-$version$targit.tar.sign - * Download and verify a released compressed tarball: ext=lz comp=lzip # ext=bz2 comp=bzip2 # ext=xz comp=xz wget https://linux-libre.fsfla.org/pub/linux-libre/releases\ /$version/linux-libre-$version.tar.{$ext,sign} && $comp -d < linux-libre-$version.tar.$ext | gpg --verify linux-libre-$version.tar.sign - # no $targit -- Alexandre Oliva, happy hacker https://FSFLA.org/blogs/lxo/ Free Software Activist GNU Toolchain Engineer Disinformation flourishes because many people care deeply about injustice but very few check the facts. Ask me about <https://stallmansupport.org>
[-- Attachment #1: Type: text/plain, Size: 1896 bytes --] On 2021-05-01, Leo Famulari wrote: > On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote: >> Pragmatically speaking, on slower platforms this is a huge resource >> overhead. So much so that ci.guix.gnu.org *usually* times out when >> generating the linux-libre aarch64 tarballs: >> >> https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic > > Thanks for letting me know. I didn't know this was happening. > > The immediate solution is for me to make sure the tarballs have built > before committing the updates. I already do this for x86_64 and I can > start doing it for aarch64 too. This has definitely helped sometimes, thanks! I even saw a substitute of linux-libre for aarch64 earlier today! :) Still, I'm noticing another problem with the way way the tarballs are generated on ci.guix.gnu.org ... When it generates a tarball, all the various packages independently try to recreate the source tarball; so you have at least fours jobs ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers", "linux-libre-bpf") all concurrently trying to build the very same very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them might succeed, but the others may not, and even though one of them succeeded, none of the failing ones retry... Not knowing exactly how ci.guix.gnu.org works, would it make sense to create a tarball package instead of the ... computed origin(?) tarball, so it could be better represented in the package dependency graph, and the various linux-libre-* packages can wait till it is available rather than all trying to recreate the same thing? That still requires the tarball generation to not time out in the first place, but maybe it would help with the resource limitations a bit to only build the source tarball once per architecture? live well, vagrant [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 227 bytes --]
On Sat, May 29, 2021 at 09:50:36PM -0700, Vagrant Cascadian wrote: > When it generates a tarball, all the various packages independently try > to recreate the source tarball; so you have at least fours jobs > ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers", > "linux-libre-bpf") all concurrently trying to build the very same > very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them > might succeed, but the others may not, and even though one of them > succeeded, none of the failing ones retry... > > Not knowing exactly how ci.guix.gnu.org works, would it make sense to > create a tarball package instead of the ... computed origin(?) tarball, > so it could be better represented in the package dependency graph, and > the various linux-libre-* packages can wait till it is available rather > than all trying to recreate the same thing? If I understand correctly, this was recently fixed in Cuirass: https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=d1a95e8b33b454a45bda506a22a8b9d9d2c8b16e
Hello, >> Not knowing exactly how ci.guix.gnu.org works, would it make sense to >> create a tarball package instead of the ... computed origin(?) tarball, >> so it could be better represented in the package dependency graph, and >> the various linux-libre-* packages can wait till it is available rather >> than all trying to recreate the same thing? > > If I understand correctly, this was recently fixed in Cuirass: > > https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=d1a95e8b33b454a45bda506a22a8b9d9d2c8b16e With the recent Cuirass commits, the notion of build dependencies have been introduced. If you have a look to this page: https://ci.guix.gnu.org/build/497096/details, you will see that the "git" build won't start until all the other builds it depends are successfully completed (xmlto, subversion, tk ...). The problem with linux-libre and a few other packages is that some of the derivations they depend do not correspond to proper packages that are identified and built by Cuirass. You will see here: https://ci.guix.gnu.org/build/512266/details that linux-libre has no build dependencies. If we could translate the computed origin method into a package, I think it could solve this issue. I also noticed a similar issue with ublock-origin and the make-chromium-extension method. Thanks, Mathieu
[-- Attachment #1: Type: text/plain, Size: 3206 bytes --] Turning this conversation into a bug, original thread around here: https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427 On 2021-05-29, Vagrant Cascadian wrote: > On 2021-05-01, Leo Famulari wrote: >> On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote: >>> Pragmatically speaking, on slower platforms this is a huge resource >>> overhead. So much so that ci.guix.gnu.org *usually* times out when >>> generating the linux-libre aarch64 tarballs: >>> >>> https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic >> >> Thanks for letting me know. I didn't know this was happening. >> >> The immediate solution is for me to make sure the tarballs have built >> before committing the updates. I already do this for x86_64 and I can >> start doing it for aarch64 too. > > This has definitely helped sometimes, thanks! I even saw a substitute of > linux-libre for aarch64 earlier today! :) > > Still, I'm noticing another problem with the way way the tarballs are > generated on ci.guix.gnu.org ... > > When it generates a tarball, all the various packages independently try > to recreate the source tarball; so you have at least fours jobs > ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers", > "linux-libre-bpf") all concurrently trying to build the very same > very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them > might succeed, but the others may not, and even though one of them > succeeded, none of the failing ones retry... > > Not knowing exactly how ci.guix.gnu.org works, would it make sense to > create a tarball package instead of the ... computed origin(?) tarball, > so it could be better represented in the package dependency graph, and > the various linux-libre-* packages can wait till it is available rather > than all trying to recreate the same thing? > > That still requires the tarball generation to not time out in the first > place, but maybe it would help with the resource limitations a bit to > only build the source tarball once per architecture? This seems to still be an issue for ci.guix.gnu.org, but the linux-libre* substitutes for aarch64 seem to be available on https://bordeaux.guix.gnu.org ... $ guix weather linux-libre linux-libre-arm64-generic computing 2 package derivations for aarch64-linux... looking for 2 store items on https://ci.guix.gnu.org... https://ci.guix.gnu.org 0.0% substitutes available (0 out of 2) unknown substitute sizes 0.0 MiB on disk (uncompressed) 0.740 seconds per request (0.7 seconds in total) 1.4 requests per second 0.0% (0 out of 2) of the missing items are queued 1 queued builds aarch64-linux: 1 (100.0%) build rate: .00 builds per hour x86_64-linux: 0.00 builds per hour aarch64-linux: 0.00 builds per hour i686-linux: 0.00 builds per hour looking for 2 store items on https://bordeaux.guix.gnu.org... https://bordeaux.guix.gnu.org 100.0% substitutes available (2 out of 2) 83.9 MiB of nars (compressed) 202.2 MiB on disk (uncompressed) (continuous integration information unavailable) live well, vagrant [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 227 bytes --]
[-- Attachment #1: Type: text/plain, Size: 295 bytes --] On 2021-08-01, Vagrant Cascadian wrote: > Turning this conversation into a bug, original thread around here: > > https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427 For reference: bug#49810: source tarballs potentially built for each derivation live well, vagrant [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 227 bytes --]