From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 8A9NA0RpRGHXwwAAgWs5BA (envelope-from ) for ; Fri, 17 Sep 2021 12:09:08 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id MHaDOkNpRGE/HQAAbx9fmQ (envelope-from ) for ; Fri, 17 Sep 2021 10:09:07 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 82B47FAF0 for ; Fri, 17 Sep 2021 12:09:07 +0200 (CEST) Received: from localhost ([::1]:56388 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mRAnq-0001Rk-OO for larch@yhetil.org; Fri, 17 Sep 2021 06:09:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40912) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mRAnf-0001Rc-Ao for guix-devel@gnu.org; Fri, 17 Sep 2021 06:08:55 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:45075) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mRAnc-0004ww-Tf for guix-devel@gnu.org; Fri, 17 Sep 2021 06:08:55 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 6120827BBE1; Fri, 17 Sep 2021 11:08:48 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 25dd04ab; Fri, 17 Sep 2021 10:08:47 +0000 (UTC) References: User-agent: mu4e 1.6.5; emacs 27.2 From: Christopher Baines To: zimoun Subject: Re: guix weather -m etc/sources-manifest.scm and CI? Date: Fri, 17 Sep 2021 10:49:02 +0100 In-reply-to: Message-ID: <87sfy3y0nn.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1631873347; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=HvYi/b0ZPohYEs+j0QVWPXh/pw071RSh0OJ9yimlAVo=; b=BTCw1w9mkhcJpPjHvyCX5LntWWpcpZ/L/A8nhoACXULQynrFxqZb1vQbQfCcbXY8EjEYxk zEZ1JL+XYne5pJTzBqCyKoshhkg3ykEUP2uaEBS1hvg/2R4DmCcI/kQgAfM727xtXdGcEr WhuZoOAG30KEn4WpR9a7C4+FtBvmQB8cR3ga+OLWyi4ZFTZsgcYKo+8zt0F7EqZ8lGt9lY qmz6bOPWF5zc93suluXaGUUGpMmesBk1RHMYVeqI97cnHP1Y4I/txFtOzu1k4bomsMRJ+x YZNtbZ6twGrSy0+1lDpBc/texKpghNqXGr02tDHOLoNyx6SGl22pB1P93fzhhg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1631873347; a=rsa-sha256; cv=none; b=sualNZSryIt7oSq0yIYw7uudKAmQrH1CEHdV8H3hNMldzzZtr2A5G8fC2HjP6Dq0Qw7sQv XvmhIGIRn+EagP5milfBXryftv7cbmlNYr94S5xSucOt1b5RGOsQLpiRLmgzHdkoMMEeK0 tUQZ5Op2cgz4TahEERnwzJcW/VBezU6Aq73m8gUzD+dTKDcOZrgoTxERJTUpwY1cHYvirG P57KjDV82wsYiqaS8UzTnqCmcpR5KOCwApg9SUsU+At7xRW/jrs6rObNk9eMNjyrN+OdqR 3azIsRoEqbZg/9qARZ4kqsNF5WpjfvHaE1TF9K+p5ZV0p5ElVYFmqTTMArGwVw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.49 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 82B47FAF0 X-Spam-Score: -3.49 X-Migadu-Scanner: scn0.migadu.com X-TUID: uHsWdm8c20lW --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable zimoun writes: > Playing with the new 'etc/sources-manifest.scm', using fb32a38, I get: > > $ guix weather -m ~/src/guix/guix/etc/source-manifest.scm > computing 16,831 package derivations for x86_64-linux... > looking for 16,831 store items on https://ci.guix.gnu.org... > https://ci.guix.gnu.org > 74.6% substitutes available (12,556 out of 16,831) > at least 65,367.1 MiB of nars (compressed) > 81,988.1 MiB on disk (uncompressed) > 0.095 seconds per request (1,606.8 seconds in total) > 10.5 requests per second > > 0.0% (0 out of 4,275) of the missing items are queued > 5 queued builds > aarch64-linux: 4 (80.0%) > powerpc64le-linux: 1 (20.0%) > build rate: .00 builds per hour > powerpc64le-linux: 0.00 builds per hour > aarch64-linux: 0.00 builds per hour > i686-linux: 0.00 builds per hour > x86_64-linux: 0.00 builds per hour > looking for 16,831 store items on https://bordeaux.guix.gnu.org... > https://bordeaux.guix.gnu.org > 99.8% substitutes available (16,804 out of 16,831) > 62,195.0 MiB of nars (compressed) > 108,212.7 MiB on disk (uncompressed) > 0.049 seconds per request (829.2 seconds in total) > 20.3 requests per second > (continuous integration information unavailable) > > > The questions are: > > Why ci.guix.gnu.org contains only 75%? And bordeaux almost everything? > (I guess the missing ones on bordeaux are corner cases as icecat, linux-l= ibre). bordeaux.guix.gnu.org is hopefully only missing substitutes for things where there's actually an issue building them. It's possible not to guess though and instead ask guix weather what is missing. I used time-machine as well for the latest commit processed by data.guix.gnu.org, just so that the list doesn't include builds which haven't started yet. =E2=86=92 guix time-machine --commit=3D33bc3fb2a5f30a6e21f1b8d6d43867d921bd= 951c -- weather --substitute-urls=3Dhttps://bordeaux.guix.gnu.org --display= -missing -m ./etc/source-manifest.scm computing 16,850 package derivations for x86_64-linux... looking for 16,850 store items on https://bordeaux.guix.gnu.org... https://bordeaux.guix.gnu.org 99.8% substitutes available (16,822 out of 16,850) 62,377.9 MiB of nars (compressed) 108,593.9 MiB on disk (uncompressed) 0.160 seconds per request (4.5 seconds in total) 6.3 requests per second (continuous integration information unavailable) Substitutes are missing for the following items: /gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz = = i686-linux /gnu/store/9fgglvi13vhpc63knf15dipzmmck6ia9-mini-os-git-checkout = = i586-gnu /gnu/store/pb5jmi9zalg6xylzsjmrskwxs0kar97l-fossil-src-2.11.tar.gz = = armhf-linux /gnu/store/8s5d2hjd6nhzf6m9dlxrykk8ijkf62pi-texlive-marginnote-51265-chec= kout = armhf-linux /gnu/store/cp3ka40bhb28rrmyj4mzf9xjhi0ssxjx-CombBLAS_beta_16_2.tgz = = x86_64-linux /gnu/store/f1h94axw82id0k8c2lippg73sqlibqs8-dovecot-trees-2.1.0.tar.gz = = aarch64-linux /gnu/store/f12l7dg2z1xyf2wcw9g3v7jwpgd8m5zv-tla2tools-1.8.0-checkout = = i686-linux /gnu/store/53j9996hdgnmhgzswjjggdz9wnv29p5b-jpegsrc.v9d.tar.gz = = i586-gnu /gnu/store/80s1m4q2hnjfbqzw3fhywvsyim2b00cd-gcc-4.7.4.tar.bz2 = = x86_64-linux /gnu/store/csv5xca0p8w5jqqx53szy2dja8lwxma2-unicode-blocks.txt = = i686-linux /gnu/store/xn62dzq9hw3qnvmbxyxjkvhlacs72rz7-canada1500.zip = = armhf-linux /gnu/store/6vqin3by3nkn0sxhgwnzi9l7gflpfw1q-gcc-vc4-6.5.0-checkout = = x86_64-linux /gnu/store/zxfkf2bzq7pp7nhmbdgzvmjp0iv46wds-propeller-gcc-b4f45a4725e0b6d= 0af59e594c4e3e35ca4105867-checkout = i686-linux /gnu/store/g98l0m8qsfdqybak2jz3ma2miv9bki4j-emacs-danneskjold-theme-0.0.0= -2.e4d1f2c-checkout = x86_64-linux /gnu/store/bznj81ls02r7kwld6338dhba0pql7nik-Rserve_1.8-6.tar.gz = = i686-linux /gnu/store/fb0rqvv38pf02qv3zjvjsindjdhlbpca-rda_1.0.2-2.1.tar.gz = = x86_64-linux /gnu/store/m5jn626l55nwxhgs405byka7pzkg1arz-d3.v6.js = = powerpc64le-linux /gnu/store/bf496ni94ik19gdxkgp0vddkfxbid2vf-d3.v5.js = = mips64el-linux /gnu/store/c0kig783lv56vv0yc6i8rsvhaxdp52sd-paml4.9e.tgz = = mips64el-linux /gnu/store/17pi83jai4wlnnv1dilqwv1xv6g35d7x-plink-1.07-src.zip = = aarch64-linux /gnu/store/c3k6gxx7yha95f4avia11ls71ndzfx34-ggtree_3.0.3.tar.gz = = x86_64-linux /gnu/store/g2fhcc16yk5asszzmih0cpkwqb3b9bb3-clusterProfiler_4.0.4.tar.gz = = x86_64-linux /gnu/store/7n2rk9n46am8qx03ppycvkcdbnh6qzkq-eye-1.4.1.tar.gz = = aarch64-linux /gnu/store/izhvfxx0iiwv8p1xr8ag5mpvsi86x9kx-weightwatcher-1.12.tar.gz = = mips64el-linux /gnu/store/l3r5j8r4f6fh6bxa9va3681n3bclxpdq-stuff-1.26.0.tar.gz = = i686-linux /gnu/store/ymip20qmlz320r2zx0hhfxn70428n76z-skymaker-3.10.5.tar.gz = = armhf-linux /gnu/store/xqc7n8fsq5zxag37a2ay88cx6c3zsm51-mpfrcx-0.6.3.tar.gz = = x86_64-linux /gnu/store/yf2dgn6jcdwvbnzcjwcadc8lwr9ks0wr-eclib-20190909-checkout = = mips64el-linux If you visit https://data.guix.gnu.org and add the store path to the URL, you can get links to the failed builds ([1] for example). 1: https://data.guix.gnu.org/gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp= 3-1.1.4_src.tgz > Does it make sense to duplicate the storage of all these origins? > > Using extensively "guix time-machine", I note that a lot of > derivations are missing and thus they are built locally, which is > costly on poor machine. Could we reduce the duplication and so save > some space in order to systematically keep these derivations? It > would greatly ease the Guix experience for "guix time-machine" users. > :-) In terms of duplication, I still think that an argument can be made that bordeaux.guix.gnu.org is providing value, even if it's doing some of the same stuff as ci.guix.gnu.org. As for keeping build results, everything that's ever been built for bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is still around. In some ways, this is because deleting things is a bit more difficult, as the files aren't in the store, you can't just do a guix gc. However, I do want to try to never delete anything. That's going to require a bit of work as the local storage on the bayfront machine will be all used up at some point, but I do have the beginnings of a plan to avoid this an keep building things. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmFEaSxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdR0A//fluCyK7m64reMpOypw9zq4YEOBxWttfk u7cbkq7eZt54teck38RY4wezQscbbzDOulZ3/RtG8Mx7j8W4SZzFdMu2MNtWWYOJ 6d/8SSji9MeKAWa2IAIdn6eqKZgXC3m+aP6p9jVR1F/yaR+BC7mmfc8VoOA495G5 efUVUZHSKdrApV8HF1vbgYeF5sH5xSFHN3PTMakPLxgaVhIn24A83vVtBQZRJfuI nr8YkLq3glMKtD47i+8cXIXUzZtfPaY67wGNi0n8QUC8hRpclgiQiGbBoTjYkYjl J+o8glGXgstpgt3T8DO3MCqrM4fD1jX6xqhPiQbIv5pcjJb7kP+21dXS1EQddOc1 wugYq/sLcy0jUxLhLpEyb8PX/4Pa+2/RoJJFnXLdsAbPCBV3mByLWPJAe6K8zRnb I1u9l+/j41F37QCF1LhbfRYt1BU4eEYx7gEGTeKkJtsj2rYi7ar/ISYax2irT62Q wuKMWUSqiUz3W9FteLXK1voOn1C+eMGbFoabHNnacrVc89mOtcwScj7AsHn8fQdq 5fm9TnXQnEEpm8c1exM/BoxQbfc7XPJQAZxwJ0tOxvXIzQ5H4UDB3hsz4W/417jk PJxElv1ntV9Y0c1M6mDdFRYqKx+zSEP9MYJtnCsjVV4ZqKDdQCuSspoMBwqUG8wW Ye8JZmVx9Ek= =F+mD -----END PGP SIGNATURE----- --=-=-=--