From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eLovKh8g8WG8awAAgWs5BA (envelope-from ) for ; Wed, 26 Jan 2022 11:19:11 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id yPuqJx8g8WE6hQAA9RJhRA (envelope-from ) for ; Wed, 26 Jan 2022 11:19:11 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 51AB62C330 for ; Wed, 26 Jan 2022 11:19:11 +0100 (CET) Received: from localhost ([::1]:49544 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nCfOQ-00074G-Gt for larch@yhetil.org; Wed, 26 Jan 2022 05:19:10 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42502) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nCfOI-00071y-DL for bug-guix@gnu.org; Wed, 26 Jan 2022 05:19:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:58109) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nCfOI-00089Z-36 for bug-guix@gnu.org; Wed, 26 Jan 2022 05:19:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nCfOH-0008BA-UQ for bug-guix@gnu.org; Wed, 26 Jan 2022 05:19:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 26 Jan 2022 10:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53533 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Leo Famulari Received: via spool by 53533-submit@debbugs.gnu.org id=B53533.164319231131401 (code B ref 53533); Wed, 26 Jan 2022 10:19:01 +0000 Received: (at 53533) by debbugs.gnu.org; 26 Jan 2022 10:18:31 +0000 Received: from localhost ([127.0.0.1]:51012 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCfNn-0008AP-3z for submit@debbugs.gnu.org; Wed, 26 Jan 2022 05:18:31 -0500 Received: from mail-io1-f49.google.com ([209.85.166.49]:34542) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCfNl-0008AC-KX for 53533@debbugs.gnu.org; Wed, 26 Jan 2022 05:18:30 -0500 Received: by mail-io1-f49.google.com with SMTP id i62so11350104ioa.1 for <53533@debbugs.gnu.org>; Wed, 26 Jan 2022 02:18:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=UPxSSeE8C5LBi73/ayWes6ABMVxIxT4umpwGjoiqk/c=; b=o+NMb0Z9maks04qUelv7u4nHGSSNt4r8kzsQY+bEHefHjiEBWDC+XzKYge/C7ZREu7 my7hsfSwDYSmWBvw7bVcj4iI+500i4HNgy7KqMWq2Lj30yDlKMzrbgZ2CEyXX1zJGfoh Uw5/+fQug8GMZuBfjdgQ8LWe224I+mu2omYxk7JUH5GH3x6UXaUUZ2HwlOWEePoLhgdj dVrBpJ85parTYJMLc13X/iZAxHgPyGyC609zseTx2CDJWYPe4IBl6vJ5I9DB68EM92fb N+hY4c9wM2QoG+xoTW7tQsCOLZ7HvIPxQ7N79udzkvrhkP2CEy38/I2GpymW8DyI6rRB AOVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=UPxSSeE8C5LBi73/ayWes6ABMVxIxT4umpwGjoiqk/c=; b=u083ucrkZZJ/reb/3qZLxpfjDGZWHTWHnOSuxOribRF4yawRhY7g4BYSqVcpvFnGI2 kxvN9Mzwoy2Hqd7O2pUKd7XqmIrjorJ56AEdMCq4zLDnXexY+4Yjw9S4sRYSnTOY0AiB xm0w87et4KIpAWP77hJfV3ScYbDeSrmmWaJTfrlADo8Uo+x7ldgnPb6HOwjFy7qwvDZv gv+lOj/ZDVjDwAPCW7xfUx7dgD7suN4Bb29pM6p6PxG0mo67OEVqZvBFSRgKPyJ+EfLL cNfkspPtyex8ytcj7CTPpS8l8IkzPUjrQZIAxhFyNonY27Ez/1436QlR8KG4UKqYsSIM 7z2A== X-Gm-Message-State: AOAM530P7YVNdw6ijLOxpd1u+nondEfJpEoSjFYir7r+67UbLSD/MwaG 9vt1bM3vCvzUnYozTcGzpZNNSBrH/KkD3WOMM40= X-Google-Smtp-Source: ABdhPJyW7IBASB2u5tcWBKWMjarRh2V6w5knQx9QStA7FkffOR6oqL+tAQl7zuW3a595pYQD1swZrXM3kWleVFnQnDw= X-Received: by 2002:a05:6602:1591:: with SMTP id e17mr13311634iow.171.1643192303535; Wed, 26 Jan 2022 02:18:23 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: zimoun Date: Wed, 26 Jan 2022 11:18:12 +0100 Message-ID: Content-Type: text/plain; charset="UTF-8" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sharlatan Hellseher , 53533@debbugs.gnu.org, Mathieu Othacehe Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1643192351; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=UPxSSeE8C5LBi73/ayWes6ABMVxIxT4umpwGjoiqk/c=; b=j1++hbl48EzcvOF2D7mfe+rZiTOCy02KLL2NqE1EznmxOYbIvB8N7lDdv3k9o0xDZL76qi wIbwU4xrrelo2vavXiG3gC9mpSGYMvaTRS7l9mxQ0he7qE1y6UyHmoyMI90wG9ClO2Kwkb pN12mxd0vDm3vD3sSZX1WQC4eAC5Adb7gF5L5HtXAPdXH9llgbGa4CBuJsuqmanZegVxwd bN6l62CVAhMDJo82ZmfvInQWXe0zEenbbJMoxWWsj1vgxxKqHPKb49HYazQGHYy2EZynjV tDiszXI/uBoohZIp8/EW2RZMEgqAORXY7th/TploKBGCuh/YV4tkun3/oLq0DQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1643192351; a=rsa-sha256; cv=none; b=IeMmOPgrefO3kPzflxTDfglv/rf/09F+g8dNlrYkPVhQu5uyXbBVfnpvHy7x+21qlm8V3x SParPcCteywm2BfVHHqHyf8auQpXEC6bzns+LhDGAgbcvJnctmmwpl97BTWIGP3ia+7ArN mcbHKbaVdCmFsRE4AdxE0mk/3Dc6wqmENZU/Fy+Lf3JsNUEBIPLHO2cNql1LfYCQSdibnG UuUUh0imS/KdK6OeCM1prkaVtBgNaoHkRXpQ7zlxtQAajD4o72IauocaYJQh1rCzVhPmAL p5x5V0LMpsR0GIoNmPAm2Jw1xAAYqQOI+SrM4SAY0dGLw8x5IMtMsBhQc4zs4w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=o+NMb0Z9; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.53 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=o+NMb0Z9; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 51AB62C330 X-Spam-Score: -2.53 X-Migadu-Scanner: scn0.migadu.com X-TUID: 05azccrTJjfS Hi, On Wed, 26 Jan 2022 at 03:53, Leo Famulari wrote: > It's not easy to make sure that all packages build successfully. I've > never seen 100% of packages build successfully since I joined Guix in > 2015. It's a nice goal but it requires some work... a lot more work. I think this goal is impossible by design of the current workflow. Do not take me wrong, I am saying that some clean is not necessary. My aim with this email is to point that the annoyance of broken packages comes from our current workflow and our scaling up, yeah! :-) > Everyone is invited to help. Probably, the first step is to remove > several hundred packages that don't build; it might even be a couple > thousand. We agreed to do that in 2019 with a reminder on June, 20th 2020: and nothing happened. Because the issue is the current modelling: master/staging/core-updates. Continuing with that workflow, it is not realistic to purge broken packages. Here, we have to make a choice: a) current workflow with broken packages b) adopt another workflow The pro for a) is that it is already working enough, we have the habits, etc. And the cons of a) are some packages break sometimes. My rough guess is, these days, ~5% of packages are broken per revision. And my guess is that this percentage of broken package stays the same as the number of packages increases; other said, more packages often means more users, so this ~5% means more "visibility" to hit a broken package. For b), the switch comes with a cost and I am not convinced all people are agreeing if this cost would be counter-balanced. More details about such discussion are for instance in this thread: and the archive guix-devel contains many similar discussions. Nothing new. ;-) > > I would like to conclude from the CI is which commit broke how many > > packages. > > Agreed, this is a very important missing feature. Also, we need the > capability to compare commits in terms of CI results. The idea was to have this feature provided by the Data Service. However, AFAIK, many pieces are since then broken between ci.guix.gnu.org and data.guix.gnu.org. For instance, using this: you can detect which commit breaks the package. Well, 2 important notes: 1. All is Scheduled because data.guix.gnu.org is fetching data from the other CI managed by the Build Coordinator, not from the CI managed by Cuirass. 2. The commits listed there are an approximation. Basically, there are based on the mailing list guix-commits and a complete batch pushed as once is considered as a change, and more than often, this batch contains irrelevant change, see details there: > > Some missing practice of packaging: > > - Some essential message of the reason why tests were disabled and any > > sort of suggestions on how to > > make them enabled. Contact upstream if required. > > Everyone is supposed to do this when writing packages. It's a failure of > the code review process if that is not happening. I would not say it is a failure of the code review process. Instead, I would say it is a failure of the current workflow. On one hand, the project cannot complain that not enough people are doing reviewing (see this thread [1]), when on the other hand, potential reviewer would have to build many more than just the change itself, especially when many potential reviewers do not have enough power at hand (see this thread [2]). 1: 2: Note that since I joined the project (around 2018), we are discussing about more automation using patchwork and other similar tools. For instance, Chris started this initiative: which would really help to improve the CI process. However, we, as a project, are not putting enough love in such initiative. Well, the hidden point barely discussed is twofold: necessity of a powerful infrastructure for building more when the resource are somehow splitted and strong integration with current CI. Both require many more resources: manpower and money. That's why, I think the current workflow will make more visible broken packages. > That doesn't mean that Guix QA is perfect, but rather that your > measurement is inaccurate and misleading. Yes, this 1.1% does not make sense. Firstly, counting the commits containing "Fix build" completely underestimates the number of real fixes, because some fixes contains instead "Fix tests", some fixes are update, etc. Secondly, the important number is the number of broken packages at one instant t (revision) and for that, the Git history is useless. Basically, the important number is the ratio between the green vs red bullets [3]. Other said, compare the coverage, say, using "guix weather" for past revisions. I guess, ~5% of packages are broken, on average. 3: My 2 cents. Cheers, simon PS: The discussion usually happens on guix-devel and not in the bug tracker. Other said, since it does not appear to me an actionable bug report, I would like to close it.