From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark H Weaver Subject: Re: Merging staging? Date: Thu, 20 Dec 2018 20:17:07 -0500 Message-ID: <8736qradap.fsf@netris.org> References: <87o99gvyce.fsf@netris.org> <6AD6EA9A-7634-4A8E-8A59-F3D35BC0F82A@lepiller.eu> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:40123) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ga9SE-0002mG-W2 for guix-devel@gnu.org; Thu, 20 Dec 2018 20:18:19 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ga9SB-0002Dk-A1 for guix-devel@gnu.org; Thu, 20 Dec 2018 20:18:18 -0500 Received: from world.peace.net ([64.112.178.59]:42014) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ga9SB-0002C3-4z for guix-devel@gnu.org; Thu, 20 Dec 2018 20:18:15 -0500 In-Reply-To: <6AD6EA9A-7634-4A8E-8A59-F3D35BC0F82A@lepiller.eu> (Julien Lepiller's message of "Thu, 20 Dec 2018 23:45:00 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Julien Lepiller Cc: guix-devel@gnu.org Hi Julien, I've rearranged your reply from "top-posting" style to "bottom-posting" style. Please consider using bottom-posting in the future. I wrote: > Julien Lepiller writes: > >> I'd like to get staging merged soon, as it wasn't for quite some >> time. Here are some stats about the current state of substitutes for >> staging: >> >> According to guix weather, we have: >> >> | architecture | berlin | hydra | >> +--------------+--------+-------+ >> | x86_64 | 36.5% | 81.7% | >> | i686 | 23.8% | 71.0% | >> | aarch64 | 22.2% | 00.0% | >> | armhf | 17.0% | 45.6% | >> >> What should the next step be? > > I think we should wait until the coverage on armhf and aarch64 have > become larger, for the sake of users on those systems. > > Also, I've seen some commits that make me wonder if hydra is still > being configured as an authorized substitute server on new Guix > installations. > Do you know? > > If 'berlin' is the only substitute server by default, then we certainly > need to wait for those numbers to get higher, no? > > What do you think? Julien Lepiller responded: > I agree, but I wonder if there is a reason for these to be so low? It's a good question. I have several hypotheses: * Unfortunately, it is fairly common for builds for important core packages to spuriously fail, often due to unreliable test suites, and to cause thousands of other important dependent packages to fail. When this happens on Hydra, I can see what's going on, and restart the build and all of its dependents. I wouldn't be surprised if some important core packages spuriously failed to build on Berlin, but we have no effective way to see what happened there. If that's the case, the 'guix weather' numbers above might never get much higher no matter how long we wait. * Berlin's build slots may have been occupied for long periods of time by 'test.*' jobs stuck in an endless "waiting for udevd..." loop, as described in . Hydra's web interface allows me to monitor active jobs and manually kill those stuck jobs when I find them. I don't know how to do that on Berlin. * Especially on armhf and aarch64, where Berlin has very little build capacity, and new builds are being added to Berlin's build queue must faster than they can be built, it is quite possible that Berlin is spending most of its effort on long-outdated builds. On Hydra, I can see when this is happening, and often intervene by cancelling large numbers of outdated builds on armhf, so that it remains focused on the most popular and up-to-date packages. * On WIP branches like 'core-updates' and 'staging', when a new evaluation is done, I cancel all outdated Hydra jobs on those branches. I don't know if anything similar is done on Berlin. In summary, there are several things that I regularly do to make efficient use of Hydra's limited build capacity. I periodically look at Berlin's web interface to see how it has progressed, but it is currently mostly a black box to me. I see no effective way to focus its limited resources on the most important builds, or to see when build slots are stuck. Regards, Mark