From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id qTdjLfTwOWUlBAEA9RJhRA:P1 (envelope-from ) for ; Thu, 26 Oct 2023 06:54:12 +0200 Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id qTdjLfTwOWUlBAEA9RJhRA (envelope-from ) for ; Thu, 26 Oct 2023 06:54:12 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E6E50421AF for ; Thu, 26 Oct 2023 06:54:11 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=phyd0mLE; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1698296052; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=7X9vi4KUksRVpGVI+dnD9Kjjl9sAc9gxDM3IN0HrcH0=; b=pDSNxSSf1ryO9QB1wVbtg7nis86iDD3eK3zCkFmfHZ5OUiFJz/qKADiuSdxCwjojMCYZz+ +6bwR3tuJMdYoshbg1aG2bHHXbHN8e3s9iKbZyJ172LDiAG5gWsgZ4+V25uas1XEikpLu+ TV0Yi8N2IFnPf4/yvRADHVaGp0eyNSF0JWVRs+8gNghZ4yUCg1rQWE4sbO6jfj3cY6ulJz SDOjYzYwL6JWSDUwvSuUt0FIiX1rj7TsdaRtwJHBdDH/ILB2ZAxUbIiVRkuVxzCBAzAWGC wsI9jrR4Yfb4SxFfNd8Qi9gfXcsxqtX6IvQXUqZa25h3XffabDrhSRLjLisQ8Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=phyd0mLE; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1698296052; a=rsa-sha256; cv=none; b=flfw8mDLrU7Hxy/hO9UHW5Mo3WFLtpdFf6n/dI0M4iEHz7CGAkSKmp3PoXjcoDHfJ9l+27 ljTq583uGivHI7LbQDFGxK4DF+h4TDXUf16aSg6zkR3bDQsHTL0rxGsJ2KujmKorw18YI9 Rm47IcBgkJyoXZmrLEWEnEk4QLFZK+MUDn3dmgTzwyp3Q994FGKxhm3Ffs9ioY+rbPeA34 75SfIGCjhsXvnaT6ze5CzmkbJaekPZ91ftkuWOMa8EpNfGQ1o5Q2HawBIcdZSmKF1pYQM7 UKlHg/xxcgFJttOVfCilaELaqD5KkYCWZuoVyfuLPTfEL3QjyK6SOmwmOfqcyw== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qvrvH-00081P-Jy; Thu, 26 Oct 2023 00:24:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qvrv6-000817-VG for guix-patches@gnu.org; Thu, 26 Oct 2023 00:24:33 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qvrv6-0005dw-3m for guix-patches@gnu.org; Thu, 26 Oct 2023 00:24:32 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qvrva-0001bK-Ku for guix-patches@gnu.org; Thu, 26 Oct 2023 00:25:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#66720] [PATCH] gnu: icecat: honor parallel-job-count. Resent-From: Eric Bavier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 26 Oct 2023 04:25:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66720 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: =?UTF-8?Q?Cl=C3=A9ment?= Lassieur Cc: Jonathan Brielmaier , 66720@debbugs.gnu.org Received: via spool by 66720-submit@debbugs.gnu.org id=B66720.16982942586090 (code B ref 66720); Thu, 26 Oct 2023 04:25:02 +0000 Received: (at 66720) by debbugs.gnu.org; 26 Oct 2023 04:24:18 +0000 Received: from localhost ([127.0.0.1]:60396 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qvrur-0001a9-MB for submit@debbugs.gnu.org; Thu, 26 Oct 2023 00:24:18 -0400 Received: from mout01.posteo.de ([185.67.36.65]:43245) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qvruo-0001Zt-Ut for 66720@debbugs.gnu.org; Thu, 26 Oct 2023 00:24:15 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 2A98E240028 for <66720@debbugs.gnu.org>; Thu, 26 Oct 2023 06:23:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1698294218; bh=OgIrnDYnF0ylSH5gShP+AeleQlh0AOpLylQSItvruqU=; h=Message-ID:Subject:From:To:Cc:Date:Content-Transfer-Encoding: MIME-Version:From; b=phyd0mLEkmSGTiKCZJoC9oa4n4XMeu1metGTl/gGAfSx/1qjvGlfFnI6wOCxDL5Gq bKH+fiD+E4Jy5tsODu5xrwlCZNPjlPMsHqtoh/3RLWiMRMxn643gD4z4BrJcFsxXrK xkzqQMi4xipbj7F3BT0pMY7MT3nfwKnEwzqM1V0ID+2b8OXmTwZZAA9ziCcrTv8sfs jorTudVozHRgLJbmhMp4KMGPguHMLd6C2TAABlLAXugsQlEEjPunlHQ4AVrRfAfvYz rqTFk7P9SuPGvhleNJHqAX52C7VX6ipRs8X1Cf+81LOl51kViAPDY1CzFnrIYYmBb4 R2MDsl3dei0XA== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4SGCN866G0z9rxB; Thu, 26 Oct 2023 06:23:36 +0200 (CEST) Message-ID: <921ec2aa3bb7640b9aa27873022f410f187631b3.camel@posteo.net> From: Eric Bavier Date: Thu, 26 Oct 2023 04:23:30 +0000 In-Reply-To: <877cnay0qs.fsf@lassieur.org> References: <64b0fef1896198626ba2248438c3253fabe4e075.1698117931.git.bavier@posteo.net> <877cnay0qs.fsf@lassieur.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx2.migadu.com X-Migadu-Spam-Score: 2.70 X-Spam-Score: 2.70 X-Migadu-Queue-Id: E6E50421AF X-TUID: dbvmaO9Egm+L Hello Cl=C3=A9ment, Thank you for your reply. =20 On Wed, 2023-10-25 at 21:12 +0200, Cl=C3=A9ment Lassieur wrote: > Eric Bavier writes: >=20 > > - ;; mach will use parallel build if possible by de= fault > > - `(,@(if parallel-build? > > - '() > > - '("-j1")) > > + ;; mach will use a wide parallel build if possibl= e by > > + ;; default, so reign it in if requested. >=20 > It seems like Icecat makes a choice based on available memory. Why do > you want to override this with something that would potentially not work > if memory is lacking? I think our concerns roughly overlap. I wasn't aware that it considers available memory. It didn't seem that way to me. I will typically set `--cores=3D2` for guix builds on my system, to otherwise keep it available for other use. Recently, with no substitute available, I found my system grinding to a halt while building icecat. It was using every core on the system, and filling all of my RAM (I had not activated a swap space at this point). I think this is the code in question, from ./python/mozbuild/mozbuild/build_commands.py: if num_jobs =3D=3D 0: = =20 if job_size =3D=3D 0: = =20 job_size =3D 2.0 if self.substs.get("CC_TYPE") =3D=3D "gcc" else 1.0 = # GiB=20 = = = =20 cpus =3D multiprocessing.cpu_count() = =20 if not psutil or not job_size: = =20 num_jobs =3D cpus = =20 else: = =20 mem_gb =3D psutil.virtual_memory().total / 1024 ** 3 = =20 from_mem =3D round(mem_gb / job_size) = =20 num_jobs =3D max(1, min(cpus, from_mem)) = =20 print( = =20 " Parallelism determined by memory: using %d jobs for %d cores "= =20 "based on %.1f GiB RAM and estimated job size of %.1f GiB" = =20 % (num_jobs, cpus, mem_gb, job_size) = =20 ) So there's no fancy load balancing going on, just based on total virtual memory, assuming 2GiB per build job. For a dedicated build machine this is probably fine, or in the situation you bring up, where total system memory = is lacking. But it is not great when available free memory is lacking, such a= s an in-use desktop system. While it's not a perfect proxy for system load, I feel like, in general, packages should honor the `--cores=3DN` build option. If they take other things into consideration, we should try to work with that too. Perhaps we could figure something out such that `--cores=3DN` sets an upper limit and icecat's `mach` is able to cap that further based on system virtual memory? `~Eric