From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id uC1ZDF4FS2SObgEASxT56A (envelope-from ) for ; Fri, 28 Apr 2023 01:29:34 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id GIRIDF4FS2T2xgAA9RJhRA (envelope-from ) for ; Fri, 28 Apr 2023 01:29:34 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EB67B2776 for ; Fri, 28 Apr 2023 01:29:33 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1psB2m-0002x2-8s; Thu, 27 Apr 2023 19:28:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1psB2k-0002ws-Br for guix-devel@gnu.org; Thu, 27 Apr 2023 19:28:54 -0400 Received: from mailout.easymail.ca ([64.68.200.34]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1psB2c-0007xV-Jj for guix-devel@gnu.org; Thu, 27 Apr 2023 19:28:53 -0400 Received: from localhost (localhost [127.0.0.1]) by mailout.easymail.ca (Postfix) with ESMTP id 2068069E30; Thu, 27 Apr 2023 23:28:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bokr.com; s=easymail; t=1682638123; bh=uuP7AEU3MhUtB6c0hIeiTL1DM0zNoXQ8z3TERh8uQhc=; h=From:Date:To:Cc:Subject:References:In-Reply-To:From; b=DpQA/iqBffZFpcdPmo2fBvT3PV3bDWm1ry2fLWsSJGE2GIB/fmVrwCYPFClbQ/2Pt PfekAP/0yhuGRL3l4tgXV1DT7QjuuMQUe6riugnlSNoO4X0/IFQRUBXHviKM6Y1l28 iJLbtq3fUQURluNq2nKHZBvXLGd8xHQGIFFdmRx01lDT9E68T6AtQrPerxBWe9H+PT WAahL81n2Kqj0Ln+zwM80LBfQCTmT5XrsJ/3eGjMq2Dr/p86e0r3hsjPUcSLOu5CVA adN9wAgyPzWrlnkqOatjg2PgMKeMys5fjeji8aVbDYs39dEiqw+53oJZqrjyz9jVr9 siVP52uhGdoLA== X-Virus-Scanned: Debian amavisd-new at emo07-pco.easydns.vpn Received: from mailout.easymail.ca ([127.0.0.1]) by localhost (emo07-pco.easydns.vpn [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZtAOIC74VUyQ; Thu, 27 Apr 2023 23:28:42 +0000 (UTC) Received: from localhost (m37-199-0-166.cust.tele2.se [37.199.0.166]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mailout.easymail.ca (Postfix) with ESMTPSA id 414AF69E28; Thu, 27 Apr 2023 23:28:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bokr.com; s=easymail; t=1682638122; bh=uuP7AEU3MhUtB6c0hIeiTL1DM0zNoXQ8z3TERh8uQhc=; h=From:Date:To:Cc:Subject:References:In-Reply-To:From; b=W2Bx+wXFGd9HobRpYQBzKNtCXj7iVU1IimO9lZrtaVlpWQNFyfsl5x4KkmjFmU77j ikuvCfAKhx+Axf1E7ovZanijFUYy8LMaqRyLO4aOfD/d7+8rfIuwYo557TvWFNyIu8 fnAwqOkPZgQxKIf1NHOsffC16kFdh7JNaJ+MV2euQO6S41gmTY37RUSkTCEYmlszwW 9bZBmIrXAxj1rHfcxv8sAFdbDb7yXkp0yXsaWEVD0TnOn0LNIEi1sCQNg7wu/Kj/4J SDyQs5aSAw4h3rgJ929YBl+CGVM5LhJ0T9n0oocs1/huVTOUuQ25oHkWCnN+1wakgA VcuYZDnePTBYw== From: bokr@bokr.com Date: Fri, 28 Apr 2023 01:28:26 +0200 To: Simon Josefsson Cc: Andreas Enge , guix-devel@gnu.org Subject: Re: Build dependency inflation (was: Re: Core-updates merge) Message-ID: <20230427232826.GA3911@LionPure> References: <87a5ytw6nd.fsf@kaka.sjd.se> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <87a5ytw6nd.fsf@kaka.sjd.se> User-Agent: Mutt/1.10.1 (2018-07-13) Received-SPF: pass client-ip=64.68.200.34; envelope-from=bokr@bokr.com; helo=mailout.easymail.ca X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Seal: i=1; s=key1; d=yhetil.org; t=1682638174; a=rsa-sha256; cv=none; b=Vpn3dUnxPN98rT6eKaYydW0kkpylxPLwHenb0DMtJTdwADUrI6Tb/Qw/QTHTLUpbSh/B3w dvCcyAH9ad6/SRpTFRJgHEYNwaaEkGNWmSb3SynqR8Nxyu8e3egjJhlxR5i1U/0ZFoE435 6jaWaCWVxMk8yqPuk9W2qY66SmWw+t1kEGSiI95B3IpZS5N3l83epUEa0m4AO1zBlIhQSo dqU7qGwBQgvUnooR3NXKfUj2I6B/3uRqHyrwTCQj485CVw+kfazeZKj73P68NtRcqZpsKk po3lMPTQk4tQScic3plYr1pGgGB7ygCMVC+7RWzLmOdL3Uxn5VFAAHi9FnJtGQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=bokr.com header.s=easymail header.b="DpQA/iqB"; dkim=fail ("headers rsa verify failed") header.d=bokr.com header.s=easymail header.b=W2Bx+wXF; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1682638174; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=00ITJ7/zrfAsMIz4El9c+8oK3d06MYWZ0RC5wCJnCg0=; b=Nr6DhFm1cpY1RGE1/fkvQcDhnEeFSpfU+sD61TdqvtVpUg3xzVEqTqj3oD6/Kuuej5NKZJ SUFB4UM3tzjAAXcq7DxwPnsjj0oWZhd0fQJbEjh9YYh2Vqr+ffKqAFry/rEdTD4eeeM7w8 8/TwdAnsKwUWV2H/1br32bQDTi+tDo1jvX1/9yQeCloVjP6eEX+xI2arwL28eHjt5HFziG levP/z97fnN6V1tpjgNEJVV10NFINNedZ3GTwnRWzOBuq2j79JStrHwro6UzyoBmPgHF80 mGasgH81IYtrD81dNlrspm6mqp8K9ZuyaNxRhiEyuCV2aBazNLZDlxYA8wFvpQ== X-Migadu-Scanner: scn1.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=bokr.com header.s=easymail header.b="DpQA/iqB"; dkim=fail ("headers rsa verify failed") header.d=bokr.com header.s=easymail header.b=W2Bx+wXF; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 2.80 X-Spam-Score: 2.80 X-Migadu-Queue-Id: EB67B2776 X-TUID: t4GG/0ccMILy Hi, TL;DR: If we have 22k sources, how about expressing dependency as a 22k x 22k matrix of weights Wij and dependency by multiplication of 22k x 1 matrix os sources variables ordered Sj by immediate dependency first. I guess the Sj column matrix is transposed to do the multiplication, so the Wij j's and Sj j's match. Start with weights of only 1 0r 0. Later maybe measures of compile times? The diagonal Wjj would be all ones, and counts of ones in vertical columns Wkj would be maximal if that Sj was directly depended on for a lot of k's Reordering the Wjj and Sj sort of like solving simultaneous equations and colorizing somehow to show the clusters of Wij dependency bits vs a backgrond of zero bits might make an interesting display. WDYT? Any GnuPlot whizzes out there? :-) Regards, Bengt Richter On +2023-04-27 19:56:38 +0200, Simon Josefsson via Development of GNU Guix and the GNU System distribution. wrote: > Andreas Enge writes: > > > - Too much in Guix depends on too much else, which makes building things > > needlessly entangled; in particular time zone data should not be referred > > to by packages, but be loaded at runtime (Leo Famulari). > > This is an important open problem -- is there any way to attack this > problem in a systematic way? I guess it is hard to understand which > packages ends up depending on what since it is a large graph with long > cycles, and also to understand which build depencies are essential and > which are superficial, and thus consequently challenging to know where > to start working to reduce build dependencies? > > I wonder if it is possible to graph all the build dependencies, and do > something like a monte-carlo what-if simulation: randomly pick one > build-dependency from the entire build-dependency graph and remove it, > and recompute the graph. If the difference between these two graphs > leads to a significantly lower total build computational cost than > before, we may be on to something. My theory is that "true" build > dependencies will show up in so many places that removing just one > instance of it will not affect total build time. But "needless" build > dependencies may occur just once or few times, and this approach would > catch low-hanging fruit to work on. Maybe the simulation needs to be > done on more than just removing one build-dependency, it could play > what-if games removing two build-dependencies etc, or three random > build-dependencies, and so on. Maybe my idea is flawed, and this will > only lead to a list of build-dependencies that are impossible to get rid > off anyway. > > Is there some other method to understand what build dependencies would > be important to remove, to speed up total rebuild time? > > Maybe we could analyze how much of a particular build-dependency > actually is used by a particular build? By looking into file-access > patterns etc. > > /Simon