From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id OLa4CFGcvWEeqwAAgWs5BA (envelope-from ) for ; Sat, 18 Dec 2021 09:31:13 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id gEl4BFGcvWGNKgAA1q6Kng (envelope-from ) for ; Sat, 18 Dec 2021 08:31:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BD30435BDC for ; Sat, 18 Dec 2021 09:31:12 +0100 (CET) Received: from localhost ([::1]:45134 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1myV7Y-0006JV-0t for larch@yhetil.org; Sat, 18 Dec 2021 03:31:12 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38850) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1myV7O-0006J8-Ub for guix-patches@gnu.org; Sat, 18 Dec 2021 03:31:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:57933) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1myV7O-0005rE-MK for guix-patches@gnu.org; Sat, 18 Dec 2021 03:31:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1myV7O-0001gf-Gj for guix-patches@gnu.org; Sat, 18 Dec 2021 03:31:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#51838] [PATCH v5 07/45] guix: node-build-system: Add #:absent-dependencies argument. Resent-From: Liliana Marie Prikler Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 18 Dec 2021 08:31:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51838 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Timothy Sample Cc: 51838@debbugs.gnu.org, Pierre Langlois , Jelle Licht , Philip McGrath Received: via spool by 51838-submit@debbugs.gnu.org id=B51838.16398162266443 (code B ref 51838); Sat, 18 Dec 2021 08:31:02 +0000 Received: (at 51838) by debbugs.gnu.org; 18 Dec 2021 08:30:26 +0000 Received: from localhost ([127.0.0.1]:41246 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1myV6n-0001fq-Vi for submit@debbugs.gnu.org; Sat, 18 Dec 2021 03:30:26 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:37629) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1myV6l-0001fd-Pa for 51838@debbugs.gnu.org; Sat, 18 Dec 2021 03:30:24 -0500 Received: by mail-wr1-f67.google.com with SMTP id t26so8472545wrb.4 for <51838@debbugs.gnu.org>; Sat, 18 Dec 2021 00:30:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=HPBT34Bd4d/385NLcMxIMiBAZiH9adqlM2nGkEuWMFs=; b=j6QMYGIAnwK5tBEMP6AT7UOqoAK0vfBP9l/xOII+hYbWq3TZUhnPtGY4OQUvSSCQiC LZmOJrD9sR//IoZUiDO9Kpzw8XrFjntXh+W2oPTBC7sOx0XTvrn5WUpQscX+A/z5SbqY zc39NXxYR/CUXdkyyPwmocb/WayyL0Ps+mtW2evwKopvbMMzJf3eVGlumhMayi9Sd84C Dh9dmzobtSUXFXqTNU6KHJDY5btr5IqBy+hDbOpCCvOYe8PU3Y/SgEAB5ZcSLtz6lwuQ IW4NYgFG2t6IdhdALHYhenSMNCF3t0c1+dZf26oIii+0uOjzMRq5p7x8B+3ltwqZ19U8 vs+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=HPBT34Bd4d/385NLcMxIMiBAZiH9adqlM2nGkEuWMFs=; b=bSdS0atMZnvVyatutR43d1bvqhclq3OqZmFwytbMZ9DwfMBCHzkX9P8Cs07jys2Gf0 kM+EhKJNrZ6wNq9sHlTTHvBWh7BCR9Gal4yFf7CNKGNj6IXW1W8g4cySBrLucz0R65JI LEu1fO6vLWiYi4GynxFo0mKNx9vTbutXbBrmmQctyLyrjo1dhr6bDSuLcvdJVyy90yWC Lh869haow32/IIoe9ccOe4YSA7IWoeINkAaOaW78BmK4D3+hnZLuBSwx7Q7TBxYbYkFK O0ZqPY6/p1rqX/urbgI1cPoa4sDW6+Z07/ik3xnA4an5l+eQDsFjYlRVVBRrW8aiTq4o PvzQ== X-Gm-Message-State: AOAM530BzLfPhlLBd6tp2eAsOEVOyCaIZWlAV/coYIHi+O+tLMInBa9k 87HDPOtpimS/8Rmp+FGWmis= X-Google-Smtp-Source: ABdhPJy5phsWOVWcOmFbLVsYyxTTm3nyXrlp61a/gC9DTiCAa1qkf6LISfI/sX9qAqaKoqg+ddo7QA== X-Received: by 2002:a5d:588c:: with SMTP id n12mr1449453wrf.363.1639816217893; Sat, 18 Dec 2021 00:30:17 -0800 (PST) Received: from nijino.fritz.box (85-127-52-93.dsl.dynamic.surfer.at. [85.127.52.93]) by smtp.gmail.com with ESMTPSA id w25sm9627241wmk.20.2021.12.18.00.30.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 18 Dec 2021 00:30:17 -0800 (PST) Message-ID: <815f327e36ecd24066179586997947ecd8f31150.camel@gmail.com> From: Liliana Marie Prikler Date: Sat, 18 Dec 2021 09:30:16 +0100 In-Reply-To: <871r2a7hme.fsf@ngyro.com> References: <20211213060107.129223-1-philip@philipmcgrath.com> <20211217020325.520821-1-philip@philipmcgrath.com> <20211217020325.520821-8-philip@philipmcgrath.com> <87k0g36xp3.fsf@ngyro.com> <871r2a7hme.fsf@ngyro.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.42.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639816272; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=HPBT34Bd4d/385NLcMxIMiBAZiH9adqlM2nGkEuWMFs=; b=YE+QwkEE5kI225p9woGVdwVNpEu3xOWvdN31O2IDJbwe2JjJ+gWKY3mhMucKdMcBncBGoU BE6eJ/xYr9X6/u5pGw1LOotG+CYeX3FpGuAJO4752ZhacD0YVULMMJiNZ0Nmgm+X+kETuH +CtlkfBBsTvoJHF5Kg2qI8gBujPneieNqfOz8bJg1Ve3V44ZPIqjKju6JAsGxRoJRZvvOF ZK0y96imnWQzmeHOryoe11iGYiVk1gVIe0JHvnZQ3BIIHG2tIWonRCRic3wL56p2N/Purh 4j0uwXy3QEzNFRxr3bikMnywSGPfXWIaDtaFvng+2oCK3PcH1kv0eHYhDItDKw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639816272; a=rsa-sha256; cv=none; b=J8nMbVhk/wNIr3JSm+NGl5u+bG+anxAnKSbxRngfdnKlCRNXZKCtt2nyPoO1HlqWHlV0KO GdaTMir9rAVK0wXpaXEMsuZ9yOkQ+5WC6ImBw2yiiFPzarF2zSwvzLL/VuBELJXyUHZ1OJ i5RHVMRMG1WOuSltMlh/T3Q7brvaALzWrYAtZ2eArLS242jgfw7UnU3rjC/KwuKGQIkllE CAajLi2xNSD+s/bgEt+9kockUoJOiBThRh2VNy+1tvYp+s/9gtXCKYAQ+AkZk8sB2q1OJK HMUAmaTps1xpGpn0O/JPVz2YSXaIej8aE0A99Nx2TmaugKJVKZs4+vwpzclugQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=j6QMYGIA; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.20 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=j6QMYGIA; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: BD30435BDC X-Spam-Score: -3.20 X-Migadu-Scanner: scn0.migadu.com X-TUID: nXpyEERYZTWv Hi, Am Freitag, dem 17.12.2021 um 21:48 -0500 schrieb Timothy Sample: > Liliana Marie Prikler writes: > > > For the GNU build system (and likewise meson-build-system), the > > default behaviour if you haven't specified anything as per upstream > > conventions is typically to error if the package is required and > > omit it if it's not.  The default behaviour of node-build-system > > (and likewise cargo and most other build systems that come with the > > advertisement of "we know package managers better than people who > > actually produce usefulpackage managers) is "Oh my god, you don't > > have an exact copy of the machine that built this stuff locally, I > > am going to barf huge walls of noise at you".  Therefore, we can't > > meaningfully compare those build systems in terms of strategies. > > NPM packages tend to wildly over-specify their dependencies.  We > already remove dependency version checking, and before this change, > many of our packages skipped any kind of dependency checking by > skipping the configure phase altogether.  To me, the ‘#:absent- > dependencies’ approach tries to work around the dependency over- > specification by listing exactly those things that are only there to > elicit a useless “Oh my god [...], I’m going to barf huge walls of > noise” message.  The rest of the dependencies are those that the Guix > package author deemed required (or at least important).  Basically, > ‘#:absent-dependencies’ helps us translate between the NPM culture of > over-specification (which is really a culture of prioritizing package > author over package user) and the GNU culture of “DWIM” dependencies. Except that it's not. The current workaround is "I know this thing's going to barf at me, so I prepare an umbrella and hope it has no holes". > > If we really want some static verification for node-build-system, I > > think we should take that as an approach rather than hard-coding > > (absent) dependencies literally everywhere. > > We need some way to know what to statically verify.  We can’t > magically know what’s important and what isn’t.  The two options in > this thread are ‘#:absent-dependencies’, and only checking what’s > already in the package’s inputs.  What worries me about the second > approach is that it offers no help when updating a package.  With > ‘#:absent-dependencies’, if the developer adds a new dependency that > really is required, we will get a build-time failure letting us > know.  Whoever is updating the package can fix it before even > committing anything.  If we just check the inputs, that’s not the > case, and we might end up with Philip’s “mysterious runtime error, > potentially many steps down a dependency chain.”  Hopefully tests > would catch it, but I like the extra assurance. That's why I didn't want to default to "do nothing", but to *warn* about missing dependencies in configure. Then whoever bumps the package will at least know which warnings are produced if they do so and they can cross-check by manually building past versions. Including #:absent-dependencies is no safe-guard against a failure here anyway. A dependency that was optional in V1 might become required in V2. > Another benefit is that it would help us gain knowledge as to which > NPM packages are often used but not actually required (e.g., NPM > publishing tools).  With this knowledge, we could write a clever NPM > importer that ignored obviously inessential dependencies. > > I guess I’m starting to sound like a broken record now – this is > basically what we covered before!  :)  Maybe we’re in need of a fresh > perspective.  (If anyone is reading along and has thoughts, feel free > to chime in!) I think the NPM convention is to put everything you need "at build time, but not at runtime" into dev-dependencies, no? In any case, one approach I could offer is to sanity-check by searching for require() statements and trying them in a controlled node environment. This could look something like eval("try { var dep = require('" + dependency + "'); true } catch (e) { false; }") Once we know where require statements are made and whether they succeed, we can start estimating the impact of a missing dependency. For this, it'd be nice to have a full function call graph, in which a node is coloured dirty if it has a failing require statement, lies within a module that has one or calls into a dirty node. However, as a primitive approximation we can also count the node modules with failing requires against those that don't. We set an arbitrary threshold of allowed failures, e.g. 0.42, and then check that whatever value we obtain from the above is lower than that. While that would be nice and all, I think the overall issue with the current node implementation in Guix is that 'configure' and 'sanity- check' are the same phase, so you have to disable both or none. I think we could easily do with a configure phase that does nothing, not even warn about a missing dependency, and a sanity-check phase that requires every dependency mentioned in package.json to be met. Packagers would then outright delete sanity-check as they do for python and as they did for configure (but not have configure fail due to it!) or deliberately rewrite the package.json for the sanity check and dropping absent dependencies, i.e. what you do minus the keyword. If later needed for the purposes of an importer, we would then still have that database and could at some point introduce the key #:insane- requirements. WDYT?