From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id MK7CJ261T1/GGAAA0tVLHw (envelope-from ) for ; Wed, 02 Sep 2020 15:08:30 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id OF2NI261T192RwAAB5/wlQ (envelope-from ) for ; Wed, 02 Sep 2020 15:08:30 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 23FD29402C8 for ; Wed, 2 Sep 2020 15:08:30 +0000 (UTC) Received: from localhost ([::1]:50738 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kDUNB-00088N-36 for larch@yhetil.org; Wed, 02 Sep 2020 11:08:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42092) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kDUN0-00085y-BY for guix-devel@gnu.org; Wed, 02 Sep 2020 11:08:18 -0400 Received: from mail-qt1-x82c.google.com ([2607:f8b0:4864:20::82c]:40766) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kDUMy-0007TI-Fk; Wed, 02 Sep 2020 11:08:18 -0400 Received: by mail-qt1-x82c.google.com with SMTP id v54so3787469qtj.7; Wed, 02 Sep 2020 08:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=7JmAJxdkRiW0d1seiZCCZmGC7giR4uAaJlq1o1shjig=; b=cRiFdm7Nvfm0LOaX4brix29+3qRAeNYjOQgBO87fbj67lyNmX2E0Nj0dYhZfPUgHMo XJmG5uWDOROc0vUK24PAPdtqTLNgFOCKLOIf7RdkgJq1WUE3x2vNpsAOOpoNFXxSt3/i Hr9+IPvgBRfz2FHl4cFTemxUXFBpFIBDfCkDJ27yYFyRbYsw5kQ2TRi6awp4sXx2alRi xUACXljTK5upit5FETaeMseCwgOuSGWRNdyDL1+7o1BO4ap74WT62ft9mR4BjWFLzrYZ mMhztiOzTTdGCHdFbVBg8VYx1o1XejNsmC/SWI9OSpALV2qfpacDTboz0zGEHOqaSXI9 s3gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=7JmAJxdkRiW0d1seiZCCZmGC7giR4uAaJlq1o1shjig=; b=m2hUXuwBMn3wtZzQ2kRL4mkZI1o0ELFfyBXJQ6KCoyRBLea09SOJhFW5rFfpNPIMis CsTzUdJsyl3fHhC8b4aYml796yMwHvUkmJbUYgss2iavRTHtwhMdA/nsEc8/DTjzP0si CJZUHM1/pnT9Gs+TjcfJfpGaaXkyFvF2CF9KjSTGBYJow0FLgCkGHWbqqL/Bjcs+QUHV kxIxZzlgLBb331wxWAjQmDJCQCN7Fct8PcaqMsk0ZLWbW9Ts4hyND2RRGYn2A7EvdhBK SY/eXO0O+y982Q1RIc8nCTYh6FliOoyHgmSmjbdYxAYSpm08yekfrBsqH1eHoiHgItE5 pkwA== X-Gm-Message-State: AOAM532uZmbbXl2arrfJ8cRg5+TU1Cvzee5N3rwYxASP1DS2/2fi2urO jdgdoBUN9BC48tP4iutdWbi1cSovQikUug== X-Google-Smtp-Source: ABdhPJxUhMXPAWss60U4hGIUa5ZAqmh4aiL/L+oZHBrlb2yzaSrm4y9Klr5/0xwZX4FbI2Ihy7VhDQ== X-Received: by 2002:aed:2946:: with SMTP id s64mr7020605qtd.335.1599059293919; Wed, 02 Sep 2020 08:08:13 -0700 (PDT) Received: from hurd (dsl-10-133-254.b2b2c.ca. [72.10.133.254]) by smtp.gmail.com with ESMTPSA id 203sm4747301qke.59.2020.09.02.08.08.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Sep 2020 08:08:13 -0700 (PDT) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Dependency cycle issues when using a Gexp-based snippet References: <87h7ssx07p.fsf@gmail.com> <87wo1nzr6p.fsf@gnu.org> Date: Wed, 02 Sep 2020 11:08:27 -0400 In-Reply-To: <87wo1nzr6p.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 24 Aug 2020 23:09:50 +0200") Message-ID: <87mu286wt0.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2607:f8b0:4864:20::82c; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x82c.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=cRiFdm7N; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: -1.71 X-TUID: HG6khkK+L4oB Hi Ludovic! Thank you for the reply. Ludovic Court=C3=A8s writes: > Hello! > > maxim.cournoyer@gmail.com skribis: > >> While trying to move some of the patching done to qtbase into a snippet, >> with the goal of having at least the ./configure script runnable in a >> guix environment without having to manually run patching phases: > > [...] > >> I encountered the following issue, which seems similar to one >> encountered by Ricardo in 2016 [0]: >> >> ice-9/eval.scm:293:34: error: canonical-package: unbound variable >> hint: Did you forget a `use-modules' form? >> >> >> The origin can be correctly built at the REPL, so the problem indeed >> seems to be a dependency cycle. > > Indeed: the problem is that when loading this module, we try to resolve > one of the variables referenced in the snippet, but that variable is not > defined yet because it comes from a module that=E2=80=99s in a dependency= circle > with the one at hand. > >> Attempting a suggested fix by Ludovic in that same conversation [0], >> namely, making the snippet field of the record a thunked one: >> >> modified guix/packages.scm >> @@ -250,7 +250,8 @@ as base32. Otherwise, it must be a bytevector." >> (patches origin-patches ; list of file names >> (default '()) (delayed)) >> >> - (snippet origin-snippet (default #f)) ; sexp or #f >> + (snippet origin-snippet >> + (default #f) (thunked)) ; sexp or #f >> (patch-flags origin-patch-flags ; list of strings >> (default '("-p1"))) > > We should check what this change costs in CPU and memory, but it=E2=80=99s > probably worth it. As Marius noted before, the snippets for > ungoogled-chromium and linux-libre are contrived because of this > limitation. (Perhaps we can use =E2=80=98delayed=E2=80=99 instead of =E2= =80=98thunked=E2=80=99.) What is the difference between delayed and thunked? Would a thunked capture the closure of its environment while delayed not? Is the closure useful to access record-bound values such as the version field of a package? I checked the usage at compilation and run time, using the 'time' command (aliased to time+ on my system), and didn't find any meaningful difference whether the snippet is made a thunked or delayed field, or none (current situation): On current master: --8<---------------cut here---------------start------------->8--- time+ make -j8 2436.29user 56.47system 14:29.36elapsed 286%CPU (0avgtext+0avgdata 870828ma= xresident)k 5480inputs+405952outputs (71major+320522minor)pagefaults 0swaps time+ ./pre-inst-env guix package -A | wc -l 9.87user 0.24system 0:06.51elapsed 155%CPU (0avgtext+0avgdata 281564maxresi= dent)k 0inputs+0outputs (0major+25636minor)pagefaults 0swaps 14702 --8<---------------cut here---------------end--------------->8--- With delayed source field: --8<---------------cut here---------------start------------->8--- time+ make -j8 2493.40user 51.16system 16:29.33elapsed 257%CPU (0avgtext+0avgdata 842008ma= xresident)k 48888inputs+413640outputs (158major+340632minor)pagefaults 0swaps time+ ./pre-inst-env guix package -A | wc -l 9.68user 0.26system 0:06.68elapsed 148%CPU (0avgtext+0avgdata 293488maxresi= dent)k 1008inputs+0outputs (7major+42943minor)pagefaults 0swaps 14702 --8<---------------cut here---------------end--------------->8--- And finally with thunked source field: --8<---------------cut here---------------start------------->8--- time+ make -j8 2302.67user 27.44system 22:32.82elapsed 172%CPU (0avgtext+0avgdata 862336ma= xresident)k 2960inputs+416688outputs (51major+300689minor)pagefaults 0swaps time+ ./pre-inst-env guix package -A | wc -l 10.02user 0.23system 0:07.46elapsed 137%CPU (0avgtext+0avgdata 283052maxres= ident)k 2984inputs+0outputs (101major+24685minor)pagefaults 0swaps 14702 --8<---------------cut here---------------end--------------->8--- Based on this, I would choose the most capable 'mode' for the source field, which I presume is thunked. >> It now seems a new cycle is introduced because trying to build anything >> hangs using the CPU with slowly increasing memory usage: > > Hmm not sure exactly why, but look: > > + (snippet > + (with-imported-modules '((guix build utils)) > + #~(begin > + (use-modules (guix build utils)) > + ;; corelib uses bundled harfbuzz, md4, md5, sha3 > + (with-directory-excursion "src/3rdparty" > + (for-each delete-file-recursively > + (list "double-conversion" "freetype" "h= arfbuzz-ng" > + "libpng" "libjpeg" "pcre2" "sqlit= e" "xcb" > + "zlib"))) > + > + (let ((coreutils #+(canonical-package coreutils))) > + (substitute* "configure" > + (("/bin/pwd") > + (string-append coreutils "/bin/pwd"))) > + (substitute* "src/corelib/global/global.pri" > + (("/bin/ls") > + (string-append coreutils "/bin/ls")))) > + #t))))) > > Such substitutions are system-dependent; thus, they should be made in a > phase, not in a snippet. Perhaps we=E2=80=99ll sidestep the issue altoge= ther? > :-) Indeed. I didn't consider this aspect well. Apart from being inefficient (the sources of a package would be different for each system) it would still technically work, no? But yeah, that bloat is probably not worth the extra convenience (of having a source that can be built straight from 'guix build --source'). Thanks, Maxim