From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id wEJeOnMo3V8nCQAA0tVLHw (envelope-from ) for ; Fri, 18 Dec 2020 22:08:51 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id KOkSNnMo3V8eXQAAbx9fmQ (envelope-from ) for ; Fri, 18 Dec 2020 22:08:51 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5364594036A for ; Fri, 18 Dec 2020 22:08:51 +0000 (UTC) Received: from localhost ([::1]:46002 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kqNvd-0004Vu-Sf for larch@yhetil.org; Fri, 18 Dec 2020 17:08:49 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:39082) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kqNo6-0008Ip-LO for bug-guix@gnu.org; Fri, 18 Dec 2020 17:01:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56924) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kqNo6-0000mg-CI for bug-guix@gnu.org; Fri, 18 Dec 2020 17:01:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kqNo6-00084V-AB for bug-guix@gnu.org; Fri, 18 Dec 2020 17:01:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 18 Dec 2020 22:01:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 45316 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 45316@debbugs.gnu.org X-Debbugs-Original-To: bug-guix Received: via spool by submit@debbugs.gnu.org id=B.160832882030964 (code B ref -1); Fri, 18 Dec 2020 22:01:02 +0000 Received: (at submit) by debbugs.gnu.org; 18 Dec 2020 22:00:20 +0000 Received: from localhost ([127.0.0.1]:40237 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqNnP-00083M-Ua for submit@debbugs.gnu.org; Fri, 18 Dec 2020 17:00:20 -0500 Received: from lists.gnu.org ([209.51.188.17]:57304) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqNnO-00083C-Qn for submit@debbugs.gnu.org; Fri, 18 Dec 2020 17:00:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:38946) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kqNnO-000836-IK for bug-guix@gnu.org; Fri, 18 Dec 2020 17:00:18 -0500 Received: from mail-qt1-x836.google.com ([2607:f8b0:4864:20::836]:37083) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kqNnK-0000LD-8X for bug-guix@gnu.org; Fri, 18 Dec 2020 17:00:17 -0500 Received: by mail-qt1-x836.google.com with SMTP id z9so2455026qtn.4 for ; Fri, 18 Dec 2020 14:00:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=e/lVmjLLeCvEinWyi9JYGM5h7izbLSoXjALespNWLnc=; b=fZYroZQ8JINPpiC/nTonO/PhFEZqtEH5ZH4pW+2SuytLgaNDa/cwuwcue8nNawS5kc iAu/b2udU8NKLup1QPM0nbNnz2RRYqIZDyFMGBAo1HRS6QvvBkjfybM3E3WNwlz2gZFz KQPdosbrj7XO8EeTYk7o1TJbMW6SoyV9rdHiTAU8WQ1dGv2O1YkD3Ek+xle8BCc5wUFa IsfS13EPADmR9WukpzqaRwVKwpHj6Vs9FZL9kOlhaIucmjS8ZbaTV9LP4GwQ8qjjEtj5 YqWr+z7fdKS0tVSW4QfOVkMuYY8vglC+GolLkhcLe1ePuDwRBYX7bn6FUEzo2IX1rdV8 SZUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=e/lVmjLLeCvEinWyi9JYGM5h7izbLSoXjALespNWLnc=; b=MoPUxKmZyaWNM7d+Ex+1TBLliMExKWjsm7wW6J3t+pfFLAQqW0uVnFgigf/5wlmhJr SwrjyBm5fGhi7ARLIxEtpk7oFZ9OQdq/NjxcJooWuUqS0yRl3B+VGsRVUrgMsvTCj9HI /VeHDIWELnCRRCbjOOrjci+Ny3t6Um/9XaIbEa3N2sJwL5GqM0cBirjpsP9CwpeE5sQt X6VTCe3146QCcAZr8ztZSpmFhQPtJy1OYIhyh7PpAGHbFsmmR81cvPRBY/GnUXLkcwtO uJZT+faD5YbKJ+kb9q1++ryjzg4u9sS/AJyEbhC6+ChxEFJpyLrir93JrtIQIG3FcZcL mD9w== X-Gm-Message-State: AOAM532vN/mqGGxvzTVYwtCrjjOZchoESte4I87cp1taI3S8qBmLPm9j 9Np6/Gi0T4D/hxSdJPL6EWWS5N1rHCJyxQ== X-Google-Smtp-Source: ABdhPJyJCaNLj5arx6l3U0senJJ+xdhSGZIqPLthIwIkOk9X+Yz09mPGhOKfLX4X/XwEFAvYp01tBQ== X-Received: by 2002:ac8:6651:: with SMTP id j17mr6216446qtp.176.1608328812088; Fri, 18 Dec 2020 14:00:12 -0800 (PST) Received: from hurd (dsl-10-146-235.b2b2c.ca. [72.10.146.235]) by smtp.gmail.com with ESMTPSA id f5sm428387qto.67.2020.12.18.14.00.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Dec 2020 14:00:11 -0800 (PST) From: Maxim Cournoyer Date: Fri, 18 Dec 2020 17:00:10 -0500 Message-ID: <87y2huepw5.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::836; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x836.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -0.22 Authentication-Results: aspmx1.migadu.com; dkim=fail (headers rsa verify failed) header.d=gmail.com header.s=20161025 header.b=fZYroZQ8; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: 5364594036A X-Spam-Score: -0.22 X-Migadu-Scanner: scn0.migadu.com X-TUID: 0QZ0ryoGnCMh --=-=-= Content-Type: text/plain Hello Guix! tl;dr: The Emacs build system and site-start.el loader are modified so that Emacs packages are installed in their own distinct installation directory. The Emacs packages built with the Emacs built system used to be installed in a sub-directory under the share/emacs/guix.d/ directory, but this was changed in commit 65a7dd2950ca13a8b942b2836260a2192351b271 shortly after having accommodated the site-start.el machinery to enable loading packages from any profile (via the EMACSLOADPATH search path specification). While this change allowed to expose simply and directly the packages found in EMACSLOADPATH, it does introduce the risk of file name collisions when multiple Emacs packages are joined in the same profile, especially with Emacs packages increasing in complexity (e.g., using more than a single .el file!) and expecting to have both their sources and resources extracted under their own nested directory rather than as a flat collection (ELPA, MELPA). One recent example I stumbled on was attempting to use the emacs-yasnippet-snippets package along with emacs-elpy; both wanted to install a 'snippets' directory to share/emacs/site-lisp/snippets, collided and resulted in problems that prove difficult to understand. This is what motivated this patch series, where the site-start.el auxiliary code used for package discovery is extended to support packages installed in their own directory under a 'share/emacs/guix' installation prefix, via Emacs' own package library! The emacs-build-system is updated for this new installation prefix, as well as existing packages and documentation. Parting with a directly usable EMACSLOADPATH means that site-start.el *must* run for packages to appear in the load-path; that means for running a test suite, the -Q or --quick Emacs options cannot be used, since it implies --no-site-file. Benefits of using this approach: + Avoid inter-package file name collisions. + Better integration with user installed packages via M-x package-install. The Guix-installed packages are listed in M-x package-list as 'external'. Cons include: - Slightly more complex loader (although much of it is offloaded to package.el), thus slightly slower (see the comparison below). - Requires to ensure every package's test suite doesn't make use of -Q. In my opinion the benefits outweighs the cons by a comfortable margin, especially with the boring work of adapting the package collection already done. To test the performance of the new approach, the following manifest file was used to test the rebuild of the ~900 Emacs packages making use of the Emacs build system: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=emacs-packages-manifest.scm (use-modules (gnu packages) (guix build-system) (guix packages) (srfi srfi-1)) (define %broken-emacs-packages (map specification->package '("emacs-picpocket" ;tests fail "emacs-twittering-mode" ;build fails ;; Broken only on current master, without new changes. "emacs-md4rd" "emacs-el-patch" "emacs-flymake-shellcheck" ))) (define %emacs-packages (fold-packages (lambda (package lst) (if (eq? (build-system-name (package-build-system package)) 'emacs) (cons package lst) lst)) '())) (packages->manifest (lset-difference eqv? %emacs-packages %broken-emacs-packages)) --=-=-= Content-Type: text/plain A simple benchmark testing the performance of the activation of the hundreds of Emacs packages was then run using: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix environment --pure -m emacs-packages-manifest.scm \ --ad-hoc emacs [env]$ /run/setuid-programs/sudo /bin/sh -c 'echo 3 > /proc/sys/vm/drop_caches' [env]$ emacs --batch --no-site-file \ --eval="(progn (require 'guix-emacs) \ (require 'benchmark) \ (message \"(total gc-count gc-time) = %s\" \ (benchmark-run 1 (guix-emacs-autoload-packages))))" --8<---------------cut here---------------end--------------->8--- On the master branch: --8<---------------cut here---------------start------------->8--- [...] Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zotxt-autoloads... Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zoutline-autoloads... Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/ztree-autoloads... (total gc-count gc-time) = (25.242400751 13 0.189669369) --8<---------------cut here---------------end--------------->8--- Or about 0.65 s on a warm cache. On a branch with these changes: --8<---------------cut here---------------start------------->8--- Error loading autoloads: (file-missing Cannot open load file No such file or directory kotl/kotl-autoloads) Error loading autoloads: (file-missing Cannot open load file No such file or directory helm-easymenu) Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/flycheck-cpplint-0.1-1.1d8a090/flycheck-cpplint-autoloads) Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/evil-anzu-0.03/evil-anzu-autoloads) Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/erc-image-0-3.82fb387/erc-image-autoloads) ad-handle-definition: `ido-completing-read' got redefined Error loading autoloads: (file-missing Cannot open load file No such file or directory tex-site) (total gc-count gc-time) = (26.175704339 47 0.783184412) --8<---------------cut here---------------end--------------->8--- Or about 3 seconds on a warm cache. There a 6 errors that would need to be looked into, but I these look like actual packaging problems rather than new issues. The previously used way to load the autoloads, '(load f 'noerror)' would have masked them. Thanks, Maxim --=-=-=--