[PATCH] epm.el: A CLI tool for package.el

[Max Nikulin <manikulin@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4229 bytes --]

On 30/04/2023 17:39, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>>       ./epm.el -Q --epm-dir $(emacs_pkgdir)/emacs-%e install compat
> 
> Good idea. Although, we should not overdo this package management thing.
> If we really need complex functionality here, we should better just use
> cask/eldev instead of re-inventing the wheel.

I have not tried cask or eldev, so I can not reason on supposed workflow.

> Are you willing to improve the draft to be ready for upstream?

See the attachment. Interface is subject to change to better fit 
particular use cases.

> I think that we need to zoom out a bit and discuss the contexts where
> Org build system is used:
> 
> 1. During Org development, by developers who know what they are doing

...and who prefer failure to files installed to unexpected directory 
when configuration contains a mistake or it is not activated.

> 2. By ordinary users, not necessarily familiar with GNU make and all the
>     associated build process conventions.

For those who are not familiar with convention any target may be 
specified in docs, keeping usual meaning of the default "all" target.

> For example, my `package-user-dir' contains a
> number of forks with additional patches applied - it is not the
> environment I want to develop (and test) Org in.

I think, for ordinary user it is better to keep build time and runtime 
`package-user-dir' the same. Developers, especially when multiple 
versions are involved, should have isolated sandboxes similar to 
"python3 -m venv" and activating scripts setting proper environment 
variables.

> Org users will likely use make autoloads, make, make docs, and make
> repro.

make autoloads should be necessary only to run org uncompiled. My 
impression is that some bugs may exist, so make clean and make autoloads 
are necessary during updates.

> However, make repro and
> optionally make docs should avoid re-using user packages as it may cause
> inconsistent results if the `package-user-dir' is messed up.

I agree concerning "make repro", but unsure for docs.

> One way to handle the above scenarios might be your idea with AUTODEP.
> By default, it will be "auto":
>   - make compile, docs  :: Re-use default `package-user-dir'
>   - make repro    :: Auto-download and ignore `package-user-dir'
>   - other targets :: Prompt the user
> Alternatives will be meant to be used as
> 
>      AUTODEP=download/user/no make target.
> 
> triggering unconditional downloading, using `package-user-dir', and not
> using any guess, correspondingly.

In general I agree that strategy may depend on specified target. The 
only issue that make allows to specify several targets. An I am unsure 
concerning user prompt.

I have realized that as soon as build dependencies are involved, 
Makefile should use emacs -q, not emacs -Q since -Q excludes site-lisp 
directories created by e.g. elpa-compat debian package.

> I think that it is stretching a bit beyond the complexity we should
> allow within Org build system. In your scenario, I can simply do
> make cleanpkg and re-download the latest dependencies.

I would prefer clear error message that package version is not 
satisfactory. However such feature may be added later.

>> #!/bin/sh
>> ":"; # -*- mode: emacs-lisp; lexical-binding: t; -*-
>> ":"; exec emacs --script "$0" "$@"
> 
> Let's not lock to bash. AFAIK, our makefiles can currently work on
> Windows. Using /bin/sh will lead to regression.

It is POSIX shell, not BASH. I am unsure if make can be used on windows 
when e.g. cygwin is not available. Makefiles are full of POSIX tool 
invocations.

Anyway, since emacs binary is customizable in Makefile, the correct way 
to use this script from Makefile is like (perhaps with more flags)

$(EMACS_Q) --script mk/epm.el --epm-dir $(EPMDIR) missing compat

A convenience way

mk/epm.el -q report compat htmlize

is for default emacs from PATH when a user works with shell prompt.


>> 	   (dir (directory-file-name (expand-file-name fmt-expanded))))
>>        ;; `package-user-dir' ~/.emacs.d/elpa by default
>>        ;; `package-directory-list' does not include it
> 
> What does this comment refer to?

To lack of my experience with package.el and site-lisp infrastructure.

[-- Attachment #2: 0001-epm.el-A-CLI-tool-for-package.el.patch --]
[-- Type: text/x-patch, Size: 9177 bytes --]

From 6e0d73abf527901df080f0f5d7d272722d89c87a Mon Sep 17 00:00:00 2001
From: Max Nikulin <manikulin@gmail.com>
Date: Wed, 3 May 2023 18:39:49 +0700
Subject: [PATCH] epm.el: A CLI tool for package.el

* mk/epm.el: A helper to install build time dependencies from ELPA.
---
 mk/epm.el | 243 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 243 insertions(+)
 create mode 100755 mk/epm.el

diff --git a/mk/epm.el b/mk/epm.el
new file mode 100755
index 000000000..2816702bb
--- /dev/null
+++ b/mk/epm.el
@@ -0,0 +1,243 @@
+#!/bin/sh
+":"; # -*- mode: emacs-lisp; lexical-binding: t; -*-
+":"; exec emacs --script "$0" "$@"
+;;; epm.el --- Emacs package management helper for Org Mode
+
+;; Copyright (C) 2023 Free Software Foundation, Inc.
+
+;; Author: Max Nikulin <manikulin@gmail.com>
+;; Created: 2 May 2023
+;; Keywords: maint, tools
+;; Package-Requires: ((emacs "26.1"))
+;; URL: https://orgmode.org
+;; Version: 0.1
+
+;; This file is not part of GNU Emacs.
+;;
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+;; epm.el is an attempt to create a tool that allows simple package
+;; management outside of interactive Emacs session.  Its purpose
+;; is dependency management for make based workflow for building
+;; and testing and for continuous integration (CI) systems.
+;; It install not available yet packages from ELPA.
+;;
+;; Before compiling or running Org mode uncompiled it is necessary
+;; to install dependencies.  If libraries are already available
+;; in your `load-path' then the following commands should be no-op.
+;;
+;; As a user who prefers to use Org mode version from git
+;; likely you would prefer default ~/emacs.d/elpa directory
+;; for installed packages
+;;
+;;     mk/epm.el -q install compat
+;;
+;; If you are a developer and you need to separate environment from
+;; main Emacs configuration, you may choose some alternative
+;; package directory and use %e substitution for Emacs version.
+;;
+;;     export EPMDIR="$HOME/.cache/epm/emacs-%e"
+;;     /path/to/emacs-src/emacs -q --script mk/epm.el install compat
+;;
+;; or
+;;
+;;     /path/to/emacs-src/emacs -q --script mk/epm.el \
+;;         --epm-dir "$HOME/.cache/epm/emacs-%e" install compat
+;;
+;; Of course, you should specify location to overriden `package-user-dir'
+;; in local.mk.
+;;
+;; When a bug is suspected, it is better to install dependencies
+;; e.g. to TMPDIR to avoid issues with content of user init directory.
+;;
+;; If you prefer to avoid ELPA packages and this script, you still have
+;; --directory/-L option and EMACSLOADPATH environment variable
+;; to specify where required libraries may be loaded.
+;;
+;; Since required dependency may be installed e.g. as elpa-compat Debian
+;; package, it is not recommended to use --quick/-Q or --no-site-lisp/-nsl
+;; options, prefer --no-user-init/-q instead unless you suspect some
+;; issue with site-lisp directories.
+;;
+;; Limitations:
+;; - package.el allows to check if minimum version requirement is satisfied
+;;   for a package, but I have not found API to check it for a library from
+;;   `load-path'.
+;; - Upgrading of a package is not implemented and I am unsure if convenient
+;;   API exists.
+;; - Ideally istead of library list it should be possible to specify .el file
+;;   and dependencies should be taken from the Package-Requires header.
+;;
+;; To get list of libraries that are not available run
+;;
+;;     mk/epm.el -q missing compat
+;;
+;; Non-zero exit code means missing dependencies, its list is printed to stdout.
+;; To check which file will be loaded try
+;;
+;;     mk/epm.el -q report htmlize compat
+;;
+;; Overview of available commands are provided by
+;;
+;;     mk/epm.el -q help
+
+;;; Code:
+
+(require 'format-spec)
+(require 'package)
+(require 'subr-x)
+
+(defvar epm-dir nil
+  "Overrides `package-user-dir' and EPMDIR environment.")
+
+(defun epm--get-script-name ()
+  "Guess command line argument spefifying this script.
+
+Real argument is not available:
+`argi' is \"-scriptload\", `argval' is local variable of `command-line-1',
+`load-file-name' is absolute path, `file-relative-name' is too aggressive
+and adds \"..\" to root."
+  (let ((relative (file-relative-name load-file-name
+                                      command-line-default-directory)))
+    (if (string-suffix-p load-file-name relative)
+        load-file-name
+      relative)))
+
+(defvar epm-script-name (epm--get-script-name)
+  "Name of epm.el as it appears in Emacs command line options")
+
+(defun epm-nonempty-p (s)
+  (and s (not (string-empty-p s))))
+
+(defun epm-init ()
+  (unless (epm-nonempty-p epm-dir)
+    (setq epm-dir (getenv "EPMDIR")))
+  (when (epm-nonempty-p epm-dir)
+    (let* ((fmt-expanded (format-spec epm-dir `((?e . ,emacs-version))))
+	   (dir (directory-file-name (expand-file-name fmt-expanded command-line-default-directory))))
+      ;; `package-user-dir' ~/.emacs.d/elpa by default even with -Q
+      ;; `package-directory-list' does not include `package-user-dir'.
+      (setq package-user-dir dir)))
+  ;; TODO (load site-run-file 'no-error 'no-message)
+  ;; may be necessary to load elpa-* deb packages when -Q option
+  ;; is used. See Info node "(elisp) Init File".
+  (package-initialize))
+
+(defun epm-library-unavailable-p (lib)
+  (unless (locate-library lib)
+    lib))
+
+(defun epm-missing (libs)
+  ;; TODO consider `require' catching load errors
+  (delq nil (mapcar #'epm-library-unavailable-p libs)))
+
+(defun epm-cmd-help (_cmd _args)
+  "List commands."
+  (princ (format "\
+Usage: %s [--dbg|--debug-on-error] [--epm-dir DIR] COMMAND ARGS...
+   or: %s --script %s [--dbg|--debug-on-error] [--epm-dir DIR] COMMAND ARGS...
+
+A CLI tool to install ELPA packages.
+
+Any Emacs option may be specified, e.g. --quck,-Q, --no-init-file,-q,
+or --directory,-L DIR
+
+--dbg, --debug-on-error
+    Enable `debug-on-error'
+
+--epm-dir DIR
+    Set `package-user-dir'.
+    \"%%e\" is replaced by `emacs-version'.
+    Alternatively EPMDIR environment variable may be specified.
+\n"
+                 epm-script-name (car command-line-args) epm-script-name))
+  (pcase-dolist (`(,name . ,func) epm-commands)
+    (princ name)
+    (terpri)
+    (princ
+     (replace-regexp-in-string
+      "\\`\\|\n" "\\1    "
+      (documentation func) 'fixedcase nil))
+    (terpri)
+    (terpri)))
+
+(defun epm-cmd-missing (_ libs)
+  "Report not installed libraries and exit with non-zero code."
+  (let ((missing (epm-missing libs)))
+    (when missing
+      (princ (mapconcat #'identity missing " "))
+      (terpri)
+      (kill-emacs 1))))
+
+(defun epm-cmd-install (_ libs)
+  "Install packages from LIBS that are not available yet"
+  ;; TODO force option or update command
+  (let ((missing (epm-missing libs)))
+    (when missing
+      (package-refresh-contents)
+      (make-directory package-user-dir 'parents))
+    (dolist (pkg missing)
+      (package-install (intern pkg)))))
+
+(defun epm-cmd-report (_ libs)
+  "Report paths of available libraries"
+  (princ (format "package-user-dir: %s\n" package-user-dir))
+  ;; (princ (format "load-path: %s\n" load-path))
+  (dolist (name libs)
+    ;; (version-to-list version)
+    (princ (format "%-20s %s " name
+		   (if (package-installed-p (intern name))
+		       "package "
+		     "        ")))
+    (princ (locate-library name))
+    (terpri)))
+
+(defvar epm-commands
+  '(("help" . epm-cmd-help)
+    ("install" . epm-cmd-install)
+    ("missing" . epm-cmd-missing)
+    ("report" . epm-cmd-report)))
+
+(defun epm-command-line-function ()
+  "Handle command line options and arguments specific to epm.
+
+Implements a handler for `command-line-functions'."
+  ;; There is no easy to determine if "--" argument has been processed earlier.
+  ;; TODO "--option=value" is handled by `command-line-1' only for standard arguments.
+  (pcase argi
+    ("--epm-dir"
+     (setq epm-dir (pop command-line-args-left))
+     t)
+    ((or "--dbg" "--debug-on-error")
+     ;; -d is handled as --display, --debug as --debug-init
+     (setq debug-on-error t)
+     t)
+    ((pred (string-match-p "\\`[^-]"))
+     (epm-init)
+     (let ((func (cdr (assoc argi epm-commands)))
+           (cmd-args command-line-args-left))
+       (if (not func)
+           (error "Unknown command %s" argi)
+         (setq command-line-args-left nil)
+         (funcall func argi cmd-args)))
+     t)))
+
+(push #'epm-command-line-function command-line-functions)
+
+;; Local Variables:
+;; no-byte-compile: t
+;; End:
+;;; epm.el ends here
-- 
2.25.1