`make' written in elisp

`make' written in elisp

[Michael Schierl]

Hi,

I'm not sure if I am right here ("Emacs development discussions."
Discussions about developing Emacs, developing for Emacs or developing with
Emacs...?) so if i'm wrong I'd like if you pointed me into the right
direction...

For most users of Emacs who are not on *ix, it is harder to install new
packages than for those on *ix since most (GNU) Emacs "extensions" are
shipped with makefiles or autoconf scripts. This works flawlessly on Unix
but not on other systems - especially because most of the simpler scripts
make assumptions where your emacs binary or your site-lisp directory is.
You can adjust this for every single package, but what for? most makefiles
don't do more than runninge batch-byte-compile for the .el files, makeinfo
for the .texi files and install-info for the resulting info files (plus
copying the files where they belong).

So I tried to write a framework completely in Elisp (except makeinfo at the
moment, because `texinfo-format-buffer' seems to be just
crashy/buggy/whatever when applying it to "in-the-wild" texinfo files)
called elMake¹.

Its syntax is similar to that of apache ant, except it is in sexps instead
of xml tags (Users writing Elisp should be familiar with that syntax).

It seems to work quite well so far (example "elMakefiles" for jde, auctex
and of course elmake itself seem to work as desired).

You can have a look at it at http://elmake.sourceforge.net/ (the website is
quite "minimalistic", but what do you need more for a 42KB download?).


So, and why do i write here? I'd like to know

- what you think of that idea
- if you could think to use it for your own emacs addons (if you maintain 
  any)
- what is bad at the particular implementation
- if you find any bugs
- what features are missing
- how to install such a thing in your opinion (atm it is a .el file to be 
  loaded and pressed C-x C-e at the correct position)
- and everything else you can think of (if related to elMake).

For answers you might use the elmake-devl mailing list at sourceforge, but
of course I'll read answers on this list as well.

TIA,

Michael

¹ hit 2 on Google for "elmake" is the (empty) sf.net project site atm...

Re: `make' written in elisp

[Richard Stallman]

Have you designed this so that the same packages can use both this and
their current configure files?  Use of a configure file and `make
install' is the GNU standard, and we don't want any packages that now
support the standard to stop supporting it.

Re: `make' written in elisp

[Michael Schierl]

Richard Stallman schrieb:
> 
> Have you designed this so that the same packages can use both this and
> their current configure files?  

Yes. There is no change needed for the packages i tested until now
(except placing an additional file called elMakefile into the install
directory). And if a package would require to change its directory
structure/whatever, that is clearly a fault of elMake then and needs to
be fixed.

This will of course mean for developers to maintain two different
install methods.

> Use of a configure file and `make
> install' is the GNU standard, and we don't want any packages that now
> support the standard to stop supporting it.

Sure. I don't want that either.

Michael

Re: `make' written in elisp

[Richard Stallman]

    Yes. There is no change needed for the packages i tested until now
    (except placing an additional file called elMakefile into the install
    directory). And if a package would require to change its directory
    structure/whatever, that is clearly a fault of elMake then and needs to
    be fixed.

Then it might be a good idea.  Can you show me an example of an
elMakefile?

Re: `make' written in elisp

[Michael Schierl]

Richard Stallman schrieb:
> 
>     Yes. There is no change needed for the packages i tested until now
>     (except placing an additional file called elMakefile into the install
>     directory). And if a package would require to change its directory
>     structure/whatever, that is clearly a fault of elMake then and needs to
>     be fixed.
> 
> Then it might be a good idea.  Can you show me an example of an
> elMakefile?

Surely. Directly from CVS:

<URL:http://cvs.sourceforge.net/viewcvs.py/*checkout*/elmake/elmake/samples/auctex.elMake_?content-type=text%2Fplain&rev=1.1>
<URL:http://cvs.sourceforge.net/viewcvs.py/*checkout*/elmake/elmake/samples/emptyB.elMake?content-type=text%2Fplain&rev=1.2>
<URL:http://cvs.sourceforge.net/viewcvs.py/*checkout*/elmake/elmake/samples/jde.elMake_?content-type=text%2Fplain&rev=1.1>

A "general" sample:

<URL:http://cvs.sourceforge.net/viewcvs.py/*checkout*/elmake/elmake/samples/sample.elMake?content-type=text%2Fplain&rev=1.3>

And the one used for "bootstrapping" elMake itself:

<URL:http://cvs.sourceforge.net/viewcvs.py/*checkout*/elmake/elmake/elMakefile?content-type=text%2Fplain&rev=1.5>


You might as well download
<URL:http://prdownloads.sourceforge.net/elmake/elmake-0.1.zip> 
They are all in there as well.

Michael

Re: `make' written in elisp

[Stefan Monnier]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

> For most users of Emacs who are not on *ix, it is harder to install new
> packages than for those on *ix since most (GNU) Emacs "extensions" are
> shipped with makefiles or autoconf scripts. This works flawlessly on Unix
> but not on other systems - especially because most of the simpler scripts
> make assumptions where your emacs binary or your site-lisp directory is.
> You can adjust this for every single package, but what for? most makefiles
> don't do more than runninge batch-byte-compile for the .el files, makeinfo
> for the .texi files and install-info for the resulting info files (plus
> copying the files where they belong).

For the same reason, I wrote the install.el elisp package which can take
a single .el file or a tarball of an elisp package and install it.

It is a completely different solution to your problem, but I think that
your code addresses just the issues that are left in install.el.

It's still very simple (or rather: simplistic) and needs more work, but it
might be interesting to integrate it with your code.

I have attached my latest install.el.


        Stefan

[-- Attachment #2: install.el --]
[-- Type: application/emacs-lisp, Size: 21208 bytes --]

[-- Attachment #3: Type: text/plain, Size: 141 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: `make' written in elisp

[Richard Stallman]

A year after you posted it, I finally read the source code of install.el.
(I was too backlogged to read source code last January.)

I found that it was extremely clean and appealing code.
If a package system for Emacs can mean something like this,
I am in favor of it.

I would suggest some small changes.

* Instead of asking the user every time whether to install for
everyone or just for him, I think it would be better to have different
command names, or use a prefix argument to mean "global".  It is
not feasible to use the commands from programs if they always
ask questions.

* It seems to assume that auto-compression-mode is enabled, that
just writing a file whose name ends in .gz will compress it.

* A number of functions need doc strings.

Here's the version that was posted a year ago; does anyone want
to work on it further?


;;; install.el --- Package to ease installation of Elisp packages

;; Copyright (C) 2001, 2003  Stefan Monnier

;; Author: Stefan Monnier <monnier@cs.yale.edu>

;; This file is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 2, or (at your option)
;; any later version.

;; This file is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to
;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
;; Boston, MA 02111-1307, USA.

;;; Commentary:

;; This little package is meant to ease up the task of installing
;; third party ELisp packages.  I.e. it takes care of placing it in
;; an appropriate location, finds out what code is necessary to get
;; the package activated, sets up your .emacs file to activate the
;; package, and byte-compiles the package.

;; It should work on both single-file packages and tarballs.

;; On tarball packages, it does a bit of guess work to figure out
;; where are which files and how to use them.  This is bound to
;; fail sometimes.

;; Tested on:
;; + ProofGeneral
;; + sml-mode
;; + AUCTeX   (missed the info page)
;; + X-Symbol (as an XEmacs package)
;; + Gnus     (but doesn't install the info doc :-( )
;; + BBDB     (misses the main info page)
;; - WhizzyTeX (needs to hack the perl script and stuff :-( )
;; ? ECB
;; ? JDEE
;; ? preview-latex
;; ? VM
;; ? mmm-mode
;; ? Semantic

;; The on-disk structure is as follows:
;; - there are two area: the `home' and the `site' each with their
;;   respective directory (~/lib/emacs and /usr/share/emacs/site-lisp)
;;   and file (.emacs and site-start).
;; - There is a distinction between installing and activating.
;;   Installing only places the files on disk, whereas activating sets up
;;   autoloads and friends.
;; - Activation is done on a directory by directory basis.  Each directory
;;   has an `autoloads' file.  Loading it activates the package(s)
;;   in directory.
;; - Single-file packages are placed together in the toplevel directory
;;   whereas tarball-packages are placed in their own subdirectory (so they
;;   can be activated independently).

;;; Todo:

;; - don't ask whether to activate site-wide packages installed in home.
;; - Create Info from Texinfo when needed.
;; - Try harder to find Info files such as doc/auctex.
;; - UI to (un)install and (de)activate packages, get a list, ...
;; - If a single-file package lacks ;;;###autoload, try to add them
;;   based on the Commentary section or something.
;; - don't leave out reams of `autoloads~' backup files.

;;; Code:

(require 'em-glob)

(defgroup install nil
  "Elisp package installation")

(defmacro install-filter (list exp)
  (declare (debug t))
  `(let ((res nil))
     (dolist (x ,list (nreverse res))
       (if ,exp (push x res)))))

(defcustom install-site-file
  (or
   (locate-file (or site-run-file "site-start") load-path load-suffixes)
   (let ((lp (mapcar 'abbreviate-file-name load-path)))
     ;; Prefer non-user directories.
     (setq lp (or (install-filter lp (not (string-match "\\`~/" x))) lp))
     ;; Prefer site-lisp directories.
     (setq lp (or (install-filter lp (string-match "/site-lisp\\'" x)) lp))
     ;; Prefer shorter directory names (i.e. parents rather than subdirs).
     (setq lp (sort lp (lambda (d1 d2) (< (length d1) (length d2)))))
     ;; 
     (expand-file-name (concat (or site-run-file "site-start") ".el") (car lp))))
  "Site-wide customization file."
  :type 'file)

(defcustom install-site-dir (file-name-directory install-site-file)
  "Directory where site-wide packages should be installed."
  :type 'directory)

(defcustom install-home-file (or user-init-file
				 (convert-standard-filename "~/.emacs"))
  "Main customization file into which Install should place `load' commands."
  :type 'file)

(defcustom install-home-dir
  ;; FIXME: We should be careful never to choose one of Emacs's own
  ;; directories, even if the user installed Emacs in his home dir.
  (let ((lp (mapcar 'abbreviate-file-name load-path)))
    ;; Only consider writable directories.
    (setq lp (install-filter lp (file-writable-p x)))
    ;; Only consider user directories.
    (setq lp (install-filter lp (string-match "\\`~/" x)))
    ;; Prefer shorter directory names (i.e. parents rather than subdirs).
    (setq lp (sort lp (lambda (d1 d2) (< (length d1) (length d2)))))
    ;; Default to ~/lib/emacs.
    (if (or (null lp)
	    ;; If it's a subdir of lib/emacs, use lib/emacs.  This can happen
	    ;; because Install does not automatically add lib/emacs to the
	    ;; load-path if it only installs tar packages underneath.
	    (string-match "\\`~/lib/emacs/" (car lp)))
	"~/lib/emacs/"
      (car lp)))
  "Directory into which elisp packages should be placed by Install."
  :type 'directory)

(defcustom install-autoload-file "autoloads"
  "Name of autoload files used by Install.")

(defcustom install-compress-source-files nil ;; ".gz"
  "If non-nil, Install will try to compress file.")

(defcustom install-byte-compile t
  "If non-nil, elisp files are byte-compiled during installation."
  :type 'boolean)

;;

(defun install-get-dir ()
  "Return the directory into which to install packages."
  (or (and (file-writable-p install-site-dir)
	   (y-or-n-p "Install site-wide? ")
	   install-site-dir)
      (progn
	(unless (file-writable-p install-home-dir)
	  (setq install-home-dir
		(let ((default-directory install-home-dir))
		  (read-directory-name "Directory to install into: ")))
	  (unless (file-directory-p install-home-dir)
	    (make-directory install-home-dir t)))
	install-home-dir)))

(defun install-get-file ()
  "Return the file into which to activate packages."
  (or (and (file-writable-p install-site-file)
	   (y-or-n-p "Activate site-wide? ")
	   install-site-file)
      install-home-file))

(defmacro install-with-file (file &rest body)
  (declare (debug t) (indent 1))
  `(let ((install-with-existing-file (find-buffer-visiting ,file)))
     (with-current-buffer (or install-with-existing-file
			      (find-file-noselect ,file))
       (prog1 (save-current-buffer ,@body)
	 (unless install-with-existing-file
	   (kill-buffer (current-buffer)))))))

;;;###autoload
(defun install-file (file)
  (interactive "fFile to install: ")
  (with-current-buffer (find-file-noselect file)
    (install-buffer)))


;;;###autoload
(defun install-buffer ()
  "Install the current elisp buffer as a package.
The package is install in `install-home-dir', autoloads are added
to the `install-autoload-file' in that directory and the
`install-custom-file' is then updated to load these autoloads."
  (interactive)
  (cond
   ((derived-mode-p 'tar-mode) (install-tar-buffer))
   ((not (derived-mode-p 'emacs-lisp-mode))
    (error "I only know how to install tar.gz and elisp files."))
   (t
    (let* ((install-dir (install-get-dir))
	   (package (file-name-nondirectory buffer-file-name))
	   (file (expand-file-name package install-dir))
	   (autoload (expand-file-name install-autoload-file install-dir)))
      (when (and install-compress-source-files
		 (string-match "\\.el\\'" file))
	(setq file (concat file (if (stringp install-compress-source-files)
				    install-compress-source-files ".gz"))))
      ;; Install the elisp file.
      (write-region (point-min) (point-max) file)
      ;; Extract the autoloads into a separate file.
      (install-update-autoloads autoload)
      ;; Activate.
      (install-activate autoload)
      ;; Finally, byte compile.  In the present case (a single-file package),
      ;; this could be done before activation.
      (if install-byte-compile
	  (byte-compile-file file))))))

(defun install-tar-buffer ()
  "Like `install-buffer' but for a tar package rather than single file."
  (let* ((name (file-name-nondirectory buffer-file-name))
	 ;; Strip off ".tar.gz", ".tar", ".tgz", ".tar.Z", ...
	 (name (if (string-match "\\.[tT][^.]+\\(\\.[^.]+\\)?\\'" name)
		   (substring name 0 (match-beginning 0)) name))
	 (install-dir (install-get-dir))
	 (default-directory (expand-file-name name install-dir)))
    ;; Install the files.
    ;; FIXME: check what `tar-untar-buffer' does with symlinks and stuff.
    ;; FIXME: the dir might already exist.
    (make-directory default-directory)
    (tar-untar-buffer)
    (let ((files (directory-files default-directory
				  nil "\\`\\([^.]\\|\\.[^.]\\|\\.\\..\\)" t)))
      ;; If the tar file already had everything under a single directory,
      ;; remove the redundant level of directory.
      (when (and (= (length files) 1) (file-directory-p (car files)))
	(let* ((f (car files))
	       ;; Keep the longest name of the two, assuming that the
	       ;; difference is that the longer one has a version number.
	       (final (if (> (length name) (length f)) name f))
	       (temp (if (= (length name) (length f)) (concat f ".tmp") f)))
	  ;; FIXME: the dir might already exist.
	  (rename-file f (expand-file-name temp install-dir))
	  (setq default-directory install-dir)
	  (delete-directory name)
	  ;; FIXME: the dir might already exist.
	  (unless (equal final temp) (rename-file temp final))
	  (setq name final)
	  (setq default-directory (expand-file-name final)))))
    (install-directory-inplace)))

;;;###autoload
(defun install-directory-inplace ()
  "Prepare and activate the current directory for use by Emacs.
Sets up the autoload files, activates them and byte-compiles if needed."
  (interactive)
  ;; Extract the autoloads.
  (install-setup-tree)
  ;; Activate the package.
  (install-activate (expand-file-name (install-get-activation-file)))
  ;; Finally, byte-compile the files.
  (if install-byte-compile
      (install-byte-compile-dir)))

(defun install-dirs-of-files (files)
  "Return a list of subdirs containing elisp files."
  (let ((dirs nil)
	(ignore (regexp-opt
		 (cons
		  ;; Ignore contrib directories because they tend to contain
		  ;; either less-debugged code, or packages that might
		  ;; already be installed and can thus interfere.
		  "contrib/"
		  (let ((exts nil))
		    (dolist (ext completion-ignored-extensions exts)
		      (if (eq (aref ext (1- (length ext))) ?/)
			  (push ext exts))))))))
    ;; Collect the dirs that hold elisp files.
    (dolist (file files dirs)
      (let ((dir (file-name-directory file)))
	(unless (or (member dir dirs)
		    (and dir (string-match ignore dir)))
	  (push dir dirs))))))

(defun install-find-elisp-dirs ()
  "Return a list of subdirs containing elisp files."
  (install-dirs-of-files (install-glob "**/*.el")))

(defun install-byte-compile-dir ()
  "Byte compile all elisp files under the current directory."
  (let ((load-path (append (mapcar (lambda (dir)
				     (if dir
					 (expand-file-name dir)
				       default-directory))
				   (install-find-elisp-dirs))
			   load-path)))
    (byte-recompile-directory default-directory 0)))

(defun install-glob (pattern)
  (let ((res (eshell-extended-glob pattern)))
    (if (listp res) res)))

(defun install-get-activation-file ()
  "Return the file to load to activate the package.
This is usually \"./autoloads\", but it can also be \"lisp/foo-site.el\"."
  (if (file-exists-p install-autoload-file)
      install-autoload-file
    (or (car (install-glob (concat "**/" install-autoload-file)))
	(car (install-glob "**/auto-autoloads.el"))
	(car (install-glob "**/*-site.el")))))

(defun install-setup-tree ()
  (eshell-glob-initialize)
  ;; Look for elisp files.
  (let ((dirs (install-find-elisp-dirs))
	(autoload-files nil)
	(toplevel nil))
    ;; Prepare each elisp subdir and collect info along the way.
    (dolist (dir dirs)
      (let ((default-directory (expand-file-name (or dir default-directory))))
	;; Remove *.elc files, in case they were not compiled for our version.
	(mapc 'delete-file (install-glob "*.elc"))
	;; Extract autoloads.
	(let ((sites (or (install-glob "auto-autoloads.el")
			 (install-glob "*-site.el"))))
	  (if (= 1 (length sites))
	      ;; Some packages come with a <pkg>-site.el file instead
	      ;; of using autoloads.  In that case, just load that file.
	      (push (concat dir (car sites)) autoload-files)
	    ;; Otherwise.  Make an autoloads file and load it.
	    ;; FIXME: Don't make hundreds of autoload files.
	    (let ((exists (file-exists-p install-autoload-file)))
	      (if (not (install-update-autoloads install-autoload-file))
		  ;; Don't stupidly add empty autoloads files.
		  (unless exists (delete-file install-autoload-file))
		(push (concat dir install-autoload-file) autoload-files)))))))
    (mapc 'install-ensure-autoloads-file autoload-files)
    ;; Setup the toplevel activation file.
    (if (and (= 1 (length autoload-files))
	     (equal (car autoload-files) (install-get-activation-file)))
	(setq toplevel (car autoload-files))
      (setq toplevel install-autoload-file)
      (dolist (file autoload-files)
	(unless (equal file toplevel)
	  (install-activate
	   `(expand-file-name
	     ,(file-relative-name
	       (expand-file-name file)
	       (file-name-directory (expand-file-name toplevel)))
	     (file-name-directory load-file-name))
	   toplevel))))
    ;; Make up an info/dir file if necessary and register the info dirs.
    (let ((info-dirs (install-make-info)))
      (when info-dirs
	(install-with-file toplevel
	  (unless (derived-mode-p 'emacs-lisp-mode) (emacs-lisp-mode))
	  (goto-char (point-min))
	  (unless (re-search-forward "(add-to-list[ \t\n]+'Info-default-directory-list" nil t)
	    (forward-comment (point-max))
	    (while (re-search-backward "^\f" nil t))
	    (unless (bolp) (newline))
	    (let ((top-dir (file-name-directory (expand-file-name toplevel))))
	      (dolist (dir info-dirs)
		(setq dir (expand-file-name (or dir default-directory)))
		(if (equal dir top-dir)
		    (insert "(add-to-list 'Info-default-directory-list (file-name-directory load-file-name))\n")
		  (let ((text (pp-to-string (file-relative-name dir top-dir))))
		    (if (string-match "\n\\'" text)
			(setq text (substring text 0 -1)))
		    (insert "(add-to-list 'Info-default-directory-list\n"
			    "             (expand-file-name " text
			    " (file-name-directory load-file-name)))\n")))))
	    (save-buffer)))))))

(defun install-ensure-autoloads-file (file)
  "Make sure that the autoload file FILE exists and if not create it."
  (install-with-file file
    (unless (derived-mode-p 'emacs-lisp-mode) (emacs-lisp-mode))
    (when buffer-read-only
      (set-file-modes buffer-file-name
		      (logior ?\200 (file-modes buffer-file-name)))
      (toggle-read-only))
    (goto-char (point-min))
    ;; Insert a little boiler plate if there's nothing yet.
    (when (eobp)
      (insert ";;; " (file-name-nondirectory file)
	      " --- automatically extracted autoloads\n"
	      ";;\n"
	      ";;; Code:\n\n"
	      "\f\n;; Local Variables:\n"
	      ";; version-control: never\n"
	      ";; no-byte-compile: t\n"
	      ";; no-update-autoloads: t\n"
	      ";; End:\n"
	      ";;; " (file-name-nondirectory file)
	      " ends here\n")
     (goto-char (point-min)))
    ;; Make sure it will setup the load path properly.
    (unless (re-search-forward "\\<load-file-name\\>" nil t)
      (forward-comment (point-max))
      (while (re-search-backward "^\f" nil t))
      (unless (bolp) (newline))
      (unless (eq (char-before (1- (point))) ?\n) (newline))
      (insert ";; Tell Emacs to look for elisp files in this directory."
	      ;; Add some sort of signature.
	      "  -- Install\n")
      (insert "(add-to-list 'load-path
              (or (file-name-directory load-file-name) (car load-path)))\n\n")
      (save-buffer)))
  file)

(defvar generated-autoload-file)

(defun install-update-autoloads (autoload)
  "Update file AUTOLOAD.  This will create the file if necessary.
Returns non-nil if there is anything autoloaded into it."
  (setq autoload (expand-file-name autoload))
  (let ((bufp (find-buffer-visiting autoload)))
    (let ((generated-autoload-file (install-ensure-autoloads-file autoload)))
      ;; (update-file-autoloads file)
      (update-directory-autoloads (file-name-directory autoload)))
    ;; Make sure the file sets up the load-path appropriately.
    (with-current-buffer (find-file-noselect autoload)
      (unless (derived-mode-p 'emacs-lisp-mode) (emacs-lisp-mode))
      (goto-char (point-min))
      (re-search-forward "^\f" nil t) ;Find the first autoload entry.
      (forward-comment (point-max))
      (prog1 (not (eobp))
	(unless bufp (kill-buffer (current-buffer)))))))

(defun install-activate (autoload &optional into)
  "Update INTO to make sure it loads AUTOLOAD.
AUTOLOAD can be an expression.
If it is a string, this also loads it into the currently running Emacs.
If provided, INTO specifies the file which should load AUTOLOAD.
The default is to use `install-get-file'."
  (when (stringp autoload)
    (setq autoload (abbreviate-file-name autoload))
    (load autoload))
  (install-with-file (or into (install-get-file))
    (unless (derived-mode-p 'emacs-lisp-mode) (emacs-lisp-mode))
    (save-excursion
      (let ((text (pp-to-string autoload)))
	(if (string-match "\n\\'" text)
	    (setq text (substring text 0 -1)))
	(goto-char (point-min))
	(unless (re-search-forward (regexp-quote text) nil t)
	  (goto-char (point-min))
	  (forward-comment (point-max))
	  (while (re-search-backward "^\f" nil t))
	  (unless (bolp) (newline))
	  ;; Pass `install' as argument to load: this both makes Emacs
	  ;; ignore the load if the file is missing and is used as a marker
	  ;; indicating that this load statement was introduced by us.
	  (insert "(load " text " 'install)\n")
	  (save-buffer))))))

;;;###autoload
(defun install-list-packages ()
  "Show the installed packages."
  (interactive)
  (dired (install-get-dir)))

;; Info files and DIR files.
;; Some of this should probably be moved to info.el.

(defconst install-info-dir "-*- Text -*-\n\x1f\n\
File: dir	Node: Top	This is the top of the INFO tree\
\n\n* Menu:\n\n"
  "Text content of a barebones empty `info/dir' file.")

(defun install-find-info-files ()
  (let ((files (or (install-glob "**/*.info*")
		   (install-glob "**/info/*")))
	(tmp nil))
    (dolist (f files)
      (unless (or (member f '("dir" "localdir"))
		  (and (string-match "-[0-9]+" f)
		       (member (replace-match "" t t f) files))
		  (not (string-match "\\.info\\>\\|\\(\\`\\|/\\)[^.]+\\(\\'\\|\\.\\(gz\\|Z\\)\\)" f)))
	(push f tmp)))
    tmp))

(defun install-make-info ()
  "Make an info/dir file if necessary and return the info directories."
  ;; FIXME: This should create the info files from the Texinfo files
  ;; if necessary !!
  ;; Problems to do that:
  ;; - detect when necessary.  E.g. BBDB comes with an info page for
  ;;   the bbdb-filters stuff, but the main bbdb doc is in texinfo.
  ;; - figure out how to makeinfo the thing.  E.g. AucTeX comes with
  ;;   a whole bunch of Texinfo files and it's really not clear which
  ;;   is the right one.
  ;; - The info file might be there, but not found.  E.e. AucTeX has its
  ;;   page in doc/auctex.
  (let* ((files (install-find-info-files))
	 (dirs (install-dirs-of-files files))
	 (dir-files nil))
    ;; Remove files that were in ignored directories.
    (dolist (file files)
      (unless (member (file-name-directory file) dirs)
	(setq files (delq file files))))
    ;; Check that there's something to do.
    (when files
      (assert dirs)
      (dolist (dir dirs)
	(if (file-exists-p (expand-file-name "dir" dir))
	    (push (expand-file-name "dir" dir) dir-files)))
      (unless dir-files
	;; Pick the dir closest to the toplevel to put the main dir file.
	(setq dirs (sort dirs (lambda (s1 s2) (< (length s1) (length s2)))))
	(install-with-file (expand-file-name "dir" (car dirs))
	  (assert (= (point-min) (point-max)))
	  (insert install-info-dir)
	  (narrow-to-region (point) (point-max))
	  (dolist (file files)
	    (let ((section "Miscellaneous")
		  (entry nil))
	      (install-with-file file
		(goto-char (point-min))
		(if (not (re-search-forward
			  (concat "^START-INFO-DIR-ENTRY\n"
				  "\\([* \t].*\n\\)+"
				  "END-INFO-DIR-ENTRY$") nil t))
		    ;; No entry in the file, let's build a default one.
		    (let ((base (file-name-nondirectory
				 (file-name-sans-extension file))))
		      (setq entry (concat "* " (upcase base)
					  ": (" base ").\n")))
		  (setq entry (match-string 1))
		  (goto-char (point-min))
		  (when (re-search-forward
			 "^INFO-DIR-SECTION[ \t]+\\(.*[^ \t\n]\\)" nil t)
		    (setq section (match-string 1)))))
	      (goto-char (point-min))
	      (unless (search-forward entry nil t)
		(unless (re-search-forward (concat "^" (regexp-quote section) "[ \t]*\n") nil 'move)
		  (unless (bobp) (newline))
		  (insert section) (newline))
		(insert entry))))
	  (save-buffer)
	  (kill-buffer (current-buffer))))
      dirs)))
	      
	      
\f
(provide 'install)
;;; install.el ends here

Re: `make' written in elisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> A year after you posted it, I finally read the source code of install.el.
> (I was too backlogged to read source code last January.)
>
> I found that it was extremely clean and appealing code.  If a
> package system for Emacs can mean something like this, I am in favor
> of it.

Judging from the comments, this installer might more or less get along
at least with some XEmacs packages, as well as some other packages
with a regular enough structure (AUCTeX has in the mean time gotten a
new autoconf-based installer very similar to that of preview-latex, so
the latest versions probably will not work with install.el).

As far as I understand the XEmacs package system (at which I have
taken a cursory glance at best), it consists of several parts:

a) a directory layout.  XEmacs packages have a standard way of placing
Elisp, info, data and other files in a layout under the top package
directory.  In addition there are some standard file names for
initializing of packages and installation.  In short, this part of the
package system is basically just file layout conventions relative to a
package-specific top directory.

We don't have any conventions for packages intended for Emacs as far
as I can see.  It certainly would not be bad to have something like
that: if people make code for Emacs, and if we can agree on some
layout that makes it easier for stuff to work out of the box, it
certainly might be worthwhile to state some conventions.

b) version management.  Versions must basically be floats, higher
versions correspond to later versions, are usually based on some CVS
archive and not at all with actual package versions.  Uh.

c) a package manager that will download and upgrade packages.  It
might do so in user-specific and system-wide packages.

d) central download archives that will provide all packages, updated
by packagers (often different from the actual creators of software) on
central servers.

e) it does not offer AFAIR: dependencies and standard tests (for
executables, directories and other stuff) that autoconf provides.  A
bit of framework for that might be nice for obliterating some
necessities for installation.  Even though installing MSYS is not
particularly hard, it still seems to be a barrier for some people.


I am not convinced about the value of the centralized server approach
to package management from XEmacs: it does not seem to lend itself
overly well to getting reasonably updated packages from third
developers (who often have their own download areas from which they
offer their stuff, and those would not normally be visible to the
package manager that just looks at one server for all packages at one
time IIUC).

But I do think that we should try to offer some way of installing
external Lisp code if one has acquired it as a zip or tar archive in
some manner already.

And in those respects where the directory structure and layout
conventions of an XEmacs package would seem reasonable, there seems
little point in inventing something else.

Stefan, having met those problems when writing install.el already,
would probably be a lot more qualified to comment on my impressions
and on the versatility of the package layout he has been catering for.

> I would suggest some small changes.

[...]

All of those certainly sound reasonable.  In particular, I'd second
having separate commands for installing system-wide and user-local
packages.  IIRC (I might well be mistaken), XEmacs automatically
picked one or the other automatically, based on whether one had
previous private packages, and that was less than great.  In
particular, I tend to install a CVS version of Emacs into its own tree
(from my own user), and this means that I can, as a user, install
packages into this Emacs' system tree.  But that does not imply that I
don't want to install some stuff also in my user-local tree (which
would get shared by the regular Emacs as well).

Having separate commands would appear to be beneficial.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: `make' written in elisp

[Ralf Angeli]

* David Kastrup (2005-01-03) writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> I found that it was extremely clean and appealing code.  If a
>> package system for Emacs can mean something like this, I am in favor
>> of it.
[...]
> But I do think that we should try to offer some way of installing
> external Lisp code if one has acquired it as a zip or tar archive in
> some manner already.

Have you ever looked at source-based package management systems like
FreeBSD's ports system or NetBSD's pkgsrc?  I think something like
this would be suited very well for installing Emacs packages.  They
basically download the source code from the original site, patch it if
necessary and call commands for building it, e.g. `configure', `make'
and `make install'.  Thus, an installation like the one of AUCTeX
would not be a problem.  In fact, FreeBSD's port for AUCTeX is
ridiculously simple.

-- 
Ralf

Re: `make' written in elisp

[David Kastrup]

Ralf Angeli <angeli@iwi.uni-sb.de> writes:

> * David Kastrup (2005-01-03) writes:
>
>> Richard Stallman <rms@gnu.org> writes:
>>
>>> I found that it was extremely clean and appealing code.  If a
>>> package system for Emacs can mean something like this, I am in favor
>>> of it.
> [...]
>> But I do think that we should try to offer some way of installing
>> external Lisp code if one has acquired it as a zip or tar archive in
>> some manner already.
>
> Have you ever looked at source-based package management systems like
> FreeBSD's ports system or NetBSD's pkgsrc?  I think something like
> this would be suited very well for installing Emacs packages.  They
> basically download the source code from the original site, patch it
> if necessary and call commands for building it, e.g. `configure',
> `make' and `make install'.  Thus, an installation like the one of
> AUCTeX would not be a problem.  In fact, FreeBSD's port for AUCTeX
> is ridiculously simple.

The problem is that the only reliable common available infrastructure
we have for all operating systems on which Emacs is able to run, is
Emacs.  And that takes us back to the subject title of this posting.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: `make' written in elisp

[Ralf Angeli]

* David Kastrup (2005-01-03) writes:

> Ralf Angeli <angeli@iwi.uni-sb.de> writes:

[source-based package management systems]

> The problem is that the only reliable common available infrastructure
> we have for all operating systems on which Emacs is able to run, is
> Emacs.  And that takes us back to the subject title of this posting.

Assuming the `make' part is solved by install.el or elmake, we "only"
need an Elisp-based substitute for `configure'.  If a package like
AUCTeX doesn't provide it, one has to require external tools.  I don't
see a problem here.  The point is that the package system should be
able to support `configure'.

-- 
Ralf

Re: `make' written in elisp

[Richard Stallman]

    Assuming the `make' part is solved by install.el or elmake, we "only"
    need an Elisp-based substitute for `configure'.

That is too heavy-weight; I am not interested in installing that kind
of package system in Emacs.

Re: `make' written in elisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     Assuming the `make' part is solved by install.el or elmake, we "only"
>     need an Elisp-based substitute for `configure'.
>
> That is too heavy-weight; I am not interested in installing that kind
> of package system in Emacs.

Well, at least a standard function for finding the path name of an
executable in PATH and/or in general a file in a set of locations
might be nice to have.

Of course a complete autoconf system would be ridiculous.  I guess
that once we start talking about actually compiling stuff (and
autoconf's main point is about figuring compiler specialties), we
might as well assume a complete GNU build system to be present, so we
would not need to implement this in Emacs.

But as long as we are just talking about figuring out some locations
and the availability/name of some executables, there is little point
not to have a few available functions for that.  Even though it might
in some cases be more convenient if they run at runtime instead of
install time, making it possible to switch between setups/PATHs
without reinstalling/reconfiguring (or have an explicit command
PACKAGENAME-reconfigure for reconfiguring?).

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: `make' written in elisp

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 03 Jan 2005 09:02:37 +0100
> Cc: Ralf Angeli <angeli@iwi.uni-sb.de>, emacs-devel@gnu.org
> 
> Well, at least a standard function for finding the path name of an
> executable in PATH and/or in general a file in a set of locations
> might be nice to have.

We already have that, I think:

    locate-file-internal is a built-in function in `C source code'.
    (locate-file-internal FILENAME PATH &optional SUFFIXES PREDICATE)

    Search for FILENAME through PATH.
    If SUFFIXES is non-nil, it should be a list of suffixes to append to
    file name when searching.
    If non-nil, PREDICATE is used instead of `file-readable-p'.
    PREDICATE can also be an integer to pass to the access(2) function,
    in which case file-name-handlers are ignored.

Isn't this what you wanted?

Re: `make' written in elisp

[Stefan Monnier]

>> Well, at least a standard function for finding the path name of an
>> executable in PATH and/or in general a file in a set of locations
>> might be nice to have.

> We already have that, I think:

>     locate-file-internal is a built-in function in `C source code'.
>     (locate-file-internal FILENAME PATH &optional SUFFIXES PREDICATE)

I'd rather advertise locate-file.  There's also executable-find.


        Stefan

Re: `make' written in elisp

[Ralf Angeli]

* Richard Stallman (2005-01-03) writes:

>     Assuming the `make' part is solved by install.el or elmake, we "only"
>     need an Elisp-based substitute for `configure'.
>
> That is too heavy-weight; I am not interested in installing that kind
> of package system in Emacs.

At least it should be able to call things like `configure' which is a
mandatory step in the build process of packages like AUCTeX and
preview-latex.  During this step the values of some important
variables are determined and written to the package's site
configuration file.  For example XEmacs' package system suffers from
its inability to do that.  AUCTeX users who want to use the package
provided by the XEmacs package system have to do the respective
configuration manually which the configure script does automatically.

-- 
Ralf

Re: `make' written in elisp

[Richard Stallman]

    At least it should be able to call things like `configure' which is a
    mandatory step in the build process of packages like AUCTeX and
    preview-latex.  During this step the values of some important
    variables are determined and written to the package's site
    configuration file.

Why is it important for Emacs Lisp packages to do that sort of thing?

Re: `make' written in elisp

[Ralf Angeli]

* Richard Stallman (2005-01-03) writes:

>     At least it should be able to call things like `configure' which is a
>     mandatory step in the build process of packages like AUCTeX and
>     preview-latex.  During this step the values of some important
>     variables are determined and written to the package's site
>     configuration file.
>
> Why is it important for Emacs Lisp packages to do that sort of thing?

Automatically determining variables specific to the site the package
is being installed at configuration time is a means to cut down load
times.  There are things where it is not efficient or feasible to
carry out the respective tests each time the package is loaded.

As an example, in AUCTeX we are currently using `configure' to
determine locations where TeX trees are located.  The resulting value
is used at runtime for speeding up file searches.

Another example (which is not yet implemented) is the determination of
programs for viewing DVI or PDF files on different platforms.
`configure' can check which alternative is present on the system at
hand and insert the choice into the site configuration file.  This
would allow us to get rid of the different platform-specific files we
now have.

Besides writing values to init files the configuration process is used
to check if external tools required for building or running the
package are present and provide necessary features.  A special case
might be preview-latex which has a TeX part besides the Elisp part.
For the respective file the target directory for its installation is
determined by `configure'.

-- 
Ralf

Re: `make' written in elisp

[David Kastrup]

Ralf Angeli <angeli@iwi.uni-sb.de> writes:

> * Richard Stallman (2005-01-03) writes:
>
>>     At least it should be able to call things like `configure'
>>     which is a mandatory step in the build process of packages like
>>     AUCTeX and preview-latex.  During this step the values of some
>>     important variables are determined and written to the package's
>>     site configuration file.
>>
>> Why is it important for Emacs Lisp packages to do that sort of
>> thing?
>
> Automatically determining variables specific to the site the package
> is being installed at configuration time is a means to cut down load
> times.  There are things where it is not efficient or feasible to
> carry out the respective tests each time the package is loaded.
>
> As an example, in AUCTeX we are currently using `configure' to
> determine locations where TeX trees are located.  The resulting
> value is used at runtime for speeding up file searches.

I am not actually convinced that this is a good idea: I'd prefer it
that _if_ AUCTeX is made to match some package layout, then the whole
installation/configuration will be performed by Lisp.

Probably cached in customize variables, and with an option to redo the
detection on demand (in case you want to experiment with different
setups of TeX distributions or similar).

I'd really like it if we could skip all of the autoconf magic (which
is particularly painful and error-prone dealing with paths with spaces
and backslashes) for a reasonably easy Elisp solution.  It would
appear that with some of the mentioned functions in this thread, this
might be possible.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: `make' written in elisp

[Richard Stallman]

    Automatically determining variables specific to the site the package
    is being installed at configuration time is a means to cut down load
    times.  There are things where it is not efficient or feasible to
    carry out the respective tests each time the package is loaded.

I am surprised to hear that.  Many Emacs Lisp packages search for
files they need to use, and they generally do so either when first
loaded into a session, or each time they are executed.  We have never
tried to make any of them save the results between sessions, but it
seems to be fast enough.

Is AUCTeX doing something that is particularly slow?

    Besides writing values to init files the configuration process is used
    to check if external tools required for building or running the
    package are present and provide necessary features.  A special case
    might be preview-latex which has a TeX part besides the Elisp part.

Why is it necessary to do this?
It seems to me that it would be just fine
to look for these things when the user tries to use them.

to try to use the command when the user asks to do

Re: `make' written in elisp

[Ralf Angeli]

* Richard Stallman (2005-01-04) writes:

> I am surprised to hear that.  Many Emacs Lisp packages search for
> files they need to use, and they generally do so either when first
> loaded into a session, or each time they are executed.  We have never
> tried to make any of them save the results between sessions, but it
> seems to be fast enough.
>
> Is AUCTeX doing something that is particularly slow?

That depends on how you define slow.  I just extracted the relevant
code for testing purposes and using `time' with the resulting shell
script I got the following execution times on a Pentium M 1,7
(currently running at 600MHz):

  real    0m0.114s
  user    0m0.084s
  sys     0m0.018s

In the meantime I reimplemented the functionality in Elisp.  Using

  (abs (- (prog1 (float-time (current-time)) (TeX-input-dirs))
          (float-time (current-time))))

for measuring the execution time I got an average of about 0.06
seconds for executing the function `TeX-input-dirs'.  While I am not
really fond of having the function executed every time the package is
loaded, it is probably an acceptable delay.  And, as David already
mentioned, we could find ways to save its output and subsequently
inhibit its execution until a user calls some reconfiguration
procedure.

>     Besides writing values to init files the configuration process is used
>     to check if external tools required for building or running the
>     package are present and provide necessary features.  A special case
>     might be preview-latex which has a TeX part besides the Elisp part.
>
> Why is it necessary to do this?
> It seems to me that it would be just fine
> to look for these things when the user tries to use them.
>
> to try to use the command when the user asks to do

I don't think it is feasible to check for program versions every time
a program is called.  And instead of letting the user alone with error
messages which don't give him a hint that the program is too old, I
consider it better practice to tell him when he tries to install the
software which relies on the external program.

-- 
Ralf

Re: `make' written in elisp

[Stefan]

[-- Attachment #1: Type: text/plain, Size: 4493 bytes --]

> Judging from the comments, this installer might more or less get along
> at least with some XEmacs packages, as well as some other packages
> with a regular enough structure (AUCTeX has in the mean time gotten a
> new autoconf-based installer very similar to that of preview-latex, so
> the latest versions probably will not work with install.el).

It might still work, depending on how the configuration is done.  I tend to
think that most of the autoconfiguration should be done upon use rather
than during installation, which basically implies "in elisp rather than in
autoconf".

> a) a directory layout.  XEmacs packages have a standard way of placing
> Elisp, info, data and other files in a layout under the top package
> directory.  In addition there are some standard file names for
> initializing of packages and installation.  In short, this part of the
> package system is basically just file layout conventions relative to a
> package-specific top directory.

> We don't have any conventions for packages intended for Emacs as far
> as I can see.  It certainly would not be bad to have something like
> that: if people make code for Emacs, and if we can agree on some
> layout that makes it easier for stuff to work out of the box, it
> certainly might be worthwhile to state some conventions.

Yes, `install' does not impose a particular structure (it just looks
recursively for *.el files).  But it would benefit from some conventions
(so it can safely ignore the contrib/foo.el or compat/bar.el files).
I can't find any documentation about the in-package layout used by XEmacs,
tho.  Anybody knows where it's described?

> b) version management.  Versions must basically be floats, higher
> versions correspond to later versions, are usually based on some CVS
> archive and not at all with actual package versions.  Uh.

Right, it seems to be used only to discover whether a newer version is
available or not.

> c) a package manager that will download and upgrade packages.  It
> might do so in user-specific and system-wide packages.

> d) central download archives that will provide all packages, updated
> by packagers (often different from the actual creators of software) on
> central servers.

> e) it does not offer AFAIR: dependencies and standard tests (for
> executables, directories and other stuff) that autoconf provides.  A
> bit of framework for that might be nice for obliterating some
> necessities for installation.  Even though installing MSYS is not
> particularly hard, it still seems to be a barrier for some people.

`install' doesn't get involved with version management, dependencies, ...
It only tries to make it easier for users to install and setup external
packages.  A package manager to also allow activation/deactivation/deletion
would be a natural addition.

> I am not convinced about the value of the centralized server approach
> to package management from XEmacs: it does not seem to lend itself

Since `install' specifically targets external packages, a centralized server
doesn't make much sense indeed.  For XEmacs, it was probably
a good simplification.

> But I do think that we should try to offer some way of installing
> external Lisp code if one has acquired it as a zip or tar archive in
> some manner already.

Not just tarballs but also single elisp files.  Most elisp packages are
single files.

> And in those respects where the directory structure and layout
> conventions of an XEmacs package would seem reasonable, there seems
> little point in inventing something else.

Agreed.

> Stefan, having met those problems when writing install.el already,
> would probably be a lot more qualified to comment on my impressions
> and on the versatility of the package layout he has been catering for.

The DWIM-aspect of `install' (just look for *.el and *.info files) only
works up to a point.  I've bumped into problems where some *.el files should
be ignored, or other problems where the *.info file doesn't have the `info'
extension, or where the info file simply doesn't exist and it's not clear
how to build it based on a bunch of *.texi files, ...

We could keep adding intelligence and heuristics, but it makes more sense to
devise some conventions.

> All of those certainly sound reasonable.  In particular, I'd second having
> separate commands for installing system-wide and user-local packages.

Agreed, it makes more sense.


        Stefan


PS: I attached my latest version, which is still "work in progress".



[-- Attachment #2: install.el --]
[-- Type: application/emacs-lisp, Size: 22759 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: `make' written in elisp

[Stephen J. Turnbull]

>>>>> "Stefan" == Stefan  <monnier@iro.umontreal.ca> writes:

    Stefan> I tend to think that most of the autoconfiguration should
    Stefan> be done upon use rather than during installation, which
    Stefan> basically implies "in elisp rather than in autoconf".

A word to the wise: This is expensive at startup, and will probably
take some effort to shake out.  Mike Sperber put in many hours
removing unnecessary stats from our naive package location code,
despite the very regular layout of our package hierarchy.  It's still
pretty straightforward, but the efficiency is not an accident.

It would theoretically be possible to have a cache which collects all
the auto-autoloads, but we felt that was over-engineering; startup is
not something that should happen very often in Emacs.

    dak> a) a directory layout.  XEmacs packages have a standard way
    dak> of placing Elisp, info, data and other files in a layout
    dak> under the top package directory.

XEmacs's layout is inverted from what makes sense for third-party
packages: we have a standard GNU Emacs-style tree with lisp/$package,
info/$package.info*, etc as needed.  Really, however, third-party
packages want $package/{lisp,lib-src,etc,...}.  The Emacspeak and JDEE
maintainers absolutely insist on this, for two examples.

XEmacs is moving toward flattening out the core lisp, and eventually
inverting the tree for packages.  The only exceptions will be package
meta-data (manifests used to ensure that package upgrades and removals
get rid of everything installed by unwanted packages), and info files.

This is probably not an appropriate strategy for Emacs, but you should
be aware that XEmacs is not necessarily going to stick with the
current structure.

    dak> In addition there are some standard file names for
    dak> initializing of packages and installation.  In short, this
    dak> part of the package system is basically just file layout
    dak> conventions relative to a package-specific top directory.

We have auto-autoloads.el, custom-loads.el, and _pkg.el.  The first is
what it looks like, the second handles attaching Custom's menus and
groups and so on, and the third contains meta-data which actually is
inserted into auto-autoloads.el, and never actually loaded itself.

    Stefan> I can't find any documentation about the in-package layout
    Stefan> used by XEmacs, tho.  Anybody knows where it's described?

The hierarchy is simple, and not really explicitly documented.  There
is some information on internal structure, especially for the source
trees, in XEmacs Users' Guide (node Packages) and Lispref (node
Packaging), see http://www.xemacs.org/Documentation/21.5/html/.  There
may be some commentary in the Internals Manual in the "Future Work"
nodes, but they're quite disorganized, and mostly written by Ben who
is not authoritative for the packages.

The installed package hierarchy is as described above:
$top/{etc,info,lib-src,lisp,man,pkginfo,...} with packages constrained
to install into like-named subdirectories, except for $top/info, where
the name should be $package.info*, and $top/pkginfo, which contains
the MANIFEST.$package files used when uninstalling.

We're policy-free with respect to source directory organization.
Basically in source the required files are an XEmacs-style Makefile
and a metadata file, package-info.in.  The three files referred to
above are automatically generated.

    dak> b) version management.  Versions must basically be floats,
    dak> higher versions correspond to later versions, are usually
    dak> based on some CVS archive and not at all with actual package
    dak> versions.  Uh.

    Stefan> Right, it seems to be used only to discover whether a
    Stefan> newer version is available or not.

Yeah, we need to do something about that.  I don't know what Steve
Baur was thinking.

    dak> c) a package manager that will download and upgrade packages.
    dak> It might do so in user-specific and system-wide packages.

XEmacs's is currently in flux.  Its #1 design bug is absence of an
HTTP transport, both because it makes EFS a single point of failure,
and because of the firewall issue.

However, unless there's some easily findable index of URLs for
downloadable packages, this is not terribly useful, I think.

    dak> I am not convinced about the value of the centralized server
    dak> approach to package management from XEmacs: it does not seem
    dak> to lend itself

    Stefan> Since `install' specifically targets external packages, a
    Stefan> centralized server doesn't make much sense indeed.  For
    Stefan> XEmacs, it was probably a good simplification.

The centralized server is _not_ designed as part of the package
management system, but as a service provided to the community.  Due to
the convenience for maintainers who don't necessarily have commit
access to the upstream project (assuming it exists), it has become a
dependency.  Doing something about that would require some effort.

There is one point that needs to be mentioned, however, and that is
that for complex packages that depend on other packages, the "calling
macros from byte-compiled code" kind of bug has dropped from FAQ to
fossil status.

    dak> But I do think that we should try to offer some way of
    dak> installing external Lisp code if one has acquired it as a zip
    dak> or tar archive in some manner already.

    Stefan> Not just tarballs but also single elisp files.  Most elisp
    Stefan> packages are single files.

This we really are not set up to handle efficiently.  I think we
should, so I'll be interested to see how you handle it.

    Stefan> The DWIM-aspect of `install' (just look for *.el and
    Stefan> *.info files) only works up to a point.  I've bumped into
    Stefan> problems where some *.el files should be ignored, or other

We handle this by having an explicit list as a make variable.  Usually
the default setting of "all .el files" works fine, of course, and
that's easily implemented with `wildcard'.

    Stefan> problems where the *.info file doesn't have the `info'
    Stefan> extension, or where the info file simply doesn't exist and
    Stefan> it's not clear how to build it based on a bunch of *.texi
    Stefan> files, ...

Yep.  Info files and package-specific data (eg, pixmaps, the Yow!
database, SGML DTDs, JDEE's Java code, etc) are the reason for 1/2 of
the hair in the package build infrastructure.  Most of the rest has to
do with supporting building on Windows.

    Stefan> We could keep adding intelligence and heuristics, but it
    Stefan> makes more sense to devise some conventions.

I agree in theory, but in practice, you'll run into resistance (ie,
they just won't bother) from 3rd-party maintainers, all the more since
you're focusing on "external" packages.

    dak> All of those certainly sound reasonable.  In particular, I'd
    dak> second having separate commands for installing system-wide
    dak> and user-local packages.

    Stefan> Agreed, it makes more sense.

XEmacs experience with "guessing" the target tree has not been
positive; we probably should consider moving in that direction
ourselves.


-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;

Re: `make' written in elisp

[Stefan Monnier]

Stefan> I tend to think that most of the autoconfiguration should
Stefan> be done upon use rather than during installation, which
Stefan> basically implies "in elisp rather than in autoconf".

> A word to the wise: This is expensive at startup, and will probably
> take some effort to shake out.  Mike Sperber put in many hours
> removing unnecessary stats from our naive package location code,
> despite the very regular layout of our package hierarchy.  It's still
> pretty straightforward, but the efficiency is not an accident.

I think we're not talking about the same thing.  IIUC You're talking about
the time it takes for XEmacs to find all the packages and update the
load-path accordingly.  `install' does not suffer from this problem since it
instead requires the user to write this list directly in his .emacs file
(which also has the advantage that the user can choose between Gnus-5.8 and
Gnus-5.10 when both are installed).  Also `install' only handles external
packages which are likely to be much fewer than for XEmacs-with-Sumo.

What I was talking about is things like latex-preview searching for some
LaTeX files.  This is not done at startup: it is currently done at install
time, and I was arguing about that it should be done when latex-preview is
loaded.  Of course it may also take too much time, but doing it at use time
has the advantage that it does not require a re-install whenever LaTeX files
are moved from /usr/local/texmf to /usr/texmf, or /usr/lib/texmf, or
/usr/share/texmf, or /opt/texmf, or god knows where else.

Stefan> I can't find any documentation about the in-package layout
Stefan> used by XEmacs, tho.  Anybody knows where it's described?
[...]
> We're policy-free with respect to source directory organization.
> Basically in source the required files are an XEmacs-style Makefile
> and a metadata file, package-info.in.  The three files referred to
> above are automatically generated.

So it's similar to `install' in this sense, except that you require
a Makefile to describe where the various files are located.
That sounds like a good way to do it (tho for `install', it would make more
sense to use a .el file for that info since `install' does the job of
`make' for typical simple packages).

> There is one point that needs to be mentioned, however, and that is
> that for complex packages that depend on other packages, the "calling
> macros from byte-compiled code" kind of bug has dropped from FAQ to
> fossil status.

So there is some amount of version-dependency checking now?

Stefan> Not just tarballs but also single elisp files.  Most elisp
Stefan> packages are single files.

> This we really are not set up to handle efficiently.  I think we
> should, so I'll be interested to see how you handle it.

Take a look at install.el.  The case of single files was the first that
worked.  It's much simpler than tarballs: place it in the desired directory,
add the autoloads to the directory's autoload file, byte-compile, done.

Remember, `install' aims for simplicity and tries as much as possible to
stick to what a real user would manually do.  I haven't tried to handle
autoconf'd packages yet, but there's no reason why `install' couldn't just
say: oh wait, there's a `configure' file, let's just run
"EMACS=<myself> ./configure --prefix=<myownpackagedir>; make; make install".


        Stefan

Re: `make' written in elisp

[Stephen J. Turnbull]

BTW, I'm going to be out of town for a week, so any further responses
will be delayed.

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

    Stefan> What I was talking about is things like latex-preview
    Stefan> searching for some LaTeX files.  This is not done at
    Stefan> startup: it is currently done at install time, and I was
    Stefan> arguing about that it should be done when latex-preview is
    Stefan> loaded.  Of course it may also take too much time,

That's my point.  People are just not going to find that acceptable,
at least they didn't with year 2000 hardware with XEmacs.  Even if
it's done while you're getting morning coffee---because it will take
longer than that.  Waiting for it to load every time you reload AUCTeX
will wear out the AUCTeX developers.

    Stefan> So [XEmacs packaging is] similar to `install' in this
    Stefan> sense, except that you require a Makefile to describe
    Stefan> where the various files are located.  That sounds like a
    Stefan> good way to do it (tho for `install', it would make more
    Stefan> sense to use a .el file for that info since `install' does
    Stefan> the job of `make' for typical simple packages).

Agreed.

    >> There is one point that needs to be mentioned, however, and
    >> that is that for complex packages that depend on other
    >> packages, the "calling macros from byte-compiled code" kind of
    >> bug has dropped from FAQ to fossil status.

    Stefan> So there is some amount of version-dependency checking
    Stefan> now?

No.  These are package dependencies, not version dependencies.  If a
package uses macros from another one, it is placed in the "REQUIRES"
make variable.  Then the libraries from those REQUIRE'd packages are
preloaded using the -l argument to emacs.

Theoretically these could be autogenerated, but the dependency
trackers we've seen so far have all been buggy, and miss more
dependencies than when doing it by hand.

    Stefan> Remember, `install' aims for simplicity and tries as much
    Stefan> as possible to stick to what a real user would manually
    Stefan> do.

Sure.  I don't think XEmacs will change its commitment to a more
sophisticated ("intrusive and inflexible" if you prefer :-) packaging
system.  I think we should be able to achieve interoperability in most
cases though.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: `make' written in elisp

[Stefan]

Stefan> What I was talking about is things like latex-preview
Stefan> searching for some LaTeX files.  This is not done at
Stefan> startup: it is currently done at install time, and I was
Stefan> arguing about that it should be done when latex-preview is
Stefan> loaded.  Of course it may also take too much time,

> That's my point.  People are just not going to find that acceptable,
> at least they didn't with year 2000 hardware with XEmacs.

But you're comparing apples and oranges.  With XEmacs you're talking about
lots and lots of searches at every startup, whereas with AUCTeX we're talking
about several searches done only when the feature is used.

So, yes, maybe it'll be slow, but the XEmacs experience with its package
system is no evidence of it.

Stefan> So there is some amount of version-dependency checking
Stefan> now?

> No.  These are package dependencies, not version dependencies.  If a
> package uses macros from another one, it is placed in the "REQUIRES"
> make variable.  Then the libraries from those REQUIRE'd packages are
> preloaded using the -l argument to emacs.

> Theoretically these could be autogenerated, but the dependency
> trackers we've seen so far have all been buggy, and miss more
> dependencies than when doing it by hand.

The way to fix those things is to call them bugs.  The packages should have
(require 'foobar) or (eval-when-compile (require 'foobar)) so that you get
a clean error rather than a miscompile.

Of course, you don't always have much control over the upstream code, but
you can apply local patches.  The only case where (require 'foobar) can
still lead to miscompile is when old versions of `foobar' don't provide
the macro, which is why I thought you used version-dependency checking.


        Stefan

Re: `make' written in elisp

[Miles Bader]

> > That's my point.  People are just not going to find that acceptable,
> > at least they didn't with year 2000 hardware with XEmacs.
> 
> But you're comparing apples and oranges.  With XEmacs you're talking about
> lots and lots of searches at every startup, whereas with AUCTeX we're talking
> about several searches done only when the feature is used.

Yeah, anyway don't we already do this sort of thing for configuring
e.g., grep at runtime?  Searching for a few programs and invoking them
to test options, _the first time you use a specific feature_, doesn't
seems like a big deal at all.

Does auxtex require something else?

-Miles

patch for locate-file-completion?

[Nic Ferrier]

I think locate-file-completion is broken, here's how:

    M-x load-library somepath-that-should-complete TAB

Mine causes an exception because of a nil path element in my
load-path. I'm not sure why I've got a nil path element in my
load-path but the docs for load-path say it is legal to have a nil
element.

Apologies if it's not really a problem, if there's some reason for the
exception I'd like to know.

If not, and there is a problem, here's a patch that fixes it:


Index: files.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/files.el,v
retrieving revision 1.686
diff -c -r1.686 files.el
*** files.el	27 Mar 2004 04:05:23 -0000	1.686
--- files.el	31 Mar 2004 22:26:37 -0000
***************
*** 617,623 ****
  	  (string-dir (file-name-directory string)))
        (dolist (dir (car path-and-suffixes))
  	(if string-dir (setq dir (expand-file-name string-dir dir)))
! 	(when (file-directory-p dir)
  	  (dolist (file (file-name-all-completions
  			 (file-name-nondirectory string) dir))
  	    (push (if string-dir (concat string-dir file) file) names)
--- 617,623 ----
  	  (string-dir (file-name-directory string)))
        (dolist (dir (car path-and-suffixes))
  	(if string-dir (setq dir (expand-file-name string-dir dir)))
! 	(when (and dir (file-directory-p dir))
  	  (dolist (file (file-name-all-completions
  			 (file-name-nondirectory string) dir))
  	    (push (if string-dir (concat string-dir file) file) names)

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk

Re: patch for locate-file-completion?

[Richard Stallman]

It looks to me like right fix is this one.
Does this look right to you?

diff -c /home/rms/emacs/lisp/files.el.\~80\~ /home/rms/emacs/lisp/files.el
*** /home/rms/emacs/lisp/files.el.~80~	Mon Mar 29 23:58:25 2004
--- /home/rms/emacs/lisp/files.el	Thu Apr  1 10:31:51 2004
***************
*** 616,621 ****
--- 616,623 ----
  	  (suffix (concat (regexp-opt (cdr path-and-suffixes) t) "\\'"))
  	  (string-dir (file-name-directory string)))
        (dolist (dir (car path-and-suffixes))
+ 	(unless dir
+ 	  (setq dir default-directory))
  	(if string-dir (setq dir (expand-file-name string-dir dir)))
  	(when (file-directory-p dir)
  	  (dolist (file (file-name-all-completions

Diff finished.  Thu Apr  1 10:32:37 2004

Re: `make' written in elisp

[Eric M. Ludlam]

  [ ... ]
>>     Besides writing values to init files the configuration process is used
>>     to check if external tools required for building or running the
>>     package are present and provide necessary features.  A special case
>>     might be preview-latex which has a TeX part besides the Elisp part.
>>
>> Why is it necessary to do this?
>> It seems to me that it would be just fine
>> to look for these things when the user tries to use them.
>>
>> to try to use the command when the user asks to do
>
>I don't think it is feasible to check for program versions every time
>a program is called.  And instead of letting the user alone with error
>messages which don't give him a hint that the program is too old, I
>consider it better practice to tell him when he tries to install the
>software which relies on the external program.

Hi,

  On the topic of checking version numbers at runtime, I wrote a tool
to do this for me which is in the CEDET stuff I maintain.  Many tools
have a variable, for example, foo.el, might have foo-version be a
string that is the version number.

  This tool can scan for that, and make sure the version you want is
compatible via various settings.  I also use it to manager save file
version numbers so that upgrades can occur among various tools that
are not a part of Emacs.

  The one dilemma is that a package author has to explicitly specify
when a version is no longer compatible with older versions.  ie,
current version is 2.4, but tools depending on 1.5 won't work with
2.4, this needs to be specified for inversion to handle all cases.

  Here is a CVS link:

http://cvs.sourceforge.net/viewcvs.py/cedet/cedet/common/inversion.el?rev=1.20&view=log

Enjoy
Eric

-- 
          Eric Ludlam:                 zappo@gnu.org, eric@siege-engine.com
   Home: http://www.ludlam.net            Siege: www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: www.gnu.org