Re: Elpa packages and macro dependencies.

[Achim Gratz <Stromeko@nexgo.de>]

Stefan Monnier writes:
> Indeed.  I think your situation could also be handled correctly by my
> suggestion:
>
>    E.g. we could add to bytecomp.el the ability to force `require' to
>    reload a package if it's not already loaded from the file that
>    locate-library returns.

That doesn't work when defcustoms, autoloads and defvars change in the
new version.  I've been through this in my attempts to make Org more
resistant to such mishaps.  In a proper package system there was a way
to unload a package cleanly, then fly in the new package and plug it in.
We don't have that now, instead we have "packages" that really aren't
(because they are still built-in) and some infrastructure is missing to
unload and reload a complete package cleanly.

This is what is used in Org to get rid of any built-in stuff that might
be delivered with the Emacs before running batch tests:

--8<---------------cut here---------------start------------->8---
;;
;; Remove Org remnants built into Emacs
;;

;; clean load-path
(setq load-path
      (delq nil (mapcar
                 (function (lambda (p)
                             (unless (string-match "lisp\\(/packages\\)?/org$" p)
                               p)))
                 load-path)))
;; remove property list to defeat cus-load and remove autoloads
(mapatoms (function  (lambda (s)
                       (let ((sn (symbol-name s)))
                         (when (string-match "^\\(org\\|ob\\|ox\\)\\(-.*\\)?$" sn)
                           (setplist s nil)
                           (when (eq 'autoload (car-safe s))
                             (unintern s)))))))

;; we should now start from a clean slate
--8<---------------cut here---------------end--------------->8---

This has had little exposure beyond actual batch testing, but something
very close to this might actually work in most cases.  For bytecomp the
critical part would be to know which features and namespaces to nuke,
but if it's only used for packages the information could be delivered
with the package metadata perhaps.

Here's an experimental advice (not currently implemented in Org) to
recognize a changed load-path and re-load a feature in this case to take
care of some issues at runtime:

--8<---------------cut here---------------start------------->8---
;; some parts of Org might already have been used from a different
;; place, try to reload these parts from the current load-path
(require 'loadhist)
(defadvice require (before org-require-reload-when-shadowed
			   (feature &optional filename noerror)
			   activate compile preactivate)
  "Check whether a required feature has been shadowed by changing
`load-path' after it has been loaded and reload that feature from
current load-path in this case."
  (when (featurep feature)
    (let ((feature-name (or filename (symbol-name feature))))
      (when (string-match "^\\(org\\|ob\\)\\(-.*\\)?$" feature-name)
	(let ((feature-lib  (file-name-directory (or (locate-library feature-name) "")))
	      (feature-dir  (file-name-directory (feature-file feature))))
	  ;(message "require-reload-when shadowed %s\n\t%s\n\t%s" feature-name feature-lib feature-dir)
	  (when (not (string= feature-lib feature-dir))
	    (message "Reloading %s" feature-name)
	    (unload-feature feature t)))))))
--8<---------------cut here---------------end--------------->8---

Using just unload it doesn't catch some corner cases with defcustom and
autoloads as mentioned before so it would likely not be good enough for
byte-compiling, but for "close enough" versions in the load-path this
actually fixes up things without getting in the way.

And finally org-reload (as currently implemented in Org) that tries to
do the right thing or at least warn if the load-path is set up in
strange ways so that not all features from Org are found in the same
place:

--8<---------------cut here---------------start------------->8---
;;;###autoload
(defun org-reload (&optional uncompiled)
  "Reload all org lisp files.
With prefix arg UNCOMPILED, load the uncompiled versions."
  (interactive "P")
  (require 'loadhist)
  (let* ((org-dir     (org-find-library-dir "org"))
	 (contrib-dir (or (org-find-library-dir "org-contribdir") org-dir))
	 (feature-re "^\\(org\\|ob\\|ox\\)\\(-.*\\)?")
	 (remove-re (mapconcat 'identity
			       (mapcar (lambda (f) (concat "^" f "$"))
				       (list (if (featurep 'xemacs)
						 "org-colview"
					       "org-colview-xemacs")
					     "org" "org-loaddefs" "org-version"))
			       "\\|"))
	 (feats (delete-dups
		 (mapcar 'file-name-sans-extension
			 (mapcar 'file-name-nondirectory
				 (delq nil
				       (mapcar 'feature-file
					       features))))))
	 (lfeat (append
		 (sort
		  (setq feats
			(delq nil (mapcar
				   (lambda (f)
				     (if (and (string-match feature-re f)
					      (not (string-match remove-re f)))
					 f nil))
				   feats)))
		  'string-lessp)
		 (list "org-version" "org")))
	 (load-suffixes (when (boundp 'load-suffixes) load-suffixes))
	 (load-suffixes (if uncompiled (reverse load-suffixes) load-suffixes))
	 load-uncore load-misses)
    (setq load-misses
	  (delq 't
		(mapcar (lambda (f)
			  (or (org-load-noerror-mustsuffix (concat org-dir f))
			      (and (string= org-dir contrib-dir)
				   (org-load-noerror-mustsuffix (concat contrib-dir f)))
			      (and (org-load-noerror-mustsuffix (concat (org-find-library-dir f) f))
				   (add-to-list 'load-uncore f 'append)
				   't)
			      f))
			lfeat)))
    (if load-uncore
	(message "The following feature%s found in load-path, please check if that's correct:\n%s"
		 (if (> (length load-uncore) 1) "s were" " was") load-uncore))
    (if load-misses
	(message "Some error occurred while reloading Org feature%s\n%s\nPlease check *Messages*!\n%s"
		 (if (> (length load-misses) 1) "s" "") load-misses (org-version nil 'full))
      (message "Successfully reloaded Org\n%s" (org-version nil 'full)))))
--8<---------------cut here---------------end--------------->8---


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables