Elpa packages and macro dependencies.

Elpa packages and macro dependencies.

[bruce.connor.am]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

I'm running into a problem, but I'm not sure it's actually a bug, so I'm
bringing it here.

Say a user has an Elpa package installed, called A (version 1.0) and would
like to install package B.
Package B has a (require 'A) statement, because it uses one of A's macros.
However, B lists (A "1.1") in its PackageRequires, because it uses some new
feature of this macro which was added in version 1.1.

1. Our innocent user then launches `list-packages', and selects to install
B.
2. Package.el knows that B depends on A-1.1, so it upgrades A first.
3. However, _if I understand things correctly_, package.el doesn't reload A
after upgrading it, which means that the old definitions are still active!
4. Once it proceeds to byte-compile the newly installed B, the (require 'A)
statement won't load anything because the old A is already loaded. And the
byte-compilation of B will use the old macro definition, despite B
specifically listing A-1.1 as a dependency!
5. Restarting Emacs won't fix it because B was already compiled wrong. The
user has to restart Emacs, AND reinstall (or recompile) B.

Am I correct in my understanding?
I believe this would be avoided by using (load "A") instead of (require
'A). Is this advisable?
Would this problem go away if B used (eval-when-compile (require 'A))?

Best and Thanks,
Artur

[-- Attachment #2: Type: text/html, Size: 1635 bytes --]

Re: Elpa packages and macro dependencies.

[Glenn Morris]

bruce.connor.am@gmail.com wrote:

> I'm running into a problem, but I'm not sure it's actually a bug, so I'm
> bringing it here.

Well, it's
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10125
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18443
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18448

Re: Elpa packages and macro dependencies.

[bruce.connor.am]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

I see. Thanks for the pointers!

2014-10-16 17:09 GMT+01:00 Glenn Morris <rgm@gnu.org>:

> bruce.connor.am@gmail.com wrote:
>
> > I'm running into a problem, but I'm not sure it's actually a bug, so I'm
> > bringing it here.
>
> Well, it's
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10125
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18443
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18448
>

[-- Attachment #2: Type: text/html, Size: 1056 bytes --]

Re: Elpa packages and macro dependencies.

[Thierry Volpiatto]

Glenn Morris <rgm@gnu.org> writes:

> bruce.connor.am@gmail.com wrote:
>
>> I'm running into a problem, but I'm not sure it's actually a bug, so I'm
>> bringing it here.
>
> Well, it's
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10125
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18443
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18448

Seems this problem is fixed for async package users (need to require
async.bytecomp though).

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

>> I'm running into a problem, but I'm not sure it's actually a bug, so I'm
>> bringing it here.

> Well, it's
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10125
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18443
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18448

Indeed.  I think your situation could also be handled correctly by my
suggestion:

   E.g. we could add to bytecomp.el the ability to force `require' to
   reload a package if it's not already loaded from the file that
   locate-library returns.


-- Stefan

Re: Elpa packages and macro dependencies.

[bruce.connor.am]

[-- Attachment #1: Type: text/plain, Size: 585 bytes --]

>
> Indeed.  I think your situation could also be handled correctly by my
> suggestion:
>
>    E.g. we could add to bytecomp.el the ability to force `require' to
>    reload a package if it's not already loaded from the file that
>    locate-library returns.
>
> Yes, that's what I'm resorting to now. I'm writing the package so that it
reloads
itself before doing anything if `locate-library' returns a newer version.

It's a little annoying that it has to visit a file and check every time
it's used,
but it only happens at byte-compile time, so it's not a huge deal.

Cheers,
Artur

[-- Attachment #2: Type: text/html, Size: 994 bytes --]

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> Yes, that's what I'm resorting to now. I'm writing the package so that
> it reloads itself before doing anything if `locate-library' returns
> a newer version.
> It's a little annoying that it has to visit a file and check every
> time it's used, but it only happens at byte-compile time, so it's not
> a huge deal.

The idea would be to make this automatic in the byte-compiler, so the
packages don't need to worry about it.

Would you like to work on a patch for that?


        Stefan

Re: Elpa packages and macro dependencies.

[bruce.connor.am]

> The idea would be to make this automatic in the byte-compiler, so the
> packages don't need to worry about it.
>
> Would you like to work on a patch for that?
>

Sure, I’ll give it a try.

Re: Elpa packages and macro dependencies.

[bruce.connor.am]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

> > The idea would be to make this automatic in the byte-compiler, so the
> > packages don't need to worry about it.

Actually, now that I think about this,  wouldn't it make more sense to fix
this
on package.el instead of the bytecompiler?

Before upgrading a package, package.el could check if the package is already
loaded and then force-reload it after the installation. If there's a way of
knowing
the features provided by a package (look at filenames?) this would be
trivial.

[-- Attachment #2: Type: text/html, Size: 592 bytes --]

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

>> > The idea would be to make this automatic in the byte-compiler, so the
>> > packages don't need to worry about it.
> Actually, now that I think about this, wouldn't it make more sense to
> fix this on package.el instead of the bytecompiler?

Don't take my word too literally.  Maybe this can be done all in
package.el without touching the byte-compiler, indeed.

> Before upgrading a package, package.el could check if the package is already
> loaded and then force-reload it after the installation.

The question is how to do the "force-reload".  Once we know how to do
that, there are various options as to when to do this force-reload, and
I think this is an easier problem.


        Stefan

Re: Elpa packages and macro dependencies.

[bruce.connor.am]

[-- Attachment #1: Type: text/plain, Size: 550 bytes --]

> > Before upgrading a package, package.el could check if the package is
already
> > loaded and then force-reload it after the installation.
>
> The question is how to do the "force-reload".  Once we know how to do
> that, there are various options as to when to do this force-reload, and
> I think this is an easier problem.

Just calling (load "file") is what I had in mind.
It's enough to redefine macros, which are the biggest issue IMO.

It won't redefine defcustom/var, but I'm not even convinced that it would
be a good idea to redefine them.

[-- Attachment #2: Type: text/html, Size: 696 bytes --]

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

>> > Before upgrading a package, package.el could check if the package
>> > is already loaded and then force-reload it after the installation.
>> The question is how to do the "force-reload".  Once we know how to do
>> that, there are various options as to when to do this force-reload, and
>> I think this is an easier problem.
> Just calling (load "file") is what I had in mind.
> It's enough to redefine macros, which are the biggest issue IMO.

There are some issues to deal with:
- We shouldn't load "file" if it's not already loaded.
- Packages can include several files, so we have to figure out the set
  of files to load and (here's the rub) the order in which to load them.

> It won't redefine defcustom/var, but I'm not even convinced that it would
> be a good idea to redefine them.

It can be important in some cases, but admittedly, it's not clear how to
do it safely.


        Stefan

Re: Elpa packages and macro dependencies.

[Nicolas Richard]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> - Packages can include several files, so we have to figure out the set
>   of files to load and (here's the rub) the order in which to load them.

It could be approximated by doing it in the order found in load-history.

-- 
Nicolas Richard

Re: Elpa packages and macro dependencies.

[Artur Malabarba]

>> - Packages can include several files, so we have to figure out the set
>>   of files to load and (here's the rub) the order in which to load them.
>
> It could be approximated by doing it in the order found in load-history.
>

Thanks for the idea, Nicolas. I was struggling a bit with that part.

Here's what I had in mind. (I can turn this in to a patch, of course)

Inside the package-activate-1 function, immediately after loading the
“autoloads” file (which happens invariably), we call load on each file
in this package corresponding to features already loaded. Here's the
relevant snippet:

________________________________

(with-demoted-errors
  (load (expand-file-name autoload-name pkg-dir) nil t)
  ;; Call `load' on all files in `pkg-dir' which correspond to
  ;; provided features. Skip autoloads file since we already
  ;; evaluated it above.
  (mapcar (lambda (file) (load (expand-file-name file pkg-dir) nil t))
          ;; The autoloads file is usually not a feature, but better stay safe.
          (remove autoload-name (package-list-loaded-files pkg-dir))))

________________________________

The package-list-loaded-files function is then defined as follows. We
don't need to worry about inter-package dependency here, package.el
has already taken care of calling package-install in the right order.

We DO need to worry about intra-package dependencies (which
files require which), so we load them in the same order in which they
were originally added to features (thanks to Nicolas' idea). This should
be correct as long as the order in which files require each other doesn't
change between versions. And even when it does change, this can only
cause problems in situations where the current implementation already
causes problems anyway.

________________________________

(defun package-list-loaded-files (dir)
  "List all files in DIR which correspond to loaded features.
Returns the `file-name-base' of each file, sorted by most
recently loaded last."
  (mapcar
   ;; Turn the sorted list of cons back into a list of files.
   #'car
   (sort
    (remove nil (mapcar
                 (lambda (x) (let* ((name (file-name-base x))
                               (hist (memq (intern-soft name) features)))
                          ;; Return (FILENAME . HISTORY-POSITION)
                          (when hist
                            (cons name (length hist)))))
                 (directory-files "./" nil "^[^\\.].*\\.el\\'" 'nosort)))
    ;; The cdr is the position in history
    (lambda (x y) (< (cdr x) (cdr y))))))

________________________________

Haven't had time to test a full installation yet. But I've verified
package-list-loaded-files works.

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> (with-demoted-errors
>   (load (expand-file-name autoload-name pkg-dir) nil t)
>   ;; Call `load' on all files in `pkg-dir' which correspond to
>   ;; provided features. Skip autoloads file since we already
>   ;; evaluated it above.
>   (mapcar (lambda (file) (load (expand-file-name file pkg-dir) nil t))
>           ;; The autoloads file is usually not a feature, but better stay safe.
>           (remove autoload-name (package-list-loaded-files pkg-dir))))

Pretty hackish, but it might work.
We might also want to force the `autoload' calls in autoload-name to
replace the existing definitions with autoloads.


        Stefan

Re: Elpa packages and macro dependencies.

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

> > (with-demoted-errors
> >   (load (expand-file-name autoload-name pkg-dir) nil t)
> >   ;; Call `load' on all files in `pkg-dir' which correspond to
> >   ;; provided features. Skip autoloads file since we already
> >   ;; evaluated it above.
> >   (mapcar (lambda (file) (load (expand-file-name file pkg-dir) nil t))
> >           ;; The autoloads file is usually not a feature, but better
stay safe.
> >           (remove autoload-name (package-list-loaded-files pkg-dir))))
>
> Pretty hackish, but it might work.

For what it's worth, it doesn't look too hackish to me.
Emacs already works under the assumption that features correspond to
filenames and that filenames are unique. We're just building on that
assumption.

> We might also want to force the `autoload' calls in autoload-name to
> replace the existing definitions with autoloads.

Maybe I misunderstood your earlier e-mail, but didn't you show that
autoloads updated just fine with regular loading?

[-- Attachment #2: Type: text/html, Size: 1209 bytes --]

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> Maybe I misunderstood your earlier e-mail, but didn't you show that
> autoloads updated just fine with regular loading?

No, they only update a function if the definition was either
non-existent or was itself an autoload.  If the function already has
a non-autoload definition, then `autoload' does nothing.


        Stefan

Re: Elpa packages and macro dependencies.

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

2014-10-20 10:33 GMT+01:00 Artur Malabarba <bruce.connor.am@gmail.com>:

> Here's what I had in mind. [...]

So, where are we on this?

I'm attaching a proper patch.
To reiterate, this makes package.el, after installing a package, call
`load` on any file which corresponds to an already loaded fature.
This should fix packages whose upgrade is messed up because of macro
dependencies.

Is it ok to apply or is it too hacky?

[-- Attachment #2: 0001-lisp-emacs-lisp-package.el-Reload-loaded-packages-af.patch --]
[-- Type: application/octet-stream, Size: 2819 bytes --]

From d3f79fe7d21f804a90ac055bddb0427e0d6da19d Mon Sep 17 00:00:00 2001
From: Artur Malabarba <bruce.connor.am@gmail.com>
Date: Thu, 13 Nov 2014 11:51:35 +0000
Subject: [PATCH] lisp/emacs-lisp/package.el: Reload loaded packages after
 installation.

---
 lisp/emacs-lisp/package.el | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 4e5c397..34e200e 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -524,9 +524,16 @@ Return the max version (as a string) if the package is held at a lower version."
       (error "Internal error: unable to find directory for `%s'"
 	     (package-desc-full-name pkg-desc)))
     ;; Add to load path, add autoloads, and activate the package.
-    (let ((old-lp load-path))
-      (with-demoted-errors
-        (load (expand-file-name (format "%s-autoloads" name) pkg-dir) nil t))
+    (let ((old-lp load-path)
+          (autoloads-file (format "%s-autoloads" name)))
+      (with-demoted-errors (format "Error loading %s: %%s" name)
+        (load (expand-file-name autoloads-file pkg-dir) nil t)
+        ;; Call `load' on all files in `pkg-dir' which correspond to
+        ;; provided features. Skip autoloads file since we already
+        ;; evaluated it above.
+        (mapc (lambda (file) (load (expand-file-name file pkg-dir) nil t))
+              ;; The autoloads file is usually not a feature, but better stay safe.
+              (remove autoloads-file (package-list-loaded-files pkg-dir))))
       (when (and (eq old-lp load-path)
                  (not (or (member pkg-dir load-path)
                           (member pkg-dir-dir load-path))))
@@ -543,6 +550,24 @@ Return the max version (as a string) if the package is held at a lower version."
     ;; Don't return nil.
     t))
 
+(defun package-list-loaded-files (dir)
+  "List all files in DIR which correspond to loaded features.
+Returns the `file-name-base' of each file, sorted by most
+recently loaded last."
+  (mapcar
+   ;; Turn the list of cons back into a list of files.
+   #'car
+   (sort
+    (remove nil (mapcar 
+                 (lambda (x) (let* ((name (file-name-base x))
+                               (hist (memq (intern-soft name) features)))
+                          ;; Return (FILENAME . HISTORY-POSITION)
+                          (when hist
+                            (cons name (length hist)))))
+                 (directory-files "./" nil "^[^\\.].*\\.el\\'" 'nosort)))
+    ;; The cdr is the position in history
+    (lambda (x y) (< (cdr x) (cdr y))))))
+
 (defun package-built-in-p (package &optional min-version)
   "Return true if PACKAGE is built-in to Emacs.
 Optional arg MIN-VERSION, if non-nil, should be a version list
-- 
1.7.12.4

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> +        ;; Call `load' on all files in `pkg-dir' which correspond to
> +        ;; provided features. Skip autoloads file since we already
> +        ;; evaluated it above.

Please add a to this comment a short explanation for why we do
those loads.

Also I think we should not pay attention to features here, only to file names.

One more thing: we should do those loads *after* the code that updates
load-path.

> +                 (lambda (x) (let* ((name (file-name-base x))
> +                               (hist (memq (intern-soft name) features)))

I.e. here we should use `load-history' rather than `features'.

One other thing: we could probably get something more reliable by doing
the following:
1- before loading the autoloads file (and before updating load-path), get
   do the `directory-files' call
2- use `locate-library' to see which of those files have previous
   "conflicting" files.
3- check in load-history which of those conflicting files have been
   already loaded.
4- then load autoloads and updates load-path.
5- then load those files identified at step 3.

Maybe we could use the same mechanism after byte-compilation, so as to
re-re-load the file after it's been byte-compiled.  Tho this might just
be asking for trouble.


        Stefan

Re: Elpa packages and macro dependencies.

[Artur Malabarba]

2014-11-13 17:34 GMT+00:00 Stefan Monnier <monnier@iro.umontreal.ca>:
> One other thing: we could probably get something more reliable by doing
> the following:
> 1- before loading the autoloads file (and before updating load-path), get
>    do the `directory-files' call
> 2- use `locate-library' to see which of those files have previous
>    "conflicting" files.
> 3- check in load-history which of those conflicting files have been
>    already loaded.
> 4- then load autoloads and updates load-path.
> 5- then load those files identified at step 3.
>
> Maybe we could use the same mechanism after byte-compilation, so as to
> re-re-load the file after it's been byte-compiled.  Tho this might just
> be asking for trouble.

Ok, I think the current version should adhere to all of this.
Is it ok for me to push to a branch on the origin, or should I just
keep posting patches on this thread?

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> Ok, I think the current version should adhere to all of this.
> Is it ok for me to push to a branch on the origin, or should I just
> keep posting patches on this thread?

Either way is fine,


        Stefan

Re: Elpa packages and macro dependencies.

[Achim Gratz]

Stefan Monnier writes:
> The question is how to do the "force-reload".  Once we know how to do
> that, there are various options as to when to do this force-reload, and
> I think this is an easier problem.

To me it seems that the discussion has veered from the original topic to
a second(ary) problem here.

The first problem is to ensure that the package is byte-compiled
correctly.  To that end, I still think it needs to be sent off to an
Emacs instance that is either freshly started or forked from some
known-good state.  The dependencies of the to-be-compiled package should
be activated before compilation starts.

The second problem is how (if at all) to re-load the newly compiled
package into the Emacs instance the user has been using.  You could
start with the org-reload implementation I posted (which is also in
Emacs).  In a nutshell, it checks which files have previously been
loaded and re-loads them.  This won't always work depending on what
exactly has changed between versions.  Sometimes doing another reload is
in order and sometimes you just have to restart.  You'll likely have to
delegate some of that responsibility to the package, since you generally
can't know what exactly to do in which order for each package.  Then
again, the package maintainer may not know either…


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> The first problem is to ensure that the package is byte-compiled
> correctly.

Indeed.

> The second problem is how (if at all) to re-load the newly compiled
> package into the Emacs instance the user has been using.

That's right.

*BUT*

If we can solve the second problem, then the first problem is solved as
well (by reloading the new package(s) before compiling).


        Stefan

Re: Elpa packages and macro dependencies.

[Achim Gratz]

Stefan Monnier writes:
>> The first problem is to ensure that the package is byte-compiled
>> correctly.
>
> Indeed.
>
>> The second problem is how (if at all) to re-load the newly compiled
>> package into the Emacs instance the user has been using.
>
> That's right.
>
> *BUT*
>
> If we can solve the second problem, then the first problem is solved as
> well (by reloading the new package(s) before compiling).

No it isn't.  It's quite easy to get Emacs into a state where you cannot
correctly compile anything for instance, although usually the problems
will be more subtle.

Yes, you can make some problems with macros disappear by re-loading
(from source) the package before byte compilation, but it doesn't solve
the first problem completely.  So we might just as well recognize that
these are two different problems to solve that allow for different
solutions.  The first problem can be solved with another Emacs instance,
while the second problem must be solved in the same Emacs instance.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

>> If we can solve the second problem, then the first problem is solved as
>> well (by reloading the new package(s) before compiling).
> No it isn't.  It's quite easy to get Emacs into a state where you cannot
> correctly compile anything for instance, although usually the problems
> will be more subtle.

What kind of situations/problems are you thinking of?


        Stefan

Re: Elpa packages and macro dependencies.

[Achim Gratz]

Stefan Monnier writes:
>> No it isn't.  It's quite easy to get Emacs into a state where you cannot
>> correctly compile anything for instance, although usually the problems
>> will be more subtle.
>
> What kind of situations/problems are you thinking of?

Clever advises or aliases the user has set up that change what gets
compiled.  Another possibility are manually loaded packages that
misguide feature checks at compile time.  The old cl macros were a rich
source of potential problems if you weren't careful, but the advent of
cl-lib has made things much cleaner.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> Clever advises or aliases the user has set up that change what gets
> compiled.  Another possibility are manually loaded packages that
> misguide feature checks at compile time.  The old cl macros were a rich
> source of potential problems if you weren't careful, but the advent of
> cl-lib has made things much cleaner.

I can think of many different hypothetical ways it can fail, indeed, but
I'm wondering about actual cases.


        Stefan

Re: Elpa packages and macro dependencies.

[Achim Gratz]

Stefan Monnier writes:
>> Clever advises or aliases the user has set up that change what gets
>> compiled.  Another possibility are manually loaded packages that
>> misguide feature checks at compile time.  The old cl macros were a rich
>> source of potential problems if you weren't careful, but the advent of
>> cl-lib has made things much cleaner.
>
> I can think of many different hypothetical ways it can fail, indeed, but
> I'm wondering about actual cases.

Let's just say that a certain class of mysterious and usually
non-reproducible failures have stopped to be reported ever since I've
made sure that Org gets compiled and tested in a fresh and clean Emacs
instance.  We get such reports now and then from people who are trying
to use ELPA and our usual recommendation is to install the package from
"emacs -Q" which is a bit painful for someone who was hoping to have an
easier install method.

Now, Org probably wasn't what Tom had in mind when writing package.el,
but if more of Emacs' built-in packages get moved to ELPA (I don't know
if that is the plan), then there will be quite a few more such packages.

I think it's worth keeping the current simplicity of package.el for
those packages that don't need anything more complicated, so one thing
that comes to mind is to move these things to the package, maybe in the
form of callbacks.  If you look at org-reload it could very well be
factored out into a function that takes a few lists or regexes.  The
same should be possible for the advice to require.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Elpa packages and macro dependencies.

[Achim Gratz]

Stefan Monnier writes:
> Indeed.  I think your situation could also be handled correctly by my
> suggestion:
>
>    E.g. we could add to bytecomp.el the ability to force `require' to
>    reload a package if it's not already loaded from the file that
>    locate-library returns.

That doesn't work when defcustoms, autoloads and defvars change in the
new version.  I've been through this in my attempts to make Org more
resistant to such mishaps.  In a proper package system there was a way
to unload a package cleanly, then fly in the new package and plug it in.
We don't have that now, instead we have "packages" that really aren't
(because they are still built-in) and some infrastructure is missing to
unload and reload a complete package cleanly.

This is what is used in Org to get rid of any built-in stuff that might
be delivered with the Emacs before running batch tests:

--8<---------------cut here---------------start------------->8---
;;
;; Remove Org remnants built into Emacs
;;

;; clean load-path
(setq load-path
      (delq nil (mapcar
                 (function (lambda (p)
                             (unless (string-match "lisp\\(/packages\\)?/org$" p)
                               p)))
                 load-path)))
;; remove property list to defeat cus-load and remove autoloads
(mapatoms (function  (lambda (s)
                       (let ((sn (symbol-name s)))
                         (when (string-match "^\\(org\\|ob\\|ox\\)\\(-.*\\)?$" sn)
                           (setplist s nil)
                           (when (eq 'autoload (car-safe s))
                             (unintern s)))))))

;; we should now start from a clean slate
--8<---------------cut here---------------end--------------->8---

This has had little exposure beyond actual batch testing, but something
very close to this might actually work in most cases.  For bytecomp the
critical part would be to know which features and namespaces to nuke,
but if it's only used for packages the information could be delivered
with the package metadata perhaps.

Here's an experimental advice (not currently implemented in Org) to
recognize a changed load-path and re-load a feature in this case to take
care of some issues at runtime:

--8<---------------cut here---------------start------------->8---
;; some parts of Org might already have been used from a different
;; place, try to reload these parts from the current load-path
(require 'loadhist)
(defadvice require (before org-require-reload-when-shadowed
			   (feature &optional filename noerror)
			   activate compile preactivate)
  "Check whether a required feature has been shadowed by changing
`load-path' after it has been loaded and reload that feature from
current load-path in this case."
  (when (featurep feature)
    (let ((feature-name (or filename (symbol-name feature))))
      (when (string-match "^\\(org\\|ob\\)\\(-.*\\)?$" feature-name)
	(let ((feature-lib  (file-name-directory (or (locate-library feature-name) "")))
	      (feature-dir  (file-name-directory (feature-file feature))))
	  ;(message "require-reload-when shadowed %s\n\t%s\n\t%s" feature-name feature-lib feature-dir)
	  (when (not (string= feature-lib feature-dir))
	    (message "Reloading %s" feature-name)
	    (unload-feature feature t)))))))
--8<---------------cut here---------------end--------------->8---

Using just unload it doesn't catch some corner cases with defcustom and
autoloads as mentioned before so it would likely not be good enough for
byte-compiling, but for "close enough" versions in the load-path this
actually fixes up things without getting in the way.

And finally org-reload (as currently implemented in Org) that tries to
do the right thing or at least warn if the load-path is set up in
strange ways so that not all features from Org are found in the same
place:

--8<---------------cut here---------------start------------->8---
;;;###autoload
(defun org-reload (&optional uncompiled)
  "Reload all org lisp files.
With prefix arg UNCOMPILED, load the uncompiled versions."
  (interactive "P")
  (require 'loadhist)
  (let* ((org-dir     (org-find-library-dir "org"))
	 (contrib-dir (or (org-find-library-dir "org-contribdir") org-dir))
	 (feature-re "^\\(org\\|ob\\|ox\\)\\(-.*\\)?")
	 (remove-re (mapconcat 'identity
			       (mapcar (lambda (f) (concat "^" f "$"))
				       (list (if (featurep 'xemacs)
						 "org-colview"
					       "org-colview-xemacs")
					     "org" "org-loaddefs" "org-version"))
			       "\\|"))
	 (feats (delete-dups
		 (mapcar 'file-name-sans-extension
			 (mapcar 'file-name-nondirectory
				 (delq nil
				       (mapcar 'feature-file
					       features))))))
	 (lfeat (append
		 (sort
		  (setq feats
			(delq nil (mapcar
				   (lambda (f)
				     (if (and (string-match feature-re f)
					      (not (string-match remove-re f)))
					 f nil))
				   feats)))
		  'string-lessp)
		 (list "org-version" "org")))
	 (load-suffixes (when (boundp 'load-suffixes) load-suffixes))
	 (load-suffixes (if uncompiled (reverse load-suffixes) load-suffixes))
	 load-uncore load-misses)
    (setq load-misses
	  (delq 't
		(mapcar (lambda (f)
			  (or (org-load-noerror-mustsuffix (concat org-dir f))
			      (and (string= org-dir contrib-dir)
				   (org-load-noerror-mustsuffix (concat contrib-dir f)))
			      (and (org-load-noerror-mustsuffix (concat (org-find-library-dir f) f))
				   (add-to-list 'load-uncore f 'append)
				   't)
			      f))
			lfeat)))
    (if load-uncore
	(message "The following feature%s found in load-path, please check if that's correct:\n%s"
		 (if (> (length load-uncore) 1) "s were" " was") load-uncore))
    (if load-misses
	(message "Some error occurred while reloading Org feature%s\n%s\nPlease check *Messages*!\n%s"
		 (if (> (length load-misses) 1) "s" "") load-misses (org-version nil 'full))
      (message "Successfully reloaded Org\n%s" (org-version nil 'full)))))
--8<---------------cut here---------------end--------------->8---


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> That doesn't work when defcustoms, autoloads and defvars change in the
> new version.

Good point.  Indeed defvars and defcustoms can be problematic.

OTOH I can't see why autoloads would be problematic.  Do you happen to
remember of a concrete case, or at least the general shape of
a concrete case?


        Stefan

Re: Elpa packages and macro dependencies.

[Achim Gratz]

Stefan Monnier writes:
>> That doesn't work when defcustoms, autoloads and defvars change in the
>> new version.
>
> Good point.  Indeed defvars and defcustoms can be problematic.
>
> OTOH I can't see why autoloads would be problematic.  Do you happen to
> remember of a concrete case, or at least the general shape of
> a concrete case?

Again, this in particular is only really a problem for built-in packages
since there's no way to make Emacs completely forget about the built-in
package when you happen to install a newer version of it via ELPA later
on.  When an autoloaded function is moved into a different file in the
new version, the new autoload definition is ignored (because there
already is one) and if you try to invoke the function the wrong file
gets loaded so it doesn't work (the original autoload definition is
always read during Emacs' startup).  It's admittedly a rare occurence,
but if you split a single-file package into multiple files or if you
were to do some refactoring (like the Org 7.x->8.x transistion) it can
bite you.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Elpa packages and macro dependencies.

[Phillip Lord]

Achim Gratz <Stromeko@nexgo.de> writes:
> Again, this in particular is only really a problem for built-in packages
> since there's no way to make Emacs completely forget about the built-in
> package when you happen to install a newer version of it via ELPA later
> on.  When an autoloaded function is moved into a different file in the
> new version, the new autoload definition is ignored (because there
> already is one) and if you try to invoke the function the wrong file
> gets loaded so it doesn't work (the original autoload definition is
> always read during Emacs' startup).  It's admittedly a rare occurence,
> but if you split a single-file package into multiple files or if you
> were to do some refactoring (like the Org 7.x->8.x transistion) it can
> bite you.


I've been bitten by this one.

Would it not be possible to have package.el check when it installs
packages from ~/.emacs.d if that package is also in Emacs core and then
remove the relevant paths from load-path.

So if I have in my .emacs

(require 'org)
(require 'org-latex) 

would work iff org is only in core. But would crash with "cannot open
load file" if org is loaded from ELPA. If it doesn't crash in my .emacs
I am going to get strange errors later anyway, from loading two versions
of org.

All of this could be calculated when a new package is installed, it
seems to me.

Phil

Re: Elpa packages and macro dependencies.

[Tom Tromey]

>>>>> "Phillip" == Phillip Lord <phillip.lord@newcastle.ac.uk> writes:

Phillip> Would it not be possible to have package.el check when it
Phillip> installs packages from ~/.emacs.d if that package is also in
Phillip> Emacs core and then remove the relevant paths from load-path.

FWIW my original intent was that packages in this category (both
distributed with emacs and separately as well) would be managed in the
Emacs install tree using package.el, following the usual activation
approach.

I never implemented the needed changes on the Emacs side.  I don't think
anybody else did either.

And while this would be an improvement (IMO) it also carries a cost --
you can't refer to these packages before package-initialize is called.
So, there's some startup ordering trickiness.

Also this doesn't solve all problems.  Upgrading packages will be
troublesome as long as there's no way to unload a package.  Though
perhaps async compilation plus offering to restart Emacs would be good
enough.

Tom

Re: Elpa packages and macro dependencies.

[Stefan Monnier]

> Again, this in particular is only really a problem for built-in packages
> since there's no way to make Emacs completely forget about the built-in
> package when you happen to install a newer version of it via ELPA later
> on.  When an autoloaded function is moved into a different file in the
> new version, the new autoload definition is ignored (because there
> already is one) and if you try to invoke the function the wrong file
> gets loaded so it doesn't work (the original autoload definition is
> always read during Emacs' startup).

That doesn't sound right:

    ELISP> (autoload 'sm-foo "toto")
    (autoload "toto" nil nil nil)
    
    ELISP> (autoload 'sm-foo "titi")
    (autoload "titi" nil nil nil)
    
    ELISP> (symbol-function 'sm-foo)
    (autoload "titi" nil nil nil)
    
    ELISP> 

Clearly, the second autoload overrides the first.

I think that the problem only shows up if your autoloaded functions has
already been loaded, in which case `autoload' doesn't do anything.


        Stefan

Re: Elpa packages and macro dependencies.

[bruce.connor.am]

Achim Gratz:
> That doesn't work when defcustoms, autoloads and defvars change in the
> new version.

It would be good to fix autoloads indeed, but you can't force-reload
defcustoms, or you'll be erasing user customizations.
You could reload them only if its value is still the standard value,
but then users who has customized it would still run into the same
problem.

That said, IIUC, defcustoms and defvars aren't really a huge problem.
They're only problematic if the package author changes something in a
backwards-incompatible way (thus making an old defcustom not work with
new code). And even then, things should resume normality after a restart.


Or am I missing something?


The problem with macros is that they break installation of packages,
and it happens even if all package authors respect backwards
compatibility.