PACKAGE-FEATURES, and hot update of Emacs packages

PACKAGE-FEATURES, and hot update of Emacs packages

[Qiantan Hong]

package-reinstall seems to update the external UNIX file system only,
and don’t update the package loaded in the Elisp image properly.
I didn’t find a built-in way to do so, which sucks because one need
to restart Emacs (and lost the image state) to update packages.

It turns out that Emacs seems to have almost all the necessary facility
to make hot update happens, the only missing one
seems to be package-features (that finds out features loaded from the package).
Then to really update p, one just need to

(mapc (lambda (feature) (unload-feature feature t)) (package-features p))
(package-reinstall p)
(require p)

(assuming p is the same as the “entry” feature of the whole package)

Currently I use an adhoc version
(defun package-features (p)
  (cl-remove-if-not
   (lambda (feature)
     (s-prefix-p (symbol-name p) (symbol-name feature)))
   features))

It works pretty well, I’m able to update all of my everyday packages successfully,
which shows that hot update is very doable. Now for the rare case
that the default behavior result in inconsistent state, package maintainer
can always define feature-UNLOAD-FUNCTION to make it work nicely.

Now back to the only missing pieces, how much effort it’d take to have
a proper PACKAGE-FEATURES? Is it good to have it in core? 
(I suppose it’d be a minor change to package.el).

Best,
Qiantan

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Eli Zaretskii]

> From: Qiantan Hong <qthong@stanford.edu>
> Date: Sat, 27 Nov 2021 09:31:10 +0000
> 
> Then to really update p, one just need to
> 
> (mapc (lambda (feature) (unload-feature feature t)) (package-features p))
> (package-reinstall p)
> (require p)

Is it really 100% safe to unload features, in the sense that reloading
the package will then behave as if Emacs was restarted?

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Qiantan Hong]

> Is it really 100% safe to unload features, in the sense that reloading
> the package will then behave as if Emacs was restarted?
In my experience, in most cases default behavior does the right thing,
and as I have commented
> Now for the rare case
> that the default behavior result in inconsistent state, package maintainer
> can always define feature-UNLOAD-FUNCTION to make it work nicely.


And I think having default behavior to be safe in many cases is already
very notable. AFAIK in UNIX almost every package maintainer need to write
some script manually, so it’s pretty reasonable to shift the responsibility
of “100% safe” to package maintainers.

Best,
Qiantan

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Qiantan Hong]

Maybe we can also have a :unloadable option so that package authors
can declare that the package is tested to be 100% safe unloadable,
and we give a warning when attempting to unload package without the flag.

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Phil Sainty]

On 2021-11-27 22:31, Qiantan Hong wrote:
> Then to really update p, one just need to
> 
> (mapc (lambda (feature) (unload-feature feature t)) (package-features 
> p))
> (package-reinstall p)
> (require p)

Unloading a feature will trash the user's configuration, and so when
you load the updated library only the default config will be set --
unless the user used `eval-after-load' (which certainly can't be
assumed).  As such, I don't think that could work as a reliable "hot
update" process.

My approach to this general issue (in libraries I've written) has been
to detect at load time if the user had an older version of the library
loaded, and to perform any appropriate in-place upgrades.  E.g.:
https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/so-long.el?id=aebba085cba1#n2012

This does require that the code knows both the previously-loaded and
newly-loaded version numbers, which can be tricky if the previous
version of the code wasn't storing its version; so I think it would be
great if Emacs *automatically* tracked this information for packages
(or any library that supplies a version) so that authors could write
such update code without needing to have previously prepared for it.

I don't think this currently happens.  I can see the functions
`package-get-version' and `package-desc-version' (which can be used
with `package-alist'), but haven't spotted anything which is storing
the version of a package that is/was loaded (and I don't think this
information is as easily accessible when loading byte-compiled files,
as version comments don't appear in the .elc file).

I suppose (package-desc-version (package-load-descriptor DIR)) can be
used for ELPA packages installed in the standard way, so maybe there's
scope for using that; but I suspect that'd be too brittle an approach,
as there are many non-standard ways of loading things.


-Phil

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Phil Sainty]

On 2021-11-28 01:16, Phil Sainty wrote:
> I suppose (package-desc-version (package-load-descriptor DIR)) can be
> used for ELPA packages installed in the standard way, so maybe there's
> scope for using that; but I suspect that'd be too brittle an approach,
> as there are many non-standard ways of loading things.

I phrased that final bit very confusingly.

I just meant that a file which is intended to be installed as an
ELPA package might actually be loaded directly from some non-ELPA
source, or even evaluated via `eval-buffer', and so an approach
which depended on a *-pkg.el being present in the same directory
could fail in those situations, causing inconsistent behaviour.

-Phil

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Qiantan Hong]

> Unloading a feature will trash the user's configuration, and so when
> you load the updated library only the default config will be set --
> unless the user used `eval-after-load' (which certainly can't be
> assumed).  As such, I don't think that could work as a reliable "hot
> update" process.
I think if the user doesn’t use eval-after-load
(which I assumed is actually used a lot because the popularity
of lazy-loading in init.el) and instead do 
(require f) …configs…
eagerly, that makes the init feature depends on feature f.
To have consistent behavior we just need to also reload
all dependents of the package.

> My approach to this general issue (in libraries I've written) has been
> to detect at load time if the user had an older version of the library
> loaded, and to perform any appropriate in-place upgrades.  E.g.:
> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/so-long.el?id=aebba085cba1#n2012
This require the user to reload the file.
Currently if the old package is not unloaded, the feature is still
present and (require p) does nothing.
It might be trivial to do so for a single-file/feature package,
but less so for a multi-file or even directory-nested one.
Is there a reasonable way to do it?

It seems that an adhoc way would be to remove 
(package-features p) from features without actually unloading,
and then (require p). It also requires PACKAGE-FEATURES!

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Tim Cross]

Phil Sainty <psainty@orcon.net.nz> writes:

> On 2021-11-27 22:31, Qiantan Hong wrote:
>> Then to really update p, one just need to
>> (mapc (lambda (feature) (unload-feature feature t)) (package-features p))
>> (package-reinstall p)
>> (require p)
>
> Unloading a feature will trash the user's configuration, and so when
> you load the updated library only the default config will be set --
> unless the user used `eval-after-load' (which certainly can't be
> assumed).  As such, I don't think that could work as a reliable "hot
> update" process.
>
> My approach to this general issue (in libraries I've written) has been
> to detect at load time if the user had an older version of the library
> loaded, and to perform any appropriate in-place upgrades.  E.g.:
> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/so-long.el?id=aebba085cba1#n2012
>
> This does require that the code knows both the previously-loaded and
> newly-loaded version numbers, which can be tricky if the previous
> version of the code wasn't storing its version; so I think it would be
> great if Emacs *automatically* tracked this information for packages
> (or any library that supplies a version) so that authors could write
> such update code without needing to have previously prepared for it.
>
> I don't think this currently happens.  I can see the functions
> `package-get-version' and `package-desc-version' (which can be used
> with `package-alist'), but haven't spotted anything which is storing
> the version of a package that is/was loaded (and I don't think this
> information is as easily accessible when loading byte-compiled files,
> as version comments don't appear in the .elc file).
>
> I suppose (package-desc-version (package-load-descriptor DIR)) can be
> used for ELPA packages installed in the standard way, so maybe there's
> scope for using that; but I suspect that'd be too brittle an approach,
> as there are many non-standard ways of loading things.
>
>

While I like the idea of being able to use package version to help
manage this, the problem is there is no standardisation of package
version formats. This makes calculation of which version is before/after
another version unreliable.

Personally, I wish the package systems used a consistent semantic
versioning approach. In addition to determining which package version is
later than another version, it would also provide the ability to have
more fine grain control over applying updates i.e. could set the system
to automatically apply minor patch updates, but not major version
updates that may contain breaking changes. 

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Phil Sainty]

On 2021-11-28 14:04, Tim Cross wrote:
> While I like the idea of being able to use package version to help
> manage this, the problem is there is no standardisation of package
> version formats. This makes calculation of which version is 
> before/after
> another version unreliable.

Version comparisons are very well defined; but I guess you're referring
to MELPA's horribly-broken approach to versions here?  A test with a
package installed from MELPA confirms that `package-desc-version' and
`package-get-version' do pick up on whatever version MELPA assigned, so
I guess that's unreliable as you say.  Along with my earlier misgivings
about assuming that any given library had been installed as a package in
the first place, I agree that this doesn't seem like a viable approach.

The code itself would include the *real* version number, though, so in
principle I think it would be possible to utilise that; but I think it's
currently only exposed in the .el source.

If the .elc file contained the version number as well, I think that
load-time version-update tracking would become possible?


-Phil

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Stefan Monnier]

> While I like the idea of being able to use package version to help
> manage this, the problem is there is no standardisation of package
> version formats. This makes calculation of which version is before/after
> another version unreliable.

FWIW, `package.el` does impose some convention for version
representation (basically, whatever can be parsed by `version-to-list`)
which provides a proper ordering between versions.

[ I don't have much to add to the rest of the discussion, sorry.  ]


        Stefan

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Phil Sainty]

On 2021-11-28 09:36, Qiantan Hong wrote:
> This require the user to reload the file.

Yes indeed.  Or rather, it *supports* them reloading the file.

> Currently if the old package is not unloaded, the feature is still
> present and (require p) does nothing.

Certainly `require' won't do anything, but nothing prevents you from
using `load-library' for for the same list of libraries that you were
planning to unload.

You can't be certain that's going to be safe, though.  Reloading a
library is expected to be safe when the code is unchanged, but there's
no requirement that I'm aware of to support loading new code over the
old code -- that will often be fine in practice, but there's loads of
scope for it not to be.

And as before, I don't think it's valid to unload things (even if you
could be confident that everything supported being unloaded) because
you destroy the configuration.

I can't think of a way to reliably do what you're suggesting in the
general case.  Each approach will work in certain circumstances, but
I don't think any approach is *generally* reliable.

Libraries could probably declare themselves as supporting reloading
if the author believed that to be the case, and you could have an
automated reload sequence if everything claimed to support it.


-Phil

Re: PACKAGE-FEATURES, and hot update of Emacs packages

[Qiantan Hong]

>> Currently if the old package is not unloaded, the feature is still
>> present and (require p) does nothing.
> 
> Certainly `require' won't do anything, but nothing prevents you from
> using `load-library' for for the same list of libraries that you were
> planning to unload.
For a multi-file package, does (load-library entry-feature) also
loads the other files? I suppose it doesn’t because usually
the entry-feature use REQUIRE instead of LOAD-LIBRARY
to load its “components”.

An adhoc solution would be to temporarily override the definition
of REQUIRE that perform LOAD-LIBRARY if the feature to be
required is part of the package. It seems that we still need
PACKAGE-FEATURES to test it… still, it sounds super ad-hoc,
not sure how bad an idea it is.

Or maybe we can upgrade REQUIRE itself so that it reloads
when a feature file is modified?

> You can't be certain that's going to be safe, though.  Reloading a
> library is expected to be safe when the code is unchanged, but there's
> no requirement that I'm aware of to support loading new code over the
> old code -- that will often be fine in practice, but there's loads of
> scope for it not to be.
Your so-long library seems to have provided an excellent example
of supporting it, so I suppose it’s very doable in general.

> And as before, I don't think it's valid to unload things (even if you
> could be confident that everything supported being unloaded) because
> you destroy the configuration.
> 
> I can't think of a way to reliably do what you're suggesting in the
> general case.  Each approach will work in certain circumstances, but
> I don't think any approach is *generally* reliable.
As I suggested, reloading all packages/features dependent on a particular
package seems to do the trick, at least it will recover the init configs.
The conceptual model is also very clear — the operation basically
recover a part of the image state from external file system, 
sort of like loading an incremental image.
It might also be useful when the user corrupted the state of a particular
package and want to recover it.
In case that such recursive reloading might result in reloading too many
packages, we might provide a switch for recursive/non-recursive reloading,
and present a list of packages to be reloaded for the user to confirm.

Maybe we can have both :unloadable and :reloadable,
reload should be used in possible, and unload-then-load
can be used as a fallback or rescuing operation.


Best,
Qiantan