Updating Elisp files while Emacs is running

Updating Elisp files while Emacs is running

[Philipp Stephani]

Hi,

OS-level package managers such as Debian's dpkg generally replace files
unconditionally when upgrading. Typically this is not a problem because
running binaries will continue to use the in-memory image, and daemons can
get restarted or sent a signal without user interaction. However, for Emacs
the situation seems different: Emacs not only needs its (dumped) binary,
but also relies on Elisp files. For Elisp files already loaded when the
upgrade is running this typically doesn't cause problems, but when Elisp
files aren't already loaded, replacing them causes problems because the
directories containing them are versioned (see e.g. lispdir in configure.ac),
so that when the version changes (even for minor changes, such as updating
from the recent pretest-1 to pretest-2), the files are no longer at the
expected location, and libraries can't be loaded any more, either manually
or via autoloads. Similar things happen for system-wide ELPA packages (in
the package-directory-list directories), because the names of these
directories are also versioned. How could this be changed so that
unattended upgrades changing the Emacs version are possible while Emacs is
running? Could the Elisp file directories get names that don't include the
version number (maybe hidden behind a configure option)? Is it possible to
install system-wide ELPA packages into directories that don't contain
version numbers? Or is there another way, e.g. using a signal to
reinitialize the load-path and the package system? I'm aware these
suggestions would still cause issues if e.g. Elisp files get moved around
or the byte code format changes, but that seems to be comparatively rare.

Thanks,
Philipp

Re: Updating Elisp files while Emacs is running

[Phillip Lord]

Philipp Stephani <p.stephani2@gmail.com> writes:
> OS-level package managers such as Debian's dpkg generally replace files
> unconditionally when upgrading. Typically this is not a problem because
> running binaries will continue to use the in-memory image, and daemons can
> get restarted or sent a signal without user interaction. However, for Emacs
> the situation seems different: Emacs not only needs its (dumped) binary,
> but also relies on Elisp files. For Elisp files already loaded when the
> upgrade is running this typically doesn't cause problems, but when Elisp
> files aren't already loaded, replacing them causes problems because the
> directories containing them are versioned (see e.g. lispdir in configure.ac),
> so that when the version changes (even for minor changes, such as updating
> from the recent pretest-1 to pretest-2), the files are no longer at the
> expected location, and libraries can't be loaded any more, either manually
> or via autoloads. Similar things happen for system-wide ELPA packages (in
> the package-directory-list directories), because the names of these
> directories are also versioned. How could this be changed so that
> unattended upgrades changing the Emacs version are possible while Emacs is
> running? Could the Elisp file directories get names that don't include the
> version number (maybe hidden behind a configure option)? Is it possible to
> install system-wide ELPA packages into directories that don't contain
> version numbers? Or is there another way, e.g. using a signal to
> reinitialize the load-path and the package system? I'm aware these
> suggestions would still cause issues if e.g. Elisp files get moved around
> or the byte code format changes, but that seems to be comparatively rare.


I am not sure I see the issue. When updates to the core occur, then the
version number changes, so new files will not overwrite existing ones,
nor will the load-path be changed.

Likewise, for ELPA, as far as I know -- the load-path is set once during
package-initialize, so if a new version of a package gets added, Emacs
will run with the old.

If you want an existing Emacs to move to the new packages, that would be
hard to get right; only a restart is going to make sense here.

Why is that a problem?

Re: Updating Elisp files while Emacs is running

[Philipp Stephani]

Phillip Lord <phillip.lord@russet.org.uk> schrieb am Mo., 14. März 2016 um
15:48 Uhr:

>
>
>
>
> Philipp Stephani <p.stephani2@gmail.com> writes:
> > OS-level package managers such as Debian's dpkg generally replace files
> > unconditionally when upgrading. Typically this is not a problem because
> > running binaries will continue to use the in-memory image, and daemons
> can
> > get restarted or sent a signal without user interaction. However, for
> Emacs
> > the situation seems different: Emacs not only needs its (dumped) binary,
> > but also relies on Elisp files. For Elisp files already loaded when the
> > upgrade is running this typically doesn't cause problems, but when Elisp
> > files aren't already loaded, replacing them causes problems because the
> > directories containing them are versioned (see e.g. lispdir in
> configure.ac),
> > so that when the version changes (even for minor changes, such as
> updating
> > from the recent pretest-1 to pretest-2), the files are no longer at the
> > expected location, and libraries can't be loaded any more, either
> manually
> > or via autoloads. Similar things happen for system-wide ELPA packages (in
> > the package-directory-list directories), because the names of these
> > directories are also versioned. How could this be changed so that
> > unattended upgrades changing the Emacs version are possible while Emacs
> is
> > running? Could the Elisp file directories get names that don't include
> the
> > version number (maybe hidden behind a configure option)? Is it possible
> to
> > install system-wide ELPA packages into directories that don't contain
> > version numbers? Or is there another way, e.g. using a signal to
> > reinitialize the load-path and the package system? I'm aware these
> > suggestions would still cause issues if e.g. Elisp files get moved around
> > or the byte code format changes, but that seems to be comparatively rare.
>
>
> I am not sure I see the issue. When updates to the core occur, then the
> version number changes, so new files will not overwrite existing ones,
> nor will the load-path be changed.
>

That is exactly the problem. Either the version number shouldn't change, or
the load-path needs to be updated. The old files are gone, they can't be
accessed any more after the upgrade.


>
> Likewise, for ELPA, as far as I know -- the load-path is set once during
> package-initialize, so if a new version of a package gets added, Emacs
> will run with the old.
>

Not if the files of the old package have been removed.


>
> If you want an existing Emacs to move to the new packages, that would be
> hard to get right; only a restart is going to make sense here.
>
> Why is that a problem?
>

Because Emacs expects the old files to be there, but they have been
removed. Sorry for being unclear: the old files get removed during the
upgrade.

Re: Updating Elisp files while Emacs is running

[Phillip Lord]

Philipp Stephani <p.stephani2@gmail.com> writes:
>> If you want an existing Emacs to move to the new packages, that would be
>> hard to get right; only a restart is going to make sense here.
>>
>> Why is that a problem?
>>
>
> Because Emacs expects the old files to be there, but they have been
> removed. Sorry for being unclear: the old files get removed during the
> upgrade.

You want to load lisp files from two version of Emacs into the same
Emacs? This is not going to end well, I fear.


Phil

Re: Updating Elisp files while Emacs is running

[Philipp Stephani]

Phillip Lord <phillip.lord@russet.org.uk> schrieb am Mo., 14. März 2016 um
19:13 Uhr:

> Philipp Stephani <p.stephani2@gmail.com> writes:
> >> If you want an existing Emacs to move to the new packages, that would be
> >> hard to get right; only a restart is going to make sense here.
> >>
> >> Why is that a problem?
> >>
> >
> > Because Emacs expects the old files to be there, but they have been
> > removed. Sorry for being unclear: the old files get removed during the
> > upgrade.
>
> You want to load lisp files from two version of Emacs into the same
> Emacs? This is not going to end well, I fear.
>
>
Better than not loading anything at all though. It gives users the chance
to perform an orderly shutdown of Emacs. With the files absent, often even
kill-emacs fails because there's something autoloaded in kill-emacs-hook
that can't be loaded any more. The intention is not to keep instances with
inconsistent Elisp libraries loaded for a long time, it's more to not
immediately crash and burn and allow the user to finish they work before
restarting.

Re: Updating Elisp files while Emacs is running

[Bob Proulx]

Philipp Stephani wrote:
> Phillip Lord schrieb:
> > I am not sure I see the issue. When updates to the core occur, then the
> > version number changes, so new files will not overwrite existing ones,
> > nor will the load-path be changed.
> 
> That is exactly the problem. Either the version number shouldn't change, or
> the load-path needs to be updated. The old files are gone, they can't be
> accessed any more after the upgrade.

A little more detail here would help.  I am guessing that it wasn't a
packaged emacs upgrade that has caused you problems but instead you
removed a version of emacs while that version of emacs was still
running and that removal caused your problem.

Multiple versions may be installed side by side without issues.
Upgrades can install newer versions but will not remove older
versions.  That way programs that are still running will continue to
access the associated files for that specific version.

> > If you want an existing Emacs to move to the new packages, that would be
> > hard to get right; only a restart is going to make sense here.
> >
> > Why is that a problem?
> 
> Because Emacs expects the old files to be there, but they have been
> removed. Sorry for being unclear: the old files get removed during the
> upgrade.

An package upgrade will not remove previous versions.  However if you
also told it to remove as well then that removal might be explicitly done.

  $ apt-cache show emacs | grep Depends:
  Depends: emacs24 | emacs24-lucid | emacs24-nox

Previously:

  Depends: emacs23 | emacs23-lucid | emacs23-nox

In Debian an 'upgrade' of the system will not be able to upgrade from
one packaged version of emacs to the next because doing so requires
adding the new package version and 'upgrade' does not allow any
changes to the installed package list, just upgrades of what is
already installed.  I always perform an 'upgrade' before running
'dist-upgrade'.  A 'dist-upgrade' is required to allow the
installation or removal of different packages.  On a dist-upgrade the
emacs package is upgraded with the new Depends: which pulls in the new
version of emacs$VERSION.  The previous emacs$OLDVERSION is left
behind on the system.  Running emacs processes will continue to use
the old files.  Any new invocations will launch the new version.

Now here is what I think may have happened.  If subsequently an
'apt-get autoremove' (I always use 'apt-get autoremove --purge' along
with version control of /etc using the 'etckeeper' package) then if
nothing else Depends: upon 'emacs$OLDVERSION' and it was marked as
automatically installed as a dependency then it will be removed
because nothing else installed depends upon it.  (If you manually
'apt-get install emacs23' then that package will be marked as manually
installed and will never be a candidate for autoremoval.)  Using
autoremove after a dist-upgrade is a good way to clean lint from your
system.  Otherwise old versions of packages pile up and you find you
have emacs versions 23, 22, 21, and so forth still hanging around on
your system.  I used to be there before I started routinely using
autoremove.

As a shortcut on newer apt-get versions you can use the option
'apt-get install --auto-remove' to combine these operations all at one
time.  If this is done then the old package will be removed at the
same time.  The old package will be listed in the review list asking
for confirmation before performing the action.  The expectation is
that when reviewing the list of packages for removal you will see that
it is removing something that you are still using and choose not to
remove it.  If something is listed as a candidate for autoremoval
simply mark it as manually installed.  'apt-get install emacs23' is
sufficient.  There is also the 'apt-mark' command to list and
explicitly manage this.  Then repeate the autoremove again, see that
the packages you don't want removed are no longer listed as
candidates, and confirm the removal of the now shorter list.

Did I get close? :-)

Bob

P.S. I know the current upstream emacs version is 25.  But I am
showing things as they are packaged today.  Which is why I tried to
make it somewhat generic with emacs$VERSION and emacs$OLDVERSION.

Re: Updating Elisp files while Emacs is running

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Mon, 14 Mar 2016 13:41:17 +0000
> 
> Could the Elisp file directories get names that don't include the
> version number (maybe hidden behind a configure option)? Is it possible to
> install system-wide ELPA packages into directories that don't contain
> version numbers? Or is there another way, e.g. using a signal to
> reinitialize the load-path and the package system? I'm aware these
> suggestions would still cause issues if e.g. Elisp files get moved around
> or the byte code format changes, but that seems to be comparatively rare.

The (unversioned) directory that Emacs provides for these purposes is
site-lisp.

Re: Updating Elisp files while Emacs is running

[Philipp Stephani]

Eli Zaretskii <eliz@gnu.org> schrieb am Mo., 14. März 2016 um 17:49 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Mon, 14 Mar 2016 13:41:17 +0000
> >
> > Could the Elisp file directories get names that don't include the
> > version number (maybe hidden behind a configure option)? Is it possible
> to
> > install system-wide ELPA packages into directories that don't contain
> > version numbers? Or is there another way, e.g. using a signal to
> > reinitialize the load-path and the package system? I'm aware these
> > suggestions would still cause issues if e.g. Elisp files get moved around
> > or the byte code format changes, but that seems to be comparatively rare.
>
> The (unversioned) directory that Emacs provides for these purposes is
> site-lisp.
>
>
But that doesn't contain files that are part of Emacs, and ELPA packages
are still in versioned directories, no?

Re: Updating Elisp files while Emacs is running

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Mon, 14 Mar 2016 17:22:36 +0000
> 
>  The (unversioned) directory that Emacs provides for these purposes is
>  site-lisp.
> 
> But that doesn't contain files that are part of Emacs, and ELPA packages are still in versioned directories, no?

AFAIK, if you put files into site-lisp, they will shadow those in the
versioned lisp directory (because site-lisp is earlier on load-path).
I hope the same happens with ELPA, but I don't really know.

Re: Updating Elisp files while Emacs is running

[Philipp Stephani]

Eli Zaretskii <eliz@gnu.org> schrieb am Mo., 14. März 2016 um 19:05 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Mon, 14 Mar 2016 17:22:36 +0000
> >
> >  The (unversioned) directory that Emacs provides for these purposes is
> >  site-lisp.
> >
> > But that doesn't contain files that are part of Emacs, and ELPA packages
> are still in versioned directories, no?
>
> AFAIK, if you put files into site-lisp, they will shadow those in the
> versioned lisp directory (because site-lisp is earlier on load-path).
>

Or I could just overwrite the paths in configure.ac to not include a
version number. What I'd suggest would be to introduce a configure option
to do just this. Or are there any problems with changing these paths?


> I hope the same happens with ELPA, but I don't really know.
>
>
The problem with ELPA is different: the subdirectories of the package
directories are documented to always include the version number (
https://www.gnu.org/software/emacs/manual/html_node/emacs/Package-Files.html).
From looking at the code I don't think that's actually a hard requirement,
but I'd rather not rely on undocumented behavior.

Re: Updating Elisp files while Emacs is running

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Mon, 14 Mar 2016 18:44:49 +0000
> 
>  AFAIK, if you put files into site-lisp, they will shadow those in the
>  versioned lisp directory (because site-lisp is earlier on load-path).
> 
> Or I could just overwrite the paths in configure.ac to not include a version number. What I'd suggest would be
> to introduce a configure option to do just this.

What for?  We already have environment variables that you can use to
do the same without changing the build.

> Or are there any problems with changing these paths?

I don't understand why you don't want to use site-lisp, which was
invented for that very purpose (among other things).  But that's me.

>  I hope the same happens with ELPA, but I don't really know.
> 
> The problem with ELPA is different: the subdirectories of the package directories are documented to always
> include the version number
> (https://www.gnu.org/software/emacs/manual/html_node/emacs/Package-Files.html). From looking at the
> code I don't think that's actually a hard requirement, but I'd rather not rely on undocumented behavior. 

Then I guess there's something missing in how ELPA packages are
deployed.

Re: Updating Elisp files while Emacs is running

[Joost Kremers]

Phillip Lord wrote:
> Likewise, for ELPA, as far as I know -- the load-path is set once during
> package-initialize, so if a new version of a package gets added, Emacs
> will run with the old.

Except that its directory will have been deleted. This means that if the
package is autoloaded (which is common) and you haven't actually *used*
(i.e., loaded) the package before upgrading, then when you do use it,
Emacs will look in the old location, won't find the package and throw up
an error.



-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)

Re: Updating Elisp files while Emacs is running

[Stefan Monnier]

> OS-level package managers such as Debian's dpkg generally replace files
> unconditionally when upgrading.

I think we'd be better off discussing it with concrete cases at hand.
Let's try a few cases:
- You have Emacs-24.5 running and upgrade to Emacs-25.1.  In this case,
  making your Emacs process load files from the 25.1 version is risky,
  because there can be (and there are) incompatibilities.  You might
  still want to do it, since the incompatibilities won't always bite
  you, and it can be better than failing right away.

  Note that the files have to be elsewhere, because Debian lets you have
  both Emacs-24.5 and Emacs-25.1 installed at the same time.

- You have Emacs-24.4 running and upgrade to Emacs-24.5.  In this case,
  there should be very few incompatibilities, so it might make sense to
  try and redirect the running process to use the new 24.5 files.

  One possible solution here is for the Debian package to use
  directory names like /usr/share/emacs/24/... instead of
  /usr/share/emacs/24.4/... since Debian doesn't let you have both 24.4
  and 24.5 installed at the same time (contrary to the default Emacs
  build's presumption).

- You have ELPA package sm-c-mode-3.4 installed (and an Emacs is
  running with its load-path pointing to it) and upgrade it to
  sm-c-mode-4.0.  This can happen if you upgrade from a separate process.
  Here, there are 2 subcases:
  - you've already used sm-c-mode in this process, so most/all the files
    are already loaded.  In that case you should be fine and there's
    most likely nothing to do.
  - you haven't used sm-c-mode yet in this process, so it would be both
    desirable and likely safe to reload the new sm-c-mode-autoloads.el
    so as to redirect future uses of sm-c-mode to load from the new
    4.0 version.

I'm not sure how best to handle this, but maybe one way to accommodate
this is to add some hook to `load' to handle the situation of
a directory having gone missing.

E.g. one easy/safe hook could be a load-file-not-found-hook called when
load fails to find the file, so you could add ad-hoc code to it that
tries to detect the above situations, updates the load-path accordingly,
and tries again.

A slightly more intrusive attempt might be
a load-path-directory-not-found-hook called when a dir in load-path
doesn't exist.  This would try to solve the main problem I see with the
previous hook: when /usr/share/emacs/24.4 is missing, Emacs may
erroneously find some old/unrelated file with the same name further down
the load-path and would hence fail to run load-file-not-found-hook.
But the problem with load-path-directory-not-found-hook would then be
that it's currently considered perfectly normal for load-path to contain
non-existing directories, so it could break some setups if not done
carefully enough.


        Stefan

Re: Updating Elisp files while Emacs is running

[Bob Proulx]

Stefan Monnier wrote:
> > OS-level package managers such as Debian's dpkg generally replace files
> > unconditionally when upgrading.

But the Debian packages generally install new files when upgrading and
leave the old files behind.  (As I said in much more detail in my
other posting.)

> I think we'd be better off discussing it with concrete cases at hand.

Agreed.

> Let's try a few cases:
> - You have Emacs-24.5 running and upgrade to Emacs-25.1.  In this case,
>   making your Emacs process load files from the 25.1 version is risky,
>   because there can be (and there are) incompatibilities.  You might
>   still want to do it, since the incompatibilities won't always bite
>   you, and it can be better than failing right away.
> 
>   Note that the files have to be elsewhere, because Debian lets you have
>   both Emacs-24.5 and Emacs-25.1 installed at the same time.

The only way this case can occur is if the user removes emacs24 after
having installed emacs25.  But they shouldn't remove emacs24 if they
are still using it.  I think this is a local user error.  The default
actions are okay.

> - You have Emacs-24.4 running and upgrade to Emacs-24.5.  In this case,
>   there should be very few incompatibilities, so it might make sense to
>   try and redirect the running process to use the new 24.5 files.
>
>   One possible solution here is for the Debian package to use
>   directory names like /usr/share/emacs/24/... instead of
>   /usr/share/emacs/24.4/... since Debian doesn't let you have both 24.4
>   and 24.5 installed at the same time (contrary to the default Emacs
>   build's presumption).

This would only come up with upgrading from one distribution release
to another.  At which time a reboot is needed to boot the new kernel.
Because of the release schedules for both Emacs and distributions such
as Debian this is an unlikely case as Emacs will almost always have
rolled to a new major version.

Users of the daily bleeding edge Unstable or the rolling release
candidate Testing are expected to know a little more about how things
work in order to administer their system.  Being the daily bleeding
edge bits Unstable is not for the casual user and there is often much
breakage.  Even the rolling testing candidate Testing goes through
transitions which requires advanced knowledge of the system to unbreak
it at times.  Life is a compromise.  The upgrades that are very well
tested are the release upgrades.  But users want early access to the
development bits which are less well polished.  There aren't enough
volunteer resources to polish every package every day to release
quality.  You could search for CUT (Constantly Usable Testing) for a
proposal to deliver it but it has not yet been implemented.

> - You have ELPA package sm-c-mode-3.4 installed (and an Emacs is
>...

ELPA may be problematic and a place where improvements could be made
but the original poster indicated they were using Debian packages.  In
which case ELPA wasn't the fault.

Bob