Automatically saving package sources

Automatically saving package sources

[sbaugh]

It is nice for installed ELPA packages to be reinstalled when Emacs is
upgraded or changed, since there can be changes to autoload generation,
the byte compiler, the native compiler, or other things which will
change the output from the package installation.

Furthermore, it is nice (where possible) for this reinstallation to
happen with the same version of the package rather than by fetching the
latest version from ELPA.  This minimizes the number of changes that
happen at once.

However, right now there is no easy way to reinstall a package from the
same sources it was originally installed from, because those sources are
not saved separately from the package itself.  They are all mixed
together in the installed package directory.  This makes it impossible
to produce a "clean slate" environment for reinstalling the package,
which (since packages can create arbitrary files during
byte-compilation) means the package can't be reinstalled in a way that's
identical to its original installation.

If package.el supported saving the package source separately, however,
it could simply reinstall from that.

I would like to add an option package-save-sources which makes
package.el automatically save the package source for the currently
installed package version in a separate directory.  Perhaps in
~/.emacs.d/elpa-sources.

When package-save-sources is enabled, then package-reinstall would
reinstall a package from the saved source rather than upgrading the
package.  package-upgrade would delete the saved source and fetch a new
version.

Does this seem like a reasonable feature to add?

As an additional benefit, elpa-sources would be useful for reviewing
changes in the source when upgrading packages.  It could be optionally
maintained as a VCS repository, with packages saved to a constant path
based on their name (e.g. elpa-sources/xref.el instead of
elpa-sources/xref-1.2.3/xref.el).  Then a VCS could be used to review
the diff resulting from an upgrade.  This would be useful for any user
who wants to scrutinize package code changes before installing them.

Re: Automatically saving package sources

[Stefan Kangas]

sbaugh@catern.com writes:

> It is nice for installed ELPA packages to be reinstalled when Emacs is
> upgraded or changed, since there can be changes to autoload generation,
> the byte compiler, the native compiler, or other things which will
> change the output from the package installation.

Yup, this is a long-standing problem.

I think there are some bugs in the bug tracker about it too.  Maybe they
can give some ideas.

> Furthermore, it is nice (where possible) for this reinstallation to
> happen with the same version of the package rather than by fetching the
> latest version from ELPA.  This minimizes the number of changes that
> happen at once.
>
> However, right now there is no easy way to reinstall a package from the
> same sources it was originally installed from, because those sources are
> not saved separately from the package itself.  They are all mixed
> together in the installed package directory.  This makes it impossible
> to produce a "clean slate" environment for reinstalling the package,
> which (since packages can create arbitrary files during
> byte-compilation) means the package can't be reinstalled in a way that's
> identical to its original installation.
>
> If package.el supported saving the package source separately, however,
> it could simply reinstall from that.
>
> I would like to add an option package-save-sources which makes
> package.el automatically save the package source for the currently
> installed package version in a separate directory.  Perhaps in
> ~/.emacs.d/elpa-sources.
>
> When package-save-sources is enabled, then package-reinstall would
> reinstall a package from the saved source rather than upgrading the
> package.  package-upgrade would delete the saved source and fetch a new
> version.
>
> Does this seem like a reasonable feature to add?

Yes, I think it could be useful.  I'm copying in Stefan Monnier too.

One complication that immediately comes to mind is how to find the
sources of those packages from *Help*.  I think it will expect to find
them in ~/.emacs.d/elpa rather than ~/.emacs.d/elpa-sources if the
package is loaded from there.  Maybe some special handling is needed.

Also, do we need to put it behind an option?  I guess I don't understand
the use-case for _not_ wanting this feature.

> As an additional benefit, elpa-sources would be useful for reviewing
> changes in the source when upgrading packages.  It could be optionally
> maintained as a VCS repository, with packages saved to a constant path
> based on their name (e.g. elpa-sources/xref.el instead of
> elpa-sources/xref-1.2.3/xref.el).  Then a VCS could be used to review
> the diff resulting from an upgrade.  This would be useful for any user
> who wants to scrutinize package code changes before installing them.

Re: Automatically saving package sources

[Stefan Monnier]

>> It is nice for installed ELPA packages to be reinstalled when Emacs is
>> upgraded or changed, since there can be changes to autoload generation,
>> the byte compiler, the native compiler, or other things which will
>> change the output from the package installation.

FWIW, native-compiled files do get regenerated (and from the `.el`
rather than from the `.elc`).  So the problem is more for autoloads and
`.elc` files.  And we do have `package-recompile` since Emacs-29.1.

>> However, right now there is no easy way to reinstall a package from the
>> same sources it was originally installed from, because those sources are
>> not saved separately from the package itself.  They are all mixed
>> together in the installed package directory.  This makes it impossible
>> to produce a "clean slate" environment for reinstalling the package,
>> which (since packages can create arbitrary files during
>> byte-compilation) means the package can't be reinstalled in a way that's
>> identical to its original installation.

While that's true in theory, in practice `package-recompile` should work
fine for all but the "weirdest" packages.

>> I would like to add an option package-save-sources which makes
>> package.el automatically save the package source for the currently
>> installed package version in a separate directory.  Perhaps in
>> ~/.emacs.d/elpa-sources.

It could make sense to keep the tarballs, indeed.
I would see it as a cache: when asking to reinstall, we'd try and
re-fetch that specific package version, first looking for a local copy
in our cache, then looking for it in the repository ((Non)GNU ELPA does
keep a number of older versions, even though they're not listed in
`archive-contents`).
And if that fails, there's still `package-recompile` or an upgrade.


        Stefan

Re: Automatically saving package sources

[Philip Kaludercic]

sbaugh@catern.com writes:

> As an additional benefit, elpa-sources would be useful for reviewing
> changes in the source when upgrading packages.  It could be optionally
> maintained as a VCS repository, with packages saved to a constant path
> based on their name (e.g. elpa-sources/xref.el instead of
> elpa-sources/xref-1.2.3/xref.el).  Then a VCS could be used to review
> the diff resulting from an upgrade.  This would be useful for any user
> who wants to scrutinize package code changes before installing them.

This would partially overlap with the functionality provided by
package-vc.  It might be possible to re-use the functionality
wrt. building packages (e.g. building manuals, running extra commands),
that the ELPA build server takes care of when installing a package from
a tarball.

It might be, that all one needs would be to do a package-install-file?

Also, in the specific case of xref, you'd have the issue that xref.el is
maintained in emacs.git, and you cannot just checkout a subtree with
Git, so this would require tracking the entire Emacs repository,
presumably with individual checkouts for each core package you would
want to install, since the revisions one would be using wouldn't be the
same for all of them.

-- 
Philip Kaludercic