[package-vc] Consider cleaning up files from install process

[package-vc] Consider cleaning up files from install process

[Joseph Turner]

package-vc-install currently adds a number of files to a package's
repository which are not tracked by VC.  It is inconvenience to add each
of them to a .gitignore file (or equivalent in other VC systems).  Are
they necessary?  Are you open to removing them in a clean up phase?

For example, when following these instructions to install hyperdrive.el:
https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc

the git repository ends up with these extra untracked files:

dir
hyperdrive-autoloads.el
hyperdrive-pkg.el
hyperdrive.info

Best,

Joseph

Re: [package-vc] Consider cleaning up files from install process

[Philip Kaludercic]

Joseph Turner <joseph@breatheoutbreathe.in> writes:

> package-vc-install currently adds a number of files to a package's
> repository which are not tracked by VC.  It is inconvenience to add each
> of them to a .gitignore file (or equivalent in other VC systems).  Are
> they necessary?  Are you open to removing them in a clean up phase?

Initially package-vc would automatically add these as ignored files
using `vc-ignore', but I decided against keeping that behaviour in
e08e9bc40, due to issues I should have documented at the time.

> For example, when following these instructions to install hyperdrive.el:
> https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc
>
> the git repository ends up with these extra untracked files:

"Sadly", all of these files are necessary for package.el:

> dir

This file is used by info to indicate that there is a manual file that
should be listed in in (dir) Top.

> hyperdrive-autoloads.el

This file contains all the autoload information, including function
and user options stubs, that allow the user to invoke the package
directly without a (require 'hyperdrive)

> hyperdrive-pkg.el

This contains the package descriptor, that tracks the basic package
metadata used to decide what package to load.

> hyperdrive.info

This is the compiled manual, generated from a .texi or .org file in your
repository.  Without this file, there is no manual.

So in summary, this is the difference between just cloning a repository
and adding it to your `load-path'.  Unless I am forgetting something,
there shouldn't be any more files than these, so adding them to your
.gitignore should be a constant effort and should solve the issue.

> Best,
>
> Joseph

Re: [package-vc] Consider cleaning up files from install process

[Adam Porter]

I noticed this problem myself when I first tried using package-vc-install.

An alternative would be to follow Quelpa's example: rather than loading 
the package directly from the git repository's directory, the package's 
files could be copied into a new directory in the elpa/ directory, and 
Emacs could load the package from there.

As a developer, I would prefer that approach, because I don't usually 
want Emacs to load whatever commit I may have checked out in a package's 
repository.  I want to install a commit of a package into my Emacs 
configuration and have that one loaded until I install a different one.

In other words, the way package-vc-install currently works leaves me 
vulnerable to this scenario:

1.  I use package-vc-install to install one of my packages from its 
local repository directory.
2.  I check out a feature branch of that package to work on a new 
feature, saving some files but not loading any of the changed code.
3.  I restart Emacs (e.g. maybe I shut down the system and turned it 
back on the next day).
4.  That work-in-progress feature branch of my package gets loaded into 
Emacs automatically (which may be entirely broken, being a WIP).

Whereas if I use Quelpa, it installs the files a package's recipe 
specifies into the expected location in the elpa/ directory, and that 
remains independent of whatever I may have checked out in the package's 
local git directory.  My Emacs configuration and installed packages 
remain consistent regardless of whatever in-between state I may have 
left a package's repository in.

Philip, would you be willing to consider switching to that model for 
package-vc-install, or offering it as an option?

Thanks,
Adam

Re: [package-vc] Consider cleaning up files from install process

[Philip Kaludercic]

Adam Porter <adam@alphapapa.net> writes:

> I noticed this problem myself when I first tried using package-vc-install.
>
> An alternative would be to follow Quelpa's example: rather than
> loading the package directly from the git repository's directory, the
> package's files could be copied into a new directory in the elpa/
> directory, and Emacs could load the package from there.
>
> As a developer, I would prefer that approach, because I don't usually
> want Emacs to load whatever commit I may have checked out in a
> package's repository.  I want to install a commit of a package into my
> Emacs configuration and have that one loaded until I install a
> different one.

Isn't this workflow already supported by `package-install-file'?  And
otherwise, there wouldn't be much of a point to just replicate the
functionality of an existing package manager.

> In other words, the way package-vc-install currently works leaves me
> vulnerable to this scenario:
>
> 1.  I use package-vc-install to install one of my packages from its
> local repository directory.
> 2.  I check out a feature branch of that package to work on a new
> feature, saving some files but not loading any of the changed code.
> 3.  I restart Emacs (e.g. maybe I shut down the system and turned it
> back on the next day).
> 4.  That work-in-progress feature branch of my package gets loaded
> into Emacs automatically (which may be entirely broken, being a WIP).

I understand your issue, but anecdotally, I can also give an example
where I wanted to change some functionality, by M-x find-function'ing
the source buffer and then loosing the change after upgrading the
package.

> Whereas if I use Quelpa, it installs the files a package's recipe
> specifies into the expected location in the elpa/ directory, and that
> remains independent of whatever I may have checked out in the
> package's local git directory.  My Emacs configuration and installed
> packages remain consistent regardless of whatever in-between state I
> may have left a package's repository in.
>
> Philip, would you be willing to consider switching to that model for
> package-vc-install, or offering it as an option?

Can you describe the interface you are imagining.  From what I
understand, it should be possible to reproduce what you need by
combining `package-vc-checkout' and `package-install-file'?

> Thanks,
> Adam

Re: [package-vc] Consider cleaning up files from install process

[Adam Porter]

On 9/19/23 08:50, Philip Kaludercic wrote:

>> An alternative would be to follow Quelpa's example: rather than
>> loading the package directly from the git repository's directory, the
>> package's files could be copied into a new directory in the elpa/
>> directory, and Emacs could load the package from there.
>>
>> As a developer, I would prefer that approach, because I don't usually
>> want Emacs to load whatever commit I may have checked out in a
>> package's repository.  I want to install a commit of a package into my
>> Emacs configuration and have that one loaded until I install a
>> different one.
> 
> Isn't this workflow already supported by `package-install-file'?  And
> otherwise, there wouldn't be much of a point to just replicate the
> functionality of an existing package manager.

That doesn't seem to do the same thing: it doesn't install from a git 
repo, per se, but from local files or directories:

   (package-install-file FILE)

   Install a package from FILE.
   The file can either be a tar file, an Emacs Lisp file, or a
   directory.

What I like to do is to install the package from the git repo as a, 
well, "normal package," i.e. as if it were installed from ELPA.  That's 
how Quelpa works: it produces, e.g. 
"~/.emacs.d/elpa/PACKAGE-VERSION/PACKAGE{,...}.el".  I don't have to 
have a local clone of the repo first.

(Also, the name of the command suggests that it only installs from files 
rather than directories; I didn't even realize it could install from 
directories until now.)

>> In other words, the way package-vc-install currently works leaves me
>> vulnerable to this scenario:
>>
>> 1.  I use package-vc-install to install one of my packages from its
>> local repository directory.
>> 2.  I check out a feature branch of that package to work on a new
>> feature, saving some files but not loading any of the changed code.
>> 3.  I restart Emacs (e.g. maybe I shut down the system and turned it
>> back on the next day).
>> 4.  That work-in-progress feature branch of my package gets loaded
>> into Emacs automatically (which may be entirely broken, being a WIP).
> 
> I understand your issue, but anecdotally, I can also give an example
> where I wanted to change some functionality, by M-x find-function'ing
> the source buffer and then loosing the change after upgrading the
> package.

IMHO that seems like a sort of user-error, to leave uncommitted changes 
in a repo that's under the control of a system that doesn't take such 
changes into account when operating on the repo.  IOW, if I install a 
package with command package-FOO, I expect the files it makes to be 
under its purview, not mine; they are not my personal data files, but 
belong to Emacs and its package.el library, the same as if I used "M-x 
package-install".

To put it another way, it seems like a bit of an anti-pattern to develop 
a package that's stored in "~/.emacs.d/elpa"--that directory is supposed 
to be for installed packages' files, and it should be safe to wipe that 
directory out at any time and reinstall packages from their 
sources--after all, "M-x package-delete" shouldn't be wiping out user 
data.  Development on packages should be done on clones stored outside 
of "~/.emacs.d", e.g. in "~/src" or whatever one uses.  IMHO, of 
course--Emacs is all about user freedom--yet I think we should encourage 
the better practices.

>> Whereas if I use Quelpa, it installs the files a package's recipe
>> specifies into the expected location in the elpa/ directory, and that
>> remains independent of whatever I may have checked out in the
>> package's local git directory.  My Emacs configuration and installed
>> packages remain consistent regardless of whatever in-between state I
>> may have left a package's repository in.
>>
>> Philip, would you be willing to consider switching to that model for
>> package-vc-install, or offering it as an option?
> 
> Can you describe the interface you are imagining.  From what I
> understand, it should be possible to reproduce what you need by
> combining `package-vc-checkout' and `package-install-file'?

Well, I would expect it to act similarly to Quelpa:

1. Clone the git repo (shallowly, at least by default) to a directory 
(perhaps a temporary one, but at least one stored in the appropriate XDG 
cache directory, outside of user-emacs-directory).

2.  Call package-install-file on that worktree.

3.  The package ends up installed into package-user-dir as if it were 
installed with package-install from ELPA.

Then if the user wants to modify the package, he should either a) use a 
local clone wherever he keeps such projects, or b) configure package-vc 
to clone the repo to such a directory, and then a command could automate 
finding the source files in that directory rather than the ones in 
package-user-dir which are loaded into Emacs.  (IIRC Straight, for 
example, does something like this, but I don't use it myself.)

Do these explanations clarify what I mean?  Let me know if I need to try 
again.  :)

--Adam

Re: [package-vc] Consider cleaning up files from install process

[Philip Kaludercic]

Adam Porter <adam@alphapapa.net> writes:

> On 9/19/23 08:50, Philip Kaludercic wrote:
>
>>> An alternative would be to follow Quelpa's example: rather than
>>> loading the package directly from the git repository's directory, the
>>> package's files could be copied into a new directory in the elpa/
>>> directory, and Emacs could load the package from there.
>>>
>>> As a developer, I would prefer that approach, because I don't usually
>>> want Emacs to load whatever commit I may have checked out in a
>>> package's repository.  I want to install a commit of a package into my
>>> Emacs configuration and have that one loaded until I install a
>>> different one.
>> Isn't this workflow already supported by `package-install-file'?
>> And
>> otherwise, there wouldn't be much of a point to just replicate the
>> functionality of an existing package manager.
>
> That doesn't seem to do the same thing: it doesn't install from a git
> repo, per se, but from local files or directories:
>
>   (package-install-file FILE)
>
>   Install a package from FILE.
>   The file can either be a tar file, an Emacs Lisp file, or a
>   directory.
>
> What I like to do is to install the package from the git repo as a,
> well, "normal package," i.e. as if it were installed from ELPA.
> That's how Quelpa works: it produces,
> e.g. "~/.emacs.d/elpa/PACKAGE-VERSION/PACKAGE{,...}.el".  I don't have
> to have a local clone of the repo first.
>
> (Also, the name of the command suggests that it only installs from
> files rather than directories; I didn't even realize it could install
> from directories until now.)
>
>>> In other words, the way package-vc-install currently works leaves me
>>> vulnerable to this scenario:
>>>
>>> 1.  I use package-vc-install to install one of my packages from its
>>> local repository directory.
>>> 2.  I check out a feature branch of that package to work on a new
>>> feature, saving some files but not loading any of the changed code.
>>> 3.  I restart Emacs (e.g. maybe I shut down the system and turned it
>>> back on the next day).
>>> 4.  That work-in-progress feature branch of my package gets loaded
>>> into Emacs automatically (which may be entirely broken, being a WIP).
>> I understand your issue, but anecdotally, I can also give an example
>> where I wanted to change some functionality, by M-x find-function'ing
>> the source buffer and then loosing the change after upgrading the
>> package.
>
> IMHO that seems like a sort of user-error, to leave uncommitted
> changes in a repo that's under the control of a system that doesn't
> take such changes into account when operating on the repo.  IOW, if I
> install a package with command package-FOO, I expect the files it
> makes to be under its purview, not mine; they are not my personal data
> files, but belong to Emacs and its package.el library, the same as if
> I used "M-x package-install".

I don't believe in these kinds of distinctions.  I can change what I
want, the only question I have to keep in mind is what the chances of
loosing these changes is going to be.

> To put it another way, it seems like a bit of an anti-pattern to
> develop a package that's stored in "~/.emacs.d/elpa"--that directory
> is supposed to be for installed packages' files, and it should be safe
> to wipe that directory out at any time and reinstall packages from
> their sources--after all, 

Is that documented anywhere?  I know it can be done, if your
configuration is written accordingly, but if you see installed packages
as not being part of your domain, then transgressing layers of
abstraction and accessing, then deleting the "internal" representation
of package.el would seem to also be wrong?

>                           "M-x package-delete" shouldn't be wiping out
> user data.  Development on packages should be done on clones stored
> outside of "~/.emacs.d", e.g. in "~/src" or whatever one uses.  IMHO,
> of course--Emacs is all about user freedom--yet I think we should
> encourage the better practices.

I don't see it that way, but the technical background for this decision
is that source packages -- written with the intention of loading the
source code directly from the checkout -- should be loadable without
having to require package-vc, since package-vc is just an extension of
package.el that allows for an alternative way of installing packages.

>>> Whereas if I use Quelpa, it installs the files a package's recipe
>>> specifies into the expected location in the elpa/ directory, and that
>>> remains independent of whatever I may have checked out in the
>>> package's local git directory.  My Emacs configuration and installed
>>> packages remain consistent regardless of whatever in-between state I
>>> may have left a package's repository in.
>>>
>>> Philip, would you be willing to consider switching to that model for
>>> package-vc-install, or offering it as an option?
>> Can you describe the interface you are imagining.  From what I
>> understand, it should be possible to reproduce what you need by
>> combining `package-vc-checkout' and `package-install-file'?
>
> Well, I would expect it to act similarly to Quelpa:
>
> 1. Clone the git repo (shallowly, at least by default) to a directory
> (perhaps a temporary one, but at least one stored in the appropriate
> XDG cache directory, outside of user-emacs-directory).
>
> 2.  Call package-install-file on that worktree.
>
> 3.  The package ends up installed into package-user-dir as if it were
> installed with package-install from ELPA.

How does this look like:

--8<---------------cut here---------------start------------->8---
(defun package-vc-install-copy (pkg-desc)
  "Fetch the package sources of PKG-DESC and install a copy."
  (interactive (list (package-vc--read-package-desc "Fetch package source: ")))
  (let* ((copy (copy-package-desc pkg-desc))
	 (package-dir (expand-file-name
		       (symbol-name (package-desc-name pkg-desc))
		       (make-temp-file "package-vc-" t)))
	 (package-vc-register-as-project nil))
    (setf (package-desc-dir copy) package-dir)
    (save-window-excursion
      (package-vc-checkout pkg-desc package-dir)
      (package-install-from-buffer))))
--8<---------------cut here---------------end--------------->8---

I'd ideally pass :last-release to package-vc-checkout, but I have
detected a bug with this setup that should be fixed.

> Then if the user wants to modify the package, he should either a) use
> a local clone wherever he keeps such projects, or b) configure
> package-vc to clone the repo to such a directory, and then a command
> could automate finding the source files in that directory rather than
> the ones in package-user-dir which are loaded into Emacs.  (IIRC
> Straight, for example, does something like this, but I don't use it
> myself.)

To my understanding Straight maintains all packages in separate
repositories, then symlinks the files that are to be loaded.  So just
like with package-vc, you always have direct access to the source code
under VC.

> Do these explanations clarify what I mean?  Let me know if I need to
> try again.  :)

The main question I have is what advantage or disadvantage this would
provide over Quelpa?  The functionality you describe misses the point of
what I see to be the point of package-vc, as it wasn't written to be a
replacement of Straight or Quelpa (I have never used either of the two),
but just as a tool to streamline fetching and preparing packages
directly from source code repositories.

> --Adam

Re: [package-vc] Consider cleaning up files from install process

[Stefan Monnier via Emacs development discussions.]

> For example, when following these instructions to install hyperdrive.el:
> https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc
>
> the git repository ends up with these extra untracked files:
>
> dir
> hyperdrive-autoloads.el
> hyperdrive-pkg.el
> hyperdrive.info

Please report this as a bug in the Hyperdrive repository: its
`.gitignore` should be adjusted to ignore those files.


        Stefan

Re: [package-vc] Consider cleaning up files from install process

[Stefan Monnier via Emacs development discussions.]

> In other words, the way package-vc-install currently works leaves me
> vulnerable to this scenario:
>
> 1.  I use package-vc-install to install one of my packages from its local
>    repository directory.
> 2.  I check out a feature branch of that package to work on a new feature,
>    saving some files but not loading any of the changed code.
> 3.  I restart Emacs (e.g. maybe I shut down the system and turned it back on
>    the next day).
> 4.  That work-in-progress feature branch of my package gets loaded into
>     Emacs automatically (which may be entirely broken, being a WIP).

Not sure why you consider that a problem.  That's the whole purpose of
installing straight from the VCS rather than from a tarball.

If you don't want your Emacs to use that work-in-progress feature
branch, then work on it elsewhere than in the VCS clone that was created
specifically to be used in-place.
E.g. use `git worktree`.

> Philip, would you be willing to consider switching to that model for
> package-vc-install, or offering it as an option?

The option already exists, it's called `git worktree` :-)

Switching to "that model" wholesale would prevent the current use where
you can edit the *live* files.


        Stefan

Re: [package-vc] Consider cleaning up files from install process

[Stefan Monnier via Emacs development discussions.]

> Well, I would expect it to act similarly to Quelpa:
>
> 1. Clone the git repo (shallowly, at least by default) to a directory
>    (perhaps a temporary one, but at least one stored in the appropriate XDG
>    cache directory, outside of user-emacs-directory).
>
> 2.  Call package-install-file on that worktree.
>
> 3.  The package ends up installed into package-user-dir as if it were
>     installed with package-install from ELPA.

So that treats the Git repository as a sort of "distribution site for
tarball-like thingies".  If you like it, use Quelpa (or argue for ELPA to
support other repository formats).
That's not the purpose of `package-vc`.

`package-vc`s purpose is to install the source so you can hack on it.


        Stefan

Re: [package-vc] Consider cleaning up files from install process

[Joseph Turner]

> > For example, when following these instructions to install hyperdrive.el:
> > https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc
> >
> > the git repository ends up with these extra untracked files:
> >
> > dir
> > hyperdrive-autoloads.el
> > hyperdrive-pkg.el
> > hyperdrive.info

> Please report this as a bug in the Hyperdrive repository: its
> `.gitignore` should be adjusted to ignore those files.

Good to know. We have updated the hyperdrive.el .gitignore.

Are all package authors are expected to add these files to .gitignore
(or equivalent in other VCS)?  If so, we should probably put a notice in
the package-vc documentation or somewhere else. What do you suggest?

Thank you!!

Joseph

Re: [package-vc] Consider cleaning up files from install process

[Joseph Turner]

> > In other words, the way package-vc-install currently works leaves me
> > vulnerable to this scenario:
> >
> > 1.  I use package-vc-install to install one of my packages from its local
> >    repository directory.
> > 2.  I check out a feature branch of that package to work on a new feature,
> >    saving some files but not loading any of the changed code.
> > 3.  I restart Emacs (e.g. maybe I shut down the system and turned it back on
> >    the next day).
> > 4.  That work-in-progress feature branch of my package gets loaded into
> >     Emacs automatically (which may be entirely broken, being a WIP).

> Not sure why you consider that a problem.  That's the whole purpose of
> installing straight from the VCS rather than from a tarball.

> If you don't want your Emacs to use that work-in-progress feature
> branch, then work on it elsewhere than in the VCS clone that was created
> specifically to be used in-place.
> E.g. use `git worktree`.

> > Philip, would you be willing to consider switching to that model for
> > package-vc-install, or offering it as an option?

> The option already exists, it's called `git worktree` :-)

I had not considered `git worktree` as a solution.  Very neat.

> Switching to "that model" wholesale would prevent the current use where
> you can edit the *live* files.

Thank you!

Joseph

Re: [package-vc] Consider cleaning up files from install process

[Joseph Turner]

> > For example, when following these instructions to install hyperdrive.el:
> > https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc
> >
> > the git repository ends up with these extra untracked files:
> >
> > dir
> > hyperdrive-autoloads.el
> > hyperdrive-pkg.el
> > hyperdrive.info

> Please report this as a bug in the Hyperdrive repository: its
> `.gitignore` should be adjusted to ignore those files.

Good to know. We have updated the hyperdrive.el .gitignore.

Are all package authors are expected to add these files to .gitignore
(or equivalent in other VCS)?  If so, we should probably put a notice in
the package-vc documentation or somewhere else.  What do you suggest?

Thank you!

Joseph

Re: [package-vc] Consider cleaning up files from install process

[Philip Kaludercic]

Joseph Turner <joseph@breatheoutbreathe.in> writes:

>> > For example, when following these instructions to install hyperdrive.el:
>> > https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc
>> >
>> > the git repository ends up with these extra untracked files:
>> >
>> > dir
>> > hyperdrive-autoloads.el
>> > hyperdrive-pkg.el
>> > hyperdrive.info
>
>> Please report this as a bug in the Hyperdrive repository: its
>> `.gitignore` should be adjusted to ignore those files.
>
> Good to know. We have updated the hyperdrive.el .gitignore.
>
> Are all package authors are expected to add these files to .gitignore
> (or equivalent in other VCS)?  If so, we should probably put a notice in
> the package-vc documentation or somewhere else.  What do you suggest?

This is not related to package-vc, it is also the recommended practice
for regular ELPA packages.

> Thank you!
>
> Joseph

Re: [package-vc] Consider cleaning up files from install process

[Joseph Turner]

On October 4, 2023 8:23:07 AM PDT, Philip Kaludercic <philipk@posteo.net> wrote:
>Joseph Turner <joseph@breatheoutbreathe.in> writes:
>
>>> > For example, when following these instructions to install hyperdrive.el:
>>> > https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc
>>> >
>>> > the git repository ends up with these extra untracked files:
>>> >
>>> > dir
>>> > hyperdrive-autoloads.el
>>> > hyperdrive-pkg.el
>>> > hyperdrive.info
>>
>>> Please report this as a bug in the Hyperdrive repository: its
>>> `.gitignore` should be adjusted to ignore those files.
>>
>> Good to know. We have updated the hyperdrive.el .gitignore.
>>
>> Are all package authors are expected to add these files to .gitignore
>> (or equivalent in other VCS)?  If so, we should probably put a notice in
>> the package-vc documentation or somewhere else.  What do you suggest?
>
>This is not related to package-vc, it is also the recommended practice
>for regular ELPA packages.

Would you mind sharing a link to this recommendation? I searched the Emacs/Elisp info manuals as well as the ELPA README.

>> Thank you!
>>
>> Joseph

Re: [package-vc] Consider cleaning up files from install process

[Philip Kaludercic]

Joseph Turner <joseph@breatheoutbreathe.in> writes:

> On October 4, 2023 8:23:07 AM PDT, Philip Kaludercic <philipk@posteo.net> wrote:
>>Joseph Turner <joseph@breatheoutbreathe.in> writes:
>>
>>>> > For example, when following these instructions to install hyperdrive.el:
>>>> > https://ushin.org/hyperdrive/hyperdrive-manual.html#package_002dvc
>>>> >
>>>> > the git repository ends up with these extra untracked files:
>>>> >
>>>> > dir
>>>> > hyperdrive-autoloads.el
>>>> > hyperdrive-pkg.el
>>>> > hyperdrive.info
>>>
>>>> Please report this as a bug in the Hyperdrive repository: its
>>>> `.gitignore` should be adjusted to ignore those files.
>>>
>>> Good to know. We have updated the hyperdrive.el .gitignore.
>>>
>>> Are all package authors are expected to add these files to .gitignore
>>> (or equivalent in other VCS)?  If so, we should probably put a notice in
>>> the package-vc documentation or somewhere else.  What do you suggest?
>>
>>This is not related to package-vc, it is also the recommended practice
>>for regular ELPA packages.
>
> Would you mind sharing a link to this recommendation? I searched the Emacs/Elisp info manuals as well as the ELPA README.

I don't know of any link, what I meant with "recommendation" was "it is
frequently recommended when reviewing a package".

>>> Thank you!
>>>
>>> Joseph

Re: [package-vc] Consider cleaning up files from install process

[Adam Porter]

Hi Philip,

On 9/20/23 03:13, Philip Kaludercic wrote:

>> IMHO that seems like a sort of user-error, to leave uncommitted
>> changes in a repo that's under the control of a system that doesn't
>> take such changes into account when operating on the repo.  IOW, if I
>> install a package with command package-FOO, I expect the files it
>> makes to be under its purview, not mine; they are not my personal data
>> files, but belong to Emacs and its package.el library, the same as if
>> I used "M-x package-install".
> 
> I don't believe in these kinds of distinctions.  I can change what I
> want, the only question I have to keep in mind is what the chances of
> loosing these changes is going to be.

I think I understand what you mean.  However, that seems to conflict 
with the idea of loading packages directly from git worktrees, 
regardless of their status.  That is, when I install a package into my 
Emacs configuration, I intend for the version I installed to be 
available when I start Emacs, not whatever state I might have left the 
repository in when I last touched it.  If I had last checked out an 
in-progress feature branch and then shut down my computer, I wouldn't 
want Emacs to attempt to load that next time.

>> To put it another way, it seems like a bit of an anti-pattern to
>> develop a package that's stored in "~/.emacs.d/elpa"--that directory
>> is supposed to be for installed packages' files, and it should be safe
>> to wipe that directory out at any time and reinstall packages from
>> their sources--after all,
> 
> Is that documented anywhere?  I know it can be done, if your
> configuration is written accordingly, but if you see installed packages
> as not being part of your domain, then transgressing layers of
> abstraction and accessing, then deleting the "internal" representation
> of package.el would seem to also be wrong?

I don't know if that idea is documented anywhere, but it seems implicit 
to me by reason of users not necessarily needing to be aware of what 
files and directories get created when installing a package.  If a user 
only used the package-install and package-delete commands, and didn't 
look at the filesystem or messages output, he wouldn't know nor care 
where the packages' files were.  (I regularly advocate committing one's 
"elpa/" directory to git along with one's "init.el", and in so doing 
I've learned how many users aren't aware of how it all works.)

>>> Can you describe the interface you are imagining.  From what I
>>> understand, it should be possible to reproduce what you need by
>>> combining `package-vc-checkout' and `package-install-file'?
>>
>> Well, I would expect it to act similarly to Quelpa:
>>
>> 1. Clone the git repo (shallowly, at least by default) to a directory
>> (perhaps a temporary one, but at least one stored in the appropriate
>> XDG cache directory, outside of user-emacs-directory).
>>
>> 2.  Call package-install-file on that worktree.
>>
>> 3.  The package ends up installed into package-user-dir as if it were
>> installed with package-install from ELPA.
> 
> How does this look like:
> 
> --8<---------------cut here---------------start------------->8---
> (defun package-vc-install-copy (pkg-desc)
>    "Fetch the package sources of PKG-DESC and install a copy."
>    (interactive (list (package-vc--read-package-desc "Fetch package source: ")))
>    (let* ((copy (copy-package-desc pkg-desc))
> 	 (package-dir (expand-file-name
> 		       (symbol-name (package-desc-name pkg-desc))
> 		       (make-temp-file "package-vc-" t)))
> 	 (package-vc-register-as-project nil))
>      (setf (package-desc-dir copy) package-dir)
>      (save-window-excursion
>        (package-vc-checkout pkg-desc package-dir)
>        (package-install-from-buffer))))
> --8<---------------cut here---------------end--------------->8---
> 
> I'd ideally pass :last-release to package-vc-checkout, but I have
> detected a bug with this setup that should be fixed.

I tested that command in a clean Emacs 29.1, and it seems to work well. 
Thanks.

The only suggestion I have would be for an optional persistent directory 
used for keeping the git repositories around as a cache (i.e. outside of 
user-emacs-directory, likely in $XDG_CACHE_HOME), so that when upgrading 
the package, the whole repo wouldn't need to be cloned again, e.g. 
Quelpa configures this in the `quelpa-dir' option.

> The main question I have is what advantage or disadvantage this would
> provide over Quelpa?  The functionality you describe misses the point of
> what I see to be the point of package-vc, as it wasn't written to be a
> replacement of Straight or Quelpa (I have never used either of the two),
> but just as a tool to streamline fetching and preparing packages
> directly from source code repositories.

The advantage over Quelpa would simply be to not require Quelpa.  :) 
I'd like for this functionality to be built-in to Emacs.  I thought 
that's what package-vc was basically intended to do (i.e. give a user a 
git repo URL and he can install the package into his Emacs with a single 
command, as if he had cloned it manually and ran package-install on it), 
but maybe I misunderstood.

Thanks for your help,
Adam

Re: [package-vc] Consider cleaning up files from install process

[Philip Kaludercic]

Adam Porter <adam@alphapapa.net> writes:

> Hi Philip,
>
> On 9/20/23 03:13, Philip Kaludercic wrote:
>
>>> IMHO that seems like a sort of user-error, to leave uncommitted
>>> changes in a repo that's under the control of a system that doesn't
>>> take such changes into account when operating on the repo.  IOW, if I
>>> install a package with command package-FOO, I expect the files it
>>> makes to be under its purview, not mine; they are not my personal data
>>> files, but belong to Emacs and its package.el library, the same as if
>>> I used "M-x package-install".
>> I don't believe in these kinds of distinctions.  I can change what I
>> want, the only question I have to keep in mind is what the chances of
>> loosing these changes is going to be.
>
> I think I understand what you mean.  However, that seems to conflict
> with the idea of loading packages directly from git worktrees,
> regardless of their status.  That is, when I install a package into my
> Emacs configuration, I intend for the version I installed to be
> available when I start Emacs, not whatever state I might have left the
> repository in when I last touched it.  If I had last checked out an
> in-progress feature branch and then shut down my computer, I wouldn't
> want Emacs to attempt to load that next time.

It is not that I don't get your argument, I think it is a fair point; my
issue is that this ends up just being a discussion of anecdotal
use-cases.  I can point out that if I check out a feature branch, I
would like to use it, that if I modify a function, I'd like to test it.
If I just want to hack on something that I am not using, I wouldn't be
using package-vc.  But I suspect that this doesn't mean that much to
you, because we just have different workflows.

>>> To put it another way, it seems like a bit of an anti-pattern to
>>> develop a package that's stored in "~/.emacs.d/elpa"--that directory
>>> is supposed to be for installed packages' files, and it should be safe
>>> to wipe that directory out at any time and reinstall packages from
>>> their sources--after all,
>> Is that documented anywhere?  I know it can be done, if your
>> configuration is written accordingly, but if you see installed packages
>> as not being part of your domain, then transgressing layers of
>> abstraction and accessing, then deleting the "internal" representation
>> of package.el would seem to also be wrong?
>
> I don't know if that idea is documented anywhere, but it seems
> implicit to me by reason of users not necessarily needing to be aware
> of what files and directories get created when installing a package.
> If a user only used the package-install and package-delete commands,
> and didn't look at the filesystem or messages output, he wouldn't know
> nor care where the packages' files were.  

I don't think I agree.  Just because a user doesn't know what "apt
install coreutils" does, doesn't mean that the system should have to be
resilient to a "rm -rf {,/usr{,/local}}/bin".  It would be a different
matter if the user could reasonably assume that elpa/ is a cache
directory, but I don't see why that should be the case.

>                                           (I regularly advocate
> committing one's "elpa/" directory to git along with one's "init.el",
> and in so doing I've learned how many users aren't aware of how it all
> works.)

That is something i wouldn't recommend, but mostly because I avoid
adding auto-generated files to a git repository.

>>>> Can you describe the interface you are imagining.  From what I
>>>> understand, it should be possible to reproduce what you need by
>>>> combining `package-vc-checkout' and `package-install-file'?
>>>
>>> Well, I would expect it to act similarly to Quelpa:
>>>
>>> 1. Clone the git repo (shallowly, at least by default) to a directory
>>> (perhaps a temporary one, but at least one stored in the appropriate
>>> XDG cache directory, outside of user-emacs-directory).
>>>
>>> 2.  Call package-install-file on that worktree.
>>>
>>> 3.  The package ends up installed into package-user-dir as if it were
>>> installed with package-install from ELPA.
>> How does this look like:
>> --8<---------------cut here---------------start------------->8---
>> (defun package-vc-install-copy (pkg-desc)
>>    "Fetch the package sources of PKG-DESC and install a copy."
>>    (interactive (list (package-vc--read-package-desc "Fetch package source: ")))
>>    (let* ((copy (copy-package-desc pkg-desc))
>> 	 (package-dir (expand-file-name
>> 		       (symbol-name (package-desc-name pkg-desc))
>> 		       (make-temp-file "package-vc-" t)))
>> 	 (package-vc-register-as-project nil))
>>      (setf (package-desc-dir copy) package-dir)
>>      (save-window-excursion
>>        (package-vc-checkout pkg-desc package-dir)
>>        (package-install-from-buffer))))
>> --8<---------------cut here---------------end--------------->8---
>> I'd ideally pass :last-release to package-vc-checkout, but I have
>> detected a bug with this setup that should be fixed.
>
> I tested that command in a clean Emacs 29.1, and it seems to work
> well. Thanks.
>
> The only suggestion I have would be for an optional persistent
> directory used for keeping the git repositories around as a cache
> (i.e. outside of user-emacs-directory, likely in $XDG_CACHE_HOME), so
> that when upgrading the package, the whole repo wouldn't need to be
> cloned again, e.g. Quelpa configures this in the `quelpa-dir' option.

Currently I wouldn't want to add this to package-vc itself, but I could
imagine adding a package to ELPA that would provide the functionality
with all the related user option.

>> The main question I have is what advantage or disadvantage this would
>> provide over Quelpa?  The functionality you describe misses the point of
>> what I see to be the point of package-vc, as it wasn't written to be a
>> replacement of Straight or Quelpa (I have never used either of the two),
>> but just as a tool to streamline fetching and preparing packages
>> directly from source code repositories.
>
> The advantage over Quelpa would simply be to not require Quelpa.  :)
> I'd like for this functionality to be built-in to Emacs.  I thought
> that's what package-vc was basically intended to do (i.e. give a user
> a git repo URL and he can install the package into his Emacs with a
> single command, as if he had cloned it manually and ran
> package-install on it), but maybe I misunderstood.

The background for package-vc is that it is a further iteration on my
site-lisp package.  I wrote that to automatically manage (add to
`load-path', byte-compile, scrape for autoloads) my packages, that I
load directly from their Git repositories, because I want M-x
find-function to work.  Following Stefan Monnier advice, the approach
with package-vc was to delegate loading the package.el, and add an
alternative "back-end" for fetching packages + means for contributing
local changes back upstream (package-vc-prepare-patch, and why
package-vc-install fetches an index of package specifications from ELPA
servers).  Installing packages that are not listed in GNU or NonGNU ELPA
is neat, but not my focus behind developing the package.  That remains
streamlining the process of contributing to packages and hacking
directly on the code that is being loaded.

> Thanks for your help,
> Adam