package-vc support for :files keyword

package-vc support for :files keyword

[Daniel Fleischer]

Hi, I've been testing package-vc support in Emacs 29. It's clear that
this feature is inspired by 3rd party packages that help users install
packages from external forges. Also, it makes sense that the Melpa
recipe specification compatibility is not a goal. However, I do think
the `:files' keyword is very useful to have, and was wondering whether
it's planned, maybe in master?

Thanks, 

-- 
Daniel Fleischer

Re: package-vc support for :files keyword

[Philip Kaludercic]

Daniel Fleischer <danflscr@gmail.com> writes:

> Hi, I've been testing package-vc support in Emacs 29. It's clear that
> this feature is inspired by 3rd party packages that help users install
> packages from external forges. 

I have never used any of the other packages managers, primarily because
they seem to peruse different goals.  The primary motivation for
package-vc is to make it easy to hack on packages that are already
available on an ELPA and contribute these changes upstream via commands
like `package-vc-prepare-patch'.  The fact that it can be used as the
primary method for package.el to fetch sources is an "unintended"
consequence.

>                                Also, it makes sense that the Melpa
> recipe specification compatibility is not a goal. 

Right, as the primary technical inspiration for package-vc has been
elpa-admin, and therefore we (re-)use GNU ELPA-style package
specifications.

>                                                   However, I do think
> the `:files' keyword is very useful to have, and was wondering whether
> it's planned, maybe in master?

No, just like `:ignored-files' I do not see a sensible way to implement
it without compromising on the primary goal mentioned above.  Generally
speaking, this is a style of package development that {GNU,NonGNU} ELPA
would like to advise against perusing, in favour of a
one-repo-per-package approach.  Do you have a specific reason why you
think that something like `:{ignored-,}files' are useful in general
(i.e. not just because of the contingent fact that MELPA has popularised
this form of development).

But just like :make and :shell-command, or use-package support was not
intended in the beginning, I don't insist on anything as long as a
good compromise can be found.  I just have my doubts, since supporting
this would probably run against a number of basic assumptions that
package-vc was written around.
    
> Thanks, 

Re: package-vc support for :files keyword

[Daniel Fleischer]

Philip Kaludercic <philipk@posteo.net> writes:

> Right, as the primary technical inspiration for package-vc has been
> elpa-admin, and therefore we (re-)use GNU ELPA-style package
> specifications.

Hi, I wasn't aware there's a style guide for ELPA packages.

> No, just like `:ignored-files' I do not see a sensible way to implement
> it without compromising on the primary goal mentioned above.

I don't understand this part: `:ignored-files' is part of the ELPA
documentation and you are saying it's still not supported?

> Generally speaking, this is a style of package development that
> {GNU,NonGNU} ELPA would like to advise against perusing, in favour of
> a one-repo-per-package approach. Do you have a specific reason why you
> think that something like `:{ignored-,}files' are useful in general
> (i.e. not just because of the contingent fact that MELPA has
> popularised this form of development).

One example is tests; having test files either in the root folder or in
a folder called "tests", how do you prevent them from being evaluated?
They could have additional dev dependencies that you might not want to
have. What is the ELPA guideline about that?

Re: package-vc support for :files keyword

[Philip Kaludercic]

(If possible, try to use a wide reply when responding to these messages,
so that I (and anyone else who would want to participate but it not
subscribed) get notified directly).

Daniel Fleischer <danflscr@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Right, as the primary technical inspiration for package-vc has been
>> elpa-admin, and therefore we (re-)use GNU ELPA-style package
>> specifications.
>
> Hi, I wasn't aware there's a style guide for ELPA packages.

This is not really a style-guide (which is something I would interpret
as concerning the contents of a package), but just a matter of how it
would be best for packages and the package archive to interact.

>> No, just like `:ignored-files' I do not see a sensible way to implement
>> it without compromising on the primary goal mentioned above.
>
> I don't understand this part: `:ignored-files' is part of the ELPA
> documentation and you are saying it's still not supported?

It is not supported by package-vc.  If you check (emacs) Fetching
Package Sources, then you'll see the supported keywords listed under the
"Specifying Package Sources" section.  Generally speaking, package-vc
is a subset of the keywords defined by elpa-admin.

>> Generally speaking, this is a style of package development that
>> {GNU,NonGNU} ELPA would like to advise against perusing, in favour of
>> a one-repo-per-package approach. Do you have a specific reason why you
>> think that something like `:{ignored-,}files' are useful in general
>> (i.e. not just because of the contingent fact that MELPA has
>> popularised this form of development).
>
> One example is tests; having test files either in the root folder or in
> a folder called "tests", how do you prevent them from being evaluated?
> They could have additional dev dependencies that you might not want to
> have. What is the ELPA guideline about that?

Test files are not evaluated/loaded, unless the user does so explicitly
or if the package developer were to force it via autoloads.  In the
former case, I would say that it is probably intended behaviour, in the
latter case this is a mistake made by the maintainer.

If the file is located under a tests/ directory (or any other name),
then it will not be added to the `load-path' to begin with.

The point here is, that if a user wants to contribute to a package with
tests, the tests shouldn't be hidden or even removed.  That is why I'd
even say it would be better to add the tests to the main lisp directory,
so that it is even easier to load them.

-- 
Philip Kaludercic

Re: package-vc support for :files keyword

[Daniel Fleischer]

Philip Kaludercic <philipk@posteo.net> writes:

> (If possible, try to use a wide reply when responding to these messages,
> so that I (and anyone else who would want to participate but it not
> subscribed) get notified directly).

Will do.


I need to learn more about this topic; I learned about `.elpaignore'
which could be what I need. So I won't discuss specific features at this
point.

I will leave one point for thought: trying to gap the difference between
ELPA packages and the "others" (melpa) could benefit everyone; it can
lower the technical bar for external packages to join ELPA or non-GNU
ELPA; packages will still be reviewed upon submission but things would
work more smoothly as there will be only one recipe style.

Thanks, 

-- 
Daniel Fleischer

Re: package-vc support for :files keyword

[Philip Kaludercic]

Daniel Fleischer <danflscr@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> (If possible, try to use a wide reply when responding to these messages,
>> so that I (and anyone else who would want to participate but it not
>> subscribed) get notified directly).
>
> Will do.

Thanks.

>
> I need to learn more about this topic; I learned about `.elpaignore'
> which could be what I need. 

Right, package.el parses .elpaignore files to exclude files from byte
compilation, and nothing else.  

>                             So I won't discuss specific features at this
> point.

I am happy to help if there is anything else I can clarify.

> I will leave one point for thought: trying to gap the difference between
> ELPA packages and the "others" (melpa) could benefit everyone; it can
> lower the technical bar for external packages to join ELPA or non-GNU
> ELPA; packages will still be reviewed upon submission but things would
> work more smoothly as there will be only one recipe style.

The thing is, most of the time all a package needs to be added to GNU or
NonGNU ELPA is just a URL.  That is why package-vc-install can
frequently handle installing a package by just proving a URL to a
repository.  Just check the elpa-packages files for the two archives:

https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/elpa-packages
https://git.savannah.gnu.org/cgit/emacs/nongnu.git/tree/elpa-packages

I don't think will be possible for ELPA-style package specifications and
MELPA-style package recipes to be merged, nor do I see a urgent need for
that to happen.  The two archives don't only differ in superficial
points like here, in the way packages are represented and listed, but in
other fine technical details and philosophical matters.

> Thanks, 

-- 
Philip Kaludercic

Re: package-vc support for :files keyword

[Tony Zorman]

Hi,

Philip Kaludercic <philipk@posteo.net> writes:
> But just like :make and :shell-command, or use-package support was not
> intended in the beginning, I don't insist on anything as long as a
> good compromise can be found.  I just have my doubts, since supporting
> this would probably run against a number of basic assumptions that
> package-vc was written around.

here's an idea: why not create e.g. a variable to decide when
:shell-command is executed? Moving the execution to right after cloning
the repository, instead of before building documentation, would enable
one to easily emulate :files directives, as well as other keywords that
need to be executed before things actually get built.

Alternatively one could have a second :shell-command like keyword.

  Tony

-- 
Tony Zorman | https://tony-zorman.com/

Re: package-vc support for :files keyword

[Philip Kaludercic]

Tony Zorman <tonyzorman@mailbox.org> writes:

> Hi,
>
> Philip Kaludercic <philipk@posteo.net> writes:
>> But just like :make and :shell-command, or use-package support was not
>> intended in the beginning, I don't insist on anything as long as a
>> good compromise can be found.  I just have my doubts, since supporting
>> this would probably run against a number of basic assumptions that
>> package-vc was written around.
>
> here's an idea: why not create e.g. a variable to decide when
> :shell-command is executed? Moving the execution to right after cloning
> the repository, instead of before building documentation, would enable
> one to easily emulate :files directives, as well as other keywords that
> need to be executed before things actually get built.
>
> Alternatively one could have a second :shell-command like keyword.

My issue with the first idea would be that this would create a greater
discrepancy between what elpa-admin and package-vc do.  So if anything,
I think only the second option, e.g. :early-shell-command, would be
viable.

In both cases, what would you imagine that the command would do?  If it
just calls "rm foo.el bar.el ...", then we have an issue when upgrading,
because there would at least be a merge conflict any time the other
files are modified upstream.  Upgrading isn't easy the way it is, but
raising the necessity for manual intervention every time is something
I'd like to avoid.  Also, what would a :early-shell-command be used for
on the ELPA build server?

>   Tony

-- 
Philip Kaludercic

Re: package-vc support for :files keyword

[Tony Zorman]

On Mon, Sep 18 2023 09:10, Philip Kaludercic wrote:
> Tony Zorman <tonyzorman@mailbox.org> writes:
>> here's an idea: why not create e.g. a variable to decide when
>> :shell-command is executed? Moving the execution to right after cloning
>> the repository, instead of before building documentation, would enable
>> one to easily emulate :files directives, as well as other keywords that
>> need to be executed before things actually get built.
>>
>> Alternatively one could have a second :shell-command like keyword.
>
> My issue with the first idea would be that this would create a greater
> discrepancy between what elpa-admin and package-vc do.  So if anything,
> I think only the second option, e.g. :early-shell-command, would be
> viable.

Fair enough.

> In both cases, what would you imagine that the command would do?  If it
> just calls "rm foo.el bar.el ...", then we have an issue when upgrading,
> because there would at least be a merge conflict any time the other
> files are modified upstream.  Upgrading isn't easy the way it is, but
> raising the necessity for manual intervention every time is something
> I'd like to avoid.

I think that

  $ rm foo.el
  $ git update-index --assume-unchanged foo.el

should work. It should merge cleanly (I've tried this out just now, and
it worked, but I may have overlooked something). If it's part of the
package description, then updating should work out of the box, since
package-vc-upgrade also calls package-vc--unpack-1, which would execute
the respective :early-shell-command again.

> Also, what would a :early-shell-command be used for
> on the ELPA build server?

That I don't know, I've never consciously looked at the ELPA build
server.

-- 
Tony Zorman | https://tony-zorman.com/

Re: package-vc support for :files keyword

[Philip Kaludercic]

Tony Zorman <tonyzorman@mailbox.org> writes:

> On Mon, Sep 18 2023 09:10, Philip Kaludercic wrote:
>> Tony Zorman <tonyzorman@mailbox.org> writes:
>>> here's an idea: why not create e.g. a variable to decide when
>>> :shell-command is executed? Moving the execution to right after cloning
>>> the repository, instead of before building documentation, would enable
>>> one to easily emulate :files directives, as well as other keywords that
>>> need to be executed before things actually get built.
>>>
>>> Alternatively one could have a second :shell-command like keyword.
>>
>> My issue with the first idea would be that this would create a greater
>> discrepancy between what elpa-admin and package-vc do.  So if anything,
>> I think only the second option, e.g. :early-shell-command, would be
>> viable.
>
> Fair enough.
>
>> In both cases, what would you imagine that the command would do?  If it
>> just calls "rm foo.el bar.el ...", then we have an issue when upgrading,
>> because there would at least be a merge conflict any time the other
>> files are modified upstream.  Upgrading isn't easy the way it is, but
>> raising the necessity for manual intervention every time is something
>> I'd like to avoid.
>
> I think that
>
>   $ rm foo.el
>   $ git update-index --assume-unchanged foo.el
>
> should work. It should merge cleanly (I've tried this out just now, and
> it worked, but I may have overlooked something). If it's part of the
> package description, then updating should work out of the box, since
> package-vc-upgrade also calls package-vc--unpack-1, which would execute
> the respective :early-shell-command again.

Being a git-specific command, this shouldn't be added to package-vc
directly.  If there is a VCS agnostic/generalisable way of doing this,
then it could be added to VC.

But for now, if I understand you correctly, you are suggesting that
users give package specifications like this:

(foo :url "https://some.vcs/repository"
     ;; ...
     :early-shell-command "rm [all the files]; git update-index --assume-unchanged [all the files]")

where [all the files] might change between updates.

At this point I continue to question the utility of emulating
MELPA-style :files attributes, unless there are concrete usability
issues.

For the record, these are all the repositories in {Non,}GNU ELPA that
develop multiple packages in a single repository:

GNU ELPA:
http://www.dr-qubit.org/git/predictive.git
https://github.com/oantolin/embark
https://github.com/abo-abo/swiper
https://github.com/abo-abo/hydra
https://github.com/alphapapa/taxy.el.git

NonGNU ELPA:
https://github.com/lewang/flx
https://github.com/emacs-helm/helm
https://github.com/magit/magit
https://github.com/brianc/jade-mode

From what I understand, there is no technical necessity for this mode of
development?  I wonder how difficult it would be to push for a
one-package-one-repo approach.

>> Also, what would a :early-shell-command be used for
>> on the ELPA build server?
>
> That I don't know, I've never consciously looked at the ELPA build
> server.

-- 
Philip Kaludercic

Re: package-vc support for :files keyword

[Adam Porter]

Hi Philip, et al,

> For the record, these are all the repositories in {Non,}GNU ELPA that
> develop multiple packages in a single repository:
> 
> GNU ELPA:
> http://www.dr-qubit.org/git/predictive.git
> https://github.com/oantolin/embark
> https://github.com/abo-abo/swiper
> https://github.com/abo-abo/hydra
> https://github.com/alphapapa/taxy.el.git
> 
> From what I understand, there is no technical necessity for this mode of
> development?  I wonder how difficult it would be to push for a
> one-package-one-repo approach.

Please note out that while `taxy' and `taxy-magit-section' are both 
developed in "taxy.el.git", they are in separate branches, so there is 
no need to build the two packages from a single set of files by 
excluding some files and then the others.

I've chosen to keep these packages in the same repo because they are so 
closely related.  I'd like to be able to keep this arrangement.

Having said that, while I wouldn't personally object to dropping support 
for building multiple packages from a single branch (since I don't do it 
myself), I wouldn't favor doing so, because existing packages do, and it 
would create more work for the authors to have to split them up.

Maybe it would be reasonable to make a new policy against building 
multiple packages from a single branch, while "grandfathering" in the 
existing packages that do so, if it would solve a problem for ELPA.

Thanks for your work,
Adam

Re: package-vc support for :files keyword

[Philip Kaludercic]

Adam Porter <adam@alphapapa.net> writes:

> Hi Philip, et al,
>
>> For the record, these are all the repositories in {Non,}GNU ELPA that
>> develop multiple packages in a single repository:
>> GNU ELPA:
>> http://www.dr-qubit.org/git/predictive.git
>> https://github.com/oantolin/embark
>> https://github.com/abo-abo/swiper
>> https://github.com/abo-abo/hydra
>> https://github.com/alphapapa/taxy.el.git
>> From what I understand, there is no technical necessity for this
>> mode of
>> development?  I wonder how difficult it would be to push for a
>> one-package-one-repo approach.
>
> Please note out that while `taxy' and `taxy-magit-section' are both
> developed in "taxy.el.git", they are in separate branches, so there is
> no need to build the two packages from a single set of files by
> excluding some files and then the others.

Oops, I just wrote a quick script that compared URLs but did not check
what :branch they are developed on.

> I've chosen to keep these packages in the same repo because they are
> so closely related.  I'd like to be able to keep this arrangement.

That is totally fine, would you mind sharing your setup, in case someone
else is interested in this approach as well?

> Having said that, while I wouldn't personally object to dropping
> support for building multiple packages from a single branch (since I
> don't do it myself), I wouldn't favor doing so, because existing
> packages do, and it would create more work for the authors to have to
> split them up.

That is the issue, and I certainly don't want to be the one to blame for
breakage, be it for package developers let alone users.

> Maybe it would be reasonable to make a new policy against building
> multiple packages from a single branch, while "grandfathering" in the
> existing packages that do so, if it would solve a problem for ELPA.

What do you mean by "grandfathering"?

> Thanks for your work,
> Adam

Re: package-vc support for :files keyword

[Adam Porter]

Hi Philip,

On 9/19/23 03:37, Philip Kaludercic wrote:

>> Please note out that while `taxy' and `taxy-magit-section' are both
>> developed in "taxy.el.git", they are in separate branches, so there is
>> no need to build the two packages from a single set of files by
>> excluding some files and then the others.
> 
> Oops, I just wrote a quick script that compared URLs but did not check
> what :branch they are developed on.
> 
>> I've chosen to keep these packages in the same repo because they are
>> so closely related.  I'd like to be able to keep this arrangement.
> 
> That is totally fine, would you mind sharing your setup, in case someone
> else is interested in this approach as well?

I'm not sure what you're asking for, but I'll be glad to share.  All I 
did was create an orphan branch in the same repository and add the 
"taxy-magit-section.el" and associated files there, as if it were a 
separate repo.

>> Having said that, while I wouldn't personally object to dropping
>> support for building multiple packages from a single branch (since I
>> don't do it myself), I wouldn't favor doing so, because existing
>> packages do, and it would create more work for the authors to have to
>> split them up.
> 
> That is the issue, and I certainly don't want to be the one to blame for
> breakage, be it for package developers let alone users.
> 
>> Maybe it would be reasonable to make a new policy against building
>> multiple packages from a single branch, while "grandfathering" in the
>> existing packages that do so, if it would solve a problem for ELPA.
> 
> What do you mean by "grandfathering"?

It's an expression; in this context, it would mean to allow the packages 
that already work this way to continue doing so, while requiring newly 
submitted packages to only build one package per branch (or repo, 
depending on the policy).

Re: package-vc support for :files keyword

[Philip Kaludercic]

Adam Porter <adam@alphapapa.net> writes:

> Hi Philip,
>
> On 9/19/23 03:37, Philip Kaludercic wrote:
>
>>> Please note out that while `taxy' and `taxy-magit-section' are both
>>> developed in "taxy.el.git", they are in separate branches, so there is
>>> no need to build the two packages from a single set of files by
>>> excluding some files and then the others.
>> Oops, I just wrote a quick script that compared URLs but did not
>> check
>> what :branch they are developed on.
>> 
>>> I've chosen to keep these packages in the same repo because they are
>>> so closely related.  I'd like to be able to keep this arrangement.
>> That is totally fine, would you mind sharing your setup, in case
>> someone
>> else is interested in this approach as well?
>
> I'm not sure what you're asking for, but I'll be glad to share.  All I
> did was create an orphan branch in the same repository and add the
> "taxy-magit-section.el" and associated files there, as if it were a
> separate repo.

What I meant was if you just had multiple, separate checkouts of the
same repository or were using something more fancy like git worktrees.

>>> Having said that, while I wouldn't personally object to dropping
>>> support for building multiple packages from a single branch (since I
>>> don't do it myself), I wouldn't favor doing so, because existing
>>> packages do, and it would create more work for the authors to have to
>>> split them up.
>> That is the issue, and I certainly don't want to be the one to blame
>> for
>> breakage, be it for package developers let alone users.
>> 
>>> Maybe it would be reasonable to make a new policy against building
>>> multiple packages from a single branch, while "grandfathering" in the
>>> existing packages that do so, if it would solve a problem for ELPA.
>> What do you mean by "grandfathering"?
>
> It's an expression; in this context, it would mean to allow the
> packages that already work this way to continue doing so, while
> requiring newly submitted packages to only build one package per
> branch (or repo, depending on the policy).

FWIW I have been bringing this up whenever a package like this was
proposed, but usually they refer to existing users on MELPA and then I
don't want to bother them any further.

Re: package-vc support for :files keyword

[Adam Porter]

On 9/19/23 08:56, Philip Kaludercic wrote:

>> I'm not sure what you're asking for, but I'll be glad to share.  All I
>> did was create an orphan branch in the same repository and add the
>> "taxy-magit-section.el" and associated files there, as if it were a
>> separate repo.
> 
> What I meant was if you just had multiple, separate checkouts of the
> same repository or were using something more fancy like git worktrees.

Yes, sometimes I use worktrees.  I don't have a canonical "workflow," as 
it's evolved over time and continues to.

Re: package-vc support for :files keyword

[Tony Zorman]

On Mon, Sep 18 2023 15:52, Philip Kaludercic wrote:
>> I think that
>>
>>   $ rm foo.el
>>   $ git update-index --assume-unchanged foo.el
>>
>> should work. It should merge cleanly (I've tried this out just now, and
>> it worked, but I may have overlooked something). If it's part of the
>> package description, then updating should work out of the box, since
>> package-vc-upgrade also calls package-vc--unpack-1, which would execute
>> the respective :early-shell-command again.
>
> Being a git-specific command, this shouldn't be added to package-vc
> directly.  If there is a VCS agnostic/generalisable way of doing this,
> then it could be added to VC.

Ah, sometimes I forget that there are VCs other than Git—sorry :)

> But for now, if I understand you correctly, you are suggesting that
> users give package specifications like this:
>
> (foo :url "https://some.vcs/repository"
>      ;; ...
>      :early-shell-command "rm [all the files]; git update-index --assume-unchanged [all the files]")
>
> where [all the files] might change between updates.

Yes, exactly.

> At this point I continue to question the utility of emulating
> MELPA-style :files attributes, unless there are concrete usability
> issues.
>
> For the record, these are all the repositories in {Non,}GNU ELPA that
> develop multiple packages in a single repository:
>
> [… 12 lines elided …]
>
> From what I understand, there is no technical necessity for this mode of
> development?  I wonder how difficult it would be to push for a
> one-package-one-repo approach.

This is not just for multiple packages in a single repository—at least
one has to somewhat broaden what "multiple packages" means. Some
packages include small shims for bigger projects, and inadvertently
require them as dependencies. The original issue[1] on the
vc-use-package repo mentions org-ql[2], more specifically its helm
integration in the form of helm-org-ql.el. Some people might not want to
pull down helm as a dependency just for one file that they are not going
to use anyways.

I'm not sure how common of a situation this actually is, but at least
for the big completion frameworks—helm and ivy—it's not totally unheard
of.

[1]: https://github.com/slotThe/vc-use-package/issues/12
[2]: https://github.com/alphapapa/org-ql/

-- 
Tony Zorman | https://tony-zorman.com/

Re: package-vc support for :files keyword

[Philip Kaludercic]

Tony Zorman <tonyzorman@mailbox.org> writes:

> On Mon, Sep 18 2023 15:52, Philip Kaludercic wrote:
>>> I think that
>>>
>>>   $ rm foo.el
>>>   $ git update-index --assume-unchanged foo.el
>>>
>>> should work. It should merge cleanly (I've tried this out just now, and
>>> it worked, but I may have overlooked something). If it's part of the
>>> package description, then updating should work out of the box, since
>>> package-vc-upgrade also calls package-vc--unpack-1, which would execute
>>> the respective :early-shell-command again.
>>
>> Being a git-specific command, this shouldn't be added to package-vc
>> directly.  If there is a VCS agnostic/generalisable way of doing this,
>> then it could be added to VC.
>
> Ah, sometimes I forget that there are VCs other than Git—sorry :)
>
>> But for now, if I understand you correctly, you are suggesting that
>> users give package specifications like this:
>>
>> (foo :url "https://some.vcs/repository"
>>      ;; ...
>>      :early-shell-command "rm [all the files]; git update-index --assume-unchanged [all the files]")
>>
>> where [all the files] might change between updates.
>
> Yes, exactly.
>
>> At this point I continue to question the utility of emulating
>> MELPA-style :files attributes, unless there are concrete usability
>> issues.
>>
>> For the record, these are all the repositories in {Non,}GNU ELPA that
>> develop multiple packages in a single repository:
>>
>> [… 12 lines elided …]
>>
>> From what I understand, there is no technical necessity for this mode of
>> development?  I wonder how difficult it would be to push for a
>> one-package-one-repo approach.
>
> This is not just for multiple packages in a single repository—at least
> one has to somewhat broaden what "multiple packages" means. Some
> packages include small shims for bigger projects, and inadvertently
> require them as dependencies. The original issue[1] on the
> vc-use-package repo mentions org-ql[2], more specifically its helm
> integration in the form of helm-org-ql.el. Some people might not want to
> pull down helm as a dependency just for one file that they are not going
> to use anyways.
>
> I'm not sure how common of a situation this actually is, but at least
> for the big completion frameworks—helm and ivy—it's not totally unheard
> of.

Hmm, this is interesting example that I was not familiar with.  As an
alternative idea, do you think that using `:ignored-files' like
elpa-admin.el could be useful?  You could exclude all the files with
"soft-dependencies", that wouldn't be scraped in `package-vc--unpack-1'
when looking for dependency files.

Alternatively, since I am still not convinced that having
:early-shell-command built-in to package-vc is a good idea, one could
scatter a number of hooks around the code that could be used to extend
installation procedure with additional functionality, like a
:early-shell-command.

> [1]: https://github.com/slotThe/vc-use-package/issues/12
> [2]: https://github.com/alphapapa/org-ql/

Re: package-vc support for :files keyword

[Adam Porter]

FWIW, it seems like a bit of an anti-pattern for shell commands to be 
part of Elisp packages' recipes.  I can see how it may be necessary for 
certain ones that support external software that may need to build a 
module, etc, but it seems like that sort of thing should be kept to a 
minimum due to, e.g. potential security issues with executing code at 
install time.  Or am I missing something?  :)

--Adam

Re: package-vc support for :files keyword

[Philip Kaludercic]

Adam Porter <adam@alphapapa.net> writes:

> FWIW, it seems like a bit of an anti-pattern for shell commands to be
> part of Elisp packages' recipes.  I can see how it may be necessary
> for certain ones that support external software that may need to build
> a module, etc, but it seems like that sort of thing should be kept to
> a minimum due to, e.g. potential security issues with executing code
> at install time.  Or am I missing something?  :)

It usually is a hack, and it isn't enabled by default either.  Emacs 30
has a special user option to regulate it.  I certainly don't recommend
using it, and am not that enthusiastic about promoting it.

> --Adam

Re: package-vc support for :files keyword

[Tony Zorman]

On Tue, Sep 19 2023 08:47, Philip Kaludercic wrote:
> Tony Zorman <tonyzorman@mailbox.org> writes:
>
> [… 42 lines elided …]
>
>> This is not just for multiple packages in a single repository—at least
>> one has to somewhat broaden what "multiple packages" means. Some
>> packages include small shims for bigger projects, and inadvertently
>> require them as dependencies. The original issue[1] on the
>> vc-use-package repo mentions org-ql[2], more specifically its helm
>> integration in the form of helm-org-ql.el. Some people might not want to
>> pull down helm as a dependency just for one file that they are not going
>> to use anyways.
>>
>> I'm not sure how common of a situation this actually is, but at least
>> for the big completion frameworks—helm and ivy—it's not totally unheard
>> of.
>
> Hmm, this is interesting example that I was not familiar with.  As an
> alternative idea, do you think that using `:ignored-files' like
> elpa-admin.el could be useful?  You could exclude all the files with
> "soft-dependencies", that wouldn't be scraped in `package-vc--unpack-1'
> when looking for dependency files.

Overall, I think this would be the better (even best) approach, yes.

(I also consider executing external shell commands during build-time an
anti pattern, I just got the feeling that making package-vc support
`:ignored-files' wasn't on the table—glad to realise that I was wrong!)

-- 
Tony Zorman | https://tony-zorman.com/

Re: package-vc support for :files keyword

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

Tony Zorman <tonyzorman@mailbox.org> writes:

> On Tue, Sep 19 2023 08:47, Philip Kaludercic wrote:
>> Tony Zorman <tonyzorman@mailbox.org> writes:
>>
>> [… 42 lines elided …]
>>
>>> This is not just for multiple packages in a single repository—at least
>>> one has to somewhat broaden what "multiple packages" means. Some
>>> packages include small shims for bigger projects, and inadvertently
>>> require them as dependencies. The original issue[1] on the
>>> vc-use-package repo mentions org-ql[2], more specifically its helm
>>> integration in the form of helm-org-ql.el. Some people might not want to
>>> pull down helm as a dependency just for one file that they are not going
>>> to use anyways.
>>>
>>> I'm not sure how common of a situation this actually is, but at least
>>> for the big completion frameworks—helm and ivy—it's not totally unheard
>>> of.
>>
>> Hmm, this is interesting example that I was not familiar with.  As an
>> alternative idea, do you think that using `:ignored-files' like
>> elpa-admin.el could be useful?  You could exclude all the files with
>> "soft-dependencies", that wouldn't be scraped in `package-vc--unpack-1'
>> when looking for dependency files.
>
> Overall, I think this would be the better (even best) approach, yes.

Here is a sketch of how that could look like.  Can you test it to see if
it works:


[-- Attachment #2: Type: text/plain, Size: 3169 bytes --]

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index a8393cb7e75..07e660d9f33 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -213,7 +213,7 @@ package-vc--desc->spec
 name for PKG-DESC."
   (alist-get
    (setq name (or name (package-desc-name pkg-desc)))
-   (if (and (package-desc-archive pkg-desc)
+   (if (and pkg-desc (package-desc-archive pkg-desc)
             (not (alist-get name package-vc-selected-packages
                             nil nil #'string=)))
        (alist-get (intern (package-desc-archive pkg-desc))
@@ -501,7 +501,8 @@ package-vc--unpack-1
 autoloads, generating a package description file (used to
 identify a package as a VC package later on), building
 documentation and marking the package as installed."
-  (let (missing)
+  (let ((pkg-spec (package-vc--desc->spec pkg-desc))
+        missing)
     ;; Remove any previous instance of PKG-DESC from `package-alist'
     (let ((pkgs (assq (package-desc-name pkg-desc) package-alist)))
       (when pkgs
@@ -510,17 +511,27 @@ package-vc--unpack-1
     ;; In case the package was installed directly from source, the
     ;; dependency list wasn't know beforehand, and they might have
     ;; to be installed explicitly.
-    (let ((deps '()))
+    (let ((ignored-files
+           (mapconcat
+            (lambda (ignore)
+              (wildcard-to-regexp
+               (if (string-match-p "\\`/" ignore)
+                   (concat pkg-dir ignore)
+                 (concat "*/" ignore))))
+            (plist-get pkg-spec :ignored-files)
+            "\\|"))
+          (deps '()))
       (dolist (file (directory-files pkg-dir t "\\.el\\'" t))
-        (with-temp-buffer
-          (insert-file-contents file)
-          (when-let* ((require-lines (lm-header-multiline "package-requires")))
-            (thread-last
-              (mapconcat #'identity require-lines " ")
-              package-read-from-string
-              package--prepare-dependencies
-              (nconc deps)
-              (setq deps)))))
+        (unless (string-match-p ignored-files file)
+          (with-temp-buffer
+            (insert-file-contents file)
+            (when-let* ((require-lines (lm-header-multiline "package-requires")))
+              (thread-last
+                (mapconcat #'identity require-lines " ")
+                package-read-from-string
+                package--prepare-dependencies
+                (nconc deps)
+                (setq deps))))))
       (dolist (dep deps)
         (cl-callf version-to-list (cadr dep)))
       (setf missing (package-vc-install-dependencies (delete-dups deps)))
@@ -529,8 +540,7 @@ package-vc--unpack-1
                           missing)))
 
     (let ((default-directory (file-name-as-directory pkg-dir))
-          (pkg-file (expand-file-name (package--description-file pkg-dir) pkg-dir))
-          (pkg-spec (package-vc--desc->spec pkg-desc)))
+          (pkg-file (expand-file-name (package--description-file pkg-dir) pkg-dir)))
       ;; Generate autoloads
       (let* ((name (package-desc-name pkg-desc))
              (auto-name (format "%s-autoloads.el" name))

[-- Attachment #3: Type: text/plain, Size: 250 bytes --]


> (I also consider executing external shell commands during build-time an
> anti pattern, I just got the feeling that making package-vc support
> `:ignored-files' wasn't on the table—glad to realise that I was wrong!)

Why did you think so?

Re: package-vc support for :files keyword

[Tony Zorman]

On Wed, Sep 20 2023 07:32, Philip Kaludercic wrote:
> Here is a sketch of how that could look like.  Can you test it to see if
> it works:
>
> [… 71 lines elided …]

Yup, works beautifully; thanks!

>> (I also consider executing external shell commands during build-time an
>> anti pattern, I just got the feeling that making package-vc support
>> `:ignored-files' wasn't on the table—glad to realise that I was wrong!)
>
> Why did you think so?

I perhaps read your first email[1] more negatively than you intended it
to be.

[1]: https://lists.gnu.org/archive/html/emacs-devel/2023-06/msg00247.html

-- 
Tony Zorman | https://tony-zorman.com/

Re: package-vc support for :files keyword

[Philip Kaludercic]

Tony Zorman <tonyzorman@mailbox.org> writes:

> On Wed, Sep 20 2023 07:32, Philip Kaludercic wrote:
>> Here is a sketch of how that could look like.  Can you test it to see if
>> it works:
>>
>> [… 71 lines elided …]
>
> Yup, works beautifully; thanks!

OK, I'll convert this into a proper patch and update the documentation.

>>> (I also consider executing external shell commands during build-time an
>>> anti pattern, I just got the feeling that making package-vc support
>>> `:ignored-files' wasn't on the table—glad to realise that I was wrong!)
>>
>> Why did you think so?
>
> I perhaps read your first email[1] more negatively than you intended it
> to be.

My point there was that using :ignored-files as a ":files" substitute
won't work, but I understand how my statement there could be misread.
Sorry about that.

> [1]: https://lists.gnu.org/archive/html/emacs-devel/2023-06/msg00247.html

Re: package-vc support for :files keyword

[Philip Kaludercic]

Tony Zorman <tonyzorman@mailbox.org> writes:

> On Wed, Sep 20 2023 07:32, Philip Kaludercic wrote:
>> +        (unless (string-match-p ignored-files file)
>
> One thing that did jump out to me just now, which I hadn't considered
> before: when ignored-files is the empty string, this will always match
> any file—not what we want :)

Do you mean because the empty glob expression doesn't match everything,
but the empty regular expression does?

The easiest solution seems to be to just check if the empty string is
listed and handle that separately (whatever the correct behaviour is in
that case), but on the other hand, why should a user list an empty
string in the list of ignored files?

-- 
Philip Kaludercic

Re: package-vc support for :files keyword

[Tony Zorman]

On Sun, Sep 24 2023 16:31, Philip Kaludercic wrote:
> Tony Zorman <tonyzorman@mailbox.org> writes:
>
>> On Wed, Sep 20 2023 07:32, Philip Kaludercic wrote:
>>> +        (unless (string-match-p ignored-files file)
>>
>> One thing that did jump out to me just now, which I hadn't considered
>> before: when ignored-files is the empty string, this will always match
>> any file—not what we want :)
>
> Do you mean because the empty glob expression doesn't match everything,
> but the empty regular expression does?
>
> The easiest solution seems to be to just check if the empty string is
> listed and handle that separately (whatever the correct behaviour is in
> that case), but on the other hand, why should a user list an empty
> string in the list of ignored files?

The point I was making was that `(mapconcat (λ…) nil) ≡ ""`, since
mapconcat always returns a string. Even if there were no :ignored-files
listed, the ignored-files variable would then be the empty string,
matching each and every file.

-- 
Tony Zorman | https://tony-zorman.com/

Re: package-vc support for :files keyword

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

Tony Zorman <tonyzorman@mailbox.org> writes:

> On Sun, Sep 24 2023 16:31, Philip Kaludercic wrote:
>> Tony Zorman <tonyzorman@mailbox.org> writes:
>>
>>> On Wed, Sep 20 2023 07:32, Philip Kaludercic wrote:
>>>> +        (unless (string-match-p ignored-files file)
>>>
>>> One thing that did jump out to me just now, which I hadn't considered
>>> before: when ignored-files is the empty string, this will always match
>>> any file—not what we want :)
>>
>> Do you mean because the empty glob expression doesn't match everything,
>> but the empty regular expression does?
>>
>> The easiest solution seems to be to just check if the empty string is
>> listed and handle that separately (whatever the correct behaviour is in
>> that case), but on the other hand, why should a user list an empty
>> string in the list of ignored files?
>
> The point I was making was that `(mapconcat (λ…) nil) ≡ ""`, since
> mapconcat always returns a string. Even if there were no :ignored-files
> listed, the ignored-files variable would then be the empty string,
> matching each and every file.

Oops, right, in that case this should be preferable:


[-- Attachment #2: Type: text/plain, Size: 3287 bytes --]

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index a8393cb7e75..d100e0bcb58 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -213,7 +213,7 @@ package-vc--desc->spec
 name for PKG-DESC."
   (alist-get
    (setq name (or name (package-desc-name pkg-desc)))
-   (if (and (package-desc-archive pkg-desc)
+   (if (and pkg-desc (package-desc-archive pkg-desc)
             (not (alist-get name package-vc-selected-packages
                             nil nil #'string=)))
        (alist-get (intern (package-desc-archive pkg-desc))
@@ -501,7 +501,8 @@ package-vc--unpack-1
 autoloads, generating a package description file (used to
 identify a package as a VC package later on), building
 documentation and marking the package as installed."
-  (let (missing)
+  (let ((pkg-spec (package-vc--desc->spec pkg-desc))
+        missing)
     ;; Remove any previous instance of PKG-DESC from `package-alist'
     (let ((pkgs (assq (package-desc-name pkg-desc) package-alist)))
       (when pkgs
@@ -510,17 +511,29 @@ package-vc--unpack-1
     ;; In case the package was installed directly from source, the
     ;; dependency list wasn't know beforehand, and they might have
     ;; to be installed explicitly.
-    (let ((deps '()))
+    (let ((ignored-files
+           (if (plist-get pkg-spec :ignored-files)
+               (mapconcat
+                (lambda (ignore)
+                  (wildcard-to-regexp
+                   (if (string-match-p "\\`/" ignore)
+                       (concat pkg-dir ignore)
+                     (concat "*/" ignore))))
+                (plist-get pkg-spec :ignored-files)
+                "\\|")
+             regexp-unmatchable))
+          (deps '()))
       (dolist (file (directory-files pkg-dir t "\\.el\\'" t))
-        (with-temp-buffer
-          (insert-file-contents file)
-          (when-let* ((require-lines (lm-header-multiline "package-requires")))
-            (thread-last
-              (mapconcat #'identity require-lines " ")
-              package-read-from-string
-              package--prepare-dependencies
-              (nconc deps)
-              (setq deps)))))
+        (unless (string-match-p ignored-files file)
+          (with-temp-buffer
+            (insert-file-contents file)
+            (when-let* ((require-lines (lm-header-multiline "package-requires")))
+              (thread-last
+                (mapconcat #'identity require-lines " ")
+                package-read-from-string
+                package--prepare-dependencies
+                (nconc deps)
+                (setq deps))))))
       (dolist (dep deps)
         (cl-callf version-to-list (cadr dep)))
       (setf missing (package-vc-install-dependencies (delete-dups deps)))
@@ -529,8 +542,7 @@ package-vc--unpack-1
                           missing)))
 
     (let ((default-directory (file-name-as-directory pkg-dir))
-          (pkg-file (expand-file-name (package--description-file pkg-dir) pkg-dir))
-          (pkg-spec (package-vc--desc->spec pkg-desc)))
+          (pkg-file (expand-file-name (package--description-file pkg-dir) pkg-dir)))
       ;; Generate autoloads
       (let* ((name (package-desc-name pkg-desc))
              (auto-name (format "%s-autoloads.el" name))

Re: package-vc support for :files keyword

[Tony Zorman]

On Wed, Sep 27 2023 14:03, Philip Kaludercic wrote:
> Oops, right, in that case this should be preferable:
>
> [… 73 lines elided …]

Works great, thanks!

-- 
Tony Zorman | https://tony-zorman.com/

Re: package-vc support for :files keyword

[Jonas Bernoulli]

Tony Zorman <tonyzorman@mailbox.org> writes:

>> From what I understand, there is no technical necessity for this mode of
>> development?  I wonder how difficult it would be to push for a
>> one-package-one-repo approach.
>
> This is not just for multiple packages in a single repository—at least
> one has to somewhat broaden what "multiple packages" means. Some
> packages include small shims for bigger projects, and inadvertently
> require them as dependencies. The original issue[1] on the
> vc-use-package repo mentions org-ql[2], more specifically its helm
> integration in the form of helm-org-ql.el. Some people might not want to
> pull down helm as a dependency just for one file that they are not going
> to use anyways.
>
> I'm not sure how common of a situation this actually is, but at least
> for the big completion frameworks—helm and ivy—it's not totally unheard
> of.

Here's a complete list for all of these packages that are available on
Melpa.  Obviously not all of these pairings fall into the "foo and
helm-foo share a repository" category, but you can get an idea of what
other reasons exist for splitting a repository into multiple packages,
based on the names of the packages/libraries.  I have included links to
the repositories, so you can quickly jump there, when only looking at
the names is not enough.

| Primary package (141)   | Secondary package               | Repository                                                |
|-------------------------+---------------------------------+----------------------------------------------------------------|
| 750words                | ox-750words                     | https://github.com/zzamboni/750words-client                    |
| ac-php                  | ac-php-core                     | https://github.com/xcwen/ac-php                                |
| ac-php                  | company-php                     | https://github.com/xcwen/ac-php                                |
| agda2-mode              | annotation                      | https://github.com/agda/agda                                   |
| agda2-mode              | eri                             | https://github.com/agda/agda                                   |
| ameba                   | flycheck-ameba                  | https://github.com/crystal-ameba/ameba.el                      |
| axiom-environment       | company-axiom                   | https://bitbucket.org/pdo/axiom-environment                    |
| axiom-environment       | ob-axiom                        | https://bitbucket.org/pdo/axiom-environment                    |
| biblio                  | biblio-core                     | https://github.com/cpitclaudel/biblio.el                       |
| bibtex-completion       | helm-bibtex                     | https://github.com/tmalsburg/helm-bibtex                       |
| bibtex-completion       | ivy-bibtex                      | https://github.com/tmalsburg/helm-bibtex                       |
| boa-ide                 | boa-mode                        | https://github.com/boalang/syntax-highlight                    |
| bufler                  | helm-bufler                     | https://github.com/alphapapa/bufler.el                         |
| button-lock             | wiki-nav                        | https://github.com/rolandwalker/button-lock                    |
| calfw                   | calfw-cal                       | https://github.com/kiwanami/emacs-calfw                        |
| calfw                   | calfw-howm                      | https://github.com/kiwanami/emacs-calfw                        |
| calfw                   | calfw-ical                      | https://github.com/kiwanami/emacs-calfw                        |
| calfw                   | calfw-org                       | https://github.com/kiwanami/emacs-calfw                        |
| cardano-tx              | bech32                          | https://github.com/Titan-C/cardano.el                          |
| cardano-tx              | cardano-wallet                  | https://github.com/Titan-C/cardano.el                          |
| cardano-tx              | cbor                            | https://github.com/Titan-C/cardano.el                          |
| cardano-tx              | ouroboros                       | https://github.com/Titan-C/cardano.el                          |
| cardano-tx              | readable-numbers                | https://github.com/Titan-C/cardano.el                          |
| cfn-mode                | flycheck-cfn                    | https://gitlab.com/worr/cfn-mode                               |
| chatgpt-shell           | dall-e-shell                    | https://github.com/xenodium/chatgpt-shell                      |
| chatgpt-shell           | ob-chatgpt-shell                | https://github.com/xenodium/chatgpt-shell                      |
| chatgpt-shell           | ob-dall-e-shell                 | https://github.com/xenodium/chatgpt-shell                      |
| chatgpt-shell           | shell-maker                     | https://github.com/xenodium/chatgpt-shell                      |
| chronometrist           | chronometrist-key-values        | https://codeberg.org/contrapunctus/chronometrist               |
| chronometrist           | chronometrist-spark             | https://codeberg.org/contrapunctus/chronometrist               |
| circe                   | tracking                        | https://github.com/emacs-circe/circe                           |
| citar                   | citar-embark                    | https://github.com/emacs-citar/citar                           |
| clojure-essential-ref   | clojure-essential-ref-nov       | https://github.com/p3r7/clojure-essential-ref                  |
| clojure-mode            | clojure-mode-extra-font-locking | https://github.com/clojure-emacs/clojure-mode                  |
| codesearch              | projectile-codesearch           | https://github.com/abingham/emacs-codesearch                   |
| colorless-themes        | lavenderless-theme              | https://git.sr.ht/~lthms/colorless-themes.el                   |
| colorless-themes        | nordless-theme                  | https://git.sr.ht/~lthms/colorless-themes.el                   |
| compile-multi           | compile-multi-all-the-icons     | https://github.com/mohkale/compile-multi                       |
| compile-multi           | compile-multi-embark            | https://github.com/mohkale/compile-multi                       |
| compile-multi           | consult-compile-multi           | https://github.com/mohkale/compile-multi                       |
| copyit                  | copyit-pandoc                   | https://github.com/zonuexe/emacs-copyit                        |
| crystal-mode            | flycheck-crystal                | https://github.com/crystal-lang-tools/emacs-crystal-mode       |
| ctl-mode                | gs-mode                         | https://github.com/yyr/emacs-grads                             |
| dash                    | dash-functional                 | https://github.com/magnars/dash.el                             |
| datetime-format         | timecop                         | https://github.com/emacs-php/emacs-datetime                    |
| ddskk                   | ccc                             | https://github.com/skk-dev/ddskk                               |
| ddskk                   | cdb                             | https://github.com/skk-dev/ddskk                               |
| deferred                | concurrent                      | https://github.com/kiwanami/emacs-deferred                     |
| dired-hacks             | dired-avfs                      | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-collapse                  | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-filter                    | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-hacks-utils               | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-list                      | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-narrow                    | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-open                      | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-rainbow                   | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-ranger                    | https://github.com/Fuco1/dired-hacks                           |
| dired-hacks             | dired-subtree                   | https://github.com/Fuco1/dired-hacks                           |
| dired-rsync             | dired-rsync-transient           | https://github.com/stsquad/dired-rsync                         |
| distel-completion-lib   | auto-complete-distel            | https://github.com/sebastiw/distel-completion                  |
| distel-completion-lib   | company-distel                  | https://github.com/sebastiw/distel-completion                  |
| dix                     | dix-evil                        | https://github.com/unhammer/dix                                |
| dogears                 | helm-dogears                    | https://github.com/alphapapa/dogears.el                        |
| dylan                   | dime                            | https://github.com/dylan-lang/dylan-emacs-support              |
| eclim                   | ac-emacs-eclim                  | https://github.com/emacs-eclim/emacs-eclim                     |
| eclim                   | company-emacs-eclim             | https://github.com/emacs-eclim/emacs-eclim                     |
| el-secretario           | el-secretario-elfeed            | https://git.sr.ht/~zetagon/el-secretario                       |
| el-secretario           | el-secretario-mu4e              | https://git.sr.ht/~zetagon/el-secretario                       |
| el-secretario           | el-secretario-notmuch           | https://git.sr.ht/~zetagon/el-secretario                       |
| el-secretario           | el-secretario-org               | https://git.sr.ht/~zetagon/el-secretario                       |
| elfeed                  | elfeed-web                      | https://github.com/skeeto/elfeed                               |
| elfeed-tube             | elfeed-tube-mpv                 | https://github.com/karthink/elfeed-tube                        |
| emacsql                 | emacsql-mysql                   | https://github.com/magit/emacsql                               |
| emacsql                 | emacsql-pg                      | https://github.com/magit/emacsql                               |
| emacsql                 | emacsql-psql                    | https://github.com/magit/emacsql                               |
| emacsql                 | emacsql-sqlite                  | https://github.com/magit/emacsql                               |
| emacsql                 | emacsql-sqlite-builtin          | https://github.com/magit/emacsql                               |
| emacsql                 | emacsql-sqlite-module           | https://github.com/magit/emacsql                               |
| embark                  | avy-embark-collect              | https://github.com/oantolin/embark                             |
| embark                  | embark-consult                  | https://github.com/oantolin/embark                             |
| espotify                | consult-spotify                 | https://codeberg.org/jao/espotify                              |
| espotify                | ivy-spotify                     | https://codeberg.org/jao/espotify                              |
| esqlite                 | esqlite-helm                    | https://github.com/mhayashi1120/Emacs-esqlite                  |
| evil                    | evil-test-helpers               | https://github.com/emacs-evil/evil                             |
| ewal                    | ewal-doom-themes                | https://github.com/cyruseuros/ewal                             |
| ewal                    | ewal-evil-cursors               | https://github.com/cyruseuros/ewal                             |
| ewal                    | ewal-spacemacs-themes           | https://github.com/cyruseuros/ewal                             |
| flutter                 | flutter-l10n-flycheck           | https://github.com/amake/flutter.el                            |
| flx                     | flx-ido                         | https://github.com/lewang/flx                                  |
| flycheck-aspell         | flymake-aspell                  | https://github.com/leotaku/flycheck-aspell                     |
| flyspell-correct        | flyspell-correct-avy-menu       | https://github.com/d12frosted/flyspell-correct                 |
| flyspell-correct        | flyspell-correct-helm           | https://github.com/d12frosted/flyspell-correct                 |
| flyspell-correct        | flyspell-correct-ivy            | https://github.com/d12frosted/flyspell-correct                 |
| flyspell-correct        | flyspell-correct-popup          | https://github.com/d12frosted/flyspell-correct                 |
| friendly-shell          | friendly-remote-shell           | https://github.com/p3r7/friendly-shell                         |
| friendly-shell          | friendly-shell-command          | https://github.com/p3r7/friendly-shell                         |
| frimacs                 | ob-fricas                       | https://github.com/pdo/frimacs                                 |
| fsharp-mode             | eglot-fsharp                    | https://github.com/fsharp/emacs-fsharp-mode                    |
| fuz                     | helm-fuz                        | https://github.com/rustify-emacs/fuz.el                        |
| git-emacs               | git-blamed                      | https://github.com/tsgates/git-emacs                           |
| gitlab                  | helm-gitlab                     | https://github.com/nlamirault/emacs-gitlab                     |
| gitlab                  | ivy-gitlab                      | https://github.com/nlamirault/emacs-gitlab                     |
| go-mode                 | go-guru                         | https://github.com/dominikh/go-mode.el                         |
| go-mode                 | go-rename                       | https://github.com/dominikh/go-mode.el                         |
| groovy-mode             | grails-mode                     | https://github.com/Groovy-Emacs-Modes/groovy-emacs-modes       |
| haskell-emacs           | haskell-emacs-base              | https://github.com/knupfer/haskell-emacs                       |
| haskell-emacs           | haskell-emacs-text              | https://github.com/knupfer/haskell-emacs                       |
| helm                    | helm-core                       | https://github.com/emacs-helm/helm                             |
| historian               | ivy-historian                   | https://github.com/PythonNut/historian.el                      |
| hydra                   | lv                              | https://github.com/abo-abo/hydra                               |
| indent-lint             | flycheck-indent                 | https://github.com/conao3/indent-lint.el                       |
| islisp-mode             | inferior-islisp                 | https://gitlab.com/sasanidas/islisp-mode                       |
| ivy                     | counsel                         | https://github.com/abo-abo/swiper                              |
| ivy                     | ivy-avy                         | https://github.com/abo-abo/swiper                              |
| ivy                     | ivy-hydra                       | https://github.com/abo-abo/swiper                              |
| ivy                     | swiper                          | https://github.com/abo-abo/swiper                              |
| jade-mode               | stylus-mode                     | https://github.com/brianc/jade-mode                            |
| jade-mode               | sws-mode                        | https://github.com/brianc/jade-mode                            |
| jedi                    | jedi-core                       | https://github.com/tkf/emacs-jedi                              |
| jump                    | inflections                     | https://github.com/eschulte/jump.el                            |
| kaesar                  | kaesar-file                     | https://github.com/mhayashi1120/Emacs-kaesar                   |
| kaesar                  | kaesar-mode                     | https://github.com/mhayashi1120/Emacs-kaesar                   |
| kaesar                  | kaesar-pbkdf2                   | https://github.com/mhayashi1120/Emacs-kaesar                   |
| kaleidoscope            | kaleidoscope-evil-state-flash   | https://git.madhouse-project.org/algernon/kaleidoscope.el      |
| keg                     | flycheck-keg                    | https://github.com/conao3/keg.el                               |
| keg                     | keg-mode                        | https://github.com/conao3/keg.el                               |
| kubel                   | kubel-evil                      | https://github.com/abrochard/kubel                             |
| kubernetes              | kubernetes-evil                 | https://github.com/kubernetes-el/kubernetes-el                 |
| langtool                | langtool-popup                  | https://github.com/mhayashi1120/Emacs-langtool                 |
| lean-mode               | company-lean                    | https://github.com/leanprover/lean-mode                        |
| lean-mode               | helm-lean                       | https://github.com/leanprover/lean-mode                        |
| load-theme-buffer-local | color-theme-buffer-local        | https://github.com/vic/color-theme-buffer-local                |
| loopy                   | loopy-dash                      | https://github.com/okamsn/loopy                                |
| mac-pseudo-daemon       | osx-pseudo-daemon               | https://github.com/DarwinAwardWinner/mac-pseudo-daemon         |
| magit                   | git-commit                      | https://github.com/magit/magit                                 |
| magit                   | magit-section                   | https://github.com/magit/magit                                 |
| major-mode-hydra        | pretty-hydra                    | https://github.com/jerrypnz/major-mode-hydra.el                |
| maxima                  | company-maxima                  | https://gitlab.com/sasanidas/maxima                            |
| media-progress          | media-progress-dired            | https://github.com/jumper047/media-progress                    |
| media-progress          | media-progress-dirvish          | https://github.com/jumper047/media-progress                    |
| merlin                  | merlin-ac                       | https://github.com/ocaml/merlin                                |
| merlin                  | merlin-company                  | https://github.com/ocaml/merlin                                |
| merlin                  | merlin-iedit                    | https://github.com/ocaml/merlin                                |
| metal-archives          | metal-archives-shopping-list    | https://github.com/seblemaguer/metal-archives.el               |
| monorepo-cadadr         | bibliothek                      | https://github.com/cadadr/elisp                                |
| monorepo-cadadr         | forecast                        | https://github.com/cadadr/elisp                                |
| monorepo-cadadr         | org-variable-pitch              | https://github.com/cadadr/elisp                                |
| monorepo-cadadr         | paper-theme                     | https://github.com/cadadr/elisp                                |
| monorepo-zenitani       | smart-compile                   | https://github.com/zenitani/elisp                              |
| musicbrainz             | listenbrainz                    | https://github.com/zzkt/metabrainz                             |
| nameframe               | nameframe-perspective           | https://github.com/john2x/nameframe                            |
| nameframe               | nameframe-project               | https://github.com/john2x/nameframe                            |
| nameframe               | nameframe-projectile            | https://github.com/john2x/nameframe                            |
| nand2tetris             | company-nand2tetris             | https://github.com/CestDiego/nand2tetris.el                    |
| nand2tetris             | nand2tetris-assembler           | https://github.com/CestDiego/nand2tetris.el                    |
| native-complete         | company-native-complete         | https://github.com/CeleritasCelery/emacs-native-shell-complete |
| nixos-options           | company-nixos-options           | https://github.com/travisbhartwell/nix-emacs                   |
| nixos-options           | helm-nixos-options              | https://github.com/travisbhartwell/nix-emacs                   |
| nixos-options           | nix-sandbox                     | https://github.com/travisbhartwell/nix-emacs                   |
| opener                  | evil-opener                     | https://github.com/0robustus1/opener.el                        |
| org-multi-wiki          | helm-org-multi-wiki             | https://github.com/akirak/org-multi-wiki                       |
| org-pdftools            | org-noter-pdftools              | https://github.com/fuxialexander/org-pdftools                  |
| org-project-capture     | org-category-capture            | https://github.com/colonelpanic8/org-project-capture           |
| org-project-capture     | org-projectile                  | https://github.com/colonelpanic8/org-project-capture           |
| org-project-capture     | org-projectile-helm             | https://github.com/colonelpanic8/org-project-capture           |
| org-ql                  | helm-org-ql                     | https://github.com/alphapapa/org-ql                            |
| org-recent-headings     | helm-org-recent-headings        | https://github.com/alphapapa/org-recent-headings               |
| org-roam-ql             | org-roam-ql-ql                  | https://github.com/ahmed-shariff/org-roam-ql                   |
| org-starter             | org-starter-swiper              | https://github.com/akirak/org-starter                          |
| org2blog                | metaweblog                      | https://github.com/org2blog/org2blog                           |
| ox-twiki                | ox-tiddly                       | https://github.com/dfeich/org8-wikiexporters                   |
| package-lint            | package-lint-flymake            | https://github.com/purcell/package-lint                        |
| pest-mode               | flycheck-pest                   | https://github.com/ksqsf/pest-mode                             |
| pest-mode               | flymake-pest                    | https://github.com/ksqsf/pest-mode                             |
| phpactor                | company-phpactor                | https://github.com/emacs-php/phpactor.el                       |
| phpstan                 | flycheck-phpstan                | https://github.com/emacs-php/phpstan.el                        |
| phpstan                 | flymake-phpstan                 | https://github.com/emacs-php/phpstan.el                        |
| pig-mode                | pig-snippets                    | https://github.com/motus/pig-mode                              |
| pollen-mode             | company-pollen                  | https://github.com/lijunsong/pollen-mode                       |
| prescient               | company-prescient               | https://github.com/radian-software/prescient.el                |
| prescient               | corfu-prescient                 | https://github.com/radian-software/prescient.el                |
| prescient               | ivy-prescient                   | https://github.com/radian-software/prescient.el                |
| prescient               | selectrum-prescient             | https://github.com/radian-software/prescient.el                |
| prescient               | vertico-prescient               | https://github.com/radian-software/prescient.el                |
| processing-mode         | processing-snippets             | https://github.com/ptrv/processing2-emacs                      |
| projection              | projection-multi                | https://github.com/mohkale/projection                          |
| prosjekt                | dirtree-prosjekt                | https://github.com/abingham/prosjekt                           |
| prosjekt                | helm-prosjekt                   | https://github.com/abingham/prosjekt                           |
| psalm                   | flycheck-psalm                  | https://github.com/emacs-php/psalm.el                          |
| request                 | request-deferred                | https://github.com/tkf/emacs-request                           |
| restclient              | restclient-helm                 | https://github.com/pashky/restclient.el                        |
| restclient              | restclient-jq                   | https://github.com/pashky/restclient.el                        |
| rinari                  | ruby-compilation                | https://github.com/eschulte/rinari                             |
| ripgrep                 | projectile-ripgrep              | https://github.com/nlamirault/ripgrep.el                       |
| rtags                   | ac-rtags                        | https://github.com/Andersbakken/rtags                          |
| rtags                   | company-rtags                   | https://github.com/Andersbakken/rtags                          |
| rtags                   | flycheck-rtags                  | https://github.com/Andersbakken/rtags                          |
| rtags                   | helm-rtags                      | https://github.com/Andersbakken/rtags                          |
| rtags                   | ivy-rtags                       | https://github.com/Andersbakken/rtags                          |
| rtags                   | rtags-xref                      | https://github.com/Andersbakken/rtags                          |
| semaphore               | semaphore-promise               | https://github.com/webnf/semaphore.el                          |
| sift                    | projectile-sift                 | https://github.com/nlamirault/sift.el                          |
| smart-mode-line         | smart-mode-line-powerline-theme | https://github.com/Malabarba/smart-mode-line                   |
| solidity-mode           | company-solidity                | https://github.com/ethereum/emacs-solidity                     |
| solidity-mode           | solidity-flycheck               | https://github.com/ethereum/emacs-solidity                     |
| sourcekit               | company-sourcekit               | https://github.com/nathankot/company-sourcekit                 |
| speechd                 | speechd-el                      | https://github.com/brailcom/speechd-el                         |
| springboard             | ido-springboard                 | https://github.com/jwiegley/springboard                        |
| stan-mode               | company-stan                    | https://github.com/stan-dev/stan-mode                          |
| stan-mode               | eldoc-stan                      | https://github.com/stan-dev/stan-mode                          |
| stan-mode               | flycheck-stan                   | https://github.com/stan-dev/stan-mode                          |
| stan-mode               | stan-snippets                   | https://github.com/stan-dev/stan-mode                          |
| term-manager            | term-project                    | https://github.com/colonelpanic8/term-manager                  |
| term-manager            | term-projectile                 | https://github.com/colonelpanic8/term-manager                  |
| tern                    | tern-auto-complete              | https://github.com/ternjs/tern                                 |
| tmsu                    | ol-tmsu                         | https://github.com/vifon/tmsu.el                               |
| transient-extras        | transient-extras-a2ps           | https://github.com/haji-ali/transient-extras                   |
| transient-extras        | transient-extras-lp             | https://github.com/haji-ali/transient-extras                   |
| tree-edit               | evil-tree-edit                  | https://github.com/ethan-leba/tree-edit                        |
| tree-sitter             | tsc                             | https://github.com/emacs-tree-sitter/elisp-tree-sitter         |
| treemacs                | treemacs-all-the-icons          | https://github.com/Alexander-Miller/treemacs                   |
| treemacs                | treemacs-evil                   | https://github.com/Alexander-Miller/treemacs                   |
| treemacs                | treemacs-icons-dired            | https://github.com/Alexander-Miller/treemacs                   |
| treemacs                | treemacs-magit                  | https://github.com/Alexander-Miller/treemacs                   |
| treemacs                | treemacs-persp                  | https://github.com/Alexander-Miller/treemacs                   |
| treemacs                | treemacs-perspective            | https://github.com/Alexander-Miller/treemacs                   |
| treemacs                | treemacs-projectile             | https://github.com/Alexander-Miller/treemacs                   |
| treemacs                | treemacs-tab-bar                | https://github.com/Alexander-Miller/treemacs                   |
| vdm-mode                | flycheck-vdm                    | https://github.com/peterwvj/vdm-mode                           |
| vdm-mode                | vdm-comint                      | https://github.com/peterwvj/vdm-mode                           |
| vdm-mode                | vdm-snippets                    | https://github.com/peterwvj/vdm-mode                           |
| w3m                     | shimbun                         | https://github.com/emacs-w3m/emacs-w3m                         |
| wgrep                   | wgrep-ack                       | https://github.com/mhayashi1120/Emacs-wgrep                    |
| wgrep                   | wgrep-ag                        | https://github.com/mhayashi1120/Emacs-wgrep                    |
| wgrep                   | wgrep-deadgrep                  | https://github.com/mhayashi1120/Emacs-wgrep                    |
| wgrep                   | wgrep-helm                      | https://github.com/mhayashi1120/Emacs-wgrep                    |
| wgrep                   | wgrep-pt                        | https://github.com/mhayashi1120/Emacs-wgrep                    |
| wildcharm-theme         | wildcharm-light-theme           | https://github.com/habamax/wildcharm-theme                     |
| window-jump             | spaces                          | https://github.com/chumpage/chumpy-windows                     |
| window-jump             | splitter                        | https://github.com/chumpage/chumpy-windows                     |
| xwwp                    | xwwp-follow-link-helm           | https://github.com/canatella/xwwp                              |
| xwwp                    | xwwp-follow-link-ivy            | https://github.com/canatella/xwwp                              |
| ycm                     | company-ycm                     | https://github.com/neuromage/ycm.el                            |
| ycmd                    | company-ycmd                    | https://github.com/abingham/emacs-ycmd                         |
| ycmd                    | flycheck-ycmd                   | https://github.com/abingham/emacs-ycmd                         |
| zetteldesk              | zetteldesk-info                 | https://github.com/Vidianos-Giannitsis/zetteldesk.el           |
| zetteldesk              | zetteldesk-kb                   | https://github.com/Vidianos-Giannitsis/zetteldesk.el           |
| zetteldesk              | zetteldesk-ref                  | https://github.com/Vidianos-Giannitsis/zetteldesk.el           |
| zetteldesk              | zetteldesk-remark               | https://github.com/Vidianos-Giannitsis/zetteldesk.el           |
| zk                      | zk-desktop                      | https://github.com/localauthor/zk                              |
| zk                      | zk-index                        | https://github.com/localauthor/zk                              |

Re: package-vc support for :files keyword

[Stefan Kangas]

Jonas Bernoulli <jonas@bernoul.li> writes:

> Tony Zorman <tonyzorman@mailbox.org> writes:
>
>> This is not just for multiple packages in a single repository—at least
>> one has to somewhat broaden what "multiple packages" means. Some
>> packages include small shims for bigger projects, and inadvertently
>> require them as dependencies. The original issue[1] on the
>> vc-use-package repo mentions org-ql[2], more specifically its helm
>> integration in the form of helm-org-ql.el. Some people might not want to
>> pull down helm as a dependency just for one file that they are not going
>> to use anyways.
>>
>> I'm not sure how common of a situation this actually is, but at least
>> for the big completion frameworks—helm and ivy—it's not totally unheard
>> of.

If a user uses foo, and also bar, then foo may support bar optionally,
or the other way around.  We have ways of dealing with that without an
explicit dependency, including e.g. autoloads and `eval-after-load'.
The user will simply install both foo and bar, and things should ideally
work as expected, including their integration.  See for example
use-package-ensure-system-package.el.

Is there any reason why that can't work?

A separate but related issue is that we should really teach package.el
to deal with optional dependencies.  I personally like Debian's model
with "Recommends" and "Suggests" sections.

> Here's a complete list for all of these packages that are available on
> Melpa.  Obviously not all of these pairings fall into the "foo and
> helm-foo share a repository" category, but you can get an idea of what
> other reasons exist for splitting a repository into multiple packages,
> based on the names of the packages/libraries.  I have included links to
> the repositories, so you can quickly jump there, when only looking at
> the names is not enough.

Having reviewed this list, my conclusion remains that there is usually
no need for splitting up packages like this.

Re: package-vc support for :files keyword

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Jonas Bernoulli <jonas@bernoul.li> writes:
>
>> Tony Zorman <tonyzorman@mailbox.org> writes:
>>
>>> This is not just for multiple packages in a single repository—at least
>>> one has to somewhat broaden what "multiple packages" means. Some
>>> packages include small shims for bigger projects, and inadvertently
>>> require them as dependencies. The original issue[1] on the
>>> vc-use-package repo mentions org-ql[2], more specifically its helm
>>> integration in the form of helm-org-ql.el. Some people might not want to
>>> pull down helm as a dependency just for one file that they are not going
>>> to use anyways.
>>>
>>> I'm not sure how common of a situation this actually is, but at least
>>> for the big completion frameworks—helm and ivy—it's not totally unheard
>>> of.
>
> If a user uses foo, and also bar, then foo may support bar optionally,
> or the other way around.  We have ways of dealing with that without an
> explicit dependency, including e.g. autoloads and `eval-after-load'.
> The user will simply install both foo and bar, and things should ideally
> work as expected, including their integration.  See for example
> use-package-ensure-system-package.el.
>
> Is there any reason why that can't work?
>
> A separate but related issue is that we should really teach package.el
> to deal with optional dependencies.  I personally like Debian's model
> with "Recommends" and "Suggests" sections.

What is the difference between the two?

>> Here's a complete list for all of these packages that are available on
>> Melpa.  Obviously not all of these pairings fall into the "foo and
>> helm-foo share a repository" category, but you can get an idea of what
>> other reasons exist for splitting a repository into multiple packages,
>> based on the names of the packages/libraries.  I have included links to
>> the repositories, so you can quickly jump there, when only looking at
>> the names is not enough.
>
        > Having reviewed this list, my conclusion remains that there is usually
> no need for splitting up packages like this.

I might be mistaken, but I believe that MELPA and specifically
package-lint advise against using {with,}eval-after-load, encouraging
splitting up packages like these.