Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>     Allow for packages to be installed directly from VCS

Yay!

> +(defcustom package-devel-dir (expand-file-name "devel" package-user-dir)
> +  "Directory containing the user's Emacs Lisp package checkouts.
> +The directory name should be absolute.
> +Apart from this directory, Emacs also looks for system-wide
> +packages in `package-directory-list'."
> +  :type 'directory
> +  :initialize #'custom-initialize-delay
> +  :set-after '(package-user-dir)
> +  :risky t
> +  :version "29.1")

Since this is not autoloaded, it should not need (nor want) `custom-initialize-delay`.

> @@ -560,7 +571,9 @@ This is, approximately, the inverse of `version-to-list'.
>  This is the name of the package with its version appended."
>    (format "%s-%s"
>            (package-desc-name pkg-desc)
> -          (package-version-join (package-desc-version pkg-desc))))
> +          (if (eq (package-desc-kind pkg-desc) 'source)
> +              "devel"
> +            (package-version-join (package-desc-version pkg-desc)))))

Currently the way `package.el` handles the use of Git-installed packages
is that it allows a package's directory to be just `<pkg>` instead of
`<pkg>-<vers>`.

So maybe we could do the same here.

> +(declare-function vc-working-revision "vc" (file &optional backend))
> +(defun package-devel-commit (pkg)
> +  "Extract the commit of a development package PKG."
> +  (cl-assert (eq (package-desc-kind pkg) 'source))
> +  (require 'vc)
> +  (cl-loop with dir = (package-desc-dir pkg)
> +           for file in (directory-files dir t "\\.el\\'" t)
> +           when (vc-working-revision file) return it
> +           finally return "unknown"))

Any chance we could use `vc-working-revision` on the package's root
directory to avoid this loop?

> @@ -681,6 +704,14 @@ return it."
>                               (read (current-buffer)))
>                              (error "Can't find define-package in %s" pkg-file))))
>            (setf (package-desc-dir pkg-desc) pkg-dir)
> +          (when (file-exists-p (expand-file-name
> +                                (symbol-name (package-desc-name pkg-desc))
> +                                package-devel-dir))
> +            ;; XXX: This check seems dirty, there should be a better
> +            ;; way to deduce if a package is in the devel directory.
> +            (setf (package-desc-kind pkg-desc) 'source)
> +            (push (cons :commit (package-devel-commit pkg-desc))
> +                  (package-desc-extras pkg-desc)))
>            (if (file-exists-p signed-file)
>                (setf (package-desc-signed pkg-desc) t))
>            pkg-desc)))))

Hmm... why do we need to know if a package is in the devel directory?

> +         ;; In case the package was installed directly from source, the
> +         ;; dependency list wasn't know beforehand, and they might have
> +         ;; to be installed explicitly.
> +         (let (deps)
> +           (dolist (file (directory-files pkg-dir t "\\.el\\'" t))
> +             (with-temp-buffer
> +               (insert-file-contents file)
> +               (when-let* ((require-lines (lm-header-multiline "package-requires")))
> +                 (thread-last
> +                   (mapconcat #'identity require-lines " ")
> +                   package-read-from-string
> +                   package--prepare-dependencies
> +                   (nconc deps)
> +                   (setq deps)))))

Hmm... I expected here we'd open the `<pkg>.el` file and call `package-buffer-info`.

Also, I'm wondering if `package-unpack` is the right place to put this
code.  Maybe it should be in a completely separate function:
`package-unpack` takes a buffer containing the package, whereas for
those VCS-packages we start from something quite different, and the
equivalent to the `pkg-desc` we pass for installation should be fairly
different as well (should more similar to the "package specs" used in
`elpa-admin.el`).  So I think it'll want a completely separate code path.

IOW, I'm not sure `package-fetch` should call `package-install`.
I suspect it would work as well or better if it did most of the
installation with its own fresh new code, and only hooks into the rest
of the code once the files are extracted and a `<pkg>-pkg.el`
was generated.

Also, unless the new code is small, it should likely live in another
file.  `package.el` tends to get loaded at startup in all sessions
(unless you use `package-quickstart`), so we should try and trim it down
rather than grow it.  Most of its code (`package-install`,
`list-packages`, ...) should be in a separate file from the one that
provides `package-activate-all`, I think.

Admittedly, maybe the better way is not to move `package-install` out
into a `package-extra.el` but rather to move `package-activate-all` to
a preloaded `package-hooks.el`.

> +    ((when-let* ((desc (cadr (assoc name-or-url package-archive-contents
> +                                    #'string=)))
> +                 (spec (or (alist-get :vc (package-desc-extras desc))
> +                           (user-error "Package has no VC header"))))

Hmmm... so IIUC you intend to change `elpa-admin.el` to include a `:vc`
extra info in the `archive-contents` file for every package?
Any reason to make that a plain string (that we then need to parse back
into a list) rather than a list?


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +(defcustom package-devel-dir (expand-file-name "devel" package-user-dir)
>> +  "Directory containing the user's Emacs Lisp package checkouts.
>> +The directory name should be absolute.
>> +Apart from this directory, Emacs also looks for system-wide
>> +packages in `package-directory-list'."
>> +  :type 'directory
>> +  :initialize #'custom-initialize-delay
>> +  :set-after '(package-user-dir)
>> +  :risky t
>> +  :version "29.1")
>
> Since this is not autoloaded, it should not need (nor want) `custom-initialize-delay`.

Ok.

>> @@ -560,7 +571,9 @@ This is, approximately, the inverse of `version-to-list'.
>>  This is the name of the package with its version appended."
>>    (format "%s-%s"
>>            (package-desc-name pkg-desc)
>> -          (package-version-join (package-desc-version pkg-desc))))
>> +          (if (eq (package-desc-kind pkg-desc) 'source)
>> +              "devel"
>> +            (package-version-join (package-desc-version pkg-desc)))))
>
> Currently the way `package.el` handles the use of Git-installed packages
> is that it allows a package's directory to be just `<pkg>` instead of
> `<pkg>-<vers>`.
>
> So maybe we could do the same here.

I guess the question here is if you want to make it explicit that
e.g. when removing a package with package-delete, you will be removing a
local checkout that might have modifications.  If not, it would probably
make sense to add warnings where necessary.

>> +(declare-function vc-working-revision "vc" (file &optional backend))
>> +(defun package-devel-commit (pkg)
>> +  "Extract the commit of a development package PKG."
>> +  (cl-assert (eq (package-desc-kind pkg) 'source))
>> +  (require 'vc)
>> +  (cl-loop with dir = (package-desc-dir pkg)
>> +           for file in (directory-files dir t "\\.el\\'" t)
>> +           when (vc-working-revision file) return it
>> +           finally return "unknown"))
>
> Any chance we could use `vc-working-revision` on the package's root
> directory to avoid this loop?

If you pass the backend directly to vc-working-revision, and set the
file to the expanded file name of the current directory, it seems to
work (at least for git and SVN).  Would it make sense to add that as a
first step, then fall back to the loop?

>> @@ -681,6 +704,14 @@ return it."
>>                               (read (current-buffer)))
>>                              (error "Can't find define-package in %s" pkg-file))))
>>            (setf (package-desc-dir pkg-desc) pkg-dir)
>> +          (when (file-exists-p (expand-file-name
>> +                                (symbol-name (package-desc-name pkg-desc))
>> +                                package-devel-dir))
>> +            ;; XXX: This check seems dirty, there should be a better
>> +            ;; way to deduce if a package is in the devel directory.
>> +            (setf (package-desc-kind pkg-desc) 'source)
>> +            (push (cons :commit (package-devel-commit pkg-desc))
>> +                  (package-desc-extras pkg-desc)))
>>            (if (file-exists-p signed-file)
>>                (setf (package-desc-signed pkg-desc) t))
>>            pkg-desc)))))
>
> Hmm... why do we need to know if a package is in the devel directory?

Currently the :kind is not stored, so we have to infer it somehow.  I am
certain this could also be solve some other way, but I would also like
to allow for simply placing a directory into elpa/devel to be handled as
a package, as before with my site-lisp.el proposal.

>> +         ;; In case the package was installed directly from source, the
>> +         ;; dependency list wasn't know beforehand, and they might have
>> +         ;; to be installed explicitly.
>> +         (let (deps)
>> +           (dolist (file (directory-files pkg-dir t "\\.el\\'" t))
>> +             (with-temp-buffer
>> +               (insert-file-contents file)
>> +               (when-let* ((require-lines (lm-header-multiline "package-requires")))
>> +                 (thread-last
>> +                   (mapconcat #'identity require-lines " ")
>> +                   package-read-from-string
>> +                   package--prepare-dependencies
>> +                   (nconc deps)
>> +                   (setq deps)))))
>
> Hmm... I expected here we'd open the `<pkg>.el` file and call `package-buffer-info`.

That was my first thought too, but are we always certain that <pkg>.el
is the main file?

> Also, I'm wondering if `package-unpack` is the right place to put this
> code.  Maybe it should be in a completely separate function:
> `package-unpack` takes a buffer containing the package, whereas for
> those VCS-packages we start from something quite different, and the
> equivalent to the `pkg-desc` we pass for installation should be fairly
> different as well (should more similar to the "package specs" used in
> `elpa-admin.el`).  So I think it'll want a completely separate code path.
>
> IOW, I'm not sure `package-fetch` should call `package-install`.
> I suspect it would work as well or better if it did most of the
> installation with its own fresh new code, and only hooks into the rest
> of the code once the files are extracted and a `<pkg>-pkg.el`
> was generated.

I will experiment with how viable this is.

> Also, unless the new code is small, it should likely live in another
> file.  `package.el` tends to get loaded at startup in all sessions
> (unless you use `package-quickstart`), so we should try and trim it down
> rather than grow it.  Most of its code (`package-install`,
> `list-packages`, ...) should be in a separate file from the one that
> provides `package-activate-all`, I think.

Currently, I don't expect it to grow too much beyond the current
proposal.  If it does turn out that the missing features are too large,
pulling it all out into something like package-vc.el/package-extra.el
seems reasonable (and perhaps perhaps even fit for distribution via
ELPA).

> Admittedly, maybe the better way is not to move `package-install` out
> into a `package-extra.el` but rather to move `package-activate-all` to
> a preloaded `package-hooks.el`.
>
>> +    ((when-let* ((desc (cadr (assoc name-or-url package-archive-contents
>> +                                    #'string=)))
>> +                 (spec (or (alist-get :vc (package-desc-extras desc))
>> +                           (user-error "Package has no VC header"))))
>
> Hmmm... so IIUC you intend to change `elpa-admin.el` to include a `:vc`
> extra info in the `archive-contents` file for every package?
> Any reason to make that a plain string (that we then need to parse back
> into a list) rather than a list?

That would also be possible, I believe I initially assumed that :vc
would contain the contents of the header, and forgot to revise that
assumption later on.

>
>         Stefan
>

-- 
	Philip Kaludercic

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>>> @@ -560,7 +571,9 @@ This is, approximately, the inverse of `version-to-list'.
>>>  This is the name of the package with its version appended."
>>>    (format "%s-%s"
>>>            (package-desc-name pkg-desc)
>>> -          (package-version-join (package-desc-version pkg-desc))))
>>> +          (if (eq (package-desc-kind pkg-desc) 'source)
>>> +              "devel"
>>> +            (package-version-join (package-desc-version pkg-desc)))))
>>
>> Currently the way `package.el` handles the use of Git-installed packages
>> is that it allows a package's directory to be just `<pkg>` instead of
>> `<pkg>-<vers>`.
>>
>> So maybe we could do the same here.
>
> I guess the question here is if you want to make it explicit that
> e.g. when removing a package with package-delete, you will be removing a
> local checkout that might have modifications.  If not, it would probably
> make sense to add warnings where necessary.

I don't understand why using `<pkg>-devel` instead of `<pkg>` makes it
easier to know that "you will be removing a local checkout that might
have modifications"

>>> +(declare-function vc-working-revision "vc" (file &optional backend))
>>> +(defun package-devel-commit (pkg)
>>> +  "Extract the commit of a development package PKG."
>>> +  (cl-assert (eq (package-desc-kind pkg) 'source))
>>> +  (require 'vc)
>>> +  (cl-loop with dir = (package-desc-dir pkg)
>>> +           for file in (directory-files dir t "\\.el\\'" t)
>>> +           when (vc-working-revision file) return it
>>> +           finally return "unknown"))
>>
>> Any chance we could use `vc-working-revision` on the package's root
>> directory to avoid this loop?
>
> If you pass the backend directly to vc-working-revision, and set the
> file to the expanded file name of the current directory, it seems to
> work (at least for git and SVN).  Would it make sense to add that as a
> first step, then fall back to the loop?

Since you say it works, yes.  We could even drop the loop altogether
(and either not accept working with those VC backends where it doesn't
work, or improve those backends which we do want to work).

>>> @@ -681,6 +704,14 @@ return it."
>>>                               (read (current-buffer)))
>>>                              (error "Can't find define-package in %s" pkg-file))))
>>>            (setf (package-desc-dir pkg-desc) pkg-dir)
>>> +          (when (file-exists-p (expand-file-name
>>> +                                (symbol-name (package-desc-name pkg-desc))
>>> +                                package-devel-dir))
>>> +            ;; XXX: This check seems dirty, there should be a better
>>> +            ;; way to deduce if a package is in the devel directory.
>>> +            (setf (package-desc-kind pkg-desc) 'source)
>>> +            (push (cons :commit (package-devel-commit pkg-desc))
>>> +                  (package-desc-extras pkg-desc)))
>>>            (if (file-exists-p signed-file)
>>>                (setf (package-desc-signed pkg-desc) t))
>>>            pkg-desc)))))
>>
>> Hmm... why do we need to know if a package is in the devel directory?
>
> Currently the :kind is not stored, so we have to infer it somehow.  I am
> certain this could also be solve some other way, but I would also like
> to allow for simply placing a directory into elpa/devel to be handled as
> a package, as before with my site-lisp.el proposal.

IIUC for those VCS-installed packages we generate the <pkg>-pkg.el file
ourselves, so it should be fairly easy to make sure that file provides
the needed info so we don't need to guess.

>>> +         ;; In case the package was installed directly from source, the
>>> +         ;; dependency list wasn't know beforehand, and they might have
>>> +         ;; to be installed explicitly.
>>> +         (let (deps)
>>> +           (dolist (file (directory-files pkg-dir t "\\.el\\'" t))
>>> +             (with-temp-buffer
>>> +               (insert-file-contents file)
>>> +               (when-let* ((require-lines (lm-header-multiline "package-requires")))
>>> +                 (thread-last
>>> +                   (mapconcat #'identity require-lines " ")
>>> +                   package-read-from-string
>>> +                   package--prepare-dependencies
>>> +                   (nconc deps)
>>> +                   (setq deps)))))
>>
>> Hmm... I expected here we'd open the `<pkg>.el` file and call
>> `package-buffer-info`.
>
> That was my first thought too, but are we always certain that <pkg>.el
> is the main file?

For the rare cases where this is not the case, the `:main-file` property
of the package spec should tell us which file it is.  I'd rather not try
and guess.

> That would also be possible, I believe I initially assumed that :vc
> would contain the contents of the header, and forgot to revise that
> assumption later on.

I think making it hold the package spec (as given in `elpa-packages`)
would be a natural starting point.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> @@ -560,7 +571,9 @@ This is, approximately, the inverse of `version-to-list'.
>>>>  This is the name of the package with its version appended."
>>>>    (format "%s-%s"
>>>>            (package-desc-name pkg-desc)
>>>> -          (package-version-join (package-desc-version pkg-desc))))
>>>> +          (if (eq (package-desc-kind pkg-desc) 'source)
>>>> +              "devel"
>>>> +            (package-version-join (package-desc-version pkg-desc)))))
>>>
>>> Currently the way `package.el` handles the use of Git-installed packages
>>> is that it allows a package's directory to be just `<pkg>` instead of
>>> `<pkg>-<vers>`.
>>>
>>> So maybe we could do the same here.
>>
>> I guess the question here is if you want to make it explicit that
>> e.g. when removing a package with package-delete, you will be removing a
>> local checkout that might have modifications.  If not, it would probably
>> make sense to add warnings where necessary.
>
> I don't understand why using `<pkg>-devel` instead of `<pkg>` makes it
> easier to know that "you will be removing a local checkout that might
> have modifications"

The -devel doesn't necessarily indicate that directly, I just meant it
was worth somehow explicitly indicating that this is a VCS-package.

>>>> +(declare-function vc-working-revision "vc" (file &optional backend))
>>>> +(defun package-devel-commit (pkg)
>>>> +  "Extract the commit of a development package PKG."
>>>> +  (cl-assert (eq (package-desc-kind pkg) 'source))
>>>> +  (require 'vc)
>>>> +  (cl-loop with dir = (package-desc-dir pkg)
>>>> +           for file in (directory-files dir t "\\.el\\'" t)
>>>> +           when (vc-working-revision file) return it
>>>> +           finally return "unknown"))
>>>
>>> Any chance we could use `vc-working-revision` on the package's root
>>> directory to avoid this loop?
>>
>> If you pass the backend directly to vc-working-revision, and set the
>> file to the expanded file name of the current directory, it seems to
>> work (at least for git and SVN).  Would it make sense to add that as a
>> first step, then fall back to the loop?
>
> Since you say it works, yes.  We could even drop the loop altogether
> (and either not accept working with those VC backends where it doesn't
> work, or improve those backends which we do want to work).

As I presume that in most cases Git is used, and this appears to be
working with vc-git, this should be ok.

>>>> @@ -681,6 +704,14 @@ return it."
>>>>                               (read (current-buffer)))
>>>>                              (error "Can't find define-package in %s" pkg-file))))
>>>>            (setf (package-desc-dir pkg-desc) pkg-dir)
>>>> +          (when (file-exists-p (expand-file-name
>>>> +                                (symbol-name (package-desc-name pkg-desc))
>>>> +                                package-devel-dir))
>>>> +            ;; XXX: This check seems dirty, there should be a better
>>>> +            ;; way to deduce if a package is in the devel directory.
>>>> +            (setf (package-desc-kind pkg-desc) 'source)
>>>> +            (push (cons :commit (package-devel-commit pkg-desc))
>>>> +                  (package-desc-extras pkg-desc)))
>>>>            (if (file-exists-p signed-file)
>>>>                (setf (package-desc-signed pkg-desc) t))
>>>>            pkg-desc)))))
>>>
>>> Hmm... why do we need to know if a package is in the devel directory?
>>
>> Currently the :kind is not stored, so we have to infer it somehow.  I am
>> certain this could also be solve some other way, but I would also like
>> to allow for simply placing a directory into elpa/devel to be handled as
>> a package, as before with my site-lisp.el proposal.
>
> IIUC for those VCS-installed packages we generate the <pkg>-pkg.el file
> ourselves, so it should be fairly easy to make sure that file provides
> the needed info so we don't need to guess.

Yes, but again, if I just clone a project into elpa/devel, this won't
work.  It could be argued that this shouldn't be supported because it is
a different issue, and the elpa/... directories maintained by
package.el, but in that case I would like some other way to have
package.el manage non-versioned local source.

>>>> +         ;; In case the package was installed directly from source, the
>>>> +         ;; dependency list wasn't know beforehand, and they might have
>>>> +         ;; to be installed explicitly.
>>>> +         (let (deps)
>>>> +           (dolist (file (directory-files pkg-dir t "\\.el\\'" t))
>>>> +             (with-temp-buffer
>>>> +               (insert-file-contents file)
>>>> +               (when-let* ((require-lines (lm-header-multiline "package-requires")))
>>>> +                 (thread-last
>>>> +                   (mapconcat #'identity require-lines " ")
>>>> +                   package-read-from-string
>>>> +                   package--prepare-dependencies
>>>> +                   (nconc deps)
>>>> +                   (setq deps)))))
>>>
>>> Hmm... I expected here we'd open the `<pkg>.el` file and call
>>> `package-buffer-info`.
>>
>> That was my first thought too, but are we always certain that <pkg>.el
>> is the main file?
>
> For the rare cases where this is not the case, the `:main-file` property
> of the package spec should tell us which file it is.  I'd rather not try
> and guess.

Ok, I didn't know that :main-file was propagated into the package
specification.  In that case there should be no issue.

>> That would also be possible, I believe I initially assumed that :vc
>> would contain the contents of the header, and forgot to revise that
>> assumption later on.
>
> I think making it hold the package spec (as given in `elpa-packages`)
> would be a natural starting point.

Ok.

-- 
	Philip Kaludercic

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> The -devel doesn't necessarily indicate that directly, I just meant it
> was worth somehow explicitly indicating that this is a VCS-package.

But what I'm saying is: why not put no "-*" at all, since that's what
`package.el` has been using so far for those cases (tho these had to be
"installed" by hand)?

>> IIUC for those VCS-installed packages we generate the <pkg>-pkg.el file
>> ourselves, so it should be fairly easy to make sure that file provides
>> the needed info so we don't need to guess.
>
> Yes, but again, if I just clone a project into elpa/devel, this won't
> work.  It could be argued that this shouldn't be supported because it is
> a different issue, and the elpa/... directories maintained by
> package.el, but in that case I would like some other way to have
> package.el manage non-versioned local source.

I'm sorry, I don't understand what you're saying here.  Who&how does
"just clone a project into elpa/devel" and what do you mean by "elpa/devel"?

>> For the rare cases where this is not the case, the `:main-file` property
>> of the package spec should tell us which file it is.  I'd rather not try
>> and guess.
> Ok, I didn't know that :main-file was propagated into the package
> specification.  In that case there should be no issue.

The package specification is written by hand into `elpa-packages`, so
nothing propagates it from elsewhere into the spec, AFAICT (other than
a human, that is).

I think we need to clarify our assumptions because we seem to fail to
understand each other.

The way I imagine it working goes as follows:

- `elpa-admin.el` will be changed so that the spec from `elpa-packages`
  is propagated into the metadata in `archive-contents`.

- when VCS-installing a package, we basically start with that spec (tho
  it can be reduced to a simple URL in many cases).

- those VCS-packages are installed by cloning them into a directory with
  name `<pkg>`.

- During installation we auto-produce a `<pkg>-pkg.el` file (and the
  `<pkg>-autoloads.el`, of course), so that `package-activate-all` will
  do the right thing.

- We somehow/somewhere/somewhen update these `<pkg>-pkg.el` and
  `<pkg>-autoloads.el` files sometimes after a `git pull` or equivalent.
  A simple way would be to provide some `package-vcs-update` command or
  somesuch (which would also recompile the changed `.el` files).

- We could also arrange for `package-activate` to try and detect the
  case where the files were updated manually via `git pull` and emit
  a warning prompting the user to refresh that package's generated files.
  This same command which refreshes the generated files of
  a VCS-installed package could probably be used by users who want to do
  the `git clone` by hand.

BTW, I think we should choose some `package-<foo>-` prefix for the
vars&functions related to this new "install from VCS" functionality.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The -devel doesn't necessarily indicate that directly, I just meant it
>> was worth somehow explicitly indicating that this is a VCS-package.
>
> But what I'm saying is: why not put no "-*" at all, since that's what
> `package.el` has been using so far for those cases (tho these had to be
> "installed" by hand)?

I get that, my question still remains.  Or to rephrase it, what is the
advantage of displaying package names with the same names as the 
directories they are found in?

>>> IIUC for those VCS-installed packages we generate the <pkg>-pkg.el file
>>> ourselves, so it should be fairly easy to make sure that file provides
>>> the needed info so we don't need to guess.
>>
>> Yes, but again, if I just clone a project into elpa/devel, this won't
>> work.  It could be argued that this shouldn't be supported because it is
>> a different issue, and the elpa/... directories maintained by
>> package.el, but in that case I would like some other way to have
>> package.el manage non-versioned local source.
>
> I'm sorry, I don't understand what you're saying here.  Who&how does
> "just clone a project into elpa/devel" and what do you mean by "elpa/devel"?

What I meant was that all directories in ~/.emacs.d/elpa/devel could be
automatically detected and loaded.  One case where this could be useful
for anyone using submodules in a versioned configuration.

>>> For the rare cases where this is not the case, the `:main-file` property
>>> of the package spec should tell us which file it is.  I'd rather not try
>>> and guess.
>> Ok, I didn't know that :main-file was propagated into the package
>> specification.  In that case there should be no issue.
>
> The package specification is written by hand into `elpa-packages`, so
> nothing propagates it from elsewhere into the spec, AFAICT (other than
> a human, that is).
>
> I think we need to clarify our assumptions because we seem to fail to
> understand each other.
>
> The way I imagine it working goes as follows:
>
> - `elpa-admin.el` will be changed so that the spec from `elpa-packages`
>   is propagated into the metadata in `archive-contents`.
>
> - when VCS-installing a package, we basically start with that spec (tho
>   it can be reduced to a simple URL in many cases).
>
> - those VCS-packages are installed by cloning them into a directory with
>   name `<pkg>`.
>
> - During installation we auto-produce a `<pkg>-pkg.el` file (and the
>   `<pkg>-autoloads.el`, of course), so that `package-activate-all` will
>   do the right thing.
>
> - We somehow/somewhere/somewhen update these `<pkg>-pkg.el` and
>   `<pkg>-autoloads.el` files sometimes after a `git pull` or equivalent.
>   A simple way would be to provide some `package-vcs-update` command or
>   somesuch (which would also recompile the changed `.el` files).
>
> - We could also arrange for `package-activate` to try and detect the
>   case where the files were updated manually via `git pull` and emit
>   a warning prompting the user to refresh that package's generated files.
>   This same command which refreshes the generated files of
>   a VCS-installed package could probably be used by users who want to do
>   the `git clone` by hand.

Yes, exactly (and I agree with all of the above too).  Something like
checking the mtime of files and/or comparing the auto-loaded files on
saving or on startup.

> BTW, I think we should choose some `package-<foo>-` prefix for the
> vars&functions related to this new "install from VCS" functionality.

If it is possible to extract the relevant functionality form package.el
to package-<foo>.el, then of course, but I can also try to stick to this
otherwise.  Do you think this should also include renaming package-fetch?

>
>         Stefan
>

-- 
	Philip Kaludercic

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-02-15 17:13:56] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> The -devel doesn't necessarily indicate that directly, I just meant it
>>> was worth somehow explicitly indicating that this is a VCS-package.
>> But what I'm saying is: why not put no "-*" at all, since that's what
>> `package.el` has been using so far for those cases (tho these had to be
>> "installed" by hand)?
> I get that, my question still remains.  Or to rephrase it, what is the
> advantage of displaying package names with the same names as the 
> directories they are found in?

Not sure what "display" has to do with it.  The name of the package is
beyond our control.  So the choice is not what to display but what name
to give to the directory.  `<pkg>` is a name that already works.
`<pkg>-devel` is a name that currently doesn't work.

In any case, this is not an important issue, I think (at least, as long
as "devel" is not a valid version name).

>>>> IIUC for those VCS-installed packages we generate the <pkg>-pkg.el file
>>>> ourselves, so it should be fairly easy to make sure that file provides
>>>> the needed info so we don't need to guess.
>>>
>>> Yes, but again, if I just clone a project into elpa/devel, this won't
>>> work.  It could be argued that this shouldn't be supported because it is
>>> a different issue, and the elpa/... directories maintained by
>>> package.el, but in that case I would like some other way to have
>>> package.el manage non-versioned local source.
>>
>> I'm sorry, I don't understand what you're saying here.  Who&how does
>> "just clone a project into elpa/devel" and what do you mean by "elpa/devel"?
>
> What I meant was that all directories in ~/.emacs.d/elpa/devel could be
> automatically detected and loaded.

That's already the case if `~/.emacs.d/elpa/devel` is in
`package-directory-list`, so I don't understand how that relates to what
we're discussing.

> One case where this could be useful for anyone using submodules in
> a versioned configuration.

Again, I don't understand the relationship.  Maybe if you could give
a concrete example scenario?

>> BTW, I think we should choose some `package-<foo>-` prefix for the
>> vars&functions related to this new "install from VCS" functionality.
> If it is possible to extract the relevant functionality form package.el
> to package-<foo>.el, then of course, but I can also try to stick to this
> otherwise.  Do you think this should also include renaming package-fetch?

Yes.  Depending on how good "-<foo>-" is, we may want to add shorter
aliases, of course, but I think that using a `package-<foo>-` prefix
will help clarify the design.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-02-15 17:13:56] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> The -devel doesn't necessarily indicate that directly, I just meant it
>>>> was worth somehow explicitly indicating that this is a VCS-package.
>>> But what I'm saying is: why not put no "-*" at all, since that's what
>>> `package.el` has been using so far for those cases (tho these had to be
>>> "installed" by hand)?
>> I get that, my question still remains.  Or to rephrase it, what is the
>> advantage of displaying package names with the same names as the 
>> directories they are found in?
>
> Not sure what "display" has to do with it.  The name of the package is
> beyond our control.  So the choice is not what to display but what name
> to give to the directory.  `<pkg>` is a name that already works.
> `<pkg>-devel` is a name that currently doesn't work.
>
> In any case, this is not an important issue, I think (at least, as long
> as "devel" is not a valid version name).

I now understand the confusion, I had assumed we were discussion how
package names are presented by completing-read calls, without
considering that `package-desc-full-name' is invoked elsewhere too.
In that case, yes, the change should be reverted.

>>>>> IIUC for those VCS-installed packages we generate the <pkg>-pkg.el file
>>>>> ourselves, so it should be fairly easy to make sure that file provides
>>>>> the needed info so we don't need to guess.
>>>>
>>>> Yes, but again, if I just clone a project into elpa/devel, this won't
>>>> work.  It could be argued that this shouldn't be supported because it is
>>>> a different issue, and the elpa/... directories maintained by
>>>> package.el, but in that case I would like some other way to have
>>>> package.el manage non-versioned local source.
>>>
>>> I'm sorry, I don't understand what you're saying here.  Who&how does
>>> "just clone a project into elpa/devel" and what do you mean by "elpa/devel"?
>>
>> What I meant was that all directories in ~/.emacs.d/elpa/devel could be
>> automatically detected and loaded.
>
> That's already the case if `~/.emacs.d/elpa/devel` is in
> `package-directory-list`, so I don't understand how that relates to what
> we're discussing.

Unless I am mistaken, `package--make-autoloads-and-stuff' is currently
only invoked by `package-unpack', that in turn is only called by the
`package-install-from-*' functions.  Am I missing something, or how
would the autoload file be generated if I just place a repository into
~/.emacs.d/elpa/devel?

Of course if this complicates things too much, I won't insist on it, as
alternative solutions exists.

>> One case where this could be useful for anyone using submodules in
>> a versioned configuration.
>
> Again, I don't understand the relationship.  Maybe if you could give
> a concrete example scenario?

Say a repository consists of only an init.el, and a few submoduled git
repositories under /elpa/devel/.  These wouldn't include the -pkg.el or
-autoload.el files, so they should be generated as soon as necessary.

>>> BTW, I think we should choose some `package-<foo>-` prefix for the
>>> vars&functions related to this new "install from VCS" functionality.
>> If it is possible to extract the relevant functionality form package.el
>> to package-<foo>.el, then of course, but I can also try to stick to this
>> otherwise.  Do you think this should also include renaming package-fetch?
>
> Yes.  Depending on how good "-<foo>-" is, we may want to add shorter
> aliases, of course, but I think that using a `package-<foo>-` prefix
> will help clarify the design.

Ok.

>
>         Stefan
>

-- 
	Philip Kaludercic

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>>>> I'm sorry, I don't understand what you're saying here.  Who&how does
>>>> "just clone a project into elpa/devel" and what do you mean by "elpa/devel"?
>>>
>>> What I meant was that all directories in ~/.emacs.d/elpa/devel could be
>>> automatically detected and loaded.
>>
>> That's already the case if `~/.emacs.d/elpa/devel` is in
>> `package-directory-list`, so I don't understand how that relates to what
>> we're discussing.
>
> Unless I am mistaken, `package--make-autoloads-and-stuff' is currently
> only invoked by `package-unpack', that in turn is only called by the
> `package-install-from-*' functions.  Am I missing something, or how
> would the autoload file be generated if I just place a repository into
> ~/.emacs.d/elpa/devel?

I don't think we should spend too much time trying to support the case
where a user only does `git clone` and expects Emacs to notice the new
package and set it up magically.  Instead, I'd expect
`package-activate-all` to warn the user about a missing `<pkg>-pkg.el`
when collecting the available packages and about a missing
`<pkg>-autoloads.el` file when trying to activate a given package.

Then the user can use some `package-<foo>-refresh` or somesuch to
(re)create those files (and (re)compile the `.el` files along the way
and (re)generate the docs, ...).

> Say a repository consists of only an init.el, and a few submoduled git
> repositories under /elpa/devel/.  These wouldn't include the -pkg.el or
> -autoload.el files, so they should be generated as soon as necessary.

Let those users write specific code for that if they so wish.
Generating the `<pkg>-pkg.el` file as well as the `<pkg>-autoloads.el`
cannot be done fully automatically in all cases.  More specifically it
may require the package's spec to find the `:main-file` and other such
things.  I don't think we should aim for that level of automation yet.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>> I'm sorry, I don't understand what you're saying here.  Who&how does
>>>>> "just clone a project into elpa/devel" and what do you mean by "elpa/devel"?
>>>>
>>>> What I meant was that all directories in ~/.emacs.d/elpa/devel could be
>>>> automatically detected and loaded.
>>>
>>> That's already the case if `~/.emacs.d/elpa/devel` is in
>>> `package-directory-list`, so I don't understand how that relates to what
>>> we're discussing.
>>
>> Unless I am mistaken, `package--make-autoloads-and-stuff' is currently
>> only invoked by `package-unpack', that in turn is only called by the
>> `package-install-from-*' functions.  Am I missing something, or how
>> would the autoload file be generated if I just place a repository into
>> ~/.emacs.d/elpa/devel?
>
> I don't think we should spend too much time trying to support the case
> where a user only does `git clone` and expects Emacs to notice the new
> package and set it up magically.  Instead, I'd expect
> `package-activate-all` to warn the user about a missing `<pkg>-pkg.el`
> when collecting the available packages and about a missing
> `<pkg>-autoloads.el` file when trying to activate a given package.
>
> Then the user can use some `package-<foo>-refresh` or somesuch to
> (re)create those files (and (re)compile the `.el` files along the way
> and (re)generate the docs, ...).

This seems reasonable, I will try to implement it in the next few days.

>> Say a repository consists of only an init.el, and a few submoduled git
>> repositories under /elpa/devel/.  These wouldn't include the -pkg.el or
>> -autoload.el files, so they should be generated as soon as necessary.
>
> Let those users write specific code for that if they so wish.
> Generating the `<pkg>-pkg.el` file as well as the `<pkg>-autoloads.el`
> cannot be done fully automatically in all cases.  More specifically it
> may require the package's spec to find the `:main-file` and other such
> things.  I don't think we should aim for that level of automation yet.

My init.el begins with this blob that does the job well enough (it was
also the basis for the site-lisp.el patch I proposed last year):

--8<---------------cut here---------------start------------->8---
(eval-and-compile                       ;for flymake
  (require 'autoload)
  (let* ((backup-inhibited t)
         (dir (locate-user-emacs-file "site-lisp"))
         (load (expand-file-name "autoload.el" dir)))
    (dolist (dir (directory-files dir t "^[^.]"))
      (when (file-directory-p dir)
        (add-to-list 'load-path dir)
        (make-directory-autoloads dir load)
        (byte-recompile-directory dir)))
    (add-to-list 'load-path dir)
    (load load nil t)
    (kill-buffer (find-buffer-visiting load))))
--8<---------------cut here---------------end--------------->8---

So if I want to try some package out from source, all I need to do is
clone the repository into ~/.emacs.d/site-lisp and everything else is
done automatically when Emacs is restarted (or the snippet is
evaluated).

-- 
	Philip Kaludercic

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> --8<---------------cut here---------------start------------->8---
> (eval-and-compile                       ;for flymake
>   (require 'autoload)
>   (let* ((backup-inhibited t)
>          (dir (locate-user-emacs-file "site-lisp"))
>          (load (expand-file-name "autoload.el" dir)))
>     (dolist (dir (directory-files dir t "^[^.]"))
>       (when (file-directory-p dir)
>         (add-to-list 'load-path dir)
>         (make-directory-autoloads dir load)
>         (byte-recompile-directory dir)))
>     (add-to-list 'load-path dir)
>     (load load nil t)
>     (kill-buffer (find-buffer-visiting load))))
> --8<---------------cut here---------------end--------------->8---

I think we could aim to make it easier for users to do this kind of
thing, but I don't think we want to impose the cost of
`make-directory-autoloads` and `byte-recompile-directory` at every startup.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> --8<---------------cut here---------------start------------->8---
>> (eval-and-compile                       ;for flymake
>>   (require 'autoload)
>>   (let* ((backup-inhibited t)
>>          (dir (locate-user-emacs-file "site-lisp"))
>>          (load (expand-file-name "autoload.el" dir)))
>>     (dolist (dir (directory-files dir t "^[^.]"))
>>       (when (file-directory-p dir)
>>         (add-to-list 'load-path dir)
>>         (make-directory-autoloads dir load)
>>         (byte-recompile-directory dir)))
>>     (add-to-list 'load-path dir)
>>     (load load nil t)
>>     (kill-buffer (find-buffer-visiting load))))
>> --8<---------------cut here---------------end--------------->8---
>
> I think we could aim to make it easier for users to do this kind of
> thing, but I don't think we want to impose the cost of
> `make-directory-autoloads` and `byte-recompile-directory` at every startup.

In that case, I would suggest considering my site-lisp.el proposal from
last year again.  The reason I dropped it was because it was implied
that the functionality should be merged into package.el, but it seems
that this wouldn't be the right way to go about it either.

-- 
	Philip Kaludercic

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>>> --8<---------------cut here---------------start------------->8---
>>> (eval-and-compile                       ;for flymake
>>>   (require 'autoload)
>>>   (let* ((backup-inhibited t)
>>>          (dir (locate-user-emacs-file "site-lisp"))
>>>          (load (expand-file-name "autoload.el" dir)))
>>>     (dolist (dir (directory-files dir t "^[^.]"))
>>>       (when (file-directory-p dir)
>>>         (add-to-list 'load-path dir)
>>>         (make-directory-autoloads dir load)
>>>         (byte-recompile-directory dir)))
>>>     (add-to-list 'load-path dir)
>>>     (load load nil t)
>>>     (kill-buffer (find-buffer-visiting load))))
>>> --8<---------------cut here---------------end--------------->8---
>>
>> I think we could aim to make it easier for users to do this kind of
>> thing, but I don't think we want to impose the cost of
>> `make-directory-autoloads` and `byte-recompile-directory` at every startup.
>
> In that case, I would suggest considering my site-lisp.el proposal from
> last year again.  The reason I dropped it was because it was implied
> that the functionality should be merged into package.el, but it seems
> that this wouldn't be the right way to go about it either.

The way I see it the package-from-source thingy you're working on in
`feature/package+vc` provides a big part of that functionality and could
even provide "all of it".  I just that the "all of it" needs to be
controlled by a config var because it would slow down startup too much
for those who prefer a slightly less automatic setup where they need to
manually ask Emacs to refresh a package after it has been
modified/installed outside of Emacs's knowledge.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Augusto Stoffel]

On Wed, 16 Feb 2022 at 21:56, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> I don't think we should spend too much time trying to support the case
> where a user only does `git clone` and expects Emacs to notice the new
> package and set it up magically.  Instead, I'd expect
> `package-activate-all` to warn the user about a missing `<pkg>-pkg.el`
> when collecting the available packages and about a missing
> `<pkg>-autoloads.el` file when trying to activate a given package.
>
> Then the user can use some `package-<foo>-refresh` or somesuch to
> (re)create those files (and (re)compile the `.el` files along the way
> and (re)generate the docs, ...).
>
>> Say a repository consists of only an init.el, and a few submoduled git
>> repositories under /elpa/devel/.  These wouldn't include the -pkg.el or
>> -autoload.el files, so they should be generated as soon as necessary.
>
> Let those users write specific code for that if they so wish.
> Generating the `<pkg>-pkg.el` file as well as the `<pkg>-autoloads.el`
> cannot be done fully automatically in all cases.  More specifically it
> may require the package's spec to find the `:main-file` and other such
> things.  I don't think we should aim for that level of automation yet.

It seems reasonable that installing a new package from source will take
a certain number of steps.  On the other hand, there is also the case of
personal configuration files to consider.

Parts of my config are under ~/.emacs.d/lisp and I would like to
autoload (and perhaps byte-compile) definitions from there files without
having to worry about anything.  I believe such a feature would make
sense for lots of people.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> Let those users write specific code for that if they so wish.
>> Generating the `<pkg>-pkg.el` file as well as the `<pkg>-autoloads.el`
>> cannot be done fully automatically in all cases.  More specifically it
>> may require the package's spec to find the `:main-file` and other such
>> things.  I don't think we should aim for that level of automation yet.
>
> It seems reasonable that installing a new package from source will take
> a certain number of steps.  On the other hand, there is also the case of
> personal configuration files to consider.
>
> Parts of my config are under ~/.emacs.d/lisp and I would like to
> autoload (and perhaps byte-compile) definitions from there files without
> having to worry about anything.  I believe such a feature would make
> sense for lots of people.

I'm not opposed to feature in itself.  I'm saying we should first focus
on the less automatic workflow.  Also I'd probably prefer the more
automatic workflow to be optional, unless we can make it sufficiently
cheap (updating autoloads files at every startup is not an option in
this respect).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>     Allow for packages to be installed directly from VCS
>
> Yay!

I would to bring up the "feature/package+vc" branch once more and ask if
it would be possible for the feature to be merged before Emacs 29 is cut
off.

To recapitulate: The current set of features this branch would add are
as follows:

- The ability to install a package directly from source using
  `package-vc-fetch' (aliased to `package-checkout').  This
  functionality is ideally VC generic.

- The ability to update a package using `package-upgrade'[0]

- Package metadata can either be inferred from the package URL (see
  `package-vc-heusitic-alist') or via explicit hints from an ELPA
  server.  I plan to add the necessary features to GNU and NonGNU ELPA
  in time so that the heuristics can be avoided.

- The ability to (i) contact, (ii) send bug reports and (iii) patches
  (using the new `vc-patch-prepare') to package maintainers.

Some changes had to be made to package.el for these features to work,
but most of the heavy-lifting is done in package-vc.el.

Features I have thought about implementing but haven't gotten around
/managed to are:

- Checking if local modifications have been made to the tarball-based
  installation and offering to transpose these into the source
  installation, making it easier to submit these as patches.

- Offering some way of detecting revisions that bump the package version
  and to switch between these.

- Better integration with package specifications that modify the package
  before bundling it into a tarball (e.g. moving files around or running
  custom commands).  There is a conflict here between providing packages
  that work just like their tarballs while also trying to avoid
  situations where arbitrary code is executed on installing a package --
  though that has arguably always been an issue in general.

If it is imaginable for the branch to be merged, I'd like to ask anyone
interested in testing the features and giving feedback on what could be
added.  Given some ack, I'd also like to start writing some
documentation for the manual.

[0] This feature is currently broken because it tries to use `vc-pull'
    which refuses to work if any changes have been made.  This is an
    issue as package-vc-fetch tries to modify the list of ignored files
    in a repository (think of .gitignore).  It will either be necessary
    to change how vc-pull works or try to do something with
    vc-log-incoming + vc-merge.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> - The ability to install a package directly from source using
>   `package-vc-fetch' (aliased to `package-checkout').  This
>   functionality is ideally VC generic.
>
> - The ability to update a package using `package-upgrade'[0]
>
> - Package metadata can either be inferred from the package URL (see
>   `package-vc-heusitic-alist') or via explicit hints from an ELPA
>   server.  I plan to add the necessary features to GNU and NonGNU ELPA
>   in time so that the heuristics can be avoided.
>
> - The ability to (i) contact, (ii) send bug reports and (iii) patches
>   (using the new `vc-patch-prepare') to package maintainers.

Sounds like great functionality, but I wonder whether the security
implications have been discussed?  Today, we use GNU ELPA as a filter of
sorts and people rely on code there not being compromised.

I assume "hints from an ELPA server" is basically a list of links to git
repositories?  If that's the case, then we may well end up with pointing
users towards repos that have been compromised.

If we don't have such a list, then adding the basic functionality sounds
useful anyway -- that is, allowing users to say `M-x
package-install-from-repo' or something and then they type in the URL of
that repo -- that's fine, and leaves the security implications to the
user (where they already are today for people that install from external
repos).

But if we list these repos in `M-x list-packages', that's a very
different issue.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> - The ability to install a package directly from source using
>>   `package-vc-fetch' (aliased to `package-checkout').  This
>>   functionality is ideally VC generic.
>>
>> - The ability to update a package using `package-upgrade'[0]
>>
>> - Package metadata can either be inferred from the package URL (see
>>   `package-vc-heusitic-alist') or via explicit hints from an ELPA
>>   server.  I plan to add the necessary features to GNU and NonGNU ELPA
>>   in time so that the heuristics can be avoided.
>>
>> - The ability to (i) contact, (ii) send bug reports and (iii) patches
>>   (using the new `vc-patch-prepare') to package maintainers.
>
> Sounds like great functionality, but I wonder whether the security
> implications have been discussed?  Today, we use GNU ELPA as a filter of
> sorts and people rely on code there not being compromised.

It seems to me that fetching a package from source is no more dangerous
than fetching a tarball, seeing as the tarball is automatically
generated from the repository.

But I might be mistaken, in which case I'd be interested in what should
be done to improve security.

> I assume "hints from an ELPA server" is basically a list of links to git
> repositories?  

The hints would be an entry in the package property list consisting of
the following items:

    (VC-BACKEND REPOSITORY DIRECTORY BRANCH)

Packages with either this metadata or that are caught by the heuristic
are automatically proposed as completion options when invoking
`package-vc-fetch'.  Of course it is also possible to give an arbitrary
URL, but at that point you should know what you are doing.

>                If that's the case, then we may well end up with pointing
> users towards repos that have been compromised.

Just so that we are on the same page, what are we talking about when
using the term "compromised"?  In what sense would this not affect a
package that we fetch directly from source but not a tarball?

> If we don't have such a list, then adding the basic functionality sounds
> useful anyway -- that is, allowing users to say `M-x
> package-install-from-repo' or something and then they type in the URL of
> that repo -- that's fine, and leaves the security implications to the
> user (where they already are today for people that install from external
> repos).

Yes.

> But if we list these repos in `M-x list-packages', that's a very
> different issue.

No, the list-packages would still only list packages that are listed in
the package archive + packages that have already been installed
using `package-vc-fetch'.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> It seems to me that fetching a package from source is no more dangerous
> than fetching a tarball, seeing as the tarball is automatically
> generated from the repository.

It doesn't matter much whether it's a tar ball or a git repo (although
there is signing of the tar balls), but whether there's any oversight at
all or not.  All commits to Non/GNU ELPA end up on a mailing list, which
provides a smidgen of transparency, which is better than none.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> It seems to me that fetching a package from source is no more dangerous
>> than fetching a tarball, seeing as the tarball is automatically
>> generated from the repository.
>
> It doesn't matter much whether it's a tar ball or a git repo (although
> there is signing of the tar balls), but whether there's any oversight at
> all or not.  All commits to Non/GNU ELPA end up on a mailing list, which
> provides a smidgen of transparency, which is better than none.

OK, in that case my proposal shouldn't be any more dangerous than what
is already going on.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> OK, in that case my proposal shouldn't be any more dangerous than what
> is already going on.

You have `package-vc-fetch' which autocompletes to a list of packages,
so it's more dangerous than what we have today.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> OK, in that case my proposal shouldn't be any more dangerous than what
>> is already going on.
>
> You have `package-vc-fetch' which autocompletes to a list of packages,
> so it's more dangerous than what we have today.

Yes, but the list of packages that are proposed are just a subset of the
packages that are already in the archive?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> Yes, but the list of packages that are proposed are just a subset of the
> packages that are already in the archive?

Oh, I thought they were packages that weren't part of the archive
already.  But in that case, I think it's fine.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Yes, but the list of packages that are proposed are just a subset of the
>> packages that are already in the archive?
>
> Oh, I thought they were packages that weren't part of the archive
> already.  But in that case, I think it's fine.

Good.  So if there is a principal interest in the feature, I would start
writing the documentation.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Yes, but the list of packages that are proposed are just a subset of the
>>> packages that are already in the archive?
>>
>> Oh, I thought they were packages that weren't part of the archive
>> already.  But in that case, I think it's fine.
>
> Good.  So if there is a principal interest in the feature, I would start
> writing the documentation.

I've pushed the documentation to feature/package+vc but forgot to ping
this thread.  For the sake of convenience, and to recapitulate the new
features, this is what it would look like:

--8<---------------cut here---------------start------------->8---
@node Package from Source
@section Package from Source
@cindex package source vcs git @c "git" is not technically correct
                               @c but it is a popular term

  By default @code{package-install} will download a Tarball from a
package archive and install the files therein contained.  Most of the
time this is just what you want.  One exception is when you decide to
hack on the source code of a package, and would like to share these
changes with other users.  In that case you usually want to fetch and
work on the upstream source, so that you can prepare a usable patch.

@findex package-vc-install
  One way to do this is to use @code{package-vc-install}, to fetch the
source code for a package directly from source.  The command will also
automatically ensure that all files are byte-compiled and auto-loaded,
just like with a regular package.  From this point on the package can
be regarded just like any other package, that can be updated (using
@code{package-update}), deleted (using @code{package-delete}) and
viewed in the package listing.

@findex package-contact-maintainer
@findex package-report-bug
@findex package-vc-prepare-patch
  With the source checkout, you might want to reproduce a bug against
the current development head or implement a new feature to scratch an
itch.  If the package metadata indicates that a maintainer can be
contacted via Email, you can use the commands
@code{package-contact-maintainer} to send them a message, or
@code{package-report-bug} to report a bug that will include all the
user options that you have customised.  Patches can be sent out using
@code{package-vc-prepare-patch}, that makes use of
@code{vc-prepare-patch} under the hold (@pxref{Preparing Patches}).

@findex package-vc-link-directory
@findex package-vc-refresh
  If you maintain your own packages you might want to use a local
checkout instead of cloning a remote repository.  This can be done
using @code{package-vc-link-directory}, that creates a symbolic link
from the package directory (@pxref{Package Files}) to your checkout
and initialises the code.  Note that if changes are made such as
adding autoloads, you should use @code{package-vc-refresh} to repeat
the initialisation.
--8<---------------cut here---------------end--------------->8---

Fetching or installing package dev source from VCS: names

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Given what this command does, according to the documentation just
posted, I suggest we call it

   package-dev-source

or

   package-upstream-source

to correspond clearly to what it does.

Corresponding index entries could be these two.

   @cindex package development source
   @cindex package upstream source

Whether it is good to include them depends on what the
sorted index will look like with them and without them.
One or both might be redundant, or miht not.

In the case of NonGNU ELPA, which repo does this access?
Our repo?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Fetching or installing package dev source from VCS: names

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Given what this command does, according to the documentation just
> posted, I suggest we call it
>
>    package-dev-source
>
> or
>
>    package-upstream-source
>
> to correspond clearly to what it does.

Which command specifically?  `package-vc-install' or some other command?

> Corresponding index entries could be these two.
>
>    @cindex package development source
>    @cindex package upstream source
>
> Whether it is good to include them depends on what the
> sorted index will look like with them and without them.
> One or both might be redundant, or miht not.

OK, I will try that out.

> In the case of NonGNU ELPA, which repo does this access?
> Our repo?

That is open, the repository it uses depends on the metadata that ELPA
provides.  If that is missing it uses a heuristic to guess a repository
from the URL, in case contains a link to a "well-known" source force.

As I was planning to work on adding the Metadata to ELPA, I'm curios to
know what you think would be preferable?  From the tone of your question
I would guess the ELPA mirror, right?  If so, I'd be interested what
difference it would make.

Re: Fetching or installing package dev source from VCS: names

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > In the case of NonGNU ELPA, which repo does this access?
  > > Our repo?

  > That is open, the repository it uses depends on the metadata that ELPA
  > provides.  If that is missing it uses a heuristic to guess a repository
  > from the URL, in case contains a link to a "well-known" source force.

I think NonGNU ELPA always has its own repo for every package.  For
some packages, that repo gets updated automatically from an outside
repo.  For a package that is updated automatically like that, I think
it is best to use the outside repo.

When the NonGNU ELPA repo for the package is NOT updated
automatically, I think this should use the NonGNU ELPA repo.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Fetching or installing package dev source from VCS: names

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > In the case of NonGNU ELPA, which repo does this access?
>   > > Our repo?
>
>   > That is open, the repository it uses depends on the metadata that ELPA
>   > provides.  If that is missing it uses a heuristic to guess a repository
>   > from the URL, in case contains a link to a "well-known" source force.
>
> I think NonGNU ELPA always has its own repo for every package.  For
> some packages, that repo gets updated automatically from an outside
> repo.  For a package that is updated automatically like that, I think
> it is best to use the outside repo.

Unless I am mistaken, this is the default for NonGNU ELPA.

> When the NonGNU ELPA repo for the package is NOT updated
> automatically, I think this should use the NonGNU ELPA repo.

As I said, my understanding is that this would only apply to GNU ELPA.

Re: Fetching or installing package dev source from VCS: names

[Stefan Monnier]

>> When the NonGNU ELPA repo for the package is NOT updated
>> automatically, I think this should use the NonGNU ELPA repo.
> As I said, my understanding is that this would only apply to GNU ELPA.

While it's currently the case, I would be surprised if we won't ever get
into the situation where a maintainerless NonGNU ELPA packages needs
changes, and where we end up hosting them directly in `nongnu.git`  :-(


        Stefan

Re: Fetching or installing package dev source from VCS: names

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think NonGNU ELPA always has its own repo for every package.  For
  > > some packages, that repo gets updated automatically from an outside
  > > repo.  For a package that is updated automatically like that, I think
  > > it is best to use the outside repo.

  > Unless I am mistaken, this is the default for NonGNU ELPA.

It may be the usual case for NonGNU ELPA, but it cannot be the default
as such.  In order to use automatic updating of a package in NonGNU
ELPA, we need a certain commitment from the developer to cooperate
with what NonGNU ELPA needs.

So we also have the option of NOT updating the NonGNU ELPA version
automatically.  That is the fallback option, which we can always use
if we can't use the more convenient (for us) automatic updating.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

package-contact-maintainer

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

`package-contact-maintainer' does a useful job in an overengineered
way.  To facilitate sending email to the maintainers of a package, it
would be cleaner to have a command `package-maintainer' that puts the
maintainer(s)' address(es) in the kill ring.  Then you could yank that
in your favorite mail reader.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: package-contact-maintainer

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> `package-contact-maintainer' does a useful job in an overengineered
> way.  To facilitate sending email to the maintainers of a package, it
> would be cleaner to have a command `package-maintainer' that puts the
> maintainer(s)' address(es) in the kill ring.  Then you could yank that
> in your favorite mail reader.

I like the idea, but I don't think I would have guessed that a command
named `package-maintainers' would do that.  Not that adding an item is
that drastic of a move anyway.  Then again, mixing in the word "kill"
(kill-package-maintainers, package-kill-maintainers,
package-maintainers-kill, ...) doesn't help either...

Re: package-contact-maintainer

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> I like the idea, but I don't think I would have guessed that a command
> named `package-maintainers' would do that.  Not that adding an item is
> that drastic of a move anyway.  Then again, mixing in the word "kill"
> (kill-package-maintainers, package-kill-maintainers,
> package-maintainers-kill, ...) doesn't help either...

See:

    (apropos-function "copy.*-as-kill$")

Re: package-contact-maintainer

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> I like the idea, but I don't think I would have guessed that a command
>> named `package-maintainers' would do that.  Not that adding an item is
>> that drastic of a move anyway.  Then again, mixing in the word "kill"
>> (kill-package-maintainers, package-kill-maintainers,
>> package-maintainers-kill, ...) doesn't help either...
>
> See:
>
>     (apropos-function "copy.*-as-kill$")

Thanks, but upon thinking about it I think the command is actually
unnecessary, considering that you can get the same information from
`describe-package'.  I've removed it for now from the branch.

Fetching or installing package dev source from VCS: manual style

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The proposed manual text includes several sentences which use passive
voice and would be shorter in active voice.  Occasionally the use of
passive voice is best approach, but usually not, so please try making
them active and see if that makes the text more readable.

Please don't refer to a maintainer, singular, as "them".
There are many other ways to write that sentence, and some
don't use a pronoun at all.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Fetching or installing package dev source from VCS: manual style

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> The proposed manual text includes several sentences which use passive
> voice and would be shorter in active voice.  Occasionally the use of
> passive voice is best approach, but usually not, so please try making
> them active and see if that makes the text more readable.
>
> Please don't refer to a maintainer, singular, as "them".
> There are many other ways to write that sentence, and some
> don't use a pronoun at all.

Thank you for the input, how do you like this:

--8<---------------cut here---------------start------------->8---
@node Package from Source
@section Package from Source
@cindex package development source
@cindex package upstream source
@cindex package git source @c "git" is not technically correct

  By default @code{package-install} will download a tarball from a
package archive and install the files therein contained.  Most of the
time this is just what you want.  One exception is when you decide to
hack on the source code of a package, and would like to share these
changes with other users.  In that case you usually want to fetch and
work on the upstream source, so that you can prepare a usable patch.

@findex package-vc-install
  One way to do this is to use @code{package-vc-install}, to fetch the
source code for a package directly from source.  The command will also
automatically ensure that all files are byte-compiled and auto-loaded,
just like with a regular package.  Packages installed this way behave
just like any other package.  You can update them using
@code{package-update} or @code{package-update-all} and delete them
again using @code{package-delete}.  They are even displayed in the
regular package listing.

@findex package-report-bug
@findex package-vc-prepare-patch
  With the source checkout, you might want to reproduce a bug against
the current development head or implement a new feature to scratch an
itch.  If the package metadata indicates how to contact the
maintainer, you can use the command @code{package-report-bug} to
report a bug via Email.  This report will include all the user options
that you have customised.  If you have made a change you wish to share
with the maintainers, first commit your changes then use the command
@code{package-vc-prepare-patch} to share it.  @xref{Preparing Patches}.

@findex package-vc-link-directory
@findex package-vc-refresh
  If you maintain your own packages you might want to use a local
checkout instead of cloning a remote repository.  You can do this by
using @code{package-vc-link-directory}, which creates a symbolic link
from the package directory (@pxref{Package Files}) to your checkout
and initialises the code.  Note that you might have to use
@code{package-vc-refresh} to repeat the initialisation and update the
autoloads.
--8<---------------cut here---------------end--------------->8---

Re: Fetching or installing package dev source from VCS: manual style

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

THANK YOU for working on this *amazing* feature!

Below, I suggest some optional, minor edits.

> By default @code{package-install} will download a tarball from a
> package archive and install the files therein contained.

will download -> downloads (+ install -> installs)
the files therein contained -> its files

> Most of the time this is just what you want.

most of the time -> often
just ->

> One exception is when you decide to hack on the source code of a
> package, and would like to share these changes with other users.  In
> that case you usually want to fetch and work on the upstream source,
> so that you can prepare a usable patch.

If you want to hack on the source code of a package, and perhaps share
your changes with others, you may prefer to fetch and work on the
upstream source, so that you can prepare a patch.

...

P.S. I wonder if this new feature integrates with version control logs.
That alone, having any kind of "what is new" screen, would make me
switch everything to use the new system.  I think the "blind updates"
that we have today do not match "Emacs spirit" at all.

Rudy
-- 
"Logic is a science of the necessary laws of thought, without which no
employment of the understanding and the reason takes place."
-- Immanuel Kant, 1785

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: Fetching or installing package dev source from VCS: manual style

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
> THANK YOU for working on this *amazing* feature!

I am glad it is of use!

> Below, I suggest some optional, minor edits.
>
>> By default @code{package-install} will download a tarball from a
>> package archive and install the files therein contained.
>
> will download -> downloads (+ install -> installs)
> the files therein contained -> its files

Applied.

>> Most of the time this is just what you want.
>
> most of the time -> often
> just ->

Perhaps the entire sentence is unnecessary?  

>> One exception is when you decide to hack on the source code of a
>> package, and would like to share these changes with other users.  In
>> that case you usually want to fetch and work on the upstream source,
>> so that you can prepare a usable patch.
>
> If you want to hack on the source code of a package, and perhaps share
> your changes with others, you may prefer to fetch and work on the
> upstream source, so that you can prepare a patch.

I get the impression that the sentence is too long, perhaps it can be
cut down?  How is this:

    By default @code{package-install} downloads a Tarball from a package
  archive and installs its files.  This might be inadequate if you wish
  to hack on the package sources and add new features.  In that case
  others, you may prefer to directly fetch and work on the upstream
  source.  This will make it easier to develop patches and report bugs.

It might also be good to mention that you can "downgrade" to regular
packages after having installed a source package, but that this could
lead to issues if user options have been changed.  Yet there is no
inherent conflict between the two types of packages.

> ...
>
> P.S. I wonder if this new feature integrates with version control logs.
> That alone, having any kind of "what is new" screen, would make me
> switch everything to use the new system.  I think the "blind updates"
> that we have today do not match "Emacs spirit" at all.

It wouldn't be difficult to integrate 'vc-log-incoming' into some
command like 'package-vc-whats-new'.

> Rudy

Re: Fetching or installing package dev source from VCS: manual style

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> [...]  This might be inadequate if you wish to hack on the package
> sources and add new features.

I will let *you* cook. :)  One last quick comment:

The new version says "hack on the package sources and add new features",
but someone might want to fix a bug, perhaps a typo, not add a new
feature.  Or they might want to simply read the code to learn!  The old
version said it much more *beautifully*, "hack on the source code of a
package, and perhaps share your changes with others".

> It wouldn't be difficult to integrate 'vc-log-incoming' into some
> command like 'package-vc-whats-new'.

We would still need wire that up to e.g. 'package-update-all' so that it
shows What's New for *all* packages instead of, wait for it, ...

One package to update.  Do it? (y or n)

:-)

Rudy
-- 
"Be especially critical of any statement following the word
'obviously.'"
-- Anna Pell Wheeler, 1883-1966

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: Fetching or installing package dev source from VCS: manual style

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> [...]  This might be inadequate if you wish to hack on the package
>> sources and add new features.
>
> I will let *you* cook. :)  One last quick comment:
>
> The new version says "hack on the package sources and add new features",
> but someone might want to fix a bug, perhaps a typo, not add a new
> feature.  Or they might want to simply read the code to learn!  The old
> version said it much more *beautifully*, "hack on the source code of a
> package, and perhaps share your changes with others".

How about "hack on the package sources and add new features to share
with others"?

>> It wouldn't be difficult to integrate 'vc-log-incoming' into some
>> command like 'package-vc-whats-new'.
>
> We would still need wire that up to e.g. 'package-update-all' so that it
> shows What's New for *all* packages instead of, wait for it, ...
>
> One package to update.  Do it? (y or n)

Hmm, I see what you mean but I don't know if this is a good idea in
general when using `package-update-all' to have a log buffer pop up for
every package.

> :-)
>
> Rudy

Re: Fetching or installing package dev source from VCS: manual style

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> How about "hack on the package sources and add new features to share
> with others"?

Honestly?  The term "new features" has become the central mantra of all
modern "agile" software (read: "shit").  With free software, we can have
a more mature approach.

One might want to hack on a package to learn from it.  Or to discuss it
with a friend.  Or to add some tests.  Or simply to have fun!

(If I had to pick two verbs for the sentence, I would go with "hack" and
"share".  No "new features" needed.)

> Hmm, I see what you mean but I don't know if this is a good idea in
> general when using `package-update-all' to have a log buffer pop up
> for every package.

I imagine a list of packages, similar to 'list-packages', e.g.

| Package       | Current |   New | Details          | Description 
|---------------+---------+-------+------------------|------------------
| [[ vertico ]] |    0.27 |  0.28 | [[ 17 changes ]] | VERTical Inte...
| [[ magit   ]] |   3.2.1 | 3.3.0 | [[ 80 changes ]] | A Git porcela...

where [[ ... ]] denotes a button.

A single click on the "17 changes" button would then open the version
control log with the 17 commits between Vertico 0.27 and 0.28, Emacs
experience!

Rudy
-- 
"'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and
if it were so, it would be; but as it isn't, it ain't.  That's logic.'"
-- Lewis Carroll, Through the Looking Glass, 1871/1872

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: Fetching or installing package dev source from VCS: manual style

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> How about "hack on the package sources and add new features to share
>> with others"?
>
> Honestly?  The term "new features" has become the central mantra of all
> modern "agile" software (read: "shit").  With free software, we can have
> a more mature approach.
>
> One might want to hack on a package to learn from it.  Or to discuss it
> with a friend.  Or to add some tests.  Or simply to have fun!
>
> (If I had to pick two verbs for the sentence, I would go with "hack" and
> "share".  No "new features" needed.)

I have no strong opinions on the specific phrasing, as I am not
convinced that "new features" is anything more than a descriptive term.
But if you prefer it, we can change it to "hack on packages and share
your changes with others".

>> Hmm, I see what you mean but I don't know if this is a good idea in
>> general when using `package-update-all' to have a log buffer pop up
>> for every package.
>
> I imagine a list of packages, similar to 'list-packages', e.g.
>
> | Package       | Current |   New | Details          | Description 
> |---------------+---------+-------+------------------|------------------
> | [[ vertico ]] |    0.27 |  0.28 | [[ 17 changes ]] | VERTical Inte...
> | [[ magit   ]] |   3.2.1 | 3.3.0 | [[ 80 changes ]] | A Git porcela...
>
> where [[ ... ]] denotes a button.
>
> A single click on the "17 changes" button would then open the version
> control log with the 17 commits between Vertico 0.27 and 0.28, Emacs
> experience!

I am uncertain if vc would allow for such a thing to be inspected right
now.  I'd like to be able to get package-vc into a functional state
before Emacs 29 is cut off, so I would consider this at a later point.

> Rudy

Re: Fetching or installing package dev source from VCS: manual style

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > --8<---------------cut here---------------start------------->8---
  > @node Package from Source
  > @section Package from Source
  > @cindex package development source
  > @cindex package upstream source
  > @cindex package git source @c "git" is not technically correct

That improves the @cindex commands.  But please look at the whole
manual's index and see where these two show up.  If two of them appear
close together, there is no need for both.  If they are separated  by
several other commands, then it is useful to have both.

I used to check the whole index for redundant index entries like this
before each Emacs release.

  > @findex package-vc-install
  >   One way to do this is to use @code{package-vc-install}, to fetch the
  > source code for a package directly from source.

I contend that `package-dev-source' or `package-upstream-source'
is a better name than `package-vc-install'.

They are better because they describe the crucial difference for which
you would use that command: to get the development source rather than
the current release source.  It is true that the development source
will usually come from a VCS, but that's an secondary detail, not the
crucial point.

The current version's source may be stored in a VCS also.  Isn't that
normally the case for NonGNU ELPA?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Fetching or installing package dev source from VCS: manual style

[Stefan Monnier]

>   > @findex package-vc-install
>   >   One way to do this is to use @code{package-vc-install}, to fetch the
>   > source code for a package directly from source.
>
> I contend that `package-dev-source' or `package-upstream-source'
> is a better name than `package-vc-install'.
>
> They are better because they describe the crucial difference for which
> you would use that command: to get the development source rather than
> the current release source.  It is true that the development source
> will usually come from a VCS, but that's an secondary detail, not the
> crucial point.

I don't have a strong opinion either way, but I'll note that the purpose
of this new feature is to fetch&install the code from a VCS.  It's not
to fetch the "development" code instead of the "release" code (that's
already served by the difference between GNU ELPA and GNU-devel ELPA).

E.g. so you can have local changes while still tracking upstream
updates.  There are some technical hurdles that make it harder for
`package-vc-install` to track upstream "releases" rather than upstream
"development" (at least in the general case: in a case like Org where
releases come from a separate branch it can be pretty easy), but it's
definitely doable and I actually hope we'll be able to add such
a feature at some point.


        Stefan

Re: Fetching or installing package dev source from VCS: manual style

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > --8<---------------cut here---------------start------------->8---
>   > @node Package from Source
>   > @section Package from Source
>   > @cindex package development source
>   > @cindex package upstream source
>   > @cindex package git source @c "git" is not technically correct
>
> That improves the @cindex commands.  But please look at the whole
> manual's index and see where these two show up.  If two of them appear
> close together, there is no need for both.  If they are separated  by
> several other commands, then it is useful to have both.

3 matches for "Package from Source" in buffer: *info*
   1511:* package development source:            Package from Source.
   1515:* package git source:                    Package from Source.
   1525:* package upstream source:               Package from Source.

They seem close-by, so I am not sure if it is worth it.

> I used to check the whole index for redundant index entries like this
> before each Emacs release.

I have to admit that I have never used the concept index directly.

>   > @findex package-vc-install
>   >   One way to do this is to use @code{package-vc-install}, to fetch the
>   > source code for a package directly from source.
>
> I contend that `package-dev-source' or `package-upstream-source'
> is a better name than `package-vc-install'.

So would you have the entire package renamed?  I guess "package-vc" or
"package-dev" aren't that different, and I'm fine with both names.

> They are better because they describe the crucial difference for which
> you would use that command: to get the development source rather than
> the current release source.  It is true that the development source
> will usually come from a VCS, but that's an secondary detail, not the
> crucial point.
>
> The current version's source may be stored in a VCS also.  Isn't that
> normally the case for NonGNU ELPA?

Yes.

Re: Fetching or installing package dev source from VCS: manual style

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Wed, 19 Oct 2022 06:58:02 +0000
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> >   > --8<---------------cut here---------------start------------->8---
> >   > @node Package from Source
> >   > @section Package from Source
> >   > @cindex package development source
> >   > @cindex package upstream source
> >   > @cindex package git source @c "git" is not technically correct
> >
> > That improves the @cindex commands.  But please look at the whole
> > manual's index and see where these two show up.  If two of them appear
> > close together, there is no need for both.  If they are separated  by
> > several other commands, then it is useful to have both.
> 
> 3 matches for "Package from Source" in buffer: *info*
>    1511:* package development source:            Package from Source.
>    1515:* package git source:                    Package from Source.
>    1525:* package upstream source:               Package from Source.
> 
> They seem close-by, so I am not sure if it is worth it.

What's worse, they all point to the same place in the text.

What I recommend doing in these cases is to reshuffle the words:

 @cindex package development source
 @cindex upstream source, for packages
 @cindex git source of package @c "git" is not technically correct

This way, typing "package TAB" will still show all of them, but if
someone thinks about "upstream source", for example, they can type
"upstream source TAB" and find this.

> I have to admit that I have never used the concept index directly.

It is not very useful, except for finding duplicate and redundant
entries.

Re: Fetching or installing package dev source from VCS: manual style

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 19 Oct 2022 06:58:02 +0000
>> 
>> Richard Stallman <rms@gnu.org> writes:
>> 
>> >   > --8<---------------cut here---------------start------------->8---
>> >   > @node Package from Source
>> >   > @section Package from Source
>> >   > @cindex package development source
>> >   > @cindex package upstream source
>> >   > @cindex package git source @c "git" is not technically correct
>> >
>> > That improves the @cindex commands.  But please look at the whole
>> > manual's index and see where these two show up.  If two of them appear
>> > close together, there is no need for both.  If they are separated  by
>> > several other commands, then it is useful to have both.
>> 
>> 3 matches for "Package from Source" in buffer: *info*
>>    1511:* package development source:            Package from Source.
>>    1515:* package git source:                    Package from Source.
>>    1525:* package upstream source:               Package from Source.
>> 
>> They seem close-by, so I am not sure if it is worth it.
>
> What's worse, they all point to the same place in the text.
>
> What I recommend doing in these cases is to reshuffle the words:
>
>  @cindex package development source
>  @cindex upstream source, for packages
>  @cindex git source of package @c "git" is not technically correct

This looks a lot better:

3 matches for "Package from Source" in buffer: *info*
    800:* git source of package:                 Package from Source.
   1513:* package development source:            Package from Source.
   2083:* upstream source, for packages:         Package from Source.


> This way, typing "package TAB" will still show all of them, but if
> someone thinks about "upstream source", for example, they can type
> "upstream source TAB" and find this.

That is also good.

>> I have to admit that I have never used the concept index directly.
>
> It is not very useful, except for finding duplicate and redundant
> entries.

Re: Fetching or installing package dev source from VCS: manual style

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> 3 matches for "Package from Source" in buffer: *info*
  > >>    1511:* package development source:            Package from Source.
  > >>    1515:* package git source:                    Package from Source.
  > >>    1525:* package upstream source:               Package from Source.

The first two are so close that they are redundant; it is better to
have just one of them.  I think "development" is the more important of
the two.

The third one is 10 lines away.  Maybe that is far enough not to be
redundant.  It is a judgment call.

  > 3 matches for "Package from Source" in buffer: *info*
  >     800:* git source of package:                 Package from Source.
  >    1513:* package development source:            Package from Source.
  >    2083:* upstream source, for packages:         Package from Source.

Rotating the words in an index entry makes another index entry.
It can be useful to have multiple rotations of the same words.

For instance, "development source, of package" might be useful in
addition to "package development source".  "Upstream source, of
package" might be useful in addition to "package upstream source".

Each of them is useful to have in the index if more than a few users
will look for the term that way in the index, and it isn't redundant
with other index entries for the same place.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Multiple index entries

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> 3 matches for "Package from Source" in buffer: *info*
  > >>    1511:* package development source:            Package from Source.
  > >>    1515:* package git source:                    Package from Source.
  > >>    1525:* package upstream source:               Package from Source.

The first two are so close that they are redundant; it is better to
have just one of them.  I think "development" is the more important of
the two.

The third one is 10 lines away.  Maybe that is far enough not to be
redundant.  It is a judgment call.

  > 3 matches for "Package from Source" in buffer: *info*
  >     800:* git source of package:                 Package from Source.
  >    1513:* package development source:            Package from Source.
  >    2083:* upstream source, for packages:         Package from Source.

Rotating the words in an index entry makes another index entry
which normally would appear far away in the index.
It can be useful to index multiple rotations of the same expression.

For instance, "development source, of package" might be useful in
addition to "package development source".  "Upstream source, of
package" might be useful in addition to "package upstream source".

Each of them is useful to have in the index if more than a few users
will look for the term that way in the index, and it isn't redundant
with other index entries that would be close by.

Back when I readied the Emacs Manual for publication in each version,
I used to check the whole concept index for various blemishes,
and redundant index entries was one of them.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

"Package from Source"

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The node name "Package from Source" is somewhat confusing.  This
command fetches the source from the development repo.  Sometimes a
user will want that, but sometimes a user will want to look at or edit
the source of the current release of the package.

Let's keep these two issues separate:
1) To see the package source files in a repo to edit them.
2) Which version to use.

In other words, there should be a command to check out the current version 
sources, and a command to check out the development version sources.
Or maybe one command that can do either of those two.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: "Package from Source"

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> The node name "Package from Source" is somewhat confusing.  This
> command fetches the source from the development repo.  Sometimes a
> user will want that, but sometimes a user will want to look at or edit
> the source of the current release of the package.

Do you have a suggestion for a better name?  I'd gladly apply it if it
makes sense.

> Let's keep these two issues separate:
> 1) To see the package source files in a repo to edit them.
> 2) Which version to use.

Right, 1) is implemented and could be merged if there is an ack from
above™ and 2) has not been implemented yet, but I plan to do so when I
find the time.

> In other words, there should be a command to check out the current version 
> sources, and a command to check out the development version sources.
> Or maybe one command that can do either of those two.

I can imagine that both are possible with a single command, that would
take a prefix argument when invoked interactively.  Another idea would
be to use a user option.

Re: "Package from Source"

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > In other words, there should be a command to check out the current version 
  > > sources, and a command to check out the development version sources.
  > > Or maybe one command that can do either of those two.

  > I can imagine that both are possible with a single command, that would
  > take a prefix argument when invoked interactively.

A prefix argument would work.

I tend to think that this difference is important enough
that it is better to have two different command names.

                                                        Another idea would
  > be to use a user option.

I think we should encourage people to make this choice on a
package-by-package basis.  A user option would push users
to handle all packages the same way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Installation from ELPA

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Does installation of a package from ELPA normally load only
the binary?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Installation from ELPA

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> cc: emacs-devel@gnu.org
> Date: Sun, 23 Oct 2022 15:11:50 -0400
> 
> Does installation of a package from ELPA normally load only
> the binary?

What do you mean by "binary" in this context?  We are talking about
Lisp programs, so there should be no "binaries" in the usual sense of
this word.

Re: Installation from ELPA

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Does installation of a package from ELPA normally load only
  > > the binary?

  > What do you mean by "binary" in this context?  We are talking about
  > Lisp programs, so there should be no "binaries" in the usual sense of
  > this word.

In general, "binaries" in the context of discussions of source code
means any transformed files made from source code, which are not
themselves source code.

It would have been more precise and rigorous to write "non-source
files", but I don't think it would actually be clearer.

The non-source files generated from Emacs Lisp code are truly
binaries.  Byte-compiled files represent compiled functions using
strings of binary data that contain the byte codes.  There are also
native-compiled files, which contain binary code for machine
instructions.

This question is pertinent to the issue of what ways of
fetching a package from ELPA would be useful to have.
So I hope someone will post the answer.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Installation from ELPA

[Stefan Monnier]

Richard Stallman [2022-10-23 15:11:50] wrote:
> Does installation of a package from ELPA normally load only
> the binary?

No, it normally fetches a tarball which contains pretty much the same
contents as the upstream's Git repository.  We then byte-compile
the ELisp files locally.


        Stefan

Re: Installation from ELPA

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, it normally fetches a tarball which contains pretty much the same
  > contents as the upstream's Git repository.  We then byte-compile
  > the ELisp files locally.

That means the user does get the source by default with the corrent
design.  And could edit them, too.

That mans the node name "Package from Source" is quite misleading.

Using the proposed package-repo command would create a checkout
of the ELPA repo, which is different from the current way of
installing a package, and that might be useful.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Fetching or installing package dev source from VCS: manual style

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   If you have made a change you wish to share
  > with the maintainers, first commit your changes then use the command
  > @code{package-vc-prepare-patch} to share it.  @xref{Preparing Patches}.

This presumes that a user who suggests a patch to a program will or
should commit that change to some repo in order to email the a patch.
This is an extra step and seems like a gratuitous requirement.

When I write and suggest a patch in a program that I don't intend to
maintain a version of, I don't even think of doing that.  I use M-x
diff to make the patch to email, because that is the natural Emacs
way.

I think the command should be named `package-prepare-patch', and it
should be able to handle either way of working, with version control
or without.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Fetching or installing package dev source from VCS: manual style

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 18 Oct 2022 08:05:29 -0400
> 
>   >   If you have made a change you wish to share
>   > with the maintainers, first commit your changes then use the command
>   > @code{package-vc-prepare-patch} to share it.  @xref{Preparing Patches}.
> 
> This presumes that a user who suggests a patch to a program will or
> should commit that change to some repo in order to email the a patch.
> This is an extra step and seems like a gratuitous requirement.

It is quite an accepted practice these days.  We prefer to get such
patches because they are easier to install -- a single command applies
the patch and makes a commit with the log message provided with the
patch.

On the contributor's side, committing is easy, and if the contributor
wants, he or she can back out the commit right away, or do it on a
scratch branch, or another of the gazillion techniques for that.

Or request for "git format-patch" in CONTRIBUTE assumes this workflow.

Re: Fetching or installing package dev source from VCS: manual style

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   >   If you have made a change you wish to share
>   > with the maintainers, first commit your changes then use the command
>   > @code{package-vc-prepare-patch} to share it.  @xref{Preparing Patches}.
>
> This presumes that a user who suggests a patch to a program will or
> should commit that change to some repo in order to email the a patch.
> This is an extra step and seems like a gratuitous requirement.
>
> When I write and suggest a patch in a program that I don't intend to
> maintain a version of, I don't even think of doing that.  I use M-x
> diff to make the patch to email, because that is the natural Emacs
> way.
>
> I think the command should be named `package-prepare-patch', and it
> should be able to handle either way of working, with version control
> or without.

IMO in that case Emacs already does everything you need and there is no
point in adding an additional command.  `package-prepare-patch' would be
part of package.el, and couldn't make use of the versioning history to
generate a diff.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 92 bytes --]


Here is a patch that would add all the necessary metadata for package-vc
to work to ELPA:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-package-vc-to-package-metadata.patch --]
[-- Type: text/x-patch, Size: 1451 bytes --]

From 98ac710ca97ebca5be1f9bf49b40a701c541b21d Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Mon, 17 Oct 2022 00:05:17 +0200
Subject: [PATCH] Add 'package-vc' to package metadata

* elpa-admin.el (elpaa--metadata): Add :upstream property.
---
 elpa-admin.el | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/elpa-admin.el b/elpa-admin.el
index 054c9bda86..f22c345cd9 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -1074,6 +1074,18 @@ PKG is the name of the package and DIR is the directory where it is."
           (unless found-url
             ;; Provide a good default URL.
             (push (cons :url (elpaa--default-url pkg)) extras))
+          ;; Add `package-vc' data
+          (let ((spec (cdr pkg-spec))
+                (ups (list :upstream 'Git nil nil nil)))
+            (if (null (plist-get spec :url))
+                (setf (nth 2 ups) (concat "https://git.savannah.gnu.org/git/"
+                                          elpaa--gitrepo)
+                      (nth 3 ups) (plist-get spec :lisp-dir)
+                      (nth 4 ups) (concat elpaa--branch-prefix pkg))
+              (setf (nth 2 ups) (plist-get spec :url)
+                    (nth 3 ups) (plist-get spec :lisp-dir)
+                    (nth 4 ups) (plist-get spec :branch)))
+            (push ups extras))
           (list simple
 		(package-version-join version)
 		(package-desc-summary pkg-desc)
-- 
2.38.0


[-- Attachment #3: Type: text/plain, Size: 2187 bytes --]


This does not special optimisation to minimise the data.  I guess this
is also a good point to hear objections to the current way the VC data
is handled.  Here is an example entry from "archive-contents", note the
new `:upstream' entry in the metadata field (the last element of the
vector):

--8<---------------cut here---------------start------------->8---
 (setup .
        [(1 3 0)
         ((emacs
           (26 1)))
         "Helpful Configuration Macro" tar
         ((:url . "https://git.sr.ht/~pkal/setup")
          (:keywords "lisp" "local")
          (:maintainer "Philip Kaludercic" . "~pkal/public-inbox@lists.sr.ht")
          (:authors
-->        ("Philip Kaludercic" . "philipk@posteo.net"))
          (:upstream Git "https://git.sr.ht/~pkal/setup" nil nil)
          (:commit . "eece09d1151fd641f31d738b8c62742918993e95"))])
--8<---------------cut here---------------end--------------->8---

or

--8<---------------cut here---------------start------------->8---
 (smalltalk-mode .
                 [(4 0)
                  nil "Major mode for the GNU Smalltalk programming language" tar
                  ((:maintainer "Derek Zhou" . "derek@3qin.us")
                   (:url . "https://elpa.gnu.org/packages/smalltalk-mode.html")
-->                (:upstream Git "https://git.savannah.gnu.org/git/emacs/elpa.git" nil "externals/smalltalk-mode")
                   (:commit . "570b705db9a02bb48cd61652639401715f419447"))])
--8<---------------cut here---------------end--------------->8---

The value corresponding to :upstream is a list of the form

    (VC-BACKEND REPOSITORY-URL SUBDIRECTORY BRANCH)

I have been wondering if it would be better to use a {a,p}list instead,
which would be more flexible if changes would become necessary in the
future.  Perhaps I am even missing something right now?

At the same time I don't mean to burden the ELPA server by increasing
the size of the file that is regularly queried.  Perhaps REPOSITORY-URL
could have special values such as `:url' to indicate that the package
URL is the same as the repository and `:elpa'/`:nongnu' to indicate that
the default repository is being used.  Or is this just premature
optimisation?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> The value corresponding to :upstream is a list of the form
>
>     (VC-BACKEND REPOSITORY-URL SUBDIRECTORY BRANCH)
>
> I have been wondering if it would be better to use a {a,p}list instead,
> which would be more flexible if changes would become necessary in the
> future.  Perhaps I am even missing something right now?
>
> At the same time I don't mean to burden the ELPA server by increasing
> the size of the file that is regularly queried.  Perhaps REPOSITORY-URL
> could have special values such as `:url' to indicate that the package
> URL is the same as the repository and `:elpa'/`:nongnu' to indicate that
> the default repository is being used.  Or is this just premature
> optimisation?

I think it's important to make the format both extensible and readable,
so I'd probably lean towards an alist or plist.

If the reason not to do that is a concern that the file gets
bigger/slower, I think it would be good to produce some numbers.
Without that, it does feel a bit like premature optimization.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> +          ;; Add `package-vc' data
> +          (let ((spec (cdr pkg-spec))
> +                (ups (list :upstream 'Git nil nil nil)))
> +            (if (null (plist-get spec :url))
> +                (setf (nth 2 ups) (concat "https://git.savannah.gnu.org/git/"
> +                                          elpaa--gitrepo)
> +                      (nth 3 ups) (plist-get spec :lisp-dir)
> +                      (nth 4 ups) (concat elpaa--branch-prefix pkg))
> +              (setf (nth 2 ups) (plist-get spec :url)
> +                    (nth 3 ups) (plist-get spec :lisp-dir)
> +                    (nth 4 ups) (plist-get spec :branch)))
> +            (push ups extras))

I think rather than invent a new format and duplicate that info into
`archive-contents`, pushing us to worry about its size, I'd prefer to
just stash a copy of `elpa-packages` alongside `archive-contents`.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +          ;; Add `package-vc' data
>> +          (let ((spec (cdr pkg-spec))
>> +                (ups (list :upstream 'Git nil nil nil)))
>> +            (if (null (plist-get spec :url))
>> +                (setf (nth 2 ups) (concat "https://git.savannah.gnu.org/git/"
>> +                                          elpaa--gitrepo)
>> +                      (nth 3 ups) (plist-get spec :lisp-dir)
>> +                      (nth 4 ups) (concat elpaa--branch-prefix pkg))
>> +              (setf (nth 2 ups) (plist-get spec :url)
>> +                    (nth 3 ups) (plist-get spec :lisp-dir)
>> +                    (nth 4 ups) (plist-get spec :branch)))
>> +            (push ups extras))
>
> I think rather than invent a new format and duplicate that info into
> `archive-contents`, pushing us to worry about its size, I'd prefer to
> just stash a copy of `elpa-packages` alongside `archive-contents`.

That sounds like a good idea, as long as we are fine with
"elpa-packages" being a public interface that couldn't just be changed
at will.

I'll try it out and report back.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-10-17 17:21:35] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> +          ;; Add `package-vc' data
>>> +          (let ((spec (cdr pkg-spec))
>>> +                (ups (list :upstream 'Git nil nil nil)))
>>> +            (if (null (plist-get spec :url))
>>> +                (setf (nth 2 ups) (concat "https://git.savannah.gnu.org/git/"
>>> +                                          elpaa--gitrepo)
>>> +                      (nth 3 ups) (plist-get spec :lisp-dir)
>>> +                      (nth 4 ups) (concat elpaa--branch-prefix pkg))
>>> +              (setf (nth 2 ups) (plist-get spec :url)
>>> +                    (nth 3 ups) (plist-get spec :lisp-dir)
>>> +                    (nth 4 ups) (plist-get spec :branch)))
>>> +            (push ups extras))
>>
>> I think rather than invent a new format and duplicate that info into
>> `archive-contents`, pushing us to worry about its size, I'd prefer to
>> just stash a copy of `elpa-packages` alongside `archive-contents`.
>
> That sounds like a good idea, as long as we are fine with
> "elpa-packages" being a public interface that couldn't just be changed
> at will.

We can massage it a bit to make it more "future proof".
E.g. we can start by removing the `:core` packages :-)


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-10-17 17:21:35] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> +          ;; Add `package-vc' data
>>>> +          (let ((spec (cdr pkg-spec))
>>>> +                (ups (list :upstream 'Git nil nil nil)))
>>>> +            (if (null (plist-get spec :url))
>>>> +                (setf (nth 2 ups) (concat "https://git.savannah.gnu.org/git/"
>>>> +                                          elpaa--gitrepo)
>>>> +                      (nth 3 ups) (plist-get spec :lisp-dir)
>>>> +                      (nth 4 ups) (concat elpaa--branch-prefix pkg))
>>>> +              (setf (nth 2 ups) (plist-get spec :url)
>>>> +                    (nth 3 ups) (plist-get spec :lisp-dir)
>>>> +                    (nth 4 ups) (plist-get spec :branch)))
>>>> +            (push ups extras))
>>>
>>> I think rather than invent a new format and duplicate that info into
>>> `archive-contents`, pushing us to worry about its size, I'd prefer to
>>> just stash a copy of `elpa-packages` alongside `archive-contents`.
>>
>> That sounds like a good idea, as long as we are fine with
>> "elpa-packages" being a public interface that couldn't just be changed
>> at will.
>
> We can massage it a bit to make it more "future proof".
> E.g. we can start by removing the `:core` packages :-)

What would you replace it with?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> E.g. we can start by removing the `:core` packages :-)
> What would you replace it with?

With nothing.  I presume here that your code doesn't know how to usefully
install, say, the `erc` package from `emacs.git`, so we just won't offer
the ability to install this package directly from Git.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> +          ;; Add `package-vc' data
>>> +          (let ((spec (cdr pkg-spec))
>>> +                (ups (list :upstream 'Git nil nil nil)))
>>> +            (if (null (plist-get spec :url))
>>> +                (setf (nth 2 ups) (concat "https://git.savannah.gnu.org/git/"
>>> +                                          elpaa--gitrepo)
>>> +                      (nth 3 ups) (plist-get spec :lisp-dir)
>>> +                      (nth 4 ups) (concat elpaa--branch-prefix pkg))
>>> +              (setf (nth 2 ups) (plist-get spec :url)
>>> +                    (nth 3 ups) (plist-get spec :lisp-dir)
>>> +                    (nth 4 ups) (plist-get spec :branch)))
>>> +            (push ups extras))
>>
>> I think rather than invent a new format and duplicate that info into
>> `archive-contents`, pushing us to worry about its size, I'd prefer to
>> just stash a copy of `elpa-packages` alongside `archive-contents`.
>
> That sounds like a good idea, as long as we are fine with
> "elpa-packages" being a public interface that couldn't just be changed
> at will.
>
> I'll try it out and report back.

I have pushed a first draft and it appears to work.  It doesn't yet
handle all attribute (such as :doc), but that should be doable when I
find the time.

I wanted to add support for elpa-admin.el, and it will boil down to
something like this:

  (with-temp-buffer
    (let ((filename (expand-file-name "elpa-packages" dir))
          (standard-output (current-buffer)))
      ;; Remove and compress the contents of the "elpa-pacakges"
      (prin1 (elpaa--get-specs))
      (write-region nil nil filename)))

but I am uncertain where to add it, so that it only runs when necessary.
Do you have any ideas?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> I have pushed a first draft and it appears to work.  It doesn't yet
> handle all attribute (such as :doc), but that should be doable when I
> find the time.

Side note: installing packages from Git (like what you're implementing
in the `package+vc` branch) is already one of the functionalities
supported by `elpa-admin.el` so it would be good to try and share/reuse
that code as much as possible.

Actually part of `elpa-admin.el` might also be helpful for the
ELPA-bundling feature.

So we should move more of `elpa-admin.el` into Emacs's own code.

> I wanted to add support for elpa-admin.el,

Not sure what you mean.

> and it will boil down to
> something like this:
>
>   (with-temp-buffer
>     (let ((filename (expand-file-name "elpa-packages" dir))
>           (standard-output (current-buffer)))
>       ;; Remove and compress the contents of the "elpa-pacakges"
>       (prin1 (elpaa--get-specs))
>       (write-region nil nil filename)))
>
> but I am uncertain where to add it, so that it only runs when necessary.
> Do you have any ideas?

Are you talking about the code which will run on `elpa.gnu.org` that is
supposed to push some kind of copy of `elpa-packages` to the web site so
clients can download it?

We just need to export this as a function which we can then run from
a cron job (tho you could also just call it from
`elpaa-batch-make-all-packages`, so it's run as part of the main cron
job).  But note that we can't just write out (elpaa--get-specs): we also
need to remove the `:core` packages and replace the nil `:url`s with
URLs pointing to `elpa.git` or `nongnu.git` (with corresponding `:branch`).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I have pushed a first draft and it appears to work.  It doesn't yet
>> handle all attribute (such as :doc), but that should be doable when I
>> find the time.
>
> Side note: installing packages from Git (like what you're implementing
> in the `package+vc` branch) is already one of the functionalities
> supported by `elpa-admin.el` so it would be good to try and share/reuse
> that code as much as possible.

That is only possible insofar I don't have to rely on Git features,
since package-vc intends to be VC-generic.

> Actually part of `elpa-admin.el` might also be helpful for the
> ELPA-bundling feature.
>
> So we should move more of `elpa-admin.el` into Emacs's own code.

As part of `package-vc' or as a new module?

>> I wanted to add support for elpa-admin.el,
>
> Not sure what you mean.
>
>> and it will boil down to
>> something like this:
>>
>>   (with-temp-buffer
>>     (let ((filename (expand-file-name "elpa-packages" dir))
>>           (standard-output (current-buffer)))
>>       ;; Remove and compress the contents of the "elpa-pacakges"
>>       (prin1 (elpaa--get-specs))
>>       (write-region nil nil filename)))
>>
>> but I am uncertain where to add it, so that it only runs when necessary.
>> Do you have any ideas?
>
> Are you talking about the code which will run on `elpa.gnu.org` that is
> supposed to push some kind of copy of `elpa-packages` to the web site so
> clients can download it?

Yes exactly.

> We just need to export this as a function which we can then run from
> a cron job (tho you could also just call it from
> `elpaa-batch-make-all-packages`, so it's run as part of the main cron
> job).  But note that we can't just write out (elpaa--get-specs): we also
> need to remove the `:core` packages and replace the nil `:url`s with
> URLs pointing to `elpa.git` or `nongnu.git` (with corresponding `:branch`).

I would prefer the latter and will try that out.

>
>         Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-10-19 07:08:34] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I have pushed a first draft and it appears to work.  It doesn't yet
>>> handle all attribute (such as :doc), but that should be doable when I
>>> find the time.
>> Side note: installing packages from Git (like what you're implementing
>> in the `package+vc` branch) is already one of the functionalities
>> supported by `elpa-admin.el` so it would be good to try and share/reuse
>> that code as much as possible.
> That is only possible insofar I don't have to rely on Git features,
> since package-vc intends to be VC-generic.

I'm thinking of things like the code that handles `:doc` or `:make`.
These don't care about Git.

I don't think that the code in `elpa-admin.el` can be used as-is, but it
would be good to try and change them in tandem (maybe by extracting the
common code into some new file?).

>> Actually part of `elpa-admin.el` might also be helpful for the
>> ELPA-bundling feature.
>> So we should move more of `elpa-admin.el` into Emacs's own code.
> As part of `package-vc' or as a new module?

I think either way is fine.  The whole `elpa-admin.el` contains things
that are irrelevant to Emacs itself, so we'll likely keep a separate
file like that, but the code that's useful for other uses
(i.e. package-vc and/or bundled-ELPA) should be moved out (tho
elpa-admin.el has to be backward compatible to some extent, so we'll
have to keep a "local" copy of whatever is moved to a common file
hosted in emacs.git, at least for a few years).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-10-19 07:08:34] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> I have pushed a first draft and it appears to work.  It doesn't yet
>>>> handle all attribute (such as :doc), but that should be doable when I
>>>> find the time.
>>> Side note: installing packages from Git (like what you're implementing
>>> in the `package+vc` branch) is already one of the functionalities
>>> supported by `elpa-admin.el` so it would be good to try and share/reuse
>>> that code as much as possible.
>> That is only possible insofar I don't have to rely on Git features,
>> since package-vc intends to be VC-generic.
>
> I'm thinking of things like the code that handles `:doc` or `:make`.
> These don't care about Git.
>
> I don't think that the code in `elpa-admin.el` can be used as-is, but it
> would be good to try and change them in tandem (maybe by extracting the
> common code into some new file?).

The only thing is that we will probably need to prompt the user if they
are OK with executing some command or using Makefile, right?  I can try
to re-use elpa-admin.el but the code will then have to be wrapped for
"regular" use by some safety precautions.

>>> Actually part of `elpa-admin.el` might also be helpful for the
>>> ELPA-bundling feature.
>>> So we should move more of `elpa-admin.el` into Emacs's own code.
>> As part of `package-vc' or as a new module?
>
> I think either way is fine.  The whole `elpa-admin.el` contains things
> that are irrelevant to Emacs itself, so we'll likely keep a separate
> file like that, but the code that's useful for other uses
> (i.e. package-vc and/or bundled-ELPA) should be moved out (tho
> elpa-admin.el has to be backward compatible to some extent, so we'll
> have to keep a "local" copy of whatever is moved to a common file
> hosted in emacs.git, at least for a few years).

Sounds good.  I'll first try and see if the code can be added to
package-vc.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> I don't think that the code in `elpa-admin.el` can be used as-is, but it
>> would be good to try and change them in tandem (maybe by extracting the
>> common code into some new file?).
> The only thing is that we will probably need to prompt the user if they
> are OK with executing some command or using Makefile, right?

Could be, I haven't thought about it.
[ `elpa-admin.el` runs all these commands via Bubblewrap, and
  `package-vc` will compile the files, which can end up running any code
  whatsoever anyway, so I'm not sure how important that is.  ]

> I can try to re-use elpa-admin.el but the code will then have to be
> wrapped for "regular" use by some safety precautions.

Yes, the code will definitely need adjustments for use in this
other context.  That shouldn't prevent sharing it, tho.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-10-19 07:08:34] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> I have pushed a first draft and it appears to work.  It doesn't yet
>>>> handle all attribute (such as :doc), but that should be doable when I
>>>> find the time.
>>> Side note: installing packages from Git (like what you're implementing
>>> in the `package+vc` branch) is already one of the functionalities
>>> supported by `elpa-admin.el` so it would be good to try and share/reuse
>>> that code as much as possible.
>> That is only possible insofar I don't have to rely on Git features,
>> since package-vc intends to be VC-generic.
>
> I'm thinking of things like the code that handles `:doc` or `:make`.
> These don't care about Git.

For something like :make to work, we would also require :renames, right?
But if that is added, then the version control could break.

> I don't think that the code in `elpa-admin.el` can be used as-is, but it
> would be good to try and change them in tandem (maybe by extracting the
> common code into some new file?).
>
>>> Actually part of `elpa-admin.el` might also be helpful for the
>>> ELPA-bundling feature.
>>> So we should move more of `elpa-admin.el` into Emacs's own code.
>> As part of `package-vc' or as a new module?
>
> I think either way is fine.  The whole `elpa-admin.el` contains things
> that are irrelevant to Emacs itself, so we'll likely keep a separate
> file like that, but the code that's useful for other uses
> (i.e. package-vc and/or bundled-ELPA) should be moved out (tho
> elpa-admin.el has to be backward compatible to some extent, so we'll
> have to keep a "local" copy of whatever is moved to a common file
> hosted in emacs.git, at least for a few years).

How does this look like:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-elpa-admin.el-elpaa-batch-make-all-packages-Generate.patch --]
[-- Type: text/x-patch, Size: 1834 bytes --]

From e4a2b7cdcf5fa62f301730db6b1e33d521d9d96f Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Fri, 21 Oct 2022 23:18:30 +0200
Subject: [PATCH] * elpa-admin.el (elpaa-batch-make-all-packages): Generate
 elpa-packages.eld

---
 elpa-admin.el | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/elpa-admin.el b/elpa-admin.el
index 054c9bda86..fc08dcb540 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -764,11 +764,25 @@ of the current `process-environment'.  Return the modified copy."
 
 (defun elpaa-batch-make-all-packages (&rest _)
   "Check all the packages and build the relevant new tarballs."
-  (let* ((specs (elpaa--get-specs)))
+  (let* ((git-sv "https://git.sv.gnu.org/")
+         (specs (elpaa--get-specs)))
     (dolist (spec specs)
       (condition-case err
           (elpaa--make-one-package spec)
-        (error (message "Build error for %s: %S" (car spec) err))))))
+        (error (message "Build error for %s: %S" (car spec) err))))
+    (with-temp-buffer
+      ;; Remove and compress the contents of the "elpa-packages"
+      (let ((standard-output (current-buffer)))
+        (dolist (spec specs)
+          (when (eq (cadr spec) :core)  ;handle "core" packages
+            (setf (cdr spec)
+                  (append
+                   (list :url (format "%s/git/%s" git-sv elpaa--gitrepo)
+                         :branch (concat elpaa--branch-prefix (car spec)))
+                   (cdddr spec)))))
+        (prin1 specs))
+      (write-region nil nil (expand-file-name "elpa-packages.eld" elpaa--release-subdir))
+      (write-region nil nil (expand-file-name "elpa-packages.eld" elpaa--devel-subdir)))))
 
 (defun elpaa-batch-make-one-package (&rest _)
   "Build the new tarballs (if needed) for one particular package."
-- 
2.38.0


[-- Attachment #3: Type: text/plain, Size: 248 bytes --]


It currently makes the assumption that :core is the first element in the
property list for each package specification.  This appears to be the
convention and makes the code simpler, but if you think it is not
justified, that part can be reworked.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> I'm thinking of things like the code that handles `:doc` or `:make`.
>> These don't care about Git.
> For something like :make to work, we would also require :renames, right?

No, it's done before renames.

> But if that is added, then the version control could break.

In the part of `elpa-admin.el` designed for "install from Git"
(i.e. the part that matches package+vc's goal), `:rename` is not taken
into account.  I don't think we can reasonably do so, as you point out,
so I think `:renames` should be labeled as something we can't support
(and eventually deprecated for that reason).

> How does this look like:

I'm a bit confused: it seems to do for `:core` what I expected to happen
for packages with a nil `:url` (and it doesn't drop `:core` packages,
like I'd expect).

Other than that, it looks OK (in the nitpick area, I'd move the code to
its own function and then call it from `elpaa-batch-make-all-packages`).

I like the `.eld` extension, and I guess "elpa-packages.eld" is as good
a name as any.

> It currently makes the assumption that :core is the first element in the
> property list for each package specification.

That's right: that element should be either `:core` or `:url`.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I'm thinking of things like the code that handles `:doc` or `:make`.
>>> These don't care about Git.
>> For something like :make to work, we would also require :renames, right?
>
> No, it's done before renames.
>
>> But if that is added, then the version control could break.
>
> In the part of `elpa-admin.el` designed for "install from Git"
> (i.e. the part that matches package+vc's goal), `:rename` is not taken
> into account.  I don't think we can reasonably do so, as you point out,
> so I think `:renames` should be labeled as something we can't support
> (and eventually deprecated for that reason).

OK, so I won't add support for that right now.

>> How does this look like:
>
> I'm a bit confused: it seems to do for `:core` what I expected to happen
> for packages with a nil `:url` (and it doesn't drop `:core` packages,
> like I'd expect).

That was my mistake, I misremembered that core packages aren't mirrored
in their own branches in elpa.git.  I've removed it here:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Generate-elpa-packages.eld-with-package-data.patch --]
[-- Type: text/x-patch, Size: 2187 bytes --]

From fe8530cde40d921b96ec29513d16d91114468e75 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Fri, 21 Oct 2022 23:18:30 +0200
Subject: [PATCH] Generate elpa-packages.eld with package data

* elpa-admin.el
(elpaa-publish-package-specs): Process specs and write file.
(elpaa-batch-make-all-packages): Call 'elpaa-publish-package-specs'.
---
 elpa-admin.el | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/elpa-admin.el b/elpa-admin.el
index 054c9bda86..00c3393499 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -762,13 +762,36 @@ of the current `process-environment'.  Return the modified copy."
 	  (list pkgname))
       spec)))
 
+(defun elpaa-publish-package-specs (specs)
+  "Process and publish SPECS in elpa-packages.eld files."
+  (with-temp-buffer
+    ;; Remove :core packages, handle :url nil and and compress the
+    ;; contents of the "elpa-packages"
+    (prin1
+     (mapcan
+      (lambda (spec)
+        (cond
+         ((and (plist-member (cdr spec) :url)
+               (null (plist-get (cdr spec) :url)))
+          `((,(car spec)
+             :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
+             :branch ,(concat elpaa--branch-prefix (car spec))
+             ,@(cdddr spec))))
+         ((plist-member (cdr spec) :core) nil)
+         ((list spec))))
+      specs)
+     (current-buffer))
+    (write-region nil nil (expand-file-name "elpa-packages.eld" elpaa--release-subdir))
+    (write-region nil nil (expand-file-name "elpa-packages.eld" elpaa--devel-subdir))))
+
 (defun elpaa-batch-make-all-packages (&rest _)
   "Check all the packages and build the relevant new tarballs."
-  (let* ((specs (elpaa--get-specs)))
+  (let ((specs (elpaa--get-specs)))
     (dolist (spec specs)
       (condition-case err
           (elpaa--make-one-package spec)
-        (error (message "Build error for %s: %S" (car spec) err))))))
+        (error (message "Build error for %s: %S" (car spec) err))))
+    (elpaa-publish-package-specs specs)))
 
 (defun elpaa-batch-make-one-package (&rest _)
   "Build the new tarballs (if needed) for one particular package."
-- 
2.38.0


[-- Attachment #3: Type: text/plain, Size: 451 bytes --]


> Other than that, it looks OK (in the nitpick area, I'd move the code to
> its own function and then call it from `elpaa-batch-make-all-packages`).

Done.

> I like the `.eld` extension, and I guess "elpa-packages.eld" is as good
> a name as any.
>
>> It currently makes the assumption that :core is the first element in the
>> property list for each package specification.
>
> That's right: that element should be either `:core` or `:url`.

Great.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> That was my mistake, I misremembered that core packages aren't mirrored
> in their own branches in elpa.git.  I've removed it here:

LGTM, thanks!
Feel free to push.

> +         ((and (plist-member (cdr spec) :url)
> +               (null (plist-get (cdr spec) :url)))
> +          `((,(car spec)
> +             :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
> +             :branch ,(concat elpaa--branch-prefix (car spec))
> +             ,@(cdddr spec))))

The `cdddr` presumes the `:url nil` is at the very beginning (which is
a valid assumption, indeed), whereas the `plist-member/get` dance tries
to deal with the more general case.  I think you can simply that to

    (pcase spec
      (`(:url nil . ,props) ...)
      ...)


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> That was my mistake, I misremembered that core packages aren't mirrored
>> in their own branches in elpa.git.  I've removed it here:
>
> LGTM, thanks!
> Feel free to push.
>
>> +         ((and (plist-member (cdr spec) :url)
>> +               (null (plist-get (cdr spec) :url)))
>> +          `((,(car spec)
>> +             :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
>> +             :branch ,(concat elpaa--branch-prefix (car spec))
>> +             ,@(cdddr spec))))
>
> The `cdddr` presumes the `:url nil` is at the very beginning (which is
> a valid assumption, indeed), whereas the `plist-member/get` dance tries
> to deal with the more general case.  I think you can simply that to
>
>     (pcase spec
>       (`(:url nil . ,props) ...)
>       ...)

Good point, I will apply this suggestion and then push the change.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +          ;; Add `package-vc' data
>> +          (let ((spec (cdr pkg-spec))
>> +                (ups (list :upstream 'Git nil nil nil)))
>> +            (if (null (plist-get spec :url))
>> +                (setf (nth 2 ups) (concat "https://git.savannah.gnu.org/git/"
>> +                                          elpaa--gitrepo)
>> +                      (nth 3 ups) (plist-get spec :lisp-dir)
>> +                      (nth 4 ups) (concat elpaa--branch-prefix pkg))
>> +              (setf (nth 2 ups) (plist-get spec :url)
>> +                    (nth 3 ups) (plist-get spec :lisp-dir)
>> +                    (nth 4 ups) (plist-get spec :branch)))
>> +            (push ups extras))
>
> I think rather than invent a new format and duplicate that info into
> `archive-contents`, pushing us to worry about its size, I'd prefer to
> just stash a copy of `elpa-packages` alongside `archive-contents`.

One thing I recently noticed are :url entries of the form

    "bzr::https://bzr.savannah.gnu.org/r/org-edna-el"

or

    "hg::https://hg.savannah.nongnu.org/hgweb/enwc/"

How are these handled?  I couldn't find anything specific to this in
elpa-admin.el.  Is this a Git feature?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> One thing I recently noticed are :url entries of the form
>
>     "bzr::https://bzr.savannah.gnu.org/r/org-edna-el"
>
> or
>
>     "hg::https://hg.savannah.nongnu.org/hgweb/enwc/"
>
> How are these handled?  I couldn't find anything specific to this in
> elpa-admin.el.  Is this a Git feature?

Yes, it's a Git feature.  Git will use the `git-remote-FOO` extension to
access those repositories.  In my experience it tends not to work super
reliably and those extensions aren't installed in `elpa.gnu.org` so we
don't auto-update from those repositories.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> One thing I recently noticed are :url entries of the form
>>
>>     "bzr::https://bzr.savannah.gnu.org/r/org-edna-el"
>>
>> or
>>
>>     "hg::https://hg.savannah.nongnu.org/hgweb/enwc/"
>>
>> How are these handled?  I couldn't find anything specific to this in
>> elpa-admin.el.  Is this a Git feature?
>
> Yes, it's a Git feature.  Git will use the `git-remote-FOO` extension to
> access those repositories.  In my experience it tends not to work super
> reliably and those extensions aren't installed in `elpa.gnu.org` so we
> don't auto-update from those repositories.

That also means that these repositories should either be filtered out
from elpa-packages.eld or the format should indicate that VC backend is used.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-10-24 15:35:42] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> One thing I recently noticed are :url entries of the form
>>>     "bzr::https://bzr.savannah.gnu.org/r/org-edna-el"
>>>
>>> or
>>>
>>>     "hg::https://hg.savannah.nongnu.org/hgweb/enwc/"
>>>
>>> How are these handled?  I couldn't find anything specific to this in
>>> elpa-admin.el.  Is this a Git feature?
>>
>> Yes, it's a Git feature.  Git will use the `git-remote-FOO` extension to
>> access those repositories.  In my experience it tends not to work super
>> reliably and those extensions aren't installed in `elpa.gnu.org` so we
>> don't auto-update from those repositories.
>
> That also means that these repositories should either be filtered out
> from elpa-packages.eld or the format should indicate that VC backend is used.

The current format will work for those users who have the corresponding
`git-remote-<BACKEND>` adapter installed, but yes I guess we need to
refine a bit what we mean by a "URL".  Currently it's actually not
really a URL but the "name" of a Git repository, tho it happens to look
very much like a URL.

Maybe we could use something like

    `:url (hg "https://hg.savannah.nongnu.org/hgweb/enwc/")`


-- Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-10-24 15:35:42] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> One thing I recently noticed are :url entries of the form
>>>>     "bzr::https://bzr.savannah.gnu.org/r/org-edna-el"
>>>>
>>>> or
>>>>
>>>>     "hg::https://hg.savannah.nongnu.org/hgweb/enwc/"
>>>>
>>>> How are these handled?  I couldn't find anything specific to this in
>>>> elpa-admin.el.  Is this a Git feature?
>>>
>>> Yes, it's a Git feature.  Git will use the `git-remote-FOO` extension to
>>> access those repositories.  In my experience it tends not to work super
>>> reliably and those extensions aren't installed in `elpa.gnu.org` so we
>>> don't auto-update from those repositories.
>>
>> That also means that these repositories should either be filtered out
>> from elpa-packages.eld or the format should indicate that VC backend is used.
>
> The current format will work for those users who have the corresponding
> `git-remote-<BACKEND>` adapter installed, but yes I guess we need to
> refine a bit what we mean by a "URL".  Currently it's actually not
> really a URL but the "name" of a Git repository, tho it happens to look
> very much like a URL.
>
> Maybe we could use something like
>
>     `:url (hg "https://hg.savannah.nongnu.org/hgweb/enwc/")`

Looks fine in principle, Would this only apply to non-Git repositories
that just happen to look like a URL?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-10-24 20:34:31] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Philip Kaludercic [2022-10-24 15:35:42] wrote:
>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>> One thing I recently noticed are :url entries of the form
>>>>>     "bzr::https://bzr.savannah.gnu.org/r/org-edna-el"
>>>>>
>>>>> or
>>>>>
>>>>>     "hg::https://hg.savannah.nongnu.org/hgweb/enwc/"
>>>>>
>>>>> How are these handled?  I couldn't find anything specific to this in
>>>>> elpa-admin.el.  Is this a Git feature?
>>>>
>>>> Yes, it's a Git feature.  Git will use the `git-remote-FOO` extension to
>>>> access those repositories.  In my experience it tends not to work super
>>>> reliably and those extensions aren't installed in `elpa.gnu.org` so we
>>>> don't auto-update from those repositories.
>>>
>>> That also means that these repositories should either be filtered out
>>> from elpa-packages.eld or the format should indicate that VC backend is used.
>>
>> The current format will work for those users who have the corresponding
>> `git-remote-<BACKEND>` adapter installed, but yes I guess we need to
>> refine a bit what we mean by a "URL".  Currently it's actually not
>> really a URL but the "name" of a Git repository, tho it happens to look
>> very much like a URL.
>>
>> Maybe we could use something like
>>
>>     `:url (hg "https://hg.savannah.nongnu.org/hgweb/enwc/")`
>
> Looks fine in principle, Would this only apply to non-Git repositories
> that just happen to look like a URL?

All non-Git repositories have such a URL, AFAIK, but yes, we'd use that
for all non-Git repositories (all 3 of them :-).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-10-24 20:34:31] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> Philip Kaludercic [2022-10-24 15:35:42] wrote:
>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>>> One thing I recently noticed are :url entries of the form
>>>>>>     "bzr::https://bzr.savannah.gnu.org/r/org-edna-el"
>>>>>>
>>>>>> or
>>>>>>
>>>>>>     "hg::https://hg.savannah.nongnu.org/hgweb/enwc/"
>>>>>>
>>>>>> How are these handled?  I couldn't find anything specific to this in
>>>>>> elpa-admin.el.  Is this a Git feature?
>>>>>
>>>>> Yes, it's a Git feature.  Git will use the `git-remote-FOO` extension to
>>>>> access those repositories.  In my experience it tends not to work super
>>>>> reliably and those extensions aren't installed in `elpa.gnu.org` so we
>>>>> don't auto-update from those repositories.
>>>>
>>>> That also means that these repositories should either be filtered out
>>>> from elpa-packages.eld or the format should indicate that VC backend is used.
>>>
>>> The current format will work for those users who have the corresponding
>>> `git-remote-<BACKEND>` adapter installed, but yes I guess we need to
>>> refine a bit what we mean by a "URL".  Currently it's actually not
>>> really a URL but the "name" of a Git repository, tho it happens to look
>>> very much like a URL.
>>>
>>> Maybe we could use something like
>>>
>>>     `:url (hg "https://hg.savannah.nongnu.org/hgweb/enwc/")`
>>
>> Looks fine in principle, Would this only apply to non-Git repositories
>> that just happen to look like a URL?
>
> All non-Git repositories have such a URL, AFAIK, but yes, we'd use that
> for all non-Git repositories (all 3 of them :-).

Another possibility would be to add another property like :vc-backend.
We could also use the opportunity to extend the "elpa-packages.eld"
format (not necessarily "elpa-packages" itself) to have some metadata
like a format-version or default backend.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Tim Cross]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> It seems to me that fetching a package from source is no more dangerous
>> than fetching a tarball, seeing as the tarball is automatically
>> generated from the repository.
>
> It doesn't matter much whether it's a tar ball or a git repo (although
> there is signing of the tar balls), but whether there's any oversight at
> all or not.  All commits to Non/GNU ELPA end up on a mailing list, which
> provides a smidgen of transparency, which is better than none.

Signing of tar balls may help a bit as it will typically mean you have
to both compromise the source repository and compromise the keys used
for signing. It raises the bar a bit. 

Posting commits to a mail list might provide some transparency, but next
to nothing for security. In fact, I would argue it is extremely
dangerous to assume it does anything for security. Just because
something is sent to a mailing list doesn't mean anyone actually looks
at it or performs any assessment.

I'm not sure anything Philip is proposing is making the situation
significantly worse because we are not doing anything proactive with
respect to security anyway. We are largely hoping 'someone else' has
reviewed it and flagged any security issues. While this might occur from
time to time, its ad hoc nature means it cannot be assumed to occur. 

We should educate users that all these methods, regardless of source,
have security implications. We should actively discourage any assumption
that we are similar to Apple whereby you can (supposedly) assume some
level of confidence regarding packages install from their App store.

The only level of confidence we can really provide wrt GNU ELPA and
nonGNU ELPA is that the packages in those repositories have acceptable
licenses. 

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> If we don't have such a list, then adding the basic functionality sounds
> useful anyway -- that is, allowing users to say `M-x
> package-install-from-repo' or something and then they type in the URL of
> that repo -- that's fine, and leaves the security implications to the
> user (where they already are today for people that install from external
> repos).

Indeed there are 2 different steps:
- installing from a particular "URL" (well, a URL plus some extra side
  info, tho that side info can be empty in many cases).  AFAIK that's
  what Philip's code currently offers.
- provide some way to let the user specify a package name and let
  something else map that to a "URL".  This is the more risky step and
  I don't think his code implements that yet.  Not sure how to address
  the security issue at that step, other than by dumping the problem
  onto the users: show them the URL and ask them if they're OK with it.

But as Philip points out, the (Non)GNU ELPA packages, while signed and
all, just blindly pull from those same URLs to build the tarballs, so
the difference is not as large as it seems.

> But if we list these repos in `M-x list-packages', that's a very
> different issue.

It also depends on where the list comes from.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If we don't have such a list, then adding the basic functionality sounds
>> useful anyway -- that is, allowing users to say `M-x
>> package-install-from-repo' or something and then they type in the URL of
>> that repo -- that's fine, and leaves the security implications to the
>> user (where they already are today for people that install from external
>> repos).
>
> Indeed there are 2 different steps:
> - installing from a particular "URL" (well, a URL plus some extra side
>   info, tho that side info can be empty in many cases).  AFAIK that's
>   what Philip's code currently offers.

Correct.

> - provide some way to let the user specify a package name and let
>   something else map that to a "URL".  This is the more risky step and
>   I don't think his code implements that yet.  Not sure how to address
>   the security issue at that step, other than by dumping the problem
>   onto the users: show them the URL and ask them if they're OK with it.

This is implemented, the "something else" is just the package metadata.
To me there seems to be no difference between trusting an archive that
a tarball is safe or that a repository it points to is safe.

> But as Philip points out, the (Non)GNU ELPA packages, while signed and
> all, just blindly pull from those same URLs to build the tarballs, so
> the difference is not as large as it seems.

If it would make any difference, it would also be possible to inhibit
the generation of autoloads.

>> But if we list these repos in `M-x list-packages', that's a very
>> different issue.
>
> It also depends on where the list comes from.
>
>
>         Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Tim Cross]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> - The ability to install a package directly from source using
>>   `package-vc-fetch' (aliased to `package-checkout').  This
>>   functionality is ideally VC generic.
>>
>> - The ability to update a package using `package-upgrade'[0]
>>
>> - Package metadata can either be inferred from the package URL (see
>>   `package-vc-heusitic-alist') or via explicit hints from an ELPA
>>   server.  I plan to add the necessary features to GNU and NonGNU ELPA
>>   in time so that the heuristics can be avoided.
>>
>> - The ability to (i) contact, (ii) send bug reports and (iii) patches
>>   (using the new `vc-patch-prepare') to package maintainers.
>
> Sounds like great functionality, but I wonder whether the security
> implications have been discussed?  Today, we use GNU ELPA as a filter of
> sorts and people rely on code there not being compromised.
>
> I assume "hints from an ELPA server" is basically a list of links to git
> repositories?  If that's the case, then we may well end up with pointing
> users towards repos that have been compromised.
>
> If we don't have such a list, then adding the basic functionality sounds
> useful anyway -- that is, allowing users to say `M-x
> package-install-from-repo' or something and then they type in the URL of
> that repo -- that's fine, and leaves the security implications to the
> user (where they already are today for people that install from external
> repos).
>
> But if we list these repos in `M-x list-packages', that's a very
> different issue.


I think it is very dangerous to suggest there is ANY security here, even
with GNU ELPA packages.

- There is no formal security review of packages
- There is no review before packages are updated. If a repository is
  compromised and that compromise has not been detected, an update can
  still occur and introduced compromised code into GNU ELPA. 

Far better to just educate users that ANY package they install could
contain malicious code.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Tim Cross <theophilusx@gmail.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> - The ability to install a package directly from source using
>>>   `package-vc-fetch' (aliased to `package-checkout').  This
>>>   functionality is ideally VC generic.
>>>
>>> - The ability to update a package using `package-upgrade'[0]
>>>
>>> - Package metadata can either be inferred from the package URL (see
>>>   `package-vc-heusitic-alist') or via explicit hints from an ELPA
>>>   server.  I plan to add the necessary features to GNU and NonGNU ELPA
>>>   in time so that the heuristics can be avoided.
>>>
>>> - The ability to (i) contact, (ii) send bug reports and (iii) patches
>>>   (using the new `vc-patch-prepare') to package maintainers.
>>
>> Sounds like great functionality, but I wonder whether the security
>> implications have been discussed?  Today, we use GNU ELPA as a filter of
>> sorts and people rely on code there not being compromised.
>>
>> I assume "hints from an ELPA server" is basically a list of links to git
>> repositories?  If that's the case, then we may well end up with pointing
>> users towards repos that have been compromised.
>>
>> If we don't have such a list, then adding the basic functionality sounds
>> useful anyway -- that is, allowing users to say `M-x
>> package-install-from-repo' or something and then they type in the URL of
>> that repo -- that's fine, and leaves the security implications to the
>> user (where they already are today for people that install from external
>> repos).
>>
>> But if we list these repos in `M-x list-packages', that's a very
>> different issue.
>
>
> I think it is very dangerous to suggest there is ANY security here, even
> with GNU ELPA packages.
>
> - There is no formal security review of packages
> - There is no review before packages are updated. If a repository is
>   compromised and that compromise has not been detected, an update can
>   still occur and introduced compromised code into GNU ELPA. 
>
> Far better to just educate users that ANY package they install could
> contain malicious code.

Is this feasible?  Should this be a (long-)term goal, or will ELPA
always be a "use at your own risk" kind of thing?

Also, could you clarify what you mean by "formal security review"?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Tim Cross]

Philip Kaludercic <philipk@posteo.net> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>>
>> Far better to just educate users that ANY package they install could
>> contain malicious code.
>
> Is this feasible?  Should this be a (long-)term goal, or will ELPA
> always be a "use at your own risk" kind of thing?
>
> Also, could you clarify what you mean by "formal security review"?

The amount of resources needed to ensure code has no security issues is
immense - well of course you cannot always guarantee no security issues,
but just having sufficient analysis to provide a reasonable level of
confidence is significant. To make it worse, any such analysis needs to
be done after every code update.

While a casual read through of the code may identify many glaring
security problems, it will miss less obvious issues. TO make matters
harder, actively malicious code will be crafted to hide its intent as
much as possible, so is unlikely to be identified just scanning through
the code.

A formal review would likely involve the following

- Formal analysis of the code. More than just a casual read through of
  the code. This would involve reading the code with a focus on security
  and verifying you understood what it did by tracing data flow all the
  way through. This would include any packages/modules the main package
  also uses (if those package versions have also undergone a formal
  security review, you can leverage off that information and would not
  be required to perform such a review on them again. It also involves
  reading assuming there are problems and your trying to find them.

- Use of various networking tools (like wireshark) to identify and
  understand any network traffic. Verify none of that traffic contains
  data which could be considered sensitive.

- Understanding of all data storage used by the system and ensure where
  such storage involves any sensitive data that it is documented and
  storage protection (e.g. hashing or encryption) is implemented or
  easily implemented/turned on.

- Actively attempt to use (abuse) the code for malicious purposes and
  look for possible security flaws. For example, supplying input with
  malicious content such as embedded code, suspect URLs, embedded
  commands etc. 

- Review of the installation and usage documentation to ensure there are
  not recommendations or requirements which would further reduce the
  user's security position.

in addition, there is some housekeeping tasks which would also be
required, including

- Tracking when reviews are done and by who

- Maintaining a registry of identified security issues. This would
  likely involve a high level description, severity ranking (i.e. CVSS),
  date discovered, date maintainers notified and possibly date for
  public release. Might want ot also indicate if issue is known to be
  actively exploited.

- Formal process for handling and managing security issues

What makes this extremely challenging is that the reviews should be
done by people without a vested interest in the package. Package authors
and maintainers are not always the best people to perform such reviews
because they are too close to the code (very similar to using external
testers).

The real challenge here is that this stuff is a hard skill which few
developers have. Finding sufficient numbers of people with these skills
who are able to work on just GNU ELPA packages would be next to
impossible. We simply don't have the numbers necessary.

Consider the commercial 'app stores' such as Apple's, MS's Azure store
or even Google Android store. These companies put significant resources
into trying to ensure the apps in these stores are 'safe' (to varying
degrees), yet we regularly see reports of multiple apps being removed
from these stores for security reasons or about apps which have been in
the store for some time and later found to be malicious. Or for
something perhaps a little closer to GNU ELPA, consider the issues
experienced by NPM and teh supply chain issues which have occurred there
- especially with respect to those cases of trusted and widely used
packages which have been compromised and that compromise was not
detected for a bit of time. The NPM ecosystem is putting a lot of
resources into enforcing better security practices by package
maintainers as this is where many of the failures have occurred.  It
hasn't been the actual package repository (i.e. think GNU ELPA), but the
source repositories (i.e. github, gitlab, sourceHut, random Git host)
which have been compromised. The level of sound security practices
employed by package maintainers varies widely. Some of the things NPM
are doing to try and improve security in this area may well be
informative for GNU ELPA and non-GNU ELPA. For example, sending an email
to package maintainer whenever their package is updated in ELPA,
requiring all source repositories to be protected with 2FA etc. 

The main thing protecting GNU Emacs and the ELPA infrastructure is low
return on investment. Malicious intent is like many things, there has to
be sufficient reward for the effort involved. The 'out run the lion'
principal also factors in here (you don't need to out run the lion, you
just need to not be the slowest person trying to outrun the lion). Right
now, the rewards seem low and there are easier targets out
there. Similar to the fallacy that there are no viruses on Apple or GNU
Linux, it isn't simply that we are more secure, we just don't provide
sufficient return for the effort investment.  

The other reason we should encourage the user to be proactive here and
not have them develop the expectation that all packages in GNU ELPA are
'safe' is because a lot really depends on each individual user and how
they use Emacs and in what environment. If your just using Emacs for
basic code editing at home when working on hobby projects, your risk
profile is likely low. However, if your a leading researcher in a
extremely competitive industry, you might be a much richer target worth
pursuit. If you also use Emacs to read your email, browse the web, as
your window manager, to interact with databases and for accessing remote
sensitive servers/data stores, then perhaps someone might target Emacs
as a way to gain access or exploit your data/system. This level of
targeting of researchers by well resourced industrial espionage groups
(often government backed) has been of sufficient concern that the 'five
eyes' have issued a number of security alerts to various government and
private research organisation over the last few years.  

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think it is very dangerous to suggest there is ANY security here, even
  > with GNU ELPA packages.

  > - There is no formal security review of packages
  > - There is no review before packages are updated. If a repository is
  >   compromised and that compromise has not been detected, an update can
  >   still occur and introduced compromised code into GNU ELPA. 

I think we had better do something about this.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> - The ability to update a package using `package-upgrade'[0]

[...]

> [0] This feature is currently broken because it tries to use `vc-pull'
>     which refuses to work if any changes have been made.  This is an
>     issue as package-vc-fetch tries to modify the list of ignored files
>     in a repository (think of .gitignore).  It will either be necessary
>     to change how vc-pull works or try to do something with
>     vc-log-incoming + vc-merge.

Does anyone know if it is possible to do this?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Sat, 15 Oct 2022 15:52:28 +0000
> 
> Philip Kaludercic <philipk@posteo.net> writes:
> 
> > - The ability to update a package using `package-upgrade'[0]
> 
> [...]
> 
> > [0] This feature is currently broken because it tries to use `vc-pull'
> >     which refuses to work if any changes have been made.  This is an
> >     issue as package-vc-fetch tries to modify the list of ignored files
> >     in a repository (think of .gitignore).  It will either be necessary
> >     to change how vc-pull works or try to do something with
> >     vc-log-incoming + vc-merge.
> 
> Does anyone know if it is possible to do this?

Where is the code which fails vc-pull if there are changes in the
working tree?  is it specific to some backend, perhaps?  The doc
string says:

  On a non-distributed version control system, update the current
  fileset to the tip revisions.  For each unchanged and unlocked
  file, this simply replaces the work file with the latest revision
  on its branch.  If the file contains changes, any changes in the
  tip revision are merged into the working file.

Note that last sentence.  So this ought to work, and if it doesn't,
it's a bug, I guess?

From the shell prompt, "git pull" will fail if the changes are in the
same file as one of those whose changes are brought in by the pull,
but either using --rebase or maybe the stash/pull/unstash dance should
take care of that, I guess?

Or am I missing something?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Sean Whitton]

Hello,

On Sat 15 Oct 2022 at 07:22PM +03, Eli Zaretskii wrote:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 15 Oct 2022 15:52:28 +0000
>>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>> > - The ability to update a package using `package-upgrade'[0]
>>
>> [...]
>>
>> > [0] This feature is currently broken because it tries to use `vc-pull'
>> >     which refuses to work if any changes have been made.  This is an
>> >     issue as package-vc-fetch tries to modify the list of ignored files
>> >     in a repository (think of .gitignore).  It will either be necessary
>> >     to change how vc-pull works or try to do something with
>> >     vc-log-incoming + vc-merge.
>>
>> Does anyone know if it is possible to do this?
>
> Where is the code which fails vc-pull if there are changes in the
> working tree?  is it specific to some backend, perhaps?  The doc
> string says:
>
>   On a non-distributed version control system, update the current
>   fileset to the tip revisions.  For each unchanged and unlocked
>   file, this simply replaces the work file with the latest revision
>   on its branch.  If the file contains changes, any changes in the
>   tip revision are merged into the working file.
>
> Note that last sentence.  So this ought to work, and if it doesn't,
> it's a bug, I guess?
>
> From the shell prompt, "git pull" will fail if the changes are in the
> same file as one of those whose changes are brought in by the pull,
> but either using --rebase or maybe the stash/pull/unstash dance should
> take care of that, I guess?

I think only if they're nearby in the file.  Changes at the other end of
the file, 'git pull' will just merge in.

-- 
Sean Whitton

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> I think only if they're nearby in the file.  Changes at the other end of
> the file, 'git pull' will just merge in.

In my experience, the distance doesn't matter.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:
> From the shell prompt, "git pull" will fail if the changes are in the
> same file as one of those whose changes are brought in by the pull,
> but either using --rebase or maybe the stash/pull/unstash dance should
> take care of that, I guess?

There is also git pull --autostash

And if you use Mercurial, merge without conflict always works with
hg pull -u.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
> Cc: Philip Kaludercic <philipk@posteo.net>, monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> Date: Sun, 16 Oct 2022 09:10:34 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > From the shell prompt, "git pull" will fail if the changes are in the
> > same file as one of those whose changes are brought in by the pull,
> > but either using --rebase or maybe the stash/pull/unstash dance should
> > take care of that, I guess?
> 
> There is also git pull --autostash

But --autostash is only valid when using --rebase, isn't it?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[tomas]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

On Sun, Oct 16, 2022 at 11:15:58AM +0300, Eli Zaretskii wrote:
> > From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>

[...]

> > There is also git pull --autostash
> 
> But --autostash is only valid when using --rebase, isn't it?

AFAIK not (anymore). It seems to be a historical artifact.

Still, of course, autostash leaves the user to cope with potential
conflicts. If there are conflicts, there's no way around them, of
course, but having the warning before ("you have changes in your
local repo, watch out!") seems somehow friendlier to me. Autostash
"magic" upon an unsuspecting user sounds like a pretty sharp tool.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> Date: Sun, 16 Oct 2022 11:29:30 +0200
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>, philipk@posteo.net,
> 	monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From:  <tomas@tuxteam.de>
> 
> > > There is also git pull --autostash
> > 
> > But --autostash is only valid when using --rebase, isn't it?
> 
> AFAIK not (anymore). It seems to be a historical artifact.

Maybe your Git is too new.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[tomas]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

On Sun, Oct 16, 2022 at 01:31:51PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 16 Oct 2022 11:29:30 +0200
> > Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>, philipk@posteo.net,
> > 	monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > From:  <tomas@tuxteam.de>
> > 
> > > > There is also git pull --autostash
> > > 
> > > But --autostash is only valid when using --rebase, isn't it?
> > 
> > AFAIK not (anymore). It seems to be a historical artifact.
> 
> Maybe your Git is too new.

Or your's too old :)

It seems to have changed around 2.26 (spring 2020 or so). Mine is
vanilla Debian stable (aka Bullseye), git 2.30 (end 2020 or so).

I always thought Debian was for us old folks ;-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 15 Oct 2022 15:52:28 +0000
>> 
>> Philip Kaludercic <philipk@posteo.net> writes:
>> 
>> > - The ability to update a package using `package-upgrade'[0]
>> 
>> [...]
>> 
>> > [0] This feature is currently broken because it tries to use `vc-pull'
>> >     which refuses to work if any changes have been made.  This is an
>> >     issue as package-vc-fetch tries to modify the list of ignored files
>> >     in a repository (think of .gitignore).  It will either be necessary
>> >     to change how vc-pull works or try to do something with
>> >     vc-log-incoming + vc-merge.
>> 
>> Does anyone know if it is possible to do this?
>
> Where is the code which fails vc-pull if there are changes in the
> working tree?  is it specific to some backend, perhaps?  The doc
> string says:
>
>   On a non-distributed version control system, update the current
>   fileset to the tip revisions.  For each unchanged and unlocked
>   file, this simply replaces the work file with the latest revision
>   on its branch.  If the file contains changes, any changes in the
>   tip revision are merged into the working file.
>
> Note that last sentence.  So this ought to work, and if it doesn't,
> it's a bug, I guess?

I have re-tested it and I was mistaken, the issue was with Git.  Sadly I
cannot reproduce it right now, but I will try to re-trigger it.

> From the shell prompt, "git pull" will fail if the changes are in the
> same file as one of those whose changes are brought in by the pull,
> but either using --rebase or maybe the stash/pull/unstash dance should
> take care of that, I guess?

Right, I believe it was something along these lines but this is
something that is difficult to automate.

> Or am I missing something?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 16 Oct 2022 22:22:42 +0000
> 
> > From the shell prompt, "git pull" will fail if the changes are in the
> > same file as one of those whose changes are brought in by the pull,
> > but either using --rebase or maybe the stash/pull/unstash dance should
> > take care of that, I guess?
> 
> Right, I believe it was something along these lines but this is
> something that is difficult to automate.

Can you elaborate on the "difficult" part?  You could always stash any
local changes before pulling, then un-stashing them after.  This
should work reliably, I think?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sun, 16 Oct 2022 22:22:42 +0000
>> 
>> > From the shell prompt, "git pull" will fail if the changes are in the
>> > same file as one of those whose changes are brought in by the pull,
>> > but either using --rebase or maybe the stash/pull/unstash dance should
>> > take care of that, I guess?
>> 
>> Right, I believe it was something along these lines but this is
>> something that is difficult to automate.
>
> Can you elaborate on the "difficult" part?  You could always stash any
> local changes before pulling, then un-stashing them after.  This
> should work reliably, I think?

Difficult in the sense that to my knowledge there is no --dwim flag to
tell Git to try and avert a conflict at any cost.  Instead you would
have to know what the right operation is depending on the current state
of the repository and the kinds of conflicts that pulling creates.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 17 Oct 2022 06:27:04 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Can you elaborate on the "difficult" part?  You could always stash any
> > local changes before pulling, then un-stashing them after.  This
> > should work reliably, I think?
> 
> Difficult in the sense that to my knowledge there is no --dwim flag to
> tell Git to try and avert a conflict at any cost.  Instead you would
> have to know what the right operation is depending on the current state
> of the repository and the kinds of conflicts that pulling creates.

If the changes are in the same parts of a file, merge conflicts are
inevitable, and are a necessary part of this kind of operation.  So
why is that perceived as a problem?

I think we should try to avoid conflicts and failures where a
simple-minded merge will do.  But if a conflict is inevitable, we
should punt and let the user fix that.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Mon, 17 Oct 2022 06:27:04 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Can you elaborate on the "difficult" part?  You could always stash any
>> > local changes before pulling, then un-stashing them after.  This
>> > should work reliably, I think?
>> 
>> Difficult in the sense that to my knowledge there is no --dwim flag to
>> tell Git to try and avert a conflict at any cost.  Instead you would
>> have to know what the right operation is depending on the current state
>> of the repository and the kinds of conflicts that pulling creates.
>
> If the changes are in the same parts of a file, merge conflicts are
> inevitable, and are a necessary part of this kind of operation.  So
> why is that perceived as a problem?
>
> I think we should try to avoid conflicts and failures where a
> simple-minded merge will do.  But if a conflict is inevitable, we
> should punt and let the user fix that.

The issue I believe I had was that package-vc uses vc-ignore to add a
few files to the list of ignored files, and this could trigger an
"unnecessary" conflict when updating the package.  But I believe that
this only occurs if the .gitignore file was modified in upstream source.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> The issue I believe I had was that package-vc uses vc-ignore to add a
> few files to the list of ignored files,

Maybe we should refrain from doing it and presume it's something which
the upstream should do for us (and then try and make the behavior
livable if the upstream doesn't comply).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The issue I believe I had was that package-vc uses vc-ignore to add a
>> few files to the list of ignored files,
>
> Maybe we should refrain from doing it and presume it's something which
> the upstream should do for us (and then try and make the behavior
> livable if the upstream doesn't comply).

Done.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If the main purpose of this feature is for people to test, debug and
develop the development version, I think it is wiser not to speak
of "installing" from VCS.

Presenting the feature as a way to "install" would encourage people
who are not really thinking of testing, debugging or developping the
package, and motivated only by a vague wish for "the latest thing."

I suggest that we are better off if we avoid encouraging them.

One way to avoid that is to have this feature simply check out the
package's development sources.  If you know how to work with Emacs
Lisp, you will not find it hard to byte compile that and use it.  We
can even document the natural ways to proceed and do those things, and
people who are really prepared to test it will do them.  But newbies
probably won't go that far.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> If the main purpose of this feature is for people to test, debug and
> develop the development version,

It is not.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If the main purpose of this feature is for people to test, debug and
  > > develop the development version,

  > It is not.

That's the idea I understood from Philip's writing.  If you think its
main purpose is something else, would you please say what that is?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If the main purpose of this feature is for people to test, debug and
> develop the development version, I think it is wiser not to speak
> of "installing" from VCS.

Not necessarily, another advantage might include the ability to maintain
personal changes that don't get overridden by updates.  There are
certainly other reasons to do so that I cannot think of right now.

In general I would like to see something like "package-vc" being
regarded as a means to make software freedom more practical/perceptible.
Emacs per se already does a great job in this regard, by enabling the
user to study the source of any command or functionality with minimal
overhead (usually just a M-x find-function away), but I believe that
features like this can improve this spirit further.

> Presenting the feature as a way to "install" would encourage people
> who are not really thinking of testing, debugging or developping the
> package, and motivated only by a vague wish for "the latest thing."

I agree that people might think of this idea, but then again what is the
issue if they do?  The installation is parallel (but preferred when
loading) to any other installation, be it local or system-wide, and if
they decide to revert they just have to uninstall the "source" package.

> I suggest that we are better off if we avoid encouraging them.
>
> One way to avoid that is to have this feature simply check out the
> package's development sources.  If you know how to work with Emacs
> Lisp, you will not find it hard to byte compile that and use it.  We
> can even document the natural ways to proceed and do those things, and
> people who are really prepared to test it will do them.  But newbies
> probably won't go that far.

I proposed a library along those lines last year that would automate
this (it was called "site-lisp.el" in case you want to look the
discussion up).  It automatically byte-compiles, prepares autoloads and
adds the directory to the load path for all files/directories in
~/.config/emacs/site-lisp.  I still use it for a number of my personal
projects, but it doesn't do the same thing as package-vc, so I wouldn't
consider them to be mutually exclusive.  What I do believe is that by
default, what you have to do to prepare and load some foreign code is
unnecessarily cumbersome/repetitive.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If the main purpose of this feature is for people to test, debug and
  > > develop the development version, I think it is wiser not to speak
  > > of "installing" from VCS.

  > Not necessarily, another advantage might include the ability to maintain
  > personal changes that don't get overridden by updates.

That would be a useful feature.  But I think it would be unfortunate
(and not what users want) to tell them, "To make and save your own
patches, switch to the development version of the package."

It would be better to offer a way that you can install your own
patches _in the released version_, which is probably the version you
were using and wrote the patches for, and then preserve these patches
when you install new released versions.

  > In general I would like to see something like "package-vc" being
  > regarded as a means to make software freedom more practical/perceptible.

That is a good general goal, but I don't think it applies to this
question.  Whether you want to try a development version is one
question; whether you want to make patches (and desire to preserve
them across updates) is another question.  Let's make the latter feature
available for the reeased package also.

I understand that may be some extra work.  However, merging your local
patches with upgrades is not always going to be trivial and
trouble-free.  Often there will be no conflict, but sometimes there
will be one.  One way or another, you will need to be prepared for that
if you start preserving your patches across upgrades

  > > Presenting the feature as a way to "install" would encourage people
  > > who are not really thinking of testing, debugging or developping the
  > > package, and motivated only by a vague wish for "the latest thing."

  > I agree that people might think of this idea, but then again what is the
  > issue if they do?

They are likely to get different behavior, and more bugs.
There is a good reason for making releases: in general, that makes
things more reliable for users.  That applies to packages as well
as to Emacs overall.  Running the development sources of anything
carries greater risks.  Some users want to take those risks, but
it would be a mistake to urge all users to do that.

  > I proposed a library along those lines last year that would automate
  > this (it was called "site-lisp.el" in case you want to look the
  > discussion up).  It automatically byte-compiles, prepares autoloads and
  > adds the directory to the load path for all files/directories in
  > ~/.config/emacs/site-lisp.

This leads me to ask two questions:

Would my proposed package-vc-dev command, plus this, add up to the 
currently proposed package-vc-install command?

Would this be just what the user wants
for preserving patches in packages installed from the release?

By the way, GNU ELPA and NonGNU ELPA both have a repo which
holds the current released version of each package.  Imagine
a package-vc-release command that does a checkout of the
released package from ELPA, in the same way.  That would help
people add patches to the released version, and would merge
changes when a new version is released.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > If the main purpose of this feature is for people to test, debug and
>   > > develop the development version, I think it is wiser not to speak
>   > > of "installing" from VCS.
>
>   > Not necessarily, another advantage might include the ability to maintain
>   > personal changes that don't get overridden by updates.
>
> That would be a useful feature.  But I think it would be unfortunate
> (and not what users want) to tell them, "To make and save your own
> patches, switch to the development version of the package."
>
> It would be better to offer a way that you can install your own
> patches _in the released version_, which is probably the version you
> were using and wrote the patches for, and then preserve these patches
> when you install new released versions.

That is a long term goal, but if the foundation is to be merged before
the next release, I have other issues that I'd like to address now and
then return to this for Emacs 30.

>   > In general I would like to see something like "package-vc" being
>   > regarded as a means to make software freedom more practical/perceptible.
>
> That is a good general goal, but I don't think it applies to this
> question.  Whether you want to try a development version is one
> question; whether you want to make patches (and desire to preserve
> them across updates) is another question.  Let's make the latter feature
> available for the reeased package also.
>
> I understand that may be some extra work.  However, merging your local
> patches with upgrades is not always going to be trivial and
> trouble-free.  Often there will be no conflict, but sometimes there
> will be one.  One way or another, you will need to be prepared for
> that if you start preserving your patches across upgrades
>
>   > > Presenting the feature as a way to "install" would encourage people
>   > > who are not really thinking of testing, debugging or developping the
>   > > package, and motivated only by a vague wish for "the latest thing."
>
>   > I agree that people might think of this idea, but then again what is the
>   > issue if they do?
>
> They are likely to get different behavior, and more bugs.
> There is a good reason for making releases: in general, that makes
> things more reliable for users.  That applies to packages as well
> as to Emacs overall.  Running the development sources of anything
> carries greater risks.  Some users want to take those risks, but
> it would be a mistake to urge all users to do that.

I agree in principle, but with two caveats.  Consider that most people
use package archives like MELPA, that distribute the most recent commit
by default, so this is not unfamiliar to many users.  The second point
is that due to the usage of version control, the user is able to revert
to a previous commit or even use their own branch where they pick and
choose the commits they are not interested in.

>   > I proposed a library along those lines last year that would automate
>   > this (it was called "site-lisp.el" in case you want to look the
>   > discussion up).  It automatically byte-compiles, prepares autoloads and
>   > adds the directory to the load path for all files/directories in
>   > ~/.config/emacs/site-lisp.
>
> This leads me to ask two questions:
>
> Would my proposed package-vc-dev command, plus this, add up to the 
> currently proposed package-vc-install command?

I did not quite understand the point of `package-vc-dev', so I cannot
comment on it.  What I can say that `package-vc-install' "minus"
"site-lisp" would be a function that would just use package metadata to
clone repositories into a specific directory.

> Would this be just what the user wants
> for preserving patches in packages installed from the release?

Again, I am not sure.

> By the way, GNU ELPA and NonGNU ELPA both have a repo which
> holds the current released version of each package.  Imagine
> a package-vc-release command that does a checkout of the
> released package from ELPA, in the same way.  That would help
> people add patches to the released version, and would merge
> changes when a new version is released.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It would be better to offer a way that you can install your own
  > > patches _in the released version_, which is probably the version you
  > > were using and wrote the patches for, and then preserve these patches
  > > when you install new released versions.

  > That is a long term goal, but if the foundation is to be merged before
  > the next release, I have other issues that I'd like to address now and
  > then return to this for Emacs 30.

I'm not quite sure what that means -- what is the "foundation", and why
is there a hurry to merge it in?

But there is a general principle that it is better to get the external
interfaces right before installing something new.  Changing those
interfaces now won't require people to change habits; changing them
later could do so.

The code for checking out the current-version repo of an ELPA package
should be very little different from the code for checking out the
upstream repo.  If the latter is already working, getting the former
to work equally well shouldn't take long.  Would you please do that?

Once we support choosing between the two repos for a package, the name
`package-vc-install' will be ambiguous -- it could fit either of those
two cases.  It would not be a clear name to use.  So let's not define
anything now with that name.

I suggest the names `package-repo' and `package-dev-repo' for checking
out an ELPA package in these two ways, current repo and upstream/dev
repo.  (I previously envisaged other names, but don't worry about
those.)  Each of these names is clearer than `package-vc-install', and
they will make the distinction (choice of repo) clear too.

I think `package-dev-repo' would be the command now called
`package-vc-install'.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > It would be better to offer a way that you can install your own
>   > > patches _in the released version_, which is probably the version you
>   > > were using and wrote the patches for, and then preserve these patches
>   > > when you install new released versions.
>
>   > That is a long term goal, but if the foundation is to be merged before
>   > the next release, I have other issues that I'd like to address now and
>   > then return to this for Emacs 30.
>
> I'm not quite sure what that means -- what is the "foundation", and why
> is there a hurry to merge it in?

"Hurry" is the wrong way to look at it, I was just looking forward to
having a basic feature like this added to Emacs 29.  Of course this
shouldn't be rushed, but I believe the branch is currently in a
functional state that would be useful.

> But there is a general principle that it is better to get the external
> interfaces right before installing something new.  Changing those
> interfaces now won't require people to change habits; changing them
> later could do so.
>
> The code for checking out the current-version repo of an ELPA package
> should be very little different from the code for checking out the
> upstream repo.  If the latter is already working, getting the former
> to work equally well shouldn't take long.  Would you please do that?

It can be done, but I believe it would be best to add those kinds of
annotations to the server side (elpa-admin.el).  The issue is that
figuring out what version was used to release a package is non-trivial
using just vc.

> Once we support choosing between the two repos for a package, the name
> `package-vc-install' will be ambiguous -- it could fit either of those
> two cases.  It would not be a clear name to use.  So let's not define
> anything now with that name.

The reason I chose this name is because of the symmetry to
`package-install', and I still think this is nice.

> I suggest the names `package-repo' and `package-dev-repo' for checking
> out an ELPA package in these two ways, current repo and upstream/dev
> repo.  (I previously envisaged other names, but don't worry about
> those.)  Each of these names is clearer than `package-vc-install', and
> they will make the distinction (choice of repo) clear too.

In practice there is really no great difference between the "current
repo" and a "upstream/dev repo".

> I think `package-dev-repo' would be the command now called
> `package-vc-install'.

I am fine with renaming the file "package-vc" to "package-dev", but the
"-repo" suffix is not clear to me.  "-install" is clear in the sense
that it will fetch and activate the package, just like
"package-install", but "-repo" could do anything from opening the URL in
an external browser or printing some message in the echo area.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> It can be done, but I believe it would be best to add those kinds of
> annotations to the server side (elpa-admin.el).  The issue is that
> figuring out what version was used to release a package is non-trivial
> using just vc.

FWIW, the more I think about it, the more I think `package-vc` should
always use Git locally (regardless of the VCS used upstream).  This will
also make it easier to share the code with `elpa-admin.el` (e.g. to
figure out which commit is the latest release).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It can be done, but I believe it would be best to add those kinds of
>> annotations to the server side (elpa-admin.el).  The issue is that
>> figuring out what version was used to release a package is non-trivial
>> using just vc.
>
> FWIW, the more I think about it, the more I think `package-vc` should
> always use Git locally (regardless of the VCS used upstream).  This will
> also make it easier to share the code with `elpa-admin.el` (e.g. to
> figure out which commit is the latest release).

I am not sure about that, this would break the contribution workflow for
anyone interested in sending back patches to a non-Git project.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-10-26 15:28:04] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> It can be done, but I believe it would be best to add those kinds of
>>> annotations to the server side (elpa-admin.el).  The issue is that
>>> figuring out what version was used to release a package is non-trivial
>>> using just vc.
>>
>> FWIW, the more I think about it, the more I think `package-vc` should
>> always use Git locally (regardless of the VCS used upstream).  This will
>> also make it easier to share the code with `elpa-admin.el` (e.g. to
>> figure out which commit is the latest release).
>
> I am not sure about that, this would break the contribution workflow for
> anyone interested in sending back patches to a non-Git project.

AFAIK this is sufficiently hypothetical that I'm really not convinced
it's worth the extra work (and the risk of incompatibilities between
the `elpa-admin.el` behavior and that if `package-vc`, e.g. where they
might disagree on which commit is "the" release).

We can try to add some support for other local-VCS afterwards, but it
shouldn't be a priority.

Also, AFAIK the `git-remote-hg` adapter and the `git-remote-bzr` adapter
are bidirectional, so they are still compatible with some workflows.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-10-26 15:28:04] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> It can be done, but I believe it would be best to add those kinds of
>>>> annotations to the server side (elpa-admin.el).  The issue is that
>>>> figuring out what version was used to release a package is non-trivial
>>>> using just vc.
>>>
>>> FWIW, the more I think about it, the more I think `package-vc` should
>>> always use Git locally (regardless of the VCS used upstream).  This will
>>> also make it easier to share the code with `elpa-admin.el` (e.g. to
>>> figure out which commit is the latest release).
>>
>> I am not sure about that, this would break the contribution workflow for
>> anyone interested in sending back patches to a non-Git project.
>
> AFAIK this is sufficiently hypothetical that I'm really not convinced
> it's worth the extra work (and the risk of incompatibilities between
> the `elpa-admin.el` behavior and that if `package-vc`, e.g. where they
> might disagree on which commit is "the" release).

I don't know, to me this sounds more like an argument against merging
elpa-admin.el and package-vc.el...

> We can try to add some support for other local-VCS afterwards, but it
> shouldn't be a priority.

So you would say that the support for other VCS should be removed?

> Also, AFAIK the `git-remote-hg` adapter and the `git-remote-bzr` adapter
> are bidirectional, so they are still compatible with some workflows.

Fair enough.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> AFAIK this is sufficiently hypothetical that I'm really not convinced
>> it's worth the extra work (and the risk of incompatibilities between
>> the `elpa-admin.el` behavior and that if `package-vc`, e.g. where they
>> might disagree on which commit is "the" release).
>
> I don't know, to me this sounds more like an argument against merging
> elpa-admin.el and package-vc.el...

It's not about sharing the code here (for which I have lobbied in other
messages), but about sharing the behavior.
`elpa-admin.el` de facto defines a "protocol" that upstream maintainers
have to follow.  If `package-vc` doesn't follow the same protocol,
end-users will suffer unexpected discrepancies.

>> We can try to add some support for other local-VCS afterwards, but it
>> shouldn't be a priority.
> So you would say that the support for other VCS should be removed?

Not necessarily removed, but considered second class for now.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> AFAIK this is sufficiently hypothetical that I'm really not convinced
>>> it's worth the extra work (and the risk of incompatibilities between
>>> the `elpa-admin.el` behavior and that if `package-vc`, e.g. where they
>>> might disagree on which commit is "the" release).
>>
>> I don't know, to me this sounds more like an argument against merging
>> elpa-admin.el and package-vc.el...
>
> It's not about sharing the code here (for which I have lobbied in other
> messages), but about sharing the behavior.
> `elpa-admin.el` de facto defines a "protocol" that upstream maintainers
> have to follow.  If `package-vc` doesn't follow the same protocol,
> end-users will suffer unexpected discrepancies.

That is sort of the question I was implying previously (when mentioning
adding meta data), as to whether or not there is a "protocol" or if the
two format just coincidentally share a part of a common structure.

>>> We can try to add some support for other local-VCS afterwards, but it
>>> shouldn't be a priority.
>> So you would say that the support for other VCS should be removed?
>
> Not necessarily removed, but considered second class for now.

I will have to think about it, it would be sad to have to give that up.

>         Stefan

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Do we really want to rely on the package spec for that?
>> It seems the easiest way to share this information.  Do you have a
>> different idea?
>
> Yes: use the `elpa-admin.el` code to compute that info :-)

It might be possible to use the "previous-revision" VC method to step
through the history to find the same information too... I would prefer
that to avoid a strict Git dependency.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>>>> Do we really want to rely on the package spec for that?
>>> It seems the easiest way to share this information.  Do you have a
>>> different idea?
>>
>> Yes: use the `elpa-admin.el` code to compute that info :-)
>
> It might be possible to use the "previous-revision" VC method to step
> through the history to find the same information too... I would prefer
> that to avoid a strict Git dependency.

That can work when there's a single line of development.  Once you throw
merges into the mix, you'd have to reproduce exactly the algorithm used
by Git, which can become quite challenging.

To be honest, at this point I consider non-Git VCS a bit like XEmacs:
they sure have some really interesting technical characteristics yet
trying to support them is counter-productive.
[ I do hope Git will be replaced by something better, of course, but
  Hg and Brz have already proved they're not Git's successor.  ]


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > To be honest, at this point I consider non-Git VCS a bit like XEmacs:
  > they sure have some really interesting technical characteristics yet
  > trying to support them is counter-productive.

The GNU Project uses CVS.  We need Emacs to continue to support it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: philipk@posteo.net, emacs-devel@gnu.org
> Date: Tue, 01 Nov 2022 12:46:42 -0400
> 
>   > To be honest, at this point I consider non-Git VCS a bit like XEmacs:
>   > they sure have some really interesting technical characteristics yet
>   > trying to support them is counter-productive.
> 
> The GNU Project uses CVS.  We need Emacs to continue to support it.

Agreed.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: philipk@posteo.net, emacs-devel@gnu.org
>> Date: Tue, 01 Nov 2022 12:46:42 -0400
>> 
>>   > To be honest, at this point I consider non-Git VCS a bit like XEmacs:
>>   > they sure have some really interesting technical characteristics yet
>>   > trying to support them is counter-productive.
>> 
>> The GNU Project uses CVS.  We need Emacs to continue to support it.
>
> Agreed.

On that topic, are there any Emacs packages being developed using CVS
that I could use for testing?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> The GNU Project uses CVS.  We need Emacs to continue to support it.

I don't think anyone disagrees with that (though some people might ask
why GNU is still using CVS).

But isn't this discussion about package-vc.el?  I don't think I
understand how a convenience feature intended to simplify Emacs
package installation and development could affect the GNU project as a
whole.  Are there any GNU packages for Emacs that still use CVS?  If
yes, why don't they use Git?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Tue, 1 Nov 2022 19:35:19 +0100
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, philipk@posteo.net,
>  emacs-devel@gnu.org
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > The GNU Project uses CVS.  We need Emacs to continue to support it.
> 
> I don't think anyone disagrees with that (though some people might ask
> why GNU is still using CVS).

That question is not really relevant to the issue at hand.

> But isn't this discussion about package-vc.el?  I don't think I
> understand how a convenience feature intended to simplify Emacs
> package installation and development could affect the GNU project as a
> whole.  Are there any GNU packages for Emacs that still use CVS?  If
> yes, why don't they use Git?

Again, not relevant.  package-vc should support all the VCS backends
that VC supports, for as long as Emacs supports that VCS in general.
(If you want to argue for removing support for CVS, you may, but
that's a separate discussion.)

I understand Stefan's opinion, but it isn't our policy, at least not
yet, not until and unless we decide to drop support for all the VCSes
except Git.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> I understand Stefan's opinion, but it isn't our policy, at least not
> yet, not until and unless we decide to drop support for all the VCSes
> except Git.

To be clear, I do not advocate to drop support for non-Git VCS.
I find that *improving* the support for non-Git VCS (at this time) is
ultimately counter-productive.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Stefan Kangas <stefankangas@gmail.com>,  rms@gnu.org,
>   philipk@posteo.net,  emacs-devel@gnu.org
> Date: Tue, 01 Nov 2022 15:04:27 -0400
> 
> > I understand Stefan's opinion, but it isn't our policy, at least not
> > yet, not until and unless we decide to drop support for all the VCSes
> > except Git.
> 
> To be clear, I do not advocate to drop support for non-Git VCS.
> I find that *improving* the support for non-Git VCS (at this time) is
> ultimately counter-productive.

No one was talking about improving support of VCSes beyond what we
support for Git.  But we should strive to have all the VCSes supported
the same, and that cannot be considered counter-productive, because
that's what VC is about, by design.  So when we add some feature for
Git, we should try supporting it for other VCSes as well, as a matter
of policy.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> > Are there any GNU packages for Emacs that still use CVS?  If
> > yes, why don't they use Git?
>
> Again, not relevant.  package-vc should support all the VCS backends
> that VC supports, for as long as Emacs supports that VCS in general.

Of course it is relevant.  AFAIU, Stefan M argued that it's *not very
important*, not that it shouldn't be supported at all.  To determine
if it is important or not, the questions I asked are exactly the ones
that need to be answered.

Actually, I could rephrase it: Is there any Emacs Lisp package out
there that use CVS?  I don't know of any, myself.

> (If you want to argue for removing support for CVS, you may, but
> that's a separate discussion.)

I don't think anyone wants to do that.  I wrote that in my previous message.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Stefan Kangas [2022-11-01 20:26:11] wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>> > Are there any GNU packages for Emacs that still use CVS?  If
>> > yes, why don't they use Git?
>>
>> Again, not relevant.  package-vc should support all the VCS backends
>> that VC supports, for as long as Emacs supports that VCS in general.
>
> Of course it is relevant.  AFAIU, Stefan M argued that it's *not very
> important*, not that it shouldn't be supported at all.  To determine
> if it is important or not, the questions I asked are exactly the ones
> that need to be answered.

It goes further that that, by the way: the few times I have to interact
with an Hg or Bzr repository, I don't use Hg or Bzr, I use Git.

Back when I was using Bzr instead, I did the same: I used Bzr to access
Svn, Hg, and Git repositories.

[ Some tools go even a step further and decouple the tool used locally
  from the format used locally, so you can use the Brz tool on a local
  Git clone IIUC.  ]

So `package-vc` doesn't need to support all VCS because we can rely on
bridges to decouple the tool used locally from the format of
the repository (tho, those adapters/bridges tend to be a bit fiddly,
admittedly).

> Actually, I could rephrase it: Is there any Emacs Lisp package out
> there that use CVS?  I don't know of any, myself.

Good question.  I don't think there's any such thing in MELPA, at
least :-)

The only non-Git repositories I know of are either Hg or Bzr.
I wouldn't be surprised if there's some package stored in Svn
somewhere, tho.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> So `package-vc` doesn't need to support all VCS because we can rely on
> bridges to decouple the tool used locally from the format of
> the repository (tho, those adapters/bridges tend to be a bit fiddly,
> admittedly).

So why doesn't vc.el just do this too?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-11-01 22:19:10] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> So `package-vc` doesn't need to support all VCS because we can rely on
>> bridges to decouple the tool used locally from the format of
>> the repository (tho, those adapters/bridges tend to be a bit fiddly,
>> admittedly).
>
> So why doesn't vc.el just do this too?

`vc.el` has to work with an existing local clone, whereas for
`package-vc` we could choose to impose a particular local VCS tool
because we're in control of the creation of the local clone.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> So why doesn't vc.el just do this too?
> `vc.el` has to work with an existing local clone, whereas for
> `package-vc` we could choose to impose a particular local VCS tool
> because we're in control of the creation of the local clone.

Oh, and another reason is that `vc.el` has supported that, so not
supporting it would be a regression, whereas `package-vc` is new, so
even if it only supports Git, it's better than before.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> So why doesn't vc.el just do this too?
>> `vc.el` has to work with an existing local clone, whereas for
>> `package-vc` we could choose to impose a particular local VCS tool
>> because we're in control of the creation of the local clone.

That would break the command `package-vc-link-directory', that re-uses a
local clone by linking it into the package directory.

> Oh, and another reason is that `vc.el` has supported that, so not
> supporting it would be a regression, whereas `package-vc` is new, so
> even if it only supports Git, it's better than before.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-11-02 08:01:55] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> So why doesn't vc.el just do this too?
>>> `vc.el` has to work with an existing local clone, whereas for
>>> `package-vc` we could choose to impose a particular local VCS tool
>>> because we're in control of the creation of the local clone.
> That would break the command `package-vc-link-directory', that re-uses a
> local clone by linking it into the package directory.

So what?  Until now this command has failed in all contexts because it
didn't exist.  :-)
And I suspect it will still fail in many cases because it will depend on
the extent of VC coverage for any given VCS.
Will it work DaRCS (which does have a VC backend, tho not included in
Emacs) or Pijul?


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-11-02 08:01:55] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>> So why doesn't vc.el just do this too?
>>>> `vc.el` has to work with an existing local clone, whereas for
>>>> `package-vc` we could choose to impose a particular local VCS tool
>>>> because we're in control of the creation of the local clone.
>> That would break the command `package-vc-link-directory', that re-uses a
>> local clone by linking it into the package directory.
>
> So what?  Until now this command has failed in all contexts because it
> didn't exist.  :-)

^^ I am not sure I follow what you mean?

> And I suspect it will still fail in many cases because it will depend on
> the extent of VC coverage for any given VCS.
> Will it work DaRCS (which does have a VC backend, tho not included in
> Emacs) or Pijul?

I haven't tried it, but I do believe it should work, because the only
critical new VC function is `vc-clone' that isn't used by
`package-vc-link-directory'.

>
>         Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > `vc.el` has to work with an existing local clone, whereas for
  > `package-vc` we could choose to impose a particular local VCS tool
  > because we're in control of the creation of the local clone.

It is true that the local clone is created by `package-vc'.
However, it has to clone a remote repo into that.

We should not arbitrarily limit what VCS the remote repo can use,
given that we already have the facilities (in VC itself) to easily
avoid such limits.

  > Oh, and another reason is that `vc.el` has supported that, so not
  > supporting it would be a regression, whereas `package-vc` is new, so
  > even if it only supports Git, it's better than before.

Just because a certain command is new does not mean it can be limited
to git.  Emacs supports VCS-independent commands and that is a
valuable advanage.  We should extend it to new features whenever that
is feasible.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>   > `vc.el` has to work with an existing local clone, whereas for
>   > `package-vc` we could choose to impose a particular local VCS tool
>   > because we're in control of the creation of the local clone.
>
> It is true that the local clone is created by `package-vc'.
> However, it has to clone a remote repo into that.
>
> We should not arbitrarily limit what VCS the remote repo can use,
> given that we already have the facilities (in VC itself) to easily
> avoid such limits.

Many modern VCS can clone locally a repository using the format of
another VCS.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We should not arbitrarily limit what VCS the remote repo can use,
  > > given that we already have the facilities (in VC itself) to easily
  > > avoid such limits.

  > Many modern VCS can clone locally a repository using the format of
  > another VCS.

It is fine to use that capability in cases where it is applicable
to implement the flexibility we should have.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  philipk@posteo.net,
>   emacs-devel@gnu.org
> Date: Tue, 01 Nov 2022 16:26:30 -0400
> 
> So `package-vc` doesn't need to support all VCS because we can rely on
> bridges to decouple the tool used locally from the format of
> the repository (tho, those adapters/bridges tend to be a bit fiddly,
> admittedly).

Sorry, I disagree.  package-vc _does_ need to support all VCSes, or at
least it should try.  The only way I will agree to what you say is if
we change our general VC policy of supporting VCSes.

> The only non-Git repositories I know of are either Hg or Bzr.

Not relevant.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Alfred M. Szmidt]

   The only non-Git repositories I know of are either Hg or Bzr.
   I wouldn't be surprised if there's some package stored in Svn
   somewhere, tho.

vc-fossil is maintained in Fossil, not git.

There are still projects, not just GNU, that use CVS activley and are
quite happy with it.  This "git or bust" attitude really must stop.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Tue, 1 Nov 2022 20:26:11 +0100
> Cc: rms@gnu.org, monnier@iro.umontreal.ca, philipk@posteo.net, 
> 	emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > Are there any GNU packages for Emacs that still use CVS?  If
> > > yes, why don't they use Git?
> >
> > Again, not relevant.  package-vc should support all the VCS backends
> > that VC supports, for as long as Emacs supports that VCS in general.
> 
> Of course it is relevant.  AFAIU, Stefan M argued that it's *not very
> important*, not that it shouldn't be supported at all.

That's not what he said.

Anyway, my point is that our current policy is to try to support all
the VCSes in VC at the same level.  We don't always succeed, for
various reasons, but we don't consider it "counter-productive" nor
"waste of effort" to do so.  So "importance" is not a factor here.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Dmitry Gutov]

On 02.11.2022 05:25, Eli Zaretskii wrote:
> Anyway, my point is that our current policy is to try to support all
> the VCSes in VC at the same level.

...in VC.

I'm not sure we should consider package-vc to be part of VC.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Dmitry Gutov [2022-11-02 12:18:03] wrote:
> On 02.11.2022 05:25, Eli Zaretskii wrote:
>> Anyway, my point is that our current policy is to try to support all
>> the VCSes in VC at the same level.
> ...in VC.
> I'm not sure we should consider package-vc to be part of VC.

Indeed, it's not.  It's a package which uses VC, quite different.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Stefan Kangas <stefankangas@gmail.com>,
>   rms@gnu.org,  philipk@posteo.net,  emacs-devel@gnu.org
> Date: Wed, 02 Nov 2022 08:45:09 -0400
> 
> Dmitry Gutov [2022-11-02 12:18:03] wrote:
> > On 02.11.2022 05:25, Eli Zaretskii wrote:
> >> Anyway, my point is that our current policy is to try to support all
> >> the VCSes in VC at the same level.
> > ...in VC.
> > I'm not sure we should consider package-vc to be part of VC.
> 
> Indeed, it's not.  It's a package which uses VC, quite different.

Not for the issue at hand, it isn't.  The issue at hand is our policy,
not anything else.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> Date: Wed, 2 Nov 2022 12:18:03 +0200
> Cc: rms@gnu.org, monnier@iro.umontreal.ca, philipk@posteo.net,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 02.11.2022 05:25, Eli Zaretskii wrote:
> > Anyway, my point is that our current policy is to try to support all
> > the VCSes in VC at the same level.
> 
> ...in VC.
> 
> I'm not sure we should consider package-vc to be part of VC.

From where I stand, package-vc is part of the same policy.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 2 Nov 2022 12:18:03 +0200
>> Cc: rms@gnu.org, monnier@iro.umontreal.ca, philipk@posteo.net,
>>  emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> 
>> On 02.11.2022 05:25, Eli Zaretskii wrote:
>> > Anyway, my point is that our current policy is to try to support all
>> > the VCSes in VC at the same level.
>> 
>> ...in VC.
>> 
>> I'm not sure we should consider package-vc to be part of VC.
>
> From where I stand, package-vc is part of the same policy.

I would also like this to be the case.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The GNU Project uses CVS.  We need Emacs to continue to support it.

  > I don't think anyone disagrees with that (though some people might ask
  > why GNU is still using CVS).

Ifa person asked that question as simple curiosity, it might have a simple
answer or it might be "historical reasons not worth explaining."

Nowever, the words you used suggest that people would be making
a demand that the GNU Project change its practices to adopt
the practices they prefer.  That is not nice.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>   > > The GNU Project uses CVS.  We need Emacs to continue to support it.
>
>   > I don't think anyone disagrees with that (though some people might ask
>   > why GNU is still using CVS).
>
> Ifa person asked that question as simple curiosity, it might have a simple
> answer or it might be "historical reasons not worth explaining."
>
> Nowever, the words you used suggest that people would be making
> a demand that the GNU Project change its practices to adopt
> the practices they prefer.  That is not nice.

I don't think there's any demand to change in the question.  I think
it's rather a statement of surprise since in the very vast majority of
cases more modern VCS are so much more convenient that it's hard to
imagine that the inertia is high enough to keep using CVS (or that
there may actually be reasons beside inertia).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   I think
  > it's rather a statement of surprise since in the very vast majority of
  > cases more modern VCS are so much more convenient

That is a matter of opinion.  Some people find git convenient --
others don't.  Emacs handles all the version control systems.  Please
do not push downgrade its support for some of them.  Use the one you
prefer, but don't put pressure on people who prefer a different one.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1114 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> AFAIK this is sufficiently hypothetical that I'm really not convinced
>>>> it's worth the extra work (and the risk of incompatibilities between
>>>> the `elpa-admin.el` behavior and that if `package-vc`, e.g. where they
>>>> might disagree on which commit is "the" release).
>>>
>>> I don't know, to me this sounds more like an argument against merging
>>> elpa-admin.el and package-vc.el...
>>
>> It's not about sharing the code here (for which I have lobbied in other
>> messages), but about sharing the behavior.
>> `elpa-admin.el` de facto defines a "protocol" that upstream maintainers
>> have to follow.  If `package-vc` doesn't follow the same protocol,
>> end-users will suffer unexpected discrepancies.
>
> That is sort of the question I was implying previously (when mentioning
> adding meta data), as to whether or not there is a "protocol" or if the
> two format just coincidentally share a part of a common structure.

Here is what I had in mind.  With the following patch for
elpa-admin.git:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-elpa-admin.el-elpaa-publish-package-specs-Update-fil.patch --]
[-- Type: text/x-patch, Size: 2207 bytes --]

From dd2bbf65ce18cebc28b1118a31ca5d117df1b5c0 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Fri, 28 Oct 2022 19:19:21 +0200
Subject: [PATCH] * elpa-admin.el (elpaa--publish-package-specs): Update file
 format

---
 elpa-admin.el | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/elpa-admin.el b/elpa-admin.el
index 81eb0b9f91..f45a596671 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -792,18 +792,25 @@ of the current `process-environment'.  Return the modified copy."
   "Process and publish SPECS in elpa-packages.eld files."
   (with-temp-buffer
     ;; Remove :core packages, handle :url nil and and compress the
-    ;; contents of the "elpa-packages"
+    ;; contents of the "elpa-packages".  Note that elpa-packages.eld
+    ;; does not use the same format as "elpa-packages" in
+    ;; {nongnu,elpa}.git.  The file is intended to be used by
+    ;; package-vc.el.
     (prin1
-     (mapcan
-      (lambda (spec)
-        (pcase spec
-          (`(,name :url nil . ,rest)
-           `((,name :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
-                    :branch ,(concat elpaa--branch-prefix (car spec))
-                    ,@rest)))
-          (`(,_ :core ,_ . ,_) nil)       ;not supported
-          (_ (list spec))))
-      specs)
+     (list (mapcan
+            (lambda (spec)
+              (let ((rel (ignore-errors (elpaa--get-last-release spec))))
+                (when rel
+                  (nconc spec (list :release-rev (cdr rel)))))
+              (pcase spec
+                (`(,name :url nil . ,rest)
+                 `((,name :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
+                          :branch ,(concat elpaa--branch-prefix (car spec))
+                          ,@rest)))
+                (`(,_ :core ,_ . ,_) nil)       ;not supported
+                (_ (list spec))))
+            specs)
+           :version 1 :default-vc 'Git)
      (current-buffer))
     (write-region nil nil (expand-file-name "elpa-packages.eld" elpaa--release-subdir))
     (write-region nil nil (expand-file-name "elpa-packages.eld" elpaa--devel-subdir))))
-- 
2.38.0


[-- Attachment #3: Type: text/plain, Size: 38 bytes --]


and these changes to package-vc.el:


[-- Attachment #4: Type: text/plain, Size: 3827 bytes --]

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index 8e4f2819db..e995853768 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -56,6 +56,9 @@ package-vc
   :group 'package
   :version "29.1")
 
+(defconst package-vc-elpa-packages-version 1
+  "Version number of the package specification format understood by package-vc.")
+
 (defcustom package-vc-heuristic-alist
   `((,(rx bos "http" (? "s") "://"
           (or (: (? "www.") "github.com"
@@ -144,6 +147,34 @@ package-vc-archive-spec-alist
 
 All other values are ignored.")
 
+(defvar package-vc-archive-data-alist nil
+  "List of package specification archive metadata.
+Each element of the list has the form (ARCHIVE . PLIST), where
+PLIST keys are one of:
+
+        `:version' (integer)
+
+Indicating the version of the file formatting, to be compared
+with `package-vc-elpa-packages-version'.
+
+        `:vc-backend' (symbol)
+
+A symbol indicating what the default VC backend to use if a
+package specification does not indicate anything.  The value
+ought to be a member of `vc-handled-backends'.  If missing,
+`vc-clone' will fall back onto `package-vc-default-backend'.
+
+All other values are ignored.")
+
+(define-inline package-vc-query-archive-data (archive prop)
+  "Query the property PROP for the package specification for PKG-DESC.
+If no package specification can be determined, the function will
+return nil."
+  (inline-letevals (archive prop)
+    (inline-quote (plist-get
+                   (alist-get ,archive package-vc-archive-data-alist)
+                   ,prop))))
+
 (defun package-vc-desc->spec (pkg-desc &optional name)
   "Retrieve the package specification for PKG-DESC.
 The optional argument NAME can be used to override the default
@@ -171,9 +202,23 @@ package-vc--read-archive-data
     (when (file-exists-p contents-file)
       (with-temp-buffer
         (let ((coding-system-for-read 'utf-8))
-          (insert-file-contents contents-file))
-        (setf (alist-get (intern archive) package-vc-archive-spec-alist)
-              (read (current-buffer)))))))
+          (insert-file-contents contents-file)
+          ;; The response from the server is expected to have the form
+          ;;
+          ;;    ((("foo" :url "..." ...) ...)
+          ;;     :version 1
+          ;;     :default-vc Git)
+          (let ((spec (read (current-buffer))))
+            (when (= package-vc-elpa-packages-version
+                     (plist-get (cdr spec) :version))
+              (setf (alist-get (intern archive) package-vc-archive-spec-alist)
+                    (car spec)))
+            (setf (alist-get (intern archive) package-vc-archive-data-alist)
+                  (cdr spec))
+            (when-let ((default-vc (plist-get (cdr spec) :default-vc))
+                       ((not (memq default-vc vc-handled-backends))))
+              (warn "Archive `%S' expects missing VC backend %S"
+                    archive (plist-get (cdr spec) :default-vc)))))))))
 
 (defun package-vc--download-and-read-archives (&optional async)
   "Download specifications of all `package-archives' and read them.
@@ -374,6 +419,10 @@ package-vc-unpack
       (unless (file-exists-p repo-dir)
         (make-directory (file-name-directory repo-dir) t)
         (let ((backend (or (package-vc-guess-backend url)
+                           (plist-get (alist-get (package-desc-archive pkg-desc)
+                                                 package-vc-archive-data-alist
+                                                 nil nil #'string=)
+                                      :vc-backend)
                            package-vc-default-backend)))
           (unless (vc-clone url backend repo-dir (or rev branch))
             (error "Failed to clone %s from %s" name url))))

[-- Attachment #5: Type: text/plain, Size: 489 bytes --]


A more forwards-compatible file format would be defined that would
uncouple the format used by elpa-admin.el and package-vc.el the more I
think about this, the more I think it is better to take this path
instead of sharing code between the two.  Sure, this means that some
things have to be duplicated, but we are talking about two different
scenarios (activating source controlled code and packaging tarballs)
that might have a lot in common but still ought to be considered
separately.

Merging feature/package+vc

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> A more forwards-compatible file format would be defined that would
> uncouple the format used by elpa-admin.el and package-vc.el the more I
> think about this, the more I think it is better to take this path
> instead of sharing code between the two.  Sure, this means that some
> things have to be duplicated, but we are talking about two different
> scenarios (activating source controlled code and packaging tarballs)
> that might have a lot in common but still ought to be considered
> separately.

Unless I have forgotten anything, this should be the last issue
remaining to be decided before the branch can be merged.  It also seems
there were no objections to the current state of the branch as explained
in <87bkqmqpvb.fsf@posteo.net>.

The only other point I can recall, though I wouldn't call it an issue,
is the name of `package-vc-install'.  As I've previously argued, I like
the symmetry to `package-install', but it might make sense to add an
alias.  Previously there was an alias named `package-checkout', but I
decided to remove it because I wasn't satisfied with the choice, but
perhaps something like `package-checkout-sources' or
`package-fetch-sources' could help discoverability/intuition?

Re: Merging feature/package+vc

[Stefan Monnier]

> The only other point I can recall, though I wouldn't call it an issue,
> is the name of `package-vc-install'.

FWIW, I like the `package-vc` prefix for that functionality.

> `package-fetch-sources' could help discoverability/intuition?

I think it gives the wrong intuition.  `package-install` also fetches
the source code.  The difference is whether we get the VCS info with it.

When I describe the way I install packages with `elpa-admin.el`
I usually say that I "install from Git", so maybe for discoverability
purposes we could use something like `package-install-from-vcs` or
`package-install-from-git` (not sure it's worth the trouble,
personally).


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> +     (list (mapcan
> +            (lambda (spec)
> +              (let ((rel (ignore-errors (elpaa--get-last-release spec))))
> +                (when rel
> +                  (nconc spec (list :release-rev (cdr rel)))))
> +              (pcase spec
> +                (`(,name :url nil . ,rest)
> +                 `((,name :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
> +                          :branch ,(concat elpaa--branch-prefix (car spec))
> +                          ,@rest)))
> +                (`(,_ :core ,_ . ,_) nil)       ;not supported
> +                (_ (list spec))))
> +            specs)
> +           :version 1 :default-vc 'Git)

IMO we should strive to reduce a package's spec.  Ideally they should
just be a URL.  We're not there yet, but the above adds info which we
can (and do in `elpa-admin.el`) extract from the VCS, so it's going
"backward".  IOW, it's making things worse rather than better.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +     (list (mapcan
>> +            (lambda (spec)
>> +              (let ((rel (ignore-errors (elpaa--get-last-release spec))))
>> +                (when rel
>> +                  (nconc spec (list :release-rev (cdr rel)))))
>> +              (pcase spec
>> +                (`(,name :url nil . ,rest)
>> +                 `((,name :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
>> +                          :branch ,(concat elpaa--branch-prefix (car spec))
>> +                          ,@rest)))
>> +                (`(,_ :core ,_ . ,_) nil)       ;not supported
>> +                (_ (list spec))))
>> +            specs)
>> +           :version 1 :default-vc 'Git)
>
> IMO we should strive to reduce a package's spec.  Ideally they should
> just be a URL.  

I agree.

>                 We're not there yet, but the above adds info which we
> can (and do in `elpa-admin.el`) extract from the VCS, so it's going
> "backward".  IOW, it's making things worse rather than better.

You mean the addition of :release-rev, right?  Yes, it could be
extracted from the VCS, but finding a generic system is tricky as you
have mentioned.  How about a new VC method `last-change' that takes a
region and returns the last revision that affected it.  Any backend that
supports `annotate' ought to be able to determine it, right?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>>                 We're not there yet, but the above adds info which we
>> can (and do in `elpa-admin.el`) extract from the VCS, so it's going
>> "backward".  IOW, it's making things worse rather than better.
>
> You mean the addition of :release-rev, right?

Yes.

> Yes, it could be extracted from the VCS, but finding a generic system
> is tricky as you have mentioned.  How about a new VC method
> `last-change' that takes a region and returns the last revision that
> affected it.  Any backend that supports `annotate' ought to be able to
> determine it, right?

Fine by me.  As mentioned earlier, there's a good chance that if you use
a different VCS than Git, this method will sometimes end up selecting
a different commit than `elpa-admin.el`, but if we insist on supporting
a local VCS different from the one used by `elpa-admin.el`, then we
probably have to live with that.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>                 We're not there yet, but the above adds info which we
>>> can (and do in `elpa-admin.el`) extract from the VCS, so it's going
>>> "backward".  IOW, it's making things worse rather than better.
>>
>> You mean the addition of :release-rev, right?
>
> Yes.
>
>> Yes, it could be extracted from the VCS, but finding a generic system
>> is tricky as you have mentioned.  How about a new VC method
>> `last-change' that takes a region and returns the last revision that
>> affected it.  Any backend that supports `annotate' ought to be able to
>> determine it, right?
>
> Fine by me.  As mentioned earlier, there's a good chance that if you use
> a different VCS than Git, this method will sometimes end up selecting
> a different commit than `elpa-admin.el`, but if we insist on supporting
> a local VCS different from the one used by `elpa-admin.el`, then we
> probably have to live with that.

I guess so.  Here is a possible patch that should behave close enough to
elpa-admin.el using "blame" (an obvious exception is the missing handler
for the new :merge property, but I have been wondering if it might also
be fair to always add "--first-parent" for Git).

The default handler just wraps vc-annotate, so it is a bit more fragile.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Extract-last-source-package-release-from-local-VCS-d.patch --]
[-- Type: text/x-patch, Size: 10416 bytes --]

From 30f1e7c1e93dda496412f76f70b2f49b30407b11 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Sun, 30 Oct 2022 11:43:11 +0100
Subject: [PATCH] Extract last source package release from local VCS data

* lisp/emacs-lisp/package-vc.el (package-vc-archive-spec-alist):
Unmention :release-rev
(package-vc-desc->spec): Fall back on other archives if a
specification is missing.
(package-vc-main-file): Add new function, copying the behaviour of
elpa-admin.el.
(package-vc-generate-description-file): Use 'package-vc-main-file'.
(package-vc-unpack): Handle special value ':last-release'.
(package-vc-release-rev): Add new function using 'last-change'.
(package-vc-install): Pass ':last-release' as REV instead of a
release.
* lisp/vc/vc-git.el (vc-git-last-change): Add Git 'last-change'
implementation.
* lisp/vc/vc.el (vc-default-last-change): Add default 'last-change'
implementation.

This attempts to replicate the behaviour of elpa-admin.el's
"elpaa--get-last-release-commit".
---
 lisp/emacs-lisp/package-vc.el | 88 +++++++++++++++++++++++------------
 lisp/vc/vc-git.el             | 17 +++++++
 lisp/vc/vc.el                 | 18 +++++++
 3 files changed, 94 insertions(+), 29 deletions(-)

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index 3816c6152d..6597989777 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -139,12 +139,6 @@ package-vc-archive-spec-alist
 metadata.  If not given, the assumed default is the package named
 with \".el\" concatenated to the end.
 
-        `:release-rev' (string)
-
-A revision string indicating the revision used for the current
-release in the package archive.  If missing or nil, no release
-was made.
-
         `:vc-backend' (symbol)
 
 A symbol indicating what the VC backend to use for cloning a
@@ -179,8 +173,10 @@ package-vc-desc->spec
 name for PKG-DESC."
   (alist-get
    (or name (package-desc-name pkg-desc))
-   (alist-get (intern (package-desc-archive pkg-desc))
-              package-vc-archive-spec-alist)
+   (if (package-desc-archive pkg-desc)
+       (alist-get (intern (package-desc-archive pkg-desc))
+                  package-vc-archive-spec-alist)
+     (mapcan #'append (mapcar #'cdr package-vc-archive-spec-alist)))
    nil nil #'string=))
 
 (define-inline package-vc-query-spec (pkg-desc prop)
@@ -258,6 +254,20 @@ package-vc-version
            return it
            finally return "0"))
 
+(defun package-vc-main-file (pkg-desc)
+  "Return the main file for PKG-DESC."
+  (cl-assert (package-vc-p pkg-desc))
+  (let ((pkg-spec (package-vc-desc->spec pkg-desc)))
+    (or (plist-get pkg-spec :main-file)
+        (expand-file-name
+         (format "%s.el" (package-desc-name pkg-desc))
+         (file-name-concat
+          (or (package-desc-dir pkg-desc)
+              (expand-file-name
+               (package-desc-name pkg-desc)
+               package-user-dir))
+          (plist-get pkg-spec :lisp-dir))))))
+
 (defun package-vc-generate-description-file (pkg-desc pkg-file)
   "Generate a package description file for PKG-DESC.
 The output is written out into PKG-FILE."
@@ -265,18 +275,13 @@ package-vc-generate-description-file
     ;; Infer the subject if missing.
     (unless (package-desc-summary pkg-desc)
       (setf (package-desc-summary pkg-desc)
-            (or (package-desc-summary pkg-desc)
-                (and-let* ((pkg (cadr (assq name package-archive-contents))))
-                  (package-desc-summary pkg))
-                (and-let* ((pkg-spec (package-vc-desc->spec pkg-desc))
-                           (main-file (plist-get pkg-spec :main-file)))
-                  (lm-summary main-file))
-                (and-let* ((main-file (expand-file-name
-                                       (format "%s.el" name)
-                                       (package-desc-dir pkg-desc)))
-                           ((file-exists-p main-file)))
-                  (lm-summary main-file))
-                package--default-summary)))
+            (let ((main-file (package-vc-main-file pkg-desc)))
+              (or (package-desc-summary pkg-desc)
+                  (and-let* ((pkg (cadr (assq name package-archive-contents))))
+                    (package-desc-summary pkg))
+                  (and main-file (file-exists-p main-file)
+                       (lm-summary main-file))
+                  package--default-summary))))
     (let ((print-level nil)
           (print-quoted t)
           (print-length nil))
@@ -424,9 +429,16 @@ package-vc-unpack
                                                  nil nil #'string=)
                                       :vc-backend)
                            package-vc-default-backend)))
-          (unless (vc-clone url backend repo-dir (or rev branch))
+          (unless (vc-clone url backend repo-dir
+                            (or (and (not (eq rev :last-release)) rev) branch))
             (error "Failed to clone %s from %s" name url))))
 
+      ;; Check out the latest release if requested
+      (when (eq rev :last-release)
+        (if-let ((release-rev (package-vc-release-rev pkg-desc)))
+            (vc-retrieve-tag pkg-dir release-rev)
+          (message "No release revision was found, continuing...")))
+
       (unless (eq pkg-dir repo-dir)
         ;; Link from the right position in `repo-dir' to the package
         ;; directory in the ELPA store.
@@ -466,6 +478,22 @@ package-vc--archives-initialize
   (unless package-vc-archive-data-alist
     (package-vc--download-and-read-archives)))
 
+(defun package-vc-release-rev (pkg-desc)
+  "Find the latest revision that bumps the \"Version\" tag for PKG-DESC.
+If no such revision can be found, return nil."
+  (with-current-buffer (find-file-noselect (package-vc-main-file pkg-desc))
+    (vc-buffer-sync)
+    (save-excursion
+      (goto-char (point-min))
+      (let ((case-fold-search t))
+        (when (re-search-forward (concat (lm-get-header-re "version") ".*$")
+                                 (lm-code-start) t)
+          (ignore-error vc-not-supported
+            (vc-call-backend (vc-backend (buffer-file-name))
+                             'last-change
+                             (match-beginning 0)
+                             (match-end 0))))))))
+
 ;;;###autoload
 (defun package-vc-install (name-or-url &optional name rev backend)
   "Fetch the source of NAME-OR-URL.
@@ -477,9 +505,11 @@ package-vc-install
 metadata will be consulted for the URL.  An explicit revision can
 be requested using REV.  If the command is invoked with a prefix
 argument, the revision used for the last release in the package
-archive is used.  If a NAME-OR-URL is a URL, that is to say a
-string, the VC backend used to clone the repository can be set by
-BACKEND.  If missing, `package-vc-guess-backend' will be used."
+archive is used.  This can also be reproduced by passing the
+special value `:last-release' as REV.  If a NAME-OR-URL is a URL,
+that is to say a string, the VC backend used to clone the
+repository can be set by BACKEND.  If missing,
+`package-vc-guess-backend' will be used."
   (interactive
    (progn
      ;; Initialize the package system to get the list of package
@@ -490,11 +520,7 @@ package-vc-install
                     "Fetch package source (name or URL): " packages))
             (name (file-name-base input)))
        (list input (intern (string-remove-prefix "emacs-" name))
-             (and current-prefix-arg
-                  (or (package-vc-query-spec
-                       (cadr (assoc input package-archive-contents #'string=))
-                       :release-rev)
-                      (user-error "No release revision was found")))))))
+             (and current-prefix-arg :last-release)))))
   (package-vc--archives-initialize)
   (cond
    ((and-let* ((stringp name-or-url)
@@ -511,6 +537,10 @@ package-vc-install
          (setf (package-desc-kind copy) 'vc)
          copy)
        (or (package-vc-desc->spec (cadr desc))
+           (and-let* ((extras (package-desc-extras (cadr desc)))
+                      (url (alist-get :url extras))
+                      (backend (package-vc-guess-backend url)))
+             (list :vc-backend backend :url url))
            (user-error "Package has no VC data"))
        rev)))
    ((user-error "Unknown package to fetch: %s" name-or-url))))
diff --git a/lisp/vc/vc-git.el b/lisp/vc/vc-git.el
index 6137ce75ce..cd62effd08 100644
--- a/lisp/vc/vc-git.el
+++ b/lisp/vc/vc-git.el
@@ -1632,6 +1632,23 @@ vc-git-annotate-extract-revision-at-line
 		    (expand-file-name fname (vc-git-root default-directory))))
 	  revision)))))
 
+(defun vc-git-last-change (from to)
+  (vc-buffer-sync)
+  (let ((file (file-relative-name
+               (buffer-file-name)
+               (vc-git-root (buffer-file-name))))
+        (start (line-number-at-pos from t))
+        (end (line-number-at-pos to t)))
+    (with-temp-buffer
+      (when (vc-git--out-ok
+             "blame" "--porcelain"
+             (format "-L%d,%d" start end)
+             file)
+        (goto-char (point-min))
+        (save-match-data
+          (when (looking-at "\\`\\([[:alnum:]]+\\)[[:space:]]+")
+            (match-string 1)))))))
+
 ;;; TAG/BRANCH SYSTEM
 
 (declare-function vc-read-revision "vc"
diff --git a/lisp/vc/vc.el b/lisp/vc/vc.el
index 38209ef39e..c8d28c144b 100644
--- a/lisp/vc/vc.el
+++ b/lisp/vc/vc.el
@@ -448,6 +448,11 @@
 ;; - mergebase (rev1 &optional rev2)
 ;;
 ;;   Return the common ancestor between REV1 and REV2 revisions.
+;;
+;; - last-change (from to)
+;;
+;;   Return the most recent revision that made a change between FROM
+;;   and TO.
 
 ;; TAG/BRANCH SYSTEM
 ;;
@@ -3584,6 +3589,19 @@ vc-clone
                            remote directory rev)))
             (throw 'ok res)))))))
 
+(declare-function log-view-current-tag "log-view" (&optional pos))
+(defun vc-default-last-change (_backend from to)
+  "Default `last-change' implementation.
+FROM and TO are used as region markers"
+  (save-window-excursion
+    (let* ((buf (window-buffer (vc-region-history from to)))
+           (proc (get-buffer-process buf)))
+      (cl-assert (processp proc))
+      (while (accept-process-output proc))
+      (with-current-buffer buf
+        (prog1 (log-view-current-tag)
+          (kill-buffer))))))
+
 \f
 
 ;; These things should probably be generally available
-- 
2.38.0


[-- Attachment #3: Type: text/plain, Size: 353 bytes --]


Invoking `package-vc-install' with a prefix argument will now check out
the specific commit that bumps the version tag.  At least for git, the
slight problem here is that this means the head is in a detached state,
not connected to any specific branch.  I don't know if there is any
elegant solution to this problem, or if it should even be "solved".

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> I guess so.  Here is a possible patch that should behave close enough to
> elpa-admin.el using "blame"

Why do you use `blame` rather than `log` like `elpa-admin.el`?

> (an obvious exception is the missing handler for the new :merge
> property, but I have been wondering if it might also be fair to always
> add "--first-parent" for Git).

We considered using `--first-parent` but it tends to point to more
recent merge commits than to the actual commit that bumped the
`Version:`.  If you add to that the fact that order of parents in Git is
somewhat arbitrary and rarely taken into consideration, I'd rather not
go there.

For `:merge` we didn't really have a choice, and (to make up for that)
we get to control how the merge is done and thus which one is the
first parent.

> The default handler just wraps vc-annotate, so it is a bit more fragile.

Hmm... the code I see in your patch uses `vc-region-history` (which is
only supported for Git and Hg, currently, and is fairly difficult to
support in a generic way) rather than `vc-annotate`.
Am I missing something?

> Invoking `package-vc-install' with a prefix argument will now check out
> the specific commit that bumps the version tag.  At least for git, the
> slight problem here is that this means the head is in a detached state,
> not connected to any specific branch.  I don't know if there is any
> elegant solution to this problem, or if it should even be "solved".

I suspect a better option is to use something like `git reset` instead
of `git checkout`, so we end up at the right revision but still on the
main branch.  But yes, it's kind of ugly if you want to try and preserve
local changes.  I think doing it right requires distinguishing whether
we're moving forward or backward (moving forward can be done with `git
merge`, which is well-behaved, whereas moving backward is poorly
supported, AFAICT).

You might want to take a look at `elpaa--select-revision` where I try to
solve this problem (for a slightly more restricted case, admittedly, but
it's already unsatisfactory).

Maybe there's a better way to do that?


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I guess so.  Here is a possible patch that should behave close enough to
>> elpa-admin.el using "blame"
>
> Why do you use `blame` rather than `log` like `elpa-admin.el`?

I tried to use log like in elpa-admin.el, but the issue was I had to
either

1. Define the generic interface to search for a line using a regular
   expression that is not an Elisp regular expression, and could
   conceivably differ between different VCS that might also implement
   'last-change'.

2. Deal with the issue that the Version header might move between
   revisions, meaning that a line range wouldn't do a good job at
   capturing the right commit.

>> (an obvious exception is the missing handler for the new :merge
>> property, but I have been wondering if it might also be fair to always
>> add "--first-parent" for Git).
>
> We considered using `--first-parent` but it tends to point to more
> recent merge commits than to the actual commit that bumped the
> `Version:`.  If you add to that the fact that order of parents in Git is
> somewhat arbitrary and rarely taken into consideration, I'd rather not
> go there.
>
> For `:merge` we didn't really have a choice, and (to make up for that)
> we get to control how the merge is done and thus which one is the
> first parent.

I see.

>> The default handler just wraps vc-annotate, so it is a bit more fragile.
>
> Hmm... the code I see in your patch uses `vc-region-history` (which is
> only supported for Git and Hg, currently, and is fairly difficult to
> support in a generic way) rather than `vc-annotate`.
> Am I missing something?

Uh, that is my mistake.  I started writing that code yesterday (I
believe?) and simply forgot what I had used.  I'll try to translate that
into vc-annotate before pushing anything.

>> Invoking `package-vc-install' with a prefix argument will now check out
>> the specific commit that bumps the version tag.  At least for git, the
>> slight problem here is that this means the head is in a detached state,
>> not connected to any specific branch.  I don't know if there is any
>> elegant solution to this problem, or if it should even be "solved".
>
> I suspect a better option is to use something like `git reset` instead
> of `git checkout`, so we end up at the right revision but still on the
> main branch.  But yes, it's kind of ugly if you want to try and preserve
> local changes.  I think doing it right requires distinguishing whether
> we're moving forward or backward (moving forward can be done with `git
> merge`, which is well-behaved, whereas moving backward is poorly
> supported, AFAICT).

Checking out a specific revision is currently only done right after
cloning, so this is always a reset.  The issue here is that I am trying
to stay generic and was using 'vc-retrieve-tag' (but perhaps
'vc-checkout' would be better), so this is an issue that might have to
be tackled in vc-git.el...

> You might want to take a look at `elpaa--select-revision` where I try to
> solve this problem (for a slightly more restricted case, admittedly, but
> it's already unsatisfactory).
>
> Maybe there's a better way to do that?

Nothing immediate I can think of.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

Philip Kaludercic [2022-10-30 14:15:07] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I guess so.  Here is a possible patch that should behave close enough to
>>> elpa-admin.el using "blame"
>>
>> Why do you use `blame` rather than `log` like `elpa-admin.el`?
>
> I tried to use log like in elpa-admin.el, but the issue was I had to
> either
>
> 1. Define the generic interface to search for a line using a regular
>    expression that is not an Elisp regular expression, and could
>    conceivably differ between different VCS that might also implement
>    'last-change'.
>
> 2. Deal with the issue that the Version header might move between
>    revisions, meaning that a line range wouldn't do a good job at
>    capturing the right commit.

FWIW, I suspect that your `git blame` will give the same answer as the
corresponding `git log` (it will probably end up using the same code
under the hood), but I don't understand what you mean above.  I think
`git log` would accept the exact same `-L` as you use for `git blame`
and will deal with "the issue that the Version header might move" in the
exact same way.

>> Hmm... the code I see in your patch uses `vc-region-history` (which is
>> only supported for Git and Hg, currently, and is fairly difficult to
>> support in a generic way) rather than `vc-annotate`.
>> Am I missing something?
>
> Uh, that is my mistake.  I started writing that code yesterday (I
> believe?) and simply forgot what I had used.  I'll try to translate that
> into vc-annotate before pushing anything.

[ Just to be clear, I wasn't implying a preference between `vc-annotate`
  and `vc-region-history`.  As fallback, they both seem fine to me.  ]

> Checking out a specific revision is currently only done right after
> cloning, so this is always a reset.  The issue here is that I am trying
> to stay generic and was using 'vc-retrieve-tag' (but perhaps
> 'vc-checkout' would be better), so this is an issue that might have to
> be tackled in vc-git.el...

I find this part of VC beyond repair: AFAICT the notion of
"tag/snapshot/..."  has been defined "on the fly" with a narrow view of
what was needed for one particular situation at a particular time.

It's probably worth "throwing it out" (well, keeping it on life support)
and introducing a whole new set of operations based on a design that
tries to accommodate all the various related notions out there.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-10-30 14:15:07] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> I guess so.  Here is a possible patch that should behave close enough to
>>>> elpa-admin.el using "blame"
>>>
>>> Why do you use `blame` rather than `log` like `elpa-admin.el`?
>>
>> I tried to use log like in elpa-admin.el, but the issue was I had to
>> either
>>
>> 1. Define the generic interface to search for a line using a regular
>>    expression that is not an Elisp regular expression, and could
>>    conceivably differ between different VCS that might also implement
>>    'last-change'.
>>
>> 2. Deal with the issue that the Version header might move between
>>    revisions, meaning that a line range wouldn't do a good job at
>>    capturing the right commit.
>
> FWIW, I suspect that your `git blame` will give the same answer as the
> corresponding `git log` (it will probably end up using the same code
> under the hood), but I don't understand what you mean above.  I think
> `git log` would accept the exact same `-L` as you use for `git blame`
> and will deal with "the issue that the Version header might move" in the
> exact same way.

Well apparently not, but I am not entirely sure what the issue was.  If
you want to, I could take a better look at it and figure out if I missed
something.

>>> Hmm... the code I see in your patch uses `vc-region-history` (which is
>>> only supported for Git and Hg, currently, and is fairly difficult to
>>> support in a generic way) rather than `vc-annotate`.
>>> Am I missing something?
>>
>> Uh, that is my mistake.  I started writing that code yesterday (I
>> believe?) and simply forgot what I had used.  I'll try to translate that
>> into vc-annotate before pushing anything.
>
> [ Just to be clear, I wasn't implying a preference between `vc-annotate`
>   and `vc-region-history`.  As fallback, they both seem fine to me.  ]

I think vc-annotate is preferable, as it is more widespread.

>> Checking out a specific revision is currently only done right after
>> cloning, so this is always a reset.  The issue here is that I am trying
>> to stay generic and was using 'vc-retrieve-tag' (but perhaps
>> 'vc-checkout' would be better), so this is an issue that might have to
>> be tackled in vc-git.el...
>
> I find this part of VC beyond repair: AFAICT the notion of
> "tag/snapshot/..."  has been defined "on the fly" with a narrow view of
> what was needed for one particular situation at a particular time.
>
> It's probably worth "throwing it out" (well, keeping it on life support)
> and introducing a whole new set of operations based on a design that
> tries to accommodate all the various related notions out there.

I agree, but that issue goes beyond the proposal of feature/package+vc.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> FWIW, I suspect that your `git blame` will give the same answer as the
>> corresponding `git log` (it will probably end up using the same code
>> under the hood), but I don't understand what you mean above.  I think
>> `git log` would accept the exact same `-L` as you use for `git blame`
>> and will deal with "the issue that the Version header might move" in the
>> exact same way.
>
> Well apparently not, but I am not entirely sure what the issue was.  If
> you want to, I could take a better look at it and figure out if I missed
> something.

Not worth it.  As I said, I think it'll give the same info anyway.

>> I find this part of VC beyond repair: AFAICT the notion of
>> "tag/snapshot/..."  has been defined "on the fly" with a narrow view of
>> what was needed for one particular situation at a particular time.
>>
>> It's probably worth "throwing it out" (well, keeping it on life support)
>> and introducing a whole new set of operations based on a design that
>> tries to accommodate all the various related notions out there.
>
> I agree, but that issue goes beyond the proposal of feature/package+vc.

No doubt :-)


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> FWIW, I suspect that your `git blame` will give the same answer as the
>>> corresponding `git log` (it will probably end up using the same code
>>> under the hood), but I don't understand what you mean above.  I think
>>> `git log` would accept the exact same `-L` as you use for `git blame`
>>> and will deal with "the issue that the Version header might move" in the
>>> exact same way.
>>
>> Well apparently not, but I am not entirely sure what the issue was.  If
>> you want to, I could take a better look at it and figure out if I missed
>> something.
>
> Not worth it.  As I said, I think it'll give the same info anyway.
>
>>> I find this part of VC beyond repair: AFAICT the notion of
>>> "tag/snapshot/..."  has been defined "on the fly" with a narrow view of
>>> what was needed for one particular situation at a particular time.
>>>
>>> It's probably worth "throwing it out" (well, keeping it on life support)
>>> and introducing a whole new set of operations based on a design that
>>> tries to accommodate all the various related notions out there.
>>
>> I agree, but that issue goes beyond the proposal of feature/package+vc.
>
> No doubt :-)

OK, I've merged master into feature/package+vc, resolving all conflicts
and would be prepared to merge feature/package+vc if there are no
further objections.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> OK, I've merged master into feature/package+vc, resolving all conflicts
> and would be prepared to merge feature/package+vc if there are no
> further objections.

No objections on my side,


        Stefan

feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> OK, I've merged master into feature/package+vc, resolving all conflicts
>> and would be prepared to merge feature/package+vc if there are no
>> further objections.
>
> No objections on my side,

I have merged the branch onto master, so if anyone wants to fix,
add/remove or improve anything, feel free to do so.

Re: feature/package-vc has been merged

[Stefan Monnier]

> I have merged the branch onto master,

Yay!  Thanks!


        Stefan

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Richard Stallman <rms@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 04 Nov 2022 18:01:09 +0000
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> OK, I've merged master into feature/package+vc, resolving all conflicts
> >> and would be prepared to merge feature/package+vc if there are no
> >> further objections.
> >
> > No objections on my side,
> 
> I have merged the branch onto master, so if anyone wants to fix,
> add/remove or improve anything, feel free to do so.

Thanks.  A few comments about the documentation (you will see that I
fixed some of the issues where I could):

> (defcustom package-vc-repository-store
>   (expand-file-name "emacs/vc-packages" (xdg-data-home))
>   "Directory used by `package-vc--unpack' to store repositories."

The doc string of a user option should not reference an internal
function, it should reference user-visible features.  Would it be
correct to say that this directory is used by package-vc to store
repositories of packages it fetches and/or installs?

> (defun package-vc-ensure-packages ()
>   "Ensure source packages specified in `package-vc-selected-packages'."

This doc string should explain what does this function ensure.
"Ensure source" doesn't clarify that.  I think, but cannot be sure,
this should say something like

  Ensure packages specified in `package-vc-selected-packages' are installed."

> (defcustom package-vc-selected-packages '()
>   "List of packages that must be installed.
> Each member of the list is of the form (NAME . SPEC), where NAME
> is a symbol designating the package and SPEC is one of:
> 
> - nil, if any package version can be installed;
> - a version string, if that specific revision is to be installed;
> - a property list of the form described in
>   `package-vc-archive-spec-alist', giving a package
>   specification.

There's no variable package-vc-archive-spec-alist.  Did you mean
package-vc--archive-spec-alist instead?  In that case, I think the
format of that list should be spelled out in this doc string, as the
referenced variable is an internal one.

> This user option differs from `package-selected-packages' in that
> it is meant to be specified manually.  You can also use the
> function `package-vc-selected-packages' to apply the changes."

The function package-vc-selected-packages mentioned in the last
sentence doesn't seem to exist.  Also, what does it mean "to apply the
changes" -- apply the changes to what?

> (defvar package-vc--archive-spec-alist nil
>   "List of package specifications for each archive.
> The list maps each package name, as a string, to a plist.
> Valid keys include
> 
>         `:url' (string)

The "Valid keys" part is "out of the blue" here.  Keys of what?  If
that's a reference to the "plist" part preceding it, then we aren't
talking about a plist, we are talking about keyword/value pairs,
right?

> (defun package-vc--version (pkg)
>   "Extract the commit of a development package PKG."

This doc string doesn't seem to match what the function does.

> (defun package-vc--build-documentation (pkg-desc file)
>   "Build documentation FILE for PKG-DESC."
>   (let ((pkg-dir (package-desc-dir pkg-desc)))
>     (when (string-match-p "\\.org\\'" file)
>       (require 'ox)
>       (require 'ox-texinfo)
>       (with-temp-buffer
>         (insert-file-contents file)
>         (setq file (make-temp-file "ox-texinfo-"))
>         (org-export-to-file 'texinfo file)))
>     (call-process "install-info" nil nil nil
>                   file pkg-dir)))

I'm confused by this function.  Does org-export-to-file produce a
.texi file or an Info file?  In any case, the semantics is confusing:
if FILE has the .org extension, the function installs a file whose
name is not FILE, otherwise the function installs FILE.  This seems to
conflate source and destination files in a confusing way.

> (defun package-vc-update (pkg-desc)
>   "Attempt to update the package PKG-DESC."
>   ;; HACK: To run `package-vc--unpack-1' after checking out the new
>   ;; revision, we insert a hook into `vc-post-command-functions', and
>   ;; remove it right after it ran.  To avoid running the hook multiple
>   ;; times or even for the wrong repository (as `vc-pull' is often
>   ;; asynchronous), we extract the relevant arguments using a pseudo
>   ;; filter for `vc-filter-command-function', executed only for the
>   ;; side effect, and store them in the lexical scope.  When the hook
>   ;; is run, we check if the arguments are the same (`eq') as the ones
>   ;; previously extracted, and only in that case will be call
>   ;; `package-vc--unpack-1'.  Ugh...

Shouldn't this use unwind-protect, to make sure the hacked
post-command-hook doesn't leak out?

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Richard Stallman <rms@gnu.org>,  emacs-devel@gnu.org
>> Date: Fri, 04 Nov 2022 18:01:09 +0000
>> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>> >> OK, I've merged master into feature/package+vc, resolving all
>> >> conflicts
>> >> and would be prepared to merge feature/package+vc if there are no
>> >> further objections.
>> >
>> > No objections on my side,
>> 
>> I have merged the branch onto master, so if anyone wants to fix,
>> add/remove or improve anything, feel free to do so.
>
> Thanks.  A few comments about the documentation (you will see that I
> fixed some of the issues where I could):
>
>> (defcustom package-vc-repository-store
>>   (expand-file-name "emacs/vc-packages" (xdg-data-home))
>>   "Directory used by `package-vc--unpack' to store repositories."
>
> The doc string of a user option should not reference an internal
> function, it should reference user-visible features.  Would it be
> correct to say that this directory is used by package-vc to store
> repositories of packages it fetches and/or installs?

You are right, I forgot about this when renaming package-vc-unpack to
package-vc--unpack.  How about

  "Directory used to store clone repositories.  
                                                
This directory is only used for packages with a special `:lisp-dir'
entry in their package specification (for details on package
specifications see `package-vc-selected-packages').  Repositories for
packages like these are cloned into the directory specified here, and
their `:lisp-dir' is then linked back into the user elpa directory."

>> (defun package-vc-ensure-packages ()
>>   "Ensure source packages specified in `package-vc-selected-packages'."
>
> This doc string should explain what does this function ensure.
> "Ensure source" doesn't clarify that.  I think, but cannot be sure,
> this should say something like

My intention was for it to be read like "Ensure (source packages)
specified in ...".  But yes, the phrasing is inelegant.

>   Ensure packages specified in `package-vc-selected-packages' are
> installed."

It is a bit long, but checkdoc doesn't appear to complain.

>> (defcustom package-vc-selected-packages '()
>>   "List of packages that must be installed.
>> Each member of the list is of the form (NAME . SPEC), where NAME
>> is a symbol designating the package and SPEC is one of:
>> 
>> - nil, if any package version can be installed;
>> - a version string, if that specific revision is to be installed;
>> - a property list of the form described in
>>   `package-vc-archive-spec-alist', giving a package
>>   specification.
>
> There's no variable package-vc-archive-spec-alist.  Did you mean
> package-vc--archive-spec-alist instead?  In that case, I think the
> format of that list should be spelled out in this doc string, as the
> referenced variable is an internal one.

You are right (same mistake as before), the documentation for package
specifications should be moved from the internal variable to either
'package-vc-selected-packages' or some other place.  It might not be bad
to document it in package.texi either?  Or would that be too Lisp-y for
the Emacs manual?

>> This user option differs from `package-selected-packages' in that
>> it is meant to be specified manually.  You can also use the
>> function `package-vc-selected-packages' to apply the changes."
>
> The function package-vc-selected-packages mentioned in the last
> sentence doesn't seem to exist.  Also, what does it mean "to apply the
> changes" -- apply the changes to what?

It was supposed to be `package-vc-ensure-packages'.  How about
rephrasing that last sentence into "By calling the function
`package-vc-selected-packages", the packages specified in this list can
also be installed.

>> (defvar package-vc--archive-spec-alist nil
>>   "List of package specifications for each archive.
>> The list maps each package name, as a string, to a plist.
>> Valid keys include
>> 
>>         `:url' (string)
>
> The "Valid keys" part is "out of the blue" here.  Keys of what?  If
> that's a reference to the "plist" part preceding it, then we aren't
> talking about a plist, we are talking about keyword/value pairs,
> right?

Yes, these are plist keys.  Should we say something like "Valid keys for
these plists include:".  But as I have said above, the documentation for
package specifications (which is what the plist is) should be documented
somewhere else.

>>   "Extract the commit of a development package PKG."
>
> This doc string doesn't seem to match what the function does.

I was confused at first, as I was assuming you were talking about
`package-vc-commit', but it appears I accidentally copied the docstring
into `package-vc--version'.  A better documentation string here would be

     "Return the version number for the source package PKG."

>> (defun package-vc--build-documentation (pkg-desc file)
>>   "Build documentation FILE for PKG-DESC."
>>   (let ((pkg-dir (package-desc-dir pkg-desc)))
>>     (when (string-match-p "\\.org\\'" file)
>>       (require 'ox)
>>       (require 'ox-texinfo)
>>       (with-temp-buffer
>>         (insert-file-contents file)
>>         (setq file (make-temp-file "ox-texinfo-"))
>>         (org-export-to-file 'texinfo file)))
>>     (call-process "install-info" nil nil nil
>>                   file pkg-dir)))
>
> I'm confused by this function.  Does org-export-to-file produce a
> .texi file or an Info file?  In any case, the semantics is confusing:
> if FILE has the .org extension, the function installs a file whose
> name is not FILE, otherwise the function installs FILE.  This seems to
> conflate source and destination files in a confusing way.

I can expand the documentation here.

"file" is the input, either a .texi or an .org file.  If it is an .org
file, we want to convert it to .texi first (which is what the (when ...)
block does), and write the output into a temporary file.  The temporary
file is used as an input to install-info, that builds the .info file.

>> (defun package-vc-update (pkg-desc)
>>   "Attempt to update the package PKG-DESC."
>>   ;; HACK: To run `package-vc--unpack-1' after checking out the new
>>   ;; revision, we insert a hook into `vc-post-command-functions', and
>>   ;; remove it right after it ran.  To avoid running the hook multiple
>>   ;; times or even for the wrong repository (as `vc-pull' is often
>>   ;; asynchronous), we extract the relevant arguments using a pseudo
>>   ;; filter for `vc-filter-command-function', executed only for the
>>   ;; side effect, and store them in the lexical scope.  When the hook
>>   ;; is run, we check if the arguments are the same (`eq') as the ones
>>   ;; previously extracted, and only in that case will be call
>>   ;; `package-vc--unpack-1'.  Ugh...
>
> Shouldn't this use unwind-protect, to make sure the hacked
> post-command-hook doesn't leak out?

The issue is that vc-pull *can be* asynchronous, so wrapping it in a
`unwind-protect' would un-modify the hook too soon.  But you are right
that if vc-pull were to fail, the hook would remain inside of
`vc-post-command-functions' for too long.

As I said in that comment

   "If there is a better way to do this, it should be done."

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 05 Nov 2022 16:43:45 +0000
> 
> >> (defcustom package-vc-repository-store
> >>   (expand-file-name "emacs/vc-packages" (xdg-data-home))
> >>   "Directory used by `package-vc--unpack' to store repositories."
> >
> > The doc string of a user option should not reference an internal
> > function, it should reference user-visible features.  Would it be
> > correct to say that this directory is used by package-vc to store
> > repositories of packages it fetches and/or installs?
> 
> You are right, I forgot about this when renaming package-vc-unpack to
> package-vc--unpack.  How about
> 
>   "Directory used to store clone repositories.  
>                                                 
> This directory is only used for packages with a special `:lisp-dir'
> entry in their package specification (for details on package
> specifications see `package-vc-selected-packages').  Repositories for
> packages like these are cloned into the directory specified here, and
> their `:lisp-dir' is then linked back into the user elpa directory."

Instead of "specified here", please use "specified by this variable".
And I don't think I understand what the "their `:lisp-dir' is then
linked back into the user elpa directory" part wants to say.

I also have a more general question: you say this "is only used for
packages with a special `:lisp-dir' entry", and that begs the
question: where will we clone a package that doesn't have the
:lisp-dir entry?

> >> (defcustom package-vc-selected-packages '()
> >>   "List of packages that must be installed.
> >> Each member of the list is of the form (NAME . SPEC), where NAME
> >> is a symbol designating the package and SPEC is one of:
> >> 
> >> - nil, if any package version can be installed;
> >> - a version string, if that specific revision is to be installed;
> >> - a property list of the form described in
> >>   `package-vc-archive-spec-alist', giving a package
> >>   specification.
> >
> > There's no variable package-vc-archive-spec-alist.  Did you mean
> > package-vc--archive-spec-alist instead?  In that case, I think the
> > format of that list should be spelled out in this doc string, as the
> > referenced variable is an internal one.
> 
> You are right (same mistake as before), the documentation for package
> specifications should be moved from the internal variable to either
> 'package-vc-selected-packages' or some other place.  It might not be bad
> to document it in package.texi either?  Or would that be too Lisp-y for
> the Emacs manual?

It's okay to have this in package.texi, but perhaps with fewer
details.  Users should customize such complex variables via Custom
anyway.

> >> This user option differs from `package-selected-packages' in that
> >> it is meant to be specified manually.  You can also use the
> >> function `package-vc-selected-packages' to apply the changes."
> >
> > The function package-vc-selected-packages mentioned in the last
> > sentence doesn't seem to exist.  Also, what does it mean "to apply the
> > changes" -- apply the changes to what?
> 
> It was supposed to be `package-vc-ensure-packages'.  How about
> rephrasing that last sentence into "By calling the function
> `package-vc-selected-packages", the packages specified in this list can
> also be installed.

Just avoid the passive voice:

  If you want to also install the packages in this list, use
  `package-vc-ensure-packages'.

> >> (defvar package-vc--archive-spec-alist nil
> >>   "List of package specifications for each archive.
> >> The list maps each package name, as a string, to a plist.
> >> Valid keys include
> >> 
> >>         `:url' (string)
> >
> > The "Valid keys" part is "out of the blue" here.  Keys of what?  If
> > that's a reference to the "plist" part preceding it, then we aren't
> > talking about a plist, we are talking about keyword/value pairs,
> > right?
> 
> Yes, these are plist keys.  Should we say something like "Valid keys for
> these plists include:".

I would say explicitly "plist of key/value pairs".

> >>   "Extract the commit of a development package PKG."
> >
> > This doc string doesn't seem to match what the function does.
> 
> I was confused at first, as I was assuming you were talking about
> `package-vc-commit', but it appears I accidentally copied the docstring
> into `package-vc--version'.  A better documentation string here would be
> 
>      "Return the version number for the source package PKG."

That's fine.

> >> (defun package-vc--build-documentation (pkg-desc file)
> >>   "Build documentation FILE for PKG-DESC."
> >>   (let ((pkg-dir (package-desc-dir pkg-desc)))
> >>     (when (string-match-p "\\.org\\'" file)
> >>       (require 'ox)
> >>       (require 'ox-texinfo)
> >>       (with-temp-buffer
> >>         (insert-file-contents file)
> >>         (setq file (make-temp-file "ox-texinfo-"))
> >>         (org-export-to-file 'texinfo file)))
> >>     (call-process "install-info" nil nil nil
> >>                   file pkg-dir)))
> >
> > I'm confused by this function.  Does org-export-to-file produce a
> > .texi file or an Info file?  In any case, the semantics is confusing:
> > if FILE has the .org extension, the function installs a file whose
> > name is not FILE, otherwise the function installs FILE.  This seems to
> > conflate source and destination files in a confusing way.
> 
> I can expand the documentation here.
> 
> "file" is the input, either a .texi or an .org file.

If FILE is input, then saying "Build documentation FILE" is
misleading: there's no need to build FILE, it already exists.  You
should say something like

  Build documentation for package PKG-DESC from documentation source in FILE.

> If it is an .org file, we want to convert it to .texi first (which
> is what the (when ...)  block does), and write the output into a
> temporary file.  The temporary file is used as an input to
> install-info, that builds the .info file.

Ah, so there's a problem here, I think.  install-info doesn't produce
Info files, it only installs existing Info files and updates the DIR
files with the entries of the installed Info files.  So the step of
producing Info files from Texinfo is missing here.

> >> (defun package-vc-update (pkg-desc)
> >>   "Attempt to update the package PKG-DESC."
> >>   ;; HACK: To run `package-vc--unpack-1' after checking out the new
> >>   ;; revision, we insert a hook into `vc-post-command-functions', and
> >>   ;; remove it right after it ran.  To avoid running the hook multiple
> >>   ;; times or even for the wrong repository (as `vc-pull' is often
> >>   ;; asynchronous), we extract the relevant arguments using a pseudo
> >>   ;; filter for `vc-filter-command-function', executed only for the
> >>   ;; side effect, and store them in the lexical scope.  When the hook
> >>   ;; is run, we check if the arguments are the same (`eq') as the ones
> >>   ;; previously extracted, and only in that case will be call
> >>   ;; `package-vc--unpack-1'.  Ugh...
> >
> > Shouldn't this use unwind-protect, to make sure the hacked
> > post-command-hook doesn't leak out?
> 
> The issue is that vc-pull *can be* asynchronous, so wrapping it in a
> `unwind-protect' would un-modify the hook too soon.  But you are right
> that if vc-pull were to fail, the hook would remain inside of
> `vc-post-command-functions' for too long.
> 
> As I said in that comment
> 
>    "If there is a better way to do this, it should be done."

If you can use unwind-protect at least in the synchronous case, it
would be better, I think.

And if the async pull could take a long time, I think it's a problem
to have the additional entry in the post-command-hook for all that
time.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sat, 05 Nov 2022 16:43:45 +0000
>> 
>> >> (defcustom package-vc-repository-store
>> >>   (expand-file-name "emacs/vc-packages" (xdg-data-home))
>> >>   "Directory used by `package-vc--unpack' to store repositories."
>> >
>> > The doc string of a user option should not reference an internal
>> > function, it should reference user-visible features.  Would it be
>> > correct to say that this directory is used by package-vc to store
>> > repositories of packages it fetches and/or installs?
>> 
>> You are right, I forgot about this when renaming package-vc-unpack to
>> package-vc--unpack.  How about
>> 
>>   "Directory used to store clone repositories.  
>>                                                 
>> This directory is only used for packages with a special `:lisp-dir'
>> entry in their package specification (for details on package
>> specifications see `package-vc-selected-packages').  Repositories for
>> packages like these are cloned into the directory specified here, and
>> their `:lisp-dir' is then linked back into the user elpa directory."
>
> Instead of "specified here", please use "specified by this variable".
> And I don't think I understand what the "their `:lisp-dir' is then
> linked back into the user elpa directory" part wants to say.



> I also have a more general question: you say this "is only used for
> packages with a special `:lisp-dir' entry", and that begs the
> question: where will we clone a package that doesn't have the
> :lisp-dir entry?

Directly into the elpa directory.

>> >> (defcustom package-vc-selected-packages '()
>> >>   "List of packages that must be installed.
>> >> Each member of the list is of the form (NAME . SPEC), where NAME
>> >> is a symbol designating the package and SPEC is one of:
>> >> 
>> >> - nil, if any package version can be installed;
>> >> - a version string, if that specific revision is to be installed;
>> >> - a property list of the form described in
>> >>   `package-vc-archive-spec-alist', giving a package
>> >>   specification.
>> >
>> > There's no variable package-vc-archive-spec-alist.  Did you mean
>> > package-vc--archive-spec-alist instead?  In that case, I think the
>> > format of that list should be spelled out in this doc string, as the
>> > referenced variable is an internal one.
>> 
>> You are right (same mistake as before), the documentation for package
>> specifications should be moved from the internal variable to either
>> 'package-vc-selected-packages' or some other place.  It might not be bad
>> to document it in package.texi either?  Or would that be too Lisp-y for
>> the Emacs manual?
>
> It's okay to have this in package.texi, but perhaps with fewer
> details.  Users should customize such complex variables via Custom
> anyway.

OK, I will think about this.

>> >> This user option differs from `package-selected-packages' in that
>> >> it is meant to be specified manually.  You can also use the
>> >> function `package-vc-selected-packages' to apply the changes."
>> >
>> > The function package-vc-selected-packages mentioned in the last
>> > sentence doesn't seem to exist.  Also, what does it mean "to apply the
>> > changes" -- apply the changes to what?
>> 
>> It was supposed to be `package-vc-ensure-packages'.  How about
>> rephrasing that last sentence into "By calling the function
>> `package-vc-selected-packages", the packages specified in this list can
>> also be installed.
>
> Just avoid the passive voice:
>
>   If you want to also install the packages in this list, use
>   `package-vc-ensure-packages'.

Done.


>> >> (defun package-vc--build-documentation (pkg-desc file)
>> >>   "Build documentation FILE for PKG-DESC."
>> >>   (let ((pkg-dir (package-desc-dir pkg-desc)))
>> >>     (when (string-match-p "\\.org\\'" file)
>> >>       (require 'ox)
>> >>       (require 'ox-texinfo)
>> >>       (with-temp-buffer
>> >>         (insert-file-contents file)
>> >>         (setq file (make-temp-file "ox-texinfo-"))
>> >>         (org-export-to-file 'texinfo file)))
>> >>     (call-process "install-info" nil nil nil
>> >>                   file pkg-dir)))
>> >
>> > I'm confused by this function.  Does org-export-to-file produce a
>> > .texi file or an Info file?  In any case, the semantics is confusing:
>> > if FILE has the .org extension, the function installs a file whose
>> > name is not FILE, otherwise the function installs FILE.  This seems to
>> > conflate source and destination files in a confusing way.
>> 
>> I can expand the documentation here.
>> 
>> "file" is the input, either a .texi or an .org file.
>
> If FILE is input, then saying "Build documentation FILE" is
> misleading: there's no need to build FILE, it already exists.  You
> should say something like
>
>   Build documentation for package PKG-DESC from documentation source in FILE.

Right, done.

>> If it is an .org file, we want to convert it to .texi first (which
>> is what the (when ...)  block does), and write the output into a
>> temporary file.  The temporary file is used as an input to
>> install-info, that builds the .info file.
>
> Ah, so there's a problem here, I think.  install-info doesn't produce
> Info files, it only installs existing Info files and updates the DIR
> files with the entries of the installed Info files.  So the step of
> producing Info files from Texinfo is missing here.

Oh yes, I have to add a makeinfo call somewhere inbetween.  The code was
based on elpa-admin.el, and it seems I misread it.

>> >> (defun package-vc-update (pkg-desc)
>> >>   "Attempt to update the package PKG-DESC."
>> >>   ;; HACK: To run `package-vc--unpack-1' after checking out the new
>> >>   ;; revision, we insert a hook into `vc-post-command-functions', and
>> >>   ;; remove it right after it ran.  To avoid running the hook multiple
>> >>   ;; times or even for the wrong repository (as `vc-pull' is often
>> >>   ;; asynchronous), we extract the relevant arguments using a pseudo
>> >>   ;; filter for `vc-filter-command-function', executed only for the
>> >>   ;; side effect, and store them in the lexical scope.  When the hook
>> >>   ;; is run, we check if the arguments are the same (`eq') as the ones
>> >>   ;; previously extracted, and only in that case will be call
>> >>   ;; `package-vc--unpack-1'.  Ugh...
>> >
>> > Shouldn't this use unwind-protect, to make sure the hacked
>> > post-command-hook doesn't leak out?
>> 
>> The issue is that vc-pull *can be* asynchronous, so wrapping it in a
>> `unwind-protect' would un-modify the hook too soon.  But you are right
>> that if vc-pull were to fail, the hook would remain inside of
>> `vc-post-command-functions' for too long.
>> 
>> As I said in that comment
>> 
>>    "If there is a better way to do this, it should be done."
>
> If you can use unwind-protect at least in the synchronous case, it
> would be better, I think.

The issue is that vc-pull doesn't indicate if the command was run
asynchronously or not.

> And if the async pull could take a long time, I think it's a problem
> to have the additional entry in the post-command-hook for all that
> time.

That is why the function tries to identify who invoked the hook and only
run it (once) in that case.

For this to be done properly the 'pull' method for VC would probably
have to be changed/replaced to make programmatic invocations more
predictable.

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 11:43:15 +0000
> 
> >>   "Directory used to store clone repositories.  
> >>                                                 
> >> This directory is only used for packages with a special `:lisp-dir'
> >> entry in their package specification (for details on package
> >> specifications see `package-vc-selected-packages').  Repositories for
> >> packages like these are cloned into the directory specified here, and
> >> their `:lisp-dir' is then linked back into the user elpa directory."
> >
> > Instead of "specified here", please use "specified by this variable".
> > And I don't think I understand what the "their `:lisp-dir' is then
> > linked back into the user elpa directory" part wants to say.
> 
> 
> 
> > I also have a more general question: you say this "is only used for
> > packages with a special `:lisp-dir' entry", and that begs the
> > question: where will we clone a package that doesn't have the
> > :lisp-dir entry?
> 
> Directly into the elpa directory.

And the "linked back" means a symlink? or something else?

Anyway, I think all of this should be explained in the doc string.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sun, 06 Nov 2022 11:43:15 +0000
>> 
>> >>   "Directory used to store clone repositories.  
>> >>                                                 
>> >> This directory is only used for packages with a special `:lisp-dir'
>> >> entry in their package specification (for details on package
>> >> specifications see `package-vc-selected-packages').  Repositories for
>> >> packages like these are cloned into the directory specified here, and
>> >> their `:lisp-dir' is then linked back into the user elpa directory."
>> >
>> > Instead of "specified here", please use "specified by this variable".
>> > And I don't think I understand what the "their `:lisp-dir' is then
>> > linked back into the user elpa directory" part wants to say.
>> 
>> 
>> 
>> > I also have a more general question: you say this "is only used for
>> > packages with a special `:lisp-dir' entry", and that begs the
>> > question: where will we clone a package that doesn't have the
>> > :lisp-dir entry?
>> 
>> Directly into the elpa directory.
>
> And the "linked back" means a symlink? or something else?

Right.  If the package specification has a custom lisp directory, clone
the directory elsewhere and will then symlink [actual repo] + [lisp dir]
into ~/.config/emacs/elpa (or whatever directory is used).

> Anyway, I think all of this should be explained in the doc string.

Will do.

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 15:28:59 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Philip Kaludercic <philipk@posteo.net>
> >> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> >> Date: Sun, 06 Nov 2022 11:43:15 +0000
> >> 
> >> >>   "Directory used to store clone repositories.  
> >> >>                                                 
> >> >> This directory is only used for packages with a special `:lisp-dir'
> >> >> entry in their package specification (for details on package
> >> >> specifications see `package-vc-selected-packages').  Repositories for
> >> >> packages like these are cloned into the directory specified here, and
> >> >> their `:lisp-dir' is then linked back into the user elpa directory."
> >> >
> >> > Instead of "specified here", please use "specified by this variable".
> >> > And I don't think I understand what the "their `:lisp-dir' is then
> >> > linked back into the user elpa directory" part wants to say.
> >> 
> >> 
> >> 
> >> > I also have a more general question: you say this "is only used for
> >> > packages with a special `:lisp-dir' entry", and that begs the
> >> > question: where will we clone a package that doesn't have the
> >> > :lisp-dir entry?
> >> 
> >> Directly into the elpa directory.
> >
> > And the "linked back" means a symlink? or something else?
> 
> Right.  If the package specification has a custom lisp directory, clone
> the directory elsewhere and will then symlink [actual repo] + [lisp dir]
> into ~/.config/emacs/elpa (or whatever directory is used).
> 
> > Anyway, I think all of this should be explained in the doc string.
> 
> Will do.

And symlinks could be problematic on MS-Windows.  Maybe allow to copy
instead.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> > I also have a more general question: you say this "is only used
>> >> > for
>> >> > packages with a special `:lisp-dir' entry", and that begs the
>> >> > question: where will we clone a package that doesn't have the
>> >> > :lisp-dir entry?
>> >> 
>> >> Directly into the elpa directory.
>> >
>> > And the "linked back" means a symlink? or something else?
>> 
>> Right.  If the package specification has a custom lisp directory, clone
>> the directory elsewhere and will then symlink [actual repo] + [lisp
>> dir]
>> into ~/.config/emacs/elpa (or whatever directory is used).
>> 
>> > Anyway, I think all of this should be explained in the doc string.
>> 
>> Will do.
>
> And symlinks could be problematic on MS-Windows.  Maybe allow to copy
> instead.

That I did not know.  The issue with copying is that you loose vc
information.

When do symlinks create issues under windows?

Perhaps there is a better way to solve this issue in general?  I would
have to look at how complicated it would be to make package.el load a
subdirectory instead of the immediate directory under "elpa".

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 15:58:11 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And symlinks could be problematic on MS-Windows.  Maybe allow to copy
> > instead.
> 
> That I did not know.  The issue with copying is that you loose vc
> information.

Why do they need VC information inside ~/.emacs.d/ ?

> When do symlinks create issues under windows?

Creating symlinks requires privileges that not every user has.

> Perhaps there is a better way to solve this issue in general?  I would
> have to look at how complicated it would be to make package.el load a
> subdirectory instead of the immediate directory under "elpa".

Yes, that'd be more portable, I think.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sun, 06 Nov 2022 15:58:11 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > And symlinks could be problematic on MS-Windows.  Maybe allow to copy
>> > instead.
>> 
>> That I did not know.  The issue with copying is that you loose vc
>> information.
>
> Why do they need VC information inside ~/.emacs.d/ ?

While not strictly necessary, a big part of the motivation of package-vc
is the ability to easily work on packages that are usable as regular
packages.  If I M-x find-function into a ~/.emacs.d/elpa/ and make a few
changes, having to distinguish between packages between packages with
:lisp-dir attributes and those without seems unnatural.

>> When do symlinks create issues under windows?
>
> Creating symlinks requires privileges that not every user has.

Oh, I see.

>> Perhaps there is a better way to solve this issue in general?  I would
>> have to look at how complicated it would be to make package.el load a
>> subdirectory instead of the immediate directory under "elpa".
>
> Yes, that'd be more portable, I think.

It looks like it is doable.  There might be a few tricky edge-cases, but
nothing impossible.

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 16:42:37 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> That I did not know.  The issue with copying is that you loose vc
> >> information.
> >
> > Why do they need VC information inside ~/.emacs.d/ ?
> 
> While not strictly necessary, a big part of the motivation of package-vc
> is the ability to easily work on packages that are usable as regular
> packages.  If I M-x find-function into a ~/.emacs.d/elpa/ and make a few
> changes, having to distinguish between packages between packages with
> :lisp-dir attributes and those without seems unnatural.

Sorry, I don't follow: what does find-function etc. have to do with
having the VCS data inside ~/.emacs.d/elpa/?  By "vc information" you
meant what Git stores in the .git directory, yes?  I'm asking why it's
important to have that in ~/.emacs.d/elpa/.

Or maybe I don't understand what you mean by "work on packages that
are usable as regular packages"?

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sun, 06 Nov 2022 16:42:37 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> That I did not know.  The issue with copying is that you loose vc
>> >> information.
>> >
>> > Why do they need VC information inside ~/.emacs.d/ ?
>> 
>> While not strictly necessary, a big part of the motivation of package-vc
>> is the ability to easily work on packages that are usable as regular
>> packages.  If I M-x find-function into a ~/.emacs.d/elpa/ and make a few
>> changes, having to distinguish between packages between packages with
>> :lisp-dir attributes and those without seems unnatural.
>
> Sorry, I don't follow: what does find-function etc. have to do with
> having the VCS data inside ~/.emacs.d/elpa/?  By "vc information" you
> meant what Git stores in the .git directory, yes?  I'm asking why it's
> important to have that in ~/.emacs.d/elpa/.
>
> Or maybe I don't understand what you mean by "work on packages that
> are usable as regular packages"?

Let's say I notice a something I would want to change/add/fix in a
package I am using.  find-function is just one way I would query Emacs
to open up the source, then make a few changes.  If I decide that these
are worth up-streaming, it is nice to just commit them right away and
call `package-vc-prepare-patch' to send the maintainer an email.  That
last part won't work if the files were copied, because in that case I'll
have to copy my changes back to the actual repository, create a commit
in there, then send out the patch, which (right now) couldn't make
immediate use of the package metadata, so you'd have to fall back to
`vc-prepare-patch' and find out who the maintainer is manually.

All of this would only apply to packages with external `:lisp-dir's,
which doesn't immediately interest a user/developer.  Having to keep
this in mind would pointlessly expose an internal detail of package-vc
that I'd like to avoid.

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 17:31:22 +0000
> 
> Let's say I notice a something I would want to change/add/fix in a
> package I am using.  find-function is just one way I would query Emacs
> to open up the source, then make a few changes.  If I decide that these
> are worth up-streaming, it is nice to just commit them right away and
> call `package-vc-prepare-patch' to send the maintainer an email.

How many package users do that?

And if you think many do, why not clone the repository directly into
~/.emacs.d/elpa/?

> All of this would only apply to packages with external `:lisp-dir's,
> which doesn't immediately interest a user/developer.  Having to keep
> this in mind would pointlessly expose an internal detail of package-vc
> that I'd like to avoid.

But it is us who introduced and support :lisp-dir.  If we think it's a
leaky abstraction, we could decide not to support it.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sun, 06 Nov 2022 17:31:22 +0000
>> 
>> Let's say I notice a something I would want to change/add/fix in a
>> package I am using.  find-function is just one way I would query Emacs
>> to open up the source, then make a few changes.  If I decide that these
>> are worth up-streaming, it is nice to just commit them right away and
>> call `package-vc-prepare-patch' to send the maintainer an email.
>
> How many package users do that?

I know of at least one for certain, and that is me.

But I have had discussions with other users from time to time where the
same thing comes up.  There has also been interest in alternative
package managers that are principally incompatible with package.el
(e.g. elpaca, straight) that advertise the fact that they allow users to
easily hack on packages.

> And if you think many do, why not clone the repository directly into
> ~/.emacs.d/elpa/?

Because that won't take care of scraping for autoloads, byte
compilation and installing missing dependencies.

>> All of this would only apply to packages with external `:lisp-dir's,
>> which doesn't immediately interest a user/developer.  Having to keep
>> this in mind would pointlessly expose an internal detail of package-vc
>> that I'd like to avoid.
>
> But it is us who introduced and support :lisp-dir.  If we think it's a
> leaky abstraction, we could decide not to support it.

You mean as in only allowing for packages to distribute lisp code in the
root directory of the repository?  That would pointlessly break too many
packages that decide to structure their file hierarchy for whatever
reason.

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 18:35:10 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And if you think many do, why not clone the repository directly into
> > ~/.emacs.d/elpa/?
> 
> Because that won't take care of scraping for autoloads, byte
> compilation and installing missing dependencies.

I don't see why.  Please elaborate how having the repository inside
~/.emacs.d gets in the way of these activities.

> >> All of this would only apply to packages with external `:lisp-dir's,
> >> which doesn't immediately interest a user/developer.  Having to keep
> >> this in mind would pointlessly expose an internal detail of package-vc
> >> that I'd like to avoid.
> >
> > But it is us who introduced and support :lisp-dir.  If we think it's a
> > leaky abstraction, we could decide not to support it.
> 
> You mean as in only allowing for packages to distribute lisp code in the
> root directory of the repository?  That would pointlessly break too many
> packages that decide to structure their file hierarchy for whatever
> reason.

Is that what :lisp-dir is about? then the doc strings in package-vc.el
doesn't even hint about that.  In particular, there's nothing there
about the root directory of the repository.  (Not that I understand
why having Lisp files in a subdirectory of the repository would be a
problem that needs an explicit configuration of the package, probably
missing something else again.)

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sun, 06 Nov 2022 18:35:10 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > And if you think many do, why not clone the repository directly into
>> > ~/.emacs.d/elpa/?
>> 
>> Because that won't take care of scraping for autoloads, byte
>> compilation and installing missing dependencies.
>
> I don't see why.  Please elaborate how having the repository inside
> ~/.emacs.d gets in the way of these activities.

It doesn't get in the way, the issue just is that if you were to just
clone a package right into .emacs.d, you would still have to do all
these steps individually and manually.

>> >> All of this would only apply to packages with external `:lisp-dir's,
>> >> which doesn't immediately interest a user/developer.  Having to keep
>> >> this in mind would pointlessly expose an internal detail of package-vc
>> >> that I'd like to avoid.
>> >
>> > But it is us who introduced and support :lisp-dir.  If we think it's a
>> > leaky abstraction, we could decide not to support it.
>> 
>> You mean as in only allowing for packages to distribute lisp code in the
>> root directory of the repository?  That would pointlessly break too many
>> packages that decide to structure their file hierarchy for whatever
>> reason.
>
> Is that what :lisp-dir is about? then the doc strings in package-vc.el
> doesn't even hint about that.  In particular, there's nothing there
> about the root directory of the repository.  

The docstring for `package-vc--archive-spec-alist' has the following
(I'm still looking for a better way to document this):

 `:lisp-dir' (string)
    The repository-relative name of the directory to use for loading the Lisp
    sources.  If not given, the value defaults to the root directory
    of the repository.

One example where this is being used is for org-mode:
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/.

>                                              (Not that I understand
> why having Lisp files in a subdirectory of the repository would be a
> problem that needs an explicit configuration of the package, probably
> missing something else again.)

Maybe I am mistaken, but having a directory in `load-path' doesn't mean
all sub-directories are automatically indexed, right?

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 07 Nov 2022 08:42:18 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Philip Kaludercic <philipk@posteo.net>
> >> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> >> Date: Sun, 06 Nov 2022 18:35:10 +0000
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > And if you think many do, why not clone the repository directly into
> >> > ~/.emacs.d/elpa/?
> >> 
> >> Because that won't take care of scraping for autoloads, byte
> >> compilation and installing missing dependencies.
> >
> > I don't see why.  Please elaborate how having the repository inside
> > ~/.emacs.d gets in the way of these activities.
> 
> It doesn't get in the way, the issue just is that if you were to just
> clone a package right into .emacs.d, you would still have to do all
> these steps individually and manually.

Which steps are those, and why do we have to do it manually?

> >> You mean as in only allowing for packages to distribute lisp code in the
> >> root directory of the repository?  That would pointlessly break too many
> >> packages that decide to structure their file hierarchy for whatever
> >> reason.
> >
> > Is that what :lisp-dir is about? then the doc strings in package-vc.el
> > doesn't even hint about that.  In particular, there's nothing there
> > about the root directory of the repository.  
> 
> The docstring for `package-vc--archive-spec-alist' has the following
> (I'm still looking for a better way to document this):
> 
>  `:lisp-dir' (string)
>     The repository-relative name of the directory to use for loading the Lisp
>     sources.  If not given, the value defaults to the root directory
>     of the repository.

Ah, it's a misunderstanding.  See below.

> >                                              (Not that I understand
> > why having Lisp files in a subdirectory of the repository would be a
> > problem that needs an explicit configuration of the package, probably
> > missing something else again.)
> 
> Maybe I am mistaken, but having a directory in `load-path' doesn't mean
> all sub-directories are automatically indexed, right?

We have a standard solution for that:
normal-top-level-add-subdirs-to-load-path.  We install in any
directory that needs this a file called subdirs.el with the following
contents:

  (if (fboundp 'normal-top-level-add-subdirs-to-load-path)
      (normal-top-level-add-subdirs-to-load-path))

Example of directories which need this is the site-lisp directory.

Why cannot we do something like this in this case?

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Mon, 07 Nov 2022 08:42:18 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Philip Kaludercic <philipk@posteo.net>
>> >> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> >> Date: Sun, 06 Nov 2022 18:35:10 +0000
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> > And if you think many do, why not clone the repository directly into
>> >> > ~/.emacs.d/elpa/?
>> >> 
>> >> Because that won't take care of scraping for autoloads, byte
>> >> compilation and installing missing dependencies.
>> >
>> > I don't see why.  Please elaborate how having the repository inside
>> > ~/.emacs.d gets in the way of these activities.
>> 
>> It doesn't get in the way, the issue just is that if you were to just
>> clone a package right into .emacs.d, you would still have to do all
>> these steps individually and manually.
>
> Which steps are those, and why do we have to do it manually?

Let us assume `default-directory' is (locate-user-emacs-file "elpa").
If I run "M-! git clone https://some.git.host.com/path/to/repo/foo.git",
then I'll just have a directory called "foo", right?  If I want to byte
compile the files I'd e.g. have to open foo in Dired, mark all Emacs
Lisp files, byte compile them, then run something like
`make-directory-autoloads' myself.  Then I'd have to find the main file,
check the dependency list and run M-x package-install on every one that
is missing, again one-by-one.

You don't have to do this for `package-install', because it invokes
`package-unpack' that takes care of those details.  As
package-vc-install doesn't use prepared tarballs, the equivalent process
is a bit different (thus we have `package-vc-unpack'), but the intention
is the same.  Bundle all the repetitive task into a single command.

>> >> You mean as in only allowing for packages to distribute lisp code in the
>> >> root directory of the repository?  That would pointlessly break too many
>> >> packages that decide to structure their file hierarchy for whatever
>> >> reason.
>> >
>> > Is that what :lisp-dir is about? then the doc strings in package-vc.el
>> > doesn't even hint about that.  In particular, there's nothing there
>> > about the root directory of the repository.  
>> 
>> The docstring for `package-vc--archive-spec-alist' has the following
>> (I'm still looking for a better way to document this):
>> 
>>  `:lisp-dir' (string)
>>     The repository-relative name of the directory to use for loading the Lisp
>>     sources.  If not given, the value defaults to the root directory
>>     of the repository.
>
> Ah, it's a misunderstanding.  See below.
>
>> >                                              (Not that I understand
>> > why having Lisp files in a subdirectory of the repository would be a
>> > problem that needs an explicit configuration of the package, probably
>> > missing something else again.)
>> 
>> Maybe I am mistaken, but having a directory in `load-path' doesn't mean
>> all sub-directories are automatically indexed, right?
>
> We have a standard solution for that:
> normal-top-level-add-subdirs-to-load-path.  We install in any
> directory that needs this a file called subdirs.el with the following
> contents:
>
>   (if (fboundp 'normal-top-level-add-subdirs-to-load-path)
>       (normal-top-level-add-subdirs-to-load-path))
>
> Example of directories which need this is the site-lisp directory.

I was not familiar with this function.

> Why cannot we do something like this in this case?

I would have to try this out, but my worry is that in some cases this
could add too many non-lisp directories.

It still seems more elegant to encode what the lisp directory is in the
package description.

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 07 Nov 2022 16:58:51 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> >> > And if you think many do, why not clone the repository directly into
> >> >> > ~/.emacs.d/elpa/?
> >> >> 
> >> >> Because that won't take care of scraping for autoloads, byte
> >> >> compilation and installing missing dependencies.
> >> >
> >> > I don't see why.  Please elaborate how having the repository inside
> >> > ~/.emacs.d gets in the way of these activities.
> >> 
> >> It doesn't get in the way, the issue just is that if you were to just
> >> clone a package right into .emacs.d, you would still have to do all
> >> these steps individually and manually.
> >
> > Which steps are those, and why do we have to do it manually?
> 
> Let us assume `default-directory' is (locate-user-emacs-file "elpa").
> If I run "M-! git clone https://some.git.host.com/path/to/repo/foo.git",
> then I'll just have a directory called "foo", right?  If I want to byte
> compile the files I'd e.g. have to open foo in Dired, mark all Emacs
> Lisp files, byte compile them, then run something like
> `make-directory-autoloads' myself.  Then I'd have to find the main file,
> check the dependency list and run M-x package-install on every one that
> is missing, again one-by-one.
> 
> You don't have to do this for `package-install', because it invokes
> `package-unpack' that takes care of those details.  As
> package-vc-install doesn't use prepared tarballs, the equivalent process
> is a bit different (thus we have `package-vc-unpack'), but the intention
> is the same.  Bundle all the repetitive task into a single command.

OK, but my question was why all of this gets magically done when you
clone the repository outside ~/.emacs.d/elpa, but does not get done
when you clone it inside?  I thought this is what you were alluding to
when I asked why not clone into ~/.emacs.d.

> >   (if (fboundp 'normal-top-level-add-subdirs-to-load-path)
> >       (normal-top-level-add-subdirs-to-load-path))
> >
> > Example of directories which need this is the site-lisp directory.
> 
> I was not familiar with this function.
> 
> > Why cannot we do something like this in this case?
> 
> I would have to try this out, but my worry is that in some cases this
> could add too many non-lisp directories.

The function can be easily extended to only add directories in which
we have *.el files.

> It still seems more elegant to encode what the lisp directory is in the
> package description.

You consider manual configuration of a package to be more elegant than
automatically finding the directories to put on load-path?

Re: feature/package-vc has been merged

[Stefan Monnier]

>> It still seems more elegant to encode what the lisp directory is in the
>> package description.
>
> You consider manual configuration of a package to be more elegant than
> automatically finding the directories to put on load-path?

Note that we want a single description per package, which can be shared
by (Non)GNU ELPA and `package-vc`, so this imposes
additional constraints.

Also automation is nice, but it has a cost (e.g. many packages have
ELisp files in both a `lisp/` subdirectory and a `test/` subdirectory:
which one should we pick?), and since packages where ELisp files are in
subdirectories are a fairly small fraction, I decided for (Non)GNU ELPA
that it was cleaner to use a manual config than to try and be
(too) clever.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Which steps are those, and why do we have to do it manually?
>> 
>> Let us assume `default-directory' is (locate-user-emacs-file "elpa").
>> If I run "M-! git clone https://some.git.host.com/path/to/repo/foo.git",
>> then I'll just have a directory called "foo", right?  If I want to byte
>> compile the files I'd e.g. have to open foo in Dired, mark all Emacs
>> Lisp files, byte compile them, then run something like
>> `make-directory-autoloads' myself.  Then I'd have to find the main file,
>> check the dependency list and run M-x package-install on every one that
>> is missing, again one-by-one.
>> 
>> You don't have to do this for `package-install', because it invokes
>> `package-unpack' that takes care of those details.  As
>> package-vc-install doesn't use prepared tarballs, the equivalent process
>> is a bit different (thus we have `package-vc-unpack'), but the intention
>> is the same.  Bundle all the repetitive task into a single command.
>
> OK, but my question was why all of this gets magically done when you
> clone the repository outside ~/.emacs.d/elpa, but does not get done
> when you clone it inside?  I thought this is what you were alluding to
> when I asked why not clone into ~/.emacs.d.

Oh no, I'm sorry if I implied this somehow.  This is neither the case in
or outside of ~/.emacs.d/elpa.  Somehow, someone has to do all of this.
And one someone is package-vc.

>> >   (if (fboundp 'normal-top-level-add-subdirs-to-load-path)
>> >       (normal-top-level-add-subdirs-to-load-path))
>> >
>> > Example of directories which need this is the site-lisp directory.
>> 
>> I was not familiar with this function.
>> 
>> > Why cannot we do something like this in this case?
>> 
>> I would have to try this out, but my worry is that in some cases this
>> could add too many non-lisp directories.
>
> The function can be easily extended to only add directories in which
> we have *.el files.

This might work well enough most of the time, but I don't think we
always want this.

>> It still seems more elegant to encode what the lisp directory is in the
>> package description.
>
> You consider manual configuration of a package to be more elegant than
> automatically finding the directories to put on load-path?

Eh, I am not sure what we are talking about anymore :/ I think it is
best for me to reiterate.  Both elpa.git and nongnu.git have package
specifications in "elpa-packages", that among other things may contain
properties indicating what directory the lisp files are located in.
elpa-admin.el uses this information to prepare the tarballs for
distribution.

These package specifications are now also published in a way that
package-vc can process them.  For GNU ELPA you can find these here:
https://elpa.gnu.org/packages/elpa-packages.eld.  After cloning a
package repository, package-vc can use the `:lisp-dir' property from a
package specification to figure out what directory is to be added to the
load-path, where the files are that should be byte-compiled,
auto-loaded, etc.

I am proposing we make use of this available information instead of
guessing or making broad assumptions like that all lisp code is of
interest.

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 07 Nov 2022 18:10:04 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > OK, but my question was why all of this gets magically done when you
> > clone the repository outside ~/.emacs.d/elpa, but does not get done
> > when you clone it inside?  I thought this is what you were alluding to
> > when I asked why not clone into ~/.emacs.d.
> 
> Oh no, I'm sorry if I implied this somehow.  This is neither the case in
> or outside of ~/.emacs.d/elpa.  Somehow, someone has to do all of this.
> And one someone is package-vc.

So my proposal is to clone the repositories below ~/.emacs.d/elpa/,
and thus solve the problem of linking/copying of files from the
repository to .emacs.d (and several others, which you and Stefan for
some reason don't consider important enough).

And with that, let me bow out of this, since I think you understand
well enough what I proposed and can make your own decisions.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Mon, 07 Nov 2022 18:10:04 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > OK, but my question was why all of this gets magically done when you
>> > clone the repository outside ~/.emacs.d/elpa, but does not get done
>> > when you clone it inside?  I thought this is what you were alluding to
>> > when I asked why not clone into ~/.emacs.d.
>> 
>> Oh no, I'm sorry if I implied this somehow.  This is neither the case in
>> or outside of ~/.emacs.d/elpa.  Somehow, someone has to do all of this.
>> And one someone is package-vc.
>
> So my proposal is to clone the repositories below ~/.emacs.d/elpa/,
> and thus solve the problem of linking/copying of files from the
> repository to .emacs.d (and several others, which you and Stefan for
> some reason don't consider important enough).
>
> And with that, let me bow out of this, since I think you understand
> well enough what I proposed and can make your own decisions.

I have implemented the necessary changes, and it appears to work.  One
reservation I have is the following change:

--8<---------------cut here---------------start------------->8---
@@ -1090,17 +1096,19 @@ package-generate-autoloads
          (backup-inhibited t)
          (version-control 'never))
     (loaddefs-generate
-     pkg-dir output-file
-     nil
-     "(add-to-list 'load-path (directory-file-name
-                         (or (file-name-directory #$) (car load-path))))")
+     (file-name-concat pkg-dir lisp-dir)
+     output-file nil
+     (prin1-to-string
+      `(add-to-list 'load-path
+                    (directory-file-name
+                     ,(file-name-concat pkg-dir lisp-dir)))))
     (let ((buf (find-buffer-visiting output-file)))
       (when buf (kill-buffer buf)))
     auto-name))
--8<---------------cut here---------------end--------------->8---

Git tells me that Lars was wrote the "(or (file-name-directory #$) (car
load-path))" bit, so I've added him to the CC's: Can you explain what
the intention was here, and if my suggested replacement would break
anything?

Re: feature/package-vc has been merged

[Stefan Monnier]

>      (loaddefs-generate
> -     pkg-dir output-file
> -     nil
> -     "(add-to-list 'load-path (directory-file-name
> -                         (or (file-name-directory #$) (car load-path))))")
> +     (file-name-concat pkg-dir lisp-dir)
> +     output-file nil
> +     (prin1-to-string
> +      `(add-to-list 'load-path
> +                    (directory-file-name
> +                     ,(file-name-concat pkg-dir lisp-dir)))))
>      (let ((buf (find-buffer-visiting output-file)))
>        (when buf (kill-buffer buf)))
>      auto-name))
> --8<---------------cut here---------------end--------------->8---
>
> Git tells me that Lars was wrote the "(or (file-name-directory #$) (car
> load-path))" bit, so I've added him to the CC's: Can you explain what
> the intention was here, and if my suggested replacement would break
> anything?

I suspect he just copy/pasted the code from elsewhere.
The point is to make the file work regardless of where it is whereas
your replacement makes it work only in the location we used to
generate it.

You can/should replace #$ with `load-file-name`, which will make it
behave better w.r.t byte-compilation.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1371 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>      (loaddefs-generate
>> -     pkg-dir output-file
>> -     nil
>> -     "(add-to-list 'load-path (directory-file-name
>> -                         (or (file-name-directory #$) (car load-path))))")
>> +     (file-name-concat pkg-dir lisp-dir)
>> +     output-file nil
>> +     (prin1-to-string
>> +      `(add-to-list 'load-path
>> +                    (directory-file-name
>> +                     ,(file-name-concat pkg-dir lisp-dir)))))
>>      (let ((buf (find-buffer-visiting output-file)))
>>        (when buf (kill-buffer buf)))
>>      auto-name))
>> --8<---------------cut here---------------end--------------->8---
>>
>> Git tells me that Lars was wrote the "(or (file-name-directory #$) (car
>> load-path))" bit, so I've added him to the CC's: Can you explain what
>> the intention was here, and if my suggested replacement would break
>> anything?
>
> I suspect he just copy/pasted the code from elsewhere.
> The point is to make the file work regardless of where it is whereas
> your replacement makes it work only in the location we used to
> generate it.

I see, that makes sense.

> You can/should replace #$ with `load-file-name`, which will make it
> behave better w.r.t byte-compilation.

I suspected so much from lread.c, but wanted to make sure.

So in that case the following should suffice:


[-- Attachment #2: Type: text/plain, Size: 1068 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 0881626e92..67ce30a93c 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1103,8 +1103,17 @@ package-generate-autoloads
      (package-lisp-dir pkg-desc)
      output-file nil
      (prin1-to-string
-      `(add-to-list 'load-path
-                    ,(package-lisp-dir pkg-desc))))
+      `(add-to-list
+        'load-path
+        (file-name-concat
+         ;; Add the directory that will contain the autoload file to
+         ;; the load path.  We don't hard-code `pkg-dir', to avoid
+         ;; issues if the package directory is moved around.
+         (or (and load-file-name (file-name-directory load-file-name))
+             (car load-path))
+         ;; In case the package specification indicates that the lisp
+         ;; files are found in a subdirectory, append that path.
+         ,(alist-get :lisp-dir (package-desc-extras pkg-desc))))))
     (let ((buf (find-buffer-visiting output-file)))
       (when buf (kill-buffer buf)))
     auto-name))

[-- Attachment #3: Type: text/plain, Size: 20 bytes --]


>
>         Stefan

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  emacs-devel@gnu.org,  Lars
>  Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 09 Nov 2022 08:15:52 +0000
> 
> +      `(add-to-list
> +        'load-path
> +        (file-name-concat

Why file-name-concat and not expand-file-name?

> +         ;; In case the package specification indicates that the lisp
> +         ;; files are found in a subdirectory, append that path.
                                                              ^^^^
The entity you are talking about here is not a "path", AFAIU, it's a
directory name.

Thanks.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  emacs-devel@gnu.org,  Lars
>>  Ingebrigtsen <larsi@gnus.org>
>> Date: Wed, 09 Nov 2022 08:15:52 +0000
>> 
>> +      `(add-to-list
>> +        'load-path
>> +        (file-name-concat
>
> Why file-name-concat and not expand-file-name?

Just because (file-name-concat foo nil) ≡ foo, but upon reflection I
think you are right in saying that `expand-file-name' would be cleaner.

>> +         ;; In case the package specification indicates that the lisp
>> +         ;; files are found in a subdirectory, append that path.
>                                                               ^^^^
> The entity you are talking about here is not a "path", AFAIU, it's a
> directory name.

Right, thanks for noticing.  Will fix both.

> Thanks.

Re: feature/package-vc has been merged

[Stefan Monnier]

> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index 0881626e92..67ce30a93c 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -1103,8 +1103,17 @@ package-generate-autoloads
>       (package-lisp-dir pkg-desc)
>       output-file nil
>       (prin1-to-string
> -      `(add-to-list 'load-path
> -                    ,(package-lisp-dir pkg-desc))))
> +      `(add-to-list
> +        'load-path
> +        (file-name-concat
> +         ;; Add the directory that will contain the autoload file to
> +         ;; the load path.  We don't hard-code `pkg-dir', to avoid
> +         ;; issues if the package directory is moved around.
> +         (or (and load-file-name (file-name-directory load-file-name))
> +             (car load-path))
> +         ;; In case the package specification indicates that the lisp
> +         ;; files are found in a subdirectory, append that path.
> +         ,(alist-get :lisp-dir (package-desc-extras pkg-desc))))))
>      (let ((buf (find-buffer-visiting output-file)))
>        (when buf (kill-buffer buf)))
>      auto-name))

I don't understand why we have code relating to `package-vc` in
`package.el`.

Please be careful to try to correctly preserve the *exact* name added to
`load-path` (most importantly whether it ends in a slash or not),
because some other code relies on it (e.g. to try and avoid having both
`/foo/bar` and `/foo/bar/` or to remove entries).


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
>> index 0881626e92..67ce30a93c 100644
>> --- a/lisp/emacs-lisp/package.el
>> +++ b/lisp/emacs-lisp/package.el
>> @@ -1103,8 +1103,17 @@ package-generate-autoloads
>>       (package-lisp-dir pkg-desc)
>>       output-file nil
>>       (prin1-to-string
>> -      `(add-to-list 'load-path
>> -                    ,(package-lisp-dir pkg-desc))))
>> +      `(add-to-list
>> +        'load-path
>> +        (file-name-concat
>> +         ;; Add the directory that will contain the autoload file to
>> +         ;; the load path.  We don't hard-code `pkg-dir', to avoid
>> +         ;; issues if the package directory is moved around.
>> +         (or (and load-file-name (file-name-directory load-file-name))
>> +             (car load-path))
>> +         ;; In case the package specification indicates that the lisp
>> +         ;; files are found in a subdirectory, append that path.
>> +         ,(alist-get :lisp-dir (package-desc-extras pkg-desc))))))
>>      (let ((buf (find-buffer-visiting output-file)))
>>        (when buf (kill-buffer buf)))
>>      auto-name))
>
> I don't understand why we have code relating to `package-vc` in
> `package.el`.

I have made :lisp-dir a general property of a package description.  This
might be set in package-vc when generating the <PKG>-pkg.el, but in
principle any (non-vc) package could make use of this to indicate a
subdirectory where lisp files are stored.  For now this is something
that will probably only interest package-vc.

> Please be careful to try to correctly preserve the *exact* name added to
> `load-path` (most importantly whether it ends in a slash or not),
> because some other code relies on it (e.g. to try and avoid having both
> `/foo/bar` and `/foo/bar/` or to remove entries).

Good point, I'll make sure to check that.

Re: feature/package-vc has been merged

[Stefan Monnier]

> I have made :lisp-dir a general property of a package description.  This
> might be set in package-vc when generating the <PKG>-pkg.el, but in
> principle any (non-vc) package could make use of this to indicate a
> subdirectory where lisp files are stored.  For now this is something
> that will probably only interest package-vc.

Hmm... but if a package does that in an ELPA archive (i.e. the
`:lisp-dir` thingy appears in the `archive-contents`), then this package
will not be installed correctly when installed with an older
Emacs, right?

IOW, it's an extension of the ELPA protocol.

I do think it's getting time to update the protocol based on the
experience gained over the last 10 years, but I'm really not convinced
`:lisp-dir` is a good addition.

E.g. a thing I'd find more interesting would be to let the tarball provide
its own <PKG>-autoloads.el (which is in charge of adding whichever
lisp-dirs it wants).


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I have made :lisp-dir a general property of a package description.  This
>> might be set in package-vc when generating the <PKG>-pkg.el, but in
>> principle any (non-vc) package could make use of this to indicate a
>> subdirectory where lisp files are stored.  For now this is something
>> that will probably only interest package-vc.
>
> Hmm... but if a package does that in an ELPA archive (i.e. the
> `:lisp-dir` thingy appears in the `archive-contents`), then this package
> will not be installed correctly when installed with an older
> Emacs, right?
>
> IOW, it's an extension of the ELPA protocol.

Strictly speaking yes, but without any specification it is hard to say.
All this is intended as is an extra field that is stored in a package
description file.

Do you have any other suggestion that could be implemented to solve the
immediate problem that package-vc has (lisp files are located in some
sub-directory and we want to avoid loading package-vc during
initialisation).

> I do think it's getting time to update the protocol based on the
> experience gained over the last 10 years, but I'm really not convinced
> `:lisp-dir` is a good addition.
>
> E.g. a thing I'd find more interesting would be to let the tarball provide
> its own <PKG>-autoloads.el (which is in charge of adding whichever
> lisp-dirs it wants).

That sounds like a better idea.

Re: feature/package-vc has been merged

[Stefan Monnier]

Philip Kaludercic [2022-11-09 19:04:56] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I have made :lisp-dir a general property of a package description.  This
>>> might be set in package-vc when generating the <PKG>-pkg.el, but in
>>> principle any (non-vc) package could make use of this to indicate a
>>> subdirectory where lisp files are stored.  For now this is something
>>> that will probably only interest package-vc.
>>
>> Hmm... but if a package does that in an ELPA archive (i.e. the
>> `:lisp-dir` thingy appears in the `archive-contents`), then this package
>> will not be installed correctly when installed with an older
>> Emacs, right?
>>
>> IOW, it's an extension of the ELPA protocol.
>
> Strictly speaking yes, but without any specification it is hard to say.
> All this is intended as is an extra field that is stored in a package
> description file.
>
> Do you have any other suggestion that could be implemented to solve the
> immediate problem that package-vc has (lisp files are located in some
> sub-directory and we want to avoid loading package-vc during
> initialisation).

I don't understand the problem you're trying to solve: package-vc
already has access to this `:lisp-dir` info and it can generate the
autoloads file on its own, so I can't see why `package.el` would need to
know about it.

>> I do think it's getting time to update the protocol based on the
>> experience gained over the last 10 years, but I'm really not convinced
>> `:lisp-dir` is a good addition.
>>
>> E.g. a thing I'd find more interesting would be to let the tarball provide
>> its own <PKG>-autoloads.el (which is in charge of adding whichever
>> lisp-dirs it wants).
>
> That sounds like a better idea.

But for `package-vc` that's already the case since it has complete control
over how the `<foo>-pkg.el` and the `<foo>-autoloads.el` are generated.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2022-11-09 19:04:56] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> I have made :lisp-dir a general property of a package description.  This
>>>> might be set in package-vc when generating the <PKG>-pkg.el, but in
>>>> principle any (non-vc) package could make use of this to indicate a
>>>> subdirectory where lisp files are stored.  For now this is something
>>>> that will probably only interest package-vc.
>>>
>>> Hmm... but if a package does that in an ELPA archive (i.e. the
>>> `:lisp-dir` thingy appears in the `archive-contents`), then this package
>>> will not be installed correctly when installed with an older
>>> Emacs, right?
>>>
>>> IOW, it's an extension of the ELPA protocol.
>>
>> Strictly speaking yes, but without any specification it is hard to say.
>> All this is intended as is an extra field that is stored in a package
>> description file.
>>
>> Do you have any other suggestion that could be implemented to solve the
>> immediate problem that package-vc has (lisp files are located in some
>> sub-directory and we want to avoid loading package-vc during
>> initialisation).
>
> I don't understand the problem you're trying to solve: package-vc
> already has access to this `:lisp-dir` info and it can generate the
> autoloads file on its own, so I can't see why `package.el` would need to
> know about it.

You are right, it is not necessary but...

>>> I do think it's getting time to update the protocol based on the
>>> experience gained over the last 10 years, but I'm really not convinced
>>> `:lisp-dir` is a good addition.
>>>
>>> E.g. a thing I'd find more interesting would be to let the tarball provide
>>> its own <PKG>-autoloads.el (which is in charge of adding whichever
>>> lisp-dirs it wants).
>>
>> That sounds like a better idea.
>
> But for `package-vc` that's already the case since it has complete control
> over how the `<foo>-pkg.el` and the `<foo>-autoloads.el` are generated.

..., this is handled by `package-vc--unpack-1' which up until now did
not depend on a package specification, just a package description.  This
is so, so that it can be invoked by `package-vc-refresh' or
`package-vc-install-from-checkout' which all don't necessarily have
specifications.  It would be imaginable to store the specification in a
file like <PKG>-spec.eld, but there are too many files already so this
is just getting more and more messy.

If you believe that this is not worth it or shortsighted on my part,
I'll implement the code necessary for what you suggest to work.

Re: feature/package-vc has been merged

[Stefan Monnier]

>> But for `package-vc` that's already the case since it has complete control
>> over how the `<foo>-pkg.el` and the `<foo>-autoloads.el` are generated.
>
> ..., this is handled by `package-vc--unpack-1' which up until now did
> not depend on a package specification, just a package description.  This
> is so, so that it can be invoked by `package-vc-refresh' or
> `package-vc-install-from-checkout' which all don't necessarily have
> specifications.  It would be imaginable to store the specification in a
> file like <PKG>-spec.eld, but there are too many files already so this
> is just getting more and more messy.
>
> If you believe that this is not worth it or shortsighted on my part,
> I'll implement the code necessary for what you suggest to work.

Ah... so this is extra info stored in the `<pkg>-pkg.el` file but only
for packages installed by `package-vc`.  So it's not expected to appear
in `<pkg>-pkg.el` coming from ELPA repositories.  But then it should not
be used in `package.el`, only in `package-vc-...` functions, right?

BTW, I'm beginning to sense that maybe instead of storing `:lisp-dir` in
the package description (of package-vc-installed-packages), we
should/could store there the whole package-spec.  This way
`package-vc-refresh` can use the package-spec and throw away the rest of
the package description.

For `package-vc-install-from-checkout`, we need to "invent" a package
spec anyway, right?


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> But for `package-vc` that's already the case since it has complete control
>>> over how the `<foo>-pkg.el` and the `<foo>-autoloads.el` are generated.
>>
>> ..., this is handled by `package-vc--unpack-1' which up until now did
>> not depend on a package specification, just a package description.  This
>> is so, so that it can be invoked by `package-vc-refresh' or
>> `package-vc-install-from-checkout' which all don't necessarily have
>> specifications.  It would be imaginable to store the specification in a
>> file like <PKG>-spec.eld, but there are too many files already so this
>> is just getting more and more messy.
>>
>> If you believe that this is not worth it or shortsighted on my part,
>> I'll implement the code necessary for what you suggest to work.
>
> Ah... so this is extra info stored in the `<pkg>-pkg.el` file but only
> for packages installed by `package-vc`.  So it's not expected to appear
> in `<pkg>-pkg.el` coming from ELPA repositories.  

Right.

>                                                   But then it should not
> be used in `package.el`, only in `package-vc-...` functions, right?

I have attached the entire patch below.  There are a few points where we
want to have regular package.el functions also respect :lisp-dir, but
these changes mostly consist of just replacing package-desc-dir with an
auxiliary function that returns the lisp directory.

> BTW, I'm beginning to sense that maybe instead of storing `:lisp-dir` in
> the package description (of package-vc-installed-packages), we
> should/could store there the whole package-spec.  This way
> `package-vc-refresh` can use the package-spec and throw away the rest of
> the package description.

That would also be possible, I'd have to think about it and try it out.

> For `package-vc-install-from-checkout`, we need to "invent" a package
> spec anyway, right?

Well no, because that function uses `package-vc--unpack-1' that is
invoked during the regular installation after all the information from
the specification has been made use of.  But even if that weren't the
case, we'd just use the "empty" specification in this case,  because the
argument to `package-vc-install-from-checkout' is a lisp directory to
begin with.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-specifying-a-lisp-dir-for-package-descriptions.patch --]
[-- Type: text/x-diff, Size: 11518 bytes --]

From 1eeca3607efcc08b3534e9bbdaa57d2bd62839ec Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Tue, 8 Nov 2022 23:45:35 +0100
Subject: [PATCH] Allow specifying a :lisp-dir for package descriptions

* lisp/emacs-lisp/package-vc.el (package-vc-repository-store): Remove
obsolete variable.
(package-vc--unpack-1): Respect :lisp-dir.
(package-vc--unpack): Add :lisp-dir to the package description if
necessary.
* lisp/emacs-lisp/package.el (package-lisp-dir): Add new inline
function.
(package--reload-previously-loaded): Use 'package-lisp-dir'.
(package-activate-1): Use 'package-lisp-dir'.
(package-generate-autoloads): Change first parameter from NAME to
PKG-DESC.
(package--make-autoloads-and-stuff): Use 'package-lisp-dir'.
(package--compile): Use 'package-lisp-dir'.
(package--native-compile-async): Use 'package-lisp-dir'.
(package--delete-directory): Remove 'package-vc-p' check and drop
second parameter.
(package-delete): Remove second argument when invoking
'package--delete-directory'.
(package-recompile): Use 'package-lisp-dir'.
---
 lisp/emacs-lisp/package-vc.el | 34 +++++-------------
 lisp/emacs-lisp/package.el    | 65 ++++++++++++++++++++---------------
 2 files changed, 46 insertions(+), 53 deletions(-)

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index f8948905ea..93a96abb68 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -100,12 +100,6 @@ package-vc-heuristic-alist
                                               vc-handled-backends)))
   :version "29.1")
 
-(defcustom package-vc-repository-store
-  (expand-file-name "emacs/vc-packages" (xdg-data-home))
-  "Directory used by to store repositories."
-  :type 'directory
-  :version "29.1")
-
 (defcustom package-vc-default-backend 'Git
   "Default VC backend used when cloning a package repository.
 If no repository type was specified or could be guessed by
@@ -386,7 +380,7 @@ package-vc--unpack-1
   ;; dependency list wasn't know beforehand, and they might have
   ;; to be installed explicitly.
   (let (deps)
-    (dolist (file (directory-files pkg-dir t "\\.el\\'" t))
+    (dolist (file (directory-files (package-lisp-dir pkg-desc) t "\\.el\\'" t))
       (with-temp-buffer
         (insert-file-contents file)
         (when-let* ((require-lines (lm-header-multiline "package-requires")))
@@ -402,10 +396,9 @@ package-vc--unpack-1
      (package-compute-transaction nil (delete-dups deps))))
 
   (let ((default-directory (file-name-as-directory pkg-dir))
-        (name (package-desc-name pkg-desc))
         (pkg-file (expand-file-name (package--description-file pkg-dir) pkg-dir)))
     ;; Generate autoloads
-    (package-generate-autoloads name pkg-dir)
+    (package-generate-autoloads pkg-desc pkg-dir)
 
     ;; Generate package file
     (package-vc--generate-description-file pkg-desc pkg-file)
@@ -492,28 +485,17 @@ package-vc--unpack
   (pcase-let* (((map :url :lisp-dir) pkg-spec)
                (name (package-desc-name pkg-desc))
                (dirname (package-desc-full-name pkg-desc))
-               (pkg-dir (expand-file-name dirname package-user-dir))
-               (real-dir (if (null lisp-dir)
-                             pkg-dir
-                           (unless (file-exists-p package-vc-repository-store)
-                             (make-directory package-vc-repository-store t))
-                           (file-name-concat
-                            package-vc-repository-store
-                            ;; FIXME: We aren't sure this directory
-                            ;; will be unique, but we can try other
-                            ;; names to avoid an unnecessary error.
-                            (file-name-base url)))))
+               (pkg-dir (expand-file-name dirname package-user-dir)))
     (setf (package-desc-dir pkg-desc) pkg-dir)
     (when (file-exists-p pkg-dir)
       (if (yes-or-no-p "Overwrite previous checkout?")
-          (package--delete-directory pkg-dir pkg-desc)
+          (package--delete-directory pkg-dir)
         (error "There already exists a checkout for %s" name)))
-    (package-vc--clone pkg-desc pkg-spec real-dir rev)
-    (unless (eq pkg-dir real-dir)
-      ;; Link from the right position in `repo-dir' to the package
-      ;; directory in the ELPA store.
-      (make-symbolic-link (file-name-concat real-dir lisp-dir) pkg-dir))
+    (package-vc--clone pkg-desc pkg-spec pkg-dir rev)
 
+    (when lisp-dir
+      (push (cons :lisp-dir lisp-dir)
+            (package-desc-extras pkg-desc)))
     (package-vc--unpack-1 pkg-desc pkg-dir)))
 
 (defun package-vc--read-package-name (prompt &optional allow-url installed)
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index a7bcdd214c..bf6849af65 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -462,6 +462,15 @@ package-vc-p
   (inline-letevals (pkg-desc)
     (inline-quote (eq (package-desc-kind ,pkg-desc) 'vc))))
 
+(define-inline package-lisp-dir (pkg-desc)
+  "Return the directory with Lisp files for PKG-DESC."
+  (inline-letevals (pkg-desc)
+    (inline-quote
+     (let* ((extras (package-desc-extras ,pkg-desc))
+            (lisp-dir (alist-get :lisp-dir extras))
+            (dir (package-desc-dir ,pkg-desc)))
+       (file-name-directory (file-name-concat dir lisp-dir))))))
+
 (cl-defstruct (package-desc
                ;; Rename the default constructor from `make-package-desc'.
                (:constructor package-desc-create)
@@ -827,7 +836,7 @@ package--reload-previously-loaded
 byte-compilation of the new package to fail."
   (with-demoted-errors "Error in package--load-files-for-activation: %s"
     (let* (result
-           (dir (package-desc-dir pkg-desc))
+           (dir (package-lisp-dir pkg-desc))
            ;; A previous implementation would skip `dir' itself.
            ;; However, in normal use reloading from the same directory
            ;; never happens anyway, while in certain cases external to
@@ -891,7 +900,7 @@ package-activate-1
           (package--reload-previously-loaded pkg-desc))
         (with-demoted-errors "Error loading autoloads: %s"
           (load (package--autoloads-file-name pkg-desc) nil t))
-        (add-to-list 'load-path (directory-file-name pkg-dir)))
+        (add-to-list 'load-path (package-lisp-dir pkg-desc)))
       ;; Add info node.
       (when (file-exists-p (expand-file-name "dir" pkg-dir))
         ;; FIXME: not the friendliest, but simple.
@@ -1080,9 +1089,10 @@ package-autoload-ensure-default-file
 (defvar autoload-timestamps)
 (defvar version-control)
 
-(defun package-generate-autoloads (name pkg-dir)
-  "Generate autoloads in PKG-DIR for package named NAME."
-  (let* ((auto-name (format "%s-autoloads.el" name))
+(defun package-generate-autoloads (pkg-desc pkg-dir)
+  "Generate autoloads for PKG-DESC in PKG-DIR."
+  (let* ((name (package-desc-name pkg-desc))
+         (auto-name (format "%s-autoloads.el" name))
          ;;(ignore-name (concat name "-pkg.el"))
          (output-file (expand-file-name auto-name pkg-dir))
          ;; We don't need 'em, and this makes the output reproducible.
@@ -1090,17 +1100,29 @@ package-generate-autoloads
          (backup-inhibited t)
          (version-control 'never))
     (loaddefs-generate
-     pkg-dir output-file
-     nil
-     "(add-to-list 'load-path (directory-file-name
-                         (or (file-name-directory #$) (car load-path))))")
+     (package-lisp-dir pkg-desc)
+     output-file nil
+     (prin1-to-string
+      `(add-to-list
+        'load-path
+        ;; Add the directory that will contain the autoload file to
+        ;; the load path.  We don't hard-code `pkg-dir', to avoid
+        ;; issues if the package directory is moved around.
+        ,(if-let ((base '(or (and load-file-name (file-name-directory load-file-name))
+                             (car load-path)))
+                  (extras (package-desc-extras pkg-desc))
+                  (lisp-dir (alist-get :lisp-dir extras)))
+             ;; In case the package specification indicates that the lisp
+             ;; files are found in a subdirectory, append that directory.
+             `(expand-file-name ,lisp-dir ,base)
+           base))))
     (let ((buf (find-buffer-visiting output-file)))
       (when buf (kill-buffer buf)))
     auto-name))
 
 (defun package--make-autoloads-and-stuff (pkg-desc pkg-dir)
   "Generate autoloads, description file, etc., for PKG-DESC installed at PKG-DIR."
-  (package-generate-autoloads (package-desc-name pkg-desc) pkg-dir)
+  (package-generate-autoloads pkg-desc pkg-dir)
   (let ((desc-file (expand-file-name (package--description-file pkg-dir)
                                      pkg-dir)))
     (unless (file-exists-p desc-file)
@@ -1118,7 +1140,7 @@ package--compile
   (let ((byte-compile-ignore-files (package--parse-elpaignore pkg-desc))
         (warning-minimum-level :error)
         (load-path load-path))
-    (byte-recompile-directory (package-desc-dir pkg-desc) 0 t)))
+    (byte-recompile-directory (package-lisp-dir pkg-desc) 0 t)))
 
 (defun package--native-compile-async (pkg-desc)
   "Native compile installed package PKG-DESC asynchronously.
@@ -1126,7 +1148,7 @@ package--native-compile-async
 `package-activate-1'."
   (when (native-comp-available-p)
     (let ((warning-minimum-level :error))
-      (native-compile-async (package-desc-dir pkg-desc) t))))
+      (native-compile-async (package-lisp-dir pkg-desc) t))))
 
 ;;;; Inferring package from current buffer
 (defun package-read-from-string (str)
@@ -2419,7 +2441,7 @@ package--newest-p
 
 (declare-function comp-el-to-eln-filename "comp.c")
 (defvar package-vc-repository-store)
-(defun package--delete-directory (dir pkg-desc)
+(defun package--delete-directory (dir)
   "Delete PKG-DESC directory DIR recursively.
 Clean-up the corresponding .eln files if Emacs is native
 compiled."
@@ -2427,18 +2449,7 @@ package--delete-directory
     (cl-loop
      for file in (directory-files-recursively dir "\\.el\\'")
      do (comp-clean-up-stale-eln (comp-el-to-eln-filename file))))
-  (if (and (package-vc-p pkg-desc)
-           (require 'package-vc)   ;load `package-vc-repository-store'
-           (file-in-directory-p dir package-vc-repository-store))
-      (progn
-        (delete-directory
-         (expand-file-name
-          (car (file-name-split
-                (file-relative-name dir package-vc-repository-store)))
-          package-vc-repository-store)
-         t)
-        (delete-file (directory-file-name dir)))
-    (delete-directory dir t)))
+  (delete-directory dir t))
 
 
 (defun package-delete (pkg-desc &optional force nosave)
@@ -2493,7 +2504,7 @@ package-delete
                   (package-desc-name pkg-used-elsewhere-by)))
           (t
            (add-hook 'post-command-hook #'package-menu--post-refresh)
-           (package--delete-directory dir pkg-desc)
+           (package--delete-directory dir)
            ;; Remove NAME-VERSION.signed and NAME-readme.txt files.
            ;;
            ;; NAME-readme.txt files are no longer created, but they
@@ -2549,7 +2560,7 @@ package-recompile
     ;; load them (in case they contain byte code/macros that are now
     ;; invalid).
     (dolist (elc (directory-files-recursively
-                  (package-desc-dir pkg-desc) "\\.elc\\'"))
+                  (package-lisp-dir pkg-desc) "\\.elc\\'"))
       (delete-file elc))
     (package--compile pkg-desc)))
 
-- 
2.35.1

Re: feature/package-vc has been merged

[Stefan Monnier]

> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index a7bcdd214c..bf6849af65 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -462,6 +462,15 @@ package-vc-p
>    (inline-letevals (pkg-desc)
>      (inline-quote (eq (package-desc-kind ,pkg-desc) 'vc))))
>  
> +(define-inline package-lisp-dir (pkg-desc)
> +  "Return the directory with Lisp files for PKG-DESC."
> +  (inline-letevals (pkg-desc)
> +    (inline-quote
> +     (let* ((extras (package-desc-extras ,pkg-desc))
> +            (lisp-dir (alist-get :lisp-dir extras))
> +            (dir (package-desc-dir ,pkg-desc)))
> +       (file-name-directory (file-name-concat dir lisp-dir))))))

Any reason why this needs to be inlined?  I'd expect the inlining to
have a negligible effect since the body contains its fair share of
further function calls, none of which are conditional and AFAICT all the
calls to this pass a variable as argument, so the inlining will not
expose further optimization opportunities.

> @@ -827,7 +836,7 @@ package--reload-previously-loaded
>  byte-compilation of the new package to fail."
>    (with-demoted-errors "Error in package--load-files-for-activation: %s"
>      (let* (result
> -           (dir (package-desc-dir pkg-desc))
> +           (dir (package-lisp-dir pkg-desc))
>             ;; A previous implementation would skip `dir' itself.
>             ;; However, in normal use reloading from the same directory
>             ;; never happens anyway, while in certain cases external to

Why do we need this change?

> @@ -891,7 +900,7 @@ package-activate-1
>            (package--reload-previously-loaded pkg-desc))
>          (with-demoted-errors "Error loading autoloads: %s"
>            (load (package--autoloads-file-name pkg-desc) nil t))
> -        (add-to-list 'load-path (directory-file-name pkg-dir)))
> +        (add-to-list 'load-path (package-lisp-dir pkg-desc)))
>        ;; Add info node.
>        (when (file-exists-p (expand-file-name "dir" pkg-dir))
>          ;; FIXME: not the friendliest, but simple.

This should really not be needed (actually, this `add-to-list` is not
needed at all for any package (re)installed with Emacs≥26 (or maybe even
25, can't remember)).  The "normal" behavior is that it's the autoloads
file which adds to `load-path` (which makes it possible for that
autoloads file to add the `:lisp-dir` instead of the root dir, indeed).

> @@ -1080,9 +1089,10 @@ package-autoload-ensure-default-file
>  (defvar autoload-timestamps)
>  (defvar version-control)
>  
> -(defun package-generate-autoloads (name pkg-dir)
> -  "Generate autoloads in PKG-DIR for package named NAME."
> -  (let* ((auto-name (format "%s-autoloads.el" name))
> +(defun package-generate-autoloads (pkg-desc pkg-dir)
> +  "Generate autoloads for PKG-DESC in PKG-DIR."
> +  (let* ((name (package-desc-name pkg-desc))
> +         (auto-name (format "%s-autoloads.el" name))
>           ;;(ignore-name (concat name "-pkg.el"))
>           (output-file (expand-file-name auto-name pkg-dir))
>           ;; We don't need 'em, and this makes the output reproducible.

I thought an alternative was for `package-vc.el` to call this function
with the `:lisp-dir` as `pkg-dir`, so we don't need to change this part
of the code.

> @@ -1118,7 +1140,7 @@ package--compile
>    (let ((byte-compile-ignore-files (package--parse-elpaignore pkg-desc))
>          (warning-minimum-level :error)
>          (load-path load-path))
> -    (byte-recompile-directory (package-desc-dir pkg-desc) 0 t)))
> +    (byte-recompile-directory (package-lisp-dir pkg-desc) 0 t)))

Why do we need this?  AFAIK this recurses into subdirectories so using
`package-desc-dir` still compiles all the files just fine, no?

>  (defun package--native-compile-async (pkg-desc)
>    "Native compile installed package PKG-DESC asynchronously.
> @@ -1126,7 +1148,7 @@ package--native-compile-async
>  `package-activate-1'."
>    (when (native-comp-available-p)
>      (let ((warning-minimum-level :error))
> -      (native-compile-async (package-desc-dir pkg-desc) t))))
> +      (native-compile-async (package-lisp-dir pkg-desc) t))))

Same here.

> @@ -2419,7 +2441,7 @@ package--newest-p
>  
>  (declare-function comp-el-to-eln-filename "comp.c")
>  (defvar package-vc-repository-store)
> -(defun package--delete-directory (dir pkg-desc)
> +(defun package--delete-directory (dir)
>    "Delete PKG-DESC directory DIR recursively.
>  Clean-up the corresponding .eln files if Emacs is native
>  compiled."

IIUC this part of the change is because `package-delete` does not delete
package-vc packages, right?  If so, I support that 100%:
Packages installed with `package-install` can be deleted without too
much fuss because they usually don't contain any important info, but for
`package-vc` this is completely different and we should either never
delete them or at least not without a minimum of sanity checks
(uncommited changes, stashes, additional branches) and very
explicit prompt.

> @@ -2549,7 +2560,7 @@ package-recompile
>      ;; load them (in case they contain byte code/macros that are now
>      ;; invalid).
>      (dolist (elc (directory-files-recursively
> -                  (package-desc-dir pkg-desc) "\\.elc\\'"))
> +                  (package-lisp-dir pkg-desc) "\\.elc\\'"))
>        (delete-file elc))
>      (package--compile pkg-desc)))

Same as above.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
>> index a7bcdd214c..bf6849af65 100644
>> --- a/lisp/emacs-lisp/package.el
>> +++ b/lisp/emacs-lisp/package.el
>> @@ -462,6 +462,15 @@ package-vc-p
>>    (inline-letevals (pkg-desc)
>>      (inline-quote (eq (package-desc-kind ,pkg-desc) 'vc))))
>>  
>> +(define-inline package-lisp-dir (pkg-desc)
>> +  "Return the directory with Lisp files for PKG-DESC."
>> +  (inline-letevals (pkg-desc)
>> +    (inline-quote
>> +     (let* ((extras (package-desc-extras ,pkg-desc))
>> +            (lisp-dir (alist-get :lisp-dir extras))
>> +            (dir (package-desc-dir ,pkg-desc)))
>> +       (file-name-directory (file-name-concat dir lisp-dir))))))
>
> Any reason why this needs to be inlined?  I'd expect the inlining to
> have a negligible effect since the body contains its fair share of
> further function calls, none of which are conditional and AFAICT all the
> calls to this pass a variable as argument, so the inlining will not
> expose further optimization opportunities.

You are probably right, it would be better to convert this into a
regular function.

>> @@ -827,7 +836,7 @@ package--reload-previously-loaded
>>  byte-compilation of the new package to fail."
>>    (with-demoted-errors "Error in package--load-files-for-activation: %s"
>>      (let* (result
>> -           (dir (package-desc-dir pkg-desc))
>> +           (dir (package-lisp-dir pkg-desc))
>>             ;; A previous implementation would skip `dir' itself.
>>             ;; However, in normal use reloading from the same directory
>>             ;; never happens anyway, while in certain cases external to
>
> Why do we need this change?

Because we are only interested in the sub-directory containing the lisp
files.

>> @@ -891,7 +900,7 @@ package-activate-1
>>            (package--reload-previously-loaded pkg-desc))
>>          (with-demoted-errors "Error loading autoloads: %s"
>>            (load (package--autoloads-file-name pkg-desc) nil t))
>> -        (add-to-list 'load-path (directory-file-name pkg-dir)))
>> +        (add-to-list 'load-path (package-lisp-dir pkg-desc)))
>>        ;; Add info node.
>>        (when (file-exists-p (expand-file-name "dir" pkg-dir))
>>          ;; FIXME: not the friendliest, but simple.
>
> This should really not be needed (actually, this `add-to-list` is not
> needed at all for any package (re)installed with Emacs≥26 (or maybe even
> 25, can't remember)).  The "normal" behavior is that it's the autoloads
> file which adds to `load-path` (which makes it possible for that
> autoloads file to add the `:lisp-dir` instead of the root dir, indeed).

I see what you mean.  But this would be change that is unrelated to
package-vc, so it could just be removed directly on master.

>> @@ -1080,9 +1089,10 @@ package-autoload-ensure-default-file
>>  (defvar autoload-timestamps)
>>  (defvar version-control)
>>  
>> -(defun package-generate-autoloads (name pkg-dir)
>> -  "Generate autoloads in PKG-DIR for package named NAME."
>> -  (let* ((auto-name (format "%s-autoloads.el" name))
>> +(defun package-generate-autoloads (pkg-desc pkg-dir)
>> +  "Generate autoloads for PKG-DESC in PKG-DIR."
>> +  (let* ((name (package-desc-name pkg-desc))
>> +         (auto-name (format "%s-autoloads.el" name))
>>           ;;(ignore-name (concat name "-pkg.el"))
>>           (output-file (expand-file-name auto-name pkg-dir))
>>           ;; We don't need 'em, and this makes the output reproducible.
>
> I thought an alternative was for `package-vc.el` to call this function
> with the `:lisp-dir` as `pkg-dir`, so we don't need to change this part
> of the code.

I might be missing something, but the previous signature was missing a
package description object that the change required.

If you think it is better, I could rename the function and add a
compatibility alias.

>> @@ -1118,7 +1140,7 @@ package--compile
>>    (let ((byte-compile-ignore-files (package--parse-elpaignore pkg-desc))
>>          (warning-minimum-level :error)
>>          (load-path load-path))
>> -    (byte-recompile-directory (package-desc-dir pkg-desc) 0 t)))
>> +    (byte-recompile-directory (package-lisp-dir pkg-desc) 0 t)))
>
> Why do we need this?  AFAIK this recurses into subdirectories so using
> `package-desc-dir` still compiles all the files just fine, no?

Same as above, if we know all the lisp code is located in one
sub-directory, there is no need to compile everything in the
directory -- which might contain test files or other scripts that were
not meant to be compiled.

>>  (defun package--native-compile-async (pkg-desc)
>>    "Native compile installed package PKG-DESC asynchronously.
>> @@ -1126,7 +1148,7 @@ package--native-compile-async
>>  `package-activate-1'."
>>    (when (native-comp-available-p)
>>      (let ((warning-minimum-level :error))
>> -      (native-compile-async (package-desc-dir pkg-desc) t))))
>> +      (native-compile-async (package-lisp-dir pkg-desc) t))))
>
> Same here.

Same here.

>> @@ -2419,7 +2441,7 @@ package--newest-p
>>  
>>  (declare-function comp-el-to-eln-filename "comp.c")
>>  (defvar package-vc-repository-store)
>> -(defun package--delete-directory (dir pkg-desc)
>> +(defun package--delete-directory (dir)
>>    "Delete PKG-DESC directory DIR recursively.
>>  Clean-up the corresponding .eln files if Emacs is native
>>  compiled."
>
> IIUC this part of the change is because `package-delete` does not delete
> package-vc packages, right?  If so, I support that 100%:

This change reverts a previous feature, back when package-vc didn't load
sub-directories, but cloned the repository into some XDG directory and
created a symbolic link from ~/.emacs.d/elpa to the right sub-directory.
To do that, knowledge of what package was been deleted was required:

-  (if (and (package-vc-p pkg-desc)
-           (require 'package-vc)   ;load `package-vc-repository-store'
-           (file-in-directory-p dir package-vc-repository-store))
-      (progn
-        (delete-directory
-         (expand-file-name
-          (car (file-name-split
-                (file-relative-name dir package-vc-repository-store)))
-          package-vc-repository-store)
-         t)
-        (delete-file (directory-file-name dir)))
-    (delete-directory dir t)))
+  (delete-directory dir t))

Now that this isn't being done anymore, I could revert the change and
simplify the code back to the original state (actually it has become
slightly more complicated since, because I noticed that
`package-vc-install-from-checkout' requires some special handling, but
that doesn't require a package description to be detected).

> Packages installed with `package-install` can be deleted without too
> much fuss because they usually don't contain any important info, but for
> `package-vc` this is completely different and we should either never
> delete them or at least not without a minimum of sanity checks
> (uncommited changes, stashes, additional branches) and very
> explicit prompt.

This is currently not done, but could be added.  I am imagining checking
of the package directory is the root directory of a repository (does
that work for all VCS?), and if so double-prompting the user.  But I
would be opposed to preventing users from deleting packages installed
using `package-vc', if only it would go against my workflow of fetching
the sources for a package, preparing and sending a patch, and then
reverting to the release package.

>> @@ -2549,7 +2560,7 @@ package-recompile
>>      ;; load them (in case they contain byte code/macros that are now
>>      ;; invalid).
>>      (dolist (elc (directory-files-recursively
>> -                  (package-desc-dir pkg-desc) "\\.elc\\'"))
>> +                  (package-lisp-dir pkg-desc) "\\.elc\\'"))
>>        (delete-file elc))
>>      (package--compile pkg-desc)))
>
> Same as above.

The explanation remains the same.

>
>         Stefan

Re: feature/package-vc has been merged

[Stefan Monnier]

>>> @@ -827,7 +836,7 @@ package--reload-previously-loaded
>>>  byte-compilation of the new package to fail."
>>>    (with-demoted-errors "Error in package--load-files-for-activation: %s"
>>>      (let* (result
>>> -           (dir (package-desc-dir pkg-desc))
>>> +           (dir (package-lisp-dir pkg-desc))
>>>             ;; A previous implementation would skip `dir' itself.
>>>             ;; However, in normal use reloading from the same directory
>>>             ;; never happens anyway, while in certain cases external to
>>
>> Why do we need this change?
>
> Because we are only interested in the sub-directory containing the lisp
> files.

But is there any harm in considering the whole parent directory instead?

>>> @@ -891,7 +900,7 @@ package-activate-1
>>>            (package--reload-previously-loaded pkg-desc))
>>>          (with-demoted-errors "Error loading autoloads: %s"
>>>            (load (package--autoloads-file-name pkg-desc) nil t))
>>> -        (add-to-list 'load-path (directory-file-name pkg-dir)))
>>> +        (add-to-list 'load-path (package-lisp-dir pkg-desc)))
>>>        ;; Add info node.
>>>        (when (file-exists-p (expand-file-name "dir" pkg-dir))
>>>          ;; FIXME: not the friendliest, but simple.
>>
>> This should really not be needed (actually, this `add-to-list` is not
>> needed at all for any package (re)installed with Emacs≥26 (or maybe even
>> 25, can't remember)).  The "normal" behavior is that it's the autoloads
>> file which adds to `load-path` (which makes it possible for that
>> autoloads file to add the `:lisp-dir` instead of the root dir, indeed).
>
> I see what you mean.  But this would be change that is unrelated to
> package-vc, so it could just be removed directly on master.

Removing it would is indeed a decision unrelated to `package-vc`.
But in the mean time you can simply not change that code.

>>> @@ -1080,9 +1089,10 @@ package-autoload-ensure-default-file
>>>  (defvar autoload-timestamps)
>>>  (defvar version-control)
>>>  
>>> -(defun package-generate-autoloads (name pkg-dir)
>>> -  "Generate autoloads in PKG-DIR for package named NAME."
>>> -  (let* ((auto-name (format "%s-autoloads.el" name))
>>> +(defun package-generate-autoloads (pkg-desc pkg-dir)
>>> +  "Generate autoloads for PKG-DESC in PKG-DIR."
>>> +  (let* ((name (package-desc-name pkg-desc))
>>> +         (auto-name (format "%s-autoloads.el" name))
>>>           ;;(ignore-name (concat name "-pkg.el"))
>>>           (output-file (expand-file-name auto-name pkg-dir))
>>>           ;; We don't need 'em, and this makes the output reproducible.
>>
>> I thought an alternative was for `package-vc.el` to call this function
>> with the `:lisp-dir` as `pkg-dir`, so we don't need to change this part
>> of the code.
>
> I might be missing something, but the previous signature was missing a
> package description object that the change required.

No, I mean that the change should not be needed (and hence the change
in signature shouldn't be needed either).

>> Why do we need this?  AFAIK this recurses into subdirectories so using
>> `package-desc-dir` still compiles all the files just fine, no?
> Same as above, if we know all the lisp code is located in one
> sub-directory, there is no need to compile everything in the
> directory -- which might contain test files or other scripts that were
> not meant to be compiled.

In elpa-admin.el I compile all those ancillary files as well.
I'm not sure we should refrain from compiling them, actually.
[ But yes: their compilation will fail more often.  I consider it
  a problem in the packages.  ]

>> IIUC this part of the change is because `package-delete` does not delete
>> package-vc packages, right?  If so, I support that 100%:
>
> This change reverts a previous feature, back when package-vc didn't load
> sub-directories, but cloned the repository into some XDG directory and
> created a symbolic link from ~/.emacs.d/elpa to the right sub-directory.
> To do that, knowledge of what package was been deleted was required:

Good, thanks.

> This is currently not done, but could be added.  I am imagining checking
> of the package directory is the root directory of a repository (does
> that work for all VCS?), and if so double-prompting the user.  But I
> would be opposed to preventing users from deleting packages installed
> using `package-vc', if only it would go against my workflow of fetching
> the sources for a package, preparing and sending a patch, and then
> reverting to the release package.

I'm not saying we should prevent it, but just that we should be careful
to only delete with the full, express, and informed consent of the users.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> @@ -827,7 +836,7 @@ package--reload-previously-loaded
>>>>  byte-compilation of the new package to fail."
>>>>    (with-demoted-errors "Error in package--load-files-for-activation: %s"
>>>>      (let* (result
>>>> -           (dir (package-desc-dir pkg-desc))
>>>> +           (dir (package-lisp-dir pkg-desc))
>>>>             ;; A previous implementation would skip `dir' itself.
>>>>             ;; However, in normal use reloading from the same directory
>>>>             ;; never happens anyway, while in certain cases external to
>>>
>>> Why do we need this change?
>>
>> Because we are only interested in the sub-directory containing the lisp
>> files.
>
> But is there any harm in considering the whole parent directory instead?

As mentioned below, I think the harm is that unintended error could
appear.  But I get your argument too, that mistakes should be fixed in
general and having these pop up during byte compilation is a good way to
make these more noticeable...

>>>> @@ -891,7 +900,7 @@ package-activate-1
>>>>            (package--reload-previously-loaded pkg-desc))
>>>>          (with-demoted-errors "Error loading autoloads: %s"
>>>>            (load (package--autoloads-file-name pkg-desc) nil t))
>>>> -        (add-to-list 'load-path (directory-file-name pkg-dir)))
>>>> +        (add-to-list 'load-path (package-lisp-dir pkg-desc)))
>>>>        ;; Add info node.
>>>>        (when (file-exists-p (expand-file-name "dir" pkg-dir))
>>>>          ;; FIXME: not the friendliest, but simple.
>>>
>>> This should really not be needed (actually, this `add-to-list` is not
>>> needed at all for any package (re)installed with Emacs≥26 (or maybe even
>>> 25, can't remember)).  The "normal" behavior is that it's the autoloads
>>> file which adds to `load-path` (which makes it possible for that
>>> autoloads file to add the `:lisp-dir` instead of the root dir, indeed).
>>
>> I see what you mean.  But this would be change that is unrelated to
>> package-vc, so it could just be removed directly on master.
>
> Removing it would is indeed a decision unrelated to `package-vc`.
> But in the mean time you can simply not change that code.

OK, I'll revert that change.

>>>> @@ -1080,9 +1089,10 @@ package-autoload-ensure-default-file
>>>>  (defvar autoload-timestamps)
>>>>  (defvar version-control)
>>>>  
>>>> -(defun package-generate-autoloads (name pkg-dir)
>>>> -  "Generate autoloads in PKG-DIR for package named NAME."
>>>> -  (let* ((auto-name (format "%s-autoloads.el" name))
>>>> +(defun package-generate-autoloads (pkg-desc pkg-dir)
>>>> +  "Generate autoloads for PKG-DESC in PKG-DIR."
>>>> +  (let* ((name (package-desc-name pkg-desc))
>>>> +         (auto-name (format "%s-autoloads.el" name))
>>>>           ;;(ignore-name (concat name "-pkg.el"))
>>>>           (output-file (expand-file-name auto-name pkg-dir))
>>>>           ;; We don't need 'em, and this makes the output reproducible.
>>>
>>> I thought an alternative was for `package-vc.el` to call this function
>>> with the `:lisp-dir` as `pkg-dir`, so we don't need to change this part
>>> of the code.
>>
>> I might be missing something, but the previous signature was missing a
>> package description object that the change required.
>
> No, I mean that the change should not be needed (and hence the change
> in signature shouldn't be needed either).

If there is any place where :lisp-dir this is needed, then here, because
this is the place where the auto-load is generated containing the
`load-path' modification.  If I don't have the package description, then
I cannot infer the right sub-directory.

>>> Why do we need this?  AFAIK this recurses into subdirectories so using
>>> `package-desc-dir` still compiles all the files just fine, no?
>> Same as above, if we know all the lisp code is located in one
>> sub-directory, there is no need to compile everything in the
>> directory -- which might contain test files or other scripts that were
>> not meant to be compiled.
>
> In elpa-admin.el I compile all those ancillary files as well.
> I'm not sure we should refrain from compiling them, actually.
> [ But yes: their compilation will fail more often.  I consider it
>   a problem in the packages.  ]
>
>>> IIUC this part of the change is because `package-delete` does not delete
>>> package-vc packages, right?  If so, I support that 100%:
>>
>> This change reverts a previous feature, back when package-vc didn't load
>> sub-directories, but cloned the repository into some XDG directory and
>> created a symbolic link from ~/.emacs.d/elpa to the right sub-directory.
>> To do that, knowledge of what package was been deleted was required:
>
> Good, thanks.
>
>> This is currently not done, but could be added.  I am imagining checking
>> of the package directory is the root directory of a repository (does
>> that work for all VCS?), and if so double-prompting the user.  But I
>> would be opposed to preventing users from deleting packages installed
>> using `package-vc', if only it would go against my workflow of fetching
>> the sources for a package, preparing and sending a patch, and then
>> reverting to the release package.
>
> I'm not saying we should prevent it, but just that we should be careful
> to only delete with the full, express, and informed consent of the users.

I agree.

Re: feature/package-vc has been merged

[Stefan Monnier]

> As mentioned below, I think the harm is that unintended error could
> appear.  But I get your argument too, that mistakes should be fixed in
> general and having these pop up during byte compilation is a good way to
> make these more noticeable...

Either way is fine by me, every use of `lisp-dir` should come with
a comment justifying it, IMO.

>>>>> -(defun package-generate-autoloads (name pkg-dir)
>>>>> -  "Generate autoloads in PKG-DIR for package named NAME."
>>>>> -  (let* ((auto-name (format "%s-autoloads.el" name))
>>>>> +(defun package-generate-autoloads (pkg-desc pkg-dir)
>>>>> +  "Generate autoloads for PKG-DESC in PKG-DIR."
>>>>> +  (let* ((name (package-desc-name pkg-desc))
>>>>> +         (auto-name (format "%s-autoloads.el" name))
>>>>>           ;;(ignore-name (concat name "-pkg.el"))
>>>>>           (output-file (expand-file-name auto-name pkg-dir))
>>>>>           ;; We don't need 'em, and this makes the output reproducible.
>>>>
>>>> I thought an alternative was for `package-vc.el` to call this function
>>>> with the `:lisp-dir` as `pkg-dir`, so we don't need to change this part
>>>> of the code.
>>>
>>> I might be missing something, but the previous signature was missing a
>>> package description object that the change required.
>>
>> No, I mean that the change should not be needed (and hence the change
>> in signature shouldn't be needed either).
>
> If there is any place where :lisp-dir this is needed, then here, because
> this is the place where the auto-load is generated containing the
> `load-path' modification.  If I don't have the package description, then
> I cannot infer the right sub-directory.

I don't understand: in my mental model, package-vc would call
(package-generate-autoloads 'org "/foo/bar/org/lisp/") and that would
generate the right autoloads file with the right modification of
`load-path`, and then `package-vc` would just need to create an
additional /foo/bar/org/org-autoloads.el file which simply loads
/foo/bar/org/lisp/org-autoloads.el.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> As mentioned below, I think the harm is that unintended error could
>> appear.  But I get your argument too, that mistakes should be fixed in
>> general and having these pop up during byte compilation is a good way to
>> make these more noticeable...
>
> Either way is fine by me, every use of `lisp-dir` should come with
> a comment justifying it, IMO.

I've decided to remove the function entirely.

>>>>>> -(defun package-generate-autoloads (name pkg-dir)
>>>>>> -  "Generate autoloads in PKG-DIR for package named NAME."
>>>>>> -  (let* ((auto-name (format "%s-autoloads.el" name))
>>>>>> +(defun package-generate-autoloads (pkg-desc pkg-dir)
>>>>>> +  "Generate autoloads for PKG-DESC in PKG-DIR."
>>>>>> +  (let* ((name (package-desc-name pkg-desc))
>>>>>> +         (auto-name (format "%s-autoloads.el" name))
>>>>>>           ;;(ignore-name (concat name "-pkg.el"))
>>>>>>           (output-file (expand-file-name auto-name pkg-dir))
>>>>>>           ;; We don't need 'em, and this makes the output reproducible.
>>>>>
>>>>> I thought an alternative was for `package-vc.el` to call this function
>>>>> with the `:lisp-dir` as `pkg-dir`, so we don't need to change this part
>>>>> of the code.
>>>>
>>>> I might be missing something, but the previous signature was missing a
>>>> package description object that the change required.
>>>
>>> No, I mean that the change should not be needed (and hence the change
>>> in signature shouldn't be needed either).
>>
>> If there is any place where :lisp-dir this is needed, then here, because
>> this is the place where the auto-load is generated containing the
>> `load-path' modification.  If I don't have the package description, then
>> I cannot infer the right sub-directory.
>
> I don't understand: in my mental model, package-vc would call
> (package-generate-autoloads 'org "/foo/bar/org/lisp/") and that would
> generate the right autoloads file with the right modification of
> `load-path`, and then `package-vc` would just need to create an
> additional /foo/bar/org/org-autoloads.el file which simply loads
> /foo/bar/org/lisp/org-autoloads.el.

I wanted to place the autoload file in top-level package directory, but
I guess if make sure :lisp-dir gets respected during activation, then
this could be reverted.  I'd have to check how reliable this is.

Re: feature/package-vc has been merged

[Stefan Monnier]

> I wanted to place the autoload file in top-level package directory, but
> I guess if make sure :lisp-dir gets respected during activation, then
> this could be reverted.  I'd have to check how reliable this is.

I think package activation should not pay attention to `lisp-dir` or
more generally it should not pay attention to *how* the package was
installed (whether from `package-install`, `package-vc-install`, or
manually, or via some other system).

Which is why the simplest accommodation is to use 2 autoload files:
the main one in :lisp-dir and a proxy in the root.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I wanted to place the autoload file in top-level package directory, but
>> I guess if make sure :lisp-dir gets respected during activation, then
>> this could be reverted.  I'd have to check how reliable this is.
>
> I think package activation should not pay attention to `lisp-dir` or
> more generally it should not pay attention to *how* the package was
> installed (whether from `package-install`, `package-vc-install`, or
> manually, or via some other system).
>
> Which is why the simplest accommodation is to use 2 autoload files:
> the main one in :lisp-dir and a proxy in the root.

I have made these changes and pushed them to the scratch branch.  The
proxy file might be a hack, but it seems like the better solution in the
end.

Re: feature/package-vc has been merged

[Stefan Monnier]

>>> As mentioned below, I think the harm is that unintended error could
>>> appear.  But I get your argument too, that mistakes should be fixed in
>>> general and having these pop up during byte compilation is a good way to
>>> make these more noticeable...
>> Either way is fine by me, every use of `lisp-dir` should come with
>> a comment justifying it, IMO.
> I've decided to remove the function entirely.

I don't understand what you're referring to.  I was not talking about
any particular function but about all uses of the (package-vc-specific)
`:lisp-dir` information in `package.el`.


        Stefan

Updating the "ELPA Protocol"

[Philip Kaludercic]

> I do think it's getting time to update the protocol based on the
> experience gained over the last 10 years, but I'm really not convinced
> `:lisp-dir` is a good addition.

Oh: Are there any other things you think should be changed given the
chance?

Re: Updating the "ELPA Protocol"

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

>> I do think it's getting time to update the protocol based on the
>> experience gained over the last 10 years, but I'm really not convinced
>> `:lisp-dir` is a good addition.
>
> Oh: Are there any other things you think should be changed given the
> chance?

One issue I have been told is the issue of renaming a package, so that
package.el can transparently handle the rename.  That appears to not be
supported.

Re: Updating the "ELPA Protocol"

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> One issue I have been told is the issue of renaming a package, so that
> package.el can transparently handle the rename.  That appears to not be
> supported.

How about using something like Debian's transitional packages, where you
replace the old package with an empty package that just depends on the
new package?  Would that work?

Re: Updating the "ELPA Protocol"

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> One issue I have been told is the issue of renaming a package, so that
>> package.el can transparently handle the rename.  That appears to not be
>> supported.
>
> How about using something like Debian's transitional packages, where you
> replace the old package with an empty package that just depends on the
> new package?  Would that work?

It might, but then the user is required to remove the old package and
select the new one, right?

Re: Updating the "ELPA Protocol"

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> How about using something like Debian's transitional packages, where you
>> replace the old package with an empty package that just depends on the
>> new package?  Would that work?
>
> It might, but then the user is required to remove the old package and
> select the new one, right?

AFAIU, the new renamed package will be automatically installed as a
dependency of the new version so in that sense I don't think the user
would need to do anything.  The user will potentially have to update
their customization, in case the package prefix changes and symbols are
renamed.

The user will also have to manually delete the old package and manually
select the new one at some point.  This should merely be a cleanup
though, and in theory the user can continue as before as long as the
transitional package remains in the archive.

Re: Updating the "ELPA Protocol"

[Stefan Monnier]

>>> One issue I have been told is the issue of renaming a package, so that
>>> package.el can transparently handle the rename.  That appears to not be
>>> supported.

I think we'd need more details and concrete examples to judge how best
handle such renamings.  The problem I see is that in Emacs, names are
very visible: the package name almost inevitably affect the ELisp
files's names, which themselves affect the functions and vars defined
therein.

For that reason handling the renaming only in ELPA is rarely sufficient.
And also for that reason, renamings are rare.

>> How about using something like Debian's transitional packages, where you
>> replace the old package with an empty package that just depends on the
>> new package?  Would that work?
> It might, but then the user is required to remove the old package and
> select the new one, right?

Compared to all the adjustments they may need to do do their config,
I suspect this is rather minor (and as the other Stefan mentions, it's
not indispensable).


        Stefan

Re: Updating the "ELPA Protocol"

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> One issue I have been told is the issue of renaming a package, so that
>>>> package.el can transparently handle the rename.  That appears to not be
>>>> supported.
>
> I think we'd need more details and concrete examples to judge how best
> handle such renamings.  The problem I see is that in Emacs, names are
> very visible: the package name almost inevitably affect the ELisp
> files's names, which themselves affect the functions and vars defined
> therein.
>
> For that reason handling the renaming only in ELPA is rarely sufficient.
> And also for that reason, renamings are rare.

From what I know this happens from time to time on MELPA.  The changes
usually have to be backwards-compatible or the renames must be trivial
(a change like "foo" -> "foo-mode" or vice versa).

>>> How about using something like Debian's transitional packages, where you
>>> replace the old package with an empty package that just depends on the
>>> new package?  Would that work?
>> It might, but then the user is required to remove the old package and
>> select the new one, right?
>
> Compared to all the adjustments they may need to do do their config,
> I suspect this is rather minor (and as the other Stefan mentions, it's
> not indispensable).

True, but I can imagine it causing confusion, when you find the actually
selected package being empty and the real package listed as a "mere"
dependency.

Re: Updating the "ELPA Protocol"

[Stefan Monnier]

>> I think we'd need more details and concrete examples to judge how best
>> handle such renamings.  The problem I see is that in Emacs, names are
>> very visible: the package name almost inevitably affect the ELisp
>> files's names, which themselves affect the functions and vars defined
>> therein.
>>
>> For that reason handling the renaming only in ELPA is rarely sufficient.
>> And also for that reason, renamings are rare.
>
> From what I know this happens from time to time on MELPA.  The changes
> usually have to be backwards-compatible or the renames must be trivial
> (a change like "foo" -> "foo-mode" or vice versa).

Interesting.  It would be great to have a (more or less exhaustive) list
of renamings that took place in MELPA.

> True, but I can imagine it causing confusion, when you find the actually
> selected package being empty and the real package listed as a "mere"
> dependency.

I don't find it particularly confusing in Debian, because the new empty
package usually describes itself fairly clearly as some kind of
"transition package" where it say gives instructions on how to go about
it (that you can remove it, etc..).


        Stefan

Re: Updating the "ELPA Protocol"

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I think we'd need more details and concrete examples to judge how best
>>> handle such renamings.  The problem I see is that in Emacs, names are
>>> very visible: the package name almost inevitably affect the ELisp
>>> files's names, which themselves affect the functions and vars defined
>>> therein.
>>>
>>> For that reason handling the renaming only in ELPA is rarely sufficient.
>>> And also for that reason, renamings are rare.
>>
>> From what I know this happens from time to time on MELPA.  The changes
>> usually have to be backwards-compatible or the renames must be trivial
>> (a change like "foo" -> "foo-mode" or vice versa).
>
> Interesting.  It would be great to have a (more or less exhaustive) list
> of renamings that took place in MELPA.

This is not really exhaustive, but a good starting point:

https://github.com/melpa/melpa/search?o=desc&q=rename&s=committer-date&type=commits

There are 234 commits with the word "rename".  Some are unrelated
renames because MELPA has a single repository for admin scripts and
specs, but most appear to be real package renames ("string-edit" ->
"string-edit-at-point", "farmhouse-theme" -> "farmhouse-themes",
"goldendict" -> "external-dict").

>> True, but I can imagine it causing confusion, when you find the actually
>> selected package being empty and the real package listed as a "mere"
>> dependency.
>
> I don't find it particularly confusing in Debian, because the new empty
> package usually describes itself fairly clearly as some kind of
> "transition package" where it say gives instructions on how to go about
> it (that you can remove it, etc..).

Hmm, I think I agree if we ensure that the transition package has a
clear description that would indicate it is a replacement.

Re: Updating the "ELPA Protocol"

[Jonas Bernoulli]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Interesting.  It would be great to have a (more or less exhaustive) list
> of renamings that took place in MELPA.

Grep <melpa>/recipes or ":old-names".

Re: Updating the "ELPA Protocol"

[Jonas Bernoulli]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> One issue I have been told is the issue of renaming a package, so that
>> package.el can transparently handle the rename.  That appears to not be
>> supported.
>
> How about using something like Debian's transitional packages, where you
> replace the old package with an empty package that just depends on the
> new package?  Would that work?

I would prefer if package.el took care of it.
`package-update-all' could ask:

  Package `foo' has been renamed to `foobar'.
  When a package is renamed, then names of variables and functions
  often change as well, making it necessary to adjust configuration.

  Would you like to replace `foo' with `foobar' now?
  yes/no/no, don't ask again

Somewhat related, I have again started working on moving Melpa
("unstable") away from using timestamps as version strings.  Eventually
users will have to update from, e.g., 20230101.123 to the smaller 0.2.3.
Here too I hope we could add a special case to package.el.  It wouldn't
have to remain in place forever; other *elpa are unlikely to repeat the
mistake of using timestamps.

And while I am at it, the most important "feature" I am hoping for is
for package.el to be added to GNU Elpa, so that we can start relying on
the newly added features, without having to wait half a decade.

Re: Updating the "ELPA Protocol"

[Stefan Monnier]

>> How about using something like Debian's transitional packages, where you
>> replace the old package with an empty package that just depends on the
>> new package?  Would that work?
>
> I would prefer if package.el took care of it.
> `package-update-all' could ask:
>
>   Package `foo' has been renamed to `foobar'.
>   When a package is renamed, then names of variables and functions
>   often change as well, making it necessary to adjust configuration.
>
>   Would you like to replace `foo' with `foobar' now?
>   yes/no/no, don't ask again

This discussion of replacement reminds me of the more general problem of
restructuring where a package may split into 2 (or more), or the reverse.

The combination of dependencies and renaming/replacement may cover all
those cases if used wisely, but we should pay attention to the details
so that it does.

> Somewhat related, I have again started working on moving Melpa
> ("unstable") away from using timestamps as version strings.  Eventually
> users will have to update from, e.g., 20230101.123 to the smaller 0.2.3.
> Here too I hope we could add a special case to package.el.

We may have a few problematic packages which use versions that look very
much like Melpa's, but my gut feeling is that it shouldn't be too hard
to come up with a heuristic that can handle it correctly enough, yes.


        Stefan

Re: Updating the "ELPA Protocol"

[Jonas Bernoulli]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> How about using something like Debian's transitional packages, where you
>>> replace the old package with an empty package that just depends on the
>>> new package?  Would that work?
>>
>> I would prefer if package.el took care of it.
>> `package-update-all' could ask:
>>
>>   Package `foo' has been renamed to `foobar'.
>>   When a package is renamed, then names of variables and functions
>>   often change as well, making it necessary to adjust configuration.
>>
>>   Would you like to replace `foo' with `foobar' now?
>>   yes/no/no, don't ask again
>
> This discussion of replacement reminds me of the more general problem of
> restructuring where a package may split into 2 (or more), or the reverse.
>
> The combination of dependencies and renaming/replacement may cover all
> those cases if used wisely, but we should pay attention to the details
> so that it does.

A common variant of "or the reverse" is when a package absorbs a
formerly independent package.  evil-collection has absorbed several
mode-specific packages that uses to developed independently.

Re: feature/package-vc has been merged

[Stefan Monnier]

Philip Kaludercic [2022-11-06 18:35:10] wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>>> From: Philip Kaludercic <philipk@posteo.net>
>>> Cc: monnier@iro.umontreal.ca,  rms@gnu.org,  emacs-devel@gnu.org
>>> Date: Sun, 06 Nov 2022 17:31:22 +0000
>>> 
>>> Let's say I notice a something I would want to change/add/fix in a
>>> package I am using.  find-function is just one way I would query Emacs
>>> to open up the source, then make a few changes.  If I decide that these
>>> are worth up-streaming, it is nice to just commit them right away and
>>> call `package-vc-prepare-patch' to send the maintainer an email.
>>
>> How many package users do that?
>
> I know of at least one for certain, and that is me.

I do too.

Also, the Borg, the Straight, and the DELPS package managers implemented
that same functionality, so there's clearly a public for it.

>> And if you think many do, why not clone the repository directly into
>> ~/.emacs.d/elpa/?
>
> Because that won't take care of scraping for autoloads, byte
> compilation and installing missing dependencies.

Also because packages in there are expected to be installed under the
<PKG>-<VERSION> directory, whereas when installing from Git a package
will be updated in place so its version number will keep changing, and
it would be inconvenient to rename its directory every time.


        Stefan

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 20:30:07 -0500
> 
> >> And if you think many do, why not clone the repository directly into
> >> ~/.emacs.d/elpa/?
> >
> > Because that won't take care of scraping for autoloads, byte
> > compilation and installing missing dependencies.
> 
> Also because packages in there are expected to be installed under the
> <PKG>-<VERSION> directory, whereas when installing from Git a package
> will be updated in place so its version number will keep changing, and
> it would be inconvenient to rename its directory every time.

Then how about removing that requirement for packages installed
directly from their upstream repositories?

Re: feature/package-vc has been merged

[Stefan Monnier]

>> >> And if you think many do, why not clone the repository directly into
>> >> ~/.emacs.d/elpa/?
>> >
>> > Because that won't take care of scraping for autoloads, byte
>> > compilation and installing missing dependencies.
>> 
>> Also because packages in there are expected to be installed under the
>> <PKG>-<VERSION> directory, whereas when installing from Git a package
>> will be updated in place so its version number will keep changing, and
>> it would be inconvenient to rename its directory every time.
>
> Then how about removing that requirement for packages installed
> directly from their upstream repositories?

Not sure what would be the benefit.
I personally like the distinction between `~/.emacs.d/elpa` (for
packages that are installed as "black boxes") and those installed from
Git which are stored in a "regular" location (i.e. where I keep my
source code rather than my config).

The downside would be increased mutual dependency between the package-vc
and package-nonvc code.


        Stefan

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: philipk@posteo.net,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 06 Nov 2022 23:43:55 -0500
> 
> >> >> And if you think many do, why not clone the repository directly into
> >> >> ~/.emacs.d/elpa/?
> >> >
> >> > Because that won't take care of scraping for autoloads, byte
> >> > compilation and installing missing dependencies.
> >> 
> >> Also because packages in there are expected to be installed under the
> >> <PKG>-<VERSION> directory, whereas when installing from Git a package
> >> will be updated in place so its version number will keep changing, and
> >> it would be inconvenient to rename its directory every time.
> >
> > Then how about removing that requirement for packages installed
> > directly from their upstream repositories?
> 
> Not sure what would be the benefit.

Simplicity (only one location for installing packages) and solution
for problems that otherwise need non-trivial tinkering.  See the
preceding discussions.

> I personally like the distinction between `~/.emacs.d/elpa` (for
> packages that are installed as "black boxes") and those installed from
> Git which are stored in a "regular" location (i.e. where I keep my
> source code rather than my config).

I fail to recognize a significant advantage of this, and I definitely
don't see how this would justify all the complications we've been
discussing lately here.

> The downside would be increased mutual dependency between the package-vc
> and package-nonvc code.

Why increased?  And why downside?  Emacs infrastructure packages are
inter-dependent anyway, and we never saw that as a significant
disadvantage.

Re: feature/package-vc has been merged

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I personally like the distinction between `~/.emacs.d/elpa` (for
> packages that are installed as "black boxes") and those installed from
> Git which are stored in a "regular" location (i.e. where I keep my
> source code rather than my config).

Yes, doing it the other way would also be pretty confusing: how to
remember which directories in ~/.emacs.d/elpa are git repositories?

It would also break my use case: in combination with (use-package foo
:ensure t), a simple "rm -rf ~/.emacs.d/elpa" and restarting Emacs
lets me reinstall (and upgrade) all packages automatically.  I do that
pretty frequently.  I guess use-package must grow support for
package-vc though.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> I personally like the distinction between `~/.emacs.d/elpa` (for
>> packages that are installed as "black boxes") and those installed from
>> Git which are stored in a "regular" location (i.e. where I keep my
>> source code rather than my config).
>
> Yes, doing it the other way would also be pretty confusing: how to
> remember which directories in ~/.emacs.d/elpa are git repositories?
>
> It would also break my use case: in combination with (use-package foo
> :ensure t), a simple "rm -rf ~/.emacs.d/elpa" and restarting Emacs
> lets me reinstall (and upgrade) all packages automatically.  I do that
> pretty frequently.  I guess use-package must grow support for
> package-vc though.

That ought to be pretty simple.  All it will require is an additional
keyword to indicate that `package-vc-install' should be used.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sun, 06 Nov 2022 20:30:07 -0500
>> 
>> >> And if you think many do, why not clone the repository directly into
>> >> ~/.emacs.d/elpa/?
>> >
>> > Because that won't take care of scraping for autoloads, byte
>> > compilation and installing missing dependencies.
>> 
>> Also because packages in there are expected to be installed under the
>> <PKG>-<VERSION> directory, whereas when installing from Git a package
>> will be updated in place so its version number will keep changing, and
>> it would be inconvenient to rename its directory every time.
>
> Then how about removing that requirement for packages installed
> directly from their upstream repositories?

That is already the case.  Package-vc creates a directory <PKG>, without
any <VERSION>.

Re: feature/package-vc has been merged

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> >>> Let's say I notice a something I would want to change/add/fix in a
> >>> package I am using.  find-function is just one way I would query Emacs
> >>> to open up the source, then make a few changes.  If I decide that these
> >>> are worth up-streaming, it is nice to just commit them right away and
> >>> call `package-vc-prepare-patch' to send the maintainer an email.
> >>
> >> How many package users do that?
> >
> > I know of at least one for certain, and that is me.
>
> I do too.
>
> Also, the Borg, the Straight, and the DELPS package managers implemented
> that same functionality, so there's clearly a public for it.

Me too.  Actually, I thought that was the entire motivation for
package-vc.  Is there any other reason to have it?

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> >>> Let's say I notice a something I would want to change/add/fix in a
>> >>> package I am using.  find-function is just one way I would query Emacs
>> >>> to open up the source, then make a few changes.  If I decide that these
>> >>> are worth up-streaming, it is nice to just commit them right away and
>> >>> call `package-vc-prepare-patch' to send the maintainer an email.
>> >>
>> >> How many package users do that?
>> >
>> > I know of at least one for certain, and that is me.
>>
>> I do too.
>>
>> Also, the Borg, the Straight, and the DELPS package managers implemented
>> that same functionality, so there's clearly a public for it.
>
> Me too.  Actually, I thought that was the entire motivation for
> package-vc.  Is there any other reason to have it?

It was my personal motivation, but I know that some people prefer using
version controlled package sources to improve the reproducability of an
Emacs configuration.

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

>>> >> How many package users do that?
>>> >
>>> > I know of at least one for certain, and that is me.
>>>
>>> I do too.
>>>
>>> Also, the Borg, the Straight, and the DELPS package managers implemented
>>> that same functionality, so there's clearly a public for it.
>>
>> Me too.  Actually, I thought that was the entire motivation for
>> package-vc.  Is there any other reason to have it?
>
> It was my personal motivation, but I know that some people prefer using
> version controlled package sources to improve the reproducability of an
> Emacs configuration.

This and the lower the entry level/bar is to contribute the more user
potentially contribute. For me since moving to Borg my contributions
have gone greatly compared to using package.el where there was less user
control and a higher level effort I have to take to get to the packages
sources and it's version control.

However this process also depends on how good the system is, magit,
forge and epkg are very good.
From my past experience vc didn't came close to that. package-vc also
still depends on the centralized package information and doesn't include
other sources such as melpa by default.

Br,

Björn 

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> However this process also depends on how good the system is, magit,
> forge and epkg are very good.
> From my past experience vc didn't came close to that. 

Package-vc just uses vc for all version control operations, but there is
no requirement to use vc yourself.  If a Git repository is cloned, there
is nothing stopping you from using Magit to create a commit.

The only exception might be if you want to use
`package-vc-prepare-patch', which does in fact depend on
`vc-prepare-patch'.

>                                                       package-vc also
> still depends on the centralized package information and doesn't include
> other sources such as melpa by default.

It is as centralised as package management itself.  Each archive may
list package specifications, next to the archive contents.  As the
feature is new, MELPA hasn't probably had the time to configure the
file, but I hope that might change in the future.

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> However this process also depends on how good the system is, magit,
>> forge and epkg are very good.
>> From my past experience vc didn't came close to that. 
>
> Package-vc just uses vc for all version control operations, but there is
> no requirement to use vc yourself.  If a Git repository is cloned, there
> is nothing stopping you from using Magit to create a commit.

It is about Magit's ui to work with the package, view it,
e.g. it's current status without all it's current issues/bugs, commits
and so on.

Not just about creating a commit.

Br,

Björn

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> However this process also depends on how good the system is, magit,
>>> forge and epkg are very good.
>>> From my past experience vc didn't came close to that. 
>>
>> Package-vc just uses vc for all version control operations, but there is
>> no requirement to use vc yourself.  If a Git repository is cloned, there
>> is nothing stopping you from using Magit to create a commit.
>
> It is about Magit's ui to work with the package, view it,
> e.g. it's current status without all it's current issues/bugs, commits
                           ^
                           without?
> and so on.
>
> Not just about creating a commit.

Can you give me a link or some good example to understand this better?

> Br,
>
> Björn

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

>>
>> It is about Magit's ui to work with the package, view it,
>> e.g. it's current status without all it's current issues/bugs, commits
>                            ^
>                            without?
>> and so on.
>>
>> Not just about creating a commit.
>
> Can you give me a link or some good example to understand this better?

Hm I think borgs manual about updating drones is a good example[1].
Read the sections and maybe try it for your self. You can read quite
good what changed on each package you have in your setup, decide if you
pull then update, update with git pull, and then call `borg-build` to
update your package. All done asynchronous without making you wait for
it done be done.

If you have your own changes, you can see in the magit-status buffer the
differentiating commits etc.
In case there were missing features in the package or other issues you
can  press ' f f to view them.

I would recommend to setup a repository of any package you use with
magit and forge, you can see that way that working with a package and
updating it works. 


[1] https://emacsmirror.net/manual/borg/Updating-drones.html

Br,

Björn 

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> I would recommend to setup a repository of any package you use with
> magit and forge, you can see that way that working with a package and
> updating it works. 

I have never gotten "forge" to work, and it isn't available on GNU ELPA
or NonGNU ELPA to begin with.

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> I would recommend to setup a repository of any package you use with
>> magit and forge, you can see that way that working with a package and
>> updating it works. 
>
> I have never gotten "forge" to work, and it isn't available on GNU ELPA
> or NonGNU ELPA to begin with.

It is available on Melpa? Just add it.
It did work fine for others, just ask in case you have issues.

Br,

Björn

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> I would recommend to setup a repository of any package you use with
>>> magit and forge, you can see that way that working with a package and
>>> updating it works. 
>>
>> I have never gotten "forge" to work, and it isn't available on GNU ELPA
>> or NonGNU ELPA to begin with.
>
> It is available on Melpa? Just add it.
> It did work fine for others, just ask in case you have issues.

I guess so, but I barley have the time to work on package-vc these days
(and there is still work to be done on that front), so playing around
with other package managers that all have to be configured, and have
different priorities to what I am doing is not something I am interested
in doing right now.

That being said, if you have any concrete suggestions on what package-vc
ought to do, coming from Borg, I'd be happy to hear them.

Re: feature/package-vc has been merged

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Please do not encourage people to load packages from MELPA.  MELPA
does not cooperate with us.  Not in legal matters, not in ethical
matters, and not in technical matters of development.

A given package that happens to be in MELPA may be perfectly fine in
and of itself, or it may have problems of one kind of the other.

If you come across a package in MELPA that has no particular problems,
we can DTRT to put it in either GNU ELPA or NonGNU ELPA.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package-vc has been merged

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Please do not encourage people to load packages from MELPA.  MELPA
> does not cooperate with us.  Not in legal matters, not in ethical
> matters, and not in technical matters of development.

MELPA is not mentioned anywhere, and no special accommodations have been
made for MELPA (or other archive for that matter).  That being said, I
would like to see MELPA adding support for the package specification
format that this features has introduced.  I am doubtful this will
happen any time soon, because their "recipe language" and the ELPA
package specification language differ, so someone will have to figure
out how to translate the one onto the other.

All package-vc does support is installing a package directly from a URL.
This might even help people who "rely" on MELPA to drop the archive,
because it makes it easier to "manually" install those one or two
packages that haven't been added to {Non,}GNU ELPA yet.

> A given package that happens to be in MELPA may be perfectly fine in
> and of itself, or it may have problems of one kind of the other.
>
> If you come across a package in MELPA that has no particular problems,
> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.

I have been doing that for over a year now.  All, but a negligible
amount, of packages on MELPA are legally/ethically fine.  The quality
differs a lot, but in principle it would be possible to add everything
to NonGNU ELPA.

Re: feature/package-vc has been merged

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I'm sorry if you thought that my message against recommending MELPA
was directed at you:

  > > Please do not encourage people to load packages from MELPA.  MELPA
  > > does not cooperate with us.  Not in legal matters, not in ethical
  > > matters, and not in technical matters of development.

I was replying to a message from someone else which mentioned MELPA in
glowing terms.  It's not your doing.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package-vc has been merged

[Björn Bidar]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Please do not encourage people to load packages from MELPA.  MELPA
> does not cooperate with us.  Not in legal matters, not in ethical
> matters, and not in technical matters of development.

What justifies this kind of gaslighting against Melpa? You might not
like to hear it but without Melpa Emacs wouldn't be were it is now..

> A given package that happens to be in MELPA may be perfectly fine in
> and of itself, or it may have problems of one kind of the other.
>
> If you come across a package in MELPA that has no particular problems,
> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.

It's perfectly fine that is on Melpa, not everyone likes the mailing
list based approach of Gnu.
Offer other options such as a Gitlab or Gitea instance instead of
antiquated Savanah (or make it more modern in other ways)
and people might move elsewhere.

Br,

Björn

Re: feature/package-vc has been merged

[tomas]

[-- Attachment #1: Type: text/plain, Size: 190 bytes --]

On Fri, Nov 11, 2022 at 08:44:34PM +0200, Björn Bidar wrote:

[...]

> Offer other options such as a Gitlab or Gitea instance [...]

Shudder.

(now we're 1:1 ;-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: feature/package-vc has been merged

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What justifies this kind of gaslighting against Melpa? 

That is a nasty thing to say, because "Gaslighting" means a kind of
lying.  Our objections to MELPA are about MELPA's policies and
actions.  You can verify for yourself what we say.

We reject MELPA because it includes packages that do not meet our
standards.  For instance, it accepts packages that violate our our
moral standard: nonfree programs are an injustice and it is wrong to
recommend people use one.  MELPA includes packages that recommend
specific nonfree programs.

We are fighting to get rid of nonfree software, so we have a policy
that we don't refer people to sites that recommend nonfree programs.

You don't have to agree with our policies or the values they are based
on.  But this list is the wrong place to argue against them.  If you
want to do that, you can do it on gnu-misc-discuss@gnu.org.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package-vc has been merged

[Björn Bidar]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > What justifies this kind of gaslighting against Melpa? 
>
> That is a nasty thing to say, because "Gaslighting" means a kind of
> lying.  Our objections to MELPA are about MELPA's policies and
> actions.  You can verify for yourself what we say.
> We reject MELPA because it includes packages that do not meet our
> standards.  For instance, it accepts packages that violate our our
> moral standard: nonfree programs are an injustice and it is wrong to
> recommend people use one.  MELPA includes packages that recommend
> specific nonfree programs.


Yes as you were implying that all programs on Melpa are bad, as said
earlier a lot of reasons why people use Melpa is because there's no
other option provided by Emacs/Gnu to contribute in other ways than
mailing-list.
While for some that is for many it is not.

> moral standard: nonfree programs are an injustice and it is wrong to
> recommend people use one.  MELPA includes packages that recommend
> specific nonfree programs.

I don't recommend non free programs, Melpa doesn't even contain non free programs.

You reject it, you can't speak for everyone but I assume you speak for
yourself and the Gnu project.

> We are fighting to get rid of nonfree software, so we have a policy
> that we don't refer people to sites that recommend nonfree programs.

I get that people need to be able to interact with these kinds of
platfroms still. While we probably all non free programs will cease to
exist this won't happen any day. Either you can use Emacs in your day to
day work or you can't.

> You don't have to agree with our policies or the values they are based
> on.  But this list is the wrong place to argue against them.  If you
> want to do that, you can do it on gnu-misc-discuss@gnu.org.

Well you started it.

Your narrow minded view hurts your mission in the end..
Gplv3 all over again..

Sometimes it might not hurt to argue a little diplomatic.

Br,

Björn

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: emacs-devel@gnu.org
> Date: Sat, 12 Nov 2022 08:30:39 +0200
> 
> Your narrow minded view hurts your mission in the end..
> Gplv3 all over again..

Warning: this kind of personal attacks are not tolerated here.  Please
avoid that in the future.

Re: feature/package-vc has been merged

[Björn Bidar]

I didn't mean to attack but to criticize him, I didn't mean to
personally attack him with my language, I'm sorry.

Br,

Björn

Re: feature/package-vc has been merged

[Björn Bidar]

I didn't mean to attack but to criticize him, I didn't mean to
personally attack him with my language, I'm sorry.

Br,

Björn

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>> Please do not encourage people to load packages from MELPA.  MELPA
>> does not cooperate with us.  Not in legal matters, not in ethical
>> matters, and not in technical matters of development.
>
> What justifies this kind of gaslighting against Melpa? 

Wikipedia defines gaslighting as:

    Gaslighting is a colloquialism, loosely defined as manipulating
    someone so as to make them question their own reality [...]

so I am not sure how this applies to this thread.

>                                                        You might not
> like to hear it but without Melpa Emacs wouldn't be were it is now..

This is a counterfactual discussion, because it cannot be said if MELPA
was a necessary or contingent fact.  I agree that MELPA provided an
important service in collecting the number of packages that it did, but
if NonGNU ELPA had been created over 10 years ago with the regular GNU
ELPA, perhaps it would have been enough?

That being said, if I had a single-use time machine I wouldn't waste it
on finding out insignificant something like this.

>> A given package that happens to be in MELPA may be perfectly fine in
>> and of itself, or it may have problems of one kind of the other.
>>
>> If you come across a package in MELPA that has no particular problems,
>> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.
>
> It's perfectly fine that is on Melpa, not everyone likes the mailing
> list based approach of Gnu.
> Offer other options such as a Gitlab or Gitea instance instead of
> antiquated Savanah (or make it more modern in other ways)
> and people might move elsewhere.

I am afraid you have some misunderstandings regarding GNU ELPA (and I
suppose NonGNU ELPA as well).  GNU ELPA packages can and often are
developed on PR-based forges, where the state is synchronised into
elpa.git/nongnu.git, where the packages are build and distributed.
There is no need to use mailing lists -- except maybe to announce a
package and to request it be added to an archive.  But am I understating
your correctly that that is really the point you are objecting to?

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Richard Stallman <rms@gnu.org> writes:
>>
>>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>> [[[ whether defending the US Constitution against all enemies,     ]]]
>>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>>
>>> Please do not encourage people to load packages from MELPA.  MELPA
>>> does not cooperate with us.  Not in legal matters, not in ethical
>>> matters, and not in technical matters of development.
>>
>> What justifies this kind of gaslighting against Melpa? 
>
> Wikipedia defines gaslighting as:
>
>     Gaslighting is a colloquialism, loosely defined as manipulating
>     someone so as to make them question their own reality [...]
>
> so I am not sure how this applies to this thread.

I'm sorry but English isn't my mother tongue.. From my pov he wrote
misleading statements about Melpa which did sound like gaslighting to me.

>>                                                        You might not
>> like to hear it but without Melpa Emacs wouldn't be were it is now..
>
> This is a counterfactual discussion, because it cannot be said if MELPA
> was a necessary or contingent fact.  I agree that MELPA provided an
> important service in collecting the number of packages that it did, but
> if NonGNU ELPA had been created over 10 years ago with the regular GNU
> ELPA, perhaps it would have been enough?

Some have issues with the FSF, RMS etc. staying out of the whole thing
was convenient for some.
Even if you ignore that Melpa was more convinient to use unless there's
a more modern way to interact to with ELPA.

> That being said, if I had a single-use time machine I wouldn't waste it
> on finding out insignificant something like this.
>

Nothing to argue about that.

>>> A given package that happens to be in MELPA may be perfectly fine in
>>> and of itself, or it may have problems of one kind of the other.
>>>
>>> If you come across a package in MELPA that has no particular problems,
>>> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.
>>
>> It's perfectly fine that is on Melpa, not everyone likes the mailing
>> list based approach of Gnu.
>> Offer other options such as a Gitlab or Gitea instance instead of
>> antiquated Savanah (or make it more modern in other ways)
>> and people might move elsewhere.
>
> I am afraid you have some misunderstandings regarding GNU ELPA (and I
> suppose NonGNU ELPA as well).  GNU ELPA packages can and often are
> developed on PR-based forges, where the state is synchronised into
> elpa.git/nongnu.git, where the packages are build and distributed.
> There is no need to use mailing lists -- except maybe to announce a
> package and to request it be added to an archive.  But am I understating
> your correctly that that is really the point you are objecting to?

I'm sorry I wasn't aware of that, I assumed that using Github to develop
the package is enough to disqualify it.

I am objecting against the assumption Melpa equals bad. I can understand
the issue with some of it's packages or even the place of distribution
but it hard to replace a platform like Github for the network effect it
has.

Br,

Björn

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Richard Stallman <rms@gnu.org>,  stefankangas@gmail.com,
>   monnier@iro.umontreal.ca,  eliz@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 12 Nov 2022 15:01:24 +0200
> 
> I am objecting against the assumption Melpa equals bad. I can understand
> the issue with some of it's packages or even the place of distribution
> but it hard to replace a platform like Github for the network effect it
> has.

No one says MELPA equals bad, and Github is not the issue here.  We
just avoid recommending people to use MELPA because doing so could
accidentally cause someone to install non-free packages, or packages
that proliferate or require non-free programs.  It's a preference
thing, not a damnation.

Re: feature/package-vc has been merged

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Richard Stallman <rms@gnu.org>,  stefankangas@gmail.com,
>>   monnier@iro.umontreal.ca,  eliz@gnu.org,  emacs-devel@gnu.org
>> Date: Sat, 12 Nov 2022 15:01:24 +0200
>> 
>> I am objecting against the assumption Melpa equals bad. I can understand
>> the issue with some of it's packages or even the place of distribution
>> but it hard to replace a platform like Github for the network effect it
>> has.
>
> No one says MELPA equals bad, and Github is not the issue here.  We

Than I misunderstood RMS when he said:
>>Please do not encourage people to load packages from MELPA.

> just avoid recommending people to use MELPA because doing so could
> accidentally cause someone to install non-free packages, or packages
> that proliferate or require non-free programs.  It's a preference
> thing, not a damnation.

I wasn't aware of packages in MELPA that do such a thing, especially
about non-free packages.

From my knowledge Melpa doesn't contain those packages nomore than Elpa
already does (Microsoft Exchange support, excorporate).

Can you name such packages?

In the end people either can use Emacs on day to day work where they are
forced to use such systems that interact with non-free services or not.

Br,

Björn

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: philipk@posteo.net,  rms@gnu.org,  stefankangas@gmail.com,
>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 12 Nov 2022 15:41:36 +0200
> 
> > No one says MELPA equals bad, and Github is not the issue here.  We
> 
> Than I misunderstood RMS when he said:
> >>Please do not encourage people to load packages from MELPA.

"Not to encourage" is a far cry from "bad".

> > just avoid recommending people to use MELPA because doing so could
> > accidentally cause someone to install non-free packages, or packages
> > that proliferate or require non-free programs.  It's a preference
> > thing, not a damnation.
> 
> I wasn't aware of packages in MELPA that do such a thing, especially
> about non-free packages.
> 
> >From my knowledge Melpa doesn't contain those packages nomore than Elpa
> already does (Microsoft Exchange support, excorporate).
> 
> Can you name such packages?

No, I cannot.  I don't track MELPA.  You can find them in past
discussions, though.

> In the end people either can use Emacs on day to day work where they are
> forced to use such systems that interact with non-free services or not.

We don't (and cannot) force people to do (or avoid doing) anything,
neither can we hide the fact that MELPA exists.  We just _encourage_
them to try to use only free software.  It's a difference between
expressing an opinion and violently enforcing that opinion on others.

Re: feature/package-vc has been merged

[Po Lu]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> I'm sorry but English isn't my mother tongue.. From my pov he wrote
> misleading statements about Melpa which did sound like gaslighting to me.

The GNU project exists to provide a free operating system.  In addition,
it has the larger goal of eliminating non-free software.

By allowing packages that promote the use of non-free software (and even
non-free operating systems!), and enabling their convenient installation
from Emacs, the MELPA developers are making it harder for us to achieve
that goal.  So please don't be surprised if we are not willing to refer
our users to them.

> Some have issues with the FSF, RMS etc. staying out of the whole thing
> was convenient for some.

Those ``issues'' held by ``some'' do not influence decision-making on
this list much, as they simply are of no relevance to Emacs, or free
software in general.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Richard Stallman <rms@gnu.org> writes:
>>>
>>>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>>> [[[ whether defending the US Constitution against all enemies,     ]]]
>>>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>>>
>>>> Please do not encourage people to load packages from MELPA.  MELPA
>>>> does not cooperate with us.  Not in legal matters, not in ethical
>>>> matters, and not in technical matters of development.
>>>
>>> What justifies this kind of gaslighting against Melpa? 
>>
>> Wikipedia defines gaslighting as:
>>
>>     Gaslighting is a colloquialism, loosely defined as manipulating
>>     someone so as to make them question their own reality [...]
>>
>> so I am not sure how this applies to this thread.
>
> I'm sorry but English isn't my mother tongue.. From my pov he wrote
> misleading statements about Melpa which did sound like gaslighting to me.

Forgive me for guessing, but could your native language be German?  I'm
just inferring from the name.  If so, what did you want to say?
Vielleicht verstehe ich so besser was du meinst?

>>>                                                        You might not
>>> like to hear it but without Melpa Emacs wouldn't be were it is now..
>>
>> This is a counterfactual discussion, because it cannot be said if MELPA
>> was a necessary or contingent fact.  I agree that MELPA provided an
>> important service in collecting the number of packages that it did, but
>> if NonGNU ELPA had been created over 10 years ago with the regular GNU
>> ELPA, perhaps it would have been enough?
>
> Some have issues with the FSF, RMS etc. staying out of the whole thing
> was convenient for some.
> Even if you ignore that Melpa was more convinient to use unless there's
> a more modern way to interact to with ELPA.

I have floated the idea of creating an Emacs package for submitting ELPA
packages, that would help automatise the repetitive questions, such as
have you signed the FSF CA if you want to add a package to GNU ELPA, are
all the dependencies available, has basic code style been respected that
checkdoc and byte compilation can detect, etc.  

Another idea I have heard been suggested is creating a separate issue
tracker for ELPA submissions and issues.  I am not sure if this would
help that much, but I guess some people avoid the mailing list because
they don't want to initiate a long discussion.

>> That being said, if I had a single-use time machine I wouldn't waste it
>> on finding out insignificant something like this.
>>
>
> Nothing to argue about that.
>
>>>> A given package that happens to be in MELPA may be perfectly fine in
>>>> and of itself, or it may have problems of one kind of the other.
>>>>
>>>> If you come across a package in MELPA that has no particular problems,
>>>> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.
>>>
>>> It's perfectly fine that is on Melpa, not everyone likes the mailing
>>> list based approach of Gnu.
>>> Offer other options such as a Gitlab or Gitea instance instead of
>>> antiquated Savanah (or make it more modern in other ways)
>>> and people might move elsewhere.
>>
>> I am afraid you have some misunderstandings regarding GNU ELPA (and I
>> suppose NonGNU ELPA as well).  GNU ELPA packages can and often are
>> developed on PR-based forges, where the state is synchronised into
>> elpa.git/nongnu.git, where the packages are build and distributed.
>> There is no need to use mailing lists -- except maybe to announce a
>> package and to request it be added to an archive.  But am I understating
>> your correctly that that is really the point you are objecting to?
>
> I'm sorry I wasn't aware of that, I assumed that using Github to develop
> the package is enough to disqualify it.

No, that is the great thing about Git.  I can clone and hack on a
package that is hosted on GitHub, without ever having to accept the
execution on Non Free Javascript on my device.  Sure, the GNU project
would advise against using GitHub for several reasons, but as long as
you don't force others to use Non-Free Software, it is not a
deal-breaker.

Just take a look at the current list of packages included in ELPA:

  https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/elpa-packages

There are plenty of packages that are developed on GitHub or GitLab.
Almost none are currently maintained on Savannah.  Luckily more and more
are also appearing on freedom respecting sites like Sourcehut.

(I really don't know where this kind of misinformation stems from.  I
have heard it too, and was scared at first.  But it turns out that
people who haven't quite understand the arguments keep arguing against
strawmen in their own minds.)

> I am objecting against the assumption Melpa equals bad. I can understand
> the issue with some of it's packages or even the place of distribution
> but it hard to replace a platform like Github for the network effect it
> has.

The issue was just that Emacs doesn't want to refer to MELPA, because
the two projects clash in their respective interests.  My understanding
is that MELPA tries to be exhaustive, while Emacs/ELPA prioritises that
all software can be used without loss of functionality on a fully free
system.  A choice has to be made.

IMO this is often the result of "bad" software choices.  The point is
not to ignore non-free software and pretend it doesn't exist.
Proprietary software is a means of exercising control over a user, and
some people are stuck in dominating environments, where the lack of
software freedom is symptomatic for their entire predicament, not
necessarily the cause of it.

I always like the example of browse-url, which has a user option
`browse-url-browser-function'.  Among other things, you could set it to
the function `browse-url-chrome'.  But wait, isn't Chrome the famous
non-free browser that spies on its users and one day might even make
watching an advisement mandatory?  Sure, but all browse-url does it
provide a generic way of opening a URL in some external program.  If the
user has to use Chrome, that is bad, but they don't have much of a
choice.  But for free people, at home or in less restrictive
environments this doesn't make their "browse-url'ing" any worse.  Chrome
is a possible, but not a necessary way to implement the feature.  I
still get to use Firefox or eww.  So everyone is happy, because the
functionality is generic and not called something like
"browse-using-google-chrome-and-only-google-chrome" -- which wouldn't
even make sense in this context.

The simple fact is that MELPA insists on distributing software that make
these mistakes instead of trying to find a compromise position that can
help people bound to non-free systems make the most out of Emacs, while
not placing the rest at a functional disadvantage.

In my eyes this is more than reasonable.

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>
>>>> Richard Stallman <rms@gnu.org> writes:
>>>>
>>>>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>>>> [[[ whether defending the US Constitution against all enemies,     ]]]
>>>>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>>>>
>>>>> Please do not encourage people to load packages from MELPA.  MELPA
>>>>> does not cooperate with us.  Not in legal matters, not in ethical
>>>>> matters, and not in technical matters of development.
>>>>
>>>> What justifies this kind of gaslighting against Melpa? 
>>>
>>> Wikipedia defines gaslighting as:
>>>
>>>     Gaslighting is a colloquialism, loosely defined as manipulating
>>>     someone so as to make them question their own reality [...]
>>>
>>> so I am not sure how this applies to this thread.
>>
>> I'm sorry but English isn't my mother tongue.. From my pov he wrote
>> misleading statements about Melpa which did sound like gaslighting to me.
>
> Forgive me for guessing, but could your native language be German?  I'm
> just inferring from the name.  If so, what did you want to say?
> Vielleicht verstehe ich so besser was du meinst?

Ja meine Muttersprache ist Deutsch, vielleicht geht es besser so.
Ich habe es so verstanden das man Melpa nicht nennen sollte weil Melpa
nicht kooperiert mit Gnu, was meiner Meinung nach nicht war ist
bzw. mir neu wäre. 

Beide Quellen enhalten die nicht freie Software verwenden (Melpa:
Lastpass, Elpa Excange support).

Dadurch fällt mir als einziger Unterschied Github bzw. Forge basierte
Entwicklung und Mailinglisten basiertes Model.

OT: Ich spreche Deutsch selten in den letzten Tagen, habe generell
manchmal Probleme mich auszudrücken es ist nicht nur das Englische, AHDS
sei dank.

>>>>                                                        You might not
>>>> like to hear it but without Melpa Emacs wouldn't be were it is now..
>>>
>>> This is a counterfactual discussion, because it cannot be said if MELPA
>>> was a necessary or contingent fact.  I agree that MELPA provided an
>>> important service in collecting the number of packages that it did, but
>>> if NonGNU ELPA had been created over 10 years ago with the regular GNU
>>> ELPA, perhaps it would have been enough?
>>
>> Some have issues with the FSF, RMS etc. staying out of the whole thing
>> was convenient for some.
>> Even if you ignore that Melpa was more convinient to use unless there's
>> a more modern way to interact to with ELPA.
>
> I have floated the idea of creating an Emacs package for submitting ELPA
> packages, that would help automatise the repetitive questions, such as
> have you signed the FSF CA if you want to add a package to GNU ELPA, are
> all the dependencies available, has basic code style been respected that
> checkdoc and byte compilation can detect, etc.  

That sounds like a good idea, some kind of CI that checks the packages
would be nice, the Ci can run on the creation of a request or on
whitelist.

I think for a lot of people the way that the FSF acts or just the name
leaves a bad taste in their mouth. Personally I think it quite sad that
there isn't more corporation, I wish the FSF and FSFE would push for
more free software in government and elsewhere around the world.
In a lot environments uncertainty around free software especially after
GPLv3 was released created by issues.
A lot of places I've worked at had almost an allergy against things such
as GPLv3.

> Another idea I have heard been suggested is creating a separate issue
> tracker for ELPA submissions and issues.  I am not sure if this would
> help that much, but I guess some people avoid the mailing list because
> they don't want to initiate a long discussion.

If debbugs would be list a little modern such things would be easier,
just create a bug at the Gnu bugtracker under the ELPA product. 

>>
>> Nothing to argue about that.
>>
>>>>> A given package that happens to be in MELPA may be perfectly fine in
>>>>> and of itself, or it may have problems of one kind of the other.
>>>>>
>>>>> If you come across a package in MELPA that has no particular problems,
>>>>> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.
>>>>
>>>> It's perfectly fine that is on Melpa, not everyone likes the mailing
>>>> list based approach of Gnu.
>>>> Offer other options such as a Gitlab or Gitea instance instead of
>>>> antiquated Savanah (or make it more modern in other ways)
>>>> and people might move elsewhere.
>>>
>>> I am afraid you have some misunderstandings regarding GNU ELPA (and I
>>> suppose NonGNU ELPA as well).  GNU ELPA packages can and often are
>>> developed on PR-based forges, where the state is synchronised into
>>> elpa.git/nongnu.git, where the packages are build and distributed.
>>> There is no need to use mailing lists -- except maybe to announce a
>>> package and to request it be added to an archive.  But am I understating
>>> your correctly that that is really the point you are objecting to?
>>
>> I'm sorry I wasn't aware of that, I assumed that using Github to develop
>> the package is enough to disqualify it.
>
> No, that is the great thing about Git.  I can clone and hack on a
> package that is hosted on GitHub, without ever having to accept the
> execution on Non Free Javascript on my device.  Sure, the GNU project
> would advise against using GitHub for several reasons, but as long as
> you don't force others to use Non-Free Software, it is not a
> deal-breaker.
>
> Just take a look at the current list of packages included in ELPA:
>
>   https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/elpa-packages
>
> There are plenty of packages that are developed on GitHub or GitLab.
> Almost none are currently maintained on Savannah.  Luckily more and more
> are also appearing on freedom respecting sites like Sourcehut.
>
> (I really don't know where this kind of misinformation stems from.  I
> have heard it too, and was scared at first.  But it turns out that
> people who haven't quite understand the arguments keep arguing against
> strawmen in their own minds.)

Yeah I understand that, I use Git in a similar way,  I have my own
mirrors but use Gitlab/Github for the network effect in the communities
I need it.

But the misinformation came at least from my side out of the issue
that I wasn't aware that Melpa contains packages that engages with
non-free software at least not to the extend that Emacs already does.
Like there are Windows build for macos and Windows, Melpa contains
packages for that interact with such operating systems in the same way.

So Github was only remaining thing that is left as an issue.
To be honest it makes sense since relaying it as a central hub is just
bad no matter your position of free software..

>> I am objecting against the assumption Melpa equals bad. I can understand
>> the issue with some of it's packages or even the place of distribution
>> but it hard to replace a platform like Github for the network effect it
>> has.
>
> The issue was just that Emacs doesn't want to refer to MELPA, because
> the two projects clash in their respective interests.  My understanding
> is that MELPA tries to be exhaustive, while Emacs/ELPA prioritises that
> all software can be used without loss of functionality on a fully free
> system.  A choice has to be made.
> IMO this is often the result of "bad" software choices.  The point is
> not to ignore non-free software and pretend it doesn't exist.
> Proprietary software is a means of exercising control over a user, and
> some people are stuck in dominating environments, where the lack of
> software freedom is symptomatic for their entire predicament, not
> necessarily the cause of it.
>

It is not just bad software choices but also idealism vs reality.
I can try to change the predicament that I'm tied to some non free
programs or system but at some point my means are exhausted.
First I need to have the means to do it, for example: I'm a software
engineer I try to find alternatives, setup my own systems if needed and
find out what is the best tool for what I want to do.
But a lot of people don't have that power either because they don't have
the resources or their environment forces them.
For example at work or because the government doesn't offer free
alternatives.

I respect people such as RMS for sacrificing the convenience of using
only free systems but I think that doesn't work for most.

So to be able to keep using free software their are some Emacs packages
or programs that interface with non-free systems. Referencing Melpa for
such packages seem It is not just bad software choices but also idealism vs reality.
I can try to change the predicament that I'm tied to some non free
programs or system but at some point my means are exhausted.
First I need to have the means to do it, for example: I'm a software
engineer I try to find alternatives, setup my own systems if needed and
find out what is the best tool for what I want to do.
But a lot of people don't have that power either because they don't have
the resources or their environment forces them.
For example at work or because the government doesn't offer free
alternatives.

I respect people such as RMS for sacrificing the convenience of using
only free systems but I think that doesn't work for most.

So to be able to keep using free software their are some Emacs packages
or programs that interface with non-free systems. Referencing Melpa for
such packages seem fine for me, except for instances where Elpa contains
these packages themselves such as for Exchange support (Excorporate).

That Emacs supports Windows, MacOS or other non-free platforms has a very
similar reason I think.fine for me, except for instances where Elpa contains
these packages themselves such as for Exchange support (Excorporate).

That Emacs supports Windows, MacOS or other non-free platforms has a very
similar reason I think.


> I always like the example of browse-url, which has a user option
> `browse-url-browser-function'.  Among other things, you could set it to
> the function `browse-url-chrome'.  But wait, isn't Chrome the famous
> non-free browser that spies on its users and one day might even make
> watching an advisement mandatory?  Sure, but all browse-url does it
> provide a generic way of opening a URL in some external program.  If the
> user has to use Chrome, that is bad, but they don't have much of a
> choice.  But for free people, at home or in less restrictive
> environments this doesn't make their "browse-url'ing" any worse.  Chrome
> is a possible, but not a necessary way to implement the feature.  I
> still get to use Firefox or eww.  So everyone is happy, because the
> functionality is generic and not called something like
> "browse-using-google-chrome-and-only-google-chrome" -- which wouldn't
> even make sense in this context.

Yeah it is kinda strange, like why don't use your operating system for
such choices, no need to have a function for that.

> The simple fact is that MELPA insists on distributing software that make
> these mistakes instead of trying to find a compromise position that can
> help people bound to non-free systems make the most out of Emacs, while
> not placing the rest at a functional disadvantage.
>
> In my eyes this is more than reasonable.

I think people should have the ability to choose what they want to use,
so unless the package is malicious I see no issue with people using a
package.
Doesn't mean that I like it thou.


Thanks for talking Br,

Björn

Re: feature/package-vc has been merged

[Stefan Monnier]

> Beide Quellen enhalten die nicht freie Software verwenden (Melpa:
> Lastpass, Elpa Excange support).

FWIW, the reasoning behind accepting a package which requires an
Exchange server, is that the Exchange server runs on another machine
over which you have no control anyway, so whether it runs Free software
or proprietary software does not make any difference to you.

In other words, the concern w.r.t Free Software is whether the software
*you* run is Free.

I don't fully agree with the distinction, FWIW.


        Stefan

Re: feature/package-vc has been merged

[Björn Bidar]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Beide Quellen enhalten die nicht freie Software verwenden (Melpa:
>> Lastpass, Elpa Excange support).
>
> FWIW, the reasoning behind accepting a package which requires an
> Exchange server, is that the Exchange server runs on another machine
> over which you have no control anyway, so whether it runs Free software
> or proprietary software does not make any difference to you.

Ah that explain is so any package that interacts with non-free cloud
system is fine e.g. twitter-mode, spotify etc.

> In other words, the concern w.r.t Free Software is whether the software
> *you* run is Free.
>

That's the case for both Elpa and Melpa that was the original reason why
I didn't good the distinction initially.

Br,

Björn

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>
>>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>>
>>>>> Richard Stallman <rms@gnu.org> writes:
>>>>>
>>>>>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>>>>> [[[ whether defending the US Constitution against all enemies,     ]]]
>>>>>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>>>>>
>>>>>> Please do not encourage people to load packages from MELPA.  MELPA
>>>>>> does not cooperate with us.  Not in legal matters, not in ethical
>>>>>> matters, and not in technical matters of development.
>>>>>
>>>>> What justifies this kind of gaslighting against Melpa? 
>>>>
>>>> Wikipedia defines gaslighting as:
>>>>
>>>>     Gaslighting is a colloquialism, loosely defined as manipulating
>>>>     someone so as to make them question their own reality [...]
>>>>
>>>> so I am not sure how this applies to this thread.
>>>
>>> I'm sorry but English isn't my mother tongue.. From my pov he wrote
>>> misleading statements about Melpa which did sound like gaslighting to me.
>>
>> Forgive me for guessing, but could your native language be German?  I'm
>> just inferring from the name.  If so, what did you want to say?
>> Vielleicht verstehe ich so besser was du meinst?
>
> Ja meine Muttersprache ist Deutsch, vielleicht geht es besser so.
> Ich habe es so verstanden das man Melpa nicht nennen sollte weil Melpa
> nicht kooperiert mit Gnu, was meiner Meinung nach nicht war ist
> bzw. mir neu wäre. 
>
> Beide Quellen enhalten die nicht freie Software verwenden (Melpa:
> Lastpass, Elpa Excange support).

Ich verstehe deinen Satzbau hier nicht, anstatt einem Objekt nach
"enthalten," fängt ein Untersatz an ", die nicht...".  Willst du sagen,
dass beide Archive nur Freie Software enthalten?

> Dadurch fällt mir als einziger Unterschied Github bzw. Forge basierte
> Entwicklung und Mailinglisten basiertes Model.
>
> OT: Ich spreche Deutsch selten in den letzten Tagen, habe generell
> manchmal Probleme mich auszudrücken es ist nicht nur das Englische, AHDS
> sei dank.

I'll translate this for the others on the list:

     My native language is German, so perhaps this is better.  I
     understood that one shouldn't mention MELPA because MELPA doesn't
     cooperate with GNU, which in my opinion isn't true or rather would
     be new to me.

MELPA ist ein eigenständiges Projekt welches seine eigenen Ziele hat,
welche nicht zu 100% mit dem GNU Projekt im Einklang stehen.  Es gibt
fundamentale Meinungsverschiedenheiten welche seit Jahren bestehen.

MELPA is an independent Projekt with their own Goals.  There aren't
100% aligned with those of the GNU Project.  There have been fundamental
differences of opinion that have existed for years.

     Both sources contain which don't use Free Software (MELPA:
     Lastpass, ELPA Exchange support) [this is a literal translation, I
     can't make sense of the sentence in German either].

     Thereby the only difference I can notice is GitHub, or rather a
     Forge-based development vs. a mailing list model.

Abgesehen von den technischen unterschieden, gibt es verschiedene
Guidelines zwischen den Projekten, welche Pakete angenommen werden.
Dahingegen sind die Fragen der Umsetzung (Mailing List vs PR-basierte
Webseite) recht nebensächlich.

Setting aside technical differences, there are different Guidelines what
packages are accepted.  Compared to that the question of how the
projects are organised (Mailing List vs PR-based Website) is not that
important.

GNU ELPA: https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/README#n330
MELPA: https://raw.githubusercontent.com/melpa/melpa/master/CONTRIBUTING.org

>>>>>                                                        You might not
>>>>> like to hear it but without Melpa Emacs wouldn't be were it is now..
>>>>
>>>> This is a counterfactual discussion, because it cannot be said if MELPA
>>>> was a necessary or contingent fact.  I agree that MELPA provided an
>>>> important service in collecting the number of packages that it did, but
>>>> if NonGNU ELPA had been created over 10 years ago with the regular GNU
>>>> ELPA, perhaps it would have been enough?
>>>
>>> Some have issues with the FSF, RMS etc. staying out of the whole thing
>>> was convenient for some.
>>> Even if you ignore that Melpa was more convinient to use unless there's
>>> a more modern way to interact to with ELPA.
>>
>> I have floated the idea of creating an Emacs package for submitting ELPA
>> packages, that would help automatise the repetitive questions, such as
>> have you signed the FSF CA if you want to add a package to GNU ELPA, are
>> all the dependencies available, has basic code style been respected that
>> checkdoc and byte compilation can detect, etc.  
>
> That sounds like a good idea, some kind of CI that checks the packages
> would be nice, the Ci can run on the creation of a request or on
> whitelist.

That can be difficult on a free-form mailing list like this one, unless
there is some formal indication (which is something a package like this
could provide).

> I think for a lot of people the way that the FSF acts or just the name
> leaves a bad taste in their mouth. Personally I think it quite sad that
> there isn't more corporation, I wish the FSF and FSFE would push for
> more free software in government and elsewhere around the world.
> In a lot environments uncertainty around free software especially after
> GPLv3 was released created by issues.
> A lot of places I've worked at had almost an allergy against things such
> as GPLv3.

I wish the entire GNU Project would be more integrated towards the
creation of a GNU Operating system, but that really is an off-topic
issue for this list.

>> Another idea I have heard been suggested is creating a separate issue
>> tracker for ELPA submissions and issues.  I am not sure if this would
>> help that much, but I guess some people avoid the mailing list because
>> they don't want to initiate a long discussion.
>
> If debbugs would be list a little modern such things would be easier,
> just create a bug at the Gnu bugtracker under the ELPA product. 

What do you have in mind specifically when you say "modern"?

The Guix people have been using a separate different front end that
/looks/ more modern, that still is debbugs AFAIK:
https://issues.guix.gnu.org/, and the source code is here:
https://git.elephly.net/gitweb.cgi?p=software/mumi.git.

>>>>>> A given package that happens to be in MELPA may be perfectly fine in
>>>>>> and of itself, or it may have problems of one kind of the other.
>>>>>>
>>>>>> If you come across a package in MELPA that has no particular problems,
>>>>>> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.
>>>>>
>>>>> It's perfectly fine that is on Melpa, not everyone likes the mailing
>>>>> list based approach of Gnu.
>>>>> Offer other options such as a Gitlab or Gitea instance instead of
>>>>> antiquated Savanah (or make it more modern in other ways)
>>>>> and people might move elsewhere.
>>>>
>>>> I am afraid you have some misunderstandings regarding GNU ELPA (and I
>>>> suppose NonGNU ELPA as well).  GNU ELPA packages can and often are
>>>> developed on PR-based forges, where the state is synchronised into
>>>> elpa.git/nongnu.git, where the packages are build and distributed.
>>>> There is no need to use mailing lists -- except maybe to announce a
>>>> package and to request it be added to an archive.  But am I understating
>>>> your correctly that that is really the point you are objecting to?
>>>
>>> I'm sorry I wasn't aware of that, I assumed that using Github to develop
>>> the package is enough to disqualify it.
>>
>> No, that is the great thing about Git.  I can clone and hack on a
>> package that is hosted on GitHub, without ever having to accept the
>> execution on Non Free Javascript on my device.  Sure, the GNU project
>> would advise against using GitHub for several reasons, but as long as
>> you don't force others to use Non-Free Software, it is not a
>> deal-breaker.
>>
>> Just take a look at the current list of packages included in ELPA:
>>
>>   https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/elpa-packages
>>
>> There are plenty of packages that are developed on GitHub or GitLab.
>> Almost none are currently maintained on Savannah.  Luckily more and more
>> are also appearing on freedom respecting sites like Sourcehut.
>>
>> (I really don't know where this kind of misinformation stems from.  I
>> have heard it too, and was scared at first.  But it turns out that
>> people who haven't quite understand the arguments keep arguing against
>> strawmen in their own minds.)
>
> Yeah I understand that, I use Git in a similar way,  I have my own
> mirrors but use Gitlab/Github for the network effect in the communities
> I need it.
>
> But the misinformation came at least from my side out of the issue
> that I wasn't aware that Melpa contains packages that engages with
> non-free software at least not to the extend that Emacs already does.
> Like there are Windows build for macos and Windows, Melpa contains
> packages for that interact with such operating systems in the same way.

Richard went into that issue in a parallel thread just yesterday:
https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg00792.html:

  Our general policy makes a subtle distinction between these two cases:

  1. If a nonfree program FOO is not well known, we don't even mention that
  it exists.  Because we don't want to promote using FOO.

  2. If a nonfree program FOO is well known and widely used, something to
  help and encourage FOO's users to use some GNU packages along with FOO
  is good.

  3. Anything that would encourage the existing users of some GNU packages
  to use FOO with them is bad.

> So Github was only remaining thing that is left as an issue.
> To be honest it makes sense since relaying it as a central hub is just
> bad no matter your position of free software..
>
>>> I am objecting against the assumption Melpa equals bad. I can understand
>>> the issue with some of it's packages or even the place of distribution
>>> but it hard to replace a platform like Github for the network effect it
>>> has.
>>
>> The issue was just that Emacs doesn't want to refer to MELPA, because
>> the two projects clash in their respective interests.  My understanding
>> is that MELPA tries to be exhaustive, while Emacs/ELPA prioritises that
>> all software can be used without loss of functionality on a fully free
>> system.  A choice has to be made.
>> IMO this is often the result of "bad" software choices.  The point is
>> not to ignore non-free software and pretend it doesn't exist.
>> Proprietary software is a means of exercising control over a user, and
>> some people are stuck in dominating environments, where the lack of
>> software freedom is symptomatic for their entire predicament, not
>> necessarily the cause of it.
>
> It is not just bad software choices but also idealism vs reality.
> I can try to change the predicament that I'm tied to some non free
> programs or system but at some point my means are exhausted.
> First I need to have the means to do it, for example: I'm a software
> engineer I try to find alternatives, setup my own systems if needed and
> find out what is the best tool for what I want to do.
> But a lot of people don't have that power either because they don't have
> the resources or their environment forces them.
> For example at work or because the government doesn't offer free
> alternatives.
>
> I respect people such as RMS for sacrificing the convenience of using
> only free systems but I think that doesn't work for most.
>
> So to be able to keep using free software their are some Emacs packages
> or programs that interface with non-free systems. Referencing Melpa for
> such packages seem It is not just bad software choices but also idealism vs reality.
> I can try to change the predicament that I'm tied to some non free
> programs or system but at some point my means are exhausted.
> First I need to have the means to do it, for example: I'm a software
> engineer I try to find alternatives, setup my own systems if needed and
> find out what is the best tool for what I want to do.
> But a lot of people don't have that power either because they don't have
> the resources or their environment forces them.
> For example at work or because the government doesn't offer free
> alternatives.
>
> I respect people such as RMS for sacrificing the convenience of using
> only free systems but I think that doesn't work for most.
>
> So to be able to keep using free software their are some Emacs packages
> or programs that interface with non-free systems. Referencing Melpa for
> such packages seem fine for me, except for instances where Elpa contains
> these packages themselves such as for Exchange support (Excorporate).
>
> That Emacs supports Windows, MacOS or other non-free platforms has a very
> similar reason I think.fine for me, except for instances where Elpa contains
> these packages themselves such as for Exchange support (Excorporate).
>
> That Emacs supports Windows, MacOS or other non-free platforms has a very
> similar reason I think.

This discussion should probably continue on emacs-tangents@gnu.org, if
you want to.

>> I always like the example of browse-url, which has a user option
>> `browse-url-browser-function'.  Among other things, you could set it to
>> the function `browse-url-chrome'.  But wait, isn't Chrome the famous
>> non-free browser that spies on its users and one day might even make
>> watching an advisement mandatory?  Sure, but all browse-url does it
>> provide a generic way of opening a URL in some external program.  If the
>> user has to use Chrome, that is bad, but they don't have much of a
>> choice.  But for free people, at home or in less restrictive
>> environments this doesn't make their "browse-url'ing" any worse.  Chrome
>> is a possible, but not a necessary way to implement the feature.  I
>> still get to use Firefox or eww.  So everyone is happy, because the
>> functionality is generic and not called something like
>> "browse-using-google-chrome-and-only-google-chrome" -- which wouldn't
>> even make sense in this context.
>
> Yeah it is kinda strange, like why don't use your operating system for
> such choices, no need to have a function for that.

You can do that too, in fact that is what the default implementation
`browse-url-default-browser' does.

>> The simple fact is that MELPA insists on distributing software that make
>> these mistakes instead of trying to find a compromise position that can
>> help people bound to non-free systems make the most out of Emacs, while
>> not placing the rest at a functional disadvantage.
>>
>> In my eyes this is more than reasonable.
>
> I think people should have the ability to choose what they want to use,
> so unless the package is malicious I see no issue with people using a
> package.
> Doesn't mean that I like it thou.

OK, that has always been possible, because you can add MELPA, and even
if MELPA went down, you could install it manually.  Nobody is inhibited
from installing and using non-free software, but the GNU project isn't
interested in helping them either.

> Thanks for talking Br,
>
> Björn

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>
>>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>>
>>>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>>>
>>>>>> Richard Stallman <rms@gnu.org> writes:
>>>>>>
>>>>>>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>>>>>> [[[ whether defending the US Constitution against all enemies,     ]]]
>>>>>>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>>>>>>
>>>>>>> Please do not encourage people to load packages from MELPA.  MELPA
>>>>>>> does not cooperate with us.  Not in legal matters, not in ethical
>>>>>>> matters, and not in technical matters of development.
>>>>>>
>>>>>> What justifies this kind of gaslighting against Melpa? 
>>>>>
>>>>> Wikipedia defines gaslighting as:
>>>>>
>>>>>     Gaslighting is a colloquialism, loosely defined as manipulating
>>>>>     someone so as to make them question their own reality [...]
>>>>>
>>>>> so I am not sure how this applies to this thread.
>>>>
>>>> I'm sorry but English isn't my mother tongue.. From my pov he wrote
>>>> misleading statements about Melpa which did sound like gaslighting to me.
>>>
>>> Forgive me for guessing, but could your native language be German?  I'm
>>> just inferring from the name.  If so, what did you want to say?
>>> Vielleicht verstehe ich so besser was du meinst?
>>
>> Ja meine Muttersprache ist Deutsch, vielleicht geht es besser so.
>> Ich habe es so verstanden das man Melpa nicht nennen sollte weil Melpa
>> nicht kooperiert mit Gnu, was meiner Meinung nach nicht war ist
>> bzw. mir neu wäre. 
>>
>> Beide Quellen enhalten die nicht freie Software verwenden (Melpa:
>> Lastpass, Elpa Excange support).
>
> Ich verstehe deinen Satzbau hier nicht, anstatt einem Objekt nach
> "enthalten," fängt ein Untersatz an ", die nicht...".  Willst du sagen,
> dass beide Archive nur Freie Software enthalten?

Ja genau, beide Quellen enthalten nur freie Software. Beide Quellen
haben aber Software die mit nicht freier Software interagiert.

I translate this my self: Yes both sources contain only free software,
but both contain software that interacts with non-free software.

Anyway you don't have to write in German just for me, it's fine.

>> Dadurch fällt mir als einziger Unterschied Github bzw. Forge basierte
>> Entwicklung und Mailinglisten basiertes Model.
>>
>> OT: Ich spreche Deutsch selten in den letzten Tagen, habe generell
>> manchmal Probleme mich auszudrücken es ist nicht nur das Englische, AHDS
>> sei dank.
>
> I'll translate this for the others on the list:
>
>      My native language is German, so perhaps this is better.  I
>      understood that one shouldn't mention MELPA because MELPA doesn't
>      cooperate with GNU, which in my opinion isn't true or rather would
>      be new to me.
>
> MELPA ist ein eigenständiges Projekt welches seine eigenen Ziele hat,
> welche nicht zu 100% mit dem GNU Projekt im Einklang stehen.  Es gibt
> fundamentale Meinungsverschiedenheiten welche seit Jahren bestehen.
>
> MELPA is an independent Projekt with their own Goals.  There aren't
> 100% aligned with those of the GNU Project.  There have been fundamental
> differences of opinion that have existed for years.
>
>      Both sources contain which don't use Free Software (MELPA:
>      Lastpass, ELPA Exchange support) [this is a literal translation, I
>      can't make sense of the sentence in German either].
>
>      Thereby the only difference I can notice is GitHub, or rather a
>      Forge-based development vs. a mailing list model.
>
> Abgesehen von den technischen unterschieden, gibt es verschiedene
> Guidelines zwischen den Projekten, welche Pakete angenommen werden.
> Dahingegen sind die Fragen der Umsetzung (Mailing List vs PR-basierte
> Webseite) recht nebensächlich.
>
> Setting aside technical differences, there are different Guidelines what
> packages are accepted.  Compared to that the question of how the
> projects are organised (Mailing List vs PR-based Website) is not that
> important.
>
> GNU ELPA: https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/README#n330
> MELPA:
> https://raw.githubusercontent.com/melpa/melpa/master/CONTRIBUTING.org

>>>>>>                                                        You might not
>>>>>> like to hear it but without Melpa Emacs wouldn't be were it is now..
>>>>>
>>>>> This is a counterfactual discussion, because it cannot be said if MELPA
>>>>> was a necessary or contingent fact.  I agree that MELPA provided an
>>>>> important service in collecting the number of packages that it did, but
>>>>> if NonGNU ELPA had been created over 10 years ago with the regular GNU
>>>>> ELPA, perhaps it would have been enough?
>>>>
>>>> Some have issues with the FSF, RMS etc. staying out of the whole thing
>>>> was convenient for some.
>>>> Even if you ignore that Melpa was more convinient to use unless there's
>>>> a more modern way to interact to with ELPA.
>>>
>>> I have floated the idea of creating an Emacs package for submitting ELPA
>>> packages, that would help automatise the repetitive questions, such as
>>> have you signed the FSF CA if you want to add a package to GNU ELPA, are
>>> all the dependencies available, has basic code style been respected that
>>> checkdoc and byte compilation can detect, etc.  
>>
>> That sounds like a good idea, some kind of CI that checks the packages
>> would be nice, the Ci can run on the creation of a request or on
>> whitelist.
>
> That can be difficult on a free-form mailing list like this one, unless
> there is some formal indication (which is something a package like this
> could provide).
>
>> I think for a lot of people the way that the FSF acts or just the name
>> leaves a bad taste in their mouth. Personally I think it quite sad that
>> there isn't more corporation, I wish the FSF and FSFE would push for
>> more free software in government and elsewhere around the world.
>> In a lot environments uncertainty around free software especially after
>> GPLv3 was released created by issues.
>> A lot of places I've worked at had almost an allergy against things such
>> as GPLv3.
>
> I wish the entire GNU Project would be more integrated towards the
> creation of a GNU Operating system, but that really is an off-topic
> issue for this list.
>
>>> Another idea I have heard been suggested is creating a separate issue
>>> tracker for ELPA submissions and issues.  I am not sure if this would
>>> help that much, but I guess some people avoid the mailing list because
>>> they don't want to initiate a long discussion.
>>
>> If debbugs would be list a little modern such things would be easier,
>> just create a bug at the Gnu bugtracker under the ELPA product. 
>
> What do you have in mind specifically when you say "modern"?
>
> The Guix people have been using a separate different front end that
> /looks/ more modern, that still is debbugs AFAIK:
> https://issues.guix.gnu.org/, and the source code is here:
> https://git.elephly.net/gitweb.cgi?p=software/mumi.git.

Yes something like your example, a ui that allows contribution without
email and looks more modern. Both debbugs and the mailman2 that used by
Gnu also doesn't scale/look good on high dpi screens.
Mailman2 is EOL in any case.

>>>>>>> A given package that happens to be in MELPA may be perfectly fine in
>>>>>>> and of itself, or it may have problems of one kind of the other.
>>>>>>>
>>>>>>> If you come across a package in MELPA that has no particular problems,
>>>>>>> we can DTRT to put it in either GNU ELPA or NonGNU ELPA.
>>>>>>
>>>>>> It's perfectly fine that is on Melpa, not everyone likes the mailing
>>>>>> list based approach of Gnu.
>>>>>> Offer other options such as a Gitlab or Gitea instance instead of
>>>>>> antiquated Savanah (or make it more modern in other ways)
>>>>>> and people might move elsewhere.
>>>>>
>>>>> I am afraid you have some misunderstandings regarding GNU ELPA (and I
>>>>> suppose NonGNU ELPA as well).  GNU ELPA packages can and often are
>>>>> developed on PR-based forges, where the state is synchronised into
>>>>> elpa.git/nongnu.git, where the packages are build and distributed.
>>>>> There is no need to use mailing lists -- except maybe to announce a
>>>>> package and to request it be added to an archive.  But am I understating
>>>>> your correctly that that is really the point you are objecting to?
>>>>
>>>> I'm sorry I wasn't aware of that, I assumed that using Github to develop
>>>> the package is enough to disqualify it.
>>>
>>> No, that is the great thing about Git.  I can clone and hack on a
>>> package that is hosted on GitHub, without ever having to accept the
>>> execution on Non Free Javascript on my device.  Sure, the GNU project
>>> would advise against using GitHub for several reasons, but as long as
>>> you don't force others to use Non-Free Software, it is not a
>>> deal-breaker.
>>>
>>> Just take a look at the current list of packages included in ELPA:
>>>
>>>   https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/elpa-packages
>>>
>>> There are plenty of packages that are developed on GitHub or GitLab.
>>> Almost none are currently maintained on Savannah.  Luckily more and more
>>> are also appearing on freedom respecting sites like Sourcehut.
>>>
>>> (I really don't know where this kind of misinformation stems from.  I
>>> have heard it too, and was scared at first.  But it turns out that
>>> people who haven't quite understand the arguments keep arguing against
>>> strawmen in their own minds.)
>>
>> Yeah I understand that, I use Git in a similar way,  I have my own
>> mirrors but use Gitlab/Github for the network effect in the communities
>> I need it.
>>
>> But the misinformation came at least from my side out of the issue
>> that I wasn't aware that Melpa contains packages that engages with
>> non-free software at least not to the extend that Emacs already does.
>> Like there are Windows build for macos and Windows, Melpa contains
>> packages for that interact with such operating systems in the same way.
>
> Richard went into that issue in a parallel thread just yesterday:
> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg00792.html:
>
>   Our general policy makes a subtle distinction between these two cases:
>
>   1. If a nonfree program FOO is not well known, we don't even mention that
>   it exists.  Because we don't want to promote using FOO.
>
>   2. If a nonfree program FOO is well known and widely used, something to
>   help and encourage FOO's users to use some GNU packages along with FOO
>   is good.
>
>   3. Anything that would encourage the existing users of some GNU packages
>   to use FOO with them is bad.


OK I don't see anything against cooperating with Gnu in Melpa, the only
difference is the barrier of entry for packages that interact with
non-free systems, especially the amount of questioning that a package
has go too but that is subjective I think.

>> So Github was only remaining thing that is left as an issue.
>> To be honest it makes sense since relaying it as a central hub is just
>> bad no matter your position of free software..
>>
>>>> I am objecting against the assumption Melpa equals bad. I can understand
>>>> the issue with some of it's packages or even the place of distribution
>>>> but it hard to replace a platform like Github for the network effect it
>>>> has.
>>>
>>> The issue was just that Emacs doesn't want to refer to MELPA, because
>>> the two projects clash in their respective interests.  My understanding
>>> is that MELPA tries to be exhaustive, while Emacs/ELPA prioritises that
>>> all software can be used without loss of functionality on a fully free
>>> system.  A choice has to be made.
>>> IMO this is often the result of "bad" software choices.  The point is
>>> not to ignore non-free software and pretend it doesn't exist.
>>> Proprietary software is a means of exercising control over a user, and
>>> some people are stuck in dominating environments, where the lack of
>>> software freedom is symptomatic for their entire predicament, not
>>> necessarily the cause of it.
>>
>> It is not just bad software choices but also idealism vs reality.
>> I can try to change the predicament that I'm tied to some non free
>> programs or system but at some point my means are exhausted.
>> First I need to have the means to do it, for example: I'm a software
>> engineer I try to find alternatives, setup my own systems if needed and
>> find out what is the best tool for what I want to do.
>> But a lot of people don't have that power either because they don't have
>> the resources or their environment forces them.
>> For example at work or because the government doesn't offer free
>> alternatives.
>>
>> I respect people such as RMS for sacrificing the convenience of using
>> only free systems but I think that doesn't work for most.
>>
>> So to be able to keep using free software their are some Emacs packages
>> or programs that interface with non-free systems. Referencing Melpa for
>> such packages seem It is not just bad software choices but also idealism vs reality.
>> I can try to change the predicament that I'm tied to some non free
>> programs or system but at some point my means are exhausted.
>> First I need to have the means to do it, for example: I'm a software
>> engineer I try to find alternatives, setup my own systems if needed and
>> find out what is the best tool for what I want to do.
>> But a lot of people don't have that power either because they don't have
>> the resources or their environment forces them.
>> For example at work or because the government doesn't offer free
>> alternatives.
>>
>> I respect people such as RMS for sacrificing the convenience of using
>> only free systems but I think that doesn't work for most.
>>
>> So to be able to keep using free software their are some Emacs packages
>> or programs that interface with non-free systems. Referencing Melpa for
>> such packages seem fine for me, except for instances where Elpa contains
>> these packages themselves such as for Exchange support (Excorporate).
>>
>> That Emacs supports Windows, MacOS or other non-free platforms has a very
>> similar reason I think.fine for me, except for instances where Elpa contains
>> these packages themselves such as for Exchange support (Excorporate).
>>
>> That Emacs supports Windows, MacOS or other non-free platforms has a very
>> similar reason I think.
>
> This discussion should probably continue on emacs-tangents@gnu.org, if
> you want to.
>

Ok

Br,

Björn

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> I translate this my self: Yes both sources contain only free software,
> but both contain software that interacts with non-free software.

I believe that Stefan explained this, in distinguishing between software
that you have to run on your own system and a fixed service that runs on
non-free software.  A web browser is not at fault when requesting a
website from a non-free web server.

> Anyway you don't have to write in German just for me, it's fine.

Ok.

>> What do you have in mind specifically when you say "modern"?
>>
>> The Guix people have been using a separate different front end that
>> /looks/ more modern, that still is debbugs AFAIK:
>> https://issues.guix.gnu.org/, and the source code is here:
>> https://git.elephly.net/gitweb.cgi?p=software/mumi.git.
>
> Yes something like your example, a ui that allows contribution without
> email and looks more modern. Both debbugs and the mailman2 that used by
> Gnu also doesn't scale/look good on high dpi screens.
> Mailman2 is EOL in any case.

Then it might be worth convincing whoever is responsible to try setting
up mumi.  There has also been the discussion of moving to SourceHut,
which should also fix the issues you have.

>> Richard went into that issue in a parallel thread just yesterday:
>> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg00792.html:
>>
>>   Our general policy makes a subtle distinction between these two
>> cases:
>>
>>   1. If a nonfree program FOO is not well known, we don't even
>> mention that
>>   it exists.  Because we don't want to promote using FOO.
>>
>>   2. If a nonfree program FOO is well known and widely used,
>> something to
>>   help and encourage FOO's users to use some GNU packages along with
>> FOO
>>   is good.
>>
>>   3. Anything that would encourage the existing users of some GNU
>> packages
>>   to use FOO with them is bad.
>
> OK I don't see anything against cooperating with Gnu in Melpa, the only
> difference is the barrier of entry for packages that interact with
> non-free systems, especially the amount of questioning that a package
> has go too but that is subjective I think.

Are you saying that GNU ELPA or MELPA go through more "questioning"?

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> I translate this my self: Yes both sources contain only free software,
>> but both contain software that interacts with non-free software.
>
> I believe that Stefan explained this, in distinguishing between software
> that you have to run on your own system and a fixed service that runs on
> non-free software.  A web browser is not at fault when requesting a
> website from a non-free web server.

What if the software only implements non-free standards such as Exchange?

>> Anyway you don't have to write in German just for me, it's fine.
>
> Ok.
>
>>> What do you have in mind specifically when you say "modern"?
>>>
>>> The Guix people have been using a separate different front end that
>>> /looks/ more modern, that still is debbugs AFAIK:
>>> https://issues.guix.gnu.org/, and the source code is here:
>>> https://git.elephly.net/gitweb.cgi?p=software/mumi.git.
>>
>> Yes something like your example, a ui that allows contribution without
>> email and looks more modern. Both debbugs and the mailman2 that used by
>> Gnu also doesn't scale/look good on high dpi screens.
>> Mailman2 is EOL in any case.
>
> Then it might be worth convincing whoever is responsible to try setting
> up mumi.  There has also been the discussion of moving to SourceHut,
> which should also fix the issues you have.

ok. For me anything is fine but I there are others.

>>> Richard went into that issue in a parallel thread just yesterday:
>>> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg00792.html:
>>>
>>>   Our general policy makes a subtle distinction between these two
>>> cases:
>>>
>>>   1. If a nonfree program FOO is not well known, we don't even
>>> mention that
>>>   it exists.  Because we don't want to promote using FOO.
>>>
>>>   2. If a nonfree program FOO is well known and widely used,
>>> something to
>>>   help and encourage FOO's users to use some GNU packages along with
>>> FOO
>>>   is good.
>>>
>>>   3. Anything that would encourage the existing users of some GNU
>>> packages
>>>   to use FOO with them is bad.
>>
>> OK I don't see anything against cooperating with Gnu in Melpa, the only
>> difference is the barrier of entry for packages that interact with
>> non-free systems, especially the amount of questioning that a package
>> has go too but that is subjective I think.
>
> Are you saying that GNU ELPA or MELPA go through more "questioning"?


I'm saying that packages that interface with non-free formats or systems
have less questioning in Melpa.
In Elpa a package has to justify why it should be added when it
interfaces with non-free systems.

Br,

Björn

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> I translate this my self: Yes both sources contain only free software,
>>> but both contain software that interacts with non-free software.
>>
>> I believe that Stefan explained this, in distinguishing between
>> software
>> that you have to run on your own system and a fixed service that
>> runs on
>> non-free software.  A web browser is not at fault when requesting a
>> website from a non-free web server.
>
> What if the software only implements non-free standards such as Exchange?

The term "non-free" doesn't apply to standards.  How do you read the
source and share modifications of a protocol specification?

>>>> Richard went into that issue in a parallel thread just yesterday:
>>>> https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg00792.html:
>>>>
>>>>   Our general policy makes a subtle distinction between these two
>>>> cases:
>>>>
>>>>   1. If a nonfree program FOO is not well known, we don't even
>>>> mention that
>>>>   it exists.  Because we don't want to promote using FOO.
>>>>
>>>>   2. If a nonfree program FOO is well known and widely used,
>>>> something to
>>>>   help and encourage FOO's users to use some GNU packages along with
>>>> FOO
>>>>   is good.
>>>>
>>>>   3. Anything that would encourage the existing users of some GNU
>>>> packages
>>>>   to use FOO with them is bad.
>>>
>>> OK I don't see anything against cooperating with Gnu in Melpa, the
>>> only
>>> difference is the barrier of entry for packages that interact with
>>> non-free systems, especially the amount of questioning that a package
>>> has go too but that is subjective I think.
>>
>> Are you saying that GNU ELPA or MELPA go through more "questioning"?
>
> I'm saying that packages that interface with non-free formats or systems
> have less questioning in Melpa.
> In Elpa a package has to justify why it should be added when it
> interfaces with non-free systems.

I guess that is true, but that is the issue too.

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

>>> I believe that Stefan explained this, in distinguishing between
>>> software
>>> that you have to run on your own system and a fixed service that
>>> runs on
>>> non-free software.  A web browser is not at fault when requesting a
>>> website from a non-free web server.
>>
>> What if the software only implements non-free standards such as Exchange?
>
> The term "non-free" doesn't apply to standards.  How do you read the
> source and share modifications of a protocol specification?

Maybe I should have phrased it differently but the mapi protocol to
connect to Exchange is only implemented in Microsoft Exchange.
Unless there's a foss software that basically reimplements Microsoft
Exchange having a package that implements such standards in Elpa is like having
a package that happens to be able to talk to the api of services such as
spotify.  

But even for protocols or file type standards there is documentation that is
needed to implement such.

Br,

Björn

Re: feature/package-vc has been merged

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

On Mon, Nov 14, 2022 at 12:19:17AM +0200, Björn Bidar wrote:

[...]

> Maybe I should have phrased it differently but the mapi protocol to
> connect to Exchange is only implemented in Microsoft Exchange.

As usual, reality isn't as clear cut as to fit into such a small
sentence.

There are several free projects which implement parts of MAPI, to
several degrees of completeness. Parts of MAPI are publically
released.

Sometimes, a combination of reverse engineering, very perseverant
lobbying work by free software associations and the ultimate
threat of anti-trust action end up cooking a big vendor (think
Samba).

Big vendors think strategically. Free software advocates have
to, too.

Nowadays, big vendor's strategies are shifting away from software
and protocol secrecy towards the methods of surveillance capitalism
(what Zuboff calls "the means of behavioral modification").

I think those are more insidious (cf. Microsoft and Github).

But back to your assertion above: I'd say "all generalizations
suck" :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

>>> All of this would only apply to packages with external `:lisp-dir's,
>>> which doesn't immediately interest a user/developer.  Having to keep
>>> this in mind would pointlessly expose an internal detail of package-vc
>>> that I'd like to avoid.
>>
>> But it is us who introduced and support :lisp-dir.  If we think it's a
>> leaky abstraction, we could decide not to support it.
>
> You mean as in only allowing for packages to distribute lisp code in the
> root directory of the repository?  That would pointlessly break too many
> packages that decide to structure their file hierarchy for whatever
> reason.

Could this be done similar as in Borg where you specify the lisp,
directories build steps etc. in the configuration file.
In case of Borg that it .gitmodules which would also be used in a
similar way.

From my pov if you use the package directly from the version control
system you need to take these specialties into account.
Source isn't used as is but processed by the packages build-system.
But the user also needs to take not that all the necessary tools such as
make or ninja are installed.

Br,

Björn 

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>>> All of this would only apply to packages with external `:lisp-dir's,
>>>> which doesn't immediately interest a user/developer.  Having to keep
>>>> this in mind would pointlessly expose an internal detail of package-vc
>>>> that I'd like to avoid.
>>>
>>> But it is us who introduced and support :lisp-dir.  If we think it's a
>>> leaky abstraction, we could decide not to support it.
>>
>> You mean as in only allowing for packages to distribute lisp code in the
>> root directory of the repository?  That would pointlessly break too many
>> packages that decide to structure their file hierarchy for whatever
>> reason.
>
> Could this be done similar as in Borg where you specify the lisp,
> directories build steps etc. in the configuration file.
> In case of Borg that it .gitmodules which would also be used in a
> similar way.

Package-vc has access to the ELPA package specifications that indicate
if a sub-directory is used to store Lisp code.  I have already made the
necessary changes to make use of this information.

> From my pov if you use the package directly from the version control
> system you need to take these specialties into account.
> Source isn't used as is but processed by the packages build-system.
> But the user also needs to take not that all the necessary tools such as
> make or ninja are installed.

Right, this is currently not supported.  Theoretically for security
reasons, but security and packaging in Emacs have rarely been mutual
considerations.  Adding it wouldn't be difficult, but coming up with a
sensible fallback strategy might be.

> Br,
>
> Björn 

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

>> From my pov if you use the package directly from the version control
>> system you need to take these specialties into account.
>> Source isn't used as is but processed by the packages build-system.
>> But the user also needs to take not that all the necessary tools such as
>> make or ninja are installed.
>
> Right, this is currently not supported.  Theoretically for security
> reasons, but security and packaging in Emacs have rarely been mutual
> considerations.  Adding it wouldn't be difficult, but coming up with a
> sensible fallback strategy might be.

Without such a system the package could be without use in many cases.

I noticed recently that some external packages such as projectile where
copied but not to the extend or why that they are useful.

For example Borg only works because of magit, epkg is almost useles
without Borg.

If the packages complete use case isn't meat it should at least get all
the features that it is useful without applying hacks so it can be used
in the next Emacs version.
I understand that you try to get it closer however that would then only
affect anything after Emacs 29.

Br,

Björn

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>> From my pov if you use the package directly from the version control
>>> system you need to take these specialties into account.
>>> Source isn't used as is but processed by the packages build-system.
>>> But the user also needs to take not that all the necessary tools such as
>>> make or ninja are installed.
>>
>> Right, this is currently not supported.  Theoretically for security
>> reasons, but security and packaging in Emacs have rarely been mutual
>> considerations.  Adding it wouldn't be difficult, but coming up with a
>> sensible fallback strategy might be.
>
> Without such a system the package could be without use in many cases.

Many is probably the word of contention here.  If you take a look at
elpa.git:elpa-packages, you'll find only a few :make or :shell-command
directives, none of which are critical.  nongnu.git:elpa-packages has
non at all.  One thing I worry about, but which has also been discussed
here are :renames.  E.g. Vertico uses these to move extensions from a
subdirectory to the main directory for packaging.  But moving the files
would be registered by the VCS, and could make committing changes more
difficult.  Maybe we could create symbolic/hard links instead of
renaming?

> I noticed recently that some external packages such as projectile where
> copied but not to the extend or why that they are useful.

I am afraid I don't understand the issue you are describing.  Could you
be more concrete?

> For example Borg only works because of magit, epkg is almost useles
> without Borg.

Just to clarify, I have never used Borg, straight, elpacaa, etc. so I
don't know how they work, how they are used or what terminology they
use.  I have peeked into their source code in the past, but none of that
was related to the development of package-vc.

> If the packages complete use case isn't meat it should at least get all
> the features that it is useful without applying hacks so it can be used
> in the next Emacs version.
> I understand that you try to get it closer however that would then only
> affect anything after Emacs 29.

What hacks are you referring to?  Is there a specific use-case you think
must be added for package-vc to be satisfactory?

> Br,
>
> Björn

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>>> From my pov if you use the package directly from the version control
>>>> system you need to take these specialties into account.
>>>> Source isn't used as is but processed by the packages build-system.
>>>> But the user also needs to take not that all the necessary tools such as
>>>> make or ninja are installed.
>>>
>>> Right, this is currently not supported.  Theoretically for security
>>> reasons, but security and packaging in Emacs have rarely been mutual
>>> considerations.  Adding it wouldn't be difficult, but coming up with a
>>> sensible fallback strategy might be.
>>
>> Without such a system the package could be without use in many cases.
>
> Many is probably the word of contention here.  If you take a look at
> elpa.git:elpa-packages, you'll find only a few :make or :shell-command
> directives, none of which are critical.  nongnu.git:elpa-packages has
> non at all.  One thing I worry about, but which has also been discussed
> here are :renames.  E.g. Vertico uses these to move extensions from a
> subdirectory to the main directory for packaging.  But moving the files
> would be registered by the VCS, and could make committing changes more
> difficult.  Maybe we could create symbolic/hard links instead of
> renaming?

Most packages work fine with if they are used in place.

>> I noticed recently that some external packages such as projectile where
>> copied but not to the extend or why that they are useful.
>
> I am afraid I don't understand the issue you are describing.  Could you
> be more concrete?

The package is copied but not as good because in the end it misses some
features, it doesn't feel as polished. I don't have a direct example
except missing features like no build system integration.
It is just to bare bone.

>> For example Borg only works because of magit, epkg is almost useles
>> without Borg.
>
> Just to clarify, I have never used Borg, straight, elpacaa, etc. so I
> don't know how they work, how they are used or what terminology they
> use.  I have peeked into their source code in the past, but none of that
> was related to the development of package-vc.

That's to bad I think it very helpful to improve on such packages or
even just adapt them instead of reinventing them.

>> If the packages complete use case isn't meat it should at least get all
>> the features that it is useful without applying hacks so it can be used
>> in the next Emacs version.
>> I understand that you try to get it closer however that would then only
>> affect anything after Emacs 29.
>
> What hacks are you referring to?  Is there a specific use-case you think
> must be added for package-vc to be satisfactory?

First of all I Didn't know about these elpa-packages but there are two
cases where I think custom commands could be needed such as packages with native modules or
some that require bootstrapping their build system by means such as
autotools.

Reunsing melpa package generation instructions just might not work as
good going by just using the package sources in place such as how borg
does it.

Br,

Björn

Re: feature/package-vc has been merged

[Stefan Monnier]

>>> I noticed recently that some external packages such as projectile where
>>> copied but not to the extend or why that they are useful.
>> I am afraid I don't understand the issue you are describing.  Could you
>> be more concrete?
> The package is copied but not as good because in the end it misses some
> features, it doesn't feel as polished. I don't have a direct example
> except missing features like no build system integration.
> It is just to bare bone.

I don't know what you mean by "The package is copied".  Then when you
say "not as good": as good as what?  Same for "as polished": as polished
as what?

I'm also not sure even what you mean by "the package": are you talking
about the tarball for `projectile` on NonGNU ELPA?

> First of all I Didn't know about these elpa-packages but there are two
> cases where I think custom commands could be needed such as packages
> with native modules or some that require bootstrapping their build
> system by means such as autotools.

All the packages I have found which used autotools were easy to convert
to rely on the normal build process for ELPA packages.

For those packages which require building other artifacts, such as
`pdf-tools` or `libpq`, the current build recipe does not support that
at all because it was designed for elpa.gnu.org where we don't
distribute binaries compiled for specific architectures.

So instead, the build needs to happen during `package-install` and that
depends on the ELPA protocol which does not include any special support
for that :-(
Luckily it can be solved by appropriate ELisp code in the package, which
can trigger compilation of the artifact either during `package-install`
or later when the package is actually used.

It's not ideal, admittedly, but given the requirement to have something
that can work across various packaging systems (`package-install`,
`package-vc-install`, and possibly others), it's not that bad.

I'm hoping that we'll develop a "standard" way to do it in the form of
some ELisp function/macro so that `pdf-tools`, `libpq`, etc... can
simply include a line or two of ELisp code which says how to compile
their artifact (possibly just running `make`, or `ninja`, ...).


        Stefan

Re: feature/package-vc has been merged

[Björn Bidar]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> I noticed recently that some external packages such as projectile where
>>>> copied but not to the extend or why that they are useful.
>>> I am afraid I don't understand the issue you are describing.  Could you
>>> be more concrete?
>> The package is copied but not as good because in the end it misses some
>> features, it doesn't feel as polished. I don't have a direct example
>> except missing features like no build system integration.
>> It is just to bare bone.
>
> I don't know what you mean by "The package is copied".  Then when you
> say "not as good": as good as what?  Same for "as polished": as polished
> as what?
>
> I'm also not sure even what you mean by "the package": are you talking
> about the tarball for `projectile` on NonGNU ELPA?
>
>> First of all I Didn't know about these elpa-packages but there are two
>> cases where I think custom commands could be needed such as packages
>> with native modules or some that require bootstrapping their build
>> system by means such as autotools.
>
> All the packages I have found which used autotools were easy to convert
> to rely on the normal build process for ELPA packages.
>
> For those packages which require building other artifacts, such as
> `pdf-tools` or `libpq`, the current build recipe does not support that
> at all because it was designed for elpa.gnu.org where we don't
> distribute binaries compiled for specific architectures.
> So instead, the build needs to happen during `package-install` and that
> depends on the ELPA protocol which does not include any special support
> for that :-(
> Luckily it can be solved by appropriate ELisp code in the package, which
> can trigger compilation of the artifact either during `package-install`
> or later when the package is actually used.
> It's not ideal, admittedly, but given the requirement to have something
> that can work across various packaging systems (`package-install`,
> `package-vc-install`, and possibly others), it's not that bad.


The side effect of that process is that there's no standartized way for
these packages to require native modules is that building those packages
in build systems isn't standartized, these packages have to be convinced
to not build binaries while Emacs run but take the prebuild binaries
from the package manager.

For packages provided by the package manager or which live for other
reasons in /usr there no separate path for native packages/modules.
Unless a separate load path is added manually they are not found in /usr/lib.

> I'm hoping that we'll develop a "standard" way to do it in the form of
> some ELisp function/macro so that `pdf-tools`, `libpq`, etc... can
> simply include a line or two of ELisp code which says how to compile
> their artifact (possibly just running `make`, or `ninja`, ...).

I'm not sure if that is always the best solution, maybe Melpa isn't the
best place such packages.

But just don't run them inside Emacs itself and not synchronously.

Br,

Björn

Re: feature/package-vc has been merged

[Stefan Monnier]

> The side effect of that process is that there's no standartized way for
> these packages to require native modules is that building those packages
> in build systems isn't standartized, these packages have to be convinced
> to not build binaries while Emacs run but take the prebuild binaries
> from the package manager.

I'm not completely sure I understand to what you're alluding, but indeed
it causes extra work for things like Debian where they would benefit
from having a standard way to build (and locate) the extra artifacts
like the module of `pdf-tools`.

Currently packages like `pdf-tools` and `libpq` are fairly unusual but
I'd encourage the development of a standardized way to handle them.
Then Debian packagers would be able to take advantage of it.

>> I'm hoping that we'll develop a "standard" way to do it in the form of
>> some ELisp function/macro so that `pdf-tools`, `libpq`, etc... can
>> simply include a line or two of ELisp code which says how to compile
>> their artifact (possibly just running `make`, or `ninja`, ...).
> I'm not sure if that is always the best solution,

But that's what we can get fairly easily without disruption :-)

> maybe Melpa isn't the best place such packages.

Not sure what Melpa has to do with this, but if you mean the whole of
the ELPA infrastructure, then I think this is largely irrelevant: ELPA
is popular and users do want to install `pdf-tools` via `M-x
package-install`, so while there may be a better solution, we still need
to solve this problem.

> But just don't run them inside Emacs itself and not synchronously.

I lost you here.  I have no idea what "them" refers to.


        Stefan

Re: feature/package-vc has been merged

[Björn Bidar]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The side effect of that process is that there's no standartized way for
>> these packages to require native modules is that building those packages
>> in build systems isn't standartized, these packages have to be convinced
>> to not build binaries while Emacs run but take the prebuild binaries
>> from the package manager.
>
> I'm not completely sure I understand to what you're alluding, but indeed
> it causes extra work for things like Debian where they would benefit
> from having a standard way to build (and locate) the extra artifacts
> like the module of `pdf-tools`.
>
> Currently packages like `pdf-tools` and `libpq` are fairly unusual but
> I'd encourage the development of a standardized way to handle them.
> Then Debian packagers would be able to take advantage of it.

I'm exactly talking about you reference but with addition that there's
no standard load-path for native modules. Regular lisp modules which are
unless native compilation is used interpreted or byte compiled go to
/usr/share.

But native modules should go to /usr/lib/emacs/site-lisp or something
similar in name.

But Emacs doesn't configure a load path for such packages by default.

>>> I'm hoping that we'll develop a "standard" way to do it in the form of
>>> some ELisp function/macro so that `pdf-tools`, `libpq`, etc... can
>>> simply include a line or two of ELisp code which says how to compile
>>> their artifact (possibly just running `make`, or `ninja`, ...).
>> I'm not sure if that is always the best solution,
>
> But that's what we can get fairly easily without disruption :-)
>
>> maybe Melpa isn't the best place such packages.
>
> Not sure what Melpa has to do with this, but if you mean the whole of
> the ELPA infrastructure, then I think this is largely irrelevant: ELPA
> is popular and users do want to install `pdf-tools` via `M-x
> package-install`, so while there may be a better solution, we still need
> to solve this problem.

Melpa probably doesn't have anything directly to do with it but a lot of
packages that use native modules came from their but that has probably
other reasons such as not assigning the copyright to the fsf.

>> But just don't run them inside Emacs itself and not synchronously.
>
> I lost you here.  I have no idea what "them" refers to.

Any process that involves package-vc by it building the package or
fetching their sources.

Br,

Björn.

Re: feature/package-vc has been merged

[Stefan Monnier]

> I'm exactly talking about you reference but with addition that there's
> no standard load-path for native modules. Regular lisp modules which are
> unless native compilation is used interpreted or byte compiled go to
> /usr/share.

Ah, that.  Yes, we have a problem here because we just search along `load-path`
but this normally holds architecture-independent files, whereas modules
are architecture-dependent.

I suspect nobody took up that challenge yet because it's quite rare
nowadays to share /usr or $HOME across several machines and even more so
across machines using different architectures.

>>> But just don't run them inside Emacs itself and not synchronously.
>> I lost you here.  I have no idea what "them" refers to.
> Any process that involves package-vc by it building the package or
> fetching their sources.

Ah, that's a small matter of programming.


        Stefan

Re: feature/package-vc has been merged

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Philip Kaludercic <philipk@posteo.net>,  Eli Zaretskii <eliz@gnu.org>,
>   rms@gnu.org,  emacs-devel@gnu.org
> Date: Thu, 10 Nov 2022 01:40:33 +0200
> 
> I'm exactly talking about you reference but with addition that there's
> no standard load-path for native modules. Regular lisp modules which are
> unless native compilation is used interpreted or byte compiled go to
> /usr/share.
> 
> But native modules should go to /usr/lib/emacs/site-lisp or something
> similar in name.
> 
> But Emacs doesn't configure a load path for such packages by default.

Emacs looks for modules in the same directories as Lisp files: along
load-path.  So I don't understand why you say that there's no standard
path for native modules: there certainly is, from my POV.

If Emacs can find Lisp files that are part of a package, Emacs should
also be able to find the modules that are part of the package.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>
>>>>> From my pov if you use the package directly from the version control
>>>>> system you need to take these specialties into account.
>>>>> Source isn't used as is but processed by the packages build-system.
>>>>> But the user also needs to take not that all the necessary tools such as
>>>>> make or ninja are installed.
>>>>
>>>> Right, this is currently not supported.  Theoretically for security
>>>> reasons, but security and packaging in Emacs have rarely been mutual
>>>> considerations.  Adding it wouldn't be difficult, but coming up with a
>>>> sensible fallback strategy might be.
>>>
>>> Without such a system the package could be without use in many cases.
>>
>> Many is probably the word of contention here.  If you take a look at
>> elpa.git:elpa-packages, you'll find only a few :make or :shell-command
>> directives, none of which are critical.  nongnu.git:elpa-packages has
>> non at all.  One thing I worry about, but which has also been discussed
>> here are :renames.  E.g. Vertico uses these to move extensions from a
>> subdirectory to the main directory for packaging.  But moving the files
>> would be registered by the VCS, and could make committing changes more
>> difficult.  Maybe we could create symbolic/hard links instead of
>> renaming?
>
> Most packages work fine with if they are used in place.

What do you mean by "in place" here?

>>> I noticed recently that some external packages such as projectile where
>>> copied but not to the extend or why that they are useful.
>>
>> I am afraid I don't understand the issue you are describing.  Could you
>> be more concrete?
>
> The package is copied but not as good because in the end it misses some
> features, it doesn't feel as polished. I don't have a direct example
> except missing features like no build system integration.
> It is just to bare bone.

I am still confused as to what you are thinking about.  Setting aside
:make, :shell-command and :renames, the end experienced result of
installing a package that was prepared by elpa-admin.el or one installed
using package-vc (assuming package-vc uses the revision of the latest
release) should be identical.

>>> For example Borg only works because of magit, epkg is almost useles
>>> without Borg.
>>
>> Just to clarify, I have never used Borg, straight, elpacaa, etc. so I
>> don't know how they work, how they are used or what terminology they
>> use.  I have peeked into their source code in the past, but none of that
>> was related to the development of package-vc.
>
> That's to bad I think it very helpful to improve on such packages or
> even just adapt them instead of reinventing them.

The point of package-vc.el is to have something that explicitly extends
package.el and works in the core, in active collaboration with ELPA.
That is why the implementation is far simpler than what others have to
do, because they are fighting an up hill battle outside of the core.

>>> If the packages complete use case isn't meat it should at least get all
>>> the features that it is useful without applying hacks so it can be used
>>> in the next Emacs version.
>>> I understand that you try to get it closer however that would then only
>>> affect anything after Emacs 29.
>>
>> What hacks are you referring to?  Is there a specific use-case you think
>> must be added for package-vc to be satisfactory?
>
> First of all I Didn't know about these elpa-packages but there are two
> cases where I think custom commands could be needed such as packages with native modules or
> some that require bootstrapping their build system by means such as
> autotools.

OK, but that is something that package.el doesn't handle either right
now.  This is a shared deficit, as I had mentioned in my message to
Stefan just a few minutes ago.  I would like to address it at some
point, but probably not in time for Emacs 29.

> Reunsing melpa package generation instructions just might not work as
> good going by just using the package sources in place such as how borg
> does it.

I am still confused by your terminology.  Package-vc has nothing to do
with MELPA, we re-use the same package specifications that ELPA makes
use of, e.g. https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/elpa-packages.

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>
>>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>>
>>>>>> From my pov if you use the package directly from the version control
>>>>>> system you need to take these specialties into account.
>>>>>> Source isn't used as is but processed by the packages build-system.
>>>>>> But the user also needs to take not that all the necessary tools such as
>>>>>> make or ninja are installed.
>>>>>
>>>>> Right, this is currently not supported.  Theoretically for security
>>>>> reasons, but security and packaging in Emacs have rarely been mutual
>>>>> considerations.  Adding it wouldn't be difficult, but coming up with a
>>>>> sensible fallback strategy might be.
>>>>
>>>> Without such a system the package could be without use in many cases.
>>>
>>> Many is probably the word of contention here.  If you take a look at
>>> elpa.git:elpa-packages, you'll find only a few :make or :shell-command
>>> directives, none of which are critical.  nongnu.git:elpa-packages has
>>> non at all.  One thing I worry about, but which has also been discussed
>>> here are :renames.  E.g. Vertico uses these to move extensions from a
>>> subdirectory to the main directory for packaging.  But moving the files
>>> would be registered by the VCS, and could make committing changes more
>>> difficult.  Maybe we could create symbolic/hard links instead of
>>> renaming?
>>
>> Most packages work fine with if they are used in place.
>
> What do you mean by "in place" here?
In side the repositories on source tree.

>>>> I noticed recently that some external packages such as projectile where
>>>> copied but not to the extend or why that they are useful.
>>>
>>> I am afraid I don't understand the issue you are describing.  Could you
>>> be more concrete?
>>
>> The package is copied but not as good because in the end it misses some
>> features, it doesn't feel as polished. I don't have a direct example
>> except missing features like no build system integration.
>> It is just to bare bone.
>
> I am still confused as to what you are thinking about.  Setting aside
> :make, :shell-command and :renames, the end experienced result of
> installing a package that was prepared by elpa-admin.el or one installed
> using package-vc (assuming package-vc uses the revision of the latest
> release) should be identical.

More complex packages have a configure/prepare step, generate some lisp
code, build native parts. In the case of borg you can just add
additional steps to for example in my case build the emms libtag helper
that isn't build if you use melpa.

But you addressed that in an earlier comment. 

>>>> For example Borg only works because of magit, epkg is almost useles
>>>> without Borg.
>>>
>>> Just to clarify, I have never used Borg, straight, elpacaa, etc. so I
>>> don't know how they work, how they are used or what terminology they
>>> use.  I have peeked into their source code in the past, but none of that
>>> was related to the development of package-vc.
>>
>> That's to bad I think it very helpful to improve on such packages or
>> even just adapt them instead of reinventing them.
>
> The point of package-vc.el is to have something that explicitly extends
> package.el and works in the core, in active collaboration with ELPA.
> That is why the implementation is far simpler than what others have to
> do, because they are fighting an up hill battle outside of the core.

There might be reasons for that..
Quite often the core packages aren't as good or just to
complicated/convoluted with legacy features.

Or in case of borg compared to package-vc deeper integration into git or
other different features like building packages in a second instance
with just borg and the package.

Re: feature/package-vc has been merged

[Philip Kaludercic]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

>>>>> For example Borg only works because of magit, epkg is almost useles
>>>>> without Borg.
>>>>
>>>> Just to clarify, I have never used Borg, straight, elpacaa, etc. so I
>>>> don't know how they work, how they are used or what terminology they
>>>> use.  I have peeked into their source code in the past, but none of that
>>>> was related to the development of package-vc.
>>>
>>> That's to bad I think it very helpful to improve on such packages or
>>> even just adapt them instead of reinventing them.
>>
>> The point of package-vc.el is to have something that explicitly extends
>> package.el and works in the core, in active collaboration with ELPA.
>> That is why the implementation is far simpler than what others have to
>> do, because they are fighting an up hill battle outside of the core.
>
> There might be reasons for that..
> Quite often the core packages aren't as good or just to
> complicated/convoluted with legacy features.

My experience has been quite the opposite -- I wouldn't be a
more-or-less-regular contributor if that weren't my impression.  Core
packages are usually developed by people with a good sense of what makes
Emacs "Emacs".

Most of the time the complexity you speak of has an explanation, even if
that explanation is only due to the responsibilities of having a package
in core.

> Or in case of borg compared to package-vc deeper integration into git or
> other different features like building packages in a second instance
> with just borg and the package.

Right, that is an advantage that Borg has by /committing/ to Git,
instead of going through VC and avoiding a strict dependency on one
specific VCS.

Re: feature/package-vc has been merged

[Björn Bidar]

Philip Kaludercic <philipk@posteo.net> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>>>>>> For example Borg only works because of magit, epkg is almost useles
>>>>>> without Borg.
>>>>>
>>>>> Just to clarify, I have never used Borg, straight, elpacaa, etc. so I
>>>>> don't know how they work, how they are used or what terminology they
>>>>> use.  I have peeked into their source code in the past, but none of that
>>>>> was related to the development of package-vc.
>>>>
>>>> That's to bad I think it very helpful to improve on such packages or
>>>> even just adapt them instead of reinventing them.
>>>
>>> The point of package-vc.el is to have something that explicitly extends
>>> package.el and works in the core, in active collaboration with ELPA.
>>> That is why the implementation is far simpler than what others have to
>>> do, because they are fighting an up hill battle outside of the core.
>>
>> There might be reasons for that..
>> Quite often the core packages aren't as good or just to
>> complicated/convoluted with legacy features.
>
> My experience has been quite the opposite -- I wouldn't be a
> more-or-less-regular contributor if that weren't my impression.  Core
> packages are usually developed by people with a good sense of what makes
> Emacs "Emacs".

That seems to be really subjective a lot people use Emacs but wouldn't
without Melpa and co. Core packages often lack a lot of modern features.
But a lack of features can be a feature too thou.

> Most of the time the complexity you speak of has an explanation, even if
> that explanation is only due to the responsibilities of having a package
> in core.
>
>> Or in case of borg compared to package-vc deeper integration into git or
>> other different features like building packages in a second instance
>> with just borg and the package.
>
> Right, that is an advantage that Borg has by /committing/ to Git,
> instead of going through VC and avoiding a strict dependency on one
> specific VCS.

Which makes it by design less reproduceable unless it vc keeps track of
all the version control information like last ref of that package and
its source. But I assume that's what's done here.

Something like make bootstrap could be very useful e.g. to use your
configuration on another machine, so that emacs can start after all
packages needed in init.el are present.
https://emacsmirror.net/manual/borg/Using-your-configuration-on-another-machine.html

Br,

Björn

Re: feature/package-vc has been merged

[Stefan Monnier]

> Which makes it by design less reproduceable unless it vc keeps track of
> all the version control information like last ref of that package and
> its source. But I assume that's what's done here.

VC is just a thin layer over Git, just like Magit only much more limited
(in exchange for working with the VCSs).  AFAIK neither of them worries
about "keep[ing] track of all the version control information like last ref
of that package and its source".  They let Git worry about that.

After you `package-vc-install`, you can use Magit to your heart's
content on the resulting clone.

> Something like make bootstrap could be very useful e.g. to use your
> configuration on another machine, so that emacs can start after all
> packages needed in init.el are present.
> https://emacsmirror.net/manual/borg/Using-your-configuration-on-another-machine.html

`package-(vc-)selected-packages` is designed for that kind of purpose.


        Stefan

Re: feature/package-vc has been merged

[Stefan Monnier]

>> Without such a system the package could be without use in many cases.
> Many is probably the word of contention here.  If you take a look at
> elpa.git:elpa-packages, you'll find only a few :make or :shell-command
> directives, none of which are critical.  nongnu.git:elpa-packages has
> non at all.

Indeed.  We don't use `make` or `ninja` to compile the ELisp files,
instead we impose a standardized way to compile them.

[ And those packages which rely on special build procedures will often
  suffer from problems with the native compiler, which will lazily
  recompile the files to native code without paying attention to the
  special build requirements.  ]

> One thing I worry about, but which has also been discussed
> here are :renames.

Indeed.  Currently `elpa-admin.el` doesn't obey them when using "install
from Git" (it does obey them when building the tarballs, of course) :-(

> E.g. Vertico uses these to move extensions from a subdirectory to the
> main directory for packaging.  But moving the files would be
> registered by the VCS, and could make committing changes more
> difficult.  Maybe we could create symbolic/hard links instead
> of renaming?

Moving is definitely out of the question, but symlinks and hardlinks are
also problematic.  We should probably document that `:renames` are not
fully supported in all cases and should thus be avoided.
I currently count 6 :renames, two for `extensions/` and 4 for `docs/`.

AFAIK Those for docs are needed only because `package-install` handles
`.info` files only in the root directory of packages, but that doesn't
afflict `package-vc`, so we should be able to find a better solution.

Those for `extensions/` can be handled by adding `extensions/` to
the `load-path` in the `<pkg>-autoloads.el` generated by
`package-vc-install`.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Without such a system the package could be without use in many cases.
>> Many is probably the word of contention here.  If you take a look at
>> elpa.git:elpa-packages, you'll find only a few :make or :shell-command
>> directives, none of which are critical.  nongnu.git:elpa-packages has
>> non at all.
>
> Indeed.  We don't use `make` or `ninja` to compile the ELisp files,
> instead we impose a standardized way to compile them.
>
> [ And those packages which rely on special build procedures will often
>   suffer from problems with the native compiler, which will lazily
>   recompile the files to native code without paying attention to the
>   special build requirements.  ]

What might be interesting is providing support for running commands to
build additional software a package needs, e.g. when a package
distributed a C module.  But this wouldn't just be a package-vc thing,
but a thing that would interest all packages.

>> One thing I worry about, but which has also been discussed
>> here are :renames.
>
> Indeed.  Currently `elpa-admin.el` doesn't obey them when using "install
> from Git" (it does obey them when building the tarballs, of course) :-(
>
>> E.g. Vertico uses these to move extensions from a subdirectory to the
>> main directory for packaging.  But moving the files would be
>> registered by the VCS, and could make committing changes more
>> difficult.  Maybe we could create symbolic/hard links instead
>> of renaming?
>
> Moving is definitely out of the question, but symlinks and hardlinks are
> also problematic.  We should probably document that `:renames` are not
> fully supported in all cases and should thus be avoided.
> I currently count 6 :renames, two for `extensions/` and 4 for `docs/`.

I agree, luckily there hasn't been much need for it up until now.

> AFAIK Those for docs are needed only because `package-install` handles
> `.info` files only in the root directory of packages, but that doesn't
> afflict `package-vc`, so we should be able to find a better solution.
>
> Those for `extensions/` can be handled by adding `extensions/` to
> the `load-path` in the `<pkg>-autoloads.el` generated by
> `package-vc-install`.

OK, but how would this generalise?  Hard-coding something like "if
'extensions/' is renamed to the '', then add 'extensions/' to
`load-path'" doesn't sound desirable.

>
>         Stefan

Re: feature/package-vc has been merged

[Stefan Monnier]

>> Those for `extensions/` can be handled by adding `extensions/` to
>> the `load-path` in the `<pkg>-autoloads.el` generated by
>> `package-vc-install`.
> OK, but how would this generalise?  Hard-coding something like "if
> 'extensions/' is renamed to the '', then add 'extensions/' to
> `load-path'" doesn't sound desirable.

Agreed.  The main problem is that `:renames` doesn't say if it's done
for ELisp files or for something else.

If we knew that than it would be a simple matter of checking if the new
name is just "".

There are other packages with extra ELisp subdirectories
(e.g. `proof-general` and `hyperbole`) which handle them differently.

I'd be tempted to say that the way `corfu` and `vertico` do it
(i.e. via `:renames`) should be discouraged.  E.g. let them add
`extensions` to `load-path` "by hand" rather than get a similar result
via `:renames`.


        Stefan

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> I have merged the branch onto master, so if anyone wants to fix,
> add/remove or improve anything, feel free to do so.

OMG, the time has come!  Thank you.

Rudy
-- 
"'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and
if it were so, it would be; but as it isn't, it ain't.  That's logic.'"
-- Lewis Carroll, Through the Looking Glass, 1871/1872

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Rudolf Adamkovič <salutis@me.com> writes:

> OMG, the time has come!  Thank you.

I would like to share first-time user experience with you.  Perhaps you
will find it interesting.  Here it goes:

I found `package-vc-selected-packages' in seconds thanks to its familiar
name, nicely matching `package-selected-packages'.  Well done!  It
surprised me, though, that the variable does not accept bare package
names, such as `modus-themes' instead of `(modus-themes . nil)'.

Then I looked for `package-vc-install-selected-packages`, and troubles
started.  I found no such function, so I decided to consult the
documentation for `package-vc-selected-packages' which says:

  You can also use the function ‘package-vc-selected-packages’ to apply
  the changes.

Confused, I could not find any such function.  I also tried `M-x
package-vc TAB' but that did not help either.  So, I read the source
code, I found `package-vc-ensure-packages', and that worked.  Phew!

I then added `package-vc-ensure-packages' to my Emacs init file, but
Emacs did not launch cleanly, for it could not find the function.  So, I
added an explicit `require' for `package-vc'.

Finally, it all worked.  I installed Modus Themes!

Next, I looked for `package-vc-update', but found nothing.  I then found
`package-vc-refresh' but that sounded like `package-refresh-contents'
which does not update packages.  Confusing!

Still, I ran the `package-vc-refresh' function, and it prompted me with
"Refresh package:".  I hit RET, just to try something, and it failed
with

  funcall-interactively: Wrong number of arguments: #<subr
  package-vc-refresh>, 0

So, I tried again with "modus-themes", which I installed with
`package-vc-ensure-packages', and got

  execute-extended-command: Wrong type argument: listp, #s(package-desc modus-themes (3 0 0) "Elegant, highly legible and customizable themes" ((emacs (27 1))) vc nil "/Users/salutis/.emacs.d/elpa/modus-themes" ((:url . "https://git.sr.ht/~protesilaos/modus-themes") (:keywords "faces" "theme" "accessibility") (:maintainer "Modus-Themes Development" . "~protesilaos/modus-themes@lists.sr.ht") (:authors ("Protesilaos Stavrou" . "info@protesilaos.com")) (:commit . "f1149c18005d31638b3701f9a89819b133b4360e")) nil)

Ignoring the refresh woes, I suggest the following design changes:

  - allow strings in `-selected-packages' if possible
  - make `-ensure-packages' interactive for `M-x'
  - rename `-ensure-' to `-install-selected-' for consistency
  - auto-load `-install-selected-packages' for common use
  - rename `-refresh' to `-update' for consistency
  - add `-update-all' for consistency
  - fix the nonexistent function reference in the documentation

Then, the existing users of `package.el' will know how to use a large
part of `package-vc.el' without needing to learn that "ensure" means
"install-selected", "refresh" means "update", and so on.

Rudy
-- 
"Chop your own wood and it will warm you twice."
-- Henry Ford; Francis Kinloch, 1819; Henry David Thoreau, 1854

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 4009 bytes --]

Rudolf Adamkovič <salutis@me.com> writes:

> Rudolf Adamkovič <salutis@me.com> writes:
>
>> OMG, the time has come!  Thank you.
>
> I would like to share first-time user experience with you.  Perhaps you
> will find it interesting.  Here it goes:
>
> I found `package-vc-selected-packages' in seconds thanks to its familiar
> name, nicely matching `package-selected-packages'.  Well done!  It
> surprised me, though, that the variable does not accept bare package
> names, such as `modus-themes' instead of `(modus-themes . nil)'.

It would be trivial to implement, the main issue is that I wanted to use
`alist' as the user option type.  If instead the type were to be
replaced with some

    (repeat (choice symbol (cons symbol string) ...))

it might work.

But keep in mind that (modus-themes . nil) is the same as (modus-themes).

> Then I looked for `package-vc-install-selected-packages`, and troubles
> started.  I found no such function, so I decided to consult the
> documentation for `package-vc-selected-packages' which says:
>
>   You can also use the function ‘package-vc-selected-packages’ to apply
>   the changes.

> Confused, I could not find any such function.  I also tried `M-x
> package-vc TAB' but that did not help either.  So, I read the source
> code, I found `package-vc-ensure-packages', and that worked.  Phew!

Right, that was a mistake in the documentation that Eli also pointed
out.  If you think the name-symmetry between package and package-vc is
helpful, we could rename that one to
`package-vc-install-selected-packages'.

> I then added `package-vc-ensure-packages' to my Emacs init file, but
> Emacs did not launch cleanly, for it could not find the function.  So, I
> added an explicit `require' for `package-vc'.

What I had in mind was for `package-vc-selected-packages' to be used as
is.  It is an autoloaded option with a custom setter that installs all
"selected packages" as a side effect.

As the manual says, all you need to do is write

    (setopt package-vc-selected-packages '((modus-themes)))

Should that be highlighted more explicitly in the documentation?  I had
actually initially hesitated in exposing `package-vc-ensure-packages' as
a public function at all, let alone auto-loading it.

> Finally, it all worked.  I installed Modus Themes!
>
> Next, I looked for `package-vc-update', but found nothing.  I then found
> `package-vc-refresh' but that sounded like `package-refresh-contents'
> which does not update packages.  Confusing!

Hmm, I forgot about `package-refresh-contents'.  What `package-vc-refresh'
does is package installing dependencies, generating autoloads, building
the documentation, byte compiling, etc.  It exists so that if the user
changes something, they can "activate" their changes without having to
do all those things automatically.

> Still, I ran the `package-vc-refresh' function, and it prompted me with
> "Refresh package:".  I hit RET, just to try something, and it failed
> with
>
>   funcall-interactively: Wrong number of arguments: #<subr
>   package-vc-refresh>, 0

Interesting...

> So, I tried again with "modus-themes", which I installed with
> `package-vc-ensure-packages', and got
>
>   execute-extended-command: Wrong type argument: listp,
> #s(package-desc modus-themes (3 0 0) "Elegant, highly legible and
> customizable themes" ((emacs (27 1))) vc nil
> "/Users/salutis/.emacs.d/elpa/modus-themes" ((:url
> . "https://git.sr.ht/~protesilaos/modus-themes") (:keywords "faces"
> "theme" "accessibility") (:maintainer "Modus-Themes Development"
> . "~protesilaos/modus-themes@lists.sr.ht") (:authors ("Protesilaos
> Stavrou" . "info@protesilaos.com")) (:commit
> . "f1149c18005d31638b3701f9a89819b133b4360e")) nil)

Ah, the interactive spec for `package-vc-refresh' is missing a (list
...).  Is trying to apply a package description as an argument list,
which doesn't make sense.  This will fix the issue:


[-- Attachment #2: Type: text/plain, Size: 629 bytes --]

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index a0b4b03118..127a7e073f 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -692,7 +692,7 @@ package-vc-install-from-checkout
 (defun package-vc-refresh (pkg-desc)
   "Refresh the installation for package given by PKG-DESC.
 Interactively, prompt for the name of the package to refresh."
-  (interactive (package-vc--read-pkg "Refresh package: "))
+  (interactive (list (package-vc--read-pkg "Refresh package: ")))
   (package-vc--unpack-1 pkg-desc (package-desc-dir pkg-desc)))
 
 (defun package-vc--read-pkg (prompt)

[-- Attachment #3: Type: text/plain, Size: 1412 bytes --]


Perhaps the function should be called `-reinstall', or `-reactivate',
but I am uncertain about all these names too.

> Ignoring the refresh woes, I suggest the following design changes:
>
>   - allow strings in `-selected-packages' if possible

Can be done.

>   - make `-ensure-packages' interactive for `M-x'

Can be done.

>   - rename `-ensure-' to `-install-selected-' for consistency

Can be done.

>   - auto-load `-install-selected-packages' for common use

Not a fan of that, but I guess it can be done.

>   - rename `-refresh' to `-update' for consistency

That would be wrong, because `package-vc-refresh' doesn't update the
package.  There is `package-vc-update' and `package-update' (that do the
same thing for source packages).  Maybe `package-vc-update' can be made
interactive too and made to only prompt for source packages.

>   - add `-update-all' for consistency

That already exists, in `package-update-all'.  It will also update
source packages.  What you have in mind would be a
`package-vc-update-all' that would only update all the source packages.

>   - fix the nonexistent function reference in the documentation

Right, we are on that already.

> Then, the existing users of `package.el' will know how to use a large
> part of `package-vc.el' without needing to learn that "ensure" means
> "install-selected", "refresh" means "update", and so on.

I agree on these points.

> Rudy

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

Thank you for reading my first-time user experience!

> What I had in mind was for `package-vc-selected-packages' to be used
> as is.  It is an autoloaded option with a custom setter that installs
> all "selected packages" as a side effect.
>
> As the manual says, all you need to do is write
>
>     (setopt package-vc-selected-packages '((modus-themes)))
>
> [...]

As both a developer and a user, I remain cautious about any "magic" that
makes it harder to tell memory access from arbitrary execution of some
instructions.  Many new and "modern" mainstream languages, such as
Swift, make that mistake.

I like the clear distinction between `foo' and `(foo)' in Lisp.  I even
like the `*' in C that clearly says "a pointer".  Brilliant ideas lost
to the history in most modern languages.  But I digress!

I would never expect `setopt' to do networking.  In fact, I use `setq'
everywhere, like almost everybody else, because I know exactly what it
does, at the call site.  With `setopt', I can only guess.

But anyway, I have a practical reason to say all this:

I have a literate Emacs configuration, mixed with notes.  In it, any
part can add *some* package to the selected packages but any part can
also rely on the availability of *all* installed packages.

To get this freedom, I install the selected packages early on the
after-init hook.  Then, all configuration blocks, evaluated in any
order, can count on all installed packages.

Any number of times, anywhere, and evaluated in any order:

  (with-eval-after load '...
     ...do anything, all packages already installed...)

  (add-hook 'after-init-hook
            ...do anything, all packages already installed...)

Once, does not matter where:

  (add-hook 'after-init-hook
            ...install selected packages...
            -99)

Rudy
-- 
"Programming reliably -- must be an activity of an undeniably
mathematical nature […] You see, mathematics is about thinking, and
doing mathematics is always trying to think as well as possible."
-- Edsger W. Dijkstra, 1981

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
> Thank you for reading my first-time user experience!

Thank you for taking the time to write up your comments and impressions
in such detail!

>> What I had in mind was for `package-vc-selected-packages' to be used
>> as is.  It is an autoloaded option with a custom setter that installs
>> all "selected packages" as a side effect.
>>
>> As the manual says, all you need to do is write
>>
>>     (setopt package-vc-selected-packages '((modus-themes)))
>>
>> [...]
>
> As both a developer and a user, I remain cautious about any "magic" that
> makes it harder to tell memory access from arbitrary execution of some
> instructions.  Many new and "modern" mainstream languages, such as
> Swift, make that mistake.

The thing is the user options aren't strictly variables.  The usually
are, but they are explicitly meant to operate on a higher abstraction
level, sort of even in a separate name space.

But of course, I understand that not everyone feels comfortable with
this.  So while I insist that package-vc-selected-packages ought to be a
user option, I have also made `package-vc-install-selected-packages'
autoloaded (I have yet to push all the commits) and invocable as a
regular function

> I like the clear distinction between `foo' and `(foo)' in Lisp.  I even
> like the `*' in C that clearly says "a pointer".  Brilliant ideas lost
> to the history in most modern languages.  But I digress!

Maybe I am missing your point, but why shouldn't `foo' and (foo) be
different?  The only context where I can think of `foo' and (foo) being
treated the same is in some APL-like language that "automatically"
upgrades an atomic variable to a 0-dimensional array.

> I would never expect `setopt' to do networking.  In fact, I use `setq'
> everywhere, like almost everybody else, because I know exactly what it
> does, at the call site.  With `setopt', I can only guess.

Right, but this is not always correct.  Usually works, but some user
options are meant to perform a computation on setting a value.  Of
course, computation is dangerous, but it might also be the only way to
provide the necessary flexibility in some cases.

> But anyway, I have a practical reason to say all this:
>
> I have a literate Emacs configuration, mixed with notes.  In it, any
> part can add *some* package to the selected packages but any part can
> also rely on the availability of *all* installed packages.
>
> To get this freedom, I install the selected packages early on the
> after-init hook.  Then, all configuration blocks, evaluated in any
> order, can count on all installed packages.
>
> Any number of times, anywhere, and evaluated in any order:
>
>   (with-eval-after load '...
>      ...do anything, all packages already installed...)
>
>   (add-hook 'after-init-hook
>             ...do anything, all packages already installed...)
>
> Once, does not matter where:
>
>   (add-hook 'after-init-hook
>             ...install selected packages...
>             -99)

I see, this is an interesting approach.  But just to make sure, can you
confirm that package-vc doesn't break any of the assumptions you make
that are necessary for your configuration to work as intended?

> Rudy

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> The thing is the user options aren't strictly variables.  The usually
> are, but they are explicitly meant to operate on a higher abstraction
> level, sort of even in a separate name space.

Gotcha!

> But of course, I understand that not everyone feels comfortable with
> this.  So while I insist that package-vc-selected-packages ought to be
> a user option, I have also made `package-vc-install-selected-packages'
> autoloaded (I have yet to push all the commits) and invocable as a
> regular function

Perfect.  Win-win.  Thank you so much!

(I hope you also marked the function as interactive.  When playing with
VC package, I have half of the history in M-x and the other half in M-:.
Switching between the two all the time confuses me to no end.)

>> I like the clear distinction between `foo' and `(foo)' in Lisp.  I even
>> like the `*' in C that clearly says "a pointer".  Brilliant ideas lost
>> to the history in most modern languages.  But I digress!
>
> Maybe I am missing your point, but why shouldn't `foo' and (foo) be
> different?  The only context where I can think of `foo' and (foo)
> being treated the same is in some APL-like language ...

I apologize for the digression!  :)

To quickly explain, consider Swift code:

  class Things {
    // stored property
    var count = 0 // set during init
  }

and

  class Things {
    // computed property
    var count: Int {
       // arbitrary code
    }
  }

One calls `things.count' in both cases, with no parentheses, so one has
no idea whether `count' is a *stored* property (true variable) with O(1)
access or a *computed* property that runs in say in O(n).  Accidentally
quadratic code in no time!  Plus, one also know nothing about side
effects.

(Further, one can also define `willSet' and `didSet' on *stored*
properties, makes things even worse.)

In C, one sees `count' or `*count' or `count()' and everything is clear
on the first sight.  Similarly, in Scheme, one sees `count' or
`(count)'.

> I see, this is an interesting approach.  But just to make sure, can
> you confirm that package-vc doesn't break any of the assumptions you
> make that are necessary for your configuration to work as intended?

Suddenly ... so many issues!  :)

First, Emacs failed with "circular loads" errors (or something like
that).  I had to comment out `(package-vc-ensure-packages)' here to work
around the problem:

(defcustom package-vc-selected-packages '()
  [...]
  :set (lambda (sym val)
         (custom-set-default sym val)
         (package-vc-ensure-packages))

I then tried to change all packages to VC at once and got:

  user-error: Package has no VC data

Which one, I wondered.

Anyway, I reverted the config and decided to proceed in smaller steps.

As a side note, while I investigated various issues, I found myself
wishing for a hyperlink to

`package-vc-archive-spec-alist'

from the help page for

`package-vc-selected-packages'.

(The help page mentions does not use a hyperlink.)

Finally, I got to MELPA packages and got stuck altoegether.

When I

(1) require `package-vc', and then
(2) run `package-refresh-contents',

I get

Debugger entered--Lisp error: (file-error
  "https://melpa.org/packages/elpa-packages.eld" "Not found")
  signal(file-error ("https://melpa.org/packages/elpa-packages.eld" "Not
  found")) package--with-response-buffer-1("https://melpa.org/packages/"
  #f(compiled-function () #<bytecode 0x1fdb28378e1ccb5e>) :file
  "elpa-packages.eld" :async nil :error-function #f(compiled-function ()
  #<bytecode -0x3d0e809e66d1a0c>) :noerror nil)
  package--download-one-archive(("melpa"
  . "https://melpa.org/packages/") "elpa-packages.eld" nil)
  (condition-case nil (package--download-one-archive archive
  "elpa-packages.eld" async) ((debug error) (message "Failed to download
  `%s' archive." (car archive)))) (let ((archive (car tail)))
  (condition-case nil (package--download-one-archive archive
  "elpa-packages.eld" async) ((debug error) (message "Failed to download
  `%s' archive." (car archive)))) (setq tail (cdr tail))) (while tail
  (let ((archive (car tail))) (condition-case nil
  (package--download-one-archive archive "elpa-packages.eld" async)
  ((debug error) (message "Failed to download `%s' archive." (car
  archive)))) (setq tail (cdr tail)))) (let ((tail package-archives))
  (while tail (let ((archive (car tail))) (condition-case nil
  (package--download-one-archive archive "elpa-packages.eld" async)
  ((debug error) (message "Failed to download `%s' archive." (car
  archive)))) (setq tail (cdr tail)))))
  package-vc--download-and-read-archives(nil)
  run-hook-with-args(package-vc--download-and-read-archives nil)
  package-refresh-contents()
  funcall-interactively(package-refresh-contents)
  call-interactively(package-refresh-contents record nil)
  command-execute(package-refresh-contents record)
  execute-extended-command(nil "package-refresh-contents" nil)
  funcall-interactively(execute-extended-command nil
  "package-refresh-contents" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Rudy
--
"'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and
if it were so, it would be; but as it isn't, it ain't.  That's logic.'"
-- Lewis Carroll, Through the Looking Glass, 1871/1872

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> The thing is the user options aren't strictly variables.  The usually
>> are, but they are explicitly meant to operate on a higher abstraction
>> level, sort of even in a separate name space.
>
> Gotcha!

Worse still, any variable, user option or not can have a side effect,
when `add-variable-watcher' is used.

>> But of course, I understand that not everyone feels comfortable with
>> this.  So while I insist that package-vc-selected-packages ought to be
>> a user option, I have also made `package-vc-install-selected-packages'
>> autoloaded (I have yet to push all the commits) and invocable as a
>> regular function
>
> Perfect.  Win-win.  Thank you so much!
>
> (I hope you also marked the function as interactive.  When playing with
> VC package, I have half of the history in M-x and the other half in M-:.
> Switching between the two all the time confuses me to no end.)

Yes, I've done it, but haven't pushed the changes yet.

>>> I like the clear distinction between `foo' and `(foo)' in Lisp.  I even
>>> like the `*' in C that clearly says "a pointer".  Brilliant ideas lost
>>> to the history in most modern languages.  But I digress!
>>
>> Maybe I am missing your point, but why shouldn't `foo' and (foo) be
>> different?  The only context where I can think of `foo' and (foo)
>> being treated the same is in some APL-like language ...
>
> I apologize for the digression!  :)
>
> To quickly explain, consider Swift code:
>
>   class Things {
>     // stored property
>     var count = 0 // set during init
>   }
>
> and
>
>   class Things {
>     // computed property
>     var count: Int {
>        // arbitrary code
>     }
>   }
>
> One calls `things.count' in both cases, with no parentheses, so one has
> no idea whether `count' is a *stored* property (true variable) with O(1)
> access or a *computed* property that runs in say in O(n).  Accidentally
> quadratic code in no time!  Plus, one also know nothing about side
> effects.
>
> (Further, one can also define `willSet' and `didSet' on *stored*
> properties, makes things even worse.)
>
> In C, one sees `count' or `*count' or `count()' and everything is clear
> on the first sight.  Similarly, in Scheme, one sees `count' or
> `(count)'.

Interesting, I didn't know this.

>> I see, this is an interesting approach.  But just to make sure, can
>> you confirm that package-vc doesn't break any of the assumptions you
>> make that are necessary for your configuration to work as intended?
>
> Suddenly ... so many issues!  :)
>
> First, Emacs failed with "circular loads" errors (or something like
> that).  I had to comment out `(package-vc-ensure-packages)' here to work
> around the problem:

> (defcustom package-vc-selected-packages '()
>   [...]
>   :set (lambda (sym val)
>          (custom-set-default sym val)
>          (package-vc-ensure-packages))

That is unusual, `package-vc-ensure-packages' (or as it is now called
`package-vc-install-selected-packages') doesn't modify
`package-vc-selected-packages' so the setter shouldn't be invoked
either?

> I then tried to change all packages to VC at once and got:
>
>   user-error: Package has no VC data
>
> Which one, I wondered.

Good point, I can add that to the error message.

> Anyway, I reverted the config and decided to proceed in smaller steps.
>
> As a side note, while I investigated various issues, I found myself
> wishing for a hyperlink to
>
> `package-vc-archive-spec-alist'
>
> from the help page for
>
> `package-vc-selected-packages'.
>
> (The help page mentions does not use a hyperlink.)

These issues have already been solved.

> Finally, I got to MELPA packages and got stuck altoegether.
>
> When I
>
> (1) require `package-vc', and then
> (2) run `package-refresh-contents',
>
> I get
>
> Debugger entered--Lisp error: (file-error
>   "https://melpa.org/packages/elpa-packages.eld" "Not found")
>   signal(file-error ("https://melpa.org/packages/elpa-packages.eld" "Not
>   found")) package--with-response-buffer-1("https://melpa.org/packages/"
>   #f(compiled-function () #<bytecode 0x1fdb28378e1ccb5e>) :file
>   "elpa-packages.eld" :async nil :error-function #f(compiled-function ()
>   #<bytecode -0x3d0e809e66d1a0c>) :noerror nil)
>   package--download-one-archive(("melpa"
>   . "https://melpa.org/packages/") "elpa-packages.eld" nil)
>   (condition-case nil (package--download-one-archive archive
>   "elpa-packages.eld" async) ((debug error) (message "Failed to download
>   `%s' archive." (car archive)))) (let ((archive (car tail)))
>   (condition-case nil (package--download-one-archive archive
>   "elpa-packages.eld" async) ((debug error) (message "Failed to download
>   `%s' archive." (car archive)))) (setq tail (cdr tail))) (while tail
>   (let ((archive (car tail))) (condition-case nil
>   (package--download-one-archive archive "elpa-packages.eld" async)
>   ((debug error) (message "Failed to download `%s' archive." (car
>   archive)))) (setq tail (cdr tail)))) (let ((tail package-archives))
>   (while tail (let ((archive (car tail))) (condition-case nil
>   (package--download-one-archive archive "elpa-packages.eld" async)
>   ((debug error) (message "Failed to download `%s' archive." (car
>   archive)))) (setq tail (cdr tail)))))
>   package-vc--download-and-read-archives(nil)
>   run-hook-with-args(package-vc--download-and-read-archives nil)
>   package-refresh-contents()
>   funcall-interactively(package-refresh-contents)
>   call-interactively(package-refresh-contents record nil)
>   command-execute(package-refresh-contents record)
>   execute-extended-command(nil "package-refresh-contents" nil)
>   funcall-interactively(execute-extended-command nil
>   "package-refresh-contents" nil)
>   call-interactively(execute-extended-command nil nil)
>   command-execute(execute-extended-command)

Do you have `debug-on-error' enabled?  I don't use MELPA, so this error
doesn't occur in my case.  I've modified `package-archives', but even so
the most I get in my case is

    Failed to download ‘melpa’ archive.

> Rudy
> --
> "'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and
> if it were so, it would be; but as it isn't, it ain't.  That's logic.'"
> -- Lewis Carroll, Through the Looking Glass, 1871/1872
>
> Rudolf Adamkovič <salutis@me.com> [he/him]
> Studenohorská 25
> 84103 Bratislava
> Slovakia

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> Yes, I've done it, but haven't pushed the changes yet.

Perfect.  Cannot wait to try the new version!

> That is unusual, `package-vc-ensure-packages' (or as it is now called
> `package-vc-install-selected-packages') doesn't modify
> `package-vc-selected-packages' so the setter shouldn't be invoked
> either?

Strange!  Something somewhere triggered that error, without me touching
the source code.  Knowing about the "magic" of reactive programming, I
commented out the "did set" action, just to get through the hump.

> Good point, I can add that to the error message.

Great idea!

> Do you have `debug-on-error' enabled?  I don't use MELPA, so this
> error doesn't occur in my case.  I've modified `package-archives', but
> even so the most I get in my case is
>
>     Failed to download ‘melpa’ archive.

I first got

  Failed to download ‘melpa-stable’ archive.
  Failed to download ‘melpa’ archive.

Then I enabled `debug-on-error' to give you as much info as I can.

Rudy
-- 
"Be especially critical of any statement following the word
'obviously.'"
-- Anna Pell Wheeler, 1883-1966

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

>> Do you have `debug-on-error' enabled?  I don't use MELPA, so this
>> error doesn't occur in my case.  I've modified `package-archives', but
>> even so the most I get in my case is
>>
>>     Failed to download ‘melpa’ archive.
>
> I first got
>
>   Failed to download ‘melpa-stable’ archive.
>   Failed to download ‘melpa’ archive.
>
> Then I enabled `debug-on-error' to give you as much info as I can.

Ah ok, but that is to be excepted for now.  I am not part of the MELPA
project, so I can't "force" them to provide package specifications.  If
they are missing, then we must rely on heuristics to download source
packages.  My hope is that they will add the file too at some point.

> Rudy

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Yes, I've done it, but haven't pushed the changes yet.
>
> Perfect.  Cannot wait to try the new version!

To avoid unnecessary an back and forth, pushing commits and reverting
them, I have pushed the proposed changes to the
"scratch/package-vc-fixes" branch, that I will rebase onto master when
the changes are ready.  If you have the time, try it out and see if the
issues you had were resolved.

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> To avoid unnecessary an back and forth, pushing commits and reverting
> them, I have pushed the proposed changes to the
> "scratch/package-vc-fixes" branch, that I will rebase onto master when
> the changes are ready.  If you have the time, try it out and see if
> the issues you had were resolved.

I have just tried it, and below, I summarize my experience.

Please note that I have not read any source code, nor followed the
dicussion word for word.  Thus, I write my reports as a new user.
Hopefully, you find them useful.

1. Simplify the configuration.
   
   (add-hook 'after-init-hook
             #'package-vc-install-selected-packages
             -99)
   
   No `require', no `closure', no nothing.  Beautiful.

   Restart Emacs.

2. Try "refreshing" (no) packages.

   M-x package-vc-refresh

   Got:

     completing-read-default: Wrong type argument: stringp, t

3. Try to install Modus Themes

   Update the configuration from `package' to `package-vc':
   
     (with-eval-after-load 'package-vc
       (add-to-list 'package-vc-selected-packages 'modus-themes))
   
   Delete `modus-themes-<version>' from `~/.emacs.d/elpa'.
   
   Restart Emacs.

   Got:
   
     user-error: Unknown package to fetch: nil
   
   *scratches head*
   
   Oh, the list!  Change the configuration again:
   
     (with-eval-after-load 'package-vc
       (add-to-list 'package-vc-selected-packages '(modus-themes)))
   
   Restart Emacs.

   Got:
   
     run-hooks: Symbol’s function definition is void:
     modus-themes-load-operandi
   
   Check `~/.emacs.d/elpa' to see if it contains `modus-themes'.

   It does.
   
   *scratches head*
   
   Check `package-vc-selected-packages':

     Its value is ((modus-themes))
   
   Try `M-x package-vc-update'.

   Got:
   
     completing-read-default: Wrong type argument: stringp, t
   
   Check `load-path' if it contains `modus-themes'.

   It does.
   
   Execute `M-: (require 'modus-themes)' followed by `M-x
   modus-themes-load-operandi'.

   The theme loads.

   *scatches head*
   
   Think: "It seems that Emacs needs to process auto-loads."
   
   Tries `package-refresh' again.

   Got:
   
     completing-read-default: Wrong type argument: stringp, t

4. Give up and report back.

   :)

Amost there!

> Ah ok, but that is to be excepted for now.  I am not part of the MELPA
> project, so I can't "force" them to provide package specifications.
> If they are missing, then we must rely on heuristics to download
> source packages.  My hope is that they will add the file too at some
> point.

I see.  One can still install MELPA packages via a URL, right?

P.S. 1

I remain unsure what `refresh' means.  According to the documentation,
it "refresh[es] the installation for [the] package".  But what does it
mean?  The command needs a more detailed documentation and perhaps also
a more descriptive name.  I understand the `update', but not the
`refresh'.

P.S. 2

I could not find `package-vc-update-all' akin `package-update-all'.  Not
that I needed it, but I looked for it in the case I manage to install
more packages.

P.S. 3

The `package-vc-selected-packages' documentation still does not link the
`package-vc-archive-spec-alist' variable.  But I noticed that the
variable has a new name, with a double dash for private use.  Either
way, I just wanted to mention this fact for completness.

Rudy
-- 
"Simplicity is complexity resolved."
-- Constantin Brâncuși, 1876-1957

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> To avoid unnecessary an back and forth, pushing commits and reverting
>> them, I have pushed the proposed changes to the
>> "scratch/package-vc-fixes" branch, that I will rebase onto master when
>> the changes are ready.  If you have the time, try it out and see if
>> the issues you had were resolved.
>
> I have just tried it, and below, I summarize my experience.
>
> Please note that I have not read any source code, nor followed the
> dicussion word for word.  Thus, I write my reports as a new user.
> Hopefully, you find them useful.
>
> 1. Simplify the configuration.
>    
>    (add-hook 'after-init-hook
>              #'package-vc-install-selected-packages
>              -99)
>    
>    No `require', no `closure', no nothing.  Beautiful.
>
>    Restart Emacs.
>
> 2. Try "refreshing" (no) packages.
>
>    M-x package-vc-refresh

As we seem to use package.el in different ways, can you clarify what
your intention is when invoking `package-vc-refresh'?  The documentation
string tries to clarify what is does, but perhaps the name is confusing:

      "Refresh the installation for package given by PKG-DESC.
    Refreshing an installation means scraping for new autoload
    cookies, re-compiling Emacs Lisp files, building and installing
    any documentation, downloading any missing dependencies.  This
    command does not fetch new revisions from a remote server.  That
    is the responsibility of `package-vc-update'.  Interactively,
    prompt for the name of the package to refresh."

Reload?  Regenerate?  Redo?

>    Got:
>
>      completing-read-default: Wrong type argument: stringp, t

Found an fixed the bug:


[-- Attachment #2: Type: text/plain, Size: 688 bytes --]

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index 726e21ee6d..70fb6ac399 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -534,7 +534,7 @@ package-vc--read-package-name
                            (and-let* ((extras (package-desc-extras (cadr pkg)))
                                       (url (alist-get :url extras))
                                       ((package-vc--guess-backend url)))))))
-                   nil (not allow-url)))
+                   (not allow-url)))
 
 (defun package-vc--read-package-desc (prompt &optional installed)
   "Query the user for a source package and return a description with PROMPT.

[-- Attachment #3: Type: text/plain, Size: 2897 bytes --]


(Btw, I am trying to force-push to the scratch branch but this appears
not to work.  Wasn't that the point of a scratch branch?)

> 3. Try to install Modus Themes
>
>    Update the configuration from `package' to `package-vc':
>    
>      (with-eval-after-load 'package-vc
>        (add-to-list 'package-vc-selected-packages 'modus-themes))
>    
>    Delete `modus-themes-<version>' from `~/.emacs.d/elpa'.
>    
>    Restart Emacs.
>
>    Got:
>    
>      user-error: Unknown package to fetch: nil
>    
>    *scratches head*
>    
>    Oh, the list!  Change the configuration again:
>    
>      (with-eval-after-load 'package-vc
>        (add-to-list 'package-vc-selected-packages '(modus-themes)))
>    
>    Restart Emacs.
>
>    Got:
>    
>      run-hooks: Symbol’s function definition is void:
>      modus-themes-load-operandi
>    
>    Check `~/.emacs.d/elpa' to see if it contains `modus-themes'.
>
>    It does.
>    
>    *scratches head*
>    
>    Check `package-vc-selected-packages':
>
>      Its value is ((modus-themes))
>    
>    Try `M-x package-vc-update'.
>
>    Got:
>    
>      completing-read-default: Wrong type argument: stringp, t
>    
>    Check `load-path' if it contains `modus-themes'.
>
>    It does.
>    
>    Execute `M-: (require 'modus-themes)' followed by `M-x
>    modus-themes-load-operandi'.
>
>    The theme loads.
>
>    *scatches head*
>    
>    Think: "It seems that Emacs needs to process auto-loads."
>    
>    Tries `package-refresh' again.
>
>    Got:
>    
>      completing-read-default: Wrong type argument: stringp, t
>
> 4. Give up and report back.
>
>    :)
>
> Amost there!
>
>> Ah ok, but that is to be excepted for now.  I am not part of the MELPA
>> project, so I can't "force" them to provide package specifications.
>> If they are missing, then we must rely on heuristics to download
>> source packages.  My hope is that they will add the file too at some
>> point.
>
> I see.  One can still install MELPA packages via a URL, right?
>
> P.S. 1
>
> I remain unsure what `refresh' means.  According to the documentation,
> it "refresh[es] the installation for [the] package".  But what does it
> mean?  The command needs a more detailed documentation and perhaps also
> a more descriptive name.  I understand the `update', but not the
> `refresh'.
>
> P.S. 2
>
> I could not find `package-vc-update-all' akin `package-update-all'.  Not
> that I needed it, but I looked for it in the case I manage to install
> more packages.
>
> P.S. 3
>
> The `package-vc-selected-packages' documentation still does not link the
> `package-vc-archive-spec-alist' variable.  But I noticed that the
> variable has a new name, with a double dash for private use.  Either
> way, I just wanted to mention this fact for completness.
>
> Rudy

Re: feature/package-vc has been merged

[Stefan Monnier]

> (Btw, I am trying to force-push to the scratch branch but this appears
> not to work.  Wasn't that the point of a scratch branch?)

The repository does not allow force pushes.  There's no finer config as
far as I know.  You can still technically force push to any branch by
deleting the branch and pushing anew, i.e. something like:

    git push origin :scratch/foo
    git push origin HEAD:scratch/foo

We only tolerate that in scratch/* branches, tho.


        Stefan

Re: feature/package-vc has been merged

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (Btw, I am trying to force-push to the scratch branch but this appears
>> not to work.  Wasn't that the point of a scratch branch?)
>
> The repository does not allow force pushes.  There's no finer config as
> far as I know.  You can still technically force push to any branch by
> deleting the branch and pushing anew, i.e. something like:
>
>     git push origin :scratch/foo
>     git push origin HEAD:scratch/foo
>
> We only tolerate that in scratch/* branches, tho.

I suspected as much.  That is why I pushed the changes under that
branch.  Thanks for the clarification, I've pushed the changes.

Re: feature/package-vc has been merged

[Philip Kaludercic]

(sorry, this message got sent out too early.  Continuing my reply here)

> 3. Try to install Modus Themes
>
>    Update the configuration from `package' to `package-vc':
>    
>      (with-eval-after-load 'package-vc
>        (add-to-list 'package-vc-selected-packages 'modus-themes))
>    
>    Delete `modus-themes-<version>' from `~/.emacs.d/elpa'.
>    
>    Restart Emacs.
>
>    Got:
>    
>      user-error: Unknown package to fetch: nil
>
>    *scratches head*
>    
>    Oh, the list!  Change the configuration again:
>    
>      (with-eval-after-load 'package-vc
>        (add-to-list 'package-vc-selected-packages '(modus-themes)))
>    
>    Restart Emacs.

Two ways to resolve this:  Either throw an error earlier, which might be
annoying but the right thing™, or dwim and accept that the value might
not be readable in ECI.

>    Got:
>    
>      run-hooks: Symbol’s function definition is void:
>      modus-themes-load-operandi
>    
>    Check `~/.emacs.d/elpa' to see if it contains `modus-themes'.
>
>    It does.
>    
>    *scratches head*
>    
>    Check `package-vc-selected-packages':
>
>      Its value is ((modus-themes))
>    
>    Try `M-x package-vc-update'.
>
>    Got:
>    
>      completing-read-default: Wrong type argument: stringp, t
>
>    Check `load-path' if it contains `modus-themes'.
>
>    It does.
>    
>    Execute `M-: (require 'modus-themes)' followed by `M-x
>    modus-themes-load-operandi'.
>
>    The theme loads.
>
>    *scatches head*
>    
>    Think: "It seems that Emacs needs to process auto-loads."
>    
>    Tries `package-refresh' again.
>
>    Got:
>    
>      completing-read-default: Wrong type argument: stringp, t

This issue should be resolved with the above patch, but the question
still remains what went wrong in the above installation.  Can you
check/post the -pkg.el and -autoloads.el file?

> 4. Give up and report back.
>
>    :)
>
> Amost there!
>
>> Ah ok, but that is to be excepted for now.  I am not part of the MELPA
>> project, so I can't "force" them to provide package specifications.
>> If they are missing, then we must rely on heuristics to download
>> source packages.  My hope is that they will add the file too at some
>> point.
>
> I see.  One can still install MELPA packages via a URL, right?

package-vc.el (just like package.el) doesn't treat MELPA packages as
special or different.  If you give `package-vc-install' a URL, it will
try to guess what has to be done -- this doesn't always work, but the
guessing can still be improved.

What you can do with `package-vc-selected-packages' is give your own
ELPA-esque specification.  That should work most of the time if a
package is not available on GNU ELPA or NonGNU ELPA.

> P.S. 1
>
> I remain unsure what `refresh' means.  According to the documentation,
> it "refresh[es] the installation for [the] package".  But what does it
> mean?  The command needs a more detailed documentation and perhaps also
> a more descriptive name.  I understand the `update', but not the
> `refresh'.

Did the above docstring clarify that point?

> P.S. 2
>
> I could not find `package-vc-update-all' akin `package-update-all'.  Not
> that I needed it, but I looked for it in the case I manage to install
> more packages.

It doesn't exist.  My idea was that package-update-all would handle
that, but there might be a value in a command that only updates source
packages?

> P.S. 3
>
> The `package-vc-selected-packages' documentation still does not link the
> `package-vc-archive-spec-alist' variable.  But I noticed that the
> variable has a new name, with a double dash for private use.  Either
> way, I just wanted to mention this fact for completness.

As mentioned above there are a few commits I haven't been able to push
because emacs.git is preventing me from force pushing.  I'll try to get
them up there some other way, and then rewrite the history before
rebasing the changes onto master.

> Rudy

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> (sorry, this message got sent out too early.  Continuing my reply
> here)

Gotcha!  Replying to both messages here.

>>    M-x package-vc-refresh
>
> As we seem to use package.el in different ways, can you clarify what
> your intention is when invoking `package-vc-refresh'?

I wanted to see what it does with empty `package-vc-selected-packages'.

Yeah, I know.  A strange "first thing one does".  :)

> The documentation string tries to clarify what is does, but perhaps
> the name is confusing:
>
>       "Refresh the installation for package given by PKG-DESC.
>     Refreshing an installation means scraping for new autoload
>     cookies, re-compiling Emacs Lisp files, building and installing
>     any documentation, downloading any missing dependencies.  This
>     command does not fetch new revisions from a remote server.  That
>     is the responsibility of `package-vc-update'.  Interactively,
>     prompt for the name of the package to refresh."
>
> Reload?  Regenerate?  Redo?

So good!  The documentation before said that it refreshes the given
package.  Hence, I kept scratching my head.  Now, it all makes sense.

As for the name, how about rebuild?  All those steps normally happen
during a build process.  I think "refresh" sounds much too close to
"update".

>>    Oh, the list!  Change the configuration again:
>>
>>      (with-eval-after-load 'package-vc
>>        (add-to-list 'package-vc-selected-packages '(modus-themes)))
>>
>>    Restart Emacs.
>
> Two ways to resolve this: Either throw an error earlier, which might
> be annoying but the right thing™, or dwim and accept that the value
> might not be readable in ECI.

Early or late, we need to show at least the traditional Emacs type
predicate error.  I say that because it took me ten minutes to find the
problem. :)

> This issue should be resolved with the above patch, but the question
> still remains what went wrong in the above installation.  Can you
> check/post the -pkg.el and -autoloads.el file?

Oops, I since then restored everything back.

No worries though, I will do more testing going forward!

> package-vc.el (just like package.el) doesn't treat MELPA packages as
> special or different.  If you give `package-vc-install' a URL, it will
> try to guess what has to be done -- this doesn't always work, but the
> guessing can still be improved.
>
> What you can do with `package-vc-selected-packages' is give your own
> ELPA-esque specification.  That should work most of the time if a
> package is not available on GNU ELPA or NonGNU ELPA.

I see.  Smart!

> Did the above [`package-vc-refresh'] docstring clarify that point?

Definitely!

>> I could not find `package-vc-update-all' akin `package-update-all'.
>> Not that I needed it, but I looked for it in the case I manage to
>> install more packages.
>
> It doesn't exist.  My idea was that package-update-all would handle
> that, but there might be a value in a command that only updates source
> packages?

I think that would make sense.

>> The `package-vc-selected-packages' documentation [...]
>
> As mentioned above there are a few commits I haven't been able to push
> because emacs.git is preventing me from force pushing.  I'll try to
> get them up there some other way, and then rewrite the history before
> rebasing the changes onto master.

So much for "scratch" branches. :-/

Rudy
-- 
"Be especially critical of any statement following the word
'obviously.'"
-- Anna Pell Wheeler, 1883-1966

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

>> The documentation string tries to clarify what is does, but perhaps
>> the name is confusing:
>>
>>       "Refresh the installation for package given by PKG-DESC.
>>     Refreshing an installation means scraping for new autoload
>>     cookies, re-compiling Emacs Lisp files, building and installing
>>     any documentation, downloading any missing dependencies.  This
>>     command does not fetch new revisions from a remote server.  That
>>     is the responsibility of `package-vc-update'.  Interactively,
>>     prompt for the name of the package to refresh."
>>
>> Reload?  Regenerate?  Redo?
>
> So good!  The documentation before said that it refreshes the given
> package.  Hence, I kept scratching my head.  Now, it all makes sense.
>
> As for the name, how about rebuild?  All those steps normally happen
> during a build process.  I think "refresh" sounds much too close to
> "update".

That sounds good, I'll rename it to `package-vc-rebuild'.  Thanks!

>>>    Oh, the list!  Change the configuration again:
>>>
>>>      (with-eval-after-load 'package-vc
>>>        (add-to-list 'package-vc-selected-packages '(modus-themes)))
>>>
>>>    Restart Emacs.
>>
>> Two ways to resolve this: Either throw an error earlier, which might
>> be annoying but the right thing™, or dwim and accept that the value
>> might not be readable in ECI.
>
> Early or late, we need to show at least the traditional Emacs type
> predicate error.  I say that because it took me ten minutes to find the
> problem. :)

I am still confused as to how the error came about in the first place

     user-error: Unknown package to fetch: nil

This means that `package-vc-install' was invoked with nil for
NAME-OR-URL, right?  I suspect that the condition in

  (unless (and name (package-installed-p name) (package-vc-p pkg-desc)) ...)
  
`package-vc-install-selected-packages' could be at fault here.  I will
think about it...

>>> I could not find `package-vc-update-all' akin `package-update-all'.
>>> Not that I needed it, but I looked for it in the case I manage to
>>> install more packages.
>>
>> It doesn't exist.  My idea was that package-update-all would handle
>> that, but there might be a value in a command that only updates source
>> packages?
>
> I think that would make sense.

OK, I'll add it then.

>>> The `package-vc-selected-packages' documentation [...]
>>
>> As mentioned above there are a few commits I haven't been able to push
>> because emacs.git is preventing me from force pushing.  I'll try to
>> get them up there some other way, and then rewrite the history before
>> rebasing the changes onto master.
>
> So much for "scratch" branches. :-/

I am not sure what you mean?  The issue has since been resolved.

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

> That sounds good, I'll rename it to `package-vc-rebuild'.  Thanks!

Awesome!  Nobody will confuse "rebuild" and "update".

> I am still confused as to how the error came about in the first place

I will re-test.

> OK, I'll add it then.

Thanks!

>>> [...] I haven't been able to push because emacs.git is preventing me
>>> from force pushing.  I'll try to get them up there some other way,
>>> and then rewrite the history before rebasing the changes onto
>>> master.
>>
>> So much for "scratch" branches. :-/
>
> I am not sure what you mean?  The issue has since been resolved.

I thought you could not force-push to the scratch branch.

Rudy
-- 
"I love deadlines.  I love the whooshing noise they make as they go by."
-- Douglas Adams, The Salmon of Doubt, 2002

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip,

I pulled the latest commits and tried to install `modus-themes',
`vterm', `mentor', and `orderless'.  Everything worked like magic.

In each case, I simply added the `vc' prefix and a pair of parentheses
in `-vc-selected-packages'.  I then restarted Emacs and everything
always installed and worked flawlessly.  The `vterm' package also
compiled its C library without any problems.  Magic!

Old:

(with-eval-after-load 'package
  (add-to-list 'package-vc-selected-packages 'modus-themes))

New:

(with-eval-after-load 'package-vc
  (add-to-list 'package-vc-selected-packages '(modus-themes)))

Then:

(add-hook 'after-init-hook
          #'package-vc-install-selected-packages
          -99)

[Simpler than `package', which needs additional babysitting with
`(package-refresh-contents)'.  Perfect!]

That said, When I updated my config to install from VC, I noticed that
`elpa' in `.emacs.d' now contained two versions of the packages, one
from `package' and one from `package-vc'.

I executed `package-autoremove', but it made me wonder if having two
versions, even if temporarily, will cause problems for the people
upgrading from `package' to `package-vc'.

As a side note, you mentioned that `package-install' works for both
"normal" and VC packages?  I guess `package-autoremove' does that too?
I looked at the documentation for both and they mention just
`package-selected-packages'.

And speaking of `-autoremove', should we also add `-vc-autoremove'?

For the only problem I have, I could not install `geiser'.  I got
"user-error: Package has no VC data", and I kept wondering what the user
should do in this situation.  I had no idea where to look.

I will try more packages in the coming days or weeks!

P.S. The documentation for the `package-vc-selected-packages' variable
contains a typo "... you cal also use ...".

Rudy
-- 
"Logic is a science of the necessary laws of thought, without which no
employment of the understanding and the reason takes place."
-- Immanuel Kant, 1785

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Stefan Monnier]

> [Simpler than `package', which needs additional babysitting with
> `(package-refresh-contents)'.  Perfect!]

That should not be necessary with `package-install` either.
If you still find it necessary, please report it as a bug (and put me
in the `X-Debbugs-Cc`).


        Stefan

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> [Simpler than `package', which needs additional babysitting with
>> `(package-refresh-contents)'.  Perfect!]
>
> That should not be necessary with `package-install` either.
> If you still find it necessary, please report it as a bug (and put me
> in the `X-Debbugs-Cc`).

TIL!  It works without any babysitting.

I simplified ...

(add-hook
 'after-init-hook
 (lambda ()
   (require 'package)
   (unless (seq-every-p #'package-installed-p package-selected-packages)
     (package-refresh-contents)
     (package-install-selected-packages t)))
 -99)

... to ...

(add-hook 'after-init-hook
          (lambda () package-install-selected-packages t)
          -99)

... and my Emacs configuration bootstrapped without any problems.

That said, the VC command still wins, for does not need the lambda. :)

Perhaps we should default the `noconfirm' parameter to `t' when the user
uses the command non-interactively?  While on it, we could also rename
it to `no-confirm' to make the spellchecker, and thus the reader,
happier.  :)

Either way, thank you for letting me know!

Rudy
-- 
"Chop your own wood and it will warm you twice."
-- Henry Ford; Francis Kinloch, 1819; Henry David Thoreau, 1854

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Stefan Kangas]

Rudolf Adamkovič <salutis@me.com> writes:

> And speaking of `-autoremove', should we also add `-vc-autoremove'?

My concern is that it seems easy to accidentally delete some changes
hidden away in a branch or stash.  Or even on the master/main branch.

Should such a command double check that no local changes exist before
autoremoving, and ask for an additional confirmation if that is the
case, perhaps?

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Stefan Kangas <stefankangas@gmail.com> writes:

>> And speaking of `-autoremove', should we also add `-vc-autoremove'?
>
> My concern is that it seems easy to accidentally delete some changes
> hidden away in a branch or stash.  Or even on the master/main branch.

Good point!

> Should such a command double check that no local changes exist before
> autoremoving, and ask for an additional confirmation if that is the
> case, perhaps?

That would make it super-duper, but I guess we cannot do that check
easily, can we?  The user can have uncommitted changes but also commits
on various branches, non-versioned patch and scratch files lying around,
and who knows what else.

Rudy
-- 
"Logic is a science of the necessary laws of thought, without which no
employment of the understanding and the reason takes place."
-- Immanuel Kant, 1785

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip,
>
> I pulled the latest commits and tried to install `modus-themes',
> `vterm', `mentor', and `orderless'.  Everything worked like magic.
>
> In each case, I simply added the `vc' prefix and a pair of parentheses
> in `-vc-selected-packages'.  I then restarted Emacs and everything
> always installed and worked flawlessly.  The `vterm' package also
> compiled its C library without any problems.  Magic!
>
> Old:
>
> (with-eval-after-load 'package
>   (add-to-list 'package-vc-selected-packages 'modus-themes))
>
> New:
>
> (with-eval-after-load 'package-vc
>   (add-to-list 'package-vc-selected-packages '(modus-themes)))
>
> Then:
>
> (add-hook 'after-init-hook
>           #'package-vc-install-selected-packages
>           -99)
>
> [Simpler than `package', which needs additional babysitting with
> `(package-refresh-contents)'.  Perfect!]

I'm very happy to hear this!

> That said, When I updated my config to install from VC, I noticed that
> `elpa' in `.emacs.d' now contained two versions of the packages, one
> from `package' and one from `package-vc'.
>
> I executed `package-autoremove', but it made me wonder if having two
> versions, even if temporarily, will cause problems for the people
> upgrading from `package' to `package-vc'.

This seems like an issue of what perspective one takes.  I don't see
package-vc as an "upgrade" over package, but as an alternative backend
for fetching source code.  Not even I plan to use it for everything,
rather I'd install a source package when I am hacking on some package,
and when I have sent out the patches and am done with it, I'd remove the
source package and be left with the previous installation.

Removing the regular package would lead to too much confusion in this
case.

Perhaps this should be documented?

> As a side note, you mentioned that `package-install' works for both
> "normal" and VC packages?  I guess `package-autoremove' does that too?
> I looked at the documentation for both and they mention just
> `package-selected-packages'.

I don't think I said "package-install works for both normal and VC
packages"?  package-install fetches a tarball, package-vc-install
fetches a repository?

> And speaking of `-autoremove', should we also add `-vc-autoremove'?

I don't think so, because -autoremove is for removing non-selected
packages, while all source packages are selected (you'll find that
package-vc uses regular tarball packages if dependencies have to be
installed).

> For the only problem I have, I could not install `geiser'.  I got
> "user-error: Package has no VC data", and I kept wondering what the user
> should do in this situation.  I had no idea where to look.

That sounds wrong, there seems to be a specification in
elpa-packages.eld:

  ("geiser" :url "https://gitlab.com/emacs-geiser/geiser.git" :lisp-dir
  "elisp" :readme "readme.org" :doc "doc/geiser.texi")

Maybe this is related to MELPA?  They don't have package specification
lists yet, and if their package is prioritised (due to their versioning
scheme), maybe this could confused package-vc?

> I will try more packages in the coming days or weeks!

Very appreciative.  But this is a good sign that after bug#59109 has
been resolved, the changes can be applied back onto master.

> P.S. The documentation for the `package-vc-selected-packages' variable
> contains a typo "... you cal also use ...".

Will fix, thanks!

> Rudy

Re: feature/package-vc has been merged

[Rudolf Adamkovič]

Philip Kaludercic <philipk@posteo.net> writes:

>> That said, When I updated my config to install from VC, I noticed
>> that `elpa' in `.emacs.d' now contained two versions of the packages,
>> one from `package' and one from `package-vc'.
>
> This seems like an issue of what perspective one takes.  I don't see
> package-vc as an "upgrade" over package, but as an alternative backend
> for fetching source code.  Not even I plan to use it for everything,
> rather I'd install a source package when I am hacking on some package,
> and when I have sent out the patches and am done with it, I'd remove
> the source package and be left with the previous installation.

I like the backend idea, but the two systems, non-VC and VC, define
separate lists of selected packages.  But then, what happens when the
user selects the same package in both lists?  Will that lead to any
problems?  If so, then we should not allow the user to do that, or deal
with it in some way.

>> As a side note, you mentioned that `package-install' works for both
>> "normal" and VC packages?  I guess `package-autoremove' does that too?
>> I looked at the documentation for both and they mention just
>> `package-selected-packages'.
>
> I don't think I said "package-install works for both normal and VC
> packages"?  package-install fetches a tarball, package-vc-install
> fetches a repository?

I apologize for confusion.

Perhaps you said that for `package-update'?  (I remain a bit confused
about which commands apply to both VC and non-VC packages.)

>> And speaking of `-autoremove', should we also add `-vc-autoremove'?
>
> I don't think so, because -autoremove is for removing non-selected
> packages, while all source packages are selected (you'll find that
> package-vc uses regular tarball packages if dependencies have to be
> installed).

Really?  I thought VC `-install' works like the non-VC kind, meaning it
does not select anything, just installs a package so that the user can
quickly try it.  But you say "all source packages are selected".

>> For the only problem I have, I could not install `geiser'.  I got
>> "user-error: Package has no VC data", and I kept wondering what the user
>> should do in this situation.  I had no idea where to look.
>
> That sounds wrong, there seems to be a specification in
> elpa-packages.eld:
>
>   ("geiser" :url "https://gitlab.com/emacs-geiser/geiser.git" :lisp-dir
>   "elisp" :readme "readme.org" :doc "doc/geiser.texi")
>
> Maybe this is related to MELPA?  They don't have package specification
> lists yet, and if their package is prioritised (due to their versioning
> scheme), maybe this could confused package-vc?

I tried the following two experiments, starting from:

(with-eval-after-load 'package
  (add-to-list
    'package-archives
    '("melpa" . "https://melpa.org/packages/"))
  (add-to-list
    'package-archives
    '("melpa-stable" . "https://stable.melpa.org/packages/")))

Experiment 1:  Add MELPA to the end of the list.

(with-eval-after-load 'package
  (add-to-list
    'package-archives
    '("melpa" . "https://melpa.org/packages/")
    t)  ; <------ `t' means the end of the list
  (add-to-list
    'package-archives
    '("melpa-stable" . "https://stable.melpa.org/packages/")
    t)) ; <------ `t' means the end of the list

This *did not* help.

Experiment 2: Comment out MELPA.

;; (with-eval-after-load 'package
;;   (add-to-list
;;     'package-archives
;;     '("melpa" . "https://melpa.org/packages/"))
;;   (add-to-list
;;     'package-archives
;;     '("melpa-stable" . "https://stable.melpa.org/packages/")))

This *did* help.

If the user has MELPA at the end of the list, it should just work.  So,
that needs fixing.  But the question becomes, what should happen if the
user has MELPA at the beginning of the list.  VC could fail, but it
could also proceed, silently or with a message.

>> I will try more packages in the coming days or weeks!
>
> Very appreciative.  But this is a good sign that after bug#59109 has
> been resolved, the changes can be applied back onto master.
>
>> P.S. The documentation for the `package-vc-selected-packages' variable
>> contains a typo "... you cal also use ...".
>
> Will fix, thanks!

Super!

Also, the documentation for the `package-vc-install' function needs some
blank lines between paragraphs.  :)

Rudy
-- 
"Strange as it may sound, the power of mathematics rests on its evasion
of all unnecessary thought and on its wonderful saving of mental
operations."
-- Ernst Mach, 1838-1916

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>> That said, When I updated my config to install from VC, I noticed
>>> that `elpa' in `.emacs.d' now contained two versions of the packages,
>>> one from `package' and one from `package-vc'.
>>
>> This seems like an issue of what perspective one takes.  I don't see
>> package-vc as an "upgrade" over package, but as an alternative backend
>> for fetching source code.  Not even I plan to use it for everything,
>> rather I'd install a source package when I am hacking on some package,
>> and when I have sent out the patches and am done with it, I'd remove
>> the source package and be left with the previous installation.
>
> I like the backend idea, but the two systems, non-VC and VC, define
> separate lists of selected packages.  But then, what happens when the
> user selects the same package in both lists?  Will that lead to any
> problems?  If so, then we should not allow the user to do that, or deal
> with it in some way.

No, I have "arrogantly" made sure that source packages are prioritised
over regular (tarball, built-in, system) packages.

>>> As a side note, you mentioned that `package-install' works for both
>>> "normal" and VC packages?  I guess `package-autoremove' does that too?
>>> I looked at the documentation for both and they mention just
>>> `package-selected-packages'.
>>
>> I don't think I said "package-install works for both normal and VC
>> packages"?  package-install fetches a tarball, package-vc-install
>> fetches a repository?
>
> I apologize for confusion.
>
> Perhaps you said that for `package-update'?  (I remain a bit confused
> about which commands apply to both VC and non-VC packages.)

Unless I have forgotten something, the only commands from package.el
that apply to both package.el and package-vc.el are

- package-delete
- package-update
- package-update-all

>>> And speaking of `-autoremove', should we also add `-vc-autoremove'?
>>
>> I don't think so, because -autoremove is for removing non-selected
>> packages, while all source packages are selected (you'll find that
>> package-vc uses regular tarball packages if dependencies have to be
>> installed).
>
> Really?  I thought VC `-install' works like the non-VC kind, meaning it
> does not select anything, just installs a package so that the user can
> quickly try it.  But you say "all source packages are selected".

If you package-install a package, it is selected and won't be removed.
Quote the docstring:

  Mark the installed package as selected by adding it to
  ‘package-selected-packages’.

The only non-selected packages are all transitive dependencies.

>>> For the only problem I have, I could not install `geiser'.  I got
>>> "user-error: Package has no VC data", and I kept wondering what the user
>>> should do in this situation.  I had no idea where to look.
>>
>> That sounds wrong, there seems to be a specification in
>> elpa-packages.eld:
>>
>>   ("geiser" :url "https://gitlab.com/emacs-geiser/geiser.git" :lisp-dir
>>   "elisp" :readme "readme.org" :doc "doc/geiser.texi")
>>
>> Maybe this is related to MELPA?  They don't have package specification
>> lists yet, and if their package is prioritised (due to their versioning
>> scheme), maybe this could confused package-vc?
>
> I tried the following two experiments, starting from:
>
> (with-eval-after-load 'package
>   (add-to-list
>     'package-archives
>     '("melpa" . "https://melpa.org/packages/"))
>   (add-to-list
>     'package-archives
>     '("melpa-stable" . "https://stable.melpa.org/packages/")))
>
> Experiment 1:  Add MELPA to the end of the list.
>
> (with-eval-after-load 'package
>   (add-to-list
>     'package-archives
>     '("melpa" . "https://melpa.org/packages/")
>     t)  ; <------ `t' means the end of the list
>   (add-to-list
>     'package-archives
>     '("melpa-stable" . "https://stable.melpa.org/packages/")
>     t)) ; <------ `t' means the end of the list
>
> This *did not* help.
>
> Experiment 2: Comment out MELPA.
>
> ;; (with-eval-after-load 'package
> ;;   (add-to-list
> ;;     'package-archives
> ;;     '("melpa" . "https://melpa.org/packages/"))
> ;;   (add-to-list
> ;;     'package-archives
> ;;     '("melpa-stable" . "https://stable.melpa.org/packages/")))
>
> This *did* help.
>
> If the user has MELPA at the end of the list, it should just work.  So,
> that needs fixing.  But the question becomes, what should happen if the
> user has MELPA at the beginning of the list.  VC could fail, but it
> could also proceed, silently or with a message.

You can see what is going on in `package-vc--desc->spec'.  What we could
do is replace the

  (if (package-desc-archive pkg-desc)
       (alist-get (intern (package-desc-archive pkg-desc))
                  package-vc--archive-spec-alist)
     (mapcan #'append (mapcar #'cdr package-vc--archive-spec-alist)))

with just

  (mapcan #'append (mapcar #'cdr package-vc--archive-spec-alist))

but that might be confusing in other situations (the code here is
preparing an alist of package names to specifications).

>>> I will try more packages in the coming days or weeks!
>>
>> Very appreciative.  But this is a good sign that after bug#59109 has
>> been resolved, the changes can be applied back onto master.
>>
>>> P.S. The documentation for the `package-vc-selected-packages' variable
>>> contains a typo "... you cal also use ...".
>>
>> Will fix, thanks!
>
> Super!
>
> Also, the documentation for the `package-vc-install' function needs some
> blank lines between paragraphs.  :)

I don't know, is that conventional?  I believe to have seen both, but
never knew which was the "right" way.

> Rudy

Re: feature/package-vc has been merged

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

>> I am still confused as to how the error came about in the first place
>
> I will re-test.

Great (thank you for taking the time and your patience, btw)!

>> OK, I'll add it then.
>
> Thanks!
>
>>>> [...] I haven't been able to push because emacs.git is preventing me
>>>> from force pushing.  I'll try to get them up there some other way,
>>>> and then rewrite the history before rebasing the changes onto
>>>> master.
>>>
>>> So much for "scratch" branches. :-/
>>
>> I am not sure what you mean?  The issue has since been resolved.
>
> I thought you could not force-push to the scratch branch.

I can delete the branch and re-push.  Not nice, but there hasn't been a
need to force-push since my last mistake.  The newest state should be
available on the scratch/package-vc-fixes.

> Rudy

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

>>> The default handler just wraps vc-annotate, so it is a bit more fragile.
>>
>> Hmm... the code I see in your patch uses `vc-region-history` (which is
>> only supported for Git and Hg, currently, and is fairly difficult to
>> support in a generic way) rather than `vc-annotate`.
>> Am I missing something?
>
> Uh, that is my mistake.  I started writing that code yesterday (I
> believe?) and simply forgot what I had used.  I'll try to translate that
> into vc-annotate before pushing anything.

This patch changes the 'last-change' interface from (from to) into (file
line), which matches how most VCS think about it to begin with, and uses
'annotate-command' in the default case:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Have-last-change-accept-a-line-number-instead-of-a-r.patch --]
[-- Type: text/x-patch, Size: 3868 bytes --]

From d33998ed3b5e05a40b9c4c1799b6e911b582ef01 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Sun, 30 Oct 2022 16:52:08 +0100
Subject: [PATCH] Have 'last-change' accept a line number instead of a range

* lisp/emacs-lisp/package-vc.el (package-vc-release-rev): Use new
signature.
* lisp/vc/vc-git.el (vc-git-last-change): Update signature
* lisp/vc/vc.el (vc-default-last-change): Update signature and use
'annotate-command'.
---
 lisp/emacs-lisp/package-vc.el |  4 ++--
 lisp/vc/vc-git.el             | 10 +++-------
 lisp/vc/vc.el                 | 29 ++++++++++++++++-------------
 3 files changed, 21 insertions(+), 22 deletions(-)

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index c3f54c1be8..52e7e25e9f 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -507,8 +507,8 @@ package-vc-release-rev
           (ignore-error vc-not-supported
             (vc-call-backend (vc-backend (buffer-file-name))
                              'last-change
-                             (match-beginning 0)
-                             (match-end 0))))))))
+                             (buffer-file-name)
+                             (line-number-at-pos nil t))))))))
 
 ;;;###autoload
 (defun package-vc-install (name-or-url &optional name rev backend)
diff --git a/lisp/vc/vc-git.el b/lisp/vc/vc-git.el
index cd62effd08..376892c720 100644
--- a/lisp/vc/vc-git.el
+++ b/lisp/vc/vc-git.el
@@ -1632,17 +1632,13 @@ vc-git-annotate-extract-revision-at-line
 		    (expand-file-name fname (vc-git-root default-directory))))
 	  revision)))))
 
-(defun vc-git-last-change (from to)
+(defun vc-git-last-change (file line)
   (vc-buffer-sync)
-  (let ((file (file-relative-name
-               (buffer-file-name)
-               (vc-git-root (buffer-file-name))))
-        (start (line-number-at-pos from t))
-        (end (line-number-at-pos to t)))
+  (let ((file (file-relative-name file (vc-git-root (buffer-file-name)))))
     (with-temp-buffer
       (when (vc-git--out-ok
              "blame" "--porcelain"
-             (format "-L%d,%d" start end)
+             (format "-L%d,+1" line)
              file)
         (goto-char (point-min))
         (save-match-data
diff --git a/lisp/vc/vc.el b/lisp/vc/vc.el
index c8d28c144b..d655a1c625 100644
--- a/lisp/vc/vc.el
+++ b/lisp/vc/vc.el
@@ -449,10 +449,10 @@
 ;;
 ;;   Return the common ancestor between REV1 and REV2 revisions.
 ;;
-;; - last-change (from to)
+;; - last-change (file line)
 ;;
-;;   Return the most recent revision that made a change between FROM
-;;   and TO.
+;;   Return the most recent revision of FILE that made a change
+;;   on LINE.
 
 ;; TAG/BRANCH SYSTEM
 ;;
@@ -3590,17 +3590,20 @@ vc-clone
             (throw 'ok res)))))))
 
 (declare-function log-view-current-tag "log-view" (&optional pos))
-(defun vc-default-last-change (_backend from to)
+(defun vc-default-last-change (_backend file line)
   "Default `last-change' implementation.
-FROM and TO are used as region markers"
-  (save-window-excursion
-    (let* ((buf (window-buffer (vc-region-history from to)))
-           (proc (get-buffer-process buf)))
-      (cl-assert (processp proc))
-      (while (accept-process-output proc))
-      (with-current-buffer buf
-        (prog1 (log-view-current-tag)
-          (kill-buffer))))))
+It returns the last revision that changed LINE number in FILE."
+  (unless (file-exists-p file)
+    (signal 'file-error "File doesn't exist"))
+  (with-temp-buffer
+    (vc-call-backend (vc-backend file) 'annotate-command
+                     file (current-buffer))
+    (goto-char (point-min))
+    (forward-line (1- line))
+    (let ((rev (vc-call-backend
+                (vc-backend file)
+                'annotate-extract-revision-at-line)))
+      (if (consp rev) (car rev) rev))))
 
 \f
 
-- 
2.38.0


[-- Attachment #3: Type: text/plain, Size: 101 bytes --]


With this, I think that the feature will work pretty well -- setting
aside the detached HEAD stuff.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>                 We're not there yet, but the above adds info which we
>>> can (and do in `elpa-admin.el`) extract from the VCS, so it's going
>>> "backward".  IOW, it's making things worse rather than better.
>>
>> You mean the addition of :release-rev, right?
>
> Yes.

Another key that could be gotten rid of is :auto-sync.  I don't know if
it is worth it, but it would be nice to be able to use `map-delete' --
is that available on the ELPA server?  I am never quite sure what
version of Emacs is currently running on there.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>>>>                 We're not there yet, but the above adds info which we
>>>> can (and do in `elpa-admin.el`) extract from the VCS, so it's going
>>>> "backward".  IOW, it's making things worse rather than better.
>>> You mean the addition of :release-rev, right?
> Another key that could be gotten rid of is :auto-sync.

Indeed.

> I don't know if it is worth it, but it would be nice to be able to use
> `map-delete' -- is that available on the ELPA server?  I am never
> quite sure what version of Emacs is currently running on there.

It's running Debian stable, hence currently Emacs-27.


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>>                 We're not there yet, but the above adds info which we
>>>>> can (and do in `elpa-admin.el`) extract from the VCS, so it's going
>>>>> "backward".  IOW, it's making things worse rather than better.
>>>> You mean the addition of :release-rev, right?
>> Another key that could be gotten rid of is :auto-sync.
>
> Indeed.
>
>> I don't know if it is worth it, but it would be nice to be able to use
>> `map-delete' -- is that available on the ELPA server?  I am never
>> quite sure what version of Emacs is currently running on there.
>
> It's running Debian stable, hence currently Emacs-27.

OK, in that case a

    (map-delete (cdr spec) :auto-sync)

ought to suffice.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

>> But there is a general principle that it is better to get the external
>> interfaces right before installing something new.  Changing those
>> interfaces now won't require people to change habits; changing them
>> later could do so.
>>
>> The code for checking out the current-version repo of an ELPA package
>> should be very little different from the code for checking out the
>> upstream repo.  If the latter is already working, getting the former
>> to work equally well shouldn't take long.  Would you please do that?
>
> It can be done, but I believe it would be best to add those kinds of
> annotations to the server side (elpa-admin.el).  The issue is that
> figuring out what version was used to release a package is non-trivial
> using just vc.

I have pushed a commit to feature/package+vc that will use the release
revision (if it is known) when `package-vc-install' is invoked with a
prefix argument.

On the server side, all that has to be added for this to work is the
following:


[-- Attachment #2: Type: text/plain, Size: 541 bytes --]

diff --git a/elpa-admin.el b/elpa-admin.el
index 7ed78430d6..eb94afb512 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -770,6 +770,9 @@ of the current `process-environment'.  Return the modified copy."
     (prin1
      (mapcan
       (lambda (spec)
+        (let ((rel (ignore-errors (elpaa--get-last-release spec))))
+          (when rel
+            (nconc spec (list :release-rev (cdr rel)))))
         (pcase spec
           (`(,name :url nil . ,rest)
            `((,name :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> diff --git a/elpa-admin.el b/elpa-admin.el
> index 7ed78430d6..eb94afb512 100644
> --- a/elpa-admin.el
> +++ b/elpa-admin.el
> @@ -770,6 +770,9 @@ of the current `process-environment'.  Return the modified copy."
>      (prin1
>       (mapcan
>        (lambda (spec)
> +        (let ((rel (ignore-errors (elpaa--get-last-release spec))))
> +          (when rel
> +            (nconc spec (list :release-rev (cdr rel)))))
>          (pcase spec
>            (`(,name :url nil . ,rest)
>             `((,name :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)

Do we really want to rely on the package spec for that?


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> diff --git a/elpa-admin.el b/elpa-admin.el
>> index 7ed78430d6..eb94afb512 100644
>> --- a/elpa-admin.el
>> +++ b/elpa-admin.el
>> @@ -770,6 +770,9 @@ of the current `process-environment'.  Return the modified copy."
>>      (prin1
>>       (mapcan
>>        (lambda (spec)
>> +        (let ((rel (ignore-errors (elpaa--get-last-release spec))))
>> +          (when rel
>> +            (nconc spec (list :release-rev (cdr rel)))))
>>          (pcase spec
>>            (`(,name :url nil . ,rest)
>>             `((,name :url ,(concat "https://git.sv.gnu.org/git/" elpaa--gitrepo)
>
> Do we really want to rely on the package spec for that?

It seems the easiest way to share this information.  Do you have a
different idea?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

>> Do we really want to rely on the package spec for that?
> It seems the easiest way to share this information.  Do you have a
> different idea?

Yes: use the `elpa-admin.el` code to compute that info :-)


        Stefan

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I have pushed a commit to feature/package+vc that will use the release
  > revision (if it is known) when `package-vc-install' is invoked with a
  > prefix argument.

It is good to offer both options.  However, there may be a bug in this
way of doing it.

The words "the release revision" are somewhat terse and I am not
entirely sure what they mean.  My guess is they mean "the revision in
the upstream repo that corresponds to the released code in ELPA."  Is
that correct?

In the usual case, the released code in ELPA does correspond to some
revision in the upstream repo.  Therefore, assuming the release
revision "is known", this implementation will find the right code.

However, in unusual cases the code in ELPA includes changes that did
not come from the upstream repo, and may not be present there.  This
feature needs to DTRT in those cases too.

To check out the latest revision in the upstream repo is certainly wrong,
and it could cause serious confusion.  The command should never do
that, not even as a fallback.

What other behavior would be a better fallback?
What is TRT in these cases?

I see two reasonable possibilities:

1. To signal an error and do nothing.

2. To check out the current release version from the ELPA repo (NOT
the upstream repo).

I think #2 is better, but #1 at least avoids confusion.

In what situations is the release revision not known?
We should think about what is TRT in those cases too.
First, what cases are they?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VACS

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I have pushed a commit to feature/package+vc that will use the release
>   > revision (if it is known) when `package-vc-install' is invoked with a
>   > prefix argument.
>
> It is good to offer both options.  However, there may be a bug in this
> way of doing it.
>
> The words "the release revision" are somewhat terse and I am not
> entirely sure what they mean.  My guess is they mean "the revision in
> the upstream repo that corresponds to the released code in ELPA."  Is
> that correct?

Yes, that is correct.

> In the usual case, the released code in ELPA does correspond to some
> revision in the upstream repo.  Therefore, assuming the release
> revision "is known", this implementation will find the right code.

No, the revision used to create a release is always 

> However, in unusual cases the code in ELPA includes changes that did
> not come from the upstream repo, and may not be present there.  This
> feature needs to DTRT in those cases too.

Ah, now I understand where you are coming from.  As I said in my
previous message, if this is the case and we have a diverged package, we
can always point to elpa.git/nongnu.git.  package-vc.el uses a new file
added to {GNU,NonGNU} ELPA that you can find here

https://elpa.gnu.org/packages/elpa-packages.eld
https://elpa.nongnu.org/nongnu/elpa-packages.eld

These consist of package specifications like

      ("ace-window" :url "https://github.com/abo-abo/ace-window" :auto-sync t)

for upstream repositories, or
                                                                              
      ("adjust-parens" :url "https://git.sv.gnu.org/git/emacs/elpa.git" :branch "externals/adjust-parens")

for repositories that are to be checked out directly from elpa.git.

> To check out the latest revision in the upstream repo is certainly wrong,
> and it could cause serious confusion.  The command should never do
> that, not even as a fallback.

Fallback from what?  If you think the ELPA mirror ought to always be
used, then why even bother with pointing to the upstream repository?  If
anyone needs the upstream repository, they can always add it themselves.
This would mean that the first package specification I have above would
just become

      ("ace-window" :url "https://git.sv.gnu.org/git/emacs/elpa.git" :branch "externals/ace-window")

The only arguments I see are to reduce the load on Savannah and to avoid
the update-lag for auto-synchronised packages between the point in time
when a commit is pushed to the upstream repository and the point it is
mirrored.

> What other behavior would be a better fallback?
> What is TRT in these cases?
       
> I see two reasonable possibilities:
>
> 1. To signal an error and do nothing.
>
> 2. To check out the current release version from the ELPA repo (NOT
> the upstream repo).
>
> I think #2 is better, but #1 at least avoids confusion.

I believe we are talking about diverged packages, right?  Just to
clarify my previous point, this ought to be detectable on the server
side of things, in which case the mirror is used -- unless we always
decide to use the mirror.  There is no need for special handling in
package-vc.el -- in fact the entire point is unrelated to the branch and
any issue blocking the merging into master.

> In what situations is the release revision not known?
> We should think about what is TRT in those cases too.
> First, what cases are they?

The release revision is the most recent release that bumped the
"Version" header, as described in (elisp) Simple Packages.  If there is
no such header, the package is not released onto ELPA, hence it wouldn't
appear in the package archive to begin with.

This is only an issue for unreleased packages, in which case we cannot
state what the release revision is supposed to be in the first place.
This is handled by the following code right now:

   (if-let ((release-rev (package-vc-release-rev pkg-DECs)))
            (vc-retrieve-tag pkg-dir release-rev)
          (message "No release revision was found, continuing..."))

I fear we might disagree as to how fatal or not this case is.  I think
it is better to complete the installation at all instead of aborting it,
which is why I just emit a message and continue on.  But I do recognise
the point that if someone wants the revision used to release a package,
then we should take that request seriously.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Stefan Monnier]

> However, in unusual cases the code in ELPA includes changes that did
> not come from the upstream repo, and may not be present there.

No.  If this were to happen, the `elpa-packages.eld` file would then
point to `elpa.git`as "the upstream", so `package-vc` would still find
the same code as the one distributed on GNU ELPA.


        Stefan

Not a prefix arg

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I have pushed a commit to feature/package+vc that will use the release
  > revision (if it is known) when `package-vc-install' is invoked with a
  > prefix argument.

It is good to offer both options.  However, the question of which
version to check out, and in which repo, makes a big difference.  A
prefix argument is easy to forget, and if you get the source for the
wrong version (wrong for your purposes), you may not recognize that it
is the wrong version.

I think we need to have two command names for these two things to do,
and each command name should make the difference clear.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I'm not quite sure what that means -- what is the "foundation", and why
  > > is there a hurry to merge it in?

                                                         Of course this
  > shouldn't be rushed, but I believe the branch is currently in a
  > functional state that would be useful.

That may be true -- I don't know, so I won't dispute it.
However, before we merge it in we should get the command
interface right.

  > > The code for checking out the current-version repo of an ELPA package
  > > should be very little different from the code for checking out the
  > > upstream repo.  If the latter is already working, getting the former
  > > to work equally well shouldn't take long.  Would you please do that?

  > It can be done, but I believe it would be best to add those kinds of
  > annotations to the server side (elpa-admin.el).  The issue is that
  > figuring out what version was used to release a package is non-trivial
  > using just vc.

I find it hard to understand that last sentence.  What does it mean,
"what version was used to release a package"?

Do you mean, "what version _in the upstream repo_ was copied into the
ELPA repo"?  If so, I think that is a spurious problem -- we don't
need to know which upstream version.  At least, not for this purpose.
It might be useful to keep track of that for some other purpose; if
so, I have nothing against doing so.  But it isn't related to this
feature.

ELPA has its own repo.  By definition, the current version of the
package its latest version of in the ELPA repo.  To get the package
source in a repo, for the current ELPA version, should simply check it
out from the ELPA repo.  This has nothing to do with the ELPA repo.

These two operations (check out from ELPA repo, and check out from
upstream repo) are different in purpose.  It is important to avoid
confusion about whether the checkout contains the current version
or the upstream repo.

So they should have different command names.

Also, the results should be distinguishable by directory name.  A
checkout from the upstream repo should have `upstream' in its name,
while a checkout from the ELPA repo should have `current' in its name.
This too will help users avoid misremembering which version they
are operating on.

I think we should separate checking out the upstream source from a
repo and "installing" for use by default in your Emacs.  There are
several reasons to want to see the upstream source, different
scenarios.

* A user might want to start using the upstream version by default.

* A user might want to work on changes in the upstream version
and try those changes from time to time, but not most of the time.

* A user who has started using the upstream version by default
might wish to disable use of that version because it has some flaw,
and reemable it later.

So there needs to be a way to enable and disable use of a specific
ELPA version checkout of a package.

I think we should separate the operations of checking out a version
from a repo, and enabling or disabling that version for use by Emacs.

  > > I think `package-dev-repo' would be the command now called
  > > `package-vc-install'.

  > I am fine with renaming the file "package-vc" to "package-dev", but the
  > "-repo" suffix is not clear to me.  "-install" is clear in the sense
  > that it will fetch and activate the package, just like
  > "package-install",

How about `checkout' instead of `repo'?  `package-checkout' to check
out from the ELPA repo, and either `package-upstream-checkout' or
`package-dev-checkout' for the upstream repo.

We could also have `package-enable-version' and `package-disable-version'
to enable and disable actual use of a package checkout by Emacs.

Is it feasible to specify which version of which package by selecting
a directory containing a checkout, or by putting point in a list of
such checkouts?  That would be a convenient interface, I think.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > I'm not quite sure what that means -- what is the "foundation", and why
>   > > is there a hurry to merge it in?
>
>                                                          Of course this
>   > shouldn't be rushed, but I believe the branch is currently in a
>   > functional state that would be useful.
>
> That may be true -- I don't know, so I won't dispute it.
> However, before we merge it in we should get the command
> interface right.

FYI since our last exchange I have implemented the features necessary to
check out the commit used to create the tarball that ELPA server would
distribute.  This is done when you invoke `package-vc-install' with a
prefix argument.

>   > > The code for checking out the current-version repo of an ELPA package
>   > > should be very little different from the code for checking out the
>   > > upstream repo.  If the latter is already working, getting the former
>   > > to work equally well shouldn't take long.  Would you please do that?
>
>   > It can be done, but I believe it would be best to add those kinds of
>   > annotations to the server side (elpa-admin.el).  The issue is that
>   > figuring out what version was used to release a package is non-trivial
>   > using just vc.
>
> I find it hard to understand that last sentence.  What does it mean,
> "what version was used to release a package"?

I meant to say "revision", sorry about that.  But as I said above, this
has now already been implemented.

> Do you mean, "what version _in the upstream repo_ was copied into the
> ELPA repo"?  If so, I think that is a spurious problem -- we don't
> need to know which upstream version.  At least, not for this purpose.
> It might be useful to keep track of that for some other purpose; if
> so, I have nothing against doing so.  But it isn't related to this
> feature.
>
> ELPA has its own repo.  By definition, the current version of the
> package its latest version of in the ELPA repo.  To get the package
> source in a repo, for the current ELPA version, should simply check it
> out from the ELPA repo.  This has nothing to do with the ELPA repo.
>
> These two operations (check out from ELPA repo, and check out from
> upstream repo) are different in purpose.  It is important to avoid
> confusion about whether the checkout contains the current version
> or the upstream repo.

I have my doubts as to how significant of a difference these two things
are.  But this isn't to be decided in package-vc.el anyway, instead the
package archive (in our case GNU ELPA and NonGNU ELPA) generates a list
of package specifications, which can either point upstream or to ELPA.

> So they should have different command names.

There are more and more packages that are automatically synchronised, as
is the case by default with NonGNU ELPA.  In that case, there is no
significant difference between the two, and introducing such a thing
would just bring about confusion.

What can be argued is that packages from GNU ELPA that are not
automatically synchronised ought to point to elpa.git.  But as I have
said, that is a decision that can be made at any point, unrelated to
package-vc.el, as this is just the "input" that an ELPA server provides.

> Also, the results should be distinguishable by directory name.  A
> checkout from the upstream repo should have `upstream' in its name,
> while a checkout from the ELPA repo should have `current' in its name.
> This too will help users avoid misremembering which version they
> are operating on.

Is this a point related to the separation of upstream and ELPA
repositories, or one in general.

> I think we should separate checking out the upstream source from a
> repo and "installing" for use by default in your Emacs.  There are
> several reasons to want to see the upstream source, different
> scenarios.

> * A user might want to start using the upstream version by default.
>
> * A user might want to work on changes in the upstream version
> and try those changes from time to time, but not most of the time.
>
> * A user who has started using the upstream version by default
> might wish to disable use of that version because it has some flaw,
> and reemable it later.
>
> So there needs to be a way to enable and disable use of a specific
> ELPA version checkout of a package.

I believe that package.el already supports this via `package-load-list'.

> I think we should separate the operations of checking out a version
> from a repo, and enabling or disabling that version for use by Emacs.
>
>   > > I think `package-dev-repo' would be the command now called
>   > > `package-vc-install'.
>
>   > I am fine with renaming the file "package-vc" to "package-dev", but the
>   > "-repo" suffix is not clear to me.  "-install" is clear in the sense
>   > that it will fetch and activate the package, just like
>   > "package-install",
>
> How about `checkout' instead of `repo'?  `package-checkout' to check
> out from the ELPA repo, and either `package-upstream-checkout' or
> `package-dev-checkout' for the upstream repo.

As Stephan has pointed out that "-checkout" is misleading since it
sounds like a command that would just clone the repository without
activating it as a package.  FWIW this is a fine functionality to have
and would be trivial to implement.

> We could also have `package-enable-version' and `package-disable-version'
> to enable and disable actual use of a package checkout by Emacs.
>
> Is it feasible to specify which version of which package by selecting
> a directory containing a checkout, or by putting point in a list of
> such checkouts?  That would be a convenient interface, I think.

As mentioned above, I believe that `package-load-list' is the right tool
for the job.  In general, package-vc.el is just an alternative method if
fetching and initialising a package, the rest is just done taken care of
by package.el, that isn't too concerned about where the package came from.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > How about `checkout' instead of `repo'?  `package-checkout' to check
  > > out from the ELPA repo, and either `package-upstream-checkout' or
  > > `package-dev-checkout' for the upstream repo.

  > As Stephan has pointed out that "-checkout" is misleading

We are miscommunicating here, I think.

                                                              since it
  > sounds like a command that would just clone the repository

Yes, exactly.  That is what I propose for this command to do: just
check out that package's repo.

                                                               without
  > activating it as a package.

Yes, exactly.  This command should check out the package, and no more.

After that, you may wish to load the code of that package, or arrange
for it to be loaded on other occasions in the future, or you may not.
We should provide easy ways to do those things if you want to, but we
shouldn't impose those things on you by default just because you
asked to see those sources in a repo.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

On 3 November 2022 04:17:47 CET, Richard Stallman <rms@gnu.org> wrote:
>[[[ To any NSA and FBI agents reading my email: please consider    ]]]
>[[[ whether defending the US Constitution against all enemies,     ]]]
>[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>  > > How about `checkout' instead of `repo'?  `package-checkout' to check
>  > > out from the ELPA repo, and either `package-upstream-checkout' or
>  > > `package-dev-checkout' for the upstream repo.
>
>  > As Stephan has pointed out that "-checkout" is misleading
>
>We are miscommunicating here, I think.
>
>                                                              since it
>  > sounds like a command that would just clone the repository
>
>Yes, exactly.  That is what I propose for this command to do: just
>check out that package's repo.
>
>                                                               without
>  > activating it as a package.
>
>Yes, exactly.  This command should check out the package, and no more.
>
>After that, you may wish to load the code of that package, or arrange
>for it to be loaded on other occasions in the future, or you may not.
>We should provide easy ways to do those things if you want to, but we
>shouldn't impose those things on you by default just because you
>asked to see those sources in a repo.
>

In that case we are simply talking about different things. My intention, which is also a popular feature among users, is to provide the ability to fetch and activate a package directly from source. Getting rid of this feature would undermine the point of the branch. What I can do, as I had previously proposed is to, is provide a second command, package-vc-checkout that does what you propose. It would take a package and a directory as arguments.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VACS

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> On 3 November 2022 04:17:47 CET, Richard Stallman <rms@gnu.org> wrote:
>>[[[ To any NSA and FBI agents reading my email: please consider    ]]]
>>[[[ whether defending the US Constitution against all enemies,     ]]]
>>[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> In that case we are simply talking about different things. My
> intention, which is also a popular feature among users, is to provide
> the ability to fetch and activate a package directly from
> source. Getting rid of this feature would undermine the point of the
> branch. What I can do, as I had previously proposed is to, is provide
> a second command, package-vc-checkout that does what you propose. It
> would take a package and a directory as arguments.

I have implemented this as a separate command that can also choose to
check out the current tip of development of the last release.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >After that, you may wish to load the code of that package, or arrange
  > >for it to be loaded on other occasions in the future, or you may not.
  > >We should provide easy ways to do those things if you want to, but we
  > >shouldn't impose those things on you by default just because you
  > >asked to see those sources in a repo.
  > >

  > In that case we are simply talking about different things.

We're talking about the same thing, but looking at it from two
different angles.

You want to do two things together: to check out source from the
upstrean repo, and set up to use that version by default in Emacs.
Now you've added a command to check it out and not do anything else.

I see them as conceptually separate.  I think we should have a command
to check out source from the upstrean repo and no more.  And then
a command to activate that version for use.

It's not a crucial difference.  Either way, the user can check it out
with or without activation.

If we include checking out the ELPA repo, then we have four
possibilities: check out the ELPA repo, check it out and activate it,
check out the upstream repo, and check it out and activate it.

Then there is an advantage in separating checkout and activation.
Then we need only two checkout commands (the upstream repo and the
ELPA repo), and one activate command that can activate either repo.

But it's not a problem if that has to be done with four commands.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > >After that, you may wish to load the code of that package, or arrange
>   > >for it to be loaded on other occasions in the future, or you may not.
>   > >We should provide easy ways to do those things if you want to, but we
>   > >shouldn't impose those things on you by default just because you
>   > >asked to see those sources in a repo.
>   > >
>
>   > In that case we are simply talking about different things.
>
> We're talking about the same thing, but looking at it from two
> different angles.

For certain values of "same thing" ^^

> You want to do two things together: to check out source from the
> upstrean repo, and set up to use that version by default in Emacs.
> Now you've added a command to check it out and not do anything else.

I want to offer the ability to do two things together.  So you can now
either

        M-x package-vc-install foo

and have everything taken care of, or

        M-x package-vc-checkout foo path/to/some/directory
        M-x package-vc-link-directory path/to/some/directory

to do the same using two separate steps.

> I see them as conceptually separate.  I think we should have a command
> to check out source from the upstrean repo and no more.  And then
> a command to activate that version for use.

And that is provided.

> It's not a crucial difference.  Either way, the user can check it out
> with or without activation.

Right.

> If we include checking out the ELPA repo, then we have four
> possibilities: check out the ELPA repo, check it out and activate it,
> check out the upstream repo, and check it out and activate it.
>
> Then there is an advantage in separating checkout and activation.
> Then we need only two checkout commands (the upstream repo and the
> ELPA repo), and one activate command that can activate either repo.
>
> But it's not a problem if that has to be done with four commands.

As you can see above, I've solved it with a three commands, where
`package-vc-install' and `package-vc-checkout' both take a prefix
argument if you want to checkout the revision of the latest release.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Sat, 05 Nov 2022 07:15:53 +0000
> 
> > You want to do two things together: to check out source from the
> > upstrean repo, and set up to use that version by default in Emacs.
> > Now you've added a command to check it out and not do anything else.
> 
> I want to offer the ability to do two things together.  So you can now
> either
> 
>         M-x package-vc-install foo
> 
> and have everything taken care of, or
> 
>         M-x package-vc-checkout foo path/to/some/directory
>         M-x package-vc-link-directory path/to/some/directory
> 
> to do the same using two separate steps.

FWIW, I find the name package-vc-link-directory sub-optimal.  There's
nothing about linking in what it does.  How about

    package-vc-install-from-checkout
or
    package-vc-install-cloned

instead?

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 05 Nov 2022 07:15:53 +0000
>> 
>> > You want to do two things together: to check out source from the
>> > upstrean repo, and set up to use that version by default in Emacs.
>> > Now you've added a command to check it out and not do anything else.
>> 
>> I want to offer the ability to do two things together.  So you can now
>> either
>> 
>>         M-x package-vc-install foo
>> 
>> and have everything taken care of, or
>> 
>>         M-x package-vc-checkout foo path/to/some/directory
>>         M-x package-vc-link-directory path/to/some/directory
>> 
>> to do the same using two separate steps.
>
> FWIW, I find the name package-vc-link-directory sub-optimal.  There's
> nothing about linking in what it does.  How about
>
>     package-vc-install-from-checkout
> or
>     package-vc-install-cloned
>
> instead?

I like both, and probably prefer the former because of the similarity to
`package-vc-checkout'.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 05 Nov 2022 11:21:04 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >>         M-x package-vc-checkout foo path/to/some/directory
> >>         M-x package-vc-link-directory path/to/some/directory
> >> 
> >> to do the same using two separate steps.
> >
> > FWIW, I find the name package-vc-link-directory sub-optimal.  There's
> > nothing about linking in what it does.  How about
> >
> >     package-vc-install-from-checkout
> > or
> >     package-vc-install-cloned
> >
> > instead?
> 
> I like both, and probably prefer the former because of the similarity to
> `package-vc-checkout'.

Fine by me, thanks.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: rms@gnu.org,  emacs-devel@gnu.org
>> Date: Sat, 05 Nov 2022 11:21:04 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >>         M-x package-vc-checkout foo path/to/some/directory
>> >>         M-x package-vc-link-directory path/to/some/directory
>> >> 
>> >> to do the same using two separate steps.
>> >
>> > FWIW, I find the name package-vc-link-directory sub-optimal.  There's
>> > nothing about linking in what it does.  How about
>> >
>> >     package-vc-install-from-checkout
>> > or
>> >     package-vc-install-cloned
>> >
>> > instead?
>> 
>> I like both, and probably prefer the former because of the
>> similarity to
>> `package-vc-checkout'.
>
> Fine by me, thanks.

Done.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I wrote

  > ELPA has its own repo.  By definition, the current version of the
  > package its latest version of in the ELPA repo.  To get the package
  > source in a repo, for the current ELPA version, should simply check it
  > out from the ELPA repo.  This has nothing to do with the ELPA repo.

but I think I meant

  > ELPA has its own repo.  By definition, the current version of the
  > package its latest version of in the ELPA repo.  To get the package
  > source in a repo, for the current ELPA version, should simply check it
  > out from the ELPA repo.  This has nothing to do with the upstream repo.

Sorry for the confusion.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Which form(s) of the code the act of loading an ELPA package installs
in the user's machine in the current master.  The source files?  The
byte-compiled files?  Native-compiled files?  It varies based on
circumstances?

Knowing this would help me understand some of the issues of
what new commands to check out the package repos ought to do
in order to fit in well with existing usage.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Which form(s) of the code the act of loading an ELPA package installs
> in the user's machine in the current master.  The source files?  The
> byte-compiled files?  Native-compiled files?  It varies based on
> circumstances?

I am sorry, I cannot parse your question.  Could you rephrase it?  In
general, the only difference between "source packages" and "regular
packages" is in the manner the source code is fetched.
`package-install' downloads and extracts a tarball, `package-vc-install'
clones a repository.

> Knowing this would help me understand some of the issues of
> what new commands to check out the package repos ought to do
> in order to fit in well with existing usage.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>     Allow for packages to be installed directly from VCS
>
> Yay!

Are there any specific issues that I have missed that would block the
merging of feature/package+vc?  I have been cleaning up minor issues for
the last few days but don't have the time to work on major features due
to university obligations.  My plan would be to start working on these
some time after the Emacs 29 merge window is closed.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Jonas Bernoulli]

I cannot find it now, but somewhere in this thread you discussed
uninstalling a vc-installed package without risking to lose any
local changes.

In Borg I handle it like this:

Installing a package detaches the gitdir from the working tree.
(Borg uses submodules, but you don't need to do that to detach the
gitdir.)

Uninstalling the package only removes the working tree.  Keeping the
gitdir preserves all branches, stashes and dangling revs.  Uninstalling
is only allowed if there are no uncommitted changes.

The gitdir remains and if the user ever installs the same package again,
they are at that time asked whether to simply restore the working tree
or to start over and discard the existing gitdir.

Re: feature/package+vc 04c4c578c7 3/4: Allow for packages to be installed directly from VCS

[Philip Kaludercic]

Jonas Bernoulli <jonas@bernoul.li> writes:

> I cannot find it now, but somewhere in this thread you discussed
> uninstalling a vc-installed package without risking to lose any
> local changes.

I suppose you mean <jwva64rhr91.fsf-monnier+emacs@gnu.org> and
<jwvy1sahlcf.fsf-monnier+emacs@gnu.org>.

> In Borg I handle it like this:
>
> Installing a package detaches the gitdir from the working tree.
> (Borg uses submodules, but you don't need to do that to detach the
> gitdir.)
>
> Uninstalling the package only removes the working tree.  Keeping the
> gitdir preserves all branches, stashes and dangling revs.  Uninstalling
> is only allowed if there are no uncommitted changes.
>
> The gitdir remains and if the user ever installs the same package again,
> they are at that time asked whether to simply restore the working tree
> or to start over and discard the existing gitdir.

From what I see this relies on the advantage of fixing a VCS like Git,
while package-vc -- to stay VCS agnostic -- keeps everything within a
directory.  I think that Stefan's suggestion to double-check if a user
really wants to delete a VC package should be enough.