Adding use-package to core

Adding use-package to core

[Stefan Kangas]

Payas Relekar <relekarpayas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>>> 1. Get use-package in ELPA
>>>> 2. Complete all documentation
>>>> 3. Prepare documentation in texinfo
>>>>    Will cross that bridge when 2 is done.
>>
>> While we are at it, is there a rationale for this order?  I mean, there
>> is no hurry, right? My misunderstanding was that the order was 2 -> 3 ->
>> 1.  Or are you planning to have use-package ready for Emacs 29?
>
> If you mean merging use-package into emacs.git, that will definitely not
> be happening (at least if I'm the only one working on it).
[...]
> From my understanding, ELPA has less stringent requirements for
> documentation and testing compared to core. Since I cannot commit enough
> time to complete all the tasks before expected 29 branch-off, ELPA is a
> good compromise IMO.

OK, I'll bite.  I have time to dedicate to this task; perhaps we can
move quickly enough for Emacs 29.  Ultimately, the decision will be with
the maintainers, but it seems to me that use-package.el itself is in
shape for inclusion.

Could you write up a summary of what needs doing before we can add
use-package to emacs.git?  Is it just a matter of writing up some more
documentation, or is there more that needs doing?

> As for the order rationale, getting use-package to ELPA means less
> friction and more users can try it out, more feedback, more eyeballs,
> basically.

If we get it into core, we can make it into a :core package.

Re: Adding use-package to core

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 3430 bytes --]

Stefan Kangas <stefankangas@gmail.com> writes:

> Payas Relekar <relekarpayas@gmail.com> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>>>> 1. Get use-package in ELPA
>>>>> 2. Complete all documentation
>>>>> 3. Prepare documentation in texinfo
>>>>>    Will cross that bridge when 2 is done.
>>>
>>> While we are at it, is there a rationale for this order?  I mean, there
>>> is no hurry, right? My misunderstanding was that the order was 2 -> 3 ->
>>> 1.  Or are you planning to have use-package ready for Emacs 29?
>>
>> If you mean merging use-package into emacs.git, that will definitely not
>> be happening (at least if I'm the only one working on it).
> [...]
>> From my understanding, ELPA has less stringent requirements for
>> documentation and testing compared to core. Since I cannot commit enough
>> time to complete all the tasks before expected 29 branch-off, ELPA is a
>> good compromise IMO.
>
> OK, I'll bite.  I have time to dedicate to this task; perhaps we can
> move quickly enough for Emacs 29.  Ultimately, the decision will be with
> the maintainers, but it seems to me that use-package.el itself is in
> shape for inclusion.

That would certainly be great1

> Could you write up a summary of what needs doing before we can add
> use-package to emacs.git?  Is it just a matter of writing up some more
> documentation, or is there more that needs doing?

Payas and I can up with the following list

  1. Get use-package in ELPA
  2. Complete all documentation
  3. Prepare documentation in texinfo
     Will cross that bridge when 2 is done.
  4. Add all relevant files to emacs.git
     TBD when 3 is done.
  5. Ensure everything loads properly

The first point has been done, but hasn't been finalised.  The most
tricky part IMO right now is to complete the .texi documentation.  Take
a look at

https://raw.githubusercontent.com/jwiegley/use-package/master/use-package.texi

I have previously commented on what has do be done here
(<87o7v37sqh.fsf@posteo.net>, <871qqhby2k.fsf@posteo.net>):

    Take a look at use-package.texi in the use-package repository.  There
    are currently two TODO that ought to be addressed.  And as the file is
    generated, the texinfo markup is probably not as idiomatic as it ought
    to be.  There are at least a few instances where @code is used instead
    of @kbd, @key or @var.  @ref where @xref/@pxref might be better.  Content-wise
    a few sections such as how to install the package will be outdated, and
    I'd rephrase the sections that mention MELPA to use ELPA examples.  I
    also notice that the spacing is inconsistent, and one should try to keep
    ensure that each full stop is followed by two spaces.

My worry here are the TODO sections -- they either have to be removed or
expanded:

--8<---------------cut here---------------start------------->8---
@node Getting Started
@chapter Getting Started

TODO@. For now, see @code{README.md}.
--8<---------------cut here---------------end--------------->8---

If the manual is pointing to the README, we might just have to convert
the Markdown document to Texi and clean it up.  My hunch is that the
README isn't written like a good manual, so it won't be that easy.

... On the other hand, if this TODO really just wants the "Getting
Started" section from the README, this should be trivial.  All one would
need is to clean up the following that I quickly converted using Pandoc:


[-- Attachment #2: Type: text/plain, Size: 1918 bytes --]

Here is the simplest @code{use-package} declaration:

@verbatim
;; This is only needed once, near the top of the file
(eval-when-compile
  ;; Following line is not needed if use-package.el is in ~/.emacs.d
  (add-to-list 'load-path "<path where use-package is installed>")
  (require 'use-package))

(use-package foo)
@end verbatim

This loads in the package @code{foo}, but only if @code{foo} is
available on your system. If not, a warning is logged to the
@code{*Messages*} buffer.

Use the @code{:init} keyword to execute code before a package is loaded.
It accepts one or more forms, up to the next keyword:

@verbatim
(use-package foo
  :init
  (setq foo-variable t))
@end verbatim

Similarly, @code{:config} can be used to execute code after a package is
loaded. In cases where loading is done lazily (see more about
autoloading below), this execution is deferred until after the autoload
occurs:

@verbatim
(use-package foo
  :init
  (setq foo-variable t)
  :config
  (foo-mode 1))
@end verbatim

As you might expect, you can use @code{:init} and @code{:config}
together:

@verbatim
(use-package color-moccur
  :commands (isearch-moccur isearch-all)
  :bind (("M-s O" . moccur)
         :map isearch-mode-map
         ("M-o" . isearch-moccur)
         ("M-O" . isearch-moccur-all))
  :init
  (setq isearch-lazy-highlight t)
  :config
  (use-package moccur-edit))
@end verbatim

In this case, I want to autoload the commands @code{isearch-moccur} and
@code{isearch-all} from @code{color-moccur.el}, and bind keys both at
the global level and within the @code{isearch-mode-map} (see next
section). When the package is actually loaded (by using one of these
commands), @code{moccur-edit} is also loaded, to allow editing of the
@code{moccur} buffer.

If you autoload no-interactive function, please use @code{:autoload}.

@verbatim
(use-package org-crypt
  :autoload org-crypt-use-before-save-magic)
@end verbatim

[-- Attachment #3: Type: text/plain, Size: 663 bytes --]


--8<---------------cut here---------------start------------->8---
@node Debugging Tools
@chapter Debugging Tools

TODO
--8<---------------cut here---------------end--------------->8---

This is a more technical topic that probably has to be written by
someone who is knowledgeable with the internals and implementation
details of use-package.  Either we find someone like that -- who has
time -- or the chapter is removed for now.

>> As for the order rationale, getting use-package to ELPA means less
>> friction and more users can try it out, more feedback, more eyeballs,
>> basically.
>
> If we get it into core, we can make it into a :core package.

Right.

Re: Adding use-package to core

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> Payas and I can up with the following list
>
>   1. Get use-package in ELPA
>   2. Complete all documentation
>   3. Prepare documentation in texinfo
>      Will cross that bridge when 2 is done.
>   4. Add all relevant files to emacs.git
>      TBD when 3 is done.
>   5. Ensure everything loads properly
>
> The first point has been done, but hasn't been finalised.

I didn't follow the discussion, but it seems like adding it to GNU ELPA
is being worked on.  That is good.

But I think one question remains, unless I missed some important part of
the discussion (and please forgive me if I did):

We need to figure out how development will proceed after adding it to
emacs.git.  I see three options:

1. Development will continue as before, in the old repository and forge
   (currently GitHub).  This case is different from Eglot, and AFAIU
   more like cc-mode, org-mode (or previously Gnus):

   - Development mainly takes place externally.
   - When a new version of use-package is released, it is manually
     synched (a.k.a. copied) to emacs.git and committed as basically
     "new version".  [So to read the full git log, one would need to
     clone use-package.git -- we don't preserve it.]
   - Someone needs to be in charge of (presumably manually) synching
     changes in emacs.git back to use-package.git.

2. Development moves to emacs.git wholesale, along the same lines as we
   did with eglot.  This would take more work, and I guess collaboration
   and active interest from John.

3. We wait until we can include GNU ELPA packages in the Emacs
   distribution.  AFAIK, this is not actively worked on, and would in
   practice mean that we just wait until someone steps up to volunteer
   for that (i.e. we effectively do nothing, for now).

On balance, the first option (1) seems to me the best one here, as
use-package development is already quite established externally and it
seems smart to leverage that existing community to our advantage.  The
main advantage we are looking for here is to ship use-package with
Emacs, and we already get that with option 1.

Perhaps, in this case, it also doesn't make sense to make use-package
into a :core package.  We would then just need to worry about synching
the latest version of use-package to emacs.git before releasing a new
version of Emacs.  We would not have to update it all the time just to
get it released on GNU ELPA.

I've CC:ed John in case he has anything to add.

>     Take a look at use-package.texi in the use-package repository.  There
>     are currently two TODO that ought to be addressed.  And as the file is
>     generated, the texinfo markup is probably not as idiomatic as it ought
>     to be.  There are at least a few instances where @code is used instead
>     of @kbd, @key or @var.  @ref where @xref/@pxref might be better.  Content-wise
>     a few sections such as how to install the package will be outdated, and
>     I'd rephrase the sections that mention MELPA to use ELPA examples.  I
>     also notice that the spacing is inconsistent, and one should try to keep
>     ensure that each full stop is followed by two spaces.

Agreed.

> My worry here are the TODO sections -- they either have to be removed or
> expanded:
>
> --8<---------------cut here---------------start------------->8---
> @node Getting Started
> @chapter Getting Started
>
> TODO@. For now, see @code{README.md}.
> --8<---------------cut here---------------end--------------->8---
>
> If the manual is pointing to the README, we might just have to convert
> the Markdown document to Texi and clean it up.  My hunch is that the
> README isn't written like a good manual, so it won't be that easy.
>
> ... On the other hand, if this TODO really just wants the "Getting
> Started" section from the README, this should be trivial.  All one would
> need is to clean up the following that I quickly converted using Pandoc:

I think we should be able to just use the section "Getting started" from
README.md.  I think we can just comment out the sections
"Issues/Requests" and "Debugging Tools" until they are actually written.

However, the documentation seems to be currently written in Org-mode,
from where it is exported using Hugo[1] to a website:

    https://jwiegley.github.io/use-package/getting-started/

If it is important to continue doing that, I think we should work with
the org-mode sources rather than the texinfo ones.  It would be useful
if John could let us know if he has any preferences in this regard.

As for the formatting issues in the org->texinfo conversion, I believe
that Org-mode has largely solved them.  With some luck, it should be a
small matter of imitating what they did.  In any case, it should be
possible to find solutions and/or workarounds.

So all in all, the documentation will take some work, but nothing too
crazy.

The harder job going forward will be keeping use-package.{org,texi}
up-to-date with README.md, if that will continue to be an important
source of documentation.  Perhaps README.md should be restricted in
scope to make that job easier.

Footnotes:
[1]  https://gohugo.io/

Re: Adding use-package to core

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Payas and I can up with the following list
>>
>>   1. Get use-package in ELPA
>>   2. Complete all documentation
>>   3. Prepare documentation in texinfo
>>      Will cross that bridge when 2 is done.
>>   4. Add all relevant files to emacs.git
>>      TBD when 3 is done.
>>   5. Ensure everything loads properly
>>
>> The first point has been done, but hasn't been finalised.
>
> I didn't follow the discussion, but it seems like adding it to GNU ELPA
> is being worked on.  That is good.

Actually it is done, but I advised Payas to postpone it until the
documentation is ready, which makes the package more useful.  Even if
use-package isn't added in time for 29, working on the issue would still
be useful for the ELPA initiative.

> But I think one question remains, unless I missed some important part of
> the discussion (and please forgive me if I did):

(None of this was discussed, to my recollection)

> We need to figure out how development will proceed after adding it to
> emacs.git.  I see three options:
>
> 1. Development will continue as before, in the old repository and forge
>    (currently GitHub).  This case is different from Eglot, and AFAIU
>    more like cc-mode, org-mode (or previously Gnus):
>
>    - Development mainly takes place externally.
>    - When a new version of use-package is released, it is manually
>      synched (a.k.a. copied) to emacs.git and committed as basically
>      "new version".  [So to read the full git log, one would need to
>      clone use-package.git -- we don't preserve it.]
>    - Someone needs to be in charge of (presumably manually) synching
>      changes in emacs.git back to use-package.git.
>
> 2. Development moves to emacs.git wholesale, along the same lines as we
>    did with eglot.  This would take more work, and I guess collaboration
>    and active interest from John.
>
> 3. We wait until we can include GNU ELPA packages in the Emacs
>    distribution.  AFAIK, this is not actively worked on, and would in
>    practice mean that we just wait until someone steps up to volunteer
>    for that (i.e. we effectively do nothing, for now).
>
> On balance, the first option (1) seems to me the best one here, as
> use-package development is already quite established externally and it
> seems smart to leverage that existing community to our advantage.  The
> main advantage we are looking for here is to ship use-package with
> Emacs, and we already get that with option 1.

FWIW I agree, and think this has worked well enough for a number of
larger packages (another example are the Modus Themes).  It is sad that
the revision history is broken, but that is a limitation of Git IMO.

> Perhaps, in this case, it also doesn't make sense to make use-package
> into a :core package.  We would then just need to worry about synching
> the latest version of use-package to emacs.git before releasing a new
> version of Emacs.  We would not have to update it all the time just to
> get it released on GNU ELPA.

I cannot evaluate this, but isn't use-package a relatively stable
package, that is mostly being bug-fixed?  There really aren't that many
new features being added all the time, so this might not be that
important.  Or am I mistaken?

> I've CC:ed John in case he has anything to add.
>
>>     Take a look at use-package.texi in the use-package repository.  There
>>     are currently two TODO that ought to be addressed.  And as the file is
>>     generated, the texinfo markup is probably not as idiomatic as it ought
>>     to be.  There are at least a few instances where @code is used instead
>>     of @kbd, @key or @var.  @ref where @xref/@pxref might be better.  Content-wise
>>     a few sections such as how to install the package will be outdated, and
>>     I'd rephrase the sections that mention MELPA to use ELPA examples.  I
>>     also notice that the spacing is inconsistent, and one should try to keep
>>     ensure that each full stop is followed by two spaces.
>
> Agreed.
>
>> My worry here are the TODO sections -- they either have to be removed or
>> expanded:
>>
>> --8<---------------cut here---------------start------------->8---
>> @node Getting Started
>> @chapter Getting Started
>>
>> TODO@. For now, see @code{README.md}.
>> --8<---------------cut here---------------end--------------->8---
>>
>> If the manual is pointing to the README, we might just have to convert
>> the Markdown document to Texi and clean it up.  My hunch is that the
>> README isn't written like a good manual, so it won't be that easy.
>>
>> ... On the other hand, if this TODO really just wants the "Getting
>> Started" section from the README, this should be trivial.  All one would
>> need is to clean up the following that I quickly converted using Pandoc:
>
> I think we should be able to just use the section "Getting started" from
> README.md.  I think we can just comment out the sections
> "Issues/Requests" and "Debugging Tools" until they are actually written.

Someone has to make that choice, and I am fine if you are the one who
decides to do so.

> However, the documentation seems to be currently written in Org-mode,
> from where it is exported using Hugo[1] to a website:
>
>     https://jwiegley.github.io/use-package/getting-started/
>
> If it is important to continue doing that, I think we should work with
> the org-mode sources rather than the texinfo ones.  It would be useful
> if John could let us know if he has any preferences in this regard.

No, we have recently decided to scrap that documentation source in
favour of the documentation generated on elpa.gnu.org -- that is if I
understood everything correctly.

> As for the formatting issues in the org->texinfo conversion, I believe
> that Org-mode has largely solved them.  With some luck, it should be a
> small matter of imitating what they did.  In any case, it should be
> possible to find solutions and/or workarounds.

Really?  How does Org-mode distinguish between @key, @kbd, @code and
@var?

> So all in all, the documentation will take some work, but nothing too
> crazy.

If you think so, that is great.

> The harder job going forward will be keeping use-package.{org,texi}
> up-to-date with README.md, if that will continue to be an important
> source of documentation.  Perhaps README.md should be restricted in
> scope to make that job easier.

I think this is a good idea in general.  Long READMEs are easy to get
lost in.  If a file asks you to read it, then it should at the very
least be kind enough to keep it short and only focus on pointing you to
the relevant information, that would ideally be found somewhere else.

Re: Adding use-package to core

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> I cannot evaluate this, but isn't use-package a relatively stable
> package, that is mostly being bug-fixed?  There really aren't that many
> new features being added all the time, so this might not be that
> important.  Or am I mistaken?

I don't think you're wrong, however I don't see any clear benefits to
making it a :core package as compared to just adding it as a normal
package, as long as it's developed externally.  It's less work, and
seems to be good enough for org-mode, for example.

But you're right that it might not be that important.  Either or works.

> No, we have recently decided to scrap that documentation source in
> favour of the documentation generated on elpa.gnu.org -- that is if I
> understood everything correctly.

OK, thanks.  I now did my homework and read the related threads (sorry
for not doing that first).

1. Does that mean that it is actually preferred to make all changes
   directly in use-package.texi now?

2. Should use-package.org just be scrapped, then?  (I guess that, if the
   answer is yes, we should first export the latest version of that file
   to use-package.texi, and then continue from there.)

3. Payas mentioned working on the texinfo sources.  What's the status of
   that work?  I'm ready to get started, but it seems better if we can
   avoid doing the same work twice, so maybe I should base it on his
   latest version?

> How does Org-mode distinguish between @key, @kbd, @code and @var?

See org-setup.org, org.org and the generated file org.texi.  Basically
they use org-mode macros to add texinfo markup.

Re: Adding use-package to core

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> I cannot evaluate this, but isn't use-package a relatively stable
>> package, that is mostly being bug-fixed?  There really aren't that many
>> new features being added all the time, so this might not be that
>> important.  Or am I mistaken?
>
> I don't think you're wrong, however I don't see any clear benefits to
> making it a :core package as compared to just adding it as a normal
> package, as long as it's developed externally.  It's less work, and
> seems to be good enough for org-mode, for example.
>
> But you're right that it might not be that important.  Either or works.

I just checked elpa-packages, assuming that the modus-themes were
externally synchronised, but apparently this isn't the case.  Maybe we
could take a look at discussion from when Modus was merged into the core,
to find arguments pro and contra.

>> No, we have recently decided to scrap that documentation source in
>> favour of the documentation generated on elpa.gnu.org -- that is if I
>> understood everything correctly.
>
> OK, thanks.  I now did my homework and read the related threads (sorry
> for not doing that first).

No problem.

> 1. Does that mean that it is actually preferred to make all changes
>    directly in use-package.texi now?

That is my understanding

> 2. Should use-package.org just be scrapped, then?  (I guess that, if the
>    answer is yes, we should first export the latest version of that file
>    to use-package.texi, and then continue from there.)

That would follow.

> 3. Payas mentioned working on the texinfo sources.  What's the status of
>    that work?  I'm ready to get started, but it seems better if we can
>    avoid doing the same work twice, so maybe I should base it on his
>    latest version?

That I cannot say.

>> How does Org-mode distinguish between @key, @kbd, @code and @var?
>
> See org-setup.org, org.org and the generated file org.texi.  Basically
> they use org-mode macros to add texinfo markup.

Oh, that is interesting.  Perhaps this would work too, my experience
with org->texi has been dissatisfactory but limited.

Re: Adding use-package to core

[Payas Relekar]

First, thank you Stefan for stepping up to do the work!

I have been quite swamped with $LIFE and haven't been able to dedicate
sufficient time to use-package upstreaming as much as I would have
liked. That makes your interest all the more welcome!

Philip Kaludercic <philipk@posteo.net> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> OK, thanks.  I now did my homework and read the related threads (sorry
>> for not doing that first).

I might be bit out of loop on today's emails, let me know if I miss
anything below :)

>> 1. Does that mean that it is actually preferred to make all changes
>>    directly in use-package.texi now?
>
> That is my understanding

+1

>> 2. Should use-package.org just be scrapped, then?  (I guess that, if the
>>    answer is yes, we should first export the latest version of that file
>>    to use-package.texi, and then continue from there.)
>
> That would follow.

Pick whichever way you're most comfortable with, as long as the end
result is acceptable by maintainers, shouldn't be a problem.

>> 3. Payas mentioned working on the texinfo sources.  What's the status of
>>    that work?  I'm ready to get started, but it seems better if we can
>>    avoid doing the same work twice, so maybe I should base it on his
>>    latest version?
>
> That I cannot say.

As mentioned above, I have been barely able to give any time to work on
this. Please go ahead with anything you would like to work on.

>>> How does Org-mode distinguish between @key, @kbd, @code and @var?
>>
>> See org-setup.org, org.org and the generated file org.texi.  Basically
>> they use org-mode macros to add texinfo markup.
>
> Oh, that is interesting.  Perhaps this would work too, my experience
> with org->texi has been dissatisfactory but limited.

I would personally like to see the package in core if at all possible,
and John has also expressed interest in that. Personally I'd prefer if
development continues in core after upstreaming, but it is upto John to
decide. I know that Modus-themes and Org both are developed out of tree,
and only merged in when the package version is bumped. Which means
history is lost, but clearly that hasn't been a problem and maintainers
of those packages prefer it.

--

Re: Adding use-package to core

[Philip Kaludercic]

Payas Relekar <relekarpayas@gmail.com> writes:

>>> OK, thanks.  I now did my homework and read the related threads (sorry
>>> for not doing that first).
>
> I might be bit out of loop on today's emails, let me know if I miss
> anything below :)

It doesn't seem like you missed anything critical.  Thank you for
confirming.

Re: Adding use-package to core

[John Wiegley]

>>>>> Payas Relekar <relekarpayas@gmail.com> writes:

> I would personally like to see the package in core if at all possible, and
> John has also expressed interest in that. Personally I'd prefer if
> development continues in core after upstreaming, but it is upto John to
> decide. I know that Modus-themes and Org both are developed out of tree, and
> only merged in when the package version is bumped. Which means history is
> lost, but clearly that hasn't been a problem and maintainers of those
> packages prefer it.

I'm entirely in support of the code and its development moving directly into
core. Whichever best supports the Emacs developers and the needs of the
community, since it is more than likely that future work will be carried out
by others. I'm ready to hand it off in whatever way is desired by the team
here.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[Stefan Kangas]

John Wiegley <johnw@gnu.org> writes:

>>>>>> Payas Relekar <relekarpayas@gmail.com> writes:
>
>> I would personally like to see the package in core if at all possible, and
>> John has also expressed interest in that. Personally I'd prefer if
>> development continues in core after upstreaming, but it is upto John to
>> decide. I know that Modus-themes and Org both are developed out of tree, and
>> only merged in when the package version is bumped. Which means history is
>> lost, but clearly that hasn't been a problem and maintainers of those
>> packages prefer it.
>
> I'm entirely in support of the code and its development moving directly into
> core. Whichever best supports the Emacs developers and the needs of the
> community, since it is more than likely that future work will be carried out
> by others. I'm ready to hand it off in whatever way is desired by the team
> here.

IMHO, for that to make sense, someone capable would have to volunteer to
maintain it on our side.  If we don't have such a volunteer, it makes
more sense to me to keep development external for now.

If we see a need to move development fully into core in the future, we
can always do that, but the reverse is harder.

(If we want to preserve history when making such a move at a later date,
we could just delete our existing copy of the file from emacs.git and
then merge the full git history, just as we did with eglot.el.)

Re: Adding use-package to core

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> John Wiegley <johnw@gnu.org> writes:
>
>>>>>>> Payas Relekar <relekarpayas@gmail.com> writes:
>>
>>> I would personally like to see the package in core if at all possible, and
>>> John has also expressed interest in that. Personally I'd prefer if
>>> development continues in core after upstreaming, but it is upto John to
>>> decide. I know that Modus-themes and Org both are developed out of tree, and
>>> only merged in when the package version is bumped. Which means history is
>>> lost, but clearly that hasn't been a problem and maintainers of those
>>> packages prefer it.
>>
>> I'm entirely in support of the code and its development moving directly into
>> core. Whichever best supports the Emacs developers and the needs of the
>> community, since it is more than likely that future work will be carried out
>> by others. I'm ready to hand it off in whatever way is desired by the team
>> here.
>
> IMHO, for that to make sense, someone capable would have to volunteer to
> maintain it on our side.  If we don't have such a volunteer, it makes
> more sense to me to keep development external for now.
>
> If we see a need to move development fully into core in the future, we
> can always do that, but the reverse is harder.

I agree, a transitory stage where use-package is still maintained
externally sounds like the safer bet for now.

> (If we want to preserve history when making such a move at a later date,
> we could just delete our existing copy of the file from emacs.git and
> then merge the full git history, just as we did with eglot.el.)

In that case, what is left is completing the .texi manual, or am I
mistaken?  After that, I suppose that placing the right files in the
right places in emacs.git will be less that a days work.

Re: Adding use-package to core

[Payas Relekar]

Philip Kaludercic <philipk@posteo.net> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> John Wiegley <johnw@gnu.org> writes:
>>
>>> I'm entirely in support of the code and its development moving directly into
>>> core. Whichever best supports the Emacs developers and the needs of the
>>> community, since it is more than likely that future work will be carried out
>>> by others. I'm ready to hand it off in whatever way is desired by the team
>>> here.
>>
>> IMHO, for that to make sense, someone capable would have to volunteer to
>> maintain it on our side.  If we don't have such a volunteer, it makes
>> more sense to me to keep development external for now.

I would like to volunteer for maintaining use-package in core or even
outside, but my current skill-level and time availability restrict in
making a commitment. However, that was always my intention when starting
this discussion. Keeping development out of tree is a good middle
ground.

>> If we see a need to move development fully into core in the future, we
>> can always do that, but the reverse is harder.
>
> I agree, a transitory stage where use-package is still maintained
> externally sounds like the safer bet for now.
>
>> (If we want to preserve history when making such a move at a later date,
>> we could just delete our existing copy of the file from emacs.git and
>> then merge the full git history, just as we did with eglot.el.)
>
> In that case, what is left is completing the .texi manual, or am I
> mistaken?  After that, I suppose that placing the right files in the
> right places in emacs.git will be less that a days work.

IIUC, yes. Completing the manual and placing the files should be all
that is needed. As mentioned before, use-package has been widely
used/tested and has been fairly stable. There should be very few
surprises expected even if we aim for 29.1.

--

Re: Adding use-package to core

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: John Wiegley <johnw@gnu.org>,  Payas Relekar <relekarpayas@gmail.com>,
>  Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel
>  <emacs-devel@gnu.org>
> Date: Sun, 13 Nov 2022 07:07:47 +0000
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> >> I'm entirely in support of the code and its development moving directly into
> >> core. Whichever best supports the Emacs developers and the needs of the
> >> community, since it is more than likely that future work will be carried out
> >> by others. I'm ready to hand it off in whatever way is desired by the team
> >> here.
> >
> > IMHO, for that to make sense, someone capable would have to volunteer to
> > maintain it on our side.  If we don't have such a volunteer, it makes
> > more sense to me to keep development external for now.
> >
> > If we see a need to move development fully into core in the future, we
> > can always do that, but the reverse is harder.
> 
> I agree, a transitory stage where use-package is still maintained
> externally sounds like the safer bet for now.

I don't share Stefan's fears.  If the only issue with moving
use-package to core is that someone must step forward to take the
responsibility for maintaining it, I think we can move it into core
without fear.  I see no reason for making a dedicated maintainer for
use-package a prerequisite for importing it, given what John says
about its stability.  It's a non-issue.

Re: Adding use-package to core

[Payas Relekar]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

>> I don't share Stefan's fears.  If the only issue with moving
>> use-package to core is that someone must step forward to take the
>> responsibility for maintaining it, I think we can move it into core
>> without fear.  I see no reason for making a dedicated maintainer for
>> use-package a prerequisite for importing it, given what John says
>> about its stability.  It's a non-issue.
>
> Why don't you?  This package has been very popular for a long time and
> at least I haven't seen anyone complain about it not being in core.
> Whatever gets included seem to freeze in time and becomes very hard to
> make non-breaking changes, and their writers probably get frustrated
> from that more easily and stop developing it.  There's ever more lines of
> code and more packages, I don't think this direction is sustainable and
> I hope you will reconsider this approach of adding everything to core
> at some point.

While Eli can speak for maintainers' perspective, here's my personal
observations:

It depends on package. Org-mode and modus-themes have only benefitted
from being included in core and their development pace has kept up if
not improved. Eglot progress may be harder to gauge, but at least in my
personal observation, it hasn't reduced.

As for use-package,
1. The package has been stable and working without many complaints for a
lot of people (myself included) for a long time. For all intents and
purposes it can be declared "done" (a.k.a. frozen) and not many will
complain.

2. John has already mentioned that he is willing to hand-over maintains
to whoever steps up to the job, and if they want to do it in core,
that's how it will be. He has also mentioned that he doesn't have
sufficient bandwidth to spend on future development of use-package hence
the former statement.

3. Considering the adoption of use-package in wider Emacs userbase, I'd
say it is already near impossible to make breaking changes. Moving to
core only makes that process more formal with decade long deprecation
policy.

4. If you need someone to complain about use-package not being in core,
you can count this mail as first. When I started out with Emacs in 2020,
use-package was already recommended way in half the packages I found,
and installing it required adding MELPA. Depending on where you stand
with technical and ethical grounds, this is already not ideal. Less
elisp needed to be understood by newbies (even if one-line MELPA
addition) the better.

5. While as a developer I agree that more code means more maintenance
burden, we are not here to reduce development burden. Only code that is
maintenance free is one that is never written, after all. However, Emacs
is there to serve the needs of users. IME declarative configuration that
use-package provides is so clearly supersior to current imperative
manner that it is worth adding more code, especially if this new code is
as battle tested as use-package is.

6. As for your last point, Emacs is not Python. One of the things I
really really like about Emacs is, even if I never install a single
package, the stock setup gives me a LOT of stuff built-in. While my
personal Emacs is riced to the brim, at work I use plain Emacs with very
very little customization (11 lines in init.el) and it does the job.

Perhaps these points can change your point of opinion. Feel free to
correct wherever necessary.

Thanks,
Payas

--

Re: Adding use-package to core

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> In that case, what is left is completing the .texi manual, or am I
> mistaken?  After that, I suppose that placing the right files in the
> right places in emacs.git will be less that a days work.

I think so, yes.

Re: Adding use-package to core

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> In that case, what is left is completing the .texi manual, or am I
>> mistaken?  After that, I suppose that placing the right files in the
>> right places in emacs.git will be less that a days work.
>
> I think so, yes.

And just to double-check, you'll start working on that?

Re: Adding use-package to core

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> And just to double-check, you'll start working on that?

Yes, I've started working on use-package.texi.

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

From: 	Eli Zaretskii
Subject: 	Re: Adding use-package to core
Date: 	Sun, 13 Nov 2022 09:31:11 +0200
> I don't share Stefan's fears.  If the only issue with moving
> use-package to core is that someone must step forward to take the
> responsibility for maintaining it, I think we can move it into core
> without fear.  I see no reason for making a dedicated maintainer for
> use-package a prerequisite for importing it, given what John says
> about its stability.  It's a non-issue.

Why don't you?  This package has been very popular for a long time and
at least I haven't seen anyone complain about it not being in core.
Whatever gets included seem to freeze in time and becomes very hard to
make non-breaking changes, and their writers probably get frustrated
from that more easily and stop developing it.  There's ever more lines of
code and more packages, I don't think this direction is sustainable and
I hope you will reconsider this approach of adding everything to core
at some point.

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Sun, 13 Nov 2022 17:11:57 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: emacs-devel@gnu.org
> 
> From: 	Eli Zaretskii
> Subject: 	Re: Adding use-package to core
> Date: 	Sun, 13 Nov 2022 09:31:11 +0200
> > I don't share Stefan's fears.  If the only issue with moving
> > use-package to core is that someone must step forward to take the
> > responsibility for maintaining it, I think we can move it into core
> > without fear.  I see no reason for making a dedicated maintainer for
> > use-package a prerequisite for importing it, given what John says
> > about its stability.  It's a non-issue.
> 
> Why don't you?  This package has been very popular for a long time and
> at least I haven't seen anyone complain about it not being in core.
> Whatever gets included seem to freeze in time and becomes very hard to
> make non-breaking changes, and their writers probably get frustrated
> from that more easily and stop developing it.  There's ever more lines of
> code and more packages, I don't think this direction is sustainable and
> I hope you will reconsider this approach of adding everything to core
> at some point.

If you have read this discussion, you've seen that the package's
author thinks it's best to move use-package into core, and describes
the package as stable without any need for significant development.
So I think your comments are based on some misunderstanding.

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 13, 2022, 11:55 by relekarpayas@gmail.com:
> While Eli can speak for maintainers' perspective, here's my personal
> observations:
>
> It depends on package. Org-mode and modus-themes have only benefitted
> from being included in core and their development pace has kept up if
> not improved. Eglot progress may be harder to gauge, but at least in my
> personal observation, it hasn't reduced.

modus-themes is simply too good and deserves being a default theme IMO.
How has org-mode have benefited from being in core?  My personal
experience and observation from others is that it's users like too keep
it relatively up to date, and installing a more recent version while there
is also a built-in one is, confusing and error-prone.
eglot is a special case, it is more or less the primary function of a
code editor these days and should get even more attention until it isn't
the 'thing.'

> As for use-package,
> 1. The package has been stable and working without many complaints for a
> lot of people (myself included) for a long time. For all intents and
> purposes it can be declared "done" (a.k.a. frozen) and not many will
> complain.

It's GH page seems to have quite a bit of issues, and I had my frustration
with it because of it's poor handling of custom settings, I think there
is a lot of room for improvement and there is no need for urgency.

> 2. John has already mentioned that he is willing to hand-over maintains
> to whoever steps up to the job, and if they want to do it in core,
> that's how it will be. He has also mentioned that he doesn't have
> sufficient bandwidth to spend on future development of use-package hence
> the former statement.

Okay, who has taken up this task for the foreseeable future? It's GH
page says otherwise.  Plus, it's author having stopped working on it
is not a great argument either.

> 3. Considering the adoption of use-package in wider Emacs user base, I'd
> say it is already near impossible to make breaking changes. Moving to
> core only makes that process more formal with decade long deprecation
> policy.

There is always oversights when it comes to volunteer software, why risk
it when it worked perfectly thus far being an Elpa package?

> 4. If you need someone to complain about use-package not being in core,
> you can count this mail as first. When I started out with Emacs in 2020,
> use-package was already recommended way in half the packages I found,
> and installing it required adding MELPA. Depending on where you stand
> with technical and ethical grounds, this is already not ideal. Less
> elisp needed to be understood by newbies (even if one-line MELPA
> addition) the better.

IIUC it is in Elpa, so this problem is solved now?  If not, then keep it
in Elpa.  Having it in core is going to improve what exactly?  It is
kind of a flavor pack.

> 5. While as a developer I agree that more code means more maintenance
> burden, we are not here to reduce development burden. Only code that is
> maintenance free is one that is never written, after all. However, Emacs
> is there to serve the needs of users. IME declarative configuration that
> use-package provides is so clearly superior to current imperative
> manner that it is worth adding more code, especially if this new code is
> as battle tested as use-package is.

IMO declarative macros are a lot more opaque and comes with other
unnecessary hurdles, so let's not call these preferences superior.
I admit it is easier to copy paste declarative configs when one has
little experience with Emacs, but this is yet another band-aid by itself.

> 6. As for your last point, Emacs is not Python. One of the things I
> really really like about Emacs is, even if I never install a single
> package, the stock setup gives me a LOT of stuff built-in. While my
> personal Emacs is riced to the brim, at work I use plain Emacs with very
> very little customization (11 lines in init.el) and it does the job.
>
> Perhaps these points can change your point of opinion. Feel free to
> correct wherever necessary.

This is all the more confusing to me; the point of use-package is to
make the usage of packages you install from wherever easier.  So if
someone is installing packages anyway, then they first install
use-package if that is the route they want.  You say you are also
fine with using the source code editing facilities already existing
in Emacs, I agree that those should come out of box and have priority
(as long as the maintenance burden isn't too much for the maintainers.)
However, use-package is not that, it is all about installing things so
let it get installed if that is the preference someone takes.  Why force
it on everyone when there is no benefit?  Better to focus effort on
editing oriented, relatively un-opinionated stuff in the core used by
everyone as well as you (like simple.el, python-mode) instead.

> Thanks,
> Payas

Thanks

Re: Adding use-package to core

[Philip Kaludercic]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> IIUC it is in Elpa, so this problem is solved now?  If not, then keep it
> in Elpa.  Having it in core is going to improve what exactly?  It is
> kind of a flavor pack.

No, the package documentation still has to be converted into .texi and
cleaned up.  This point has been blocking both the addition to ELPA and
the core.  Stefan is working on this issue right now, and when he is
done, we can add use-package both to the core and to ELPA.

Re: Adding use-package to core

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> If you have read this discussion, you've seen that the package's author
EZ> thinks it's best to move use-package into core, and describes the package
EZ> as stable without any need for significant development.

Correct, I think this is the real moment for "freezing it", as I don't forsee
more functionality being needed. There are quite a number of issues in GitHub,
though, so perhaps it would be nice to see that number significantly reduced
before including it.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[Stefan Kangas]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> I hope you will reconsider this approach of adding everything to core
> at some point.

Historically, yes, everything was added to core.  Not so much in the
last decade or so: what I see is that we are gradually working towards
reducing our scope, and moving inessential things out (see the
lisp/obsolete directory).  I think this is a good thing, and we should
continue that work.

The new things that do get added tend to be key features and libraries
where it *really* makes sense to have them in core.  We must provide
some minimal set of features by default if Emacs is to be useful to
anyone, so this is also a good thing.

But my argument was not about this.  The idea was that keeping
use-package external might make it easier for us to leverage the
community that already exists around that package (see the 150+ issues
in the tracker, open pull requests, etc.).

On the other hand, moving it to core might entice parts of that existing
community to engage with core development.  That would also be a good
thing.

So, at the end of the day, I'm sure we'll be fine either way.

Re: Adding use-package to core

[Stefan Kangas]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> How has org-mode have benefited from being in core?

Well, of course more people have had the chance to use it.  This has
helped build a fairly sizable org-mode community.

> My personal experience and observation from others is that it's users
> like too keep it relatively up to date, and installing a more recent
> version while there is also a built-in one is, confusing and
> error-prone.

That's not my experience, FWIW.  As a very heavy org-mode user with a
decent number of customizations, I've almost always been more than happy
with the stable version of org-mode shipped with Emacs.

I also can't remember having experienced any issues with installing a
new version of org-mode on top of the built-in one.  package.el
certainly has no trouble with it, AFAIK, and if there are any issues
they are merely bugs that should be fixed.

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 13, 2022, 16:48 by eliz@gnu.org:

> If you have read this discussion, you've seen that the package's
> author thinks it's best to move use-package into core, and describes
> the package as stable without any need for significant development.
> So I think your comments are based on some misunderstanding.
>

How integrated is it with custom, package, etc? Not well enough to be
considered complete IMO.

My understanding is that you will make it yet another member of the
orphanage lisp/ when it could function perfectly from ELPA.

Re: Adding use-package to core

[Eli Zaretskii]

> From: "John Wiegley" <johnw@gnu.org>
> Cc: xenodasein@tutanota.de,  emacs-devel@gnu.org
> Date: Sun, 13 Nov 2022 09:53:11 -0800
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> If you have read this discussion, you've seen that the package's author
> EZ> thinks it's best to move use-package into core, and describes the package
> EZ> as stable without any need for significant development.
> 
> Correct, I think this is the real moment for "freezing it", as I don't forsee
> more functionality being needed. There are quite a number of issues in GitHub,
> though, so perhaps it would be nice to see that number significantly reduced
> before including it.

We can do it in parallel, specifically during the pretest of Emacs 29.
Including in core doesn't mean we don't fix bugs or add features,
quite the opposite.

Re: Adding use-package to core

[Stefan Kangas]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> How integrated is it with custom, package, etc? Not well enough to be
> considered complete IMO.

I have no idea what this refers to.  Please write up the issues
concretely, and we can work on fixing them.  If there are any issues on
GitHub that are important, please point out which they are.

Thanks in advance.

Re: Adding use-package to core

[John Wiegley]

>>>>> "PR" == Payas Relekar <relekarpayas@gmail.com> writes:

PR> 4. If you need someone to complain about use-package not being in core,
PR> you can count this mail as first. When I started out with Emacs in 2020,
PR> use-package was already recommended way in half the packages I found, and
PR> installing it required adding MELPA. Depending on where you stand with
PR> technical and ethical grounds, this is already not ideal. Less elisp
PR> needed to be understood by newbies (even if one-line MELPA addition) the
PR> better.

For me, this is also the biggest argument in favor of inclusion in core: If
use-package is something you need to install (even through ELPA) before you
can bootstrap your environment, it introduces a manual step that would just be
nice for users if we could remove. But that's not an unsurmountable obstacle.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Sun, 13 Nov 2022 19:12:03 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: emacs-devel@gnu.org
> 
> Nov 13, 2022, 16:48 by eliz@gnu.org:
> 
> > If you have read this discussion, you've seen that the package's
> > author thinks it's best to move use-package into core, and describes
> > the package as stable without any need for significant development.
> > So I think your comments are based on some misunderstanding.
> >
> 
> How integrated is it with custom, package, etc? Not well enough to be
> considered complete IMO.

If something in it turns out to be incomplete, we will fix it.

> My understanding is that you will make it yet another member of the
> orphanage lisp/ when it could function perfectly from ELPA.

The packages in lisp/ aren't orphans, just look at the Git logs.  (How
can you seriously consider the Lisp packages in core "frozen" when we
average 5000 commits a year for the past ten years??  Where do all
those commits go, in your opinion?)

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 13, 2022, 18:05 by stefankangas@gmail.com:
> Well, of course more people have had the chance to use it. This has
> helped build a fairly sizable org-mode community.

I see, so significantly more people compared to before became familiar with
it when it was included in the default release; how do we know this?

If the implication here is that many a users don't get the chance to learn
about great packages in the wild, even when they are in ELPA, when these
packages are not in the default release; then this is a different problem
to address, maybe by including some kind of digest on the state of things
like Sascha Chua's site at release would help.

> > ...
> That's not my experience, FWIW. As a very heavy org-mode user with a
> decent number of customizations, I've almost always been more than happy
> with the stable version of org-mode shipped with Emacs.

Again is there any way we can compare what ratio of its users prefer the one
in the default release, genuinely asking?

I say (if there isn't some nuance I'm not seeing like reaching a much wider
audience) it is almost always more preferable to have the chance to
M-x package-install the latest (stable or devel) org at will, as this is
one of the best features of Emacs, being an interpreter environment itself.

> I also can't remember having experienced any issues with installing a
> new version of org-mode on top of the built-in one. package.el
> certainly has no trouble with it, AFAIK, and if there are any issues
> they are merely bugs that should be fixed.

True, still it would be better to avoid this wholesale.  Org is a
different kind of beast however, so I can't say I'm informed enough
to decide if it would be better for it not to be in default release
of Emacs.  Best would be to improve the interaction of built-in vs
installed business.

Re: Adding use-package to core

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> We can do it in parallel, specifically during the pretest of Emacs 29.
> Including in core doesn't mean we don't fix bugs or add features, quite the
> opposite.

Oh, sounds good then!

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[John Wiegley]

>>>>> "Edd" == Emacs development discussions <xenodasein---> writes:

Edd> How integrated is it with custom, package, etc?

This is exactly the kind of thinking I was hoping for in suggestion the move
to core. Yes, yes and yes! Integration, documentation, the ability for users
to resolve their own issues (for example, making macro expansion easier to do
for use-package), these do need more thought and care.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 13, 2022, 18:46 by johnw@gnu.org:

>>>>>> "Edd" == Emacs development discussions <xenodasein---> writes:
>>>>>>
>
> Edd> How integrated is it with custom, package, etc?
>
> This is exactly the kind of thinking I was hoping for in suggestion the move
> to core. Yes, yes and yes! Integration, documentation, the ability for users
> to resolve their own issues (for example, making macro expansion easier to do
> for use-package), these do need more thought and care.
>
> -- 
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com>                           60E1 46C4 BD1A 7AC1 4BA2
>

Using package.el and custom through their interfaces currently provides
the Emacs soft-entry into using packages from outside, use-package
provides yet another alternative in between them and manually doing
everything.  My understanding is that most people would actually prefer
solving this without tinkering in init.el but they find custom interface
inadequate so they go for the next alternative which is use-package.
I think we can bridge that gap somehow.  OTOH, use-package also works
great for people who'd like to set things through code, like how JW
uses it in their config on GH; for that way of doing things it is in much
better shape IMO, stable if you will.

Eli:
You said this can be fixed, of course but wouldn't it be much
easier to do without fighting backwards compatibility issues of
core code?

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 13, 2022, 18:31 by eliz@gnu.org:

>> Date: Sun, 13 Nov 2022 19:12:03 +0100 (CET)
>> From: xenodasein@tutanota.de
>> Cc: emacs-devel@gnu.org
>>
>> How integrated is it with custom, package, etc? Not well enough to be
>> considered complete IMO.
>>
>
> If something in it turns out to be incomplete, we will fix it.
>
>> My understanding is that you will make it yet another member of the
>> orphanage lisp/ when it could function perfectly from ELPA.
>>
>
> The packages in lisp/ aren't orphans, just look at the Git logs.  (How
> can you seriously consider the Lisp packages in core "frozen" when we
> average 5000 commits a year for the past ten years??  Where do all
> those commits go, in your opinion?)
>

There seems to be many like allout that don't receive a commit for
decades (apart from a typo fix or two). AFAICT most go to new additions.
Stefan K seems to do a lot of thankless housekeeping in that direction,
but more importantly there was the force of nature from Lars before you
ahem frustrated him away, so my criticism mostly applies to from then on;
by all means if we can get Lars back to fixing the million line
interdependent Elisp code, I'll grunt no more.  The way I see it options
are immediately to stop anything else and to hold communions and pray to
get him back, or just start downsizing.

Re: Adding use-package to core

[John Wiegley]

>>>>> xenodasein  <xenodasein@tutanota.de> writes:

> Using package.el and custom through their interfaces currently provides the
> Emacs soft-entry into using packages from outside, use-package provides yet
> another alternative in between them and manually doing everything.

To me, use-package and package.el are mainly orthogonal: Package.el is for
package management (installing, updating, removing), while use-package is for
customization beyond what Customize provides -- or at least allows you to
concentrate changes related to the same package in one place.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Sun, 13 Nov 2022 20:02:16 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: "xenodasein--- viaEmacs development discussions." <emacs-devel@gnu.org>,
> 	Eli Zaretskii <eliz@gnu.org>, stefankangas@gmail.com
> 
> You said this can be fixed, of course but wouldn't it be much
> easier to do without fighting backwards compatibility issues of
> core code?

Why would maintaining use-package in core present more
backward-compatibility issues than when it's maintained outside?

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Sun, 13 Nov 2022 20:19:00 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: emacs-devel@gnu.org
> 
> > The packages in lisp/ aren't orphans, just look at the Git logs.  (How
> > can you seriously consider the Lisp packages in core "frozen" when we
> > average 5000 commits a year for the past ten years??  Where do all
> > those commits go, in your opinion?)
> >
> 
> There seems to be many like allout that don't receive a commit for
> decades (apart from a typo fix or two).

Which means they are probably not used by many, or don't need any
changes.  That's natural.

> AFAICT most go to new additions.

Not true.

> Stefan K seems to do a lot of thankless housekeeping in that direction,

I guess you only looked at the last year or two.

> but more importantly there was the force of nature from Lars before you
> ahem frustrated him away, so my criticism mostly applies to from then on;
> by all means if we can get Lars back to fixing the million line
> interdependent Elisp code, I'll grunt no more.  The way I see it options
> are immediately to stop anything else and to hold communions and pray to
> get him back, or just start downsizing.

Once again, I'm talking about the last decade, not the last couple of
years.  You are missing the big picture.

And anyway, I don't see what all this has to do with having a package
in core or not.

RE: [External] : Re: Adding use-package to core

[Drew Adams]

> To me, use-package and package.el are mainly orthogonal:
> Package.el is for package management (installing, updating,
> removing), while use-package is for customization beyond
> what Customize provides -- or at least allows you to
> concentrate changes related to the same package in one place.

Speaking/asking from ignorance here...

1. "Customization beyond what Customize provides"

What kinds of such customization, besides the
one you call out next (#2)?

2. "allows you to concentrate changes related
to the same package in one place"

Can you be more specific here?  How does what
you have in mind differ from what customize
groups provide?

_____

For #2, a package can even have a group with
subgroups.  And a package has parent groups.
Seems to me that not only do Customize groups
let you concentrate changes in one place, but
they even let you do so in a hierarchical way
(a graph, i.e., hierarchies with sharing),
that is, change your focus of concentration.
This applies for both browsing/discovering
and changing settings.

Examples at different ends of the grouping
spectrum:

`M-x customize-group bookmark-plus' shows 114
options and faces.  Flat: no subgroups.

On the other hand, group `Icicles' has nine
subgroups. `M-x customize-group Icicles' shows
the following, where each parent group and
subgroup name links to its `customize-group'
presentation:
______

Parent groups: Matching Completion Apropos Dabbrev
               Help Recentf Minibuffer Convenience

Icicles group: 
     State : visible group members are all at standard values.

   Minibuffer input completion and cycling of completion candidates.

   See also Doc-Part1, Doc-Part2, Description, Download, Other
   Libraries by Drew, and Send Bug Report.

 hexrgb-canonicalize-defined-colors-flag 
   Non-nil means remove duplicate color names. More

 icicle-completion-style-sets 
   Possible 'completion-styles' values for when 'TAB' completion
   method is 'vanilla'.

Subgroups:
Icicles-Buffers
  Icicles preferences related to buffers.
Icicles-Completions-Display
  Icicles preferences related to display of completion candidates.
Icicles-Files
  Icicles preferences related to files.
Icicles-Key-Bindings
  Icicles preferences related to key bindings.
Icicles-Key-Completion
  Icicles preferences related to key completion
  ('icicle-complete-keys').
Icicles-Matching
  Icicles preferences related to matching input for completion.
Icicles-Minibuffer-Display
  Icicles preferences related to minibuffer display during
  completion.
Icicles-Miscellaneous
  Miscellaneous Icicles preferences.
Icicles-Searching
  Icicles preferences related to searching.
_____

A guess is that you have in mind other _kinds_
of customizations, beyond options and faces.
Is that it?

Customize is limited, but it would be good to
set straight which of its limitations
`use-package' helps overcome.

One guess would be key bindings.  (The Emacs
manual has two completely separate sections,
`Easy Customization Interface' and `Customizing
Key Bindings', with eight and ten subsections,
respectively.)  (`defcustom' now has :type
`key-sequence', but that's of course only for
customizing option values.)
_____

To be clear, I'm not making any statement about
either `use-package' or Customize.  Certainly
the Customize UI could be improved, and there
are user customizations that Customize doesn't
help with at all, OOTB.

It might be good to match some of its limitations
against what `use-package' offers to handle them.
Maybe that's the best solution for them, or maybe
it can serve as food for thought for improvement
to Customize.

Re: [External] : Re: Adding use-package to core

[North Year]

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

Drew Adams <drew.adams@oracle.com> writes:

>> To me, use-package and package.el are mainly orthogonal:
>> Package.el is for package management (installing, updating,
>> removing), while use-package is for customization beyond
>> what Customize provides – or at least allows you to
>> concentrate changes related to the same package in one place.
>
> Speaking/asking from ignorance here…
>
> 1. “Customization beyond what Customize provides”
>
> What kinds of such customization, besides the
> one you call out next (#2)?

customize.el offers a handy way for package developers
to specify options that is easier for user to tweak.

use-package has nothing to do with package developer,
it is for users. Users are easy to configure packages
in in one place
(including tweaking stuffs with customize.el by `:config')

use-package brings nothing new, it is just syntax sugar
with builtin functions (like autoload, with-eval-after-load, add-hook) stuffs.

And indeed, when using use-package at the first time, I often
need to macro-expand it to see its behaviors really
do something as expected.

> 2. “allows you to concentrate changes related
> to the same package in one place”

you can wrap all your configurations with a package in one place `(use-package xxx xxxxxxxx)',
where you customize the hooks, stuffs with customized.el,
the things need to do before loading this package, and
the things ned to do after loading this package,
the functions that need to autoloaded,
the keybindings related to this package, etc.

Re: [External] : Re: Adding use-package to core

[Matthew Carter]

Drew Adams <drew.adams@oracle.com> writes:

>> To me, use-package and package.el are mainly orthogonal:
>> Package.el is for package management (installing, updating,
>> removing), while use-package is for customization beyond
>> what Customize provides -- or at least allows you to
>> concentrate changes related to the same package in one place.
>
> Speaking/asking from ignorance here...
>
> 1. "Customization beyond what Customize provides"
>
> What kinds of such customization, besides the
> one you call out next (#2)?
>
> 2. "allows you to concentrate changes related
> to the same package in one place"
>
> Can you be more specific here?  How does what
> you have in mind differ from what customize
> groups provide?
>
> _____
>
> For #2, a package can even have a group with
> subgroups.  And a package has parent groups.
> Seems to me that not only do Customize groups
> let you concentrate changes in one place, but
> they even let you do so in a hierarchical way
> (a graph, i.e., hierarchies with sharing),
> that is, change your focus of concentration.
> This applies for both browsing/discovering
> and changing settings.
>
> Examples at different ends of the grouping
> spectrum:
>
> `M-x customize-group bookmark-plus' shows 114
> options and faces.  Flat: no subgroups.
>
> On the other hand, group `Icicles' has nine
> subgroups. `M-x customize-group Icicles' shows
> the following, where each parent group and
> subgroup name links to its `customize-group'
> presentation:
> ______
>
> Parent groups: Matching Completion Apropos Dabbrev
>                Help Recentf Minibuffer Convenience
>
> Icicles group: 
>      State : visible group members are all at standard values.
>
>    Minibuffer input completion and cycling of completion candidates.
>
>    See also Doc-Part1, Doc-Part2, Description, Download, Other
>    Libraries by Drew, and Send Bug Report.
>
>  hexrgb-canonicalize-defined-colors-flag 
>    Non-nil means remove duplicate color names. More
>
>  icicle-completion-style-sets 
>    Possible 'completion-styles' values for when 'TAB' completion
>    method is 'vanilla'.
>
> Subgroups:
> Icicles-Buffers
>   Icicles preferences related to buffers.
> Icicles-Completions-Display
>   Icicles preferences related to display of completion candidates.
> Icicles-Files
>   Icicles preferences related to files.
> Icicles-Key-Bindings
>   Icicles preferences related to key bindings.
> Icicles-Key-Completion
>   Icicles preferences related to key completion
>   ('icicle-complete-keys').
> Icicles-Matching
>   Icicles preferences related to matching input for completion.
> Icicles-Minibuffer-Display
>   Icicles preferences related to minibuffer display during
>   completion.
> Icicles-Miscellaneous
>   Miscellaneous Icicles preferences.
> Icicles-Searching
>   Icicles preferences related to searching.
> _____
>
> A guess is that you have in mind other _kinds_
> of customizations, beyond options and faces.
> Is that it?
>
> Customize is limited, but it would be good to
> set straight which of its limitations
> `use-package' helps overcome.
>
> One guess would be key bindings.  (The Emacs
> manual has two completely separate sections,
> `Easy Customization Interface' and `Customizing
> Key Bindings', with eight and ten subsections,
> respectively.)  (`defcustom' now has :type
> `key-sequence', but that's of course only for
> customizing option values.)
> _____
>
> To be clear, I'm not making any statement about
> either `use-package' or Customize.  Certainly
> the Customize UI could be improved, and there
> are user customizations that Customize doesn't
> help with at all, OOTB.
>
> It might be good to match some of its limitations
> against what `use-package' offers to handle them.
> Maybe that's the best solution for them, or maybe
> it can serve as food for thought for improvement
> to Customize.
>

I have always found M-x customize lacking, in that it expects to store
my preferences in a machine readable way (albeit, still text, but ill
formatted and automatically appended to my files) and be set up via a
slow manual process (hopping through customize menus is much slower than
just editing elisp).

Conversely, use-package allows me to set up all the options I care about
in one area, share them with others, and tweak them via the plain old
text editing vs clunky menus (I use emacs in TTY mode, maybe in GUI mode
these are less clunky).

If all customs were equivalent to a setq, I could do the old way
(organize on my own, with various (setq some-thing some-value)), however
more and more packages seem to need actual M-x customize assignment (a
simple setq won't make the change take proper effect).

Thankfully use-package accommodates this, and I can set these up in a
plain old lisp format in the :custom block the use-package provides.

In close to 15 years of using Emacs, I've never opted for M-x customize
when other alternatives would do as a user (although I aim to support it
in the non-GNU packages I provide).

-- 
Matthew Carter (m@ahungry.com)
http://ahungry.com

Re: Adding use-package to core

[Po Lu]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> Why don't you?  This package has been very popular for a long time and
> at least I haven't seen anyone complain about it not being in core.
> Whatever gets included seem to freeze in time and becomes very hard to
> make non-breaking changes

First of all, this is untrue.  I think we have always let whoever
happens to be maintaining a package in core decide which changes to
make, and that includes breaking ones.

Secondly, not making breaking changes does not cause you to ``freeze in
time''.

> and their writers probably get frustrated from that more easily and
> stop developing it.

Any statistics?

> There's ever more lines of code and more packages, I don't think this
> direction is sustainable and I hope you will reconsider this approach
> of adding everything to core at some point.

Personally, I hope that everything most people find useful will
eventually make its way into Emacs, because doing so is a direct
shortcut to making Emacs more useful for everyone.

Re: [External] : Re: Adding use-package to core

[Tim Cross]

Drew Adams <drew.adams@oracle.com> writes:

>> To me, use-package and package.el are mainly orthogonal:
>> Package.el is for package management (installing, updating,
>> removing), while use-package is for customization beyond
>> what Customize provides -- or at least allows you to
>> concentrate changes related to the same package in one place.
>
> Speaking/asking from ignorance here...
>
> 1. "Customization beyond what Customize provides"
>
> What kinds of such customization, besides the
> one you call out next (#2)?
>
> 2. "allows you to concentrate changes related
> to the same package in one place"
>
> Can you be more specific here?  How does what
> you have in mind differ from what customize
> groups provide?
>
Hi Drew,

I'm not sure I agree with the statement from the post you are responding
to and wanted to give a different perspective which might clarify why
some like use-package.

At its most fundamental level, use-package is really just syntactic
sugar. It provides a somewhat more declarative way to manage package
loading, initialisation, configuration and customisation.  It takes
functionality which largely already exists and puts it together in one
declarative 'chunk'. Much of what it does is really just convenience
macros which define a DSL for installing, loading and configuring
packages. For example, it provides convenience features for defining key
bindings, setting interpreters, autoloads, file associations, loading on
demand or loading after something else has been loaded, adding hooks,
setting customize variables, etc. Instead of having

(unless (package-installed-p 'some-package)
  (package-install 'some-package))
(require 'some-package)
(add-hook 'some-package-hook 'my-some-package-setup)
(define-key some-package-map (kbd "C-c f") 'some-mode-do-something)
...
(custom-set-variables
  `(some-package-var t))
  
I might just have

(use-package some-package
  :ensure t
  :hook (some-package . my-some-package-setup)
  :bind ("C-c f" . some-mode-do-something)
  :custom (some-package-var t))
  
which would achieve the same thing.

The 3 main reasons I use it are

1. I don't like the customize interface. This is not a criticism of
customize. I don't like similar interfaces in other editors either. I
find them slow and cumbersome.

2. I like having all the configuration for a package in one
block/stanza. I could also do this with just normal elisp, but
use-package provides some additional syntactic sugar which makes things
less verbose. It also tends to lead to a more declarative configuration.

3. It makes it easy to delay loading of packages, temporarily disabling
or defining when a package is loaded and encourages better habits by
keeping all the configuration associated with a package together so that
if I do temporarily disable it, I don't get other init failures when
other parts of my init file try to reference something which is no
longer being loaded. This isn't so much about what the package does as
much as how it encourages better practice in managing your config.. 

In short, use-package doesn't provide new functionality as much as it
provides an alternative approach to existing functionality. It adds
another choice to how users can manage their configuration. Some people
will like it, some will hate it and probably most won't really care one
way or the other.  After using it for many years, I can say it has
certainly improved the management and maintenance of my init.el

I do think it is worth adding to Emacs and because it can also manage
installing packages from ELPA, I would love it if it was in Emacs core
so I can just use it instead of first ensuring it has been installed and
then using it. 

Re: [External] : Re: Adding use-package to core

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> 1. "Customization beyond what Customize provides"

> What kinds of such customization, besides the one you call out next (#2)?

Defining custom functions, binding keys, advising functions, adding functions
onto hooks (when the hook isn't customizable), using `setq` when necessary,
etc.

use-package was originally created to eliminate repeated boilerplate in my
init.el file, because, not only do I want all of these customizations, but I
want them to load FAST and come into play only if I actually use the package.

> 2. "allows you to concentrate changes related to the same package in one
> place"

> Can you be more specific here? How does what you have in mind differ from
> what customize groups provide?

Customize group is great, but it only works for what is customizable.

I'm a huge fan of customize, btw, and have been using it for many years. The
fact that my init.el is 4.6k lines long should tell you that it doesn't
satisfy all my needs; 463 use-package declarations are needed to achieve that.

  https://github.com/jwiegley/dot-emacs/blob/master/init.el

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [External] : Re: Adding use-package to core

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On Mon, Nov 14, 2022 at 12:13 AM Matthew Carter <m@ahungry.com> wrote:

> If all customs were equivalent to a setq, I could do the old way
> (organize on my own, with various (setq some-thing some-value)), however
> more and more packages seem to need actual M-x customize assignment (a
> simple setq won't make the change take proper effect).

There's `setopt', now.

> In close to 15 years of using Emacs, I've never opted for M-x customize
> when other alternatives would do as a user

Same here (and just realized this month marks the 25th anniversary of my
emacsing).

[-- Attachment #2: Type: text/html, Size: 854 bytes --]

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 14, 2022, 00:27 by luangruo@yahoo.com:
> xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
> writes:
> > Why don't you?  This package has been very popular for a long time and
> > at least I haven't seen anyone complain about it not being in core.
> > Whatever gets included seem to freeze in time and becomes very hard to
> > make non-breaking changes
>
> First of all, this is untrue. I think we have always let whoever
> happens to be maintaining a package in core decide which changes to
> make, and that includes breaking ones.
>
> Secondly, not making breaking changes does not cause you to ``freeze in
> time''.

Something changes or it doesn't, I have no clue what you mean.

> > and their writers probably get frustrated from that more easily and
> > stop developing it.
>
> Any statistics?

I invite you to think what would it be like anytime you mention something
here the answer was any statistics, any statistics?  That this kind of
funny to imagine.

Anyway, 'statistics' in my head formed after looking through lisp files,
how many of them there, and the fact that how few people are maintaining
these, anyone can see this thanks to git.  Problem is over time commits
to core packages keep making assumptions about each other's existence
and that inter-dependency does not seem to encourage anyone to work on
them, even their original writers.  Even you are in your own X corner
and not touching that issue, except for nay-saying on this list.

> > There's ever more lines of code and more packages, I don't think this
> > direction is sustainable and I hope you will reconsider this approach
> > of adding everything to core at some point.
>
> Personally, I hope that everything most people find useful will
> eventually make its way into Emacs, because doing so is a direct
> shortcut to making Emacs more useful for everyone.

Yeah?  Who he is going to put in the cold hard work hours into
maintaining all that?  Furthermore after certain complexity it is
of no help even having numerous developers.  These are well documented
and understood facts of software development and when someone keeps
denying these things without substantial argument it displays blatant
incompetence.  I don't see how bundling millions of lines of code
together when there is already a system to distribute these as external
packages is a shortcut to usefulness for everyone (what does that even
mean?)  Anyone after some kind of shortcut to usefulness seem to simply
download VSCode and be done with it.

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 13, 2022, 20:04 by eliz@gnu.org:

>> Date: Sun, 13 Nov 2022 20:02:16 +0100 (CET)
>> From: xenodasein@tutanota.de
>> Cc: "xenodasein--- viaEmacs development discussions." <emacs-devel@gnu.org>,
>>  Eli Zaretskii <eliz@gnu.org>, stefankangas@gmail.com
>>
>> You said this can be fixed, of course but wouldn't it be much
>> easier to do without fighting backwards compatibility issues of
>> core code?
>>
>
> Why would maintaining use-package in core present more
> backward-compatibility issues than when it's maintained outside?
>

Same reason emacs-devel is not responsible for every single line of
Elisp code on the Internet?  External packages seem to get more love
from their developers.  If not, something new replaces them, people
migrate, and nobody complains to Emacs bug list about it.

Re: Adding use-package to core

[Po Lu]

xenodasein@tutanota.de writes:

> Something changes or it doesn't, I have no clue what you mean.

Most changes break nothing.

> Anyway, 'statistics' in my head formed after looking through lisp files,
> how many of them there, and the fact that how few people are maintaining
> these, anyone can see this thanks to git.

I don't see that.  What I see instead is the same as with any other
mature, well-established software: most of our code is finished and does
not require constant changes.

> Problem is over time commits to core packages keep making assumptions
> about each other's existence and that inter-dependency does not seem
> to encourage anyone to work on them, even their original writers.

Any specific examples?

Anyway, once a package is included with Emacs, and its minimum Emacs
requirement also bumped, it is fine for it to rely on the rest of Emacs
being present.  But if its maintainer decides to support older versions
of Emacs as well, then everyone else does not interfere.  See TRAMP for
an example of one such package.

> Even you are in your own X corner and not touching that issue, except
> for nay-saying on this list.

It might not seem like it, but I have a job to keep me busy, and the
amount of time I can spend on Emacs is quite limited.

Add to that the fact that Emacs 29 is about to be released, and the
major changes to the GUI code in it have inevitably led to regressions
that have to be ironed out, and you will see why most of my changes in
the past two months have been limited to minor refactorings and bug
fixes.  That approach seems to have paid off.  For example, it has led
to bugs that have lain unfound for almost 30 years being fixed (see for
example 25c6bc7a3.)

> Yeah? Who he is going to put in the cold hard work hours into
> maintaining all that?

Presumably whoever wrote the package and has *asked* us to include it
into Emacs.

> Furthermore after certain complexity it is of no help even having
> numerous developers.

[...]

> These are well documented and understood facts of software development
> and when someone keeps denying these things without substantial
> argument it displays blatant incompetence.

So by changing the repository in which some code is placed, other code
is made more complex?  By what, magic?

> I don't see how bundling millions of lines of code together when there
> is already a system to distribute these as external packages is a
> shortcut to usefulness for everyone (what does that even mean?)

You cannot seriously claim it is easier to run several commands to
unpack and install a package in the ELPA directory than to do nothing at
all.

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 14, 2022, 10:47 by luangruo@yahoo.com:
> xenodasein@tutanota.de writes:
> > Something changes or it doesn't, I have no clue what you mean.
>
> Most changes break nothing.
>
> > Anyway, 'statistics' in my head formed after looking through lisp files,
> > how many of them there, and the fact that how few people are maintaining
> > these, anyone can see this thanks to git.
>
> I don't see that. What I see instead is the same as with any other
> mature, well-established software: most of our code is finished and does
> not require constant changes.

Practically all the programming languages or programs we support keep
evolving yet our code is finished?  Surely you can see how this
contradicts the whole point of software lifecycle wisdom?

> > Problem is over time commits to core packages keep making assumptions
> > about each other's existence and that inter-dependency does not seem
> > to encourage anyone to work on them, even their original writers.
>
> Any specific examples?

The example is the commit history, and the number of people rushing in
to help maintain old Emacs code they didn't originally write.

> Anyway, once a package is included with Emacs, and its minimum Emacs
> requirement also bumped, it is fine for it to rely on the rest of Emacs
> being present. But if its maintainer decides to support older versions
> of Emacs as well, then everyone else does not interfere. See TRAMP for
> an example of one such package.

Crucial point here being Michael A. actually does the hard work and
TRAMP is not one of the orphans.

> > Even you are in your own X corner and not touching that issue, except
> > for nay-saying on this list.
>
> It might not seem like it, but I have a job to keep me busy, and the
> amount of time I can spend on Emacs is quite limited.

That is why keeping the core as simple and easy to maintain as possible
is very important.

> Add to that the fact that Emacs 29 is about to be released, and the
> major changes to the GUI code in it have inevitably led to regressions
> that have to be ironed out, and you will see why most of my changes in
> the past two months have been limited to minor refactorings and bug
> fixes. That approach seems to have paid off. For example, it has led
> to bugs that have lain unfound for almost 30 years being fixed (see for
> example 25c6bc7a3.)

Your fixes are numerous and very helpful, thanks a lot.

> > Yeah? Who he is going to put in the cold hard work hours into
> >  maintaining all that?
>
> Presumably whoever wrote the package and has *asked* us to include it
> into Emacs.

My whole complaint is that this is not happening, so the trend here
should be reducing lines of code, but opposite is happening.  This
not a good direction.

> > Furthermore after certain complexity it is of no help even having
> > numerous developers.
> > [...]
> > These are well documented and understood facts of software development
> > and when someone keeps denying these things without substantial
> > argument it displays blatant incompetence.
>
> So by changing the repository in which some code is placed, other code
> is made more complex? By what, magic?

By their interdependence increasing over time.  All code being in the
same place and the nature of free software without strict rules, seem to
lead to this result, I believe it is easy to observe from Emacs source.

Quoting from some other mail I've sense to list:
"Same reason emacs-devel is not responsible for every single line of
Elisp code on the Internet?  External packages seem to get more love
from their developers.  If not, something new replaces them, people
migrate, and nobody complains to Emacs bug list about it."

> > I don't see how bundling millions of lines of code together when there
> > is already a system to distribute these as external packages is a
> > shortcut to usefulness for everyone (what does that even mean?)
>
> You cannot seriously claim it is easier to run several commands to
> unpack and install a package in the ELPA directory than to do nothing at
> all.

Technically it is not easier but also how much harder it is to install
them is so minuscule that the maintenance burden it causes it is not
worth it.  All that effort is better spent improving Elisp, minibuffer,
simple, cc-mode... i.e. "the good parts" and let convenience projects
get externally maintained.

Re: Adding use-package to core

[Po Lu]

xenodasein@tutanota.de writes:

> Practically all the programming languages or programs we support keep
> evolving yet our code is finished?  Surely you can see how this
> contradicts the whole point of software lifecycle wisdom?

I can maybe see how it contradicts consumerist wisdom, but not any other
wisdom.

> The example is the commit history, and the number of people rushing in
> to help maintain old Emacs code they didn't originally write.

That is not a specific example, or even a useful one.

> Crucial point here being Michael A. actually does the hard work and
> TRAMP is not one of the orphans.

And Michael is the TRAMP maintainer, who asked that TRAMP be included in
Emacs.  Just like John.

> That is why keeping the core as simple and easy to maintain as possible
> is very important.

And why would adding use-package negatively affect that?

I think "core" is basically *.c, and maybe subr.el and the likes of
cl-lib.el, and everything under lisp/emacs-lisp/.  Anyone can feel free
to correct me if that's wrong.

> My whole complaint is that this is not happening, so the trend here
> should be reducing lines of code, but opposite is happening.  This
> not a good direction.

LOC is a terrible metric for almost anything.

> By their interdependence increasing over time.  All code being in the
> same place and the nature of free software without strict rules, seem to
> lead to this result, I believe it is easy to observe from Emacs source.

That only happens when we think it is okay for a specific package.  When
we think it is not, then it does not happen, and as a result TRAMP still
works on Emacs 25.1.

OTOH, it is a Good Thing for other code in Emacs 29 or 30 to be able to
use use-package, as doing so will lead to improvements in both pieces of
code.

> Quoting from some other mail I've sense to list:
> "Same reason emacs-devel is not responsible for every single line of
> Elisp code on the Internet?  External packages seem to get more love
> from their developers.  If not, something new replaces them, people
> migrate

And that is a Bad Thing, which causes trouble to countless numbers of
people.  It is why I gave up on almost all packages from ELPA, after
trying to use and like package management for several years.

> Technically it is not easier but also how much harder it is to install
> them is so minuscule that the maintenance burden it causes it is not
> worth it.

I've never seen the maintenance burden of something increase simply due
to the location of a folder changing.

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Mon, 14 Nov 2022 11:32:53 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: johnw@gnu.org, emacs-devel@gnu.org, stefankangas@gmail.com
> 
> Nov 13, 2022, 20:04 by eliz@gnu.org:
> 
> >> You said this can be fixed, of course but wouldn't it be much
> >> easier to do without fighting backwards compatibility issues of
> >> core code?
> >
> > Why would maintaining use-package in core present more
> > backward-compatibility issues than when it's maintained outside?
> 
> Same reason emacs-devel is not responsible for every single line of
> Elisp code on the Internet?  External packages seem to get more love
> from their developers.

Admitting a package into Emacs doesn't mean its developer goes away
and we take over.  If the developer can make changes in ELPA, he or
she can also make changes in Emacs.  Assuming that the developer wants
to continue developing the package, of course (if not, it means the
package would have died anyway).

IOW, I see no reason to expect that adding a package to Emacs will
slow down, let alone "freeze" its development, not because of it was
added to Emacs anyway.

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Mon, 14 Nov 2022 12:52:50 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: "xenodasein--- viaEmacs development discussions." <emacs-devel@gnu.org>,
> 	eliz@gnu.org
> 
> My whole complaint is that this is not happening, so the trend here
> should be reducing lines of code, but opposite is happening.  This
> not a good direction.

You are entitled to your opinions, but we disagree, and we have many
years of experience to present as evidence.

Given such stark differences of assumptions, opinions, and experience
in Emacs development, I see no reason to keep arguing about this.

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

Nov 14, 2022, 13:54 by eliz@gnu.org:

>> Date: Mon, 14 Nov 2022 12:52:50 +0100 (CET)
>> From: xenodasein@tutanota.de
>> Cc: "xenodasein--- viaEmacs development discussions." <emacs-devel@gnu.org>,
>>  eliz@gnu.org
>>
>> My whole complaint is that this is not happening, so the trend here
>> should be reducing lines of code, but opposite is happening.  This
>> not a good direction.
>>
>
> You are entitled to your opinions, but we disagree, and we have many
> years of experience to present as evidence.
>
> Given such stark differences of assumptions, opinions, and experience
> in Emacs development, I see no reason to keep arguing about this.
>

Experience in accomplishing what exactly?  How can I examine this
evidence?  I don't know what are your goals, is going down from 5%
percent to 4% on StackOverflow's survey this year is one of them?
I really want to see this story of success, but wherever I look
that has numbers tells that you can't compare numbers even with
Notepad++.  Perhaps you can point out what kind of thing can change
your decisions on things like this, but I find it hard to understand
how leaving whoever comes after you an even bigger Emacs despite
having said currently no one understands all parts of it.

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Mon, 14 Nov 2022 15:47:43 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: luangruo@yahoo.com, emacs-devel@gnu.org
> 
> > You are entitled to your opinions, but we disagree, and we have many
> > years of experience to present as evidence.
> >
> > Given such stark differences of assumptions, opinions, and experience
> > in Emacs development, I see no reason to keep arguing about this.
> 
> Experience in accomplishing what exactly?  How can I examine this
> evidence?  I don't know what are your goals, is going down from 5%
> percent to 4% on StackOverflow's survey this year is one of them?

The overall goal is, and always has been, to make Emacs more powerful,
more capable of supporting text-editing and text-processing
applications in more and more areas where people need such
capabilities and applications.  Examine the NEWS files over the last
20 years and ask yourself if that is not happening with every new
major release.

As for the evidence, it's right before your eyes: show me another
similar platform that lived for so many years, and after all these
years still gets developed at the same, if not higher, rate.  What
other evidence do you need?  5000 commits each year for the past 20
years cannot just come from a small band of like-thinking weirdos who
are detached from their users.

> I really want to see this story of success, but wherever I look
> that has numbers tells that you can't compare numbers even with
> Notepad++.

We develop Emacs for those who want and need to use it.  What matters
to us is not the percentage of our users between editors and IDEs,
what matters is where our users want us to advance, and which new
technologies will allow us to provide them with new and improved
capabilities, infrastructure, and applications.

> I find it hard to understand how leaving whoever comes after you an
> even bigger Emacs despite having said currently no one understands
> all parts of it.

No matter how small we make Emacs, no one will ever be able to
understand all of it.  It spans and uses too many too wide areas of
computer processing technologies for any single person to be able to
understand it.  Apart of the "usual" CS stuff, we have Unicode and
character properties, GUI and TTY display, text shaping, font
selection, image processing and display, Lisp and its compilation,
file I/O, regular expressions, and that's just a sample.  Who could
possible be an expert in all of that?

So this is a red herring.

As for "even bigger Emacs" part -- if you really believe Emacs can be
divided into core and the rest, I think you don't understand the
spirit and the heart of Emacs and its development.  (Which I can
totally feel for, since it takes a lot of time and personal
involvement to really grasp that.)  See, separating the Emacs core
from applications built on that core makes no sense, and if you try,
you will most probably kill Emacs.  If you examine carefully every
significant development in the Emacs core, you will see that each and
every one of them was always about some application or class of
applications.  Take the recent addition of tree-sitter, for example:
it would make no sense to develop that if font-lock for many
programming languages was not in core, because who in their right mind
would do all that hard work if it couldn't be immediately applied to
existing applications that are part of Emacs, and do it Right, using
all the expertise of "doing it Right" we have on board?

You should arrive at the same conclusion if you examine carefully
"core" Lisp packages, like subr.el, simple.el, etc. -- every single
part of them is there because we needed it for some application in
Emacs itself.  How could that happen if the applications were outside
of Emacs?  It is a fact that developers of 3rd-party applications tend
to solve the problem by themselves, almost never asking us to provide
new core features to help them solve those problems.  If most of Emacs
applications were outside the project, we could never have progressed
and extended our basic capabilities and infrastructure as we have them
now.

This intimate bond between the internals and the core infrastructure
on the one hand, and the applications that are built on top of that
OTOH -- this is our main strength, it's what keeps drawing in new core
developers over the years, thus keeping the project alive and kicking,
and developing at a mind-blowing rate.

And yes, this presents a problem: where do we draw the line, how do we
avoid making Emacs "too big" by adding "everything"?  Well, that's
what takes experience and intuition and gut feeling -- and a lot of
arguments like this one.  But by and large, we succeed, as the state
of Emacs today should tell us.

Whatever the difficulties to make the decision in each case, the
opinion that we should generally go in the opposite direction,
i.e. progressively remove more and more application-level code from
Emacs -- that opinion is completely wrong.  IMNSHO, anyway.

Re: Adding use-package to core

[xenodasein--- via Emacs development discussions.]

This is a real answer, I thank you for it.  FWIW, no one can claim
that you are not meticulous in your community management.  It is a
tiresome and hard task rarely done well.


Nov 14, 2022, 17:39 by eliz@gnu.org:

>> Experience in accomplishing what exactly?  How can I examine this
>> evidence?  I don't know what are your goals, is going down from 5%
>> percent to 4% on StackOverflow's survey this year is one of them?
>>
>
> The overall goal is, and always has been, to make Emacs more powerful,
> more capable of supporting text-editing and text-processing
> applications in more and more areas where people need such
> capabilities and applications.  Examine the NEWS files over the last
> 20 years and ask yourself if that is not happening with every new
> major release.
>
> As for the evidence, it's right before your eyes: show me another
> similar platform that lived for so many years, and after all these
> years still gets developed at the same, if not higher, rate.  What
> other evidence do you need?  5000 commits each year for the past 20
> years cannot just come from a small band of like-thinking weirdos who
> are detached from their users.
>
>> I really want to see this story of success, but wherever I look
>> that has numbers tells that you can't compare numbers even with
>> Notepad++.
>>
>
> We develop Emacs for those who want and need to use it.  What matters
> to us is not the percentage of our users between editors and IDEs,
> what matters is where our users want us to advance, and which new
> technologies will allow us to provide them with new and improved
> capabilities, infrastructure, and applications.
>

The problem is, more capable and powerful _compared to what?_
Comparing yourself against only yourself of yesterday is a great maxim
to have, in line with an ideal world and FSF ideals.  The reality is
that almost everyone using a different editor/ide do not actually hate
Emacs and would use it instead of Electron bloatware _if_ it solved
their problems faster, which it does, seen from how things are and my
experience is also supportive of this.  We are a practical species if
nothing else.

I use Emacs because I like it, not because it actually solves some
programming problems faster.  I am aware how it actually does many
things better after investment and when programming is your lifestyle,
but for most it is only a task.  Note that having to invest is not an
advantage or accomplishment, other programmable editors come very
close to Emacs if you program the hell out of them too.

From this perspective Emacs doesn't accomplish to actually be the
better editor overall, if we compare ourselves to others.  For example
Vim's advantages in quick remote editing or availability.  That
availability is also an accomplishment over Emacs, despite it being
the GNU editor.  I agree popularity is not always a great thing to
have and most times does not bring good things to its way.  Ideally
Emacs would be both demonstrably better yet still niche and we could
revel in our l33t productivity, without others rushing in to have some
:).


>> I find it hard to understand how leaving whoever comes after you an
>> even bigger Emacs despite having said currently no one understands
>> all parts of it.
>>
>
> No matter how small we make Emacs, no one will ever be able to
> understand all of it.  It spans and uses too many too wide areas of
> computer processing technologies for any single person to be able to
> understand it.  Apart of the "usual" CS stuff, we have Unicode and
> character properties, GUI and TTY display, text shaping, font
> selection, image processing and display, Lisp and its compilation,
> file I/O, regular expressions, and that's just a sample.  Who could
> possible be an expert in all of that?
>
> So this is a red herring.
>

Is it though?  Vim's Bram Moolenaar seems to manage it, not to mention
how GNU Emacs came to be; there are many more impressive software out
there with this singular dedicated developer.  Aiming for a single
person is indeed too bold, reducing this number to a few on the other
hand almost feels like a low hanging fruit.  Not every contributor has
to be in the top echelon of course, my impression is that you
underestimate the benefits a less complex project will bring and
instead decide it better for the users when there is more stuff all
together.  (I believe it is a correct assumption given you make it in
a time what Internet is not as ubiquitous as today.  Even then there
could be bigger alternative distributions that include optional
packages, mainly for less lucky parts of the Earth.)


> As for "even bigger Emacs" part -- if you really believe Emacs can be
> divided into core and the rest, I think you don't understand the
> spirit and the heart of Emacs and its development.  (Which I can
> totally feel for, since it takes a lot of time and personal
> involvement to really grasp that.)  See, separating the Emacs core
> from applications built on that core makes no sense, and if you try,
> you will most probably kill Emacs.  If you examine carefully every
> significant development in the Emacs core, you will see that each and
> every one of them was always about some application or class of
> applications.  Take the recent addition of tree-sitter, for example:
> it would make no sense to develop that if font-lock for many
> programming languages was not in core, because who in their right mind
> would do all that hard work if it couldn't be immediately applied to
> existing applications that are part of Emacs, and do it Right, using
> all the expertise of "doing it Right" we have on board?
>
> You should arrive at the same conclusion if you examine carefully
> "core" Lisp packages, like subr.el, simple.el, etc. -- every single
> part of them is there because we needed it for some application in
> Emacs itself.  How could that happen if the applications were outside
> of Emacs?  It is a fact that developers of 3rd-party applications tend
> to solve the problem by themselves, almost never asking us to provide
> new core features to help them solve those problems.  If most of Emacs
> applications were outside the project, we could never have progressed
> and extended our basic capabilities and infrastructure as we have them
> now.
>

I actually agree on this mostly, what I try to suggest is to narrowing
the scope of these applications, which must be maintained by
emacs-devel.  Bidirectional editing, advanced support for programming
languages etc. these are all great and must have features.  If
something isn't needed by almost every programmer or writer, package
it; this seems like a good rule of thumb?

(I guess from the timing I sounded like I am strongly against
inclusion of use-package, but this is not the case, this was more of a
general complaint.  I am somewhat against use-package because I think
it is a band-aid over the shortcomings of existing customization
facilities, but maybe Emacs should turn towards use-package style
customization I don't know.)


> This intimate bond between the internals and the core infrastructure
> on the one hand, and the applications that are built on top of that
> OTOH -- this is our main strength, it's what keeps drawing in new core
> developers over the years, thus keeping the project alive and kicking,
> and developing at a mind-blowing rate.
>
> And yes, this presents a problem: where do we draw the line, how do we
> avoid making Emacs "too big" by adding "everything"?  Well, that's
> what takes experience and intuition and gut feeling -- and a lot of
> arguments like this one.  But by and large, we succeed, as the state
> of Emacs today should tell us.
>
> Whatever the difficulties to make the decision in each case, the
> opinion that we should generally go in the opposite direction,
> i.e. progressively remove more and more application-level code from
> Emacs -- that opinion is completely wrong.  IMNSHO, anyway.
>

I do not understand what live-ness or dead-ness mean for Emacs' case,
as long as FSF and its support lives.  Because no matter what a lot of
people including me will keep using it because of its aesthetics,
community, the way it works etc.  This is indeed a great
accomplishment by itself, it is nightmarish to imagine where Emacs was
not an existing (viable) alternative.

My overall argument here is to demonstrates why I think emacs-devel's
methods are not as effective as they think, and should be more open to
different approaches.  Being in this state of being immortal (or
arguably already dead) gives you unique opportunities to be more
experimental I think.

IIUC the vision is having as many features (that mostly work) as
possible, instead of few superb ones.  Doing the opposite granted Vim
"the editor wars," giving an order of magnitude larger user base and
widespread availability.

Re: Adding use-package to core

[Michael Albinus]

Po Lu <luangruo@yahoo.com> writes:

Hi,

>> Crucial point here being Michael A. actually does the hard work and
>> TRAMP is not one of the orphans.
>
> And Michael is the TRAMP maintainer, who asked that TRAMP be included in
> Emacs.  Just like John.

Completely unrelated, and I say it just for the archives: Integration of
Tramp into Emacs 22 was negotiated by Kai Grossjohann, the that-time
maintainer of Tramp. 20+ years ago.

Best regards, Michael.

Configuration helpers (wad: Adding use-package to core)

[Stefan Monnier]

BTW,

While I'm happy to see `use-package` just reached GNU ELPA and I'm sure
even more people would be happy if it made it into Emacs itself,
`use-package` is solving an old problem and I hope someone will improve
it (or some replacement) to solve new problems.

Some of the problems I think need solving in users's config files are:

- flymake: currently, flymake tends to flag a lot of configuration code
  because that code will be riddled with `setq` on variables that don't
  exist (yet) and calls to functions that don't exist (yet) either.
  A configuration system like use-package/setup/leaf provides some of
  the info needed to resolve this problem, but those packages have yet
  to be tweaked to explain to flymake what's going on.

- idempotent: a good configuration system should make it "hard" to
  write non-idempotent code, and should aim to make it so re-reading
  your init file will give you almost the same result as
  restarting Emacs.
  E.g. if you change a hook function, the result should not be to have
  both the old hook function and the new hook function on that hook.
  Or if you remove the "add-hook" code, re-reading the init file should
  end up removing the hook that was previously added.


        Stefan

Re: Adding use-package to core

[Philip Kaludercic]

Philip Kaludercic <philipk@posteo.net> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> John Wiegley <johnw@gnu.org> writes:
>>
>>>>>>>> Payas Relekar <relekarpayas@gmail.com> writes:
>>>
>>>> I would personally like to see the package in core if at all possible, and
>>>> John has also expressed interest in that. Personally I'd prefer if
>>>> development continues in core after upstreaming, but it is upto John to
>>>> decide. I know that Modus-themes and Org both are developed out of tree, and
>>>> only merged in when the package version is bumped. Which means history is
>>>> lost, but clearly that hasn't been a problem and maintainers of those
>>>> packages prefer it.
>>>
>>> I'm entirely in support of the code and its development moving directly into
>>> core. Whichever best supports the Emacs developers and the needs of the
>>> community, since it is more than likely that future work will be carried out
>>> by others. I'm ready to hand it off in whatever way is desired by the team
>>> here.
>>
>> IMHO, for that to make sense, someone capable would have to volunteer to
>> maintain it on our side.  If we don't have such a volunteer, it makes
>> more sense to me to keep development external for now.
>>
>> If we see a need to move development fully into core in the future, we
>> can always do that, but the reverse is harder.
>
> I agree, a transitory stage where use-package is still maintained
> externally sounds like the safer bet for now.
>
>> (If we want to preserve history when making such a move at a later date,
>> we could just delete our existing copy of the file from emacs.git and
>> then merge the full git history, just as we did with eglot.el.)
>
> In that case, what is left is completing the .texi manual, or am I
> mistaken?  After that, I suppose that placing the right files in the
> right places in emacs.git will be less that a days work.

One thing I have noticed since use-package has appeared on ELPA, that
could also be tackled in time is that the package description (C-h P) is
very messy.  It starts like this:

--8<---------------cut here---------------start------------->8---
# `use-package`

[![Join the chat at https://gitter.im/use-package/Lobby](https://badges.gitter.im/use-package/Lobby.svg)](https://gitter.im/use-package/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Build Status](https://github.com/jwiegley/use-package/actions/workflows/test.yml/badge.svg)](https://github.com/jwiegley/use-package/actions)
[![MELPA](https://melpa.org/packages/use-package-badge.svg)](https://melpa.org/#/use-package)
[![MELPA Stable](https://stable.melpa.org/packages/use-package-badge.svg)](https://stable.melpa.org/#/use-package)

The `use-package` macro allows you to isolate package configuration in your
`.emacs` file in a way that is both performance-oriented and, well, tidy.  I
created it because I have over 80 packages that I use in Emacs, and things
were getting difficult to manage.  Yet with this utility my total load time is
around 2 seconds, with no loss of functionality!

**NOTE**: `use-package` is **not** a package manager! Although `use-package`
does have the useful capability to interface with package managers (see
[below](#package-installation)), its primary purpose is for the configuration
and loading of packages.

Notes for users upgrading to 2.x are located [at the bottom](#upgrading-to-2x).

- [Installing use-package](#installing-use-package)
- [Getting started](#getting-started)
- [Key-binding](#key-binding)
	+ [Binding to keymaps](#binding-to-keymaps)
	+ [Binding within local keymaps](#binding-within-local-keymaps)
- [Modes and interpreters](#modes-and-interpreters)
- [Magic handlers](#magic-handlers)
- [Hooks](#hooks)
- [Package customization](#package-customization)
  + [Customizing variables](#customizing-variables)
  + [Customizing faces](#customizing-faces)
- [Notes about lazy loading](#notes-about-lazy-loading)
- [Information about package loads](#information-about-package-loads)
--8<---------------cut here---------------end--------------->8---

And is followed by the entire manual (from the README file).  I wonder
if it would be better to set :redeem in the package specification to
ignore, and instead just display the shorter commentary section:

--8<---------------cut here---------------start------------->8---
;; The `use-package' declaration macro allows you to isolate package
;; configuration in your ".emacs" in a way that is performance-oriented and,
;; well, just tidy.  I created it because I have over 80 packages that I use
;; in Emacs, and things were getting difficult to manage.  Yet with this
;; utility my total load time is just under 1 second, with no loss of
;; functionality!
;;
;; Please see README.md from the same repository for documentation.
--8<---------------cut here---------------end--------------->8---

Perhaps just with the reference to the README.md replaced with a
reference to the forthcoming manual.

Re: Adding use-package to core

[Eli Zaretskii]

> Date: Tue, 15 Nov 2022 16:38:10 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: emacs-devel@gnu.org
> 
> > We develop Emacs for those who want and need to use it.  What matters
> > to us is not the percentage of our users between editors and IDEs,
> > what matters is where our users want us to advance, and which new
> > technologies will allow us to provide them with new and improved
> > capabilities, infrastructure, and applications.
> 
> The problem is, more capable and powerful _compared to what?_

Compared to what we did yesterday and to what our users want or may
want us to do tomorrow.  For the latter, we keep tabs on other similar
editors and on technological developments, and consider their use in
Emacs.

> I use Emacs because I like it, not because it actually solves some
> programming problems faster.  I am aware how it actually does many
> things better after investment and when programming is your lifestyle,
> but for most it is only a task.  Note that having to invest is not an
> advantage or accomplishment, other programmable editors come very
> close to Emacs if you program the hell out of them too.

We develop Emacs for those who like it and want to use it, regardless
of their reasons.  Why people like Emacs and use it is not relevant to
this discussion, because no matter what we do, we cannot change the
basic facets and look-and-feel of Emacs and its usage.  Whoever likes
it, for whatever reasons, will keep liking it, and those who don't
will only rarely change their minds.

> > No matter how small we make Emacs, no one will ever be able to
> > understand all of it.  It spans and uses too many too wide areas of
> > computer processing technologies for any single person to be able to
> > understand it.  Apart of the "usual" CS stuff, we have Unicode and
> > character properties, GUI and TTY display, text shaping, font
> > selection, image processing and display, Lisp and its compilation,
> > file I/O, regular expressions, and that's just a sample.  Who could
> > possible be an expert in all of that?
> >
> > So this is a red herring.
> 
> Is it though?  Vim's Bram Moolenaar seems to manage it, not to mention
> how GNU Emacs came to be; there are many more impressive software out
> there with this singular dedicated developer.

I don't know how Vim compares to Emacs from the technological POV, I
don't have time to study Vim that thoroughly.  Maybe Vim is less
complex and implements fewer technologies by itself, maybe the Vim
developer is much more talented and/or has more time than we do, maybe
something else.  One thing is sure: with Emacs, with the kind of core
developers we get for the past 30 years, this didn't happen even once,
and so I consider it impractical to hope that it can be done.  And
rational project management doesn't build on hopes for miracles.

> Aiming for a single person is indeed too bold, reducing this number
> to a few on the other hand almost feels like a low hanging fruit.

Once you have 2 or three people, it doesn't matter how many they are:
you need to discuss changes that span more than one area (and almost
all of them do), you need to be able to do forensics on code for which
you have no experts on board, etc.  This is where we are, and no
reasonable downscaling will ever be able to change that.

> my impression is that you underestimate the benefits a less complex
> project will bring

On what is that impression based?

It goes without saying that a significantly smaller project could be
easier to manage, but the important question is: how much smaller can
we make Emacs before it ceases being Emacs?  And my answer to that is
"not much".  Just review all the Lisp packages that currently see a
lot of commits, and you will arrive at the same conclusion: they are
all important.  (Packages that don't see changes aren't contributing
to difficulties in maintenance.)

> I actually agree on this mostly, what I try to suggest is to narrowing
> the scope of these applications, which must be maintained by
> emacs-devel.

It cannot be done.  If you narrow the scope, you lose the motivation
for development.  More importantly, you will begin losing
contributors, most of which are only interested in application levels,
and that is the slippery slope to death, because contributors are the
source of future Emacs developers and maintainers.

> Bidirectional editing, advanced support for programming languages
> etc. these are all great and must have features.  If something isn't
> needed by almost every programmer or writer, package it; this seems
> like a good rule of thumb?

We already make decisions based on our gut feeling what is and what
isn't important.  So if you agree with that in general, what is left
is differences in judgment calls, and those can never be usefully
argued about, because they are basically subjective.

And our experience is that the decision "what is needed by every
programmer or writer" is very non-trivial.  Emacs is so vast and
people's interests and needs are so diverse that one person's main
package in Emacs could be almost useless to someone else.  E.g., there
are people who will say that they use Emacs only because it has Org,
and others who don't use Org at all, but cannot imagine their lives
without VC or Tramp or Dired.

> I do not understand what live-ness or dead-ness mean for Emacs' case,
> as long as FSF and its support lives.

Death means slowdown of development rate, stagnation, loss of users
due to lack of maintenance, and then the project goes dark.  Cf
XEmacs.

> My overall argument here is to demonstrates why I think emacs-devel's
> methods are not as effective as they think, and should be more open to
> different approaches.  Being in this state of being immortal (or
> arguably already dead) gives you unique opportunities to be more
> experimental I think.

We do experiment, only somewhat cautiously.  This isn't an educational
student project; we have a lot of users who don't want us to make
fatal mistakes.  That's a huge responsibility.

> IIUC the vision is having as many features (that mostly work) as
> possible, instead of few superb ones.

But that's not the vision.  The vision is to have as many features we
_need_to_have_in_core_ as possible.  That's a tautological definition,
of course, like every good definition is ("GNU's not Unix" and all
that), which again brings us to the crucial point: it's a judgment
call what to include and what not to include, which technology to
adopt and which not.  That's a very large portion of what Emacs
maintenance is all about, day in and day out.

Re: Adding use-package to core

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> And is followed by the entire manual (from the README file).  I wonder
> if it would be better to set :redeem in the package specification to
> ignore, and instead just display the shorter commentary section:

Sounds good to me.  Would you mind installing that change?

I think we should probably just replace README.md with something much
shorter once use-package is in emacs.git.  Instead, we could point to
this URL

    https://elpa.gnu.org/packages/doc/use-package.html

if that is that supposed to be the canonical URL for documentation.

Re: Adding use-package to core

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> And is followed by the entire manual (from the README file).  I wonder
>> if it would be better to set :redeem in the package specification to
                                ^
                                this is a peculiar typo...
>> ignore, and instead just display the shorter commentary section:
>
> Sounds good to me.  Would you mind installing that change?

Can do.

> I think we should probably just replace README.md with something much
> shorter once use-package is in emacs.git.  Instead, we could point to
> this URL
>
>     https://elpa.gnu.org/packages/doc/use-package.html
>
> if that is that supposed to be the canonical URL for documentation.

At the very least there should be a note indicating that use-package is
not being developed on GitHub any more, and that bug reports or patches
should be sent to emacs-devel.

Re: Adding use-package to core

[Payas Relekar]

Philip Kaludercic <philipk@posteo.net> writes:

> At the very least there should be a note indicating that use-package is
> not being developed on GitHub any more, and that bug reports or patches
> should be sent to emacs-devel.

Once use-package is in the core, I can send a PR to update README to
point to documentation on ELPA as well as instructions to send
patches/bug reports to emacs-devel.

I can probably prepare a draft PR later today or early tomorrow, just to
get this out of the way.
--

Re: Adding use-package to core

[Stefan Kangas]

Payas Relekar <relekarpayas@gmail.com> writes:

> Once use-package is in the core, I can send a PR to update README to
> point to documentation on ELPA as well as instructions to send
> patches/bug reports to emacs-devel.
>
> I can probably prepare a draft PR later today or early tomorrow, just to
> get this out of the way.

That will be appreciated, thanks.

Re: Adding use-package to core

[Eli Zaretskii]

> From: Payas Relekar <relekarpayas@gmail.com>
> Cc: Stefan Kangas <stefankangas@gmail.com>,  John Wiegley <johnw@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel
>  <emacs-devel@gnu.org>
> Date: Wed, 16 Nov 2022 13:58:59 +0530
> 
> Philip Kaludercic <philipk@posteo.net> writes:
> 
> > At the very least there should be a note indicating that use-package is
> > not being developed on GitHub any more, and that bug reports or patches
> > should be sent to emacs-devel.
> 
> Once use-package is in the core, I can send a PR to update README to
> point to documentation on ELPA as well as instructions to send
> patches/bug reports to emacs-devel.

If use-package is in core, why should the link to its documentation
point to ELPA and not to some place in core?

Re: Adding use-package to core

[Payas Relekar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Payas Relekar <relekarpayas@gmail.com>
>> Cc: Stefan Kangas <stefankangas@gmail.com>,  John Wiegley <johnw@gnu.org>,
>>  Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel
>>  <emacs-devel@gnu.org>
>> Date: Wed, 16 Nov 2022 13:58:59 +0530
>>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>> > At the very least there should be a note indicating that use-package is
>> > not being developed on GitHub any more, and that bug reports or patches
>> > should be sent to emacs-devel.
>>
>> Once use-package is in the core, I can send a PR to update README to
>> point to documentation on ELPA as well as instructions to send
>> patches/bug reports to emacs-devel.
>
> If use-package is in core, why should the link to its documentation
> point to ELPA and not to some place in core?

Either is fine, if I know what that link would be, I'll make PR with
that.

Once it is merged and ELPA starts tracking use-package from core,
we should probably ask John to mark the repo as archived on Github as an
added incentive to direct people to emacs-devel as well as prevent
people from creating new bugs/PRs on Github. Does that sound like a good
idea?

--

Re: Adding use-package to core

[Eli Zaretskii]

> From: Payas Relekar <relekarpayas@gmail.com>
> Cc: philipk@posteo.net,  stefankangas@gmail.com,  johnw@gnu.org,
>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 16 Nov 2022 19:48:17 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If use-package is in core, why should the link to its documentation
> > point to ELPA and not to some place in core?
> 
> Either is fine, if I know what that link would be, I'll make PR with
> that.

I guess it will be to the Info manual?

> Once it is merged and ELPA starts tracking use-package from core,
> we should probably ask John to mark the repo as archived on Github as an
> added incentive to direct people to emacs-devel as well as prevent
> people from creating new bugs/PRs on Github. Does that sound like a good
> idea?

Yes, I think so.

Re: Adding use-package to core

[Payas Relekar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Payas Relekar <relekarpayas@gmail.com>
>
>> Once it is merged and ELPA starts tracking use-package from core,
>> we should probably ask John to mark the repo as archived on Github as an
>> added incentive to direct people to emacs-devel as well as prevent
>> people from creating new bugs/PRs on Github. Does that sound like a good
>> idea?
>
> Yes, I think so.

This is thinking ahead, but we'll want to think about existing open
issues and PRs before making repo archived.
--

Re: Adding use-package to core

[Philip Kaludercic]

Payas Relekar <relekarpayas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Payas Relekar <relekarpayas@gmail.com>
>>
>>> Once it is merged and ELPA starts tracking use-package from core,
>>> we should probably ask John to mark the repo as archived on Github as an
>>> added incentive to direct people to emacs-devel as well as prevent
>>> people from creating new bugs/PRs on Github. Does that sound like a good
>>> idea?
>>
>> Yes, I think so.
>
> This is thinking ahead, but we'll want to think about existing open
> issues and PRs before making repo archived.

This message has prompted me to check the open issues and PRs on GitHub,
and I was surprised to see that there are 103 open (!) issues.  Luckily
none of them appear to have the label "bug", but 32 contain the word
"error".  Skimming through the list, it appears most of these aren't
fatal, but it would be good to fix some of the issues before Emacs 29.1
is released.

Pull requests, on the other hand, seem manageable.  Currently there are
four open, two created today by Stefan Kangas, that are mainly
janitorial and I expect to be merged soon.  The others are small
features that could also be added.

Re: Adding use-package to core

[Eli Zaretskii]

> From: Payas Relekar <relekarpayas@gmail.com>
> Cc: philipk@posteo.net,  stefankangas@gmail.com,  johnw@gnu.org,
>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 16 Nov 2022 20:59:26 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Payas Relekar <relekarpayas@gmail.com>
> >
> >> Once it is merged and ELPA starts tracking use-package from core,
> >> we should probably ask John to mark the repo as archived on Github as an
> >> added incentive to direct people to emacs-devel as well as prevent
> >> people from creating new bugs/PRs on Github. Does that sound like a good
> >> idea?
> >
> > Yes, I think so.
> 
> This is thinking ahead, but we'll want to think about existing open
> issues and PRs before making repo archived.

I don't understand why.  Even if archived, it is accessible, no?

Re: Adding use-package to core

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Payas Relekar <relekarpayas@gmail.com>
>> Cc: philipk@posteo.net,  stefankangas@gmail.com,  johnw@gnu.org,
>>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Wed, 16 Nov 2022 20:59:26 +0530
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Payas Relekar <relekarpayas@gmail.com>
>> >
>> >> Once it is merged and ELPA starts tracking use-package from core,
>> >> we should probably ask John to mark the repo as archived on Github as an
>> >> added incentive to direct people to emacs-devel as well as prevent
>> >> people from creating new bugs/PRs on Github. Does that sound like a good
>> >> idea?
>> >
>> > Yes, I think so.
>> 
>> This is thinking ahead, but we'll want to think about existing open
>> issues and PRs before making repo archived.
>
> I don't understand why.  Even if archived, it is accessible, no?

AFAIK you cannot respond to an issue on GH when the repository is
archived, so that would make it difficult to tackle any issues that
would require feedback or discussion in general.

Re: Adding use-package to core

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Payas Relekar <relekarpayas@gmail.com>,  stefankangas@gmail.com,
>   johnw@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 16 Nov 2022 17:01:37 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Payas Relekar <relekarpayas@gmail.com>
> >> Cc: philipk@posteo.net,  stefankangas@gmail.com,  johnw@gnu.org,
> >>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >> Date: Wed, 16 Nov 2022 20:59:26 +0530
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> From: Payas Relekar <relekarpayas@gmail.com>
> >> >
> >> >> Once it is merged and ELPA starts tracking use-package from core,
> >> >> we should probably ask John to mark the repo as archived on Github as an
> >> >> added incentive to direct people to emacs-devel as well as prevent
> >> >> people from creating new bugs/PRs on Github. Does that sound like a good
> >> >> idea?
> >> >
> >> > Yes, I think so.
> >> 
> >> This is thinking ahead, but we'll want to think about existing open
> >> issues and PRs before making repo archived.
> >
> > I don't understand why.  Even if archived, it is accessible, no?
> 
> AFAIK you cannot respond to an issue on GH when the repository is
> archived, so that would make it difficult to tackle any issues that
> would require feedback or discussion in general.

What is the alternative?  If you copy all the issues to debbugs, you
have the same problem.  And it is impractical to expect us to solve
all the issues before importing use-package, and also impractical to
update the issues in GitHub after the import.

So I don't think there's any better solution than gradually work on
the issues via debbugs.  That is, for every issue we want to solve,
file a debbugs bug report, and then work there.

Re: Adding use-package to core

[John Wiegley]

>>>>> Payas Relekar <relekarpayas@gmail.com> writes:

> Once it is merged and ELPA starts tracking use-package from core, we should
> probably ask John to mark the repo as archived on Github as an added
> incentive to direct people to emacs-devel as well as prevent people from
> creating new bugs/PRs on Github. Does that sound like a good idea?

Whenever emacs-devel tells me it's time for this, I'm happy to mark it as
archived on GitHub. Stefan Kangas has been doing a wonderful job at closing
tickets and submitting PRs over the past couple of weeks, so we're getting
into a better state for such a handoff.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> What is the alternative?  If you copy all the issues to debbugs, you
> have the same problem.  And it is impractical to expect us to solve
> all the issues before importing use-package, and also impractical to
> update the issues in GitHub after the import.
>
> So I don't think there's any better solution than gradually work on
> the issues via debbugs.  That is, for every issue we want to solve,
> file a debbugs bug report, and then work there.

We could archive it after some grace period, say 3, 6, or N months.

At the beginning of that period, we could add a comment to all open
issues and pull requests, saying where to report the issue.  At the end,
we could do it again.  Then we archive.

Re: Adding use-package to core

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Wed, 16 Nov 2022 16:33:03 -0800
> Cc: relekarpayas@gmail.com, johnw@gnu.org, monnier@iro.umontreal.ca, 
> 	emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What is the alternative?  If you copy all the issues to debbugs, you
> > have the same problem.  And it is impractical to expect us to solve
> > all the issues before importing use-package, and also impractical to
> > update the issues in GitHub after the import.
> >
> > So I don't think there's any better solution than gradually work on
> > the issues via debbugs.  That is, for every issue we want to solve,
> > file a debbugs bug report, and then work there.
> 
> We could archive it after some grace period, say 3, 6, or N months.
> 
> At the beginning of that period, we could add a comment to all open
> issues and pull requests, saying where to report the issue.  At the end,
> we could do it again.  Then we archive.

And in the meantime someone will have to monitor the GitHub issues and
update debbugs?

Re: Adding use-package to core

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> We could archive it after some grace period, say 3, 6, or N months.
>>
>> At the beginning of that period, we could add a comment to all open
>> issues and pull requests, saying where to report the issue.  At the end,
>> we could do it again.  Then we archive.
>
> And in the meantime someone will have to monitor the GitHub issues and
> update debbugs?

In this scenario, I think so, yes.  One way of doing it just to close
any _new_ issues immediately, asking them to use bug-gnu-emacs instead.
I'm already getting all the notifications as it is, and I'd be happy to
do that for the duration.  (I'd expect a trickle rather than a storm.)

On the other hand, maybe it would be less confusing to just archive the
repository and be done with it.  If I'm not mistaken, it would be hard
to migrate all issues to debbugs automatically, since GitHub works with
user names and not email addresses.  So this option probably means
losing some bugs, and maybe frustrating some bug reporters.  Maybe it's
worth it: I don't know.

I guess ultimately what we choose to do depends how important it is for
us to preserve existing feature requests and bug reports in a form where
we can continue working on them (e.g. by closing fixed issues).

BTW, it might be worth taking a look at Eglot on GitHub to see how João
handled this: https://github.com/joaotavora/eglot

Re: Adding use-package to core

[Payas Relekar]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> We could archive it after some grace period, say 3, 6, or N months.
>>>
>>> At the beginning of that period, we could add a comment to all open
>>> issues and pull requests, saying where to report the issue.  At the end,
>>> we could do it again.  Then we archive.
>>
>> And in the meantime someone will have to monitor the GitHub issues and
>> update debbugs?
>
> In this scenario, I think so, yes.  One way of doing it just to close
> any _new_ issues immediately, asking them to use bug-gnu-emacs instead.
> I'm already getting all the notifications as it is, and I'd be happy to
> do that for the duration.  (I'd expect a trickle rather than a storm.)

Agree, and thank you for picking this up!

> On the other hand, maybe it would be less confusing to just archive the
> repository and be done with it.  If I'm not mistaken, it would be hard
> to migrate all issues to debbugs automatically, since GitHub works with
> user names and not email addresses.  So this option probably means
> losing some bugs, and maybe frustrating some bug reporters.  Maybe it's
> worth it: I don't know.

I would be in favor of this one. Mostly because it reduces
administrative burden. Also, I expect any issues serious enough will get
re-reported on debbugs anyway, and considering use-package has been
around for a while, there shouldn't be too many of them.

> I guess ultimately what we choose to do depends how important it is for
> us to preserve existing feature requests and bug reports in a form where
> we can continue working on them (e.g. by closing fixed issues).
>
> BTW, it might be worth taking a look at Eglot on GitHub to see how João
> handled this: https://github.com/joaotavora/eglot

Eglot is still being maintained by João, while use-package cannot expect
such commitment from John. If Stefan or anyone is willing to volunteer,
we can continue addressing issues/PRs on both Github as well as mailing
list, but it must be a commitment at least until the repo is archived.

There is also the fact that use-package has fewer expected changes in
future while Eglot has long road of evolution ahead. It makes sense for
eglot to keep both avenues available for users/contributors, while
use-package does not have as much need.

--

Re: Adding use-package to core

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Wed, 16 Nov 2022 23:17:18 -0800
> Cc: philipk@posteo.net, relekarpayas@gmail.com, johnw@gnu.org, 
> 	monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> We could archive it after some grace period, say 3, 6, or N months.
> >>
> >> At the beginning of that period, we could add a comment to all open
> >> issues and pull requests, saying where to report the issue.  At the end,
> >> we could do it again.  Then we archive.
> >
> > And in the meantime someone will have to monitor the GitHub issues and
> > update debbugs?
> 
> In this scenario, I think so, yes.  One way of doing it just to close
> any _new_ issues immediately, asking them to use bug-gnu-emacs instead.
> I'm already getting all the notifications as it is, and I'd be happy to
> do that for the duration.  (I'd expect a trickle rather than a storm.)

We'd need Someone(TM) to volunteer to do these mundane and ungrateful
tasks.

> On the other hand, maybe it would be less confusing to just archive the
> repository and be done with it.  If I'm not mistaken, it would be hard
> to migrate all issues to debbugs automatically, since GitHub works with
> user names and not email addresses.  So this option probably means
> losing some bugs, and maybe frustrating some bug reporters.  Maybe it's
> worth it: I don't know.

It will certainly mean less work for us, as we won't need to watch the
GitHub repository and synchronize it with debbugs.

> I guess ultimately what we choose to do depends how important it is for
> us to preserve existing feature requests and bug reports in a form where
> we can continue working on them (e.g. by closing fixed issues).

We can include a reference to the original issues in debbugs reports.
Then people who need that can look up previous discussions.

Re: Adding use-package to core

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> On the other hand, maybe it would be less confusing to just archive the
>> repository and be done with it.
>
> It will certainly mean less work for us, as we won't need to watch the
> GitHub repository and synchronize it with debbugs.

True.  So I guess we are leaning in this direction.

FWIW, the current number of open issues on GitHub is down from 150 to
103 in the last week.  That included some bug fixes, but the majority
was closing resolved and invalid issues.  With some work, it should be
possible to push that number down even further.

> We can include a reference to the original issues in debbugs reports.
> Then people who need that can look up previous discussions.

That's also true.

Re: Adding use-package to core

[John Wiegley]

>>>>> Payas Relekar <relekarpayas@gmail.com> writes:

> Eglot is still being maintained by João, while use-package cannot expect
> such commitment from John. If Stefan or anyone is willing to volunteer, we
> can continue addressing issues/PRs on both Github as well as mailing list,
> but it must be a commitment at least until the repo is archived.

I'm definitely willing to continue working with Stefan, he's making
considerable progress already on our outstanding issue list. Having an active
collaborator makes it easier to shift some of my time around, since I'm better
at being interrupt driven than scheduling time in advance. So don't count out
me out entirely just yet. :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding use-package to core

[Payas Relekar]

John Wiegley <johnw@gnu.org> writes:

>>>>>> Payas Relekar <relekarpayas@gmail.com> writes:
>
>> Eglot is still being maintained by João, while use-package cannot expect
>> such commitment from John. If Stefan or anyone is willing to volunteer, we
>> can continue addressing issues/PRs on both Github as well as mailing list,
>> but it must be a commitment at least until the repo is archived.
>
> I'm definitely willing to continue working with Stefan, he's making
> considerable progress already on our outstanding issue list. Having an active
> collaborator makes it easier to shift some of my time around, since I'm better
> at being interrupt driven than scheduling time in advance. So don't count out
> me out entirely just yet. :)

That is quite nice to hear :)

Perhaps keeping the repo active will work out better than I had thought before.
--

Re: Adding use-package to core

[Sean Whitton]

Hello,

On Mon 14 Nov 2022 at 08:27AM +08, Po Lu wrote:

> Personally, I hope that everything most people find useful will
> eventually make its way into Emacs, because doing so is a direct
> shortcut to making Emacs more useful for everyone.

We are still aiming to bundle all of GNU ELPA with releases of Emacs at
some point, right?

-- 
Sean Whitton

Re: Adding use-package to core

[Philip Kaludercic]

Sean Whitton <spwhitton@spwhitton.name> writes:

> Hello,
>
> On Mon 14 Nov 2022 at 08:27AM +08, Po Lu wrote:
>
>> Personally, I hope that everything most people find useful will
>> eventually make its way into Emacs, because doing so is a direct
>> shortcut to making Emacs more useful for everyone.
>
> We are still aiming to bundle all of GNU ELPA with releases of Emacs at
> some point, right?

Was that the plan, or to have a separate release of Emacs (fat Emacs)
with all GNU ELPA packages installed by default?

Either way, I wonder how well that would work.

Re: Adding use-package to core

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: xenodasein--- via "Emacs development discussions."
>  <emacs-devel@gnu.org>,  eliz@gnu.org,  xenodasein@tutanota.de
> Date: Fri, 18 Nov 2022 12:29:25 -0700
> 
> Hello,
> 
> On Mon 14 Nov 2022 at 08:27AM +08, Po Lu wrote:
> 
> > Personally, I hope that everything most people find useful will
> > eventually make its way into Emacs, because doing so is a direct
> > shortcut to making Emacs more useful for everyone.
> 
> We are still aiming to bundle all of GNU ELPA with releases of Emacs at
> some point, right?

Not all of it, no.  Only some of it.

RE: [External] : Re: Adding use-package to core

[Drew Adams]

> > We are still aiming to bundle all of GNU ELPA with releases of Emacs at
> > some point, right?
> 
> Was that the plan, or to have a separate release of Emacs (fat Emacs)
> with all GNU ELPA packages installed by default?
> 
> Either way, I wonder how well that would work.

IIRC, there once was a CONTRIB directory delivered
as part of Emacs.  But that was long, long ago.

And maybe it wasn't delivered with GNU Emacs per se
(dunno) but was instead set up here and there in
various labs, universities, etc.  I think it was
provided with GNU Emacs, but I may be wrong.

Re: Adding use-package to core

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sean Whitton <spwhitton@spwhitton.name>
>> Cc: xenodasein--- via "Emacs development discussions."
>>  <emacs-devel@gnu.org>,  eliz@gnu.org,  xenodasein@tutanota.de
>> Date: Fri, 18 Nov 2022 12:29:25 -0700
>> 
>> Hello,
>> 
>> On Mon 14 Nov 2022 at 08:27AM +08, Po Lu wrote:
>> 
>> > Personally, I hope that everything most people find useful will
>> > eventually make its way into Emacs, because doing so is a direct
>> > shortcut to making Emacs more useful for everyone.
>> 
>> We are still aiming to bundle all of GNU ELPA with releases of Emacs at
>> some point, right?
>
> Not all of it, no.  Only some of it.

What would the criteria for inclusion be like?

Re: Adding use-package to core

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Sean Whitton <spwhitton@spwhitton.name>,  luangruo@yahoo.com,
>   emacs-devel@gnu.org,  xenodasein@tutanota.de
> Date: Fri, 18 Nov 2022 20:43:33 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Sean Whitton <spwhitton@spwhitton.name>
> >> Cc: xenodasein--- via "Emacs development discussions."
> >>  <emacs-devel@gnu.org>,  eliz@gnu.org,  xenodasein@tutanota.de
> >> Date: Fri, 18 Nov 2022 12:29:25 -0700
> >> 
> >> Hello,
> >> 
> >> On Mon 14 Nov 2022 at 08:27AM +08, Po Lu wrote:
> >> 
> >> > Personally, I hope that everything most people find useful will
> >> > eventually make its way into Emacs, because doing so is a direct
> >> > shortcut to making Emacs more useful for everyone.
> >> 
> >> We are still aiming to bundle all of GNU ELPA with releases of Emacs at
> >> some point, right?
> >
> > Not all of it, no.  Only some of it.
> 
> What would the criteria for inclusion be like?

Packages that we'd like to have in Emacs, but for some reason are on
ELPA instead.  This would allow packages like Magit, Org, project.el,
and maybe others to stay only on ELPA.  Which is why including in a
release packages from ELPA is something we'd like to be able to do in
the first place.

ELPA admits packages that do the same job as other packages, and also
packages that do some jobs that are extremely niche jobs.  So
including everything makes very little sense to me.

Re: Adding use-package to core

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Sean Whitton <spwhitton@spwhitton.name>,  luangruo@yahoo.com,
>>   emacs-devel@gnu.org,  xenodasein@tutanota.de
>> Date: Fri, 18 Nov 2022 20:43:33 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Sean Whitton <spwhitton@spwhitton.name>
>> >> Cc: xenodasein--- via "Emacs development discussions."
>> >>  <emacs-devel@gnu.org>,  eliz@gnu.org,  xenodasein@tutanota.de
>> >> Date: Fri, 18 Nov 2022 12:29:25 -0700
>> >> 
>> >> Hello,
>> >> 
>> >> On Mon 14 Nov 2022 at 08:27AM +08, Po Lu wrote:
>> >> 
>> >> > Personally, I hope that everything most people find useful will
>> >> > eventually make its way into Emacs, because doing so is a direct
>> >> > shortcut to making Emacs more useful for everyone.
>> >> 
>> >> We are still aiming to bundle all of GNU ELPA with releases of Emacs at
>> >> some point, right?
>> >
>> > Not all of it, no.  Only some of it.
>> 
>> What would the criteria for inclusion be like?
>
> Packages that we'd like to have in Emacs, but for some reason are on
> ELPA instead.  This would allow packages like Magit, Org, project.el,
> and maybe others to stay only on ELPA.  

1. project.el and Org is already included, even developed in Emacs?

2. Are we talking about GNU ELPA or both NonGNU ELPA and GNU ELPA.
   Because fro what I understand, Magit would require a copyright
   exception to be added.

Perhaps we can take a look at the results of the Emacs Survey, when that
comes out later this month and collect a list of popular contenders?

>                                         Which is why including in a
> release packages from ELPA is something we'd like to be able to do in
> the first place.
>
> ELPA admits packages that do the same job as other packages, and also
> packages that do some jobs that are extremely niche jobs.  So
> including everything makes very little sense to me.

I agree.

Re: Adding use-package to core

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: spwhitton@spwhitton.name,  luangruo@yahoo.com,  emacs-devel@gnu.org,
>   xenodasein@tutanota.de
> Date: Sat, 19 Nov 2022 07:33:30 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> What would the criteria for inclusion be like?
> >
> > Packages that we'd like to have in Emacs, but for some reason are on
> > ELPA instead.  This would allow packages like Magit, Org, project.el,
> > and maybe others to stay only on ELPA.  
> 
> 1. project.el and Org is already included, even developed in Emacs?

The intent was to leave them only on ELPA when the way of including
ELPA packages in a release is figured out.  (And Org is definitely NOT
developed in Emacs.)

> 2. Are we talking about GNU ELPA or both NonGNU ELPA and GNU ELPA.

GNU ELPA only.

> Perhaps we can take a look at the results of the Emacs Survey, when that
> comes out later this month and collect a list of popular contenders?

We could do that, but until we have a reliable way of including ELPA
packages in a release (which should include the solution for how to
upgrade such packages after the released Emacs is installed on the
user's machine), doing these secondary jobs is IMO just a waste of
time and energy.  For example, if the solutions are far away in the
future, the list of contenders you collect now will be outdated by
then, and will need to be redone anew.

The issues we are touching here were all discussed in the past, and
the difficulties that need to be resolved were described and also
discussed.  It's nothing new, and I don't think anything's changed
since those discussions, we are still where we were back then wrt our
ability to include ELPA packages.

Re: Adding use-package to core

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: spwhitton@spwhitton.name,  luangruo@yahoo.com,  emacs-devel@gnu.org,
>>   xenodasein@tutanota.de
>> Date: Sat, 19 Nov 2022 07:33:30 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> What would the criteria for inclusion be like?
>> >
>> > Packages that we'd like to have in Emacs, but for some reason are on
>> > ELPA instead.  This would allow packages like Magit, Org, project.el,
>> > and maybe others to stay only on ELPA.  
>> 
>> 1. project.el and Org is already included, even developed in Emacs?
>
> The intent was to leave them only on ELPA when the way of including
> ELPA packages in a release is figured out.  (And Org is definitely NOT
> developed in Emacs.)

Of course.

>> 2. Are we talking about GNU ELPA or both NonGNU ELPA and GNU ELPA.
>
> GNU ELPA only.

OK, but then again, what to do about Magit being on NonGNU ELPA.

>> Perhaps we can take a look at the results of the Emacs Survey, when that
>> comes out later this month and collect a list of popular contenders?
>
> We could do that, but until we have a reliable way of including ELPA
> packages in a release (which should include the solution for how to
> upgrade such packages after the released Emacs is installed on the
> user's machine), doing these secondary jobs is IMO just a waste of
> time and energy.  For example, if the solutions are far away in the
> future, the list of contenders you collect now will be outdated by
> then, and will need to be redone anew.
>
> The issues we are touching here were all discussed in the past, and
> the difficulties that need to be resolved were described and also
> discussed.  It's nothing new, and I don't think anything's changed
> since those discussions, we are still where we were back then wrt our
> ability to include ELPA packages.

What exactly is the complication here?  Wouldn't it be possible to have
a "contrib"/"elpa"/... directory under lisp with ELPA packages that are
prepared before packaging?  Or should these packages be moved into the
core?  Would using fancy tricks like git worktrees, as done by
elpa-admin be a possible approach to tackle the issue?

Re: Adding use-package to core

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: spwhitton@spwhitton.name,  luangruo@yahoo.com,  emacs-devel@gnu.org,
>   xenodasein@tutanota.de
> Date: Sat, 19 Nov 2022 10:09:37 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 2. Are we talking about GNU ELPA or both NonGNU ELPA and GNU ELPA.
> >
> > GNU ELPA only.
> 
> OK, but then again, what to do about Magit being on NonGNU ELPA.

AFAIU, an effort to collect all the assignments for Magit is under
way, with the purpose of adding Magit to ELPA and/or Emacs.

> > We could do that, but until we have a reliable way of including ELPA
> > packages in a release (which should include the solution for how to
> > upgrade such packages after the released Emacs is installed on the
> > user's machine), doing these secondary jobs is IMO just a waste of
> > time and energy.  For example, if the solutions are far away in the
> > future, the list of contenders you collect now will be outdated by
> > then, and will need to be redone anew.
> >
> > The issues we are touching here were all discussed in the past, and
> > the difficulties that need to be resolved were described and also
> > discussed.  It's nothing new, and I don't think anything's changed
> > since those discussions, we are still where we were back then wrt our
> > ability to include ELPA packages.
> 
> What exactly is the complication here?  Wouldn't it be possible to have
> a "contrib"/"elpa"/... directory under lisp with ELPA packages that are
> prepared before packaging?  Or should these packages be moved into the
> core?  Would using fancy tricks like git worktrees, as done by
> elpa-admin be a possible approach to tackle the issue?

The basic issue is: when I install a new Emacs version, and later want
to upgrade to a newer version of a package maintained on ELPA, how
does Emacs do that, and what are user-level implications, given the
existing methods of installing Emacs, both by building it locally and
installing a distro?

Again, the details of this were described and discussed, so I suggest
to look up those past discussions and read there.

Re: Adding use-package to core

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: spwhitton@spwhitton.name,  luangruo@yahoo.com,  emacs-devel@gnu.org,
>>   xenodasein@tutanota.de
>> Date: Sat, 19 Nov 2022 10:09:37 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> 2. Are we talking about GNU ELPA or both NonGNU ELPA and GNU ELPA.
>> >
>> > GNU ELPA only.
>> 
>> OK, but then again, what to do about Magit being on NonGNU ELPA.
>
> AFAIU, an effort to collect all the assignments for Magit is under
> way, with the purpose of adding Magit to ELPA and/or Emacs.

That I did not know, but I can't find anything on the topic online either.

>> > We could do that, but until we have a reliable way of including ELPA
>> > packages in a release (which should include the solution for how to
>> > upgrade such packages after the released Emacs is installed on the
>> > user's machine), doing these secondary jobs is IMO just a waste of
>> > time and energy.  For example, if the solutions are far away in the
>> > future, the list of contenders you collect now will be outdated by
>> > then, and will need to be redone anew.
>> >
>> > The issues we are touching here were all discussed in the past, and
>> > the difficulties that need to be resolved were described and also
>> > discussed.  It's nothing new, and I don't think anything's changed
>> > since those discussions, we are still where we were back then wrt our
>> > ability to include ELPA packages.
>> 
>> What exactly is the complication here?  Wouldn't it be possible to have
>> a "contrib"/"elpa"/... directory under lisp with ELPA packages that are
>> prepared before packaging?  Or should these packages be moved into the
>> core?  Would using fancy tricks like git worktrees, as done by
>> elpa-admin be a possible approach to tackle the issue?
>
> The basic issue is: when I install a new Emacs version, and later want
> to upgrade to a newer version of a package maintained on ELPA, how
> does Emacs do that, and what are user-level implications, given the
> existing methods of installing Emacs, both by building it locally and
> installing a distro?

Ok, I get that.

> Again, the details of this were described and discussed, so I suggest
> to look up those past discussions and read there.

I'll try that too.  Do you have any specific discussions in mind.
Searching the archive can be a bit difficult when looking for general
discussions.

Re: Adding use-package to core

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

>> AFAIU, an effort to collect all the assignments for Magit is under
>> way, with the purpose of adding Magit to ELPA and/or Emacs.
>
> That I did not know, but I can't find anything on the topic online either.

I don't think anyone is working on it.  I found this in the archives:

https://lists.gnu.org/archive/html/emacs-devel/2017-07/msg00437.html

Re: Adding use-package to core

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: spwhitton@spwhitton.name,  luangruo@yahoo.com,  emacs-devel@gnu.org,
>   xenodasein@tutanota.de
> Date: Sat, 19 Nov 2022 11:02:28 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> OK, but then again, what to do about Magit being on NonGNU ELPA.
> >
> > AFAIU, an effort to collect all the assignments for Magit is under
> > way, with the purpose of adding Magit to ELPA and/or Emacs.
> 
> That I did not know, but I can't find anything on the topic online either.

Here's the last time this was discussed:

  https://lists.gnu.org/archive/html/emacs-devel/2022-09/msg00125.html

> > Again, the details of this were described and discussed, so I suggest
> > to look up those past discussions and read there.
> 
> I'll try that too.  Do you have any specific discussions in mind.
> Searching the archive can be a bit difficult when looking for general
> discussions.

I think the last time this was discussed is here:

  https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00729.html

In particular, I tried to describe the issues that bother me here:

  https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00917.html

Re: Adding use-package to core

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: spwhitton@spwhitton.name,  luangruo@yahoo.com,  emacs-devel@gnu.org,
>>   xenodasein@tutanota.de
>> Date: Sat, 19 Nov 2022 11:02:28 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> OK, but then again, what to do about Magit being on NonGNU ELPA.
>> >
>> > AFAIU, an effort to collect all the assignments for Magit is under
>> > way, with the purpose of adding Magit to ELPA and/or Emacs.
>> 
>> That I did not know, but I can't find anything on the topic online either.
>
> Here's the last time this was discussed:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2022-09/msg00125.html
>
>> > Again, the details of this were described and discussed, so I suggest
>> > to look up those past discussions and read there.
>> 
>> I'll try that too.  Do you have any specific discussions in mind.
>> Searching the archive can be a bit difficult when looking for general
>> discussions.
>
> I think the last time this was discussed is here:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00729.html
>
> In particular, I tried to describe the issues that bother me here:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00917.html

Thanks!

Re: Adding use-package to core

[Stefan Monnier]

> Packages that we'd like to have in Emacs, but for some reason are on
> ELPA instead.

Yup.  I think it would grow a bit in the same kind of way that we've
added packages to Emacs in the past with similar tradeoffs between
offering a better "out of the box" experience and increasing the size of
the Emacs tarball too much.

> ELPA admits packages that do the same job as other packages, and also
> packages that do some jobs that are extremely niche jobs.
> So including everything makes very little sense to me.

Agreed.  Also there's the question of code quality.

Think of it a bit like NonGNU ELPA: in theory we could add almost all
"Melpa packages to NonGNU ELPA" (aka "GNU ELPA packages to Emacs"), but
instead we add them one by one.


        Stefan

Re: Adding use-package to core

[Stefan Monnier]

> packages in a release (which should include the solution for how to
> upgrade such packages after the released Emacs is installed on the
> user's machine),

FWIW, this has been a solved problem already in Emacs-24: package.el has
always been able to handle the presence of both system-wide packages and
user-installed packages and to pick the most recent version.


        Stefan

Re: Adding use-package to core

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> packages in a release (which should include the solution for how to
>> upgrade such packages after the released Emacs is installed on the
>> user's machine),
>
> FWIW, this has been a solved problem already in Emacs-24: package.el has
> always been able to handle the presence of both system-wide packages and
> user-installed packages and to pick the most recent version.

This is what has been confusing me about Eli's comments.  Debian does
this all the time, there are a great number of packages that can be
installed on a system-wide basis:

--8<---------------cut here---------------start------------->8---
$ apt search elpa-

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Sorting...
Full Text Search...
elpa-a/testing 1.0.0-2 all
  functions for dealing with associative structures

elpa-ac-rtags/testing 2.38-6 all
  auto-complete back-end for RTags

elpa-ace-link/testing 0.5.0-3 all
  selecting a link to jump to

elpa-ace-popup-menu/testing 0.2.1-3 all
  replace GUI popup menu with something more efficient

elpa-ace-window/testing 0.10.0-1 all
  selecting a window to switch to
--8<---------------cut here---------------end--------------->8---

That is also why, in my eyes the issue is reduced to the matter of
deciding if the selected ELPA packages are bundled in Git (e.g. as
worktrees) or when the tarballs are generated (that distributions use to
build their packages).

But I am not done with reading through the archives, so I might just be
repeating stuff pointlessly.

Re: Adding use-package to core

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Philip Kaludercic <philipk@posteo.net>,  spwhitton@spwhitton.name,
>   luangruo@yahoo.com,  emacs-devel@gnu.org,  xenodasein@tutanota.de
> Date: Sat, 19 Nov 2022 10:28:56 -0500
> 
> > packages in a release (which should include the solution for how to
> > upgrade such packages after the released Emacs is installed on the
> > user's machine),
> 
> FWIW, this has been a solved problem already in Emacs-24: package.el has
> always been able to handle the presence of both system-wide packages and
> user-installed packages and to pick the most recent version.

It might be _almost_ solved, but I'm not sure it is 100% solved.  We
need a clear description of:

  . how are such packages installed when a release tarball is built
    and installed ("how" here means where and with which artifacts)
  . how will this work in a built but uninstalled source tree
  . how are such packages installed from a distro
  . how can they be upgraded from ELPA and downgraded back to the
    version that came with the tarball

I asked this question in the Dec 2020 discussion, and you responded
here:

  https://lists.gnu.org/archive/html/emacs-devel/2020-12/msg00923.html

The response triggered a few followup questions, and I don't think we
have satisfactory, let alone final, answers for all of those
questions.

I'm not saying this part of the issue is rocket science, but it does
have to be figured out.