Testing fontification, indentation, and buffer manipulation

Testing fontification, indentation, and buffer manipulation

[Daniele Nicolodi]

Hello,

I am hacking on beancount-mode (a mode to edit Beancount ledger files)
and I would like to write some automated tests to check the
functionality of the minor mode.  I found ERT, but it seems that it does
not offer any facility to easily test fontification, indentation, or
buffer manipulation in general.

Is there any facility that would help in writing such tests?

Thank you.

Cheers,
Dan

Re: Testing fontification, indentation, and buffer manipulation

[Yuri Khan]

On Tue, Jan 15, 2019 at 10:52 PM Daniele Nicolodi <daniele@grinta.net> wrote:

> I am hacking on beancount-mode (a mode to edit Beancount ledger files)
> and I would like to write some automated tests to check the
> functionality of the minor mode.  I found ERT, but it seems that it does
> not offer any facility to easily test fontification, indentation, or
> buffer manipulation in general.
>
> Is there any facility that would help in writing such tests?

You might want to look at ledger-mode tests. Its approach to
fontification testing is:

* Create a temporary buffer.
* Put test text into it.
* Harvest it back with fontification properties into a data structure.
* Test that this structure is equal to the golden output.

https://github.com/ledger/ledger-mode/blob/master/test/test-helper.el#L149
https://github.com/ledger/ledger-mode/blob/master/test/fontify-test.el

Re: Testing fontification, indentation, and buffer manipulation

[Helmut Eller]

On Tue, Jan 15 2019, Daniele Nicolodi wrote:

> I am hacking on beancount-mode (a mode to edit Beancount ledger files)
> and I would like to write some automated tests to check the
> functionality of the minor mode.  I found ERT, but it seems that it does
> not offer any facility to easily test fontification, indentation, or
> buffer manipulation in general.
>
> Is there any facility that would help in writing such tests?

Something that I found useful was using somewhat exotic characters to
mark positions.  E.g.

  (forth-assert-face ": →frob ;" font-lock-function-name-face)

tests that the face after the → is font-lock-function-name-face.

Or

  (forth-assert-forward-sexp " ¹if drop exit else 1+ then² bar ")

tests that if forward-sexp is called at the position marked with ¹ it
will move to ².

Helmut

Re: Testing fontification, indentation, and buffer manipulation

[Daniele Nicolodi]

On 15/01/2019 11:45, Yuri Khan wrote:
> On Tue, Jan 15, 2019 at 10:52 PM Daniele Nicolodi <daniele@grinta.net> wrote:
> 
>> I am hacking on beancount-mode (a mode to edit Beancount ledger files)
>> and I would like to write some automated tests to check the
>> functionality of the minor mode.  I found ERT, but it seems that it does
>> not offer any facility to easily test fontification, indentation, or
>> buffer manipulation in general.
>>
>> Is there any facility that would help in writing such tests?
> 
> You might want to look at ledger-mode tests. Its approach to
> fontification testing is:
> 
> * Create a temporary buffer.
> * Put test text into it.
> * Harvest it back with fontification properties into a data structure.
> * Test that this structure is equal to the golden output.
> 
> https://github.com/ledger/ledger-mode/blob/master/test/test-helper.el#L149
> https://github.com/ledger/ledger-mode/blob/master/test/fontify-test.el

Thanks Yuri, this is very helpful.

Cheers,
Dan

Re: Testing fontification, indentation, and buffer manipulation

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 2616 bytes --]

Hey Daniele.

In several of the (third-party) major-modes I author or maintain, I use
the library "assess" available on MELPA for testing fontification. See:
https://melpa.org/#/assess
For testing indentation, I've also created reference-documents and
completely reindented them in a test and then  compared the results.
This provides some basic (but not complete) coverage, because (depending
on the mode) interactive usage may give different results.
An example. Consider the following line of c/java/javascript/php/c#-
code:
    if (someBool)

The correct indentation for the following line could be:

 - the same as the preceeding line if the next line contains an opening
   bracket ({)- nested one level deeper if it contains a direct expression.

So when pressing enter... Should you indent immediately? And should you
then later on correct the indentation if a user creates a block-start
marker { ?
I'm not sure what the right answer is, but it is an example of something
which will differ in a interactive use-case vs reflowing a completed,
existing document.
If your major-mode has cases like this, you will have to write more
elaborate tests to ensure these too are handled correctly.
That said: I'm not sure if this is the recommended or canonical
approach, but combining these techniques has worked for me in my
projects.
--
Regards
Jostein Kjønigsen

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjonigsen.net


On Sat, Jan 19, 2019, at 4:07 PM, Daniele Nicolodi wrote:
> On 15/01/2019 11:45, Yuri Khan wrote:
>> On Tue, Jan 15, 2019 at 10:52 PM Daniele Nicolodi
>> <daniele@grinta.net> wrote:>> 
>>> I am hacking on beancount-mode (a mode to edit Beancount ledger
>>> files)>>> and I would like to write some automated tests to check the
>>> functionality of the minor mode.  I found ERT, but it seems that
>>> it does>>> not offer any facility to easily test fontification, indentation, or>>> buffer manipulation in general.
>>> 
>>> Is there any facility that would help in writing such tests?
>> 
>> You might want to look at ledger-mode tests. Its approach to
>> fontification testing is:
>> 
>> * Create a temporary buffer.
>> * Put test text into it.
>> * Harvest it back with fontification properties into a data
>>   structure.>> * Test that this structure is equal to the golden output.
>> 
>> https://github.com/ledger/ledger-mode/blob/master/test/test-helper.el#L149>> https://github.com/ledger/ledger-mode/blob/master/test/fontify-test.el> 
> Thanks Yuri, this is very helpful.
> 
> Cheers,
> Dan
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 4290 bytes --]

Re: Testing fontification, indentation, and buffer manipulation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If assess is useful, can we arrange to get it included in Emacs?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If assess is useful, can we arrange to get it included in Emacs?

Yes. Been meaning too for ages.

I was hoping to have it as both an ELPA package and available in core,
but there still isn't a good mechanism for that at the moment. I guess I
should just add it to core, so it is available in Emacs-27.

Phil

Re: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

> I was hoping to have it as both an ELPA package and available in core,
> but there still isn't a good mechanism for that at the moment.

Don't know about "good", but we can (and do) publish GNU ELPA packages
where the code is kept in core (i.e. in the master branch of emacs.git):

    % grep :core .../elpa/externals-list
    ;;  :core     = part of GNU Emacs repository.
    ;; :core URL must be a list of:
     ;; ("cc-mode"          :core ("lisp/progmodes/cc-align.el"
     ("cl-print"            :core "lisp/emacs-lisp/cl-print.el")
     ("flymake"             :core "lisp/progmodes/flymake.el")
     ("jsonrpc"             :core "lisp/jsonrpc.el")
     ("let-alist"           :core "lisp/emacs-lisp/let-alist.el")
     ("map"                 :core "lisp/emacs-lisp/map.el")
     ("ntlm"                :core "lisp/net/ntlm.el")
     ("python"              :core "lisp/progmodes/python.el")
     ("soap-client"         :core ("lisp/net/soap-client.el" "lisp/net/soap-inspect.el"))
     ;;("tramp"             :core
    %

It has some rough edges, which makes it currently impractical to do that
for Tramp (for example), but for simple enough cases it works OK.


        Stefan

Re: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I was hoping to have it as both an ELPA package and available in core,
>> but there still isn't a good mechanism for that at the moment.
>
> Don't know about "good", but we can (and do) publish GNU ELPA packages
> where the code is kept in core (i.e. in the master branch of emacs.git):
>
>     % grep :core .../elpa/externals-list
>     ;;  :core     = part of GNU Emacs repository.
>     ;; :core URL must be a list of:
>      ;; ("cc-mode"          :core ("lisp/progmodes/cc-align.el"
>      ("cl-print"            :core "lisp/emacs-lisp/cl-print.el")
>      ("flymake"             :core "lisp/progmodes/flymake.el")
>      ("jsonrpc"             :core "lisp/jsonrpc.el")
>      ("let-alist"           :core "lisp/emacs-lisp/let-alist.el")
>      ("map"                 :core "lisp/emacs-lisp/map.el")
>      ("ntlm"                :core "lisp/net/ntlm.el")
>      ("python"              :core "lisp/progmodes/python.el")
>      ("soap-client"         :core ("lisp/net/soap-client.el" "lisp/net/soap-inspect.el"))
>      ;;("tramp"             :core
>     %
>
> It has some rough edges, which makes it currently impractical to do that
> for Tramp (for example), but for simple enough cases it works OK.


Yeah, I saw that, and that would be a good way to go. I guess these
packages are developed in core though (i.e. it's their primary
repository)? For assess, it's nice to be able to develop them outside of
core, because that makes it easier, for example, to test against
multiple versions of Emacs (I am trying to support two major versions).

I've also got working code that takes an ELPA package and copies it into
the core build. It's quite simple to do actually, the documentation
needs improving and to plump in some configure options. It would raise
the possibility of having a "Emacs minimal" core distribution (i.e. with
no "elpa" packages). Interested?

Phil

Re: Testing fontification, indentation, and buffer manipulation

[Ted Zlatanov]

On Thu, 24 Jan 2019 10:29:29 +0000 phillip.lord@russet.org.uk (Phillip Lord) wrote: 

PL> I've also got working code that takes an ELPA package and copies it into
PL> the core build. It's quite simple to do actually, the documentation
PL> needs improving and to plump in some configure options. It would raise
PL> the possibility of having a "Emacs minimal" core distribution (i.e. with
PL> no "elpa" packages). Interested?

That would be a terrific addition to emba.gnu.org (currently it just
does the default build):

* build and test without ELPA
* build and test with ELPA (including tests from ELPA!)

Network connectivity is slightly tricky because it's through a proxy but
may not be needed to ELPA. Let me know if that sounds useful and I can
try to implement the CI side.

Thanks :)
Ted

Re: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

> I've also got working code that takes an ELPA package and copies it into
> the core build.  It's quite simple to do actually, the documentation
> needs improving and to plump in some configure options.  It would raise
> the possibility of having a "Emacs minimal" core distribution (i.e. with
> no "elpa" packages).  Interested?

I think we need that, yes (e.g. we really should bundle Company with
Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
if and how he wants such a thing.


        Stefan

Re: Testing fontification, indentation, and buffer manipulation

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 25 Jan 2019 19:46:20 -0500
> 
> > I've also got working code that takes an ELPA package and copies it into
> > the core build.  It's quite simple to do actually, the documentation
> > needs improving and to plump in some configure options.  It would raise
> > the possibility of having a "Emacs minimal" core distribution (i.e. with
> > no "elpa" packages).  Interested?
> 
> I think we need that, yes (e.g. we really should bundle Company with
> Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
> if and how he wants such a thing.

My opinions on that are known: I think that we should have everything
useful in core, and ELPA should only host packages that are "not yet"
in core, or have some other issues that prevent their move to core.
Since this is at odds with opinions of many others, I very much doubt
that my decision along these lines will stand.

Re: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've also got working code that takes an ELPA package and copies it into
>> the core build.  It's quite simple to do actually, the documentation
>> needs improving and to plump in some configure options.  It would raise
>> the possibility of having a "Emacs minimal" core distribution (i.e. with
>> no "elpa" packages).  Interested?
>
> I think we need that, yes (e.g. we really should bundle Company with
> Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
> if and how he wants such a thing.

I've pushed this to:

feature/core-elpa-by-copy

Currently, it's in as a feature

./configure --enable-elpa
./configure --enable-elpa=where-to-find-or-clone-elpa

Main features:

 - Works with externals or non-external packages.
 - Versions are specified by git SHA, so a version build will be
   repeatable.
 - Both code and tests are supported
 - It's customizable by package authors on ELPA.
 - Doesn't use package.el (I think this is an anti-feature, but tried
   that before and no one liked it)
 - It requires network access only if ELPA is not available locally, or if
   the Makefile which describes versions is updated
 - Works with make -j in which case it does a git clone which the C is building

None Features: 
 - Currently, only works with a single git repo (i.e. ELPA) which is
   hard coded.
 - Doesn't support documentation or info

To be done:
 - Give it prettier output
 - Clean ups.


Phil

Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I've also got working code that takes an ELPA package and copies it into
>>> the core build.  It's quite simple to do actually, the documentation
>>> needs improving and to plump in some configure options.  It would raise
>>> the possibility of having a "Emacs minimal" core distribution (i.e. with
>>> no "elpa" packages).  Interested?
>>
>> I think we need that, yes (e.g. we really should bundle Company with
>> Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
>> if and how he wants such a thing.
>
> I've pushed this to:
>
> feature/core-elpa-by-copy
>
> Currently, it's in as a feature
>
> ./configure --enable-elpa
> ./configure --enable-elpa=where-to-find-or-clone-elpa
>
> Main features:
>
>  - Works with externals or non-external packages.
>  - Versions are specified by git SHA, so a version build will be
>    repeatable.
>  - Both code and tests are supported
>  - It's customizable by package authors on ELPA.
>  - Doesn't use package.el (I think this is an anti-feature, but tried
>    that before and no one liked it)
>  - It requires network access only if ELPA is not available locally, or if
>    the Makefile which describes versions is updated
>  - Works with make -j in which case it does a git clone which the C is building
>
> None Features: 
>  - Currently, only works with a single git repo (i.e. ELPA) which is
>    hard coded.
>  - Doesn't support documentation or info
>
> To be done:
>  - Give it prettier output
>  - Clean ups.


Wondering whether anyone had time to give me feedback on this. It
provides a mechanism to have ELPA packages bundle with core Emacs.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

> Wondering whether anyone had time to give me feedback on this. It
> provides a mechanism to have ELPA packages bundle with core Emacs.

Sorry for taking so long.  I just took a look at it, and it looks fairly
simple and clean.  But I think the main question is *if* we really want
to use this to build the official tarballs.  I think we should, but IIRC
Eli wasn't quite as enthusiastic about it.

>  - Doesn't use package.el (I think this is an anti-feature, but tried
>    that before and no one liked it)

That doesn't ring a bell.  Could you refresh our memory why no one
liked it?

> --- /dev/null
> +++ b/notes.org
> @@ -0,0 +1,84 @@
> +
> +
> +* Working on
> +
> +Have package checkout working for external
> +Next get it working for non external so we can template

Don't know what you mean by "can template"

> +* To Fix
> +Think we are running git fetch --all on every package.

Don't know what you're referring to here either.

> +1) Scripting?
> +Guess we have to use sh

Of course, we could also use Elisp ;-)

> +2) How to get specific directory of a repository
> +
> +
> +
> +git archive --remote=phillord@git.savannah.gnu.org:/srv/git/emacs/elpa.git master packages/hydra | tar xv --strip-components=1
> +
> +Currently, this requires a member login -- it doesn't work with http
> +or the current git server.
> +
> +Would also be possible to configure this from local.

I'm not familiar with `git archive` but does this require network access?

> +3) How to work out when a directory is out of date wrt to the make
> +   file

Don't know what you mean here.

> +4) Heuristics for working out which files to copy to right place
> +
> +*-test.el
> +*-tests.el
> +/test/*.el
> +
> +to test
> +
> +All other *el to lisp
> +
> +Nothing else

Rather than heuristics, I'd go for a *rules* (which needs to be obeyed
before the package can be bundled).

But I'm not sure we should install those packages with a different
layout than for a "normal" ELPA install.
What's the advantage of using a different layout?

> +6) How to enable ELPA packages to customize the process
> +Add the option to run an core-deploy.sh file placed into the root of a
> +package.

Don't know what you mean here.

> +8) Support other repos
> +Org-mode effectively does this already

Don't know what you mean here.

> --- /dev/null
> +++ b/elpa/bin/extract-package.sh
> @@ -0,0 +1,55 @@
> +#!/bin/bash
> +
> +function grab_external {
> +    rm -rf packages/$PACKAGE*
> +    mkdir --parents $PACKAGE_LOC
> +    cwd=`pwd`
> +    cd $GIT_LOC
> +    git archive $SHA \
> +        | tar xv -C $cwd/$PACKAGE_LOC
> +    cd $cwd
> +    cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC

Rather than `cd $cwd`, I generally prefer to use (...) to save&restore
the current directory:

    cwd=`pwd`
    (cd $GIT_LOC
     git archive $SHA \
         | tar xv -C $cwd/$PACKAGE_LOC)
    cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC

If we keep the layout unchanged, we could use `git worktree add`, which
has the advantage that when running Emacs in-place, jumping to the
source in `C-h f` and friends gives you access to the Git metainfo.


        Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Wondering whether anyone had time to give me feedback on this. It
>> provides a mechanism to have ELPA packages bundle with core Emacs.
>
> Sorry for taking so long.  I just took a look at it, and it looks fairly
> simple and clean.  But I think the main question is *if* we really want
> to use this to build the official tarballs.  I think we should, but IIRC
> Eli wasn't quite as enthusiastic about it.

Indeed. It also means that git will (sometimes) be needed to build;
although this will only be true after git pulling the main git repo, so
perhaps this is not such an issue.

I would also plan to build two tar balls -- emacs-and-selected-elpa.tgz
and emacs.tgz. Over time, I think it would make sense to change these to
emacs.tgz and emacs-minimal.tgz (i.e. with ELPA becomes the default). 


>>  - Doesn't use package.el (I think this is an anti-feature, but tried
>>    that before and no one liked it)
>
> That doesn't ring a bell.  Could you refresh our memory why no one
> liked it?

A variety of reasons. It's less simple. It requires changes to
package.el (for example, what should emacs -Q do with ELPA packages. In
my original version they would be disabled, in this version they are
still on). package.el has to be launched every start up for every
emacs. And Eli in particular didn't like the increase in the number of
directories in the default load-path. This implementation does the same
thing, but it doesn't have to -- we could copy all the lisp files into
the same directory.



>> --- /dev/null
>> +++ b/notes.org
>> @@ -0,0 +1,84 @@
>> +
>> +
>> +* Working on
>> +
>> +Have package checkout working for external
>> +Next get it working for non external so we can template
>
> Don't know what you mean by "can template"

Sorry the notes are a bit out of date (bad habit of mine, and I wasn't
expecting anyone to read this file, cause it will go before merge).

This is working now (neither queue nor darkroom are external).


>> +* To Fix
>> +Think we are running git fetch --all on every package.
>
> Don't know what you're referring to here either.


Also fixed. Getting the dependencies was a bit tricky. 


>> +1) Scripting?
>> +Guess we have to use sh
>
> Of course, we could also use Elisp ;-)

If I remember correctly, all of this happens before the full Emacs is
dumped. It seemed cleaner to do this to me.


>> +2) How to get specific directory of a repository
>> +
>> +
>> +
>> +git archive
>> --remote=phillord@git.savannah.gnu.org:/srv/git/emacs/elpa.git
>> master packages/hydra | tar xv --strip-components=1
>> +
>> +Currently, this requires a member login -- it doesn't work with http
>> +or the current git server.
>> +
>> +Would also be possible to configure this from local.
>
> I'm not familiar with `git archive` but does this require network
> access?

git archive gives a bare, not with a .git, directory.

I had to drop that command anyway. Git doesn't allow checkout of an
arbitrary SHA from a remote repo (because it would allow checking out
commits that had been deleted otherwise, I believe). So instead, now,
the makefile checkouts out the whole of ELPA, and then uses git archive
to get a specific SHA from local.

Obviously, this is a fairly network intensive way of getting just a few
files; but it's a fixed sized problem.


>> +3) How to work out when a directory is out of date wrt to the make
>> +   file
>
> Don't know what you mean here.


That's fixed also. Packages should updated from git (and git fetch run)
if Makefile is updated. So, this means a git fetch after everytime Emacs
is ./configure is run.

>
>> +4) Heuristics for working out which files to copy to right place
>> +
>> +*-test.el
>> +*-tests.el
>> +/test/*.el
>> +
>> +to test
>> +
>> +All other *el to lisp
>> +
>> +Nothing else
>
> Rather than heuristics, I'd go for a *rules* (which needs to be obeyed
> before the package can be bundled).

ELPA already uses heuristics (see admin/ert-support.el). Rules would
make this simpler, of course, at the cost of, well, another rule.


> But I'm not sure we should install those packages with a different
> layout than for a "normal" ELPA install.
> What's the advantage of using a different layout?

I haven't checked to see who uses them or why, so I don't know.

>
>> +6) How to enable ELPA packages to customize the process
>> +Add the option to run an core-deploy.sh file placed into the root of a
>> +package.
>
> Don't know what you mean here.


That bits out of date, and supported another way.

By default, EMACS/elpa/bin/package-makefile.mk is copied into a package
after it is pulled out of git and it is this that copies files from the
elpa directory into the core Emacs layout. If a package wants to
override this behaviour, it could include it's own package-makefile.mk.


>> +8) Support other repos
>> +Org-mode effectively does this already
>
> Don't know what you mean here.

Currently, this will only take files from the ELPA repo, but there is
not reason that this needs to be so. Files could come another repo (such
as org-mode's own). My guess is that there is no particular reason to
support this at the moment.


>> --- /dev/null
>> +++ b/elpa/bin/extract-package.sh
>> @@ -0,0 +1,55 @@
>> +#!/bin/bash
>> +
>> +function grab_external {
>> +    rm -rf packages/$PACKAGE*
>> +    mkdir --parents $PACKAGE_LOC
>> +    cwd=`pwd`
>> +    cd $GIT_LOC
>> +    git archive $SHA \
>> +        | tar xv -C $cwd/$PACKAGE_LOC
>> +    cd $cwd
>> +    cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC
>
> Rather than `cd $cwd`, I generally prefer to use (...) to save&restore
> the current directory:
>
>     cwd=`pwd`
>     (cd $GIT_LOC
>      git archive $SHA \
>          | tar xv -C $cwd/$PACKAGE_LOC)
>     cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC

Oh, yes, hadn't thought of that.


> If we keep the layout unchanged, we could use `git worktree add`, which
> has the advantage that when running Emacs in-place, jumping to the
> source in `C-h f` and friends gives you access to the Git metainfo.

Not sure I have understood -- you mean git subtree add perhaps? This
wouldn't work anyway, because C-h f will jump to source in
EMACS/lisp/elpa which is not created by git archive.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

>>> +1) Scripting?
>>> +Guess we have to use sh
>> Of course, we could also use Elisp ;-)
> If I remember correctly, all of this happens before the full Emacs is
> dumped.  It seemed cleaner to do this to me.

I'm fine with using `sh`, but I think that it'd be perfectly OK to delay
this elpa-inclusion to after `src/emacs` is dumped.

> That's fixed also. Packages should updated from git (and git fetch run)
> if Makefile is updated. So, this means a git fetch after everytime Emacs
> is ./configure is run.

I think a plain `make` shouldn't perform a `git fetch` every time the
Makefile changes.  It's OK to require a `git fetch` in that case, but
if/when the `git fetch` happens should be under the control of the user
(i.e. should require some explicit step like `make update-elpa`).

>> But I'm not sure we should install those packages with a different
>> layout than for a "normal" ELPA install.
>> What's the advantage of using a different layout?
> I haven't checked to see who uses them or why, so I don't know.

I think there's a misunderstanding: I was not talking about the layout
used internally by the packages, but the layout used by your code (where
you put some stuff in lisp/ and other in test/ so the layout of the
package's files is modified compared to what happens in a normal ELPA
install).

> Currently, this will only take files from the ELPA repo, but there is
> not reason that this needs to be so. Files could come another repo (such
> as org-mode's own). My guess is that there is no particular reason to
> support this at the moment.

I see no need to support fetching from other Git repositories, indeed,
except in order to use a local mirror, of course.

> This wouldn't work anyway, because C-h f will jump to source in
> EMACS/lisp/elpa which is not created by git archive.

What I meant is that it'd be much better if we could make sure that
EMACS/lisp/elpa has the relevant .git metadata.


        Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>>> +1) Scripting?
>>>> +Guess we have to use sh
>>> Of course, we could also use Elisp ;-)
>> If I remember correctly, all of this happens before the full Emacs is
>> dumped.  It seemed cleaner to do this to me.
>
> I'm fine with using `sh`, but I think that it'd be perfectly OK to delay
> this elpa-inclusion to after `src/emacs` is dumped.

The `sh` is written now anyway and it's fairly simple, so I am less
worried about it. It does make the dependencies easier, which Emacs
multi-directory structure doesn't make easier. Especially, since the git
clone (or update as necessary) currently happens during the C build. In
practice, this means the network time has a small impact on the overall build.


>> That's fixed also. Packages should updated from git (and git fetch run)
>> if Makefile is updated. So, this means a git fetch after everytime Emacs
>> is ./configure is run.
>
> I think a plain `make` shouldn't perform a `git fetch` every time the
> Makefile changes.  It's OK to require a `git fetch` in that case, but
> if/when the `git fetch` happens should be under the control of the user
> (i.e. should require some explicit step like `make update-elpa`).

The problem is that if the Makefile updates, then it is possible that
it contains an sha-1 that doesn't exist in the ELPA clone, because the
makefile is new. If this happens the git archive command will break. I'd
rather have the build just work. Apart from convienience, if it does
not, emacs-devel will get a lot of "emacs build is broken" errors after
an update.

I could move the dependency to Makefile.in I guess? Then a simple
./configure wouldn't change things, but any textual change in
Makefile.in would. Or, I could check the repo for the SHA-1 first and if
this doesn't exist, the run git fetch.




>>> But I'm not sure we should install those packages with a different
>>> layout than for a "normal" ELPA install.
>>> What's the advantage of using a different layout?
>> I haven't checked to see who uses them or why, so I don't know.
>
> I think there's a misunderstanding: I was not talking about the layout
> used internally by the packages, but the layout used by your code (where
> you put some stuff in lisp/ and other in test/ so the layout of the
> package's files is modified compared to what happens in a normal ELPA
> install).


Oh, sorry. Think we are good, then. At the moment, there is a single
layout for any ELPA package (unless it provides it's own
package-makefile.mk).

>> Currently, this will only take files from the ELPA repo, but there is
>> not reason that this needs to be so. Files could come another repo (such
>> as org-mode's own). My guess is that there is no particular reason to
>> support this at the moment.
>
> I see no need to support fetching from other Git repositories, indeed,
> except in order to use a local mirror, of course.

Yes!



>> This wouldn't work anyway, because C-h f will jump to source in
>> EMACS/lisp/elpa which is not created by git archive.
>
> What I meant is that it'd be much better if we could make sure that
> EMACS/lisp/elpa has the relevant .git metadata.


So that you could edit EMACS/lisp/elpa files in their installed location
and update them? Or something simpler. I think that would be a big
challenge.

Simpler would be to add a file called ".this-is-not-a-source-file" and
have emacs-lisp-mode detect this and point at the real
source. My default my Makefile makes a bare clone of ELPA so there are
no source files to edit there but

./configure --enable-elpa=my-real-no-bare-elpa-clone

should also work. So there might well be real source.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

> I could move the dependency to Makefile.in I guess? Then a simple
> ./configure wouldn't change things, but any textual change in
> Makefile.in would. Or, I could check the repo for the SHA-1 first and if
> this doesn't exist, the run git fetch.

My opinion is that the problem in in having the SHAs in Makefile.in.
I think Makefile.in should refer to branches/tags rather than to SHAs.


        Stefan


PS: Also, every time Makefile.in changes, `make` re-runs config.status
which then causes all the .o files to be recompiled as well, so I'd
rather we don't change it all too often.

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think a plain `make` shouldn't perform a `git fetch` every time the
  > Makefile changes.  It's OK to require a `git fetch` in that case, but
  > if/when the `git fetch` happens should be under the control of the user
  > (i.e. should require some explicit step like `make update-elpa`).

I don't think it is right for 'make' to affect the source files people edit,
except in limited automatic ways.

There are sometimes reasons to continue building the same checkout --
perhaps making some local edits -- and excluding all other changes
that could confuse matters.
-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I could move the dependency to Makefile.in I guess? Then a simple
>> ./configure wouldn't change things, but any textual change in
>> Makefile.in would. Or, I could check the repo for the SHA-1 first and if
>> this doesn't exist, the run git fetch.
>
> My opinion is that the problem in in having the SHAs in Makefile.in.
> I think Makefile.in should refer to branches/tags rather than to SHAs.

I thought about that. Actually, the code would cope with branches and
tags, but I think they are both a bad idea.

The problem with branches is that what is on the branch changes. My
feeling is that the build process should be repeatable; so building
emacs-27.1-with-elpa.tgz should not result in different tar balls as
ELPA changes.

The problem with tags is that with ELPA as a single git, tags share a
single namespace. Otherwise, people could just add "emacs-27.1" or
whatever to say which version of the package should be for
Emacs-27.1. But this would require all ELPA packages on master to
choose the same point in time for a release and would not work at all
for packages on a branch.

Finally, neither solve the problem -- if a branch or tag is add
EMACS/elpa/Makefile.in which exists in the ELPA repo, but not in the
clone on the local machine the build will fail.



> PS: Also, every time Makefile.in changes, `make` re-runs config.status
> which then causes all the .o files to be recompiled as well, so I'd
> rather we don't change it all too often.

Oh, yes, that's true. Putting those eval calls into the Makefile is
pretty ugly. I can move the SHA data out into a secondary file no
worries.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I think a plain `make` shouldn't perform a `git fetch` every time the
>   > Makefile changes.  It's OK to require a `git fetch` in that case, but
>   > if/when the `git fetch` happens should be under the control of the user
>   > (i.e. should require some explicit step like `make update-elpa`).
>
> I don't think it is right for 'make' to affect the source files people edit,
> except in limited automatic ways.

That's not the case here. The idea is this. You (the user) manually
clones the Emacs repository. When you build it (by default) the Emacs
build process automatically clones the ELPA reposistory and takes the
files that it needs from there.

Doing this will mean that there will some Emacs-Lisp files present in a
built Emacs which are not actually source files in the sense of "the
thing you need to edit", but copies from else where. I need to think
about how to minimize the risk of this.


> There are sometimes reasons to continue building the same checkout --
> perhaps making some local edits -- and excluding all other changes
> that could confuse matters.

My build process simply runs git fetch when it thinks it needs to. Where
the fetch will come from (if anywhere) can be configured where the
developer wants. So, it's still possible to isolate yourself totally
from the main ELPA repo.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That's not the case here. The idea is this. You (the user) manually
  > clones the Emacs repository. When you build it (by default) the Emacs
  > build process automatically clones the ELPA reposistory and takes the
  > files that it needs from there.

That avoids the problem I thought I saw, but what is the purpose?  I
thought the idea of ELPA is that a user would get ELPA packages when perse
wants them -- they would not be preloaded.

If installing Emacs doesn't require a local copy of ELPA, building
Emacs should not need one.

Is this a way of handling a few packages that are included in Emacs
distros, if we maintain them in ELPA?  It makes a kind of sense for
that case, but I would expect that those testing and making Emacs
releases would need more control over which versions of packages are
included in the release.

  > My build process simply runs git fetch when it thinks it needs to. Where
  > the fetch will come from (if anywhere) can be configured where the
  > developer wants. So, it's still possible to isolate yourself totally
  > from the main ELPA repo.

It is one more thing that could trip people up, those who are not used
to it.  They would forget to do this and the build would surprise them.

In my case, I might fetch master from the Emacs repo, then build the
next day while having no network connection.  So it would fail.

For all these reasons, it would be better to copy these programs
into the Emacs repo, and just not edit them there -- only update
from the ELPA repo when appropriate.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

> The problem with branches is that what is on the branch changes. My
> feeling is that the build process should be repeatable; so building
> emacs-27.1-with-elpa.tgz should not result in different tar balls as
> ELPA changes.

By principle, GNU ELPA packages don't have to be "in-sync" with Emacs,
so this is not terribly important.

I think on Emacs `master` we won't want to use fixed SHAs but will just
use whatever's on `master` of elpa.git.  On the release branch, we'll
probably want to be more specific, using corresponding branches.

> Finally, neither solve the problem -- if a branch or tag is add
> EMACS/elpa/Makefile.in which exists in the ELPA repo, but not in the
> clone on the local machine the build will fail.

I think the use of branches/tags will make it sufficiently infrequent
that it's not a big deal.  Also the build shouldn't completely fail.


        Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

There has been a lot of discussion about this before. But it stalled
because we didn't have an implementation; this is at least something
that we can talk about concretely.

Richard Stallman <rms@gnu.org> writes:
>   > clones the Emacs repository. When you build it (by default) the Emacs
>   > build process automatically clones the ELPA reposistory and takes the
>   > files that it needs from there.
>
> That avoids the problem I thought I saw, but what is the purpose?  I
> thought the idea of ELPA is that a user would get ELPA packages when perse
> wants them -- they would not be preloaded.

Perhaps. An alternative, though, would be to reduce the size of Emacs as
a monolithic entity and have many more packages that are distributed via
ELPA. At the moment, the slow release cycle of Emacs means that many
packages are only updated on a two yearly basis.

It's also possible that the large number of packages in core Emacs is
also responsible for the slow release cycle.


> If installing Emacs doesn't require a local copy of ELPA, building
> Emacs should not need one.

Building Emacs requires lots of things that installing does not.

> Is this a way of handling a few packages that are included in Emacs
> distros, if we maintain them in ELPA?  It makes a kind of sense for
> that case, but I would expect that those testing and making Emacs
> releases would need more control over which versions of packages are
> included in the release.

Yes, although as I say, with the intention of enabling more packages to
be developed in this way. I'm not sure how many packages are currently
included in both repos (org and seq for sure).

This build system provides complete control over which versions are
included in the release; so I agree with you there.


>   > My build process simply runs git fetch when it thinks it needs to. Where
>   > the fetch will come from (if anywhere) can be configured where the
>   > developer wants. So, it's still possible to isolate yourself totally
>   > from the main ELPA repo.
>
> It is one more thing that could trip people up, those who are not used
> to it.  They would forget to do this and the build would surprise them.
>
> In my case, I might fetch master from the Emacs repo, then build the
> next day while having no network connection.  So it would fail.

Yes. Or, rather, there are circumstances under which it would
fail. Currently, it's a configure option, though. So you could still
build without.

It's also worth mentioning that you perhaps see this as more of a
problem because your life is unusual. I am rarely without a network
connection, even when travelling. I suspect this is the same for most.


> For all these reasons, it would be better to copy these programs
> into the Emacs repo, and just not edit them there -- only update
> from the ELPA repo when appropriate.

I think this is wrong. Having stuff which is not really source (i.e. it
is not the prefered location for editing) in a repo does not really make
sense. org-mode, for example, is currently in three repos (it's own,
ELPA and core). It's not something a route that you would really want to
go down; org-mode does it AFAICT only because of the advantages of being
in the default download of Emacs.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The problem with branches is that what is on the branch changes. My
>> feeling is that the build process should be repeatable; so building
>> emacs-27.1-with-elpa.tgz should not result in different tar balls as
>> ELPA changes.
>
> By principle, GNU ELPA packages don't have to be "in-sync" with Emacs,
> so this is not terribly important.

I'm unconvinced. I think a build should be repeatable. Imagine bisecting
a bug that has come up in an ELPA package when you are not sure whether
it's the package or Emacs core that has broken things.

Of course, this makes continuous integration harder. The solution there
is, I think, to run "make test" on the ELPA repository against the
current Emacs build. This already runs on head of all branches I think.

> I think on Emacs `master` we won't want to use fixed SHAs but will just
> use whatever's on `master` of elpa.git.  On the release branch, we'll
> probably want to be more specific, using corresponding branches.

That might mean multiple branches of master which would produce a very
cluttered namespace. The problem is that ELPA currently uses a different
(non git) mechanism to identify the current version of every package; so
you can't identify this from git metadata (except for SHA!).


>> Finally, neither solve the problem -- if a branch or tag is add
>> EMACS/elpa/Makefile.in which exists in the ELPA repo, but not in the
>> clone on the local machine the build will fail.
>
> I think the use of branches/tags will make it sufficiently infrequent
> that it's not a big deal.  Also the build shouldn't completely fail.

Beyond removing the configure option, it will fail at the moment. I
could do something else beyond that. But surely, by default, the build
should fail, if something fails?

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > That avoids the problem I thought I saw, but what is the purpose?  I
  > > thought the idea of ELPA is that a user would get ELPA packages when perse
  > > wants them -- they would not be preloaded.

  > Perhaps. An alternative, though, would be to reduce the size of Emacs as
  > a monolithic entity and have many more packages that are distributed via
  > ELPA. At the moment, the slow release cycle of Emacs means that many
  > packages are only updated on a two yearly basis.

I don't see how one is an alternative to the other.  They seem
to be talking about two different questions, both about ELPA
but not the same.

  > > Is this a way of handling a few packages that are included in Emacs
  > > distros, if we maintain them in ELPA?  It makes a kind of sense for
  > > that case, but I would expect that those testing and making Emacs
  > > releases would need more control over which versions of packages are
  > > included in the release.

  > Yes, although as I say, with the intention of enabling more packages to
  > be developed in this way. I'm not sure how many packages are currently
  > included in both repos (org and seq for sure).

How many there will be is a different question.  The question at hand
is how to handle merging them into Emacs and when.

I think that should be done by explicit command, not automatically or
spontaneously, and not as part of building Emacs.

  > > If installing Emacs doesn't require a local copy of ELPA, building
  > > Emacs should not need one.

  > Building Emacs requires lots of things that installing does not.

I think you have changed the subject,
but since what you said is very general, I can't be sure.

What sort of things do you mean?

  > I think this is wrong. Having stuff which is not really source (i.e. it
  > is not the prefered location for editing)

It is source code, no matter where it is located.  It is source code
even when it is a copy the user has downloaded.

Pastiching our definition of source code could be a good joke, but it
is not a serious argument for a decision about development methods.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

> I'm unconvinced. I think a build should be repeatable. Imagine bisecting
> a bug that has come up in an ELPA package when you are not sure whether
> it's the package or Emacs core that has broken things.

That's already a problem with Elisp packages (whether from GNU ELPA or
elsewhere), so the fact that we bundle some of them doesn't make much of
a difference in this respect.

Dependencies between them *should* be limited so that GNU ELPA packages
can be used with various Emacs versions (and vice-versa).

>> I think on Emacs `master` we won't want to use fixed SHAs but will just
>> use whatever's on `master` of elpa.git.  On the release branch, we'll
>> probably want to be more specific, using corresponding branches.
> That might mean multiple branches of master which would produce a very
> cluttered namespace. The problem is that ELPA currently uses a different
> (non git) mechanism to identify the current version of every package; so
> you can't identify this from git metadata (except for SHA!).

I don't know what you mean.  A branch/tag is just a name for an SHA, so
I can't see why a branch/tag wouldn't work where an SHA does.

In terms of namespace, we'd simply use an appropriate naming scheme,
of course.

>>> Finally, neither solve the problem -- if a branch or tag is add
>>> EMACS/elpa/Makefile.in which exists in the ELPA repo, but not in the
>>> clone on the local machine the build will fail.
>>
>> I think the use of branches/tags will make it sufficiently infrequent
>> that it's not a big deal.  Also the build shouldn't completely fail.
>
> Beyond removing the configure option, it will fail at the moment. I
> could do something else beyond that.  But surely, by default, the build
> should fail, if something fails?

I think currently I'd want bundled-GNU-ELPA packages to be treated as
"GNU ELPA packages pre-installed for the user's convenience" rather than
as "core packages that you can later upgrade via GNU ELPA".
So it's OK if they can't build: Emacs should still be perfectly usable
without its bundled-GNU-ELPA packages (which you can later install via
GNU ELPA anyway).

Maybe in the future, we'll want to allow some bundled-GNU-ELPA packages
to be more important (e.g. be *necessary* for example because they're
used by some core Elisp packages), but there's no need to cross that
bridge now.


        Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > That avoids the problem I thought I saw, but what is the purpose?  I
>   > > thought the idea of ELPA is that a user would get ELPA packages when perse
>   > > wants them -- they would not be preloaded.
>
>   > Perhaps. An alternative, though, would be to reduce the size of Emacs as
>   > a monolithic entity and have many more packages that are distributed via
>   > ELPA. At the moment, the slow release cycle of Emacs means that many
>   > packages are only updated on a two yearly basis.
>
> I don't see how one is an alternative to the other.  They seem
> to be talking about two different questions, both about ELPA
> but not the same.

Yes, well, that's the question at hand. Currently, ELPA is a place to
store packages that are not maintained in core. But it could also be
used to enable a much smaller core, with many more packages in ELPA.


>   > > Is this a way of handling a few packages that are included in Emacs
>   > > distros, if we maintain them in ELPA?  It makes a kind of sense for
>   > > that case, but I would expect that those testing and making Emacs
>   > > releases would need more control over which versions of packages are
>   > > included in the release.
>
>   > Yes, although as I say, with the intention of enabling more packages to
>   > be developed in this way. I'm not sure how many packages are currently
>   > included in both repos (org and seq for sure).
>
> How many there will be is a different question.  The question at hand
> is how to handle merging them into Emacs and when.
>
> I think that should be done by explicit command, not automatically or
> spontaneously, and not as part of building Emacs.

Indeed, that is the status quo. But it is problematic, because users
have to know about the package in ELPA to install it in the first
place. Org-mode is managed in both places for this reason: in core Emacs
because it comes "pre-installed" with all Emacs releases; in ELPA
because newer version of org-mode can be installed easily by users.

A smaller "core" would allow a more rapid release cycle. Many of the
packages could be maintained independently; missing a release cycle
would no longer mean users waiting a year or two for an update.


>   > > If installing Emacs doesn't require a local copy of ELPA, building
>   > > Emacs should not need one.
>
>   > Building Emacs requires lots of things that installing does not.
>
> I think you have changed the subject,
> but since what you said is very general, I can't be sure.
>
> What sort of things do you mean?

All the dev-tools -- compilers, headers, autoconf.


>   > I think this is wrong. Having stuff which is not really source (i.e. it
>   > is not the prefered location for editing)
>
> It is source code, no matter where it is located.  It is source code
> even when it is a copy the user has downloaded.
>
> Pastiching our definition of source code could be a good joke, but it
> is not a serious argument for a decision about development methods.

I am happy to use an alternative label if you have one. A flaw with this
alternative system is that, for example, the "jump to source" link in
*Help* will take you to a piece of lisp that you do not want to edit
because it may be over written.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I'm unconvinced. I think a build should be repeatable. Imagine bisecting
>> a bug that has come up in an ELPA package when you are not sure whether
>> it's the package or Emacs core that has broken things.
>
> That's already a problem with Elisp packages (whether from GNU ELPA or
> elsewhere), so the fact that we bundle some of them doesn't make much of
> a difference in this respect.

It is, but, now that ELPA package will be in the tarball download; I
think when someone says "bug X, affecting package Y, in Emacs-26.Z" that
should give you enough information to try and reproduce the error.

Without a repeatable build, you will also need to know what verison
package Y is, if and only if it is a "core ELPA" package.


> Dependencies between them *should* be limited so that GNU ELPA packages
> can be used with various Emacs versions (and vice-versa).

I think that this is a necessity at the moment. My own feeling is that
ELPA/package.el should support Emacs versions -- that it should be
possible to publish an ELPA package to a specific version of Emacs. That
is for the future however.


>>> I think on Emacs `master` we won't want to use fixed SHAs but will just
>>> use whatever's on `master` of elpa.git.  On the release branch, we'll
>>> probably want to be more specific, using corresponding branches.
>> That might mean multiple branches of master which would produce a very
>> cluttered namespace. The problem is that ELPA currently uses a different
>> (non git) mechanism to identify the current version of every package; so
>> you can't identify this from git metadata (except for SHA!).
>
> I don't know what you mean.  A branch/tag is just a name for an SHA, so
> I can't see why a branch/tag wouldn't work where an SHA does.

This is how a tag is implemented not what it is. A tag is a meaningful
name, living in a single namespace for an entire repository, which can
refer to different versions over time. A SHA key is not meaningful,
which while it lives in a single namespace has been written to attempt
to be world-unique, and cannot refer to different versions over time.


> In terms of namespace, we'd simply use an appropriate naming scheme,
> of course.

I have extensive experience with naming schemes and "simply" is rarely a
term I would apply; especially, in this case, where many packages
inhabit a single namespace. This rules out the obvious scheme of just
using the Emacs version number.

My naming scheme would be to use a stable, meaningless identifier.

Regardless, the code works either way, because as you say an tag is
implemented as an SHA.


>>>> Finally, neither solve the problem -- if a branch or tag is add
>>>> EMACS/elpa/Makefile.in which exists in the ELPA repo, but not in the
>>>> clone on the local machine the build will fail.
>>>
>>> I think the use of branches/tags will make it sufficiently infrequent
>>> that it's not a big deal.  Also the build shouldn't completely fail.
>>
>> Beyond removing the configure option, it will fail at the moment. I
>> could do something else beyond that.  But surely, by default, the build
>> should fail, if something fails?
>
> I think currently I'd want bundled-GNU-ELPA packages to be treated as
> "GNU ELPA packages pre-installed for the user's convenience" rather than
> as "core packages that you can later upgrade via GNU ELPA".
> So it's OK if they can't build: Emacs should still be perfectly usable
> without its bundled-GNU-ELPA packages (which you can later install via
> GNU ELPA anyway).

Currently, to achieve this on a clone of Emacs, you'd have to
reconfigure. With "./configure", my build would not install ELPA
packages and would (when I've written the code!) only produce a single
source tarball. With "./configure --enable-elpa", it would install ELPA
and produce two source tarballs, but would fail to build without ELPA.

Happy to put another mechanism in, but not sure what that would be.


> Maybe in the future, we'll want to allow some bundled-GNU-ELPA packages
> to be more important (e.g. be *necessary* for example because they're
> used by some core Elisp packages), but there's no need to cross that
> bridge now.

Ironically, the package that stimulated this discussion was "assess.el"
which is a testing package. From a user perspective, of course, it makes
no difference but for the developer it's exactly the sort of package
that you would want to make a dependency to many other
packages. Likewise, seq.el (which is why Nicolas wants to move it from
ELPA to core!).

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

>>> I'm unconvinced. I think a build should be repeatable. Imagine bisecting
>>> a bug that has come up in an ELPA package when you are not sure whether
>>> it's the package or Emacs core that has broken things.
>>
>> That's already a problem with Elisp packages (whether from GNU ELPA or
>> elsewhere), so the fact that we bundle some of them doesn't make much of
>> a difference in this respect.
>
> It is, but, now that ELPA package will be in the tarball download; I
> think when someone says "bug X, affecting package Y, in Emacs-26.Z" that
> should give you enough information to try and reproduce the error.

For the actual release, I'd expect the branch/tag to be very precise,
indeed (maybe even an SHA at that point).

> Without a repeatable build, you will also need to know what verison
> package Y is, if and only if it is a "core ELPA" package.

We could also consider such bug reports exactly like we do now: treat
the bundled GNU ELPA packages as if they weren't bundled, and ask the
user what is the version of the GNU ELPA package affected.

>>> That might mean multiple branches of master which would produce a very
>>> cluttered namespace. The problem is that ELPA currently uses a different
>>> (non git) mechanism to identify the current version of every package; so
>>> you can't identify this from git metadata (except for SHA!).
>> I don't know what you mean.  A branch/tag is just a name for an SHA, so
>> I can't see why a branch/tag wouldn't work where an SHA does.
> This is how a tag is implemented not what it is. A tag is a meaningful
> name, living in a single namespace for an entire repository, which can
> refer to different versions over time. A SHA key is not meaningful,
> which while it lives in a single namespace has been written to attempt
> to be world-unique, and cannot refer to different versions over time.

I understand this difference but I don't see how it relates to what you
said above.

> I have extensive experience with naming schemes and "simply" is rarely a
> term I would apply; especially, in this case, where many packages
> inhabit a single namespace. This rules out the obvious scheme of just
> using the Emacs version number.
> My naming scheme would be to use a stable, meaningless identifier.

I was thinking of something like "<scopingprefix>/<pkgname>/<emacsbranch>"

E.g. bundled/company/emacs-27

[ I had suggested that in addition to `emacs-27` we also create a branch
  `emacs-27.1` (created when we get into the final phase where we only
  allow commits that are explicitly allowed by the maintainer), so we
  could have bundled/company/emacs-27.1 as the branch pointing to the
  "final" version of company bundled with the Emacs-27.1 release.  ]

In order to reduce the number of such branches, we'd likely want some
fallback scheme where we have "<scopingprefix>/<emacsbranch>"
when "<scopingprefix>/<pkgname>/<emacsbranch>" doesn't exist, and
finally use just `master` when "<scopingprefix>/<emacsbranch>" doesn't
exist either.

> Regardless, the code works either way, because as you say an tag is
> implemented as an SHA.

Indeed.  All this discussion is really about not needing to pull from
the repository as part of a normal `make`, but moving this operation to
a separate invocation explicitly requesting it (call it `make
update-elpa` or something).

> Currently, to achieve this on a clone of Emacs, you'd have to
> reconfigure. With "./configure", my build would not install ELPA
> packages and would (when I've written the code!) only produce a single
> source tarball. With "./configure --enable-elpa", it would install ELPA
> and produce two source tarballs, but would fail to build without ELPA.
>
> Happy to put another mechanism in, but not sure what that would be.

Not sure what it should look like at the code level, but I think we
could live without "./configure --enable-elpa".  Instead, we'd have:

    git clone; ./configure; make

build Emacs normally but including warnings about GNU ELPA packages not
being found.

    git clone; ./configure; make update-elpa; make

build Emacs with the all the bundled GNU ELPA packages.  And subsequent

    git pull; ./configure; make

will build with those same bundled GNU ELPA packages, potentially
emitting some warnings about *some* GNU ELPA packages not being found.

>> Maybe in the future, we'll want to allow some bundled-GNU-ELPA packages
>> to be more important (e.g. be *necessary* for example because they're
>> used by some core Elisp packages), but there's no need to cross that
>> bridge now.
> Ironically, the package that stimulated this discussion was
> "assess.el" which is a testing package. From a user perspective, of
> course, it makes no difference but for the developer it's exactly the
> sort of package that you would want to make a dependency to many other
> packages. Likewise, seq.el (which is why Nicolas wants to move it from
> ELPA to core!).

I know: I don't mean to rule out having GNU ELPA packages that are
indispensable, but I think we should take it one step at a time,
otherwise the whole thing might just be rejected.

FWIW, seq.el *is* (and always has been, IIRC) in core.

As for assess.el, having it in GNU ELPA is OK as long as it is not
needed when *building* Emacs's core packages.  Basically, it should be
OK if we can do (like we do with ERT): `make lisp; make elpa; make
tests` (tho we'll probably have to work extra to try and make sure that
`make tests` nicely skips those tests that require assess.el when
assess.el is missing).


        Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, well, that's the question at hand. Currently, ELPA is a place to
  > store packages that are not maintained in core. But it could also be
  > used to enable a much smaller core, with many more packages in ELPA.

I think of this as a matter of details, quantitative rather than
qualitative.

  > > I think that should be done by explicit command, not automatically or
  > > spontaneously, and not as part of building Emacs.

  > Indeed, that is the status quo. But it is problematic, because users
  > have to know about the package in ELPA to install it in the first
  > place.

I think we are miscommunicating.  I'm talking about the procedure for
building the core Emacs release.  That is what was brought up before.

You seem to be talking now about how users use a given ELPA package.
These are fundamentally separate issues.

How to inform users about ELPA packages is also a separate issue.

  > A smaller "core" would allow a more rapid release cycle. Many of the
  > packages could be maintained independently; missing a release cycle
  > would no longer mean users waiting a year or two for an update.

Maybe you're right, but that's a different issue.  We can discuss that
later -- for now, let's focus on the question of building the Emacs
core.

  > > I think you have changed the subject,
  > > but since what you said is very general, I can't be sure.
  > >
  > > What sort of things do you mean?

  > All the dev-tools -- compilers, headers, autoconf.

To build a package requires having the build tools loaded.
That is natural.

What ELPA contains is not build tools for Emacs,
but add-ons for Emacs.  It is not natural that building
program A requires all its add-ons to be loaded.,


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Yes, well, that's the question at hand. Currently, ELPA is a place to
>   > store packages that are not maintained in core. But it could also be
>   > used to enable a much smaller core, with many more packages in ELPA.
>
> I think of this as a matter of details, quantitative rather than
> qualitative.
>
>   > > I think that should be done by explicit command, not automatically or
>   > > spontaneously, and not as part of building Emacs.
>
>   > Indeed, that is the status quo. But it is problematic, because users
>   > have to know about the package in ELPA to install it in the first
>   > place.
>
> I think we are miscommunicating.  I'm talking about the procedure for
> building the core Emacs release.  That is what was brought up before.
>
> You seem to be talking now about how users use a given ELPA package.
> These are fundamentally separate issues.
>
> How to inform users about ELPA packages is also a separate issue.
>
>   > A smaller "core" would allow a more rapid release cycle. Many of the
>   > packages could be maintained independently; missing a release cycle
>   > would no longer mean users waiting a year or two for an update.
>
> Maybe you're right, but that's a different issue.  We can discuss that
> later -- for now, let's focus on the question of building the Emacs
> core.
>
>   > > I think you have changed the subject,
>   > > but since what you said is very general, I can't be sure.
>   > >
>   > > What sort of things do you mean?
>
>   > All the dev-tools -- compilers, headers, autoconf.
>
> To build a package requires having the build tools loaded.
> That is natural.
>
> What ELPA contains is not build tools for Emacs,
> but add-ons for Emacs.  It is not natural that building
> program A requires all its add-ons to be loaded.,


Richard

I think we are going around in circles here, so perhaps I can spell out
what I am suggesting.

Emacs is currently what we might call the "real core" and a large number
of packages. Then there is another number of packages in ELPA. Some
packages are in both. It's not necessarily clear why packages are in one
of the other. Operationally, packages in core are available in the
default download, while packages in ELPA are not; packages in ELPA are
easy to update by users between Emacs releases, packages in core are
not.  The only way to square this circle is to put packages in both ELPA
and core.

The changes I am suggesting would (eventually when finished) produce two
tar balls "emacs" and "emacs-with-elpa". The latter would come with
additional ELPA packages; it would only be generated with a configure
option but, of course, to build in this way would require access to
those ELPA packages. To build only the former would not require access
to those ELPA packages.

The eventual aim would be an Emacs where many of the packages currently
in core, become routinely updatable between release cycles. I think this
would appeal to many developers who would like their code to get into
the world quicker; but would still prefer the visibility that being in
the default download.

I think this is a good idea, but others do not. I hope you can see why,
though, I am talking about release cycles, access to ELPA packages and
building Emacs all at the same time. They are intimately linked in this
case.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> That's already a problem with Elisp packages (whether from GNU ELPA or
>>> elsewhere), so the fact that we bundle some of them doesn't make much of
>>> a difference in this respect.
>>
>> It is, but, now that ELPA package will be in the tarball download; I
>> think when someone says "bug X, affecting package Y, in Emacs-26.Z" that
>> should give you enough information to try and reproduce the error.
>
> For the actual release, I'd expect the branch/tag to be very precise,
> indeed (maybe even an SHA at that point).

Yes, that would also work and would be close enough not to worry me.


>> Without a repeatable build, you will also need to know what verison
>> package Y is, if and only if it is a "core ELPA" package.
>
> We could also consider such bug reports exactly like we do now: treat
> the bundled GNU ELPA packages as if they weren't bundled, and ask the
> user what is the version of the GNU ELPA package affected.

Yes, could do, although it would put slightly more work on the
user. Having said that, (one of) the point behind this is to make
packages in the default download easier to update, so you'd probably
have to ask this anyway.


>>>> That might mean multiple branches of master which would produce a very
>>>> cluttered namespace. The problem is that ELPA currently uses a different
>>>> (non git) mechanism to identify the current version of every package; so
>>>> you can't identify this from git metadata (except for SHA!).
>>> I don't know what you mean.  A branch/tag is just a name for an SHA, so
>>> I can't see why a branch/tag wouldn't work where an SHA does.
>> This is how a tag is implemented not what it is. A tag is a meaningful
>> name, living in a single namespace for an entire repository, which can
>> refer to different versions over time. A SHA key is not meaningful,
>> which while it lives in a single namespace has been written to attempt
>> to be world-unique, and cannot refer to different versions over time.
>
> I understand this difference but I don't see how it relates to what you
> said above.

Namespaces can get messed up, people can use them wrong. With an SHA
key, you don't have to worry about getting the naming scheme right (or
indeed read the documentation on the naming scheme. You just use
something that you, by definition, have anyway.


>
>> I have extensive experience with naming schemes and "simply" is rarely a
>> term I would apply; especially, in this case, where many packages
>> inhabit a single namespace. This rules out the obvious scheme of just
>> using the Emacs version number.
>> My naming scheme would be to use a stable, meaningless identifier.
>
> I was thinking of something like "<scopingprefix>/<pkgname>/<emacsbranch>"
>
> E.g. bundled/company/emacs-27
>
> [ I had suggested that in addition to `emacs-27` we also create a branch
>   `emacs-27.1` (created when we get into the final phase where we only
>   allow commits that are explicitly allowed by the maintainer), so we
>   could have bundled/company/emacs-27.1 as the branch pointing to the
>   "final" version of company bundled with the Emacs-27.1 release.  ]

I think this all sounds entirely reasonable.


> In order to reduce the number of such branches, we'd likely want some
> fallback scheme where we have "<scopingprefix>/<emacsbranch>"
> when "<scopingprefix>/<pkgname>/<emacsbranch>" doesn't exist, and
> finally use just `master` when "<scopingprefix>/<emacsbranch>" doesn't
> exist either.

It would depend exactly how this was implemented I think. I would not
want this to always work -- I mean, if someone gets it wrong and types
"bundled/company/emacs27" (missing the dash) as their tag, having an
automatic fallback will mean this form of error is not picked up.



>> Regardless, the code works either way, because as you say an tag is
>> implemented as an SHA.
>
> Indeed.  All this discussion is really about not needing to pull from
> the repository as part of a normal `make`, but moving this operation to
> a separate invocation explicitly requesting it (call it `make
> update-elpa` or something).
>
>> Currently, to achieve this on a clone of Emacs, you'd have to
>> reconfigure. With "./configure", my build would not install ELPA
>> packages and would (when I've written the code!) only produce a single
>> source tarball. With "./configure --enable-elpa", it would install ELPA
>> and produce two source tarballs, but would fail to build without ELPA.
>>
>> Happy to put another mechanism in, but not sure what that would be.
>
> Not sure what it should look like at the code level, but I think we
> could live without "./configure --enable-elpa".  Instead, we'd have:
>
>     git clone; ./configure; make
>
> build Emacs normally but including warnings about GNU ELPA packages not
> being found.
>
>     git clone; ./configure; make update-elpa; make
>
> build Emacs with the all the bundled GNU ELPA packages.  And subsequent
>
>     git pull; ./configure; make
>
> will build with those same bundled GNU ELPA packages, potentially
> emitting some warnings about *some* GNU ELPA packages not being found

I really struggle with this, to be honest, and cannot see how it is
nicer. The key difference between a configure option is that you enable
it once and then it is automatic (and breaks on failure). With a make
target, you have to run it periodically at unpredictable times because
you don't know when it's needed. For me, warning messages are likely to
be lost in the haze of make -j 40.

The only good time it seems easier is stuck on a plane without
network. With your system, you just don't run "make update-elpa". With
mine, you have to run ./configure (forcing a recompile) at the start of
the flight and ./configure --enable-elpa at the end.




>>> Maybe in the future, we'll want to allow some bundled-GNU-ELPA packages
>>> to be more important (e.g. be *necessary* for example because they're
>>> used by some core Elisp packages), but there's no need to cross that
>>> bridge now.
>> Ironically, the package that stimulated this discussion was
>> "assess.el" which is a testing package. From a user perspective, of
>> course, it makes no difference but for the developer it's exactly the
>> sort of package that you would want to make a dependency to many other
>> packages. Likewise, seq.el (which is why Nicolas wants to move it from
>> ELPA to core!).
>
> I know: I don't mean to rule out having GNU ELPA packages that are
> indispensable, but I think we should take it one step at a time,
> otherwise the whole thing might just be rejected.

As always, your greater knowledge on these things is happily accepted.

>
> FWIW, seq.el *is* (and always has been, IIRC) in core.

Yes, this is correct. IIRC, though, for a while it was in the Emacs on
master, rather than in a released version, and in ELPA for the current
release.


> As for assess.el, having it in GNU ELPA is OK as long as it is not
> needed when *building* Emacs's core packages.  Basically, it should be
> OK if we can do (like we do with ERT): `make lisp; make elpa; make
> tests` (tho we'll probably have to work extra to try and make sure that
> `make tests` nicely skips those tests that require assess.el when
> assess.el is missing).

Possible, of course, but I'd have to build knowledge of assess into the
test apparatus to do that, I think. Probably easier to go the seq.el
route and install it into ELPA and core.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

> The only good time it seems easier is stuck on a plane without network.

Working on Emacs without network access is very common for me.
So it's an important use-case.

I don't really care if there's a configure flag to enable/disable the
GNU ELPA integration, but I'd want to enable it, and I'd also want my
builds to keep working without any extra hassle when I'm offline.

Also, as a general rule, for technical reasons as well as for privacy
reasons we consider it important not to initiate network connections
unless the user explicitly requested that network connection.
[ Same reason why Emacs doesn't offer to auto-update packages at
  startup.  ]


        Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The changes I am suggesting would (eventually when finished) produce two
  > tar balls "emacs" and "emacs-with-elpa". The latter would come with
  > additional ELPA packages; it would only be generated with a configure
  > option but, of course, to build in this way would require access to
  > those ELPA packages. To build only the former would not require access
  > to those ELPA packages.

That's not unreasonable.  I'm not arguing against that.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 11 Mar 2019 23:38:24 -0400
> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
>   > The changes I am suggesting would (eventually when finished) produce two
>   > tar balls "emacs" and "emacs-with-elpa". The latter would come with
>   > additional ELPA packages; it would only be generated with a configure
>   > option but, of course, to build in this way would require access to
>   > those ELPA packages. To build only the former would not require access
>   > to those ELPA packages.
> 
> That's not unreasonable.  I'm not arguing against that.

I will note that this is going where XEmacs was before, and I don't
think the result was encouraging, FWIW.

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Robert Pluim]

>>>>> On Tue, 12 Mar 2019 17:42:31 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Richard Stallman <rms@gnu.org> Date: Mon, 11 Mar 2019
    >> 23:38:24 -0400 Cc: monnier@IRO.UMontreal.CA,
    >> emacs-devel@gnu.org
    >> 
    >> > The changes I am suggesting would (eventually when finished)
    >> produce two > tar balls "emacs" and "emacs-with-elpa". The
    >> latter would come with > additional ELPA packages; it would
    >> only be generated with a configure > option but, of course, to
    >> build in this way would require access to > those ELPA
    >> packages. To build only the former would not require access >
    >> to those ELPA packages.
    >> 
    >> That's not unreasonable.  I'm not arguing against that.

    Eli> I will note that this is going where XEmacs was before, and I
    Eli> don't think the result was encouraging, FWIW.

Back when I still used XEmacs, downloading the 'sumo' tarball with all
the packages was the second step Iʼd perform after downloading the
sources, so in practice splitting them out didnʼt do much for me. OTOH
I donʼt think XEmacs ever had anything like elpa.

BTW, I may have missed part of the discussion, but why do we need two
tar balls? Could we not just have a 'download a bunch of elpa
packages' elisp function instead? [1] (Iʼll note that I essentially do
this manually for things like org-mode which are in core and elpa).

Robert

Footnotes: 
[1]  Iʼm spoiled: Iʼm rarely without an Internet connection

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: rms@gnu.org,  emacs-devel@gnu.org,  monnier@IRO.UMontreal.CA,  phillip.lord@russet.org.uk
> Date: Tue, 12 Mar 2019 16:52:31 +0100
> 
>     Eli> I will note that this is going where XEmacs was before, and I
>     Eli> don't think the result was encouraging, FWIW.
> 
> Back when I still used XEmacs, downloading the 'sumo' tarball with all
> the packages was the second step Iʼd perform after downloading the
> sources, so in practice splitting them out didnʼt do much for me.

That's what I meant: any serious user always wanted the sumo tarball.

> OTOH I donʼt think XEmacs ever had anything like elpa.

I'm not sure this matters.  Downloading files was possible then as
well, albeit more manually.  And XEmacs did have a package system.

> BTW, I may have missed part of the discussion, but why do we need two
> tar balls?

Because not everyone is on-line all the time.

XEmacs packages (was: Core ELPA was: Testing fontification, indentation, and buffer manipulation)

[Stefan Monnier]

> I will note that this is going where XEmacs was before, and I don't
> think the result was encouraging, FWIW.

My recollection of it is that they had some problems with it, especially
at the beginning, but I don't think they regretted it at all.

And our situation is quite different:

- They did a "big change" where they split the previously-combined
  XEmacs into XEmacs-core on one side and the sumo-meta-package on
  the other.  That was obviously painful because it was a big change,
  and also because they tried to make the XEmacs-core part as small
  as possible.
  In contrast, I haven't heard anyone suggest such a drastic change:
  only a gradual change by moving some packages from emacs.git to
  elpa.git bit by bit.

- I think XEmacs packages were more costly than ELPA packages in terms
  of maintenance.  2 reasons for that: one is that they were
  pre-compiled (which implies more work to package and more
  compatibility problems) and the other is that several of those
  XEmacs-packages were maintained by other people than the "upstream
  maintainer".


-- Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 12 Mar 2019 17:42:31 +0200, Eli Zaretskii <eliz@gnu.org> said:
>
>     Eli> I will note that this is going where XEmacs was before, and I
>     Eli> don't think the result was encouraging, FWIW.
>
> Back when I still used XEmacs, downloading the 'sumo' tarball with all
> the packages was the second step Iʼd perform after downloading the
> sources, so in practice splitting them out didnʼt do much for me. OTOH
> I donʼt think XEmacs ever had anything like elpa.
>
> BTW, I may have missed part of the discussion, but why do we need two
> tar balls? Could we not just have a 'download a bunch of elpa
> packages' elisp function instead? [1] (Iʼll note that I essentially do
> this manually for things like org-mode which are in core and elpa).
>
> [1]  Iʼm spoiled: Iʼm rarely without an Internet connection


Hard to see this as "spoiled" these days. Covers a large part of the
user base. The only thing that I don't like about auto downloading is
that the Emacs you get would depend on the day your first ran it.  Of
course, two Emacs installed on different days would end up as the same
thing after M-x package-update.

Imagine someone starting Emacs-25 at the end of a debian or ubuntu
release cycle; would all the packages on ELPA still work? So, I think
this would involve updating the package.el protocol to support Emacs
version numbers would be a necessity for this.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: rms@gnu.org, emacs-devel@gnu.org, monnier@IRO.UMontreal.CA,
>> phillip.lord@russet.org.uk
>> Date: Tue, 12 Mar 2019 16:52:31 +0100
>> 
>>     Eli> I will note that this is going where XEmacs was before, and I
>>     Eli> don't think the result was encouraging, FWIW.
>> 
>> Back when I still used XEmacs, downloading the 'sumo' tarball with all
>> the packages was the second step Iʼd perform after downloading the
>> sources, so in practice splitting them out didnʼt do much for me.
>
> That's what I meant: any serious user always wanted the sumo tarball.

Yes, I would guess. No problems there, change "emacs.tgz" and
"emacs-with-elpa.tgz" into "emacs-minimal.tgz" and "emacs.tgz".


>
>> OTOH I donʼt think XEmacs ever had anything like elpa.
>
> I'm not sure this matters.  Downloading files was possible then as
> well, albeit more manually.  And XEmacs did have a package system.
>
>> BTW, I may have missed part of the discussion, but why do we need two
>> tar balls?
>
> Because not everyone is on-line all the time.

They wouldn't need to be. Just some of the time.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> The only good time it seems easier is stuck on a plane without network.
>
> Working on Emacs without network access is very common for me.
> So it's an important use-case.
>
> I don't really care if there's a configure flag to enable/disable the
> GNU ELPA integration, but I'd want to enable it, and I'd also want my
> builds to keep working without any extra hassle when I'm offline.

Yes, I can see the use case; at the same token I'd much prefer the build
to break rather than provide warnings.

> Also, as a general rule, for technical reasons as well as for privacy
> reasons we consider it important not to initiate network connections
> unless the user explicitly requested that network connection.
> [ Same reason why Emacs doesn't offer to auto-update packages at
>   startup.  ]

Yes, indeed, I can see this.

What about this behaviour:

./configure    (as present -- no ELPA)
./configure --enable-elpa[=some-location]   (ELPA build support)

make elpa-update           (clone or update ELPA to some-location or default)
make                       (build Emacs with ELPA, break if not)
make DEMOTE_ELPA_ERROR=1   (build Emacs with ELPA, warning if not)


The last two could, of course, be swapped:

make                       (build Emacs with ELPA, warning if not)
make ELPA_ERROR=1          (build Emacs with ELPA, break if not)


A simple way to avoid having to run "make elpa-update" manually would be
to use git hooks so that any update of your Emacs clone, would update
your ELPA clone also.

Happy to do either, or what ever variation of the TUI you think makes
sense.

Phil

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > That's not unreasonable.  I'm not arguing against that.

  > I will note that this is going where XEmacs was before, and I don't
  > think the result was encouraging, FWIW.

I don't have an opinion, practically, on whether to do this.
I only said I don't think it is unreasonable.
-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Stefan Monnier]

>> Also, as a general rule, for technical reasons as well as for privacy
>> reasons we consider it important not to initiate network connections
>> unless the user explicitly requested that network connection.
>> [ Same reason why Emacs doesn't offer to auto-update packages at
>>   startup.  ]
>
> Yes, indeed, I can see this.
>
> What about this behaviour:
>
> ./configure    (as present -- no ELPA)
> ./configure --enable-elpa[=some-location]   (ELPA build support)
>
> make elpa-update           (clone or update ELPA to some-location or default)
> make                       (build Emacs with ELPA, break if not)
> make DEMOTE_ELPA_ERROR=1   (build Emacs with ELPA, warning if not)

Sounds OK (assuming `make` doesn't pull from elpa).


        Stefan

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Also, as a general rule, for technical reasons as well as for privacy
>>> reasons we consider it important not to initiate network connections
>>> unless the user explicitly requested that network connection.
>>> [ Same reason why Emacs doesn't offer to auto-update packages at
>>>   startup.  ]
>>
>> Yes, indeed, I can see this.
>>
>> What about this behaviour:
>>
>> ./configure    (as present -- no ELPA)
>> ./configure --enable-elpa[=some-location]   (ELPA build support)
>>
>> make elpa-update           (clone or update ELPA to some-location or default)
>> make                       (build Emacs with ELPA, break if not)
>> make DEMOTE_ELPA_ERROR=1   (build Emacs with ELPA, warning if not)
>
> Sounds OK (assuming `make` doesn't pull from elpa).

It wont need to.

I thought that this would be a pain, but actually there is a better way
to keep the two repo's in sync -- just configure the emacs git repo to
pull elpa everytime it is pulled.

Phil