Git version of ELPA

Git version of ELPA

[Stefan Monnier]

OK, I think it's high time we do this move.

Does someone have an up to date Git version of the current `elpa'
branch, ideally with most of the relevant externally available packages
merged in as "git subtree"s.


        Stefan

Re: Git version of ELPA

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Does someone have an up to date Git version of the current `elpa'
> branch, ideally with most of the relevant externally available packages
> merged in as "git subtree"s.

How about <git://repo.or.cz/emacs/elpa>?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Git version of ELPA

[Stefan Monnier]

>> Does someone have an up to date Git version of the current `elpa'
>> branch, ideally with most of the relevant externally available packages
>> merged in as "git subtree"s.
> How about <git://repo.or.cz/emacs/elpa>?

Does it have the externally available packages merged as "git subtree"s?


        Stefan

Re: Git version of ELPA

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Does someone have an up to date Git version of the current `elpa'
>>> branch, ideally with most of the relevant externally available packages
>>> merged in as "git subtree"s.
>> How about <git://repo.or.cz/emacs/elpa>?
>
> Does it have the externally available packages merged as "git subtree"s?

As if, yes.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Git version of ELPA

[Stefan Monnier]

>>>> Does someone have an up to date Git version of the current `elpa'
>>>> branch, ideally with most of the relevant externally available packages
>>>> merged in as "git subtree"s.
>>> How about <git://repo.or.cz/emacs/elpa>?
>> Does it have the externally available packages merged as "git subtree"s?
> As if, yes.

Hmm.. thanks, it does seem to be working fairly well in my
limited tests.

I've pushed it to the master branch of emacs/elpa.git.  Please don't
push to it yet.

Now, we need to update the scripts that update the elpa.gnu.org
website accordingly.


        Stefan

Re: Git version of ELPA

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 03 Aug 2013 00:28:27 -0400
> Cc: emacs-devel@gnu.org
> 
> >>>> Does someone have an up to date Git version of the current `elpa'
> >>>> branch, ideally with most of the relevant externally available packages
> >>>> merged in as "git subtree"s.
> >>> How about <git://repo.or.cz/emacs/elpa>?
> >> Does it have the externally available packages merged as "git subtree"s?
> > As if, yes.
> 
> Hmm.. thanks, it does seem to be working fairly well in my
> limited tests.
> 
> I've pushed it to the master branch of emacs/elpa.git.  Please don't
> push to it yet.
> 
> Now, we need to update the scripts that update the elpa.gnu.org
> website accordingly.

When all this is ready, please post some instructions for how to clone
the new repository with git, and anything else a git newbie should
know to pull from and push to that repo.  TIA.

Re: Git version of ELPA

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>> Does someone have an up to date Git version of the current `elpa'
>>>>> branch, ideally with most of the relevant externally available packages
>>>>> merged in as "git subtree"s.
>>>> How about <git://repo.or.cz/emacs/elpa>?
>>> Does it have the externally available packages merged as "git subtree"s?
>> As if, yes.
>
> Hmm.. thanks, it does seem to be working fairly well in my
> limited tests.

If having externally available packages merged as subtrees means that
the repo should include the revision histories from the upstream
repositories, it doesn't. At least, not for js2-mode and company.

Similarly, directories packages/{company,js2-mode} don't include the
additional files and directories present in upstream repos but not in
ELPA, containing tests, README.md and a few other supporting files.

I imagine that's going to complicate the merging procedure.

Re: Git version of ELPA

[Stefan Monnier]

> I imagine that's going to complicate the merging procedure.

After having tried the "subtree" merging a few times, I think it'll
be manageable (and is a problem which won't recur, so I'm not too
worried).


        Stefan

Re: Git version of ELPA

[Stefan Monnier]

>> I imagine that's going to complicate the merging procedure.
> After having tried the "subtree" merging a few times, I think it'll
> be manageable (and is a problem which won't recur, so I'm not too
> worried).

Of course, if there's a better Git tree available, I'll happily switch
to that one.


        Stefan

Re: Git version of ELPA

[Dmitry Gutov]

On 12.08.2013 04:10, Stefan Monnier wrote:
>> I imagine that's going to complicate the merging procedure.
>
> After having tried the "subtree" merging a few times, I think it'll
> be manageable (and is a problem which won't recur, so I'm not too
> worried).

So, you've tried merging commits from upstream that include files not 
present in ELPA? And splitting commits from ELPA to push them to 
upstream repositories? And both work fine?

 > Of course, if there's a better Git tree available, I'll happily switch
 > to that one.

A little while back Jorgen wrote a recipe how to do that: 
http://lists.gnu.org/archive/html/emacs-devel/2013-03/msg00235.html

I've yet to wrap my head around this git subtree business properly, but 
if it helps, I can try to follow it myself, and publish the result.

It raises a few questions, though. Up until now, the elpa branch has 
been receiving its contents in preprocessed form:
1) Some files and directories removed.
2) -pkg.el files pre-generated and included.

For example, just compare

http://repo.or.cz/w/emacs.git/tree/dbb5d60608ae6b99910b0c374f82b63fde526abf:/packages/yasnippet

and

https://github.com/capitaomorte/yasnippet/

My question is, will elpa publishing script work if packages/yasnippet 
contains (just) the files from the upstream repository? I imagine 
there'll have to be some adaptation.

Re: Git version of ELPA

[Stefan Monnier]

> So, you've tried merging commits from upstream that include files not
> present in ELPA? And splitting commits from ELPA to push them to upstream
> repositories? And both work fine?

Haven't tested any of them, since neither of them is supported by the
current Bazaar setup: it can't be worse than what we have in this respect.

>> Of course, if there's a better Git tree available, I'll happily switch
>> to that one.
> A little while back Jorgen wrote a recipe how to do that:
> http://lists.gnu.org/archive/html/emacs-devel/2013-03/msg00235.html
> I've yet to wrap my head around this git subtree business properly, but if
> it helps, I can try to follow it myself, and publish the result.

That would be nice.  I did not insist on it, because my tests showed
that you can "subtree merge" an external branch after-the-fact: with
Bazaar this resulted in serious problems, whereas with Git this is
handled sanely (not as well as having the external branch merged
right from the start, of course, but good enough).

> It raises a few questions, though. Up until now, the elpa branch has been
> receiving its contents in preprocessed form:
> 1) Some files and directories removed.
> 2) -pkg.el files pre-generated and included.

The `elpa' code will always tend to include local changes.  Less is
better, but they're a fact of life.

> For example, just compare
> http://repo.or.cz/w/emacs.git/tree/dbb5d60608ae6b99910b0c374f82b63fde526abf:/packages/yasnippet
> and
> https://github.com/capitaomorte/yasnippet/

yasnippet is a problem, indeed.  Someone needs to sync up the two trees
(and take responsibility to keep them in sync in the future).


        Stefan

Re: Git version of ELPA

[Dmitry Gutov]

On 12.08.2013 18:24, Stefan Monnier wrote:
>> So, you've tried merging commits from upstream that include files not
>> present in ELPA? And splitting commits from ELPA to push them to upstream
>> repositories? And both work fine?
>
> Haven't tested any of them, since neither of them is supported by the
> current Bazaar setup: it can't be worse than what we have in this respect.

I'm making this emphasis because git subtree has been mentioned several 
times, and its apparent advantages include being able to merge in both 
directions. If you don't have external repos' histories, apparently 
splitting commits and pushing them back won't work:

http://stackoverflow.com/questions/5760331/git-subtree-is-not-retaining-history-so-i-cannot-push-subtree-changes-how-can-i

>>> Of course, if there's a better Git tree available, I'll happily switch
>>> to that one.
>> A little while back Jorgen wrote a recipe how to do that:
>> http://lists.gnu.org/archive/html/emacs-devel/2013-03/msg00235.html
>> I've yet to wrap my head around this git subtree business properly, but if
>> it helps, I can try to follow it myself, and publish the result.
>
> That would be nice.  I did not insist on it, because my tests showed
> that you can "subtree merge" an external branch after-the-fact: with
> Bazaar this resulted in serious problems, whereas with Git this is
> handled sanely (not as well as having the external branch merged
> right from the start, of course, but good enough).

With Bazaar, I always just copied the most recent versions of files in a 
package, more or less manually, then marked them in `vc-dir' buffer and 
committed the result as the new revision. I don't think that doing only 
a little better than that is good enough.

>> It raises a few questions, though. Up until now, the elpa branch has been
>> receiving its contents in preprocessed form:
>> 1) Some files and directories removed.
>> 2) -pkg.el files pre-generated and included.
>
> The `elpa' code will always tend to include local changes.  Less is
> better, but they're a fact of life.

Inside package directories? How necessary is that? If you mean bugfixes 
touching several packages at once, they should be prime candidates for 
splitting and pushing upstream (I think git subtree supports that, not 
100% sure).

On the subject of extra files, more specifically:
1) Can we do without README and -pkg.el files?
2) Can we handle having README.md (and probably other formats), 
.travis.yml, Makefile, extra dirs and .el files not meant for end users? 
Like `doc', `extras', `yasnippet-debug.el' and `yasnippet-tests.el' in 
the yasnippet repository on GitHub.

Probably not, but Melpa does this okay. Aside from not having to pull 
from hundreds of external repositories, I think ELPA should work the 
same way (but also retain and use the Version header values).

If implementing it is going to be a lot of work, are we willing to take 
package-build.el from the Melpa project and adapt it, or reuse some of 
its pieces? If maybe, do we care about copyright assignments for its code?

>> For example, just compare
>> http://repo.or.cz/w/emacs.git/tree/dbb5d60608ae6b99910b0c374f82b63fde526abf:/packages/yasnippet
>> and
>> https://github.com/capitaomorte/yasnippet/
>
> yasnippet is a problem, indeed.  Someone needs to sync up the two trees
> (and take responsibility to keep them in sync in the future).

We can try to ping Joao, but that's a separate issue.

My intention here was to illustrate the difference between the file 
trees, not between their contents.

Re: Git version of ELPA

[Stefan Monnier]

> I'm making this emphasis because git subtree has been mentioned several
> times, and its apparent advantages include being able to merge in both
> directions. If you don't have external repos' histories, apparently
> splitting commits and pushing them back won't work:

I'm mostly concerned about keeping the `elpa' code up-to-date,
i.e. merging from external repositories into elpa, not the other
way around.
Git's subtrees handle this well.

To merge elpa changes back into the external repository, the best option
is to cherry-pick patches.  I don't think there's any VCS that can
handle this side of the sync semi-automatically, because I don't know of
any VCS that handles two-way syncs (except maybe for the special case where
both sides are meant to be exact mirrors, but even those can be
tricky).  So subtrees don't make any difference in this respect, AFAIK.

>> That would be nice.  I did not insist on it, because my tests showed
>> that you can "subtree merge" an external branch after-the-fact: with
>> Bazaar this resulted in serious problems, whereas with Git this is
>> handled sanely (not as well as having the external branch merged
>> right from the start, of course, but good enough).

> With Bazaar, I always just copied the most recent versions of files
> in a package, more or less manually, then marked them in `vc-dir' buffer and
> committed the result as the new revision. I don't think that doing
> only a little better than that is good enough.

With subtrees, I can do

   bzr merge --subtree <external-branch>

and it will work whatever has changed since the last merge of
that branch.  But that was already true to some extent with Bzr.
The difference is that this now works without "bzr join" and its bugs
(and also that most external branches are using Git, so we also avoid
the bugs of bzr-git).

I.e. instead of "bzr join" which can only add a new tree, I can do

   bzr merge --subtree <external-branch>

for a branch that I've never merged before: for lack of a common
ancestor it can't know what's new and what isn't, but it does
the sane thing: a 2-way "merge".
   
>> The `elpa' code will always tend to include local changes.  Less is
>> better, but they're a fact of life.
> Inside package directories?

Yes.

> How necessary is that?

Indispensible.

> If you mean bugfixes touching several packages at once,

Not only.  The difference can have to do with packaging, or with
non-copyright-assigned code, ...

In any case it's the responsibility of the package's maintainer to feed
elpa changes back to the external branch, if any.

> they should be prime candidates for splitting and pushing upstream (I
> think git subtree supports that, not 100% sure).

Sure.

> On the subject of extra files, more specifically:
> 1) Can we do without README and -pkg.el files?

The -pkg.el is the one that holds the metadata, so it's pretty
important.  We could get by without it if <pkg>/<pkg>.el contains the
corresponding info in the usual single-file format, but currently the
presence of <pkg>-pkg.el is used to indicate that this is a multifile
package rather than a singlefile package, so we'd need that
info elsewhere.

Mostly same story for README, except that it's a bit easier (the code
I have might already fallback to reading the Commentary section of
<pkg>/<pkg>.el if the README file is missing).

> 2) Can we handle having README.md (and probably other formats),

Yes.  Patches welcome (tho you might like to wait before the new Git
repository is in place, since the code is being rewritten as we speak).

> .travis.yml, Makefile, extra dirs and .el files not meant for end
> users? Like `doc', extras', `yasnippet-debug.el' and
> `yasnippet-tests.el' in the yasnippet repository on GitHub.

Currently, you can have extra (ignored) files only for singlefile
packages.  Multifile packages will package up whatever is present.
But it should be easy to add some way to list files that should be
skipped.  IOW, same as above "patch is welcome, tho you might like to
wait a bit".

> Probably not, but Melpa does this okay. Aside from not having to pull from
> hundreds of external repositories, I think ELPA should work the same way
> (but also retain and use the Version header values).

That's partly the way it's heading.

> If implementing it is going to be a lot of work, are we willing to take
> package-build.el from the Melpa project and adapt it, or reuse some of its
> pieces?

Hmm... I didn't think about that.  Kinda stupid, eh?
It's probably a bit late, tho (I have most of the old code adjusted
already).  In any case, for enhancements, it's probably a good idea to
look there.

> If maybe, do we care about copyright assignments for its code?

We would, of course.

> My intention here was to illustrate the difference between the file trees,
> not between their contents.

You'd have to ask Yidong, but I have the impression that yasnippet was
not installed in GNU ELPA in a straightforward way to start with
(usually I try to limit changes to copyright-fix up and things like
that); but sadly I can't remember details about that.


        Stefan

Re: Git version of ELPA

[Achim Gratz]

Dmitry Gutov writes:
> Probably not, but Melpa does this okay. Aside from not having to pull
> from hundreds of external repositories, I think ELPA should work the
> same way (but also retain and use the Version header values).
>
> If implementing it is going to be a lot of work, are we willing to
> take package-build.el from the Melpa project and adapt it, or reuse
> some of its pieces? If maybe, do we care about copyright assignments
> for its code?

Melpa in its current form doesn't work with packages needing a build
system or more generally auto-generated files other than *.elc.  Since
this screws over everyone using their VCS The Right Way(TM), I don't
think this is a particularly good model to follow.  ELPA needs to
distribute _releases_ of packages, which necessarily means to add or
even alter files from the development tree.  I'll read the whole thread
again, but it seems that the structure envisioned for ELPA so far has
not directly considered the consequences of that fact.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: Git version of ELPA

[Dmitry Gutov]

On 13.08.2013 04:42, Stefan Monnier wrote:
> To merge elpa changes back into the external repository, the best option
> is to cherry-pick patches.  I don't think there's any VCS that can
> handle this side of the sync semi-automatically, because I don't know of
> any VCS that handles two-way syncs (except maybe for the special case where
> both sides are meant to be exact mirrors, but even those can be
> tricky).  So subtrees don't make any difference in this respect, AFAIK.

 From your initial message in this thread, I thought we were talking 
about the "git subtree" command, which is a plugin that was merged into 
core not so long ago.

I've played around with it a bit, and it does seem to help with syncing 
both ways, even when commits in the parent repo touch several subtrees.

To pull:

git subtree pull --prefix packages/js2-mode git@github.com:mooz/js2-mode.git

To split commits and push them back:

git subtree push --prefix packages/js2-mode 
git@github.com:mooz/js2-mode.git

The second command, from what I can tell, does require that the subtree 
had been properly imported from the remote repository.

Notes:
1) I tried this on toy repos, not real ones. I can send them for you to 
examine, if you like.
2) If packages/js2-mode and git@github.com:mooz/js2-mode.git differ in 
files they contain, I imagine we'll have more errors or conflicts to 
deal with.

>>> That would be nice.  I did not insist on it, because my tests showed
>>> that you can "subtree merge" an external branch after-the-fact: with
>>> Bazaar this resulted in serious problems, whereas with Git this is
>>> handled sanely (not as well as having the external branch merged
>>> right from the start, of course, but good enough).

AFAIK, "git subtree" uses and builds on the subtree merge strategy.

> With subtrees, I can do
>
>     bzr merge --subtree <external-branch>
>
> and it will work whatever has changed since the last merge of
> that branch.  But that was already true to some extent with Bzr.
> The difference is that this now works without "bzr join" and its bugs
> (and also that most external branches are using Git, so we also avoid
> the bugs of bzr-git).
>
> I.e. instead of "bzr join" which can only add a new tree, I can do
>
>     bzr merge --subtree <external-branch>
>
> for a branch that I've never merged before: for lack of a common
> ancestor it can't know what's new and what isn't, but it does
> the sane thing: a 2-way "merge".

I'm assuming you meant "git merge" in all of the above.

>>> The `elpa' code will always tend to include local changes.  Less is
>>> better, but they're a fact of life.
>> Inside package directories?
>
> Yes.
>
>> How necessary is that?
>
> Indispensible.
>
>> If you mean bugfixes touching several packages at once,
>
> Not only.  The difference can have to do with packaging, or with
> non-copyright-assigned code, ...

I imagine that projects that have this issue (Org, for one... any other 
examples?) already maintain a separate branch in the upstream repository 
(called 'gnu', for example), and this branch is copyright-clean already, 
so our syncing workflow shouldn't be concerned about that.

Projects that don't, can adopt this practice going forward.

Syncing between the master and gnu branches in the upstream repo should 
be a) easier, b) usually none of our business (but we'd still be able to 
make some fixes, if they are meant to be pushed upstream).

> In any case it's the responsibility of the package's maintainer to feed
> elpa changes back to the external branch, if any.

Maybe so. But 'git subtree' seems to make this less painful, as long as 
ELPA doesn't go deliberately out of sync.

> The -pkg.el is the one that holds the metadata, so it's pretty
> important.  We could get by without it if <pkg>/<pkg>.el contains the
> corresponding info in the usual single-file format, but currently the
> presence of <pkg>-pkg.el is used to indicate that this is a multifile
> package rather than a singlefile package, so we'd need that
> info elsewhere.

That's what I meant, yes. We'll need a mechanism to include/exclude 
files anyway.

> Mostly same story for README, except that it's a bit easier (the code
> I have might already fallback to reading the Commentary section of
> <pkg>/<pkg>.el if the README file is missing).

It does, e.g. for the websocket package.

>> 2) Can we handle having README.md (and probably other formats),
>
> Yes.  Patches welcome (tho you might like to wait before the new Git
> repository is in place, since the code is being rewritten as we speak).

If we're okay with showing non-preprocessed Markdown, Org and similar 
files as plain text, that should be easy. Markdown reads fine that way, 
at least.

>> .travis.yml, Makefile, extra dirs and .el files not meant for end
>> users? Like `doc', extras', `yasnippet-debug.el' and
>> `yasnippet-tests.el' in the yasnippet repository on GitHub.
>
> Currently, you can have extra (ignored) files only for singlefile
> packages.  Multifile packages will package up whatever is present.
> But it should be easy to add some way to list files that should be
> skipped.  IOW, same as above "patch is welcome, tho you might like to
> wait a bit".

I'd welcome a suggestion for the exact mechanism. List them in a file 
called '.elpa-includes'?

>> If implementing it is going to be a lot of work, are we willing to take
>> package-build.el from the Melpa project and adapt it, or reuse some of its
>> pieces?
>
> Hmm... I didn't think about that.  Kinda stupid, eh?
> It's probably a bit late, tho (I have most of the old code adjusted
> already).  In any case, for enhancements, it's probably a good idea to
> look there.
>
>> If maybe, do we care about copyright assignments for its code?
>
> We would, of course.

I wasn't sure, because the contents of admin/archive-contents.el just 
live on the server, and as such are different from the packages and the 
Emacs sources.

> You'd have to ask Yidong, but I have the impression that yasnippet was
> not installed in GNU ELPA in a straightforward way to start with
> (usually I try to limit changes to copyright-fix up and things like
> that); but sadly I can't remember details about that.

We'll need to deal with it eventually, but this version is not actually 
too old. Yanippet development has been relatively slow the last few months.

Re: Git version of ELPA

[Dmitry Gutov]

Achim Gratz <Stromeko@nexgo.de> writes:

> Melpa in its current form doesn't work with packages needing a build
> system or more generally auto-generated files other than *.elc.  Since
> this screws over everyone using their VCS The Right Way(TM), I don't
> think this is a particularly good model to follow.  ELPA needs to
> distribute _releases_ of packages, which necessarily means to add or
> even alter files from the development tree.

If using VCS The Right Way means not having auto-generated files inside
the upstreadm repositories, then they shouldn't be inside the elpa
branch, either.

The packages' files get modified/added during the build process anyway,
both in ELPA and MELPA.

IOW, we're probably in agreement, but I think ELPA is farther from The
Right Way that Melpa, since the former requires -pkg.el files to be
present, and they are usually auto-generated.

Re: Git version of ELPA

[Stefan Monnier]

> 2) If packages/js2-mode and git@github.com:mooz/js2-mode.git differ in files
> they contain, I imagine we'll have more errors or conflicts to deal with.

Yes and no: the "push" would simply force the github version to have
the same content as the elpa version.  No conflict, but an undesirable
outcome, so you end up having to do it somewhat by hand (you can use
"git subtree split" to help, but there's still a manual intervention
needed).

>> In any case it's the responsibility of the package's maintainer to feed
>> elpa changes back to the external branch, if any.
> Maybe so. But 'git subtree' seems to make this less painful, as long as ELPA
> doesn't go deliberately out of sync.

Yes.  The elpa-diffs email as well (a copy is sent to the maintainer).

>>> 2) Can we handle having README.md (and probably other formats),
>> Yes.  Patches welcome (tho you might like to wait before the new Git
>> repository is in place, since the code is being rewritten as we speak).
> If we're okay with showing non-preprocessed Markdown, Org and similar files
> as plain text, that should be easy. Markdown reads fine that way, at least.

Hopefully we can do better than that, but as a first step, yes, that's
definitely fine.

>> Currently, you can have extra (ignored) files only for singlefile
>> packages.  Multifile packages will package up whatever is present.
>> But it should be easy to add some way to list files that should be
>> skipped.  IOW, same as above "patch is welcome, tho you might like to
>> wait a bit".
> I'd welcome a suggestion for the exact mechanism.

A simple solution is to not remove those files from the `elpa' branch.
I.e. consider it as a "local change".  It might lead to spurious
conflicts when merging, tho.

> List them in a file called .elpa-includes'?

I'd rather have a list of exclusions than a list of inclusions, but
other than that I guess that'd be right.  So we could easily handle
a list of exclusions by passing the list to "tar".


        Stefan

Re: Git version of ELPA

[Dmitry Gutov]

On 14.08.2013 18:30, Stefan Monnier wrote:
>> 2) If packages/js2-mode and git@github.com:mooz/js2-mode.git differ in files
>> they contain, I imagine we'll have more errors or conflicts to deal with.
>
> Yes and no: the "push" would simply force the github version to have
> the same content as the elpa version.

The push just after the removal commit would end up fine. But if you 
don't do it, later you may end up having to fight the divergence each 
time you go back and forth.

> No conflict, but an undesirable
> outcome, so you end up having to do it somewhat by hand (you can use
> "git subtree split" to help, but there's still a manual intervention
> needed).

I can't confidently say it's always so (in certain good conditions), but 
I've seen it succeed without manual intervention.

And "git subtree push" also seems to call "git subtree split" when required.

>>> In any case it's the responsibility of the package's maintainer to feed
>>> elpa changes back to the external branch, if any.
>> Maybe so. But 'git subtree' seems to make this less painful, as long as ELPA
>> doesn't go deliberately out of sync.
>
> Yes.  The elpa-diffs email as well (a copy is sent to the maintainer).

Never seen that working before. Is that a recent change?

>>> Currently, you can have extra (ignored) files only for singlefile
>>> packages.  Multifile packages will package up whatever is present.
>>> But it should be easy to add some way to list files that should be
>>> skipped.  IOW, same as above "patch is welcome, tho you might like to
>>> wait a bit".
>> I'd welcome a suggestion for the exact mechanism.
>
> A simple solution is to not remove those files from the `elpa' branch.
> I.e. consider it as a "local change".  It might lead to spurious
> conflicts when merging, tho.

Not sure I understand. I didn't suggest removing them.
What changes, and "local" to what?

If you mean the file listing the ignores, I'm sure it'll be fine to keep 
it in the upstream repo, too. Maybe Melpa even ends up using them, too.

>> List them in a file called .elpa-includes'?
>
> I'd rather have a list of exclusions than a list of inclusions, but
> other than that I guess that'd be right.  So we could easily handle
> a list of exclusions by passing the list to "tar".

Exclusions are fine by me, too. So, file name ".elpaignore", syntax 
similar to ".gitignore" (one glob per line)?

Re: Git version of ELPA

[Achim Gratz]

Dmitry Gutov writes:
> If using VCS The Right Way means not having auto-generated files inside
> the upstreadm repositories, then they shouldn't be inside the elpa
> branch, either.

Good, so then how does ELPA build the release versions from the raw
repository content?

> The packages' files get modified/added during the build process anyway,
> both in ELPA and MELPA.

True, but AFAIK that only means adding a *-pkg.el file and selecting
which files end up in the package archive.  They both don't build a
distribution version, but expect to be able to use the sources directly.

> IOW, we're probably in agreement, but I think ELPA is farther from The
> Right Way that Melpa, since the former requires -pkg.el files to be
> present, and they are usually auto-generated.

At least for Org, there are provisions to build the necessary version
and autoload files via elisp, this could be extended to the *-pkg.el
file.  Another option would be to use the existing makefile that creates
a valid package archive.  But the ELPA packager would need to make use
of those functions and the mechanism to do so should be general enough
to be usable by all packages that need it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Git version of ELPA

[Stefan Monnier]

>>> 2) If packages/js2-mode and git@github.com:mooz/js2-mode.git differ
>>> in files they contain, I imagine we'll have more errors or conflicts
>>> to deal with.
>> Yes and no: the "push" would simply force the github version to have
>> the same content as the elpa version.
> The push just after the removal commit would end up fine.

No, by definition "push" gives you an exact copy at the other end.

>> Yes.  The elpa-diffs email as well (a copy is sent to the maintainer).
> Never seen that working before. Is that a recent change?

Fairly recent, yes.  Of course, now it's broken again by the change to Git.

>>>> Currently, you can have extra (ignored) files only for singlefile
>>>> packages.  Multifile packages will package up whatever is present.
>>>> But it should be easy to add some way to list files that should be
>>>> skipped.  IOW, same as above "patch is welcome, tho you might like to
>>>> wait a bit".
>>> I'd welcome a suggestion for the exact mechanism.
>> A simple solution is to not remove those files from the `elpa' branch.
>> I.e. consider it as a "local change".  It might lead to spurious
>> conflicts when merging, tho.
> Not sure I understand. I didn't suggest removing them.
> What changes, and "local" to what?

Changes, as in "file removal", "file renaming", ...
Local to the `elpa' version.

> If you mean the file listing the ignores, I'm sure it'll be fine to keep it
> in the upstream repo, too. Maybe Melpa even ends up using them, too.

No, I mean not have any listing at all.  Just remove from the `elpa'
branch the files you don't want to see in the packages.

> Exclusions are fine by me, too. So, file name ".elpaignore", syntax similar
> to ".gitignore" (one glob per line)?

Syntax: whatever "tar" accepts.


        Stefan

Re: Git version of ELPA

[Dmitry Gutov]

On 14.08.2013 22:02, Stefan Monnier wrote:
>>>> 2) If packages/js2-mode and git@github.com:mooz/js2-mode.git differ
>>>> in files they contain, I imagine we'll have more errors or conflicts
>>>> to deal with.
>>> Yes and no: the "push" would simply force the github version to have
>>> the same content as the elpa version.
>> The push just after the removal commit would end up fine.
>
> No, by definition "push" gives you an exact copy at the other end.

Yes, sorry. You're right, of course.

As long as you've resolved all the conflicts during the previous merge, 
and the remote head is a proper ancestor of the local head, the push 
will succeed.
Which, in the described situation, is arguably worse than failure.

>>> A simple solution is to not remove those files from the `elpa' branch.
>>> I.e. consider it as a "local change".  It might lead to spurious
>>> conflicts when merging, tho.
>> Not sure I understand. I didn't suggest removing them.
>> What changes, and "local" to what?
>
> Changes, as in "file removal", "file renaming", ...
> Local to the `elpa' version.

Ah, ok. I guess the "not" in "to not remove" above was a typo.

>> Exclusions are fine by me, too. So, file name ".elpaignore", syntax similar
>> to ".gitignore" (one glob per line)?
>
> Syntax: whatever "tar" accepts.

Sounds good.

Speaking of README file formats, maybe your current solution is good 
enough. The Melpa guys have intentionally settled on the same approach 
(use the Commentary from <package-name>.el):

https://github.com/milkypostman/melpa/issues/522
https://github.com/milkypostman/melpa/pull/366

The idea is that README.md/org/etc in the root of the directory serves 
as an introduction, and it usually contains a section "how to install".

The package description buffer, on the other hand, would be most useful 
if it has a short description of what the package is about and how to 
use it *once it's installed*.

I'm not sure if I fully agree with that reasoning, but doing readme 
generation the same way across package repositories would be good.

Re: Git version of ELPA

[Stephen J. Turnbull]

Stefan Monnier writes:

 > > The push just after the removal commit would end up fine.
 > 
 > No, by definition "push" gives you an exact copy at the other end.

By definition in bzr, but not in git.  It may be that in the default
configuration of recent git-push is restricted to fast-forward, thus
producing an identical branch, though.

Re: Git version of ELPA

[Achim Gratz]

Stephen J. Turnbull writes:
> Stefan Monnier writes:
>
>  > > The push just after the removal commit would end up fine.
>  > 
>  > No, by definition "push" gives you an exact copy at the other end.
>
> By definition in bzr, but not in git.  It may be that in the default
> configuration of recent git-push is restricted to fast-forward, thus
> producing an identical branch, though.

Git can be either-or: by default it aborts if the push is not
fast-forward.  You can force it to drop the non-merged commits on the
remote side and follow through with the push, though.  If you are sure
that you always want an _exact_ copy of the whole repository at the
remote end, you can declare that remote a mirror.  You can either effect
this ad-hoc from the commandline or configure git to always behave that
way, as you prefer.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Git version of ELPA

[Dmitry Gutov]

On 14.08.2013 23:46, Dmitry Gutov wrote:
> Speaking of README file formats, maybe your current solution is good
> enough. The Melpa guys have intentionally settled on the same approach
> (use the Commentary from <package-name>.el):
>
> https://github.com/milkypostman/melpa/issues/522
> https://github.com/milkypostman/melpa/pull/366
>
> The idea is that README.md/org/etc in the root of the directory serves
> as an introduction, and it usually contains a section "how to install".
>
> The package description buffer, on the other hand, would be most useful
> if it has a short description of what the package is about and how to
> use it *once it's installed*.

Cases in point:

http://elpa.gnu.org/packages/ggtags.html
http://elpa.gnu.org/packages/ack.html
http://elpa.gnu.org/packages/js2-mode.html

link to http://elpa.gnu.org/ and mention M-x list-packages, both of 
which are somewhat extraneous.

http://elpa.gnu.org/packages/coffee-mode.html

recommends cloning from GitHub and installing manually.

http://elpa.gnu.org/packages/company.html

avoids this problem largely accidentally, by having a homepage separate 
from the GitHub repo (which is relatively unusual).

Insidentally, this change (de728884) also broke the link from js2-mode's 
README.md to http://elpa.gnu.org/packages/js2-mode.html, because the 
surrounding text depends on the latter page having the description from 
Commentary. But no matter, that happens.

Re: Git version of ELPA

[Dmitry Gutov]

On 14.08.2013 18:30, Stefan Monnier wrote:
> So we could easily handle
> a list of exclusions by passing the list to "tar".

How does this look to you?

Downsides, so far:

1) Incessant "tar: Removing leading `../' from member names" warnings.
2) .elpaignore file itself is included. Maybe even not a downside.

diff --git a/GNUmakefile b/GNUmakefile
index cc06e48..4a9f185 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -48,8 +48,11 @@ process-archive:
  	@cd $(ARCHIVE_TMP)/packages; \
  	  for pt in *; do \
  	      if [ -d $$pt ]; then \
+		  cd $$pt; \
  		  echo "Creating tarball $${pt}.tar" && \
-		  tar -cf $${pt}.tar $$pt --remove-files; \
+		  tar -cf ../$${pt}.tar ../$$pt \
+		  $$(if [ -f .elpaignore ]; then echo "-X .elpaignore"; fi;); \
+		  cd ..; rm -r $${pt}; \
  	      fi; \
  	  done
  	mkdir -p archive/packages