Re: Git version of ELPA

[Dmitry Gutov <dgutov@yandex.ru>]

On 13.08.2013 04:42, Stefan Monnier wrote:
> To merge elpa changes back into the external repository, the best option
> is to cherry-pick patches.  I don't think there's any VCS that can
> handle this side of the sync semi-automatically, because I don't know of
> any VCS that handles two-way syncs (except maybe for the special case where
> both sides are meant to be exact mirrors, but even those can be
> tricky).  So subtrees don't make any difference in this respect, AFAIK.

 From your initial message in this thread, I thought we were talking 
about the "git subtree" command, which is a plugin that was merged into 
core not so long ago.

I've played around with it a bit, and it does seem to help with syncing 
both ways, even when commits in the parent repo touch several subtrees.

To pull:

git subtree pull --prefix packages/js2-mode git@github.com:mooz/js2-mode.git

To split commits and push them back:

git subtree push --prefix packages/js2-mode 
git@github.com:mooz/js2-mode.git

The second command, from what I can tell, does require that the subtree 
had been properly imported from the remote repository.

Notes:
1) I tried this on toy repos, not real ones. I can send them for you to 
examine, if you like.
2) If packages/js2-mode and git@github.com:mooz/js2-mode.git differ in 
files they contain, I imagine we'll have more errors or conflicts to 
deal with.

>>> That would be nice.  I did not insist on it, because my tests showed
>>> that you can "subtree merge" an external branch after-the-fact: with
>>> Bazaar this resulted in serious problems, whereas with Git this is
>>> handled sanely (not as well as having the external branch merged
>>> right from the start, of course, but good enough).

AFAIK, "git subtree" uses and builds on the subtree merge strategy.

> With subtrees, I can do
>
>     bzr merge --subtree <external-branch>
>
> and it will work whatever has changed since the last merge of
> that branch.  But that was already true to some extent with Bzr.
> The difference is that this now works without "bzr join" and its bugs
> (and also that most external branches are using Git, so we also avoid
> the bugs of bzr-git).
>
> I.e. instead of "bzr join" which can only add a new tree, I can do
>
>     bzr merge --subtree <external-branch>
>
> for a branch that I've never merged before: for lack of a common
> ancestor it can't know what's new and what isn't, but it does
> the sane thing: a 2-way "merge".

I'm assuming you meant "git merge" in all of the above.

>>> The `elpa' code will always tend to include local changes.  Less is
>>> better, but they're a fact of life.
>> Inside package directories?
>
> Yes.
>
>> How necessary is that?
>
> Indispensible.
>
>> If you mean bugfixes touching several packages at once,
>
> Not only.  The difference can have to do with packaging, or with
> non-copyright-assigned code, ...

I imagine that projects that have this issue (Org, for one... any other 
examples?) already maintain a separate branch in the upstream repository 
(called 'gnu', for example), and this branch is copyright-clean already, 
so our syncing workflow shouldn't be concerned about that.

Projects that don't, can adopt this practice going forward.

Syncing between the master and gnu branches in the upstream repo should 
be a) easier, b) usually none of our business (but we'd still be able to 
make some fixes, if they are meant to be pushed upstream).

> In any case it's the responsibility of the package's maintainer to feed
> elpa changes back to the external branch, if any.

Maybe so. But 'git subtree' seems to make this less painful, as long as 
ELPA doesn't go deliberately out of sync.

> The -pkg.el is the one that holds the metadata, so it's pretty
> important.  We could get by without it if <pkg>/<pkg>.el contains the
> corresponding info in the usual single-file format, but currently the
> presence of <pkg>-pkg.el is used to indicate that this is a multifile
> package rather than a singlefile package, so we'd need that
> info elsewhere.

That's what I meant, yes. We'll need a mechanism to include/exclude 
files anyway.

> Mostly same story for README, except that it's a bit easier (the code
> I have might already fallback to reading the Commentary section of
> <pkg>/<pkg>.el if the README file is missing).

It does, e.g. for the websocket package.

>> 2) Can we handle having README.md (and probably other formats),
>
> Yes.  Patches welcome (tho you might like to wait before the new Git
> repository is in place, since the code is being rewritten as we speak).

If we're okay with showing non-preprocessed Markdown, Org and similar 
files as plain text, that should be easy. Markdown reads fine that way, 
at least.

>> .travis.yml, Makefile, extra dirs and .el files not meant for end
>> users? Like `doc', extras', `yasnippet-debug.el' and
>> `yasnippet-tests.el' in the yasnippet repository on GitHub.
>
> Currently, you can have extra (ignored) files only for singlefile
> packages.  Multifile packages will package up whatever is present.
> But it should be easy to add some way to list files that should be
> skipped.  IOW, same as above "patch is welcome, tho you might like to
> wait a bit".

I'd welcome a suggestion for the exact mechanism. List them in a file 
called '.elpa-includes'?

>> If implementing it is going to be a lot of work, are we willing to take
>> package-build.el from the Melpa project and adapt it, or reuse some of its
>> pieces?
>
> Hmm... I didn't think about that.  Kinda stupid, eh?
> It's probably a bit late, tho (I have most of the old code adjusted
> already).  In any case, for enhancements, it's probably a good idea to
> look there.
>
>> If maybe, do we care about copyright assignments for its code?
>
> We would, of course.

I wasn't sure, because the contents of admin/archive-contents.el just 
live on the server, and as such are different from the packages and the 
Emacs sources.

> You'd have to ask Yidong, but I have the impression that yasnippet was
> not installed in GNU ELPA in a straightforward way to start with
> (usually I try to limit changes to copyright-fix up and things like
> that); but sadly I can't remember details about that.

We'll need to deal with it eventually, but this version is not actually 
too old. Yanippet development has been relatively slow the last few months.