Keeping changes in sync with upstream projects

Keeping changes in sync with upstream projects

[Lluís]

This is something that I've wanted to ask since quite a long time, and now that
CEDET 1.0 seems to be closer, it is time to think of a solution.

So, the problem:

Emacs ships with code from other relatively large projects that have their own
upstream repositories.

Synchronizing emacs upstream to/from those project's upstream should be as
automated as possible (i.e., it is not desirable to diff emacs' version with
upstream's and integrate changes manually).


High-level requirement:

Sync back and forth emacs' upstream with project's on a per-project basis.


Specific requirements:

         * Sync an emacs subtree with project's upstream (assuming such projects
           have a subtree of their own on emacs upstream)
         * Keep track (on one or both sides of) of "remote" changesets:
                * merged
                * discarded
                * temporally hold (cherrypicking)


I think this would cover most aspects of cross-project syncing, and after
talking with Eric (some time ago), there is no problem for CEDET to switch from
its current CVS into another VCS.

My question is, which VCS should be used and how in order to facilitate merges
between emacs and this kind of projects (CEDET, in this case)? Because, frankly,
I had a real bad time manually merging changes from emacs trunk into CEDET :)


If I remember well, bazaar has no support for subtree merging nor changeset
cherrypicking, although I might well be completely wrong.


Any ideas on how to ease it?

I can think of tagging "local" changesets with references to "remote"
changesets, but this would need some specialized tools.

For example, if Emacs were in subversion, and the project to sync was in its own
subtree, I could add three properties to the root of the subtree (svn propedit):
         * remote repository URI
         * merged changesets
         * discarded changesets
         * hold changesets

Then a script can be written to check all changesets on the remote repository
that are not listed on any of the other three properties, meaning that they are
pending for merge.


Thanks,
        Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Keeping changes in sync with upstream projects

[Stefan Monnier]

These are very good questions, thanks for asking.
The Gnus guys have their own answer to that (not sure how advanced it
is by now), so maybe they could say how well it works for them.

The 2-way sync is the harder part.  And doing it *with* differences
(aka "discarded/held changesets"), it's even worse.
So I think the right course of action is something like:

- start with 4 version of the code:

  [upstream] <-> [clean upstream] <-> [clean Emacs] <-> [Emacs]

  where "clean upstream" means "upstream without the patches that
  shouldn't make it into Emacs" and "clean Emacs" means "Emacs without
  the changes that shouldn't make it into upstream".  I.e. [clean
  upstream] and [clean Emacs] should be virtually identical, except
  maybe for file layout and VCS.

- first simplification: reject Emacs patches that can't make it
  into upstream.  I.e. enforce [clean Emacs] == [Emacs].
  That means that when an Emacs patch is unacceptable upstream, it needs
  to be resolved somehow: either one side convinces the other, or the
  patch needs to be reworked.  Typically such changes are linked to
  features that don't exist/work in XEmacs or in older Emacsen, so the
  problem can resolved by adding compatibility code.

- second simplification: don't install into [upstream] patches that
  should make it into Emacs; instead install them into [clean upstream].

- the flow is now limited:

  [upstream] <- [clean upstream] <-> [Emacs]

- so [upstream] is basically a normal feature-branch that tracks the
  "trunk" aka [clean upstream].  Any DVCS will handle that just fine.

- so all that's left is the core 2-way sync.  I don't know of any
  easy/right way to do that.

- of course, the best is to add a third simplification: let upstream use
  [Emacs] for development, so the "2-way" sync is a nop.


        Stefan

Re: Keeping changes in sync with upstream projects

[mathias]

Zitat von Stefan Monnier <monnier@iro.umontreal.ca>:

> - of course, the best is to add a third simplification: let upstream use
>   [Emacs] for development, so the "2-way" sync is a nop.
>
or simpler, Emacs developer could use [clean upstream] for development  
of the package.

(This is what Git developer do. Parts of Git (git-gui, gitk) are  
developed in their repository. From time to time the Git maintainer  
pulls and merges them into the Git repository.)

Mathias

Re: Keeping changes in sync with upstream projects

[Stefan Monnier]

>> - of course, the best is to add a third simplification: let upstream use
>> [Emacs] for development, so the "2-way" sync is a nop.
> or simpler, Emacs developer could use [clean upstream] for development of
> the package.

Same thing, yes.  Tho that requires write access to their repository.
It will also hopefully be made easier if we move such large thingies
outside of the main Emacs-Bzr tree with the help of a package manager
(which would come with some mechanism to auto-fetch&update those
packages we want to be bundled with Emacs).


        Stefan