When do we merge the commits from emacs-25 onto master?

When do we merge the commits from emacs-25 onto master?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

I recently fixed a package.el bug on the emacs-25 branch, but it's
affecting a lot of people who are following master (and a lot of automated
tests out there run on master) so I'd like the fix to be applied on master
too.

Do we periodically merge stuff from the emacs-25 branch onto master? Or do
we merge only once (when the release is finalised)?

[-- Attachment #2: Type: text/html, Size: 393 bytes --]

Re: When do we merge the commits from emacs-25 onto master?

[Eli Zaretskii]

> Date: Mon, 23 Nov 2015 20:36:50 +0000
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> 
> Do we periodically merge stuff from the emacs-25 branch onto master?

Yes.

> Or do we merge only once (when the release is finalised)?

No.

Re: When do we merge the commits from emacs-25 onto master?

[Paul Eggert]

On 11/23/2015 12:36 PM, Artur Malabarba wrote:
> Do we periodically merge stuff from the emacs-25 branch onto master?

Yes. And you're welcome to do it. We're better off if more people know 
how to do it. Please see admin/notes/git-workflow. It needs updating (at 
least to change "emacs-24" to "emacs-25"), and you can do that before 
merging....

Re: When do we merge the commits from emacs-25 onto master?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

Thanks Paul and Eli.

> Yes. And you're welcome to do it. We're better off if more people know
how to do it. Please see admin/notes/git-workflow. It needs updating (at
least to change "emacs-24" to "emacs-25"), and you can do that before
merging....

Would someone kindly do that for me this time? :-)
Let's just say my laptop is on quarantine this week.
I promise I'll do it next week then (and learn how to do it in the
process).

[-- Attachment #2: Type: text/html, Size: 536 bytes --]

Re: When do we merge the commits from emacs-25 onto master?

[David Engster]

Artur Malabarba writes:
> Thanks Paul and Eli.
>
>> Yes. And you're welcome to do it. We're better off if more people know how to
> do it. Please see admin/notes/git-workflow. It needs updating (at least to
> change "emacs-24" to "emacs-25"), and you can do that before merging....
>
> Would someone kindly do that for me this time? :-)

We still haven't figured out how to deal with the generated ChangeLogs,
so I'm not sure we should merge at the moment.

-David

Re: ChangeLog and the release branch

[Dmitry Gutov]

On 11/24/2015 04:25 PM, Artur Malabarba wrote:

> I thought if we just generated the changelogs independently for each of
> the branches we would be fine (i.e., don't merge to master the changelog
> commits on emacs-25).

I suppose gitmerge.el could try and ignore ChangeLog.2. But then any 
edits we make to the commit messages by changing ChangeLog.2 in emacs-25 
branch, will be discarded with it.

ChangeLog and the release branch (was: When do we merge the commits from emacs-25 onto master?)

[Artur Malabarba]

David Engster <deng@randomsample.de> writes:

> We still haven't figured out how to deal with the generated ChangeLogs,
> so I'm not sure we should merge at the moment.

Could you elaborate on what the issue is?

I thought if we just generated the changelogs independently for each of
the branches we would be fine (i.e., don't merge to master the changelog
commits on emacs-25).
Or is there something I overlooked about the process?

Re: ChangeLog and the release branch (was: When do we merge the commits from emacs-25 onto master?)

[Eli Zaretskii]

> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Date: Tue, 24 Nov 2015 14:25:19 +0000
> Cc: David Engster <deng@randomsample.de>
> 
> David Engster <deng@randomsample.de> writes:
> 
> > We still haven't figured out how to deal with the generated ChangeLogs,
> > so I'm not sure we should merge at the moment.
> 
> Could you elaborate on what the issue is?

They were described in the discussion starting here:

  http://lists.gnu.org/archive/html/emacs-devel/2015-11/msg01461.html

> I thought if we just generated the changelogs independently for each of
> the branches we would be fine (i.e., don't merge to master the changelog
> commits on emacs-25).

We could do that, but then the manual corrections to ChangeLog.2 made
on the branch would have to be done again, manually, on master.

Re: When do we merge the commits from emacs-25 onto master?

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Date: Mon, 23 Nov 2015 23:43:21 +0100
> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel <emacs-devel@gnu.org>
> 
> We still haven't figured out how to deal with the generated ChangeLogs,
> so I'm not sure we should merge at the moment.

Would it work to somehow teach gitmerge.el to use the (fixed) text
from ChangeLog.2 as the merge-commit log message, instead of the
original log message at emacs-25 commit time?

Re: When do we merge the commits from emacs-25 onto master?

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Date: Mon, 23 Nov 2015 23:43:21 +0100
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel <emacs-devel@gnu.org>
>> 
>> We still haven't figured out how to deal with the generated ChangeLogs,
>> so I'm not sure we should merge at the moment.
>
> Would it work to somehow teach gitmerge.el to use the (fixed) text
> from ChangeLog.2 as the merge-commit log message, instead of the
> original log message at emacs-25 commit time?

I'm not entirely sure what you mean, but the original commits will enter
'master' through the merge, so gitlog-to-changelog will generate entries
for them.

I think at first we must decide between two options:

i) Generate ChangeLog.2 separately for the two branches and exclude it
   from merges.

ii) Devise some workflow which will allow to merge ChangeLog.2 from
   emacs-25 into master. Cherry-picks from master to emacs-25 are also a
   problem, obviously, but they will comprise far less commits
   (hopefully), so I think it's OK if they involve more work.

Option i) is simple, but will create more work for the people fixing the
ChangeLogs. Most importantly, when emacs-25 is merged into master, all
the ChangeLogs will have to be fixed again. It will be easier if merging
and ChangeLog-fixing is done at the same time by one person, since then
one can pretty much copy&paste, but I'm not sure how realistic that
is.

We could also do an extreme version of i): only create ChangeLog.2 for
emacs-25 once, immediately before the release. We will have fixed
ChangeLogs for all the commits already in master; the job is then to
extract only those which are in emacs-25. Maybe gitlog-to-changelog
could mark those commits with a special string? We could remove that
mark again when emacs 25.1 is released.

I already described one workflow for ii) in the other thread. Different
workflows are possible, of course, but I'd claim they all will be of the
same complexity.

-David

Re: When do we merge the commits from emacs-25 onto master?

[Paul Eggert]

On 11/24/2015 10:32 AM, Eli Zaretskii wrote:
>> From: David Engster <deng@randomsample.de>
>> Date: Mon, 23 Nov 2015 23:43:21 +0100
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel <emacs-devel@gnu.org>
>>
>> We still haven't figured out how to deal with the generated ChangeLogs,
>> so I'm not sure we should merge at the moment.
> Would it work to somehow teach gitmerge.el to use the (fixed) text
> from ChangeLog.2 as the merge-commit log message, instead of the
> original log message at emacs-25 commit time?

A clever idea, and something like that might work, but it sounds 
brittle. Often, changes to ChangeLog.2 aren't isolated to single 
commits, and offhand I don't see how in general to figure out which 
parts of ChangeLog.2 should be matched to which commits.

How about the following idea instead.  Let's put the emacs-25 branch in 
charge of ChangeLog.2, and put the master branch in charge of a new file 
ChangeLog.3 that exists only in the master branch. We can do this by 
updating CHANGELOG_HISTORY_INDEX_MAX to 3 only in the master.  Then 
merges from emacs-25 to master should be trivial, as people who want to 
manually edit ChangeLog.2 or to update it via 'make change-history' 
should do so only in the emacs-25 branch. We'll need to modify 'make 
ChangeLog' so that in the master it ignores commit messages merged from 
the emacs-25 branch (I assume this is doable but I haven't checked 
this). Backporting changes will still be a hassle (we'll need to run 
'make change-history' in the master, and then manually update 
ChangeLog.3 to remove references to the changes being backported to 
emacs-25), but that's OK.

Re: When do we merge the commits from emacs-25 onto master?

[Eli Zaretskii]

> Cc: bruce.connor.am@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 24 Nov 2015 11:46:11 -0800
> 
> > Would it work to somehow teach gitmerge.el to use the (fixed) text
> > from ChangeLog.2 as the merge-commit log message, instead of the
> > original log message at emacs-25 commit time?
> 
> A clever idea, and something like that might work, but it sounds 
> brittle. Often, changes to ChangeLog.2 aren't isolated to single 
> commits, and offhand I don't see how in general to figure out which 
> parts of ChangeLog.2 should be matched to which commits.

We shouldn't pay attention to _commits_ that change ChangeLog.2, we
should instead match Git log messages against entries in the
ChangeLog.2 the file in the working tree.

> How about the following idea instead.  Let's put the emacs-25 branch in 
> charge of ChangeLog.2, and put the master branch in charge of a new file 
> ChangeLog.3 that exists only in the master branch. We can do this by 
> updating CHANGELOG_HISTORY_INDEX_MAX to 3 only in the master.  Then 
> merges from emacs-25 to master should be trivial, as people who want to 
> manually edit ChangeLog.2 or to update it via 'make change-history' 
> should do so only in the emacs-25 branch. We'll need to modify 'make 
> ChangeLog' so that in the master it ignores commit messages merged from 
> the emacs-25 branch (I assume this is doable but I haven't checked 
> this). Backporting changes will still be a hassle (we'll need to run 
> 'make change-history' in the master, and then manually update 
> ChangeLog.3 to remove references to the changes being backported to 
> emacs-25), but that's OK.

You mean, "make change-history" on master will update ChangeLog.3 and
the same command on the branch will update ChangeLog.2?  But then how
to prevent the commit log messages from commits merged onto master
from appearing in ChangeLog.3?

Re: When do we merge the commits from emacs-25 onto master?

[Paul Eggert]

On 11/24/2015 12:14 PM, Eli Zaretskii wrote:
>> >We'll need to modify 'make
>> >ChangeLog' so that in the master it ignores commit messages merged from
>> >the emacs-25 branch (I assume this is doable but I haven't checked
>> >this)....
> You mean, "make change-history" on master will update ChangeLog.3 and
> the same command on the branch will update ChangeLog.2?

Yes.

> But then how
> to prevent the commit log messages from commits merged onto master
> from appearing in ChangeLog.3?

That's the modification to 'make ChangeLog' I mentioned above. Again, I 
am handwaving a bit here, but it should be doable.  Merging from 
emacs-25 to master should look like this:

1. make change-history

2. Now ChangeLog.3 is up-to-date.  Merge emacs-25 into the master; this 
updates ChangeLog.2 without changing ChangeLog.3.

3. Now ChangeLog.2 is up-to-date, but 'make change-history' would screw 
up ChangeLog.3.  Prevent this screwup by editing ChangeLog.3 to say that 
it records revisions to the current master commit (inclusive).

Easy, huh?  :-)

Re: When do we merge the commits from emacs-25 onto master?

[David Engster]

Paul Eggert writes:
> 1. make change-history
>
> 2. Now ChangeLog.3 is up-to-date.  Merge emacs-25 into the master;
> this updates ChangeLog.2 without changing ChangeLog.3.
>
> 3. Now ChangeLog.2 is up-to-date, but 'make change-history' would
> screw up ChangeLog.3.  Prevent this screwup by editing ChangeLog.3 to
> say that it records revisions to the current master commit
> (inclusive).
>
> Easy, huh?  :-)

So in the end, you have ChangeLog.2 and ChangeLog.3 in master which are
tracking changes from different branches in parallel, i.e., for the same
time span. Isn't that a problem?

-David

Re: When do we merge the commits from emacs-25 onto master?

[Paul Eggert]

David Engster wrote:
> So in the end, you have ChangeLog.2 and ChangeLog.3 in master which are
> tracking changes from different branches in parallel, i.e., for the same
> time span. Isn't that a problem?

Only in explaining it.  :-)

ChangeLog entries' time stamps are not necessarily in time order even now.  For 
example, master's ChangeLog.2 currently lists two 2015-11-06 patches by Dmitry 
Gutov before it lists a 2015-11-08 patch by me, so it's not strictly in reverse 
time order as one might naively expect.  This is a natural consequence of how 
Git works.

Similarly, as far as ChangeLog files are concerned we can consider the master 
branch's patches to be "after" emacs-25's patches logically, even if they're not 
after emacs-25's patches temporally (or in Git's order, for that matter). 
People who care more about temporal order, or about Git's order, can look at the 
timestamps (and/or at the Git log).  People who care about what's in Emacs 25.0 
versus 25.1 can look in ChangeLog.2 versus ChangeLog.3.

Perhaps we should rename ChangeLog.2 to ChangeLog.25.0?

Re: When do we merge the commits from emacs-25 onto master?

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  bruce.connor.am@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 25 Nov 2015 01:35:44 +0100
> 
> Paul Eggert writes:
> > 1. make change-history
> >
> > 2. Now ChangeLog.3 is up-to-date.  Merge emacs-25 into the master;
> > this updates ChangeLog.2 without changing ChangeLog.3.
> >
> > 3. Now ChangeLog.2 is up-to-date, but 'make change-history' would
> > screw up ChangeLog.3.  Prevent this screwup by editing ChangeLog.3 to
> > say that it records revisions to the current master commit
> > (inclusive).
> >
> > Easy, huh?  :-)
> 
> So in the end, you have ChangeLog.2 and ChangeLog.3 in master which are
> tracking changes from different branches in parallel, i.e., for the same
> time span. Isn't that a problem?

We won't release from master, so the problem is not so significant, I
think.

When branching, we might need to do something about that, but that's a
rare event.

Re: When do we merge the commits from emacs-25 onto master?

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, bruce.connor.am@gmail.com,
>  emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 24 Nov 2015 22:32:37 -0800
> 
> Perhaps we should rename ChangeLog.2 to ChangeLog.25.0?

Yes, probably.

Re: ChangeLog and the release branch

[Yagnesh Raghava Yakkala]

Hello Eli,

>> I thought if we just generated the changelogs independently for each of
>> the branches we would be fine (i.e., don't merge to master the changelog
>> commits on emacs-25).
>
> We could do that, but then the manual corrections to ChangeLog.2 made
> on the branch would have to be done again, manually, on master.

I may be totally wrong, but I think we can have a clean solution by not fixing
Changelog manually, but storing the diff of Changelog (of that commit) in git
notes.

And while generating Changelog

- retrieve changelog from commit message
- apply diff from git notes (if available)
- Finally store it in the Changelog file.

IIUC, this could work on any branch.

Yagnesh.

Re: ChangeLog and the release branch

[Eli Zaretskii]

> From: Yagnesh Raghava Yakkala <hi@yagnesh.org>
> Cc: Artur Malabarba <bruce.connor.am@gmail.com>,  deng@randomsample.de,  emacs-devel@gnu.org
> Date: Fri, 27 Nov 2015 02:37:48 +0530
> 
> I may be totally wrong, but I think we can have a clean solution by not fixing
> Changelog manually, but storing the diff of Changelog (of that commit) in git
> notes.
> 
> And while generating Changelog
> 
> - retrieve changelog from commit message
> - apply diff from git notes (if available)
> - Finally store it in the Changelog file.
> 
> IIUC, this could work on any branch.

How do you store the diffs in Git notes?  You can't easily write diffs
by hand, so some script is needed to generate them and put them there.

Other than that, perhaps this is what we should do.

Thanks.

Re: ChangeLog and the release branch

[David Engster]

Eli Zaretskii writes:
>> From: Yagnesh Raghava Yakkala <hi@yagnesh.org>
>> Cc: Artur Malabarba <bruce.connor.am@gmail.com>,
>> deng@randomsample.de, emacs-devel@gnu.org
>
>> Date: Fri, 27 Nov 2015 02:37:48 +0530
>> 
>> I may be totally wrong, but I think we can have a clean solution by not fixing
>> Changelog manually, but storing the diff of Changelog (of that commit) in git
>> notes.
>> 
>> And while generating Changelog
>> 
>> - retrieve changelog from commit message
>> - apply diff from git notes (if available)
>> - Finally store it in the Changelog file.
>> 
>> IIUC, this could work on any branch.
>
> How do you store the diffs in Git notes?  You can't easily write diffs
> by hand, so some script is needed to generate them and put them there.

I don't think you would store a diff there but the complete, corrected
ChangeLog.

> Other than that, perhaps this is what we should do.

If someone is volunteering to actually set this up and deal with the
aftermath of people struggling with 'git notes', then yes. Otherwise, I
think that Paul's suggestion to simply work with two separate ChangeLog
files is the best yet, but it would also need someone to make the
necessary changes in the gitlog-to-* scripts (skipping the commits that
originate from emacs-25).

-David

Re: ChangeLog and the release branch

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Yagnesh Raghava Yakkala <hi@yagnesh.org>,  bruce.connor.am@gmail.com,  emacs-devel@gnu.org, Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 28 Nov 2015 12:50:46 +0100
> 
> > How do you store the diffs in Git notes?  You can't easily write diffs
> > by hand, so some script is needed to generate them and put them there.
> 
> I don't think you would store a diff there but the complete, corrected
> ChangeLog.

That, too, needs some help, no?  It's easy to format ChangeLog entries
when actually editing a ChangeLog file.  It's probably harder to do
that when editing a Git note.

> If someone is volunteering to actually set this up and deal with the
> aftermath of people struggling with 'git notes', then yes.

Could you tell what that "struggling" entails?  I never used Git
notes, so I don't know what difficulties and gotchas this could
present.

Re: ChangeLog and the release branch

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: Yagnesh Raghava Yakkala <hi@yagnesh.org>,
>> bruce.connor.am@gmail.com, emacs-devel@gnu.org, Paul Eggert
>
>> <eggert@cs.ucla.edu>
>> Date: Sat, 28 Nov 2015 12:50:46 +0100
>> 
>> > How do you store the diffs in Git notes?  You can't easily write diffs
>> > by hand, so some script is needed to generate them and put them there.
>> 
>> I don't think you would store a diff there but the complete, corrected
>> ChangeLog.
>
> That, too, needs some help, no?  It's easy to format ChangeLog entries
> when actually editing a ChangeLog file.  It's probably harder to do
> that when editing a Git note.

Yes, AFAIK there's no support for git notes in Emacs.

>> If someone is volunteering to actually set this up and deal with the
>> aftermath of people struggling with 'git notes', then yes.
>
> Could you tell what that "struggling" entails?  I never used Git
> notes, so I don't know what difficulties and gotchas this could
> present.

See above: no support in plain Emacs, so people would have to do it via
CLI or some other means. And last I checked, which admittedly was years
ago, notes were poorly supported by git. Notes are a bit similar to
tags, meaning you have to explicitly push to the 'refs/tags'
refspec. Fortunately, you don't have to know that, because 'git push'
has the '--tags' switch which does that automatically. But there's no
such switch for notes, so you have to explicitly push to
'refs/notes'. It even gets more weird when you want to 'fetch' notes,
which requires you to say something like

  git fetch origin refs/notes/*:refs/notes/*

On top of that, there's no real support for merging notes. Maybe things
have improved in the meantime, but we can't count on people to always
use up-to-date git versions.

-David

Re: ChangeLog and the release branch

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

>>> If someone is volunteering to actually set this up and deal with the
>>> aftermath of people struggling with 'git notes', then yes.
>> 
>> Could you tell what that "struggling" entails?  I never used Git
>> notes, so I don't know what difficulties and gotchas this could
>> present.

> See above: no support in plain Emacs, so people would have to do it via CLI
> or some other means. And last I checked, which admittedly was years ago,
> notes were poorly supported by git. Notes are a bit similar to tags, meaning
> you have to explicitly push to the 'refs/tags' refspec. Fortunately, you
> don't have to know that, because 'git push' has the '--tags' switch which
> does that automatically. But there's no such switch for notes, so you have
> to explicitly push to 'refs/notes'. It even gets more weird when you want to
> 'fetch' notes, which requires you to say something like

I've tried using Git notes in a real setting before, and they are indeed a
pain, requiring expertise on the part of everyone to get it right.

They are great for attaching local notes to commits, like the output from
build logs and tests, for future reference. But they are somewhat poor in a
collaborative setting. Or, they were the last time I tried to use them.

John