fixing an Elpa package

fixing an Elpa package

[Eric Abrahamsen]

I've got a package (Gnorb) in Elpa that's stagnating because I can't
update it using Git. I'm pretty close to bailing on this, but wanted to
float a question first in case it's easier to resolve than I thought.

I originally incorporated the package into Elpa as a subtree, squashed
into one commit, from a repository elsewhere on my filesystem that is
linked with Github. Since then there are now four commits to this
subtree in Elpa, two of them squashes from the external repository.

In the meantime, my old computer died and I got a new one. So I
re-cloned both the Elpa repository and the Github-based repository onto
the new machine. If I add my Github-based repository as a remote in the
Elpa repository, then fetch and subtree-merge, Git tells me there are no
common commits, and wants to merge in all (317) commits from the
external repository. I guess that makes sense from Git's point of view,
but I don't want to pollute Elpa with all those commits. I'm not sure it
would even work, either.

Granted, it was probably a bad idea to take the subtree approach to
begin with. But there it is, and I've mostly lost patience with the
whole thing. If there's any means of recovering from this -- even if
it's a bit of a pain in the ass -- I would be very happy to do that, and
I hope someone might be willing to share some recipes.

Thanks,
Eric

Re: fixing an Elpa package

[Stefan Monnier]

> I originally incorporated the package into Elpa as a subtree, squashed
> into one commit, from a repository elsewhere on my filesystem that is
> linked with Github. Since then there are now four commits to this
> subtree in Elpa, two of them squashes from the external repository.

If I didn't warn you not to squash, then I'm sorry I failed to do that.
Why did you squash in the first place?

> In the meantime, my old computer died and I got a new one.  So I
> re-cloned both the Elpa repository and the Github-based repository onto
> the new machine. If I add my Github-based repository as a remote in the
> Elpa repository, then fetch and subtree-merge, Git tells me there are no
> common commits, and wants to merge in all (317) commits from the
> external repository.

Is the Github repository using different hashes from the old one from
which you merged (with squashing)?

> I guess that makes sense from Git's point of view,

If the hashes are the same, then I don't see why it makes sense.

> but I don't want to pollute Elpa with all those commits.

Merging subtrees with full history, is the way I recommend people do it
and the way it's done for most of the current subtrees, AFAIK.
After all, if it's installed as an "external" we also end up with "all
those commits".

> I'm not sure it would even work, either.

I can't see why not.

> Granted, it was probably a bad idea to take the subtree approach to
> begin with.

It's not set in stone.  So to fix this, we can either do a proper
(non-squashed) "git subtree" merge, or we can switch to a (non-squashed
either) external tree.


        Stefan

Re: fixing an Elpa package

[Eric Abrahamsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I originally incorporated the package into Elpa as a subtree, squashed
>> into one commit, from a repository elsewhere on my filesystem that is
>> linked with Github. Since then there are now four commits to this
>> subtree in Elpa, two of them squashes from the external repository.
>
> If I didn't warn you not to squash, then I'm sorry I failed to do that.
> Why did you squash in the first place?

Probably no good reason -- maybe I thought it was tidier. This was my
first time doing anything like this, and I was flailing a bit.

>> In the meantime, my old computer died and I got a new one.  So I
>> re-cloned both the Elpa repository and the Github-based repository onto
>> the new machine. If I add my Github-based repository as a remote in the
>> Elpa repository, then fetch and subtree-merge, Git tells me there are no
>> common commits, and wants to merge in all (317) commits from the
>> external repository.
>
> Is the Github repository using different hashes from the old one from
> which you merged (with squashing)?
>
>> I guess that makes sense from Git's point of view,
>
> If the hashes are the same, then I don't see why it makes sense.

About 275 commits from the external repo went into Elpa as a single
squashed commit, ab3b913. The file contents are the same, but I assumed
Git saw no correspondence between the one squashed and many un-squashed
commits, and told me I was starting over.

>> but I don't want to pollute Elpa with all those commits.
>
> Merging subtrees with full history, is the way I recommend people do it
> and the way it's done for most of the current subtrees, AFAIK.
> After all, if it's installed as an "external" we also end up with "all
> those commits".
>
>> I'm not sure it would even work, either.
>
> I can't see why not.
>
>> Granted, it was probably a bad idea to take the subtree approach to
>> begin with.
>
> It's not set in stone.  So to fix this, we can either do a proper
> (non-squashed) "git subtree" merge, or we can switch to a (non-squashed
> either) external tree.

Okay, I'm glad it's salvageable. I don't actually care if it's a subtree
or external, but I've got the subtree-related commands written down, so
might as well stick with that. I think that, so long as I don't squash
this time around, I can avoid stepping on my own feet again.

What's the next step? Commit a removal of the whole subtree, and start
over?

Semi-related question: if a users reports an emacs bug with my package
in the package header, or it gets tagged later, will an email be
automatically sent to me as maintainer?

Thanks for your help,
Eric

Re: fixing an Elpa package

[Stefan Monnier]

>>> I guess that makes sense from Git's point of view,
>> If the hashes are the same, then I don't see why it makes sense.
> About 275 commits from the external repo went into Elpa as a single
> squashed commit, ab3b913. The file contents are the same, but I assumed
> Git saw no correspondence between the one squashed and many un-squashed
> commits, and told me I was starting over.

I think the situation is not quite like you say.  I tried

   git subtree merge --squash -P packages/gnorb gnorb/master

and it did not try to re-add everything.  It did try to add too much
("git diff | wc" is a whole 80KB), because in

   % git log packages/gnorb/
   [...]
   commit ce7004456df8d17d1b1bb9b1feab3ddafb1e078a
   Author: Eric Abrahamsen <eric@ericabrahamsen.net>
   Date:   Sat Oct 25 08:19:41 2014 -0700
   
       Merging Gnorb commits up to 1.0.1

you somehow managed to "merge" without keeping track of the metadata
("git subtree merge --squash" doesn't keep all commits, but it does
keep the hashes in the commit messages, so "git merge" doesn't
understand what's going on, but "git subtree merge --squash" normally
does, although in the above commit there's no such tracking, so maybe
you didn't use "git subtree merge --squash").

> What's the next step? Commit a removal of the whole subtree, and start
> over?

I installed a dummy commit which merges the tree to which you apparently
sync'd in the above commit, so the above

   git subtree merge --squash -P packages/gnorb gnorb/master

now results in much fewer conflicts ("git diff | wc" is a mere 5KB).
So now you just have to use the above command (or another one if you
want to merge another revision than "gnorb/master"), then resolve the
conflicts, then commit and push.
If you need more help, you know where to find me ;-)

> Semi-related question: if a users reports an emacs bug with my package
> in the package header, or it gets tagged later, will an email be
> automatically sent to me as maintainer?

No, we currently don't have such a mechanism in place, sadly.


        Stefan

Re: fixing an Elpa package

[Eric Abrahamsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>>> I guess that makes sense from Git's point of view,
>>> If the hashes are the same, then I don't see why it makes sense.
>> About 275 commits from the external repo went into Elpa as a single
>> squashed commit, ab3b913. The file contents are the same, but I assumed
>> Git saw no correspondence between the one squashed and many un-squashed
>> commits, and told me I was starting over.
>
> I think the situation is not quite like you say.  I tried
>
>    git subtree merge --squash -P packages/gnorb gnorb/master
>
> and it did not try to re-add everything.  It did try to add too much
> ("git diff | wc" is a whole 80KB), because in
>
>    % git log packages/gnorb/
>    [...]
>    commit ce7004456df8d17d1b1bb9b1feab3ddafb1e078a
>    Author: Eric Abrahamsen <eric@ericabrahamsen.net>
>    Date:   Sat Oct 25 08:19:41 2014 -0700
>    
>        Merging Gnorb commits up to 1.0.1
>
> you somehow managed to "merge" without keeping track of the metadata
> ("git subtree merge --squash" doesn't keep all commits, but it does
> keep the hashes in the commit messages, so "git merge" doesn't
> understand what's going on, but "git subtree merge --squash" normally
> does, although in the above commit there's no such tracking, so maybe
> you didn't use "git subtree merge --squash").

Thanks, I didn't realize that's how it worked. Beats me how I got it
into that state, though.

>> What's the next step? Commit a removal of the whole subtree, and start
>> over?
>
> I installed a dummy commit which merges the tree to which you apparently
> sync'd in the above commit, so the above
>
>    git subtree merge --squash -P packages/gnorb gnorb/master
>
> now results in much fewer conflicts ("git diff | wc" is a mere 5KB).
> So now you just have to use the above command (or another one if you
> want to merge another revision than "gnorb/master"), then resolve the
> conflicts, then commit and push.
> If you need more help, you know where to find me ;-)

Okay, thanks for this. Just so I'm very clear: the above command uses
--squash so that Git would pay attention to the metadata in your dummy
commit, but I should not be using --squash from here on out, is that
correct?

I ran the command you listed above, resolved the conflicts, and
committed. That gave me one squashed commit containing all the new
commits from my external repo, and another merge commit (this second
commit was huge, and looked like most of the code from the package).

The external and Elpa trees were not in sync, though -- "diff -r" gave
me some differences. Running the subtree merge just told me that the
subtree was already at 4e7039a15, and didn't do anything else.

I ran the merge again without --squash, just to see what would happen,
and that appeared to pull in most of the remaining differences. Once I'd
resolved conflicts and committed, though, all of the individual commits
from the external repo were pulled into Elpa. So maybe I'm meant to keep
--squash in after all?

The top of the log now reads:
47f3dce Merge commit '4e7039a15b47244e7bd2c580d8bce976a6116b5a'
d3ce5e0 Merge commit 'be7fc18f06df6a73214c03e9f28640ec77b24264'
be7fc18 Squashed 'packages/gnorb/' changes from 321b23b..4e7039a

With all the commits from the external repo below that.

And, amazingly, there is still a two-line diff (in gnorb-bbdb.el)
between the external and Elpa repos. I don't get it.

Sorry to keep on with this...

Eric

Re: fixing an Elpa package

[Stefan Monnier]

> Okay, thanks for this. Just so I'm very clear: the above command uses
> --squash so that Git would pay attention to the metadata in your dummy
> commit, but I should not be using --squash from here on out, is that
> correct?

I'm not sure, to tell you the truth.  I never use --squash, so I'm not
very familiar with it.  But with the current tree, I see that

   git subtree merge --squash -P packages/gnorb gnorb/master

gives me relatively few conflicts, whereas

   git subtree merge -P packages/gnorb gnorb/master

gives me a load of conflicts, so it seems that if you don't provide the
"--squash" argument, Git assumes that you haven't used "--squash" in the
past either and doesn't look for the commit messages that "--squash"
uses to try and find the common ancestor.

IOW, if you've used --squash in the past, it's best to keep doing so.

You can switch to the non-squash option, of course, if you want.
Basically, you can do a

   git subtree merge -P packages/gnorb 321b23b1ad1b770e2b2bd27921f069b9394ca4d0
   git diff | patch -R -p1
   git commit -am 'Dummy merge to convert to non-squash subtree'

where 321b23b1ad1b770e2b2bd27921f069b9394ca4d0 is the revision
up-to-which the branch has already been (squash-)merged.

> I ran the command you listed above, resolved the conflicts, and
> committed. That gave me one squashed commit containing all the new
> commits from my external repo, and another merge commit (this second
> commit was huge, and looked like most of the code from the package).

Sounds right.

> The external and Elpa trees were not in sync, though -- "diff -r" gave
> me some differences.

These should only be differences due to the commits I installed directly
into elpa.git, which you maybe haven't merged in the same way into your tree?
[ Just guessing here.  ]


        Stefan

Re: fixing an Elpa package

[Eric Abrahamsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Okay, thanks for this. Just so I'm very clear: the above command uses
>> --squash so that Git would pay attention to the metadata in your dummy
>> commit, but I should not be using --squash from here on out, is that
>> correct?
>
> I'm not sure, to tell you the truth.  I never use --squash, so I'm not
> very familiar with it.  But with the current tree, I see that
>
>    git subtree merge --squash -P packages/gnorb gnorb/master
>
> gives me relatively few conflicts, whereas
>
>    git subtree merge -P packages/gnorb gnorb/master
>
> gives me a load of conflicts, so it seems that if you don't provide the
> "--squash" argument, Git assumes that you haven't used "--squash" in the
> past either and doesn't look for the commit messages that "--squash"
> uses to try and find the common ancestor.
>
> IOW, if you've used --squash in the past, it's best to keep doing so.
>
> You can switch to the non-squash option, of course, if you want.
> Basically, you can do a
>
>    git subtree merge -P packages/gnorb 321b23b1ad1b770e2b2bd27921f069b9394ca4d0
>    git diff | patch -R -p1
>    git commit -am 'Dummy merge to convert to non-squash subtree'
>
> where 321b23b1ad1b770e2b2bd27921f069b9394ca4d0 is the revision
> up-to-which the branch has already been (squash-)merged.
>
>> I ran the command you listed above, resolved the conflicts, and
>> committed. That gave me one squashed commit containing all the new
>> commits from my external repo, and another merge commit (this second
>> commit was huge, and looked like most of the code from the package).
>
> Sounds right.
>
>> The external and Elpa trees were not in sync, though -- "diff -r" gave
>> me some differences.
>
> These should only be differences due to the commits I installed directly
> into elpa.git, which you maybe haven't merged in the same way into your tree?
> [ Just guessing here.  ]

Okay, I stuck with squashing since switching to unsquashing looked like
it was going to get ugly. I just pushed, and hopefully will not break
anything. There's still a one-line cosmetic difference in gnorb-org.el
between Elpa and the external repo, but I'm going to close one eye and
pretend I don't see it.

Thanks a lot for your help,

Eric

Re: fixing an Elpa package

[Stefan Monnier]

> Okay, I stuck with squashing since switching to unsquashing looked like
> it was going to get ugly. I just pushed, and hopefully will not break
> anything.

Looks good, thanks.

> There's still a one-line cosmetic difference in gnorb-org.el
> between Elpa and the external repo, but I'm going to close one eye and
> pretend I don't see it.

You can also just fix that change ;-)

> Thanks a lot for your help,

My pleasure,


        Stefan