splitting up and sorting commits?

splitting up and sorting commits?

[Csepp]

Hey all!

I'm working on a fairly sizeable MirageOS branch, just getting the
hello-world kernel running involved adding about 40 packages.  Very
often I run into a scenario where an imported package needs some other
package to compile, and then that needs another, and then that another,
and so on and so on, so by the time I can commit the first I have a
plethora of new packages that should in theory all get their own
commits.
There are two problems with this:
* Splitting up the commit is a pain, even with git add --patch, because
hand editing the diff sucks and splitting does not work, possibly due to
there not being enough space between defines for git's taste.
* It is very easy for package to get added before their dependencies, so
even though by the end of the commit chain everything builds perfectly
fine, there are intermediate commits that can't be tested on their own.

How should one solve this?  I already spent way too much time on a
script that I foolishly thought would be able to automatically sort
commits based on their dependencies, but now I'm throwing in the towel,
it's not getting anywhere.

Re: splitting up and sorting commits?

[zimoun]

Hi,

On mer., 02 nov. 2022 at 00:05, Csepp <raingloom@riseup.net> wrote:

> I'm working on a fairly sizeable MirageOS branch, just getting the
> hello-world kernel running involved adding about 40 packages.  Very
> often I run into a scenario where an imported package needs some other
> package to compile, and then that needs another, and then that another,
> and so on and so on, so by the time I can commit the first I have a
> plethora of new packages that should in theory all get their own
> commits.
> There are two problems with this:
> * Splitting up the commit is a pain, even with git add --patch, because
> hand editing the diff sucks and splitting does not work, possibly due to
> there not being enough space between defines for git's taste.
> * It is very easy for package to get added before their dependencies, so
> even though by the end of the commit chain everything builds perfectly
> fine, there are intermediate commits that can't be tested on their own.
>
> How should one solve this?  I already spent way too much time on a
> script that I foolishly thought would be able to automatically sort
> commits based on their dependencies, but now I'm throwing in the towel,
> it's not getting anywhere.

Well, I do not know if I am answering the questions. :-)

Maybe what you are looking for is ./etc/committer.scm which tries to
split the complete change to small chunks and commit them.

Recently, I was updating Julia packages and I hit something similar.
Well, I have started with “./pre-inst-env guix refresh -u -m
julia-packages.scm” and then “./pre-inst-env ./etc/commiter.scm” which
leads to ~140 commits.  Next, I add the missing packages (committing
them individually) such that the final state just builds and I fix some
tests etc.

Last, I rebase all and edit the history.  Using Magit and Emacs, it is
really straightforward.  The main issue with large change is on one hand
to have that each atomic commit builds fine and on the other hand keep
each commit as atomic as possible.  Well, from my point of view, it is
tradeoff – personally, I am ready to have some commits that do not build
and keep them atomic.

Well, I have also spent too much time trying to find the correct order.
I also tried to an ugly script using the DAG to sort the packages but
then some updates require other specific version… and it is nearly
impossible to satisfy all the constraints.


Cheers,
simon

Re: splitting up and sorting commits?

[Liliana Marie Prikler]

Am Mittwoch, dem 02.11.2022 um 00:05 +0000 schrieb Csepp:
> Hey all!
> 
> I'm working on a fairly sizeable MirageOS branch, just getting the
> hello-world kernel running involved adding about 40 packages.  Very
> often I run into a scenario where an imported package needs some
> other package to compile, and then that needs another, and then that
> another, and so on and so on, so by the time I can commit the first I
> have a plethora of new packages that should in theory all get their
> own commits.
> There are two problems with this:
> * Splitting up the commit is a pain, even with git add --patch,
> because hand editing the diff sucks and splitting does not work,
> possibly due to there not being enough space between defines for
> git's taste.
> * It is very easy for package to get added before their dependencies,
> so even though by the end of the commit chain everything builds
> perfectly fine, there are intermediate commits that can't be tested
> on their own.
> 
> How should one solve this?  I already spent way too much time on a
> script that I foolishly thought would be able to automatically sort
> commits based on their dependencies, but now I'm throwing in the
> towel, it's not getting anywhere.
zimoun mentioned etc/committer.scm, which is nice if it works for you,
but perhaps doesn't fully answer your question.  The pauper's solution
would be to keep a separate "scratch pad" file containing a bunch of
assorted guix packages, ordered in a way that all packages are defined
before they're used as inputs – you can easily maintain this order by
insertion sort.  Then, once the big package finally builds and you're
content with it, you can copy the definitions to the correct file(s) in
the guix tree one by one.

Cheers

Re: splitting up and sorting commits?

[zimoun]

Hi Liliana,

On Thu, 03 Nov 2022 at 21:59, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

>                                                  The pauper's solution
> would be to keep a separate "scratch pad" file containing a bunch of
> assorted guix packages, ordered in a way that all packages are defined
> before they're used as inputs – you can easily maintain this order by
> insertion sort.

I did something similar that for some Julia packages, but it can be
tedious depending on the number of packages.  Well, it could be cool to
have something like

    guix graph -m some-packages.scm --sort

returning the list of packages sorted (topological order).


Cheers,
simon

Re: splitting up and sorting commits?

[Liliana Marie Prikler]

Am Freitag, dem 04.11.2022 um 10:20 +0100 schrieb zimoun:
> Hi Liliana,
> 
> On Thu, 03 Nov 2022 at 21:59, Liliana Marie Prikler
> <liliana.prikler@gmail.com> wrote:
> 
> >                                                  The pauper's
> > solution would be to keep a separate "scratch pad" file containing
> > a bunch of assorted guix packages, ordered in a way that all
> > packages are defined before they're used as inputs – you can easily
> > maintain this order by insertion sort.
> 
> I did something similar that for some Julia packages, but it can be
> tedious depending on the number of packages.  Well, it could be cool
> to have something like
> 
>     guix graph -m some-packages.scm --sort
> 
> returning the list of packages sorted (topological order).
Sorting packages in files sounds like a good application for guix style
imho.  We might even want different orders like topological or
alphabetical.  Guix graph can aid in that, but it'd return a graph
rather than the raw package code.

Cheers

Re: splitting up and sorting commits?

[Andreas Enge]

Hello,

Am Wed, Nov 02, 2022 at 12:05:54AM +0000 schrieb Csepp:
> * It is very easy for package to get added before their dependencies, so
> even though by the end of the commit chain everything builds perfectly
> fine, there are intermediate commits that can't be tested on their own.

maybe it can be handled by a different workflow? I usually use the git stash
to perform a depth first traversal of the dependency graph like so:
- Add package A. Try to build it and see that it needs dependency B.
  "git stash".
- Add package B *instead*. Try to build it...
(da capo ad libitum)
- "git commit" with package B.
- "git stash pop". Try to build package A etc.
- "git commit" with package A.

The only drawback is that B and A are often defined next to each other,
which creates a spurious merge conflict during "git stash pop", but that
is easy to solve and requires an additional "git stash drop" in the end.

Sometimes I even give up on package A in the end, but then at least B
has been added to Guix :)

Andreas

Re: splitting up and sorting commits?

[Csepp]

Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> Am Wed, Nov 02, 2022 at 12:05:54AM +0000 schrieb Csepp:
>> * It is very easy for package to get added before their dependencies, so
>> even though by the end of the commit chain everything builds perfectly
>> fine, there are intermediate commits that can't be tested on their own.
>
> maybe it can be handled by a different workflow? I usually use the git stash
> to perform a depth first traversal of the dependency graph like so:
> - Add package A. Try to build it and see that it needs dependency B.
>   "git stash".
> - Add package B *instead*. Try to build it...
> (da capo ad libitum)
> - "git commit" with package B.
> - "git stash pop". Try to build package A etc.
> - "git commit" with package A.
>
> The only drawback is that B and A are often defined next to each other,
> which creates a spurious merge conflict during "git stash pop", but that
> is easy to solve and requires an additional "git stash drop" in the end.
>
> Sometimes I even give up on package A in the end, but then at least B
> has been added to Guix :)
>
> Andreas

I tried that, the merge conflicts drove me nuts.  And juggling the stash
stack was a nightmare.

Instead I ended up committing each package individually without caring
about dependencies, then doing an interactive rebase where every command
was edit.  If I saw warnings about undefined variables I ran git rebase
--edit-todo **without** magit, because it made simple textual edits
needlessly hard (this is why I use Kakoune and Emacs side by side),
opened .git/rebase-merge/done, cut the last line, pasted it in todo
after its dependency, ran git reset --hard "HEAD^" **ONLY** if HEAD was
actually that commit, if it was not yet committed because there was a
merge conflict, I ran git rebase --skip.  Yep, figuring out when to
reset and when to skip took a few tries. :)