Emacs git repo mangled

Emacs git repo mangled

[Michael Albinus]

Hi,

In the Emacs git repo there are commits, which do not show all
files. Example:

--8<---------------cut here---------------start------------->8---
# git checkout 6e0ad2ac68820ef197116ea4149ee60f40797a32
# ls -l lisp/*.el
-rw-rw-r--. 1 albinus albinus   79434 Mar 15  2021 lisp/cus-load.el
-rw-rw-r--. 1 albinus albinus   21486 Mar 15  2021 lisp/dired-loaddefs.el
-rw-rw-r--. 1 albinus albinus   41469 Mar 15  2021 lisp/finder-inf.el
-rw-rw-r--. 1 albinus albinus     812 Mar 15  2021 lisp/htmlfontify-loaddefs.el
-rw-rw-r--. 1 albinus albinus   13025 Mar 15  2021 lisp/ibuffer-loaddefs.el
-rw-rw-r--. 1 albinus albinus 1411746 Mar 15  2021 lisp/loaddefs.el
-rw-rw-r--. 1 albinus albinus    2674 Mar 15  2021 lisp/ps-print-loaddefs.el
-rw-rw-r--. 1 albinus albinus     450 Mar 15  2021 lisp/subdirs.el
# ls -al src/*.[hc]
-rw-rw-r--. 1 albinus albinus   1084 Mar 15  2021 src/buildobj.h
-rw-rw-r--. 1 albinus albinus  61138 Mar 15  2021 src/config.h
-rw-rw-r--. 1 albinus albinus   3673 Mar 15  2021 src/epaths.h
-rw-rw-r--. 1 albinus albinus 241162 Mar 15  2021 src/globals.h
--8<---------------cut here---------------end--------------->8---

Looks like such commits belong to the merge of eglot.

Sadly, this makes it almost impossible to bisect the sources (this is
how I found the problem).

Best regards, Michael.

Re: Emacs git repo mangled

[Gerd Möllmann]

> Sadly, this makes it almost impossible to bisect the sources (this is
> how I found the problem).

Does git bisect --first-parent help?  In Magit B -p B.  I've used that 
in the past to ignore merges from 28 to master.

Re: Emacs git repo mangled

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

Hi Gerd,

>> Sadly, this makes it almost impossible to bisect the sources (this is
>> how I found the problem).
>
> Does git bisect --first-parent help?

Yes, "git bisect start --first-parent" seems to do the trick. When the
guilty commit is in the merge, you get at least an indication for this.

Shall we document this trick? admin/notes/repo is very quiet about
bisecting in general.

Best regards, Michael.

Re: Emacs git repo mangled

[Gerd Möllmann]

> Shall we document this trick? admin/notes/repo is very quiet about
> bisecting in general.

Does something like this suffice?  I don't know at the moment what more 
to say about this.

diff --git a/admin/notes/repo b/admin/notes/repo
index 2185c5a003..a65a87bfc6 100644
--- a/admin/notes/repo
+++ b/admin/notes/repo
@@ -128,6 +128,10 @@ again.
  This is a semi-automated way to find the revision that introduced a bug.
  Browse 'git help bisect' for technical instructions.

+Depending on what you want to do, it can be helpful to use the option
+'--first-parent' with 'git bisect', which makes bisect ignore commits
+coming into a branch from merges.
+
  * Maintaining ChangeLog history

  Older ChangeLog entries are kept in history files named ChangeLog.1,

Re: Emacs git repo mangled

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

Hi Gerd,

> +Depending on what you want to do, it can be helpful to use the option
> +'--first-parent' with 'git bisect', which makes bisect ignore commits
> +coming into a branch from merges.

I'd rather say 'git start --first-parent'. There's no other bisect
subcommand with that option.

Otherwise it LGTM.

Best regards, Michael.

Re: Emacs git repo mangled

[Gerd Möllmann]

On 01.11.22 10:15, Michael Albinus wrote:
> I'd rather say 'git start --first-parent'. There's no other bisect
> subcommand with that option.
> 
> Otherwise it LGTM.

Thanks.  It's in.

Re: Emacs git repo mangled

[Gregory Heytings]

>> + Depending on what you want to do, it can be helpful to use the option
>> + '--first-parent' with 'git bisect', which makes bisect ignore commits
>> + coming into a branch from merges.
>
> I'd rather say 'git start --first-parent'. There's no other bisect 
> subcommand with that option.
>

Doing that is wrong, alas.  Limiting bisection to first parents will often 
produce a wrong result.  TRT here is to follow Linus' advice, to which I 
pointed in my other post, namely to manually mark the last commit of the 
eglot branch before it was merged as "good".

Re: Emacs git repo mangled

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

Hi Gregory,

>>> + Depending on what you want to do, it can be helpful to use the option
>>> + '--first-parent' with 'git bisect', which makes bisect ignore commits
>>> + coming into a branch from merges.
>>
>> I'd rather say 'git start --first-parent'. There's no other bisect
>> subcommand with that option.
>
> Doing that is wrong, alas.  Limiting bisection to first parents will
> often produce a wrong result.  TRT here is to follow Linus' advice, to
> which I pointed in my other post, namely to manually mark the last
> commit of the eglot branch before it was merged as "good".

This approach would raise other problems.

- You might not know that you are in a merged subtree. It took me two
  days until I realized this (hmm, this could mean I'm too stupid).

- If the bad commit is inside the merge, you won't see it, because you
  have marked the whole merged subtree as good (by marking the last
  commit of the merged subtree).

- It would require manual actions, because first you need to determine
  the range of the merged subtree in order to mark last commit of this.

Best regards, Michael.

Re: Emacs git repo mangled

[Gregory Heytings]

>> Doing that is wrong, alas.  Limiting bisection to first parents will 
>> often produce a wrong result.  TRT here is to follow Linus' advice, to 
>> which I pointed in my other post, namely to manually mark the last 
>> commit of the eglot branch before it was merged as "good".
>
> This approach would raise other problems.
>
> - You might not know that you are in a merged subtree. It took me two 
> days until I realized this (hmm, this could mean I'm too stupid).
>

I don't know what happened exactly, but the Eglot tree has another root, 
so when you are inside that branch you should not see any of the files 
that are part of the Emacs tree (e.g. "Makefile" and "configure").  What 
was the cause of your confusion?

>
> - If the bad commit is inside the merge, you won't see it, because you 
> have marked the whole merged subtree as good (by marking the last commit 
> of the merged subtree).
>

By definition, the bad commit cannot be inside the merged Eglot tree, 
because that three contains only Eglot, not Emacs.  It bad commit could be 
the merge commit, but that one is not excluded during the bisection if you 
mark the last commit before the merge as "good".

>
> - It would require manual actions, because first you need to determine 
> the range of the merged subtree in order to mark last commit of this.
>

That is correct, and it is the price to pay to preserve history when a 
tree with another root is merged.  Perhaps we could maintain a list of 
such merges somewhere, with the commit SHA of the last commit before each 
merge.  Or perhaps even a commented script, that would do a "git bisect 
good ..." for each such commit.

Re: Emacs git repo mangled

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

Hi Gregory,

> I don't know what happened exactly, but the Eglot tree has another
> root, so when you are inside that branch you should not see any of the
> files that are part of the Emacs tree (e.g. "Makefile" and
> "configure").  What was the cause of your confusion?

Exactly this: all files were lost, I thought. I'm not so fluent with
git, and so I did panic.

>> - If the bad commit is inside the merge, you won't see it, because
>> you have marked the whole merged subtree as good (by marking the
>> last commit of the merged subtree).
>
> By definition, the bad commit cannot be inside the merged Eglot tree,
> because that three contains only Eglot, not Emacs.  It bad commit
> could be the merge commit, but that one is not excluded during the
> bisection if you mark the last commit before the merge as "good".

That's the eglot case. I was speaking about a merge in general.

>> - It would require manual actions, because first you need to
>> determine the range of the merged subtree in order to mark last
>> commit of this.
>
> That is correct, and it is the price to pay to preserve history when a
> tree with another root is merged.

Sigh.

> Perhaps we could maintain a list of such merges somewhere, with the
> commit SHA of the last commit before each merge.  Or perhaps even a
> commented script, that would do a "git bisect good ..." for each such
> commit.

Don't know, I let it to the git aficionados.

Best regards, Michael.

Re: Emacs git repo mangled

[Gregory Heytings]

>> By definition, the bad commit cannot be inside the merged Eglot tree, 
>> because that tree contains only Eglot, not Emacs.  The bad commit could 
>> be the merge commit, but that one is not excluded during the bisection 
>> if you mark the last commit before the merge as "good".
>
> That's the eglot case. I was speaking about a merge in general.
>

Yes, and it's because of merges in general that using --first-parent is 
not a good idea.

>> Perhaps we could maintain a list of such merges somewhere, with the 
>> commit SHA of the last commit before each merge.  Or perhaps even a 
>> commented script, that would do a "git bisect good ..." for each such 
>> commit.
>
> Don't know, I let it to the git aficionados.
>

Eglot's merge (0186faf2a1) is the first case in Emacs' history in which 
another root commit (1e5b753bf4) was added to the repository.  So at the 
moment there is only one such command to type when starting a bisection: 
git bisect good 806734c1b1.

Eli, what do you think of adding an admin/git-bisect script to do that? 
In the future, if other similar merges are done, it would suffice to add 
another such line in that file.

Re: Emacs git repo mangled

[Eli Zaretskii]

> Date: Tue, 01 Nov 2022 16:47:57 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
>  emacs-devel@gnu.org
> 
> Eli, what do you think of adding an admin/git-bisect script to do that? 

Fine by me, if it's useful.

Re: Emacs git repo mangled

[Gregory Heytings]

>> Eli, what do you think of adding an admin/git-bisect script to do that?
>
> Fine by me, if it's useful.
>

Thanks, now done (208f0578d1).  Michael, you can now use 
admin/git-bisect-start.

Re: Emacs git repo mangled

[Gerd Möllmann]

> Thanks, now done (208f0578d1).  Michael, you can now use 
> admin/git-bisect-start.

I'm not opposing anything, but I wonder if mentioning of --first-parent
somewhere might be useful to others, because I find it pretty useful.

This is why I am using it, all the time so far, in Emacs, which is
admittedly a small sample size:

We have master with frequent merges from emacs-28.  If I know that a
problem exists in master, but not in emacs-28, or if I suspect that is
the case, I don't want to waste time checking commits incoming from
emacs-28.

You have to know what you're doing, as always, and YMMV, and blahblah.

Re: Emacs git repo mangled

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]


>
> I'm not opposing anything, but I wonder if mentioning of --first-parent 
> somewhere might be useful to others, because I find it pretty useful.
>
> This is why I am using it, all the time so far, in Emacs, which is 
> admittedly a small sample size:
>
> We have master with frequent merges from emacs-28.  If I know that a 
> problem exists in master, but not in emacs-28, or if I suspect that is 
> the case, I don't want to waste time checking commits incoming from 
> emacs-28.
>
> You have to know what you're doing, as always, and YMMV, and blahblah.
>

😉 I think the --first-parent is doing too much for your use case, because 
it will also skip other merges that are not from emacs-28.  Moreover git 
bisect is "fast", in a typical bisection only a few commits are checked. 
So I don't think that excluding merge branches without being 110% certain 
that the bad commit is not in these branches is a good idea.  That being 
said, I'm of course not against mentioning --first-parent either.

Re: Emacs git repo mangled

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

Hi Gregory,

> Eglot's merge (0186faf2a1) is the first case in Emacs' history in
> which another root commit (1e5b753bf4) was added to the repository.
> So at the moment there is only one such command to type when starting
> a bisection: git bisect good 806734c1b1.

Wouldn't his mean that all other commits prior the eglot merge would be
regarded as "good", when bisecting? I doubt this is what we want.

Best regards, Michael.

Re: Emacs git repo mangled

[Gregory Heytings]

>> Eglot's merge (0186faf2a1) is the first case in Emacs' history in which 
>> another root commit (1e5b753bf4) was added to the repository. So at the 
>> moment there is only one such command to type when starting a 
>> bisection: git bisect good 806734c1b1.
>
> Wouldn't his mean that all other commits prior the eglot merge would be 
> regarded as "good", when bisecting? I doubt this is what we want.
>

No, it means that all commits between Eglot's root (1e5b753bf4) and 
Eglot's merge (0186faf2a1), except Eglot's merge itself, are regarded as 
"good".  IOW, git bisect will never "descend" into Eglot's branch, and 
only into that branch, which is exactly what we want.

Re: Emacs git repo mangled

[Eli Zaretskii]

> Date: Tue, 01 Nov 2022 19:41:41 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
>  emacs-devel@gnu.org
> 
> > Wouldn't his mean that all other commits prior the eglot merge would be 
> > regarded as "good", when bisecting? I doubt this is what we want.
> >
> 
> No, it means that all commits between Eglot's root (1e5b753bf4) and 
> Eglot's merge (0186faf2a1), except Eglot's merge itself, are regarded as 
> "good".  IOW, git bisect will never "descend" into Eglot's branch, and 
> only into that branch, which is exactly what we want.

IMO, this interpretation should be in the comments inside the script.

Re: Emacs git repo mangled

[Gregory Heytings]

>>> Wouldn't his mean that all other commits prior the eglot merge would be
>>> regarded as "good", when bisecting? I doubt this is what we want.
>>
>> No, it means that all commits between Eglot's root (1e5b753bf4) and 
>> Eglot's merge (0186faf2a1), except Eglot's merge itself, are regarded 
>> as "good".  IOW, git bisect will never "descend" into Eglot's branch, 
>> and only into that branch, which is exactly what we want.
>
> IMO, this interpretation should be in the comments inside the script.
>

It is:

# Start a git bisection, and prune the branches that are the result of
# merging external trees into the Emacs repository.

[...]

# Prune commits 1e5b753bf4..806734c1b1 introduced by 0186faf2a1 (Eglot
# merge on Oct 20 2022)
git bisect good 806734c1b1

Re: Emacs git repo mangled

[Phil Sainty]

On 2022-11-02 22:14, Gregory Heytings wrote:
> # Prune commits 1e5b753bf4..806734c1b1 introduced by 0186faf2a1 (Eglot
> # merge on Oct 20 2022)
> git bisect good 806734c1b1

Let's not use abbreviated references; they may conflict in the future.

Re: Emacs git repo mangled

[Gregory Heytings]

>> # Prune commits 1e5b753bf4..806734c1b1 introduced by 0186faf2a1 (Eglot
>> # merge on Oct 20 2022)
>> git bisect good 806734c1b1
>
> Let's not use abbreviated references; they may conflict in the future.
>

It's unlikely IMO, but okay, done.

Re: Emacs git repo mangled

[Eli Zaretskii]

> Date: Wed, 02 Nov 2022 09:14:32 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: michael.albinus@gmx.de, gerd.moellmann@gmail.com, emacs-devel@gnu.org
> 
> >> No, it means that all commits between Eglot's root (1e5b753bf4) and 
> >> Eglot's merge (0186faf2a1), except Eglot's merge itself, are regarded 
> >> as "good".  IOW, git bisect will never "descend" into Eglot's branch, 
> >> and only into that branch, which is exactly what we want.
> >
> > IMO, this interpretation should be in the comments inside the script.
> 
> It is:

Not in enough detail.  Never mind, I added what I wanted myself.

Thanks.

Re: Emacs git repo mangled

[Gregory Heytings]

>
> Not in enough detail.  Never mind, I added what I wanted myself.
>

Thanks, now I see what you mean.  That file is meant to be used for other 
(future) similar merges, so I tried to improve your improvement and to 
make it more generic, in a "Commentary" section.

I'm thinking of also adding a (long) list of known bad commits to that 
file (commits on which Emacs does not build), are you okay with that idea?

Re: Emacs git repo mangled

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

Hi Gregory,

> No, it means that all commits between Eglot's root (1e5b753bf4) and
> Eglot's merge (0186faf2a1), except Eglot's merge itself, are regarded
> as "good".  IOW, git bisect will never "descend" into Eglot's branch,
> and only into that branch, which is exactly what we want.

That doesn't work for me. Test:

- Start with current master (8721e87a6ec0874057f83f54498a0e3a64475a53).
  Everything is clean.

--8<---------------cut here---------------start------------->8---
[albinus@gandalf emacs-bisect]$ git status
On branch master
Your branch is up to date with 'origin/master'.

You are currently bisecting, started from branch 'master'.
  (use "git bisect reset" to get back to the original branch)

nothing to commit, working tree clean
--8<---------------cut here---------------end--------------->8---

- Start bisecting with the script.

--8<---------------cut here---------------start------------->8---
[albinus@gandalf emacs-bisect]$ admin/git-bisect-start
Already on 'master'
Your branch is up to date with 'origin/master'.
status: waiting for both good and bad commits
status: waiting for bad commit, 1 good commit known
--8<---------------cut here---------------end--------------->8---

- Mark master tip as bad.

--8<---------------cut here---------------start------------->8---
[albinus@gandalf emacs-bisect]$ git bisect bad master
Bisecting: 80597 revisions left to test after this (roughly 16 steps)
[2d7117fe05f034bf2ab796b639041504c4216ac5] (Fx_show_tip): Set string to " " if empty.
--8<---------------cut here---------------end--------------->8---

- Mark a commit as good which is before the eglot merge. The idea is to
  find a bad commit which is also before the eglot merge.

--8<---------------cut here---------------start------------->8---
[albinus@gandalf emacs-bisect]$ git bisect good 55c9238189795448075e2d4af93a7b29a505f23c
Bisecting: 15456 revisions left to test after this (roughly 14 steps)
error: The following untracked working tree files would be overwritten by checkout:
	doc/misc/modus-themes.texi
	lisp/cedet/semantic/grammar-wy.el
Please move or remove them before you switch branches.
Aborting
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Re: Emacs git repo mangled

[Gregory Heytings]

>
> That doesn't work for me. Test:
>
> - Start with current master (8721e87a6ec0874057f83f54498a0e3a64475a53). 
> Everything is clean.
>
> [...]
>
> [albinus@gandalf emacs-bisect]$ git bisect good 55c9238189795448075e2d4af93a7b29a505f23c
> Bisecting: 15456 revisions left to test after this (roughly 14 steps)
> error: The following untracked working tree files would be overwritten by checkout:
> 	doc/misc/modus-themes.texi
> 	lisp/cedet/semantic/grammar-wy.el
> Please move or remove them before you switch branches.
> Aborting
>

That's not a problem due to the script AFAIU.  I don't know what you did 
exactly, but with a fresh clone, after your recipe I see:

Bisecting: 15456 revisions left to test after this (roughly 14 steps)
[a1333fe6cfb3abe6ec2111dca43b059510984906] Merge branch 'master' of git.savannah.gnu.org:/srv/git/emacs

and no error.  Did you clean before starting the bisection?

Re: Emacs git repo mangled

[Phil Sainty]

[-- Attachment #1: Type: text/plain, Size: 2043 bytes --]

On 2022-11-02 05:47, Gregory Heytings wrote:
> Eglot's merge (0186faf2a1) is the first case in Emacs' history in
> which another root commit (1e5b753bf4) was added to the repository.
> So at the moment there is only one such command to type when starting
> a bisection: git bisect good 806734c1b1.
> 
> Eli, what do you think of adding an admin/git-bisect script to do
> that? In the future, if other similar merges are done, it would
> suffice to add another such line in that file.

As another opt-in alternative, the git bisect commands for starting
and iterating on a bisection run a "git checkout" and so we can use
the post-checkout git hook to automate this.  I'm not aware of any
better choice[1] of hook here, so we need it to first ascertain that
the checkout is even involved in a git bisect, but I've attached a
proof of concept which works for me.  It's a bit fiddly in that it
means the "git bisect good <ref>" runs before the triggering "git
bisect start" command has finished (as can be observed in the output),
but this appears to be ok in practice.  The hook also has no idea
when the bisect has been completed, as all the BISECT* files are
still present when "git bisect reset" does its checkout; but because
the hook adds to the bisect log, it can detect whether or not a NEW
bisect has started, which is the important thing.

Client-side git hooks must be installed manually by the user (they
cannot be supplied automatically while cloning), so it would be up to
individuals to follow some documentation to do this; but the benefit
of a hook-based approach is that it should work regardless of how the
"git bisect start" is triggered (rather than relying on people using
the command line), which benefits people using Magit's bisect UI (to
pick an obvious example).

If we provide something like this, I suggest we also comment the file
of known-good commits with all the details of each entry, and just
grep out the comment lines when processing it.


-Phil

[1]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: post-checkout --]
[-- Type: text/x-shellscript; name=post-checkout, Size: 2073 bytes --]

#!/bin/sh

## General information on Git post-checkout hooks:
#
# This hook is invoked when a git checkout is run after having updated
# the worktree. The hook is given three parameters: the ref of the
# previous HEAD, the ref of the new HEAD (which may or may not have
# changed), and a flag indicating whether the checkout was a branch
# checkout (changing branches, flag=1) or a file checkout (retrieving
# a file from the index, flag=0). This hook cannot affect the outcome
# of git checkout.
#
# It is also run after git clone, unless the --no-checkout (-n) option
# is used. The first parameter given to the hook is the null-ref, the
# second the ref of the new HEAD and the flag is always 1.
#
# This hook can be used to perform repository validity checks,
# auto-display differences from the previous HEAD if different, or set
# working dir metadata properties.

head_old=$1
head_new=$2
flag=$3

# We are interested only in branch checkouts.
if [ "$flag" != "1" ]; then
    exit 0
fi

# n.b. pwd will be this working copy's root directory.
root=$(pwd)

# printf "$0: %s - %s - %s\n" "$@"
# ls -l "$root"/.git/BISECT*

if [ -f "$root/.git/BISECT_LOG" ]; then
    # Currently bisecting.  If the ONLY thing which has happened so
    # far is the "git bisect start" command, then mark our always-good
    # commits as "good".  (This will append to the log, after which
    # the following test will fail.)
    tail -1 "$root/.git/BISECT_LOG" | grep -q '^git bisect .*\bstart\b' \
        && [ -f "$root/etc/git-bisect-good-refs" ] \
        && cp "$root/etc/git-bisect-good-refs" "$root/.git/BISECT_EMACS_GOOD_REFS" \
        && printf %s\\n "Marking known-good references in Emacs repository." \
        && cat "$root/.git/BISECT_EMACS_GOOD_REFS" | xargs git bisect good \
        && git bisect log
else
    # Not currently bisecting; just tidy up.
    [ -f "$root/.git/BISECT_EMACS_GOOD_REFS" ] \
        && rm "$root/.git/BISECT_EMACS_GOOD_REFS"
fi

exit 0

# etc/git-bisect-good-refs is a file containing this commit ref:
# 806734c1b1f433de43d59d9a5e3a1e89d64315f6

Re: Emacs git repo mangled

[Robert Pluim]

>>>>> On Wed, 02 Nov 2022 15:11:06 +1300, Phil Sainty <psainty@orcon.net.nz> said:

    Phil> Client-side git hooks must be installed manually by the user (they
    Phil> cannot be supplied automatically while cloning), so it would be up to
    Phil> individuals to follow some documentation to do this; but the benefit
    Phil> of a hook-based approach is that it should work regardless of how the
    Phil> "git bisect start" is triggered (rather than relying on people using
    Phil> the command line), which benefits people using Magit's bisect UI (to
    Phil> pick an obvious example).

Thatʼs not an issue: autogen.sh currently installs some client-side
hooks already.

Robert
-- 

Re: Emacs git repo mangled

[Andreas Schwab]

On Okt 31 2022, Michael Albinus wrote:

> Sadly, this makes it almost impossible to bisect the sources (this is
> how I found the problem).

As a workaround, you could git bisect skip those commits.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Emacs git repo mangled

[Michael Albinus]

Andreas Schwab <schwab@linux-m68k.org> writes:

Hi Andreas,

>> Sadly, this makes it almost impossible to bisect the sources (this is
>> how I found the problem).
>
> As a workaround, you could git bisect skip those commits.

Yep, this would help. But if the merge to be skipped ranges over many
commits (like the eglot merge), it might be a long way to run, unless
you know where the merge starts and finishes.

Best regards, Michael.

Re: Emacs git repo mangled

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2083 bytes --]

I'm away from my machine right now, but I remember having left a link to
the scripts (a gist repository) i used, as well as a "practice run" of the
merge for inspection in a branch called scratch/eglot2emacs. I had no
comments except for a request of a tweak in an author's email address. I
then replaced that branch with feature/eglot2emacs, which i generated by
exactly the same method.

I don't understand what the problem is, because as far as i remember, the
merged branch contains only commits to two files, lisp/progmodes/eglot.el
and doc/misc/eglot.texi.

João



On Mon, Oct 31, 2022, 09:50 Michael Albinus <michael.albinus@gmx.de> wrote:

> Hi,
>
> In the Emacs git repo there are commits, which do not show all
> files. Example:
>
> --8<---------------cut here---------------start------------->8---
> # git checkout 6e0ad2ac68820ef197116ea4149ee60f40797a32
> # ls -l lisp/*.el
> -rw-rw-r--. 1 albinus albinus   79434 Mar 15  2021 lisp/cus-load.el
> -rw-rw-r--. 1 albinus albinus   21486 Mar 15  2021 lisp/dired-loaddefs.el
> -rw-rw-r--. 1 albinus albinus   41469 Mar 15  2021 lisp/finder-inf.el
> -rw-rw-r--. 1 albinus albinus     812 Mar 15  2021
> lisp/htmlfontify-loaddefs.el
> -rw-rw-r--. 1 albinus albinus   13025 Mar 15  2021 lisp/ibuffer-loaddefs.el
> -rw-rw-r--. 1 albinus albinus 1411746 Mar 15  2021 lisp/loaddefs.el
> -rw-rw-r--. 1 albinus albinus    2674 Mar 15  2021
> lisp/ps-print-loaddefs.el
> -rw-rw-r--. 1 albinus albinus     450 Mar 15  2021 lisp/subdirs.el
> # ls -al src/*.[hc]
> -rw-rw-r--. 1 albinus albinus   1084 Mar 15  2021 src/buildobj.h
> -rw-rw-r--. 1 albinus albinus  61138 Mar 15  2021 src/config.h
> -rw-rw-r--. 1 albinus albinus   3673 Mar 15  2021 src/epaths.h
> -rw-rw-r--. 1 albinus albinus 241162 Mar 15  2021 src/globals.h
> --8<---------------cut here---------------end--------------->8---
>
> Looks like such commits belong to the merge of eglot.
>
> Sadly, this makes it almost impossible to bisect the sources (this is
> how I found the problem).
>
> Best regards, Michael.
>
>

[-- Attachment #2: Type: text/html, Size: 2600 bytes --]

Re: Emacs git repo mangled

[Stefan Kangas]

João Távora <joaotavora@gmail.com> writes:

> I don't understand what the problem is, because as far as i remember, the
> merged branch contains only commits to two files, lisp/progmodes/eglot.el
> and doc/misc/eglot.texi.

AFAICT, the first commit on the eglot branch that was merged
(1e5b753bf46a) does not start out from some commit on master, so the
rest of the tree just won't be there.  Something to remember in future
merges, I think, but not the end of the world.

git bisect skip seems appropriate here.

Re: Emacs git repo mangled

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On Mon, Oct 31, 2022 at 4:04 PM Stefan Kangas <stefankangas@gmail.com>
wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > I don't understand what the problem is, because as far as i remember,
the
> > merged branch contains only commits to two files,
lisp/progmodes/eglot.el
> > and doc/misc/eglot.texi.
>
> AFAICT, the first commit on the eglot branch that was merged
> (1e5b753bf46a) does not start out from some commit on master, so the
> rest of the tree just won't be there.  Something to remember in future
> merges, I think, but not the end of the world.

How would one proceed in future merges of git-versioned packages
that started outside the Emacs mainline?  Are you suggesting it's possible
(in a future merge) to bring that first commit out of some commit on
master?
If so, how?

I think git bisect --first-parent should be behave correctly, but I haven't
tested.  Maybe there's a way to make that the default for git bisections in
the Emacs repo?

João

PS: This is the thread where the merge technique was discussed and
the provisional results presented, if it helps anyone:
https://lists.gnu.org/archive/html/emacs-devel/2022-09/msg01583.html

[-- Attachment #2: Type: text/html, Size: 1723 bytes --]

Re: Emacs git repo mangled

[Stefan Kangas]

João Távora <joaotavora@gmail.com> writes:

> How would one proceed in future merges of git-versioned packages
> that started outside the Emacs mainline?  Are you suggesting it's possible
> (in a future merge) to bring that first commit out of some commit on
> master?

Unfortunately, I don't have a recipe handy, but AFAIU this should indeed
be possible.  Something we could look into properly the next time we do
this, perhaps.

Re: Emacs git repo mangled

[Gregory Heytings]

>
> Looks like such commits belong to the merge of eglot.
>
> Sadly, this makes it almost impossible to bisect the sources (this is 
> how I found the problem).
>

See https://lore.kernel.org/lkml/alpine.LFD.2.00.0901111113150.6528@localhost.localdomain/

Re: Emacs git repo mangled

[Payas Relekar]

Michael Albinus <michael.albinus@gmx.de> writes:

Adding João Távora as maintainer of Eglot.

I did write initial sctipts to merge eglot to emacs.git, and might be
able to take a look tomorrow.

However, considering these commits are already merged, I am not aware of
what the policy is on history rewrites, João might be best to suggest a solution.

> Hi,
>
> In the Emacs git repo there are commits, which do not show all
> files. Example:
>
> # git checkout 6e0ad2ac68820ef197116ea4149ee60f40797a32
> # ls -l lisp/*.el
> -rw-rw-r--. 1 albinus albinus   79434 Mar 15  2021 lisp/cus-load.el
> -rw-rw-r--. 1 albinus albinus   21486 Mar 15  2021 lisp/dired-loaddefs.el
> -rw-rw-r--. 1 albinus albinus   41469 Mar 15  2021 lisp/finder-inf.el
> -rw-rw-r--. 1 albinus albinus     812 Mar 15  2021 lisp/htmlfontify-loaddefs.el
> -rw-rw-r--. 1 albinus albinus   13025 Mar 15  2021 lisp/ibuffer-loaddefs.el
> -rw-rw-r--. 1 albinus albinus 1411746 Mar 15  2021 lisp/loaddefs.el
> -rw-rw-r--. 1 albinus albinus    2674 Mar 15  2021 lisp/ps-print-loaddefs.el
> -rw-rw-r--. 1 albinus albinus     450 Mar 15  2021 lisp/subdirs.el
> # ls -al src/*.[hc]
> -rw-rw-r--. 1 albinus albinus   1084 Mar 15  2021 src/buildobj.h
> -rw-rw-r--. 1 albinus albinus  61138 Mar 15  2021 src/config.h
> -rw-rw-r--. 1 albinus albinus   3673 Mar 15  2021 src/epaths.h
> -rw-rw-r--. 1 albinus albinus 241162 Mar 15  2021 src/globals.h
>
> Looks like such commits belong to the merge of eglot.
>
> Sadly, this makes it almost impossible to bisect the sources (this is
> how I found the problem).
>
> Best regards, Michael.
>
>

--

Re: Emacs git repo mangled

[João Távora]

Payas Relekar <relekarpayas@gmail.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
> Adding João Távora as maintainer of Eglot.
>
> I did write initial sctipts to merge eglot to emacs.git, and might be
> able to take a look tomorrow.
>
> However, considering these commits are already merged, I am not aware of
> what the policy is on history rewrites, João might be best to suggest
> a solution.

I've now looked at the situation, and I still don't understand what's
wrong.  As I expected the Eglot branch feature/eglot2emacs is just a
branch that shares no initial history with Emacs's mainline until the
merge with commit feature/eglot2texi manual, whose firstcommit, authored
by Eli, branches off the mainline in baa39e48.

Until that merge feature/eglot2emacs contains no files but
lisp/progmodes/eglot.el so that ls output, after you 'git checkout' to
one of feature/eglot2emacs's commits is for untracked files in your
working directory.

I detect nothing wrong in the tree: it's exactly what I would expect it
to be.  But I don't know exactly what Michael is trying to do.

Graphically and summarily, this is how the tree looks like now.

HTH



            master        f/eglot-texi-manual    f/eglot2emacs
               .                  .                 .

               |                  .                 .
               |
final merge    |                  .                 .
(1e5b753bf4)   *<-------\
               |         ---------+                 .
               |                  |
               |     merge with   *<--\             .
               |     manual branch|    --------\
               |     (1e5b753bf4) |             ----\
emacs history >*                  |                 |
               |                  *                 |
               | tweaks to manual |                 |
               |                  *                 |
               |                  |                 *
emacs history >*                  *                 |   lots of eglot
               |                  |                 |   history (includes
               |    Eli's commit  |                 *   some branching, but
               |    (1e5b753bf4)  *                 |   majority is linear).
               |                  |                 |
               |                 /                  *
emacs history >*             /---                   |
               |         /---                       *
               |     /---                           |
               | /---                               |
               +-                                   |
               |                                    |
               |                                    *  Initial commit creating
emacs history> *                                       lisp/progmodes/eglot.el
(baa39e48)     |                                       (1e5b753bf4)
               *
               |
               .
               .
               .

Re: Emacs git repo mangled

[Michael Albinus]

João Távora <joaotavora@gmail.com> writes:

Hi João,

> I've now looked at the situation, and I still don't understand what's
> wrong.  As I expected the Eglot branch feature/eglot2emacs is just a
> branch that shares no initial history with Emacs's mainline until the
> merge with commit feature/eglot2texi manual, whose firstcommit, authored
> by Eli, branches off the mainline in baa39e48.
>
> Until that merge feature/eglot2emacs contains no files but
> lisp/progmodes/eglot.el so that ls output, after you 'git checkout' to
> one of feature/eglot2emacs's commits is for untracked files in your
> working directory.
>
> I detect nothing wrong in the tree: it's exactly what I would expect it
> to be.  But I don't know exactly what Michael is trying to do.

There's nothing wrong with the eglot merge AFAICS. But the respective
commits are useless when you run "git bisect" and you land there,
because Emacs cannot be compiled based on any of those commits. That was
my pain.

Best regards, Michael.

Re: Emacs git repo mangled

[Stefan Monnier]

> I've now looked at the situation, and I still don't understand what's
> wrong.

Nothing's wrong: the Git history looks fine to me.
The problem is fundamental to `git bisect` (which starts from the
assumption that the history is linear, which is a lie).
That's what `git bisect skip` is for.
I'm sure there are other tricks one can use to teach `git bisect` to
avoid certain commits and/or branches, if needed.


        Stefan

Re: Emacs git repo mangled

[Andreas Schwab]

On Okt 31 2022, Stefan Monnier wrote:

> Nothing's wrong: the Git history looks fine to me.
> The problem is fundamental to `git bisect` (which starts from the
> assumption that the history is linear, which is a lie).

That's not true.  Bisection handles nonlinear history very well.  It
just assumes *sane* history.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Emacs git repo mangled

[Stefan Monnier]

Andreas Schwab [2022-10-31 19:33:13] wrote:
> On Okt 31 2022, Stefan Monnier wrote:
>> Nothing's wrong: the Git history looks fine to me.
>> The problem is fundamental to `git bisect` (which starts from the
>> assumption that the history is linear, which is a lie).
> That's not true.  Bisection handles nonlinear history very well.

I did not say it doesn't handle nonlinear history well.  I said it
assumes that the history is linear (it's inherent to the notion of
"bisecting").  Indeed, in practice the way Git turns the non-linear
history into a linear pseudo-history usually works very well.
But there are no miracles.


        Stefan

Re: Emacs git repo mangled

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Andreas Schwab [2022-10-31 19:33:13] wrote:
>> On Okt 31 2022, Stefan Monnier wrote:
>>> Nothing's wrong: the Git history looks fine to me.
>>> The problem is fundamental to `git bisect` (which starts from the
>>> assumption that the history is linear, which is a lie).
>> That's not true.  Bisection handles nonlinear history very well.
>
> I did not say it doesn't handle nonlinear history well.  I said it
> assumes that the history is linear (it's inherent to the notion of
> "bisecting").

Not at all. See "Bisection algorithm discussed" in [1]. `git bisect'
would be very dumb if it wouldn't take advantage of merge commits.

As for bisecting Emacs, it's a real PITA because there are so many
broken commits plus you really need a full rebuild for each step to
avoid false results.

One thing that would be useful is a list of bad commits on a format that
could be easily used with `git bisect skip'.

1. https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html

Re: Emacs git repo mangled

[Andreas Schwab]

On Okt 31 2022, Stefan Monnier wrote:

> assumes that the history is linear (it's inherent to the notion of
> "bisecting").

It is not.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."