crash on git-fast-import

crash on git-fast-import

[Rob Herring]

Hi,

lei started crashing on me today in git-fast-import. Nothing changed
on my system that I'm aware of. At first, I thought something stale
was lingering after my GNOME session died yesterday. I've since
rebooted and wiped out ~/.local/share/lei/store, but the issue
remains. I'm running commit e7c8ab553ee2 ("eml: reuse ->decode
buffer").

Here's an example:

$ lei up ~/Mail/from-me/
# https://lore.kernel.org/all/ limiting to 2024-01-21 16:33 -0600 and newer
# /home/rob/.local/share/lei/store 0/0
# /usr/bin/curl -gSf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=(dt%3A20240206145113..+AND+(f%3Arobh%40kernel.org+OR+f%3Arobh%2Bdt%40kernel.org)+AND+NOT+(nq%3AAcked-by+OR+nq%3AReviewed-by))+AND+dt%3A20240121223337..
fatal: Expected committer but didn't get one
fast-import: dumping crash report to
/home/rob/.local/share/lei/store/local/0.git/fast_import_crash_99633
E: import done: write to fast-import failed: Illegal seek at
/home/rob/proj/public-inbox/lib/PublicInbox/Import.pm line 88, <$fh>
line 2.

2024-03-06T14:51:14Z 99572 waitpid(99633) => -1 (No child processes)
at /home/rob/proj/public-inbox/lib/PublicInbox/IO.pm line 61.
99598 lei_xsearch 0 wq_worker: query_remote_mboxrd: E: write to
fast-import failed: Broken pipe at
/home/rob/proj/public-inbox/lib/PublicInbox/Import.pm line 88, <$fh>
line 2.
# 0 written to /home/rob/Mail/from-me/ (0 matches, 1 duplicates)


If the output is json, it works:

$ lei q -I https://lore.kernel.org/all/ 'f:robh d:1.day.ago..' > /dev/null
# /usr/bin/curl -gSf -s -d ''
https://lore.kernel.org/all/?x=m&q=f%3Arobh+dt%3A20240305154252..


Any ideas or tips for debugging this?

Rob

Re: crash on git-fast-import

[Eric Wong]

Rob Herring <robh@kernel.org> wrote:
> fatal: Expected committer but didn't get one
> fast-import: dumping crash report to
> /home/rob/.local/share/lei/store/local/0.git/fast_import_crash_99633

Are you able to share the contents of that crash file?

> E: import done: write to fast-import failed: Illegal seek at
> /home/rob/proj/public-inbox/lib/PublicInbox/Import.pm line 88, <$fh>
> line 2.

Oops, that error reporting is wrong, there :x  Will fix.

> Any ideas or tips for debugging this?

I'll try to look at dealing with this soon; occupied with several offline
problems :<

Re: crash on git-fast-import

[Rob Herring]

On Thu, Mar 7, 2024 at 9:20 PM Eric Wong <e@80x24.org> wrote:
>
> Rob Herring <robh@kernel.org> wrote:
> > fatal: Expected committer but didn't get one
> > fast-import: dumping crash report to
> > /home/rob/.local/share/lei/store/local/0.git/fast_import_crash_99633
>
> Are you able to share the contents of that crash file?

I wiped out the store with that particular one. They are all pretty
consistent in the errors, so here's another one:

fast-import crash report:
    fast-import process: 2487295
    parent process     : 2487077
    at 2024-03-07 13:31:06 +0000

fatal: Expected committer but didn't get one

Most Recent Commands Before Crash
---------------------------------
  blob
  mark :1
  data 12367
  get-mark :1
  reset refs/heads/main

  commit refs/heads/main
*

Active Branch LRU
-----------------
    active_branches = 0 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Inactive Branches
-----------------
refs/heads/main:
  status      : dirty
  tip commit  : 0000000000000000000000000000000000000000
  old tree    : 0000000000000000000000000000000000000000
  cur tree    : 0000000000000000000000000000000000000000
  commit clock: 0
  last pack   :


Marks
-----
:1 db9c47576664b9a87f6484c482a597fc3f959929

-------------------
END OF CRASH REPORT

>
> > E: import done: write to fast-import failed: Illegal seek at
> > /home/rob/proj/public-inbox/lib/PublicInbox/Import.pm line 88, <$fh>
> > line 2.
>
> Oops, that error reporting is wrong, there :x  Will fix.
>
> > Any ideas or tips for debugging this?
>
> I'll try to look at dealing with this soon; occupied with several offline
> problems :<

No worries.

In my attempt to avoid the problem, with this:

lei q -I https://lore.kernel.org/all/ --no-local --dedup=mid
--no-import-remote --no-save -o mail 'nq:"Reviewed-by: Rob Herring"
nq:diff d:16.weeks.ago..4.weeks.ago'

It works the first time, but when mail/ exists on subsequent runs, I
get the same crash.

Rob

Re: crash on git-fast-import

[Eric Wong]

Rob Herring <robh@kernel.org> wrote:
> Most Recent Commands Before Crash
> ---------------------------------
>   blob
>   mark :1
>   data 12367
>   get-mark :1
>   reset refs/heads/main

Ah, it looks like init.defaultBranch != master is completely
broken :x   You should be able to workaround the problem by
leaving defaultBranch unset for now.

Will try to fix ASAP but offline problems persist :<

[PATCH] import: fix handling of init.defaultBranch

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Will try to fix ASAP but offline problems persist :<

Fortunately it turned out to be a simple fix :x

-------8<-------
Subject: [PATCH] import: fix handling of init.defaultBranch

We must chomp the newline in the branch name if it's set.

Reported-by: Rob Herring <robh@kernel.org>
Link: https://public-inbox.org/meta/CAL_JsqK7P4gjLPyvzxNEcYmxT4j6Ah5f3Pz1RqDHxmysTg3aEg@mail.gmail.com/
Fixes: 73830410e4336b77 (treewide: use run_qx where appropriate, 2023-10-27)
---
 lib/PublicInbox/Import.pm | 1 +
 t/clone-coderepo.t        | 8 +++++---
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index a951874b..ed34d548 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -26,6 +26,7 @@ sub default_branch () {
 	state $default_branch = do {
 		my $h = run_qx([qw(git config --global init.defaultBranch)],
 				 { GIT_CONFIG => undef });
+		chomp $h;
 		$h eq '' ? 'refs/heads/master' : "refs/heads/$h";
 	}
 }
diff --git a/t/clone-coderepo.t b/t/clone-coderepo.t
index c0951941..c6180fc4 100644
--- a/t/clone-coderepo.t
+++ b/t/clone-coderepo.t
@@ -7,6 +7,7 @@ use PublicInbox::Import;
 use File::Temp;
 use File::Path qw(remove_tree);
 use PublicInbox::SHA qw(sha1_hex);
+use PublicInbox::IO;
 require_mods(qw(json Plack::Builder HTTP::Date HTTP::Status));
 require_git_http_backend;
 require_git '1.8.5';
@@ -38,9 +39,10 @@ my $t0 = time - 1;
 my $m; # manifest hashref
 
 {
-	my $rdr = {};
-	my $fi_data = './t/git.fast-import-data';
-	open $rdr->{0}, '<', $fi_data or xbail "open($fi_data): $!";
+	my $fi_data = PublicInbox::IO::try_cat './t/git.fast-import-data';
+	my $db = PublicInbox::Import::default_branch;
+	$fi_data =~ s!\brefs/heads/master\b!$db!gs;
+	my $rdr = { 0 => \$fi_data };
 	my @git = ('git', "--git-dir=$pa");
 	xsys_e([@git, qw(fast-import --quiet)], undef, $rdr);
 	xsys_e([qw(/bin/cp -Rp a.git b.git)], undef, { -C => "$tmpdir/src" });