Frequent git repository corruptions

Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

Hi Guix!

This might be unrelated to Guix but since I'm in the dark here, I
figured I'd rather ask, maybe someone ran into a similar issue.

For the past 2-4 weeks I've had at least 4 Git repositories that got
corrupted for no apparent reason.  Even a checkout that I had never
touched.

When it happens, errors show up on most git commands, usually `git diff`
but sometimes also `git status`:

--8<---------------cut here---------------start------------->8---
> git diff
error: object file .git/objects/e0/1bd6c3c77309b13dab22f9e0de33576ede944f is empty
fatal: unable to read e01bd6c3c77309b13dab22f9e0de33576ede944f
--8<---------------cut here---------------end--------------->8---

A quick Internet search led me to realize there is little to do to
recover the corrupted repository beside cloning it again.  I haven't
lost anything so far, knock on wood.

My first thought was a hard drive issue.  I have a NVMe SAMSUNG
MZVLW256HEHP.
I ran some smartmontools test, everything looks fine.

But now that I think about it, I did not notice any other issue than
within the .git folder, so maybe it's a... Git issue?

Would it be possible that the Guix package of Git be the culprit?
I run git 2.19.0 and a repo got corrupted just this morning.

I use emacs-magit all the time.  Not sure it's related considering that
a repository I never edited also got corrupted.

Anyone?

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Frequent git repository corruptions

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> For the past 2-4 weeks I've had at least 4 Git repositories that got
> corrupted for no apparent reason.  Even a checkout that I had never
> touched.
>
> When it happens, errors show up on most git commands, usually `git diff`
> but sometimes also `git status`:
>
> --8<---------------cut here---------------start------------->8---
>> git diff
> error: object file .git/objects/e0/1bd6c3c77309b13dab22f9e0de33576ede944f is empty
> fatal: unable to read e01bd6c3c77309b13dab22f9e0de33576ede944f
> --8<---------------cut here---------------end--------------->8---
>
> A quick Internet search led me to realize there is little to do to
> recover the corrupted repository beside cloning it again.  I haven't
> lost anything so far, knock on wood.
>
> My first thought was a hard drive issue.  I have a NVMe SAMSUNG
> MZVLW256HEHP.
> I ran some smartmontools test, everything looks fine.

Is the file system corrupt?  (What does fsck say?)
Did the system die without a proper shutdown?

-- 
Ricardo

Re: Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 211 bytes --]


> Is the file system corrupt?  (What does fsck say?)

All good.

> Did the system die without a proper shutdown?

Well, yes, a couple of times I suppose.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Frequent git repository corruptions

[Mark H Weaver]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> This might be unrelated to Guix but since I'm in the dark here, I
> figured I'd rather ask, maybe someone ran into a similar issue.
>
> For the past 2-4 weeks I've had at least 4 Git repositories that got
> corrupted for no apparent reason.  Even a checkout that I had never
> touched.

I would test the machine's RAM, preferably with a tool like memtest86
that runs directly from GRUB.  Alas, we do not yet have a package for it
in Guix.

       Mark

Re: Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 79 bytes --]

Will do, thanks for the hint.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 146 bytes --]

Wait... there is no memtest86 in Grub by default.  How do you install this Grub module
on Guix?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Frequent git repository corruptions

[Tobias Geerinckx-Rice]

Pierre,

Pierre Neidhardt wrote:
> Wait... there is no memtest86 in Grub by default.  How do you 
> install this Grub module
> on Guix?

Memtest86+ isn't a GRUB module, it's a bootable (as one would a 
kernel) stand-alone executable.

You can build it manually for now¹ or download a precompiled 
binary² and boot it from your existing GRUB's command line.

Or boot anything like System Rescue CD or similar, which almost 
always include it out of the box.

Kind regards,

T G-R

1. It's been on my packaging to-do list for a while. I'll give it 
a stab this weekend.
2. Since it runs without an OS, the binary doesn't need te be 
patched or otherwise Guixified.

Re: Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 134 bytes --]

Oh, stupid me, I confused this for a Grub module.  Makes total sense, thanks Tobias!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]


> You can build it manually for now¹ or download a precompiled binary² and boot it
> from your existing GRUB's command line.

Actually, how do you do that?

I copied the binary to `/memtest.bin` then rebooted and tried the following Grub
command (my root partition is labelled "guixsd"):

--8<---------------cut here---------------start------------->8---
search --label --set=root guixsd
linux16 /memtest.bin
--8<---------------cut here---------------end--------------->8---

Then grub complains, if I recall correctly:

--8<---------------cut here---------------start------------->8---
error: premature end of file.
--8<---------------cut here---------------end--------------->8---

Any idea?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Frequent git repository corruptions

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

On Sat, 29 Sep 2018 14:50:37 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> > You can build it manually for now¹ or download a precompiled
> > binary² and boot it from your existing GRUB's command line.  
> 
> Actually, how do you do that?
> 
> I copied the binary to `/memtest.bin` then rebooted and tried the
> following Grub command (my root partition is labelled "guixsd"):
> 
> --8<---------------cut here---------------start------------->8---
> search --label --set=root guixsd
> linux16 /memtest.bin
> --8<---------------cut here---------------end--------------->8---
> 
> Then grub complains, if I recall correctly:
> 
> --8<---------------cut here---------------start------------->8---
> error: premature end of file.
> --8<---------------cut here---------------end--------------->8---
> 
> Any idea?
> 

Do you have UEFI? I read that Memtest86+ does not work with (U)EFI,
only the commercial/free (as in beer) versions of Memtest86 work with
UEFI:

https://askubuntu.com/questions/917961/can-i-boot-memtest86-if-im-using-uefi

https://wiki.ubuntuusers.de/memtest/ (German)

Though that's just what I read, I have no further knowledge/experience
with it.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 200 bytes --]

Yup, that's what I've found out as well.

There is an AUR package for memtest86 (no +) on EFI:

	https://aur.archlinux.org/packages/memtest86-efi/

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Frequent git repository corruptions

[Tobias Geerinckx-Rice]

Pierre Neidhardt wrote:
> Yup, that's what I've found out as well.
>
> There is an AUR package for memtest86 (no +) on EFI:
>
> 	https://aur.archlinux.org/packages/memtest86-efi/

Say no more :-) That is proprietary software.

Kind regards,

T G-R

Re: Frequent git repository corruptions

[Tobias Geerinckx-Rice]

Tobias Geerinckx-Rice wrote:
> Pierre Neidhardt wrote:
>> Yup, that's what I've found out as well.
>>
>> There is an AUR package for memtest86 (no +) on EFI:
>>
>> 	https://aur.archlinux.org/packages/memtest86-efi/
>
> Say no more :-) That is proprietary software.

...or so I was told yesterday. Looking at the website now I'm not 
so sure: there seems to be a GPL2 'Free' edition.

This whole family is very confusing.

Kind regards,

T G-R

Re: Frequent git repository corruptions

[Tobias Geerinckx-Rice]

Tobias Geerinckx-Rice wrote:
> Pierre Neidhardt wrote:
>> Yup, that's what I've found out as well.
>>
>> There is an AUR package for memtest86 (no +) on EFI:
>>
>> 	https://aur.archlinux.org/packages/memtest86-efi/

I stand by my initial pronouncement:

> Say no more :-) That is proprietary software.

Only the old, unmaintained (and notably non-UEFI) version 4 is 
GPL2. Newer versions are mystery meat.[1]

Thus endeth the noise,

T G-R

1: https://www.memtest86.com/features.htm

Re: Frequent git repository corruptions

[Mark H Weaver]

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Tobias Geerinckx-Rice wrote:
>> Pierre Neidhardt wrote:
>>> Yup, that's what I've found out as well.
>>>
>>> There is an AUR package for memtest86 (no +) on EFI:
>>>
>>> 	https://aur.archlinux.org/packages/memtest86-efi/
>
> I stand by my initial pronouncement:
>
>> Say no more :-) That is proprietary software.
>
> Only the old, unmaintained (and notably non-UEFI) version 4 is
> GPL2. Newer versions are mystery meat.[1]

Oh, that's unfortunate :-(

Does anyone know of free software that can test all of the RAM in a
machine and that runs on systems with UEFI?

Failing that, there's 'memtester' which runs as a normal program under
Linux (the kernel).  This method is unable to test RAM that's allocated
to the kernel or to other processes, but if you stop as many services as
possible and run it from a text terminal, it should be able to test most
of the RAM.

We don't yet have 'memtester' in Guix, but it looks like it should be
easy to package.

      Mark

Re: Frequent git repository corruptions

[Tobias Geerinckx-Rice]

Mark H. Weaver wrote:
> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>> Only the old, unmaintained (and notably non-UEFI) version 4 is
>> GPL2. Newer versions are mystery meat.[1]
>
> Oh, that's unfortunate :-(

Indeed: I think I've packaged memtest86+, but have no machine upon 
which to run my brand-new package. That does not inspire 
confidence.

I'll give memtester a try. Thanks!

Kind regards,

T G-R

Re: Frequent git repository corruptions

[Mark H Weaver]

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Mark H. Weaver wrote:
>> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>>> Only the old, unmaintained (and notably non-UEFI) version 4 is
>>> GPL2. Newer versions are mystery meat.[1]
>>
>> Oh, that's unfortunate :-(
>
> Indeed: I think I've packaged memtest86+, but have no machine upon
> which to run my brand-new package. That does not inspire confidence.

If you send me a patch to add 'memtest86+', I'd be glad to test it on my
Thinkpad X200.

> I'll give memtester a try. Thanks!

Thank you, Tobias!

     Mark

Re: Frequent git repository corruptions

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 200 bytes --]

I've let memtester run over 4GB with 100 iterations for approx 10 hours. I
int erupted before the end.
It did not report any issue as far as I know.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]