bug#41710: Possibly found a bug while doing a "guix pull"

bug#41710: Possibly found a bug while doing a "guix pull"

[Léon Lain Delysid]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

Hello!
I just tried to "guix pull" on my Debian Buster GNU/Linux system. Twice it
made the system completely freeze for a few seconds and then reboot. And
that third time it didn't crash the system but the guix pull failed and
returned this error:
https://pastebin.com/vhDR8gDC

Can somebody help me please?

(Before doing the third "pull", I did an "apt update" and "upgrade", but I
wasn't that much behind in updates, I had done it less than a week ago
last. My last "guix pull" was made at the same time as the "apt update", a
few days ago.)

Hoping I can resolve this soon or that my feedback was positive for the
development of Guix. =)
I can provide log debug info (if there is any) upon request.

Best regards!

[-- Attachment #2: Type: text/html, Size: 925 bytes --]

bug#41710: Possibly found a bug while doing a "guix pull"

[Julien Lepiller]

Le 4 juin 2020 10:38:09 GMT-04:00, "Léon Lain Delysid" <leon.lain.delysid@gmail.com> a écrit :
>Hello!
>I just tried to "guix pull" on my Debian Buster GNU/Linux system. Twice
>it
>made the system completely freeze for a few seconds and then reboot.
>And
>that third time it didn't crash the system but the guix pull failed and
>returned this error:
>https://pastebin.com/vhDR8gDC
>
>Can somebody help me please?
>
>(Before doing the third "pull", I did an "apt update" and "upgrade",
>but I
>wasn't that much behind in updates, I had done it less than a week ago
>last. My last "guix pull" was made at the same time as the "apt
>update", a
>few days ago.)
>
>Hoping I can resolve this soon or that my feedback was positive for the
>development of Guix. =)
>I can provide log debug info (if there is any) upon request.
>
>Best regards!

Sorry pastebin is not tor friendly, I couldn't look at it.

For the freezing part, could it be that you have relatively low free ram (<2GB) and no swap? Or maybe /tmp is mounted as a tmpfs and for some reason it takes a lot of space?

You might need to use substitutes when runnning guix pull. I don't have that much ram either on some computers, so I always take a look at https://ci.guix.gnu.org/jobset/guix-modular-master and choose the latest commit that was built for my architeeture. Then eg. guix pull --commit=c987b72

This will allow guix to download substitutes for guix pull instead of building locally, which doesn't require any ram usage.

HTH!

bug#41710: Possibly found a bug while doing a "guix pull"

[zimoun]

Dear,

On Thu, 4 Jun 2020 at 17:42, Julien Lepiller <julien@lepiller.eu> wrote:
> Le 4 juin 2020 10:38:09 GMT-04:00, "Léon Lain Delysid" <leon.lain.delysid@gmail.com> a écrit :

> >I just tried to "guix pull" on my Debian Buster GNU/Linux system. Twice
> >it
> >made the system completely freeze for a few seconds and then reboot.

What do you mean by "then reboot"?
Does it reboot by itself or do you manually force the reboot?

> >And
> >that third time it didn't crash the system but the guix pull failed and
> >returned this error:
> >https://pastebin.com/vhDR8gDC

How do you start guix-daemon?


> >(Before doing the third "pull", I did an "apt update" and "upgrade",
> >but I
> >wasn't that much behind in updates, I had done it less than a week ago
> >last. My last "guix pull" was made at the same time as the "apt
> >update", a
> >few days ago.)

What is your "guix describe"?

>
> >Hoping I can resolve this soon or that my feedback was positive for the
> >development of Guix. =)
> >I can provide log debug info (if there is any) upon request.
> >
> >Best regards!
>
> Sorry pastebin is not tor friendly, I couldn't look at it.

Julien, here the paste:

--8<---------------cut here---------------start------------->8---
./guix/store.scm:1035:9: ERROR:
  1. &store-protocol-error:
      message: "error parsing derivation
`/gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv':
expected string `Derive(['"
      status: 1
guix pull: error: You found a bug: the program
'/gnu/store/kpxami25fi3mrxb37sfbbx2s366chpk5-compute-guix-derivation'
failed to compute the derivation for Guix (version:
"415b90f24ee83059102f5fb0ce74a2bc0954fe58"; system: "x86_64-linux";
host version: "018cffc9c9e5a5855733f5f45a1c4d396bb6a321"; pull-version: 1).
Please report it by email to <bug-guix@gnu.org>.
--8<---------------cut here---------------end--------------->8---

First time, I see that.

All the best,
simon

bug#41710: Possibly found a bug while doing a "guix pull"

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> ./guix/store.scm:1035:9: ERROR:
>   1. &store-protocol-error:
>       message: "error parsing derivation
> `/gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv':
> expected string `Derive(['"
>       status: 1
> guix pull: error: You found a bug: the program
> '/gnu/store/kpxami25fi3mrxb37sfbbx2s366chpk5-compute-guix-derivation'
> failed to compute the derivation for Guix (version:
> "415b90f24ee83059102f5fb0ce74a2bc0954fe58"; system: "x86_64-linux";
> host version: "018cffc9c9e5a5855733f5f45a1c4d396bb6a321"; pull-version: 1).
> Please report it by email to <bug-guix@gnu.org>.

Léon, is this .drv file empty by any chance?

If the answer is yes, that suggests a file system corruption.  We see
reports like this sometimes.  It could maybe happen if the file system
was not cleanly unmounted.

The solution may be to run:

  guix gc -D /gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv

before re-running ‘guix pull’.

Could you try that and report back?

Thanks,
Ludo’.

bug#41710: Possibly found a bug while doing a "guix pull"

[Léon Lain Delysid]

[-- Attachment #1.1: Type: text/plain, Size: 1619 bytes --]

Hello, here's the result of the command you asked me to run. I executed it,
then did a "guix pull" again and got the same result. But the corrupted
filesystem idea might be that one. Can you give me the best command to run
a repair of the filesystem?
[image: Screenshot from 2020-06-07 03-05-33.jpg]

[image: Screenshot from 2020-06-07 02-50-54.jpg]

On Sat, Jun 6, 2020 at 3:22 PM Ludovic Courtès <ludo@gnu.org> wrote:

> Hi,
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > ./guix/store.scm:1035:9: ERROR:
> >   1. &store-protocol-error:
> >       message: "error parsing derivation
> >
> `/gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv':
> > expected string `Derive(['"
> >       status: 1
> > guix pull: error: You found a bug: the program
> > '/gnu/store/kpxami25fi3mrxb37sfbbx2s366chpk5-compute-guix-derivation'
> > failed to compute the derivation for Guix (version:
> > "415b90f24ee83059102f5fb0ce74a2bc0954fe58"; system: "x86_64-linux";
> > host version: "018cffc9c9e5a5855733f5f45a1c4d396bb6a321"; pull-version:
> 1).
> > Please report it by email to <bug-guix@gnu.org>.
>
> Léon, is this .drv file empty by any chance?
>
> If the answer is yes, that suggests a file system corruption.  We see
> reports like this sometimes.  It could maybe happen if the file system
> was not cleanly unmounted.
>
> The solution may be to run:
>
>   guix gc -D
> /gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv
>
> before re-running ‘guix pull’.
>
> Could you try that and report back?
>
> Thanks,
> Ludo’.
>

[-- Attachment #1.2: Type: text/html, Size: 2362 bytes --]

[-- Attachment #2: Screenshot from 2020-06-07 02-50-54.jpg --]
[-- Type: image/jpeg, Size: 566809 bytes --]

[-- Attachment #3: Screenshot from 2020-06-07 03-05-33.jpg --]
[-- Type: image/jpeg, Size: 942993 bytes --]

bug#41710: Possibly found a bug while doing a "guix pull"

[Léon Lain Delysid]

[-- Attachment #1.1: Type: text/plain, Size: 1920 bytes --]

Hello,
Sorry for responding so late, I was busy otherwise.
I have more than enough RAM, 16 GB. And also lots of swap space. And my
/tmp takes 104KB of space.



[image: Screenshot from 2020-06-07 03-05-33.jpg]

On Thu, Jun 4, 2020 at 5:41 PM Julien Lepiller <julien@lepiller.eu> wrote:

> Le 4 juin 2020 10:38:09 GMT-04:00, "Léon Lain Delysid" <
> leon.lain.delysid@gmail.com> a écrit :
> >Hello!
> >I just tried to "guix pull" on my Debian Buster GNU/Linux system. Twice
> >it
> >made the system completely freeze for a few seconds and then reboot.
> >And
> >that third time it didn't crash the system but the guix pull failed and
> >returned this error:
> >https://pastebin.com/vhDR8gDC
> >
> >Can somebody help me please?
> >
> >(Before doing the third "pull", I did an "apt update" and "upgrade",
> >but I
> >wasn't that much behind in updates, I had done it less than a week ago
> >last. My last "guix pull" was made at the same time as the "apt
> >update", a
> >few days ago.)
> >
> >Hoping I can resolve this soon or that my feedback was positive for the
> >development of Guix. =)
> >I can provide log debug info (if there is any) upon request.
> >
> >Best regards!
>
> Sorry pastebin is not tor friendly, I couldn't look at it.
>
> For the freezing part, could it be that you have relatively low free ram
> (<2GB) and no swap? Or maybe /tmp is mounted as a tmpfs and for some reason
> it takes a lot of space?
>
> You might need to use substitutes when runnning guix pull. I don't have
> that much ram either on some computers, so I always take a look at
> https://ci.guix.gnu.org/jobset/guix-modular-master and choose the latest
> commit that was built for my architeeture. Then eg. guix pull
> --commit=c987b72
>
> This will allow guix to download substitutes for guix pull instead of
> building locally, which doesn't require any ram usage.
>
> HTH!
>

[-- Attachment #1.2: Type: text/html, Size: 2778 bytes --]

[-- Attachment #2: Screenshot from 2020-06-07 03-05-33.jpg --]
[-- Type: image/jpeg, Size: 942993 bytes --]

bug#41710: Possibly found a bug while doing a "guix pull"

[Léon Lain Delysid]

[-- Attachment #1.1: Type: text/plain, Size: 2801 bytes --]

Hi!
Sorry for the late response, I was busy otherwise.
By "then reboot", I meant it rebooted by itself. The first two times I
tried the "guix pull".

"How do you start guix-daemon?"
I didn't change any of the defaults regarding this. So I guess the daemon
must start at startup with the other daemons.

"guix describe" gives this output:
$ guix describe
Generation 12 mai 31 2020 11:28:03 (current)
  guix 018cffc
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 018cffc9c9e5a5855733f5f45a1c4d396bb6a321


[image: Screenshot from 2020-06-07 03-05-33.jpg]

On Thu, Jun 4, 2020 at 8:11 PM zimoun <zimon.toutoune@gmail.com> wrote:

> Dear,
>
> On Thu, 4 Jun 2020 at 17:42, Julien Lepiller <julien@lepiller.eu> wrote:
> > Le 4 juin 2020 10:38:09 GMT-04:00, "Léon Lain Delysid" <
> leon.lain.delysid@gmail.com> a écrit :
>
> > >I just tried to "guix pull" on my Debian Buster GNU/Linux system. Twice
> > >it
> > >made the system completely freeze for a few seconds and then reboot.
>
> What do you mean by "then reboot"?
> Does it reboot by itself or do you manually force the reboot?
>
> > >And
> > >that third time it didn't crash the system but the guix pull failed and
> > >returned this error:
> > >https://pastebin.com/vhDR8gDC
>
> How do you start guix-daemon?
>
>
> > >(Before doing the third "pull", I did an "apt update" and "upgrade",
> > >but I
> > >wasn't that much behind in updates, I had done it less than a week ago
> > >last. My last "guix pull" was made at the same time as the "apt
> > >update", a
> > >few days ago.)
>
> What is your "guix describe"?
>
> >
> > >Hoping I can resolve this soon or that my feedback was positive for the
> > >development of Guix. =)
> > >I can provide log debug info (if there is any) upon request.
> > >
> > >Best regards!
> >
> > Sorry pastebin is not tor friendly, I couldn't look at it.
>
> Julien, here the paste:
>
> --8<---------------cut here---------------start------------->8---
> ./guix/store.scm:1035:9: ERROR:
>   1. &store-protocol-error:
>       message: "error parsing derivation
>
> `/gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv':
> expected string `Derive(['"
>       status: 1
> guix pull: error: You found a bug: the program
> '/gnu/store/kpxami25fi3mrxb37sfbbx2s366chpk5-compute-guix-derivation'
> failed to compute the derivation for Guix (version:
> "415b90f24ee83059102f5fb0ce74a2bc0954fe58"; system: "x86_64-linux";
> host version: "018cffc9c9e5a5855733f5f45a1c4d396bb6a321"; pull-version: 1).
> Please report it by email to <bug-guix@gnu.org>.
> --8<---------------cut here---------------end--------------->8---
>
> First time, I see that.
>
> All the best,
> simon
>

[-- Attachment #1.2: Type: text/html, Size: 4141 bytes --]

[-- Attachment #2: Screenshot from 2020-06-07 03-05-33.jpg --]
[-- Type: image/jpeg, Size: 942993 bytes --]

bug#41710: Possibly found a bug while doing a "guix pull"

[Ludovic Courtès]

Hi Léon,

Léon Lain Delysid <leon.lain.delysid@gmail.com> skribis:

> Hello, here's the result of the command you asked me to run. I executed it,
> then did a "guix pull" again and got the same result.

Did you try re-running ‘guix pull’ after the ‘guix gc -D’ command?  (I
don’t see it on the screenshot.)

> But the corrupted filesystem idea might be that one. Can you give me
> the best command to run a repair of the filesystem?

If the suggestion above is not above, then you could try running this as
root:

  guix build --repair \
    /gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv

Let me know how it goes.

HTH,
Ludo’.

bug#41710: Possibly found a bug while doing a "guix pull"

[Léon Lain Delysid]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

Actually, the biggest screenshot of the two was the end of the output of
the "guix pull" command after doing the "guix gc -D" command.

And the "build --repair" command resulted in this output:
guix build: error: error parsing derivation
`/gnu/store/8ablj2a1ihn32ilnfw1c3brpv9l7akzc-guile-json-4.0.1.drv':
expected string `Derive(['

So, that file was empty as
well: /gnu/store/8ablj2a1ihn32ilnfw1c3brpv9l7akzc-guile-json-4.0.1.drv
So I ran the "guix gc -D" on it, then "guix pull" again, and it found other
empty .drv's. I repeated this scheme until the "guix pull" command was
successful.

Thank you people so very much for your kind help!
My system is now bug free again. I don't know if I discovered real bugs, or
if my system crashing twice during a pull command somehow broke it, but I
hope this feedback helped. Maybe one day I will find out why my system
crashed and rebooted by itself a minute later. After all, it could have
been caused by something else on my system at that moment…

Have a good day! =)


On Sun, Jun 7, 2020 at 10:06 PM Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Léon,
>
> Léon Lain Delysid <leon.lain.delysid@gmail.com> skribis:
>
> > Hello, here's the result of the command you asked me to run. I executed
> it,
> > then did a "guix pull" again and got the same result.
>
> Did you try re-running ‘guix pull’ after the ‘guix gc -D’ command?  (I
> don’t see it on the screenshot.)
>
> > But the corrupted filesystem idea might be that one. Can you give me
> > the best command to run a repair of the filesystem?
>
> If the suggestion above is not above, then you could try running this as
> root:
>
>   guix build --repair \
>
> /gnu/store/m5djg8zwgl7nh0rm1av4nnlbwgwi0fwg-guix-daemon-1.1.0-4.bdc801e.drv
>
> Let me know how it goes.
>
> HTH,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2440 bytes --]

bug#41710: Possibly found a bug while doing a "guix pull"

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

Hi,

On Mon, 8 Jun 2020 01:15:46 +0200
Léon Lain Delysid <leon.lain.delysid@gmail.com> wrote:

>or
> if my system crashing twice during a pull command somehow broke it, 

Probably.

>but I
> hope this feedback helped. 

It sure helped.  It's good to know that that can happen.

I remember the first time I used Guix, I picked some file system that would
keep doing that: leave empty files if the system crashed (among lots of other
things).  And that system crashed a lot.  I had the same result as you,
and a lot of additional problems.

Back then we already improved a lot of places that were really really
important (added fsync calls), so the remaining places should be quite
harmless--like this one.  Because of Guix, you can always rebuild
/gnu/store just as it was--after a long build time maybe, but it's possible
(could be made a LOT more usable, though).

(fsync degrades performance, so it makes no sense to fsync for /gnu/store)

I think we can't really do more without imposing undue mainentance burden on
us (for something the file system shouldn't be doing in the first place),
or we could recommend another file system or different file system options
in the manual.  What would the latter be?

Also, how it the world didn't the file system checker fsck

(1) automatically run and
(2) fix this

in your case?

I mean what if those were user documents, which *couldn't* be restored so
easily?  Weird...

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#41710: Possibly found a bug while doing a "guix pull"

[Ludovic Courtès]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Mon, 8 Jun 2020 01:15:46 +0200
> Léon Lain Delysid <leon.lain.delysid@gmail.com> wrote:
>
>>or
>> if my system crashing twice during a pull command somehow broke it, 
>
> Probably.
>
>>but I
>> hope this feedback helped. 
>
> It sure helped.  It's good to know that that can happen.
>
> I remember the first time I used Guix, I picked some file system that would
> keep doing that: leave empty files if the system crashed (among lots of other
> things).  And that system crashed a lot.  I had the same result as you,
> and a lot of additional problems.
>
> Back then we already improved a lot of places that were really really
> important (added fsync calls), so the remaining places should be quite
> harmless--like this one.  Because of Guix, you can always rebuild
> /gnu/store just as it was--after a long build time maybe, but it's possible
> (could be made a LOT more usable, though).
>
> (fsync degrades performance, so it makes no sense to fsync for /gnu/store)
>
> I think we can't really do more without imposing undue mainentance burden on
> us (for something the file system shouldn't be doing in the first place),
> or we could recommend another file system or different file system options
> in the manual.  What would the latter be?
>
> Also, how it the world didn't the file system checker fsck
>
> (1) automatically run and
> (2) fix this
>
> in your case?

Yeah, that’s really weird.  I never experienced it first-hand, but it’s
not the first time we have such a report.

Ext4 & co. reportedly can leave empty files upon crashes; perhaps that’s
a problem with those file systems (though I’ve always used ext2/3/4 and
never had this problem myself, but that’s not statically significant).

Anyway, closing.

Thank you, Léon!

Ludo’.

bug#41710: Possibly found a bug while doing a "guix pull"

[Léon Lain Delysid]

[-- Attachment #1: Type: text/plain, Size: 2709 bytes --]

That's really weird. I thought EXT3/4 was the best of the best? If the
files were user documents, I would be pissed. Well, we can't have perfect
computer programs out of the box.
And then again, I did say my system crashed in a very violent way (twice).
I had only one window open: the terminal with the "guix pull" running, but
it might have been some other program running in the background that froze
my system. The command was running, and then all of a sudden, the whole
system froze. The screen was frozen, the mouse, and even the keyboard. Then
after a minute, it rebooted by itself. That's a pretty violent crash.

On Wed, Jun 10, 2020, 15:25 Ludovic Courtès <ludo@gnu.org> wrote:

> Hello,
>
> Danny Milosavljevic <dannym@scratchpost.org> skribis:
>
> > On Mon, 8 Jun 2020 01:15:46 +0200
> > Léon Lain Delysid <leon.lain.delysid@gmail.com> wrote:
> >
> >>or
> >> if my system crashing twice during a pull command somehow broke it,
> >
> > Probably.
> >
> >>but I
> >> hope this feedback helped.
> >
> > It sure helped.  It's good to know that that can happen.
> >
> > I remember the first time I used Guix, I picked some file system that
> would
> > keep doing that: leave empty files if the system crashed (among lots of
> other
> > things).  And that system crashed a lot.  I had the same result as you,
> > and a lot of additional problems.
> >
> > Back then we already improved a lot of places that were really really
> > important (added fsync calls), so the remaining places should be quite
> > harmless--like this one.  Because of Guix, you can always rebuild
> > /gnu/store just as it was--after a long build time maybe, but it's
> possible
> > (could be made a LOT more usable, though).
> >
> > (fsync degrades performance, so it makes no sense to fsync for
> /gnu/store)
> >
> > I think we can't really do more without imposing undue mainentance
> burden on
> > us (for something the file system shouldn't be doing in the first place),
> > or we could recommend another file system or different file system
> options
> > in the manual.  What would the latter be?
> >
> > Also, how it the world didn't the file system checker fsck
> >
> > (1) automatically run and
> > (2) fix this
> >
> > in your case?
>
> Yeah, that’s really weird.  I never experienced it first-hand, but it’s
> not the first time we have such a report.
>
> Ext4 & co. reportedly can leave empty files upon crashes; perhaps that’s
> a problem with those file systems (though I’ve always used ext2/3/4 and
> never had this problem myself, but that’s not statically significant).
>
> Anyway, closing.
>
> Thank you, Léon!
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 3414 bytes --]