bug#39727: statx error running 'guix gc' on CentOS 7

bug#39727: statx error running 'guix gc' on CentOS 7

[Paul Garlick]

Hi Guix,

After a 'guix pull' today to commit
536cc4aae5b58b45b974530646a4916a29a8aa6c I noticed that 'guix gc' fails
with the message:

guix gc: error: statting `/gnu/store/.links/0pck...': Invalid argument

The system is running CentOS 7:

$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

A temporary fix is to remove 'statx' from the list of functions checked
in 
config-daemon.ac (line 96):

-  statvfs nanosleep strsignal statx])
+ statvfs nanosleep strsignal])

This could a problem with the kernel version or coreutils [0] in CentOS
7.  

It seems that HAVE_STATX is set in the guix build process but then
runtime calls to statx generate errors.

Best regards,

Paul.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1760300

bug#39727: statx error running 'guix gc' on CentOS 7

[Ludovic Courtès]

Hi,

Paul Garlick <pgarlick@tourbillion-technology.com> skribis:

> After a 'guix pull' today to commit
> 536cc4aae5b58b45b974530646a4916a29a8aa6c I noticed that 'guix gc' fails
> with the message:
>
> guix gc: error: statting `/gnu/store/.links/0pck...': Invalid argument

This was during the “removing unused link” phase, right?

> The system is running CentOS 7:
>
> $ cat /etc/centos-release
> CentOS Linux release 7.7.1908 (Core)

What does “uname -r” return?

This is most likely an issue with the ancient kernel being used.

It would be nice if you could try running a C program that does
something like this:

	struct statx st;
	if (statx(AT_FDCWD, "/",
		  AT_SYMLINK_NOFOLLOW | AT_STATX_DONT_SYNC,
		  STATX_SIZE | STATX_NLINK, &st) == -1)
          printf ("failed: %m\n");

It should fail similarly.  Then you can try commenting out
AT_STATX_DONT_SYNC and see whether it fails.

Let me know how it goes!

Thanks,
Ludo’.

bug#39727: statx error running 'guix gc' on CentOS 7

[Paul Garlick]

Hi Ludo,

> This was during the “removing unused link” phase, right?
> 
Yes, that's it.

> The system is running CentOS 7:
> 
> $ cat /etc/centos-release
> CentOS Linux release 7.7.1908 (Core)
> 
> What does “uname -r” return?

3.10.0-1062.12.1.el7.x86_64

> Let me know how it goes!

I compiled a C program with the following code using gcc-toolchain
9.2.0:

#define _GNU_SOURCE
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>

int main(void)
{
  struct statx st;
  if (statx(AT_FDCWD, "/",
            AT_SYMLINK_NOFOLLOW | AT_STATX_DONT_SYNC,
            /* AT_SYMLINK_NOFOLLOW, */
            STATX_SIZE | STATX_NLINK, &st) == -1)
    printf ("failed: %m\n");

  return 0;
}

Initially, with the AT_STATX_DONT_SYNC flag, the output is:

$ failed: Invalid argument

Then, without the  AT_STATX_DONT_SYNC flag, the program runs and there
is no output.

Best regards,

Paul.

bug#39727: statx error running 'guix gc' on CentOS 7

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

Hi Paul,

Paul Garlick <pgarlick@tourbillion-technology.com> skribis:

> Initially, with the AT_STATX_DONT_SYNC flag, the output is:
>
> $ failed: Invalid argument
>
> Then, without the  AT_STATX_DONT_SYNC flag, the program runs and there
> is no output.

Great.  Could you apply the following patch, run the daemon with:

  sudo -E ./pre-inst-env guix-daemon --build-users-group=…

then run:

  guix gc -C42

and confirm that it works for you?

Thanks!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1233 bytes --]

diff --git a/nix/libstore/gc.cc b/nix/libstore/gc.cc
index 77d7fa2dc7..8bc4e01eb0 100644
--- a/nix/libstore/gc.cc
+++ b/nix/libstore/gc.cc
@@ -581,15 +581,27 @@ void LocalStore::removeUnusedLinks(const GCState & state)
 #ifdef HAVE_STATX
 # define st_size stx_size
 # define st_nlink stx_nlink
+	static int statx_flags = AT_SYMLINK_NOFOLLOW | AT_STATX_DONT_SYNC;
 	struct statx st;
-	if (statx(AT_FDCWD, path.c_str(),
-		  AT_SYMLINK_NOFOLLOW | AT_STATX_DONT_SYNC,
-		  STATX_SIZE | STATX_NLINK, &st) == -1)
+
+	if (statx(AT_FDCWD, path.c_str(), statx_flags,
+		  STATX_SIZE | STATX_NLINK, &st) == -1) {
+	    if (errno == EINVAL) {
+		/* Old 3.10 kernels (CentOS 7) don't support
+		   AT_STATX_DONT_SYNC, so try again without it.  */
+		statx_flags &= ~AT_STATX_DONT_SYNC;
+		if (statx(AT_FDCWD, path.c_str(), statx_flags,
+			  STATX_SIZE | STATX_NLINK, &st) == -1)
+		    throw SysError(format("statting `%1%'") % path);
+	    } else {
+		throw SysError(format("statting `%1%'") % path);
+	    }
+	}
 #else
         struct stat st;
         if (lstat(path.c_str(), &st) == -1)
-#endif
             throw SysError(format("statting `%1%'") % path);
+#endif
 
         if (st.st_nlink != 1) {
             actualSize += st.st_size;

bug#39727: statx error running 'guix gc' on CentOS 7

[Paul Garlick]

Hi Ludo,


> Great.  Could you apply the following patch, run the daemon with:
> 
>   sudo -E ./pre-inst-env guix-daemon --build-users-group=…
> 
> then run:
> 
>   guix gc -C42

Yes, all is good.  

I have re-built guix with your patch and started the daemon.  Now 'guix
gc' runs as expected:

.
.
.
deleted or invalidated more than 42 bytes; stopping
deleting `/gnu/store/trash'
deleting unused links...
note: currently hard linking saves 3264.68 MiB
guix gc: freed 0.91850 MiBs

Many thanks!  CentOS 7 will be around for a few years yet.

Best regards,

Paul.

bug#39727: statx error running 'guix gc' on CentOS 7

[Ludovic Courtès]

Hi Paul,

Paul Garlick <pgarlick@tourbillion-technology.com> skribis:

>> Great.  Could you apply the following patch, run the daemon with:
>> 
>>   sudo -E ./pre-inst-env guix-daemon --build-users-group=…
>> 
>> then run:
>> 
>>   guix gc -C42
>
> Yes, all is good.  
>
> I have re-built guix with your patch and started the daemon.  Now 'guix
> gc' runs as expected:

Great, pushed as 513c0a0f4602018a49d8fd2dfa24670a3fa08ac9.

Thanks,
Ludo’.