unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* Test failures with 1.7.0
@ 2021-12-08  1:07 Julien Moutinho
  2021-12-08  4:08 ` Eric Wong
  0 siblings, 1 reply; 8+ messages in thread
From: Julien Moutinho @ 2021-12-08  1:07 UTC (permalink / raw)
  To: meta

Hi!

I'm trying to package the new 1.7.0 public-inbox into NixOS
at https://github.com/NixOS/nixpkgs/pull/104457
but two reviewers are facing different test failures
that I cannot reproduce:

One reviewer and a bot fail with:
> t/lei_to_mail.t .............. 1/? Use of uninitialized value in open at t/lei_to_mail.t line 263.
> Bailout called.  Further testing stopped:  No such file or directory
> FAILED--Further testing stopped: No such file or directory

And another reviewer fails 3 times with:
> t/extsearch.t              (Wstat: 2048 Tests: 145 Failed: 8)
>   Failed tests:  68-69, 75-76, 86, 98, 102, 139
>   Non-zero exit status: 8
> t/imapd.t                  (Wstat: 256 Tests: 186 Failed: 1)
>   Failed test:  183
>   Non-zero exit status: 1
> t/nntpd.t                  (Wstat: 256 Tests: 110 Failed: 1)
>   Failed test:  104
>   Non-zero exit status: 1
> Files=149, Tests=4730, 107 wallclock secs ( 1.14 usr  0.21 sys + 50.13 cusr 54.98 csys = 106.46 CPU)
> Result: FAIL
> Failed 3/149 test programs. 10/4730 subtests failed.

Nix being Nix we can assume the exact same code is used for all,
but our systems are different (eg. CPU(s), filesystem(s), etc.)

Could you provide some help?
Thanks

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Test failures with 1.7.0
  2021-12-08  1:07 Test failures with 1.7.0 Julien Moutinho
@ 2021-12-08  4:08 ` Eric Wong
  2021-12-08 10:56   ` Dominique Martinet
  0 siblings, 1 reply; 8+ messages in thread
From: Eric Wong @ 2021-12-08  4:08 UTC (permalink / raw)
  To: Julien Moutinho; +Cc: meta

Julien Moutinho <julm+public-inbox@sourcephile.fr> wrote:
> Hi!
> 
> I'm trying to package the new 1.7.0 public-inbox into NixOS
> at https://github.com/NixOS/nixpkgs/pull/104457
> but two reviewers are facing different test failures
> that I cannot reproduce:
> 
> One reviewer and a bot fail with:
> > t/lei_to_mail.t .............. 1/? Use of uninitialized value in open at t/lei_to_mail.t line 263.
> > Bailout called.  Further testing stopped:  No such file or directory
> > FAILED--Further testing stopped: No such file or directory
> 
> And another reviewer fails 3 times with:
> > t/extsearch.t              (Wstat: 2048 Tests: 145 Failed: 8)
> >   Failed tests:  68-69, 75-76, 86, 98, 102, 139
> >   Non-zero exit status: 8
> > t/imapd.t                  (Wstat: 256 Tests: 186 Failed: 1)
> >   Failed test:  183
> >   Non-zero exit status: 1
> > t/nntpd.t                  (Wstat: 256 Tests: 110 Failed: 1)
> >   Failed test:  104
> >   Non-zero exit status: 1

Sorry for the problems....

I wonder if it's a missing dependency that the tests forget to
account for...  Can they run the tests individually using "prove"
and show more output?

   e.g.: make && prove -bvw t/nntpd.t

Also, "./lei.sh sucks"  will dump the relevant deps+versions
(either Inline::C or Socket::MsgHdr is required for lei to work)

> Nix being Nix we can assume the exact same code is used for all,
> but our systems are different (eg. CPU(s), filesystem(s), etc.)

Perhaps you guys can compare installed package lists easily and
help narrow it down?  There's a lot of optional stuff in
public-inbox since we try to support some old systems and users
who don't want extra dependencies; but yes, it gets difficult to
support so many possible combinations.

> Could you provide some help?

Hope the above was helpful, in any case I should be around-ish
the next few days.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Test failures with 1.7.0
  2021-12-08  4:08 ` Eric Wong
@ 2021-12-08 10:56   ` Dominique Martinet
  2021-12-08 18:22     ` [PATCH] nodatacow: quiet chattr errors [was: Test failures with 1.7.0] Eric Wong
  2021-12-09  1:37     ` Test failures with 1.7.0 Julien Moutinho
  0 siblings, 2 replies; 8+ messages in thread
From: Dominique Martinet @ 2021-12-08 10:56 UTC (permalink / raw)
  To: Eric Wong; +Cc: Julien Moutinho, meta

Hi,

Eric Wong wrote on Wed, Dec 08, 2021 at 04:08:36AM +0000:
> > And another reviewer fails 3 times with:
> > > t/extsearch.t              (Wstat: 2048 Tests: 145 Failed: 8)
> > >   Failed tests:  68-69, 75-76, 86, 98, 102, 139
> > >   Non-zero exit status: 8
> > > t/imapd.t                  (Wstat: 256 Tests: 186 Failed: 1)
> > >   Failed test:  183
> > >   Non-zero exit status: 1
> > > t/nntpd.t                  (Wstat: 256 Tests: 110 Failed: 1)
> > >   Failed test:  104
> > >   Non-zero exit status: 1

(I'm that one)

So instead of giving the output of prove -bvw t/nntpd.t (I couldn't
reproduce manually..), I'll go straight to the solution: it looks like
there's a problem with the chattr fallback when btrfs is present on the
machine (to disable cow, lib/PublicInbox/NDC_PP.pm)

I had initial messages about chattr not being found; adding chattr to
the build inputs gave another error message that chattr +C doesn't work
on tmpfs.
The problem here seems to be that the fallback here just checks
/proc/self/mounts and calls chattr "just in case", saying it's ok if it
fails, but some tests probably rely on stderr output being empty?

If so it looks more like a test problem than anything else to me, but
perhaps the fallback path could be a bit more prudent in its calling of
chattr (or simply try silencing its output?)

For testing, being a bit forceful and removing that chattr call made all
the tests pass for me.


Now the $100 question is I don't know why the Inline::C version wasn't
used; this one properly calls statfs() and does the ioctl directly only
if required so would have worked.
As far as I can see we're installing Inline::C in the build chroot, I
see it in the PERL5LIB paths; and make/gcc are also available so it
probably should work. Is there a way to check that?


> Sorry for the problems....
> 
> I wonder if it's a missing dependency that the tests forget to
> account for...  Can they run the tests individually using "prove"
> and show more output?
> 
>    e.g.: make && prove -bvw t/nntpd.t
> 
> Also, "./lei.sh sucks"  will dump the relevant deps+versions
> (either Inline::C or Socket::MsgHdr is required for lei to work)

Here's what this gives me:
---
lei 1.7.0
perl 5.34.0 / Linux 5.10.81 / x86_64 ptrsize=8 nproc=4
git 2.33.1 / JSON::PP 4.06
SQLite 3.32.3, DBI 1.643, DBD::SQLite 1.66
Xapian 1.4.18, bindings: Search::Xapian 1.2.25.4
public-inbox blob OIDs of loaded features:
  2c9c4395c02edbb86683573612f5b454115ce719 PublicInbox/Address.pm
  0f002e5e62c260fbb5078bbbb25c67b1468fbbec PublicInbox/Config.pm
  bacc9cdda12498abbb0ada5d2a2e2faec10190f2 PublicInbox/ContentHash.pm
  bf8c4466218a37a43688aaaca2b063c65e98c86e PublicInbox/DS.pm
  9206da9cb710186462ffefd10cb4af0764f7e615 PublicInbox/DirIdle.pm
  485f637a3e7b41a9117cb37a9acfd8e7649dadfa PublicInbox/Eml.pm
  80fc7364fc9a1943759585c30487ab0c5d36ca0c PublicInbox/EmlContentFoo.pm
  309f80dbfcf001e51f130cddee1a14ee38303dd3 PublicInbox/Git.pm
  3e299448b334629fb6427e7df02976381ec19242 PublicInbox/IPC.pm
  60ce7b66419b3ee0e9ad2dca4b93644497539378 PublicInbox/Import.pm
  ffe26a44ab3d9e38d46447fd69543e0a9d4e5c8d PublicInbox/In2Tie.pm
  1579d500136a8f814f532275f486bc117c3e0441 PublicInbox/Inbox.pm
  887025de5dedb608576b2263b4f20c0596fce8e5 PublicInbox/LEI.pm
  30bb1a4579c74d8245d1eec7b66b9687bd526e71 PublicInbox/LeiExternal.pm
  fa0e78667b3b051cdf3dda96d0b8fba24e1f19f2 PublicInbox/LeiHelp.pm
  352ee60131aaf6bc9baabfbd4ae735c4a0f06568 PublicInbox/LeiQuery.pm
  8e866fc96655a9dd5a6342de43217da0a57e14f5 PublicInbox/LeiSucks.pm
  7cedc3493f872df866755e1c2c9d5d545df7c319 PublicInbox/Listener.pm
  0ee2a8bd60bd964a13c0c24f958f43e0ca9a6b5b PublicInbox/Lock.pm
  f82194a347b867aa22d416b75576e3ac624527ee PublicInbox/MDA.pm
  35b517e09f5d12ff264bcd0bd8afe77f1b96e69d PublicInbox/MID.pm
  dd28417b70c043b092f231147d82580ba94e31cf PublicInbox/MsgIter.pm
  5ee087fd50fea18612798d094046b4ada12d16e1 PublicInbox/MsgTime.pm
  615bc450998beea1e82606db35c0d456450d966b PublicInbox/OnDestroy.pm
  30ad949dd027153f30835f928c962d00e5d9f82a PublicInbox/Over.pm
  4c434566d31f90104d921b76adf1844a9d6aefde PublicInbox/PktOp.pm
  97e9c268f8d1e91aa0f8c1225112dc42cd623768 PublicInbox/ProcessPipe.pm
  523003b3c269ae42e9ee210e91415b2846735daa PublicInbox/Search.pm
  81e5a1b1dd88e99d25bf37aadbb1f520e8a6503f PublicInbox/Sigfd.pm
  260ce6bb065e930adfd9030d50f5a06359e809ce PublicInbox/Smsg.pm
  6ca1ca2a0be3538b3cb9dc663422698b6b94952b PublicInbox/Spawn.pm
  c00385b94db84b63facf7a8d57296ac76b3b1421 PublicInbox/Syscall.pm
  3040dd77d457d02f4517088edcad4f4cf858228f PublicInbox/Tmpfile.pm
  950bd17052a569b7e6792f875791d801525d821d PublicInbox/WQWorker.pm
Let us know how it sucks!  Please include the above and any other
relevant information when sending plain-text mail to us at:
meta@public-inbox.org -- archives: https://public-inbox.org/meta/
---

> > Nix being Nix we can assume the exact same code is used for all,
> > but our systems are different (eg. CPU(s), filesystem(s), etc.)
> 
> Perhaps you guys can compare installed package lists easily and
> help narrow it down?  There's a lot of optional stuff in
> public-inbox since we try to support some old systems and users
> who don't want extra dependencies; but yes, it gets difficult to
> support so many possible combinations.

That's the thing with nixos, in theory the package list really should be
identical, so we're down to environmental differences (like me and btrfs
present in /proc/self/mounts).
The build happens in a chroot with just the "input" packages present
(which also incidentally makes it somewhat harder to debug, I couldn't
reproduce my error manually...).


> > One reviewer and a bot fail with:
> > > t/lei_to_mail.t .............. 1/? Use of uninitialized value in open at t/lei_to_mail.t line 263.
> > > Bailout called.  Further testing stopped:  No such file or directory
> > > FAILED--Further testing stopped: No such file or directory

(waiting to see if the ones who produce this can give more details about
their environment)


Thanks,
-- 
Dominique Martinet | Asmadeus

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH] nodatacow: quiet chattr errors [was: Test failures with 1.7.0]
  2021-12-08 10:56   ` Dominique Martinet
@ 2021-12-08 18:22     ` Eric Wong
  2021-12-08 21:14       ` Dominique Martinet
  2021-12-09  1:37     ` Test failures with 1.7.0 Julien Moutinho
  1 sibling, 1 reply; 8+ messages in thread
From: Eric Wong @ 2021-12-08 18:22 UTC (permalink / raw)
  To: Dominique Martinet; +Cc: Julien Moutinho, meta

Dominique Martinet <asmadeus@codewreck.org> wrote:
> Eric Wong wrote on Wed, Dec 08, 2021 at 04:08:36AM +0000:
> > > And another reviewer fails 3 times with:
> > > > t/extsearch.t              (Wstat: 2048 Tests: 145 Failed: 8)
> > > >   Failed tests:  68-69, 75-76, 86, 98, 102, 139
> > > >   Non-zero exit status: 8
> > > > t/imapd.t                  (Wstat: 256 Tests: 186 Failed: 1)
> > > >   Failed test:  183
> > > >   Non-zero exit status: 1
> > > > t/nntpd.t                  (Wstat: 256 Tests: 110 Failed: 1)
> > > >   Failed test:  104
> > > >   Non-zero exit status: 1
> 
> (I'm that one)
> 
> So instead of giving the output of prove -bvw t/nntpd.t (I couldn't
> reproduce manually..), I'll go straight to the solution: it looks like
> there's a problem with the chattr fallback when btrfs is present on the
> machine (to disable cow, lib/PublicInbox/NDC_PP.pm)

Hi, thanks for the extra info.

Note: You might want to rm -rf ~/.cache/public-inbox/inline-c or
try a different HOME directory to get a clean repro (more on
this later).

I also forget to note, setting TAIL='tail -F' can dump daemon
output for tests (works better for GNU tail w/ inotify support).

> I had initial messages about chattr not being found; adding chattr to
> the build inputs gave another error message that chattr +C doesn't work
> on tmpfs.
> The problem here seems to be that the fallback here just checks
> /proc/self/mounts and calls chattr "just in case", saying it's ok if it
> fails, but some tests probably rely on stderr output being empty?

Yes on tests requiring stderr to be empty.  Below is a patch
which should fix it; however it should only be calling chattr on
btrfs mounts.

So I wonder if our /proc/self/mounts parsing is broken...
Can you share your contents of /proc/self/mounts?
(maybe spaces/odd chars aren't handled properly...)

You can also try:

  BTRFS_TESTDIR=/path/to/your/btrfs-mount prove -bvw t/nodatacow.t

?

> If so it looks more like a test problem than anything else to me, but
> perhaps the fallback path could be a bit more prudent in its calling of
> chattr (or simply try silencing its output?)

Agreed, the patch below silences stderr.

> For testing, being a bit forceful and removing that chattr call made all
> the tests pass for me.

> Now the $100 question is I don't know why the Inline::C version wasn't
> used; this one properly calls statfs() and does the ioctl directly only
> if required so would have worked.
> As far as I can see we're installing Inline::C in the build chroot, I
> see it in the PERL5LIB paths; and make/gcc are also available so it
> probably should work. Is there a way to check that?

For public-inbox-* stuff, Inline::C isn't enabled by default
even if installed.   Either ~/.cache/public-inbox/inline-c needs
to exist or PERL_INLINE_DIRECTORY must be set in env.

However, lei will auto-create ~/.cache/public-inbox/inline-c
if Socket::MsgHdr is missing (since lei is new w/ no existing users).
Thus running lei will enable Inline::C for public-inbox-* commands.

<snip>

Thanks again for all the extra info.  Hopefully this below is
one step towards making things work more smoothly, but I also
suspect /proc/self/mount parsing is off...

------8<-----
Subject: [PATCH] nodatacow: quiet chattr errors

Maybe this helps when chattr is missing or our /proc/self/mount
parsing is broken, somehow.

Link: https://public-inbox.org/meta/YbCPWGaJEkV6eWfo@codewreck.org/
---
 lib/PublicInbox/NDC_PP.pm | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/lib/PublicInbox/NDC_PP.pm b/lib/PublicInbox/NDC_PP.pm
index 57abccbe0625..a84682d371c8 100644
--- a/lib/PublicInbox/NDC_PP.pm
+++ b/lib/PublicInbox/NDC_PP.pm
@@ -16,11 +16,13 @@ sub nodatacow_dir ($) {
 		# weird chars are escaped as octal
 		$mnt_path =~ s/\\(0[0-9]{2})/chr(oct($1))/egs;
 		$mnt_path .= '/' unless $mnt_path =~ m!/\z!;
-		if (index($path, $mnt_path) == 0) {
-			# error goes to stderr, but non-fatal for us
-			system('chattr', '+C', $path);
-			last;
-		}
+		next if index($path, $mnt_path) < 0;
+
+		open my $olderr, '>&', \*STDERR or die "dup(STDERR): $!";
+		open STDERR, '>', '/dev/null' or die "open(/dev/null): $!";
+		system(qw(chattr +C), $path); # ignore error
+		open STDERR, '>&', $olderr or die "dup2(..,STDERR): $!";
+		last;
 	}
 }
 

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] nodatacow: quiet chattr errors [was: Test failures with 1.7.0]
  2021-12-08 18:22     ` [PATCH] nodatacow: quiet chattr errors [was: Test failures with 1.7.0] Eric Wong
@ 2021-12-08 21:14       ` Dominique Martinet
  2021-12-08 22:01         ` Dominique Martinet
  0 siblings, 1 reply; 8+ messages in thread
From: Dominique Martinet @ 2021-12-08 21:14 UTC (permalink / raw)
  To: Eric Wong; +Cc: Julien Moutinho, meta

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]

Eric Wong wrote on Wed, Dec 08, 2021 at 06:22:47PM +0000:
> Note: You might want to rm -rf ~/.cache/public-inbox/inline-c or
> try a different HOME directory to get a clean repro (more on
> this later).

Ah, the builder does not have a home ($HOME points to a non-existing
directory), so that'd explain why inline-c isn't used for tests.

I'll try giving it one, in my opinion it's more representative to test
with inline-c working.

> I also forget to note, setting TAIL='tail -F' can dump daemon
> output for tests (works better for GNU tail w/ inotify support).

Thanks!

> > I had initial messages about chattr not being found; adding chattr to
> > the build inputs gave another error message that chattr +C doesn't work
> > on tmpfs.
> > The problem here seems to be that the fallback here just checks
> > /proc/self/mounts and calls chattr "just in case", saying it's ok if it
> > fails, but some tests probably rely on stderr output being empty?
> 
> Yes on tests requiring stderr to be empty.  Below is a patch
> which should fix it; however it should only be calling chattr on
> btrfs mounts.

I'll give this a try as well.

> So I wonder if our /proc/self/mounts parsing is broken...
> Can you share your contents of /proc/self/mounts?
> (maybe spaces/odd chars aren't handled properly...)

It's a tad big, I've put it down as attachment.

I had missed the `index($path, $mnt_path) == 0` check of the script,
so I agree it should only be called on a btrfs mount...
But the problem in my case is that / is btrfs, so it will always match,
even if the build itself happens in /build (which is a tmpfs)

So the check doesn't just need to check if it matches, but it must be
the longest match's type that is considered e.g. loop through all the
entries, remember the longest match's type, and then check if that is
btrfs.

> You can also try:
> 
>   BTRFS_TESTDIR=/path/to/your/btrfs-mount prove -bvw t/nodatacow.t

I'll try something similar as well.

It turns out the other people with failing test are both using zfs, so
the last problem might be somewhat related; I'll give that a try too.


I'll send another reply when I'm done, hopefully before work :)

-- 
Dominique

[-- Attachment #2: proc mounts file --]
[-- Type: text/plain, Size: 41817 bytes --]

/dev/dm-2 / btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/05c1kr7hvjy8igz0m1msjg6qfplj2wz7-perl5.34.0-HTTP-Cookies-6.09 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/091dni27c2w1d8y248d8jr8y1ll3jdk2-perl5.34.0-Email-MIME-1.949 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0c7k0znjz1j67vyh7rkas655daxh0dd5-git-2.33.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0cnvyl1npancz3smm2v1qigp1f1lif58-perl5.34.0-Plack-Middleware-ReverseProxy-0.16 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0cz35r5126dziwxz6cs6rqh3lcmb81ig-perl5.34.0-Module-Implementation-0.09 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0dbkbza24kinyv0sb21hdmd0n4cs7vly-perl5.34.0-Params-ValidationCompiler-0.30 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0f7w6agnx47hq22drpy6bd0ha455hfp8-perl5.34.0-IO-Handle-Util-0.02 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0fi0432kdh46x9kbngnmz2y7z0q68cdz-xz-5.2.5-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0grbkshm5cqa6bpx80xk3nrnkji9fsnw-perl5.34.0-Cookie-Baker-0.11 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0hw94rkaqg2l23ajh20qyy4l5qdvnsz7-zstd-1.5.0-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0j6sw51j043rvrv6csg9wkxzjslj8bcj-hook btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0q85a7ygm5s6ym7phb2vnvmp7fgnfilr-perl5.34.0-Email-MIME-ContentType-1.024 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/0rizskpri8d8qawx6qjqcnvlxcvzr1bm-keyutils-1.6.3-lib btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/1hdx7bxjwqrfnalalkgbwi32l45h8z7b-perl5.34.0-Mail-IMAPClient-3.42 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/1l4r0r4ab3v3a3ppir4jwiah3icalk9d-zlib-1.2.11 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/1m35q3vym6v2wpfm6drlay2hkaqyc3bb-nghttp2-1.43.0-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/1yywwq3awjfvrpv3yfa4rvl27zmp9nij-perl5.34.0-HTML-Parser-3.75 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/1zcjxz5z0wvm43z2bq8vziwf4d9l01cr-perl5.34.0-Stream-Buffered-0.03 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/1zsj6v2ry3kirgfdr1qqjsg90ws4f5fm-tzdata-2021c btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/29m4iqpk6s03ld2lz2ydnr254k3jgv5d-gettext-0.21 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/2hgrzcmdn8rsd3v1axxd9c1rklbll6va-git-2.33.1-doc btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/2kl7mfz0ii381n58i719qrvk7rfqb49i-perl5.34.0-MIME-Types-2.18 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/2qb32vc3g91rvv83myywzzrgmgaimf03-openssh-8.8p1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/2s6k3rvk7x936218b32n04kl385m1w8x-libssh2-1.10.0-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/2zs5kkw0lwivr5rs2qzix0iwzgmxxf9v-perl5.34.0-Sub-Install-0.928 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/3g6fhn6pplnfjz8m4mzg500sj2y86fzp-linux-pam-1.5.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/3gq0lgn8620kw2j9axbgxcn1nb4jyizn-xapian-1.4.18 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/3ibnw61rlgj2lj5hycy2dn3ybpq7wapm-libev-4.33 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/3igs7wc8wys0n5w53binlz3af7qhidhj-perl5.34.0-Pegex-0.75 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/3q65fybq3dlzzy0jjabncg0f9vw8dc9k-perl5.34.0-DBI-1.643 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/4kx13qnwgwyb1zq57x0gnvzvdaq91m5r-curl-7.79.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/4nkxq7xgqpyq2v2zd1p25bq7f6kck8l1-perl5.34.0-Module-Runtime-0.016 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/4nw7qp9mhx4yfgga66664717qnkb96yv-util-linux-2.37.2-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/545xan6vhb42f39rjn9ni7rhmpqv8n37-perl5.34.0-HTTP-Request-AsCGI-1.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/55icsrszy4idkdwjqh7nvr9b3qnlgwaz-perl5.34.0-Encode-Locale-1.05 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/5ana57yikvgq34zbfdn6khldr09pm0aq-perl5.34.0-CGI-4.51 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/5jpkzx56n4p9aspnz00dsil44bxv13hb-perl5.34.0-IO-Tty-1.15 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/5vzawr6hfi91c8ndf7cj0lfisjn3mckr-perl5.34.0-Sub-Uplevel-0.2800 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/5xwvhw1maymf68qp44mhyvcykprxc5cx-expand-response-params btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/5ylz3dd6fdch62jznchcg85d4p5jmv2y-perl5.34.0-LWP-MediaTypes-6.04 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/61wgnz3xxfmabwxay9gpy200y07xqipl-perl5.34.0-Eval-Closure-0.14 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/649xkl75pcg9iwcfrq7bcpm459s5d0nx-curl-7.79.1-man btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/65jw2h2al04lnwx38r06n2jjbh96ah32-busybox-static-x86_64-unknown-linux-musl-1.34.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/67x6kxbanrqafx3hg7pb3bc83i3d1v3f-gzip-1.11 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/6h32ddjbnjng7gbdla7ag3hmxygm0wq5-perl5.34.0-common-sense-3.75 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/6irxz4fbf1d1ac7wvdjf8cqb3sgmnvg8-zlib-1.2.11-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/6j67z3vp0b89xvklfxgghp6va1xzn5lm-perl5.34.0-Data-OptList-0.110 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/6kgaicmynv53fa6cjabixp4a11girx5d-perl5.34.0-Test-Needs-0.002006 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/6kgfmzx90c1a6afqnbkz6qprkzss476k-mime-types-9 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/6s81ck7vrma1i6pbxxpdy2324xhz1py7-perl5.34.0-IPC-Run-20200505.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/6znzq5yfa9g004acgiz1gmd0kwd399qs-audit-2.8.5 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/71pachqc22wlvf3xjhwjh2rqbl6l3ngg-diffutils-3.8 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/724iipkqbcrh4gaiv8xi4crwckj3lzs8-perl5.34.0-Linux-Inotify2-2.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/72vdginhi637lxyii5fnpmvf4jzc4gbk-ncurses-6.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/7fpwn95ial009l1y8zy1jvswjzcn0nff-perl5.34.0-Sub-Exporter-Progressive-0.001013 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/7kw4fpy721jv2jr5ili6kyv3nqrxylyx-perl5.34.0-LWP-Protocol-http10-6.03 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/7lacd75rxv7mqbrm44xhclx3da7h199b-perl5.34.0-IO-HTML-1.004 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/7n91vqn1kb6vrk11sr30wca9qayis5s1-lua-5.2.4 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/7rfaw11na5ajdgwr55ffzwfibbrdpk8z-glibc-2.33-56-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/7vq4ail5lyw7n9qrmwf3y5879fdsjb4j-perl5.34.0-Plack-1.0048 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/7z5ai3qarfa3xvw0rfpy9dnqqilwjppf-binutils-wrapper-2.35.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/8334m6k80h0fd5acqkmp8hfrcgagm684-perl5.34.0-URI-5.05 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/84k3qb7hpv231p52zmrsxd4x186vc0zm-libkrb5-1.18 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/8jqps5s2wbxqlwiq7wnv2vhh4148q3ak-perl5.34.0-Test-Requires-0.11 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/8vdbp4jzbj82d8c4wlqd4pxrxirf3mnd-perl5.34.0-Any-URI-Escape-0.01 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/90y23lrznwmkdnczk1dzdsq4m35zj8ww-bash-interactive-5.1-p8 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/93igc1gfl7zw6yayxvai014h125cw0x0-perl5.34.0-Package-Stash-0.39 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/9hhv4qc4fb07jr0cfnif3fcym397xp4h-perl5.34.0-TermReadKey-2.38 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/9m6wkb91sjifc50m1fdgky9abnhk75zm-systemd-minimal-249.5 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/9mp06ni69a44dmrjhn28mn15brdry52w-gnused-4.8 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/9ppi191zsi7zvynkm8vy2bi22lci9iwg-bzip2-1.0.6.0.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/9xlk6zzmcf9yh89pva7ffza3xk6zwvag-libfido2-1.8.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ai4y385z26ka668xhg21g3bpprmk32cs-perl5.34.0-FCGI-ProcManager-0.28 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/aq5k7hrbq50mry8xlxx71b08m80vcpjr-perl5.34.0-DBD-SQLite-1.66 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/aqkrqs09i9wzrr5s2pfr96yhnyw1laqg-perl5.34.0-HTTP-Negotiate-6.01 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/b0436zyxfg3n29vya7rbps8cchxh6ncj-libidn-1.38 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/baacy0v9iz6m045hnffa0fi0zr6wnnzz-openssl-1.1.1l-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/bl8czy1y3qb9j3phbrqklg8j2r6p8i1i-perl5.34.0-Test-MockTime-HiRes-0.08 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/bmv164bdc1dmw67pa2bbh6rai4ra9w73-perl5.34.0-Dist-CheckConflicts-0.11 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/bvn90wp9d97ay7z0i3rfkp2kjdxcv7jl-perl5.34.0-XXX-0.35 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/byixsnjgvzxy9rx0jadwjdyzs0rj75g5-zstd-1.5.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/byzidx23xdyp1why6h4hkq3812bgf243-perl5.34.0-FCGI-0.79 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/bzg7k9pp9jifybp7128bvm1szyhg5hdb-perl5.34.0-CPAN-Meta-Check-0.014 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/c1v3ghcmqplp79dpialpx8v9bnmx3j34-perl5.34.0-Try-Tiny-0.30 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/c5bd4lrnsck51cll6s14wkp93ni4zj1j-glibc-2.33-56-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/c88mwrfh761csp2n74ddgrxl6706amid-readline-8.1p0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/c9f15p1kwm0mw5p13wsnvd1ixrhbhb12-gcc-10.3.0-lib btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ca8bls3hlz043czn2myhp6lh8mah89vq-hook btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/cdkpv9b9ln34050yi4j8f3y8fsl9ra7r-nghttp2-1.43.0-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/chddm1kigcgj62qkps2dhvc0g2yqrf8j-perl5.34.0-HTML-Tagset-3.20 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/d0aj1lpk19jns5s3qnnk79w23mmqvdmy-libpipeline-1.5.3 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/d27pv0vwsawvdgwafsras3zfv06barpq-perl5.34.0-MRO-Compat-0.13 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/d65wn83yzlxdl36k6vs1mnhycyrmnggh-kbd-2.4.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/d78xlj699g6nnnn3qwa7d9x1cfivyakj-libidn-1.38-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/d9nqdg99263q2hdj3chgq58ryzsxrx0d-perl5.34.0-Exception-Class-1.44 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/da9l6ifizmh883j94gzx5n51h8lrrp00-gcc-wrapper-10.3.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/dcb3y8l0zc2xv2crxdsfxfll9xadsp77-pcre2-10.37 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/f2m83ypabavw5ria5fqzf56gcjg42qi8-perl5.34.0-Class-Inspector-1.36 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/f3vr8qxbqzv81mdyd47nxcmp4bjbnmwf-perl5.34.0-HTTP-Server-Simple-PSGI-0.16 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/fa793x4d84wslc2pj9spq1gc5j3b1sl1-perl5.34.0-namespace-clean-0.27 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/fic8j6k9hsgx3f89kavlkc4jqqn51s8l-db-5.3.28 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/fijrw5ir2ajkx55sxy3mwxr3p3f7p5vm-perl5.34.0-Apache-LogFormat-Compiler-0.36 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/fjiwamxwjifspds1cy0q9fk10cnw55qh-perl5.34.0-libwww-perl-6.49 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/fl56aypsak2a8ka4khsafacqvm742jn6-perl5.34.0-Log-Dispatch-2.70 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/fnvnllcgsxzl5w99sil57nfjkjypyhrd-perl5.34.0-HTML-TagCloud-0.38 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/fvprxgcxf4px865gdjd81fbwnxcjrg41-coreutils-9.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/g0bvs0w27ijhw6s1lqgpscg54k4spvcr-perl5.34.0-Sub-Exporter-0.987 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/g46cby42hjz6ksbwikmwqw1ha2nb6ngp-perl5.34.0-HTTP-Entity-Parser-0.25 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/gcy03zd1c0xjzmzv30hxapf0nlbq5s1c-libkrb5-1.18-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/gdrwgfz3lr0dvrkwprqv2gd7d02fbww7-libffi-3.4.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/gjmf46lar6hi5qdmsf1vckwfs01p90rj-libseccomp-2.5.2-lib btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/gjyk1v4ri3k1sdl2xnpjk942nsl402v9-libcbor-0.8.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/gmnh4jfjhx83aggwgwzcnrwmpmqr8fwf-gnutar-1.34 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/gmzhclix3kzhir5jmmwakwhpg6j5zwf1-acl-2.3.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/h1yj1jkx57dsj4czl0mpiy2zlc3ybysw-perl5.34.0-Test-RequiresInternet-0.05 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/h6d0c7lnxx65i925xmnxgwdfdvc7hc2p-gcc-10.3.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/h97sr1q1rpv1ry83031q51jbkba7q0m4-bzip2-1.0.6.0.2-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/hggi0lvp1d5vyzkxssv7ybp2kk9if9fk-perl5.34.0-Test-Deep-1.130 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/hjp3p5szxam9wzx7mqb6dj8mzy27npzw-brotli-1.0.9-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/hl09r5178s52savxh4pmkg86jjlakd4p-openssl-1.1.1l btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/hwgcj5gjp0a4z8mfn3b8y04x28smcxmw-perl5.34.0-IO-String-1.08 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/hz9zz1pba77x1vfv1hx9nmyxi9wj127d-perl5.34.0-WWW-RobotRules-6.02 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/i783jhamhw6wb12qqjasa53hfgffwqfn-perl5.34.0-HTTP-Server-Simple-0.52 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/iin3z0l3wmi21h2z20bnl0zkhh8jhvzb-kexec-tools-2.0.23 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ikvp5db9hygc14da45lvxi1c9b4ylna9-pcre-8.44 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ilcwbbjd20yi6904fni1plz3c0cnvk7m-perl5.34.0-JSON-MaybeXS-1.004003 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/in0c2wbqmbxi5kyajh76wkxpcg1pii30-perl5.34.0-Crypt-PasswdMD5-1.40 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/iniphhwn91s358y4akbccahgxjgmz1k4-expat-2.4.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/iqj4idp0pf84hhwmslhkm5jza8vackmm-perl5.34.0-Filesys-Notify-Simple-0.14 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ir1nx41pprndndnaqqrzpdx60p178d32-db-4.8.30 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/j5v7dhq6nnvhvslzfk6ica51ahyk1dyk-gdbm-1.20 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/j7nqx56g28jr2ka1sk590q74d0rhv6dm-perl5.34.0-File-ShareDir-Install-0.13 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/jmwnx0256q8jxxq0bdw98hhgxhx6g7rl-perl5.34.0-asa-1.04 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/jp0a4523pcaksa4almi01371z9vw8i09-libidn-1.38-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/js66s0xwjnzg0ggi2lq9bcvlk6x2za13-binutils-2.35.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/k0qa3rjifblr2vrgx4g54a59zxlfhg90-xz-5.2.5 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/k80ap8bdw4dqs0771a4hwg67z77wbll8-perl5.34.0-Email-Date-Format-1.005 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/kd14wd2wfmb56zpv5y71yq2lqs11l06k-attr-2.5.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/khv33kn327fpdx2311bcjj98kigd2rac-shadow-4.8.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ki2qi1gnjn40ac7fqdz8bw0ncdvdr8kv-man-db-2.9.4 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/kraak5cql3zjqsb3jjv8d7i0qq30ikzs-curl-7.79.1-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ksfz9v7lr3nwhg16sv36rizzll3n31ha-perl5.34.0-Params-Util-1.102 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ksqy6mszsld4z3w8ybxa2vkjf5cqxw3f-c-ares-1.17.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/l3a4yxvd952r0bdpnwii3mn74dafkzcb-perl5.34.0-Test-SharedFork-0.35 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/l4ahhym84c1c57g549qa1pjb3h72dcf4-perl5.34.0-Log-Dispatch-Array-1.003 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/lf6zhg0mxghj1cjngbgj5mfdmzj51a3l-perl5.34.0-Net-SSLeay-1.88 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/lhambyc1v2c7qzzr5sq7p449xs1j6pg8-gnugrep-3.7 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/lkcsgql5nx2fq0h86fyf859cgkz1xcsq-perl5.34.0-Text-Unidecode-1.30 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/lsrahrxyjnsnxzwiacf3dpf0cs7wyn28-perl5.34.0-Inline-0.86 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/m4cpim69iv3mbxxdgvf5cph0hv1abyg5-sqlite-3.36.0-bin btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/m7prz5hx0v9sxj1kdavlh137zhny0vfv-perl5.34.0-Specio-0.46 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/m9lg325s9icr09d1yyc7n26j65alyfaw-perl5.34.0-HTTP-Daemon-6.01 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/mdsjr15hahrkv33k609kvicicsxjh40z-nghttp2-1.43.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/mfnfwmffw8vflgkp9i07c0cxrppw90gg-curl-7.79.1-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/mq9pycra4p4i0gl79gxwzw603dx49n72-stdenv-linux btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/n72iwxf0z2znppa2yr7ndkk502vvznqq-libssh2-1.10.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/n8ifh1z6wkz96j745hsd9p13agdp7md1-perl5.34.0-File-Listing-6.14 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/nalr2sh5p2vjd06kplvna8yqbdpykpkw-perl5.34.0-Test-TCP-2.22 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ni7avvif6pafhqlknzy9yr1cnv9kn702-brotli-1.0.9-lib btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/nii22vkgxhl5x3kkvgddq9yfglpn133z-perl5.34.0-POSIX-strftime-Compiler-0.44 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/nirbgzlcw8zzv3ra6qz2vwi0lw6rzbbg-perl5.34.0-TimeDate-2.33 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/nlvg888wp8jlpmz7bycgc3xfna9xjymz-libcap-2.49-lib btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/nnqmxaydza237jn0dnjvcqqq0f4vvaar-perl5.34.0-Hash-MultiValue-0.16 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ns92nil6y1iff1hvvxh8k6rgfy01qzym-perl5.34.0-HTTP-MultiPartParser-0.02 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/nwbwp5pr2z3wyf6sbwrgc9lncabd26nr-perl5.34.0-namespace-autoclean-0.29 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ny5qpm0y2h44j8kqh740zy6zbf0gzipj-perl5.34.0-Authen-Simple-0.5 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/p2m569gcvx0hr9x0my2002qa01sk351b-groff-1.22.4 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/p7j7hzy5arwx62rd7l67j1b3rii1v2kl-perl5.34.0-Email-MIME-Encodings-1.315 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/pmmlsbl7zf4mda7k6z09qxbmjd11n82r-perl5.34.0-Net-HTTP-6.19 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/psqacrv7k5fxz6mdiawc28sxcdchb4c9-ed-1.17 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/pybllh3323clcqrmaq3vsp4afhwxc5fv-perl5.34.0-YAML-PP-0.026 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/pyk9bdmc36d29367w3rhmn4a9xxnpl7j-perl5.34.0-File-ShareDir-1.118 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/q0zgpg2v1lkaa0j5yw10v9ydlqb4bx0b-perl5.34.0-Authen-Simple-Passwd-0.6 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/qbdsd82q5fyr0v31cvfxda0n0h7jh03g-libunistring-0.9.10 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/qf26s1qwwrifgjvjdqlkdwdfrlq7slh7-perl5.34.0-Sub-Identify-0.14 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/qg8jgfxhn3zmrvh8w52xybijf5nw8h9d-perl5.34.0-Sub-Quote-2.006006 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/qzr7r4w5gm5m20afn2wz4vlv7ah4sr89-gnumake-4.3 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/r5niwjr8r8qags2bzv9z583r9vajxag3-patchelf-0.13 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/r7va5lgm2nlfffvgarl6r738d625s3iy-brotli-1.0.9 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/rhjqb9k87d3v2s4vpjal7sn8p28gdma2-perl5.34.0-IO-Socket-SSL-2.068 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/rnx655nq2qs53yb5arv2gapa91r1wsbn-findutils-4.8.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/rpkbgh8y5k1yvsq04zls6r5003s0a9d5-sqlite-3.36.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/s2pzral55ps1n389m6270f1n4mpic8fl-sqlite-3.36.0-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/s41hl4f321ifhchzc963kkipr7rgkq2p-perl5.34.0-Test-Warn-0.36 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/s92s53kaiapsvw9kyk24sgcgi4ys6rlj-perl5.34.0-Email-MessageID-1.406 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/s97m83hrzzsj3d4pjvzk7r3jvddhm52w-perl5.34.0-Devel-StackTrace-2.04 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/scz4zbxirykss3hh5iahgl39wk9wpaps-libidn2-2.3.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/skc5bc5ih1s19hbsjmbvn9jwzr81h4ga-kmod-29 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/sn0h44frwy9rfmzf0f5xs26h9gj9qxc7-nghttp2-1.43.0-lib btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/sqn31ly001033hsz0dpxwcsay5qdbk2w-gawk-5.1.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/sx3ggwg4aszamg4kcz2q629n3gw6l9xw-perl5.34.0-Devel-StackTrace-AsHTML-0.15 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/syx9zczfg4p6x013lyg4j6nr1fqj4l9m-openssl-1.1.1l-dev btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/v7c1gih8m690hhm504cxisp6b1pwzxjs-perl5.34.0-Params-Validate-1.30 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/vafyy0lx82py8403bvkaki34i5bpin2n-linux-headers-5.14 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/vh8vr0avs8585clpw9k183sc2kb64v0l-perl5.34.0-Test-Fatal-0.016 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/vid3qqjlrjgld5qyiazi74qy6sd7kfad-perl5.34.0-Class-Accessor-0.51 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/vq8fq8p13wvgfaylncnbqjjdxqxxn5p2-perl5.34.0-CGI-Emulate-PSGI-0.23 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/w0hmsxmgsbv7vzf2y4lyx3lnhmr5d4vx-perl5.34.0-Email-Simple-2.216 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/wh9cwmidi60x1c6q42hp1ifjbz889chv-perl5.34.0-CGI-Fast-2.15 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/wi9g0kykpwajbbi01gxr8i6dxmg3zija-perl5.34.0-Parse-RecDescent-1.967015 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/wp10khjwd6f3ypw9pxgfj1rhscb1m896-perl5.34.0-Inline-C-0.81 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/wrjn1knw7ras686pr3nbaqgxqvr395ad-readline-6.3p08 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ww7nzw9jdzghpwxzn56nwl0k6gxska6l-perl5.34.0-HTTP-Date-6.05 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/x3nivnifc568rwficrczwz4gsn98872z-perl5.34.0-HTTP-Headers-Fast-0.22 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/x5d4w3xnlvsg7ihywgbfwg0ylmbybam0-perl5.34.0-Class-Data-Inheritable-0.08 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/x8m12f6ppfyk28zh1i03nwimyig7d4vx-perl5.34.0-Search-Xapian-1.2.25.4 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/xf9gvzb883bf4391wvyz82bzlz0c8rzy-libcap-ng-0.8.2 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/xqchk05qaxvz1x43ki64qsizsz7yf2k0-perl5.34.0-Mozilla-CA-20200520 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/xra89qnn4m2lfjhcid1b5yd2z7jxd5sl-perl5.34.0-WWW-Form-UrlEncoded-0.26 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/xvyzi7cr0icnyavi5pm9rywjc4d8l7sx-libedit-20210714-3.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/xzrmb5fj7hw44a6fwp99b9ivw5dlzqd0-perl5.34.0-HTTP-Message-6.26 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/y3nw0pf6r4xrhyfxm670mlasjs26zl5b-perl5.34.0-Email-Address-XS-1.04 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/y53ma353xbpv6nnjzxhwcid087h6dp36-perl5.34.0-Devel-GlobalDestruction-0.14 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/y64kgxy85z6j1fdbh9lsfjzwzbwyz81w-highlight-4.1 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/ywmf6vnn7rxh95mmzlfx7yb431mgx6kl-perl5.34.0-Role-Tiny-2.001004 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/yxflij8cg4fgnzqmda91jx4d94jvkjf5-util-linux-2.37.2-lib btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/zjm4xv4nr872mdhvv3j22bzb08rgf1hk-patch-2.7.6 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /nix/store/zp0pqbkcxi79ar8rca9sykqq90104n89-perl5.34.0-B-Hooks-EndOfScope-0.24 btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
/dev/dm-2 /bin/sh btrfs rw,relatime,degraded,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/root 0 0
tmpfs /build tmpfs rw,relatime,size=14680064k 0 0
devtmpfs /dev/full devtmpfs rw,nosuid,size=814296k,nr_inodes=2032488,mode=755 0 0
devtmpfs /dev/kvm devtmpfs rw,nosuid,size=814296k,nr_inodes=2032488,mode=755 0 0
devtmpfs /dev/null devtmpfs rw,nosuid,size=814296k,nr_inodes=2032488,mode=755 0 0
devtmpfs /dev/random devtmpfs rw,nosuid,size=814296k,nr_inodes=2032488,mode=755 0 0
devtmpfs /dev/tty devtmpfs rw,nosuid,size=814296k,nr_inodes=2032488,mode=755 0 0
devtmpfs /dev/urandom devtmpfs rw,nosuid,size=814296k,nr_inodes=2032488,mode=755 0 0
devtmpfs /dev/zero devtmpfs rw,nosuid,size=814296k,nr_inodes=2032488,mode=755 0 0
none /proc proc rw,relatime 0 0
none /dev/shm tmpfs rw,relatime 0 0
none /dev/pts devpts rw,relatime,mode=620,ptmxmode=000 0 0

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] nodatacow: quiet chattr errors [was: Test failures with 1.7.0]
  2021-12-08 21:14       ` Dominique Martinet
@ 2021-12-08 22:01         ` Dominique Martinet
  0 siblings, 0 replies; 8+ messages in thread
From: Dominique Martinet @ 2021-12-08 22:01 UTC (permalink / raw)
  To: Eric Wong; +Cc: Julien Moutinho, meta

Dominique Martinet wrote on Thu, Dec 09, 2021 at 06:14:36AM +0900:
> I'll try giving it one, in my opinion it's more representative to test
> with inline-c working.

So giving tests a home appear to make another test hang
(t/lei-refresh-mail-sync.t)

I've run out of time, will provide more traces tonight

> > Yes on tests requiring stderr to be empty.  Below is a patch
> > which should fix it; however it should only be calling chattr on
> > btrfs mounts.
> 
> I'll give this a try as well.

This patch makes the tests pass as expected.

> > You can also try:
> > 
> >   BTRFS_TESTDIR=/path/to/your/btrfs-mount prove -bvw t/nodatacow.t
> 
> I'll try something similar as well.

I can confirm this one works as well after installing chattr and running
on btrfs, so there's no problem if mounts parsing is fixed.

I'd say this hints at a problem so we're probably better off not
silencing chattr warnings, but would also need to check if the chattr
binary is present... Probably not worth the hassle, I don't know.


> It turns out the other people with failing test are both using zfs, so
> the last problem might be somewhat related; I'll give that a try too.

Julien could also reproduce this one by running on ZFS, we'll provide
more traces when we have them!
-- 
Dominique

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Test failures with 1.7.0
  2021-12-08 10:56   ` Dominique Martinet
  2021-12-08 18:22     ` [PATCH] nodatacow: quiet chattr errors [was: Test failures with 1.7.0] Eric Wong
@ 2021-12-09  1:37     ` Julien Moutinho
  2021-12-09  2:53       ` Dominique Martinet
  1 sibling, 1 reply; 8+ messages in thread
From: Julien Moutinho @ 2021-12-09  1:37 UTC (permalink / raw)
  To: Dominique Martinet; +Cc: Eric Wong, meta

Without Inline::C, and without Nix's sandbox
--------------------------------------------

I get 3 tests to fail (in addition to those currently disabled in Nixpkgs:
https://github.com/NixOS/nixpkgs/pull/104457/files#diff-77474340b38081fea7ce3b112c1bc060d3a95cc5e781915463e2f3fc5556da7bR34-R69
ie. convert-compact, search, v2writable, www_listing, eml, hl_mod,
lei-mirror and spawn):
$ nix develop -f . public-inbox
$ doCheck= bash pkgs/development/perl-modules/generic/builder.sh
$ make -C public-inbox-1.7.0 test
> [...]
> Test Summary Report
> -------------------
> t/nntpd-v2.t               (Wstat: 256 Tests: 113 Failed: 1)
>   Failed test:  102
>   Non-zero exit status: 1                                       
> t/nntpd.t                  (Wstat: 256 Tests: 110 Failed: 1)
>   Failed test:  99
>   Non-zero exit status: 1
> t/solver_git.t             (Wstat: 1280 Tests: 13 Failed: 5)
>   Failed tests:  9-13
>   Non-zero exit status: 5
>   Parse errors: No plan found in TAP output                     
> Files=149, Tests=4658, 296 wallclock secs ( 1.78 usr  0.56 sys + 116.73 cusr 165.82 csys = 284.89 CPU)
> Result: FAIL
> Failed 3/149 test programs. 7/4658 subtests failed.

More precisely:
$ (cd public-inbox-1.7.0; perl -I$out/lib/perl5/site_perl t/nntpd-v2.t )
> [...]
> ok 101 - new article retrieved after compact
> not ok 102 - no deleted files
> #   Failed test 'no deleted files'
> #   at ./t/nntpd.t line 363.
> #     Structures begin differing at:
> #          $got->[0] = '-nntpd    3130714 julm    4u      REG               0,53     77189  2606550 /tmp/nix-shell.D0VGvX (deleted)
> #     '
> #     $expected->[0] = Does not exist
> # [
> #   '-nntpd    3130714 julm    4u      REG               0,53     77189  2606550 /tmp/nix-shell.D0VGvX (deleted)
> # '
> # ]

$ (cd public-inbox-1.7.0; perl -I$out/lib/perl5/site_perl t/nntpd.t )
> [...]
> not ok 99 - no deleted files
> #   Failed test 'no deleted files'
> #   at t/nntpd.t line 363.
> #     Structures begin differing at:
> #          $got->[0] = '-nntpd    3131133 julm    4u      REG               0,53     77189  2606550 /tmp/nix-shell.D0VGvX (deleted)
> #     '
> #     $expected->[0] = Does not exist
> # [
> #   '-nntpd    3131133 julm    4u      REG               0,53     77189  2606550 /tmp/nix-shell.D0VGvX (deleted)
> # '
> # ]

$ (cd public-inbox-1.7.0; perl -I$out/lib/perl5/site_perl t/solver_git.t )
> ok 1 - use PublicInbox::Inbox;
> ok 2 - use PublicInbox::V2Writable;
> ok 3 - use PublicInbox::Git;
> ok 4 - use PublicInbox::SolverGit;
> ok 5 - use PublicInbox::WWW;
> ok 6 # skip Inline::C unconfigured/missing (mkdir -p ~/.cache/public-inbox/inline-c) OR Socket::MsgHdr missing
> ok 7 # skip Inline::C unconfigured/missing (mkdir -p ~/.cache/public-inbox/inline-c) OR Socket::MsgHdr missing
> ok 8 # skip Inline::C unconfigured/missing (mkdir -p ~/.cache/public-inbox/inline-c) OR Socket::MsgHdr missing
> not ok 9 - solved a blob!
> #   Failed test 'solved a blob!'
> #   at t/solver_git.t line 151.
> not ok 10 - got a git object for the blob
> #   Failed test 'got a git object for the blob'
> #   at t/solver_git.t line 153.
> #          got: ''
> #     expected: 'PublicInbox::Git'
> not ok 11 - resolved blob to unabbreviated identifier
> #   Failed test 'resolved blob to unabbreviated identifier'
> #   at t/solver_git.t line 154.
> #          got: undef
> #     expected: '69df7d565d49fbaaeb0a067910f03dc22cd52bd0'
> not ok 12 - type specified
> #   Failed test 'type specified'
> #   at t/solver_git.t line 155.
> #          got: undef
> #     expected: 'blob'
> not ok 13 - size returned
> #   Failed test 'size returned'
> #   at t/solver_git.t line 156.
> #          got: undef
> #     expected: '4405'
> Can't call method "cat_file" on an undefined value at t/solver_git.t line 158.
> # Tests were run but no plan was declared and done_testing() was not seen.

I can also reproduce Infinisil's test failure with:
$ (cd public-inbox-1.7.0; TMPDIR=/var/tmp perl -I$out/lib/perl5/site_perl t/lei_to_mail.t )
> ok 96 - got Maildir callback
> Use of uninitialized value in open at t/lei_to_mail.t line 263. 
> Bail out!  No such file or directory 

$ mount | grep 'on /var/tmp'
> losurdo/var/tmp on /var/tmp type zfs (rw,relatime,xattr,posixacl)



With Inline::C, and without Nix's sandbox
-----------------------------------------
I get one more test to fail:
$ export PERL_INLINE_DIRECTORY=$PWD/inline-c; rm -rf $PERL_INLINE_DIRECTORY; mkdir $PERL_INLINE_DIRECTORY
$ make -C public-inbox-1.7.0 test
> [...]
> Test Summary Report
> -------------------
> t/lei-q-save.t             (Wstat: 512 Tests: 108 Failed: 2)
>   Failed tests:  95, 97
>   Non-zero exit status: 2

More precisely:
$ (cd public-inbox-1.7.0; perl -I$out/lib/perl5/site_perl t/lei-q-save.t )
> [...]
> ok 93 - lei up $TMPDIR/lei-daemon/s (absolute lei up)
> execve: No such file or directory
> fork_exec blib/script/perl -i -p -e s/\[/\0/ /tmp/nix-shell.EKTZrS/pi-lei-q-save-3131696-HNqt/lei-daemon/.local/share/lei/saved-searches/after-af0b1d56d93b52910b7172f052d9b9e6c7b3411a060b50006e417c9aa83f5ce2/lei.saved-search failed: No such file or directory

Which is an artifact of my testing method, because with an absolute perl:
$ (cd public-inbox-1.7.0; $(which perl) -I$out/lib/perl5/site_perl t/lei-q-save.t )
> [...]
> not ok 95 - git config parse error shown w/ lei ls-search
> #   Failed test 'git config parse error shown w/ lei ls-search'
> #   at t/lei-q-save.t line 228.
> #                   'fatal: ligne 3 de config incorrecte dans le fichier /tmp/nix-shell.EKTZrS/pi-lei-q-save-3132353-xECZ/lei-daemon/.local/share/lei/saved-searches/after-c8d60f185616bece7957076be4604781d6cae876642a4faa127a2fcfc9647fc8/lei.saved-search
> # git config -z -l --includes --file=/tmp/nix-shell.EKTZrS/pi-lei-q-save-3132353-xECZ/lei-daemon/.cache/lei/saved-tmp.3132468.1639007508.config failed: $?=32768
> # '
> #     doesn't match '(?^:bad config line.*?\/tmp\/nix\-shell\.EKTZrS\/pi\-lei\-q\-save\-3132353\-xECZ\/lei\-daemon\/\.local\/share\/lei\/saved\-searches\/after\-c8d60f185616bece7957076be4604781d6cae876642a4faa127a2fcfc9647fc8\/lei\.saved\-search)'
> ok 96 - lei up --all (up works with bad config)
> not ok 97 - git config parse error shown w/ lei up
> #   Failed test 'git config parse error shown w/ lei up'
> #   at t/lei-q-save.t line 231.
> #                   'fatal: ligne 3 de config incorrecte dans le fichier /tmp/nix-shell.EKTZrS/pi-lei-q-save-3132353-xECZ/lei-daemon/.local/share/lei/saved-searches/after-c8d60f185616bece7957076be4604781d6cae876642a4faa127a2fcfc9647fc8/lei.saved-search
> # git config -z -l --includes --file=/tmp/nix-shell.EKTZrS/pi-lei-q-save-3132353-xECZ/lei-daemon/.cache/lei/saved-tmp.3132468.1639007508.config failed: $?=32768
> # '
> #     doesn't match '(?^:bad config line.*?\/tmp\/nix\-shell\.EKTZrS\/pi\-lei\-q\-save\-3132353\-xECZ\/lei\-daemon\/\.local\/share\/lei\/saved\-searches\/after\-c8d60f185616bece7957076be4604781d6cae876642a4faa127a2fcfc9647fc8\/lei\.saved\-search)'
It succeeds if I use LANG=C instead of LANG=fr_FR.UTF-8



With Inline::C, and with Nix's sandbox
--------------------------------------

Using (in pkgs/servers/mail/public-inbox/default.nix):
> preCheck = ''
>   perl certs/create-certs.perl
>   export HOME=$(mktemp -d)
>   mkdir -p $HOME/.cache/public-inbox/inline-c
> '';

$ nix -L build -f . public-inbox
> [...]
> perl5.34.0-public-inbox> t/lei-refresh-mail-sync.t .... ok
Hangs like forever after this line, while on t/lei-sigpipe.t:

$ ps auxwwf
> root     1755997  0.0  0.0 453172  2800 ?        Ssl  déc.04   0:03 nix-daemon --daemon
> root     1811429  0.0  0.0 453652  1928 ?        Ssl  déc.04   0:00  \_ nix-daemon 1757028
> root     1811455  0.0  0.0 453652  2004 ?        Ssl  déc.04   0:00  \_ nix-daemon 1757031
> root     1839837  0.0  0.0 453652  1832 ?        Ssl  déc.04   0:00  \_ nix-daemon 1757027
> root     1839841  0.0  0.0 453652  1996 ?        Ssl  déc.04   0:00  \_ nix-daemon 1757030
> root     1839856  0.0  0.0 453652  1832 ?        Ssl  déc.04   0:00  \_ nix-daemon 1757029
> root     3135741  0.0  0.2 603352 17080 ?        Ssl  01:05   0:00  \_ nix-daemon 3135672
> nixbld1  3137110  0.0  0.0   5200  4072 ?        Ss   01:05   0:01      \_ bash -e /nix/store/qf3mzpvsmkrw963xchbivcci06078n13-builder.sh
> nixbld1  3140619  0.0  0.0   4996  4012 ?        S    01:05   0:00          \_ make SHELL=/nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8/bin/bash VERBOSE=y test
> nixbld1  3140633  0.0  0.1  17188 13352 ?        S    01:05   0:01          |   \_ /nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/bin/perl -MExtUtils::Command::MM -MTest::Harness -e undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch') t/admin.t t/altid.t t/altid_v2.t t/cgi.t t/cmd_ipc.t t/content_hash.t t/dir_idle.t t/edit.t t/eml_content_disposition.t t/eml_content_type.t t/epoll.t t/extindex-psgi.t t/extsearch.t t/fake_inotify.t t/feed.t t/gcf2.t t/gcf2_client.t t/httpd.t t/idx_stack.t t/imap.t t/imap_searchqp.t t/imapd-tls.t t/imapd.t t/inbox_idle.t t/index-git-times.t t/indexlevels-mirror.t t/ipc.t t/kqnotify.t t/lei-auto-watch.t t/lei-convert.t t/lei-daemon.t t/lei-export-kw.t t/lei-externals.t t/lei-import-http.t t/lei-import-imap.t t/lei-import-maildir.t t/lei-import-nntp.t t/lei-import.t t/lei-index.t t/lei-inspect.t t/lei-lcat.t t/lei-p2q.t t/lei-q-kw.t t/lei-q-remote-import.t t/lei-q-save.t t/lei-q-thread.t t/lei-refresh-mail-sync.t t/lei-sigpipe.t t/lei-tag.t t/lei-up.t t/lei-watch.t t/lei.t t/lei_dedupe.t t/lei_external.t t/lei_lcat.t t/lei_mail_sync.t t/lei_overview.t t/lei_saved_search.t t/lei_store.t t/lei_to_mail.t t/lei_xsearch.t t/mbox_lock.t t/mbox_reader.t t/mdir_reader.t t/mime.t t/miscsearch.t t/net_reader-imap.t t/nntpd-tls.t t/nntpd.t t/nodatacow.t t/on_destroy.t t/plack.t t/psgi_attach.t t/psgi_bad_mids.t t/psgi_mount.t t/psgi_multipart_not.t t/psgi_scan_all.t t/psgi_search.t t/psgi_v2.t t/rename_noreplace.t t/reply.t t/search-thr-index.t t/shared_kv.t t/solver_git.t t/thread-index-gap.t t/uri_imap.t t/uri_nntps.t t/v2dupindex.t t/www_altid.t t/xcpdb-reshard.t t/www_static.t t/watch_multiple_headers.t t/watch_maildir_v2.t t/watch_maildir.t t/watch_imap.t t/watch_filter_rubylang.t t/view.t t/v2reindex.t t/v2mirror.t t/v2mda.t t/v2index-late-dupe.t t/v2-add-remove-add.t t/v1reindex.t t/v1-add-remove-add.t t/time.t t/thread-cycle.t t/spamcheck_spamc.t t/sigfd.t t/replace.t t/reindex-time-range.t t/qspawn.t t/purge.t t/psgi_text.t t/precheck.t t/over.t t/nulsubject.t t/nntpd-v2.t t/nntp.t t/multi-mid.t t/msgtime.t t/msgmap.t t/msg_iter.t t/mid.t t/mda_filter_rubylang.t t/mda.t t/linkify.t t/init.t t/indexlevels-mirror-v1.t t/inbox.t t/import.t t/imap_tracker.t t/hval.t t/httpd-unix.t t/httpd-https.t t/httpd-corner.t t/gzip_filter.t t/git.t t/filter_vger.t t/filter_subjecttag.t t/filter_rubylang.t t/filter_mirror.t t/filter_base.t t/emergency.t t/ds-poll.t t/ds-leak.t t/ds-kqxs.t t/config_limiter.t t/config.t t/address.t
> nixbld1  3145719  0.0  0.4  44576 35640 ?        S    01:09   0:01          |       \_ t/lei-sigpipe.t
> nixbld1  3145743  0.0  0.1  14388 10388 ?        S    01:09   0:00          |           \_ /nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/bin/perl -w -I/nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0/lib/perl5/site_perl -I/nix/store/1hdx7bxjwqrfnalalkgbwi32l45h8z7b-perl5.34.0-Mail-IMAPClient-3.42/lib/perl5/site_perl -I/nix/st blib/script/lei q -q -t z:1..
> nixbld1  3145724  0.0  0.5  50504 42160 ?        S    01:09   0:00          \_ lei-daemon /build/pi-lei-sigpipe-7522-WoHO/lei-daemon/xdg_run/lei/5.seq.sock
> nixbld1  3145730  0.0  0.4  50092 39736 ?        S    01:09   0:00              \_ lei/store /build/pi-lei-sigpipe-7522-WoHO/lei-daemon/.local/share

t/lei-sigpipe.t is on:
> 01:30:29.508334 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000360>
> 01:30:30.509837 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000186>
> 01:30:31.510259 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7fffffff5090) = 0 <1.000175>
> [...]

lei-daemon store is on:
> 01:31:03.834708 epoll_wait(5,

Looks like it is waiting on sysread() in
https://public-inbox.org/public-inbox.git/tree/t/lei-sigpipe.t#n35

This test does succeed outside Nix's sandbox:
$ (cd public-inbox-1.7.0; export PERL_INLINE_DIRECTORY=$PWD/inline-c; rm -rf $PERL_INLINE_DIRECTORY; mkdir $PERL_INLINE_DIRECTORY; prove -bvw t/lei-sigpipe.t )
> t/lei-sigpipe.t ..               
> ok 1 - lei import $TMPDIR/lei-daemon/big.eml
> ok 2 - read one byte             
> ok 3 - signaled                  
> ok 4 - got SIGPIPE               
> ok 5 - quiet after sigpipe 
> ok 6 - read one byte
> ok 7 - signaled -f mboxcl2       
> ok 8 - got SIGPIPE -f mboxcl2    
> ok 9 - quiet after sigpipe -f mboxcl2
> ok 10 - read one byte
> ok 11 - signaled -f text         
> ok 12 - got SIGPIPE -f text
> ok 13 - quiet after sigpipe -f text
> ok 14 - lei daemon-pid (daemon-pid after t/lei-sigpipe.t:44)
> ok 15 - daemon running after t/lei-sigpipe.t:44
> ok 16 - lei daemon-kill (daemon-kill after t/lei-sigpipe.t:44)
> ok 17 - t/lei-sigpipe.t:44 daemon stopped
> ok 18 - t/lei-sigpipe.t:44 daemon XDG_RUNTIME_DIR/lei/errors.log empty
> 1..18
> ok
> All tests successful.
> Files=1, Tests=18,  7 wallclock secs ( 0.06 usr  0.06 sys +  3.44 cusr  2.73 csys =  6.29 CPU)
> Result: PASS

More surprisingly, it even succeeds when run manually
inside the hanging Nix sandbox:
$ sudo nsenter --target 3137110 --all -S 1000 -G 100 $(readlink -e $(which bash))
$ . /build/env-vars
$ cd /build
$ export HOME=$(mktemp -d)
$ mkdir -p $HOME/.cache/public-inbox/inline-c
$ LANG=C prove -bvw t/lei-sigpipe.t
> t/lei-sigpipe.t .. 
> ok 1 - lei import $TMPDIR/lei-daemon/big.eml
> ok 2 - read one byte
> ok 3 - signaled 
> ok 4 - got SIGPIPE 
> ok 5 - quiet after sigpipe 
> ok 6 - read one byte
> ok 7 - signaled -f mboxcl2
> ok 8 - got SIGPIPE -f mboxcl2
> ok 9 - quiet after sigpipe -f mboxcl2
> ok 10 - read one byte
> ok 11 - signaled -f text
> ok 12 - got SIGPIPE -f text
> ok 13 - quiet after sigpipe -f text
> ok 14 - lei daemon-pid (daemon-pid after t/lei-sigpipe.t:44)
> ok 15 - daemon running after t/lei-sigpipe.t:44
> ok 16 - lei daemon-kill (daemon-kill after t/lei-sigpipe.t:44)
> ok 17 - t/lei-sigpipe.t:44 daemon stopped
> ok 18 - t/lei-sigpipe.t:44 daemon XDG_RUNTIME_DIR/lei/errors.log empty
> 1..18
> ok
> All tests successful.
> Files=1, Tests=18,  4 wallclock secs ( 0.06 usr  0.06 sys +  1.23 cusr  1.48 csys =  2.83 CPU)
> Result: PASS

Even more strange, Dominique was able to reproduce
the hang this morning, but no longer tonight..

Cheers,
Julien

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Test failures with 1.7.0
  2021-12-09  1:37     ` Test failures with 1.7.0 Julien Moutinho
@ 2021-12-09  2:53       ` Dominique Martinet
  0 siblings, 0 replies; 8+ messages in thread
From: Dominique Martinet @ 2021-12-09  2:53 UTC (permalink / raw)
  To: Julien Moutinho; +Cc: Eric Wong, meta

Julien Moutinho wrote on Thu, Dec 09, 2021 at 02:37:43AM +0100:
> I can also reproduce Infinisil's test failure with:
> $ (cd public-inbox-1.7.0; TMPDIR=/var/tmp perl -I$out/lib/perl5/site_perl t/lei_to_mail.t )
> > ok 96 - got Maildir callback
> > Use of uninitialized value in open at t/lei_to_mail.t line 263. 
> > Bail out!  No such file or directory 

I got curious on this one.
strace tells me:
----
2813384 renameat2(AT_FDCWD, "/tank/pi-lei_to_mail-2813384-n7sk/maildir/tmp/badc0ffee", AT_FDCWD, "/tank/pi-lei_to_mail-2813384-n7sk/maildir/cur/badc0ffee:2,", RENAME_NOREPLACE) = -1 EINVAL (Invalid argument)
2813384 openat(AT_FDCWD, "/tank/pi-lei_to_mail-2813384-n7sk/maildir/new/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
2813384 newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=2, ...}, AT_EMPTY_PATH) = 0
2813384 brk(0x44f4000)                  = 0x44f4000
2813384 getdents64(4, 0x44b3e40 /* 2 entries */, 131072) = 48
2813384 getdents64(4, 0x44b3e40 /* 0 entries */, 131072) = 0
2813384 close(4)                        = 0
2813384 openat(AT_FDCWD, "/tank/pi-lei_to_mail-2813384-n7sk/maildir/cur/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
2813384 newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=2, ...}, AT_EMPTY_PATH) = 0
2813384 getdents64(4, 0x44b3e40 /* 2 entries */, 131072) = 48
2813384 getdents64(4, 0x44b3e40 /* 0 entries */, 131072) = 0
2813384 close(4)                        = 0
2813384 write(2, "Use of uninitialized value in op"..., 64) = 64
2813384 openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2813384 getpid()                        = 2813384
2813384 getpid()                        = 2813384
2813384 getpid()                        = 2813384
2813384 write(5, "Bail out!  No such file or direc"..., 37) = 37
----

So this one is a real bug, this appears to fix it:
----
From 50a63628d505ca1c8d36f94ab5703f87a2c5e415 Mon Sep 17 00:00:00 2001
From: Dominique Martinet <asmadeus@codewreck.org>
Date: Thu, 9 Dec 2021 11:50:51 +0900
Subject: [PATCH] syscall: fallback to rename on renameat2 EINVAL

ZFS appears to incorrectly return EINVAL on renameat2 when the operation is not
supported:
renameat2(AT_FDCWD, "...", AT_FDCWD, "...", RENAME_NOREPLACE) = -1 EINVAL

Fall back to the racy rename in this case as well:

diff --git a/lib/PublicInbox/Syscall.pm b/lib/PublicInbox/Syscall.pm
index c00385b94db8..78f926ac38f0 100644
--- a/lib/PublicInbox/Syscall.pm
+++ b/lib/PublicInbox/Syscall.pm
@@ -15,7 +15,7 @@ package PublicInbox::Syscall;
 use strict;
 use v5.10.1;
 use parent qw(Exporter);
-use POSIX qw(ENOENT EEXIST ENOSYS O_NONBLOCK);
+use POSIX qw(ENOENT EEXIST ENOSYS EINVAL O_NONBLOCK);
 use Config;
 
 # $VERSION = '0.25'; # Sys::Syscall version
@@ -312,7 +312,7 @@ sub rename_noreplace ($$) {
 		my $ret = syscall($SYS_renameat2, -100, $old, -100, $new, 1);
 		if ($ret == 0) {
 			1; # like rename() perlop
-		} elsif ($! == ENOSYS) {
+		} elsif ($! == ENOSYS || $! == EINVAL) {
 			undef $SYS_renameat2;
 			_rename_noreplace_racy($old, $new);
 		} else {
----

> This test does succeed outside Nix's sandbox:
> $ (cd public-inbox-1.7.0; export PERL_INLINE_DIRECTORY=$PWD/inline-c; rm -rf $PERL_INLINE_DIRECTORY; mkdir $PERL_INLINE_DIRECTORY; prove -bvw t/lei-sigpipe.t )
> > t/lei-sigpipe.t ..               
> > ok 1 - lei import $TMPDIR/lei-daemon/big.eml
> > ok 2 - read one byte             
> > ok 3 - signaled                  
> > ok 4 - got SIGPIPE               
> > ok 5 - quiet after sigpipe 
> > ok 6 - read one byte
> > ok 7 - signaled -f mboxcl2       
> > ok 8 - got SIGPIPE -f mboxcl2    
> > ok 9 - quiet after sigpipe -f mboxcl2
> > ok 10 - read one byte
> > ok 11 - signaled -f text         
> > ok 12 - got SIGPIPE -f text
> > ok 13 - quiet after sigpipe -f text
> > ok 14 - lei daemon-pid (daemon-pid after t/lei-sigpipe.t:44)
> > ok 15 - daemon running after t/lei-sigpipe.t:44
> > ok 16 - lei daemon-kill (daemon-kill after t/lei-sigpipe.t:44)
> > ok 17 - t/lei-sigpipe.t:44 daemon stopped
> > ok 18 - t/lei-sigpipe.t:44 daemon XDG_RUNTIME_DIR/lei/errors.log empty
> > 1..18
> > ok
> > All tests successful.
> > Files=1, Tests=18,  7 wallclock secs ( 0.06 usr  0.06 sys +  3.44 cusr  2.73 csys =  6.29 CPU)
> > Result: PASS
> 
> More surprisingly, it even succeeds when run manually
> inside the hanging Nix sandbox:
> $ sudo nsenter --target 3137110 --all -S 1000 -G 100 $(readlink -e $(which bash))
> $ . /build/env-vars
> $ cd /build
> $ export HOME=$(mktemp -d)
> $ mkdir -p $HOME/.cache/public-inbox/inline-c
> $ LANG=C prove -bvw t/lei-sigpipe.t
> > t/lei-sigpipe.t .. 
> > ok 1 - lei import $TMPDIR/lei-daemon/big.eml
> > ok 2 - read one byte
> > ok 3 - signaled 
> > ok 4 - got SIGPIPE 
> > ok 5 - quiet after sigpipe 
> > ok 6 - read one byte
> > ok 7 - signaled -f mboxcl2
> > ok 8 - got SIGPIPE -f mboxcl2
> > ok 9 - quiet after sigpipe -f mboxcl2
> > ok 10 - read one byte
> > ok 11 - signaled -f text
> > ok 12 - got SIGPIPE -f text
> > ok 13 - quiet after sigpipe -f text
> > ok 14 - lei daemon-pid (daemon-pid after t/lei-sigpipe.t:44)
> > ok 15 - daemon running after t/lei-sigpipe.t:44
> > ok 16 - lei daemon-kill (daemon-kill after t/lei-sigpipe.t:44)
> > ok 17 - t/lei-sigpipe.t:44 daemon stopped
> > ok 18 - t/lei-sigpipe.t:44 daemon XDG_RUNTIME_DIR/lei/errors.log empty
> > 1..18
> > ok
> > All tests successful.
> > Files=1, Tests=18,  4 wallclock secs ( 0.06 usr  0.06 sys +  1.23 cusr  1.48 csys =  2.83 CPU)
> > Result: PASS
> 
> Even more strange, Dominique was able to reproduce
> the hang this morning, but no longer tonight..

Yes, I don't get it, it hanged once but no longer hangs, so as much as
I'd have liked to investigate I'm a bit stuck.

With this, I can confirm running with inline-c also makes the tests that
failed with the btrfs chattr call also pass.
So all that's left is fix the proc mounts parsing there :)

-- 
Dominique

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-12-09  2:54 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-08  1:07 Test failures with 1.7.0 Julien Moutinho
2021-12-08  4:08 ` Eric Wong
2021-12-08 10:56   ` Dominique Martinet
2021-12-08 18:22     ` [PATCH] nodatacow: quiet chattr errors [was: Test failures with 1.7.0] Eric Wong
2021-12-08 21:14       ` Dominique Martinet
2021-12-08 22:01         ` Dominique Martinet
2021-12-09  1:37     ` Test failures with 1.7.0 Julien Moutinho
2021-12-09  2:53       ` Dominique Martinet

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).