bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Rutger Helling]

Hey Guix,

I was surprised to learn that Ceph creates Btrfs subvolumes during its 
tests.
This is problematic because on Btrfs regular users can create 
subvolumes, but they cannot delete them.
This means I had to manually delete the following subvolumes as root:

ID 1744 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
ID 1745 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1
ID 1746 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/test_subvol
ID 1747 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/sync_snap_test
ID 1748 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/async_snap_test
ID 1749 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/current
ID 1750 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/snap_1
ID 1751 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/test_subvol
ID 1752 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/sync_snap_test
ID 1753 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/async_snap_test
ID 1754 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/current
ID 1755 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/snap_1
ID 1756 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/test_subvol
ID 1757 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/sync_snap_test
ID 1758 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/async_snap_test

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Ludovic Courtès]

Hi Rutger,

Rutger Helling <rhelling@mykolab.com> skribis:

> I was surprised to learn that Ceph creates Btrfs subvolumes during its
> tests.
> This is problematic because on Btrfs regular users can create
> subvolumes, but they cannot delete them.
> This means I had to manually delete the following subvolumes as root:
>
> ID 1744 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
> ID 1745 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1

Does that mean that wiping out /tmp/guix-build-ceph-*, as happens when
build finishes, isn’t enough to get rid of those volumes?

Ludo’.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Rutger Helling]

Hi Ludo,

I'll try building again later (could be a while, Ceph takes a really 
long time to build), but I highly doubt that that's the case.
Btrfs subvolumes cannot be deleted with tools such as rm/rmdir, even as 
root. You have to run 'btrfs subvolume delete /subvolume' as root.
Although a regular user may create subvolumes, they cannot delete it 
(unless the USER_SUBVOL_RM_ALLOWED mount option is on).
Since the building is done as a guixbuilder user, I don't see how that 
would work.
And even if it did, if a build failed for example you'd still be left 
with a bunch of subvolumes in /tmp that can only be deleted as root.

On 2017-12-12 13:52, ludo@gnu.org wrote:

> Hi Rutger,
> 
> Rutger Helling <rhelling@mykolab.com> skribis:
> 
>> I was surprised to learn that Ceph creates Btrfs subvolumes during its
>> tests.
>> This is problematic because on Btrfs regular users can create
>> subvolumes, but they cannot delete them.
>> This means I had to manually delete the following subvolumes as root:
>> 
>> ID 1744 gen 819598 top level 257 path
>> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
>> ID 1745 gen 819598 top level 257 path
>> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1
> 
> Does that mean that wiping out /tmp/guix-build-ceph-*, as happens when
> build finishes, isn't enough to get rid of those volumes?
> 
> Ludo'.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Rutger Helling]

Actually, I just tested it with the following:

$ cd /tmp
$ sudo -s -u guixbuilder01
bash-4.4$ guix environment -C --ad-hoc btrfs-progs
guixbuilder01@guixsd /tmp [env]#  btrfs subvol create test-snapshot
Create subvolume './test-snapshot'
guixbuilder01@guixsd /tmp [env]#  btrfs subvol del test-snapshot
Delete subvolume (no-commit): '/tmp/test-snapshot'
ERROR: cannot delete '/tmp/test-snapshot': Operation not permitted
guixbuilder01@guixsd /tmp [env]# exit
exit
bash-4.4$ guix environment -C --ad-hoc coreutils
guixbuilder01@guixsd /tmp [env]#  rm -rfv test-snapshot/
rm: cannot remove 'test-snapshot/': Operation not permitted
guixbuilder01@guixsd /tmp [env]#  rmdir  -v test-snapshot/
rmdir: removing directory, 'test-snapshot/'
rmdir: failed to remove 'test-snapshot/': Operation not permitted
guixbuilder01@guixsd /tmp [env]#

On 2017-12-12 14:10, Rutger Helling wrote:

> Hi Ludo,
> 
> I'll try building again later (could be a while, Ceph takes a really 
> long time to build), but I highly doubt that that's the case.
> Btrfs subvolumes cannot be deleted with tools such as rm/rmdir, even as 
> root. You have to run 'btrfs subvolume delete /subvolume' as root.
> Although a regular user may create subvolumes, they cannot delete it 
> (unless the USER_SUBVOL_RM_ALLOWED mount option is on).
> Since the building is done as a guixbuilder user, I don't see how that 
> would work.
> And even if it did, if a build failed for example you'd still be left 
> with a bunch of subvolumes in /tmp that can only be deleted as root.
> 
> On 2017-12-12 13:52, ludo@gnu.org wrote:
> 
> Hi Rutger,
> 
> Rutger Helling <rhelling@mykolab.com> skribis:
> 
> I was surprised to learn that Ceph creates Btrfs subvolumes during its
> tests.
> This is problematic because on Btrfs regular users can create
> subvolumes, but they cannot delete them.
> This means I had to manually delete the following subvolumes as root:
> 
> ID 1744 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
> ID 1745 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1
> Does that mean that wiping out /tmp/guix-build-ceph-*, as happens when
> build finishes, isn't enough to get rid of those volumes?
> 
> Ludo'.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

Rutger Helling <rhelling@mykolab.com> writes:

> Hey Guix,
>
> I was surprised to learn that Ceph creates Btrfs subvolumes during its 
> tests.
> This is problematic because on Btrfs regular users can create 
> subvolumes, but they cannot delete them.

Ugh.  My build machines do not use Btrfs so I hadn't noticed.  Will look
into disabling these tests.

Actually I'm tempted to just disable tests, since they are extremely
I/O intensive and some are flaky (about a 50% success rate on Hydra).
I've also been unable to update to the latest versions due to test suite
changes.

Maybe a system test is a decent compromise, although that will require
writing service definitions for the various daemons.  It's probably a
better use of our time than fighting the comprehensive test suite.

Do you intend to use Ceph on Guix, or is it for a dependent package?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Ludovic Courtès]

Rutger Helling <rhelling@mykolab.com> skribis:

> Actually, I just tested it with the following:
>
> $ cd /tmp
> $ sudo -s -u guixbuilder01
> bash-4.4$ guix environment -C --ad-hoc btrfs-progs
> guixbuilder01@guixsd /tmp [env]#  btrfs subvol create test-snapshot
> Create subvolume './test-snapshot'
> guixbuilder01@guixsd /tmp [env]#  btrfs subvol del test-snapshot
> Delete subvolume (no-commit): '/tmp/test-snapshot'
> ERROR: cannot delete '/tmp/test-snapshot': Operation not permitted
> guixbuilder01@guixsd /tmp [env]# exit
> exit
> bash-4.4$ guix environment -C --ad-hoc coreutils
> guixbuilder01@guixsd /tmp [env]#  rm -rfv test-snapshot/
> rm: cannot remove 'test-snapshot/': Operation not permitted
> guixbuilder01@guixsd /tmp [env]#  rmdir  -v test-snapshot/
> rmdir: removing directory, 'test-snapshot/'
> rmdir: failed to remove 'test-snapshot/': Operation not permitted

So does guix-daemon systematically leave /tmp/guix-build-ceph* behind it?

Ludo’.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Tobias Geerinckx-Rice]

Ludovic Courtès wrote on 12/12/17 at 17:03:
> So does guix-daemon systematically leave /tmp/guix-build-ceph* behind it?

Almost certainly. I can confirm several hundred previously unknown
subvolumes filling up my btrfs substitute server. Time to clean up.

Kind regards,

T G-R

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Ludovic Courtès]

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> Ludovic Courtès wrote on 12/12/17 at 17:03:
>> So does guix-daemon systematically leave /tmp/guix-build-ceph* behind it?
>
> Almost certainly. I can confirm several hundred previously unknown
> subvolumes filling up my btrfs substitute server. Time to clean up.

Ouch.  I find this Btrfs behavior highly questionable.

Let’s make sure our packages don’t leak Btrfs subvolumes then!

Ludo’.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Rutger Helling]

Hey everyone,

Thanks for all the replies.

@Marius: It's a dependency for multipath-tools.

@Ludo: Like Tobias mentioned it does indeed systematically leave behind 
subvolumes that can only be manually deleted as root.
Thankfully this is the only package in which I've noticed this behavior.

I've disabled the tests entirely myself. This allowed me to build 
multipath-tools without any problems.
Maybe it's better to do that for now to make sure Btrfs users don't run 
into trouble and later disable the individual offending tests?

On 2017-12-12 21:19, ludo@gnu.org wrote:

> Tobias Geerinckx-Rice <me@tobias.gr> skribis:
> 
> Ludovic Courtès wrote on 12/12/17 at 17:03: So does guix-daemon 
> systematically leave /tmp/guix-build-ceph* behind it?
> Almost certainly. I can confirm several hundred previously unknown
> subvolumes filling up my btrfs substitute server. Time to clean up.

Ouch.  I find this Btrfs behavior highly questionable.

Let's make sure our packages don't leak Btrfs subvolumes then!

Ludo'.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Rutger Helling]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

This bug has been open for quite a while now.
If there are no objections in the next few days, I'd like to disable the
tests entirely for now with a FIXME on finding and disabling only the
offending tests.

On Tue, 12 Dec 2017 21:19:07 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Tobias Geerinckx-Rice <me@tobias.gr> skribis:
> 
> > Ludovic Courtès wrote on 12/12/17 at 17:03:  
> >> So does guix-daemon systematically leave /tmp/guix-build-ceph*
> >> behind it?  
> >
> > Almost certainly. I can confirm several hundred previously unknown
> > subvolumes filling up my btrfs substitute server. Time to clean
> > up.  
> 
> Ouch.  I find this Btrfs behavior highly questionable.
> 
> Let’s make sure our packages don’t leak Btrfs subvolumes then!
> 
> Ludo’.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On Wed, Dec 27, 2017 at 11:50:39PM +0100, Rutger Helling wrote:
> This bug has been open for quite a while now.
> If there are no objections in the next few days, I'd like to disable the
> tests entirely for now with a FIXME on finding and disabling only the
> offending tests.

That sounds fine for now.

> 
> On Tue, 12 Dec 2017 21:19:07 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
> 
> > Tobias Geerinckx-Rice <me@tobias.gr> skribis:
> > 
> > > Ludovic Courtès wrote on 12/12/17 at 17:03:  
> > >> So does guix-daemon systematically leave /tmp/guix-build-ceph*
> > >> behind it?  
> > >
> > > Almost certainly. I can confirm several hundred previously unknown
> > > subvolumes filling up my btrfs substitute server. Time to clean
> > > up.  
> > 
> > Ouch.  I find this Btrfs behavior highly questionable.
> > 
> > Let’s make sure our packages don’t leak Btrfs subvolumes then!
> > 
> > Ludo’.
> 



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 339 bytes --]

Rutger,

Rutger Helling wrote on 27/12/17 at 23:50:
> This bug has been open for quite a while now.
> If there are no objections in the next few days, I'd like to disable the
> tests entirely for now with a FIXME on finding and disabling only the
> offending tests.
No objections if it's in a few days.

Kind regards,

T G-R


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

Rutger Helling <rhelling@mykolab.com> writes:

> This bug has been open for quite a while now.
> If there are no objections in the next few days, I'd like to disable the
> tests entirely for now with a FIXME on finding and disabling only the
> offending tests.

I dug pretty deep into the Ceph test code a few days ago without
being able to pinpoint which test(s) are the culprit for this.  Btrfs
subvolume operations only occur in FileStore.cc and objectstoretool.py
IIRC, but attempts at disabling their tests was unfruitful.

So, disabling tests altogether sounds good to me.  Make sure to add a
link to this bug report with the FIXME.

Thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 225 bytes --]

Tobias Geerinckx-Rice wrote on 28/12/17 at 00:16:
> No objections if it's in a few days.

Actually, the fix I was hacking on appears as dead an end as all the others.

No objections any day.

Kind regards,

T G-R


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Rutger Helling]

[-- Attachment #1: Type: text/plain, Size: 54 bytes --]

Thanks for all the replies. I've disabled the tests.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Ludovic Courtès]

Rutger Helling <rhelling@mykolab.com> skribis:

> Thanks for all the replies. I've disabled the tests.

Thank you.  Remember to close the bug by emailing
NUMBER-done@debbugs.gnu.org as I did here.  :-)

Ludo’.

bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.

[Rutger Helling]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

I left it open because I was under the impression that some people were
still working on a fix for the tests. But since there doesn't seem to be
a short-term solution it's probably better to close it. There's a TODO
comment now anyway.

On Mon, 08 Jan 2018 15:29:35 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Rutger Helling <rhelling@mykolab.com> skribis:
> 
> > Thanks for all the replies. I've disabled the tests.  
> 
> Thank you.  Remember to close the bug by emailing
> NUMBER-done@debbugs.gnu.org as I did here.  :-)
> 
> Ludo’.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]