Moving /gnu/store safely?

Moving /gnu/store safely?

[Phil]

Hi all,

My growing /gnu/store needed to be moved to another location.  I couldn't
find any specific instructions on doing this but think I've managed to
cobble together a way of doing this safely without breaking store properties.

There were some gremlins tho - on restarting Guix the /gnu/store was
seen to grow to approximately double it's size.

I seem to have tamed this with the garbage collection, but would be interested
in people's thoughts if this is a reasonable way to do this, or if you
have any hints or gotchas I may have missed?

Stop all Guix services and mounts
    systemctl stop guix-daemon.service
    systemctl stop gnu-store.mount
    systemctl status guix-daemon.service
    systemctl status gnu-store.mount
    
Assuming that you have a /some_other_disk mount with more room:
    mkdir /some_other_disk/gnu
    rsync -va /gnu/store /some_other_disk/gnu/
    
Move the old store create the correct directory for the new
    cd /gnu/store
    mv store store.old
    mkdir store
    chmod 01775 store
    chown root:guixbuild store
    
Change the Guix configuration file - /etc/systemd/system/gnu-store.mount - under [Mount]:
    What=/some_other_disk/gnu/store
    
Reload configuration and restart the daemon
    systemctl daemon-reload
    systemctl start guix-daemon.service
    systemctl status guix-daemon.service
    systemctl status gnu-store.mount

When you're sure everything looks good
rm -rf /gnu/store.old

Like I said this worked and guix pull/install/etc worked fine afterwards
- previous installs were recognized etc.

When I ran:
guix gc --verify

It returned success (0 ret code)

I noted however that the new /gnu/store was approx double in size - it
looked like it had created new links to represent store items.

I then ran:
guix gc --collect-garbage

This seemed to remove about half the size of the new store - suggesting
to me that stale links were now removed.

Is my method sane?

Is there a better way of doing such a move to avoid the doubling of size?


Thanks,
Phil

Re: Moving /gnu/store safely?

[Julien Lepiller]

I suppose the doubling of size is due to deduplication not working well with rsync: we use hard links for identical files, but I suppose rsync doesn't recreate the hard links?

Maybe try guix gc --optimize?

Le 3 mars 2021 08:44:56 GMT-05:00, Phil <phil@beadling.co.uk> a écrit :
>Hi all,
>
>My growing /gnu/store needed to be moved to another location.  I
>couldn't
>find any specific instructions on doing this but think I've managed to
>cobble together a way of doing this safely without breaking store
>properties.
>
>There were some gremlins tho - on restarting Guix the /gnu/store was
>seen to grow to approximately double it's size.
>
>I seem to have tamed this with the garbage collection, but would be
>interested
>in people's thoughts if this is a reasonable way to do this, or if you
>have any hints or gotchas I may have missed?
>
>Stop all Guix services and mounts
>    systemctl stop guix-daemon.service
>    systemctl stop gnu-store.mount
>    systemctl status guix-daemon.service
>    systemctl status gnu-store.mount
>    
>Assuming that you have a /some_other_disk mount with more room:
>    mkdir /some_other_disk/gnu
>    rsync -va /gnu/store /some_other_disk/gnu/
>    
>Move the old store create the correct directory for the new
>    cd /gnu/store
>    mv store store.old
>    mkdir store
>    chmod 01775 store
>    chown root:guixbuild store
>    
>Change the Guix configuration file -
>/etc/systemd/system/gnu-store.mount - under [Mount]:
>    What=/some_other_disk/gnu/store
>    
>Reload configuration and restart the daemon
>    systemctl daemon-reload
>    systemctl start guix-daemon.service
>    systemctl status guix-daemon.service
>    systemctl status gnu-store.mount
>
>When you're sure everything looks good
>rm -rf /gnu/store.old
>
>Like I said this worked and guix pull/install/etc worked fine
>afterwards
>- previous installs were recognized etc.
>
>When I ran:
>guix gc --verify
>
>It returned success (0 ret code)
>
>I noted however that the new /gnu/store was approx double in size - it
>looked like it had created new links to represent store items.
>
>I then ran:
>guix gc --collect-garbage
>
>This seemed to remove about half the size of the new store - suggesting
>to me that stale links were now removed.
>
>Is my method sane?
>
>Is there a better way of doing such a move to avoid the doubling of
>size?
>
>
>Thanks,
>Phil

Re: Moving /gnu/store safely?

[André Batista]

Hi Phil,

qua 03 mar 2021 às 13:44:56 (1614789896), phil@beadling.co.uk enviou:
> [...]     
> Assuming that you have a /some_other_disk mount with more room:
>     mkdir /some_other_disk/gnu
>     rsync -va /gnu/store /some_other_disk/gnu/
>     
> Move the old store create the correct directory for the new
>     cd /gnu/store
              ^^^^^   
I assume the above command should be 'cd /gnu', right?

>     mv store store.old
>     mkdir store
>     chmod 01775 store
>     chown root:guixbuild store
> 
> Is my method sane?

I don't know, but if it works as expected and without harmful side effects..

Cheers,

Re: Moving /gnu/store safely?

[Phil]

André Batista writes:

>> Move the old store create the correct directory for the new
>>     cd /gnu/store
>               ^^^^^   
> I assume the above command should be 'cd /gnu', right?

Yes - thanks for spotting.  It was a typo in the e-mail only.

Re: Moving /gnu/store safely?

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

Phil,

Barring unnoticed devils in details your process looks safe. 
Running ‘guix gc --verify’ is a good idea; add 
‘=repair[,contents]’ if you're paranoid.

Phil 写道：
> There were some gremlins tho - on restarting Guix the /gnu/store 
> was
> seen to grow to approximately double it's size.

[...]

>     rsync -va /gnu/store /some_other_disk/gnu/

As gremlin's go this one is pretty tame, even cuddly: the Guix 
daemon deduplicates identical files, and creates hard links 
instead, saving a good amount of space.  That's all.

By default, rsync does not preserve hard links (because of 
potential pathological performance issues, IIRC).  This won't 
break /gnu/store: indentical files are still identical, they just 
take up more space.

To preserve them, you can use:

  -H, --hard-links
    This tells rsync to look for hard-linked files in the
    source and link together the corresponding files on the
    destination.  Without this option, hard-linked files in the
    source are treated as though they were separate files.

(I add ’-HAX’ as a matter of habit when making full-system 
back-ups.)

> I noted however that the new /gnu/store was approx double in 
> size - it
> looked like it had created new links to represent store items.

No, the links were *lost* and replaced by separate files during 
rsyncing.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Moving /gnu/store safely?

[Phil]

Thanks v much for the explanations.

Tobias Geerinckx-Rice writes:

> As gremlin's go this one is pretty tame, even cuddly: the Guix daemon
> deduplicates identical files, and creates hard links instead, saving a
> good amount of space.  That's all.
>
> By default, rsync does not preserve hard links (because of potential

Heh heh - yes I was surprised (and relieved) how quickly this gremlin
was dealt with just by scanning the manual and throwing some commands
together - this makes perfect sense so each hardlink just becomes a new file on the target.

I'm assuming the doubling in size is more fluke than by design?
i.e. there is no reason to assume each file is duplicated exactly once
in Guix?  It just turns out that hard linking was saving approx 50% by
chance in my specific store - it could have been 30% or 70% or whatever
and depends on your store contents?

> No, the links were *lost* and replaced by separate files during
> rsyncing.
>

Got it.  At the risk of speculating, would you expect the subsequent
execution of the garbage collector to deduplicate the identical rsync'd files
again?  I'm 100% clear on why the problem happened, just want to check I
understand why it went away which I ran 'guix gc'.

If I look at the link count on the files in my new /gnu/store I can see
some files have a count >1, which certainly suggests that Guix has
performed some sort of deduplication after the rsync?