/gnu/store/.links/

/gnu/store/.links/

[Pjotr Prins]

What is

  ls -1 /gnu/store/.links/|wc -l
  495938

Never saw it before. Does this scale?

Pj.

Re: /gnu/store/.links/

[Ricardo Wurmus]

Hi Pjotr,

> What is
>
>   ls -1 /gnu/store/.links/|wc -l
>   495938
>
> Never saw it before. Does this scale?

It’s used for optional file deduplication.  It is enabled by default,
but you can disable it with a daemon option on file systems that
deduplicate data at the block level.

I don’t know about scalability.  This number is still well below the
limits of ext4 file systems, but accessing a big directory listing like
that can be slow.  I would feel a little better about this if we split
it up into different prefix directories (like it’s done for browser
caches).  I don’t think it’s necessary, though.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: /gnu/store/.links/

[Pjotr Prins]

On Fri, Feb 09, 2018 at 01:11:23PM +0100, Ricardo Wurmus wrote:
> 
> Hi Pjotr,
> 
> > What is
> >
> >   ls -1 /gnu/store/.links/|wc -l
> >   495938
> >
> > Never saw it before. Does this scale?
> 
> It’s used for optional file deduplication.  It is enabled by default,
> but you can disable it with a daemon option on file systems that
> deduplicate data at the block level.

Hmmm. I think this is better handled at the file system level if
people want deduplication. These systems will be more common.

> I don’t know about scalability.  This number is still well below the
> limits of ext4 file systems, but accessing a big directory listing like
> that can be slow.  I would feel a little better about this if we split
> it up into different prefix directories (like it’s done for browser
> caches).  I don’t think it’s necessary, though.

For ext4 it is going to be an issue. Anyway, we'll see what happens.
Thanks for explaining.

Pj.

Re: /gnu/store/.links/

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Fri, Feb 09, 2018 at 01:11:23PM +0100, Ricardo Wurmus wrote:

[...]

>> I don’t know about scalability.  This number is still well below the
>> limits of ext4 file systems, but accessing a big directory listing like
>> that can be slow.  I would feel a little better about this if we split
>> it up into different prefix directories (like it’s done for browser
>> caches).  I don’t think it’s necessary, though.
>
> For ext4 it is going to be an issue. Anyway, we'll see what happens.

In practice, when the maximum number of links is reached, we simply
transparently skip deduplication.  See this commit:

  commit 12b6c951cf5ca6055a22a2eec85665353f5510e5
  Author: Ludovic Courtès <ludo@gnu.org>
  Date:   Fri Oct 28 20:34:15 2016 +0200

      daemon: Do not error out when deduplication fails due to ENOSPC.

      This solves a problem whereby if /gnu/store/.links had enough entries,
      ext4's directory index would be full, leading to link(2) returning
      ENOSPC.

      * nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
      ENOSPC from link(2), print a message and return instead of throwing a
      'SysError'.

It does scale well, and it’s been here “forever”.

If you’re wondering how much gets deduplicated, see
<https://lists.gnu.org/archive/html/guix-devel/2014-09/msg00422.html>.
:-)

Ludo’.

Re: /gnu/store/.links/

[Pjotr Prins]

On Fri, Feb 09, 2018 at 06:00:02PM +0100, Ludovic Courtès wrote:
> In practice, when the maximum number of links is reached, we simply
> transparently skip deduplication.  See this commit:
> 
>   commit 12b6c951cf5ca6055a22a2eec85665353f5510e5
>   Author: Ludovic Courtès <ludo@gnu.org>
>   Date:   Fri Oct 28 20:34:15 2016 +0200
> 
>       daemon: Do not error out when deduplication fails due to ENOSPC.
> 
>       This solves a problem whereby if /gnu/store/.links had enough entries,
>       ext4's directory index would be full, leading to link(2) returning
>       ENOSPC.
> 
>       * nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
>       ENOSPC from link(2), print a message and return instead of throwing a
>       'SysError'.
> 
> It does scale well, and it’s been here “forever”.

OK. My mindset is probably ext2...

> If you’re wondering how much gets deduplicated, see
> <https://lists.gnu.org/archive/html/guix-devel/2014-09/msg00422.html>.
> :-)

Fancy that :)

Pj.

Re: /gnu/store/.links/

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> On Fri, Feb 09, 2018 at 01:11:23PM +0100, Ricardo Wurmus wrote:
>
> [...]
>
>>> I don’t know about scalability.  This number is still well below the
>>> limits of ext4 file systems, but accessing a big directory listing like
>>> that can be slow.  I would feel a little better about this if we split
>>> it up into different prefix directories (like it’s done for browser
>>> caches).  I don’t think it’s necessary, though.
>>
>> For ext4 it is going to be an issue. Anyway, we'll see what happens.
>
> In practice, when the maximum number of links is reached, we simply
> transparently skip deduplication.

Ideally, we should at some point change the daemon to break
/gnu/store/.links up into several subdirectories, as is done for log
files in /var/log/guix/drvs.  The main complication is dealing with the
transition between the old layout and the new.

       Mark

Re: /gnu/store/.links/

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 277 bytes --]

On Fri, Feb 09, 2018 at 03:24:00PM +0100, Pjotr Prins wrote:
> Hmmm. I think this is better handled at the file system level if
> people want deduplication. These systems will be more common.

In general, yes! But filesystems with this feature are still not widely
deployed...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]