Re: 01/01: nginx: berlin: Disable narinfo caching altogether.

Re: 01/01: nginx: berlin: Disable narinfo caching altogether.

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> civodul pushed a commit to branch master
> in repository maintenance.
>
> commit 8379ba4119e51151d93589a6ef57cb159d94e9f2
> Author: Ludovic Courtès <ludo@gnu.org>
> Date:   Thu Jun 21 11:41:06 2018 +0200
>
>     nginx: berlin: Disable narinfo caching altogether.
>     
>     This is a followup to ebbe4c7f402b6d9cf9c6c2ecf120f49697ab2c49.
>     
>     * hydra/nginx/berlin-locations.conf (.narinfo): Disable caching.
>     * hydra/nginx/berlin.conf: Remove 'proxy_cache_path' directive
>     for narinfos.

What's the rationale for this change?

Although 'guix publish' does its own caching, I would expect 'nginx' to
handle cache hits far more efficiently than guix publish.  Therefore, I
expect this change to result in higher CPU usage for a given amount of
traffic.  I also expect that cache hits on narinfos are an important
common case for our substitute servers.

Does this change bring advantages that outweigh the loss of efficiency?

       Mark

Re: 01/01: nginx: berlin: Disable narinfo caching altogether.

[Ludovic Courtès]

Hello Mark,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> civodul pushed a commit to branch master
>> in repository maintenance.
>>
>> commit 8379ba4119e51151d93589a6ef57cb159d94e9f2
>> Author: Ludovic Courtès <ludo@gnu.org>
>> Date:   Thu Jun 21 11:41:06 2018 +0200
>>
>>     nginx: berlin: Disable narinfo caching altogether.
>>     
>>     This is a followup to ebbe4c7f402b6d9cf9c6c2ecf120f49697ab2c49.
>>     
>>     * hydra/nginx/berlin-locations.conf (.narinfo): Disable caching.
>>     * hydra/nginx/berlin.conf: Remove 'proxy_cache_path' directive
>>     for narinfos.
>
> What's the rationale for this change?

From commit ebbe4c7f402b6d9cf9c6c2ecf120f49697ab2c49:

    Somehow nginx appeared to be caching narinfos for longer than needed,
    which defeated the atime-based cache eviction strategy of 'guix
    publish'.

In this case, I noticed on berlin that nginx was caching 404s for
narinfos longer than expected, for reasons I could not elucidate.  Plus
there’s this atime story.

(Narinfo caching was critical when we were using Hydra, because Hydra
would compute them all the fly, which included computing signatures over
whole nars.)

> Although 'guix publish' does its own caching, I would expect 'nginx' to
> handle cache hits far more efficiently than guix publish.  Therefore, I
> expect this change to result in higher CPU usage for a given amount of
> traffic.  I also expect that cache hits on narinfos are an important
> common case for our substitute servers.
>
> Does this change bring advantages that outweigh the loss of efficiency?

I’m not sure the loss of efficiency is noticeable (‘guix publish’ uses
sendfile(2) on 200, and stat(2) is all it takes to distinguish between
200 and 404, so it’s not that bad.)

Anyway, nothing’s set in stone.  It did seem like the right thing to do
now, but if it turned out to be wrong, we can revisit this.

In fact, I would very much welcome input from people who are used to
tweaking nginx more than I am!

Thanks for your feedback,
Ludo’.

Re: 01/01: nginx: berlin: Disable narinfo caching altogether.

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> civodul pushed a commit to branch master
>>> in repository maintenance.
>>>
>>> commit 8379ba4119e51151d93589a6ef57cb159d94e9f2
>>> Author: Ludovic Courtès <ludo@gnu.org>
>>> Date:   Thu Jun 21 11:41:06 2018 +0200
>>>
>>>     nginx: berlin: Disable narinfo caching altogether.
>>>     
>>>     This is a followup to ebbe4c7f402b6d9cf9c6c2ecf120f49697ab2c49.
>>>     
>>>     * hydra/nginx/berlin-locations.conf (.narinfo): Disable caching.
>>>     * hydra/nginx/berlin.conf: Remove 'proxy_cache_path' directive
>>>     for narinfos.
>>
>> What's the rationale for this change?
>
> From commit ebbe4c7f402b6d9cf9c6c2ecf120f49697ab2c49:
>
>     Somehow nginx appeared to be caching narinfos for longer than needed,
>     which defeated the atime-based cache eviction strategy of 'guix
>     publish'.
>
> In this case, I noticed on berlin that nginx was caching 404s for
> narinfos longer than expected, for reasons I could not elucidate.  Plus
> there’s this atime story.

Although you didn't mention it here, I now remember one reason why it's
a problem for narinfos to be in the cache longer than expected: because
if a narinfo is available but the corresponding NAR isn't, it causes
problems on the client side.  If that's still the case, then it
certainly justifies this change.

      Thanks,
        Mark

Re: 01/01: nginx: berlin: Disable narinfo caching altogether.

[Ludovic Courtès]

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> ludo@gnu.org (Ludovic Courtès) writes:
>>>
>>>> civodul pushed a commit to branch master
>>>> in repository maintenance.
>>>>
>>>> commit 8379ba4119e51151d93589a6ef57cb159d94e9f2
>>>> Author: Ludovic Courtès <ludo@gnu.org>
>>>> Date:   Thu Jun 21 11:41:06 2018 +0200
>>>>
>>>>     nginx: berlin: Disable narinfo caching altogether.
>>>>     
>>>>     This is a followup to ebbe4c7f402b6d9cf9c6c2ecf120f49697ab2c49.
>>>>     
>>>>     * hydra/nginx/berlin-locations.conf (.narinfo): Disable caching.
>>>>     * hydra/nginx/berlin.conf: Remove 'proxy_cache_path' directive
>>>>     for narinfos.
>>>
>>> What's the rationale for this change?
>>
>> From commit ebbe4c7f402b6d9cf9c6c2ecf120f49697ab2c49:
>>
>>     Somehow nginx appeared to be caching narinfos for longer than needed,
>>     which defeated the atime-based cache eviction strategy of 'guix
>>     publish'.
>>
>> In this case, I noticed on berlin that nginx was caching 404s for
>> narinfos longer than expected, for reasons I could not elucidate.  Plus
>> there’s this atime story.
>
> Although you didn't mention it here, I now remember one reason why it's
> a problem for narinfos to be in the cache longer than expected: because
> if a narinfo is available but the corresponding NAR isn't, it causes
> problems on the client side.  If that's still the case, then it
> certainly justifies this change.

Yes, that too.

We’ll see how it goes and if it performs badly let’s revert or adjust.

Ludo’.