Re: CDN performance

["Ludovic Courtès" <ludo@gnu.org>]

Hello!

Chris Marusich <cmmarusich@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:


[...]

> Judging by that email thread, one of the reasons why Debian considered
> using a CDN was because they felt that the cost, in terms of people
> power, of maintaining their own "proto-CDN" infrastructure had grown too
> great.  I believe it!  I think it would be ill-advised for the Guix
> project to expend effort and capital on building and maintaining its own
> CDN.  I think it would be wiser to focus on developing a decentralized
> substitute solution (GNUnet, IPFS, etc.).
>
> That said, I still think that today Guix should provide a third-party
> CDN option.  For many Guix users, a CDN would improve performance and
> availability of substitutes.  Contracting with a third party to provide
> the CDN service would require much less effort and capital than building
> and maintaining a CDN from scratch.  This would also enable the project
> to focus more on building a decentralized substitute solution.  And once
> that decentralized solution is ready, it will be easy to just "turn off"
> the CDN.
>
> I also understand Hartmut's concerns.  The risks he points out are
> valid.  Because of those risks, even if we make a third-party CDN option
> available, some people will choose not to use it.  For that reason, we
> should not require Guix users to use a third-party CDN, just as we do
> not require them to use substitutes from our build farm.
>
> However, not everyone shares the same threat model.  For example,
> although some people choose not to trust substitutes from our build
> farm, still others do.  The choice is based on one's own individual
> situation.  Similarly, if we make a third-party CDN option available and
> explain the risks of using it, Guix users will be able to make an
> educated decision for themselves about whether or not to use it.

That summarizes the situation very well.

In Paris, there was consensus on two things: that a CDN would be helpful
for performance and availability (though availability could suddenly be
worse if we run… out of money), and that it raises privacy concerns.

Based on this, some suggested having a way to opt out of the CDN
distribution: for example, we might make ci.guix.info point to the CDN
and also advertise berlin.guixsd.org as a way to bypass it.

Of course in parallel we want to do our best to develop decentralized
solutions.

> However, I believe you when you say that it's slow the first time you
> download the substitute from berlin.  What path does the data take from
> its origin through berlin?  If berlin needs to download the initial file
> from another server, perhaps the connection between berlin and that
> server is the bottleneck?  Maybe we should discuss that in a different
> email thread, though.

On the first hit, nginx fetches the file from ‘guix publish’, which is
also on berlin.  ‘guix publish’ gets the file from local storage and
passes it with sendfile(2), so it’s as fast as can be.

Thanks for the info!

Ludo’.