[PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Ludovic Courtès]

Hello Guix!

These patches (actually the last one) switch Guix to default to
<https://ci.guix.info> for substitutes, in preparation for the
upcoming 0.16.0 release (hopefully this week!).

Rationale:

  • berlin.guixsd.org generally performs better than hydra.gnu.org;

  • berlin supports x86, ARMv7, and AArch64 (hydra lacks AArch64);

  • berlin has much more storage space than hydra;

  • berlin runs Cuirass on GuixSD, so software-wise, it corresponds
    to what we want to push; Cuirass still has limitations in some
    ways compared to Hydra, but we’re getting there, and
    hydra.gnu.org is still up and running so we can still use it,
    e.g., to track build progress until Cuirass is as convenient.

For the domain name I initially wanted “ci.guix.gnu.org” but we failed
to set that up.  Oh well, I think that’s OK.

This change modifies the manual.  Translations will thus be stale;
Julien, do you think it’d be reasonable to push a new pre-release
to the Translation Project?  Or should be just live with the
slight inaccuracy?

Thanks,
Ludo’.

Ludovic Courtès (3):
  etc: Add "ci.guix.info.pub" public key file.
  Remove most references to hydra.gnu.org.
  build: Default to https://ci.guix.info for substitutes.

 Makefile.am                                   |  5 +-
 build-aux/check-available-binaries.scm        |  4 +-
 .../check-final-inputs-self-contained.scm     |  2 +-
 config-daemon.ac                              | 10 +--
 doc/contributing.texi                         |  2 +-
 doc/guix.texi                                 | 67 +++++++++----------
 etc/substitutes/ci.guix.info.pub              |  1 +
 gnu/services/base.scm                         |  4 +-
 gnu/system/install.scm                        |  2 +-
 guix/scripts/build.scm                        |  2 +-
 guix/scripts/size.scm                         |  2 +-
 guix/scripts/substitute.scm                   |  2 +-
 guix/self.scm                                 |  3 +
 guix/store.scm                                |  2 +-
 14 files changed, 51 insertions(+), 57 deletions(-)
 create mode 120000 etc/substitutes/ci.guix.info.pub

-- 
2.19.2

Using a CDN or some other mirror?

[Ludovic Courtès]

Hello,

Ludovic Courtès <ludo@gnu.org> skribis:

> These patches (actually the last one) switch Guix to default to
> <https://ci.guix.info> for substitutes, in preparation for the
> upcoming 0.16.0 release (hopefully this week!).

Right now, ci.guix.info points to berlin.guixsd.org, the front-end of
the build farm hosted at the MDC.

The previous setup was that mirror.hydra.gnu.org mirrors hydra.gnu.org
(the actual build farm front-end) using an nginx proxy:

  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf

This provides a bit of redundancy that we don’t have currently for
berlin.

Thus, I’m thinking about using a similar setup, but hosting the mirror
on some Big Corp CDN or similar.  Chris Marusich came up with a setup
along these lines a while back:

  https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html

Compared to Chris’s setup, given that ‘guix publish’ now provides
‘Cache-Control’ headers (that was not the case back then, see
<https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00360.html>),
caching in the proxy should Just Work.

I would like us to set up such a mirror for berlin and then have
ci.guix.info point to that.  The project should be able to pay the
hosting fees.

Thoughts?

Would someone like to get started?  You’ll undoubtedly get all the
appreciation of each one of us and a beverage of your choice next time
we meet!  :-)

Thanks,
Ludo’.

Re: Using a CDN or some other mirror?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> These patches (actually the last one) switch Guix to default to
>> <https://ci.guix.info> for substitutes, in preparation for the
>> upcoming 0.16.0 release (hopefully this week!).
>
> Right now, ci.guix.info points to berlin.guixsd.org, the front-end of
> the build farm hosted at the MDC.
>
> The previous setup was that mirror.hydra.gnu.org mirrors hydra.gnu.org
> (the actual build farm front-end) using an nginx proxy:
>
>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf
>
> This provides a bit of redundancy that we don’t have currently for
> berlin.
>
> Thus, I’m thinking about using a similar setup, but hosting the mirror
> on some Big Corp CDN or similar.  Chris Marusich came up with a setup
> along these lines a while back:
>
>   https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html

Large ISPs also provide CDN services.  I already contacted Deutsche
Telekom so that we can compare their CDN offer with the Amazon Cloudfont
setup that Chris has configured.

--
Ricardo

Re: Using a CDN or some other mirror?

[Hartmut Goebel]

Am 03.12.2018 um 17:12 schrieb Ludovic Courtès:
> Thus, I’m thinking about using a similar setup, but hosting the mirror
> on some Big Corp CDN or similar.

Isn't this a contradiction: Building a free infrastructure relaying on
servers from some Big Corporation? Let allow the privacy concerns
raising when delivering data via some Big Corporation.

If delivering "packages" works via static data without requiring any
additional service, we could ask universities to host Guix, too. IMHO
this is a much preferred solution since this is a decentralized publish
infrastructure already in place for many GNU/Linux distributions.

-- 
+++hartmut

| Hartmut Goebel            |                       |
| hartmut@goebel-consult.de | www.goebel-consult.de |

Re: Using a CDN or some other mirror?

[Ludovic Courtès]

Hi Hartmut,

Hartmut Goebel <hartmut@goebel-consult.de> skribis:

> Am 03.12.2018 um 17:12 schrieb Ludovic Courtès:
>> Thus, I’m thinking about using a similar setup, but hosting the mirror
>> on some Big Corp CDN or similar.
>
> Isn't this a contradiction: Building a free infrastructure relaying on
> servers from some Big Corporation? Let allow the privacy concerns
> raising when delivering data via some Big Corporation.
>
> If delivering "packages" works via static data without requiring any
> additional service, we could ask universities to host Guix, too. IMHO
> this is a much preferred solution since this is a decentralized publish
> infrastructure already in place for many GNU/Linux distributions.

As you know, berlin.guixsd.org is hosted at the Max Delbrück Center in
Berlin, a public research institute.  So in a way, we’re already doing
that.  We shouldn’t take it for granted that public institutes will
happily host our stuff and donate hardware: without Ricardo’s work and
the generosity of the MDC, we wouldn’t have anything there.

I understand the reluctance regarding “Big Corp” hosting, and I actually
share it to some extent.  However, having put much thought into it (and
also much sweat in build farm sysadmin…), I think the alternative is:
commercial hosting, or peer-to-peer.

Florian has been looking at the latter approach with IPFS, and perhaps
we’ll be able to put it in production in a few months and be happy with
it (I have good hopes given what Florian already demonstrated.)

In the meantime, we need redundant storage, high bandwidth, and high
availability.  If you know of non-profit organizations that can provide
such services, please let us know; if not, we’ll resort to a commercial
service.  The bottom line is: we cannot reasonably pretend to offer such
a service ourselves.

(Note that we’re just talking about substitute delivery—I wouldn’t want
to *build* packages on one of these commercial hosting services.)

I hope this clarifies my position.

Ludo’.

Re: Using a CDN or some other mirror?

[Pjotr Prins]

On Tue, Dec 04, 2018 at 03:05:44PM +0100, Ludovic Courtès wrote:
> Florian has been looking at the latter approach with IPFS, and perhaps
> we’ll be able to put it in production in a few months and be happy with
> it (I have good hopes given what Florian already demonstrated.)
> 
> In the meantime, we need redundant storage, high bandwidth, and high
> availability.  If you know of non-profit organizations that can provide
> such services, please let us know; if not, we’ll resort to a commercial
> service.  The bottom line is: we cannot reasonably pretend to offer such
> a service ourselves.
> 
> (Note that we’re just talking about substitute delivery—I wouldn’t want
> to *build* packages on one of these commercial hosting services.)

IPFS would be great because it will allow anyone to lookup and share
substitutes with a low barrier to entry and it allows for redundancy
too. Unlike bittorrent it comes with a built in webinterface and it is
a merkle tree with local deduplication. 

I think IPFS is a pretty solid proposition since so many are building
solutions on it.

Pj.

Re: Using a CDN or some other mirror?

[Thompson, David]

On Tue, Dec 4, 2018 at 9:06 AM Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hi Hartmut,
>
> Hartmut Goebel <hartmut@goebel-consult.de> skribis:
>
> > Am 03.12.2018 um 17:12 schrieb Ludovic Courtès:
> >> Thus, I’m thinking about using a similar setup, but hosting the mirror
> >> on some Big Corp CDN or similar.
> >
> > Isn't this a contradiction: Building a free infrastructure relaying on
> > servers from some Big Corporation? Let allow the privacy concerns
> > raising when delivering data via some Big Corporation.
> >
> > If delivering "packages" works via static data without requiring any
> > additional service, we could ask universities to host Guix, too. IMHO
> > this is a much preferred solution since this is a decentralized publish
> > infrastructure already in place for many GNU/Linux distributions.
>
> As you know, berlin.guixsd.org is hosted at the Max Delbrück Center in
> Berlin, a public research institute.  So in a way, we’re already doing
> that.  We shouldn’t take it for granted that public institutes will
> happily host our stuff and donate hardware: without Ricardo’s work and
> the generosity of the MDC, we wouldn’t have anything there.
>
> I understand the reluctance regarding “Big Corp” hosting, and I actually
> share it to some extent.  However, having put much thought into it (and
> also much sweat in build farm sysadmin…), I think the alternative is:
> commercial hosting, or peer-to-peer.
>
> Florian has been looking at the latter approach with IPFS, and perhaps
> we’ll be able to put it in production in a few months and be happy with
> it (I have good hopes given what Florian already demonstrated.)
>
> In the meantime, we need redundant storage, high bandwidth, and high
> availability.  If you know of non-profit organizations that can provide
> such services, please let us know; if not, we’ll resort to a commercial
> service.  The bottom line is: we cannot reasonably pretend to offer such
> a service ourselves.
>
> (Note that we’re just talking about substitute delivery—I wouldn’t want
> to *build* packages on one of these commercial hosting services.)
>
> I hope this clarifies my position.

Using CloudFront with a custom (non-S3) origin sounds like a
reasonable solution to me, though I understand the hesitance to use a
commercial service.

If AWS CloudFront is the path chosen, it may be worthwhile to follow
the "infrastructure as code" practice and use CloudFormation to
provision the CloudFront distribution and any other supporting
resources. The benefit is that there would be a record of exactly
*how* the project is using these commercial services and the setup
could be easily reproduced.  The timing is interesting here because I
just attended the annual AWS conference on behalf of my employer and
while I was there I felt inspired to write a Guile API for building
CloudFormation "stacks".  You can see a small sample of what it does
here: https://gist.github.com/davexunit/db4b9d3e67902216fbdbc66cd9c6413e

- Dave

Re: Using a CDN or some other mirror?

[Meiyo Peng]

Hi,

ludo@gnu.org (Ludovic Courtès) writes:

> As you know, berlin.guixsd.org is hosted at the Max Delbrück Center in
> Berlin, a public research institute.  So in a way, we’re already doing
> that.  We shouldn’t take it for granted that public institutes will
> happily host our stuff and donate hardware: without Ricardo’s work and
> the generosity of the MDC, we wouldn’t have anything there.
>
> I understand the reluctance regarding “Big Corp” hosting, and I actually
> share it to some extent.  However, having put much thought into it (and
> also much sweat in build farm sysadmin…), I think the alternative is:
> commercial hosting, or peer-to-peer.
>
> Florian has been looking at the latter approach with IPFS, and perhaps
> we’ll be able to put it in production in a few months and be happy with
> it (I have good hopes given what Florian already demonstrated.)
>
> In the meantime, we need redundant storage, high bandwidth, and high
> availability.  If you know of non-profit organizations that can provide
> such services, please let us know; if not, we’ll resort to a commercial
> service.  The bottom line is: we cannot reasonably pretend to offer such
> a service ourselves.
>
> (Note that we’re just talking about substitute delivery—I wouldn’t want
> to *build* packages on one of these commercial hosting services.)
>
> I hope this clarifies my position.

When I started to try Guix several months ago, the network speed to
substitute servers from China is very slow (<100kB/s). I don't know what
has changed but recently the network speed is about 1MB/s. Thank you all
for the improments. Hopefully a CDN will make the network even better.
I am not against using a commercial service as long as we only use them
to distribute signed packages rather than building packages.

I like the idea of IPFS. We should try it. It would be great if it works
well.

If at some point we need to setup traditional mirrors like other major
Gnu/Linux distros, I can contact my friends in China to setup mirrors in
several universities. I was a member of LUG@USTC, which provides the
largest FLOSS mirror in China.

--
Meiyo Peng

Re: Using a CDN or some other mirror?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

On Wed, Dec 05, 2018 at 10:32:02AM +0800, Meiyo Peng wrote:
> If at some point we need to setup traditional mirrors like other major
> Gnu/Linux distros, I can contact my friends in China to setup mirrors in
> several universities. I was a member of LUG@USTC, which provides the
> largest FLOSS mirror in China.

That would be cool, especially if it's hard to reach our servers from
within China.

The Nginx configuration for our mirrors is available in the Guix
"maintenance" repo:

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror-locations.conf

... along with similar files 'berlin.conf' and 'bayfront.conf' for those
servers.

You can use those files to run your own mirror by changing at least
'server_name', 'ssl_certificate', and 'ssl_certificate_key'.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Using a CDN or some other mirror?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]


Meiyo Peng <meiyo.peng@gmail.com> writes:

> When I started to try Guix several months ago, the network speed to
> substitute servers from China is very slow (<100kB/s). I don't know what
> has changed but recently the network speed is about 1MB/s. Thank you all
> for the improments. Hopefully a CDN will make the network even better.
> I am not against using a commercial service as long as we only use them
> to distribute signed packages rather than building packages.

I had a similar experience from India, with similar bandwidth on average
(<100kB/s).  It took a lot of patience to install Guix! :p

I haven't been to Asia recently, but I'm glad to hear it's getting
better!

Indeed, such a low bandwidth could be prohibitive to the adoption of
Guix outside Europe.

> I like the idea of IPFS. We should try it. It would be great if it works
> well.
>
> If at some point we need to setup traditional mirrors like other major
> Gnu/Linux distros, I can contact my friends in China to setup mirrors in
> several universities. I was a member of LUG@USTC, which provides the
> largest FLOSS mirror in China.

That'd be fantastic!

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using a CDN or some other mirror?

[Hartmut Goebel]

Am 04.12.2018 um 15:05 schrieb Ludovic Courtès:
> We shouldn’t take it for granted that public institutes will
> happily host our stuff and donate hardware

I would not expect them nor take for granted that some organization will
host and even donate a build-farm. This is indeed very generous from MDC,

As you wrote, we are in need of storage. What I'm suggesting is:

Many, many organizations support open source projects by donating
disk-space and bandwidth in form of a mirror. Supporters are internet
providers, universities, publicly founded organizations  and companies.
In former time this have been ftp-mirrors (later with an
http-interface), in the meanwhile the larger ones support rsync, too.
And many Software projects are using these kind of servers for mirroring
their software. The only restriction is that these mirrors are serving
static files only. (I assume you all know.)

> In the meantime, we need redundant storage, high bandwidth, and high
> availability.  If you know of non-profit organizations that can provide
> such services, please let us know; if not, we’ll resort to a commercial
> service.  The bottom line is: we cannot reasonably pretend to offer such
> a service ourselves.

One could get in touch with the administrators of some of the mirrors
used by other distributions. [1] has a list of Mirrors for Fedora,
including location and bandwidth. [2] is a list of mirrors used by the
community driven distribution Mageia, additionally including the
"upstream" - which might help prioritizing.

When approaching some of these server admins, we should pass them some
relevant information like expected storage demand, expected transfer
traffic from upstream, how often updates are published, how long to hold
copies, can we provide a rsync server access, etc.

I can approach two or three organizations I know of (but don't have any
contact) if I have some information to provide. A cover letter would
help a lot. For getting in tough with universities somebody working at
an university might have better chances :-)

[1] Fedora 28:
https://admin.fedoraproject.org/mirrormanager/mirrors/Fedora/28
[2] Mageia: https://mirrors.mageia.org/
[3] Debian "Secondary mirrors" in https://www.debian.org/mirror/list

-- 
+++hartmut

| Hartmut Goebel            |                       |
| hartmut@goebel-consult.de | www.goebel-consult.de |

Re: Using a CDN or some other mirror?

[Ludovic Courtès]

Hi Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> One could get in touch with the administrators of some of the mirrors
> used by other distributions. [1] has a list of Mirrors for Fedora,
> including location and bandwidth. [2] is a list of mirrors used by the
> community driven distribution Mageia, additionally including the
> "upstream" - which might help prioritizing.

It’s tempting to follow the lead of traditional distros when it comes to
mirroring.

However, Guix is very different from these: on the build farm, we build
several new store items per minute, and we aim to distribute them to our
users.  This is quite different from distros that upload .deb, .rpm,
etc. files much less frequently.  Their mirroring process is thus fairly
similar to good ol’ synchronization over rsync and the like.

For Guix I think a caching proxy and a CDN kind of model is a better
fit: it’s good fit for the publication rate of store items, and a good
fit for ‘guix publish’ which uses HTTP.  Furthermore, the build farm
doesn’t expose anything like rsync, which makes it hard to imagine a
“traditional” mirroring process.

WDYT?

Thanks,
Ludo’.

Re: Using a CDN or some other mirror?

[Hartmut Goebel]

Hi Ludo,

Am 07.12.2018 um 15:05 schrieb Ludovic Courtès:
> However, Guix is very different from these: on the build farm, we build
> several new store items per minute, and we aim to distribute them to our
> users. 
> […]  
> For Guix I think a caching proxy […] is a better 
> fit:

In this constellation an old-fashined "ftp" mirror does not work (since
we would need to convince the admins to change the setup, not just to
add another source) and the idea is void.

OTOH, maybe it's worth rethinking the premises. if some region has low
bandwidth if would still be better if one could fetch 90% of the
substitutes from nearby. Thus some daily or twice-a-day rsync would
help. It would also help if some day GuixSD gets some kind of "stable"
(and some "testing") branch and users might subscribe to this, which
would not have so many changes a day.

I'm now finishing this discussion, as it is up to you (the core team) to
decide. I just wanted to share the idea and arguments.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Using a CDN or some other mirror?

[ng0]

Hartmut Goebel transcribed 771 bytes:
> Am 03.12.2018 um 17:12 schrieb Ludovic Courtès:
> > Thus, I’m thinking about using a similar setup, but hosting the mirror
> > on some Big Corp CDN or similar.
> 
> Isn't this a contradiction: Building a free infrastructure relaying on
> servers from some Big Corporation? Let allow the privacy concerns
> raising when delivering data via some Big Corporation.
> 
> If delivering "packages" works via static data without requiring any
> additional service, we could ask universities to host Guix, too. IMHO
> this is a much preferred solution since this is a decentralized publish
> infrastructure already in place for many GNU/Linux distributions.

Regardless of me agreeing with Hartmut here, I suggest https://wasabi.com/
as an S3 compatible storage (, which is not run by Amazon.).

Re: Using a CDN or some other mirror?

[Thompson, David]

On Tue, Dec 4, 2018 at 4:15 PM <ng0@n0.is> wrote:
>
> Hartmut Goebel transcribed 771 bytes:
> > Am 03.12.2018 um 17:12 schrieb Ludovic Courtès:
> > > Thus, I’m thinking about using a similar setup, but hosting the mirror
> > > on some Big Corp CDN or similar.
> >
> > Isn't this a contradiction: Building a free infrastructure relaying on
> > servers from some Big Corporation? Let allow the privacy concerns
> > raising when delivering data via some Big Corporation.
> >
> > If delivering "packages" works via static data without requiring any
> > additional service, we could ask universities to host Guix, too. IMHO
> > this is a much preferred solution since this is a decentralized publish
> > infrastructure already in place for many GNU/Linux distributions.
>
> Regardless of me agreeing with Hartmut here, I suggest https://wasabi.com/
> as an S3 compatible storage (, which is not run by Amazon.).

But can Wasabi provide what CloudFront does?  CloudFront is distinct
from S3, and can fetch and cache data from any origin. Additionally,
it stores data on edge nodes across the globe and routes requests
accordingly to maximize download speed.

- Dave

Re: Using a CDN or some other mirror?

[ng0]

Thompson, David transcribed 1.2K bytes:
> On Tue, Dec 4, 2018 at 4:15 PM <ng0@n0.is> wrote:
> >
> > Hartmut Goebel transcribed 771 bytes:
> > > Am 03.12.2018 um 17:12 schrieb Ludovic Courtès:
> > > > Thus, I’m thinking about using a similar setup, but hosting the mirror
> > > > on some Big Corp CDN or similar.
> > >
> > > Isn't this a contradiction: Building a free infrastructure relaying on
> > > servers from some Big Corporation? Let allow the privacy concerns
> > > raising when delivering data via some Big Corporation.
> > >
> > > If delivering "packages" works via static data without requiring any
> > > additional service, we could ask universities to host Guix, too. IMHO
> > > this is a much preferred solution since this is a decentralized publish
> > > infrastructure already in place for many GNU/Linux distributions.
> >
> > Regardless of me agreeing with Hartmut here, I suggest https://wasabi.com/
> > as an S3 compatible storage (, which is not run by Amazon.).
> 
> But can Wasabi provide what CloudFront does?  CloudFront is distinct
> from S3, and can fetch and cache data from any origin. Additionally,
> it stores data on edge nodes across the globe and routes requests
> accordingly to maximize download speed.
> 
> - Dave
> 

Probably not, for some reason I thought you were discussing S3.
My bad.

Re: Using a CDN or some other mirror?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 13416 bytes --]

Hi everyone,

ludo@gnu.org (Ludovic Courtès) writes:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
> [...] I’m thinking about using a similar setup, but hosting the mirror
> on some Big Corp CDN or similar.  Chris Marusich came up with a setup
> along these lines a while back:
>
>   https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html
>
> Compared to Chris’s setup, given that ‘guix publish’ now provides
> ‘Cache-Control’ headers (that was not the case back then, see
> <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00360.html>),
> caching in the proxy should Just Work.
>
> I would like us to set up such a mirror for berlin and then have
> ci.guix.info point to that.  The project should be able to pay the
> hosting fees.
>
> Thoughts?

Regarding DNS, it would be nice if we could use an official GNU
subdomain.  If we can't use a GNU subdomain, we should at least make
sure we have some kind of DNS auto-renewal set up so that nobody can
poach our domain names.  And the operators should take appropriate
precautions when sharing any credentials used for managing it all.

Regarding CDNs, I definitely think it's worth a try!  Even Debian is
using CloudFront (cloudfront.debian.net).  In fact, email correspondence
suggests that as of 2013, Amazon may even have been paying for it!

https://lists.debian.org/debian-cloud/2013/05/msg00071.html

I wonder if Amazon would be willing to pay for our CloudFront
distribution if we asked them nicely?

In any case, before deciding to use Amazon CloudFront for ci.guix.info,
it would be prudent to estimate the cost.  CloudFront, like most Amazon
AWS services, is a "pay for what you use" model.  The pricing is here:

https://aws.amazon.com/cloudfront/pricing

To accurately estimate the cost, we need to know how many requests we
expect to receive, and how many bytes we expect to transfer out, during
a single month.  Do we have information like this for berlin today?

Although I don't doubt that a CDN will perform better than what we have
now, I do think it would be good to measure the performance so that we
know for sure the money spent is actually providing a benefit.  It would
be nice to have some data before and after to measure how availability
and performance have changed.  Apart from anecdotes, what data do we
have to determine whether performance has improved after introducing a
CDN?  For example, the following information could be useful:

  * Network load on the origin server(s)
  * Clients' latency to (the addresses pointed to by) ci.guix.info
  * Clients' throughput while downloading substitutes from ci.guix.info

We don't log or collect client metrics, and that's fine.  It could be
useful to add code to Guix to measure things like this when the user
asks to do so, but perhaps it isn't necessary.  It may be good enough if
people just volunteer to manually gather some information and share it.
For example, you can define a shell function like this:

--8<---------------cut here---------------start------------->8---
measure_get () {
curl -L \
     -o /dev/null \
     -w "url_effective: %{url_effective}\\n\
http_code: %{http_code}\\n\
num_connects: %{num_connects}\\n\
num_redirects: %{num_redirects}\\n\
remote_ip: %{remote_ip}\\n\
remote_port: %{remote_port}\\n\
size_download: %{size_download} B\\n\
speed_download: %{speed_download} B/s\\n\
time_appconnect: %{time_appconnect} s\\n\
time_connect: %{time_connect} s\\n\
time_namelookup: %{time_namelookup} s\\n\
time_pretransfer: %{time_pretransfer} s\\n\
time_redirect: %{time_redirect} s\\n\
time_starttransfer: %{time_starttransfer} s\\n\
time_total: %{time_total} s\\n" \
"$1"
}
--8<---------------cut here---------------end--------------->8---

See "man curl" for the meaning of each metric.

You can then use this function to measure a substitute download.  Here's
an example in which I download a large substitute (linux-libre) from one
of my machines in Seattle:

--8<---------------cut here---------------start------------->8---
$ measure_get https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19 2>/dev/null
url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
http_code: 200
num_connects: 1
num_redirects: 0
remote_ip: 141.80.181.40
remote_port: 443
size_download: 69899433 B
speed_download: 4945831.000 B/s
time_appconnect: 0.885277 s
time_connect: 0.459667 s
time_namelookup: 0.254210 s
time_pretransfer: 0.885478 s
time_redirect: 0.000000 s
time_starttransfer: 1.273994 s
time_total: 14.133584 s
$ 
--8<---------------cut here---------------end--------------->8---

Here, it took 0.459667 - 0.254210 = 0.205457 seconds (about 205 ms) to
establish the TCP connection after the DNS lookup.  The average
throughput was 1924285 bytes per second (about 40 megabits per second,
where 1 megabit = 10^6 bits).  It seems my connection to berlin is
already pretty good!

We can get more information about latency by using a tool like mtr:

--8<---------------cut here---------------start------------->8---
$ sudo mtr -c 10 --report-wide --tcp -P 443 berlin.guixsd.org
Start: 2018-12-08T16:57:40-0800
HOST: localhost.localdomain                        Loss%   Snt   Last   Avg  Best  Wrst StDev
[... I've omitted the intermediate hops because they aren't relevant ...]
 13.|-- 141.80.181.40                                 0.0%    10  205.0 201.9 194.9 212.8   5.6
--8<---------------cut here---------------end--------------->8---

My machine's latency to berlin is about 202 ms, which matches what we
calculated above.

For experimentation, I've set up a CloudFront distribution at
berlin-mirror.marusich.info that uses berlin.guixsd.org as its origin
server.  Let's repeat these steps to measure the performance of the
distribution from my machine's perspective (before I did this, I made
sure the GET would result in a cache hit by downloading the substitute
once before and verifying that the same remote IP address was used):

--8<---------------cut here---------------start------------->8---
$ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19 2>/dev/null
url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
http_code: 200
num_connects: 1
num_redirects: 0
remote_ip: 13.32.254.57
remote_port: 443
size_download: 69899433 B
speed_download: 9821474.000 B/s
time_appconnect: 0.607593 s
time_connect: 0.532417 s
time_namelookup: 0.511086 s
time_pretransfer: 0.608029 s
time_redirect: 0.000000 s
time_starttransfer: 0.663578 s
time_total: 7.117266 s
$ sudo mtr -c 10 --report-wide --tcp -P 443 berlin-mirror.marusich.info
Start: 2018-12-08T17:04:48-0800
HOST: localhost.localdomain                        Loss%   Snt   Last   Avg  Best  Wrst StDev
[... I've omitted the intermediate hops because they aren't relevant ...]
 14.|-- server-52-84-21-199.sea32.r.cloudfront.net    0.0%    10   19.8  20.3  14.3  28.9   4.9
--8<---------------cut here---------------end--------------->8---

Establishing the TCP connection took about 21 ms (which matches the mtr
output), and the throughput was about 79 megabits per second.  (On this
machine, 100 Mbps is the current link speed, according to dmesg output.)
This means that in my case, when using CloudFront the latency is 10x
lower, and the throughput (for a cache hit) is 2x higher, than using
berlin.guixsd.org directly!

It would be interesting to see what the performance is for others.

Ricardo Wurmus <rekado@elephly.net> writes:

> Large ISPs also provide CDN services.  I already contacted Deutsche
> Telekom so that we can compare their CDN offer with the Amazon Cloudfont
> setup that Chris has configured.

That's great!  There are many CDN services out there.  I am unfamiliar
with most of them.  It will be good to see how Deutsche Telekom's
offering compares to CloudFront.

FYI, CloudFront has edge locations in the following parts of the world:

https://aws.amazon.com/cloudfront/features/

Hartmut Goebel <hartmut@goebel-consult.de> writes:

> Am 03.12.2018 um 17:12 schrieb Ludovic Courtès:
>> Thus, I’m thinking about using a similar setup, but hosting the mirror
>> on some Big Corp CDN or similar.
>
> Isn't this a contradiction: Building a free infrastructure relaying on
> servers from some Big Corporation? Let allow the privacy concerns
> raising when delivering data via some Big Corporation.
>
> If delivering "packages" works via static data without requiring any
> additional service, we could ask universities to host Guix, too. IMHO
> this is a much preferred solution since this is a decentralized publish
> infrastructure already in place for many GNU/Linux distributions.

I understand your concern about using a third-party service.  However,
we wouldn't be using a CDN as a "software substitute", which is one of
the primary risks of using a web service today:

https://www.gnu.org/philosophy/who-does-that-server-really-serve.html

Instead, we would be using a CDN as a performance optimization that is
transparent to a Guix user.  You seem unsettled by the idea of
entrusting any part of substitute delivery to a third party, but
concretely what risks do you foresee?

Regarding your suggestion to ask universities to host mirrors (really,
caching proxies), I think it could be a good idea.  As Leo mentioned,
the configuration to set up an NGINX caching proxy of Hydra (or berlin)
is freely available in maintenance.git.  Do you think we could convince
some universities to host caching proxies that just run an NGINX web
server using those configurations?

If we can accomplish that, it may still be helpful.  If there is
interest in going down this path, I can explore some possibilities in
the Seattle area.  If the university-owned caching proxies are easily
discoverable (i.e., we list them on the website), then users might
manually set their substitute URL to point to one that's close by.

Going further, if our DNS provider supports something like "geolocation
routing" for DNS queries, we might even be able to create DNS records
for ci.guix.info that point to those universities' caching proxies.  In
this way, when a user resolves ci.guix.info, they would get the address
of a university-owned caching proxy close by.  This could have the
benefits of requiring less money than a full-fledged CDN like Amazon
CloudFront, and also decentralizing the substitute delivery, while still
remaining transparent to Guix users.  However, it would still require us
to rely on a third-party DNS service.

For example, Amazon Route 53 provides this sort of geolocation routing:

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html#routing-policy-geo

I wouldn't be surprised if there are other DNS providers out there who
offer something similar.  However, I also wouldn't be surprised if the
overall performance of CloudFront turns out to be better.

"Thompson, David" <dthompson2@worcester.edu> writes:

> If AWS CloudFront is the path chosen, it may be worthwhile to follow
> the "infrastructure as code" practice and use CloudFormation to
> provision the CloudFront distribution and any other supporting
> resources. The benefit is that there would be a record of exactly
> *how* the project is using these commercial services and the setup
> could be easily reproduced.  The timing is interesting here because I
> just attended the annual AWS conference on behalf of my employer and
> while I was there I felt inspired to write a Guile API for building
> CloudFormation "stacks".  You can see a small sample of what it does
> here: https://gist.github.com/davexunit/db4b9d3e67902216fbdbc66cd9c6413e

Nice!  That seems useful.  I will have to play with it.  I created my
distributions manually using the AWS Management Console, since it's
relatively easy to do.  I agree it would be better to practice
"infrastructure as code."

On that topic, I've also heard good things about Terraform by HashiCorp,
which is available under the Mozilla Public License 2.0:

https://github.com/hashicorp/terraform

Here is a comparison of Terraform and CloudFormation:

https://www.terraform.io/intro/vs/cloudformation.html

I looked briefly into packaging Terraform for Guix.  It's written in Go.
It seems possible, but I haven't invested enough time yet.

As a final option, since the AWS CLI is already packaged in Guix, we
could just drive CloudFormation or CloudFront directly from the CLI.

Meiyo Peng <meiyo.peng@gmail.com> writes:

> I like the idea of IPFS. We should try it. It would be great if it works
> well.
>
> If at some point we need to setup traditional mirrors like other major
> Gnu/Linux distros, I can contact my friends in China to setup mirrors in
> several universities. I was a member of LUG@USTC, which provides the
> largest FLOSS mirror in China.

IPFS would be neat.  So would Gnunet.  Heck, even a publication
mechanism using good old BitTorrent would be nice.  All of these would
require changes to Guix, I suppose.

A CDN would require no changes to Guix, and that's part of why it's so
appealing.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Using a CDN or some other mirror?

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 3938 bytes --]

Am 09.12.2018 um 04:33 schrieb Chris Marusich:
> Instead, we would be using a CDN as a performance optimization that is
> transparent to a Guix user.  You seem unsettled by the idea of
> entrusting any part of substitute delivery to a third party, but
> concretely what risks do you foresee?

I have serious privacy concerns.

TL;DR: A CDN is a centralized infrastructure, allowing to collect
information about valuable vulnerability information of almost all
Guix-users and -systems. This is might become a thread to freedom of
speech, human rights, democracy and economics. Guix should build on a
decentralized infrastructure.

A distribution provider gets a notion which system is running which
software in which version. In case of guix, the provider even gets the
exact version of the software and all its dependencies. Combining this
with the rise of IPv6, which per default uses the MAC address as part of
the IP address, actually allows identifying a single system.

This information is extremely valuable for all kinds of attackers as it
eases attacking a system a lot. This becomes a thread to

  * to opposition members, dissidents and human rights activists as the
    intelligent agencies can target these persons much more precisely,
  * to companies all over the world as many countries do industrial
    espionage.

This becomes even worst when using a CDN, since the CDN is a centralized
system: A single CDN provider gains knowledge for almost all systems all
over the world. Which means: this valuable vulnerability information is
collected at a single place. Intelligence agencies might be keen on
getting access to this information and a centralized system makes it
easy for them. And there is evidence they actually collect this
information [*].

This gets even worse when the CDN belongs to one of these companies
compiling personal profiles, like Google, Facebook or Tencent. Amazon
belongs to this group.

I have the strong opinion that Guix should build on a decentralized
infrastructure to support keeping the freedom of speech, democracy and
human rights.

[*] Actually it is known the US-American intelligence agencies have
equipment placed at Verizon to collect all kind of data [1]. One can
reason the same is true for other big providers in the US. The USA has
the FISA act AFAIU enforcing US companies to collaborate in industrial
espionage. In Germany it is known that the BND is extracting high-volume
data at the central internet exchange (DE-CIX) [2]. One can reason such
also happens in other countries, esp. members of the five-eyes, France,
Russia, China, Israel, Saudi Arabia, Iran, Irak, etc.

> Regarding your suggestion to ask universities to host mirrors (really,
> caching proxies), I think it could be a good idea.  As Leo mentioned,
> the configuration to set up an NGINX caching proxy of Hydra (or berlin)
> is freely available in maintenance.git.  Do you think we could convince
> some universities to host caching proxies that just run an NGINX web
> server using those configurations?

The difference is: For a traditional "ftp"-mirror, an organization just
needs to add another source to its existing configuration and administer
to the save way as all other mirrors. Whereas for a caching proxy they
need to change the setup of the web-server and learn how to administer
the cache. This difference might make it difficult to convince
organizations to mirror.

I could try and ask a few organizations in my area, but I would need
figures for this.


[1] https://www.bbc.com/news/world-us-canada-23123964 or search the
internet for e.g. "cia verizon espionage"
[2]
https://www.heise.de/newsticker/meldung/Gerichtsurteil-BND-darf-weiterhin-Internet-Knoten-De-CIX-anzapfen-4061494.html
[3] https://en.wikipedia.org/wiki/Foreign_Intelligence_Surveillance_Act
[4]

-- 
+++hartmut

| Hartmut Goebel            |                       |
| hartmut@goebel-consult.de | www.goebel-consult.de |


[-- Attachment #2: Type: text/html, Size: 5368 bytes --]

Re: Using a CDN or some other mirror?

[Ludovic Courtès]

Hi Hartmut,

Hartmut Goebel <hartmut@goebel-consult.de> skribis:

> Am 09.12.2018 um 04:33 schrieb Chris Marusich:
>> Instead, we would be using a CDN as a performance optimization that is
>> transparent to a Guix user.  You seem unsettled by the idea of
>> entrusting any part of substitute delivery to a third party, but
>> concretely what risks do you foresee?
>
> I have serious privacy concerns.
>
> TL;DR: A CDN is a centralized infrastructure, allowing to collect
> information about valuable vulnerability information of almost all
> Guix-users and -systems. This is might become a thread to freedom of
> speech, human rights, democracy and economics. Guix should build on a
> decentralized infrastructure.

Heck it would be ironic to find myself arguing in favor of centralized
commercial services.  So I won’t do that.  :-)

Clearly, I do understand the concerns you list.  As a maintainer, I’m
looking for solutions that can address real problems (availability of
substitutes and bandwidth) while not being a threat to our user’s
privacy and security.

The operator of a substitute server (or caching proxy), in general,
knows which IPs downloaded vulnerable software.  This is the main
threat.

This can be mitigated by talking to nearby mirrors and not just
ci.guix.info, a feature we implemented a year ago (see
<https://gnu.org/s/guix/blog/2017/reproducible-builds-a-status-update/>),
or by using several substitute servers, or by not using (or not always
using) substitutes.  Few distros have all these options.

We might also be able to somehow balance requests between several CDNs
or mirrors.

But again, medium- to long-term, the goal is to move towards IPFS or
GNUnet/Bittorrent.  IPFS is attractive because it would probably require
no modifications to ‘guix substitutes’ and only minor changes to ‘guix
publish’ since the IPFS daemon has an HTTP interface.

>> Regarding your suggestion to ask universities to host mirrors (really,
>> caching proxies), I think it could be a good idea.  As Leo mentioned,
>> the configuration to set up an NGINX caching proxy of Hydra (or berlin)
>> is freely available in maintenance.git.  Do you think we could convince
>> some universities to host caching proxies that just run an NGINX web
>> server using those configurations?
>
> The difference is: For a traditional "ftp"-mirror, an organization just
> needs to add another source to its existing configuration and administer
> to the save way as all other mirrors. Whereas for a caching proxy they
> need to change the setup of the web-server and learn how to administer
> the cache. This difference might make it difficult to convince
> organizations to mirror.
>
> I could try and ask a few organizations in my area, but I would need
> figures for this.

What would you need to know?  ‘guix weather’ can provide info about
storage size.

Thanks,
Ludo’.

Re: Using a CDN or some other mirror?

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

Hi all,

my two cents...

(I can't still help with a public cache, I hope soon...)

Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> TL;DR: A CDN is a centralized infrastructure, allowing to collect
>> information about valuable vulnerability information of almost all
>> Guix-users and -systems. This is might become a thread to freedom of
>> speech, human rights, democracy and economics. Guix should build on a
>> decentralized infrastructure.

I completely agree with you, decentralization is the solution

unfortunately the **only functioning** way is to avoid current Internet,
since it's broken (https://youbroketheinternet.org/); I see GuixSD as an
integral part of The Project Map™ https://youbroketheinternet.org/map

...but to fix the situation we need a substantial GNUnet(work) effect
and for that we _need_ GuixSD substitutes to be easily and quickly
downloaded (can we avoid this asking potential adopters to be patient or
to build?)

maybe we should divide this task in two steps:

1. distributed substitutes: caching servers hosted by a network of
friendly institutions and companies donated to GNU/GuixSD, with a
haproxy frontend for geolocated load-balancing [1]

2. decentralized substitutes: caching servers on IPFS or better (since
it allows complete anonimity) on GNUnet

> Heck it would be ironic to find myself arguing in favor of centralized
> commercial services.  So I won’t do that.  :-)

I see no problems with commercial services, _unfortunately_ nowadays
this *almost* always means centralized silos, usually exploited for
global surveillance (since Internet is broken)

[...]

> The operator of a substitute server (or caching proxy), in general,
> knows which IPs downloaded vulnerable software.  This is the main
> threat.

on Internet, and on IPFS? (sorry for the ignorance)

on GNUNet filesharing can be completely anonymous, but the performace is
degraded (so we need a large network effect here)

> This can be mitigated by talking to nearby mirrors and not just
> ci.guix.info, a feature we implemented a year ago (see
> <https://gnu.org/s/guix/blog/2017/reproducible-builds-a-status-update/>),
> or by using several substitute servers, or by not using (or not always
> using) substitutes.  Few distros have all these options.
>
> We might also be able to somehow balance requests between several CDNs
> or mirrors.

did someone explored an haproxy (with geolocation) solution?

is there a wip-haproxy attempt?

[...]

HTH
Giovanni


[1] in the next few weeks I'm going to test an haproxy instance with
geolocated ACLs following this directions
https:/www.haproxy.com/blog/use-geoip-database-within-haproxy/

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Using a CDN or some other mirror?

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Am 09.12.2018 um 14:58 schrieb Ludovic Courtès:
>> I could try and ask a few organizations in my area, but I would need
>> figures for this.
> What would you need to know?  ‘guix weather’ can provide info about
> storage size.

I don't know yet, which info the admins need for a decision. FMPOV I'd
says: Disk-space and traffic to be expected.

`guix weather` only provides the disk-space, but even this is not
obvious for me:

  13912.1 MiB of nars (compressed)
  41176.6 MiB on disk (uncompressed)

From reading the manual, I assume 13.9 GB are required on the server
(which is quite a lot IMHO). Is this correct?

-- 
+++hartmut

| Hartmut Goebel            |                       |
| hartmut@goebel-consult.de | www.goebel-consult.de |


[-- Attachment #2: Type: text/html, Size: 1435 bytes --]

Re: Using a CDN or some other mirror?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

Talking about this, I recently discussed with Ludovic the idea of compressing
nars in Lzip instead of gzip.

http://lzip.nongnu.org/lzip.html (benchmark included)

I can work on some Lzip guile-bindings, it should be quite easy, then we could
save some 10-50% (!!!) disk usage.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Compressing nars with lzip or similar

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Talking about this, I recently discussed with Ludovic the idea of compressing
> nars in Lzip instead of gzip.
>
> http://lzip.nongnu.org/lzip.html (benchmark included)
>
> I can work on some Lzip guile-bindings, it should be quite easy, then we could
> save some 10-50% (!!!) disk usage.

That would be sweet!

Though we have to keep in mind that today’s ‘guix substitute’ doesn’t
know about lzip at all (see (guix scripts substitute) and
‘call-with-decompressed-port’.)  So while lzip would be my preference,
we may not be able to use it until we can be sure our users can actually
unpack it.

Ludo’.

Re: Compressing nars with lzip or similar

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 174 bytes --]

Absolutely, but once the bindings are up and running, it should be
straightforward to update (guix scripts substitute), no?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Compressing nars with lzip or similar

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

I've done some quick research over the various options.

- lzip: better than gz and bzip2 for sure, possibly better than xz (at
least according to the author).

- plzip: for "parallel lzip".  With 4 threads I was able to compress
icecat 2.5x faster.  It used 5x more memory though.  The compression
ratio is 1-2% worse.

- lrzip: it would crash whenever I would change the compression level.
  Seems less stable.  It's as fast as plzip, while being 1-2% less
  compressed.  I don't think it's worth using.

All in all, lzip is a definite win over most options.  The main question
is: lzip or plzip?

In my opinion it makes more sense to use plzip, gotta put those cores to
some use (on the user side at least)!  Not sure what the build farm
would think about this though.

Finally, plzip can be quite memory-intensive.  When passed "--threads=1"
however, it gets closer to lzip (bit faster, bit more memory).  So maybe
the ideal would be to support plzip with a user-settable "threads" option.

Thoughts?

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Compressing nars with lzip or similar

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> I've done some quick research over the various options.
>
> - lzip: better than gz and bzip2 for sure, possibly better than xz (at
> least according to the author).
>
> - plzip: for "parallel lzip".  With 4 threads I was able to compress
> icecat 2.5x faster.  It used 5x more memory though.  The compression
> ratio is 1-2% worse.
>
> - lrzip: it would crash whenever I would change the compression level.
>   Seems less stable.  It's as fast as plzip, while being 1-2% less
>   compressed.  I don't think it's worth using.
>
> All in all, lzip is a definite win over most options.  The main question
> is: lzip or plzip?

‘guix publish’ has its own worker pool and handles parallelism
internally, so in that context plain sequential lzip would be more
appropriate IMO.

Ludo’.

Re: Compressing nars with lzip or similar

[Pierre Neidhardt]

I'll get down to this this week.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

Re: Compressing nars with lzip or similar

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Absolutely, but once the bindings are up and running, it should be
> straightforward to update (guix scripts substitute), no?

Sure!  But it will take time before we can assume all our users are
running it.

Ludo’.

Re: Using a CDN or some other mirror?

[Ludovic Courtès]

Hello,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 09.12.2018 um 14:58 schrieb Ludovic Courtès:
>>> I could try and ask a few organizations in my area, but I would need
>>> figures for this.
>> What would you need to know?  ‘guix weather’ can provide info about
>> storage size.
>
> I don't know yet, which info the admins need for a decision. FMPOV I'd
> says: Disk-space and traffic to be expected.
>
> `guix weather` only provides the disk-space, but even this is not
> obvious for me:
>
>   13912.1 MiB of nars (compressed)
>   41176.6 MiB on disk (uncompressed)
>
> From reading the manual, I assume 13.9 GB are required on the server
> (which is quite a lot IMHO). Is this correct?

If you’re running a caching proxy, you’ll need 13G.  However note that
it’s only for one architecture and one revision of Guix.  The total
space needed is obviously a function of time (number of Guix revisions
served) and number of architectures.

You could choose an expiration time in your caching proxy that satisfies
your disk space constraints, though.

Our machines that run Cuirass + ‘guix publish --cache’ need roughly
41+13G since they contain both /gnu/store and /var/cache/guix/publish.

HTH!

Ludo’.

CDN performance

[Ludovic Courtès]

Hello Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> Regarding DNS, it would be nice if we could use an official GNU
> subdomain.  If we can't use a GNU subdomain, we should at least make
> sure we have some kind of DNS auto-renewal set up so that nobody can
> poach our domain names.  And the operators should take appropriate
> precautions when sharing any credentials used for managing it all.

Agreed.

Regarding the GNU sub-domain, as I replied to Meiyo, I’m in favor of it,
all we need is someone to champion setting it up.

> Regarding CDNs, I definitely think it's worth a try!  Even Debian is
> using CloudFront (cloudfront.debian.net).  In fact, email correspondence
> suggests that as of 2013, Amazon may even have been paying for it!
>
> https://lists.debian.org/debian-cloud/2013/05/msg00071.html

(Note that debian.net is not Debian, and “there’s no cloud, only other
people’s computer” as the FSFE puts it.)

> Although I don't doubt that a CDN will perform better than what we have
> now, I do think it would be good to measure the performance so that we
> know for sure the money spent is actually providing a benefit.  It would
> be nice to have some data before and after to measure how availability
> and performance have changed.  Apart from anecdotes, what data do we
> have to determine whether performance has improved after introducing a
> CDN?  For example, the following information could be useful:
>
>   * Network load on the origin server(s)
>   * Clients' latency to (the addresses pointed to by) ci.guix.info
>   * Clients' throughput while downloading substitutes from ci.guix.info

Note that performance is one aspect; another one is availability (we’ve
seen that with the recent hydra.gnu.org outage!).  We know we’ll always
win in terms of availability by having a CDN in front of our servers.

That said, measuring performance is very useful and it’s great that we
can benefit from your expertise here!

> Here, it took 0.459667 - 0.254210 = 0.205457 seconds (about 205 ms) to
> establish the TCP connection after the DNS lookup.  The average
> throughput was 1924285 bytes per second (about 40 megabits per second,
> where 1 megabit = 10^6 bits).  It seems my connection to berlin is
> already pretty good!

Indeed.  The bandwidth problem on berlin is when you’re the first to
download a nar and it’s not been cached by nginx yet.  In that case, you
get very low bandwidth (like 10 times less than when the item is cached
by nginx.)  I’ve looked into it, went as far as strace’ing nginx, but
couldn’t find the reason of this.

Do you any idea?

> Establishing the TCP connection took about 21 ms (which matches the mtr
> output), and the throughput was about 79 megabits per second.  (On this
> machine, 100 Mbps is the current link speed, according to dmesg output.)
> This means that in my case, when using CloudFront the latency is 10x
> lower, and the throughput (for a cache hit) is 2x higher, than using
> berlin.guixsd.org directly!

Impressive.

> It would be interesting to see what the performance is for others.

I’ve tried this from home (in France, with FTTH):

--8<---------------cut here---------------start------------->8---
$ measure_get https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19  2>/dev/null
url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
http_code: 200
num_connects: 1
num_redirects: 0
remote_ip: 141.80.181.40
remote_port: 443
size_download: 69899433 B
speed_download: 1522001.000 B/s
time_appconnect: 0.178892 s
time_connect: 0.049649 s
time_namelookup: 0.000422 s
time_pretransfer: 0.178934 s
time_redirect: 0.000000 s
time_starttransfer: 0.278312 s
time_total: 45.926021 s
$ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19 2>/dev/null
url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
http_code: 200
num_connects: 1
num_redirects: 0
remote_ip: 2600:9000:2116:6e00:c:49d4:5a80:93a1
remote_port: 443
size_download: 69899433 B
speed_download: 20803402.000 B/s
time_appconnect: 0.552008 s
time_connect: 0.482477 s
time_namelookup: 0.467598 s
time_pretransfer: 0.552157 s
time_redirect: 0.000000 s
time_starttransfer: 0.735758 s
time_total: 3.360500 s
--8<---------------cut here---------------end--------------->8---

Wall-clock time is less than a tenth; woow.

Thanks for sharing your insight and scripts!

Ludo’.

Re: CDN performance

[Meiyo Peng]

Hi,

I have changed my mind. I don't care about the domain name of the
default substitute server any more. I am more worried about the network
speed within China. While I am writing this email, my computer is
downloading substitutes from ci.guix.info at an average speed of about
50kB/s.

After careful thought, I realized the new CDN won't benefit China
residents as planned. Any popular CDN outside China is significantly
throttled by ISP/GFW and the situation is worse every year. A CDN will
be a great improvement for western countries but not for many asia
countries.

I have been running a nginx caching mirror at home since the first time
I met guix. My first attempt to install guix took me one afternoon and
one night. Most of the time was wasted on downloading substitutes. After
that, I searched online and found the nginx mirror configuration in
guix/maintenance.git. Then I set up my own caching mirror. It would be
extremely painful for me to play with guix during these months without
this caching mirror. Debian was my favorite GNU/Linux distro before I
discovered Guix. It usually costs me only 30 minutes to install Debian
from scratch. Other GNU/Linux distros are about the same. If guix takes
more than 5 hours to install, most new users will be scared away.

Although now there are only a few Chinese guix users, I still prefer to
set up a caching mirror within China. The user experience improvement
will be significant. Like Hartmut Goebel described, I need information
about "expected storage demand, expected transfer traffic from upstream,
how often updates are published, how long to hold copies" so I can write
a proposal to maintainers of mirrors.ustc.edu.cn to add a new mirror for
guix. I tried the `guix weather` command, but it failed with an error:

#+BEGIN_EXAMPLE
  computing 8,821 package derivations for x86_64-linux...
  guix weather: error: lstat: No such file or directory: "/src/config/packages/patches/libjxr-add-cmakelists-file.patch"
#+END_EXAMPLE

Can anyone provide the numbers for me? The numbers of
mirror.hydra.gnu.org would be a good reference.

In addition, I think we should put up a page on guix's website providing
information about available substitute servers including address,
backend build farm, geolocation, maintainer, so users (many asia
residents) can pick the best substitute server for themselves. This will
be necessary until alternative solutions like IPFS or GNUnet are
implemented.

This email only describes the situation in China. I don't intend to
discourage you on the CDN plan. There is no conflicts between my
proposal to set up a caching mirror in China and our plan to set up a
new CDN.

--
Meiyo Peng

Re: CDN performance

[Meiyo Peng]

Hi,

Zephyr Waitzman <i@wi24rd.ml> writes:

> 大哥你的镜像速度怎么样？可以蹭网吗？（手动滑稽

My caching mirror is at home and it has an ip address of
192.168.x.x. You may connect to it if you have magic power.

BTW. Nice to see another Chinese here. What's your experience
with the default substitute server?

--
Meiyo Peng

Re: CDN performance

[Meiyo Peng]

Hi,

Zephyr Waitzman <i@wi24rd.ml> writes:

> 以前官方的慢死人，最近的话安装靠玄学（找以前邮件列表里的几个 substitude 换着用也就那样吧），偶尔会出一些（感觉不像是网络的）问题。
> 以后有时间再深入学习下。

Thank you for your feedback. That means we do need to set up a caching
mirror within China.

--
Meiyo Peng

Re: CDN performance

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1778 bytes --]

Meiyo Peng <meiyo.peng@gmail.com> writes:

> After careful thought, I realized the new CDN won't benefit China
> residents as planned. Any popular CDN outside China is significantly
> throttled by ISP/GFW and the situation is worse every year. A CDN will
> be a great improvement for western countries but not for many asia
> countries.

Could you try running the measure_get shell function I included in the
following email?

https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html

For convenience, here is the definition:

--8<---------------cut here---------------start------------->8---
measure_get () {
curl -L \
     -o /dev/null \
     -w "url_effective: %{url_effective}\\n\
http_code: %{http_code}\\n\
num_connects: %{num_connects}\\n\
num_redirects: %{num_redirects}\\n\
remote_ip: %{remote_ip}\\n\
remote_port: %{remote_port}\\n\
size_download: %{size_download} B\\n\
speed_download: %{speed_download} B/s\\n\
time_appconnect: %{time_appconnect} s\\n\
time_connect: %{time_connect} s\\n\
time_namelookup: %{time_namelookup} s\\n\
time_pretransfer: %{time_pretransfer} s\\n\
time_redirect: %{time_redirect} s\\n\
time_starttransfer: %{time_starttransfer} s\\n\
time_total: %{time_total} s\\n" \
"$1"
}
--8<---------------cut here---------------end--------------->8---

Specifically, I am curious to know what performance you get when you run

  measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19

from a computer in China.  Please be sure to run it two times in a row,
to ensure that CloudFront has cached the object.

CloudFront has edge locations in Hong Kong, so I am curious to know what
performance improvement, if any, you observe.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: CDN performance

[Meiyo Peng]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> writes:

> Meiyo Peng <meiyo.peng@gmail.com> writes:
>
>> After careful thought, I realized the new CDN won't benefit China
>> residents as planned. Any popular CDN outside China is significantly
>> throttled by ISP/GFW and the situation is worse every year. A CDN will
>> be a great improvement for western countries but not for many asia
>> countries.
>
> Could you try running the measure_get shell function I included in the
> following email?
>
> https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html
>
> For convenience, here is the definition:
>
> measure_get () {
> curl -L \
>      -o /dev/null \
>      -w "url_effective: %{url_effective}\\n\
> http_code: %{http_code}\\n\
> num_connects: %{num_connects}\\n\
> num_redirects: %{num_redirects}\\n\
> remote_ip: %{remote_ip}\\n\
> remote_port: %{remote_port}\\n\
> size_download: %{size_download} B\\n\
> speed_download: %{speed_download} B/s\\n\
> time_appconnect: %{time_appconnect} s\\n\
> time_connect: %{time_connect} s\\n\
> time_namelookup: %{time_namelookup} s\\n\
> time_pretransfer: %{time_pretransfer} s\\n\
> time_redirect: %{time_redirect} s\\n\
> time_starttransfer: %{time_starttransfer} s\\n\
> time_total: %{time_total} s\\n" \
> "$1"
> }
>
> Specifically, I am curious to know what performance you get when you run
>
>   measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>
> from a computer in China.  Please be sure to run it two times in a row,
> to ensure that CloudFront has cached the object.
>
> CloudFront has edge locations in Hong Kong, so I am curious to know what
> performance improvement, if any, you observe.

Sorry for the delay. My computer was reinstalled with Windows and taken
away by my girlfriend. So I have been waiting for my new computer that I
bought online to arrive. Finally, it arrived yesterday and I
successfully installed Guix on it.

I tested your script several times.

1. Tested today at home. China Unicom home broadband. 50Mb/s.

The result is slow as usual. curl failed once.
berlin-mirror.marusich.info is resolved to Seattle, WA, US.

#+BEGIN_EXAMPLE
  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
   55 66.6M   55 36.9M    0     0  17926      0  1:04:59  0:36:02  0:28:57 17733
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 52.85.158.151
  remote_port: 443
  size_download: 38764357 B
  speed_download: 17926.000 B/s
  time_appconnect: 6.078850 s
  time_connect: 3.006821 s
  time_namelookup: 2.659785 s
  time_pretransfer: 6.079097 s
  time_redirect: 0.000000 s
  time_starttransfer: 9.626001 s
  time_total: 2162.379211 s
  curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0   109k      0  0:10:25  0:10:25 --:--:--  241k
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 52.85.158.22
  remote_port: 443
  size_download: 69899433 B
  speed_download: 111816.000 B/s
  time_appconnect: 3.507528 s
  time_connect: 2.650373 s
  time_namelookup: 2.261801 s
  time_pretransfer: 3.507637 s
  time_redirect: 0.000000 s
  time_starttransfer: 5.995298 s
  time_total: 625.129571 s

  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0   109k      0  0:10:23  0:10:23 --:--:--  141k
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 52.85.158.22
  remote_port: 443
  size_download: 69899433 B
  speed_download: 112187.000 B/s
  time_appconnect: 2.280972 s
  time_connect: 1.407197 s
  time_namelookup: 1.056180 s
  time_pretransfer: 2.281234 s
  time_redirect: 0.000000 s
  time_starttransfer: 3.167703 s
  time_total: 623.061584 s
#+END_EXAMPLE


2. Tested 3 days ago at my office. China Telecom enterprise broadband. 50Mb/s.

Unusually fast! berlin-mirror.marusich.info is resolved to Seattle, WA,
US. I have no idea why it's so fast that day.

#+BEGIN_EXAMPLE
  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  1364k      0  0:00:50  0:00:50 --:--:-- 1352k
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 13.35.20.109
  remote_port: 443
  size_download: 69899433 B
  speed_download: 1397429.000 B/s
  time_appconnect: 2.432387 s
  time_connect: 0.200842 s
  time_namelookup: 0.000446 s
  time_pretransfer: 2.432659 s
  time_redirect: 0.000000 s
  time_starttransfer: 2.673045 s
  time_total: 50.020945 s

  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  1592k      0  0:00:42  0:00:42 --:--:-- 2506k
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 13.35.20.109
  remote_port: 443
  size_download: 69899433 B
  speed_download: 1630687.000 B/s
  time_appconnect: 0.653270 s
  time_connect: 0.209455 s
  time_namelookup: 0.001582 s
  time_pretransfer: 0.658399 s
  time_redirect: 0.000000 s
  time_starttransfer: 0.883126 s
  time_total: 42.865868 s
#+END_EXAMPLE


3. Tested today at my office. China Telecom enterprise broadband. 50Mb/s.

Slow as usual. berlin-mirror.marusich.info is still resolved to Seattle,
WA, US. I killed the program several times because it hung there with no
data transfer for a few minutes. The TCP connection was probably closed
by GFW. This is very common here.

#+BEGIN_EXAMPLE
  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  48110      0  0:24:12  0:24:12 --:--:-- 41808
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 52.85.158.151
  remote_port: 443
  size_download: 69899433 B
  speed_download: 48110.000 B/s
  time_appconnect: 0.872926 s
  time_connect: 0.282048 s
  time_namelookup: 0.000524 s
  time_pretransfer: 0.873099 s
  time_redirect: 0.000000 s
  time_starttransfer: 1.187467 s
  time_total: 1452.904154 s

  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    2 66.6M    2 1809k    0     0   5760      0  3:22:15  0:05:21  3:16:54     0^C%

  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
   52 66.6M   52 34.9M    0     0  16772      0  1:09:27  0:36:26  0:33:01     0^C%

  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  58181      0  0:20:01  0:20:01 --:--:-- 87975
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 52.85.158.22
  remote_port: 443
  size_download: 69899433 B
  speed_download: 58181.000 B/s
  time_appconnect: 2.297713 s
  time_connect: 1.904176 s
  time_namelookup: 1.727602 s
  time_pretransfer: 2.297974 s
  time_redirect: 0.000000 s
  time_starttransfer: 2.503263 s
  time_total: 1201.408929 s
#+END_EXAMPLE


Well. As you see, the network in China is both slow and unstable.

Re: CDN performance

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 8937 bytes --]

Hi Meiyo,

Thank you for sharing this information with us!

Can you also share what numbers you get when you run measure_get against
berlin.guixsd.org directly?  Clearly, the connection from you to
CloudFront is not as performant as it is for others in other parts of
the world, but I wonder if it's still better than accessing berlin
directly.  If you could run measure_get against berlin directly and
share the numbers, we can see if it represents any significant
improvement for you.

Meiyo Peng <meiyo.peng@gmail.com> writes:

> I tested your script several times.
>
> 1. Tested today at home. China Unicom home broadband. 50Mb/s.
>
> The result is slow as usual. curl failed once.
> berlin-mirror.marusich.info is resolved to Seattle, WA, US.

Well, that's not great.  Perhaps it's still better than it would be if
the DNS name resolved to a location in Europe, though.

> #+BEGIN_EXAMPLE
>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>    55 66.6M   55 36.9M    0     0  17926      0  1:04:59  0:36:02  0:28:57 17733
>   url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 52.85.158.151
>   remote_port: 443
>   size_download: 38764357 B
>   speed_download: 17926.000 B/s
>   time_appconnect: 6.078850 s
>   time_connect: 3.006821 s
>   time_namelookup: 2.659785 s
>   time_pretransfer: 6.079097 s
>   time_redirect: 0.000000 s
>   time_starttransfer: 9.626001 s
>   time_total: 2162.379211 s
>   curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

I see, this is about 143 Kbps (not Mbps) of throughput, and 347 ms after
DNS name resolution.

>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>   100 66.6M  100 66.6M    0     0   109k      0  0:10:25  0:10:25 --:--:--  241k
>   url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 52.85.158.22
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 111816.000 B/s
>   time_appconnect: 3.507528 s
>   time_connect: 2.650373 s
>   time_namelookup: 2.261801 s
>   time_pretransfer: 3.507637 s
>   time_redirect: 0.000000 s
>   time_starttransfer: 5.995298 s
>   time_total: 625.129571 s
>
>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>   100 66.6M  100 66.6M    0     0   109k      0  0:10:23  0:10:23 --:--:--  141k
>   url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 52.85.158.22
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 112187.000 B/s
>   time_appconnect: 2.280972 s
>   time_connect: 1.407197 s
>   time_namelookup: 1.056180 s
>   time_pretransfer: 2.281234 s
>   time_redirect: 0.000000 s
>   time_starttransfer: 3.167703 s
>   time_total: 623.061584 s
> #+END_EXAMPLE

897 Kbps, 351 ms after the name lookup.

> 2. Tested 3 days ago at my office. China Telecom enterprise broadband. 50Mb/s.
>
> Unusually fast! berlin-mirror.marusich.info is resolved to Seattle, WA,
> US. I have no idea why it's so fast that day.
>
> #+BEGIN_EXAMPLE
>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>   100 66.6M  100 66.6M    0     0  1364k      0  0:00:50  0:00:50 --:--:-- 1352k
>   url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 13.35.20.109
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 1397429.000 B/s
>   time_appconnect: 2.432387 s
>   time_connect: 0.200842 s
>   time_namelookup: 0.000446 s
>   time_pretransfer: 2.432659 s
>   time_redirect: 0.000000 s
>   time_starttransfer: 2.673045 s
>   time_total: 50.020945 s

11 Mbps, 200 ms after the name lookup.

>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>   100 66.6M  100 66.6M    0     0  1592k      0  0:00:42  0:00:42 --:--:-- 2506k
>   url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 13.35.20.109
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 1630687.000 B/s
>   time_appconnect: 0.653270 s
>   time_connect: 0.209455 s
>   time_namelookup: 0.001582 s
>   time_pretransfer: 0.658399 s
>   time_redirect: 0.000000 s
>   time_starttransfer: 0.883126 s
>   time_total: 42.865868 s
> #+END_EXAMPLE

13 Mbps, 208 ms after the name lookup.

> 3. Tested today at my office. China Telecom enterprise broadband. 50Mb/s.
>
> Slow as usual. berlin-mirror.marusich.info is still resolved to Seattle,
> WA, US. I killed the program several times because it hung there with no
> data transfer for a few minutes. The TCP connection was probably closed
> by GFW. This is very common here.
>
> #+BEGIN_EXAMPLE
>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>   100 66.6M  100 66.6M    0     0  48110      0  0:24:12  0:24:12 --:--:-- 41808
>   url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 52.85.158.151
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 48110.000 B/s
>   time_appconnect: 0.872926 s
>   time_connect: 0.282048 s
>   time_namelookup: 0.000524 s
>   time_pretransfer: 0.873099 s
>   time_redirect: 0.000000 s
>   time_starttransfer: 1.187467 s
>   time_total: 1452.904154 s

385 Kbps, 282 ms.

>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>     2 66.6M    2 1809k    0     0   5760      0  3:22:15  0:05:21  3:16:54     0^C%
>
>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>    52 66.6M   52 34.9M    0     0  16772      0  1:09:27  0:36:26  0:33:01     0^C%
>
>   ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>   100 66.6M  100 66.6M    0     0  58181      0  0:20:01  0:20:01 --:--:-- 87975
>   url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 52.85.158.22
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 58181.000 B/s
>   time_appconnect: 2.297713 s
>   time_connect: 1.904176 s
>   time_namelookup: 1.727602 s
>   time_pretransfer: 2.297974 s
>   time_redirect: 0.000000 s
>   time_starttransfer: 2.503263 s
>   time_total: 1201.408929 s

465 Kbps, 177 ms.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: CDN performance

[Meiyo Peng]

Hi Chris,

Thank you for your patience!

Chris Marusich <cmmarusich@gmail.com> writes:

> Can you also share what numbers you get when you run measure_get against
> berlin.guixsd.org directly?  Clearly, the connection from you to
> CloudFront is not as performant as it is for others in other parts of
> the world, but I wonder if it's still better than accessing berlin
> directly.  If you could run measure_get against berlin directly and
> share the numbers, we can see if it represents any significant
> improvement for you.

1. Tested today at home. China Unicom home broadband. 50Mb/s.

berlin.guixsd.org:

#+BEGIN_EXAMPLE
  ➜  ~ measure_get https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
   54 66.6M   54 36.3M    0     0  14981      0  1:17:45  0:42:25  0:35:20     0
  url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 141.80.181.40
  remote_port: 443
  size_download: 38141765 B
  speed_download: 14981.000 B/s
  time_appconnect: 3.228601 s
  time_connect: 2.213136 s
  time_namelookup: 0.856194 s
  time_pretransfer: 3.228820 s
  time_redirect: 0.000000 s
  time_starttransfer: 3.851583 s
  time_total: 2545.889968 s
  curl: (56) GnuTLS recv error (-54): Error in the pull function.

  ➜  ~ measure_get https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  20415      0  0:57:03  0:57:03 --:--:-- 25983
  url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 141.80.181.40
  remote_port: 443
  size_download: 69899433 B
  speed_download: 20415.000 B/s
  time_appconnect: 2.005881 s
  time_connect: 0.785257 s
  time_namelookup: 0.000520 s
  time_pretransfer: 2.006124 s
  time_redirect: 0.000000 s
  time_starttransfer: 3.031582 s
  time_total: 3423.813489 s
#+END_EXAMPLE

berlin-mirror.marusich.info:

#+BEGIN_EXAMPLE
  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  1470k      0  0:00:46  0:00:46 --:--:-- 2368k
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 13.35.20.87
  remote_port: 443
  size_download: 69899433 B
  speed_download: 1505934.000 B/s
  time_appconnect: 3.343496 s
  time_connect: 3.164926 s
  time_namelookup: 3.060655 s
  time_pretransfer: 3.343581 s
  time_redirect: 0.000000 s
  time_starttransfer: 5.766543 s
  time_total: 46.416495 s

  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  3182k      0  0:00:21  0:00:21 --:--:-- 4612k
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 13.35.20.87
  remote_port: 443
  size_download: 69899433 B
  speed_download: 3259170.000 B/s
  time_appconnect: 0.225982 s
  time_connect: 0.070428 s
  time_namelookup: 0.000483 s
  time_pretransfer: 0.226055 s
  time_redirect: 0.000000 s
  time_starttransfer: 0.306621 s
  time_total: 21.447966 s
#+END_EXAMPLE


2. Tested today at my office. China Telecom enterprise broadband. 50Mb/s.

berlin.guixsd.org:

#+BEGIN_EXAMPLE
  ➜  ~ measure_get https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  3091k      0  0:00:22  0:00:22 --:--:-- 3649k
  url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 141.80.181.40
  remote_port: 443
  size_download: 69899433 B
  speed_download: 3166021.000 B/s
  time_appconnect: 3.288213 s
  time_connect: 2.733554 s
  time_namelookup: 2.486754 s
  time_pretransfer: 3.288320 s
  time_redirect: 0.000000 s
  time_starttransfer: 3.780341 s
  time_total: 22.078489 s

  ➜  ~ measure_get https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 66.6M  100 66.6M    0     0  3499k      0  0:00:19  0:00:19 --:--:-- 4011k
  url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 141.80.181.40
  remote_port: 443
  size_download: 69899433 B
  speed_download: 3583667.000 B/s
  time_appconnect: 0.761166 s
  time_connect: 0.244415 s
  time_namelookup: 0.000981 s
  time_pretransfer: 0.761275 s
  time_redirect: 0.000000 s
  time_starttransfer: 1.247935 s
  time_total: 19.505515 s
#+END_EXAMPLE

berlin-mirror.marusich.info:

#+BEGIN_EXAMPLE
  ➜  ~ measure_get https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
   49 66.6M   49 32.8M    0     0  19012      0  1:01:16  0:30:13  0:31:03 29265
  url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 52.85.158.22
  remote_port: 443
  size_download: 34488133 B
  speed_download: 19012.000 B/s
  time_appconnect: 2.958899 s
  time_connect: 2.487483 s
  time_namelookup: 2.271520 s
  time_pretransfer: 2.959321 s
  time_redirect: 0.000000 s
  time_starttransfer: 5.447693 s
  time_total: 1813.938029 s
  curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
#+END_EXAMPLE


Although both 13.35.20.0/24 and 52.85.158.0/24 IP ranges are located at Seattle,
the result shows that the connection to 13.35.20.0/24 is significantly faster.
This is pretty normal in China.  It's definitely caused by the GFW.  Giant
internet service providers (e.g. AWS) are the primary targets of the GFW.

Re: CDN performance

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 5731 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Regarding the GNU sub-domain, as I replied to Meiyo, I’m in favor of it,
> all we need is someone to champion setting it up.

I could help with this.  Whom should I contact?

>> Regarding CDNs, I definitely think it's worth a try!  Even Debian is
>> using CloudFront (cloudfront.debian.net).  In fact, email correspondence
>> suggests that as of 2013, Amazon may even have been paying for it!
>>
>> https://lists.debian.org/debian-cloud/2013/05/msg00071.html
>
> (Note that debian.net is not Debian, and “there’s no cloud, only other
> people’s computer” as the FSFE puts it.)

I do try to avoid the term "cloud" whenever possible.  It's hard to
avoid when it's in the product name, though!  A wise man once said, "A
cloud in the mind is an obstacle to clear thinking."  ;-)

You may be right about debian.net.  I don't know who owns that domain.
It's confusing, since debian.org is definitely owned by the Debian
project, and the following page says they're using Amazon CloudFront:

https://deb.debian.org/

Maybe Debian still uses Amazon CloudFront, or maybe they don't any more.
In any case, I've found the following email thread, which documents a
thoughtful discussion regarding whether or not Debian should use a CDN.
They discussed many of the same concerns we're discussing here.

https://lists.debian.org/debian-project/2013/10/msg00029.html

A summary, in the middle of the long thread, is here:

https://lists.debian.org/debian-project/2013/10/msg00074.html

Later, part of the thread broke off and continued here:

https://lists.debian.org/debian-project/2014/02/msg00001.html

That's as far as I've read.

Judging by that email thread, one of the reasons why Debian considered
using a CDN was because they felt that the cost, in terms of people
power, of maintaining their own "proto-CDN" infrastructure had grown too
great.  I believe it!  I think it would be ill-advised for the Guix
project to expend effort and capital on building and maintaining its own
CDN.  I think it would be wiser to focus on developing a decentralized
substitute solution (GNUnet, IPFS, etc.).

That said, I still think that today Guix should provide a third-party
CDN option.  For many Guix users, a CDN would improve performance and
availability of substitutes.  Contracting with a third party to provide
the CDN service would require much less effort and capital than building
and maintaining a CDN from scratch.  This would also enable the project
to focus more on building a decentralized substitute solution.  And once
that decentralized solution is ready, it will be easy to just "turn off"
the CDN.

I also understand Hartmut's concerns.  The risks he points out are
valid.  Because of those risks, even if we make a third-party CDN option
available, some people will choose not to use it.  For that reason, we
should not require Guix users to use a third-party CDN, just as we do
not require them to use substitutes from our build farm.

However, not everyone shares the same threat model.  For example,
although some people choose not to trust substitutes from our build
farm, still others do.  The choice is based on one's own individual
situation.  Similarly, if we make a third-party CDN option available and
explain the risks of using it, Guix users will be able to make an
educated decision for themselves about whether or not to use it.

>> Here, it took 0.459667 - 0.254210 = 0.205457 seconds (about 205 ms) to
>> establish the TCP connection after the DNS lookup.  The average
>> throughput was 1924285 bytes per second (about 40 megabits per second,
>> where 1 megabit = 10^6 bits).  It seems my connection to berlin is
>> already pretty good!
>
> Indeed.  The bandwidth problem on berlin is when you’re the first to
> download a nar and it’s not been cached by nginx yet.  In that case, you
> get very low bandwidth (like 10 times less than when the item is cached
> by nginx.)  I’ve looked into it, went as far as strace’ing nginx, but
> couldn’t find the reason of this.
>
> Do you any idea?

I made a typo here.  The value "1924285" should have been "4945831",
which is what measure_get printed.  However, the intended result (40
Mbps) is still correct.

Actually, I thought 40 megabits per second was pretty great for a
single-threaded file transfer that originated in Europe (I think?) and
terminated in Seattle (via my residential Comcast downlink).  I
requested that particular file many times before that final test run, so
it was probably already cached by nginx.

However, I believe you when you say that it's slow the first time you
download the substitute from berlin.  What path does the data take from
its origin through berlin?  If berlin needs to download the initial file
from another server, perhaps the connection between berlin and that
server is the bottleneck?  Maybe we should discuss that in a different
email thread, though.

> I’ve tried this from home (in France, with FTTH):
>
> [...]
>
> speed_download: 20803402.000 B/s

Wow, that's 166 megabits per second!  I'm jealous.  :-)

> Wall-clock time is less than a tenth; woow.

I expect others will see similar performance improvements, as long as
they are close to the edge locations provided by Amazon CloudFront and
their local ISP downlink is fast enough to see a benefit.  Again, the
edge locations are here:

https://aws.amazon.com/cloudfront/features/

Here's a direct link to their current map:

https://d1.awsstatic.com/global-infrastructure/maps/CloudFront%20Network%20Map%2010.12.18.59e838df2f373247d2efaeb548076e084fd8993e.png

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: CDN performance

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 3388 bytes --]

Hi Chris,

nice to see this discussion, IMHO how GuixSD subsitutes are distributed
is a key issue in our ecosystem and is _all_ about privacy and metadata
*mass* collection

most "normal users" are not concerned about this so they are fine with
super-centralization since it's a convenience... not us :-)

personally I've come to GuixSD because I see this project as a key part
in liberating me from this class of problems

Chris Marusich <cmmarusich@gmail.com> writes:

[...]

> A summary, in the middle of the long thread, is here:
>
> https://lists.debian.org/debian-project/2013/10/msg00074.html

thank you for the reference, I've only read this summary

the key part of it IMHO is
  "Q: Do CDNs raise more security/privacy concerns than our mirrors?"

and the related subthread
 https://lists.debian.org/debian-project/2013/10/msg00033.html

the quick reply to the above question is: yes, CDNs raise more
secutiry/privacy concerns than "distributed mirrors"

obviuosly "distributed mirrors" _does_ rise some security/privacy
concerns but *centralization*... much more

[...]

> Judging by that email thread, one of the reasons why Debian considered
> using a CDN was because they felt that the cost, in terms of people
> power, of maintaining their own "proto-CDN" infrastructure had grown too
> great.

I'm still new to guixsd but understood enough to se we are much more
well equipped to maintain our distributed network of substitutes caching
servers... **transparently** configured :-)

[...]

> I also understand Hartmut's concerns.  The risks he points out are
> valid.  Because of those risks, even if we make a third-party CDN option
> available, some people will choose not to use it.

probably I'll be one of those, I'm considering to maintain a caching
substitute server in a "semi-trusted" colocated space and I'd be very
happy to share that server with the community

[...]

> However, not everyone shares the same threat model.  For example,
> although some people choose not to trust substitutes from our build
> farm, still others do.

for this very reason IMHO we should work towards a network of **very
trusted** build farms directly managed and controlled by the GuixSD
project sysadmins; if build farms will be able to quickly provide
substitutes, caching mirrors will be _much more_ effective than today

... and a network of "automated guix challenge" servers to spot
not-reproducible software in GuixSD

with a solid infrastructure of "scientifically" trustable build farms,
there are no reasons not to trust substitutes servers (this implies
working towards 100% reproducibility of GuixSD)

> The choice is based on one's own individual
> situation.  Similarly, if we make a third-party CDN option available and
> explain the risks of using it, Guix users will be able to make an
> educated decision for themselves about whether or not to use it.

that's an option... like a "last resort" in order to be able to use
guixSD :-)

we could also teach people how to setup their own caching servers and
possibly share them with the rest of the local community (possibly with
some coordination effort from the project sysadmins)

for Milan I've plans to setup such caching mirror in Jan 2019

[...]

happy hacking!
Gio

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: CDN performance

[Mark H Weaver]

Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:
> with a solid infrastructure of "scientifically" trustable build farms,
> there are no reasons not to trust substitutes servers (this implies
> working towards 100% reproducibility of GuixSD)

What does "scientifically trustable" mean?

      Mark

Re: CDN performance

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]

Hi Mark,

sorry for the late reply

Mark H Weaver <mhw@netris.org> writes:

> Giovanni Biscuolo <g@xelera.eu> writes:
>> with a solid infrastructure of "scientifically" trustable build farms,
>> there are no reasons not to trust substitutes servers (this implies
>> working towards 100% reproducibility of GuixSD)
>
> What does "scientifically trustable" mean?

I'm still not able to elaborate on that (working on it, a sort of
self-research-hack project) but I'm referencing to this message related
to reduced bootstrap tarballs:

  https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00347.html

and the related reply by Jeremiah (unfortunately cannot find it in
archives, Message-ID: <877eh81tm4.fsf@ITSx01.pdp10.guru>)

in particular Jeremiah replied this:

--8<---------------cut here---------------start------------->8---
> so, if I don't get it wrong, every skilled engineer will be able to
> build an "almost analogic" (zero bit of software preloaded) computing
> machine ad use stage0/mes [1] as the "metre" [2] to calibrate all other
> computing machines (thanks to reproducible builds)?

well, I haven't thought of it in those terms but yes I guess that is one
of the properties of the plan.
--8<---------------cut here---------------end--------------->8---

and

--8<---------------cut here---------------start------------->8---
> so, having the scientific proof that binary conforms to source, there
> will be noo need to trust (the untrastable)

Well, that is what someone else could do with it but not a direct goal
of the work.
--8<---------------cut here---------------end--------------->8---

maybe a more correct definition of the above "scientific proof" should be
"mathematical proof"

I lack the theoretical basis to be more precise now, sorry :-S

a marketing-like campaign sould be "no more trusting trust"

best regards
Giovanni

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Trustworthiness of build farms (was Re: CDN performance)

[Mark H Weaver]

Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Giovanni Biscuolo <g@xelera.eu> writes:
>>> with a solid infrastructure of "scientifically" trustable build farms,
>>> there are no reasons not to trust substitutes servers (this implies
>>> working towards 100% reproducibility of GuixSD)
>>
>> What does "scientifically trustable" mean?
>
> I'm still not able to elaborate on that (working on it, a sort of
> self-research-hack project) but I'm referencing to this message related
> to reduced bootstrap tarballs:
>
>   https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00347.html
>
> and the related reply by Jeremiah (unfortunately cannot find it in
> archives, Message-ID: <877eh81tm4.fsf@ITSx01.pdp10.guru>)
>
> in particular Jeremiah replied this:
>
>> so, if I don't get it wrong, every skilled engineer will be able to
>> build an "almost analogic" (zero bit of software preloaded) computing
>> machine ad use stage0/mes [1] as the "metre" [2] to calibrate all other
>> computing machines (thanks to reproducible builds)?
>
> well, I haven't thought of it in those terms but yes I guess that is one
> of the properties of the plan.
>
>
> and
>
>> so, having the scientific proof that binary conforms to source, there
>> will be noo need to trust (the untrastable)
>
> Well, that is what someone else could do with it but not a direct goal
> of the work.
>
> maybe a more correct definition of the above "scientific proof" should be
> "mathematical proof"

I agree that a mathematical proof is what we should be aiming for, and
the only kind of proof that I could trust in, in this scenario.

However, it's important to note several caveats:

* A mathematical proof showing that the binary conforms to the source
  requires a proof of correctness of the language implementation, which
  in turn requires formal semantics for both the source language and the
  underlying machine code.  As far as I know, the current
  bootstrappable.org effort does not include anything like this.
  Existing projects that provide this include CakeML and Jitawa.

* One must assume that the hardware behaves according to its formal
  specification.  The veracity of this assumption is not something we
  can currently verify, and even if we could, it would be invalidated if
  someone gained physical access to the machine and modified it.

* The hardware initialization program (e.g. coreboot) and the kernel
  used to perform the bootstrap must still be trusted, and unless I'm
  mistaken, the bootstrappable.org effort does not currently address
  these issues.

* The software running on the substitute servers could be compromised by
  stealing SSH keys from someone with root access.

* If the private signing key of the substitute server can be stolen,
  e.g. by gaining physical access to the machine for a short time, then
  a man-in-the-middle can deliver to users compromised binaries that
  appear to come from the substitute server itself.

* Not only the substitute server, but also all of its build slaves, must
  be trusted as well.

So, while I certainly agree that it will be a *major* improvement to
avoid the need to blindly trust precompiled bootstrap binaries, we
should be clear that the current efforts fall far short of a proof, and
there still remain several valid reasons not to place one's trust in
substitute servers.

Does that make sense?

    Regards,
      Mark

Re: CDN performance

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

Giovanni Biscuolo <g@xelera.eu> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Giovanni Biscuolo <g@xelera.eu> writes:
>>> with a solid infrastructure of "scientifically" trustable build farms,
>>> there are no reasons not to trust substitutes servers (this implies
>>> working towards 100% reproducibility of GuixSD)
>>
>> What does "scientifically trustable" mean?
>
> I'm still not able to elaborate on that (working on it, a sort of
> self-research-hack project) but I'm referencing to this message related
> to reduced bootstrap tarballs:
>
>   https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00347.html
>
> and the related reply by Jeremiah (unfortunately cannot find it in
> archives, Message-ID: <877eh81tm4.fsf@ITSx01.pdp10.guru>)

FWIW the GNU list search can take message IDs:

https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=877eh81tm4.fsf%40ITSx01.pdp10.guru&submit=Search&idxname=guix-devel

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: CDN performance

[Ricardo Wurmus]

Hi Giovanni,

> for this very reason IMHO we should work towards a network of **very
> trusted** build farms directly managed and controlled by the GuixSD
> project sysadmins; if build farms will be able to quickly provide
> substitutes, caching mirrors will be _much more_ effective than today
>
> ... and a network of "automated guix challenge" servers to spot
> not-reproducible software in GuixSD
>
> with a solid infrastructure of "scientifically" trustable build farms,
> there are no reasons not to trust substitutes servers (this implies
> working towards 100% reproducibility of GuixSD)

This sets the bar very high.  I administer berlin.guix.info /
ci.guix.info (same server) and most of the associated build nodes, but I
don’t think people should trust these computers more than their own
computers or those managed by people they personally know.

The build servers do the same work that a user would do who builds
software locally without substitutes; the builders are supposed to
behave just like any other user.  (And we can challenge their authority
with “guix challenge”.)  I want us to keep in mind that build farms
don’t necessarily deserve any more trust than other computers you don’t
control.  Substitute servers are a convenience.

To improve distribution of binaries we are committed to working on
different strategies at the same time:

- improve our processes so that non-critical package changes only hit
  the master branch when we have already built the package and made it
  available for distribution.

- improve availability of the single server berlin.guix.info by hooking
  it up to a CDN via the name ci.guix.info (from which users can easily
  opt out by changing their default substitute server to
  berlin.guix.info).

- improve redundancy by setting up an off-site fail-over for the head of
  the build farm at berlin.guix.info (such as bayfront).

- distribute build artefacts over IPFS without requiring a central
  authority

All of these things can be done in parallel by different people.  This
increases the project’s resilience and the options that users can choose
from.

--
Ricardo

guix.gnu.org sub-domain

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Regarding the GNU sub-domain, as I replied to Meiyo, I’m in favor of it,
>> all we need is someone to champion setting it up.
>
> I could help with this.  Whom should I contact?

We discussed this over the last few days in Paris and Julien (roptat on
IRC) volunteered to come up with a Knot service setup for bayfront.scm.
When that’s ready, we can contact the FSF sysadmins so they delegate to
bayfront.

I’m sure Julien wouldn’t mind getting some help or insight, so please do
get in touch!

Ludo’.

Re: guix.gnu.org sub-domain

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 271 bytes --]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> I’m sure Julien wouldn’t mind getting some help or insight, so please do
> get in touch!

OK, I'll speak privately with Julien about the DNS setup to avoid adding
noise to this email thread.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: guix.gnu.org sub-domain

[Amin Bandali]

On 2018-12-15  3:20 PM, Chris Marusich wrote:
> Hi Ludo,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I’m sure Julien wouldn’t mind getting some help or insight, so please do
>> get in touch!
>
> OK, I'll speak privately with Julien about the DNS setup to avoid adding
> noise to this email thread.
>
> -- 
> Chris
>

Hi Chris, Julien,

Any update on this?  I too had asked Ludo about DNS for guix.gnu.org a
couple of weeks ago, and he’d pointed me to the bayfront.scm [1] config
file, but I was swamped with work and only recently have found a bit of
free time on my hands.  While looking for information about Knot, I also
stumbled upon Julien’s configuration [2].  I was wondering if y’all were
able to spend any time on this, and if so, if you ended up committing
your progress anywhere?

Best,
amin

Footnotes:
[1]  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm

[2]  https://lepiller.eu/ma-configuration.html

Re: CDN performance

[Ludovic Courtès]

Hello!

Chris Marusich <cmmarusich@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:


[...]

> Judging by that email thread, one of the reasons why Debian considered
> using a CDN was because they felt that the cost, in terms of people
> power, of maintaining their own "proto-CDN" infrastructure had grown too
> great.  I believe it!  I think it would be ill-advised for the Guix
> project to expend effort and capital on building and maintaining its own
> CDN.  I think it would be wiser to focus on developing a decentralized
> substitute solution (GNUnet, IPFS, etc.).
>
> That said, I still think that today Guix should provide a third-party
> CDN option.  For many Guix users, a CDN would improve performance and
> availability of substitutes.  Contracting with a third party to provide
> the CDN service would require much less effort and capital than building
> and maintaining a CDN from scratch.  This would also enable the project
> to focus more on building a decentralized substitute solution.  And once
> that decentralized solution is ready, it will be easy to just "turn off"
> the CDN.
>
> I also understand Hartmut's concerns.  The risks he points out are
> valid.  Because of those risks, even if we make a third-party CDN option
> available, some people will choose not to use it.  For that reason, we
> should not require Guix users to use a third-party CDN, just as we do
> not require them to use substitutes from our build farm.
>
> However, not everyone shares the same threat model.  For example,
> although some people choose not to trust substitutes from our build
> farm, still others do.  The choice is based on one's own individual
> situation.  Similarly, if we make a third-party CDN option available and
> explain the risks of using it, Guix users will be able to make an
> educated decision for themselves about whether or not to use it.

That summarizes the situation very well.

In Paris, there was consensus on two things: that a CDN would be helpful
for performance and availability (though availability could suddenly be
worse if we run… out of money), and that it raises privacy concerns.

Based on this, some suggested having a way to opt out of the CDN
distribution: for example, we might make ci.guix.info point to the CDN
and also advertise berlin.guixsd.org as a way to bypass it.

Of course in parallel we want to do our best to develop decentralized
solutions.

> However, I believe you when you say that it's slow the first time you
> download the substitute from berlin.  What path does the data take from
> its origin through berlin?  If berlin needs to download the initial file
> from another server, perhaps the connection between berlin and that
> server is the bottleneck?  Maybe we should discuss that in a different
> email thread, though.

On the first hit, nginx fetches the file from ‘guix publish’, which is
also on berlin.  ‘guix publish’ gets the file from local storage and
passes it with sendfile(2), so it’s as fast as can be.

Thanks for the info!

Ludo’.

Re: Using a CDN or some other mirror?

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 3405 bytes --]

Hi Chris,

thank you for your CDN testing environment!

Chris Marusich <cmmarusich@gmail.com> writes:

[...]

> For experimentation, I've set up a CloudFront distribution at
> berlin-mirror.marusich.info that uses berlin.guixsd.org as its origin
> server.  Let's repeat these steps to measure the performance of the
> distribution from my machine's perspective (before I did this, I made
> sure the GET would result in a cache hit by downloading the substitute
> once before and verifying that the same remote IP address was used):

[...]

> It would be interesting to see what the performance is for others.

[...]

measures from my office network: Italy, 20Km north Milan, FTTC
(90Mbit/sec measured bandwidth)

measure from Berlin:

--8<---------------cut here---------------start------------->8---
url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 141.80.181.40
  remote_port: 443
  size_download: 69899433 B
  speed_download: 9051388,000 B/s
  time_appconnect: 0,229271 s
  time_connect: 0,110443 s
  time_namelookup: 0,061754 s
  time_pretransfer: 0,229328 s
  time_redirect: 0,000000 s
  time_starttransfer: 0,326907 s
  time_total: 7,722509 s
--8<---------------cut here---------------end--------------->8---

latency measured with mtr:

--8<---------------cut here---------------start------------->8---
HOST: roquette                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.38.2.1                  0.0%    10    0.3   0.4   0.3   0.4   0.0

[...]

 18.|-- 141.80.181.40              0.0%    10  112.5  77.1  55.6 201.7  47.1
--8<---------------cut here---------------end--------------->8---


from your mirror (third download):

--8<---------------cut here---------------start------------->8---
url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  http_code: 200
  num_connects: 1
  num_redirects: 0
  remote_ip: 54.230.102.61
  remote_port: 443
  size_download: 69899433 B
  speed_download: 9702091,000 B/s
  time_appconnect: 0,172660 s
  time_connect: 0,037833 s
  time_namelookup: 0,003772 s
  time_pretransfer: 0,173263 s
  time_redirect: 0,000000 s
  time_starttransfer: 0,212716 s
  time_total: 7,204574 s
--8<---------------cut here---------------end--------------->8---

latency measured with mtr:

--8<---------------cut here---------------start------------->8---
HOST: roquette                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.38.2.1                   0.0%    10    0.4   0.4   0.4   0.4   0.0

[...]

 11.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
 12.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
 13.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
 14.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
 15.|-- 52.93.58.190                0.0%    10   36.1  34.6  32.9  37.1   1.2
 16.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
--8<---------------cut here---------------end--------------->8---

100% loss?

from here it seems Berlin is as performant as CloudFront

HTH!
Gio

-- 
Giovanni Biscuolo

Xelera IT Infrastructures



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Using a CDN or some other mirror?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3982 bytes --]

Hi Giovanni,

Thank you for sharing some data with us!

Giovanni Biscuolo <g@xelera.eu> writes:

> measures from my office network: Italy, 20Km north Milan, FTTC
> (90Mbit/sec measured bandwidth)
>
> measure from Berlin:
>
> url_effective: https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 141.80.181.40
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 9051388,000 B/s

That's about 72 megabits per second.

>   time_appconnect: 0,229271 s
>   time_connect: 0,110443 s
>   time_namelookup: 0,061754 s

Latency was about 49 milliseconds (after the name lookup).

>   [...]
> latency measured with mtr:
>
> HOST: roquette                  Loss%   Snt   Last   Avg  Best  Wrst StDev
>   1.|-- 10.38.2.1                  0.0%    10    0.3   0.4   0.3   0.4   0.0
>
> [...]
>
>  18.|-- 141.80.181.40              0.0%    10  112.5  77.1  55.6 201.7  47.1
>
>
>
> from your mirror (third download):
>
> url_effective: https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
>   http_code: 200
>   num_connects: 1
>   num_redirects: 0
>   remote_ip: 54.230.102.61
>   remote_port: 443
>   size_download: 69899433 B
>   speed_download: 9702091,000 B/s

That's about 78 megabits per second, which is 8% more than 72.

>   time_appconnect: 0,172660 s
>   time_connect: 0,037833 s
>   time_namelookup: 0,003772 s

Latency was 34 milliseconds, which is 31% less than 49.

>   [...]
>
> latency measured with mtr:
>
> HOST: roquette                   Loss%   Snt   Last   Avg  Best  Wrst StDev
>   1.|-- 10.38.2.1                   0.0%    10    0.4   0.4   0.4   0.4   0.0
>
> [...]
>
>  11.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
>  12.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
>  13.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
>  14.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
>  15.|-- 52.93.58.190                0.0%    10   36.1  34.6  32.9  37.1   1.2
>  16.|-- ???                        100.0    10    0.0   0.0   0.0   0.0   0.0
>
> 100% loss?

Yes, mtr's output here is a bit surprising.

On my end, also, mtr reported similar "loss" for intermediate hops, but
in my case the final hop did not report any loss.  Deprioritization of
ICMP traffic is common in many networks, so tools like mtr and
traceroute will sometimes report surprisingly high latency or packet
loss even when the network is just fine.

The mechanism used by tools like mtr and traceroute is to repeatedly
send "probes" with monotonically increasing TTL values.  The measurement
(even when using TCP probes) relies on (1) intermediate hops correctly
returning an ICMP "time exceeded" message when the packet lands on that
hop and the TTL expires, and (2) the ICMP "time exceeded" message
getting successfully delivered back to the mtr process.

In any case, the "100% loss" metric is clearly inaccurate, since you
successfully downloaded the file at an impressive speed.  If a hop were
truly dropping 100% of the traffic, the download would have failed.  In
addition, the latency that mtr does report seems comparable to the
latency calculated from the measure_get output (which is not influenced
by the vagaries of ICMP deprioritization).

> from here it seems Berlin is as performant as CloudFront

Yes, it seems you are already well connected to the build farm!  But
still, when you used CloudFront, your throughput went up by 7%, and your
latency went down by 31%.  Even more importantly, when you downloaded
the file from CloudFront, it placed zero additional load on the build
farm because it was served from CloudFront's cache.

Again, thank you for sharing!  This is useful information.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Amin Bandali]

Hi Ludo’,

> For the domain name I initially wanted “ci.guix.gnu.org” but we failed
> to set that up.  Oh well, I think that’s OK.

What was the failure about, if you don’t mind me asking?  I’m
curious if there might be a way to salvage it.

-amin

Re: [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Ludovic Courtès]

Hi Amin,

Amin Bandali <bandali@gnu.org> skribis:

>> For the domain name I initially wanted “ci.guix.gnu.org” but we failed
>> to set that up.  Oh well, I think that’s OK.
>
> What was the failure about, if you don’t mind me asking?  I’m
> curious if there might be a way to salvage it.

I initially proposed to delegate subdomain handling for guix.gnu.org on
the machine that guix.gnu.org currently points to (aka. “bayfront”).  To
do that we’d need to coordinate with the FSF sysadmins and someone
DNS-savvy (i.e., not me :-)) should amend the configuration at
<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm>
to have a Knot service or similar.

If you’d like to help, that’d be great!

Ludo’.

Re: [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Amin Bandali]

Hi Ludo’,

On 2018-12-04  3:11 PM, Ludovic Courtès wrote:
> Hi Amin,
>
> Amin Bandali <bandali@gnu.org> skribis:
>
>>> For the domain name I initially wanted “ci.guix.gnu.org” but we failed
>>> to set that up.  Oh well, I think that’s OK.
>>
>> What was the failure about, if you don’t mind me asking?  I’m
>> curious if there might be a way to salvage it.
>
> I initially proposed to delegate subdomain handling for guix.gnu.org on
> the machine that guix.gnu.org currently points to (aka. “bayfront”).  To
> do that we’d need to coordinate with the FSF sysadmins and someone
> DNS-savvy (i.e., not me :-)) should amend the configuration at
> <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm>
> to have a Knot service or similar.
>
> If you’d like to help, that’d be great!
>
> Ludo’.

Thanks for the reply and for the link.  I’m no DNS expert either :) but
I’ll look into the DNS services section of the Guix manual and will see
if I can amend bayfront.scm.  If I get anywhere, I’ll post here and we
could then reach out to the FSF sysadmins and ask them to delegate that
subdomain to bayfront.

Best,
amin

Re: [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> These patches (actually the last one) switch Guix to default to
> <https://ci.guix.info> for substitutes, in preparation for the
> upcoming 0.16.0 release (hopefully this week!).

Who owns the guix.info domain?  Also, who owns the guixsd.org domain?

     Thanks,
       Mark

Re: [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Ricardo Wurmus]

Hi Mark,

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> These patches (actually the last one) switch Guix to default to
>> <https://ci.guix.info> for substitutes, in preparation for the
>> upcoming 0.16.0 release (hopefully this week!).
>
> Who owns the guix.info domain?

I registered it and offered it to Guix Europe (though we aren’t yet
sharing administration of the domain).

--
Ricardo

Re: [bug#33600] [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Ludovic Courtès]

Hello,

Ricardo Wurmus <rekado@elephly.net> skribis:

>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> These patches (actually the last one) switch Guix to default to
>>> <https://ci.guix.info> for substitutes, in preparation for the
>>> upcoming 0.16.0 release (hopefully this week!).
>>
>> Who owns the guix.info domain?
>
> I registered it and offered it to Guix Europe (though we aren’t yet
> sharing administration of the domain).

Could you make sure several people have access to the administration
interface?

As for guixsd.org, it was initially owned by Andreas Enge who
transferred it to Guix Europe and also provided credentials to me and I
think a couple of other people, though I forgot who.  Andreas?

We should write all this information down in guix-maintenance.git like
we did for build machines, so people know who to contact in case
something goes wrong or what to do when someone leaves the project.

Thanks,
Ludo’.

Re: [bug#33600] [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Andreas Enge]

On Tue, Dec 04, 2018 at 10:03:01AM +0100, Ludovic Courtès wrote:
> As for guixsd.org, it was initially owned by Andreas Enge who
> transferred it to Guix Europe and also provided credentials to me and I
> think a couple of other people, though I forgot who.  Andreas?

Apart from you, to Ricardo Wurmus, Christopher Baines and Manolis Ragkousis.
I am of course happy to share the credentials with more people from the
Guix sysadmin group.

Andreas

Re: [bug#33600] [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Andreas Enge]

On Mon, Dec 03, 2018 at 06:44:43PM -0500, Mark H Weaver wrote:
> Who owns the guix.info domain?  Also, who owns the guixsd.org domain?

guixsd.org is owned by the Guix Europe association. guix.info, as far as
I know, by Ricardo. Plans are to drop guixsd.org in the long run, and I
would suggest to transfer guix.info to Guix Europe at the next renewal.

Andreas

Re: [bug#33600] [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Ludovic Courtès]

Hello,

Andreas Enge <andreas@enge.fr> skribis:

> On Mon, Dec 03, 2018 at 06:44:43PM -0500, Mark H Weaver wrote:
>> Who owns the guix.info domain?  Also, who owns the guixsd.org domain?
>
> guixsd.org is owned by the Guix Europe association. guix.info, as far as
> I know, by Ricardo. Plans are to drop guixsd.org in the long run, and I
> would suggest to transfer guix.info to Guix Europe at the next renewal.

Awesome.  I’ve added a file to record that info in
guix-maintenance.git.  Please feel free to complete it; for instance
it’d be good to know the expiration date of each domain.

Thanks!

Ludo’.

Re: [bug#33600] [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Andreas Enge]

On Tue, Dec 04, 2018 at 11:28:51AM +0100, Ludovic Courtès wrote:
> Awesome.  I’ve added a file to record that info in
> guix-maintenance.git.  Please feel free to complete it; for instance
> it’d be good to know the expiration date of each domain.

Done for guixsd.org. It expires on January 27. I would be okay to let it
run out if guix.info is sufficiently marketed in the meantime.

Andreas

Re: [bug#33600] [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Tue, Dec 04, 2018 at 11:28:51AM +0100, Ludovic Courtès wrote:
>> Awesome.  I’ve added a file to record that info in
>> guix-maintenance.git.  Please feel free to complete it; for instance
>> it’d be good to know the expiration date of each domain.
>
> Done for guixsd.org. It expires on January 27. I would be okay to let it
> run out if guix.info is sufficiently marketed in the meantime.

Thanks.  I think it’s safer to keep it for a while because there may be
machine referring to berlin.guixsd.org here and there.

Ludo’.

Re: [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Meiyo Peng]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> For the domain name I initially wanted “ci.guix.gnu.org” but we failed
> to set that up.  Oh well, I think that’s OK.

I don't think ci.guix.info is a suitable name for our default substitute
server. You may think I paranoid, but if I accidentally find out
ci.guix.info is my default substitute server, I will be cautious and
verify it's integrity.

Let's use a subdomain of gnu.org. If you encounter any obstacles while
setting up the system, we'd like to help.

--
Meiyo Peng

Re: [PATCH 0/3] Defaulting to ci.guix.info (aka. berlin.guixsd.org)

[Ludovic Courtès]

Hi,

Meiyo Peng <meiyo.peng@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> For the domain name I initially wanted “ci.guix.gnu.org” but we failed
>> to set that up.  Oh well, I think that’s OK.
>
> I don't think ci.guix.info is a suitable name for our default substitute
> server. You may think I paranoid, but if I accidentally find out
> ci.guix.info is my default substitute server, I will be cautious and
> verify it's integrity.

I understand the feeling, though security-wise, what matters is the key
that you authorize with “guix archive --authorize”.

> Let's use a subdomain of gnu.org. If you encounter any obstacles while
> setting up the system, we'd like to help.

I’ve just resent a summary of what needs to be done.

Thanks for helping out!

Ludo’.