Make guix-publish's URL identical to cache file name

Make guix-publish's URL identical to cache file name

[Peng Mei Yu]

Hi,

This proposal aims to solve an old problem.  Make it easier to setup a
mirror server for the official substitute server and prevent future
complaints from China residents about network speed.

Due to the national firewall(i.e. The GFW) deployed on every backbone
networks within China and it's aggressive rules on traffic passing
through the wall, the internet connection from inner China to outside
world is extremely unreliable and slow.  This makes it a pain for
newcomers to try and play with Guix, since guix daemon loves to spawn
thousands of HTTP requests to the substitute server and download
thousands of megabytes of packages from the substitute server.  Usually
it takes a whole day to install Guix system on a new computer.  I am
serious.  I have been a Guix user for several years and actually I am
not very much bothered by the problem.  But since V1.0 release of Guix,
Guix is frequently mentioned in tech news.  When tech savvy boys hear
the cool Guix and try to install it on their computer and still cannot
complete the installation process after half a day, they will always be
pissed off by the slow network speed and complain on internet.  I
mentioned Guix several times on some websites, this makes me the first
Chinese Guix user they can find with Google and ask for help.

I am sure maintainers here know these kind of complaints.  Do you feel
it is strange that no Chinese user make new complaints in year 2020
while the Guix project is quickly progressing and becoming more popular?
That is because I decided to setup a mirror server for Chinese Guix
users after receiving several complaints online and realizing that
lobbying academic FLOSS mirror maintainers to support Guix will be
stagnant for at least several years due to these maintainers' laziness
and cowardliness and some ridiculous strict governmental regulations.
These people only want to add mirror for a project if it is as simple as
pulling static files with a cron job (usually with rsync) and serving
static files through a simple HTTP server.  HTTP reverse proxy is not an
option.  So my mirror.guix.org.cn project was started.  It's an HTTP
cache mirror of ci.guix.gnu.org, plus a git mirror of
https://git.savannah.gnu.org/cgit/guix.git.  Yes, the connection to
Savannah is also extremely slow and it makes `guix pull` unusable.

This mirror server started as an experiment and it has been working
well.  If random new user come to me and say `guix pull` is so slow or
`guix install` is so slow, I simply tell them to use mirror.guix.org.cn.
The number of active Chinese Guix users I know has increased from two to
about ten after someone's broadcast in a news group.  It is basically:
"Look.  There is thing called Guix.  Someone has setup a mirror server
for it in China."  The traffic on the server is increasing.  Network
connections are stable.  Everything is fine in this year.  However one
thing worries me.  The bandwidth of mirror.guix.org.cn is only 5Mbps
(still far more better than ci.guix.gnu.org's 30KB/s and constant
connection reset).  This is the highest bandwidth I can afford because
internet bandwidth in China is damn too expensive.  Buying higher
bandwidth is not a financially possible approach for me.  This is not a
problem in the short term but definitely be a problem in the long term.
Persuading academic FLOSS mirror maintainers to support Guix is still
the best solution for Chinese users.  Academic organizations usually
have 100Mbps bandwidth and tens of terabytes of disk.

Now, finally, we are onto the main point.  I look into guix publish's
cache directory and think that nar and narinfo files can be directly
served through a static HTTP server if we make those files' URL
identical to their on-disk file name.  The current directory structure
is like this:

--8<---------------cut here---------------start------------->8---
/var/cache/guix/publish
├── gzip
│   ├── 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.nar
│   ├── 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo
│   ├── fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31.nar
│   └── fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31.narinfo
├── hashes
│   ├── 87kif0bpf0anwbsaw0jvg8fyciw4sz67
│   └── fa6wj5bxkj5ll1d7292a70knmyl7a0cr
├── last-expiry-cleanup
└── lzip
    ├── 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.nar
    ├── 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo
    ├── fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31.nar
    └── fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31.narinfo
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
> md5sum /var/cache/guix/publish/*/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo
29cdbf041b9a304bf58f2e75ec23f18f  /var/cache/guix/publish/gzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo
29cdbf041b9a304bf58f2e75ec23f18f  /var/cache/guix/publish/lzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo
--8<---------------cut here---------------end--------------->8---

When a client tries to download
/gnu/store/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16, it sends a
request to http://example.com/87kif0bpf0anwbsaw0jvg8fyciw4sz67.narinfo
and gets the content of
/var/cache/guix/publish/gzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo:

--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16
URL: nar/gzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16
Compression: gzip
FileSize: 2284657
URL: nar/lzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16
Compression: lzip
FileSize: 1256260
NarHash: sha256:1ap2s3xz3bbp5n78v826gxagy7pic1wpgzz3ka72jdyk6qpmw3qr
NarSize: 6597040
References: ...
System: x86_64-linux
Deriver: cccyyn4xq59aimybmhlrfl2bi8kslhlm-bash-5.0.16.drv
Signature: ...
--8<---------------cut here---------------end--------------->8---

Client then sends a request to the URL as written in the URL field:
http://example.com/nar/lzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.
This URL returns the file
/var/cache/guix/publish/lzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.nar.

I propose we make the URL field in narinfo the same as nar file name on
disk.  We can change the directory structure to:

--8<---------------cut here---------------start------------->8---
/var/cache/guix/publish/nar
├── 87kif0bpf0anwbsaw0jvg8fyciw4sz67.narinfo
├── 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.nar.gz
└── 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.nar.lz
--8<---------------cut here---------------end--------------->8---

And change the URL field in narinfo to

--8<---------------cut here---------------start------------->8---
URL: 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.nar.lz
--8<---------------cut here---------------end--------------->8---

Then a mirror site can simply pull the directory
/var/cache/guix/publish/nar from the Berlin server and serve this
directory through a static HTTP server.  There will be cache misses.
But guix-daemon will safely fallback to the next server in
substitute-urls.

What's your opinion?

I have to decide next year's server specs and budget for
mirror.guix.org.cn before the Chinese shopping festival ends on November
11.  If the proposal above is doable, I will keep mirror.guix.org.cn
running for half a year and help academic mirror sites add support for
Guix in the meantime.  Otherwise I prefer to buy a prepaid three years
VPS with a 90% discount during the shopping festival.  The discount is
huge.  I don't want to miss it.


--
Peng Mei Yu

Re: Make guix-publish's URL identical to cache file name

[Jonathan Brielmaier]

Hi Peng Mei Yu,

first of all a big thank you for making the Guix experience in China
much nicer!

I'm running myself a smaller cuirass+publish server and I have
configured the caching on the nginx side and not with `guix publish`.

So my cache resides in:
/var/cache/nginx/nar/

For me it has a dramatic speed up on my German server reaching from
Germany. So 2-3MB/s on a cold cache hit versus a hot cache with 15MB/s
and even more.

Maybe you can give that a try. I'm not sure if it does help as you said
you are a bit limited by the bandwidth...

On 04.11.20 09:46, Peng Mei Yu wrote:
> I have to decide next year's server specs and budget for
> mirror.guix.org.cn before the Chinese shopping festival ends on November
> 11.  If the proposal above is doable, I will keep mirror.guix.org.cn
> running for half a year and help academic mirror sites add support for
> Guix in the meantime.  Otherwise I prefer to buy a prepaid three years
> VPS with a 90% discount during the shopping festival.  The discount is
> huge.  I don't want to miss it.

I think we should discuss if Guix can support your effort money wise.
AFAIK we have decent fundings available.

~Jonathan

Re: Make guix-publish's URL identical to cache file name

[Peng Mei Yu]

Hi Jonathan,

Jonathan Brielmaier writes:

> I'm running myself a smaller cuirass+publish server and I have
> configured the caching on the nginx side and not with `guix publish`.
>
> So my cache resides in:
> /var/cache/nginx/nar/
>
> For me it has a dramatic speed up on my German server reaching from
> Germany. So 2-3MB/s on a cold cache hit versus a hot cache with 15MB/s
> and even more.
>
> Maybe you can give that a try. I'm not sure if it does help as you said
> you are a bit limited by the bandwidth...

I am perfectly fine with any method to setup a mirror site.  But the
academic mirror sites I am talking about only want a simple approach.
They want to pull files with rsync, rclone, or FTP and serve files with
static HTTP server.  If we make Guix's mirror as simple as that, I am
confident I can persuade more mirror sites to support Guix.


> On 04.11.20 09:46, Peng Mei Yu wrote:
>> I have to decide next year's server specs and budget for
>> mirror.guix.org.cn before the Chinese shopping festival ends on November
>> 11.  If the proposal above is doable, I will keep mirror.guix.org.cn
>> running for half a year and help academic mirror sites add support for
>> Guix in the meantime.  Otherwise I prefer to buy a prepaid three years
>> VPS with a 90% discount during the shopping festival.  The discount is
>> huge.  I don't want to miss it.
>
> I think we should discuss if Guix can support your effort money wise.
> AFAIK we have decent fundings available.

That would be great.  Although I still think letting academic mirror
sites do the job and saving the money would be better.  As I said, the
ISP cost in China is expensive and academic mirror sites have plenty of
bandwidth and storage resources available.


--
Peng Mei Yu

Re: Make guix-publish's URL identical to cache file name

[Ludovic Courtès]

Hi,

Jonathan Brielmaier <jonathan.brielmaier@web.de> skribis:

> I'm running myself a smaller cuirass+publish server and I have
> configured the caching on the nginx side and not with `guix publish`.

This is suboptimal because then your /nar responses lack the
‘Content-Length’ header (‘guix publish’ cannot provide that header when
running without ‘--cache’), which in turn means no progress bar, no
estimate of the amount of data downloaded, etc.

> For me it has a dramatic speed up on my German server reaching from
> Germany. So 2-3MB/s on a cold cache hit versus a hot cache with 15MB/s
> and even more.

‘guix publish --cache’ should bring you the same benefits.  I’d still
recommend having nginx in front of it (for HTTPS, etc.), but caching can
be disabled in nginx in that case.  That’s what we do on
ci.guix.gnu.org.

Ludo’.

Re: Make guix-publish's URL identical to cache file name

[Ludovic Courtès]

Hello!

Peng Mei Yu <pengmeiyu@riseup.net> skribis:

> This proposal aims to solve an old problem.  Make it easier to setup a
> mirror server for the official substitute server and prevent future
> complaints from China residents about network speed.

Just so you know: we’re aware of this problem and agree that it needs
solving.  Ricardo (Cc’d) has helped in the past set up rsync on
ci.guix.gnu.org so that someone could rsync /var/cache/guix/publish to a
server in China.  I think we have most of the pieces more or less in
place.

> Then a mirror site can simply pull the directory
> /var/cache/guix/publish/nar from the Berlin server and serve this
> directory through a static HTTP server.  There will be cache misses.
> But guix-daemon will safely fallback to the next server in
> substitute-urls.

The simplest solution for now (I think that’s what Ricardo & co. had in
mind) would be for you to retrieve /var/cache/guix/publish on your
server, as is, and then run ‘guix publish’ on your sever: it will know
where to find files.  As I wrote to Jonathan, you can/should also run
nginx on top of that as a proxy to your local ‘guix publish’.

Ricardo, can you remind us what the next steps would be?

Does that make sense?

Thanks,
Ludo’.

Re: Make guix-publish's URL identical to cache file name

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> The simplest solution for now (I think that’s what Ricardo & co. had in
> mind) would be for you to retrieve /var/cache/guix/publish on your
> server, as is, and then run ‘guix publish’ on your sever: it will know
> where to find files.  As I wrote to Jonathan, you can/should also run
> nginx on top of that as a proxy to your local ‘guix publish’.
>
> Ricardo, can you remind us what the next steps would be?

We need to make sure that *all* the files produced by “guix publish”
have correct permissions; IIRC some of the files are not readable at all
by users other than the owner of the files.

Once that’s done we just need to start the rsync daemon again,
preferably as a shepherd service.

-- 
Ricardo

Re: Make guix-publish's URL identical to cache file name

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> The simplest solution for now (I think that’s what Ricardo & co. had in
>> mind) would be for you to retrieve /var/cache/guix/publish on your
>> server, as is, and then run ‘guix publish’ on your sever: it will know
>> where to find files.  As I wrote to Jonathan, you can/should also run
>> nginx on top of that as a proxy to your local ‘guix publish’.
>>
>> Ricardo, can you remind us what the next steps would be?
>
> We need to make sure that *all* the files produced by “guix publish”
> have correct permissions; IIRC some of the files are not readable at all
> by users other than the owner of the files.

Oops, I had forgotten, my bad.  I’ll push the attached patch later
today.

Next we’ll need to update the ‘guix’ package, restart ‘guix publish’ on
berlin, and chmod a+r -R /var/cache/guix/publish.

> Once that’s done we just need to start the rsync daemon again,
> preferably as a shepherd service.

Yes.

Sounds like we have a plan!

Peng Mei Yu: make sure to ping us in the coming weeks if you don’t hear
from us by then!

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2010 bytes --]

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index e8faf379e2..e3c8711f5b 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -583,7 +583,10 @@ requested using POOL."
      ;; guarantee the TTL (see <https://bugs.gnu.org/28664>.)
      (with-atomic-file-output nar
        (lambda (port)
-         (write-file item port))))))
+         (write-file item port)
+         ;; Make the file world-readable, contrary to what
+         ;; 'with-atomic-file-output' does.
+         (chmod port (logand #o644 (lognot (umask)))))))))
 
 (define* (bake-narinfo+nar cache item
                            #:key ttl (compressions (list %no-compression))
@@ -615,7 +618,12 @@ requested using POOL."
                                           #:nar-path nar-path
                                           #:compressions compressions
                                           #:file-sizes sizes)
-                          port)))))
+                          port)))
+
+             ;; Make the cached narinfo world-readable, contrary to what
+             ;; 'with-atomic-file-output' does, so that other users can rsync
+             ;; the whole cache.
+             (chmod port (logand #o644 (lognot (umask))))))
 
          ;; Make narinfo files for OTHERS hard links to NARINFO such that the
          ;; atime-based cache eviction considers either all the nars or none
diff --git a/tests/publish.scm b/tests/publish.scm
index e46e6256b7..cafd0f13a2 100644
--- a/tests/publish.scm
+++ b/tests/publish.scm
@@ -434,6 +434,11 @@ References: ~%"
                  (< ttl 3600)))
 
               (wait-for-file cached)
+
+              ;; Both the narinfo and nar should be world-readable.
+              (= #o644 (stat:perms (lstat cached)))
+              (= #o644 (stat:perms (lstat nar)))
+
               (let* ((body         (http-get-port url))
                      (compressed   (http-get nar-url))
                      (uncompressed (http-get (string-append base "nar/"

Re: Make guix-publish's URL identical to cache file name

[Peng Mei Yu]

Hello,

Ludovic Courtès writes:

>> Then a mirror site can simply pull the directory
>> /var/cache/guix/publish/nar from the Berlin server and serve this
>> directory through a static HTTP server.  There will be cache misses.
>> But guix-daemon will safely fallback to the next server in
>> substitute-urls.
>
> The simplest solution for now (I think that’s what Ricardo & co. had in
> mind) would be for you to retrieve /var/cache/guix/publish on your
> server, as is, and then run ‘guix publish’ on your sever: it will know
> where to find files.  As I wrote to Jonathan, you can/should also run
> nginx on top of that as a proxy to your local ‘guix publish’.

I am afraid having to install Guix and run 'guix publish' will be a
major obstacle for most mirror sites, especially when their maintainers
have no previous experience with Guix and have no interest in learning a
new software and do not want to pollute their server with an unknown
software and do not want to increase maintenance burden. This is the
simplest solution on our side, but a complex solution on mirror site
maintainers' side.

I guess a complex Nginx URL rewrite rule might also convert the URL into
real filename in the cache.  Can anyone familiar with Nginx explain?


--
Peng Mei Yu

Re: Make guix-publish's URL identical to cache file name

[Ludovic Courtès]

Hi,

Peng Mei Yu <pengmeiyu@riseup.net> skribis:

> Ludovic Courtès writes:
>
>>> Then a mirror site can simply pull the directory
>>> /var/cache/guix/publish/nar from the Berlin server and serve this
>>> directory through a static HTTP server.  There will be cache misses.
>>> But guix-daemon will safely fallback to the next server in
>>> substitute-urls.
>>
>> The simplest solution for now (I think that’s what Ricardo & co. had in
>> mind) would be for you to retrieve /var/cache/guix/publish on your
>> server, as is, and then run ‘guix publish’ on your sever: it will know
>> where to find files.  As I wrote to Jonathan, you can/should also run
>> nginx on top of that as a proxy to your local ‘guix publish’.
>
> I am afraid having to install Guix and run 'guix publish' will be a
> major obstacle for most mirror sites, especially when their maintainers
> have no previous experience with Guix and have no interest in learning a
> new software and do not want to pollute their server with an unknown
> software and do not want to increase maintenance burden. This is the
> simplest solution on our side, but a complex solution on mirror site
> maintainers' side.

Hmm yes.  I was assuming that someone running a Guix mirror wouldn’t
have troubles running Guix, but there may be cases where this is not the
case.

> I guess a complex Nginx URL rewrite rule might also convert the URL into
> real filename in the cache.  Can anyone familiar with Nginx explain?

I’m not super familiar, but I think it’s doable.

Ludo’.

Re: Make guix-publish's URL identical to cache file name

[zimoun]

Dear,

Thank you.

In case you need arguments to convince academics, this satellite Guix
project could help:

    https://hpc.guix.info/about/


On Thu, 05 Nov 2020 at 09:55, Peng Mei Yu <pengmeiyu@riseup.net> wrote:
> Jonathan Brielmaier writes:
>>
>> I think we should discuss if Guix can support your effort money wise.
>> AFAIK we have decent fundings available.
>
> That would be great.  Although I still think letting academic mirror
> sites do the job and saving the money would be better.  As I said, the
> ISP cost in China is expensive and academic mirror sites have plenty of
> bandwidth and storage resources available.

As Jonathan said, and in the meantime, maybe Guix could support your
effort money wise.  Well, I speak for myself here.

Maybe, you could send an email to guix-europe@gnu.org (association –
non-profit organization) and discuss what it is doable.


All the best,
simon

Re: Make guix-publish's URL identical to cache file name

[Peng Mei Yu]

Hello,

Ludovic Courtès writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> We need to make sure that *all* the files produced by “guix publish”
>> have correct permissions; IIRC some of the files are not readable at all
>> by users other than the owner of the files.
>
> Oops, I had forgotten, my bad.  I’ll push the attached patch later
> today.
>
> Next we’ll need to update the ‘guix’ package, restart ‘guix publish’ on
> berlin, and chmod a+r -R /var/cache/guix/publish.
>
>> Once that’s done we just need to start the rsync daemon again,
>> preferably as a shepherd service.
>
> Yes.
>
> Sounds like we have a plan!
>
> Peng Mei Yu: make sure to ping us in the coming weeks if you don’t hear
> from us by then!

Ping.  What's the status of rsync?


--
Peng Mei Yu

Re: Make guix-publish's URL identical to cache file name

[Ludovic Courtès]

Hi!

Peng Mei Yu <pengmeiyu@riseup.net> skribis:

> Ludovic Courtès writes:
>
>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>
>>> We need to make sure that *all* the files produced by “guix publish”
>>> have correct permissions; IIRC some of the files are not readable at all
>>> by users other than the owner of the files.
>>
>> Oops, I had forgotten, my bad.  I’ll push the attached patch later
>> today.
>>
>> Next we’ll need to update the ‘guix’ package, restart ‘guix publish’ on
>> berlin, and chmod a+r -R /var/cache/guix/publish.

These two issues are now fixed.

> Ping.  What's the status of rsync?

It’s not running yet.  Ricardo, do you have a diff against berlin.scm to
run it?

Thanks,
Ludo’.

Re: Make guix-publish's URL identical to cache file name

[Peng Mei Yu]

Hi,

Ludovic Courtès writes:

> Peng Mei Yu <pengmeiyu@riseup.net> skribis:
>
>> Ludovic Courtès writes:
>>
>>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>>
>>>> We need to make sure that *all* the files produced by “guix publish”
>>>> have correct permissions; IIRC some of the files are not readable at all
>>>> by users other than the owner of the files.
>>>
>>> Oops, I had forgotten, my bad.  I’ll push the attached patch later
>>> today.
>>>
>>> Next we’ll need to update the ‘guix’ package, restart ‘guix publish’ on
>>> berlin, and chmod a+r -R /var/cache/guix/publish.
>
> These two issues are now fixed.
>
>> Ping.  What's the status of rsync?
>
> It’s not running yet.  Ricardo, do you have a diff against berlin.scm to
> run it?

Ping again.


--
Peng Mei Yu