Support rsync to help Chinese users to setup mirrors

Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]


Hi folks!
We've encountered some issues when mirroring Guix replated repos in China.
https://github.com/ustclug/mirrorrequest/issues/96
https://github.com/tuna/issues/issues/199
https://github.com/ustclug/mirrorrequest/issues/96

Is there any plan to support rsync recently?
I'm doing an experiment to use Guix in our product development, but we
need China mirrors. The connection outside China is utterly unstable
because of some reasons.

In the Chinese discussion, someone mentioned that @Ludo had given them a config
script, however, it's just a reverse-proxy, and China servers have
problems to visit upstream because of the firewall.

If Guix upstream support rsync, then it's easier for us to mirror.

Feedbacks are welcome.

Best regards.


--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Support rsync to help Chinese users to setup mirrors

[Ricardo Wurmus]

Hi,

> We've encountered some issues when mirroring Guix replated repos in China.
> https://github.com/ustclug/mirrorrequest/issues/96
> https://github.com/tuna/issues/issues/199
> https://github.com/ustclug/mirrorrequest/issues/96
>
> Is there any plan to support rsync recently?

What exactly is required from us?

If it helps I would like to make it possible for others to sync the
nar cache on berlin with some other locations.

Perhaps that would already be an improvement.

--
Ricardo

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]


Hi Richardo!

Ricardo Wurmus writes:

> What exactly is required from us?

I think the maintainers of each China mirrors want to use rsync, here is
a tutorial of rsync:
https://everythinglinux.org/rsync/

If Guix upstream repo can provide rsync server, then the mirror work will be
easier for China nodes.

Best regards.


--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]


Nala Ginrut writes:

Any progress?
Please don't hesitate to tell if there's any difficulty, or more information
required.

Best regards.

> Hi Richardo!
>
> Ricardo Wurmus writes:
>
>> What exactly is required from us?
>
> I think the maintainers of each China mirrors want to use rsync, here is
> a tutorial of rsync:
> https://everythinglinux.org/rsync/
>
> If Guix upstream repo can provide rsync server, then the mirror work will be
> easier for China nodes.
>
> Best regards.


--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Support rsync to help Chinese users to setup mirrors

[Ricardo Wurmus]

Hi,

> Any progress?

Not yet as I was busy with other things.  It’s on my TODO list, though,
to share the cache directory.

> Please don't hesitate to tell if there's any difficulty, or more information
> required.

Extra information would always be useful.  I’m assuming that a direct
copy of the store would not be helpful, because that’s not what clients
request.  They fetch nars and narinfos instead, which we bake on demand
and then cache.  I’m planning to expose that cache.

-- 
Ricardo

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]


Hi Ricardo!

Ricardo Wurmus writes:

> Extra information would always be useful.  I’m assuming that a direct
> copy of the store would not be helpful, because that’s not what clients
> request.  They fetch nars and narinfos instead, which we bake on demand
> and then cache.  I’m planning to expose that cache.

How do you create this cache? Maybe we can just follow you in the same
way. We're not trying to fetch packages like regular Guix clients, we'd
like to provide a mirror/cache to help users to accelerate downloading
inside the firewall covered area.

Best regards.


--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Support rsync to help Chinese users to setup mirrors

[Ricardo Wurmus]

Nala Ginrut <mulei@gnu.org> writes:

> How do you create this cache? Maybe we can just follow you in the same
> way. We're not trying to fetch packages like regular Guix clients, we'd
> like to provide a mirror/cache to help users to accelerate downloading
> inside the firewall covered area.

Guix clients fetch binaries in nar and narinfo format.  Mirroring them
might be sufficient.  To create the cache requires running “guix
publish” with a /gnu/store, which cannot easily be mirrored as it needs
to be in sync with its database.

-- 
Ricardo

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]


Hi Richardo!

Ricardo Wurmus writes:

> Guix clients fetch binaries in nar and narinfo format.  Mirroring them
> might be sufficient.  To create the cache requires running “guix
> publish” with a /gnu/store, which cannot easily be mirrored as it needs
> to be in sync with its database.

What I expect is to mirror nar/narinfo conveniently and fast, that is
the purpose why I raise this topic. And I know to run "guix publish" to
create a repo for users, the problem I want to solve is to sync
/gnu/store with the upstream conveniently and fast.

There is the national firewall in China, so common users have difficult
to download Guix packages in a fair speed. This may hardly understand by
people outside China, but that is the fact.

We hope there's a way like rsync to sync /gnu/store with upstream, so
that we can provide a faster cache for Chinese users.


Best regards.

--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Support rsync to help Chinese users to setup mirrors

[Ricardo Wurmus]

Hi,

> There is the national firewall in China, so common users have difficult
> to download Guix packages in a fair speed. This may hardly understand by
> people outside China, but that is the fact.

I’m aware.  I lived with the firewall for ~7 years and would like to
make sure that people who are subjected to the firewall can use Guix
without being restricted.

> We hope there's a way like rsync to sync /gnu/store with upstream, so
> that we can provide a faster cache for Chinese users.

I think it may be best to just sync the cache of nars and narinfos.

Would rsync over SSH be sufficient or does it have to be rsyncd on the
default rsync port?  (SSH will be easier for me because then I don’t
need to apply for another port to be opened at the institute firewall.)

I’ve also been looking into our options for restricting rsync access via
chroot or namespaces.  Looks like the easiest way to do this is to have
an rsync user account that is restricted to a chroot with access to the
nar cache.

--
Ricardo

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]


Ricardo Wurmus writes:

> I’m aware.  I lived with the firewall for ~7 years and would like to
> make sure that people who are subjected to the firewall can use Guix
> without being restricted.

Alas, why does this happen everywhere in this decade...

>> We hope there's a way like rsync to sync /gnu/store with upstream, so
>> that we can provide a faster cache for Chinese users.
>
> I think it may be best to just sync the cache of nars and narinfos.

Yes it's better to just sync the necessary parts.

> Would rsync over SSH be sufficient or does it have to be rsyncd on the
> default rsync port?  (SSH will be easier for me because then I don’t
> need to apply for another port to be opened at the institute firewall.)

I think SSH will be interfered frequently by the firewall, because many
people use SSH for anti-circumvention. It's better to use rsyncd.
Considering there's verification, so the encryption is unnecessary.

> I’ve also been looking into our options for restricting rsync access via
> chroot or namespaces.  Looks like the easiest way to do this is to have
> an rsync user account that is restricted to a chroot with access to the
> nar cache.

I think it can be configured as anonymouse read-only authority. And
open the necessary directories.


Best regards.

--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Support rsync to help Chinese users to setup mirrors

[Ricardo Wurmus]

Nala Ginrut <mulei@gnu.org> writes:

>> Would rsync over SSH be sufficient or does it have to be rsyncd on the
>> default rsync port?  (SSH will be easier for me because then I don’t
>> need to apply for another port to be opened at the institute firewall.)
>
> I think SSH will be interfered frequently by the firewall, because many
> people use SSH for anti-circumvention. It's better to use rsyncd.
> Considering there's verification, so the encryption is unnecessary.

Oh pity.  Back then I did in fact use SSH tunnels to get around the
great firewall; VPNs were the primary target back then.

rsyncd is going to take longer as I need to make an application to the
campus IT to let them open the port.  I’ll let you know when it’s ready,
but it can take a little while.

--
Ricardo

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]


Hi Richardo!

Ricardo Wurmus writes:

> rsyncd is going to take longer as I need to make an application to the
> campus IT to let them open the port.  I’ll let you know when it’s ready,
> but it can take a little while.

Thanks in advanced!
We're trying to use Guix in our development for an experiment. If
everything goes well, we will try it in a product. Guix is worth to try
for complex system.
If the China mirror can be done, then it's a good step for us to build
Chinese Guix community.

Best regards.

--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]