Provide rsync access to GNU ELPA for mirroring it

Provide rsync access to GNU ELPA for mirroring it

[Chunyang Xu]

Hello emacs-devel!

The emacs-china user group is running an ELPA mirror [1] to provide
faster and more reliable connection for Chinese Emacs users.
Currently, it simply downloads all files through HTTP, it works well
for more than one month, however, mirroring through rsync is obviously
easier and more efficient. So please consider providing rsync access
to Gnu elpa.

[1] https://github.com/emacs-china/elpa

Thanks.

Chunyang

Re: Provide rsync access to GNU ELPA for mirroring it

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 615 bytes --]

On 2016-07-02 12:02, Chunyang Xu wrote:
> Hello emacs-devel!
> 
> The emacs-china user group is running an ELPA mirror [1] to provide
> faster and more reliable connection for Chinese Emacs users.
> Currently, it simply downloads all files through HTTP, it works well
> for more than one month, however, mirroring through rsync is obviously
> easier and more efficient. So please consider providing rsync access
> to Gnu elpa.

Is there a reason to not build the packets locally, instead of downloading them all from ELPA?
(Same for MELPA, btw; this is going to ruin download statistics)

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Provide rsync access to GNU ELPA for mirroring it

[Chunyang Xu]

On Sun, Jul 3, 2016 at 3:19 AM, Clément Pit--Claudel
<clement.pit@gmail.com> wrote:
> On 2016-07-02 12:02, Chunyang Xu wrote:
>> Hello emacs-devel!
>>
>> The emacs-china user group is running an ELPA mirror [1] to provide
>> faster and more reliable connection for Chinese Emacs users.
>> Currently, it simply downloads all files through HTTP, it works well
>> for more than one month, however, mirroring through rsync is obviously
>> easier and more efficient. So please consider providing rsync access
>> to Gnu elpa.
>
> Is there a reason to not build the packets locally, instead of downloading them all from ELPA?

Good point, actually I never think about this way before, maybe this
is a good way for Gnu ELPA. git access to savannah is very slow (< 50
KiB/s) and unreliable here, I will try this way after (if) git-clone
is done. Also, I am not sure if building locally can keep old versions
of packages like GNU ELPA.

Anyway, I still think rsync is the best way to mirror.

> (Same for MELPA, btw; this is going to ruin download statistics)

MELPA itself needs 3 hours to build its 3000+ packages. Building MELPA
packages locally is not easy.

I am not very sure if we are ruining the download statistics, but the
user-agent we used is always different from Emacs/url.el/package.el

> Clément.
>

Re: Provide rsync access to GNU ELPA for mirroring it

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 475 bytes --]

On 07/03/2016 09:16 AM, Chunyang Xu wrote:
> I am not very sure if we are ruining the download statistics, but the
> user-agent we used is always different from Emacs/url.el/package.el

All packages on MELPA get three downloads per day due to bots (corollary: a one-year-old package that has about 1000 download has more or less never been downloaded by a real user). Part of the problem is that MELPA doesn't look at user agents when counting downloads.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Provide rsync access to GNU ELPA for mirroring it

[Stefan Monnier]

>> Is there a reason to not build the packets locally, instead of
>> downloading them all from ELPA?
> Good point, actually I never think about this way before, maybe this
> is a good way for Gnu ELPA. git access to savannah is very slow (< 50
> KiB/s) and unreliable here, I will try this way after (if) git-clone
> is done. Also, I am not sure if building locally can keep old versions
> of packages like GNU ELPA.

No, indeed, the current build scripts won't always generate the same
packages: it depends on when they're run.

> Anyway, I still think rsync is the best way to mirror.

Agreed.  I'll see if I find the time to setup a read-only rsync access.
I'm sure it's not too complicated, but if someone can provide me the
typical steps and a sample config file (on a Debian machine) it would
probably nudge me towards doing it more quickly.


        Stefan

Re: Provide rsync access to GNU ELPA for mirroring it

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1094 bytes --]

On 2016-07-13 04:45, Stefan Monnier wrote:
> but if someone can provide me the
> typical steps and a sample config file (on a Debian machine) it would
> probably nudge me towards doing it more quickly.

Something like this should work, more or less:

1. Set RSYNC_ENABLE to true in /etc/default/rsync

2. Put this following in /etc/rsyncd.conf:

        pid file=/var/run/rsyncd.pid

        uid = nobody
        gid = nogroup
        list = yes
        read only = yes
        use chroot = yes # CHECK ME
        max connections = 4 # CHANGE ME

        [elpa]
            comment = GNU Elpa rsync mirror
            path = /var/www/pub # CHANGE ME
            refuse options = checksum dry-run
            dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

This is a mix of rsyncd.conf's manpage and /usr/share/doc/rsync/examples/rsyncd.conf.  You'll need to make /var/www/pub (or whichever folder you pick) world-readable, and then to figure out how to atomically update its contents (I'll trust your shell-script-fu on that one :).

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Provide rsync access to GNU ELPA for mirroring it

[Stefan Monnier]

> Something like this should work, more or less:

Thanks.  It seems to be working now, but only from behind the firewall
for now.


        Stefan

Re: Provide rsync access to GNU ELPA for mirroring it

[Stefan Monnier]

>> but if someone can provide me the
>> typical steps and a sample config file (on a Debian machine) it would
>> probably nudge me towards doing it more quickly.
> Something like this should work, more or less:

Should work now:

    rsync -av --delete elpa.gnu.org::elpa/. /mirror/elpa/.


-- Stefan

Re: Provide rsync access to GNU ELPA for mirroring it

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 549 bytes --]

On 2016-07-15 17:41, Stefan Monnier wrote:
>>> but if someone can provide me the
>>> typical steps and a sample config file (on a Debian machine) it would
>>> probably nudge me towards doing it more quickly.
>> Something like this should work, more or less:
> 
> Should work now:
> 
>     rsync -av --delete elpa.gnu.org::elpa/. /mirror/elpa/.
> 

Tested and confirmed to work nicely :)
You probably want to suggest -avz instead of -av, though, to enable compression.

Out of curiosity, how did you solve the atomic update problem?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Provide rsync access to GNU ELPA for mirroring it

[Stefan Monnier]

> Out of curiosity, how did you solve the atomic update problem?

Energetic hand waving?


        Stefan