From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 2BpgEZCNol/rVAAA0tVLHw (envelope-from ) for ; Wed, 04 Nov 2020 11:16:32 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id +IJKDZCNol80MAAA1q6Kng (envelope-from ) for ; Wed, 04 Nov 2020 11:16:32 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E3FB094065D for ; Wed, 4 Nov 2020 11:16:31 +0000 (UTC) Received: from localhost ([::1]:35306 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kaGmE-0003WG-Lb for larch@yhetil.org; Wed, 04 Nov 2020 06:16:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:47218) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kaERA-0002y1-1e for guix-devel@gnu.org; Wed, 04 Nov 2020 03:46:36 -0500 Received: from mx1.riseup.net ([198.252.153.129]:53272) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kaER7-0006KV-45 for guix-devel@gnu.org; Wed, 04 Nov 2020 03:46:35 -0500 Received: from bell.riseup.net (bell-pn.riseup.net [10.0.1.178]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.riseup.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (not verified)) by mx1.riseup.net (Postfix) with ESMTPS id 4CR0cZ5X6RzFqwG for ; Wed, 4 Nov 2020 00:46:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1604479582; bh=k6NuUzAjHNtGjdCbxm5XmwzFcX7/nYQlj54BTFA+fUo=; h=From:To:Subject:Date:From; b=jcdgcwMAB0ZLWHBkSVMKU7bqwggV8jcfx0k0RvIH77UT+0MCPedDg5Gty5KemuIUx piVChhRXmZHOw+Zx4qOZJ9L7AFVGAdWZq5uDV7LnOd+vJxwidgSorz8ux/kffqZQxE vRfuU/JHPwVvyezahIjXexIwAb7JCuyd0zj2QwSo= X-Riseup-User-ID: FD3D20B2D790883806B9D9355B27543319DEDAE4BD42208FF2CF5C51E99FC1AA Received: from [127.0.0.1] (localhost [127.0.0.1]) by bell.riseup.net (Postfix) with ESMTPSA id 4CR0cZ0PlhzJrPb for ; Wed, 4 Nov 2020 00:46:19 -0800 (PST) Received: from localhost (captain [local]) by captain (OpenSMTPD) with ESMTPA id a80221f3 for ; Wed, 4 Nov 2020 08:46:04 +0000 (UTC) From: Peng Mei Yu To: guix-devel@gnu.org Subject: Make guix-publish's URL identical to cache file name Date: Wed, 04 Nov 2020 16:46:04 +0800 Message-ID: <87o8kdpl0z.fsf@pengmeiyu.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=198.252.153.129; envelope-from=pengmeiyu@riseup.net; helo=mx1.riseup.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/04 03:46:25 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Wed, 04 Nov 2020 06:13:46 -0500 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: ns3122888.ip-94-23-21.eu Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=riseup.net header.s=squak header.b=jcdgcwMA; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: -0.21 X-TUID: vx0s1EQpvyxN Hi, This proposal aims to solve an old problem. Make it easier to setup a mirror server for the official substitute server and prevent future complaints from China residents about network speed. Due to the national firewall(i.e. The GFW) deployed on every backbone networks within China and it's aggressive rules on traffic passing through the wall, the internet connection from inner China to outside world is extremely unreliable and slow. This makes it a pain for newcomers to try and play with Guix, since guix daemon loves to spawn thousands of HTTP requests to the substitute server and download thousands of megabytes of packages from the substitute server. Usually it takes a whole day to install Guix system on a new computer. I am serious. I have been a Guix user for several years and actually I am not very much bothered by the problem. But since V1.0 release of Guix, Guix is frequently mentioned in tech news. When tech savvy boys hear the cool Guix and try to install it on their computer and still cannot complete the installation process after half a day, they will always be pissed off by the slow network speed and complain on internet. I mentioned Guix several times on some websites, this makes me the first Chinese Guix user they can find with Google and ask for help. I am sure maintainers here know these kind of complaints. Do you feel it is strange that no Chinese user make new complaints in year 2020 while the Guix project is quickly progressing and becoming more popular? That is because I decided to setup a mirror server for Chinese Guix users after receiving several complaints online and realizing that lobbying academic FLOSS mirror maintainers to support Guix will be stagnant for at least several years due to these maintainers' laziness and cowardliness and some ridiculous strict governmental regulations. These people only want to add mirror for a project if it is as simple as pulling static files with a cron job (usually with rsync) and serving static files through a simple HTTP server. HTTP reverse proxy is not an option. So my mirror.guix.org.cn project was started. It's an HTTP cache mirror of ci.guix.gnu.org, plus a git mirror of https://git.savannah.gnu.org/cgit/guix.git. Yes, the connection to Savannah is also extremely slow and it makes `guix pull` unusable. This mirror server started as an experiment and it has been working well. If random new user come to me and say `guix pull` is so slow or `guix install` is so slow, I simply tell them to use mirror.guix.org.cn. The number of active Chinese Guix users I know has increased from two to about ten after someone's broadcast in a news group. It is basically: "Look. There is thing called Guix. Someone has setup a mirror server for it in China." The traffic on the server is increasing. Network connections are stable. Everything is fine in this year. However one thing worries me. The bandwidth of mirror.guix.org.cn is only 5Mbps (still far more better than ci.guix.gnu.org's 30KB/s and constant connection reset). This is the highest bandwidth I can afford because internet bandwidth in China is damn too expensive. Buying higher bandwidth is not a financially possible approach for me. This is not a problem in the short term but definitely be a problem in the long term. Persuading academic FLOSS mirror maintainers to support Guix is still the best solution for Chinese users. Academic organizations usually have 100Mbps bandwidth and tens of terabytes of disk. Now, finally, we are onto the main point. I look into guix publish's cache directory and think that nar and narinfo files can be directly served through a static HTTP server if we make those files' URL identical to their on-disk file name. The current directory structure is like this: --8<---------------cut here---------------start------------->8--- /var/cache/guix/publish =E2=94=9C=E2=94=80=E2=94=80 gzip =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67-ba= sh-5.0.16.nar =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67-ba= sh-5.0.16.narinfo =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 fa6wj5bxkj5ll1d7292a70knmyl7a0cr-gl= ibc-2.31.nar =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 fa6wj5bxkj5ll1d7292a70knmyl7a0cr-gl= ibc-2.31.narinfo =E2=94=9C=E2=94=80=E2=94=80 hashes =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 fa6wj5bxkj5ll1d7292a70knmyl7a0cr =E2=94=9C=E2=94=80=E2=94=80 last-expiry-cleanup =E2=94=94=E2=94=80=E2=94=80 lzip =E2=94=9C=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.1= 6.nar =E2=94=9C=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.1= 6.narinfo =E2=94=9C=E2=94=80=E2=94=80 fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31= .nar =E2=94=94=E2=94=80=E2=94=80 fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31= .narinfo --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- > md5sum /var/cache/guix/publish/*/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.= 0.16.narinfo 29cdbf041b9a304bf58f2e75ec23f18f /var/cache/guix/publish/gzip/87kif0bpf0an= wbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo 29cdbf041b9a304bf58f2e75ec23f18f /var/cache/guix/publish/lzip/87kif0bpf0an= wbsaw0jvg8fyciw4sz67-bash-5.0.16.narinfo --8<---------------cut here---------------end--------------->8--- When a client tries to download /gnu/store/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16, it sends a request to http://example.com/87kif0bpf0anwbsaw0jvg8fyciw4sz67.narinfo and gets the content of /var/cache/guix/publish/gzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.n= arinfo: --8<---------------cut here---------------start------------->8--- StorePath: /gnu/store/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16 URL: nar/gzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16 Compression: gzip FileSize: 2284657 URL: nar/lzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16 Compression: lzip FileSize: 1256260 NarHash: sha256:1ap2s3xz3bbp5n78v826gxagy7pic1wpgzz3ka72jdyk6qpmw3qr NarSize: 6597040 References: ... System: x86_64-linux Deriver: cccyyn4xq59aimybmhlrfl2bi8kslhlm-bash-5.0.16.drv Signature: ... --8<---------------cut here---------------end--------------->8--- Client then sends a request to the URL as written in the URL field: http://example.com/nar/lzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16. This URL returns the file /var/cache/guix/publish/lzip/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.n= ar. I propose we make the URL field in narinfo the same as nar file name on disk. We can change the directory structure to: --8<---------------cut here---------------start------------->8--- /var/cache/guix/publish/nar =E2=94=9C=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67.narinfo =E2=94=9C=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.na= r.gz =E2=94=94=E2=94=80=E2=94=80 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.na= r.lz --8<---------------cut here---------------end--------------->8--- And change the URL field in narinfo to --8<---------------cut here---------------start------------->8--- URL: 87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16.nar.lz --8<---------------cut here---------------end--------------->8--- Then a mirror site can simply pull the directory /var/cache/guix/publish/nar from the Berlin server and serve this directory through a static HTTP server. There will be cache misses. But guix-daemon will safely fallback to the next server in substitute-urls. What's your opinion? I have to decide next year's server specs and budget for mirror.guix.org.cn before the Chinese shopping festival ends on November 11. If the proposal above is doable, I will keep mirror.guix.org.cn running for half a year and help academic mirror sites add support for Guix in the meantime. Otherwise I prefer to buy a prepaid three years VPS with a 90% discount during the shopping festival. The discount is huge. I don't want to miss it. -- Peng Mei Yu