From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id uOoeG+MUQmCyDAAA0tVLHw (envelope-from ) for ; Fri, 05 Mar 2021 11:24:19 +0000 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id eMj4FuMUQmCuJgAA1q6Kng (envelope-from ) for ; Fri, 05 Mar 2021 11:24:19 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2B64B193D9 for ; Fri, 5 Mar 2021 12:24:18 +0100 (CET) Received: from localhost ([::1]:59796 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lI8Z5-00061I-Vr for larch@yhetil.org; Fri, 05 Mar 2021 06:24:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:58190) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lI8Xu-00054Y-EW for bug-guix@gnu.org; Fri, 05 Mar 2021 06:23:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:49691) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lI8Xu-0004zd-7O for bug-guix@gnu.org; Fri, 05 Mar 2021 06:23:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lI8Xu-0003iz-3A for bug-guix@gnu.org; Fri, 05 Mar 2021 06:23:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#46942: ci.guix.gnu.org is slow from my system Resent-From: raid5atemyhomework Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 05 Mar 2021 11:23:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 46942 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 46942@debbugs.gnu.org X-Debbugs-Original-To: "bug-guix@gnu.org" Received: via spool by submit@debbugs.gnu.org id=B.161494335414266 (code B ref -1); Fri, 05 Mar 2021 11:23:01 +0000 Received: (at submit) by debbugs.gnu.org; 5 Mar 2021 11:22:34 +0000 Received: from localhost ([127.0.0.1]:33004 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lI8XR-0003i0-OH for submit@debbugs.gnu.org; Fri, 05 Mar 2021 06:22:34 -0500 Received: from lists.gnu.org ([209.51.188.17]:58550) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lI8XO-0003hp-3H for submit@debbugs.gnu.org; Fri, 05 Mar 2021 06:22:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:58142) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lI8XN-0004n0-Hk for bug-guix@gnu.org; Fri, 05 Mar 2021 06:22:29 -0500 Received: from mail-40140.protonmail.ch ([185.70.40.140]:42217) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lI8XJ-0004j5-Bm for bug-guix@gnu.org; Fri, 05 Mar 2021 06:22:29 -0500 Date: Fri, 05 Mar 2021 11:22:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail; t=1614943340; bh=1CnJThCoa27ZULKakDoCuU2UCMwhMWe29uil8hphJHY=; h=Date:To:From:Reply-To:Subject:From; b=jbpIJecvb4rgC0xVItWyOMurGBu8o0IYffhNXUHKLXRGQp8f7s1k6Q9NApViQcUOK MdwhSEP4hpPtRdhvg110KuwrrZ/rlv/pJdbvyoHah7prx5eQYRCYHJvqCSqgVwAX6G 45Pl4wfc92Hior9NKRmo4q+qRWix6MHjd7JGdxDc= Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=185.70.40.140; envelope-from=raid5atemyhomework@protonmail.com; helo=mail-40140.protonmail.ch X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" Reply-to: raid5atemyhomework X-ACL-Warn: , raid5atemyhomework From: raid5atemyhomework via Bug reports for GNU Guix X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.86 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=protonmail.com header.s=protonmail header.b=jbpIJecv; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: 2B64B193D9 X-Spam-Score: -2.86 X-Migadu-Scanner: scn0.migadu.com X-TUID: dIfv3MwAGHGr Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using= computers. One is a pure Guix System install without any channels, the other one is a = foreign Guix install with the-channel-that-cannot-be-named. ``` downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmw= qrrb8lz-wine64-6.0 ... wine64-6.0 54.4MiB = 1= 0KiB/s 01:55 [ ] 2.1% ``` This can be very slow, including as slow as 4KiB/s at times. This is fairly= awful since sometimes I get this result: ``` downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmw= qrrb8lz-wine64-6.0 ... wine64-6.0 54.4MiB = 4= 9KiB/s 06:01 [##### ] 31.9%guix substitute: error: TLS error i= n procedure 'read_from_session_record_port': Error decoding the received TL= S packet. substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 fail= ed guix package: error: some substitutes for the outputs of derivation `/gnu/s= tore/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happe= ns due to networking issues); try `--fallback' to build derivation from sou= rce ``` Because of the failed incomplete download, I have to ***restart the downloa= d again from 0%***, which is awful, awful UX. It would be really nice if: 1. The guix download process would make an effort to ***retry*** this a few= times. 2. Continue the previous download instead of restarting from 0%. The problem is not on my ISP, or at least not solely on my ISP. Doing a `wg= et` from `github.com`: ``` $ wget https://github.com/zfsonlinux/zfs/releases/download/zfs-2.0.3/zfs-2.= 0.3.tar.gz --2021-03-05 17:39:46-- https://github.com/zfsonlinux/zfs/releases/downloa= d/zfs-2.0.3/zfs-2.0.3.tar.gz Resolving github.com (github.com)... 13.229.188.59 Connecting to github.com (github.com)|13.229.188.59|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://github.com/openzfs/zfs/releases/download/zfs-2.0.3/zfs-2.= 0.3.tar.gz [following] --2021-03-05 17:39:47-- https://github.com/openzfs/zfs/releases/download/z= fs-2.0.3/zfs-2.0.3.tar.gz Reusing existing connection to github.com:443. HTTP request sent, awaiting response... 302 Found Location: https://github-releases.githubusercontent.com/437011/71526a80-6ba= 3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=3DAWS4-HMAC-SHA256&X-Amz-Credentia= l=3DAKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-D= ate=3D20210305T093947Z&X-Amz-Expires=3D300&X-Amz-Signature=3D9454446348219a= a0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-SignedHeaders=3Dho= st&actor_id=3D0&key_id=3D0&repo_id=3D437011&response-content-disposition=3D= attachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-content-type=3Dapplica= tion%2Foctet-stream [following] --2021-03-05 17:39:47-- https://github-releases.githubusercontent.com/4370= 11/71526a80-6ba3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=3DAWS4-HMAC-SHA256&= X-Amz-Credential=3DAKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_= request&X-Amz-Date=3D20210305T093947Z&X-Amz-Expires=3D300&X-Amz-Signature= =3D9454446348219aa0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-S= ignedHeaders=3Dhost&actor_id=3D0&key_id=3D0&repo_id=3D437011&response-conte= nt-disposition=3Dattachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-conte= nt-type=3Dapplication%2Foctet-stream Resolving github-releases.githubusercontent.com (github-releases.githubuser= content.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ... Connecting to github-releases.githubusercontent.com (github-releases.github= usercontent.com)|185.199.108.154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 13114404 (13M) [application/octet-stream] Saving to: =E2=80=98zfs-2.0.3.tar.gz=E2=80=99 zfs-2.0.3.tar.gz 100%[=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>] 12.51M 9.34MB= /s in 1.3s 2021-03-05 17:39:49 (9.34 MB/s) - =E2=80=98zfs-2.0.3.tar.gz=E2=80=99 saved = [13114404/13114404] ``` As can be seen above, I get 9.34MB/s elsewhere, which is better than the >6= 0Mbit/s promised by my ISP. It's possible that the problem is between my ISP and ci.guix.gnu.org. If I = `wget` directly: ``` $ wget https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wi= ne64-6.0 --2021-03-05 18:56:21-- https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh= 76xw6hmwqrrb8lz-wine64-6.0 Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40 Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... conne= cted. HTTP request sent, awaiting response... 200 OK Length: 57048561 (54M) [application/octet-stream] Saving to: =E2=80=981bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0=E2=80=99 1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 0%[ = = ] 167.79K 13.4KB/s eta 69m 20s^C ``` HOWEVER, if I `torify wget`: ``` $ torify wget https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrr= b8lz-wine64-6.0 --2021-03-05 18:57:00-- https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh= 76xw6hmwqrrb8lz-wine64-6.0 Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40 Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... conne= cted. HTTP request sent, awaiting response... 200 OK Length: 57048561 (54M) [application/octet-stream] Saving to: =E2=80=981bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1=E2=80= =99 1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1 13%[=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D> = ] 7.13M 683KB/s eta 72s ^C ``` Which is a big WHAT THE FUCK? Tor should not be 45x faster (683KB/s) than d= irectly from my ISP (13KB/s). I am not using any special workarounds for To= r, like meek or obfs4, which makes me doubt that my ISP is attempting to ce= nsor --- if my ISP is the one doing the censoring, it would probably target= Tor first before ci.guix.gnu.org, but I can access Tor just fine without s= pecial workarounds needed for very oppressive countries. I've tried this a few times. Consistently, using `torify wget` is much fas= ter than `wget` alone, I don't have to bang my head in boredom with `torify= wget`. This is not my expectation given what I understand of the Tor netw= ork and the public network. Is `ci.guix.gnu.org` doing some kind of per-IP or per-ISP or other throttli= ng? This looks very suspicious given the massive speed difference when usi= ng Tor and not using Tor. Doing `torify guix package -m manifest.scm` does not seem to perform the do= wnload inside Tor, which makes sense since it's probably `guix-daemon` that= is doing the download. So my questions are: * WHY IS DIRECT DOWNLOAD FROM MY ISP SLOWER THAN TOR? What can I do to chec= k if it's my ISP that is attempting to censor ci.guix.gnu.org? * Is there a way to make `guix-daemon` use a Tor proxy? I have two systems= using Guix, one is a Guix System, the other is using a foreign distro, and= I'd like to adjust both to use Tor instead since it's faster. * If not, is there a way to tell `guix-daemon` that I have these `nar` fi= les and it can put them in its store somehow instead of me waiting for it t= o ***SLOOOOOOOOWLY*** download it? * Are there any mirrors I can use for substitutes instead? How do I change = my systems (both the Guix System and the foreign distro) to use the mirrors= ? Is there some configuration option to do this? * How hard would it be to make the substitute download process at least mak= e an effort to attempt to continue a failed download instead of failing imm= ediately and forcing a restart from 0%? `texlive` is something like 400Mb, = it takes hours at 10KiB/s, I once had it fail at >90% which was very painfu= l because it had to restart from 0%, I had to do `guix package -m manifest.= scm` in a loop over and over again until it finished download. It would pro= bably cut down on people hammering on the server as well. Thanks raid5atemyhomework