From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hartmut Goebel Subject: Re: Random idea about speeding up guix pull Date: Tue, 5 Sep 2017 14:23:15 +0200 Message-ID: <216b2dc5-efa3-d052-9f68-7d1792cc98ec@crazy-compilers.com> References: <87a82ay0c7.fsf@gnu.org> <6a936a81-683b-6ae3-cabc-79588a97194f@crazy-compilers.com> <87r2vmgmb9.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------7DD1CD3930A1A3A5AD88FA46" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:46416) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dpCt5-00047F-1B for guix-devel@gnu.org; Tue, 05 Sep 2017 08:23:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dpCsz-0003GR-Jh for guix-devel@gnu.org; Tue, 05 Sep 2017 08:23:27 -0400 In-Reply-To: <87r2vmgmb9.fsf@gnu.org> Content-Language: en-US List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= Cc: guix-devel This is a multi-part message in MIME format. --------------7DD1CD3930A1A3A5AD88FA46 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Am 04.09.2017 um 23:56 schrieb Ludovic Court=C3=A8s: > What it does do is maintain a cached checkout in ~/.cache/guix/pull, > which makes subsequent pulls much faster. Summary ( TL;DR): * "guix pull" should use "git fetch master" * "guix download" we can keep the current behaviour I did a series of tests * - "fetch" without any argument will fetch *all* data from *all* branches. * - "fetch master" only fetches data living on "master", other branches are ignored I compared the data fetched for a repo with status of 6bd1c41e8 (yesterday 05:29): * - "fetch" fetches 1000K * - "fetch master" fetches 755K * - "fetch --depth=3D1 master" fetches 588K (but see below) I did some more tests (see results below and=C2=A0 attached script) and h= ad the following insights: * if not checking out FETCH_HEAD after fetch, the next fetch will download all data again (compare "fetch by ref" with "fetch by ref + checkout" * --depth=3D1 will download the *whole* state (at the given ref), no matter how many of the data is already here (compare "fetch by ref + checkout" with "fetch --depth=3D1 by ref + checkout") * I was not able to create a test-case where "fetch --depth=3D1 master"= would only fetch parts of the data =E2=80=93 so this contradicts the = results when updating from 6bd1c41e8. I suggest to make "guix pull" to fetch only from "master", since this already reduces the since of downloaded data. For guix download we don't (need to) cache former downloads, thus "--depth=3D1 " would suffice. Unfortunately this only works for branches and tags, not for commit-ids (see "man git-fetch-pack" for exceptions). But most current package definitions are based on commit-ids. Thus it is not worth trying "--depth=3D1 " first. cloned repo --------------- size 45M fetch all ------------------ size 45M fetch by ref ------------------ size v0.11.0=C2=A0=C2=A0=C2=A0 26M size v0.12.0=C2=A0=C2=A0=C2=A0 32M size v0.13.0=C2=A0=C2=A0=C2=A0 40M size marker-1=C2=A0=C2=A0 45M size marker-2=C2=A0=C2=A0 45M size marker-3=C2=A0=C2=A0 45M size marker-4=C2=A0=C2=A0 45M size marker-5=C2=A0=C2=A0 45M size master=C2=A0=C2=A0=C2=A0=C2=A0 45M fetch by ref + checkout ------------------ size v0.11.0=C2=A0=C2=A0=C2=A0 26M size v0.12.0=C2=A0=C2=A0=C2=A0 11M size v0.13.0=C2=A0=C2=A0=C2=A0 12M size marker-1=C2=A0=C2=A0 8,9M size marker-2=C2=A0=C2=A0 1,1M size marker-3=C2=A0=C2=A0 856K size marker-4=C2=A0=C2=A0 856K size marker-5=C2=A0=C2=A0 1,1M size master=C2=A0=C2=A0=C2=A0=C2=A0 1,1M fetch --depth=3D1 by ref ------------------ size v0.11.0=C2=A0=C2=A0=C2=A0 9,8M size v0.12.0=C2=A0=C2=A0=C2=A0 11M size v0.13.0=C2=A0=C2=A0=C2=A0 13M size marker-1=C2=A0=C2=A0 13M size marker-2=C2=A0=C2=A0 13M size marker-3=C2=A0=C2=A0 13M size marker-4=C2=A0=C2=A0 13M size marker-5=C2=A0=C2=A0 13M size master=C2=A0=C2=A0=C2=A0=C2=A0 13M fetch --depth=3D1 by ref + checkout ------------------ size v0.11.0=C2=A0=C2=A0=C2=A0 9,8M size v0.12.0=C2=A0=C2=A0=C2=A0 3,8M size v0.13.0=C2=A0=C2=A0=C2=A0 5,6M size marker-1=C2=A0=C2=A0 4,1M size marker-2=C2=A0=C2=A0 4,1M size marker-3=C2=A0=C2=A0 4,1M size marker-4=C2=A0=C2=A0 4,1M size marker-5=C2=A0=C2=A0 4,1M size master=C2=A0=C2=A0=C2=A0=C2=A0 4,1M fetch older all and master with --depth=3D1 by ref + checkout ------------------ size master=C2=A0=C2=A0=C2=A0=C2=A0 45M size master=C2=A0=C2=A0=C2=A0=C2=A0 45M --=20 Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | --------------7DD1CD3930A1A3A5AD88FA46 Content-Type: application/x-shellscript; name="test-fetch.sh" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="test-fetch.sh" Ck9SSUdfUkVQTz1+L2RldmVsL2d1aXgKV09SS0RJUj0kVE1QL3Rlc3QtZmV0Y2gKClJFRlM9 InYwLjExLjAgdjAuMTIuMCB2MC4xMy4wIG1hcmtlci0xIG1hcmtlci0yIG1hcmtlci0zIG1h cmtlci00IG1hcmtlci01IG1hc3RlciIKCm1rZGlyIC1wICRXT1JLRElSCmNkICRXT1JLRElS CgpSRVBPPSRXT1JLRElSL3ByaW1hcnkKCmdldF9zaXplICgpIHsKICAgIGxzIC1scnQgLWgg IC5naXQvb2JqZWN0cy9wYWNrLyoucGFjayB8IHRhaWwgLTEgfCBzZWQgJ3MhICAqIVx0IWcn IHwgY3V0IC1mIDUKfQoKaW5pdF9naXRfcmVwbyAoKSB7CiAgICBuYW1lPSQxCiAgICBjZCAk V09SS0RJUgogICAgZ2l0IGluaXQgLXEgJG5hbWUKICAgIGNkICRuYW1lCiAgICBnaXQgcmVt b3RlIGFkZCBvcmlnaW4gJFJFUE8KICAgIH0KCiMgbWFrZSBhIGNsZWFuIGNsb25lIG9mIHRo ZSBvcmlnIHJlcG8KZ2l0IGNsb25lIC0tcXVpZXQgJE9SSUdfUkVQTyBwcmltYXJ5CmNkIHBy aW1hcnkKZWNobyAiYWRkaW5nIHNvbWUgYWRkaXRpb25hbCBtYWFya2VycyIKZ2l0IHRhZyBt YXJrZXItNSBIRUFEfjEwCmdpdCB0YWcgbWFya2VyLTQgSEVBRH40MApnaXQgdGFnIG1hcmtl ci0zIEhFQUR+NTAKZ2l0IHRhZyBtYXJrZXItMiBIRUFEfjEwMApnaXQgdGFnIG1hcmtlci0x IEhFQUR+MTMwCiNnaXQgZ2MgLS1wcnVuZSAtLWFnZ3Jlc3NpdmUKI2dpdCByZXBhY2sgLXEg LWFkCiNnaXQgZ2MgLS1wcnVuZSAtLWFnZ3Jlc3NpdmUKY2QgJFdPUktESVIKCmVjaG8KZWNo byAiY2xvbmVkIHJlcG8gLS0tLS0tLS0tLS0tLS0tIgpnaXQgY2xvbmUgLS1xdWlldCAkUkVQ TyBjbG9uZWQKY2QgY2xvbmVkCmVjaG8gLW4gInNpemUgIiA7IGdldF9zaXplCiNscyAtbHJ0 IC1oICAuZ2l0L29iamVjdHMvcGFjay8qLnBhY2sKCmVjaG8KZWNobyAiZmV0Y2ggYWxsIC0t LS0tLS0tLS0tLS0tLS0tLSIKaW5pdF9naXRfcmVwbyBmZXRjaC1hbGwKZ2l0IGZldGNoIC0t cXVpZXQgb3JpZ2luCmVjaG8gLW4gInNpemUgIiA7IGdldF9zaXplCiNnaXQgY2hlY2tvdXQg LUIgbWFzdGVyIEZFVENIX0hFQUQKI2dpdCByZXNldCAtLWhhcmQgSEVBRH4xMzAKI2VjaG8g LW4gInNpemUgIiA7IGdldF9zaXplCiNscyAtbHJ0IC1oICAuZ2l0L29iamVjdHMvcGFjay8q LnBhY2sKCgplY2hvCmVjaG8gImZldGNoIGJ5IHJlZiAtLS0tLS0tLS0tLS0tLS0tLS0iCmlu aXRfZ2l0X3JlcG8gZmV0Y2gtcmVmCmZvciByZWYgaW4gJFJFRlMgOyBkbwogICAgZ2l0IGZl dGNoIC0tcXVpZXQgb3JpZ2luICRyZWYKICAgIGVjaG8gLW5lICJzaXplICRyZWZcdCIgOyBn ZXRfc2l6ZQpkb25lCiNscyAtbHJ0IC1oICAuZ2l0L29iamVjdHMvcGFjay8qLnBhY2sKCgpl Y2hvCmVjaG8gImZldGNoIGJ5IHJlZiArIGNoZWNrb3V0IC0tLS0tLS0tLS0tLS0tLS0tLSIK aW5pdF9naXRfcmVwbyBmZXRjaC1yZWYtY2hlY2tvdXQKZm9yIHJlZiBpbiAkUkVGUyA7IGRv CiAgICBnaXQgZmV0Y2ggLS1xdWlldCBvcmlnaW4gJHJlZgogICAgZ2l0IGNoZWNrb3V0IC1C IG1hc3RlciBGRVRDSF9IRUFEIDI+L2Rldi9udWxsCiAgICBlY2hvIC1uZSAic2l6ZSAkcmVm XHQiIDsgZ2V0X3NpemUKZG9uZQojbHMgLWxydCAtaCAgLmdpdC9vYmplY3RzL3BhY2svKi5w YWNrCgoKZWNobwplY2hvICJmZXRjaCAtLWRlcHRoPTEgYnkgcmVmIC0tLS0tLS0tLS0tLS0t LS0tLSIKaW5pdF9naXRfcmVwbyBmZXRjaC1yZWYtZGVwdGgKZm9yIHJlZiBpbiAkUkVGUyA7 IGRvCiAgICBnaXQgZmV0Y2ggLS1kZXB0aD0xIC0tcXVpZXQgb3JpZ2luICRyZWYKICAgIGVj aG8gLW5lICJzaXplICRyZWZcdCIgOyBnZXRfc2l6ZQpkb25lCiNscyAtbHJ0IC1oICAuZ2l0 L29iamVjdHMvcGFjay8qLnBhY2sKCgplY2hvCmVjaG8gImZldGNoIC0tZGVwdGg9MSBieSBy ZWYgKyBjaGVja291dCAtLS0tLS0tLS0tLS0tLS0tLS0iCmluaXRfZ2l0X3JlcG8gIGZldGNo LXJlZi1kZXB0aC1jaGVja291dApmb3IgcmVmIGluICRSRUZTIDsgZG8KICAgIGdpdCBmZXRj aCAtLWRlcHRoPTEgLS1xdWlldCBvcmlnaW4gJHJlZgogICAgZ2l0IGNoZWNrb3V0IC1CIG1h c3RlciBGRVRDSF9IRUFEIDI+L2Rldi9udWxsCiAgICBlY2hvIC1uZSAic2l6ZSAkcmVmXHQi IDsgZ2V0X3NpemUKZG9uZQojbHMgLWxydCAtaCAgLmdpdC9vYmplY3RzL3BhY2svKi5wYWNr CgplY2hvCmVjaG8gImZldGNoIG9sZGVyIGFsbCBhbmQgbWFzdGVyIHdpdGggLS1kZXB0aD0x IGJ5IHJlZiArIGNoZWNrb3V0IC0tLS0tLS0tLS0tLS0tLS0tLSIKaW5pdF9naXRfcmVwbyAg ZmV0Y2gtcmVmLWRlcHRoLWNoZWNrb3V0LTIKcmVmPW1hcmtlci00CmdpdCBmZXRjaCAgLS1x dWlldCBvcmlnaW4gJHJlZgpnaXQgY2hlY2tvdXQgLUIgbWFzdGVyIEZFVENIX0hFQUQgMj4v ZGV2L251bGwKZWNobyAtbmUgInNpemUgJHJlZlx0IiA7IGdldF9zaXplCnJlZj1tYXJrZXIt NQpnaXQgZmV0Y2ggLS1kZXB0aD0xIC0tcXVpZXQgb3JpZ2luICRyZWYKZ2l0IGNoZWNrb3V0 IC1CIG1hc3RlciBGRVRDSF9IRUFEIDI+L2Rldi9udWxsCmVjaG8gLW5lICJzaXplICRyZWZc dCIgOyBnZXRfc2l6ZQojbHMgLWxydCAtaCAgLmdpdC9vYmplY3RzL3BhY2svKi5wYWNrCg== --------------7DD1CD3930A1A3A5AD88FA46--