Random idea about speeding up guix pull

Random idea about speeding up guix pull

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

Hi,

I've seen in Ludo's slides that speeding up guix pull is topic. Here is
a random idea on the:

"git pull" can probably be speed up by using something like

    git init .
    git remote add …
    git fetch --depth=1 origin master
    git checkout FETCH_HEAD

This will only download the top-most commit resp. commit-state.

From my mostly up-t-date clone, this method downloads only 1559 objects
and 'du -s .git' reports 13M – compared to "git pull" downloading 133284
objects and taking 49M.

We could use this for downloading sourcce-code via git (git-download).

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: 0xBF773B65.asc --]
[-- Type: application/pgp-keys, Size: 14855 bytes --]

Re: Random idea about speeding up guix pull

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1455 bytes --]

Hartmut Goebel transcribed 15K bytes:
> Hi,
> 
> I've seen in Ludo's slides that speeding up guix pull is topic. Here is
> a random idea on the:
> 
> "git pull" can probably be speed up by using something like
> 
>     git init .
>     git remote add …
>     git fetch --depth=1 origin master
>     git checkout FETCH_HEAD
> 
> This will only download the top-most commit resp. commit-state.
> 
> From my mostly up-t-date clone, this method downloads only 1559 objects
> and 'du -s .git' reports 13M – compared to "git pull" downloading 133284
> objects and taking 49M.

Yes, that would make many git clones take less space.

> We could use this for downloading sourcce-code via git (git-download).

Andy Wingo has proposed this in the past and had a patch which once
upon a time in 2015 worked. If you are motivated enough to adjust it,
it's still on the list but git-download and the other file it touches
has been changed very much since 2015.

> -- 
> Regards
> Hartmut Goebel
> 
> | Hartmut Goebel          | h.goebel@crazy-compilers.com               |
> | www.crazy-compilers.com | compilers which you thought are impossible |
> 

pub   RSA 4096/BF773B65 2013-10-05 Hartmut Goebel <h.goebel@goebel-consult.de>
> sub   RSA 4096/DDEAFF1A 2013-10-05
> > 


-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Random idea about speeding up guix pull

[Ludovic Courtès]

Heya,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> I've seen in Ludo's slides that speeding up guix pull is topic. Here is
> a random idea on the:
>
> "git pull" can probably be speed up by using something like
>
>     git init .
>     git remote add …
>     git fetch --depth=1 origin master
>     git checkout FETCH_HEAD
>
> This will only download the top-most commit resp. commit-state.

That’s roughly what ‘guix pull’ does nowadays, now that it uses
Guile-Git.

The problem is elsewhere: it’s compiling Guix’s Scheme code that takes
ages, in particular since we switch to Guile 2.2 (Guile 2.2’s fancy
compiler gives us significant speedups at run time on core Guix, but
it’s also slower when compiling simple code like package definitions.)

Ludo’.

Re: Random idea about speeding up guix pull

[Hartmut Goebel]

Am 04.09.2017 um 17:01 schrieb Ludovic Courtès:
> That’s roughly what ‘guix pull’ does nowadays, now that it uses
> Guile-Git.

Does it? I only found the call to `remote-fetch` in guix/git.scm, which
is not passed any option to.

The trick is to use `--depth=1` and fetch the one, expected commit, tag
or branch-head.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Random idea about speeding up guix pull

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 04.09.2017 um 17:01 schrieb Ludovic Courtès:
>> That’s roughly what ‘guix pull’ does nowadays, now that it uses
>> Guile-Git.
>
> Does it? I only found the call to `remote-fetch` in guix/git.scm, which
> is not passed any option to.
>
> The trick is to use `--depth=1` and fetch the one, expected commit, tag
> or branch-head.

Oh right, it doesn’t do that.

What it does do is maintain a cached checkout in ~/.cache/guix/pull,
which makes subsequent pulls much faster.

Does that make sense?

Ludo’.

Re: Random idea about speeding up guix pull

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 3264 bytes --]

Am 04.09.2017 um 23:56 schrieb Ludovic Courtès:
> What it does do is maintain a cached checkout in ~/.cache/guix/pull,
> which makes subsequent pulls much faster.

Summary ( TL;DR):

  * "guix pull" should use "git fetch master"
  * "guix download" we can keep the current behaviour

I did a series of tests

  * - "fetch" without any argument will fetch *all* data from *all*
    branches.
  * - "fetch master" only fetches data living on "master", other
    branches are ignored

I compared the data fetched for a repo with status of 6bd1c41e8
(yesterday 05:29):

  * - "fetch" fetches 1000K
  * - "fetch master" fetches 755K
  * - "fetch --depth=1 master" fetches 588K (but see below)

I did some more tests (see results below and  attached script) and had
the following insights:

  * if not checking out FETCH_HEAD after fetch, the next fetch will
    download all data again (compare "fetch by ref" with "fetch by ref +
    checkout"
  * --depth=1 will download the *whole* state (at the given ref), no
    matter how many of the data is already here (compare "fetch by ref +
    checkout" with "fetch --depth=1 by ref + checkout")
  * I was not able to create a test-case where "fetch --depth=1 master"
    would only fetch parts of the data – so this contradicts the results
    when updating from 6bd1c41e8.

I suggest to make "guix pull" to fetch only from "master", since this
already reduces the since of downloaded data.

For guix download we don't (need to) cache former downloads, thus
"--depth=1 <commit>" would suffice. Unfortunately this only works for
branches and tags, not for commit-ids (see "man git-fetch-pack" for
exceptions). But most current package definitions are based on
commit-ids. Thus it is not worth trying "--depth=1 <commit>" first.

cloned repo ---------------
size 45M

fetch all ------------------
size 45M

fetch by ref ------------------
size v0.11.0    26M
size v0.12.0    32M
size v0.13.0    40M
size marker-1   45M
size marker-2   45M
size marker-3   45M
size marker-4   45M
size marker-5   45M
size master     45M

fetch by ref + checkout ------------------
size v0.11.0    26M
size v0.12.0    11M
size v0.13.0    12M
size marker-1   8,9M
size marker-2   1,1M
size marker-3   856K
size marker-4   856K
size marker-5   1,1M
size master     1,1M

fetch --depth=1 by ref ------------------
size v0.11.0    9,8M
size v0.12.0    11M
size v0.13.0    13M
size marker-1   13M
size marker-2   13M
size marker-3   13M
size marker-4   13M
size marker-5   13M
size master     13M

fetch --depth=1 by ref + checkout ------------------
size v0.11.0    9,8M
size v0.12.0    3,8M
size v0.13.0    5,6M
size marker-1   4,1M
size marker-2   4,1M
size marker-3   4,1M
size marker-4   4,1M
size marker-5   4,1M
size master     4,1M

fetch older all and master with --depth=1 by ref + checkout
------------------
size master     45M
size master     45M

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: test-fetch.sh --]
[-- Type: application/x-shellscript, Size: 2590 bytes --]

Re: Random idea about speeding up guix pull

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 04.09.2017 um 23:56 schrieb Ludovic Courtès:
>> What it does do is maintain a cached checkout in ~/.cache/guix/pull,
>> which makes subsequent pulls much faster.
>
> Summary ( TL;DR):
>
>   * "guix pull" should use "git fetch master"
>   * "guix download" we can keep the current behaviour
>
> I did a series of tests
>
>   * - "fetch" without any argument will fetch *all* data from *all*
>     branches.
>   * - "fetch master" only fetches data living on "master", other
>     branches are ignored
>
> I compared the data fetched for a repo with status of 6bd1c41e8
> (yesterday 05:29):
>
>   * - "fetch" fetches 1000K
>   * - "fetch master" fetches 755K
>   * - "fetch --depth=1 master" fetches 588K (but see below)

Thanks for the detailed analysis!

The problem is that libgit2 doesn’t support shallow clones, and it’s
unclear whether it will support it in the future:

  https://github.com/libgit2/libgit2/issues/3058

:-/

Ludo’.

Re: Random idea about speeding up guix pull

[Hartmut Goebel]

Am 05.09.2017 um 16:33 schrieb Ludovic Courtès:
> The problem is that libgit2 doesn’t support shallow clones, and it’s
> unclear whether it will support it in the future:

Maybe I'm wrong, but to my understanding fetching a single branch/tag is
not a "shallow clone", isn't it?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Random idea about speeding up guix pull

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 05.09.2017 um 16:33 schrieb Ludovic Courtès:
>> The problem is that libgit2 doesn’t support shallow clones, and it’s
>> unclear whether it will support it in the future:
>
> Maybe I'm wrong, but to my understanding fetching a single branch/tag is
> not a "shallow clone", isn't it?

I think it is, in the sense that just a subset of the Git object graph
is fetched, but I’m not 100% sure about the terminology.

Ludo’.