First guix pull is too costly

First guix pull is too costly

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]


Hello,

Yesterday, I created a new user in my system.  Then I tried to install a
package, but found that it's not the latest, although I did sudo guix
pull before sudo guix system reconfigure.  So I tried to do guix pull,
but it seems to clone the whole repository from scratch.  But,
unfortunately, it never succeeds due to my faulty network connection.
It progresses somewhat like 5%, 7%, 20%, and my connection fails, and I
have to start again from the beginning.

Is possible to reuse the guix channel from any other user to do the
first guix pull?  Or maybe a shallow clone?

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: First guix pull is too costly

[bokr]

On +2022-06-28 14:04:52 +0600, Akib Azmain Turja wrote:
> 
> Hello,
> 
> Yesterday, I created a new user in my system.  Then I tried to install a
> package, but found that it's not the latest, although I did sudo guix
> pull before sudo guix system reconfigure.  So I tried to do guix pull,
> but it seems to clone the whole repository from scratch.  But,
> unfortunately, it never succeeds due to my faulty network connection.
> It progresses somewhat like 5%, 7%, 20%, and my connection fails, and I
> have to start again from the beginning.
> 
> Is possible to reuse the guix channel from any other user to do the
> first guix pull?  Or maybe a shallow clone?
> 
> -- 
> Akib Azmain Turja
> 
> This message is signed by me with my GnuPG key.  It's fingerprint is:
> 
>     7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

I'm trying to imagine help for this situation:

Does git have a feature for an entire pull like wget does for a file? E.g.,
    wget -c some_incompletely_downloaded_file

TIA to anyone who knows and tells :)
--
Regards,
Bengt Richter

Re: First guix pull is too costly

[zimoun]

Hi,

On Tue, 28 Jun 2022 at 14:04, Akib Azmain Turja <akib@disroot.org> wrote:

> Is possible to reuse the guix channel from any other user to do the
> first guix pull?  Or maybe a shallow clone?

Assuming alice is user already existing on the machine and bob is the
new user.  Something like should do the job,

--8<---------------cut here---------------start------------->8---
ALICE=/home/alice/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
BOB=/home/BOB/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
mkdir -p $BOB
git clone $ALICE $BOB
--8<---------------cut here---------------end--------------->8---

then adapt the permissions.


Cheers,
simon
 

Re: First guix pull is too costly

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

On Tue, Jun 28, 2022 at 01:23:00PM +0200, zimoun wrote:
> Hi,
> 
> On Tue, 28 Jun 2022 at 14:04, Akib Azmain Turja <akib@disroot.org> wrote:
> 
> > Is possible to reuse the guix channel from any other user to do the
> > first guix pull?  Or maybe a shallow clone?
> 
> Assuming alice is user already existing on the machine and bob is the
> new user.  Something like should do the job,
> 
> --8<---------------cut here---------------start------------->8---
> ALICE=/home/alice/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> BOB=/home/BOB/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> mkdir -p $BOB
> git clone $ALICE $BOB
> --8<---------------cut here---------------end--------------->8---
> 
> then adapt the permissions.
> 

There's already a copy of the guix source code at
/run/current-system/profile/share/guile/site/3.0/guix, what if we "just"
switched the new-user/new-cache experience to:

cp -r /run/current-system/profile/share/guile/site/3.0/guix/*
~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
cd ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
git init
git remote add origin https://git.savannah.gnu.org/git/guix.git
git fetch
git branch --set-upstream-to=origin/master

OK, that actually doesn't work, I just tried it in a temp folder.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: First guix pull is too costly

[zimoun]

Hi Efraim,

On Tue, 28 Jun 2022 at 15:28, Efraim Flashner <efraim@flashner.co.il> wrote:

> There's already a copy of the guix source code at
> /run/current-system/profile/share/guile/site/3.0/guix, what if we "just"
> switched the new-user/new-cache experience to:

This is only for Guix System and not on foreign distro, no?


> cp -r /run/current-system/profile/share/guile/site/3.0/guix/*
> ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> cd ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> git init
> git remote add origin https://git.savannah.gnu.org/git/guix.git
> git fetch
> git branch --set-upstream-to=origin/master

Depending on the network, the first “guix pull” is often considered as
slow [1]; especially running the command for each new user: each time,
it is a full Git clone in ~/.cache/guix/checkouts of ~400MB, if I read
correctly.

Instead of cloning each time from Savannah, maybe the clone could happen
from the cache of ’root’ and only fetch from Savannah for populate by
recent commit the cache of the regular user.  It would save some network
traffic.


1: <https://yhetil.org/guix/87ft403kol.fsf@ambrevar.xyz>


Cheers,
simon

Re: First guix pull is too costly

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Instead of cloning each time from Savannah, maybe the clone could happen
> from the cache of ’root’ and only fetch from Savannah for populate by
> recent commit the cache of the regular user.  It would save some network
> traffic.

I was thinking we could have a “Git checkout cache daemon”, so that
things currently in ~/.cache/guix/checkouts could somehow be shared
safely among users.

(That doesn’t help with the first ‘guix pull’ though.)

Ludo’.

Re: First guix pull is too costly

[zimoun]

Hi,

On ven., 01 juil. 2022 at 14:51, Ludovic Courtès <ludo@gnu.org> wrote:

> I was thinking we could have a “Git checkout cache daemon”, so that
> things currently in ~/.cache/guix/checkouts could somehow be shared
> safely among users.

Yeah, maybe Git worktree could be used.  Aide the optimization of the
first “guix pull”, the current design is not totally safe when “guix
time-machine” is intensively run:

        The issue is because concurrency.  If two time-machines are run
        concurrently, they both update ~/.cache/guix/checkouts/ and the
        end result is hard to predict.

        <https://yhetil.org/guix/86r18k1ylf.fsf@gmail.com>


> (That doesn’t help with the first ‘guix pull’ though.)

Yeah, but can we do better than mirroring Savannah and maintain a list
of supported mirrors?


Cheers,
simon

Re: First guix pull is too costly

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Tue, 28 Jun 2022 at 14:04, Akib Azmain Turja <akib@disroot.org> wrote:
>
>> Is possible to reuse the guix channel from any other user to do the
>> first guix pull?  Or maybe a shallow clone?
>
> Assuming alice is user already existing on the machine and bob is the
> new user.  Something like should do the job,
>
> --8<---------------cut here---------------start------------->8---
> ALICE=/home/alice/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> BOB=/home/BOB/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> mkdir -p $BOB
> git clone $ALICE $BOB
> --8<---------------cut here---------------end--------------->8---
>
> then adapt the permissions.
>
>
> Cheers,
> simon
>

It worked!  Thanks a lot.

Just out of curiosity, why the hash is
pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq?  Is this the hash
of the channel declaration/definition/whatsoever?

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: First guix pull is too costly

[zimoun]

Hi,

On Tue, 28 Jun 2022 at 18:49, Akib Azmain Turja <akib@disroot.org> wrote:

> Just out of curiosity, why the hash is
> pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq?  Is this the hash
> of the channel declaration/definition/whatsoever?

--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.8
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(rnrs bytevectors)
scheme@(guix-user)> ,use(gcrypt hash)
scheme@(guix-user)> (define url "https://git.savannah.gnu.org/git/guix.git")
scheme@(guix-user)> (bytevector->base32-string (sha256 (string->utf8 url)))
$1 = "pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq"
--8<---------------cut here---------------end--------------->8---

From ’url-cache-directory’ in (guix git) [1].


1: <https://git.savannah.gnu.org/cgit/guix.git/tree/guix/git.scm#n125>

Cheers,
simon

Re: First guix pull is too costly

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1412 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Tue, 28 Jun 2022 at 18:49, Akib Azmain Turja <akib@disroot.org> wrote:
>
>> Just out of curiosity, why the hash is
>> pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq?  Is this the hash
>> of the channel declaration/definition/whatsoever?
>
> --8<---------------cut here---------------start------------->8---
> $ guix repl
> GNU Guile 3.0.8
> Copyright (C) 1995-2021 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.
> scheme@(guix-user)> ,use(rnrs bytevectors)
> scheme@(guix-user)> ,use(gcrypt hash)
> scheme@(guix-user)> (define url "https://git.savannah.gnu.org/git/guix.git")
> scheme@(guix-user)> (bytevector->base32-string (sha256 (string->utf8 url)))
> $1 = "pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq"
> --8<---------------cut here---------------end--------------->8---
>
> From ’url-cache-directory’ in (guix git) [1].
>
>
> 1: <https://git.savannah.gnu.org/cgit/guix.git/tree/guix/git.scm#n125>
>
> Cheers,
> simon

Thanks!

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: First guix pull is too costly

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Assuming alice is user already existing on the machine and bob is the
> new user.  Something like should do the job,
>
> ALICE=/home/alice/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> BOB=/home/BOB/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> mkdir -p $BOB
> git clone $ALICE $BOB

Does ‘git clone’ do a better job than libgit2/Guile-Git (which is what
‘guix pull’ uses) in case it gets disconnected while fetching the repo?
Is it able to resume when libgit2 isn’t?

(I’d be surprised, but I don’t know.)

Ludo’.

Re: First guix pull is too costly

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2879 bytes --]

Hello Ludo’ and Simon,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

> Does ‘git clone’ do a better job than libgit2/Guile-Git (which is what
> ‘guix pull’ uses) in case it gets disconnected while fetching the repo?
> Is it able to resume when libgit2 isn’t?
>
> (I’d be surprised, but I don’t know.)

AFAIU `git clone` is still not able to resume after a failed connection,
there is a SoC2008 proposal [1] but it seems it was never implemented.

Thanks to this stackoverflow thread:
https://stackoverflow.com/questions/3954852/how-to-complete-a-git-clone-for-a-big-project-on-an-unstable-connection

it seems there are two available workarounds to this problem:

1. Use shallow clone:

git clone --depth=1
git fetch --unshallow

if fetch fails unshallowing due to connection problems, AFAIU the next
"unshallow" will continue fetching more objects until it's done

2. git-bundle [2] some tagged release

The bundle itself is an ordinary file, which you can download any way,
via HTTP/FTP with resume support, via BitTorrent, via rsync, etc.  You
can create clone from a bundle, fix configuration [3], and do further
fetches from a proper git repository.

A Guix repo git bundle for every new release, starting from 1.3.0
(1.4?), could be useful for people with slow or unreliable internet
connection.

Could `guix pull` be extended to automatically do this for the user?

I mean: for each channell, `guix pull` could test if the
"channel-bundle" (see below) is available and download it with a
resumable method, then use the git-bundle file to create the clone and
finally fix the remote url in the git configuration.

To define the "channel-bundle" we could add a "bundle" sexp (gexp?) to
the channels.scm file, like this:

--8<---------------cut here---------------start------------->8---

(list (channel
        (name 'guix)
        (bundle "https://git.savannah.gnu.org/guix-1.3.0.bundle")
        (url "https://git.savannah.gnu.org/git/guix.git")
        (commit
          "a3d86b341d361530127c36fbfbf77d638df2c1de")
        (introduction
          (make-channel-introduction
            "9edb3f66fd807b096b48283debdcddccfea34bad"
            (openpgp-fingerprint
              "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))

--8<---------------cut here---------------end--------------->8---

WDYT?

Ciao, Gio'

P.S.: I'm sorry I still cannot help with proper patches to `guix pull`,
I know it's too easy to just describe ideas than to implement it in
code!



[1] https://git.wiki.kernel.org/index.php/SoC2008Ideas#Restartable_Clone

[2] https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-bundle.html

[3] that means fix the url in the [remote "origin"]; we can easily
document it in our manual

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

First guix pull is too costly

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]


Hello,

Yesterday, I created a new user in my system.  Then I tried to install a
package, but found that it's not the latest, although I did sudo guix
pull before sudo guix system reconfigure.  So I tried to do guix pull,
but it seems to clone the whole repository from scratch.  But,
unfortunately, it never succeeds due to my faulty network connection.
It progresses somewhat like 5%, 7%, 20%, and my connection fails, and I
have to start again from the beginning.

Is possible to reuse the guix channel from any other user to do the
first guix pull?  Or maybe a shallow clone?

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: First guix pull is too costly

[zimoun]

Hi,

Could you avoid to cross-post the exact same message on different lists.

On Tue, 28 Jun 2022 at 15:05, Akib Azmain Turja <akib@disroot.org> wrote:
> Hello,
>
> Yesterday, I created a new user in my system.  Then I tried to install a
> package, but found that it's not the latest, although I did sudo guix
> pull before sudo guix system reconfigure.  So I tried to do guix pull,
> but it seems to clone the whole repository from scratch.  But,
> unfortunately, it never succeeds due to my faulty network connection.
> It progresses somewhat like 5%, 7%, 20%, and my connection fails, and I
> have to start again from the beginning.
>
> Is possible to reuse the guix channel from any other user to do the
> first guix pull?  Or maybe a shallow clone?

For instance, this exact same message had been sent on guix-devel:

        From: Akib Azmain Turja <akib@disroot.org>
        Subject: First guix pull is too costly
        To: guix-devel@gnu.org
        Date: Tue, 28 Jun 2022 14:04:52 +0600

    https://yhetil.org/guix/87wnd1kycb.fsf@disroot.org


All the best,
simon