unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
@ 2018-02-12  0:16 Danny Milosavljevic
  2018-02-12 15:09 ` Leo Famulari
  2020-03-22 20:48 ` Leo Famulari
  0 siblings, 2 replies; 12+ messages in thread
From: Danny Milosavljevic @ 2018-02-12  0:16 UTC (permalink / raw)
  To: 30428

git-fetch doesn't allow specifying "--depth 1".

That means the repo clones are needlessly large.

Since in packages we only need one specific commit anyhow why do we fetch
all the other commits?

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-12  0:16 bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use Danny Milosavljevic
@ 2018-02-12 15:09 ` Leo Famulari
  2018-02-12 22:59   ` Danny Milosavljevic
  2020-03-22 20:48 ` Leo Famulari
  1 sibling, 1 reply; 12+ messages in thread
From: Leo Famulari @ 2018-02-12 15:09 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 30428

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On Mon, Feb 12, 2018 at 01:16:41AM +0100, Danny Milosavljevic wrote:
> git-fetch doesn't allow specifying "--depth 1".
> 
> That means the repo clones are needlessly large.
> 
> Since in packages we only need one specific commit anyhow why do we fetch
> all the other commits?

I think it's worth adding, but as an option, because there are Git
server implementations, like JGit, that don't support shallow cloning.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-12 15:09 ` Leo Famulari
@ 2018-02-12 22:59   ` Danny Milosavljevic
  2018-02-13  1:28     ` Leo Famulari
                       ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Danny Milosavljevic @ 2018-02-12 22:59 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 30428

Hi Leo,

On Mon, 12 Feb 2018 10:09:39 -0500
Leo Famulari <leo@famulari.name> wrote:

> I think it's worth adding, but as an option, because there are Git
> server implementations, like JGit, that don't support shallow cloning.

Thanks for that!  I didn't consider that before...

Possible patch (do you know such servers and can test whether they still work?):

diff --git a/guix/build/git.scm b/guix/build/git.scm
index c1af545a7..e54d92be7 100644
--- a/guix/build/git.scm
+++ b/guix/build/git.scm
@@ -37,12 +37,18 @@ recursively.  Return #t on success, #f otherwise."
   ;; in advance anyway.
   (setenv "GIT_SSL_NO_VERIFY" "true")
 
+  (mkdir-p directory)
+
   ;; We cannot use "git clone --recursive" since the following "git checkout"
   ;; effectively removes sub-module checkouts as of Git 2.6.3.
-  (and (zero? (system* git-command "clone" url directory))
+  (and ;(zero? (system* git-command "clone" url directory))
        (with-directory-excursion directory
-         (system* git-command "tag" "-l")
-         (and (zero? (system* git-command "checkout" commit))
+         ;(system* git-command "tag" "-l")
+         (invoke git-command "init")
+         (invoke git-command "remote" "add" "origin" url)
+         (and (or (zero? (system* git-command "fetch" "--depth" "1" "origin" commit))
+                  (zero? (system* git-command "fetch" "origin" commit)))
+              (zero? (system* git-command "checkout" "FETCH_HEAD"))
               (begin
                 (when recursive?
                   ;; Now is the time to fetch sub-modules.

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-12 22:59   ` Danny Milosavljevic
@ 2018-02-13  1:28     ` Leo Famulari
  2018-02-13 14:22     ` Leo Famulari
  2018-02-14 13:58     ` Ludovic Courtès
  2 siblings, 0 replies; 12+ messages in thread
From: Leo Famulari @ 2018-02-13  1:28 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 30428

[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]

I think Google uses JGit for their public facing Git servers, but I'm not sure.

On February 12, 2018 5:59:50 PM EST, Danny Milosavljevic <dannym@scratchpost.org> wrote:
>Hi Leo,
>
>On Mon, 12 Feb 2018 10:09:39 -0500
>Leo Famulari <leo@famulari.name> wrote:
>
>> I think it's worth adding, but as an option, because there are Git
>> server implementations, like JGit, that don't support shallow
>cloning.
>
>Thanks for that!  I didn't consider that before...
>
>Possible patch (do you know such servers and can test whether they
>still work?):
>
>diff --git a/guix/build/git.scm b/guix/build/git.scm
>index c1af545a7..e54d92be7 100644
>--- a/guix/build/git.scm
>+++ b/guix/build/git.scm
>@@ -37,12 +37,18 @@ recursively.  Return #t on success, #f otherwise."
>   ;; in advance anyway.
>   (setenv "GIT_SSL_NO_VERIFY" "true")
> 
>+  (mkdir-p directory)
>+
>;; We cannot use "git clone --recursive" since the following "git
>checkout"
>   ;; effectively removes sub-module checkouts as of Git 2.6.3.
>-  (and (zero? (system* git-command "clone" url directory))
>+  (and ;(zero? (system* git-command "clone" url directory))
>        (with-directory-excursion directory
>-         (system* git-command "tag" "-l")
>-         (and (zero? (system* git-command "checkout" commit))
>+         ;(system* git-command "tag" "-l")
>+         (invoke git-command "init")
>+         (invoke git-command "remote" "add" "origin" url)
>+         (and (or (zero? (system* git-command "fetch" "--depth" "1"
>"origin" commit))
>+                  (zero? (system* git-command "fetch" "origin"
>commit)))
>+              (zero? (system* git-command "checkout" "FETCH_HEAD"))
>               (begin
>                 (when recursive?
>                   ;; Now is the time to fetch sub-modules.

[-- Attachment #2: Type: text/html, Size: 2236 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-12 22:59   ` Danny Milosavljevic
  2018-02-13  1:28     ` Leo Famulari
@ 2018-02-13 14:22     ` Leo Famulari
  2018-02-13 17:08       ` Marius Bakke
  2018-02-13 17:09       ` Marius Bakke
  2018-02-14 13:58     ` Ludovic Courtès
  2 siblings, 2 replies; 12+ messages in thread
From: Leo Famulari @ 2018-02-13 14:22 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 30428

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On Mon, Feb 12, 2018 at 11:59:50PM +0100, Danny Milosavljevic wrote:
> Leo Famulari <leo@famulari.name> wrote:
> 
> > I think it's worth adding, but as an option, because there are Git
> > server implementations, like JGit, that don't support shallow cloning.
> 
> Thanks for that!  I didn't consider that before...
> 
> Possible patch (do you know such servers and can test whether they still work?):

I think that Google's public-facing Git servers use JGit. So, perhaps
try shallow cloning Chromium and see if it works.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-13 14:22     ` Leo Famulari
@ 2018-02-13 17:08       ` Marius Bakke
  2018-02-13 18:08         ` Leo Famulari
  2018-02-13 17:09       ` Marius Bakke
  1 sibling, 1 reply; 12+ messages in thread
From: Marius Bakke @ 2018-02-13 17:08 UTC (permalink / raw)
  To: Leo Famulari, Danny Milosavljevic; +Cc: 30428



On February 13, 2018 3:22:58 PM GMT+01:00, Leo Famulari <leo@famulari.name> wrote:
>On Mon, Feb 12, 2018 at 11:59:50PM +0100, Danny Milosavljevic wrote:
>> Leo Famulari <leo@famulari.name> wrote:
>> 
>> > I think it's worth adding, but as an option, because there are Git
>> > server implementations, like JGit, that don't support shallow
>cloning.
>> 
>> Thanks for that!  I didn't consider that before...
>> 
>> Possible patch (do you know such servers and can test whether they
>still work?):
>
>I think that Google's public-facing Git servers use JGit. So, perhaps
>try shallow cloning Chromium and see if it works.

"libvpx" in Guix uses Chromiums git infrastructure and is fairly small.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-13 14:22     ` Leo Famulari
  2018-02-13 17:08       ` Marius Bakke
@ 2018-02-13 17:09       ` Marius Bakke
  1 sibling, 0 replies; 12+ messages in thread
From: Marius Bakke @ 2018-02-13 17:09 UTC (permalink / raw)
  To: Leo Famulari, Danny Milosavljevic; +Cc: 30428

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]



On February 13, 2018 3:22:58 PM GMT+01:00, Leo Famulari <leo@famulari.name> wrote:
>On Mon, Feb 12, 2018 at 11:59:50PM +0100, Danny Milosavljevic wrote:
>> Leo Famulari <leo@famulari.name> wrote:
>> 
>> > I think it's worth adding, but as an option, because there are Git
>> > server implementations, like JGit, that don't support shallow
>cloning.
>> 
>> Thanks for that!  I didn't consider that before...
>> 
>> Possible patch (do you know such servers and can test whether they
>still work?):
>
>I think that Google's public-facing Git servers use JGit. So, perhaps
>try shallow cloning Chromium and see if it works.

"libvpx" in Guix uses Chromiums git infrastructure and is fairly small.

[-- Attachment #2: Type: text/html, Size: 894 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-13 17:08       ` Marius Bakke
@ 2018-02-13 18:08         ` Leo Famulari
  0 siblings, 0 replies; 12+ messages in thread
From: Leo Famulari @ 2018-02-13 18:08 UTC (permalink / raw)
  To: Marius Bakke; +Cc: 30428

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Tue, Feb 13, 2018 at 06:08:58PM +0100, Marius Bakke wrote:
> 
> 
> On February 13, 2018 3:22:58 PM GMT+01:00, Leo Famulari <leo@famulari.name> wrote:
> >On Mon, Feb 12, 2018 at 11:59:50PM +0100, Danny Milosavljevic wrote:
> >> Leo Famulari <leo@famulari.name> wrote:
> >> 
> >> > I think it's worth adding, but as an option, because there are Git
> >> > server implementations, like JGit, that don't support shallow
> >cloning.
> >> 
> >> Thanks for that!  I didn't consider that before...
> >> 
> >> Possible patch (do you know such servers and can test whether they
> >still work?):
> >
> >I think that Google's public-facing Git servers use JGit. So, perhaps
> >try shallow cloning Chromium and see if it works.
> 
> "libvpx" in Guix uses Chromiums git infrastructure and is fairly small.

Looks like the Chromium and libvpx Git repos both support shallow
cloning now. That's great news for anyone building Chromium!

But it doesn't help find a Git server that doesn't support shallow
cloning.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-12 22:59   ` Danny Milosavljevic
  2018-02-13  1:28     ` Leo Famulari
  2018-02-13 14:22     ` Leo Famulari
@ 2018-02-14 13:58     ` Ludovic Courtès
  2018-02-18 13:38       ` Danny Milosavljevic
  2 siblings, 1 reply; 12+ messages in thread
From: Ludovic Courtès @ 2018-02-14 13:58 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 30428

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Mon, 12 Feb 2018 10:09:39 -0500
> Leo Famulari <leo@famulari.name> wrote:
>
>> I think it's worth adding, but as an option, because there are Git
>> server implementations, like JGit, that don't support shallow cloning.
>
> Thanks for that!  I didn't consider that before...
>
> Possible patch (do you know such servers and can test whether they still work?):

I think it’s a great idea.  FWIW, Andy proposed something along these
lines, but the idea was to use shallow clones for tags only because in
other cases it might not work (?):

  https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00258.html

It got stuck on a minor issue: the patch added a ‘tag’ field to
‘git-reference’, which changed the API, and I suggested making things
slightly differently:

  https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00558.html

Would you like to see if both patches could be merged?

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-14 13:58     ` Ludovic Courtès
@ 2018-02-18 13:38       ` Danny Milosavljevic
  2018-02-18 13:53         ` Ludovic Courtès
  0 siblings, 1 reply; 12+ messages in thread
From: Danny Milosavljevic @ 2018-02-18 13:38 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 30428

Hi Ludo,

On Wed, 14 Feb 2018 14:58:55 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> I think it’s a great idea.  FWIW, Andy proposed something along these
> lines, but the idea was to use shallow clones for tags only because in
> other cases it might not work (?):
> 
>   https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00258.html

Yeah, some git servers either don't support searching by commit at all
or only support it after enabling it manually in the config file
(justification is some kind of privacy thing where accidentially a
private file could have been pushed to a public repo and then
reverted - with the commit hash you'd still get to it).
 
I'd rather not do the more involved patchset at the time being.
I don't understand git all that well.

It's already much nicer just to try the shallow commit checkout and fall back
to the current way if it doesn't work - and it's low risk.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-18 13:38       ` Danny Milosavljevic
@ 2018-02-18 13:53         ` Ludovic Courtès
  0 siblings, 0 replies; 12+ messages in thread
From: Ludovic Courtès @ 2018-02-18 13:53 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 30428

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> It's already much nicer just to try the shallow commit checkout and fall back
> to the current way if it doesn't work - and it's low risk.

Oh right, I hadn’t groked that this is what your patch does.

In that case I’m all for it, it seems to be low-risk indeed.  Perhaps
leave a comment explaining that for some servers “--depth 1” won’t work
when we’re passing a commit and not a tag.

Thank you!

Ludo’.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use
  2018-02-12  0:16 bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use Danny Milosavljevic
  2018-02-12 15:09 ` Leo Famulari
@ 2020-03-22 20:48 ` Leo Famulari
  1 sibling, 0 replies; 12+ messages in thread
From: Leo Famulari @ 2020-03-22 20:48 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: 30428-done

On Mon, Feb 12, 2018 at 01:16:41AM +0100, Danny Milosavljevic wrote:
> git-fetch doesn't allow specifying "--depth 1".
> 
> That means the repo clones are needlessly large.
> 
> Since in packages we only need one specific commit anyhow why do we fetch
> all the other commits?

This was fixed in 329dabe13bf98b899b907b45565434c5140804f5. Closing.

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2020-03-22 20:49 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-02-12  0:16 bug#30428: guix git-fetch doesn't specify "--depth 1" - git clone clones a lot without any use Danny Milosavljevic
2018-02-12 15:09 ` Leo Famulari
2018-02-12 22:59   ` Danny Milosavljevic
2018-02-13  1:28     ` Leo Famulari
2018-02-13 14:22     ` Leo Famulari
2018-02-13 17:08       ` Marius Bakke
2018-02-13 18:08         ` Leo Famulari
2018-02-13 17:09       ` Marius Bakke
2018-02-14 13:58     ` Ludovic Courtès
2018-02-18 13:38       ` Danny Milosavljevic
2018-02-18 13:53         ` Ludovic Courtès
2020-03-22 20:48 ` Leo Famulari

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).