hg-fetch with subrepos

hg-fetch with subrepos

[John Soo]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

Hi guix!

Thanks and please bear with my first ever mailing list post.  I was trying to package coin3d (https://bitbucket.org/Coin3D/coin/wiki/Home) as it is now under a bsd3 license.  The hash of the repo always changes. I think this is due to the .hg files not being recursively deleted for the subrepositories (https://www.mercurial-scm.org/wiki/Subrepository). Does anyone have any insights?

Thanks!

John

[-- Attachment #2: Type: text/html, Size: 751 bytes --]

Re: hg-fetch with subrepos

[Björn Höfling]

[-- Attachment #1.1: Type: text/plain, Size: 1134 bytes --]

On Sat, 1 Dec 2018 11:10:54 -0800
John Soo <jsoo1@asu.edu> wrote:

> Hi guix!
> 
> Thanks and please bear with my first ever mailing list post.  I was
> trying to package coin3d
> (https://bitbucket.org/Coin3D/coin/wiki/Home) as it is now under a
> bsd3 license.  The hash of the repo always changes. I think this is
> due to the .hg files not being recursively deleted for the
> subrepositories (https://www.mercurial-scm.org/wiki/Subrepository).
> Does anyone have any insights?
> 

Hi John,

I'm also packaging coin3d :-)

And I stumbled upon that problem too. Ludovic explained me on IRC: The
problem is the metadata directory ".hg": It contains metadata that is
not fixed. For normal hg-repositories, it will be stripped away, but
not recursively for those with sub-repos.

I have a patch that works. I just wasn't sure if it goes to master or
to staging, as it could affect the java-packages as well.

I'm attaching what I have here, will prepare an official patch tonight
or tomorrow.

Björn

PS: With coin3d, I want to add freecad. If that is your intention too,
we should share resources.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-hg-fetch-Remove-.hg-directories-of-sub-repositories.patch --]
[-- Type: text/x-patch, Size: 1152 bytes --]

From 57167ebf39e3f10c4025cb03893456c7269f98f2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Bj=C3=B6rn=20H=C3=B6fling?=
 <bjoern.hoefling@bjoernhoefling.de>
Date: Fri, 23 Nov 2018 18:38:27 +0100
Subject: [PATCH] hg-fetch: Remove .hg directories of sub-repositories.

* guix/build/hg.scm (hg-fetch): Remove all .hg directories recursively.
---
 guix/build/hg.scm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/guix/build/hg.scm b/guix/build/hg.scm
index ea51eb670..9f493f1d6 100644
--- a/guix/build/hg.scm
+++ b/guix/build/hg.scm
@@ -45,8 +45,10 @@ Mercurial changeset identifier.  Return #t on success, #f otherwise."
   ;; The contents of '.hg' vary as a function of the current
   ;; status of the Mercurial repo.  Since we want a fixed
   ;; output, this directory needs to be taken out.
-  (with-directory-excursion directory
-    (delete-file-recursively ".hg"))
+  ;; Since the '.hg' file is also in sub-modules, we have to
+  ;; search for it in all sub-directories.
+  (for-each delete-file-recursively
+            (find-files directory "^\\.hg$" #:directories? #t))
 
   #t)
 
-- 
2.19.1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: hg-fetch with subrepos

[Ludovic Courtès]

Hello,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> And I stumbled upon that problem too. Ludovic explained me on IRC: The
> problem is the metadata directory ".hg": It contains metadata that is
> not fixed. For normal hg-repositories, it will be stripped away, but
> not recursively for those with sub-repos.
>
> I have a patch that works. I just wasn't sure if it goes to master or
> to staging, as it could affect the java-packages as well.

Such a patch can go to ‘master’: it won’t trigger any rebuild because,
by definition, the content hash of an ‘origin’ is known in advance
(these are “fixed-output derivations.”)

However, we should audit current uses of ‘hg-fetch’ with recursive
sub-repos because there hashes are most likely wrong already.

> I'm attaching what I have here, will prepare an official patch tonight
> or tomorrow.

Awesome.  FWIW this patch already LGTM.  :-)

Thanks,
Ludo’.

Re: hg-fetch with subrepos

[John Soo]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

Thanks!

That patch looks familiar :D Looking forward to it.

John

On Sun, Dec 2, 2018 at 1:59 PM Ludovic Courtès <ludo@gnu.org> wrote:

> Hello,
>
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
>
> > And I stumbled upon that problem too. Ludovic explained me on IRC: The
> > problem is the metadata directory ".hg": It contains metadata that is
> > not fixed. For normal hg-repositories, it will be stripped away, but
> > not recursively for those with sub-repos.
> >
> > I have a patch that works. I just wasn't sure if it goes to master or
> > to staging, as it could affect the java-packages as well.
>
> Such a patch can go to ‘master’: it won’t trigger any rebuild because,
> by definition, the content hash of an ‘origin’ is known in advance
> (these are “fixed-output derivations.”)
>
> However, we should audit current uses of ‘hg-fetch’ with recursive
> sub-repos because there hashes are most likely wrong already.
>
> > I'm attaching what I have here, will prepare an official patch tonight
> > or tomorrow.
>
> Awesome.  FWIW this patch already LGTM.  :-)
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 1634 bytes --]

Re: hg-fetch with subrepos

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

On Sun, 2 Dec 2018 18:38:35 +0000
John Soo <jsoo1@asu.edu> wrote:

> Thanks!
> 
> That patch looks familiar :D Looking forward to it.
> 
> John
> 
> On Sun, Dec 2, 2018 at 1:59 PM Ludovic Courtès <ludo@gnu.org> wrote:
> 
> > Hello,
> >
> > Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
> >  
> > > And I stumbled upon that problem too. Ludovic explained me on
> > > IRC: The problem is the metadata directory ".hg": It contains
> > > metadata that is not fixed. For normal hg-repositories, it will
> > > be stripped away, but not recursively for those with sub-repos.
> > >
> > > I have a patch that works. I just wasn't sure if it goes to
> > > master or to staging, as it could affect the java-packages as
> > > well.  
> >
> > Such a patch can go to ‘master’: it won’t trigger any rebuild
> > because, by definition, the content hash of an ‘origin’ is known in
> > advance (these are “fixed-output derivations.”)
> >
> > However, we should audit current uses of ‘hg-fetch’ with recursive
> > sub-repos because there hashes are most likely wrong already.
> >  
> > > I'm attaching what I have here, will prepare an official patch
> > > tonight or tomorrow.  
> >
> > Awesome.  FWIW this patch already LGTM.  :-)
> >
> > Thanks,
> > Ludo’.
> >  


Thanks John and Ludo for your reviews,

pushed to master as:

b8ca5b3146f94785c4e54ade1f64dc783a9b1a85

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

bug#33712: Some Mercurial checkouts probably have incorrect hashes [was Re: hg-fetch with subrepos]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

On Sun, Dec 02, 2018 at 02:58:52PM +0100, Ludovic Courtès wrote:
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
> > And I stumbled upon that problem too. Ludovic explained me on IRC: The
> > problem is the metadata directory ".hg": It contains metadata that is
> > not fixed. For normal hg-repositories, it will be stripped away, but
> > not recursively for those with sub-repos.

[...]

> However, we should audit current uses of ‘hg-fetch’ with recursive
> sub-repos because there hashes are most likely wrong already.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#33712: Some Mercurial checkouts probably have incorrect hashes

[Leo Famulari]

It's been a while and I think that we will have noticed any incorrect
hashes by now. Closing.