bug#70339: Constructing hg-fetch fixed-output derivation requires Mercurial

bug#70339: Constructing hg-fetch fixed-output derivation requires Mercurial

[Simon Tournier]

Hi,

For instance,

--8<---------------cut here---------------start------------->8---
$ guix build -S -d hg-commitsigs
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
3,7 MB will be downloaded:
  /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
 mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%

/gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
--8<---------------cut here---------------end--------------->8---

and it is unexpected.  The construction of the fixed-output derivation
does not need to download stuff; it only needs to compose stuff
detailing how to do.  Any download (or build) must happen when running
the derivation itself.

The issue: later – say 1 or 2 years from now – the command:

   guix time-machone --commit=929ddec -- build -S -d hg-commitsigs

will start to build all what Mercurial needs (python etc.).  If for some
reasons*, only one of Mercurial dependencies fails then we are doomed.

    ( Context: I am working on a fixed-output translator; rely on
      current strategies for downloading and do not require all the past
      stack just for downloading source code. )

I think it comes from this part:

--8<---------------cut here---------------start------------->8---
                     (hg-fetch '#$(hg-reference-url ref)
                               '#$(hg-reference-changeset ref)
                               #$output
                               #:hg-command (string-append #+hg "/bin/hg")))
--8<---------------cut here---------------end--------------->8---

from ’hg-fetch’ in (guix hg-download).  Here the #+hg is not required
because just before there is:

            (set-path-environment-variable "PATH" '("bin")
                                           (match '#+inputs
                                             (((names dirs outputs ...) ...)
                                              dirs)))

So relying on string "hg" should be enough; as it is done in
’git-fetch/in-band*’ for one example.

Do I miss something?


Cheers,
simon

*reasons of failures: See
 https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment

bug#70339: Constructing hg-fetch fixed-output derivation requires Mercurial

[Ludovic Courtès]

Hello!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> $ guix build -S -d hg-commitsigs
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 3,7 MB will be downloaded:
>   /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
>  mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%
>
> /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>
>
> and it is unexpected.

That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
me.  :-)

> I think it comes from this part:
>
>                      (hg-fetch '#$(hg-reference-url ref)
>                                '#$(hg-reference-changeset ref)
>                                #$output
>                                #:hg-command (string-append #+hg "/bin/hg")))
>
> from ’hg-fetch’ in (guix hg-download).  Here the #+hg is not required
> because just before there is:
>
>             (set-path-environment-variable "PATH" '("bin")
>                                            (match '#+inputs
>                                              (((names dirs outputs ...) ...)
>                                               dirs)))

Maybe, but one way or another, Mercurial is necessary.

Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
could create a different fixed-output derivation producing the same
result but without using Mercurial; typically, the builder of that
derivation would download from SWH.

Does that make sense?

HTH,
Ludo’.

bug#70339: Constructing hg-fetch fixed-output derivation requires Mercurial

[Simon Tournier]

Hi Ludo,

On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> > $ guix build -S -d hg-commitsigs
> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> > 3,7 MB will be downloaded:
> >   /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
> >  mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%
> >
> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
> >
> > and it is unexpected.
>
> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
> me.  :-)

There is a misunderstanding, I guess.

Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)

However, just ask what it will run (please note the dash d in guix
build -S -d hg-commitsigs) must not require to have a local copy of
Mercurial (binary).  If you still think yes, why is it not the case
for fixed-output derivations relying on the old Git builder?

> > I think it comes from this part:
> >
> >                      (hg-fetch '#$(hg-reference-url ref)
> >                                '#$(hg-reference-changeset ref)
> >                                #$output
> >                                #:hg-command (string-append #+hg "/bin/hg")))
> >
> > from ’hg-fetch’ in (guix hg-download).  Here the #+hg is not required
> > because just before there is:
[...]
> Maybe, but one way or another, Mercurial is necessary.

Again, I think it is a bug from #+hg instead of plain "hg".  Having a
local copy of Mercurial (binary) must not be required to just display
the fixed-output derivation.  For running this fixed-output
derivation, yes for sure.

> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
> could create a different fixed-output derivation producing the same
> result but without using Mercurial; typically, the builder of that
> derivation would download from SWH.
>
> Does that make sense?

Yes, it makes sense; see my very first attempt in [1] :-).

But you cannot apply this strategy for fixed-output derivations
relying on Mercurial.  You need first to build Mercurial (and thus all
the Python stack) just to display the fixed-output derivation.  Then,
yes once you have this fixed-output derivation, it is possible to
manipulate it for getting another one.

This report is about #+hg that needs to be fixed for the future.
And because of that, the strategy above for fixed-output derivations
relying on Mercurial is doomed for the past, IMHO.  Except if you have
an idea. ;-)

Cheers,
simon


1: https://gitlab.com/zimoun/guix-drv

bug#70339: Constructing hg-fetch fixed-output derivation requires Mercurial

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Hi Ludo,
>
> On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> > $ guix build -S -d hg-commitsigs
>> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> > 3,7 MB will be downloaded:
>> >   /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
>> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
>> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
>> >  mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%
>> >
>> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>> >
>> > and it is unexpected.
>>
>> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
>> me.  :-)
>
> There is a misunderstanding, I guess.
>
> Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)
>
> However, just ask what it will run (please note the dash d in guix
> build -S -d hg-commitsigs) must not require to have a local copy of
> Mercurial (binary).  If you still think yes, why is it not the case
> for fixed-output derivations relying on the old Git builder?

Oh sorry, I had missed the ‘-d’ bit.

In this case, what’s happening is grafts: Guix downloads (or builds)
Mercurial so it can compute its grafted derivation.

>> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
>> could create a different fixed-output derivation producing the same
>> result but without using Mercurial; typically, the builder of that
>> derivation would download from SWH.
>>
>> Does that make sense?
>
> Yes, it makes sense; see my very first attempt in [1] :-).

[...]

> 1: https://gitlab.com/zimoun/guix-drv

Nice!

Thanks,
Ludo’.

bug#70339: Constructing hg-fetch fixed-output derivation requires Mercurial

[Simon Tournier]

Hi,

On ven., 12 avril 2024 at 16:05, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> In this case, what’s happening is grafts: Guix downloads (or builds)
> Mercurial so it can compute its grafted derivation.

Ah indeed!  Damned. :-)

Closing.

Cheers,
simon