bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Chris Marusich]

Hi Guix!

Sometimes, a Git repository may only be available via an authenticated
SSH connection.  Even in the case of repositories that only contain
free software, this situation can arise for administrative or
compliance-related reasons.  How can one define a package in such a
situation?

This patch adds a new origin method, git-fetch/impure, which solves
that problem.  Specifically, git-fetch/impure creates a fixed-output
derivation that fetches the Git repository outside of a derivation, in
the environment of the invoking user.  In particular, this enables SSH
to communicate with the user's SSH agent, which in turn allows Git to
fetch the repository over an authenticated SSH connection.  In
addition, because it is a fixed-output derivation, the output of a
successful git-fetch/impure is guaranteed to be identical to the
output of a pure git-fetch for any given commit.

Here's a simple example:

(define-public guix-over-ssh
  (package
    (inherit guix)
    (name "guix-over-ssh")
    (source
     (origin
       (inherit (package-source guix))
       (method git-fetch/impure)
       (uri
        (git-reference
         (inherit (origin-uri (package-source guix)))
         (url "ssh://marusich@git.sv.gnu.org:/srv/git/guix.git")))))))

In this particular example, my username appears in the package
definition, but there is no reason why that has to be so.  In many
systems, it is possible to grant access to multiple users with
different SSH keys under a single shared user name.  And in other
systems, an automated build system might need to fetch sources using
its own unique system user name and SSH key.

All in all, I think this is pretty useful.  It enables developers to
define packages in environments where authenticated access to Git
repositories is required.  Please let me know what you think!

Chris Marusich (1):
  guix: Add git-fetch/impure.

 doc/guix.texi         |  24 +++++++
 guix/git-download.scm | 150 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 174 insertions(+)

-- 
2.17.0

bug#31284: Status: [PATCH 0/1] guix: Add git-fetch/impure.

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 143 bytes --]

Hi,

Please ignore this bug report.  I intended to submit it to
guix-patches@gnu.org, but I entered the wrong email address.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Thompson, David]

Hi Chris,

On Fri, Apr 27, 2018 at 4:12 AM, Chris Marusich <cmmarusich@gmail.com> wrote:
> Hi Guix!
>
> Sometimes, a Git repository may only be available via an authenticated
> SSH connection.  Even in the case of repositories that only contain
> free software, this situation can arise for administrative or
> compliance-related reasons.  How can one define a package in such a
> situation?
>
> This patch adds a new origin method, git-fetch/impure, which solves
> that problem.  Specifically, git-fetch/impure creates a fixed-output
> derivation that fetches the Git repository outside of a derivation, in
> the environment of the invoking user.  In particular, this enables SSH
> to communicate with the user's SSH agent, which in turn allows Git to
> fetch the repository over an authenticated SSH connection.  In
> addition, because it is a fixed-output derivation, the output of a
> successful git-fetch/impure is guaranteed to be identical to the
> output of a pure git-fetch for any given commit.

In general I'm opposed to including such things in Guix. When I
encountered a similar situation at work I opted to use https cloning
with a password token in the url. Then there was no external state
(like an rsa key) needed.

I understand the motivation behind asking for something like this, but
I don't think Guix proper should include it.

- Dave

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Mark H Weaver]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> writes:

> Sometimes, a Git repository may only be available via an authenticated
> SSH connection.  Even in the case of repositories that only contain
> free software, this situation can arise for administrative or
> compliance-related reasons.  How can one define a package in such a
> situation?
>
> This patch adds a new origin method, git-fetch/impure, which solves
> that problem.  Specifically, git-fetch/impure creates a fixed-output
> derivation that fetches the Git repository outside of a derivation, in
> the environment of the invoking user.  In particular, this enables SSH
> to communicate with the user's SSH agent, which in turn allows Git to
> fetch the repository over an authenticated SSH connection.

I agree with David that I'd prefer not to have something like this in
Guix.  However, I can offer another method that I've sometimes used to
build a package based on a git checkout that's not publicly available on
the network.

My hacky approach has been to manually add a tarball of the desired
sources using "guix download file:///home/mhw/foo.tar.gz" and then to
add a bogus origin but with the correct hash.  If a file with a matching
hash is already in the store, then it will be used, and the other fields
of the 'origin' will effectively be ignored.

      Mark

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1697 bytes --]

Hi David and Mark,

Thank you both for taking a look at this!

"Thompson, David" <dthompson2@worcester.edu> writes:

> When I encountered a similar situation at work I opted to use https
> cloning with a password token in the url. Then there was no external
> state (like an rsa key) needed.

This is good to know!  I hadn't considered putting the secret into the
URL.  I can see how that might be a simple and appropriate solution in
some situations.  However, it would also be nice if Guix could fetch Git
repositories over SSH using public key authentication.  In some
situations, SSH public key authentication may be the only option.

Mark H Weaver <mhw@netris.org> writes:

> My hacky approach has been to manually add a tarball of the desired
> sources using "guix download file:///home/mhw/foo.tar.gz" and then to
> add a bogus origin but with the correct hash.  If a file with a matching
> hash is already in the store, then it will be used, and the other fields
> of the 'origin' will effectively be ignored.

That's a neat trick!  It looks like it would work well for ad-hoc
hacking.  But how does it scale?  Imagine if you wanted to do this for
10 packages, or 100.  The manual upkeep could become quite painful.  It
would be so much nicer if Guix could just download the source
automatically, as usual!

You've both said that you would prefer not to add git-fetch/impure to
Guix.  Can you help me to understand why you feel that way?  I really
think it would be nice if Guix could fetch Git repositories over SSH
using public key authentication, so I'm hoping that we can talk about it
and figure out an acceptable way to implement it.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Ludovic Courtès]

Hello,

Chris Marusich <cmmarusich@gmail.com> skribis:

> You've both said that you would prefer not to add git-fetch/impure to
> Guix.  Can you help me to understand why you feel that way?  I really
> think it would be nice if Guix could fetch Git repositories over SSH
> using public key authentication, so I'm hoping that we can talk about it
> and figure out an acceptable way to implement it.

One argument against it would be that it encourages people (or at least
makes it very easy) to write origins that depend on external state, and
thus may be non-reproducible by others, and that Guix itself should
provide tools for writing reproducible build definitions.

Ludo’.

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Mark H Weaver]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> writes:

> You've both said that you would prefer not to add git-fetch/impure to
> Guix.  Can you help me to understand why you feel that way?  I really
> think it would be nice if Guix could fetch Git repositories over SSH
> using public key authentication, so I'm hoping that we can talk about it
> and figure out an acceptable way to implement it.

I thought about it some more, and found that I cannot really justify my
position on this, so I hereby drop my objection.  It's obviously not
useful for packages that will be included in Guix itself, which is our
primary focus, but I suppose it could be useful for private package
definitions.

What do you think, David?  It seems to me that password tokens in URLs
raise possible security risks, whereas public-key authentication is
generally better practice.

       Mark

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Mark H Weaver]

Hi Chris,

I don't see your patch anywhere in <https://bugs.gnu.org/31284>.
Can you resend it?

I'm curious how you arrange for a build process launched by guix-daemon
to have access to your 'ssh-agent'.  Although I've dropped my objection
to this idea in principle, I have yet to review the technical details of
how this actually works.  It might be that the details will raise
further alarm bells in my head :)

      Mark

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

Mark H Weaver <mhw@netris.org> writes:

> Hi Chris,
>
> I don't see your patch anywhere in <https://bugs.gnu.org/31284>.
> Can you resend it?
>
> I'm curious how you arrange for a build process launched by guix-daemon
> to have access to your 'ssh-agent'.  Although I've dropped my objection
> to this idea in principle, I have yet to review the technical details of
> how this actually works.  It might be that the details will raise
> further alarm bells in my head :)

The patch can be found here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=31285

I accidentally sent the cover letter to bug-guix@gnu.org instead of
guix-patches@gnu.org, which is why this bogus bug report (31284) was
opened.  I've closed this bug report in favor of the "guix-patches" one.
Sorry for the confusion!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#31284: [PATCH 0/1] guix: Add git-fetch/impure.

[Thompson, David]

On Sun, Apr 29, 2018 at 1:28 PM, Mark H Weaver <mhw@netris.org> wrote:
> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> You've both said that you would prefer not to add git-fetch/impure to
>> Guix.  Can you help me to understand why you feel that way?  I really
>> think it would be nice if Guix could fetch Git repositories over SSH
>> using public key authentication, so I'm hoping that we can talk about it
>> and figure out an acceptable way to implement it.
>
> I thought about it some more, and found that I cannot really justify my
> position on this, so I hereby drop my objection.  It's obviously not
> useful for packages that will be included in Guix itself, which is our
> primary focus, but I suppose it could be useful for private package
> definitions.
>
> What do you think, David?  It seems to me that password tokens in URLs
> raise possible security risks, whereas public-key authentication is
> generally better practice.

If I'm outvoted here then I'm OK with accepting this change. Just to
clarify, I advocate the use of password tokens in URLs for private
repositories only. I do this for non-Guix things as well in order to
improve reproducibility of internal builds.

- Dave