SSH in git-fetch

SSH in git-fetch

[Lo Peter]

Dear all,

I am experimenting with writing a package definition for an example R
package (https://github.com/jennybc/foofactors) in a PRIVATE channel,
where the R source is also at a PRIVATE github repository.
While the private channel works with proper setup of SSH key (that
needs to be in PEM format due to a bug in libssh2), I am having
trouble with the private git repository for package source.

Is it that git-fetch does not support fetching over SSH?

The package definition I have is:

(define-module (my-packages r-pkgs)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix git-download)
  #:use-module (guix utils)
  #:use-module (guix build-system r)
  #:use-module (gnu packages)
  #:use-module (gnu packages statistics))

(define-public r-foofactors
  (let ((commit "ef71e8d2e82fa80e0cfc249fd42f50c01924326d")
        (revision "1"))
    (package
      (name "r-foofactors")
      (version (git-version "0.0.0.9000" revision commit))
      (source
       (origin
         (method git-fetch)
         (uri (git-reference
               (url "git@github.com:peterloleungyau/foofactors.git")
               (commit commit)))
         (file-name (git-file-name name version))
         (sha256
          (base32
           "1hmfwac2zdl8x6r21yy5b257c4891106ana4j81hfn6rd0rl9f72"))))
      (build-system r-build-system)
      (propagated-inputs
       `(("r-forcats" ,r-forcats)))
      (home-page "https://github.com/jennybc/foofactors")
      (synopsis "A R package to make factors less aggravating.")
      (description
       "Factors have driven people to extreme measures, like ordering
custom conference ribbons and laptop stickers to express how HELLNO we
feel about stringsAsFactors. And yet, sometimes you need them. Can they
be made less maddening? Let's find out.")
      (license license:expat))))


Note that I have cloned the original public repository to my private
repository (for testing purpose). I have added the package definition
to my local ~/to_keep/projects/my-guix-pkgs/my-packages/r-pkgs.scm

Then I test building it locally with

$ guix build -L ~/to_keep/projects/my-guix-pkgs/ r-foofactors
The following derivations will be built:
   /gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-1.ef71e8d.drv
   /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv
building /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to
`/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
hint: Using 'master' as the name for the initial branch. This default
branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint:   git branch -m <name>
Initialized empty Git repository in
/gnu/store/7i9py1b47lsg2d2wqjk68ha04rv2l89i-r-foofactors-0.0.0.9000-1.ef71e8d-checkout/.git/
error: cannot run ssh: No such file or directory
fatal: unable to fork
Failed to do a shallow fetch; retrying a full fetch...
error: cannot run ssh: No such file or directory
fatal: unable to fork
git-fetch: '/gnu/store/ra24wp6glfzmpx1w6i3471aqcqqdrk96-git-minimal-2.32.0/bin/git
fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guix.gnu.org...
Trying content-addressed mirror at berlin.guix.gnu.org...
Trying to download from Software Heritage...
builder for `/gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv'
failed to produce output path `/gnu/store/7i9py1b47lsg2d2wqjk68ha04r
build of /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv
failed
View build log at
'/var/log/guix/drvs/2l/35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv.bz2'.
cannot build derivation
`/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-1.ef71e8d.drv':
1 dependencies couldn't be built
guix build: error: build of
`/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-1.ef71e8d.drv'
failed


Why is there an error of "cannot run ssh"? Is it that git-fetch does
not support fetching over SSH?

I would like to prompt the use of Guix for per-project management in
my small team of data scientists, so we would need a private channel
for a few internal R packages. The above problem is a real blocker.
Any help is greatly appreciated.

Regards,
Peter Lo

Re: SSH in git-fetch

[Leo Prikler]

Hello Peter,

Am Dienstag, den 13.07.2021, 22:38 +0800 schrieb Lo Peter:
> Dear all,
> 
> I am experimenting with writing a package definition for an example R
> package (https://github.com/jennybc/foofactors) in a PRIVATE channel,
> where the R source is also at a PRIVATE github repository.
> While the private channel works with proper setup of SSH key (that
> needs to be in PEM format due to a bug in libssh2), I am having
> trouble with the private git repository for package source.
> 
> Is it that git-fetch does not support fetching over SSH?
Indeed, git-fetch does not support fetching over SSH.

> The package definition I have is:
> 
> (define-module (my-packages r-pkgs)
>   #:use-module ((guix licenses) #:prefix license:)
>   #:use-module (guix packages)
>   #:use-module (guix download)
>   #:use-module (guix git-download)
>   #:use-module (guix utils)
>   #:use-module (guix build-system r)
>   #:use-module (gnu packages)
>   #:use-module (gnu packages statistics))
> 
> (define-public r-foofactors
>   (let ((commit "ef71e8d2e82fa80e0cfc249fd42f50c01924326d")
>         (revision "1"))
>     (package
>       (name "r-foofactors")
>       (version (git-version "0.0.0.9000" revision commit))
>       (source
>        (origin
>          (method git-fetch)
>          (uri (git-reference
>                (url "git@github.com:peterloleungyau/foofactors.git")
>                (commit commit)))
>          (file-name (git-file-name name version))
>          (sha256
>           (base32
>            "1hmfwac2zdl8x6r21yy5b257c4891106ana4j81hfn6rd0rl9f72"))))
>       (build-system r-build-system)
>       (propagated-inputs
>        `(("r-forcats" ,r-forcats)))
>       (home-page "https://github.com/jennybc/foofactors")
>       (synopsis "A R package to make factors less aggravating.")
>       (description
>        "Factors have driven people to extreme measures, like ordering
> custom conference ribbons and laptop stickers to express how HELLNO
> we
> feel about stringsAsFactors. And yet, sometimes you need them. Can
> they
> be made less maddening? Let's find out.")
>       (license license:expat))))
> 
> 
> Note that I have cloned the original public repository to my private
> repository (for testing purpose). I have added the package definition
> to my local ~/to_keep/projects/my-guix-pkgs/my-packages/r-pkgs.scm
> 
> Then I test building it locally with
> 
> $ guix build -L ~/to_keep/projects/my-guix-pkgs/ r-foofactors
> The following derivations will be built:
>    /gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-
> 0.0.0.9000-1.ef71e8d.drv
>    /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> 0.0.0.9000-1.ef71e8d-checkout.drv
> building /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> 0.0.0.9000-1.ef71e8d-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to
> `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-
> 1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
> hint: Using 'master' as the name for the initial branch. This default
> branch name
> hint: is subject to change. To configure the initial branch name to
> use in all
> hint: of your new repositories, which will suppress this warning,
> call:
> hint:
> hint:   git config --global init.defaultBranch <name>
> hint:
> hint: Names commonly chosen instead of 'master' are 'main', 'trunk'
> and
> hint: 'development'. The just-created branch can be renamed via this
> command:
> hint:
> hint:   git branch -m <name>
> Initialized empty Git repository in
> /gnu/store/7i9py1b47lsg2d2wqjk68ha04rv2l89i-r-foofactors-0.0.0.9000-
> 1.ef71e8d-checkout/.git/
> error: cannot run ssh: No such file or directory
> fatal: unable to fork
> Failed to do a shallow fetch; retrying a full fetch...
> error: cannot run ssh: No such file or directory
> fatal: unable to fork
> git-fetch: '/gnu/store/ra24wp6glfzmpx1w6i3471aqcqqdrk96-git-minimal-
> 2.32.0/bin/git
> fetch origin' failed with exit code 128
> Trying content-addressed mirror at berlin.guix.gnu.org...
> Trying content-addressed mirror at berlin.guix.gnu.org...
> Trying to download from Software Heritage...
> builder for `/gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-
> foofactors-0.0.0.9000-1.ef71e8d-checkout.drv'
> failed to produce output path `/gnu/store/7i9py1b47lsg2d2wqjk68ha04r
> build of /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> 0.0.0.9000-1.ef71e8d-checkout.drv
> failed
> View build log at
> '/var/log/guix/drvs/2l/35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> 0.0.0.9000-1.ef71e8d-checkout.drv.bz2'.
> cannot build derivation
> `/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-
> 1.ef71e8d.drv':
> 1 dependencies couldn't be built
> guix build: error: build of
> `/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-
> 1.ef71e8d.drv'
> failed
> 
> 
> Why is there an error of "cannot run ssh"? Is it that git-fetch does
> not support fetching over SSH?
Yes, git-fetch does not support fetching over SSH.  "Cannot run ssh" is
the error returned because the ssh program is missing at fetch time,
but even if it existed, you'd get a different error, namely one of
lacking keys.  You'd have to set up Guix to authenticate itself as you
for pulling the source and while that is in theory possible, there is a
potential security risk attached to most ways of solving it and no
clear path forward.

Furthermore, such a feature, were it integrated in Guix, is likely only
to be used for nonfree software and thus located closely to such
software itself.

> I would like to prompt the use of Guix for per-project management in
> my small team of data scientists, so we would need a private channel
> for a few internal R packages. The above problem is a real blocker.
> Any help is greatly appreciated.
I don't think this has to necessarily be a blocker.  You can point git-
fetch to file:// URIs, so your channel could have file:///path/to/repo
and it'd work under the assumption that your scientists run git pull on
those repos frequently enough (you could automate that with a script,
perhaps even one written in Guile/a handwritten Guix extension).  If
you have company/university intranet, you could also expose those
internal package over that on a well-known address, that's not
reachable from outside.

Regards,
Leo

Re: SSH in git-fetch

[Luis Felipe]

Hi,

On Tuesday, July 13th, 2021 at 2:38 PM, Lo Peter <peterloleungyau@gmail.com> wrote:

> Dear all,
>
> I am experimenting with writing a package definition for an example R
>
> package (https://github.com/jennybc/foofactors) in a PRIVATE channel,
>
> where the R source is also at a PRIVATE github repository.
>
> While the private channel works with proper setup of SSH key (that
>
> needs to be in PEM format due to a bug in libssh2), I am having
>
> trouble with the private git repository for package source.
>
> Is it that git-fetch does not support fetching over SSH?

For what it's worth, I have a private channel that defines packages of my private projects in GitLab. I haven't used the channel for a while, but SSH authentication worked by using "git-checkout" records instead of "origin" records in package "source" fields. For instance:

(source
 (git-checkout
  (url "git@gitlab.com:luis-felipe/guile-lab.git")
  (commit (string-append "v" version))))

Re: SSH in git-fetch

[Lo Peter]

Dear Leo,

Thanks for the clarification. While I think your suggested workaround
would probably work ok, I think that would offset much of the benefits
of Guix.
My intended use case is to use guix environment together with guix
time-machine for per-project dependency management, so each project
needs only two files (channels file and manifest file) committed to
source control, to reproduce the same set of packages in different
machines (we use multiple Jenkins instances). The private channel is
only for a few internal packages that we cannot share to the world (I
have no power to change this). It would be better that we need not
manually ensure that each machine has the package source updated.

I do hope Guix developers would reconsider supporting fetching over
SSH for git-fetch.
Meanwhile I will use the method suggested by Luis Felipe which I have
tried and can confirm that it works.

Regards,
Peter Lo

On Wed, Jul 14, 2021 at 5:47 AM Leo Prikler
<leo.prikler@student.tugraz.at> wrote:
>
> Hello Peter,
>
> Am Dienstag, den 13.07.2021, 22:38 +0800 schrieb Lo Peter:
> > Dear all,
> >
> > I am experimenting with writing a package definition for an example R
> > package (https://github.com/jennybc/foofactors) in a PRIVATE channel,
> > where the R source is also at a PRIVATE github repository.
> > While the private channel works with proper setup of SSH key (that
> > needs to be in PEM format due to a bug in libssh2), I am having
> > trouble with the private git repository for package source.
> >
> > Is it that git-fetch does not support fetching over SSH?
> Indeed, git-fetch does not support fetching over SSH.
>
> > The package definition I have is:
> >
> > (define-module (my-packages r-pkgs)
> >   #:use-module ((guix licenses) #:prefix license:)
> >   #:use-module (guix packages)
> >   #:use-module (guix download)
> >   #:use-module (guix git-download)
> >   #:use-module (guix utils)
> >   #:use-module (guix build-system r)
> >   #:use-module (gnu packages)
> >   #:use-module (gnu packages statistics))
> >
> > (define-public r-foofactors
> >   (let ((commit "ef71e8d2e82fa80e0cfc249fd42f50c01924326d")
> >         (revision "1"))
> >     (package
> >       (name "r-foofactors")
> >       (version (git-version "0.0.0.9000" revision commit))
> >       (source
> >        (origin
> >          (method git-fetch)
> >          (uri (git-reference
> >                (url "git@github.com:peterloleungyau/foofactors.git")
> >                (commit commit)))
> >          (file-name (git-file-name name version))
> >          (sha256
> >           (base32
> >            "1hmfwac2zdl8x6r21yy5b257c4891106ana4j81hfn6rd0rl9f72"))))
> >       (build-system r-build-system)
> >       (propagated-inputs
> >        `(("r-forcats" ,r-forcats)))
> >       (home-page "https://github.com/jennybc/foofactors")
> >       (synopsis "A R package to make factors less aggravating.")
> >       (description
> >        "Factors have driven people to extreme measures, like ordering
> > custom conference ribbons and laptop stickers to express how HELLNO
> > we
> > feel about stringsAsFactors. And yet, sometimes you need them. Can
> > they
> > be made less maddening? Let's find out.")
> >       (license license:expat))))
> >
> >
> > Note that I have cloned the original public repository to my private
> > repository (for testing purpose). I have added the package definition
> > to my local ~/to_keep/projects/my-guix-pkgs/my-packages/r-pkgs.scm
> >
> > Then I test building it locally with
> >
> > $ guix build -L ~/to_keep/projects/my-guix-pkgs/ r-foofactors
> > The following derivations will be built:
> >    /gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-
> > 0.0.0.9000-1.ef71e8d.drv
> >    /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> > 0.0.0.9000-1.ef71e8d-checkout.drv
> > building /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> > 0.0.0.9000-1.ef71e8d-checkout.drv...
> > guile: warning: failed to install locale
> > environment variable `PATH' set to
> > `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-
> > 1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
> > hint: Using 'master' as the name for the initial branch. This default
> > branch name
> > hint: is subject to change. To configure the initial branch name to
> > use in all
> > hint: of your new repositories, which will suppress this warning,
> > call:
> > hint:
> > hint:   git config --global init.defaultBranch <name>
> > hint:
> > hint: Names commonly chosen instead of 'master' are 'main', 'trunk'
> > and
> > hint: 'development'. The just-created branch can be renamed via this
> > command:
> > hint:
> > hint:   git branch -m <name>
> > Initialized empty Git repository in
> > /gnu/store/7i9py1b47lsg2d2wqjk68ha04rv2l89i-r-foofactors-0.0.0.9000-
> > 1.ef71e8d-checkout/.git/
> > error: cannot run ssh: No such file or directory
> > fatal: unable to fork
> > Failed to do a shallow fetch; retrying a full fetch...
> > error: cannot run ssh: No such file or directory
> > fatal: unable to fork
> > git-fetch: '/gnu/store/ra24wp6glfzmpx1w6i3471aqcqqdrk96-git-minimal-
> > 2.32.0/bin/git
> > fetch origin' failed with exit code 128
> > Trying content-addressed mirror at berlin.guix.gnu.org...
> > Trying content-addressed mirror at berlin.guix.gnu.org...
> > Trying to download from Software Heritage...
> > builder for `/gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-
> > foofactors-0.0.0.9000-1.ef71e8d-checkout.drv'
> > failed to produce output path `/gnu/store/7i9py1b47lsg2d2wqjk68ha04r
> > build of /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> > 0.0.0.9000-1.ef71e8d-checkout.drv
> > failed
> > View build log at
> > '/var/log/guix/drvs/2l/35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-
> > 0.0.0.9000-1.ef71e8d-checkout.drv.bz2'.
> > cannot build derivation
> > `/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-
> > 1.ef71e8d.drv':
> > 1 dependencies couldn't be built
> > guix build: error: build of
> > `/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-
> > 1.ef71e8d.drv'
> > failed
> >
> >
> > Why is there an error of "cannot run ssh"? Is it that git-fetch does
> > not support fetching over SSH?
> Yes, git-fetch does not support fetching over SSH.  "Cannot run ssh" is
> the error returned because the ssh program is missing at fetch time,
> but even if it existed, you'd get a different error, namely one of
> lacking keys.  You'd have to set up Guix to authenticate itself as you
> for pulling the source and while that is in theory possible, there is a
> potential security risk attached to most ways of solving it and no
> clear path forward.
>
> Furthermore, such a feature, were it integrated in Guix, is likely only
> to be used for nonfree software and thus located closely to such
> software itself.
>
> > I would like to prompt the use of Guix for per-project management in
> > my small team of data scientists, so we would need a private channel
> > for a few internal R packages. The above problem is a real blocker.
> > Any help is greatly appreciated.
> I don't think this has to necessarily be a blocker.  You can point git-
> fetch to file:// URIs, so your channel could have file:///path/to/repo
> and it'd work under the assumption that your scientists run git pull on
> those repos frequently enough (you could automate that with a script,
> perhaps even one written in Guile/a handwritten Guix extension).  If
> you have company/university intranet, you could also expose those
> internal package over that on a well-known address, that's not
> reachable from outside.
>
> Regards,
> Leo
>

Re: SSH in git-fetch

[Lo Peter]

Dear Luis,

Thanks for the suggested workaround. I have tried it and can confirm
that it works. Thanks.

Regards,
Peter Lo

On Wed, Jul 14, 2021 at 6:30 AM Luis Felipe
<luis.felipe.la@protonmail.com> wrote:
>
> Hi,
>
> On Tuesday, July 13th, 2021 at 2:38 PM, Lo Peter <peterloleungyau@gmail.com> wrote:
>
> > Dear all,
> >
> > I am experimenting with writing a package definition for an example R
> >
> > package (https://github.com/jennybc/foofactors) in a PRIVATE channel,
> >
> > where the R source is also at a PRIVATE github repository.
> >
> > While the private channel works with proper setup of SSH key (that
> >
> > needs to be in PEM format due to a bug in libssh2), I am having
> >
> > trouble with the private git repository for package source.
> >
> > Is it that git-fetch does not support fetching over SSH?
>
> For what it's worth, I have a private channel that defines packages of my private projects in GitLab. I haven't used the channel for a while, but SSH authentication worked by using "git-checkout" records instead of "origin" records in package "source" fields. For instance:
>
> (source
>  (git-checkout
>   (url "git@gitlab.com:luis-felipe/guile-lab.git")
>   (commit (string-append "v" version))))
>
>