* Fwd: SSH in git-fetch [not found] <CAEM+zmjXVqARy5uPS=Hydp49-pXDUJ1dwgF-B5SLcb13=soz3g@mail.gmail.com> @ 2021-07-13 16:07 ` Lo Peter 2021-07-14 9:18 ` Chris Marusich 0 siblings, 1 reply; 3+ messages in thread From: Lo Peter @ 2021-07-13 16:07 UTC (permalink / raw) To: help-guix ---------- Forwarded message --------- From: Lo Peter <peterloleungyau@gmail.com> Date: Tue, Jul 13, 2021 at 10:38 PM Subject: SSH in git-fetch To: <guix-devel@gnu.org> Dear all, I am experimenting with writing a package definition for an example R package (https://github.com/jennybc/foofactors) in a PRIVATE channel, where the R source is also at a PRIVATE github repository. While the private channel works with proper setup of SSH key (that needs to be in PEM format due to a bug in libssh2), I am having trouble with the private git repository for package source. Is it that git-fetch does not support fetching over SSH? The package definition I have is: (define-module (my-packages r-pkgs) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix packages) #:use-module (guix download) #:use-module (guix git-download) #:use-module (guix utils) #:use-module (guix build-system r) #:use-module (gnu packages) #:use-module (gnu packages statistics)) (define-public r-foofactors (let ((commit "ef71e8d2e82fa80e0cfc249fd42f50c01924326d") (revision "1")) (package (name "r-foofactors") (version (git-version "0.0.0.9000" revision commit)) (source (origin (method git-fetch) (uri (git-reference (url "git@github.com:peterloleungyau/foofactors.git") (commit commit))) (file-name (git-file-name name version)) (sha256 (base32 "1hmfwac2zdl8x6r21yy5b257c4891106ana4j81hfn6rd0rl9f72")))) (build-system r-build-system) (propagated-inputs `(("r-forcats" ,r-forcats))) (home-page "https://github.com/jennybc/foofactors") (synopsis "A R package to make factors less aggravating.") (description "Factors have driven people to extreme measures, like ordering custom conference ribbons and laptop stickers to express how HELLNO we feel about stringsAsFactors. And yet, sometimes you need them. Can they be made less maddening? Let's find out.") (license license:expat)))) Note that I have cloned the original public repository to my private repository (for testing purpose). I have added the package definition to my local ~/to_keep/projects/my-guix-pkgs/my-packages/r-pkgs.scm Then I test building it locally with $ guix build -L ~/to_keep/projects/my-guix-pkgs/ r-foofactors The following derivations will be built: /gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-1.ef71e8d.drv /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv building /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv... guile: warning: failed to install locale environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin' hint: Using 'master' as the name for the initial branch. This default branch name hint: is subject to change. To configure the initial branch name to use in all hint: of your new repositories, which will suppress this warning, call: hint: hint: git config --global init.defaultBranch <name> hint: hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and hint: 'development'. The just-created branch can be renamed via this command: hint: hint: git branch -m <name> Initialized empty Git repository in /gnu/store/7i9py1b47lsg2d2wqjk68ha04rv2l89i-r-foofactors-0.0.0.9000-1.ef71e8d-checkout/.git/ error: cannot run ssh: No such file or directory fatal: unable to fork Failed to do a shallow fetch; retrying a full fetch... error: cannot run ssh: No such file or directory fatal: unable to fork git-fetch: '/gnu/store/ra24wp6glfzmpx1w6i3471aqcqqdrk96-git-minimal-2.32.0/bin/git fetch origin' failed with exit code 128 Trying content-addressed mirror at berlin.guix.gnu.org... Trying content-addressed mirror at berlin.guix.gnu.org... Trying to download from Software Heritage... builder for `/gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv' failed to produce output path `/gnu/store/7i9py1b47lsg2d2wqjk68ha04r build of /gnu/store/2l35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv failed View build log at '/var/log/guix/drvs/2l/35rb0s5fjk1v8vczir6cp6lfmbbf12-r-foofactors-0.0.0.9000-1.ef71e8d-checkout.drv.bz2'. cannot build derivation `/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-1.ef71e8d.drv': 1 dependencies couldn't be built guix build: error: build of `/gnu/store/gnzw26jl9vw3z10cdnpcahd67zlf1ziy-r-foofactors-0.0.0.9000-1.ef71e8d.drv' failed Why is there an error of "cannot run ssh"? Is it that git-fetch does not support fetching over SSH? I would like to prompt the use of Guix for per-project management in my small team of data scientists, so we would need a private channel for a few internal R packages. The above problem is a real blocker. Any help is greatly appreciated. Regards, Peter Lo ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Fwd: SSH in git-fetch 2021-07-13 16:07 ` Fwd: SSH in git-fetch Lo Peter @ 2021-07-14 9:18 ` Chris Marusich 2021-07-14 10:12 ` Lo Peter 0 siblings, 1 reply; 3+ messages in thread From: Chris Marusich @ 2021-07-14 9:18 UTC (permalink / raw) To: Lo Peter; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 5901 bytes --] Hi Peter, Lo Peter <peterloleungyau@gmail.com> writes: > Is it that git-fetch does not support fetching over SSH? As Luis mentioned, for a package, if you use "git-checkout" as the origin and provide a Git SSH URL, it will work. I'm not sure if an equivalent exists for other transmission methods, but I doubt it (e.g., using Mercurial over SSH to check out a Mercurial repository, or using SFTP to fetch a release tarball over the SSH File Transfer Protocol). For channels, it will work if you just provide a Git SSH URL in your channels.scm file. I tested this just now on my own machines, and it definitely works. However, since Guix uses libgit2 under the hood, rather than invoking the "ssh" command, as you noticed there may be subtle differences between how a vanilla "ssh" command works vs. how Guix will handle the SSH connection. In any case, running an SSH agent might prove useful. In both the case of fetching a package via git-checkout and fetching a channel, Guix will attempt to fetch the repository "outside" the build environment, so if you are running an agent, it will try to use it. I'm not sure if Guix will honor your SSH config, though. If you are not wedded to SSH, then another option that may be better for reproducibility is to use HTTPS with the user and password encoded in the URL. This is nice because it doesn't require a user to to configure SSH correctly on their local machine or provision an authorized SSH key just to fetch the channel or the packages. In some situations, embedding a user name and password in the URL in the package definition may be sufficient, but in other situations, it may not meet your security requirements (the URLs will wind up in the store and thus be visible to anyone else logged into the system); it's up to you to decide what's appropriate for your situation. Leo Prikler <leo.prikler@student.tugraz.at> writes: > Yes, git-fetch does not support fetching over SSH. "Cannot run ssh" is > the error returned because the ssh program is missing at fetch time, > but even if it existed, you'd get a different error, namely one of > lacking keys. You'd have to set up Guix to authenticate itself as you > for pulling the source and while that is in theory possible, there is a > potential security risk attached to most ways of solving it and no > clear path forward. > > Furthermore, such a feature, were it integrated in Guix, is likely only > to be used for nonfree software and thus located closely to such > software itself. It sounds like you're suggesting that Guix should avoid making it easy to integrate with existing access control mechanisms because access control is only useful for non-free software. I disagree with that. Access control is useful, and it is often necessary for security or compliance reasons, even in the world of free software. The issue of controlling access to a particular repository of software is orthogonal to the question of whether that software is free software. It is common for someone to want to maintain a local copy of free software and also to control access to that copy. In many organizations, it is a hard requirement that all software is securely stored and can only be accessed by authorized entities. This is true regardless of whether the software is free software. It is also easy to imagine that some people or organizations might prefer or be required to develop all software, even free software, privately within their own secure network. If Guix makes it difficult for developers to use it in an environment where access control is required, Guix is less likely to be used in those environments, and I think that would be a missed opportunity. I don't think it helps the free software movement to encourage the idea that access control is somehow antithetical to free software. It isn't. Even the FSF controls access to its software repositories via SSH. Savannah's projects may be available anonymously via FTP or other means, but the SSH URLs are only usable if you're an FSF associate member. Similarly, great free software like Kallithea exists, which is a tool for hosting source code repositories, restricting access to them, and making it easy to collaborate with others. There are many, many free software projects that integrate well with access control systems, and it does not mean they are somehow not good members of the free software community. In short, the need for access control is not unusual, and it does not go against the spirit of free software. It should be easy to use Guix in environments where access control is required. I'm glad that channels and packages can be used over SSH and that we have a variety of options besides SSH for controlling access, too. However, SSH is a very common way to control access these days, and I think there are opportunities to improve that support in Guix. >> I would like to prompt the use of Guix for per-project management in >> my small team of data scientists, so we would need a private channel >> for a few internal R packages. The above problem is a real blocker. >> Any help is greatly appreciated. > I don't think this has to necessarily be a blocker. You can point git- > fetch to file:// URIs, so your channel could have file:///path/to/repo > and it'd work under the assumption that your scientists run git pull on > those repos frequently enough (you could automate that with a script, > perhaps even one written in Guile/a handwritten Guix extension). If > you have company/university intranet, you could also expose those > internal package over that on a well-known address, that's not > reachable from outside. These solutions can work, but because they require extra steps to implement, they are probably less attractive than using Git over SSH. -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 861 bytes --] ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Fwd: SSH in git-fetch 2021-07-14 9:18 ` Chris Marusich @ 2021-07-14 10:12 ` Lo Peter 0 siblings, 0 replies; 3+ messages in thread From: Lo Peter @ 2021-07-14 10:12 UTC (permalink / raw) To: Chris Marusich; +Cc: help-guix Dear Chris, Thanks for the suggestion. I have previously tested that fetching from private channel works (after a workaround of use a PEM format of SSH key, due to a bug in libssh2). And I have just tested Luis's suggested workaround and can confirm that it works. Although the private channel will be kept only without our organization, I think the security team will frown if there are passwords in the package definitions committed to the repo. So I think meanwhile I will use the workaround suggested by Luis. Regards, Peter Lo On Wed, Jul 14, 2021 at 5:18 PM Chris Marusich <cmmarusich@gmail.com> wrote: > > Hi Peter, > > Lo Peter <peterloleungyau@gmail.com> writes: > > > Is it that git-fetch does not support fetching over SSH? > > As Luis mentioned, for a package, if you use "git-checkout" as the > origin and provide a Git SSH URL, it will work. I'm not sure if an > equivalent exists for other transmission methods, but I doubt it (e.g., > using Mercurial over SSH to check out a Mercurial repository, or using > SFTP to fetch a release tarball over the SSH File Transfer Protocol). > For channels, it will work if you just provide a Git SSH URL in your > channels.scm file. I tested this just now on my own machines, and it > definitely works. > > However, since Guix uses libgit2 under the hood, rather than invoking > the "ssh" command, as you noticed there may be subtle differences > between how a vanilla "ssh" command works vs. how Guix will handle the > SSH connection. In any case, running an SSH agent might prove useful. > In both the case of fetching a package via git-checkout and fetching a > channel, Guix will attempt to fetch the repository "outside" the build > environment, so if you are running an agent, it will try to use it. I'm > not sure if Guix will honor your SSH config, though. > > If you are not wedded to SSH, then another option that may be better for > reproducibility is to use HTTPS with the user and password encoded in > the URL. This is nice because it doesn't require a user to to configure > SSH correctly on their local machine or provision an authorized SSH key > just to fetch the channel or the packages. In some situations, > embedding a user name and password in the URL in the package definition > may be sufficient, but in other situations, it may not meet your > security requirements (the URLs will wind up in the store and thus be > visible to anyone else logged into the system); it's up to you to decide > what's appropriate for your situation. > > Leo Prikler <leo.prikler@student.tugraz.at> writes: > > > Yes, git-fetch does not support fetching over SSH. "Cannot run ssh" is > > the error returned because the ssh program is missing at fetch time, > > but even if it existed, you'd get a different error, namely one of > > lacking keys. You'd have to set up Guix to authenticate itself as you > > for pulling the source and while that is in theory possible, there is a > > potential security risk attached to most ways of solving it and no > > clear path forward. > > > > Furthermore, such a feature, were it integrated in Guix, is likely only > > to be used for nonfree software and thus located closely to such > > software itself. > > It sounds like you're suggesting that Guix should avoid making it easy > to integrate with existing access control mechanisms because access > control is only useful for non-free software. I disagree with that. > Access control is useful, and it is often necessary for security or > compliance reasons, even in the world of free software. > > The issue of controlling access to a particular repository of software > is orthogonal to the question of whether that software is free software. > It is common for someone to want to maintain a local copy of free > software and also to control access to that copy. In many > organizations, it is a hard requirement that all software is securely > stored and can only be accessed by authorized entities. This is true > regardless of whether the software is free software. It is also easy to > imagine that some people or organizations might prefer or be required to > develop all software, even free software, privately within their own > secure network. If Guix makes it difficult for developers to use it in > an environment where access control is required, Guix is less likely to > be used in those environments, and I think that would be a missed > opportunity. > > I don't think it helps the free software movement to encourage the idea > that access control is somehow antithetical to free software. It isn't. > Even the FSF controls access to its software repositories via SSH. > Savannah's projects may be available anonymously via FTP or other means, > but the SSH URLs are only usable if you're an FSF associate member. > Similarly, great free software like Kallithea exists, which is a tool > for hosting source code repositories, restricting access to them, and > making it easy to collaborate with others. There are many, many free > software projects that integrate well with access control systems, and > it does not mean they are somehow not good members of the free software > community. > > In short, the need for access control is not unusual, and it does not go > against the spirit of free software. It should be easy to use Guix in > environments where access control is required. I'm glad that channels > and packages can be used over SSH and that we have a variety of options > besides SSH for controlling access, too. However, SSH is a very common > way to control access these days, and I think there are opportunities to > improve that support in Guix. > > >> I would like to prompt the use of Guix for per-project management in > >> my small team of data scientists, so we would need a private channel > >> for a few internal R packages. The above problem is a real blocker. > >> Any help is greatly appreciated. > > I don't think this has to necessarily be a blocker. You can point git- > > fetch to file:// URIs, so your channel could have file:///path/to/repo > > and it'd work under the assumption that your scientists run git pull on > > those repos frequently enough (you could automate that with a script, > > perhaps even one written in Guile/a handwritten Guix extension). If > > you have company/university intranet, you could also expose those > > internal package over that on a well-known address, that's not > > reachable from outside. > > These solutions can work, but because they require extra steps to > implement, they are probably less attractive than using Git over SSH. > > -- > Chris ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-07-14 10:19 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CAEM+zmjXVqARy5uPS=Hydp49-pXDUJ1dwgF-B5SLcb13=soz3g@mail.gmail.com> 2021-07-13 16:07 ` Fwd: SSH in git-fetch Lo Peter 2021-07-14 9:18 ` Chris Marusich 2021-07-14 10:12 ` Lo Peter
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).