Can't serve a custom channel on a HTTPS repository

Can't serve a custom channel on a HTTPS repository

[EuAndreh]

Hi!

I have the following "channels.scm" file:

  (cons*
   (channel
    (name 'xyz-euandreh)
    (url "git://euandreh.xyz/package-repository.git")
    (branch "main")
    (introduction
     (make-channel-introduction
      "d749e053e6db365069cb9b2ef47a78b06f9e7361"
      (openpgp-fingerprint
       "5BDA E9B8 B2F6 C6BC BB0D  6CE5 81F9 0EC3 CD35 6060"))))
   %default-channels)

And I'm using "guix time-machine" with it:

  $ guix time-machine -C channels.scm -- environment --ad-hoc hello

This works very well this way.  However if I change the Git URL to HTTPS, it
stops working:

  diff --git a/channels.scm b/channels.scm
  --- a/channels.scm
  +++ b/channels.scm
  @@ -1,7 +1,7 @@
   (cons*
    (channel
     (name 'xyz-euandreh)
  -  (url "git://euandreh.xyz/package-repository.git")
  +  (url "https://euandreh.xyz/package-repository.git")
     (branch "main")
     (introduction
      (make-channel-introduction

  $ guix time-machine -C channels.scm -- environment --ad-hoc hello
  guix time-machine: error: Git error: invalid content-type: 'application/octet-stream'

The server is configured to serve over HTTPS and the Git protocol, and I can
successfully clone either way:

  $ cd /tmp/
  $ git clone git://euandreh.xyz/package-repository.git   git
  # works
  $ git clone https://euandreh.xyz/package-repository.git https
  # also works

I see that the "%default-channel-url" [0] Guile variable points to a bare HTTPS
Git repository, and that the "Content-Type" header of those files is also
"application/octet-stream":

  (define %default-channel-url
    ;; URL of the default 'guix' channel.
    "https://git.savannah.gnu.org/git/guix.git")

It seems that I also need some kind of extra HTTPS header configuration.  Is
this an issue with the "guile-git" library?  Does Savannah has any special
treatment of HTTPS repositories, or guile-git requests?  What configuration I'm
missing from the server to be able to serve over HTTPS only?

[0]: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/channels.scm?id=68ce40cad7782e75d665283fbe67b330d9a6fdcd#n175

Re: Can't serve a custom channel on a HTTPS repository

[divoplade]

Hello,

Le lundi 26 juillet 2021 à 14:36 -0300, EuAndreh a écrit :
> Is
> this an issue with the "guile-git" library?

This is a frustrating known issue. Bare git repositories cannot be
cloned with libgit2, which is what guile-git uses.

If guix is installed on your server, maybe you could set up the cgit
service [1]. It works with cgit, because the repositories are a little
more than bare repositories.

[1] 
https://guix.gnu.org/manual/en/html_node/Version-Control-Services.html#Version-Control-Services
, section "Cgit Service"

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

> This is a frustrating known issue. Bare git repositories cannot be
> cloned with libgit2, which is what guile-git uses.

The thing is, the bare HTTPS Git repository from Guix itself does work.
I imagined that if this is an issue with guile-git, there is a server-side
workaround, since the main "guix" channel is served as a bare Git HTTPS
repository [0].

I do have CGit and the Git daemon installed on the server, but I wanted to
*remove* them, and keep just the HTTPS one.  So I'm wondering how Guix itself
can accomplish that, and what I can do to replicate it.

[0]: https://git.savannah.gnu.org/git/guix.git

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

Was anybody else able to configure a Guix channel over a bare Git HTTPS
repository, like the main Guix channel?  How did you do accomplish it?  Was any
extra server configuration required?

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

Is this question more appropriate to the guix-devel mailing list?

Re: Can't serve a custom channel on a HTTPS repository

[Leo Famulari]

On Tue, Jul 27, 2021 at 12:37:43AM +0200, divoplade wrote:
> This is a frustrating known issue. Bare git repositories cannot be
> cloned with libgit2, which is what guile-git uses.

Is there a reference for this issue, like a bug ticket?

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

After searching a bit through the archives, all I found was message [0] of
divoplade themself talking about this issue.

However, isn't the Guix main channel an HTTPS bare repository itself?  Being
so, it may be more an issue with my installation than an issue with Guix or an
underlying Git library.

From the description of Git protocols [1], I understood that the main Guix
channel is served via the "Dumb HTTP" protocol.  I'm just not yet sure how it
does it, and how I can replicate.

So it may be that the a Git repository served via the "Dumb HTTP" protocol can
work as a Guix channel, but I just don't know how yet.  I couldn't identify any
special header or file in the Guix bare repository on Savannah [2], so that's
why I'm stuck at this point.

[0]: https://yhetil.org/guix/db1fe1b8a41fdc3784d4f5156e61682c19628121.camel@divoplade.fr/t/#u
[1]: https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
[2]: https://git.savannah.gnu.org/git/guix.git/

Re: Can't serve a custom channel on a HTTPS repository

[Leo Famulari]

On Sun, Aug 08, 2021 at 02:49:07PM -0300, EuAndreh wrote:
> From the description of Git protocols [1], I understood that the main Guix
> channel is served via the "Dumb HTTP" protocol.  I'm just not yet sure how it
> does it, and how I can replicate.

I've set up a dumb HTTP Git repos, and it really is as simple as the
instructions in [1]. They are slow and inefficient but easy to set up.

> So it may be that the a Git repository served via the "Dumb HTTP" protocol can
> work as a Guix channel, but I just don't know how yet.  I couldn't identify any
> special header or file in the Guix bare repository on Savannah [2], so that's
> why I'm stuck at this point.

For questions about Savannah, try asking in the #savannah channel on
libera.chat. You might have to wait a while but the Savannah admins are
very help. Or try the savannah-hackers-public mailing list.

> [1]: https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

Does your Guix channel work on the Dumb HTTP repository?

Re: Can't serve a custom channel on a HTTPS repository

[Leo Famulari]

On Mon, Aug 09, 2021 at 10:41:19AM -0300, EuAndreh wrote:
> Does your Guix channel work on the Dumb HTTP repository?

I don't use any channels except for the default channel.

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

I will try the savannah-hackers-public mailing list too.

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

FYI, I did send a message to the mailing list.  I'm waiting on an answer,  and
I'll try to make the possible solution more broadly available, if a can end up
with one.

https://lists.gnu.org/r/savannah-hackers-public/2021-08/msg00000.html

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

Final conclusion:

libgit does support bare repositories served via HTTP, but only when using Git's
"Smart HTTP", but not the "Dumb HTTP" [0].

Take Guix's channel served on Savannah:

    $ curl -I https://git.savannah.gnu.org/git/guix.git/info/refs
    HTTP/1.1 200 OK
    Content-Type: text/plain

    $ curl -I https://git.savannah.gnu.org/git/guix.git/info/refs?service=git-upload-pack
    HTTP/1.1 200 OK
    Content-Type: application/x-git-upload-pack-advertisement

In this case, Savannah is honouring Git's query params, and speaking the
"Smart HTTP" protocol.

The problem came when I tried serving a bare Git repository over HTTP using the
"Dumb HTTP" protocol, and that libgit doesn't implement [1]:

    $ curl -I https://euandreh.xyz/package-repository.git/info/refs
    HTTP/2 200
    content-type: text/plain; charset=UTF-8

    $ curl -I https://euandreh.xyz/package-repository.git/info/refs?service=git-upload-pack
    HTTP/2 200
    content-type: text/plain; charset=UTF-8

[0]: https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
[1]: https://github.com/libgit2/libgit2/issues/4652#issuecomment-390903142

---

So my suggestion is that Guix understands how to fetch from channels that are
just tarballs.  Right now channels are married to Git, and specifically,
libgit's reimplementation of Git.  This is limiting for people who:

1) want to have bare Git repositories served via HTTP, but using only the 
   "Dumb protocol";
2) want to have Guix channels on things other than Git, like Mercurial, Fossil,
   BitKeeper, etc.;
3) want to distribute only the tarball of the channel, and not the repository
   where the code of the channel lives.

I see Guix channels understanding tarballs as a solution to the above.  One
could argue that other DVCS would be treated as second class, since no
authentication or caching would be done, but I find this reasonable, and also
decoupled from the choices of the channel provider.  As long as they can produce
a tarball with Guile files, iit works.  This is a much lower bar to creating and
maintaining Guix channels.

Re: Can't serve a custom channel on a HTTPS repository

[EuAndreh]

I've just registered one:

https://issues.guix.gnu.org/50606