bug#42286: SWH fallback fails (git-fetch)

bug#42286: SWH fallback fails (git-fetch)

[zimoun]

Dear,

I think the fallback with SWH is broken.  Here an example to reproduce.

--8<---------------cut here---------------start------------->8---
cd /tmp/
tar -xvf $(guix build -S hello)
cd hello-2.10
git init
git add *
git commit -m 'init'
--8<---------------cut here---------------end--------------->8---

Then I push this fresh repo to my own GitHub account.  Because it is
easy to switch between public and private state option, which mimics
upstream loss.

Let create the well-known "hello" recipe, which is simply a copy/paste
of (gnu packages base).

--8<---------------cut here---------------start------------->8---
(define-module (hello)
  #:use-module (guix packages)
  #:use-module (guix build-system gnu)
  #:use-module (guix git-download)
  #:use-module (guix licenses))

(define-public hi
  (package
    (name "hi")
    (version "2.10")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/zimoun/hello-example.git")
                    (commit "e1eefd033b8a2c4c81babc6fde08ebb116c6abb8")))
              (sha256
               (base32
                "1im1gglfm4k10bh4mdaqzmx3lm3kivnsmxrvl6vyvmfqqzljq75l"))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
--8<---------------cut here---------------end--------------->8---

For the sanity of the test, "guix build -L . hi" builds correctly and
returns the item: /gnu/store/jn8d031zx4znxy7s5zhj4dbr6xjsfq9v-hi-2.10
with Guix b7249aa.

The upstream source is archived on SWH with:

   guix lint -L . hi -c archival

see
https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/zimoun/hello-example.git
and the logs in https://archive.softwareheritage.org/save/#requests

Note that running again "lint -c archival" returns nothing, which means
that the linter finds it on SWH, I guess.  Well, I have not checked what
'lookup-content' returns yet.  However, I filtered the first ~2000+
'packages-with-sources' with 'lookup-content' and all the 'git-fetch'
sources are returned #f, and some 'url-fetch' not.  Another story,
another thread. :-)


Let clean the previous builds and downloads:

    guix gc -D $(guix gc --list-dead | grep -E '(hi-2|git-checkout)')

And now let turn off upstream and so the maybe bug:

--8<---------------cut here---------------start------------->8---
guix build -L . hi
The following derivations will be built:
   /gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv
   /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv
building /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
Initialized empty Git repository in /gnu/store/884nsva9r8wkp40kbqyvpj1ad57jc5dd-git-checkout/.git/
fatal: could not read Username for 'https://github.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://github.com': No such device or address
git-fetch: '/gnu/store/i2pkspl2vjvas0q85hw46y2li06rcmcg-git-minimal-2.26.2/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
Backtrace:
           4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
In ./guix/swh.scm:
   573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
   224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
In web/client.scm:
    563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
    231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)

web/client.scm:231:6: In procedure tls-wrap:
Error while printing exception.
builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
build of /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv failed
View build log at '/var/log/guix/drvs/jn/6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv.bz2'.
cannot build derivation `/gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv' failed
--8<---------------cut here---------------end--------------->8---


Let me know if moreinfo are required.

All the best,
simon

bug#42286: swh-download with https->http

[zimoun]

On Thu, 09 Jul 2020 at 02:36, zimoun <zimon.toutoune@gmail.com> wrote:

> --8<---------------cut here---------------start------------->8---
> guix build -L . hi

[...]

> Trying to download from Software Heritage...
> Backtrace:
>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> Error while printing exception.

[...]

> --8<---------------cut here---------------end--------------->8---

To avoid, this error which should be a missing certificate, I switch to
'http:' instead of 'https:' in %swh-base-url.  The error is then:

--8<---------------cut here---------------start------------->8---
Trying to download from Software Heritage...
Backtrace:
           2 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
In ./guix/swh.scm:
   573:13  1 (swh-download "https://github.com/zimoun/hello-example?" ?)
   243:16  0 (call "http://archive.softwareheritage.org/api/1/revis?" ?)

./guix/swh.scm:243:16: In procedure call:
Throw to key `swh-error' with args `("http://archive.softwareheritage.org/api/1/revision/e1eefd033b8a2c4c81babc6fde08ebb116c6abb8/" #<procedure http-get (uri #:key body port version keep-alive? headers decode-body? verify-certificate? streaming?)> #<<response> version: (1 . 1) code: 302 reason-phrase: "Found" headers: ((date . #<date nanosecond: 0 second: 59 minute: 38 hour: 1 day: 9 month: 7 year: 2020 zone-offset: 0>) (server . "Varnish") (x-varnish . "19881791") (location . #<<uri> scheme: https userinfo: #f host: "archive.softwareheritage.org" port: #f path: "/api/1/revision/e1eefd033b8a2c4c81babc6fde08ebb116c6abb8/" query: #f fragment: #f>) (content-length . 0) (connection close)) port: #<closed: file 7ffff5f1b690>>)'.
builder for `/gnu/store/9f6648zb0p9z6hxr31wz3d7fm7ndcxzx-git-checkout.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---


Because I hit the rate limit, I need to wait at least one hour...
Follow-ups soon. :-)

Cheers,
simon

bug#42286: SWH fallback fails (git-fetch)

[zimoun]

On Thu, 9 Jul 2020 at 02:39, zimoun <zimon.toutoune@gmail.com> wrote:

Well, "swh-download" perfectly works directly from the REPL.

> --8<---------------cut here---------------start------------->8---
> guix build -L . hi

[...]

> Trying to download from Software Heritage...
> Backtrace:
>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> --8<---------------cut here---------------end--------------->8---

The error is definitively something related to TLS and the gexp.  In
(guix git-download) "git-fetch", the "define build" returns a gexp and
here something is missing, even if the extension gnutls (module-ref
(resolve-interface '(gnu packages tls)) 'gnutls) is provided.  Hum, I
am not sure to understand what.

BTW, if in "git-fetch" from (guix git), I add these lines:

--8<---------------cut here---------------start------------->8---
  (setenv "GIT_SSL_NO_VERIFY" "true")

  (format #t "git-fetch~%")
  (http-get "https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/")
  (format #t "ok~%")

  (mkdir-p directory)
--8<---------------cut here---------------end--------------->8---

Then I also hit:

--8<---------------cut here---------------start------------->8---
The following derivations will be built:
   /gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv
   /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv
building /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to
`/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
git-fetch
Backtrace:
           4 (primitive-load "/gnu/store/gcr6v6p6c5gwr4l6xzqcy6wln33?")
In ice-9/eval.scm:
   293:34  3 (_ #<directory (guile-user) 7ffff5bb8f00>)
In ./guix/build/git.scm:
     44:2  2 (git-fetch "https://github.com/zimoun/hello-example.git" ?)
In web/client.scm:
    563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
    231:6  0 (tls-wrap #<closed: file 7ffff4c17150> _ # _)

web/client.scm:231:6: In procedure tls-wrap:
Error while printing exception.
builder for `/gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv'
failed with exit code 1
build of /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv failed
View build log at
'/var/log/guix/drvs/cv/p65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv.bz2'.
cannot build derivation
`/gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv': 1
dependencies couldn't be built
guix build: error: build of
`/gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv' failed
--8<---------------cut here---------------end--------------->8---

Then I have tried to turn off the certificate verification with
"#:verify-certification #f" for example in "vault-fetch" or "call"
used by "define-query" but nothing works.  Well, I am a bit
circumspect.

Therefore, I am waiting for a hint or the fix. :-)

Cheers,
simon

bug#42286: SWH fallback fails (git-fetch)

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> Trying to download from Software Heritage...
> Backtrace:
>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> Error while printing exception.
> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1

Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
Let me know if anything’s amiss.

Thanks!

Ludo’.

bug#42286: SWH fallback fails (git-fetch)

[zimoun]

Hi,

On Fri, 10 Jul 2020 at 00:24, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> Trying to download from Software Heritage...
>> Backtrace:
>>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
>> In ./guix/swh.scm:
>>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
>> In web/client.scm:
>>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>>
>> web/client.scm:231:6: In procedure tls-wrap:
>> Error while printing exception.
>> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
>
> Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
> Let me know if anything’s amiss.

Cool! Works. :-)

I was almost there. :-) The missing trick was because the Guile bug
   <https://bugs.gnu.org/40486>
I was not aware and so the new "http-get*".

Is it worth to add the test in guix-build.sh?


All the best,
simon

bug#42286: SWH fallback fails (git-fetch)

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> On Fri, 10 Jul 2020 at 00:24, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hi!
>>
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>>> Trying to download from Software Heritage...
>>> Backtrace:
>>>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
>>> In ./guix/swh.scm:
>>>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>>>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
>>> In web/client.scm:
>>>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>>>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>>>
>>> web/client.scm:231:6: In procedure tls-wrap:
>>> Error while printing exception.
>>> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
>>
>> Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
>> Let me know if anything’s amiss.
>
> Cool! Works. :-)
>
> I was almost there. :-) The missing trick was because the Guile bug
>    <https://bugs.gnu.org/40486>
> I was not aware and so the new "http-get*".

Yeah.  :-)

> Is it worth to add the test in guix-build.sh?

We don’t add tests that depend on external services, so we can’t really
do that.  Or we would need to mock the original server, SWH, etc. but
that seems tricky.

Thanks,
Ludo’.