bug#50568: Missing source code

bug#50568: Missing source code

[Ludovic Courtès]

Hello,

As of current ‘master’ (ca.  ec0e05ff306c950142c9ead7c712c749617069e7),
the following source code is missing (these are fixed-output
derivations):

   /gnu/store/3814l5f74b30yy0j84a4wrv8nl16fx26-eye-1.4.1.tar.gz.drv
   /gnu/store/fjk0q2ajk3y18yh5ngif8ir14xckxqa5-weightwatcher-1.12.tar.gz.drv
   /gnu/store/syndcxqrzqph12k22y5x6i454jajjw00-skymaker-3.10.5.tar.gz.drv
   /gnu/store/ginlb786zg4kqnzvxxfh8qx7wqajhl10-stuff-1.26.0.tar.gz.drv
   /gnu/store/hkgpygplymyyaifk6pmvs2cvmkmfhpkq-clusterProfiler_4.0.4.tar.gz.drv
   /gnu/store/ghg86w6i14jkrq8nvcazr7pvxz8i82s2-ggtree_3.0.3.tar.gz.drv
   /gnu/store/prbkg6vldsh1688fsigr33g4s8qla7rl-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout.drv

These two fail to build with a hash mismatch error:

   /gnu/store/v1h0djr2daz1ymmsb7043r79llcdamxa-canada1500.zip.drv
   /gnu/store/gj7g1hxjja58shki02hcdpbahrxwbcjr-texlive-marginnote-51265-checkout.drv

If you know how to retrieve them, please share!

Ludo’.

bug#50568: Missing source code

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

Ludovic Courtès schreef op ma 13-09-2021 om 15:16 [+0200]:
> Hello,
> 
> As of current ‘master’ (ca.  ec0e05ff306c950142c9ead7c712c749617069e7),
> the following source code is missing (these are fixed-output
> derivations):

The Wayback Machine has some copies of software at https://www.atromatic.net:

>    /gnu/store/3814l5f74b30yy0j84a4wrv8nl16fx26-eye-1.4.1.tar.gz.drv
https://web.archive.org/web/20161204131910/https://www.astromatic.net/download/eye/eye-1.4.1.tar.gz

>    /gnu/store/fjk0q2ajk3y18yh5ngif8ir14xckxqa5-weightwatcher-1.12.tar.gz.drv

https://web.archive.org/web/20161204140548/https://www.astromatic.net/download/weightwatcher/weightwatcher-1.12.tar.gz

>    /gnu/store/syndcxqrzqph12k22y5x6i454jajjw00-skymaker-3.10.5.tar.gz.drv

https://web.archive.org/web/20161206053718/https://www.astromatic.net/download/skymaker/skymaker-3.10.5.tar.gz

>    /gnu/store/ginlb786zg4kqnzvxxfh8qx7wqajhl10-stuff-1.26.0.tar.gz.drv

https://web.archive.org/web/20161204140455/https://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#50568: Missing source code

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 246 bytes --]

Ludo',

I'm afraind I only have:

Ludovic Courtès 写道：
> /gnu/store/prbkg6vldsh1688fsigr33g4s8qla7rl-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout.drv

Let me know where & how to send it.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#50568: Missing source code

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

Tobias Geerinckx-Rice via Bug reports for GNU Guix 写道：
> /gnu/store/prbkg6vldsh1688fsigr33g4s8qla7rl-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout.drv

FTR, I don't think this one's actually down.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#50568: Missing source code

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> The Wayback Machine has some copies of software at https://www.atromatic.net:

Thanks, they’re now all in store on berlin.

It’s kinda ironic that it’s the Wayback Machine that’s saving us here…
I wonder if there’s a way to construct generic web.archive.org URLs that
we could use as a fallback in (guix build download)?

Thanks,
Ludo’.

bug#50568: Missing source code

[Ludovic Courtès]

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> Tobias Geerinckx-Rice via Bug reports for GNU Guix 写道：
>> /gnu/store/prbkg6vldsh1688fsigr33g4s8qla7rl-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout.drv
>
> FTR, I don't think this one's actually down.

Oops, indeed, this one’s available.

Thanks,
Ludo’.

bug#50568: Missing source code

[zimoun]

Hi,

On Tue, 14 Sept 2021 at 10:40, Ludovic Courtès <ludo@gnu.org> wrote:

> Thanks, they’re now all in store on berlin.

Do we do the usual trick to save them on SWH?

> It’s kinda ironic that it’s the Wayback Machine that’s saving us here…
> I wonder if there’s a way to construct generic web.archive.org URLs that
> we could use as a fallback in (guix build download)?

That would help, I guess. :-)

Cheers,
simon

bug#50568: Missing source code

[pelzflorian (Florian Pelz)]

On Tue, Sep 14, 2021 at 10:38:55AM +0200, Ludovic Courtès wrote:
> Hi,
> 
> Maxime Devos <maximedevos@telenet.be> skribis:
> 
> > The Wayback Machine has some copies of software at https://www.atromatic.net:
> 
> Thanks, they’re now all in store on berlin.
> 
> It’s kinda ironic that it’s the Wayback Machine that’s saving us here…
> I wonder if there’s a way to construct generic web.archive.org URLs that
> we could use as a fallback in (guix build download)?

AFAIK Just replace the date in

On Mon, Sep 13, 2021 at 03:49:05PM +0200, Maxime Devos wrote:
> https://web.archive.org/web/20161204140455/https://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz

by the current time.

Regards,
Florian

bug#50568: Missing source code

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Tue, 14 Sept 2021 at 10:40, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Thanks, they’re now all in store on berlin.
>
> Do we do the usual trick to save them on SWH?

I think their contents are already saved on berlin, just not the
tarballs themselves.

For example:

--8<---------------cut here---------------start------------->8---
$ guix download https://web.archive.org/web/20161204140455/https://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz

Starting download of /tmp/guix-file.HTtGKM
From https://web.archive.org/web/20161204140455/https://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz...
 …6.0.tar.gz                                                                        163KiB/s 00:25 | 3.9MiB transferred
/gnu/store/l3r5j8r4f6fh6bxa9va3681n3bclxpdq-stuff-1.26.0.tar.gz
1syibi3b86z9pikhicvkkmgxm916j732fdiw0agw0lq6z13fdcjm
$ guix environment --ad-hoc disarchive -- disarchive disassemble /gnu/store/l3r5j8r4f6fh6bxa9va3681n3bclxpdq-stuff-1.26.0.tar.gz |grep swhid
                        (swhid "swh:1:dir:128bbe76a82dd0b38b725565ed703a7148257ae0"))
--8<---------------cut here---------------end--------------->8---

and:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix swh)
scheme@(guile-user)> (lookup-directory "128bbe76a82dd0b38b725565ed703a7148257ae0")
$24 = #f
--8<---------------cut here---------------end--------------->8---

Hmm actually maybe they vanished before they got a chance to be
archived?

Ludo’.

bug#50568: Missing source code

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Tue, Sep 14, 2021 at 10:38:55AM +0200, Ludovic Courtès wrote:
>> Hi,
>> 
>> Maxime Devos <maximedevos@telenet.be> skribis:
>> 
>> > The Wayback Machine has some copies of software at https://www.atromatic.net:
>> 
>> Thanks, they’re now all in store on berlin.
>> 
>> It’s kinda ironic that it’s the Wayback Machine that’s saving us here…
>> I wonder if there’s a way to construct generic web.archive.org URLs that
>> we could use as a fallback in (guix build download)?
>
> AFAIK Just replace the date in
>
> On Mon, Sep 13, 2021 at 03:49:05PM +0200, Maxime Devos wrote:
>> https://web.archive.org/web/20161204140455/https://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz
>
> by the current time.

Oh nice, that seems to work:

--8<---------------cut here---------------start------------->8---
$ guix download https://web.archive.org/web/20210911171400/https://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz

Starting download of /tmp/guix-file.QXTq4V
From https://web.archive.org/web/20210911171400/https://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz...
following redirection to `https://web.archive.org/web/20161204140455/http://www.astromatic.net/download/stuff/stuff-1.26.0.tar.gz'...
 …6.0.tar.gz                                                                        200KiB/s 00:20 | 3.9MiB transferred
/gnu/store/l3r5j8r4f6fh6bxa9va3681n3bclxpdq-stuff-1.26.0.tar.gz
1syibi3b86z9pikhicvkkmgxm916j732fdiw0agw0lq6z13fdcjm
--8<---------------cut here---------------end--------------->8---

How ’bout the attached patch?  Works for me.

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1444 bytes --]

diff --git a/guix/build/download.scm b/guix/build/download.scm
index c8ddadfdd4..140e42afec 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -678,6 +678,18 @@ and write the output to FILE."
                (false-if-exception*
                 (disarchive-assemble spec file #:resolver resolve))))))))
 
+(define (internet-archive-uri uri)
+  "Return a URI corresponding to an Internet Archive backup of URI, or #f if
+URI does not denote a Web URI."
+  (and (memq (uri-scheme uri) '(http https))
+       (let* ((now  (time-utc->date (current-time time-utc)))
+              (date (date->string now "~Y~m~d~H~M~S")))
+         ;; Note: the date in the URL can be anything and web.archive.org
+         ;; automatically redirects to the closest date.
+         (build-uri 'https #:host "web.archive.org"
+                    #:path (string-append "/web/" date "/"
+                                          (uri->string uri))))))
+
 (define* (url-fetch url file
                     #:key
                     (timeout 10) (verify-certificate? #t)
@@ -769,7 +781,9 @@ otherwise simply ignore them."
 
   (setvbuf (current-error-port) 'line)
 
-  (let try ((uri (append uri content-addressed-uris)))
+  (let try ((uri (append uri content-addressed-uris
+                   (or (and=> (internet-archive-uri (first uri)) list)
+                       '()))))
     (match uri
       ((uri tail ...)
        (or (fetch uri file)

bug#50568: Missing source code

[zimoun]

Hi,

On Thu, 16 Sept 2021 at 21:28, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
> > On Tue, 14 Sept 2021 at 10:40, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >> Thanks, they’re now all in store on berlin.
> >
> > Do we do the usual trick to save them on SWH?
>
> I think their contents are already saved on berlin, just not the
> tarballs themselves.

I mean, download the tarball from Berlin, then turn locally the source
to a Git source, archive it on SWH.
As it had been done for guile-dbi for instance.


> Hmm actually maybe they vanished before they got a chance to be
> archived?

Probably.  One thing that I have on my TODO is to write a sources.json
with all the source Berlin has.  Or as a start add the URL of Berlin
as "mirror".

Cheers,
simon

bug#50568: Missing source code

[pelzflorian (Florian Pelz)]

On Tue, Sep 14, 2021 at 10:38:55AM +0200, Ludovic Courtès wrote:
> I wonder if there’s a way to construct generic web.archive.org URLs that
> we could use as a fallback in (guix build download)?
On Thu, Sep 16, 2021 at 09:45:26PM +0200, Ludovic Courtès wrote:
> How ’bout the attached patch?  Works for me.

Thank you for implementing.  I guix pull with the patch applied, add the IP
from

guix environment --ad-hoc bind:utils -- dig web.archive.org

to /etc/hosts, set my DNS server to a non-existing 10.0.0.50, disable
IPv6, then “sudo herd stop nscd”, re-connect Network Manager (ping
gnu.org fails now) and LC_ALL=en_US.utf8 guix install hello.  It
fails.  Did I misunderstand the patch’s purpose or test in a wrong
way?  This is the output:

florian@florianacer ~$ LC_ALL=en_US.utf8 guix install hello
The following package will be installed:
   hello 2.10

substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%guix substitute: warning: ci.guix.gnu.org: host not found: Temporary failure in name resolution
substitute: 
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'...   0.0%guix substitute: warning: bordeaux.guix.gnu.org: host not found: Temporary failure in name resolution
substitute: 
The following derivations will be built:
   /gnu/store/4hsqwl19n4m2scipkzy0rsc04n30vhbk-profile.drv
   /gnu/store/xr30pd6zw8lp0xz3kq2d7gm3c8pmh674-hello-2.10.drv
   /gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv

substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'...   0.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'...   0.0%
applying 5 grafts for /gnu/store/rsizfxkag8gkbccr5d4g9xdacgwh8wya-at-spi2-core-2.34.0.drv ...
building /gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv...
\builder for `/gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv' failed to produce output path `/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz'
build of /gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv failed
View build log at '/var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2'.
applying 4 grafts for /gnu/store/8jj80f4p5bp9s09zawxmli13chrdx2ri-at-spi2-atk-2.34.1.drv ...
cannot build derivation `/gnu/store/xr30pd6zw8lp0xz3kq2d7gm3c8pmh674-hello-2.10.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/4hsqwl19n4m2scipkzy0rsc04n30vhbk-profile.drv': 1 dependencies couldn't be built
guix install: error: build of `/gnu/store/4hsqwl19n4m2scipkzy0rsc04n30vhbk-profile.drv' failed
florian@florianacer ~$ bzcat /var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
From ftp://ftp.cs.tu-berlin.de/pub/gnu/hello/hello-2.10.tar.gz...
In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
From ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/hello/hello-2.10.tar.gz...
In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
From http://ftp.gnu.org/pub/gnu/hello/hello-2.10.tar.gz...
In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
From https://ci.guix.gnu.org/file/hello-2.10.tar.gz/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i...
In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
From https://tarballs.nixos.org/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i...
In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
From https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/raw/...
In procedure getaddrinfo: Temporary failure in name resolution
Trying to use Disarchive to assemble /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz...
In procedure getaddrinfo: Temporary failure in name resolution
could not find its Disarchive specification
failed to download "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz" from "mirror://gnu/hello/hello-2.10.tar.gz"

bug#50568: Missing source code

[Ludovic Courtès]

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Tue, Sep 14, 2021 at 10:38:55AM +0200, Ludovic Courtès wrote:
>> I wonder if there’s a way to construct generic web.archive.org URLs that
>> we could use as a fallback in (guix build download)?
> On Thu, Sep 16, 2021 at 09:45:26PM +0200, Ludovic Courtès wrote:
>> How ’bout the attached patch?  Works for me.
>
> Thank you for implementing.  I guix pull with the patch applied, add the IP
> from
>
> guix environment --ad-hoc bind:utils -- dig web.archive.org
>
> to /etc/hosts, set my DNS server to a non-existing 10.0.0.50, disable
> IPv6, then “sudo herd stop nscd”, re-connect Network Manager (ping
> gnu.org fails now) and LC_ALL=en_US.utf8 guix install hello.  It
> fails.  Did I misunderstand the patch’s purpose or test in a wrong
> way?  This is the output:

The thing is that this code is used by the internal ‘guix
perform-download’ command, invoked by ‘guix-daemon’ when performing a
source download.  Thus, since you were still running the old daemon, you
were using code lacking the Internet Archive fallback.

To test it, you can first start the new daemon, along these lines:

  sudo herd stop guix-daemon
  sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild

Another option is to call ‘url-fetch’ manually at the REPL.

Thanks for testing!

Ludo’.

bug#50568: Missing source code

[pelzflorian (Florian Pelz)]

On Mon, Sep 20, 2021 at 09:25:27AM +0200, Ludovic Courtès wrote:
> The thing is that this code is used by the internal ‘guix
> perform-download’ command, invoked by ‘guix-daemon’ when performing a
> source download.

Thank you for enlightening me; I had not expected guix-daemon.

Success!  After

./bootstrap
./configure --localstatedir=/var
make
GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=y make update-guix-package

and git commit -a, guix pull, reconfigure and reboot, your
web.archive.org patch downloads from web.archive.org with the steps
from before:

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> I guix pull with the patch applied, add the IP from
>
> guix environment --ad-hoc bind:utils -- dig web.archive.org
>
> to /etc/hosts, set my DNS server to a non-existing 10.0.0.50, disable
> IPv6, then “sudo herd stop nscd”, re-connect Network Manager (ping
> gnu.org fails now) and LC_ALL=en_US.utf8 guix install hello.

/var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2
says after various failures in name resolution it got downloaded from
web.archive.org.  I enjoy GNU Hello now.  Thank you for making
reproducibility easier.

> To test it, you can first start the new daemon, along these lines:
> 
>   sudo herd stop guix-daemon
>   sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild

No, going back one generation and guix gc -D’ing the
/gnu/store/*hello-2.10* after adding web.archive.org to /etc/hosts
again, I for some reason get a

florian@florianacer ~$ bzcat /var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2
error: failed to run download program '/home/florian/src/guix/scripts/guix': Keine Berechtigung

(Keine Berechtigung is German for Permission denied) but it does not
matter, since the reconfigure and reboot works when and only when
adding web.archive.org to /etc/hosts.

Actually I will later take another look at this, because the manual
says ./pre-inst-env guix-daemon should work.

> Another option is to call ‘url-fetch’ manually at the REPL.

This I didn’t try.

Regards,
Florian

bug#50568: Missing source code

[pelzflorian (Florian Pelz)]

On Tue, Sep 21, 2021 at 12:28:50PM +0200, pelzflorian (Florian Pelz) wrote:
> On Mon, Sep 20, 2021 at 09:25:27AM +0200, Ludovic Courtès wrote:
> > To test it, you can first start the new daemon, along these lines:
> > 
> >   sudo herd stop guix-daemon
> >   sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild
> 
> No, going back one generation and guix gc -D’ing the
> /gnu/store/*hello-2.10* after adding web.archive.org to /etc/hosts
> again, I for some reason get a
> 
> florian@florianacer ~$ bzcat /var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2
> error: failed to run download program '/home/florian/src/guix/scripts/guix': Keine Berechtigung
> 
> (Keine Berechtigung is German for Permission denied) but it does not
> matter, since the reconfigure and reboot works when and only when
> adding web.archive.org to /etc/hosts.
> 
> Actually I will later take another look at this, because the manual
> says ./pre-inst-env guix-daemon should work.

There is no regression.  Without the web.archive.org patch,
`guix install hello' via

sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild

always shows the same error, regardless of whether I misdirected DNS
lookups or not.  The patch is not at fault.  Though I don’t know why.

sudo -E strace -f ./pre-inst-env guix-daemon --build-users-group=guixbuild

shows

[pid  1854] execve("/home/florian/src/guix/scripts/guix", ["guix", "perform-download", "/gnu/store/wy24jyd0mm2jlgpzsxxli"..., "/gnu/store/hbdalsf5lpf01x4dcknwx"...], 0x1d8f9a0 /* 80 vars */) = -1 EACCES (Keine Berechtigung)

<https://pubs.opengroup.org/onlinepubs/9699919799/> says:
> The exec functions shall fail if:
> …
> [EACCES]
>     The new process image file is not a regular file and the implementation does not support execution of files of its type.
> The exec functions, except for fexecve(), shall fail if:
> [EACCES]
>     Search permission is denied for a directory listed in the new process image file's path prefix, or the new process image file denies execution permission.

I run:
florian@florianacer ~$ ls -ahl /home/florian/src/guix/scripts/guix
-r-xr-xr-x 1 florian users 2,0K 21. Sep 07:30 /home/florian/src/guix/scripts/guix
florian@florianacer ~$ ls -ahl /home/florian/src/guix | grep scripts
drwxr-xr-x  2 florian users 4,0K 21. Sep 07:30 scripts/
florian@florianacer ~$ lsattr /home/florian/src/guix/scripts/guix
--------------e----- /home/florian/src/guix/scripts/guix
florian@florianacer ~ [env]$ cat test-execv.c
#include <unistd.h>

int main ()
{
  const char *const argv[] =
    {
      "guix", "perform-download",
      "/gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv",
      "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz.lock",
      NULL
    };

  execv ("/home/florian/src/guix/scripts/guix", (char *const *) argv);

  return 0;
}
florian@florianacer ~ [env]$ gcc test-execv.c -o test-execv
florian@florianacer ~ [env]$ LC_ALL=en_US.utf8 ~/test-execv

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz.lock
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://ftp.wayne.edu/gnu/hello/hello-2.10.tar.gz'...
In procedure open-file: Read-only file system: "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz.lock"
[and more expected failures …]

Whatever.  It is not important.

Regards,
Florian

bug#50568: Missing source code

[Ludovic Courtès]

Hi!

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Mon, Sep 20, 2021 at 09:25:27AM +0200, Ludovic Courtès wrote:
>> The thing is that this code is used by the internal ‘guix
>> perform-download’ command, invoked by ‘guix-daemon’ when performing a
>> source download.
>
> Thank you for enlightening me; I had not expected guix-daemon.
>
> Success!  After
>
> ./bootstrap
> ./configure --localstatedir=/var
> make
> GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=y make update-guix-package
>
> and git commit -a, guix pull, reconfigure and reboot, your
> web.archive.org patch downloads from web.archive.org with the steps
> from before:

That’s a heavy test, but great that it works as intended.  I’ll push it
soonish.

> florian@florianacer ~$ bzcat /var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2
> error: failed to run download program '/home/florian/src/guix/scripts/guix': Keine Berechtigung

That is weird indeed.

Thanks for testing!

Ludo’.

bug#50568: Missing source code

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

> As of current ‘master’ (ca.  ec0e05ff306c950142c9ead7c712c749617069e7),
> the following source code is missing (these are fixed-output
> derivations):

We now have ‘etc/source-manifest.scm’, meaning one can run¹:

  guix weather -m etc/source-manifest.scm

This manifest is now built at ci.guix.gnu.org:

  https://ci.guix.gnu.org/jobset/source

… which gives us a good overview of currently missing/mismatching
source:

  https://ci.guix.gnu.org/eval/27221?status=failed

So far 27 failures out of 16,315, roughly 0.2%.

If you have copies of the missing/modified files that show up there,
please share.

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00174.html

bug#50568: Missing source code

[Guillaume Le Vaillant]

[-- Attachment #1.1: Type: text/plain, Size: 1489 bytes --]

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> As of current ‘master’ (ca.  ec0e05ff306c950142c9ead7c712c749617069e7),
>> the following source code is missing (these are fixed-output
>> derivations):
>
> We now have ‘etc/source-manifest.scm’, meaning one can run:
>
>   guix weather -m etc/source-manifest.scm
>
> This manifest is now built at ci.guix.gnu.org:
>
>   https://ci.guix.gnu.org/jobset/source
>
> … which gives us a good overview of currently missing/mismatching
> source:
>
>   https://ci.guix.gnu.org/eval/27221?status=failed
>
> So far 27 failures out of 16,315, roughly 0.2%.
>
> If you have copies of the missing/modified files that show up there,
> please share.

Hi,

I have the following ones in the store:

 - cl-gserver-1.8.0-checkout
 - gnu-standards-2020-11-25-checkout
 - golden-utils-0.0.0-3.fe1898f-checkout
 - xorg-server_1.20.10-3.diff.gz


The attached archive contains the nars made with:

--8<---------------cut here---------------start------------->8---
guix archive --export /gnu/store/...
--8<---------------cut here---------------end--------------->8---


The signing key of the machine is:

--8<---------------cut here---------------start------------->8---
(public-key 
 (ecc 
  (curve Ed25519)
  (q #988123DC42F034138C30903B80A78A6DC8B975A20D891D8DC3DD3CC4308D0D08#)))
--8<---------------cut here---------------end--------------->8---

[-- Attachment #1.2: nars.tar.lz --]
[-- Type: application/octet-stream, Size: 940876 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#50568: Missing source code

[Ludovic Courtès]

Hi Guillaume,

Guillaume Le Vaillant <glv@posteo.net> skribis:

> I have the following ones in the store:

Thanks!  I’ve been able to address these in several ways:

>  - cl-gserver-1.8.0-checkout

  commit f7f036e7c369515ad678cc08dd5bc9e254804d38 (do take a look at
  the comment in that commit…)

>  - gnu-standards-2020-11-25-checkout

  CVS doesn’t go through the firewall at berlin, so I manually added it
  to the store (not great).

>  - golden-utils-0.0.0-3.fe1898f-checkout

  commit ef93b238f64c1a27270cae78772b60d63088a026

>  - xorg-server_1.20.10-3.diff.gz

  manually added to the store; problem is that this file has disappeared
  upstream, AFAICS, so it would be best to change this package
  definition, possibly adding copy of the diff.gz file to the repo?

Ludo’.

bug#50568: Missing source code

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

>   https://ci.guix.gnu.org/eval/27221?status=failed

We’re left with the following missing bioconductor.org tarballs:

  AnnotationFuncs_1.40.0.tar.gz
  AneuFinderData_1.18.0.tar.gz
  BiocCaseStudies_1.52.0.tar.gz
  chromstaRData_1.16.0.tar.gz
  genomationData_1.22.0.tar.gz

and a hash mismatch for:

  texlive-marginnote-51265-checkout

Zimoun, Ricardo: do you know how to address the bioconductor.org issue?

Ricardo: do you still have a copy of texlive-marginnote-51265-checkout
with the hash as it currently appears on master?  We’ll probably end up
just updating the hash, but it’d be nice to check the differences.

Once we’re done with those, we’ll have 100% source coverage on ci.guix.
At that point, we can start building Disarchive metadata.

TIA!

Ludo’.

bug#50568: Missing source code

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

Ludovic Courtès schreef op do 07-10-2021 om 15:53 [+0200]:
> Ludovic Courtès <ludo@gnu.org> skribis:
> 
> >   https://ci.guix.gnu.org/eval/27221?status=failed
> 
> We’re left with the following missing bioconductor.org tarballs:
> 
>   AnnotationFuncs_1.40.0.tar.gz
>   AneuFinderData_1.18.0.tar.gz
>   BiocCaseStudies_1.52.0.tar.gz
>   chromstaRData_1.16.0.tar.gz
>   genomationData_1.22.0.tar.gz

Running "guix build --source r-annotationfuncs r-aneufinderdata r-bioccasestudies r-chromstardata r-genomationdata"
succeeds on my computer.  They are downloaded from bayfront.guix.gnu.org.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#50568: Missing source code

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op do 07-10-2021 om 15:53 [+0200]:
>> Ludovic Courtès <ludo@gnu.org> skribis:
>> 
>> >   https://ci.guix.gnu.org/eval/27221?status=failed
>> 
>> We’re left with the following missing bioconductor.org tarballs:
>> 
>>   AnnotationFuncs_1.40.0.tar.gz
>>   AneuFinderData_1.18.0.tar.gz
>>   BiocCaseStudies_1.52.0.tar.gz
>>   chromstaRData_1.16.0.tar.gz
>>   genomationData_1.22.0.tar.gz
>
> Running "guix build --source r-annotationfuncs r-aneufinderdata r-bioccasestudies r-chromstardata r-genomationdata"
> succeeds on my computer.  They are downloaded from bayfront.guix.gnu.org.

Indeed, fixed!

Currently bayfront.guix.gnu.org doesn’t run ‘guix publish’, so it
doesn’t support the /file URL scheme, which means I had to do this
manually.  We’ll fix it!

Ludo’.

bug#50568: Missing source code

[zimoun]

Hi Ludo,

On Thu, 7 Oct 2021 at 15:53, Ludovic Courtès <ludo@gnu.org> wrote:
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
> >   https://ci.guix.gnu.org/eval/27221?status=failed
>
> We’re left with the following missing bioconductor.org tarballs:
>
>   AnnotationFuncs_1.40.0.tar.gz

I pick this one.  Cuirass says Succeeded with the log:

https://ci.guix.gnu.org/build/1000819/log/raw

Well, I do not know what is this 141.80.167.131 IP address.  Something
related to Bayfront, right?


>   AneuFinderData_1.18.0.tar.gz
>   BiocCaseStudies_1.52.0.tar.gz
>   chromstaRData_1.16.0.tar.gz
>   genomationData_1.22.0.tar.gz

> Zimoun, Ricardo: do you know how to address the bioconductor.org issue?

From the evaluation you mention above, all these had succeeded.  Could
you confirm it is fine?


BTW, for the interested reader, I would like to point this bug#39885
[1] about Bioconductor, URI, fallback and time-machine which appears
to me related.  In addition to the two options proposed, another one
not listed but discussed elsewhere is to switch Bioconductor packages
from url-fetch to git-fecth.

1: <http://issues.guix.gnu.org/issue/39885>


All the best,
simoon

bug#50568: Missing source code

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Thu, 7 Oct 2021 at 15:53, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>> >   https://ci.guix.gnu.org/eval/27221?status=failed
>>
>> We’re left with the following missing bioconductor.org tarballs:
>>
>>   AnnotationFuncs_1.40.0.tar.gz
>
> I pick this one.  Cuirass says Succeeded with the log:
>
> https://ci.guix.gnu.org/build/1000819/log/raw
>
> Well, I do not know what is this 141.80.167.131 IP address.  Something
> related to Bayfront, right?
>
>
>>   AneuFinderData_1.18.0.tar.gz
>>   BiocCaseStudies_1.52.0.tar.gz
>>   chromstaRData_1.16.0.tar.gz
>>   genomationData_1.22.0.tar.gz
>
>> Zimoun, Ricardo: do you know how to address the bioconductor.org issue?
>
> From the evaluation you mention above, all these had succeeded.  Could
> you confirm it is fine?

It is: I fetched copies from bayfront, IIRC.

> BTW, for the interested reader, I would like to point this bug#39885
> [1] about Bioconductor, URI, fallback and time-machine which appears
> to me related.  In addition to the two options proposed, another one
> not listed but discussed elsewhere is to switch Bioconductor packages
> from url-fetch to git-fecth.
>
> 1: <http://issues.guix.gnu.org/issue/39885>

Very relevant!  How can we avoid those brittle bioconductor.org URLs?

Thanks,
Ludo’.

bug#50568: Missing source code

[zimoun]

Hi,

On Tue, 12 Oct 2021 at 11:50, Ludovic Courtès <ludo@gnu.org> wrote:

>> BTW, for the interested reader, I would like to point this bug#39885
>> [1] about Bioconductor, URI, fallback and time-machine which appears
>> to me related.  In addition to the two options proposed, another one
>> not listed but discussed elsewhere is to switch Bioconductor packages
>> from url-fetch to git-fecth.
>>
>> 1: <http://issues.guix.gnu.org/issue/39885>
>
> Very relevant!  How can we avoid those brittle bioconductor.org URLs?

By addressing the mentioned bug. ;-) Do not hold your breath, it is low
in my TODO list.  The next Bioconductor release v3.14 is planned for
Wednesday October 27.  Therefore, let make it happens for v3.15. :-)


Cheers,
simon