unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / Atom feed
* bug#39885: Bioconductor URI, fallback and time-machine
@ 2020-03-03 15:59 zimoun
  2020-03-23 21:20 ` Ricardo Wurmus
                   ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: zimoun @ 2020-03-03 15:59 UTC (permalink / raw)
  To: 39885, me, rekado

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

Dear,

Currently, the URI scheme (see 'bioconductor-uri' in
guix/build-system/r.scm) is:

 https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz

which leads to 2 issues:

 1. when Bioconductor updates their release, some package versions are
updated too, and so, the upstream return 404.
 2. for this reason 1., the "guix time-machine" is broken for all the
Bioconductor packages, at least if Berlin or SWH does not have a
substitute; which is not expected for 'annotation' packages.

However, the Bioconductor archive still serves the old release, i.e.,

https://bioconductor.org/packages/3.x/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz


The ways to fix the both issues are:

 a) Add the Bioconductor release (known at packaging time) to all the
packages; provide as argument to 'bioconductor-uri'.
 b) Add more URLs to fallback.

As discussed on IRC, Tobias seems more inclined with the option a) and
I am more in favour of option b.

Attached, a quick patch showing the option b).


Please also consider #36805 which was never merged or closed.
 http://issues.guix.gnu.org/issue/36805


All the best,
simon

[-- Attachment #2: 0001-build-system-r-Use-Bioconductor-old-releases-to-fall.patch --]
[-- Type: text/x-patch, Size: 2041 bytes --]

From 87e73e02202fe5e342d68f1fb17efdd4425737cd Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Tue, 3 Mar 2020 16:53:39 +0100
Subject: [PATCH] build-system: r: Use Bioconductor old releases to fallback.

* guix/build-system/r.scm (bioconductor-uri): Extend the fallback list.
---
 guix/build-system/r.scm | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
index 2d328764b0..8638e1b888 100644
--- a/guix/build-system/r.scm
+++ b/guix/build-system/r.scm
@@ -54,15 +54,18 @@ release corresponding to NAME and VERSION."
                          ('annotation "/data/annotation")
                          ('experiment "/data/experiment")
                          (_ "/bioc"))))
-    (list (string-append "https://bioconductor.org/packages/release"
-                         type-url-part
-                         "/src/contrib/"
-                         name "_" version ".tar.gz")
-          ;; TODO: use %bioconductor-version from (guix import cran)
-          (string-append "https://bioconductor.org/packages/3.10"
-                         type-url-part
-                         "/src/contrib/Archive/"
-                         name "_" version ".tar.gz"))))
+    (append (list (string-append "https://bioconductor.org/packages/release"
+                                 type-url-part
+                                 "/src/contrib/"
+                                 name "_" version ".tar.gz"))
+            (map (lambda (release)
+                   (string-append "https://bioconductor.org/packages/"
+                                  release
+                                  type-url-part
+                                  "/src/contrib/"
+                                  name "_" version ".tar.gz"))
+                 (list (@@ (guix import cran) %bioconductor-version)
+                       "3.9" "3.8" "3.7")))))
 
 (define %r-build-system-modules
   ;; Build-side modules imported by default.
-- 
2.25.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#39885: Bioconductor URI, fallback and time-machine
  2020-03-03 15:59 bug#39885: Bioconductor URI, fallback and time-machine zimoun
@ 2020-03-23 21:20 ` Ricardo Wurmus
  2020-05-21 23:29   ` zimoun
  2020-06-24 11:07 ` zimoun
  2020-11-19 14:22 ` zimoun
  2 siblings, 1 reply; 8+ messages in thread
From: Ricardo Wurmus @ 2020-03-23 21:20 UTC (permalink / raw)
  To: zimoun; +Cc: 39885


zimoun <zimon.toutoune@gmail.com> writes:

>  1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
>  2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.
>
> However, the Bioconductor archive still serves the old release, i.e.,
>
> https://bioconductor.org/packages/3.x/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
>
> The ways to fix the both issues are:
>
>  a) Add the Bioconductor release (known at packaging time) to all the
> packages; provide as argument to 'bioconductor-uri'.
>  b) Add more URLs to fallback.
>
> As discussed on IRC, Tobias seems more inclined with the option a) and
> I am more in favour of option b.

I think option a) is more explicit, which is probably what we generally
want to future-proof the time-machine.  Fallbacks are okay in the case
of the CRAN URL where it’s not necessarily clear when a package tarball
moves from the release location to the archive.

In the case of Bioconductor URLs it seems that we can afford to be a bit
more accurate.

-- 
Ricardo

^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#39885: Bioconductor URI, fallback and time-machine
  2020-03-23 21:20 ` Ricardo Wurmus
@ 2020-05-21 23:29   ` zimoun
  0 siblings, 0 replies; 8+ messages in thread
From: zimoun @ 2020-05-21 23:29 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: 39885

Dear Ricardo,

On Mon, 23 Mar 2020 at 22:21, Ricardo Wurmus <rekado@elephly.net> wrote:

> >  a) Add the Bioconductor release (known at packaging time) to all the
> > packages; provide as argument to 'bioconductor-uri'.
> >  b) Add more URLs to fallback.
> >
> > As discussed on IRC, Tobias seems more inclined with the option a) and
> > I am more in favour of option b.
>
> I think option a) is more explicit, which is probably what we generally
> want to future-proof the time-machine.  Fallbacks are okay in the case
> of the CRAN URL where it’s not necessarily clear when a package tarball
> moves from the release location to the archive.
>
> In the case of Bioconductor URLs it seems that we can afford to be a bit
> more accurate.

We are going for option a) which means rename all the URLs, right?

Because it is a lot, I suggest to first address the bug#36805, i.e.,
provide as an argument the BioConductor version to 'bioconductor-uri'
and applies this policy to all the new packages or any update of them.

Moreover, I have suggested to reorganise bioconductor.scm,
bioinformatics.scm, cran.scm, etc. and I have not dedicated enough
time to this boring task.  But because I am working remotely
(semi-lockdown), I plan to work on it next week and so this change of
URLs could be part of the big reorganisation.

What do you think?

[1] http://issues.guix.gnu.org/issue/36805


All the best,
simon




^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#39885: Bioconductor URI, fallback and time-machine
  2020-03-03 15:59 bug#39885: Bioconductor URI, fallback and time-machine zimoun
  2020-03-23 21:20 ` Ricardo Wurmus
@ 2020-06-24 11:07 ` zimoun
  2020-06-28 20:14   ` Ludovic Courtès
  2020-11-19 14:22 ` zimoun
  2 siblings, 1 reply; 8+ messages in thread
From: zimoun @ 2020-06-24 11:07 UTC (permalink / raw)
  To: 39885, Tobias Geerinckx-Rice, Ricardo Wurmus

Dear,

The time-machine is broken for some BioConductor packages..  For an
example, consider the package "r-genomegraphs" which has been removed
from the BioConductor in 3.11 release.

(Well, now the issue is mitigated because ci.guix.gnu.org serves a lot
of upstream substitutes but ci.guix.gnu.org could be down.  Other
said, we should use the upstream resources where they are available.)


Concretely, there are 2 issues:

 a) What to do for the removed packages?  For 3.11, the list is there
[1].  Do we keep them in gnu/packages/bioconductor.scm but then
'bioconductor-uri' needs some tweaks?  Or do we transfer them to the
channel guix-past (for example)?

 b) The fallback URI in guix/build-system/r.scm(bioconductor-uri)
added by commit c586f427b4831b9b492e5b900b2226e898b8fcfa is not
correct, if I do not misread:

--8<---------------cut here---------------start------------->8---
"https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/GenomeGraphs_1.46.0.tar.gz"
404 "Not Found"
--8<---------------cut here---------------end--------------->8---

The correct seems to be (without Archive):

https://bioconductor.org/packages/3.10/bioc/src/contrib/GenomeGraphs_1.46.0.tar.gz


All the best,
simon

1: https://bioconductor.org/news/bioc_3_11_release/#deprecated-and-defunct-packages




^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#39885: Bioconductor URI, fallback and time-machine
  2020-06-24 11:07 ` zimoun
@ 2020-06-28 20:14   ` Ludovic Courtès
  2020-06-29 17:36     ` zimoun
  0 siblings, 1 reply; 8+ messages in thread
From: Ludovic Courtès @ 2020-06-28 20:14 UTC (permalink / raw)
  To: zimoun; +Cc: 39885

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

>  b) The fallback URI in guix/build-system/r.scm(bioconductor-uri)
> added by commit c586f427b4831b9b492e5b900b2226e898b8fcfa is not
> correct, if I do not misread:
>
> "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/GenomeGraphs_1.46.0.tar.gz"
> 404 "Not Found"
>
> The correct seems to be (without Archive):
>
> https://bioconductor.org/packages/3.10/bioc/src/contrib/GenomeGraphs_1.46.0.tar.gz

Could you provide a patch for this?

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#39885: Bioconductor URI, fallback and time-machine
  2020-06-28 20:14   ` Ludovic Courtès
@ 2020-06-29 17:36     ` zimoun
  2020-06-29 20:42       ` Ludovic Courtès
  0 siblings, 1 reply; 8+ messages in thread
From: zimoun @ 2020-06-29 17:36 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 39885

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

Hi Ludo,

On Sun, 28 Jun 2020 at 22:14, Ludovic Courtès <ludo@gnu.org> wrote:

> Could you provide a patch for this?

About the url, for sure, see attached.

But it does not address the root of the problem.  Well, I will try to
find a slot and propose something.


All the best,
simon

[-- Attachment #2: 0001-build-system-r-bioconductor-uri-Fix-archive-URL.patch --]
[-- Type: text/x-patch, Size: 1014 bytes --]

From c1c963a3b86e306a20c14626127e54d21843c22c Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Mon, 29 Jun 2020 19:18:20 +0200
Subject: [PATCH] build-system/r: bioconductor-uri: Fix archive URL.

* guix/build-system/r.scm (bioconductor-uri): Fix archive URL.
---
 guix/build-system/r.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
index c8ec9abd0d..5ef982d66a 100644
--- a/guix/build-system/r.scm
+++ b/guix/build-system/r.scm
@@ -61,7 +61,7 @@ release corresponding to NAME and VERSION."
           ;; TODO: use %bioconductor-version from (guix import cran)
           (string-append "https://bioconductor.org/packages/3.11"
                          type-url-part
-                         "/src/contrib/Archive/"
+                         "/src/contrib/"
                          name "_" version ".tar.gz"))))
 
 (define %r-build-system-modules

base-commit: 6ebf300959a58fd1eda875205c75d21137862285
-- 
2.26.2


^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#39885: Bioconductor URI, fallback and time-machine
  2020-06-29 17:36     ` zimoun
@ 2020-06-29 20:42       ` Ludovic Courtès
  0 siblings, 0 replies; 8+ messages in thread
From: Ludovic Courtès @ 2020-06-29 20:42 UTC (permalink / raw)
  To: zimoun; +Cc: 39885

zimoun <zimon.toutoune@gmail.com> skribis:

> From c1c963a3b86e306a20c14626127e54d21843c22c Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Mon, 29 Jun 2020 19:18:20 +0200
> Subject: [PATCH] build-system/r: bioconductor-uri: Fix archive URL.
>
> * guix/build-system/r.scm (bioconductor-uri): Fix archive URL.

Applied, thanks!

I let the rest of you discuss the other issues.  :-)

Ludo’.




^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#39885: Bioconductor URI, fallback and time-machine
  2020-03-03 15:59 bug#39885: Bioconductor URI, fallback and time-machine zimoun
  2020-03-23 21:20 ` Ricardo Wurmus
  2020-06-24 11:07 ` zimoun
@ 2020-11-19 14:22 ` zimoun
  2 siblings, 0 replies; 8+ messages in thread
From: zimoun @ 2020-11-19 14:22 UTC (permalink / raw)
  To: 39885

Hi,

Some explanations of the issue are provided here:

    <http://issues.guix.gnu.org/issue/39885>

Since we are currently updating to 3.12, maybe it is the occasion to fix
the issue.  See option a) below.


On Tue, 03 Mar 2020 at 16:59, zimoun <zimon.toutoune@gmail.com> wrote:

> Currently, the URI scheme (see 'bioconductor-uri' in
> guix/build-system/r.scm) is:
>
>  https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
> which leads to 2 issues:
>
>  1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
>
>  2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.

An example of this issue is for example:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=aee183e -- import cran -a bioconductor CATALYST -r

Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...

Starting download of /tmp/guix-file.Nxajqh
From https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz...
download failed "https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz" 404 "Not Found"
failed to download "/tmp/guix-file.Nxajqh" from "https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz"
error: failed to retrieve package information from "https://cran.r-project.org/web/packages/CATALYST/DESCRIPTION": 404 ("Not Found")
Backtrace:
           4 (primitive-load "/home/simon/.cache/guix/inferiors/vznc…")
In guix/ui.scm:
  2117:12  3 (run-guix-command _ . _)
In guix/scripts/import.scm:
   120:11  2 (guix-import . _)
In srfi/srfi-1.scm:
   586:17  1 (map1 (#f))
In guix/import/utils.scm:
    258:2  0 (package->definition _)

guix/import/utils.scm:258:2: In procedure package->definition:
Throw to key `match-error' with args `("match" "no matching pattern" #f)'.
--8<---------------cut here---------------end--------------->8---

Aside the ugly backtrace which is tracked by #44115, the main issue is
because Bioconductor updated to 3.12 and Guix is still at 3.11.

Concretely, the issue is that ’release’ in the URL:

<https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz>

now refers to 3.12 (because Bioconductor update) and Guix still think it
is 3.11 (because Guix has not yet updated; work-in-progress).  And
CATALYST in 3.12 is at version 1.14.0 against 1.12.2 for 3.11.
Therefore, the conflict and the error.

It means that while:

    (define %bioconductor-version "3.11")

is not updated to 3.12, all the Bioconductor packages are broken; in the
meaning not buildable from source.


>  a) Add the Bioconductor release (known at packaging time) to all the
> packages; provide as argument to 'bioconductor-uri'.
>  b) Add more URLs to fallback, e.g.:
>
> https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
> https://bioconductor.org/packages/3.11/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
> Attached, a quick patch showing the option b).

Then each time we update Bioconductor, we add an URL to the list.


> As discussed on IRC, Tobias seems more inclined with the option a) and
> I am more in favour of option b.

Tobias and Ricardo are in favor for a) (see this thread).  Which means a
lot of work IMHO, i.e., add 3.11 as arguments and then 3.12 to all the
Bioconductor packages and fix the importer, IIUC; while b) means do
nothing except merge the proposed patch (possibly re-worked).

Just to note that only the task to group in bioconductor.scm all the
Bioconductor packages scattered here and there is still not done, I
think option a) is not doable by hand – I do not volunteer! :-) Else,
any suggestion to script the task instead?

Since I am more in favor of b), I am less motivated to fix the a). ;-)
But I am motivated to fix the issue at hand. :-)


Other option c) is to switch all the Bioconductor to git-fetch instead
of url-fetch.  I have not checked yet how could be the transition.


> Please also consider #36805 which was never merged or closed.
>  http://issues.guix.gnu.org/issue/36805

This patch could help for option a).


WDYT?

All the best,
simon




^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2020-11-19 14:23 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-03 15:59 bug#39885: Bioconductor URI, fallback and time-machine zimoun
2020-03-23 21:20 ` Ricardo Wurmus
2020-05-21 23:29   ` zimoun
2020-06-24 11:07 ` zimoun
2020-06-28 20:14   ` Ludovic Courtès
2020-06-29 17:36     ` zimoun
2020-06-29 20:42       ` Ludovic Courtès
2020-11-19 14:22 ` zimoun

unofficial mirror of bug-guix@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-bugs/0 guix-bugs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-bugs guix-bugs/ https://yhetil.org/guix-bugs \
		bug-guix@gnu.org
	public-inbox-index guix-bugs

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.bugs
	nntp://news.gmane.io/gmane.comp.gnu.guix.bugs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git