* How can I change the Makeconf file of R to not use hardcoded, wrong include paths when compiling a package with Rcpp
@ 2020-01-13 23:49 Moritz Lell
2020-01-14 19:00 ` zimoun
0 siblings, 1 reply; 4+ messages in thread
From: Moritz Lell @ 2020-01-13 23:49 UTC (permalink / raw)
To: help-guix
Hello!
to reproduce a scientific project, I need a setup with an old R version
(3.4.3), together with other, newer libraries. I created a profile, the
manifest and package definitions are below. However, when loading the
profile I cannot compile R packages with this version:
The last lines of the output of `install.packages("Rcpp_1.0.3.tar.gz",
repos = NULL)` (I have to manually download/use the miniCRAN package
because of SSL errors while downloading via R but that is another question)
--- begin --------------------------------------------------------------
g++ -shared
-L/gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3/lib/R/lib
-L/usr/local/lib -o Rcpp.so api.o attributes.o barrier.o date.o module.o
rcpp_init.o
-L/gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3/lib/R/lib -lR
installing to
/home/moritz/IPK/R/code/Unbalanced/renv/library/R-3.4/x86_64-unknown-linux-gnu/Rcpp/libs
** R
** inst
** preparing package for lazy loading
code for methods in class “C++OverloadedMethods” was not checked for
suspicious field assignments (recommended package ‘codetools’ not
available?)
code for methods in class “RcppClass” was not checked for suspicious
field assignments (recommended package ‘codetools’ not available?)
code for methods in class “RcppClass” was not checked for suspicious
field assignments (recommended package ‘codetools’ not available?)
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error: package or namespace load failed for ‘Rcpp’ in dyn.load(file,
DLLpath = DLLpath, ...):
kann shared object
'/home/user/r-test/renv/library/R-3.4/x86_64-unknown-linux-gnu/Rcpp/libs/Rcpp.so'
nicht laden:
/gnu/store/5imx9c2y6bkd27rprrghnaw59x0svyxb-gfortran-7.4.0-lib/lib/libstdc++.so.6:
version `GLIBCXX_3.4.26' not found (required by
/home/user/r-test/renv/library/R-3.4/x86_64-unknown-linux-gnu/Rcpp/libs/Rcpp.so)
--- end ---------------------------------------------------------------
The problem is `-I/usr/local/include` in the g++ call. It refers to a
directory outside of my GUIX installation and hence I get errors because
libraries there mismatch with those inside GUIX. Apparently, GUIX libs
are used for the linker but not for the compiler.
The problem lies in the file
/gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3/etc/Makeconf
--- begin ------------------------------------------------------------
...
CPPFLAGS = -I/usr/local/include
...
FLIBS =
-L/gnu/store/5imx9c2y6bkd27rprrghnaw59x0svyxb-gfortran-7.4.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.4.0
-L/gnu/store/5imx9c2y6bkd27rprrghnaw59x0svyxb-gfortran-7.4.0-lib/lib
-lgfortran -lm -lquadmath
--- end --------------------------------------------------------------
CPPFLAGS is set to the wrong, hardcoded value, but for example FLIBS is
set correctly. How can I install R such that CPPFLAGS refers to the
/lib/include directory of my profile or doesn't refer to anything?
I can define the PKG_CXXFLAGS environment variable that can be used to
insert additional header locations for R compilation, but unfortunately
they are inserted after the R default locations. But I can use it if the
R default is set to nothing. I've also tested setting CPLUS_INCLUDE_PATH
and PKG_CPPFLAGS, to no avail.
Cheers,
Moritz
==========================================================================
Appendix: Profile manifests and other files
manifest.scm --- manifest file for guix package
packages/ --- include in GUIX_PACKAGE_PATH
packages/statistics.scm --- Checked out from the guix repo with
`git show cbe1314a7e:gnu/packages/statistics.scm` and edited
to only include r-minimal
packages/old-tzdata.scm --- for the package tzdata-2017b
-----------------------------------------------------------------------
# Commands to create the files needed to reproduce the problem
mkdir packages
cat >manifest.scm <<EOF
(specifications->manifest
'("r-minimal@3.4.3"
"glibc-utf8-locales"
"gcc-toolchain"
"gfortran-toolchain"))
EOF
# ----------------------------------------------------------------------
cat >packages/old-tzdata.scm <<EOF
(define-module (old-tz-data)
#:use-module ((guix licenses)
#:select (gpl3+ lgpl2.0+ lgpl3+ public-domain))
#:use-module (guix utils)
#:use-module (guix packages)
#:use-module (guix download)
#:use-module (guix build-system gnu)
#:use-module (guix build-system trivial)
)
(define tzdata
(package
(name "tzdata")
(version "2017b")
(source (origin
(method url-fetch)
(uri (string-append
"https://www.iana.org/time-zones/repository/releases/tzdata"
version ".tar.gz"))
(sha256
(base32
"11l0s43vx33dcs78p80122i8s5s9l1sjwkzzwh66njd35r92l97q"))))
(build-system gnu-build-system)
(arguments
'(#:tests? #f
#:make-flags (let ((out (assoc-ref %outputs "out"))
(tmp (getenv "TMPDIR")))
(list (string-append "TOPDIR=" out)
(string-append "TZDIR=" out "/share/zoneinfo")
;; Discard zic, dump, and tzselect, already
;; provided by glibc.
(string-append "ETCDIR=" tmp "/etc")
;; Likewise for the C library routines.
(string-append "LIBDIR=" tmp "/lib")
(string-append "MANDIR=" tmp "/man")
"AWK=awk"
"CC=gcc"))
#:modules ((guix build utils)
(guix build gnu-build-system)
(srfi srfi-1))
#:phases
(modify-phases %standard-phases
(replace 'unpack
(lambda* (#:key source inputs #:allow-other-keys)
(and (zero? (system* "tar" "xvf" source))
(zero? (system* "tar" "xvf" (assoc-ref inputs
"tzcode"))))))
(add-after 'install 'post-install
(lambda* (#:key outputs #:allow-other-keys)
;; Move data in the right place.
(let ((out (assoc-ref outputs "out")))
(symlink (string-append out "/share/zoneinfo")
(string-append out "/share/zoneinfo/posix"))
(delete-file-recursively
(string-append out "/share/zoneinfo-posix"))
(copy-recursively (string-append out "/share/zoneinfo-leaps")
(string-append out
"/share/zoneinfo/right"))
(delete-file-recursively
(string-append out "/share/zoneinfo-leaps")))))
(delete 'configure))))
(inputs `(("tzcode" ,(origin
(method url-fetch)
(uri (string-append
"http://www.iana.org/time-zones/repository/releases/tzcode"
version ".tar.gz"))
(sha256
(base32
"0h1d567gn8l3iqgyadcswwdy2yh07nhz3lfl8ds8saz2ajxka5sd"))))))
(home-page "https://www.iana.org/time-zones")
(synopsis "Database of current and historical time zones")
(description "The Time Zone Database (often called tz or zoneinfo)
contains code and data that represent the history of local time for many
representative locations around the globe. It is updated periodically to
reflect changes made by political bodies to time zone boundaries, UTC
offsets,
and daylight-saving rules.")
(license public-domain)))
(define-public tzdata-2017a
(package
(inherit tzdata)
(version "2017a")
(source
(origin
(method url-fetch)
(uri (string-append "https://www.iana.org/time-zones/repository"
"/releases/tzdata" version ".tar.gz"))
(sha256
(base32
"1mmv4rvcs12lrvgghw4fidczvb69yv69cmzknghcvw1c196mqfnz"))))
(inputs `(("tzcode" ,(origin
(method url-fetch)
(uri (string-append
"http://www.iana.org/time-zones/repository/releases/tzcode"
version ".tar.gz"))
(sha256
(base32
"1b1q7gnlsh5hjgs5065pvajd37rmbc3k9b8cgzad1vcrifswdwh2"))))))))
EOF
# ----------------------------------------------------------------------
cat >packages/statistics.scm <<EOF
(define-module (statistics)
#:use-module (old-tz-data)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
#:use-module (guix download)
#:use-module (guix hg-download)
#:use-module (guix utils)
#:use-module (guix build-system gnu)
#:use-module (guix build-system r)
#:use-module (guix build-system python)
#:use-module (guix build-system trivial)
#:use-module (gnu packages)
#:use-module (gnu packages check)
#:use-module (gnu packages compression)
#:use-module (gnu packages cran)
#:use-module (gnu packages curl)
#:use-module (gnu packages gcc)
#:use-module (gnu packages gtk)
#:use-module (gnu packages gettext)
#:use-module (gnu packages glib)
#:use-module (gnu packages haskell)
#:use-module (gnu packages icu4c)
#:use-module (gnu packages image)
#:use-module (gnu packages java)
#:use-module (gnu packages machine-learning)
#:use-module (gnu packages maths)
#:use-module (gnu packages multiprecision)
#:use-module (gnu packages pcre)
#:use-module (gnu packages perl)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages python)
#:use-module (gnu packages readline)
#:use-module (gnu packages ssh)
#:use-module (gnu packages texinfo)
#:use-module (gnu packages time)
#:use-module (gnu packages tls)
#:use-module (gnu packages base)
#:use-module (gnu packages web)
#:use-module (gnu packages xml)
#:use-module (gnu packages xorg)
#:use-module (srfi srfi-1))
;; Update this package together with the set of recommended packages:
r-boot,
;; r-class, r-cluster, r-codetools, r-foreign, r-kernsmooth, r-lattice,
;; r-mass, r-matrix, r-mgcv, r-nlme, r-nnet, r-rpart, r-spatial, r-survival.
(define-public r-minimal
(package
(name "r-minimal")
(version "3.4.3")
(source (origin
(method url-fetch)
(uri (string-append "mirror://cran/src/base/R-"
(version-prefix version 1) "/R-"
version ".tar.gz"))
(sha256
(base32
"09pl0w01fr09bsrwd7nz2r5psysj0z93w4chz3hm2havvqqvhg3s"))))
(build-system gnu-build-system)
(arguments
`(#:disallowed-references (,tzdata-2017a)
#:make-flags
(list (string-append "LDFLAGS=-Wl,-rpath="
(assoc-ref %outputs "out")
"/lib/R/lib")
;; This affects the embedded timestamp of only the core
packages.
"PKG_BUILT_STAMP=1970-01-01")
#:phases
(modify-phases %standard-phases
(add-before 'configure 'do-not-compress-serialized-files
(lambda* (#:key inputs #:allow-other-keys)
;; This ensures that Guix can detect embedded store references;
;; see bug #28157 for details.
(substitute* "src/library/base/makebasedb.R"
(("compress = TRUE") "compress = FALSE"))
#t))
(add-before 'configure 'patch-uname
(lambda* (#:key inputs #:allow-other-keys)
(let ((uname-bin (string-append (assoc-ref inputs "coreutils")
"/bin/uname")))
(substitute* "src/scripts/R.sh.in"
(("uname") uname-bin)))
#t))
(add-after 'unpack 'build-reproducibly
(lambda _
;; The documentation contains time stamps to demonstrate
;; documentation generation in different phases.
(substitute* "src/library/tools/man/Rd2HTML.Rd"
(("\\\\%Y-\\\\%m-\\\\%d at \\\\%H:\\\\%M:\\\\%S")
"(removed for reproducibility)"))
;; Remove timestamp from tracing environment. This fixes
;; reproducibility of "methods.rd{b,x}".
(substitute* "src/library/methods/R/trace.R"
(("dateCreated = Sys.time\\(\\)")
"dateCreated = as.POSIXct(\"1970-1-1 00:00:00\", tz =
\"UTC\")"))
;; Ensure that gzipped files are reproducible.
(substitute* '("src/library/grDevices/Makefile.in"
"doc/manual/Makefile.in")
(("R_GZIPCMD\\)" line)
(string-append line " -n")))
;; The "srcfile" procedure in "src/library/base/R/srcfile.R"
;; queries the mtime of a given file and records it in an
object.
;; This is acceptable at runtime to detect stale source files,
;; but it destroys reproducibility at build time.
;;
;; Instead of disabling this feature, which may have unexpected
;; consequences, we reset the mtime of generated files before
;; passing them to the "srcfile" procedure.
(substitute* "src/library/Makefile.in"
(("@\\(cd base && \\$\\(MAKE\\) mkdesc\\)" line)
(string-append line "\n find
$(top_builddir)/library/tools | xargs touch -d '1970-01-01'; \n"))
(("@\\$\\(MAKE\\) Rdobjects" line)
(string-append "@find $(srcdir)/tools | xargs touch -d
'1970-01-01'; \n "
line)))
(substitute* "src/library/tools/Makefile.in"
(("@\\$\\(INSTALL_DATA\\) all.R
\\$\\(top_builddir\\)/library/\\$\\(pkg\\)/R/\\$\\(pkg\\)" line)
(string-append
line
"\n find $(srcdir)/$(pkg)
$(top_builddir)/library/$(pkg) | xargs touch -d \"1970-01-01\"; \n")))
;; This library is installed using
"install_package_description",
;; so we need to pass the "builtStamp" argument.
(substitute* "src/library/tools/Makefile.in"
(("(install_package_description\\(.*\"')\\)\"" line prefix)
(string-append prefix ", builtStamp='1970-01-01')\"")))
;; R bundles an older version of help2man, which does not
respect
;; SOURCE_DATE_EPOCH. We cannot just use the latest help2man,
;; because that breaks a test.
(with-fluids ((%default-port-encoding "ISO-8859-1"))
(substitute* "tools/help2man.pl"
(("my \\$date = strftime \"%B %Y\", localtime" line)
(string-append line " 1"))))
#t))
(add-before 'configure 'set-default-pager
;; Set default pager to "cat", because otherwise it is "false",
;; making "help()" print nothing at all.
(lambda _ (setenv "PAGER" "cat") #t))
(add-before 'check 'set-timezone
;; Some tests require the timezone to be set. However, the
;; timezone may not just be "UTC", or else a brittle regression
;; test in reg-tests-1d will fail.
(lambda* (#:key inputs #:allow-other-keys)
(setenv "TZ" "UTC+1")
(setenv "TZDIR"
(string-append (assoc-ref inputs "tzdata")
"/share/zoneinfo"))
#t))
(add-after 'build 'make-info
(lambda _ (zero? (system* "make" "info"))))
(add-after 'build 'install-info
(lambda _ (zero? (system* "make" "install-info")))))
#:configure-flags
'(;; Do not build the recommended packages. The build system creates
;; random temporary directories and embeds their names in some
;; package files. We build these packages with the r-build-system
;; instead.
"--without-recommended-packages"
"--with-cairo"
"--with-blas=-lopenblas"
"--with-libpng"
"--with-jpeglib"
"--with-libtiff"
"--with-ICU"
"--enable-R-shlib"
"--enable-BLAS-shlib"
"--with-system-tre")))
;; R has some support for Java. When the JDK is available at configure
;; time environment variables pointing to the JDK will be recorded under
;; $R_HOME/etc and ./tools/getsp.java will be compiled which is used
by "R
;; CMD javareconf". "R CMD javareconf" appears to only be used to
update
;; the recorded environment variables in $R_HOME/etc. Refer to
;;
https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Java-support
;; for additional information.
;; As the JDK is a rather large input with only very limited effects
on R,
;; we decided to drop it.
(native-inputs
`(("bzip2" ,bzip2)
("perl" ,perl)
("pkg-config" ,pkg-config)
("texinfo" ,texinfo) ; for building HTML manuals
("tzdata" ,tzdata-2017a)
("xz" ,xz)))
(inputs
`(;; We need not only cairo here, but pango to ensure that tests
for the
;; "cairo" bitmapType plotting backend succeed.
("pango" ,pango)
("coreutils" ,coreutils)
("curl" ,curl)
("openblas" ,openblas)
("gfortran" ,gfortran)
("icu4c" ,icu4c)
("libjpeg" ,libjpeg)
("libpng" ,libpng)
("libtiff" ,libtiff)
("libxt" ,libxt)
("pcre" ,pcre)
("readline" ,readline)
("which" ,which)
("zlib" ,zlib)))
(native-search-paths
(list (search-path-specification
(variable "R_LIBS_SITE")
(files (list "site-library/")))))
(home-page "http://www.r-project.org/")
(synopsis "Environment for statistical computing and graphics")
(description
"R is a language and environment for statistical computing and
graphics.
It provides a variety of statistical techniques, such as linear and
nonlinear
modeling, classical statistical tests, time-series analysis, classification
and clustering. It also provides robust support for producing
publication-quality data plots. A large amount of 3rd-party packages are
available, greatly increasing its breadth and scope.")
(license license:gpl3+)))
EOF
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: How can I change the Makeconf file of R to not use hardcoded, wrong include paths when compiling a package with Rcpp
2020-01-13 23:49 How can I change the Makeconf file of R to not use hardcoded, wrong include paths when compiling a package with Rcpp Moritz Lell
@ 2020-01-14 19:00 ` zimoun
2020-01-16 0:35 ` Moritz Lell
2020-01-16 20:40 ` zimoun
0 siblings, 2 replies; 4+ messages in thread
From: zimoun @ 2020-01-14 19:00 UTC (permalink / raw)
To: Moritz Lell; +Cc: help-guix
Hi Moritz,
Hum? complicated issue... so it is more questions than concrete help. Sorry.
Currently, there is no easy path to achieve your goal, if I understand
correctly.
But that is an interesting question for improving the tools. :-)
On Tue, 14 Jan 2020 at 02:51, Moritz Lell <mlell@posteo.de> wrote:
> to reproduce a scientific project, I need a setup with an old R version
> (3.4.3), together with other, newer libraries. I created a profile, the
> manifest and package definitions are below. However, when loading the
> profile I cannot compile R packages with this version:
How many packages are required by your scientific project?
Only R?
> The last lines of the output of `install.packages("Rcpp_1.0.3.tar.gz",
> repos = NULL)` (I have to manually download/use the miniCRAN package
> because of SSL errors while downloading via R but that is another question)
I am not sure that mixing packages installed by Guix and other by R
itself do help to clean an unreproducible project, IMHO.
To me, packaging all your needs with Guix seems better. You should end
up with your own package definitions living in a Git repo (channel)
and a manifest file describing how to rebuild everything.
However, the issue is: you use an old R version with recent packages.
Not usual. ;-)
The naive solution is to re-write the dependency graph by replacing
the current r-minimal@3.6.2 with the old one r-minimal@3.4.3. But it
does not work. Sadly! The implicit inputs cannot be re-written on the
fly with the '--with-inputs' option.
Now, I list some steps that you have already done just in case others
want to reproduce and/or help. :-)
(Note the Git commit commands are launched where the Git repo was
cloned. And the Guix command are launched in /tmp/t.)
1. Find the version 3.4.3 in Guix:
git log --oneline | grep r-minimal
--8<---------------cut here---------------start------------->8---
d22546cf80 gnu: r-minimal: Update to 3.5.0.
3146aab7a8 gnu: r-minimal: Add tcl/tk support.
c43ea99b39 gnu: r-minimal: Update to 3.4.4.
a768e41523 gnu: r-minimal, r: Update to 3.4.4.
9a7b578463 gnu: r-minimal: Add bash to inputs.
0fa4702c04 gnu: r-minimal: Respect SOURCE_DATE_EPOCH in srcfile.
cbe1314a7e gnu: Update r-minimal to 3.4.3.
1344deb937 gnu: r-minimal: Update to 3.4.2.
ebbb6301a3 gnu: r-minimal: Update to 3.4.1.
174fbd5f3b gnu: r-minimal: Remove timestamp from man page.
bd3a184613 gnu: r-minimal: Do not compress serialized files.
a8cd352304 gnu: r-minimal: Work around failure to embed reference to "which".
269504a797 gnu: python-pbr-minimal: Update to 3.0.1
a71d769d1e gnu: r-minimal: Update to 3.4.0.
60c9190e21 gnu: r-minimal: Fix remaining reproducibility problems.
2d7c4ae3ee gnu: r: Rename to r-minimal.
--8<---------------cut here---------------end--------------->8---
Well, we need any commit before the "Update to 3.4.4." (a768e41523),
so for example the parent:
git log --pretty=%P -n 1 a768e41523
is the commit: 38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd.
2. Try the time machine with the commit
38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd.
Possible resources consuming... everything will be compiled as it was
in Guix at the time (March 2018) so some substitutes are not available
any more.
guix time-machine --commit=38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd \
-- environment --container --ad-hoc r-minimal -- R --version
3. In the meantime, let compile the version 3.4.3 with the current
Guix (commit 2aeca24).
git show 38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd:gnu/packages/statistics.scm
\
> /tmp/t/old-statistics.scm
# edit the module name
guix build -L . r-minimal@3.4.3
and the output is: /gnu/store/3wagrrl9gzb5p1an9i5xbm9kp3km4zim-r-minimal-3.4.3.
Compared to your output
gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3, one
difference is the tzdata package which is different, I guess.
4. Speaking about reproducibility, all the toolchain matters. :-)
guix time-machine --commit=14c272d5ce -- build -L . r-minimal@3.4.3
outputs /gnu/store/n0py7ixdqkim1lx9iis6xlr2mn7l897f-r-minimal-3.4.3.
So for example compare:
guix time-machine --commit=2aeca24 \
-- environment --ad-hoc gcc-toolchain -- gcc --version
guix time-machine --commit=14c272d5ce \
-- environment --ad-hoc gcc-toolchain -- gcc --version
And let aside the grafting system. ;-)
To be concrete, compare
# GCC 8.3.0
guix time-machine --commit=14c272d5ce -- build -L . r-minimal@3.4.3
# -> /gnu/store/n0py7ixdqkim1lx9iis6xlr2mn7l897f-r-minimal-3.4.3
guix time-machine --commit=14c272d5ce \
-- environment --container -L . --ad-hoc r-minimal@3.4.3 -- R --version
# GCC 9.2.0
guix time-machine --commit=2aeca24 -- build -L . r-minimal@3.4.3
# -> /gnu/store/3wagrrl9gzb5p1an9i5xbm9kp3km4zim-r-minimal-3.4.3
guix time-machine --commit=2aeca24 \
-- environment --container -L . --ad-hoc r-minimal@3.4.3 -- R --version
Well, my point is just to be clear about what needs to be fixed to end
up with something reproducible.
> The problem lies in the file
> /gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3/etc/Makeconf
You mean:
/gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3/lib/R/etc/Makeconf
?
> CPPFLAGS = -I/usr/local/include
Hum? need more investigations... I have no idea...
> CPPFLAGS is set to the wrong, hardcoded value, but for example FLIBS is
> set correctly. How can I install R such that CPPFLAGS refers to the
> /lib/include directory of my profile or doesn't refer to anything?
It appears to me odd that it is hard coded...
> I can define the PKG_CXXFLAGS environment variable that can be used to
> insert additional header locations for R compilation, but unfortunately
> they are inserted after the R default locations. But I can use it if the
> R default is set to nothing. I've also tested setting CPLUS_INCLUDE_PATH
> and PKG_CPPFLAGS, to no avail.
Instead, I propose to overwite the r-rcpp package by adding an
explicit reference to r-minimal and so to be able to use the option
'--with-input'.
Something like:
--8<---------------cut here---------------start------------->8---
(define-module (explicit-pkgs)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
#:use-module ((gnu packages cran) #:select (r-rcpp))
#:use-module (old-statistics)) ;need the symbol r-minimal
;;; #:use-module (gnu packages statistics) ;or here
(define-public r-rcpp-explicit
(package
(inherit r-rcpp)
(version "mine")
(inputs
`(("r-minimal" ,r-minimal)))))
--8<---------------cut here---------------end--------------->8---
Then,
guix environment --container \
-L . --with-input=r-minimal=r-minimal@3.4.3 \
--ad-hoc r-minimal r-rcpp@mine \
-- R
should spawn a R prompt using the version 3.4.3 with Rcpp inside.
But it is not working properly:
--8<---------------cut here---------------start------------->8---
> library(Rcpp)
Error in readRDS(pfile) :
cannot read workspace version 3 written by R 3.6.2; need R 3.5.0 or newer
--8<---------------cut here---------------end--------------->8---
I do not know why... need more investigations. :-)
I hope the strategy makes sense.
Well,
a) I do not know if it the good strategy; it is what I am thinking of;
b) something easier is maybe possible;
c) today, there is a discussion on IRC to see how it should be
possible to add "something" for changing the R version (e.g., see how
Python packages do with 2 vs 3).
What do you think?
All the best,
simon
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: How can I change the Makeconf file of R to not use hardcoded, wrong include paths when compiling a package with Rcpp
2020-01-14 19:00 ` zimoun
@ 2020-01-16 0:35 ` Moritz Lell
2020-01-16 20:40 ` zimoun
1 sibling, 0 replies; 4+ messages in thread
From: Moritz Lell @ 2020-01-16 0:35 UTC (permalink / raw)
To: zimoun; +Cc: help-guix
[-- Attachment #1: Type: text/plain, Size: 11362 bytes --]
Hi!
thank you for your suggestions... they help me understand the "spirit"
of this program, it is still quite new for me.
While you're right that a manifest of R and all packages is the best
way, I will still need R to compile something from time to time, for
example C++ code from within my project.
I have already found out some knobs in the R installation but I'm not
done yet. I need someone with more experience with C++ linking and
including flags to help me for this to work.
In the [R-Admin
manual](https://cran.r-project.org/doc/manuals/r-release/R-admin.pdf),
B.3.3 (p.53) they explain that they set default flags if certain
environment variables are not set.
The Nix people seemingly had similar problems and they've come up with
some shims where they set more environment variables than we do.
https://github.com/NixOS/nixpkgs/blob/eb65d1dae626f4b149566c4cbccdad7ec24af189/pkgs/applications/science/math/R/default.nix#L64
From there:
-- begin ---------------------------------------------------------------
preConfigure = ''
configureFlagsArray=(
--disable-lto
[...]
R_SHELL="${stdenv.shell}"
'' + stdenv.lib.optionalString stdenv.isDarwin ''
--without-tcltk
--without-aqua
--disable-R-framework
OBJC="clang"
CPPFLAGS="-isystem ${libcxx}/include/c++/v1" # <<<----
LDFLAGS="-L${libcxx}/lib" # <<<----
'' + ''
)
echo >>etc/Renviron.in "TCLLIBPATH=${tk}/lib" # <<<----
echo >>etc/Renviron.in "TZDIR=${tzdata}/share/zoneinfo"
'';
-- end -----------------------------------------------------------------
I tried to translate this into R with a patch versus the guix git repo,
commit 40b1cee620a55bf1fc5d8d897ed1ec147b2535c8 that I have attached to
this mail. However, I think I got some include paths wrong because I get
errors when starting R:
--- begin --------------------------------------------------------------
$ LANG=en_US.UTF-8
/gnu/store/3k62212z7ljmqaqa2vf2nfvjn2rw7q15-r-minimal-3.6.2/bin/R
R version 3.6.2 (2019-12-12) -- "Dark and Stormy Night"
[...]
Error: package or namespace load failed for ‘utils’:
.onLoad failed in loadNamespace() for 'utils', details:
call: switch(os, Linux = if (file.exists("/etc/os-release")) {
error: EXPR must be a length 1 vector
Error: package or namespace load failed for ‘stats’:
.onLoad failed in loadNamespace() for 'utils', details:
call: switch(os, Linux = if (file.exists("/etc/os-release")) {
error: EXPR must be a length 1 vector
During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
>
--- end ----------------------------------------------------------------
Are the paths that I set in the attached commit correct?
guix describe
guix 4aea90b
Repository-URL: https://git.savannah.gnu.org/git/guix.git
Branch: master
Commit: 4aea90b1876179aab8d603a42533a6bdf97ccd3c
Best,
Moritz
On 14.01.20 20:00, zimoun wrote:
> Hi Moritz,
>
> Hum? complicated issue... so it is more questions than concrete help. Sorry.
>
> Currently, there is no easy path to achieve your goal, if I understand
> correctly.
> But that is an interesting question for improving the tools. :-)
>
>
> On Tue, 14 Jan 2020 at 02:51, Moritz Lell <mlell@posteo.de> wrote:
>
>> to reproduce a scientific project, I need a setup with an old R version
>> (3.4.3), together with other, newer libraries. I created a profile, the
>> manifest and package definitions are below. However, when loading the
>> profile I cannot compile R packages with this version:
>
> How many packages are required by your scientific project?
> Only R?
>
>
>> The last lines of the output of `install.packages("Rcpp_1.0.3.tar.gz",
>> repos = NULL)` (I have to manually download/use the miniCRAN package
>> because of SSL errors while downloading via R but that is another question)
>
> I am not sure that mixing packages installed by Guix and other by R
> itself do help to clean an unreproducible project, IMHO.
> To me, packaging all your needs with Guix seems better. You should end
> up with your own package definitions living in a Git repo (channel)
> and a manifest file describing how to rebuild everything.
>
> However, the issue is: you use an old R version with recent packages.
> Not usual. ;-)
>
> The naive solution is to re-write the dependency graph by replacing
> the current r-minimal@3.6.2 with the old one r-minimal@3.4.3. But it
> does not work. Sadly! The implicit inputs cannot be re-written on the
> fly with the '--with-inputs' option.
>
>
> Now, I list some steps that you have already done just in case others
> want to reproduce and/or help. :-)
>
> (Note the Git commit commands are launched where the Git repo was
> cloned. And the Guix command are launched in /tmp/t.)
>
>
> 1. Find the version 3.4.3 in Guix:
>
> git log --oneline | grep r-minimal
>
> --8<---------------cut here---------------start------------->8---
> d22546cf80 gnu: r-minimal: Update to 3.5.0.
> 3146aab7a8 gnu: r-minimal: Add tcl/tk support.
> c43ea99b39 gnu: r-minimal: Update to 3.4.4.
> a768e41523 gnu: r-minimal, r: Update to 3.4.4.
> 9a7b578463 gnu: r-minimal: Add bash to inputs.
> 0fa4702c04 gnu: r-minimal: Respect SOURCE_DATE_EPOCH in srcfile.
> cbe1314a7e gnu: Update r-minimal to 3.4.3.
> 1344deb937 gnu: r-minimal: Update to 3.4.2.
> ebbb6301a3 gnu: r-minimal: Update to 3.4.1.
> 174fbd5f3b gnu: r-minimal: Remove timestamp from man page.
> bd3a184613 gnu: r-minimal: Do not compress serialized files.
> a8cd352304 gnu: r-minimal: Work around failure to embed reference to "which".
> 269504a797 gnu: python-pbr-minimal: Update to 3.0.1
> a71d769d1e gnu: r-minimal: Update to 3.4.0.
> 60c9190e21 gnu: r-minimal: Fix remaining reproducibility problems.
> 2d7c4ae3ee gnu: r: Rename to r-minimal.
> --8<---------------cut here---------------end--------------->8---
>
>
> Well, we need any commit before the "Update to 3.4.4." (a768e41523),
> so for example the parent:
>
> git log --pretty=%P -n 1 a768e41523
>
> is the commit: 38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd.
>
>
> 2. Try the time machine with the commit
> 38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd.
> Possible resources consuming... everything will be compiled as it was
> in Guix at the time (March 2018) so some substitutes are not available
> any more.
>
> guix time-machine --commit=38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd \
> -- environment --container --ad-hoc r-minimal -- R --version
>
>
> 3. In the meantime, let compile the version 3.4.3 with the current
> Guix (commit 2aeca24).
>
> git show 38ef437b0c08d8568d6cb1cb0ff1f5c9a66336dd:gnu/packages/statistics.scm
> \
> > /tmp/t/old-statistics.scm
> # edit the module name
> guix build -L . r-minimal@3.4.3
>
> and the output is: /gnu/store/3wagrrl9gzb5p1an9i5xbm9kp3km4zim-r-minimal-3.4.3.
>
>
> Compared to your output
> gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3, one
> difference is the tzdata package which is different, I guess.
>
>
> 4. Speaking about reproducibility, all the toolchain matters. :-)
>
> guix time-machine --commit=14c272d5ce -- build -L . r-minimal@3.4.3
>
> outputs /gnu/store/n0py7ixdqkim1lx9iis6xlr2mn7l897f-r-minimal-3.4.3.
>
>
> So for example compare:
>
> guix time-machine --commit=2aeca24 \
> -- environment --ad-hoc gcc-toolchain -- gcc --version
>
> guix time-machine --commit=14c272d5ce \
> -- environment --ad-hoc gcc-toolchain -- gcc --version
>
> And let aside the grafting system. ;-)
>
>
> To be concrete, compare
>
> # GCC 8.3.0
> guix time-machine --commit=14c272d5ce -- build -L . r-minimal@3.4.3
>
> # -> /gnu/store/n0py7ixdqkim1lx9iis6xlr2mn7l897f-r-minimal-3.4.3
>
> guix time-machine --commit=14c272d5ce \
> -- environment --container -L . --ad-hoc r-minimal@3.4.3 -- R --version
>
>
> # GCC 9.2.0
> guix time-machine --commit=2aeca24 -- build -L . r-minimal@3.4.3
>
> # -> /gnu/store/3wagrrl9gzb5p1an9i5xbm9kp3km4zim-r-minimal-3.4.3
>
> guix time-machine --commit=2aeca24 \
> -- environment --container -L . --ad-hoc r-minimal@3.4.3 -- R --version
>
>
> Well, my point is just to be clear about what needs to be fixed to end
> up with something reproducible.
>
>
>
>> The problem lies in the file
>> /gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3/etc/Makeconf
>
> You mean:
> /gnu/store/d64gfblqc5b9gmwj3rbl8nx3zwbf973b-r-minimal-3.4.3/lib/R/etc/Makeconf
> ?
>
>
>> CPPFLAGS = -I/usr/local/include
>
> Hum? need more investigations... I have no idea...
>
>
>> CPPFLAGS is set to the wrong, hardcoded value, but for example FLIBS is
>> set correctly. How can I install R such that CPPFLAGS refers to the
>> /lib/include directory of my profile or doesn't refer to anything?
>
> It appears to me odd that it is hard coded...
>
>
>> I can define the PKG_CXXFLAGS environment variable that can be used to
>> insert additional header locations for R compilation, but unfortunately
>> they are inserted after the R default locations. But I can use it if the
>> R default is set to nothing. I've also tested setting CPLUS_INCLUDE_PATH
>> and PKG_CPPFLAGS, to no avail.
>
> Instead, I propose to overwite the r-rcpp package by adding an
> explicit reference to r-minimal and so to be able to use the option
> '--with-input'.
>
> Something like:
>
> --8<---------------cut here---------------start------------->8---
> (define-module (explicit-pkgs)
> #:use-module ((guix licenses) #:prefix license:)
> #:use-module (guix packages)
> #:use-module ((gnu packages cran) #:select (r-rcpp))
> #:use-module (old-statistics)) ;need the symbol r-minimal
> ;;; #:use-module (gnu packages statistics) ;or here
>
>
> (define-public r-rcpp-explicit
> (package
> (inherit r-rcpp)
> (version "mine")
> (inputs
> `(("r-minimal" ,r-minimal)))))
> --8<---------------cut here---------------end--------------->8---
>
> Then,
>
> guix environment --container \
> -L . --with-input=r-minimal=r-minimal@3.4.3 \
> --ad-hoc r-minimal r-rcpp@mine \
> -- R
>
> should spawn a R prompt using the version 3.4.3 with Rcpp inside.
>
> But it is not working properly:
>
> --8<---------------cut here---------------start------------->8---
>> library(Rcpp)
> Error in readRDS(pfile) :
> cannot read workspace version 3 written by R 3.6.2; need R 3.5.0 or newer
> --8<---------------cut here---------------end--------------->8---
>
> I do not know why... need more investigations. :-)
>
>
> I hope the strategy makes sense.
>
>
> Well,
> a) I do not know if it the good strategy; it is what I am thinking of;
> b) something easier is maybe possible;
> c) today, there is a discussion on IRC to see how it should be
> possible to add "something" for changing the R version (e.g., see how
> Python packages do with 2 vs 3).
>
> What do you think?
>
>
> All the best,
> simon
>
[-- Attachment #2: 0001-gnu-r-minimal-Prevent-absolute-paths-in-lib-R-etc-Ma.patch --]
[-- Type: text/x-patch, Size: 2018 bytes --]
From 10635e96a2c78340c0f892572bea1491a534f1a7 Mon Sep 17 00:00:00 2001
From: Moritz Lell <mlell08@gmail.com>
Date: Thu, 16 Jan 2020 01:17:07 +0100
Subject: [PATCH] gnu: r-minimal: Prevent absolute paths in lib/R/etc/Makevars
Defining the variables CPPFLAGS, LDFLAGS and TCLLIBPATH prevents the R
build system from using wrong defaults like CPPFLAGS =
-I/usr/local/include
---
gnu/packages/statistics.scm | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/gnu/packages/statistics.scm b/gnu/packages/statistics.scm
index e7fd5bd084..ed1f1e56ed 100644
--- a/gnu/packages/statistics.scm
+++ b/gnu/packages/statistics.scm
@@ -273,6 +273,17 @@ as.POSIXct(if (\"\" != Sys.getenv(\"SOURCE_DATE_EPOCH\")) {\
;; Set default pager to "cat", because otherwise it is "false",
;; making "help()" print nothing at all.
(lambda _ (setenv "PAGER" "cat") #t))
+ (add-before 'configure 'prevent-include-defaults
+ (lambda* (#:key inputs #:allow-other-keys)
+ (setenv "CPPFLAGS" (string-append
+ "-isystem " (assoc-ref inputs "glibc") "/include"))
+ (setenv "LDFLAGS" (string-append
+ "-L" (assoc-ref inputs "glibc") "/lib"))
+ ;(setenv "TCLTK_LIBS" "") ; still sets -L/usr/lib
+ (setenv "TCLLIBPATH" (string-append
+ (assoc-ref inputs "tcl") "/lib"))
+ ; (setenv "TCLTK_CPPFLAGS" "") ; still sets -L/usr/lib -ltcl8.6 -L/usr/lib -ltk8.6 -lX11 -lXss -lXext
+ #t))
(add-before 'check 'set-timezone
;; Some tests require the timezone to be set. However, the
;; timezone may not just be "UTC", or else a brittle regression
@@ -341,6 +352,7 @@ as.POSIXct(if (\"\" != Sys.getenv(\"SOURCE_DATE_EPOCH\")) {\
("coreutils" ,coreutils)
("curl" ,curl)
("openblas" ,openblas)
+ ("glibc", glibc)
("gfortran" ,gfortran)
("icu4c" ,icu4c)
("libjpeg" ,libjpeg)
--
2.24.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: How can I change the Makeconf file of R to not use hardcoded, wrong include paths when compiling a package with Rcpp
2020-01-14 19:00 ` zimoun
2020-01-16 0:35 ` Moritz Lell
@ 2020-01-16 20:40 ` zimoun
1 sibling, 0 replies; 4+ messages in thread
From: zimoun @ 2020-01-16 20:40 UTC (permalink / raw)
To: Moritz Lell; +Cc: help-guix
On Tue, 14 Jan 2020 at 20:00, zimoun <zimon.toutoune@gmail.com> wrote:
> But it is not working properly:
>
> --8<---------------cut here---------------start------------->8---
> > library(Rcpp)
> Error in readRDS(pfile) :
> cannot read workspace version 3 written by R 3.6.2; need R 3.5.0 or newer
> --8<---------------cut here---------------end--------------->8---
>
> I do not know why... need more investigations. :-)
It is because the package 'r-rcpp@mine' has been built using
'r-minimal-3.6.2'. Examining the building log:
--8<---------------cut here---------------start------------->8---
g++ -std=gnu++11 -I"/gnu/store/...-r-minimal-3.6.2/lib/R/include"
-DNDEBUG -I../inst/include/ -I/usr/local/include -fpic -g -O2 -c
api.cpp -o api.o
g++ -std=gnu++11 -I"/gnu/store/...-r-minimal-3.6.2/lib/R/include"
-DNDEBUG -I../inst/include/ -I/usr/local/include -fpic -g -O2 -c
attributes.cpp -o attributes.o
--8<---------------cut here---------------end--------------->8---
> Well,
> a) I do not know if it the good strategy; it is what I am thinking of;
The fix seems to implement kind of 'package-with-explicit' for R. I
have just tried now and I failed. Keep you in touch. :-)
All the best,
simon
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-01-16 20:40 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-13 23:49 How can I change the Makeconf file of R to not use hardcoded, wrong include paths when compiling a package with Rcpp Moritz Lell
2020-01-14 19:00 ` zimoun
2020-01-16 0:35 ` Moritz Lell
2020-01-16 20:40 ` zimoun
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.