[PATCH] gnu: Add mongodb.

[PATCH] gnu: Add mongodb.

[Roel Janssen]

[-- Attachment #1: 0001-gnu-Add-mongodb.patch --]
[-- Type: text/x-patch, Size: 4982 bytes --]

From 47d5ad30c1e1e2a24b950236590e10bbf72c277e Mon Sep 17 00:00:00 2001
From: Roel Janssen <roel@gnu.org>
Date: Thu, 17 Mar 2016 16:06:40 +0100
Subject: [PATCH] gnu: Add mongodb.

* gnu/packages/databases.scm (mongodb): New variable.
* gnu/packages/patches/mongodb-add-version-file.patch: New file.
* gnu-system.am (dist_patch_DATA): Add patch file.
---
 gnu-system.am                                      |  1 +
 gnu/packages/databases.scm                         | 51 +++++++++++++++++++++-
 .../patches/mongodb-add-version-file.patch         | 10 +++++
 3 files changed, 60 insertions(+), 2 deletions(-)
 create mode 100644 gnu/packages/patches/mongodb-add-version-file.patch

diff --git a/gnu-system.am b/gnu-system.am
index 634093a..7517cc0 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -633,6 +633,7 @@ dist_patch_DATA =						\
   gnu/packages/patches/mpc123-initialize-ao.patch		\
   gnu/packages/patches/mplayer2-theora-fix.patch		\
   gnu/packages/patches/module-init-tools-moduledir.patch	\
+  gnu/packages/patches/mongodb-add-version-file.patch		\
   gnu/packages/patches/mumps-build-parallelism.patch		\
   gnu/packages/patches/mupen64plus-ui-console-notice.patch	\
   gnu/packages/patches/mutt-store-references.patch		\
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 755bf4f..a3a3744 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2015 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2016 Nils Gillmann <niasterisk@grrlz.net>
+;;; Copyright © 2016 Roel Janssen <roel@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -46,8 +47,8 @@
   #:use-module (gnu packages bison)
   #:use-module (gnu packages jemalloc)
   #:use-module ((guix licenses)
-                #:select (gpl2 gpl3+ lgpl2.1+ lgpl3+ x11-style non-copyleft
-                          bsd-2 bsd-3 public-domain))
+                #:select (asl2.0 agpl3 gpl2 gpl3+ lgpl2.1+ lgpl3+ x11-style
+                          non-copyleft bsd-2 bsd-3 public-domain))
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
@@ -271,6 +272,52 @@ Language.")
 as a drop-in replacement of MySQL.")
     (license gpl2)))
 
+(define-public mongodb
+  (package
+    (name "mongodb")
+    (version "3.3.3")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://github.com/mongodb/mongo/archive/r"
+                                  version ".tar.gz"))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32 "01v16j8xbry7m7apwlhiqhgx3zyyk2kadyk2sr9m6k20wnh5j24y"))
+              (patches (list (search-patch "mongodb-add-version-file.patch")))))
+    (build-system gnu-build-system)
+    (native-inputs
+     `(("scons" ,scons)
+       ("python" ,python-2)
+       ("perl" ,perl)))
+    (arguments
+     `(#:tests? #f ; There is no 'check' target.
+       #:phases
+       (modify-phases %standard-phases
+         (delete 'configure) ; There is no configure phase
+         (add-after 'unpack 'scons-propagate-environment
+           (lambda _
+             ;; Modify the SConstruct file to arrange for
+             ;; environment variables to be propagated.
+             (substitute* "SConstruct"
+               (("^env = Environment\\(")
+                "env = Environment(ENV=os.environ, "))))
+         (replace 'build
+           (lambda _
+             (zero? (system* "scons" "mongod" "mongo" "mongos"))))
+         (replace 'install
+           (lambda _
+             (let ((bin  (string-append (assoc-ref %outputs "out") "/bin")))
+               (install-file "mongod" bin)
+               (install-file "mongos" bin)
+               (install-file "mongo" bin)))))))
+    (home-page "https://www.mongodb.org/")
+    (synopsis "High performance and high availability document database")
+    (description "Mongo is a high-performance, high availability,
+schema-free document-oriented database.  A key goal of MongoDB is to bridge
+the gap between key/value stores (which are fast and highly scalable) and
+traditional RDBMS systems (which are deep in functionality).")
+    (license (list agpl3 asl2.0))))
+
 (define-public postgresql
   (package
     (name "postgresql")
diff --git a/gnu/packages/patches/mongodb-add-version-file.patch b/gnu/packages/patches/mongodb-add-version-file.patch
new file mode 100644
index 0000000..ffcb70e
--- /dev/null
+++ b/gnu/packages/patches/mongodb-add-version-file.patch
@@ -0,0 +1,10 @@
+This patch is used to replace the need for git.  It's a work-around
+for a problem described here: https://jira.mongodb.org/browse/SERVER-21317
+
+Patch by Roel Janssen <roel@gnu.org>
+--- a/version.json	1970-01-01 01:00:00.000000000 +0100
++++ b/version.json	2016-03-16 15:10:23.933578071 +0100
+@@ -0,0 +1,3 @@
++{
++    "version": "3.3.3-guix-"
++}
-- 
2.5.0


[-- Attachment #2: Type: text/plain, Size: 193 bytes --]

Dear Guix,

This is a patch to add MongoDB (server and client) tools.  I used a
three-line patch to fix the build process's reliance on Git and the
.git/ directory.

Kind regards,
Roel Janssen

Re: [PATCH] gnu: Add mongodb.

[Leo Famulari]

On Thu, Mar 17, 2016 at 04:08:24PM +0100, Roel Janssen wrote:
> Subject: [PATCH] gnu: Add mongodb.

Thanks for this patch!

> * gnu/packages/databases.scm (mongodb): New variable.
> * gnu/packages/patches/mongodb-add-version-file.patch: New file.
> * gnu-system.am (dist_patch_DATA): Add patch file.

Okay.

[...]

> +(define-public mongodb
> +  (package
> +    (name "mongodb")
> +    (version "3.3.3")
> +    (source (origin
> +              (method url-fetch)
> +              (uri (string-append "https://github.com/mongodb/mongo/archive/r"
> +                                  version ".tar.gz"))

Do you know if GitHub is the preferred or canonical source for mongodb
tarballs? Okay if so, but often the GitHub tarballs are auto-generated
snapshots, while the upstream project has specially prepared
distribution tarballs from their website.

I see a tarball here:
https://www.mongodb.org/downloads

Did you compare them? If not, will you do that and decide which is more
appropriate?

> +              (file-name (string-append name "-" version ".tar.gz"))
> +              (sha256
> +               (base32 "01v16j8xbry7m7apwlhiqhgx3zyyk2kadyk2sr9m6k20wnh5j24y"))
> +              (patches (list (search-patch "mongodb-add-version-file.patch")))))
> +    (build-system gnu-build-system)
> +    (native-inputs
> +     `(("scons" ,scons)
> +       ("python" ,python-2)
> +       ("perl" ,perl)))
> +    (arguments
> +     `(#:tests? #f ; There is no 'check' target.

Wow, I would have expected a project like this to have a test suite.

> +       #:phases
> +       (modify-phases %standard-phases
> +         (delete 'configure) ; There is no configure phase
> +         (add-after 'unpack 'scons-propagate-environment
> +           (lambda _
> +             ;; Modify the SConstruct file to arrange for
> +             ;; environment variables to be propagated.
> +             (substitute* "SConstruct"
> +               (("^env = Environment\\(")
> +                "env = Environment(ENV=os.environ, "))))

So, this saves the environment of the build process and propagates it to
run-time?

> +         (replace 'build
> +           (lambda _
> +             (zero? (system* "scons" "mongod" "mongo" "mongos"))))
> +         (replace 'install
> +           (lambda _
> +             (let ((bin  (string-append (assoc-ref %outputs "out") "/bin")))
> +               (install-file "mongod" bin)
> +               (install-file "mongos" bin)
> +               (install-file "mongo" bin)))))))

No libraries or documentation?

> +    (home-page "https://www.mongodb.org/")
> +    (synopsis "High performance and high availability document database")
> +    (description "Mongo is a high-performance, high availability,
> +schema-free document-oriented database.  A key goal of MongoDB is to bridge
> +the gap between key/value stores (which are fast and highly scalable) and
> +traditional RDBMS systems (which are deep in functionality).")
> +    (license (list agpl3 asl2.0))))
> +
>  (define-public postgresql
>    (package
>      (name "postgresql")
> diff --git a/gnu/packages/patches/mongodb-add-version-file.patch b/gnu/packages/patches/mongodb-add-version-file.patch
> new file mode 100644
> index 0000000..ffcb70e
> --- /dev/null
> +++ b/gnu/packages/patches/mongodb-add-version-file.patch
> @@ -0,0 +1,10 @@
> +This patch is used to replace the need for git.  It's a work-around
> +for a problem described here: https://jira.mongodb.org/browse/SERVER-21317
> +
> +Patch by Roel Janssen <roel@gnu.org>
> +--- a/version.json	1970-01-01 01:00:00.000000000 +0100
> ++++ b/version.json	2016-03-16 15:10:23.933578071 +0100
> +@@ -0,0 +1,3 @@
> ++{
> ++    "version": "3.3.3-guix-"
> ++}

It's okay that this is different from the "version" field in the package
definition? Just checking...

> -- 
> 2.5.0
> 

> Dear Guix,
> 
> This is a patch to add MongoDB (server and client) tools.  I used a
> three-line patch to fix the build process's reliance on Git and the
> .git/ directory.
> 
> Kind regards,
> Roel Janssen

Re: [PATCH] gnu: Add mongodb.

[Thompson, David]

Hi Roel,

2016-03-17 11:08 GMT-04:00 Roel Janssen <roel@gnu.org>:
> Dear Guix,
>
> This is a patch to add MongoDB (server and client) tools.  I used a
> three-line patch to fix the build process's reliance on Git and the
> .git/ directory.

Sorry to be the bearer of bad news, but I became suspicious about this
package recipe when I noticed that it had no inputs.  Unfortunately,
MongoDB is one of those projects that bundles the source code for all
of their dependencies in their release tarballs. Here's a list from
the src/third_party directory:

asio-asio-1-11-0
boost-1.56.0
gperftools-2.2
IntelRDFPMathLib20U1
libstemmer_c
mozjs-38
murmurhash3
pcre-8.36
pcre-8.37
s2
SConscript
shim_allocator.cpp
shim_asio.cpp
shim_boost.cpp
shim_intel_decimal128.cpp
shim_mozjs.cpp
shim_pcrecpp.cc
shim_snappy.cpp
shim_stemmer.cpp
shim_tz.cpp
shim_v8.cpp
shim_wiredtiger.cpp
shim_yaml.cpp
shim_zlib.cpp
snappy-1.1.2
tz
unicode-8.0.0
valgrind-3.11.0
wiredtiger
yaml-cpp-0.5.1
zlib-1.2.8

This is a really bad practice from a security and maintenance
perspective, so in Guix we do our best to unbundle as many things as
possible.  I recommend deleting everything above besides the "shim_*"
files using a snippet in the origin record (search for "snippet" in
gnu/packages for examples) and seeing if its possible to use
system-provided dependencies for everything.  From a quick look,
you'll probably have to package an additional library or two.

Thanks and sorry!

- Dave

Re: [PATCH] gnu: Add mongodb.

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> This is a really bad practice from a security and maintenance
> perspective, so in Guix we do our best to unbundle as many things as
> possible.  I recommend deleting everything above besides the "shim_*"
> files using a snippet in the origin record (search for "snippet" in
> gnu/packages for examples) and seeing if its possible to use
> system-provided dependencies for everything.

+1

(And sorry for the bad news, indeed.  ;-))

Ludo’.

Re: [PATCH] gnu: Add mongodb.

[Roel Janssen]

Hello David,

Thanks for your feedback, and sorry for my incomplete patch.

My reasoning for keeping the third-party libraries in the tarball was
simply because upstream made that choice.  Unbundling can and probably
will introduce differences in the behavior of the output with other
distributions and their official binaries.

I will try to unbundle these libraries from the release tarball.  This
could take some time though.

Again, sorry for the noise.

Kind regards,
Roel Janssen


Thompson, David writes:

> Hi Roel,
>
> 2016-03-17 11:08 GMT-04:00 Roel Janssen <roel@gnu.org>:
>> Dear Guix,
>>
>> This is a patch to add MongoDB (server and client) tools.  I used a
>> three-line patch to fix the build process's reliance on Git and the
>> .git/ directory.
>
> Sorry to be the bearer of bad news, but I became suspicious about this
> package recipe when I noticed that it had no inputs.  Unfortunately,
> MongoDB is one of those projects that bundles the source code for all
> of their dependencies in their release tarballs. Here's a list from
> the src/third_party directory:
>
> asio-asio-1-11-0
> boost-1.56.0
> gperftools-2.2
> IntelRDFPMathLib20U1
> libstemmer_c
> mozjs-38
> murmurhash3
> pcre-8.36
> pcre-8.37
> s2
> SConscript
> shim_allocator.cpp
> shim_asio.cpp
> shim_boost.cpp
> shim_intel_decimal128.cpp
> shim_mozjs.cpp
> shim_pcrecpp.cc
> shim_snappy.cpp
> shim_stemmer.cpp
> shim_tz.cpp
> shim_v8.cpp
> shim_wiredtiger.cpp
> shim_yaml.cpp
> shim_zlib.cpp
> snappy-1.1.2
> tz
> unicode-8.0.0
> valgrind-3.11.0
> wiredtiger
> yaml-cpp-0.5.1
> zlib-1.2.8
>
> This is a really bad practice from a security and maintenance
> perspective, so in Guix we do our best to unbundle as many things as
> possible.  I recommend deleting everything above besides the "shim_*"
> files using a snippet in the origin record (search for "snippet" in
> gnu/packages for examples) and seeing if its possible to use
> system-provided dependencies for everything.  From a quick look,
> you'll probably have to package an additional library or two.
>
> Thanks and sorry!
>
> - Dave

Re: [PATCH] gnu: Add mongodb.

[Ludovic Courtès]

Hi!

Roel Janssen <roel@gnu.org> skribis:

> My reasoning for keeping the third-party libraries in the tarball was
> simply because upstream made that choice.  Unbundling can and probably
> will introduce differences in the behavior of the output with other
> distributions and their official binaries.

When unbundling, we need to make sure to use compatible library
versions, and then it should be fine.

> I will try to unbundle these libraries from the release tarball.  This
> could take some time though.

Did you manage to make progress?  I know unbundling can often be tedious
and thankless…

Ludo’.

Re: [PATCH] gnu: Add mongodb.

[Roel Janssen]

Hello Ludo’,

Ludovic Courtès writes:

> Hi!
>
> Roel Janssen <roel@gnu.org> skribis:
>
>> My reasoning for keeping the third-party libraries in the tarball was
>> simply because upstream made that choice.  Unbundling can and probably
>> will introduce differences in the behavior of the output with other
>> distributions and their official binaries.
>
> When unbundling, we need to make sure to use compatible library
> versions, and then it should be fine.

Yes, I need to look into the differences (if there are any) of the
bundled libraries.  (I know they found significant performance
improvements with their own implementation of malloc()).

>> I will try to unbundle these libraries from the release tarball.  This
>> could take some time though.
>
> Did you manage to make progress?  I know unbundling can often be tedious
> and thankless…

Thanks for the reminder.  I haven't had time to look at this.  Will try
to make progress on this in the next week.

Kind regards,
Roel Janssen

Re: [PATCH] gnu: Add mongodb.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Thu, 17 Mar 2016 16:08:24 +0100
Roel Janssen <roel@gnu.org> wrote:

> Dear Guix,
> 
> This is a patch to add MongoDB (server and client) tools.  I used a
> three-line patch to fix the build process's reliance on Git and the
> .git/ directory.

Hey,

I've now created bug #28198 with a slightly modified version of this
patch.

I've started the process of trying to build MongoDB without using the
third party source code that's included. It's going quite slowly,
mostly because the package takes a while to build.

The patches I've sent to #28198 also include a very basic service and
system test. The system test currently fails, as the mongo client
attempts to determine a "distroName", and fails. I haven't worked out
what to do about this yet...

Thanks,

Chris

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28198] [PATCH] gnu: Add mongodb.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

On Wed, 23 Aug 2017 10:30:06 +0100
Christopher Baines <mail@cbaines.net> wrote:

> The patches I've sent to #28198 also include a very basic service and
> system test. The system test currently fails, as the mongo client
> attempts to determine a "distroName", and fails. I haven't worked out
> what to do about this yet...

I started a thread about the issue with the mongo client failing on the
mongodb-user mailing list [1], and I've now also got around to creating
a ticket on the MongoDB issue tracker about it [2].

1: https://groups.google.com/d/msg/mongodb-user/jmv1dzc0Kp4/ixvXwyVIBgAJ
2: https://jira.mongodb.org/browse/SERVER-31105

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28198] [PATCH] gnu: Add mongodb.

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> On Wed, 23 Aug 2017 10:30:06 +0100
> Christopher Baines <mail@cbaines.net> wrote:
>
>> The patches I've sent to #28198 also include a very basic service and
>> system test. The system test currently fails, as the mongo client
>> attempts to determine a "distroName", and fails. I haven't worked out
>> what to do about this yet...
>
> I started a thread about the issue with the mongo client failing on the
> mongodb-user mailing list [1], and I've now also got around to creating
> a ticket on the MongoDB issue tracker about it [2].
>
> 1: https://groups.google.com/d/msg/mongodb-user/jmv1dzc0Kp4/ixvXwyVIBgAJ
> 2: https://jira.mongodb.org/browse/SERVER-31105

Given that it’s purely for statistics, IIUC the thread above, I think
the client should have an option to *not* send that information (and it
should be opt-in IMO.)

In the meantime we could patch the code to have a hard-coded
“distroName”.

Ludo’.