[ANN] nomunofu v0.1.0

[ANN] nomunofu v0.1.0

[Amirouche Boubekki]

[-- Attachment #1.1: Type: text/plain, Size: 2964 bytes --]

I am very pleased to announce the immediate availability of nomunofu.

nomunofu is database server written in GNU Guile that is powered by
WiredTiger ordered key-value store.

It allows to store and query triples.  The goal is to make it much easier,
definitely faster to query as big as possible tuples of three items.  To
achieve that goal, the server part of the database is made very simple and
it only knows how to do pattern matching.  Also, it is possible to swap the
storage engine to something that is horizontally scalable and resilient.

The client must be smarter, and do as they please to full-fill user
requests. Today release only include a minimal Python client.  In the
future, I plan to extend the Python client to fully support SPARQL 1.1.

Preliminary tests over 100 000 and 1 000 000 triples are good looking. Next
step is to reach 1 billion triples and eventually 9 billions wikidata
triples.

You can get the code with the following command:

  git clone https://github.com/amirouche/nomunofu

After the installation of GNU Guix [0], you can do:

  make init && gunzip test.nt.gz && make index && make web

And in another terminal:

  make query

Thanks to Guix, portable binaries for amd64 Ubuntu 18.04 will be made
available in a few weeks, along with this, a docker image will be built.
The binary release will include wikidata pre-loaded.

[0] https://guix.gnu.org/download/

Here is an example ipython session:

$ ipython
Python 3.7.3 (default, Oct  7 2019, 12:56:13)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from nomunofu import Nomunofu


In [2]: from nomunofu import var


In [3]: nomunofu = Nomunofu('http://localhost:8080');


In [4]: nomunofu.query((var('uid'), "
http://www.w3.org/2000/01/rdf-schema#label",  "Belgium"))

Out[4]: [{'uid': 'http://www.wikidata.org/entity/Q31'}]

In [5]: nomunofu.query((var('uid'), "
http://www.w3.org/2000/01/rdf-schema#label",  "Belgium"), (var('about'),
"http://
   ...: schema.org/about", var('uid')))

Out[5]:
[{'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://www.wikidata.org/wiki/Special:EntityData/Q31'},
 {'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://it.wikivoyage.org/wiki/Belgio'},
 {'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://an.wikipedia.org/wiki/Belchica'},
 {'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://sl.wikipedia.org/wiki/Belgija'},
 {'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://pfl.wikipedia.org/wiki/Belgien'},
 {'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://crh.wikipedia.org/wiki/Bel%C3%A7ika'},
 {'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://fiu-vro.wikipedia.org/wiki/Belgi%C3%A4'},
 {'uid': 'http://www.wikidata.org/entity/Q31',
  'about': 'https://fr.wikipedia.org/wiki/Belgique'}
...

Cheers,

Amirouche ~ zig ~ https://hyper.dev

[-- Attachment #1.2: Type: text/html, Size: 6063 bytes --]

[-- Attachment #2: Type: text/plain, Size: 153 bytes --]

_______________________________________________
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

Re: [ANN] nomunofu v0.1.0

[Amirouche Boubekki]

Le dim. 8 déc. 2019 à 18:52, Amirouche Boubekki
<amirouche.boubekki@gmail.com> a écrit :
>
> I am very pleased to announce the immediate availability of nomunofu.
>
> nomunofu is database server written in GNU Guile that is powered by WiredTiger ordered key-value store.
>
> It allows to store and query triples.  The goal is to make it much easier, definitely faster to query as big as possible tuples of three items.  To achieve that goal, the server part of the database is made very simple and it only knows how to do pattern matching.  Also, it is possible to swap the storage engine to something that is horizontally scalable and resilient.
>

I pushed portable binaries built with gnu guix for amd64 with a small
database file. You can download it with the following command:

  $ wget https://hyper.dev/nomunofu-v0.1.3.tar.gz

The uncompressed directory is 7GB.

Once you have downloaded the tarball, you can do the following cli
dance to run the database server:

  $ tar xf nomunofu-v0.1.3.tar.gz && cd nomunofu && ./nomunofu serve 8080

The database will be available on port 8080. Then you can use the
python client to do queries.

Here is example run on the current dataset, that queries for instance
of (P31) government (Q3624078):

In [1]: from nomunofu import Nomunofu
In [2]: from nomunofu import var
In [3]: nomunofu = Nomunofu('http://localhost:8080');
In [4]: nomunofu.query((var('uid'),
'http://www.wikidata.org/prop/direct/P31',
'http://www.wikidata.org/entity/Q3624078'), (var('uid'),
'http://www.w3.org/2000/01/rdf-schema#label', var('label')))

Out[4]:
[{'uid': 'http://www.wikidata.org/entity/Q31', 'label': 'Belgium'},
 {'uid': 'http://www.wikidata.org/entity/Q183', 'label': 'Germany'},
 {'uid': 'http://www.wikidata.org/entity/Q148', 'label': 'China'},
 {'uid': 'http://www.wikidata.org/entity/Q148',
  'label': "People's Republic of China"},
 {'uid': 'http://www.wikidata.org/entity/Q801', 'label': 'Israel'},
 {'uid': 'http://www.wikidata.org/entity/Q45', 'label': 'Portugal'},
 {'uid': 'http://www.wikidata.org/entity/Q155', 'label': 'Brazil'},
 {'uid': 'http://www.wikidata.org/entity/Q916', 'label': 'Angola'},
 {'uid': 'http://www.wikidata.org/entity/Q233', 'label': 'Malta'},
 {'uid': 'http://www.wikidata.org/entity/Q878',
  'label': 'United Arab Emirates'},
 {'uid': 'http://www.wikidata.org/entity/Q686', 'label': 'Vanuatu'},
 {'uid': 'http://www.wikidata.org/entity/Q869', 'label': 'Thailand'},
 {'uid': 'http://www.wikidata.org/entity/Q863', 'label': 'Tajikistan'},
 {'uid': 'http://www.wikidata.org/entity/Q1049', 'label': 'Sudan'},
 {'uid': 'http://www.wikidata.org/entity/Q1044', 'label': 'Sierra Leone'},
 {'uid': 'http://www.wikidata.org/entity/Q912', 'label': 'Mali'},
 {'uid': 'http://www.wikidata.org/entity/Q819', 'label': 'Laos'},
 {'uid': 'http://www.wikidata.org/entity/Q298', 'label': 'Chile'},
 {'uid': 'http://www.wikidata.org/entity/Q398', 'label': 'Bahrain'},
 {'uid': 'http://www.wikidata.org/entity/Q12560', 'label': 'Ottoman Empire'}]

As of right now there is less than 10 000 000 triples that were
imported. Blank nodes are included, but only english labels are
imported.

You can find the source code at:

  https://github.com/amirouche/nomunofu

I hope you have a good day!

Amirouche ~ zig ~ https://hyper.dev

_______________________________________________
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

Re: [ANN] nomunofu v0.1.0

[Amirouche Boubekki]

I am pleased to share with you the v0.1.4 binary release. It contains
the following improvements:

- The REST API takes JSON as input, which will make it easier to
create clients in other programming languages;

- The REST API takes limit and offset as query string. The maximum
limit is 1000;

- There is better error handling, the server will return a HTTP status
code 400 if it detects an error;

- Add aggregation queries `sum`, `count` and `average`, see the Python
client (nomunofu.py) to know how to properly format the query;

- Python client method `Nomunofu.query(*patterns, limit=None,
offset=None)` returns a generator.

Also, the harmless warnings are silenced. The database files are
compatible with the previous release.

This release comes with full wikidata lexemes triples.

You can download the amd64 portable binary release plus database files
with the following command:

  wget http://hyper.dev/nomunofu-v0.1.4.tar.bz2

The directory is 11G uncompressed.

Grab the source code with the following command:

  git clone https://github.com/amirouche/nomunofu

Here is an example Python query that returns at most 5 adverbs:

In [10]: for item in nomunofu.query(
    ...:     (var('uid'), wikibase('lexicalCategory'),
'http://www.wikidata.org/entity/Q380057'),
    ...:     (var('uid'), rdfschema('label'), var('label')),
    ...:     limit=5):
    ...:     print(item)
    ...:
{'uid': 'http://www.wikidata.org/entity/L3244', 'label': 'always'}
{'uid': 'http://www.wikidata.org/entity/L4124', 'label': 'here'}
{'uid': 'http://www.wikidata.org/entity/L4326', 'label': 'often'}
{'uid': 'http://www.wikidata.org/entity/L5201', 'label': 'too'}
{'uid': 'http://www.wikidata.org/entity/L5321', 'label': 'yet'}



Cheers,



Amirouche ~ zig ~ https://hyper.dev

Re: [ANN] nomunofu v0.1.0

[tomas]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

On Thu, Dec 12, 2019 at 03:57:18PM +0100, Amirouche Boubekki wrote:
> I am pleased to share with you the v0.1.4 binary release. It contains
> the following improvements:

This sounds very intriguing. I'm drowned in customer stuff at the
moment, but perhaps you've made something for my Christmas tree :-)

Thanks
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [ANN] nomunofu v0.1.0

[Amirouche Boubekki]

Le jeu. 12 déc. 2019 à 16:58, <tomas@tuxteam.de> a écrit :
>
> On Thu, Dec 12, 2019 at 03:57:18PM +0100, Amirouche Boubekki wrote:
> > I am pleased to share with you the v0.1.4 binary release. It contains
> > the following improvements:
>
> This sounds very intriguing. I'm drowned in customer stuff at the
> moment, but perhaps you've made something for my Christmas tree :-)
>

It will be even more intriguing when you will look at v0.2.0:

  https://github.com/amirouche/nomunofu/releases/tag/v0.2.0

I did not include wikidata lexemes database files, it takes only a
couple of hours to import.

Cheers

Re: [ANN] nomunofu v0.1.0

[Amirouche Boubekki]

Hello all ;-)


I ported the code to Chez Scheme to do an apple-to-apple comparison
between GNU Guile and Chez and took the time to launch a few queries
against Virtuoso available in Ubuntu 18.04 (LTS).

Spoiler: the new code is always faster.

The hard disk is SATA, and the CPU is dubbed: Intel(R) Xeon(R) CPU
E3-1220 V2 @ 3.10GHz

I imported latest-lexeme.nt (6GB) using guile-nomunofu, chez-nomunofu
and Virtuoso:

- Chez takes 40 minutes to import 6GB
- Chez is 3 to 5 times faster than Guile
- Chez is 11% faster than Virtuoso

Regarding query time, Chez is still faster than Virtuoso with or
without cache.  The query I am testing is the following:

SELECT ?s ?p ?o
FROM <http://fu>
WHERE {
  ?s <http://purl.org/dc/terms/language> <http://www.wikidata.org/entity/Q150> .
  ?s <http://wikiba.se/ontology#lexicalCategory>
<http://www.wikidata.org/entity/Q1084> .
  ?s <http://www.w3.org/2000/01/rdf-schema#label> ?o
};

Virtuoso first query takes: 1295 msec.
The second query takes: 331 msec.
Then it stabilize around: 200 msec.

chez nomunofu takes around 200ms without cache.

There is still an optimization I can do to speed up nomunofu a little.


Happy hacking!

Re: [ANN] nomunofu v0.1.0

[Ted Thibodeau Jr]

[-- Attachment #1.1: Type: text/plain, Size: 4284 bytes --]


On Dec 22, 2019, at 03:17 PM, Amirouche Boubekki <amirouche.boubekki-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hello all ;-)
> 
> 
> I ported the code to Chez Scheme to do an apple-to-apple comparison
> between GNU Guile and Chez and took the time to launch a few queries
> against Virtuoso available in Ubuntu 18.04 (LTS).

Hi, Amirouche --

Kingsley's points about tuning Virtuoso to use available 
RAM [1] and other system resources are worth looking into, 
but a possibly more important first question is --

   Exactly what version of Virtuoso are you testing?

If you followed the common script on Ubuntu 18.04, i.e., --

   sudo apt update

   sudo apt install virtuoso-opensource

-- then you likely have version 6.1.6 of VOS, the Open Source 
Edition of Virtuoso, which shipped 2012-08-02 [2], and is far
behind the latest version of both VOS (v7.2.5+) and Enterprise 
Edition (v8.3+)!

The easiest way to confirm what you're running is to review 
the first "paragraph" of output from the command corresponding 
to the name of your Virtuoso binary --

   virtuoso-t -?

   virtuoso-iodbc-t -?

If I'm right, and you're running 6.x, you'll get much better
test results just by running a current version of Virtuoso.

You can build VOS 7.2.6+ from source [3] (we'd recommend the 
develop/7 branch [4] for the absolute latest), or download a 
precompiled binary [5] of VOS 7.2.5.1 or 7.2.6.dev.

You can also try Enterprise Edition at no cost for 30 days [5].



[1] http://vos.openlinksw.com/owiki/wiki/VOS/VirtRDFPerformanceTuning

[2] http://vos.openlinksw.com/owiki/wiki/VOS/VOSNews2012#2012-08-02%20--%20Announcing%20Virtuoso%20Open-Source%20Edition%20v6.1.6.

[3] http://vos.openlinksw.com/owiki/wiki/VOS/VOSBuild

[4] https://github.com/openlink/virtuoso-opensource/tree/develop/7

[5] https://sourceforge.net/projects/virtuoso/files/virtuoso/ 





> Spoiler: the new code is always faster.
> 
> The hard disk is SATA, and the CPU is dubbed: Intel(R) Xeon(R) CPU
> E3-1220 V2 @ 3.10GHz
> 
> I imported latest-lexeme.nt (6GB) using guile-nomunofu, chez-nomunofu
> and Virtuoso:
> 
> - Chez takes 40 minutes to import 6GB
> - Chez is 3 to 5 times faster than Guile
> - Chez is 11% faster than Virtuoso


How did you load the data?  Did you use Virtuoso's build-load
facilities?  This is the recommended method [6].

[6] http://vos.openlinksw.com/owiki/wiki/VOS/VirtBulkRDFLoader


> Regarding query time, Chez is still faster than Virtuoso with or
> without cache.  The query I am testing is the following:
> 
> SELECT ?s ?p ?o
> FROM <http://fu>
> WHERE {
>  ?s <http://purl.org/dc/terms/language> <http://www.wikidata.org/entity/Q150> .
>  ?s <http://wikiba.se/ontology#lexicalCategory>
> <http://www.wikidata.org/entity/Q1084> .
>  ?s <http://www.w3.org/2000/01/rdf-schema#label> ?o
> };
> 
> Virtuoso first query takes: 1295 msec.
> The second query takes: 331 msec.
> Then it stabilize around: 200 msec.
> 
> chez nomunofu takes around 200ms without cache.
> 
> There is still an optimization I can do to speed up nomunofu a little.
> 
> 
> Happy hacking!


I'll be interested to hear your new results, with a current build,
and with proper INI tuning in place.

Regards,

Ted



--
A: Yes.                          http://www.idallen.com/topposting.html
| Q: Are you sure?           
| | A: Because it reverses the logical flow of conversation.
| | | Q: Why is top posting frowned upon?

Ted Thibodeau, Jr.           //               voice +1-781-273-0900 x32
Senior Support & Evangelism  //        mailto:tthibodeau-HpHEqLDO2a7UEDaH6ef/NA@public.gmane.org
                             //              http://twitter.com/TallTed
OpenLink Software, Inc.      //              http://www.openlinksw.com/
         20 Burlington Mall Road, Suite 322, Burlington MA 01803
     Weblog    -- http://www.openlinksw.com/blogs/
     Community -- https://community.openlinksw.com/
     LinkedIn  -- http://www.linkedin.com/company/openlink-software/
     Twitter   -- http://twitter.com/OpenLink
     Facebook  -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers





[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2094 bytes --]

[-- Attachment #2: Type: text/plain, Size: 153 bytes --]

_______________________________________________
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

Re: [Wikidata] [ANN] nomunofu v0.1.0

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

On Sun, Dec 22, 2019 at 06:36:06PM -0500, Ted Thibodeau Jr via General Guile related discussions wrote:

[...]

> Hi, Amirouche --
> 
> Kingsley's points about tuning Virtuoso to use available 
> RAM [1] and other system resources are worth looking into, 
> but a possibly more important first question is --
> 
>    Exactly what version of Virtuoso are you testing?
> 
> If you followed the common script on Ubuntu 18.04, i.e., --
> 
>    sudo apt update
> 
>    sudo apt install virtuoso-opensource

Hi, Ted

I see you do have a stake in the game. That's fine, since having
a good technical discussion should be independent of that. So
thanks for chiming in!

> -- then you likely have version 6.1.6 of VOS, the Open Source 
> Edition of Virtuoso, which shipped 2012-08-02 [2], and is far
> behind the latest version of both VOS (v7.2.5+) and Enterprise 
> Edition (v8.3+)!

Wouldn't that be something your company could contribute to
Ubuntu/Debian? Contact the package maintainers and help keeping
the free version of Virtuoso up to snuff on those platforms?

A rising tide, etc?

> You can also try Enterprise Edition at no cost for 30 days [5].

I think it's somewhat unfair to try to compare a freshly
done free project with an established commercial one.

> I'll be interested to hear your new results, with a current build,
> and with proper INI tuning in place.

Why not assist Amirouche on that part? After all, you have
much easier access to Virtuoso expertise, and me, at least,
rather prefer Amirouche to spend his time tuning his wonderful
free creatures than tuning Virtuoso ;-P (of course, he'll end
up doing what he enjoys most, I'd hope!).

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Wikidata] [ANN] nomunofu v0.1.0

[Roel Janssen]

On Mon, 2019-12-23 at 09:56 +0100, tomas@tuxteam.de wrote:
> On Sun, Dec 22, 2019 at 06:36:06PM -0500, Ted Thibodeau Jr via General Guile
> related discussions wrote:
> 
> [...]
> 
> > Hi, Amirouche --
> > 
> > Kingsley's points about tuning Virtuoso to use available 
> > RAM [1] and other system resources are worth looking into, 
> > but a possibly more important first question is --
> > 
> >    Exactly what version of Virtuoso are you testing?
> > 
> > If you followed the common script on Ubuntu 18.04, i.e., --
> > 
> >    sudo apt update
> > 
> >    sudo apt install virtuoso-opensource
> 
> Hi, Ted
> 
> I see you do have a stake in the game. That's fine, since having
> a good technical discussion should be independent of that. So
> thanks for chiming in!
> 
> > -- then you likely have version 6.1.6 of VOS, the Open Source 
> > Edition of Virtuoso, which shipped 2012-08-02 [2], and is far
> > behind the latest version of both VOS (v7.2.5+) and Enterprise 
> > Edition (v8.3+)!
> 
> Wouldn't that be something your company could contribute to
> Ubuntu/Debian? Contact the package maintainers and help keeping
> the free version of Virtuoso up to snuff on those platforms?
> 
> A rising tide, etc?
> 
> > You can also try Enterprise Edition at no cost for 30 days [5].
> 
> I think it's somewhat unfair to try to compare a freshly
> done free project with an established commercial one.
> 
> > I'll be interested to hear your new results, with a current build,
> > and with proper INI tuning in place.
> 
> Why not assist Amirouche on that part? After all, you have
> much easier access to Virtuoso expertise, and me, at least,
> rather prefer Amirouche to spend his time tuning his wonderful
> free creatures than tuning Virtuoso ;-P (of course, he'll end
> up doing what he enjoys most, I'd hope!).
> 

This might already help:
I packaged Virtuoso OSE 7.2.5 for GNU Guix.  The documentation of Virtuoso OSE
provides enough information to tune it quite well.  (I've been running Virtuoso
OSE (from a GNU Guix install) for about three years now, and am quite happy with
its performance).

Cheers,
Roel