From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Amirouche Boubekki Newsgroups: gmane.org.wikimedia.wikidata,gmane.lisp.guile.user,gmane.org.w3c.semantic-web Subject: [ANN] nomunofu v0.1.0 Date: Sun, 8 Dec 2019 18:52:43 +0100 Message-ID: Reply-To: Discussion list for the Wikidata project Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8899249730197100670==" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="116512"; mail-complaints-to="usenet@blaine.gmane.org" To: Guile User , Discussion list for the Wikidata project , semantic-web Original-X-From: wikidata-bounces-RusutVdil2icGmH+5r0DM0B+6BGkLq7r@public.gmane.org Sun Dec 08 18:53:39 2019 Return-path: Envelope-to: goww-wikidata-l@m.gmane.org Original-Received: from lists.wikimedia.org ([208.80.154.21]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1ie0kU-000UCJ-Cr for goww-wikidata-l@m.gmane.org; Sun, 08 Dec 2019 18:53:38 +0100 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.wikimedia.org; s=wikimedia; h=Sender:Content-Type:Reply-To:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:To:Message-ID:Date:From:MIME-Version; bh=6yBvQ9oTPxSgDrWSK1oCGEe5FvEDdEB5IPE4bAoJUgM=; b=Syur2SXnLEiW3tX4j2vZFARj0Idhn79G5EZlZbJPkoMczjUo8D9y1Qdprr3MgvHfs0FucRxA2YBqIDXwHFQ1IM4qxT1PtJkyS2ujEYqUakDXXvzNrVXgNVYSxvS5Z2IigHuNsGNSbkCNaloHO1KFuXFEjGI9ws5UI1F0C+pD4JI=; Original-Received: from localhost ([::1]:39380 helo=fermium.wikimedia.org) by fermium.wikimedia.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ie0k9-000593-GC; Sun, 08 Dec 2019 17:53:17 +0000 Original-Received: from mail-ua1-x931.google.com ([2607:f8b0:4864:20::931]:38774) by fermium.wikimedia.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1ie0js-00057w-CK for Wikidata-RusutVdil2icGmH+5r0DM0B+6BGkLq7r@public.gmane.org; Sun, 08 Dec 2019 17:53:02 +0000 Original-Received: by mail-ua1-x931.google.com with SMTP id z17so4717922uac.5 for ; Sun, 08 Dec 2019 09:53:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=FWf4CI2xtWJg6jKVpSLmrAeQVArJBOskC56wQAlvAkY=; b=JO4KDXzBooHp6Ydsj+isnaDakmRsjRAuM+Q4vFkpC3/FTGw881wnlRQaTlfGFU5pRC LlGBMF1ksVrOvNSSkuGM9VZMXnwkD38XgwVFde0mRr/LJmURHdt1/QnGqaABQMy4ot07 gZWm1udvTPKEEvUdBQbOkxR4jf7C3yAqz1HkixEUZkm+l7Z2T5+Elqr/bhD9YsF0uJ5U dXWpgUqI+JFmj50UPiokNiwD15/dOU07Q8tslzgaTDIzoqHFwQ+Z9nMsWSnvEyuqbiEv j2cCYXPOK/uq12QMYXLbTGUlJIh6hF/7eXus6hDYg5I64Mx520Qt54s8xDtZdG2gxUmj 5UnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FWf4CI2xtWJg6jKVpSLmrAeQVArJBOskC56wQAlvAkY=; b=bjoBgcJc4LY0WiVQSqWhwS9oOoalKE045O82ksRH7g9W99G8FBDGSb42XIBNPH2F3+ usshu6IMq0Vw0dVsVkUCrCEDF+8QPW/PMwjKi40pkCgbotslDFjAsMAeCOmql6SWkuAM hRW0ddEMzov01IvqnsSKPvPbHIYT1t1uSompTa57DkyknLZ6zydZcsSNRD7Kei02b9Be 0ap8dCqoLEsGfr5dNS10TwzcxQ2Mbla3y7Yzhm9PDzWxq4Ry3mrgZOpwYtkzVq+T5lWe eyzK4A/9dnMnhdTkL4tnRnIDLiSu7MvQ36H2pKV1BzUC2Qw1tShHy7fQ4gW8KLI8cCLj C07g== X-Gm-Message-State: APjAAAUtwUnHda1RSdNOUOTzLtTRBokF7q6VQsAFJqUkgxqGGXg4pnNm qmzgLsAzrwXUzWsoyLvgons6Y+cOw8EIGrx+Ezw= X-Google-Smtp-Source: APXvYqyLJL2DMMqpAHDrqzvls/LGn1HdReRVcHt5Bh5Fdtn3t+8guLzu0s1UgSv4Qd70CNdb0yTP7cVUckz5oqYgQ/A= X-Received: by 2002:ab0:2814:: with SMTP id w20mr20402971uap.95.1575827574647; Sun, 08 Dec 2019 09:52:54 -0800 (PST) X-BeenThere: wikidata-RusutVdil2icGmH+5r0DM0B+6BGkLq7r@public.gmane.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion list for the Wikidata project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: wikidata-bounces-RusutVdil2icGmH+5r0DM0B+6BGkLq7r@public.gmane.org Original-Sender: "Wikidata" Xref: news.gmane.org gmane.org.wikimedia.wikidata:12803 gmane.lisp.guile.user:15943 gmane.org.w3c.semantic-web:26079 Archived-At: --===============8899249730197100670== Content-Type: multipart/alternative; boundary="000000000000d06f8d059934f2e4" --000000000000d06f8d059934f2e4 Content-Type: text/plain; charset="UTF-8" I am very pleased to announce the immediate availability of nomunofu. nomunofu is database server written in GNU Guile that is powered by WiredTiger ordered key-value store. It allows to store and query triples. The goal is to make it much easier, definitely faster to query as big as possible tuples of three items. To achieve that goal, the server part of the database is made very simple and it only knows how to do pattern matching. Also, it is possible to swap the storage engine to something that is horizontally scalable and resilient. The client must be smarter, and do as they please to full-fill user requests. Today release only include a minimal Python client. In the future, I plan to extend the Python client to fully support SPARQL 1.1. Preliminary tests over 100 000 and 1 000 000 triples are good looking. Next step is to reach 1 billion triples and eventually 9 billions wikidata triples. You can get the code with the following command: git clone https://github.com/amirouche/nomunofu After the installation of GNU Guix [0], you can do: make init && gunzip test.nt.gz && make index && make web And in another terminal: make query Thanks to Guix, portable binaries for amd64 Ubuntu 18.04 will be made available in a few weeks, along with this, a docker image will be built. The binary release will include wikidata pre-loaded. [0] https://guix.gnu.org/download/ Here is an example ipython session: $ ipython Python 3.7.3 (default, Oct 7 2019, 12:56:13) Type 'copyright', 'credits' or 'license' for more information IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help. In [1]: from nomunofu import Nomunofu In [2]: from nomunofu import var In [3]: nomunofu = Nomunofu('http://localhost:8080'); In [4]: nomunofu.query((var('uid'), " http://www.w3.org/2000/01/rdf-schema#label", "Belgium")) Out[4]: [{'uid': 'http://www.wikidata.org/entity/Q31'}] In [5]: nomunofu.query((var('uid'), " http://www.w3.org/2000/01/rdf-schema#label", "Belgium"), (var('about'), "http:// ...: schema.org/about", var('uid'))) Out[5]: [{'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://www.wikidata.org/wiki/Special:EntityData/Q31'}, {'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://it.wikivoyage.org/wiki/Belgio'}, {'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://an.wikipedia.org/wiki/Belchica'}, {'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://sl.wikipedia.org/wiki/Belgija'}, {'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://pfl.wikipedia.org/wiki/Belgien'}, {'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://crh.wikipedia.org/wiki/Bel%C3%A7ika'}, {'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://fiu-vro.wikipedia.org/wiki/Belgi%C3%A4'}, {'uid': 'http://www.wikidata.org/entity/Q31', 'about': 'https://fr.wikipedia.org/wiki/Belgique'} ... Cheers, Amirouche ~ zig ~ https://hyper.dev --000000000000d06f8d059934f2e4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I am very pleased to announce the immediate availabil= ity of nomunofu.

nomunofu is database server writt= en in GNU Guile that is powered by WiredTiger ordered key-value store.

It allows to store and query triples.=C2=A0 The goal i= s to make it much easier, definitely faster to query as big as possible tup= les of three items.=C2=A0 To achieve that goal, the server part of the data= base is made very simple and it only knows how to do pattern matching.=C2= =A0 Also, it is possible to swap the storage engine to something that is ho= rizontally scalable and resilient.

The client= must be smarter, and do as they please to full-fill user requests. Today r= elease only include a minimal Python client.=C2=A0 In the future, I plan to= extend the Python client to fully support SPARQL 1.1.

Preliminary tests over 100 000 and 1 000 000 triples are good looking= . Next step is to reach 1 billion triples and eventually 9 billions wikidat= a triples.

You can get the code with the follo= wing command:


After the installation of GNU= Guix [0], you can do:

=C2=A0 make init &= & gunzip test.nt.gz && make index && make web

And in another terminal:

=C2=A0 ma= ke query

Thanks to Guix, portable binaries for amd= 64 Ubuntu 18.04 will be made available in a few weeks, along with this, a d= ocker image will be built.=C2=A0 The binary release will include wikidata p= re-loaded.


Here is an example ipython session:

$ ipython
Python 3.7.3 (default, Oct =C2=A07 2019= , 12:56:13)
Type 'copyright', 'credits' or 'license= ' for more information
IPython 7.10.1 -- An enhanced Interactive Pyt= hon. Type '?' for help.

In [1]: from nomunofu import Nomunof= u =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0

In [2]:= from nomunofu import var =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0

In [3]: nomunofu =3D Nomunofu('h= ttp://localhost:8080'); =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0

In [4]: nomunofu.query((var('ui= d'), "http://www.w3.org/2000/01/rdf-schema#label", =C2=A0&q= uot;Belgium")) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0
Out[4]: [{'uid': 'http://www.wikidat= a.org/entity/Q31'}]

In [5]: nomunofu.query((var('uid'= ;), "http://www.w3.org/2000/01/rdf-schema#label", =C2=A0"B= elgium"), (var('about'), "http://
=C2=A0 =C2=A0...: schema.org/about&qu= ot;, var('uid'))) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0
Out[5]:
[{'uid': 'http://www.wikidata.org/entity/Q31'= ,
=C2=A0 'about': 'https://www.wikidata.org/wik= i/Special:EntityData/Q31'},
=C2=A0{'uid': 'http://www.wikida= ta.org/entity/Q31',
=C2=A0 'about': 'https://it.wikivo= yage.org/wiki/Belgio'},
=C2=A0{'uid': 'http://www.wikidata.org= /entity/Q31',
=C2=A0 'about': 'https://an.wikipedia.o= rg/wiki/Belchica'},
=C2=A0{'uid': 'http://www.wikidata.org/ent= ity/Q31',
=C2=A0 'about': 'https://sl.wikipedia.org/wi= ki/Belgija'},
=C2=A0{'uid': 'http://www.wikidata.org/entity/Q3= 1',
=C2=A0 'about': 'https://pfl.wikipedia.org/wiki/B= elgien'},
=C2=A0{'uid': 'http://www.wikidata.org/entity/Q31',
=C2=A0 'about': 'https://crh.wikipedia.org/wiki/= Bel%C3%A7ika'},
=C2=A0{'uid': 'http://www.wikidata.org/entity/= Q31',
=C2=A0 'about': 'https://fiu-vro.wikipe= dia.org/wiki/Belgi%C3%A4'},
=C2=A0{'uid': 'http://www.wikida= ta.org/entity/Q31',
=C2=A0 'about': 'https://fr.wikipedia.= org/wiki/Belgique'}
...

Chee= rs,

Amirouche ~ zig ~ https://hyper.dev
--000000000000d06f8d059934f2e4-- --===============8899249730197100670== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KV2lraWRhdGEg bWFpbGluZyBsaXN0Cldpa2lkYXRhQGxpc3RzLndpa2ltZWRpYS5vcmcKaHR0cHM6Ly9saXN0cy53 aWtpbWVkaWEub3JnL21haWxtYW4vbGlzdGluZm8vd2lraWRhdGEK --===============8899249730197100670==--