[ELPA] New package: triples

[ELPA] New package: triples

[Andrew Hyatt]

Hi everyone,

I've created a new package, triples, which can be found at 
https://github.com/ahyatt/triples.  I'd like to add this to some 
ELPA, and would prefer the gnu ELPA.  I will be talking about this 
package, and examples of its use, in the upcoming Emacs 
conference.

The package is a library which is a layer on top of sqlite, 
designed to store data in a particular format, where everything 
goes into one table. All data is expressed as "triples", a link 
between a subject, and an object, via a predicate.  This is 
similar to the RDF standard.

With the triples package, a schema can be defined, which also is 
stored in the same triples data.  The schema controls what are 
valid representations in the database.  This takes the form of 
types and their properites.  Data about entities, and their types 
and properties can be stored and retrieved according to that 
schema, and the schema can be added by other clients, letting 
packages build on top of the data by adding new types, and letting 
the data live together in a way that's difficult to do otherwise. 
For example, one package could provide a way to store contact 
information in the database.  Another package could layer on an 
annotation package onto that that lets contact and any other 
entities contain annotations.

Another way to think of this is that it's a database that allows 
independent types, each type having a predefined set of 
properties, and entities that are stored can have an arbitrary set 
of those types.

You can see a use of this in my other package that I'm developing: 
https://github.com/ahyatt/ekg.  The package does work, and I am 
using ekg regularly at the moment without issues.

I am the sole contributor, and have FSF papers.

Re: [ELPA] New package: triples

[Stefan Monnier]

> I've created a new package, triples, which can be found at
> https://github.com/ahyatt/triples.  I'd like to add this to some ELPA, and
> would prefer the gnu ELPA.  I will be talking about this package, and
> examples of its use, in the upcoming Emacs conference.

Sounds really nice.  Currently I see it depends on `emacsql`, but that
package is neither in GNU ELPA nor NonGNU ELPA.  Any chance you can
rework your package so it can use the new built-in sqlite code in Emacs-29, or
maybe the `pq` package (tho this one is currently only in GNU-devel, it
just needs a `Version:` bump to appear on GNU ELPA)?

In the mean time, we should probably add `emacsql` to (presumably Non)GNU ELPA.



        Stefan


PS: I also see `cl-lib` in the package-requires, and using a weird
non-parenthesized syntax.  Since `emacsql` depends on Emacs-25, you may
as well depend on Emacs-25 which will give you `cl-lib` anyway.

Re: [ELPA] New package: triples

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Sounds really nice.  Currently I see it depends on `emacsql`, but that
> package is neither in GNU ELPA nor NonGNU ELPA.  Any chance you can
> rework your package so it can use the new built-in sqlite code in
> Emacs-29, or maybe the `pq` package (tho this one is currently only in
> GNU-devel, it just needs a `Version:` bump to appear on GNU ELPA)?
>
> In the mean time, we should probably add `emacsql` to (presumably
> Non)GNU ELPA.

I think we are getting closer to adding it.  See the discussion here:

    https://github.com/magit/emacsql/pull/87#issuecomment-1277783891

Re: [ELPA] New package: triples

[Andrew Hyatt]

On Tue, Oct 25, 2022 at 10:52 PM Stefan Monnier 
<monnier@iro.umontreal.ca> wrote: 

>> I've created a new package, triples, which can be found at 
>> https://github.com/ahyatt/triples.  I'd like to add this to 
>> some ELPA, and would prefer the gnu ELPA.  I will be talking 
>> about this package, and examples of its use, in the upcoming 
>> Emacs conference. 
> 
> Sounds really nice.  Currently I see it depends on `emacsql`, 
> but that package is neither in GNU ELPA nor NonGNU ELPA.  Any 
> chance you can rework your package so it can use the new 
> built-in sqlite code in Emacs-29, or maybe the `pq` package (tho 
> this one is currently only in GNU-devel, it just needs a 
> `Version:` bump to appear on GNU ELPA)? 
> 
> In the mean time, we should probably add `emacsql` to 
> (presumably Non)GNU ELPA.

Yes, it's possible for me to rework it. However, it then would be 
available just to those on Emacs 29, which seems restrictive. 
Would it make sense to try to be compatible with both by wrapping 
the calls and just using whichever sqlite package is available? 
Has anyone done that before?

Anyway, it doesn't hurt for me to make a branch and just 
completely switch to the Emacs 29 sqlite to see if anything 
interesting comes up, so I'll do that soon.


>
>
>
>         Stefan
>
>
> PS: I also see `cl-lib` in the package-requires, and using a weird
> non-parenthesized syntax.  Since `emacsql` depends on Emacs-25, you may
> as well depend on Emacs-25 which will give you `cl-lib` anyway.

Re: [ELPA] New package: triples

[Stefan Monnier]

> Yes, it's possible for me to rework it. However, it then would be available
> just to those on Emacs 29, which seems restrictive.  Would it make sense to
> try to be compatible with both by wrapping the calls and just using
> whichever sqlite package is available?

That's what I was hoping for, yes.  But either way is fine by me :-)

> Anyway, it doesn't hurt for me to make a branch and just completely switch
> to the Emacs 29 sqlite to see if anything interesting comes up, so I'll do
> that soon.

It's probably a good first step in any case, before deciding if and how
to merge the two cases.


        Stefan

Re: [ELPA] New package: triples

[Andrew Hyatt]

On Wed, Oct 26, 2022 at 09:29 AM Stefan Monnier 
<monnier@iro.umontreal.ca> wrote: 

>> Yes, it's possible for me to rework it. However, it then would 
>> be available just to those on Emacs 29, which seems 
>> restrictive.  Would it make sense to try to be compatible with 
>> both by wrapping the calls and just using whichever sqlite 
>> package is available? 
> 
> That's what I was hoping for, yes.  But either way is fine by me 
> :-) 
>

I've started looking into this.  Interesting consideration is that 
emacsql and sqlite have a bit different representations: in sqlite 
everything is a string, but emacsql will store symbols, numbers 
differently than strings, which are wrapped in the database with 
extra quotes to mark them as strings.  I'll try to keep 
compatibility between these versions, which really I think means 
that our use of sqlite has to support emacsql's way of doing 
things.

About the possibility of using one or the other, whichever is 
available, how would that work in terms of package dependencies? 
I don't think it's possible to have an OR dependency specified in 
the package-dependencies line in the source, but really only one 
of emacsql or emacs29 would be required.  If you have any insight 
into this, it'd be useful.
 
>> Anyway, it doesn't hurt for me to make a branch and just 
>> completely switch to the Emacs 29 sqlite to see if anything 
>> interesting comes up, so I'll do that soon. 
> 
> It's probably a good first step in any case, before deciding if 
> and how to merge the two cases.

Great, that is what I'm doing.  When it passes tests, I'll let you 
know.

>
>
>         Stefan

Re: [ELPA] New package: triples

[Stefan Monnier]

> About the possibility of using one or the other, whichever is available, how
> would that work in terms of package dependencies? I don't think it's
> possible to have an OR dependency specified in the package-dependencies
> line in the source, but really only one of emacsql or emacs29 would be
> required.  If you have any insight into this, it'd be useful.

Package dependencies just tell the package manager to install other
packages to try and make it easier on users to get a working package.
But you don't need to have a precise/exact description of the
real dependencies in the `Package-Requires` (as long as the
correct byte-compilation of your package actually depends on some other
package, e.g. if your package uses a macro defined in the other package).

You can just remove the `emacsql` dependency from `Package-Requires` and
replace it with a runtime check for the presence of either `emacsql` or
the new sqlite support that emits an error when applicable (with
a message explaining to the user that they need to have one of those two
thingies before they can use the package).
[ Just make sure the error is not emitted just because the file(s) from
  your package are loaded, but only when some code is actually called.  ]


        Stefan

Re: [ELPA] New package: triples

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

On Thu, Oct 27, 2022 at 11:34 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > About the possibility of using one or the other, whichever is available,
> how
> > would that work in terms of package dependencies? I don't think it's
> > possible to have an OR dependency specified in the package-dependencies
> > line in the source, but really only one of emacsql or emacs29 would be
> > required.  If you have any insight into this, it'd be useful.
>
> Package dependencies just tell the package manager to install other
> packages to try and make it easier on users to get a working package.
> But you don't need to have a precise/exact description of the
> real dependencies in the `Package-Requires` (as long as the
> correct byte-compilation of your package actually depends on some other
> package, e.g. if your package uses a macro defined in the other package).
>
> You can just remove the `emacsql` dependency from `Package-Requires` and
> replace it with a runtime check for the presence of either `emacsql` or
> the new sqlite support that emits an error when applicable (with
> a message explaining to the user that they need to have one of those two
> thingies before they can use the package).
> [ Just make sure the error is not emitted just because the file(s) from
>   your package are loaded, but only when some code is actually called.  ]
>

This is now done, you can find the new code that can either use emacs 29
builtin sqlite, or emacsql, on a new branch, at
https://github.com/ahyatt/triples/tree/combined.



>
>
>         Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 2191 bytes --]

Re: [ELPA] New package: triples

[Jean Louis]

* Andrew Hyatt <ahyatt@gmail.com> [2022-10-26 03:59]:
> I've created a new package, triples, which can be found at
> https://github.com/ahyatt/triples.  I'd like to add this to some ELPA, and
> would prefer the gnu ELPA.  I will be talking about this package, and
> examples of its use, in the upcoming Emacs conference.

I like the idea of relating everything to everything.

I was experimenting with semantic triplets for PostgreSQL:

Implementation of Semantic Triplets in PostgreSQL database:
https://hyperscope.link/3/7/1/5/4/Implementation-of-Semantic-Triplets-in-PostgreSQL-database-37154.html

In this case any table, column, with its primary key may be related to
any other table, column with its primary key. That way objects and
subjects are constrained and rigid, it is not universal triplets, but
database tables dependent one. It relates only what exists in the
database. If there is `people' database then such may be related to
arbitrary other tables like `statistics'.

You have made general triplets to build any type of knowledge that is nice.

Triplets allow huge flexibility in relation assignments.

I just wonder if your "types" can simply be replaced again with
predicates.

select * from predicates;
 predicates_id | predicates_name |      predicates_description       
---------------+-----------------+-----------------------------------
             1 | is              | When subject is object.
             2 | knows           | When subject knows object.
             3 | registered      | When subject registered an object

by adding "has" predicate, one can tell:

person has age
person has name
employee has id
id is integer
employeee has manager
employee is person
manager is person

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: [ELPA] New package: triples

[Andrew Hyatt]

On Thu, Oct 27, 2022 at 06:36 AM Jean Louis <bugs@gnu.support> 
wrote: 

> * Andrew Hyatt <ahyatt@gmail.com> [2022-10-26 03:59]: 
>> I've created a new package, triples, which can be found at 
>> https://github.com/ahyatt/triples.  I'd like to add this to 
>> some ELPA, and would prefer the gnu ELPA.  I will be talking 
>> about this package, and examples of its use, in the upcoming 
>> Emacs conference. 
> 
> I like the idea of relating everything to everything. 
> 
> I was experimenting with semantic triplets for PostgreSQL: 
> 
> Implementation of Semantic Triplets in PostgreSQL database: 
> https://hyperscope.link/3/7/1/5/4/Implementation-of-Semantic-Triplets-in-PostgreSQL-database-37154.html 
> 
> In this case any table, column, with its primary key may be 
> related to any other table, column with its primary key. That 
> way objects and subjects are constrained and rigid, it is not 
> universal triplets, but database tables dependent one. It 
> relates only what exists in the database. If there is `people' 
> database then such may be related to arbitrary other tables like 
> `statistics'.

Thanks for mentioning this, it's an interesting design I'll read 
up on!
 
> 
> You have made general triplets to build any type of knowledge 
> that is nice. 
> 
> Triplets allow huge flexibility in relation assignments. 
> 
> I just wonder if your "types" can simply be replaced again with 
> predicates. 
> 
> select * from predicates; 
>  predicates_id | predicates_name |      predicates_description        
> ---------------+-----------------+----------------------------------- 
>              1 | is              | When subject is object.  2 | 
>              knows           | When subject knows object.  3 | 
>              registered      | When subject registered an object 
> 
> by adding "has" predicate, one can tell: 
> 
> person has age person has name employee has id id is integer 
> employeee has manager employee is person manager is person

We have "has" but not "is". The schema has types, those types have
properties.  So we can have a subject, "Joe Employee", who has a type
person, and a type employee, and a type manager, the properties of those
respective types.  But we don't know or enforce that
every employee is also a person.  It's perhaps a feature that can be
added later if needed; it seems possible in the current implementation.
>
> -- 
> Jean
>
> Take action in Free Software Foundation campaigns:
> https://www.fsf.org/campaigns
>
> In support of Richard M. Stallman
> https://stallmansupport.org/

Re: [ELPA] New package: triples

[Jean Louis]

* Andrew Hyatt <ahyatt@gmail.com> [2022-10-27 08:20]:
> > In this case any table, column, with its primary key may be related to
> > any other table, column with its primary key. That way objects and
> > subjects are constrained and rigid, it is not universal triplets, but
> > database tables dependent one. It relates only what exists in the
> > database. If there is `people' database then such may be related to
> > arbitrary other tables like `statistics'.
> 
> Thanks for mentioning this, it's an interesting design I'll read up
> on!

Recently I am thinking of using UUID. Any table, column, by primary
key (by it UUID or just INTEGER or else), may still have UUID. Let us
say transaction entry:

                             ID   3
                   Date created   "2022-02-27 18:57:23.52117"
                  Date modified   nil
                   User created   "maddox"
                  User modified   "maddox"
                      Date/Time   "2022-02-27 00:00:00"
                           Name   "Mobile money fee"
                    Description   "nil"
                         Amount   2310.0
                       Currency   "UGX - Ugandan shilling"
                   From account   "Jean Louis"
                     To account   "Mobile money expenses"
                           UUID   "3c171d3d-b2c8-4527-8984-34bbacc82ff2"
                  Hyperdocument   nil

Then by using UUID as subject, predicate, and UUID as object, it
becomes clear what is what. A function must search UUID among those
tables who have it. It is really fast, not noticable. But I can't know
how well it could scale with millions or billions of records. For
presentation purposes it would be fine, but for direct access, I
can't be sure of speed.

> > select * from predicates;  predicates_id | predicates_name |
> > predicates_description
> > ---------------+-----------------+-----------------------------------
> > 1 | is              | When subject is object.  2 |              knows
> > | When subject knows object.  3 |              registered      | When
> > subject registered an object
> > 
> > by adding "has" predicate, one can tell:
> > 
> > person has age person has name employee has id id is integer employeee
> > has manager employee is person manager is person
> 
> We have "has" but not "is". The schema has types, those types have
> properties.  So we can have a subject, "Joe Employee", who has a type
> person, and a type employee, and a type manager, the properties of those
> respective types.  But we don't know or enforce that
> every employee is also a person.  It's perhaps a feature that can be
> added later if needed; it seems possible in the current
> implementation.

Principle of semantic triplets is just subject, predicate and
object. I guess you allow users to add any kind of predicates, right?

Types are automatically formed by using subject, predicate, object
principle.

Reference: https://en.wikipedia.org/wiki/Semantic_triple

This should be enough by using that principle:

- job is activity
- job has status
- employee is status
- Joe is name
- person has name
- Joe has employee (meaning status)

Also see this:

OneModel — an Alternative to emacs org-mode:
https://soylentnews.org/article.pl?sid=16/04/23/0149257

OneModel - Record, manage and share any knowledge:
http://onemodel.org/1/e-9223372036854618119.html

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: [ELPA] New package: triples

[Jean Louis]

I have forgot to mention that any relationship subject, predicate,
object, itself, should be capable of becoming both subject and object.

Employee IS status
Joe HAS employee (status)
Date IS TIME, etc.
2012-10-28 IS date
(Joe HAS employee status) SINCE 2012-10-28
ABC INC. IS company
Company EMPLOYED Joe
Trevis HAS manager (status)
(Company EMPLOYED Joe) HAS Trevis
((Company EMPLOYED Joe) HAS Trevis) MANAGES Joe

and so on


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: [ELPA] New package: triples

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

Yes, this is in theory possible depending on how you use this library!
I've seen this done as CVTs (Compound Value Types).  To take your example,
(Joe HAS employee status) SINCE 2012-10-28, it'd be something like:

Subject: Joe
Predicate: /employment/status
Object: "6988a214-368c-40a2-9e51-8a7f5549de44"

Subject: "6988a214-368c-40a2-9e51-8a7f5549de44"
Predicate: /employment/start
Object: "2012-10-28"


On Thu, Oct 27, 2022 at 7:39 PM Jean Louis <bugs@gnu.support> wrote:

> I have forgot to mention that any relationship subject, predicate,
> object, itself, should be capable of becoming both subject and object.
>
> Employee IS status
> Joe HAS employee (status)
> Date IS TIME, etc.
> 2012-10-28 IS date
> (Joe HAS employee status) SINCE 2012-10-28
> ABC INC. IS company
> Company EMPLOYED Joe
> Trevis HAS manager (status)
> (Company EMPLOYED Joe) HAS Trevis
> ((Company EMPLOYED Joe) HAS Trevis) MANAGES Joe
>
> and so on
>
>
> --
> Jean
>
> Take action in Free Software Foundation campaigns:
> https://www.fsf.org/campaigns
>
> In support of Richard M. Stallman
> https://stallmansupport.org/
>

[-- Attachment #2: Type: text/html, Size: 1732 bytes --]

Re: [ELPA] New package: triples

[Jean Louis]

* Andrew Hyatt <ahyatt@gmail.com> [2022-11-03 05:39]:
> Yes, this is in theory possible depending on how you use this
> library!

Not only, but it should be possible. Relationship is also subject or
object. I find UUIDs fantastic tool for relationship management. It is
something that spans over specific databases to many other digital
things on the planet. So I have created this one:

CREATE TABLE uuid2uuid (
uuid2uuid_uuid UUID PRIMARY KEY DEFAULT gen_random_uuid(),
uuid2uuid_datecreated TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
uuid2uuid_datemodified TIMESTAMP,
uuid2uuid_usercreated TEXT NOT NULL DEFAULT current_user,
uuid2uuid_usermodified TEXT NOT NULL DEFAULT current_user,
uuid2uuid_uuidobject UUID NOT NULL,
uuid2uuid_uuidpredicate UUID NOT NULL,
uuid2uuid_uuidsubject UUID NOT NULL,
oid2oid_description TEXT
);

> I've seen this done as CVTs (Compound Value Types).

I do not know that abbreviation.

If you look at the table above, you will see UUID created for each
entry. It means again you may relate relationship UUID to other UUID
and how you define values for UUID is left to programmer.

For example ID 3 in the database:

                             ID   3
                     First name   "Vicente"
   Last name (People List Name)   "Ramirez"
                    People type   "Individual Person"
                           UUID   "f2832aa3-8cbd-4d2f-87c3-2887c41d22e2"

Thus UUID "f2832aa3-8cbd-4d2f-87c3-2887c41d22e2" may be related to
other UUID "2fc8ebd6-c24e-4a91-aaf9-42970c15866c"

One could say 

Entry 7150f343-1c54-43f7-9838-414724b93bda says that:
UUID "f2832aa3-8cbd-4d2f-87c3-2887c41d22e2" is married (predicate) to UUID "2fc8ebd6-c24e-4a91-aaf9-42970c15866c"

Now we got UUID for marriage: 7150f343-1c54-43f7-9838-414724b93bda and
UUID for child 6d7b0daf-2a6e-4fcc-9b7a-b83208fc4c5f and we can say:

Entry 0e1cdfd9-7248-4c08-8cf8-a66eae51ea3a says that:
UUID "7150f343-1c54-43f7-9838-414724b93bda" are parents (predicate) of "6d7b0daf-2a6e-4fcc-9b7a-b83208fc4c5f"

It looks not readable, though the programming algorithm is to solve
the representation.

>  To take your example,
> (Joe HAS employee status) SINCE 2012-10-28, it'd be something like:
> 
> Subject: Joe
> Predicate: /employment/status
> Object: "6988a214-368c-40a2-9e51-8a7f5549de44"

I think that /employment/status as predicate shall be simple
relationship:

Entry d7b270ba-65cb-46ba-9702-34e9bf102c47 says that:
employment (UUID) has status (another UUID)

and then 
Entry e65b3519-1009-4812-a15d-151f1627fdd9 says that:
Joe (in reality UUID) has d7b270ba-65cb-46ba-9702-34e9bf102c47

Which would mean Joe has employment status. 

> Subject: "6988a214-368c-40a2-9e51-8a7f5549de44"
> Predicate: /employment/start
> Object: "2012-10-28"

That for me would rather be:

Entry 11e7bb1b-ebe0-46c4-8706-53cceeae7b76 says that:
Employment (UUID like c9d17440-d165-49d3-b545-cd0dd0f9f4d6) has Start date (UUID)

Then
Entry 8186ab12-fbe6-40e5-b266-49d71350eb30 says that:
e65b3519-1009-4812-a15d-151f1627fdd9 (Joe's employment status) is 2022-11-03 (but again as UUID)

With or without UUID, or ID, one shall minimize the types of
predicates, maximize the relationship definition, as that personally
seem to be easier for artificial intelligence programming.

Type shall not be defined strictly in the database, but rather be
defined in triplets.

Entry abac4ea3-c534-42d3-a3e4-921ce99e7c49 says that
Start Date (11e7bb1b-ebe0-46c4-8706-53cceeae7b76) IS Date (11e7bb1b-ebe0-46c4-8706-53cceeae7b76)

then:
Date (11e7bb1b-ebe0-46c4-8706-53cceeae7b76) is Type (68ad33a3-c4f1-4781-8123-cefc31ba6de7)

then you can easily get information like "list of types", types may
have its "formats", one can see which information is "Date" or list
employees by date.

Now I am thinking it should be even simpler:

subjectobjects should by something like:

UUID 
Value

triplets should be:

UUID with predicate also being UUID relates to UUID

Then

UUID "Predicate"
UUID "Is"

UUID of "Is" relates to UUID of Predicate

That way you get even predicates as objects/subjects

If we use IDs as integers, it may get problematic.

CREATE TABLE objects (
objects_uuid UUID PRIMARY KEY DEFAULT gen_random_uuid(),
objects_datecreated TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
objects_datemodified TIMESTAMP,
objects_usercreated TEXT NOT NULL DEFAULT current_user,
objects_usermodified TEXT NOT NULL DEFAULT current_user,
objects_name TEXT NOT NULL,
objects_description TEXT
);

With those 2 tables like "objects" and "triplets" one should be able
to describe anything, and due to UUIDs, even anything across the
databases, objects need not exist or be referencable in the single
database, they can be anything in that case. 

Everything else on upper level remains for programmer to define its
representations, inputs, editing, etc. 

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/