Embedding SQLite

Embedding SQLite

[Florian Weimer]

Would an interface to SQLite be acceptable for integration into Emacs?

I'm looking into improving the performance of large nnml folders in
Gnus, and storing the .overview data in an SQLite database might be an
option.

SQLite is public domain and could be shipped with GNU Emacs.  It is
supposed to be quite portable.

Re: Embedding SQLite

[Deniz Dogan]

2010/4/17 Florian Weimer <fw@deneb.enyo.de>:
> Would an interface to SQLite be acceptable for integration into Emacs?
>
> I'm looking into improving the performance of large nnml folders in
> Gnus, and storing the .overview data in an SQLite database might be an
> option.
>
> SQLite is public domain and could be shipped with GNU Emacs.  It is
> supposed to be quite portable.
>
>
>

I for one would definitely like some SQLite support in Emacs.

-- 
Deniz Dogan

Re: Embedding SQLite

[John Wiegley]

On Apr 17, 2010, at 5:58 PM, Deniz Dogan wrote:

> I for one would definitely like some SQLite support in Emacs.

I'm very in favor of this too, but it creates a new external dependency.  Also, if we use dynamic loading and then a user doesn't have it, now they can't use a Lisp module -- something which has never been true before.  Most modules will even work on DOS.

John

Re: Embedding SQLite

[Juanma Barranquero]

On Sun, Apr 18, 2010 at 05:21, John Wiegley <jwiegley@gmail.com> wrote:

> Also, if we use dynamic loading and then a user doesn't have it, now they can't use a Lisp module -- something which has never been true before.

There are lots of image types that you cannot display if you don't
have the libraries (which, on Windows, are dynamically loaded). How is
that different? And how is that different of the packages that require
external programs?

    Juanma

Re: Embedding SQLite

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, Apr 18, 2010 at 05:52:23AM +0200, Juanma Barranquero wrote:
> On Sun, Apr 18, 2010 at 05:21, John Wiegley <jwiegley@gmail.com> wrote:
> 
> > Also, if we use dynamic loading and then a user doesn't have it, now
> > they can't use a Lisp module -- something which has never been true before.
> 
> There are lots of image types that you cannot display if you don't
> have the libraries (which, on Windows, are dynamically loaded). How is
> that different? And how is that different of the packages that require
> external programs?

I must second John's concern here. This increasing dependency on
external libraries (which is set at build time!) tends to become a
nuisance in binary distributions. For example, in Debian, Emacs already
pulls in libdbus. Now libselinux is on the horizon. Then it's sqlite.
I'd fear it won't scale.

For me the solution will be to compile Emacs from source, but that isn't a
solution for everybody.

I haven't got a good idea either. Maybe the problem is in binary
distributions themselves.

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLypsKBcgs9XrR2kYRAo+JAJwJrkgEuGnG3a/PVEiDzZ8w+xzMJwCfWtup
QFWki5gXavgopor4DyI1Hvs=
=QKU4
-----END PGP SIGNATURE-----

Re: Embedding SQLite

[joakim]

tomas@tuxteam.de writes:

> On Sun, Apr 18, 2010 at 05:52:23AM +0200, Juanma Barranquero wrote:
>> On Sun, Apr 18, 2010 at 05:21, John Wiegley <jwiegley@gmail.com> wrote:
>> 
>> > Also, if we use dynamic loading and then a user doesn't have it, now
>> > they can't use a Lisp module -- something which has never been true before.
>> 
>> There are lots of image types that you cannot display if you don't
>> have the libraries (which, on Windows, are dynamically loaded). How is
>> that different? And how is that different of the packages that require
>> external programs?
>
> I must second John's concern here. This increasing dependency on
> external libraries (which is set at build time!) tends to become a
> nuisance in binary distributions. For example, in Debian, Emacs already
> pulls in libdbus. Now libselinux is on the horizon. Then it's sqlite.
> I'd fear it won't scale.
>
> For me the solution will be to compile Emacs from source, but that isn't a
> solution for everybody.
>
> I haven't got a good idea either. Maybe the problem is in binary
> distributions themselves.

People are looking into using a similar scheme for dynamic loadable
plugins in Emacs as already used in GCC. If an acceptable solution is
found, external libraries can be plugged in at runtime.

I've looked briefly at how GCC does it, and basically they seem to check
for a symbol that indicates gpl compliance.

That said, IMHO I would want something more lispy to to store lisp
structures efficiently, and something more JDBC:ish to communicate with
RDBMS:es. If one just want name/value store, there are many other
interesting NOSQL type solutions, like Apacha Cassandra, etc.

OK, sorry for hijacking this thread.

>
> Regards
> -- tomás
-- 
Joakim Verona

FFI (was: Embedding SQLite)

[Juri Linkov]

> People are looking into using a similar scheme for dynamic loadable
> plugins in Emacs as already used in GCC. If an acceptable solution is
> found, external libraries can be plugged in at runtime.

I have information that SXEmacs already implements support for
Foreign Function Interface to several libraries including SQLite
and Imagemagick's libWand.

If using its implementation in GNU Emacs is not possible due to legal
reasons, then maybe at least it would be a good idea to implement
FFI in GNU Emacs compatible with SXEmacs.  In this case all existing
library bindings will be readily available for GNU Emacs.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: FFI

[Florian Weimer]

* Juri Linkov:

> If using its implementation in GNU Emacs is not possible due to legal
> reasons, then maybe at least it would be a good idea to implement
> FFI in GNU Emacs compatible with SXEmacs.  In this case all existing
> library bindings will be readily available for GNU Emacs.

This is only true if Emacs also implements all SXEmacs types, so it is
a bit more ambitious than just implementing the FFI.

Re: FFI

[Zajcev Evgeny]

Florian Weimer <fw@deneb.enyo.de> writes:

> * Juri Linkov:
>
>> If using its implementation in GNU Emacs is not possible due to legal
>> reasons, then maybe at least it would be a good idea to implement
>> FFI in GNU Emacs compatible with SXEmacs.  In this case all existing
>> library bindings will be readily available for GNU Emacs.
>
> This is only true if Emacs also implements all SXEmacs types, so it is
> a bit more ambitious than just implementing the FFI.

I think I can port SXEmacsen FFI to GNU Emacs with small blood.
SXEmacsen FFI does not uses lisp types (it does actually, but this is
another question), so you can create almost any data accessible from
FFI with only numbers, floats and strings.  Let me show live example:

    (setq db (sqlite-open "~/mtest.db"))
      ==> #<ffiobject type=pointer size=4 fotype=0 foptr=0xb6a6600>
    (sqlite-execute db "CREATE TABLE PARAMS
       (NAME VARCHAR(256) NULL, VALUE VARCHAR(1024) NULL)")
      ==> t
    (sqlite-execute db "INSERT INTO PARAMS (NAME,VALUE)
       VALUES (?, ?)" '("nthoteh" "HRCHRCCR"))
      ==> t
    (sqlite-rows db "select * from params")
      ==> (("nthoteh" "HRCHRCCR"))
    (sqlite-close db)
      ==> 0

As you can see there are only strings, and numbers (boolean type is
done via dinamic type/value translators implemented in ffi)

thanks

-- 
lg

Re: FFI

[Florian Weimer]

* Zajcev Evgeny:

> As you can see there are only strings, and numbers (boolean type is
> done via dinamic type/value translators implemented in ffi)

Numbers are problematic because SQLite can return 64 bit signed
integers.  AFAICS, those are readily represented on SXEmacs, but not
on Emacs.

Re: FFI

[Zajcev Evgeny]

Florian Weimer <fw@deneb.enyo.de> writes:

> * Zajcev Evgeny:
>
>> As you can see there are only strings, and numbers (boolean type is
>> done via dinamic type/value translators implemented in ffi)
>
> Numbers are problematic because SQLite can return 64 bit signed
> integers.  AFAICS, those are readily represented on SXEmacs, but not
> on Emacs.

that won't be a problem to get it working after all, you can represent
ffi data on lisp level as you want it to, using types translators.
Types translator is lisp level thingie..

my interest is to have compatible FFI in GNU Emacs and (S)XEmacs (at
least on high level, read lisp level), so two communities could reuse
bindings.  SXEmacs already have working and pretty tested interfaces
to:

 - Xlib
 - curl
 - sqlite
 - Wand        
 - libgcrypt

and few others.

Wand bindings enabled us to implement pretty astonishing image
viewer/editor inside SXEmacs, it can even do red-eye reduction on
production level and mordern liquid rescale :) - to get more about, it
click on:

http://sxemacs-en.blogspot.com/2009/12/wand-mode-image-viewer.html
and
http://sxemacs-en.blogspot.com/2009/12/wand-mode-new-features_27.html

bindings implementations mostly does not use any SXEmacs specials, the
code is very common and I'm pretty sure it can be used in GNU Emacs
without much problems.

We are also working on some complex, but interesting bindings to
libtelepathy in order to create tool for multiprotocol IM and
collaborative editing in SXEmacs.  This is pretty time consuming task
and we surely need a help, if GNU Emacs community will have same FFI
API then we could unite our efforts to enhance Emacsen..

As I noticed before, I can port and maintain FFI implementation for
GNU Emacs, but i need some kind of guaranties that community will
accept it, so my work won't be a waste of time..

thanks

-- 
lg

Re: FFI

[joakim]

Zajcev Evgeny <lg.zevlg@gmail.com> writes:

> Florian Weimer <fw@deneb.enyo.de> writes:
>
>> * Zajcev Evgeny:
>>
>>> As you can see there are only strings, and numbers (boolean type is
>>> done via dinamic type/value translators implemented in ffi)
>>
>> Numbers are problematic because SQLite can return 64 bit signed
>> integers.  AFAICS, those are readily represented on SXEmacs, but not
>> on Emacs.
>
> that won't be a problem to get it working after all, you can represent
> ffi data on lisp level as you want it to, using types translators.
> Types translator is lisp level thingie..
>
> my interest is to have compatible FFI in GNU Emacs and (S)XEmacs (at
> least on high level, read lisp level), so two communities could reuse
> bindings.  SXEmacs already have working and pretty tested interfaces
> to:
>
>  - Xlib
>  - curl
>  - sqlite
>  - Wand        
>  - libgcrypt
>
> and few others.
>
> Wand bindings enabled us to implement pretty astonishing image
> viewer/editor inside SXEmacs, it can even do red-eye reduction on
> production level and mordern liquid rescale :) - to get more about, it
> click on:
>
> http://sxemacs-en.blogspot.com/2009/12/wand-mode-image-viewer.htmland
> http://sxemacs-en.blogspot.com/2009/12/wand-mode-new-features_27.html
>
> bindings implementations mostly does not use any SXEmacs specials, the
> code is very common and I'm pretty sure it can be used in GNU Emacs
> without much problems.
>
> We are also working on some complex, but interesting bindings to
> libtelepathy in order to create tool for multiprotocol IM and
> collaborative editing in SXEmacs.  This is pretty time consuming task
> and we surely need a help, if GNU Emacs community will have same FFI
> API then we could unite our efforts to enhance Emacsen..
>
> As I noticed before, I can port and maintain FFI implementation for
> GNU Emacs, but i need some kind of guaranties that community will
> accept it, so my work won't be a waste of time..

My understanding is that a FFI for Emacs must comply to a
interface, which is not yet defined. This interface will probably be similar
to the plugin api in GCC. Basically, the dll to be loaded must export a GPL
compliance symbol. It ought to be possible to add this check to your FFI
implementation, when running in Emacs, even if you do not use it in SXEmacs.


> thanks
-- 
Joakim Verona

Re: FFI

[Zajcev Evgeny]

joakim@verona.se writes:

> Zajcev Evgeny <lg.zevlg@gmail.com> writes:
[..]
>>
>> As I noticed before, I can port and maintain FFI implementation for
>> GNU Emacs, but i need some kind of guaranties that community will
>> accept it, so my work won't be a waste of time..
>
> My understanding is that a FFI for Emacs must comply to a interface,
> which is not yet defined. This interface will probably be similar to
> the plugin api in GCC. Basically, the dll to be loaded must export a
> GPL compliance symbol. It ought to be possible to add this check to
> your FFI implementation, when running in Emacs, even if you do not
> use it in SXEmacs.

this is doable, of course GNU Emacs will have some specials about FFI
(as one you've described), but at least it will have same lisp level
API for code reuse between communities

thanks

-- 
lg

telepathy (was: FFI)

[Michael Albinus]

Zajcev Evgeny <lg.zevlg@gmail.com> writes:

> We are also working on some complex, but interesting bindings to
> libtelepathy in order to create tool for multiprotocol IM and
> collaborative editing in SXEmacs.  This is pretty time consuming task
> and we surely need a help, if GNU Emacs community will have same FFI
> API then we could unite our efforts to enhance Emacsen..

Some months ago I've started with telepathy.el, using Telepathy's D-Bus
API. This work is currently stalled due to lack of time :-(

Wouldn't it be an approach to join our efforts?

(I know, I also need to review recent patches in SXEmacs for D-Bus, first)

> thanks

Best regards, Michael.

Re: telepathy

[Zajcev Evgeny]

Michael Albinus <michael.albinus@gmx.de> writes:

> Zajcev Evgeny <lg.zevlg@gmail.com> writes:
>
>> We are also working on some complex, but interesting bindings to
>> libtelepathy in order to create tool for multiprotocol IM and
>> collaborative editing in SXEmacs.  This is pretty time consuming task
>> and we surely need a help, if GNU Emacs community will have same FFI
>> API then we could unite our efforts to enhance Emacsen..
>
> Some months ago I've started with telepathy.el, using Telepathy's D-Bus
> API. This work is currently stalled due to lack of time :-(
>
> Wouldn't it be an approach to join our efforts?

sure, I'll describe my story:

Some time ago, I've implemented partial FFI bindings to:

 - libempathy
 - libglib2
 - libgobject2
 - libdbus-glib
 - libgtk-x11
 - libmissioncontrol-client

it was enough bindings to implement skeleton for IM.  It was able to
connect, send, receive messages from any IM protocol supported by
empathy, fetch avatars, private info etc - pretty limited, but enough
functionality IM skeleton.  I've been even using it for couple of
monthes as my primary gtalk client.  But empathy got evolved, their
API was extremely unstable, finally they decided to get rid of
libempathy in favor to libtelepathy.  Now they are in active position
in deciding what to put and what to remove from libtelepathy, they are
waiting for recommendations from developers about its (libtelepathy)
functionality.  I'm now collecting some kind of 'recommendations list'
for empathy community to make their codebase reusable and more
friendly for other developers.  My most intentions is to have two
kinds of API in libtelepathy:

 - Low level D-Bus stuff, that they already have
 - Stable high level API that would resemble former libempathy

Having such diversion will allow developers (other then empathy
community) to easily implement IMs, and to lower the level at any time
they need a details.

As to Emacs, I'm very sure that model when emacs does use external
programs and wraps them into lisp level functionality is great!, but
in some circumstances that is not enough flexible, sometimes you need
direct access to API.  Each time implementing C-to-lisp wrappers for
different APIs is not that great as having solid FFI implementation,
when you can write such wrappers entirely in lisp.  So Emacsen C code
for D-Bus, ImageMagick, etc is inferior to 100% pure lisp (without
performance impact!) implementation in SXEmacs using FFI.

thanks

-- 
lg

Re: telepathy

[Michael Albinus]

Zajcev Evgeny <lg.zevlg@gmail.com> writes:

> As to Emacs, I'm very sure that model when emacs does use external
> programs and wraps them into lisp level functionality is great!, but
> in some circumstances that is not enough flexible, sometimes you need
> direct access to API.  Each time implementing C-to-lisp wrappers for
> different APIs is not that great as having solid FFI implementation,
> when you can write such wrappers entirely in lisp.

That's true when you have FFI implemented in a compatible way in SXEmacs
and GNU Emacs. For the time being, using a D-Bus API is already possible
in both Emacsen (given, that I finish my tasks on SXEmacs).

> So Emacsen C code for D-Bus, ImageMagick, etc is inferior to 100% pure
> lisp (without performance impact!) implementation in SXEmacs using
> FFI.

Yes, that would ease access to such libraries. If FFI would be available
in GNU Emacs, one could think about rewriting those libraries in Lisp.

> thanks

Best regards, Michael.

Re: telepathy (was: FFI)

[Jan Moringen]

Hi.

On Mon, 2010-04-19 at 09:44 +0200, Michael Albinus wrote:
> Zajcev Evgeny <lg.zevlg@gmail.com> writes:
> 
> > We are also working on some complex, but interesting bindings to
> > libtelepathy in order to create tool for multiprotocol IM and
> > collaborative editing in SXEmacs.  This is pretty time consuming task
> > and we surely need a help, if GNU Emacs community will have same FFI
> > API then we could unite our efforts to enhance Emacsen..
> 
> Some months ago I've started with telepathy.el, using Telepathy's D-Bus
> API. This work is currently stalled due to lack of time :-(
> 
> Wouldn't it be an approach to join our efforts?

I was also working on a D-Bus-based telepathy (or mission-control, to be
precise) library as a backend for Rudel [1]. My focus was on enumeration
of contacts and creation of telepathy tubes. I wrote transparent proxy
objects for D-Bus objects using EIEIO in the process. I can commit the
code to the Rudel repository, if you are interested.

Kind regards,
Jan

[1] http://rudel.sourceforge.net/

Re: telepathy

[Zajcev Evgeny]

Jan Moringen <jan.moringen@uni-bielefeld.de> writes:

> Hi.
[..]
>> Some months ago I've started with telepathy.el, using Telepathy's D-Bus
>> API. This work is currently stalled due to lack of time :-(
>> 
>> Wouldn't it be an approach to join our efforts?
>
> I was also working on a D-Bus-based telepathy (or mission-control, to be
> precise) library as a backend for Rudel [1]. My focus was on enumeration
> of contacts and creation of telepathy tubes. I wrote transparent proxy
> objects for D-Bus objects using EIEIO in the process. I can commit the
> code to the Rudel repository, if you are interested.

of course, please commit it.  Thanks for pointing to rudel, this is
great project!

-- 
lg

Re: telepathy

[Michael Albinus]

Jan Moringen <jan.moringen@uni-bielefeld.de> writes:

> Hi.

Hi,

> I was also working on a D-Bus-based telepathy (or mission-control, to be
> precise) library as a backend for Rudel [1]. My focus was on enumeration
> of contacts and creation of telepathy tubes. I wrote transparent proxy
> objects for D-Bus objects using EIEIO in the process. I can commit the
> code to the Rudel repository, if you are interested.

Looks very interesting. Please commit; I would like to see the code :-)

> Kind regards,
> Jan

Best regards, Michael.

Re: telepathy

[Jan Moringen]

Hi.

> > I was also working on a D-Bus-based telepathy (or mission-control, to be
> > precise) library as a backend for Rudel [1]. My focus was on enumeration
> > of contacts and creation of telepathy tubes. I wrote transparent proxy
> > objects for D-Bus objects using EIEIO in the process. I can commit the
> > code to the Rudel repository, if you are interested.
> 
> Looks very interesting. Please commit; I would like to see the code :-)

OK, after some cleanup, I committed the (still untidy) code to the Rudel
bzr repository. It can be found in the "future" (read: junkyard) branch
here:

http://rudel.bzr.sourceforge.net/bzr/rudel/branches/future

The relevant files are:

telepathy/dbus-introspection.el  some helper functions
telepathy/dbus-proxy.el          lisp proxies for D-Bus objects

telepathy/telepathy-mission-control-4.el  mission control bindings 
                                          for version 4
telepathy/telepathy-mission-control-5.el  mission control bindings 
                                          for version 5

The mission control bindings are very incomplete since they depend on
the proxy code, which is at best barely complete.

I did not commit the Rudel-specific code, since it cannot be of any use
to anyone, yet :)

If you have questions, suggestions or improvements, let me know.

Kind regards,
Jan

Re: FFI

[Florian Weimer]

* Stefan Monnier:

>> my interest is to have compatible FFI in GNU Emacs and (S)XEmacs (at
>> least on high level, read lisp level), so two communities could reuse
>> bindings.
>
> That's fine by me.  The main problem here is to figure out if and how we
> can support FFIs, not from a technical standpoint but from
> a legal/political standpoint.  Until recently RMS's word on it was just
> "no way", but there's hope we can do something like what GCC does.
>
> So, for now, we're just waiting for the word of the FSF's legal
> adviser(s?).  Don't hold your breath.  But we'll be sure to get in touch
> if/when it turns out we can finally add an FFI.

Uh-oh.  SQLite support will bring in some sort of FFI as well, albeit
rather indirectly, so I guess that means it's not possible to add it
until that is settled. 8-(

Re: FFI

[Andreas Schwab]

Florian Weimer <fw@deneb.enyo.de> writes:

> Numbers are problematic because SQLite can return 64 bit signed
> integers.

Perhaps a good opportunity to add bignum support to Emacs. :-)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Embedding SQLite

[Florian Weimer]

* John Wiegley:

> On Apr 17, 2010, at 5:58 PM, Deniz Dogan wrote:
>
>> I for one would definitely like some SQLite support in Emacs.
>
> I'm very in favor of this too, but it creates a new external
> dependency.

Optionally, SQLite comes as a single C file which could be put into
the distribution tarball.

Re: Embedding SQLite

[Leo]

On 2010-04-18 04:21 +0100, John Wiegley wrote:
>> I for one would definitely like some SQLite support in Emacs.
>
> I'm very in favor of this too, but it creates a new external
> dependency. Also, if we use dynamic loading and then a user doesn't
> have it, now they can't use a Lisp module -- something which has never
> been true before. Most modules will even work on DOS.

I have been using and sticking to an emacs-based personal information
system including calendar, org, bbdb, gnus etc and have experienced many
of its limitations. For example, syncing with other applications is a
real pain. The fact is if emacs fails to improve to meet users' needs,
they will migrate away from it. For example, by using Apple Mail for
emails, iCal for calendar and eventually leave emacs forever.

This SQLite dependency should not be treated like other dependencies. A
new PIM can be designed upon it and make emacs appeal to every computer
user.

Leo

Re: Embedding SQLite

[Florian Weimer]

* Stefan Monnier:

>> Would an interface to SQLite be acceptable for integration into Emacs?
>
> Depends: is the linkage done at the C level with "libsqlite.so" or is it
> done at the Lisp level via the command-line sqlite program?

The former, or perhaps compiling sqlite3.c and linking that if no
system library is available.

SQLite is dynamically typed, and the command line tool cannot
adequately confer type information at all times.  It's much easier at
the API level.  You also avoid most SQL injection issues.

Re: Embedding SQLite

[Ted Zlatanov]

On Sat, 17 Apr 2010 21:55:23 +0200 Florian Weimer <fw@deneb.enyo.de> wrote: 

FW> Would an interface to SQLite be acceptable for integration into Emacs?
FW> I'm looking into improving the performance of large nnml folders in
FW> Gnus, and storing the .overview data in an SQLite database might be an
FW> option.

FW> SQLite is public domain and could be shipped with GNU Emacs.  It is
FW> supposed to be quite portable.

I am in favor of doing this.  In my experience SQLite has been fast and
reliable, plus it can be controlled with SQL which is a well-known
language.  It would be nice if it was possible to serialize/deserialize
hashtables and lists directly from a SQLite table.

It may seem like "just another library" but I'm tired of the 30 ways in
Emacs to serialize data and then load it back (all of them
inefficient and incompatible with each other).  Gnus, at least, would
benefit significantly from this.

I wouldn't make it an external dependency, though.  Bring it into Emacs
itself and update the C source when an update is released.

Ted