sqlite3

sqlite3

[Lars Ingebrigtsen]

I think many user experience things in Emacs would be better if Emacs
remembered more.

The bigger systems don't have this problem -- Gnus needs a large
.newsrc.eld file, and it maintains that.

The tiny things don't really have this problem, either: You save options
with `customize-save-variable', and that fine.

It's the many things that fall between these two extremes that have the
problem: Where you want to store some state, but figuring it's just too
much work to figure out Yet Another Storage Format, but the data is too
messy to stash in somebody's .emacs file via Customize (like lists of
stuff).

So there's a bunch of stuff that Emacs just forgets when you shut down,
where it perhaps shouldn't.

I've brought this up before, but I didn't really have a solution then,
but I think I do now: sqlite3.

sqlite3 is supported on more platforms than Emacs is, the interface is
small and stable, and (best of all) somebody has already created
interface functions for Emacs (via a module):

  https://github.com/syohex/emacs-sqlite3

I think it'd be good to bring that into core, and then write a small
wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
that we don't have to write SQL all over the place.  That is:

(setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)

I.e., what Emacs needs is a persistent key/value store, and this would
give us that.

In addition, if somebody really wants to write SQL stuff (it can be very
handy for some things), having sqlite3 in there gives us that in
addition for free.

This comes with questions about how the users are supposed be able to
clear out the data, for instance, but we could have a `M-x
list-persistent-data' where the users could blow out whatever they want
to.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Yuan Fu]

> On Dec 5, 2021, at 5:51 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> I think many user experience things in Emacs would be better if Emacs
> remembered more.
> 
> The bigger systems don't have this problem -- Gnus needs a large
> .newsrc.eld file, and it maintains that.
> 
> The tiny things don't really have this problem, either: You save options
> with `customize-save-variable', and that fine.
> 
> It's the many things that fall between these two extremes that have the
> problem: Where you want to store some state, but figuring it's just too
> much work to figure out Yet Another Storage Format, but the data is too
> messy to stash in somebody's .emacs file via Customize (like lists of
> stuff).
> 
> So there's a bunch of stuff that Emacs just forgets when you shut down,
> where it perhaps shouldn't.
> 
> I've brought this up before, but I didn't really have a solution then,
> but I think I do now: sqlite3.
> 
> sqlite3 is supported on more platforms than Emacs is, the interface is
> small and stable, and (best of all) somebody has already created
> interface functions for Emacs (via a module):
> 
>  https://github.com/syohex/emacs-sqlite3
> 
> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.  That is:
> 
> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
> 
> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.
> 
> In addition, if somebody really wants to write SQL stuff (it can be very
> handy for some things), having sqlite3 in there gives us that in
> addition for free.

Cool idea. Stuff like autosave, backup and recentf could benefit from a database. Other things could be REPL histories, transient settings and histories, etc. If everyone uses sqlite, .emacs.d could be less littered, too. Especially for autosave and backup, they are a bit tricky to configure so that Emacs doesn’t throw autosave and backup files all over the place.

> This comes with questions about how the users are supposed be able to
> clear out the data, for instance, but we could have a `M-x
> list-persistent-data' where the users could blow out whatever they want
> to.

Maybe also record time automatically, so a user can clear data based on age.

Yuan

Re: sqlite3

[Lars Ingebrigtsen]

Yuan Fu <casouri@gmail.com> writes:

> Especially for autosave and backup, they are a bit tricky to configure
> so that Emacs doesn’t throw autosave and backup files all over the
> place.

Well...  not really.  Quoth the EFAQ:

   For instance, to write all these things to ‘~/.emacs.d/aux/’:

     (setq lock-file-name-transforms
           '(("\\`/.*/\\([^/]+\\)\\'" "~/.emacs.d/aux/\\1" t)))
     (setq auto-save-file-name-transforms
           '(("\\`/.*/\\([^/]+\\)\\'" "~/.emacs.d/aux/\\1" t)))
     (setq backup-directory-alist
           '((".*" . "~/.emacs.d/aux/")))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1318 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:


Having sqlite3 interface as you describe would also benefit custom a
lot, the emacs .custom file is beginning to look like the Windows
Registry:-)

Some of the nice things that we might want in  a more structured world:

A. Organize, backup and restore customization per package -- at present,
   if you use a package for a while, then delete it, its customizations
   hang around in the custom file as an example.

B. Having custom options that hold large lists e.g. your favorite RSS
   feeds, gets unweildy

and a lot more ...
> Yuan Fu <casouri@gmail.com> writes:
>
>> Especially for autosave and backup, they are a bit tricky to configure
>> so that Emacs doesn¡¯t throw autosave and backup files all over the
>> place.
>
> Well...  not really.  Quoth the EFAQ:
>
>    For instance, to write all these things to ¡®~/.emacs.d/aux/¡¯:
>
>      (setq lock-file-name-transforms
>            '(("\\`/.*/\\([^/]+\\)\\'" "~/.emacs.d/aux/\\1" t)))
>      (setq auto-save-file-name-transforms
>            '(("\\`/.*/\\([^/]+\\)\\'" "~/.emacs.d/aux/\\1" t)))
>      (setq backup-directory-alist
>            '((".*" . "~/.emacs.d/aux/")))

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: sqlite3

[Qiantan Hong]

> Having sqlite3 interface as you describe would also benefit custom a
> lot, the emacs .custom file is beginning to look like the Windows
> Registry:-)
I think most, if not all, of the possible improvements for custom are
unrelated to sqlite3.

> A. Organize, backup and restore customization per package -- at present,
>   if you use a package for a while, then delete it, its customizations
>   hang around in the custom file as an example.
We just need better custom.el, no need for sqlite3.

> B. Having custom options that hold large lists e.g. your favorite RSS
>   feeds, gets unweildy
For the UI issue, sqlite3 is again irrelevant — we just need
to think about better way to present/edit large list.

For performance issue, sqlite3 might be one solution. I’m not
sure how much it will help — if we’re printing out the list
as string then throw into sqlite3, it won’t help with performance either.

Re: sqlite3

[Stefan Monnier]

> I've brought this up before, but I didn't really have a solution then,
> but I think I do now: sqlite3.

I think the more significant benefit is to be able to correctly handle
cases where several Emacs sessions modify the persistent data.
I'm thinking of things like `savehist`.


        Stefan

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think the more significant benefit is to be able to correctly handle
> cases where several Emacs sessions modify the persistent data.
> I'm thinking of things like `savehist`.

Hm...  how will that work in an sqlite3 context?  I've worked a bit with
it, but I haven't looked into that bit.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Stefan Monnier]

Lars Ingebrigtsen [2021-12-06 06:28:55] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I think the more significant benefit is to be able to correctly handle
>> cases where several Emacs sessions modify the persistent data.
>> I'm thinking of things like `savehist`.
> Hm...  how will that work in an sqlite3 context?

No idea, but it's a really important feature in my experience.


        Stefan

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> No idea, but it's a really important feature in my experience.

Seems like it Should Just Work:  https://www.sqlite.org/faq.html#q5

(But of course, the process that writes the value last wins.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> (But of course, the process that writes the value last wins.)

If we're talking about lists that should be appended (sort of), we can
use the database as the checkpoint.  That is:

(push "foo" (persistent-value 'foo-list))

Could query the database and then add "foo" and write it back.  Barring
races, it'd "just work" for many things.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> It's the many things that fall between these two extremes that have the
> problem: Where you want to store some state, but figuring it's just too
> much work to figure out Yet Another Storage Format, but the data is too
> messy to stash in somebody's .emacs file via Customize (like lists of
> stuff).
>
> So there's a bunch of stuff that Emacs just forgets when you shut down,
> where it perhaps shouldn't.
>
> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.  That is:
>
> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>
> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.
>
> In addition, if somebody really wants to write SQL stuff (it can be very
> handy for some things), having sqlite3 in there gives us that in
> addition for free.
>
> This comes with questions about how the users are supposed be able to
> clear out the data, for instance, but we could have a `M-x
> list-persistent-data' where the users could blow out whatever they want
> to.

Wouldn't recfiles be better for this purpose?

Re: sqlite3

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Wouldn't recfiles be better for this purpose?

Are those fast?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Wouldn't recfiles be better for this purpose?
>
> Are those fast?

Not nearly as fast as sqlite3, but they should be enough for most
things.

Re: sqlite3

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Not nearly as fast as sqlite3, but they should be enough for most
> things.

Speed is a feature here -- we want packages to be able to update the
persistent storage at will, and as often as they want.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Speed is a feature here -- we want packages to be able to update the
> persistent storage at will, and as often as they want.

Will something like dbm (or gdbm) suffice then?

Re: sqlite3

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Will something like dbm (or gdbm) suffice then?

I don't see anything to recommend those over sqlite3.  (I've used
variants of those a lot over the years, and they're pretty easy to
break, while sqlite3 is pretty resilient.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Po Lu <luangruo@yahoo.com> writes:

>> Will something like dbm (or gdbm) suffice then?

> I don't see anything to recommend those over sqlite3.  (I've used
> variants of those a lot over the years, and they're pretty easy to
> break, while sqlite3 is pretty resilient.)

The advantage is that dbm (and its variants) are available anywhere, I
think.  sqlite3 might not be that ubiquitous.

Re: sqlite3

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> The advantage is that dbm (and its variants) are available anywhere, I
> think.  sqlite3 might not be that ubiquitous.

I'm not aware of any systems that don't have sqlite3 -- it's extremely
portable.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> The advantage is that dbm (and its variants) are available anywhere, I
>> think.  sqlite3 might not be that ubiquitous.

> I'm not aware of any systems that don't have sqlite3 -- it's extremely
> portable.

That's not what I meant: AFAIU, dbm is installed on almost all Unix-like
systems, which is not the case with sqlite3.

Re: sqlite3

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> That's not what I meant: AFAIU, dbm is installed on almost all Unix-like
> systems, which is not the case with sqlite3.

By "installed", do you mean "is in /usr/lib/*/libsqlite* already"?
Since so many things depend on sqlite3, I'd be surprised.

In any case, sqlite3 is available on all Unix-like systems I know of, so
installing Emacs would install the library, so I'm not sure I understand
the argument.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Tim Cross]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> That's not what I meant: AFAIU, dbm is installed on almost all Unix-like
>> systems, which is not the case with sqlite3.
>
> By "installed", do you mean "is in /usr/lib/*/libsqlite* already"?
> Since so many things depend on sqlite3, I'd be surprised.
>

While I cannot remember which package it was, one of the package.el
packages I was trying out recently had a dependency on sqlite3.
Initially, it failed on my system because it was not installed. This was
a recent Ubuntu system. Not a big deal as it was available and easy to
install, but assuming it would already be installed is perhaps assuming
too much. 

> In any case, sqlite3 is available on all Unix-like systems I know of, so
> installing Emacs would install the library, so I'm not sure I understand
> the argument.

Yes, if the package maintainers specify it as a dependency it would be
installed. It would also be installed on systems which have a way to
install build dependencies - for example, on Debian systems, if you want
to build Emacs from source rather than use the *.deb package, you can
run 'apt install build-deps emacs' and all the build dependencies will be
installed for you. (though this may install dependencies you don't want
- for example, postfix and it might miss some, like the gcc library for
native comp which the current 27.2 does not need)>

As you say, sqlite3 is used by many things, so having it as a dependency
for Emacs is probably not a big deal.

Re: sqlite3

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

> As you say, sqlite3 is used by many things, so having it as a dependency
> for Emacs is probably not a big deal.

Anyhow, it should be an optional dependency, and we should have
fallbacks in place for when it is not available.

Re: sqlite3

[Colin Baxter 😺]

>>>>> Lars Ingebrigtsen <larsi@gnus.org> writes:

You might want to confirm that using sqlite would not potentially cause
licensing problems. They do not give the explicit license on
https://sqlite.org/copyright.html but confine themselves to the
statement that their code and documentation "has been dedicated to the
public domain". Further down the page, they say sqlite is "open-source"
and further down still they say a license, what they call a "Warranty of
Title for SQLite" may be purchased.

Best wishes,

Colin Baxter.

Re: sqlite3

[Po Lu]

Colin Baxter 😺 <m43cap@yandex.com> writes:

> You might want to confirm that using sqlite would not potentially cause
> licensing problems. They do not give the explicit license on
> https://sqlite.org/copyright.html but confine themselves to the
> statement that their code and documentation "has been dedicated to the
> public domain". Further down the page, they say sqlite is "open-source"
> and further down still they say a license, what they call a "Warranty of
> Title for SQLite" may be purchased.

The Free Software Directory lists SQLite as public domain, so I think
there is no problem here: https://directory.fsf.org/wiki/SQLite

Re: sqlite3

[Jean Louis]

* Po Lu <luangruo@yahoo.com> [2021-12-06 09:21]:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> >> The advantage is that dbm (and its variants) are available anywhere, I
> >> think.  sqlite3 might not be that ubiquitous.
> 
> > I'm not aware of any systems that don't have sqlite3 -- it's extremely
> > portable.
> 
> That's not what I meant: AFAIU, dbm is installed on almost all Unix-like
> systems, which is not the case with sqlite3.

One problem could be is that it is public domain.




-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Jean Louis]

* Lars Ingebrigtsen <larsi@gnus.org> [2021-12-06 08:45]:
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Not nearly as fast as sqlite3, but they should be enough for most
> > things.
> 
> Speed is a feature here -- we want packages to be able to update the
> persistent storage at will, and as often as they want.

sqlite3 is rather for applications, not for storage of simple data.

For storing key/value entries, I recommend DBM-like databases.

DBM (computing) - Wikipedia
https://en.wikipedia.org/wiki/DBM_(computing)

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think many user experience things in Emacs would be better if Emacs
> remembered more.
>
> The bigger systems don't have this problem -- Gnus needs a large
> .newsrc.eld file, and it maintains that.

A plea from the heart: Gnus also needs a better data store, and sqlite3
is the answer to that need. The .newsrc.eld format is not okay.

Emacs *needs* to ship with default database support, and that database
should be sqlite3. Everything talks to sqlite, and Emacs should, too.

I don't have many opinions about Emacs' direction of development, and I
rarely shout about anything, but please: csv and sqlite need to be built
in. We should be able to rely on them. Please make them part of the
distribution.

Re: sqlite3

[cesar mena]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> I think many user experience things in Emacs would be better if Emacs
>> remembered more.
>>
>> The bigger systems don't have this problem -- Gnus needs a large
>> .newsrc.eld file, and it maintains that.
>
> A plea from the heart: Gnus also needs a better data store, and sqlite3
> is the answer to that need. The .newsrc.eld format is not okay.

Unclear about this comment.  Lars specifically stated that this is for
things that fall in between tiny (use customize) and bigger (gnus etc ...):

   The bigger systems don't have this problem -- Gnus needs a large
   .newsrc.eld file, and it maintains that.

   The tiny things don't really have this problem, either: You save options
   with `customize-save-variable', and that fine.

Are you proposing a move to sqlite3 for .newsrc.eld?

> Emacs *needs* to ship with default database support, and that database
> should be sqlite3. Everything talks to sqlite, and Emacs should, too.
>
> I don't have many opinions about Emacs' direction of development, and I
> rarely shout about anything, but please: csv and sqlite need to be built
> in. We should be able to rely on them. Please make them part of the
> distribution.

Re: sqlite3

[Eric Abrahamsen]

cesar mena <cesar.mena@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> I think many user experience things in Emacs would be better if Emacs
>>> remembered more.
>>>
>>> The bigger systems don't have this problem -- Gnus needs a large
>>> .newsrc.eld file, and it maintains that.
>>
>> A plea from the heart: Gnus also needs a better data store, and sqlite3
>> is the answer to that need. The .newsrc.eld format is not okay.
>
> Unclear about this comment.  Lars specifically stated that this is for
> things that fall in between tiny (use customize) and bigger (gnus etc ...):
>
>    The bigger systems don't have this problem -- Gnus needs a large
>    .newsrc.eld file, and it maintains that.
>
>    The tiny things don't really have this problem, either: You save options
>    with `customize-save-variable', and that fine.
>
> Are you proposing a move to sqlite3 for .newsrc.eld?

Well, Lars is the Gnus maintainer, not me, so all I can do is propose!
But yes I am, though only for group marks. Right now .newsrc.eld
combines concerns: it saves variables such as `gnus-server-alist' and
all the topic stuff that would be better saved as customization options
(or saved in a separate file, using the customization machinery). It
also saves group parameters, right next to the marks.

I've long felt that the data above should be kept in a separate file,
that's expected to be user-readable and even (to a certain extent)
user-editable. It's *mostly* set through Gnus' own interface, but you
could certainly tweak it by hand if you wanted. Marks, however, are
meaningless to read and practically un-editable. Their presence in
.newsrc.eld makes the file slow to open and scroll, and has given rise
to a meme of "don't touch the .newsrc.eld file, you'll destroy
everything". That in itself is a sign that we're using the wrong format
for this data.

In terms raised elsewhere in the thread: the variable data is
user-editable, and should be version-controllable. The mark data is not
realistically editable through any interface but Gnus's own UI, and is
not appropriate for version control. Ergo, it should go in its own file,
and a sqlite file would be a perfect choice for that.

It seems likely that *requiring* sqlite support is a non-starter, so
this would only be an option. But ideally I'd like the options to be 1)
keep the marks in their own text file, that contains nothing but group
names -> mark info; 2) keep the marks in recutils; 3) keep the marks in
sqlite; 4) write your own mark-saving backend.

That's it!

Eric

Re: sqlite3

[Qiantan Hong]

> Marks, however, are
> meaningless to read and practically un-editable. Their presence in
> .newsrc.eld makes the file slow to open and scroll, and has given rise
> to a meme of "don't touch the .newsrc.eld file, you'll destroy
> everything". That in itself is a sign that we're using the wrong format
> for this data.
Is sqlite3 necessary?
Can we just print it out in a separate file as s-exp?

Re: sqlite3

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2021-12-06 23:37]:
> > Marks, however, are
> > meaningless to read and practically un-editable. Their presence in
> > .newsrc.eld makes the file slow to open and scroll, and has given rise
> > to a meme of "don't touch the .newsrc.eld file, you'll destroy
> > everything". That in itself is a sign that we're using the wrong format
> > for this data.
> Is sqlite3 necessary?
> Can we just print it out in a separate file as s-exp?

I do not see the motion specifically useful only to store persistent
data related to Emacs.

Rather I see usefulness in the waste future development with databases
interfaces: inventories, product lists, invoices, quotations, contact
address books, tags, documents, all kinds of applications may be
developed when database is easily accessible from Emacs by using Emacs
Lisp.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Well, Lars is the Gnus maintainer, not me, so all I can do is propose!
> But yes I am, though only for group marks. Right now .newsrc.eld
> combines concerns: it saves variables such as `gnus-server-alist' and
> all the topic stuff that would be better saved as customization options
> (or saved in a separate file, using the customization machinery). It
> also saves group parameters, right next to the marks.

It's a bit much, but I don't think those bits can be usefully separated
out.  But things like `gnus-save-duplicate-list' (which goes to a
separate, ad-hoc file) would fit well as a simple persistent variable.

> It seems likely that *requiring* sqlite support is a non-starter, so
> this would only be an option.

We don't really require much of anything, it's just that Emacs doesn't
work very well without some things.  Like gnutls.  This would be in the
same category, so I'd expect 99.7% of Emacsen would have support for it
toot sweet.

(Except for those that hate SQL sooooo much that they can't even stand
the thought of something in Emacs talking SQL even if they can't see
it.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Well, Lars is the Gnus maintainer, not me, so all I can do is propose!
>> But yes I am, though only for group marks. Right now .newsrc.eld
>> combines concerns: it saves variables such as `gnus-server-alist' and
>> all the topic stuff that would be better saved as customization options
>> (or saved in a separate file, using the customization machinery). It
>> also saves group parameters, right next to the marks.
>
> It's a bit much, but I don't think those bits can be usefully separated
> out.  But things like `gnus-save-duplicate-list' (which goes to a
> separate, ad-hoc file) would fit well as a simple persistent variable.

If `gnus-newsrc-alist' were saved as-is, but *without* the mark data,
that mark data could be pulled out of a separate storage backend at
startup time, with a simple group-name->mark-data mapping. The
.newsrc.eld file could otherwise remain the same.

But Qiantan's message reminded me that this proposal is actually a Gnus
issue, not a sqlite issue, and shouldn't clog up this thread. I'll keep
this for later!

Re: sqlite3

[cesar mena]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Well, Lars is the Gnus maintainer, not me, so all I can do is propose!
>>> But yes I am, though only for group marks. Right now .newsrc.eld
>>> combines concerns: it saves variables such as `gnus-server-alist' and
>>> all the topic stuff that would be better saved as customization options
>>> (or saved in a separate file, using the customization machinery). It
>>> also saves group parameters, right next to the marks.

Yes, it's a bit of a big hammer. 

>> It's a bit much, but I don't think those bits can be usefully separated
>> out.  But things like `gnus-save-duplicate-list' (which goes to a
>> separate, ad-hoc file) would fit well as a simple persistent variable.
>
> If `gnus-newsrc-alist' were saved as-is, but *without* the mark data,
> that mark data could be pulled out of a separate storage backend at
> startup time, with a simple group-name->mark-data mapping. The
> .newsrc.eld file could otherwise remain the same.

One can edit flags through G p in the topic buffer; that interface would
need to be preserved.  Who knows what other secrets lurk in there.

> But Qiantan's message reminded me that this proposal is actually a Gnus
> issue, not a sqlite issue, and shouldn't clog up this thread. I'll keep
> this for later!

Yes, but in the end separating flags from .newsrc.eld is not for the
faint of heart, and I am not sure I see the benefits. 

Like you said, for later.

Re: sqlite3

[Eric Abrahamsen]

cesar mena <cesar.mena@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Well, Lars is the Gnus maintainer, not me, so all I can do is propose!
>>>> But yes I am, though only for group marks. Right now .newsrc.eld
>>>> combines concerns: it saves variables such as `gnus-server-alist' and
>>>> all the topic stuff that would be better saved as customization options
>>>> (or saved in a separate file, using the customization machinery). It
>>>> also saves group parameters, right next to the marks.
>
> Yes, it's a bit of a big hammer. 
>
>>> It's a bit much, but I don't think those bits can be usefully separated
>>> out.  But things like `gnus-save-duplicate-list' (which goes to a
>>> separate, ad-hoc file) would fit well as a simple persistent variable.
>>
>> If `gnus-newsrc-alist' were saved as-is, but *without* the mark data,
>> that mark data could be pulled out of a separate storage backend at
>> startup time, with a simple group-name->mark-data mapping. The
>> .newsrc.eld file could otherwise remain the same.
>
> One can edit flags through G p in the topic buffer; that interface would
> need to be preserved.  Who knows what other secrets lurk in there.

My proposal would only affect the save and load routines (zip and unzip
data), in-session behavior wouldn't change.

>> But Qiantan's message reminded me that this proposal is actually a Gnus
>> issue, not a sqlite issue, and shouldn't clog up this thread. I'll keep
>> this for later!
>
> Yes, but in the end separating flags from .newsrc.eld is not for the
> faint of heart, and I am not sure I see the benefits. 
>
> Like you said, for later.

I have any number of pie-in-the-sky ideas for Gnus. If I get around to
implementing half of them, I'll consider it a success :)

Re: sqlite3

[cesar mena]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> cesar mena <cesar.mena@gmail.com> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>>
>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>
>>>>> Well, Lars is the Gnus maintainer, not me, so all I can do is propose!
>>>>> But yes I am, though only for group marks. Right now .newsrc.eld
>>>>> combines concerns: it saves variables such as `gnus-server-alist' and
>>>>> all the topic stuff that would be better saved as customization options
>>>>> (or saved in a separate file, using the customization machinery). It
>>>>> also saves group parameters, right next to the marks.
>>
>> Yes, it's a bit of a big hammer. 
>>
>>>> It's a bit much, but I don't think those bits can be usefully separated
>>>> out.  But things like `gnus-save-duplicate-list' (which goes to a
>>>> separate, ad-hoc file) would fit well as a simple persistent variable.
>>>
>>> If `gnus-newsrc-alist' were saved as-is, but *without* the mark data,
>>> that mark data could be pulled out of a separate storage backend at
>>> startup time, with a simple group-name->mark-data mapping. The
>>> .newsrc.eld file could otherwise remain the same.
>>
>> One can edit flags through G p in the topic buffer; that interface would
>> need to be preserved.  Who knows what other secrets lurk in there.
>
> My proposal would only affect the save and load routines (zip and unzip
> data), in-session behavior wouldn't change.

Yes, but there are so many little details that would prop up (I
think). Like what happens if you couldn't load the marks for one of my
2k (I exaggerate ... slightly) groups?  Would you turn off expiry to
prevent the deletion of dormant articles? etc ... 

>>> But Qiantan's message reminded me that this proposal is actually a Gnus
>>> issue, not a sqlite issue, and shouldn't clog up this thread. I'll keep
>>> this for later!
>>
>> Yes, but in the end separating flags from .newsrc.eld is not for the
>> faint of heart, and I am not sure I see the benefits. 
>>
>> Like you said, for later.
>
> I have any number of pie-in-the-sky ideas for Gnus. If I get around to
> implementing half of them, I'll consider it a success :)

You're doing the lord's work. I appreciate it :)

Cheers!

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Would someone please post a concise description of the proposal that is
being considered?  There have been over 100 messages in this thread,
too many for me to read, but I'd like to know what the issue is.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Would someone please post a concise description of the proposal that
> is being considered?  There have been over 100 messages in this
> thread, too many for me to read, but I'd like to know what the issue
> is.

Essentially, some people believe Emacs needs a way to quickly persist
Lisp data to disk.

Some of those people want to use the database library sqlite3 for that
task, while others want to use (g)dbm and/or GNU recutils to perform
that task.

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Some of those people want to use the database library sqlite3 for that
  > task, while others want to use (g)dbm and/or GNU recutils to perform
  > that task.

It would be good for some supporter of each option
to present all the arguments for that option, conciesely in one message
for that option.

Whatever we do, we need to have a simple and convenient user interface
for the user to browse all these records and edit them.  If this works
conveniently enough, maybe it could replace the Customize buffer.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> It would be good for some supporter of each option to present all the
> arguments for that option, conciesely in one message for that option.

I think the discussion has degenerated into an argument over how fast
serializing hash tables with `read' and `print' is during the single day
between now and when I sent my earlier reply to you, so I can't do that
anymore.

> Whatever we do, we need to have a simple and convenient user interface
> for the user to browse all these records and edit them.  If this works
> conveniently enough, maybe it could replace the Customize buffer.

Agreed.

How to save data

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It would be good for some supporter of each option to present all the
  > > arguments for that option, conciesely in one message for that option.

  > I think the discussion has degenerated into an argument over how fast
  > serializing hash tables with `read' and `print' is during the single day
  > between now and when I sent my earlier reply to you, so I can't do that
  > anymore.

That's just one small part of the issue.  It is still possible to summarize
the overall advantages and disadvantages.  This uncertain point
can be described in one line saying "may be slow".


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Qiantan Hong]

> It would be good for some supporter of each option
> to present all the arguments for that option, conciesely in one message
> for that option.
I’m against using sqlite3, or maybe in general any database.
I support plain Emacs Lisp implementation.

My arguments:

1. Using as a persistent key value store.  However, in this case
the external database provides virtually no benefit over plain
text, because the major cost here is to print/read lisp values,
and you still need to do it when using a database because they
understand strings, not lisp values!

2. As a querying tool over some data.  In this case, the
functionality largely overlaps with LISt Processing itself.  The
argument is the same as why we don’t want to introduce JavaScript
scripting into Emacs.  If developers are lured into writing more
of their code in SQL, those parts are detached from Lisp world
and loses all the niceties from Emacs Lisp (explorability,
dynamic modifiability etc).

3. Database files are not readable directly by a text editor.

4. SQLite3 is an additional dependency. I hear people saying one
can make it optional, but I highly doubt if people are willing
to maintain 2 version of code with/without SQLite3, especially
after they starts to write SQL instead of Elisp.

5. Database is essential for manipulating trillions of records,
but that's not a typical use case of Emacs. It provides little
benefit if data fits into main memory.

Some other alternative ((g)dbm, recutils) is free from some of
those problem, but I think the following option is the best overall:

1. Implement a persistent key/value store or object store in
plain Emacs Lisp. I posted an implementation in anther thread
titled "Against sqlite3!!! (Was: sqlite3)"

2. I highly doubt if LISP is in any case insufficient for querying.
But in those case, embeding a Prolog implementation in Emacs
Lisp is a much better option than SQL.

Re: sqlite3

[Qiantan Hong]

Sorry, I mistakenly killed a line which makes it hard to read,
I add it back:

> My arguments:

(Added) There are basically two use cases for SQLite3:

> 1. Using as a persistent key value store.  However, in this case
> the external database provides virtually no benefit over plain
> text, because the major cost here is to print/read lisp values,
> and you still need to do it when using a database because they
> understand strings, not lisp values!
> 
> 2. As a querying tool over some data.  In this case, the
> functionality largely overlaps with LISt Processing itself.  The
> argument is the same as why we don’t want to introduce JavaScript
> scripting into Emacs.  If developers are lured into writing more
> of their code in SQL, those parts are detached from Lisp world
> and loses all the niceties from Emacs Lisp (explorability,
> dynamic modifiability etc).

(And I then go on to the other points)
> 
> 3. Database files are not readable directly by a text editor.
> 
> 4. ...

Re: sqlite3

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> Date: Wed, 8 Dec 2021 05:41:17 +0000
> Cc: Po Lu <luangruo@yahoo.com>, Eric Abrahamsen <eric@ericabrahamsen.net>,
>  cesar mena <cesar.mena@gmail.com>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> I’m against using sqlite3, or maybe in general any database.

Why would you be against adding to Emacs infrastructure that might be
useful for some applications?

We didn't yet decide what would be the policy and guidelines for using
this in Emacs, but there should be no doubt that when we do make those
decisions, we will take into considerations all the factors that you
mentioned, and more.  Because Lars, myself, and all the others here
are quite aware of the needs and the advantages/disadvantages, and we
use and develop Emacs long enough to understand what is and what isn't
Emacs'y.  So the decisions, when they are made, will not be silly or
haphazard.  Rest assured that where using simple text files is
adequate, no one in their right mind will advocate using a DB.

I see no reason to be "against sqlite3".  Many popular software
packages out there have similar capabilities, so why shouldn't Emacs
offer them as well?  Just because it can be abused?  If that's the
reason, we'd need to "be against" almost every advanced Emacs feature,
and I personally would go over the display engine and butcher it to
remove every single advanced feature that people are known to abuse.
Would that make sense?  I hope not.

So please, can you drop this "against sqlite3" attitude, and let Lars
finish the job he started?  Adding capabilities to Emacs, and in
particular providing us with interfaces to modern technologies, is
supposed to be a Good Thing, something that advances Emacs and makes
it a better, more capable platform for developing useful applications.
It's our way to survive in the long run.  Let's not shoot ourselves in
the foot due to myopic perspectives.

TIA

Re: sqlite3

[Alan Mackenzie]

Hello, Eli and Emacs.

On Wed, Dec 08, 2021 at 15:14:08 +0200, Eli Zaretskii wrote:
> > From: Qiantan Hong <qhong@mit.edu>
> > Date: Wed, 8 Dec 2021 05:41:17 +0000
> > Cc: Po Lu <luangruo@yahoo.com>, Eric Abrahamsen <eric@ericabrahamsen.net>,
> >  cesar mena <cesar.mena@gmail.com>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>

> > I’m against using sqlite3, or maybe in general any database.

> Why would you be against adding to Emacs infrastructure that might be
> useful for some applications?

Because it can degrade other applications.  Primarily Emacs is a
programmers' text editor, and I would like it to stay that way.  Adding
database functionality is inessential to that prime function (regardless
of how important it might be), therefore is a burden to that prime
function.  All things like gnus, org-mode are likewise inessential -
they add many megabytes to the source-code, greatly increase build time,
take up key-bindings, space in documentation (hopefully ;-) and so on.

I've never used org-mode in my life, and only tried out gnus once,
around 20 years ago, when it was too slow and too complicated for me.
Yet I have to pay the costs of these packages every time I build Emacs
bootstrap.

I'm not arguing for a removal of these things, which are clearly good.
But I think it is reasonable to question the wisdom of adding more
inessential stuff.

> We didn't yet decide what would be the policy and guidelines for using
> this in Emacs, but there should be no doubt that when we do make those
> decisions, we will take into considerations all the factors that you
> mentioned, and more.  Because Lars, myself, and all the others here
> are quite aware of the needs and the advantages/disadvantages, and we
> use and develop Emacs long enough to understand what is and what isn't
> Emacs'y.  So the decisions, when they are made, will not be silly or
> haphazard.  Rest assured that where using simple text files is
> adequate, no one in their right mind will advocate using a DB.

> I see no reason to be "against sqlite3".  Many popular software
> packages out there have similar capabilities, so why shouldn't Emacs
> offer them as well?  Just because it can be abused?  If that's the
> reason, we'd need to "be against" almost every advanced Emacs feature,
> and I personally would go over the display engine and butcher it to
> remove every single advanced feature that people are known to abuse.
> Would that make sense?  I hope not.

I disagree with you (Eli), that we should not anticipate and be wary of
how people might use things.  As an example, Richard warned me a few
years ago against introducing a hook for changes in narrowing, saying
that it was too powerful, and would end up with users using it in ways
which would cause us lots of trouble.

Similarly, look at what has happened to the humble web brower - it
started as a browser for the web, and has in recent years transformed
into an execution engine for (usually) none-free and often hostile
scripts.  Would this have happened had somebody not developed
JavaScript?  I miss the web browser of old.

Similarly, I would be very unhappy if somebody transformed Emacs into a
database engine, where its prime features were unusable without the
database features being built in.  I think this possibility, remote at
this juncture, should be borne in mind.

> So please, can you drop this "against sqlite3" attitude, and let Lars
> finish the job he started?  Adding capabilities to Emacs, and in
> particular providing us with interfaces to modern technologies, is
> supposed to be a Good Thing, something that advances Emacs and makes
> it a better, more capable platform for developing useful applications.
> It's our way to survive in the long run.  Let's not shoot ourselves in
> the foot due to myopic perspectives.

As above, I don't agree entirely with this perspective.

> TIA

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The interface to sqlite3 may be much smaller than some existing
Lisp packages.

  > Similarly, look at what has happened to the humble web brower - it
  > started as a browser for the web, and has in recent years transformed
  > into an execution engine for (usually) none-free and often hostile
  > scripts.  Would this have happened had somebody not developed
  > JavaScript?  I miss the web browser of old.

I too have come to consider the "feature" of running Javascript code
out of web pages as a very bad thing.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Eli Zaretskii]

> Date: Fri, 10 Dec 2021 21:25:05 +0000
> Cc: Qiantan Hong <qhong@mit.edu>, luangruo@yahoo.com, eric@ericabrahamsen.net,
>   cesar.mena@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > So please, can you drop this "against sqlite3" attitude, and let Lars
> > finish the job he started?  Adding capabilities to Emacs, and in
> > particular providing us with interfaces to modern technologies, is
> > supposed to be a Good Thing, something that advances Emacs and makes
> > it a better, more capable platform for developing useful applications.
> > It's our way to survive in the long run.  Let's not shoot ourselves in
> > the foot due to myopic perspectives.
> 
> As above, I don't agree entirely with this perspective.

Noted.  But I'm quite sure you are wrong.

Re: sqlite3

[Karl Fogel]

On 10 Dec 2021, Alan Mackenzie wrote:
>I've never used org-mode in my life, and only tried out gnus 
>once,
>around 20 years ago, when it was too slow and too complicated for 
>me.
>Yet I have to pay the costs of these packages every time I build 
>Emacs
>bootstrap.
>
>I'm not arguing for a removal of these things, which are clearly 
>good.
>But I think it is reasonable to question the wisdom of adding 
>more
>inessential stuff.

You've defined "inessential" in a way that happens to match your 
particular usage patterns exactly.  But for others, such as 
myself, Gnus and Org Mode are essential :-).

In a thread from 2020 ([1], "GNU Emacs raison d'etre"), I offered 
a different understanding of Emacs's essence:

It is the editing environment that best rewards sustained user 
investment.

That differs from your claim that "primarily Emacs is a 
programmers' text editor".  Programming Emacs is simply the 
highest level of investment, but it doesn't necessarly imply that 
one is using Emacs *for* computer programming -- often, I'm not. 
Naturally, users who are programmers are going to have the easiest 
time investing in their Emacs experience, both due to skills they 
have and (probably) due to being temperamentally inclined towards 
patience with such investments. 

The argument for sqlite3 (or something like it) is that it makes 
some of those investments easier -- specifically, it makes it 
easier to do things that involve selective access to large 
datasets.

A concrete example is https://code.librehq.com/kfogel/mailaprop/. 
Right now, I load all the data in to memory at startup time right, 
but it's costly -- it noticeably slows down startup.  Fortunately, 
I don't restart Emacs very often, so this is liveable.  However, 
if the dataset were 10x or 20x larger, it would be intolerable.

I could of course try out the existing 3rd-party sqlite3 library 
for Emacs; it's not like there's a huge barrier here.  But still, 
that would mean adding a dependency that I don't currently have. 
Whereas if the facility were built into Emacs, the barrier would 
be lower, so I'd be more likely to experiment with selective 
loading of such datasets.  Presumably, the same argument applies 
to others' applications as well, and we'd have the further 
advantage that everyone would be using a common facility, so it 
could be improved collaboratively.

I believe this is one of the points Lars is making.

Best regards,
-Karl

[1] 
https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01855.html
    From: Karl Fogel <kfogel@red-bean.com>
    To: <emacs-devel@gnu.org>
    Subject: Re: GNU Emacs raison d'etre
    Date: Wed, 13 May 2020 11:18:50 -0500
    Message-ID: <871rnnvmdx.fsf@red-bean.com>

Re: sqlite3

[Alan Mackenzie]

Hello, Karl.

On Tue, Dec 14, 2021 at 14:09:38 -0600, Karl Fogel wrote:
> On 10 Dec 2021, Alan Mackenzie wrote:
> >I've never used org-mode in my life, and only tried out gnus once,
> >around 20 years ago, when it was too slow and too complicated for me.
> >Yet I have to pay the costs of these packages every time I build Emacs
> >bootstrap.

> >I'm not arguing for a removal of these things, which are clearly good.
> >But I think it is reasonable to question the wisdom of adding more
> >inessential stuff.

> You've defined "inessential" in a way that happens to match your 
> particular usage patterns exactly.

Not at all.  I was using the word correctly.  In the words of Fowler,
something is essential when "we have in mind a whole that would not be
what it is to be or is or was if the part in question were wanting; the
essential thing is such that the other thing is inconceivable without
it.".  Emacs without gnus would still be Emacs.  Emacs without Lisp would
not.  In that sense Lisp is essential to Emacs but gnus isn't.

That is an entirely different question as to what is necessary.
Windscreen wipers are necessary on a motor car.  But wheels are
essential.  I'm quite prepared to accept that gnus is necessary for
Emacs.

> But for others, such as myself, Gnus and Org Mode are essential :-).

As above, they're not.  They're necessary.

I question the wisdom of adding more inessential stuff to the Emacs core.
To be perfectly blunt, it is bloat.  I'm not saying that sqlite shouldn't
be added to core.  But I am saying it should be questioned carefully,
particularly by people who are familiar with it (which I'm not).

I feel sqlite has been added to the core, merged into master, with
indecent haste, and without due reflection.  This cannot be good.  I
think there are knowledgeable people here who are against this novelty
and I think their expertise has been disregarded.  That cannot be good
for the Emacs project.

> In a thread from 2020 ([1], "GNU Emacs raison d'etre"), I offered 
> a different understanding of Emacs's essence:

> It is the editing environment that best rewards sustained user 
> investment.

> That differs from your claim that "primarily Emacs is a 
> programmers' text editor".

Clarification here.  I meant an editor used by programmers, not one
programmed by them.

> Programming Emacs is simply the highest level of investment, but it
> doesn't necessarly imply that one is using Emacs *for* computer
> programming -- often, I'm not.  Naturally, users who are programmers
> are going to have the easiest time investing in their Emacs experience,
> both due to skills they have and (probably) due to being
> temperamentally inclined towards patience with such investments. 

> The argument for sqlite3 (or something like it) is that it makes 
> some of those investments easier -- specifically, it makes it 
> easier to do things that involve selective access to large 
> datasets.

Yes.  It comes with a cost, though, and the cost and the benefit have to
be weighed against eachother.

> A concrete example is https://code.librehq.com/kfogel/mailaprop/. 
> Right now, I load all the data in to memory at startup time right, 
> but it's costly -- it noticeably slows down startup.  Fortunately, 
> I don't restart Emacs very often, so this is liveable.  However, 
> if the dataset were 10x or 20x larger, it would be intolerable.

In my mail program, mutt, I load my entire mail collection into memory
every time I start it.  That's around 140,000 mails, it's over 2 GB, and
it loads and threads in 7 or 8 seconds.  I don't know how long it would
take to load into gnus, but I suspect it would be much longer.

> I could of course try out the existing 3rd-party sqlite3 library 
> for Emacs; it's not like there's a huge barrier here.  But still, 
> that would mean adding a dependency that I don't currently have. 
> Whereas if the facility were built into Emacs, the barrier would 
> be lower, so I'd be more likely to experiment with selective 
> loading of such datasets.  Presumably, the same argument applies 
> to others' applications as well, and we'd have the further 
> advantage that everyone would be using a common facility, so it 
> could be improved collaboratively.

Yes, those are the plusses.  The downsides are the bloat (i don't know by
how much), the dilution in Emacs's purpose (it's even less the "do one
thing and do it well" than it was), and I suppose there might even be
negative security implications (it's a database after all).  Also people
will be forced to learn new tools, both inside and outside of Emacs.  I
fear it will create another elite in the Emacs project, those who have
mastered sqlite, similar to, for example, those who can read and write
the cl- library.

> I believe this is one of the points Lars is making.

> Best regards,
> -Karl

> [1] 
> https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01855.html
>     From: Karl Fogel <kfogel@red-bean.com>
>     To: <emacs-devel@gnu.org>
>     Subject: Re: GNU Emacs raison d'etre
>     Date: Wed, 13 May 2020 11:18:50 -0500
>     Message-ID: <871rnnvmdx.fsf@red-bean.com>

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: sqlite3

[Karl Fogel]

On 14 Dec 2021, Alan Mackenzie wrote:
>Yes, those are the plusses.  The downsides are the bloat (i don't 
>know by
>how much), the dilution in Emacs's purpose (it's even less the 
>"do one
>thing and do it well" than it was), and I suppose there might 
>even be
>negative security implications (it's a database after all).  Also 
>people
>will be forced to learn new tools, both inside and outside of 
>Emacs.  I
>fear it will create another elite in the Emacs project, those who 
>have
>mastered sqlite, similar to, for example, those who can read and 
>write
>the cl- library.

If it's any comfort, sqlite3 is small, both in source code and 
memory footprint.  (Famously small -- part of its purpose [1] is 
to be small enough that it can be bundled even into constrained 
environments.)  And it's *very* easy to use if one has any SQL-ish 
database experience at all.  At a time when I had zero sqlite 
experience, I switched a MySQL-based application to sqlite3 in a 
couple of hours or so; at no point was it difficult.

Anyone who basically groks SQL databases will grok sqlite very 
quickly.  It's tiny and simple.  It's nowhere near the CL library 
in terms of complexity, and I think there is no danger of creating 
a new elite here.

Best regards,
-Karl


[1] https://sqlite.org/footprint.html

Re: sqlite3

[Óscar Fuentes]

Alan Mackenzie <acm@muc.de> writes:

> In my mail program, mutt, I load my entire mail collection into memory
> every time I start it.  That's around 140,000 mails, it's over 2 GB, and
> it loads and threads in 7 or 8 seconds.

And you are concerned about the bloat added by sqlite! ;-)

What you describe is a huge waste of RAM and time, unless you need to
scan all those mails many times on every session. And even then it still
would be a waste of RAM. (I wonder how that affects the time required
for a gc cycle?)

If using a proper database you could have almost zero RAM overhead and
almost zero init time overhead.

This practice of loading everything into RAM only goes so far. Doing
what you describe would be unacceptable to me. It is important to
provide a method for efficiently handling large amounts of stored data.
It will benefit some existing packages and make feasible new features.

Re: sqlite3

[Alexandre Garreau]

Le mardo, 14-a de decembro 2021, 23-a horo kaj 1:33 CET Alan Mackenzie a 
écrit :
> 
> 
> > You've defined "inessential" in a way that happens to match your
> > particular usage patterns exactly.
> 
> Not at all.  I was using the word correctly.  In the words of Fowler,
> something is essential when "we have in mind a whole that would not be
> what it is to be or is or was if the part in question were wanting; the
> essential thing is such that the other thing is inconceivable without
> it.".  Emacs without gnus would still be Emacs.  Emacs without Lisp
> would not.  In that sense Lisp is essential to Emacs but gnus isn't.
> 
> That is an entirely different question as to what is necessary.
> Windscreen wipers are necessary on a motor car.  But wheels are
> essential.  I'm quite prepared to accept that gnus is necessary for
> Emacs.

I agree.

> > But for others, such as myself, Gnus and Org Mode are essential :-).
> 
> As above, they're not.  They're necessary.
> 
> I question the wisdom of adding more inessential stuff to the Emacs
> core. To be perfectly blunt, it is bloat.  I'm not saying that sqlite
> shouldn't be added to core.  But I am saying it should be questioned
> carefully, particularly by people who are familiar with it (which I'm
> not).

…but as said, sqlite, like *almost every* (well I doubt it’s possible to 
build emacs without ncurse support) library, is optional.  So talking 
about bloat is stretched, imho.

However, I feel like the core point that many must feel and that would 
justify all these lengthy and not always that much dense discussions is 
that SQLite is actually very useful, very demanded, very popular, and 
that’s precisely the problem.  SQLite has been described, as long with 
ZLib, as “one of the two most used software on the planet”, it’s almost 
everywhere, so the network effect is *extremely powerful* in their favor.

I feel that this power is feared, because then the fear that a dependency 
on sqlite *will* appear, as soon as it’s allowed, scares some users.  I 
understand the idea of “unlispiness” of it, personally, and I indeed feel 
we’re growing more akin to standard software than to the lisp world…

However I think the opposition has been very notably badly formulated, 
with little concrete examples of the issues, or precise and concrete 
desires (beside the “no binary files”… so you can edit with nano? well ok 
but you likely use emacs… we’re not 100% the unix philosophy either).  I 
think just integrating sqlite and *then* pointing out the issues may be 
more relevant.

I think the right mindset here has been best expressed by Qiantan, who 
tried to propose something instead of just say “no”.

> I feel sqlite has been added to the core, merged into master, with
> indecent haste, and without due reflection.  This cannot be good.  I
> think there are knowledgeable people here who are against this novelty
> and I think their expertise has been disregarded.  That cannot be good
> for the Emacs project.

Absolutely not, given the length of the discussion it’s pretty much the 
opposite.  We here have a paradox where the more we discuss it, the more 
the people who oppose talk, and the more we have the impression the 
opposition is strong, whatever are the arguments (even if they’re none, or 
they’re stated bad), and the consequence is that paradoxally we discussed 
a lot and yet we get the impression this is haste and without discussion.  
It’s the same paradox as in “[harsh criticism] —shut up” looks less like 
free expression than “[politically correct soft subpart of criticism]” 
(without “shut up” or counter-criticism).

It’s a little like unintentional, natural-occuring FUD.

Actually we won, lars added a per-file backend, so what he’s implementing 
isn’t even going to cause a dependency on sqlite anymore.  It’s even now 
actually fs-wide compatible with persist.el

I’m just sad, yet again, nobody worked with nobody else to try to 
advertise, discuss and agree the various already existing API at each 
point of the discussion.  Each person prefer their own API and well…

Re: sqlite3

[Po Lu]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> …but as said, sqlite, like *almost every* (well I doubt it’s possible to 
> build emacs without ncurse support) library, is optional.  So talking 
> about bloat is stretched, imho.

It is possible to build Emacs without ncurses.  At least on MS-DOS and
MS-Windows.

Re: sqlite3

[Po Lu]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> …but as said, sqlite, like *almost every* (well I doubt it’s possible to 
> build emacs without ncurse support) library, is optional.  So talking 
> about bloat is stretched, imho.

It is possible to build Emacs without ncurses.  At least on MS-DOS and
MS-Windows.

(The MS-DOS port simply writes to video memory?)

Re: sqlite3

[Georges Ko]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> However, I feel like the core point that many must feel and that would 
> justify all these lengthy and not always that much dense discussions is 
> that SQLite is actually very useful, very demanded, very popular, and 
> that’s precisely the problem.  SQLite has been described, as long with 
> ZLib, as “one of the two most used software on the planet”, it’s almost 
> everywhere, so the network effect is *extremely powerful* in their favor.

Yes, why would I prefer to run "sqlite3 FILE", type SQL or other
SQLite-specific commands, etc. over just opening it in Emacs to
automatically view the list of tables and press Enter to see all the
rows of a table?

Isn't it nice to be able to view & edit SQLite files as easily as .tar,
.jar or .zip?

Georges
-- 
 Georges Ko                     gko@gko.net                      2021-12-16

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021, 2-a horo kaj 36:01 CET Alexandre 
Garreau a écrit :
> However, I feel like the core point that many must feel and that would
> justify all these lengthy and not always that much dense discussions is
> that SQLite is actually very useful, very demanded, very popular, and
> that’s precisely the problem.  SQLite has been described, as long with
> ZLib, as “one of the two most used software on the planet”, it’s almost
> everywhere, so the network effect is *extremely powerful* in their
> favor.
> 
> I feel that this power is feared, because then the fear that a
> dependency on sqlite *will* appear, as soon as it’s allowed, scares
> some users.  I understand the idea of “unlispiness” of it, personally,
> and I indeed feel we’re growing more akin to standard software than to
> the lisp world…

after some coincidental experience yesterday night, I discovered this 
reaction is very very akin to the one some people may have in front of 
english, esperanto or similar language when exposed to it with a fear it 
may overcome, while they’re unused to it.  fear of powerful and good stuff, 
because they might become essential

Re: sqlite3

[Eli Zaretskii]

> Date: Tue, 14 Dec 2021 22:01:33 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, Qiantan Hong <qhong@mit.edu>, rms@gnu.org,
>   eric@ericabrahamsen.net, cesar.mena@gmail.com, emacs-devel@gnu.org,
>   luangruo@yahoo.com
> From: Alan Mackenzie <acm@muc.de>
> 
> I question the wisdom of adding more inessential stuff to the Emacs core.
> To be perfectly blunt, it is bloat.

To be perfectly blunt, please leave that worry to Lars and myself.
That's why we are here, that's why we invest such an inordinate amount
of time working on Emacs (and on top of that, more time to participate
in this futile dispute).

> I'm not saying that sqlite shouldn't be added to core.  But I am
> saying it should be questioned carefully, particularly by people who
> are familiar with it (which I'm not).

We've examined that question and decided it could be a good idea.
Please bear with us, as there's no real reasons yet to seriously
question the decision.

> I feel sqlite has been added to the core, merged into master, with
> indecent haste, and without due reflection.

Your feelings are wrong, and disclosing them here is unfair.  You have
no real evidence to back that up, and without that it is just an
unfounded accusation.

The fact that both maintainers agree on this should already tell you
that you should check your feelings very well before you decide they
are justified enough to make them public.

> This cannot be good.  I think there are knowledgeable people here
> who are against this novelty and I think their expertise has been
> disregarded.  That cannot be good for the Emacs project.

Nothing was disregarded.  Hearing different opinions and disagreeing
with them is not the same as disregard.  Please don't accuse us in
crimes we didn't commit.

> In my mail program, mutt, I load my entire mail collection into memory
> every time I start it.  That's around 140,000 mails, it's over 2 GB, and
> it loads and threads in 7 or 8 seconds.  I don't know how long it would
> take to load into gnus, but I suspect it would be much longer.

What is the memory footprint of Emacs after loading those 2GB?  And
why do you want everyone to have to keep such large mail collections
in memory?  Not everyone has enough RAM installed to afford that.

> > I could of course try out the existing 3rd-party sqlite3 library 
> > for Emacs; it's not like there's a huge barrier here.  But still, 
> > that would mean adding a dependency that I don't currently have. 
> > Whereas if the facility were built into Emacs, the barrier would 
> > be lower, so I'd be more likely to experiment with selective 
> > loading of such datasets.  Presumably, the same argument applies 
> > to others' applications as well, and we'd have the further 
> > advantage that everyone would be using a common facility, so it 
> > could be improved collaboratively.
> 
> Yes, those are the plusses.  The downsides are the bloat (i don't know by
> how much), the dilution in Emacs's purpose (it's even less the "do one
> thing and do it well" than it was), and I suppose there might even be
> negative security implications (it's a database after all).  Also people
> will be forced to learn new tools, both inside and outside of Emacs.  I
> fear it will create another elite in the Emacs project, those who have
> mastered sqlite, similar to, for example, those who can read and write
> the cl- library.

There's no reason to fear this, no more than you'd fear that librsvg
will facilitate an "elite" who can "speak SVG" to create images, and
the "plebs" who don't.

Frankly, I'm amazed at the bru-ha-ha this change triggered, and keep
being amazed each day anew.  Who'd know that Emacs is such a
scandalous environment?

Re: sqlite3

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> What is the memory footprint of Emacs after loading those 2GB?  And
> why do you want everyone to have to keep such large mail collections
> in memory?  Not everyone has enough RAM installed to afford that.

Yes, growing memory footprint is a problem that has to be tackled
somehow.  Gnus (and some other programs, but it's mainly Gnus) eating
memory is starting to strain my system, and it has 32 GB of RAM.

> Frankly, I'm amazed at the bru-ha-ha this change triggered, and keep
> being amazed each day anew.  Who'd know that Emacs is such a
> scandalous environment?

I think people were worried that sqlite3 was going to be used as the
only way to save persistent variables (and even user customizations made
through custom).

But that concern has been addressed, so I'm happy now :)

The SQLite3 support itself would be nice to keep.

Re: sqlite3

[Alan Mackenzie]

Hello, Eli.

On Wed, Dec 15, 2021 at 15:06:45 +0200, Eli Zaretskii wrote:
> > Date: Tue, 14 Dec 2021 22:01:33 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, Qiantan Hong <qhong@mit.edu>, rms@gnu.org,
> >   eric@ericabrahamsen.net, cesar.mena@gmail.com, emacs-devel@gnu.org,
> >   luangruo@yahoo.com
> > From: Alan Mackenzie <acm@muc.de>

> > I question the wisdom of adding more inessential stuff to the Emacs core.
> > To be perfectly blunt, it is bloat.

> To be perfectly blunt, please leave that worry to Lars and myself.
> That's why we are here, that's why we invest such an inordinate amount
> of time working on Emacs (and on top of that, more time to participate
> in this futile dispute).

I'll keep this reply short.

[ .... ]

> > I feel sqlite has been added to the core, merged into master, with
> > indecent haste, and without due reflection.

> Your feelings are wrong, and disclosing them here is unfair.  You have
> no real evidence to back that up, and without that it is just an
> unfounded accusation.

Normally, potential new features spend more than just a few days on a
git branch, allowing people the chance to try them out and spot any
problems.  That didn't happen with this new sqlite feature.

[ .... ]

> Frankly, I'm amazed at the bru-ha-ha this change triggered, and keep
> being amazed each day anew.  Who'd know that Emacs is such a
> scandalous environment?

This change potentially changes what Emacs _is_.  Most changes don't do
that.  It came out of the blue and was integrated into master in a few
days.  I'd be surprised if a change like this didn't generate some
amount of unease.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: sqlite3

[Eli Zaretskii]

> Date: Fri, 17 Dec 2021 10:37:55 +0000
> Cc: kfogel@red-bean.com, qhong@mit.edu, rms@gnu.org, eric@ericabrahamsen.net,
>   cesar.mena@gmail.com, emacs-devel@gnu.org, luangruo@yahoo.com
> From: Alan Mackenzie <acm@muc.de>
> 
> > > I feel sqlite has been added to the core, merged into master, with
> > > indecent haste, and without due reflection.
> 
> > Your feelings are wrong, and disclosing them here is unfair.  You have
> > no real evidence to back that up, and without that it is just an
> > unfounded accusation.
> 
> Normally, potential new features spend more than just a few days on a
> git branch, allowing people the chance to try them out and spot any
> problems.  That didn't happen with this new sqlite feature.

This sounds like a complaint against Lars working too fast and too
hard.  I'm quite sure we should be grateful instead.

If, for some personal reason, you were unable to track the discussions
when they happen, that's not a problem, because you can always voice
your opinions later.  That doesn't give any basis for castigating us
for not being slower.

> > Frankly, I'm amazed at the bru-ha-ha this change triggered, and keep
> > being amazed each day anew.  Who'd know that Emacs is such a
> > scandalous environment?
> 
> This change potentially changes what Emacs _is_.

No, it doesn't, far from it.  It adds a niche feature which might be
useful in some use cases, that's all.

Re: sqlite3

[dick]

EZ> This sounds like a complaint against Lars working too fast and too
EZ> hard.

It's less that, and more the fact that he's spraying the master with
some brutal, overwrought code reminiscent of 1990s gnus.

This would be fine if he kept it isolated, but unfortunately he
defaults-on the sqlite3 stuff, and the tests fail for systems without
the latest version of sqlite3.

Re: sqlite3

[Eli Zaretskii]

> From: dick <dick.r.chiang@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 17 Dec 2021 08:05:45 -0500
> 
> This would be fine if he kept it isolated, but unfortunately he
> defaults-on the sqlite3 stuff, and the tests fail for systems without
> the latest version of sqlite3.

If the tests fail when they shouldn't, that's a bug that should be
fixed.  We fix bugs in recently developed code all the time.  This is
an inherent part of any development.

No one is immune against leaving bugs in their code.  That there are
bugs here and there is not, and cannot be, any argument against
Lars,who IMNSHO does a great job.

So I definitely reject your conclusions, let alone the unacceptable
tone of what you wrote.

Re: sqlite3

[dick]

> So I definitely reject your conclusions, let alone the unacceptable
> tone of what you wrote.

Objection sustained.

The serfs can, at best, convince the king to create a feature branch on
savannah, and then hope against the odds that someone notices, let alone
builds off of it.

The king simply indentures beta-testers for his grace's feature idea by
fiat.

Conclusion: it's good to be the king.

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Normally, potential new features spend more than just a few days on a
  > git branch, allowing people the chance to try them out and spot any
  > problems.  That didn't happen with this new sqlite feature.

That is where I see a problem, too.

sqlite3 has problematic implications.  There may be a good solution
but we have not determined that.  Meanwhile, other solutions were
there to be considered.

We should not have rushed the decision on what to do.  We should have
taken it slow, leaving time for people to study various solutions and
make a thoughtful decision.

Perhaps that decision would have included installing sqlite3.  I won't
say it wouldn't.  But we should not repeat the way that decision was
made.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 17 Dec 2021 23:42:07 -0500
> Cc: kfogel@red-bean.com, qhong@mit.edu, eric@ericabrahamsen.net,
>  cesar.mena@gmail.com, emacs-devel@gnu.org, luangruo@yahoo.com, eliz@gnu.org
> 
>   > Normally, potential new features spend more than just a few days on a
>   > git branch, allowing people the chance to try them out and spot any
>   > problems.  That didn't happen with this new sqlite feature.
> 
> That is where I see a problem, too.

Which problem is that?

According to Git logs, the sqlite branch was created on Dec 6, and
merged to master on Dec 11.  That's almost a week.

We ask people to try a branch when it provides a significant feature
that could affect everyday use of Emacs, or if we have reasons to
believe it could mis-compile or break on some platforms or
configurations, or if it has some significant user-facing aspects for
which we'd like feedback.  That's why the bidirectional display was
originally on a branch.  That's why the PGTK build is on a branch now.

In all other cases, we use feature branches to allow the developer to
work on a feature without affecting people who track the master
branch, as long as the feature isn't mature enough to land on master.
We then merge to master when the branch is mature enough.  Which is
exactly what happened here: this is a minor feature that is almost
orthogonal to anything else in Emacs, so there was no reason to keep
it on a branch longer than it took to make it mature and stable.  We
have enough examples of features that were merged when they were less
stable than this one, we have merged stuff that broke the Emacs build
on one or more platforms.  This is normal in development, not
something to worry about.

Sorry for being blunt, but I suspect that people who demand features
to be left on branches for prolonged periods of time want to delay
those features or prevent their merge in the first place.

> sqlite3 has problematic implications.  There may be a good solution
> but we have not determined that.  Meanwhile, other solutions were
> there to be considered.

Frommy POV, the problematic implications of sqlite3, such as they are,
were adequately addressed.  Other solutions were considered in due
time, and one of them is now in the codebase.  I see no problem here.

> We should not have rushed the decision on what to do.  We should have
> taken it slow, leaving time for people to study various solutions and
> make a thoughtful decision.

We didn't rush.  There was a discussion with many participants that
took several days, opinions were heard and considered.  The decision
process was sound.  I understand that people who disagree with the
decision might feel the process was lacking, but it is not so.

> Perhaps that decision would have included installing sqlite3.  I won't
> say it wouldn't.  But we should not repeat the way that decision was
> made.

Since I see no problem with the decision-making process in this case,
I don't think I understand what exactly should not be repeated.  It
was a relatively simple decision to make, there's no reason to make a
mountain out of a molehill.

Re: sqlite3

[Jean Louis]

* Alan Mackenzie <acm@muc.de> [2021-12-11 00:26]:
> Hello, Eli and Emacs.
> 
> On Wed, Dec 08, 2021 at 15:14:08 +0200, Eli Zaretskii wrote:
> > > From: Qiantan Hong <qhong@mit.edu>
> > > Date: Wed, 8 Dec 2021 05:41:17 +0000
> > > Cc: Po Lu <luangruo@yahoo.com>, Eric Abrahamsen <eric@ericabrahamsen.net>,
> > >  cesar mena <cesar.mena@gmail.com>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> > > I’m against using sqlite3, or maybe in general any database.
> 
> > Why would you be against adding to Emacs infrastructure that might be
> > useful for some applications?
> 
> Because it can degrade other applications. 

To me the inclusion of PostgreSQL module is making my life and
business so much easier since its inception. Before that I was using
other PostgreSQL Emacs Lisp interfaceses. Usage of such module does
not degrade and did not degrade any other application that I know in
Emacs. 

sqlite3 is lite as compared to PostgreSQL and I think it is blazing
fast. Once can intersect data with speed and I doubt it would be
slower than what Emacs Lisp can do.

My experience with PostgreSQL tells me that information locating is so
much faster than if I would use plain Emacs Lisp without the database. 

> Primarily Emacs is a programmers' text editor, and I would like it
> to stay that way.

SQLite - Wikipedia
https://en.wikipedia.org/wiki/SQLite

Research the subject. Browsers are for Internet browsing and they are
using SQLite. File system is remember also a database, including text
files with ordered data. Don't be afraid.

> Adding database functionality is inessential to that prime function
> (regardless of how important it might be), therefore is a burden to
> that prime function.

Lack of well structured database and by that I mean SQL relational
databases, is and always was a burden to my programming. With
databases I have increased my productivity and minimized programming
efforts, thus gained speed and also much money. Without databases and
connection from withing Emacs I would be lost and would be spending
time. Back in time I have been using web interface on a local Apache
web server, and now I am using exclusively Emacs. I can connect to
remote databases and allow other people connect to my database and by
using Emacs edit all the information, update, delete, query and follow
up with people for sales and other purposes. 

> All things like gnus, org-mode are likewise inessential - they add
> many megabytes to the source-code, greatly increase build time, take
> up key-bindings, space in documentation (hopefully ;-) and so on.

That is possible to solve, there is minimal Emacs I guess in Debian
and this option:

./configure --without-all

and I often use it on remote servers that way.

> Similarly, I would be very unhappy if somebody transformed Emacs into a
> database engine, where its prime features were unusable without the
> database features being built in.  I think this possibility, remote at
> this juncture, should be borne in mind.

Database backed variables may be think of as global variables and they
could be quite comfortable for a programmer with the difference that
such remain persistent in background. 


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 8-a de decembro 2021, 6-a horo kaj 41:17 CET Qiantan Hong a 
écrit :
> 3. Database files are not readable directly by a text editor.

Well, if we start supporting sqlite, emacs might become such an editor!  
Imagine if when you opened a binary sqlite3 database emacs offered you the 
database, with many elisp commands to edit it by hand? think like 
phpmyadmin, but lighter

> 4. SQLite3 is an additional dependency. I hear people saying one
> can make it optional, but I highly doubt if people are willing
> to maintain 2 version of code with/without SQLite3, especially
> after they starts to write SQL instead of Elisp.

Exactly, so to make that safe we ought to make a standard abstraction over 
sqlite, that encompass some alternatives.

> 5. Database is essential for manipulating trillions of records,
> but that's not a typical use case of Emacs. It provides little
> benefit if data fits into main memory.

We could see new usages of emacs, replacing more other programs with the 
emacs way, or replacing *better* programs emacs (that is, Lars, through 
emacs) is already trying to replace, such as MUA (gnus, rmail), and 
browsers (eww).

> Some other alternative ((g)dbm, recutils) is free from some of
> those problem, but I think the following option is the best overall:
> 
> 1. Implement a persistent key/value store or object store in
> plain Emacs Lisp. I posted an implementation in anther thread
> titled "Against sqlite3!!! (Was: sqlite3)"

Weren’t there some error about using load instead of more safely reading 
file content?

> 2. I highly doubt if LISP is in any case insufficient for querying.
> But in those case, embeding a Prolog implementation in Emacs
> Lisp is a much better option than SQL.

To me prolog is a little overkill, and sql slightly more readable and 
practical to query complex values.  If you want something almost as 
practical but simpler, there’s also SparQL, which is used by WikiData and 
comes from the semantic/xml/rdf world.

However, one could just as well reimplement SQL into emacs, starting with 
a naive implementation that just do what’s asked, without formidable 
optimisation.  If anyone did that, we could even use that on top of gdbm 
and unintentionally end up with a not undecent lisp database…  And that 
could be used as a even higher sql-ish, yet lisp, abstraction to (efficient 
or not) databases.

Re: sqlite3

[Georges Ko]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> Le merkredo, 8-a de decembro 2021, 6-a horo kaj 41:17 CET Qiantan Hong a 
> écrit :
>> 3. Database files are not readable directly by a text editor.
>
> Well, if we start supporting sqlite, emacs might become such an editor!  
> Imagine if when you opened a binary sqlite3 database emacs offered you the 
> database, with many elisp commands to edit it by hand? think like 
> phpmyadmin, but lighter

I guess the first thing that would be written is a sqlite3 "REPL"
client...

Then, sqlite3 database files would be processed in the same way as tar files:

  - press 'f' in dired on a .db file:
    - the list of tables, indexes, etc. are shown
      - press some key to manage tables, indexes, ...
        - press some key to create, delete, ... tables, indexes ...
      - press some key on some table name to show table description:
         - press some key to enter edit mode to add, remove or edit
           columns, etc...
      - press some other key on some table name to display the content
        of the table in something similar to an org-mode table:
         - press some key to enter edit mode:
           - delete a line = delete that row
           - insert a line = insert that row
           - edit some fields = update these fields
           - etc.
           - once finish, press some key to commit...

   That's for one file.

   If another file is opened, then the whole 'dired' playbook can be
   applied: 
     - with two table lists buffers:
          - press some key to copy the table in the other file (think
            dired-dwim-target), 
          - press some key to create the table schema in the other
            file,
          - etc...

     - with table1 content of file A and table2 content of file B shown,
       assuming they have the same columns:
          - mark table1's rows, then press some key to copy to table2,
            as you would with file...
          - or press some key and you can happily edit/copy/paste between
            the two buffers
          - etc...

  That would work through TRAMP as well...

> We could see new usages of emacs, replacing more other programs with the 
> emacs way, or replacing *better* programs emacs (that is, Lars, through 
> emacs) is already trying to replace, such as MUA (gnus, rmail), and 
> browsers (eww).

   Let's say there are other backends (maybe something similar to Clojure
   CIDER/nrepl with JDBC clients connecting to databases and
   communicating back to Elisp) which are compatible with the stuff
   above: you end up having a universal database viewer/editor (limited
   by performance of Elisp), where you have an unified way to view and
   edit databases, where you could copy rows between two totally
   different databases by just copying/pasting lines, with Emacs doing
   the mediation between the clients...

Georges
-- 
 Georges Ko                     gko@gko.net                      2021-12-09

Re: sqlite3

[Jean Louis]

* Alexandre Garreau <galex-713@galex-713.eu> [2021-12-09 10:06]:
> Well, if we start supporting sqlite, emacs might become such an editor!  
> Imagine if when you opened a binary sqlite3 database emacs offered you the 
> database, with many elisp commands to edit it by hand? think like 
> phpmyadmin, but lighter

There are many possibilities that will open with the implementation of
a database. Editing of its fields, adding, removing, that is not
hard. I am doing that in tabulated-list-mode for PostgreSQL and it
could be somehow reworked for SQLite.

I would like that main developers implement error free insertion of
Emacs Lisp into table columns. Example is if I wish to store a
function or multiple functions into SQLite table column, that such is
not stored as string which I have to `eval' to read it, rather that
some error-free method is invented so that storage goes error-free
(correct sexp) into the column and that reading from it goes without
problems. Error reporting shall be also bound to find where, which
database column specific Emacs Lisp failed, as to quickly located it
and edit it.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Alexandre Garreau]

Le lundo, 14-a de februaro 2022, 13-a horo kaj 45:58 CET Jean Louis a 
écrit :
> * Alexandre Garreau <galex-713@galex-713.eu> [2021-12-09 10:06]:
> 
> 
> > Well, if we start supporting sqlite, emacs might become such an
> > editor!
> > Imagine if when you opened a binary sqlite3 database emacs offered you
> > the database, with many elisp commands to edit it by hand? think like
> > phpmyadmin, but lighter
> 
> There are many possibilities that will open with the implementation of
> a database. Editing of its fields, adding, removing, that is not
> hard. I am doing that in tabulated-list-mode for PostgreSQL and it
> could be somehow reworked for SQLite.
> 
> I would like that main developers implement error free insertion of
> Emacs Lisp into table columns. Example is if I wish to store a
> function or multiple functions into SQLite table column, that such is
> not stored as string which I have to `eval' to read it, rather that
> some error-free method is invented so that storage goes error-free
> (correct sexp) into the column and that reading from it goes without
> problems. Error reporting shall be also bound to find where, which
> database column specific Emacs Lisp failed, as to quickly located it
> and edit it.

to read you don’t need to eval, if you want lowerlevel stuff you could 
store bytecode, and print should be sufficient for it to be read again 
without errors shouldn’t it?

Re: sqlite3

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2021-12-08 07:36]:
> Whatever we do, we need to have a simple and convenient user interface
> for the user to browse all these records and edit them.  If this works
> conveniently enough, maybe it could replace the Customize buffer.

That is feasible to make. It could replace all of the customize. But
one does need SQL database for that, not just key/value database such
as GDBM. But I vouch to have GDBM and other DBM-like databases equally
accessible straight from Emacs and Emacs Lisp.

I am editing entries in the PostgreSQL database by using Emacs module
to access PostgreSQL, by using tabulated-list-mode, and here is the
text of one such entry.

                             ID   319284
                   Date created   "2020-04-03 14:47:16.320089+03"
                  Date modified   "2021-07-03 00:48:07.059307+03"
                         Prefix   "Dr."
                     First name   "Richard"
                   Middle names   "Matthew"
                      Last name   "Stallman"
                         Suffix   nil
                        Account   "GNU.org"
                        Company   nil
                      Member of   nil
                    Lead source   "Web site"
                          Title   nil
                     Department   nil
                      Birthdate   nil
                     Reports to   nil
                    Do not call   nil
             Invalid e-mail (1)   nil
             Invalid e-mail (2)   nil
             Invalid e-mail (3)   nil
                   Office phone   nil
                   Mobile phone   nil
                     Home phone   nil
                    Other phone   nil
                            Fax   nil
                     E-mail (1)   "rms@example.org"
                     E-mail (2)   nil
                     E-mail (3)   nil
                        Website   nil
                  Other website   nil
                  Other contact   nil
                 Contact type 1   nil
                Other contact 2   nil
                 Contact type 2   nil
                Other contact 3   nil
                 Contact type 3   nil
                Primary address   nil
                   Primary city   nil
            Primary postal code   nil
                  Primary state   nil
                Primary country   "UNITED STATES"
                  Other address   nil
                     Other city   nil
              Other postal code   nil
                    Other state   nil
                  Other country   nil
                    Description   nil
           Modified by username   "wod"
            Created by username   "wod"
                  Introduced by   "Jean Louis"
                Obsolete emails   nil
         Obsolete phone numbers   nil
                    Other names   nil
                         Tokens   "'gnu.org':5 'matthew':2 'richard':1 'rms@example.org':4 'stallman':3 'state':7 'unit':6"


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Jonas Bernoulli]

Jean Louis <bugs@gnu.support> writes:

> I am editing entries in the PostgreSQL database by using Emacs module

Where can I get that module from?

Re: sqlite3

[Stefan Monnier]

Jonas Bernoulli [2022-02-15 19:19:54] wrote:
> Jean Louis <bugs@gnu.support> writes:
>> I am editing entries in the PostgreSQL database by using Emacs module
> Where can I get that module from?

I think he's using `emacs-libpq` which is in the process of being added
to GNU ELPA (it's in `elpa.git` already, there are just some packaging
issues left to address).


        Stefan

Re: sqlite3

[Jean Louis]

* Jonas Bernoulli <jonas@bernoul.li> [2022-02-15 21:20]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > I am editing entries in the PostgreSQL database by using Emacs module
> 
> Where can I get that module from?

emacs-libpq @ Github
https://github.com/anse1/emacs-libpq



-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> Would someone please post a concise description of the proposal that is
> being considered?  There have been over 100 messages in this thread,
> too many for me to read, but I'd like to know what the issue is.

Just read the first one.  It explains everything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Pip Cet]

On Tue, Dec 7, 2021 at 5:11 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Richard Stallman <rms@gnu.org> writes:
>
> > Would someone please post a concise description of the proposal that is
> > being considered?  There have been over 100 messages in this thread,
> > too many for me to read, but I'd like to know what the issue is.
>
> Just read the first one.  It explains everything.

I disagree, FWIW. There's very little in that message that goes beyond
"let's use sqlite!" What I've been able to make of this so far is:

The proposal is to have Emacs store some user data in some binary
format that cannot be readily inspected, diffed, backed up,
version-controlled, shared, altered, or understood. (Or archived,
published, indexed, checksummed, ...)

Those are huge disadvantages. The benefits are not clear to me at all,
but they seem to revolve around the time it takes for an interrupted
or terminated session to restart?

I don't think there are any applications for which the trade off (gain
some performance; lose some user freedom) is worth it, but even if
there are, we should look harder at alternatives which allow us to
gain similar amounts of performance without littering unsuspecting
users' .emacs.ds with binary-only files.

And we need to be careful about introducing this "as an option", of
course, because that means two versions of many packages' core code
need to be maintained, only one of which will be tested by users who
have not had the time to customize the defaults.

(I believe there are also good reasons for avoiding SQL, but those are
not as important as the non-technical issues mentioned above.)

I've tested a little and at least in those initial tests, sqlite seems
to actually delete data from sqlite files when the corresponding rows
are deleted, but I'd like to mention this explicitly because I'm not
sure there's actually a guarantee of it in place: if sqlite ever keeps
storing data that the user thought they deleted, that makes it
unsuitable for storing the kind of data I would expect in .emacs.d.

Of course, none of this is meant to stop people experimenting in any
way, but I don't see any package that should realistically be able to
make use of it without an opt-in option and probably an extra warning.

Pip

Re: sqlite3

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Date: Wed, 8 Dec 2021 18:36:24 +0000
> Cc: eric@ericabrahamsen.net, cesar mena <cesar.mena@gmail.com>,
>  Richard Stallman <rms@gnu.org>, emacs-devel@gnu.org
> 
> > Just read the first one.  It explains everything.
> 
> I disagree, FWIW. There's very little in that message that goes beyond
> "let's use sqlite!"

Why is that a problem?

> The proposal is to have Emacs store some user data in some binary
> format

No, the proposal is to have Emacs _capable_ of storing data in a DB.
We didn't yet seriously discuss whether some core features will
actually use that, and if so, which ones.

> that cannot be readily inspected, diffed, backed up,
> version-controlled, shared, altered, or understood. (Or archived,
> published, indexed, checksummed, ...)

That's incorrect: tools exist that can browse sqlite databases and
compare them.

> Those are huge disadvantages. The benefits are not clear to me at all,
> but they seem to revolve around the time it takes for an interrupted
> or terminated session to restart?

No, the benefit is to be able to store and retrieve data from a
persistent DB.  Python has it, Lua has it, JS has it, so why
shouldn't Emacs be able to have it?

> I don't think there are any applications for which the trade off (gain
> some performance; lose some user freedom) is worth it, but even if
> there are, we should look harder at alternatives which allow us to
> gain similar amounts of performance without littering unsuspecting
> users' .emacs.ds with binary-only files.

If you are right, then this capability will remain largely unused.
Like Lisp threads, for example.  Let's talk in a couple of years.

> And we need to be careful about introducing this "as an option", of
> course, because that means two versions of many packages' core code
> need to be maintained, only one of which will be tested by users who
> have not had the time to customize the defaults.

No, it means the packages that need such a capability will _require_
Emacs with sqlite support.  Like any package that needs to display
SVG requires Emacs with librsvg support, for example.

> I've tested a little and at least in those initial tests, sqlite seems
> to actually delete data from sqlite files when the corresponding rows
> are deleted, but I'd like to mention this explicitly because I'm not
> sure there's actually a guarantee of it in place: if sqlite ever keeps
> storing data that the user thought they deleted, that makes it
> unsuitable for storing the kind of data I would expect in .emacs.d.

Your file system doesn't actually erase the data from the disk,
either, as you probably well know.  That doesn't mean we cannot use
disk I/O in Emacs.

> Of course, none of this is meant to stop people experimenting in any
> way, but I don't see any package that should realistically be able to
> make use of it without an opt-in option and probably an extra warning.

I don't see any reason for these precautions.  We never did that for
any other optional feature that requires an optional library.

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 8-a de decembro 2021, 19-a horo kaj 57:32 CET Eli Zaretskii a 
écrit :
> > Those are huge disadvantages. The benefits are not clear to me at all,
> > but they seem to revolve around the time it takes for an interrupted
> > or terminated session to restart?
> 
> No, the benefit is to be able to store and retrieve data from a
> persistent DB.  Python has it, Lua has it, JS has it, so why
> shouldn't Emacs be able to have it?

Because Emacs is Lisp, and is has “read” and “print”, that is, is 
homoiconic, while those are not.

But, on a similar perspective, we could argue JS has JSON, and yet all 
data is not stored as JSON (well: there is a strong tendency and desire to 
do that, as show some “modern” database such as mongodb).  Lua is very 
declarative and functional as well.

However lisp has a longer and stronger history of supporting 
inspectability, so the concern can be understood.  However, contrarily to 
the unix tradition, which modern emacses somewhat follow but which is 
younger than lisp, that doesn’t require everything to be in plaintext… 
just inspectable from lisp.  And if we build purely lisp interfaces, that 
will necessarily be the case, and we’ll even get elisp and emacs’ tools to 
inspect and treat generically sqlite databases (and that would be very 
neat! and hard to oppose to!), as long as the interface is general enough.

> > I don't think there are any applications for which the trade off (gain
> > some performance; lose some user freedom) is worth it, but even if
> > there are, we should look harder at alternatives which allow us to
> > gain similar amounts of performance without littering unsuspecting
> > users' .emacs.ds with binary-only files.
> 
> If you are right, then this capability will remain largely unused.
> Like Lisp threads, for example.  Let's talk in a couple of years.

No, absolutely not.  Not necessarily.  It could be used just because 
sqlite is a trend.  Just because it’s so much used that it looks cool.  By 
network effect.  Because people coming from different background would 
already know it (that’s an advantage for emacs), and get lazy to learn 
more lispy stuff (that’s a dramatic (but not that important) risk).

I mean many things which are not worth it are done netherless, that’s not 
an argument: imagine about if the vast majority of modern computation was 
done in ridiculously sandboxed javascript, tied to servers, and without a 
licence…

> > And we need to be careful about introducing this "as an option", of
> > course, because that means two versions of many packages' core code
> > need to be maintained, only one of which will be tested by users who
> > have not had the time to customize the defaults.
> 
> No, it means the packages that need such a capability will _require_
> Emacs with sqlite support.  Like any package that needs to display
> SVG requires Emacs with librsvg support, for example.

We could do that in some other way by taking care of atomistically not 
introduce sqlite without a higher-level “preferred” interface that would 
be encouraged for the simplest uses, and allow to select a different 
backend such as “gdbm”, “bare sexps”, etc.

If think the reason for the very emotional reaction here is the display of 
a fear, which has to be justified in some way, of the fact emacs may 
subbish a pressure to become “more like anything else”, that is less 
inspectable, less compatible, etc.

Re: sqlite3

[Pip Cet]

On Wed, Dec 8, 2021 at 6:57 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Pip Cet <pipcet@gmail.com>
> > Date: Wed, 8 Dec 2021 18:36:24 +0000
> > Cc: eric@ericabrahamsen.net, cesar mena <cesar.mena@gmail.com>,
> >  Richard Stallman <rms@gnu.org>, emacs-devel@gnu.org
> >
> > > Just read the first one.  It explains everything.sq
> >
> > I disagree, FWIW. There's very little in that message that goes beyond
> > "let's use sqlite!"
>
> Why is that a problem?

Lars said his message explains everything. I say it doesn't. Whether
that is a problem or not is up to you.

> > The proposal is to have Emacs store some user data in some binary
> > format
>
> No, the proposal is to have Emacs _capable_ of storing data in a DB.

See, apparently, not even that is made clear in the message.

However, having reread it, I maintain that Lars is indeed proposing to
have Emacs store etc. etc. I just don't see another way of reading it.

> We didn't yet seriously discuss whether some core features will
> actually use that, and if so, which ones.

I agree that the necessary discussion has not taken place.

> > that cannot be readily inspected, diffed, backed up,
> > version-controlled, shared, altered, or understood. (Or archived,
> > published, indexed, checksummed, ...)
>
> That's incorrect: tools exist that can browse sqlite databases and
> compare them.

"readily".

> > Those are huge disadvantages. The benefits are not clear to me at all,
> > but they seem to revolve around the time it takes for an interrupted
> > or terminated session to restart?
>
> No, the benefit is to be able to store and retrieve data from a
> persistent DB.  Python has it, Lua has it, JS has it, so why
> shouldn't Emacs be able to have it?

That's not the proposal as I read it.

> > I don't think there are any applications for which the trade off (gain
> > some performance; lose some user freedom) is worth it, but even if
> > there are, we should look harder at alternatives which allow us to
> > gain similar amounts of performance without littering unsuspecting
> > users' .emacs.ds with binary-only files.
>
> If you are right, then this capability will remain largely unused.
> Like Lisp threads, for example.  Let's talk in a couple of years.

I don't believe in the infinite wisdom of package maintainers,
particularly when it comes to not doing a technically nifty thing that
happens to have actual side effects.

> > And we need to be careful about introducing this "as an option", of
> > course, because that means two versions of many packages' core code
> > need to be maintained, only one of which will be tested by users who
> > have not had the time to customize the defaults.
>
> No, it means the packages that need such a capability will _require_
> Emacs with sqlite support.  Like any package that needs to display
> SVG requires Emacs with librsvg support, for example.

Same effect: users will be forced to have to deal with binary files,
because they either don't have the option not to use them or won't
notice they're using them until it's too late. Let's avoid that.

> > I've tested a little and at least in those initial tests, sqlite seems
> > to actually delete data from sqlite files when the corresponding rows
> > are deleted, but I'd like to mention this explicitly because I'm not
> > sure there's actually a guarantee of it in place: if sqlite ever keeps
> > storing data that the user thought they deleted, that makes it
> > unsuitable for storing the kind of data I would expect in .emacs.d.
>
> Your file system doesn't actually erase the data from the disk,
> either, as you probably well know.  That doesn't mean we cannot use
> disk I/O in Emacs.

I think the danger of people accidentally sharing a file without
considering what invisible extra information it might contain
(particularly if there's no good reason for such invisible information
to be present in the first place) is MUCH larger than that of people
deliberately sharing an image of their decrypted file system volume,
expecting only current file system-visible content to be preserved.

The first happens on a regular basis with blacked-out PDFs, Microsoft
Word and Microsoft Excel documents, and other such abominations. I've
not heard of the latter happening on any significant scale.

The former makes it unsuitable for .emacs.d. The latter only makes it
important to keep educating people about what file systems do and do
not do.

> > Of course, none of this is meant to stop people experimenting in any
> > way, but I don't see any package that should realistically be able to
> > make use of it without an opt-in option and probably an extra warning.
>
> I don't see any reason for these precautions.  We never did that for
> any other optional feature that requires an optional library.

You're right that features of Emacs were usually introduced by turning
them on by default, then breaking the "off" option because no one used
it anymore (see pdumper). I don't think that this approach has been
hugely successful, certainly not enough to justify repeating that
mistake in this case.

Pip

Re: sqlite3

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Date: Thu, 9 Dec 2021 19:46:22 +0000
> Cc: larsi@gnus.org, eric@ericabrahamsen.net, cesar.mena@gmail.com, rms@gnu.org, 
> 	emacs-devel@gnu.org
> 
> > > The proposal is to have Emacs store some user data in some binary
> > > format
> >
> > No, the proposal is to have Emacs _capable_ of storing data in a DB.
> 
> See, apparently, not even that is made clear in the message.

What else can be meant by a proposal to allow Emacs to create and
access databases?  There's no other reasonable way of reading that
message.

> However, having reread it, I maintain that Lars is indeed proposing to
> have Emacs store etc. etc. I just don't see another way of reading it.

Of course, if we give Emacs such a capability, it's to let it be
capable of storing etc.  What else did you expect?  It's still just a
capability, and any of its use in core warrants a separate discussion.
At that time, if you think some use of this capability is not the best
idea, please voice your opinions and propose alternatives you think
are better.  But objecting to a capability as such makes no sense to
me, because it makes Emacs a more powerful platform.

> > > that cannot be readily inspected, diffed, backed up,
> > > version-controlled, shared, altered, or understood. (Or archived,
> > > published, indexed, checksummed, ...)
> >
> > That's incorrect: tools exist that can browse sqlite databases and
> > compare them.
> 
> "readily".

Which means what exactly?  We use a tool, Diff, to compare text
files.  We use another tool, sqldiff, to compare databases.  Where's
the crucial difference that justifies rejecting the _capability_ of
working with a DB?

> > No, the benefit is to be able to store and retrieve data from a
> > persistent DB.  Python has it, Lua has it, JS has it, so why
> > shouldn't Emacs be able to have it?
> 
> That's not the proposal as I read it.

You are arguing against specific uses of this capability that no one
has yet seriously put on the table.  Please wait until such a
discussion actually happens, and propose the alternatives for that
specific application of the DB capabilities.  Talking about it now is
too soon, and is also not efficient, because there's no specific
concrete application we can discuss, and so the arguments tend to be
semi-philosophical, something that doesn't facilitate useful
discussions.

> > If you are right, then this capability will remain largely unused.
> > Like Lisp threads, for example.  Let's talk in a couple of years.
> 
> I don't believe in the infinite wisdom of package maintainers,
> particularly when it comes to not doing a technically nifty thing that
> happens to have actual side effects.

And you think by refusing to have this capability in the core we will
be able to prevent that?

> > > And we need to be careful about introducing this "as an option", of
> > > course, because that means two versions of many packages' core code
> > > need to be maintained, only one of which will be tested by users who
> > > have not had the time to customize the defaults.
> >
> > No, it means the packages that need such a capability will _require_
> > Emacs with sqlite support.  Like any package that needs to display
> > SVG requires Emacs with librsvg support, for example.
> 
> Same effect: users will be forced to have to deal with binary files,
> because they either don't have the option not to use them or won't
> notice they're using them until it's too late. Let's avoid that.

There's nothing wrong with binary files, IMO.  E.g., Git (and any
modern VCS) uses them.  Heck, even Emacs does: what are the *.elc
files?  But if I'm wrong, users will complain, and we all learn a
lesson.

> > Your file system doesn't actually erase the data from the disk,
> > either, as you probably well know.  That doesn't mean we cannot use
> > disk I/O in Emacs.
> 
> I think the danger of people accidentally sharing a file without
> considering what invisible extra information it might contain
> (particularly if there's no good reason for such invisible information
> to be present in the first place) is MUCH larger than that of people
> deliberately sharing an image of their decrypted file system volume,
> expecting only current file system-visible content to be preserved.
> 
> The first happens on a regular basis with blacked-out PDFs, Microsoft
> Word and Microsoft Excel documents, and other such abominations. I've
> not heard of the latter happening on any significant scale.

It happens all the time on any multi-user system.

> > I don't see any reason for these precautions.  We never did that for
> > any other optional feature that requires an optional library.
> 
> You're right that features of Emacs were usually introduced by turning
> them on by default, then breaking the "off" option because no one used
> it anymore (see pdumper). I don't think that this approach has been
> hugely successful, certainly not enough to justify repeating that
> mistake in this case.

I disagree with your assessments, and I think I have more experience
to base that on than you do.

Re: sqlite3

[Qiantan Hong]

> Of course, if we give Emacs such a capability, it's to let it be
> capable of storing etc.  What else did you expect?  It's still just a
> capability, and any of its use in core warrants a separate discussion.
If that’s indeed the case, then sure. I’m against using database,
but have no opinion on adding support without using it.

It follows logically from my 5 arguments that there will be
no useful case for database in Emacs, even if there appear to
be some, they could always be done better with pure Emacs Lisp.
I didn’t see any refutation to those arguments.

Resist! Is intended to constructively prove the above "could always be 
done better with pure Emacs Lisp” proposition.
If you ever find any use case that cannot be done with resist! and
can only resort to sqlite3, let me know, I’ll try to solve it,
or I’ll admit defeat.

Resist! probably still need a few versions to mature, but interface
should remain unchanged.

> At that time, if you think some use of this capability is not the best
> idea, please voice your opinions and propose alternatives you think
> are better.


What is the usage pattern of gnus registry and can it be backed using
resist!?


Best,
Qiantan

Re: sqlite3

[Michael Heerdegen]

Qiantan Hong <qhong@mit.edu> writes:

> What is the usage pattern of gnus registry and can it be backed using
> resist!?

The registry stores data about your messages: relations, marks and such.
Its mostly a function message-id -> data in the mathematical sense, so
it's stored using a big hash table.  More or less.

It's implemented using the `eieio-persistent' class.

It allows to attach additional information to messages, like user
defined marks or anything else.  Allows to jump to parent messages that
are not part of the current group.  Such things.  Gnus then knows about
all your messages and where they are.

Michael.

Re: sqlite3

[Qiantan Hong]

> On Dec 9, 2021, at 8:45 PM, Michael Heerdegen <michael_heerdegen@web.de> wrote:
> 
> It allows to attach additional information to messages, like user
> defined marks or anything else.  Allows to jump to parent messages that
> are not part of the current group.  Such things.  Gnus then knows about
> all your messages and where they are.
Thanks, I’ve read through gnus-registry.el and it seems to use
only a few of the functions from registry.el
(mostly registry-lookup-secondary, registry-insert and registry-delete),
which seems well suited to be implemented by a general key value store.

Is there anything we want to improve from the current implementation?
Resist! will eliminate the need of explicitly saving the registry, but
is loading the registry currently considered a bottleneck?

> On Dec 9, 2021, at 10:23 PM, larsi@gnus.org wrote:
> 
>> What if someone else does?  Are patches welcome?
> 
> Sure, if they handle the semantics in a nice enough way.  (I.e., not
> making it slower, have the same concurrency guarantees, allow the same
> storage browsing, etc.)
Resist! will handle the concurrency guarantee. 

Performance will be no slower than current implementation, of course. 
IDK if load is considered a bottleneck, but if so I can implement split 
bucketing which will should make the maximum overall pause time comparable 
to an SQLite3 implementation.

As for storage browsing, any tool gnus provides for now should work.
If there’s to be a database editor package in the future, I wonder if
said package can support browser Emacs Lisp image.

> On Dec 10, 2021, at 12:12 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> But the objections explicitly sounded like they were against having
> the DB access capabilities within Emacs.  At least one participant
> explicitly said so.  Which is what I criticized from my POV as one of
> the Emacs maintainers -- resisting to addition of a useful capability
> doesn't make sense.
I was and am opposing “using” databases. Maybe the expression is
awfully ambiguous, I apologize — I might not have known what I really mean
before others point out there’re two completely separate issues here.
ofc I don’t oppose adding SQLite3 support for a SQLite3 browsing/editing
package, what I’m resisting is the use of SQLite3 in Emacs Lisp applications
whose application domain has nothing to do with SQLite3 and deserve
pure lisp implementation.

Best,
Qiantan

Re: sqlite3

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>,
>         "larsi@gnus.org"
> 	<larsi@gnus.org>, Eli Zaretskii <eliz@gnu.org>
> Date: Fri, 10 Dec 2021 11:15:28 +0000
> 
> > But the objections explicitly sounded like they were against having
> > the DB access capabilities within Emacs.  At least one participant
> > explicitly said so.  Which is what I criticized from my POV as one of
> > the Emacs maintainers -- resisting to addition of a useful capability
> > doesn't make sense.
> I was and am opposing “using” databases. Maybe the expression is
> awfully ambiguous, I apologize — I might not have known what I really mean
> before others point out there’re two completely separate issues here.
> ofc I don’t oppose adding SQLite3 support for a SQLite3 browsing/editing
> package, what I’m resisting is the use of SQLite3 in Emacs Lisp applications
> whose application domain has nothing to do with SQLite3 and deserve
> pure lisp implementation.

A blanket opposition like that, which is not specific to a certain use
or a class of uses of the capability, makes no sense at all.  You are
in effect saying that there are no applications in the world that
should use a DB except those that browse DBs created by others.  IOW,
in an ideal world, you'd deny the very need for having persistent DBs
for _any_ application.  Please rethink such a general attitude.

Re: sqlite3

[Qiantan Hong]

> On Dec 10, 2021, at 3:42 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> A blanket opposition like that, which is not specific to a certain use
> or a class of uses of the capability, makes no sense at all.  You are
> in effect saying that there are no applications in the world that
> should use a DB except those that browse DBs created by others.  IOW,
> in an ideal world, you'd deny the very need for having persistent DBs
> for _any_ application.  Please rethink such a general attitude.

Please read my original arguments. The scope is limited to Emacs.

> 5. Database is essential for manipulating trillions of records,
> but that's not a typical use case of Emacs. It provides little
> benefit if data fits into main memory.
The only non typical use case of Emacs I can think of is as an
database browser/editor.

Re: sqlite3

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Michael Heerdegen <michael_heerdegen@web.de>,
>         "larsi@gnus.org"
> 	<larsi@gnus.org>,
>         "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Fri, 10 Dec 2021 12:00:08 +0000
> 
> > A blanket opposition like that, which is not specific to a certain use
> > or a class of uses of the capability, makes no sense at all.  You are
> > in effect saying that there are no applications in the world that
> > should use a DB except those that browse DBs created by others.  IOW,
> > in an ideal world, you'd deny the very need for having persistent DBs
> > for _any_ application.  Please rethink such a general attitude.
> 
> Please read my original arguments. The scope is limited to Emacs.

It makes no sense to deny any possibility of using DB storage for
Emacs.

> > 5. Database is essential for manipulating trillions of records,
> > but that's not a typical use case of Emacs. It provides little
> > benefit if data fits into main memory.
> The only non typical use case of Emacs I can think of is as an
> database browser/editor.

Then we will have to disagree, and you will have to accept that the
Emacs maintainers think differently from you, and will use these
capabilities where that is deemed useful.

Re: sqlite3

[Michael Heerdegen]

Qiantan Hong <qhong@mit.edu> writes:

> Is there anything we want to improve from the current implementation?
> Resist! will eliminate the need of explicitly saving the registry, but
> is loading the registry currently considered a bottleneck?

As a registry user I can tell that saving currently takes a lot of time.
Both operations, loading and saving, should be as fast as possible.

Michael.

Re: sqlite3

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> As a registry user I can tell that saving currently takes a lot of time.
> Both operations, loading and saving, should be as fast as possible.

BTW, AFAIU, preparing the data for saving is taking most of the time.
This could be the real bottleneck.

Michael.

Re: sqlite3

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2021-12-09 23:31]:
> It follows logically from my 5 arguments that there will be
> no useful case for database in Emacs, even if there appear to
> be some, they could always be done better with pure Emacs Lisp.
> I didn’t see any refutation to those arguments.

I am very surprised as you are author of CRDT. In fact I am not sure
if you are joking or not. The reason why I like CRDT is that people
can concurrently edit files in the same time from remote locations. It
is great for collaboration. Databases are similarly great for
collaboration. 

Personally I am using PostgreSQL Emacs module, and that is great for
collaboration. A user from Seattle, WA can edit database backed
information through his Emacs and connection to my database, while I
can be talking to person which record we are concurrently editing, and
writing notes from the call. We can both in the same, from various
locations in the world access information pertaining to specific user
and follow up with the sales or other human interaction process.

Database applications are numerous and enabling databases from within
Emacs Lisp was done long ago. Having a database built-in or interface
straight in Emacs makes Emacs more powerful, it will benefit more
users. On my side it benefits so many people. Practical example is
that many people get employed because we use database and not text
files. 

There are already Emacs packages for all kinds of databases, for
SQLite: https://melpa.org/#/?q=sqlit

And when it is built-in, with direct access it will speed up the
development of many new software.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Joost Kremers]

On Thu, Dec 09 2021, Eli Zaretskii wrote:
> What else can be meant by a proposal to allow Emacs to create and
> access databases?  There's no other reasonable way of reading that
> message.

Actually, Lars' original message was to give Emacs a persistent variable store
and to use sqlite3 for that purpose.

No-one really objects to giving Emacs some sort of persistent variable store,
and no-one objects to giving Emacs the ability to communicate directly with
sqlite. But the proposal was about doing the former by means of the latter. And
I can understand how that would raise some doubts.


-- 
Joost Kremers
Life has its moments

Re: sqlite3

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Date: Thu, 09 Dec 2021 23:50:56 +0100
> Cc: Pip Cet <pipcet@gmail.com>, emacs-devel@gnu.org
> 
> On Thu, Dec 09 2021, Eli Zaretskii wrote:
> > What else can be meant by a proposal to allow Emacs to create and
> > access databases?  There's no other reasonable way of reading that
> > message.
> 
> Actually, Lars' original message was to give Emacs a persistent variable store
> and to use sqlite3 for that purpose.
> 
> No-one really objects to giving Emacs some sort of persistent variable store,
> and no-one objects to giving Emacs the ability to communicate directly with
> sqlite. But the proposal was about doing the former by means of the latter. And
> I can understand how that would raise some doubts.

But the objections explicitly sounded like they were against having
the DB access capabilities within Emacs.  At least one participant
explicitly said so.  Which is what I criticized from my POV as one of
the Emacs maintainers -- resisting to addition of a useful capability
doesn't make sense.

For specific uses of this in Emacs, there should be separate
discussions focused on each of the uses, because the alternatives are
different in each case and so are the pros and cons to consider.  For
example, I presume there would be no issue to use that as basis for a
package that allows browsing and modifying SQLite DBs from Emacs.

Re: sqlite3

[tomas]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

On Thu, Dec 09, 2021 at 11:50:56PM +0100, Joost Kremers wrote:
> 
> On Thu, Dec 09 2021, Eli Zaretskii wrote:
> > What else can be meant by a proposal to allow Emacs to create and
> > access databases?  There's no other reasonable way of reading that
> > message.
> 
> Actually, Lars' original message was to give Emacs a persistent variable store
> and to use sqlite3 for that purpose.
> 
> No-one really objects to giving Emacs some sort of persistent variable store,
> and no-one objects to giving Emacs the ability to communicate directly with
> sqlite. But the proposal was about doing the former by means of the latter. And
> I can understand how that would raise some doubts.

Thanks, Joost. I think your short mail describes my uneasiness about
Lars's proposal better than I could.

I always have that strange feeling in situations like this.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Which means what exactly?  We use a tool, Diff, to compare text
  > files.  We use another tool, sqldiff, to compare databases.  Where's
  > the crucial difference that justifies rejecting the _capability_ of
  > working with a DB?

The crucial difference between text files and sqlite3 databases,
unless I'm mistaken about the latter, is that text files are simple
and transparent, but sqlite3 databases are not.

Because text files are simple and transparent, you can compare text
files in many different ways: reading the two side by side, using
diff, using wdiff, using M-x compare-windows, and there are probably
more options.  And you can fix them manually, too.

Because sqlite3 databases are neither simple nor transparent, doing
anything with them requires using specific sqlite3 tools.

This is a major advantage for text files.

Using database format has other advantages, and there are cases where
they are important.  Mainly when the text file is big or operating on
it is slow.

Nonetheless I contend that, for a user who stores limited quantities
of persistent Emacs data, the text format is better.

That doesn't imply Emacs should not support sqlite3.  If you happen
to store large amounts of persistent data, you might like
saving persistent Emacs data that way.

But it does imply that sqlite3 should not be the default way to store
persistent Emacs data.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> The crucial difference between text files and sqlite3 databases,
> unless I'm mistaken about the latter, is that text files are simple
> and transparent, but sqlite3 databases are not.

I think you're mistaken.  sqlite3 databases are pretty transparent.
There's trivial tools that dump the contents as a series of "insert"
statements, that are about as easy to read as Emacs Lisp.  (Well, easier
for most people, I'd guess.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think you're mistaken.  sqlite3 databases are pretty transparent.

Would you please email me an sqlite3 database file, as an attachment?

  > There's trivial tools that dump the contents as a series of "insert"
  > statements, that are about as easy to read as Emacs Lisp.

I'm sure that is helpful, but there is a big difference between
conversion to and from text, and actually being textual.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > I think you're mistaken.  sqlite3 databases are pretty transparent.
>
> Would you please email me an sqlite3 database file, as an attachment?

You probably have several of them on your computer -- it's pretty hard
to not, these days.  "find / -name '*.sqlite" will give you a list.

>   > There's trivial tools that dump the contents as a series of "insert"
>   > statements, that are about as easy to read as Emacs Lisp.
>
> I'm sure that is helpful, but there is a big difference between
> conversion to and from text, and actually being textual.

There's certainly a difference, but I'm not sure it's that big.  A dump
of an Emacs Lisp hash table is certainly "textual", but I'm pretty sure
most users won't be able to read/alter it with confidence.

To view the contents of the sqlite file, you can use `M-x
sqlite-mode-open-file' on the current Emacs trunk, or you can say

echo -e '.dump' | sqlite3 file.sqlite

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You probably have several of them on your computer -- it's pretty hard
  > to not, these days.  "find / -name '*.sqlite" will give you a list.

I did that in /etc, /usr and /lib.  The only file it found was called
README.sqlite.  It was a text file.

Instead of insisting I don't need this small favor, would you please
email me an sqlite3 database file, as an attachment?  Preferably
a rather simple example.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

Richard Stallman <rms@gnu.org> writes:

> Instead of insisting I don't need this small favor, would you please
> email me an sqlite3 database file, as an attachment?  Preferably
> a rather simple example.

Attached.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

[-- Attachment #2: multisession.sqlite --]
[-- Type: application/vnd.sqlite3, Size: 16384 bytes --]

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Instead of insisting I don't need this small favor, would you please
  > > email me an sqlite3 database file, as an attachment?  Preferably
  > > a rather simple example.

  > Attached.

Thank you.

It looks quite binary to me.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> It looks quite binary to me.

Uhm...  yes?  Nobody has said that it isn't.  Emacs has commands to list
the contents, and there's a plethora of other commands out there to do
the same.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It looks quite binary to me.

  > Uhm...  yes?  Nobody has said that it isn't.

A real text file is more robust when things go wrong.

                                                  Emacs has commands to list
  > the contents, and there's a plethora of other commands out there to do
  > the same.

Yes, someone posted here about that, a few days ago.

A binary format that you can convert to text is not as robust as real
text.  Converters are designed to handle a valid binary file.  In
general, it's not clear what is the proper conversion of a corrupted
binary file.

Thus, in general, when the file is corrupted, you can't convert it to
text.

Therefore, "text file" should be the default option for configuration
data in Emacs.  People who would rather use an sqlite3 file should
select that option explicitly.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> Therefore, "text file" should be the default option for configuration
> data in Emacs.  People who would rather use an sqlite3 file should
> select that option explicitly.

Again, for the 55th time -- this facility will not be used for storing
configuration data in Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Again, for the 55th time -- this facility will not be used for storing
  > configuration data in Emacs.

I don't recall seeing you say that before.  You may have used
different words -- you didn't use the words "configuration data" -- so
perhaps I did not understand the previous statements.

The reason I expected this to be used for configuration specifications
is that that seems like a natural thing to do.  Assuming we get this
facility into a good shape, why NOT use it for storing configuration
specs?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> The reason I expected this to be used for configuration specifications
> is that that seems like a natural thing to do.  Assuming we get this
> facility into a good shape, why NOT use it for storing configuration
> specs?

I think defcustom is fine for configuration data.  Configuration data
doesn't change a lot, and people like to put it into Git, etc.

Application session state is a different beast, and is something that
Emacs has lacked a convenient interface for, so each package has come up
with its own way to store state, or more commonly -- not bothered,
because it's so much (boring) work.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: eric@ericabrahamsen.net,  eliz@gnu.org,  cesar.mena@gmail.com,
>   pipcet@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 16 Dec 2021 06:55:41 +0100
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > The reason I expected this to be used for configuration specifications
> > is that that seems like a natural thing to do.  Assuming we get this
> > facility into a good shape, why NOT use it for storing configuration
> > specs?
> 
> I think defcustom is fine for configuration data.  Configuration data
> doesn't change a lot, and people like to put it into Git, etc.

I think basically wherever saving configuration in text form makes
sense, i.e. it is short enough and readable enough by humans, it
should be preferred, because that allows viewing it, modifying it, and
generally managing it by more general-purpose tools.

> Application session state is a different beast, and is something that
> Emacs has lacked a convenient interface for, so each package has come up
> with its own way to store state, or more commonly -- not bothered,
> because it's so much (boring) work.

Application state is indeed a different issue, that should be decided
separately on a case by case basis.  For example, should
.emacs.desktop remain in its current text form?  I personally think it
should, because it's still readable enough and performant enough, but
there could be other opinions.

Large databases are yet another different beast.  For example,
consider the mairix-related features in Emacs: we current use an
external program for that, where using a database internally from
Emacs could make at least some sense: for starters, we wouldn't be
limited by search capabilities of mairix the program, but could
instead unleash the full power of Emacs searching capabilities.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Application state is indeed a different issue, that should be decided
> separately on a case by case basis.  For example, should
> .emacs.desktop remain in its current text form?  I personally think it
> should, because it's still readable enough and performant enough, but
> there could be other opinions.

Yeah, I think .emacs.desktop is probably fine as it is.

Stefan M raised the issue about savehist -- the problem there is that is
you have two Emacsen running at the same time, the one you close last
will overwrite the data from the other instances.  Quintan's log
approach might work for that, but I'm wondering what the semantics
should be.

That is, ideally I think if you do A in Emacs 1, B in Emacs 2, and C
in the first Emacs again, the savehist should have

A
B
C

and not

A
C
B

(if you closed the first Emacs first, and then the second).

Possibly.

And perhaps...  after doing A in Emacs 1, you might be able to see that
immediately in Emacs 2?  But that could also lead to all kinds of
usability problems.

So it's no clear what the interfaces here might be for these more
complex application states, and there may not be a "one solution fits
all" approach for these.

> Large databases are yet another different beast.  For example,
> consider the mairix-related features in Emacs: we current use an
> external program for that, where using a database internally from
> Emacs could make at least some sense: for starters, we wouldn't be
> limited by search capabilities of mairix the program, but could
> instead unleash the full power of Emacs searching capabilities.

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2021-12-15 08:16]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > It looks quite binary to me.
> 
>   > Uhm...  yes?  Nobody has said that it isn't.
> 
> A real text file is more robust when things go wrong.

Not so categorically speaking. Text files depend on binary foundation
such as the file system and its storage. Change a byte by `poke' on
file system and things may go wrong. Hard disks failures will change
files as well, make them disappear or break all the data.

For things not to go wrong one has to make backups and move data to
different media.

> A binary format that you can convert to text is not as robust as real
> text.  Converters are designed to handle a valid binary file.  In
> general, it's not clear what is the proper conversion of a corrupted
> binary file.

Backup of sqlite to text is easy:

sqlite> .output "mybackup.sql"
sqlite> .dump

> Thus, in general, when the file is corrupted, you can't convert it
> to text.

When text file is really corrupted one cannot recover the data that
was there in the text file. When it is deleted, one cannot get it
(unless with undelete tools). 

It is very simple to invoke database backups. It was always
suggestion for computer users to backup their file systems, diskettes,
cassettes, to "archive" data, that was since the inception of
computers. Security of data applies equally to text files and binary
files. 

> Therefore, "text file" should be the default option for configuration
> data in Emacs.  People who would rather use an sqlite3 file should
> select that option explicitly.

I agree to that for reason of Emacs history and Emacs Lisp usage over
time. 

I don't agre to arguments for that. Databases have been designed with
integrity, safety, reliability, recoverability, fault
tolerance. Generalization which represent your personal opinions,
cannot dispute those well established, technical and known facts.


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2021-12-14 07:15]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > Instead of insisting I don't need this small favor, would you please
>   > > email me an sqlite3 database file, as an attachment?  Preferably
>   > > a rather simple example.
> 
>   > Attached.
> 
> Thank you.
> 
> It looks quite binary to me.

To open a text file, you need text editor, or pager, it is some
program. 

To open database file, you need some program.

Recommended program for GUI is sqlitebrowser


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Georges Ko]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > You probably have several of them on your computer -- it's pretty hard
>   > to not, these days.  "find / -name '*.sqlite" will give you a list.
>
> I did that in /etc, /usr and /lib.  The only file it found was called
> README.sqlite.  It was a text file.
>
> Instead of insisting I don't need this small favor, would you please
> email me an sqlite3 database file, as an attachment?  Preferably
> a rather simple example.

How about this one in https://www.sqlitetutorial.net/sqlite-sample-database/ ?

https://www.sqlitetutorial.net/wp-content/uploads/2018/03/chinook.zip

Georges
-- 
 Georges Ko                     gko@gko.net                      2021-12-13

Re: sqlite3

[Alexandre Garreau]

[-- Attachment #1.1: Type: text/plain, Size: 1731 bytes --]

Le lundo, 13-a de decembro 2021, 4-a horo kaj 44:16 CET Richard Stallman a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > You probably have several of them on your computer -- it's pretty
>   > hard
>   > to not, these days.  "find / -name '*.sqlite" will give you a list.
> 
> I did that in /etc, /usr and /lib.  The only file it found was called
> README.sqlite.  It was a text file.

They should have suggested you to look in your home, as these are portable files created 
dynamically by programs, they have no place in places you mentioned.  Maybe they would 
in /var, but most of the time they’re used to contain user data, of unprivileged programs, 
hence only in home dirs

Here what I did: find  ~/.local -name *.sqlite

And I found /home/gal/.local/share/Mumble/Mumble/mumble.sqlite

which you should most likely have, since on my computer it still includes indirect traces of 
a past discussion with you (don’t worry: only the server that was used), I’m attaching it

> Instead of insisting I don't need this small favor, would you please
> email me an sqlite3 database file, as an attachment?  Preferably
> a rather simple example.

The sad thing is this file is small but also, imho, contains essentially useless data, so it’s 
not very easy to understand…  Now the sqlite databases I’ve always known of are in 
~/.mozilla but they contain pretty sensitive information (history, passwords, etc.) but since 
you sometimes use firefox you may already have it as well, and look at yours

[-- Attachment #1.2: Type: text/html, Size: 3544 bytes --]

[-- Attachment #2: mumble.sqlite --]
[-- Type: application/vnd.sqlite3, Size: 122880 bytes --]

Re: Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > They should have suggested you to look in your home,

That is a good suggestion.  Indeed, I found some there.
My home dir has many large subdirectories so I looked
in a smaller place I expected to find some.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: pipcet@gmail.com, larsi@gnus.org, eric@ericabrahamsen.net,
> 	cesar.mena@gmail.com, emacs-devel@gnu.org
> Date: Fri, 10 Dec 2021 23:07:31 -0500
> 
> That doesn't imply Emacs should not support sqlite3.  If you happen
> to store large amounts of persistent data, you might like
> saving persistent Emacs data that way.

Which is exactly what is being planned.

> But it does imply that sqlite3 should not be the default way to store
> persistent Emacs data.

Which is exactly what is being planned.  No one has any intention to
go over all the data Emacs stores between sessions and convert all of
them to sqlite3.  We only plan to use that where a data base is large,
and thus using plain text there is unwieldy.

Re: sqlite3

[Lars Ingebrigtsen]

I've added a `sqlite-mode-open-file' command, but perhaps it'd be nice
to make `C-x C-f foo.sqlite3 RET' call that automatically?
`auto-mode-alist' wouldn't be appropriate, I guess, since we don't want
to load the files into Emacs...  So is `file-name-handler-alist' still
the way to handle stuff like that?  I thought I vaguely remembered there
being a simpler thing to handle just `find-file*' and friends, but
grepping around, I can't see anything obvious.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 11 Dec 2021 10:59:16 +0100
> 
> I've added a `sqlite-mode-open-file' command, but perhaps it'd be nice
> to make `C-x C-f foo.sqlite3 RET' call that automatically?

Probably.  But is .sqlite3 indeed the extension to use?  I don't have
even a single .sqlite3 file on my system here, but gobs of *.sqlite
files.

> `auto-mode-alist' wouldn't be appropriate, I guess, since we don't want
> to load the files into Emacs...  So is `file-name-handler-alist' still
> the way to handle stuff like that?  I thought I vaguely remembered there
> being a simpler thing to handle just `find-file*' and friends, but
> grepping around, I can't see anything obvious.

AFAIK, all our auto-mode facilities assume we visit the file first.

I won't recommend using file-name handlers for this, as it could be
problematic for local files.  How about reading just the few first
bytes of the file and recognizing its magic signature instead?

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Probably.  But is .sqlite3 indeed the extension to use?  I don't have
> even a single .sqlite3 file on my system here, but gobs of *.sqlite
> files.

Ah, I misremembered.  I do have a handful of *.sqlite3 files here, but
the vast majority are indeed called *.sqlite.

> I won't recommend using file-name handlers for this, as it could be
> problematic for local files.  How about reading just the few first
> bytes of the file and recognizing its magic signature instead?

Well, a mechanism based on the name would be OK...  but I think Stefan M
had some suggestions, so I'll continue the discussion there.  😀

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Pip Cet]

On Sat, Dec 11, 2021 at 10:00 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> I've added a `sqlite-mode-open-file' command, but perhaps it'd be nice
> to make `C-x C-f foo.sqlite3 RET' call that automatically?

What about non-local files? Currently, IIUC, sqlite-mode just doesn't
work for them, handing all file names to the sqlite library which then
fails to open them.

I'm not volunteering, but maybe we could use the sqlite3 VFS interface
to back an sqlite3 database with an Emacs buffer (or several). That
would solve both the remote access issue and make C-x C-f behave more
like it does for other files.

The code that's been merged looks good to me, for what I think you're
trying to do.

Pip

Re: sqlite3

[Lars Ingebrigtsen]

Pip Cet <pipcet@gmail.com> writes:

> What about non-local files? Currently, IIUC, sqlite-mode just doesn't
> work for them, handing all file names to the sqlite library which then
> fails to open them.

It would certainly be nice to work with non-local files, but I think
it's a bit outside what we're going for here.  The point is to have
efficient local storage, and non-local sqlite files doesn't give us that.

> I'm not volunteering, but maybe we could use the sqlite3 VFS interface
> to back an sqlite3 database with an Emacs buffer (or several). That
> would solve both the remote access issue and make C-x C-f behave more
> like it does for other files.

It's be fun to have something like that, but I don't think there's that
much utility in it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Alexandre Garreau]

Le dimanĉo, 12-a de decembro 2021, 5-a horo kaj 26:00 CET Lars 
Ingebrigtsen a écrit :
> Pip Cet <pipcet@gmail.com> writes:
> 
> 
> 
> > What about non-local files? Currently, IIUC, sqlite-mode just doesn't
> > work for them, handing all file names to the sqlite library which then
> > fails to open them.
> 
> It would certainly be nice to work with non-local files, but I think
> it's a bit outside what we're going for here.  The point is to have
> efficient local storage, and non-local sqlite files doesn't give us
> that.
> 
> 
> 
> > I'm not volunteering, but maybe we could use the sqlite3 VFS interface
> > to back an sqlite3 database with an Emacs buffer (or several). That
> > would solve both the remote access issue and make C-x C-f behave more
> > like it does for other files.
> 
> It's be fun to have something like that, but I don't think there's that
> much utility in it.

What about the utility to simply *browse* sqlite files, so to provide a 
generic tool for edition, as you suggested?

Re: sqlite3

[Lars Ingebrigtsen]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> What about the utility to simply *browse* sqlite files, so to provide a 
> generic tool for edition, as you suggested?

You'd have to copy the file to the local file system first.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Stefan Monnier]

> I've added a `sqlite-mode-open-file' command, but perhaps it'd be nice
> to make `C-x C-f foo.sqlite3 RET' call that automatically?
> `auto-mode-alist' wouldn't be appropriate, I guess, since we don't want
> to load the files into Emacs...  So is `file-name-handler-alist' still
> the way to handle stuff like that?

I don't think we currently have a way to deal with that, but it would be
good to add one, because sqlite files aren't the one files which are
typically large and for which we want to have a specialized mode which
gives access to its content without loading it all upfront
(e.g. archive-mode could benefit from such a thing).


        Stefan

Re: sqlite3

[Juri Linkov]

>> I've added a `sqlite-mode-open-file' command, but perhaps it'd be nice
>> to make `C-x C-f foo.sqlite3 RET' call that automatically?
>> `auto-mode-alist' wouldn't be appropriate, I guess, since we don't want
>> to load the files into Emacs...  So is `file-name-handler-alist' still
>> the way to handle stuff like that?
>
> I don't think we currently have a way to deal with that, but it would be
> good to add one, because sqlite files aren't the one files which are
> typically large and for which we want to have a specialized mode which
> gives access to its content without loading it all upfront
> (e.g. archive-mode could benefit from such a thing).

Also maybe doc-view to not load the whole PDF to view one image per page.

Re: sqlite3

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> Also maybe doc-view to not load the whole PDF to view one image per page.

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't think we currently have a way to deal with that, but it would be
> good to add one, because sqlite files aren't the one files which are
> typically large and for which we want to have a specialized mode which
> gives access to its content without loading it all upfront
> (e.g. archive-mode could benefit from such a thing).

And for image files, too, I guess?  They're currently loaded in the
least efficient way imaginable -- first we put all the data into a
multibyte buffer (typically using >2x the number of bytes of the
original file), and then we re-read the via from image.c and display the
image.

So perhaps the way to make this workable, we should introduce a new
thing that short-circuits most of the logic in...  `find-file-noselect'?
Like a `special-file-alist' (with the same syntax as `auto-mode-alist')
that calls out to functions to interpret the file and present us with a
buffer in the correct mode.

The buffer returned shouldn't be visiting the file, I guess?  Because we
don't want to accidentally overwrite the file contents, since what we're
displaying in the buffer is just a view into the file, and not the file
itself.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Juri Linkov]

>> I don't think we currently have a way to deal with that, but it would be
>> good to add one, because sqlite files aren't the one files which are
>> typically large and for which we want to have a specialized mode which
>> gives access to its content without loading it all upfront
>> (e.g. archive-mode could benefit from such a thing).
> ...
> So perhaps the way to make this workable, we should introduce a new
> thing that short-circuits most of the logic in...  `find-file-noselect'?
> Like a `special-file-alist' (with the same syntax as `auto-mode-alist')
> that calls out to functions to interpret the file and present us with a
> buffer in the correct mode.

Actually, `find-file-noselect' already contains what is needed,
but for directories:

  (if (file-directory-p filename)
      (or (and find-file-run-dired
	       (run-hook-with-args-until-success
		'find-directory-functions
		(if find-file-visit-truename
		    (abbreviate-file-name (file-truename filename))
		  filename)))
	  (error "%s is a directory" filename))

This could be adapted to files the same way by adding a new similar hook
`find-file-functions'.

Re: sqlite3

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> Actually, `find-file-noselect' already contains what is needed,
> but for directories:
>
>   (if (file-directory-p filename)
>       (or (and find-file-run-dired
> 	       (run-hook-with-args-until-success
> 		'find-directory-functions
> 		(if find-file-visit-truename
> 		    (abbreviate-file-name (file-truename filename))
> 		  filename)))
> 	  (error "%s is a directory" filename))

Ah, excellent.

> This could be adapted to files the same way by adding a new similar hook
> `find-file-functions'.

I think we do want an auto-mode-alist type of variable to allow people
to tweak this on that level, though.  But that's just a detail.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Stefan Monnier]

> So perhaps the way to make this workable, we should introduce a new
> thing that short-circuits most of the logic in...  `find-file-noselect'?
> Like a `special-file-alist' (with the same syntax as `auto-mode-alist')
> that calls out to functions to interpret the file and present us with a
> buffer in the correct mode.

Something like that, yes.  Modulo visiting "literally", tho.  And maybe
it might be worthwhile to have a way for those special files to
"fallthrough" to the usual behavior in case something happens.

> The buffer returned shouldn't be visiting the file, I guess?

I think it should.

> Because we don't want to accidentally overwrite the file contents,
> since what we're displaying in the buffer is just a view into the
> file, and not the file itself.

That's the responsibility of the special handler.


        Stefan

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Something like that, yes.  Modulo visiting "literally", tho.  And maybe
> it might be worthwhile to have a way for those special files to
> "fallthrough" to the usual behavior in case something happens.

Right.

>> The buffer returned shouldn't be visiting the file, I guess?
>
> I think it should.
>
>> Because we don't want to accidentally overwrite the file contents,
>> since what we're displaying in the buffer is just a view into the
>> file, and not the file itself.
>
> That's the responsibility of the special handler.

But...  the proposed handler here is for `find-file-noselect' only, not
for writing files?  Or should that also set `write-file-functions' or
something?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Juri Linkov]

> I've added a `sqlite-mode-open-file' command, but perhaps it'd be nice
> to make `C-x C-f foo.sqlite3 RET' call that automatically?
> `auto-mode-alist' wouldn't be appropriate, I guess, since we don't want
> to load the files into Emacs...  So is `file-name-handler-alist' still
> the way to handle stuff like that?  I thought I vaguely remembered there
> being a simpler thing to handle just `find-file*' and friends, but
> grepping around, I can't see anything obvious.

`sqlite-mode-open-file' works great after replacing
"sqlite_schema" with "sqlite_master":
https://sqlite.org/schematab.html

Re: sqlite3

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> `sqlite-mode-open-file' works great after replacing
> "sqlite_schema" with "sqlite_master":
> https://sqlite.org/schematab.html

Ah, "sqlite_schema" is a newer name?  OK, updated now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > But it does imply that sqlite3 should not be the default way to store
  > > persistent Emacs data.

  > Which is exactly what is being planned.

I couldn't tell that from the messagse I saw, but I'm glad people
had already made this decision.

                                             No one has any intention to
  > go over all the data Emacs stores between sessions and convert all of
  > them to sqlite3.  We only plan to use that where a data base is large,
  > and thus using plain text there is unwieldy.

I think we should define a program interface for "save this data
persistently", hook it up to _both_ formats (textual Lisp and
sqlite3), then design each piece of code that saves persistent data
to use whichever representation is selected at the time.

Eventually we might convert all the existing places that make
persistent data so that they use this interface and can save the data
either way.




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2021-12-11 07:08]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > Which means what exactly?  We use a tool, Diff, to compare text
>   > files.  We use another tool, sqldiff, to compare databases.  Where's
>   > the crucial difference that justifies rejecting the _capability_ of
>   > working with a DB?
> 
> The crucial difference between text files and sqlite3 databases,
> unless I'm mistaken about the latter, is that text files are simple
> and transparent, but sqlite3 databases are not.

To see a text file on Unix-like system, one needs at least `cat'
command, or `less', `more' and similar, and better is to have text
editor. One could click on desktop system and open such file.

There must be always some program to open such file to see it
transparently, such is normally a text editor or mentioned programs.

For database it is same, one need to have a program to open it.

Files with extension *.sqlite can be opened by such programs in
console and under GUI, one example is below.

Description     : SQLite Database browser is a light GUI editor for SQLite databases, built on top of Qt
URL             : https://sqlitebrowser.org/

> Because text files are simple and transparent,

It is to specific case to tell what is simple and transparent. Many
text files are huge and complicated, they are transparent only to
person with the knowledge, the one who knows about it. Same is with
the databases.

> you can compare text files in many different ways: reading the two
> side by side, using diff, using wdiff, using M-x compare-windows,
> and there are probably more options.  And you can fix them manually,
> too.
> 
> Because sqlite3 databases are neither simple nor transparent, doing
> anything with them requires using specific sqlite3 tools.

They are both simple and very transparent, in fact, it is a relational
database, and thus it is more transparent then a text file.

If I just open my ~/.emacs/init.el and take any entry out of it such
as:

(setq bbdb-phone-style nil)

then I cannot see directly any relation to what that would mean,
unless I have profound knowledge of it. I would first have to have
knowledge of Emacs, such as to understand `setq' and that it is
followed by variable `bbdb-phone-style' and its value `nil'. Without
profound knowledge I am not able to understand what that would
possibly mean. I have to load the bbdb package and inspect the
variable with the command C-h v on it, to understand for what it
is. Thus as human being, in order to get transparency I have to work
by relations.

I would need to understand that "bbdb-phone-style is a variable
defined in ‘bbdb.el’." and that it relates to "Phone numbering plan
assumed by BBDB.  The value ’nanp refers to the North American
Numbering Plan.  The value nil refers to a free-style numbering plan."

By using key commands I have to follow relations within Emacs to
understand one line within my init.el

Compared to the database approach, one would have in the database list
of packages such as bbdb, and then variables and each variable would
belong to specific package. I would be able to click and jump to the
package and find other variables belonging to that package. And I
would not need to use Emacs to understand it, but other tools with
access to the database.

One cannot categorically say that sqlite files are not simple and not
transparent. 

> using database format has other advantages, and there are cases
> where they are important.  mainly when the text file is big or
> operating on it is slow.

the main advantage is the rigid structure of database types. it leads
to data accuracy, easy access to data, data integrity, feasibility for
future modification. it is definitely not only for big data. 

Main advantages of databases are various, though majority of them are
not related to size of data. 

Rather to the qualities of storage, redundancy, minimization of data
inconsistencies, easier data manipulation, and in concurrent or
network databases the concurrent access to it.

> Nonetheless I contend that, for a user who stores limited quantities
> of persistent Emacs data, the text format is better.

Size of data is not relevant when comparing text and databases.

Example with text:
------------------
I can write easily in the text file, or init.el following:

(setq baud-rate "Emacs") and it will look "simple" and
transparent. 

When it comes to evaluation, later in time, it will show the error,
and user in the future may not know what happened. Even though the
variable `baud-rate' cannot be set as string, the text file allows me
to write it as string.

Example with database is that such variable is already marked as
having value `integer' thus I cannot enter nothing else but integer.

It is up to database designer, thus programmers, to define properly
database tables.

> But it does imply that sqlite3 should not be the default way to store
> persistent Emacs data.

Then it could by default store `sqlite-persistent' data, marked as
such. Properties could be given to variables and functions. 

I was thinking and will still think of it to even include function
definitions into the database, not only variable values.

Some of my database backed types have their Elisp inside of the
database. The type handles itself, and not from program which is more
general and not specific. User can invent a new type and enter
specific Elisp for that type, while using the main program for data
management.


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 
  > > The crucial difference between text files and sqlite3 databases,
  > > unless I'm mistaken about the latter, is that text files are simple
  > > and transparent, but sqlite3 databases are not.

  > To see a text file on Unix-like system, one needs at least `cat'
  > command, or `less', `more' and similar, and better is to have text
  > editor. One could click on desktop system and open such file.

THat is true, but I think we're not talking about the same question.
I'm not talking abut the question of whether there is a command to
view the format, or whether the sqlite3 files have a documented
format.

I am talking about the kind of format that the file contains,
and the implications of that format.

Decades of experiences have taught me that files with internal pointers
are more likely to get badly corrupt that sequences of characters.
Also, when part of a character-sequence file gets corrupt,
the rest of it is usually still valid data and usable.
The pointer-based file tends to become totally incomprehensible.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

persistent data feature, or sqlite3 feature?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

People proposed the goal of giving Emacs the feature of persistent
objects.  For that goal, there is no particular reason to use a
general-purpose database package.

A general-purpose database package such as sqlite3 surely does not
come with a built-in representation for storing Lisp objects in
general.  Emacs would need to have a way to encode them as byte
sequences, and store those byte sequences in the database.
(This encoding might be `prin1' or it might be something else.)

It follows that sqlite3 databases containing persistent Lisp data
stored by Emacs will not in general be naturally interoperable with
sqlite3 databases made by other programs.  Other programs won't know
about this encoding.  If the other program is meant to be a general
tool, such as a database explorer, it could I suppose operate on Emacs
database files, but anything that really wants to understand its data
won't interoperate with Emacs.

It follows that there is no particular benefit from using a
general-purpose database system such as sqlite3 _for the purpose of holding
persistent Lisp data_.

People are also talking about wanting the feature of using Emacs to operate
on sqlite3 databases for various purposes.

That feature could be useful, but my point is _it's a different
feature_.

This discussion is switching between two different features,

1. Persistent objects.

2. Access to arbitrary sqlite3 databases.

and that makes it incoherent.

We could implement either feature.  We could implement both.
But one of them should not control the design of the other!

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature, or sqlite3 feature?

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2021-12-10 07:54]:
> A general-purpose database package such as sqlite3 surely does not
> come with a built-in representation for storing Lisp objects in
> general.  Emacs would need to have a way to encode them as byte
> sequences, and store those byte sequences in the database.
> (This encoding might be `prin1' or it might be something else.)

Not that it exists that I know, and not that it would be hard to make
specific Emacs Lisp type for SQLite

In my personal usage I am converting Emacs Lisp by using
prin1-to-string into the text type of a storage in database and
getting it back from there by using eval.

> It follows that sqlite3 databases containing persistent Lisp data
> stored by Emacs will not in general be naturally interoperable with
> sqlite3 databases made by other programs.

It does not follow. SQLite databases are manageable by other
programs. That data has been written as Lisp is not relevant. Also
text files with Emacs Lisp are not easily run by other programs.

> Other programs won't know about this encoding.  If the other program
> is meant to be a general tool, such as a database explorer, it could
> I suppose operate on Emacs database files, but anything that really
> wants to understand its data won't interoperate with Emacs.

Other programs don't understand Emacs Lisp, true. But other programs
for database management can easily operate on SQLite files created by
Emacs. Interoperability is thus built-in.

> It follows that there is no particular benefit from using a
> general-purpose database system such as sqlite3 _for the purpose of
> holding persistent Lisp data_.

I am doing it all the time, since years already. It is clear that you
do not see the benefit, this is because lack of experience with
databases. Your personal opinion cannot however dispute the fact that
storing information in the databases, especially into relational
databases brings many benefits to computer users, including to
programmers. 

Imagine Emacs profiles, instead of text files, user could press "N"
for new profile or "C" to copy a profile. Profile could contain
packages A, B, C, D, while other profile could contain only C, D
packages; including one profile could have quite different
configuration made for the work in other time zone, different city and
different language, with different input systems and different spell
checking; other profile could be quite different. Simple click could
switch between profiles with the certainty and reliability of data
stored. 

File system is a database. Though maybe not as sophisticated as
relational databases.

For example, we would have such a trouble finding all files on the
file system stored on Mon 14 Feb 2022 05:04:15 PM EAT and it would
take quite long time.

Using more sophisticated databases to store data can only be
helpful. Databases have indexes, and finding objects (files) stored
after Mon 14 Feb 2022 05:04:42 PM EAT would be a breeze.

Numerous intersections queries are available in relational databases.

> 1. Persistent objects.
> 
> 2. Access to arbitrary sqlite3 databases.
> 
> and that makes it incoherent.
> 
> We could implement either feature.  We could implement both.
> But one of them should not control the design of the other!

I don't think those are incoherent. I think that (2) access to
arbitrary sqlite3 databases provides (1) persistent objects by usage
of Emacs Lisp or Emacs C programming (even better); and C programming
should use arbitrary sqlite3 database with the difference that it
would take care of conformity of sexps stored.

By implementing (2) direct Emacs access to arbitraty sqlite3 database
(which already exists through Emacs packages) one achieves (1) to
create persistent objects or anything else one wish and wants.


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If we provide a feature for storing persistent Lisp data, I contend
that the default option should be to store it in a purely textual
file.

Many users will not store large amounts of data, and a textual file
would be much less risk-prone than any other method.

If we decide also to implement the feature of sqlite3 database access,
we could offer the option of storing persistent Lisp data in such a
database.  Users should be able to choose one storage format or the
other for each database.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> If we provide a feature for storing persistent Lisp data, I contend
> that the default option should be to store it in a purely textual
> file.
>
> Many users will not store large amounts of data, and a textual file
> would be much less risk-prone than any other method.

There will, of course, be large amounts of data, and that will be slow.
(People already do this with a number things, and the results are
predictably sluggish.)

> If we decide also to implement the feature of sqlite3 database access,
> we could offer the option of storing persistent Lisp data in such a
> database.  Users should be able to choose one storage format or the
> other for each database.

You'll get a slower and messier interface.  If you're (optionally)
storing the data in different data files, which ones has precedence?
Will all these backends have the same semantics wrt. concurrent updates
from different Emacs instances?  Will they offer the same introspection
opportunities (like "delete all stored data that refer to 'foo'"?)

Offering a number of backend storage options is just needlessly
complicating things, and we'll end up with a big non-performant mess, as
usual in Emacs.

For this feature to be attractive, there's a bunch of performance issues
and features that have to be present, and they will be in my
implementation, based on storing using sqlite3.  I won't be adding any
other backends for this.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: persistent data feature

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> You'll get a slower and messier interface.  If you're (optionally)
> storing the data in different data files, which ones has precedence?

I disagree with the messier part: that is purely an implementation
detail, and no such implementation exists yet.

But for uses such as storing a list of recently used Emoji, is a "slower
interface" really such an important downside?  Especially since the list
of recently used Emoji will be stored in a ring of some arbitrary (but
small) size.

I suspect that the use-cases which require storing a large amount of
data (such as the Gnus registry) will generally opt to use sqlite3
directly, instead of going through the "sticky value" system (which
involves printing Lisp data to a string, storing it as a string in the
database, and reading it back again).

> Will all these backends have the same semantics wrt. concurrent updates
> from different Emacs instances?

We could either make that undefined behaviour (the easy way out), or
design a locking mechanism for the text-based backend.  Perhaps the
existing file locking mechanism could be used for this.

> Will they offer the same introspection opportunities (like "delete all
> stored data that refer to 'foo'"?)

Sure, why not?  Since SQLite does not know about Lisp objects, I don't
see what advantage it can have WRT introspecting the contents of Lisp
objects.

> I won't be adding any other backends for this.

What if someone else does?  Are patches welcome?

Re: persistent data feature

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> But for uses such as storing a list of recently used Emoji, is a "slower
> interface" really such an important downside?  Especially since the list
> of recently used Emoji will be stored in a ring of some arbitrary (but
> small) size.

Yeah, most bits that are stored will be small.  But in aggregate, if
this feature is used as much as I think it will be, then the storage
will be large.

> I suspect that the use-cases which require storing a large amount of
> data (such as the Gnus registry) will generally opt to use sqlite3
> directly, instead of going through the "sticky value" system (which
> involves printing Lisp data to a string, storing it as a string in the
> database, and reading it back again).

Yup.

> What if someone else does?  Are patches welcome?

Sure, if they handle the semantics in a nice enough way.  (I.e., not
making it slower, have the same concurrency guarantees, allow the same
storage browsing, etc.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: persistent data feature

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 10 Dec 2021 07:23:29 +0100
> Cc: Richard Stallman <rms@gnu.org>, eric@ericabrahamsen.net,
>  cesar.mena@gmail.com, emacs-devel@gnu.org, Pip Cet <pipcet@gmail.com>,
>  eliz@gnu.org
> 
> > What if someone else does?  Are patches welcome?
> 
> Sure, if they handle the semantics in a nice enough way.  (I.e., not
> making it slower, have the same concurrency guarantees, allow the same
> storage browsing, etc.)

I agree.

Re: persistent data feature

[Stefan Monnier]

>> If we provide a feature for storing persistent Lisp data, I contend
>> that the default option should be to store it in a purely textual file.
>>
>> Many users will not store large amounts of data, and a textual file
>> would be much less risk-prone than any other method.
>
> There will, of course, be large amounts of data, and that will be slow.
> (People already do this with a number things, and the results are
> predictably sluggish.)

To help me understand this discussion, I think it would help me to have
examples of such large databases currently implemented as text files,
along with an idea of what "large" means in this context (how many MBs)
and where the "slow"ness manifests itself.


        Stefan

Re: persistent data feature

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> To help me understand this discussion, I think it would help me to have
> examples of such large databases currently implemented as text files,
> along with an idea of what "large" means in this context (how many MBs)
> and where the "slow"ness manifests itself.

Well, it has mostly limited itself because storing a lot of data in
Emacs is slow, so we don't do that.  But when storing data is both
convenient (you don't have to arrange for a file to be saved, etc) and
fast, we'll be storing more information.

But the current large data structures are in things like the Gnus
registry, which can grow arbitrarily large (and slow, unless you trim
it).  And my .ecompleterc is 4MB.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: persistent data feature

[Stefan Monnier]

> But the current large data structures are in things like the Gnus
> registry, which can grow arbitrarily large (and slow, unless you trim
> it).  And my .ecompleterc is 4MB.

4MB doesn't sound like it should be a problem, and I don't see much
benefit in having ecompleterc grow much further (by its nature it
contains a lot of "irrelevant" data and it's good to trim it every once
in a while for that reason).  Also having it too large would slow down
completion itself, so I suspect that storing it in a sql database would
not help.

The gnus-registry is a much more compelling example (especially since
there's no reason we should regularly scan it from start to end, we
only really ever need to look up specific entries, so the indexing of
the database should allow answering those queries efficiently without
having to read the whole DB from the disk (tho nowadays reading 100MB
from the disk is basically instantaneous anyway).

So, IIRC the only current example is gnus-registry.  That sounds good.


        Stefan

Re: persistent data feature

[Matt Armstrong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But the current large data structures are in things like the Gnus
>> registry, which can grow arbitrarily large (and slow, unless you trim
>> it).  And my .ecompleterc is 4MB.
>
> 4MB doesn't sound like it should be a problem, and I don't see much
> benefit in having ecompleterc grow much further (by its nature it
> contains a lot of "irrelevant" data and it's good to trim it every once
> in a while for that reason).  Also having it too large would slow down
> completion itself, so I suspect that storing it in a sql database would
> not help.
>
> The gnus-registry is a much more compelling example (especially since
> there's no reason we should regularly scan it from start to end, we
> only really ever need to look up specific entries, so the indexing of
> the database should allow answering those queries efficiently without
> having to read the whole DB from the disk (tho nowadays reading 100MB
> from the disk is basically instantaneous anyway).
>
> So, IIRC the only current example is gnus-registry.  That sounds good.

Another example is Org Roam (https://www.orgroam.com/), which uses
sqlite to manage its index of Org metadata.

Re: persistent data feature

[Stefan Monnier]

> Another example is Org Roam (https://www.orgroam.com/), which uses
> sqlite to manage its index of Org metadata.

Could you give details?

I don't know what is Org-roam (and the above web-page describes it in
terms that don't speak to me very much).  What does it store in the
database?  Why did it choose to use sqlite rather than plain text
(e.g. simple personal preference on the part of the author, or was
a text file tried at first but proved too inefficient, if so which
part)?  What kind of database size is common?


        Stefan

Re: persistent data feature

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Another example is Org Roam (https://www.orgroam.com/), which uses
>> sqlite to manage its index of Org metadata.
>
> Could you give details?
>
> I don't know what is Org-roam (and the above web-page describes it in
> terms that don't speak to me very much).  What does it store in the
> database?  Why did it choose to use sqlite rather than plain text
> (e.g. simple personal preference on the part of the author, or was
> a text file tried at first but proved too inefficient, if so which
> part)?  What kind of database size is common?

Org-roam stores Org headline metadata like headline title, todo-state,
scheduling info, properties, tags, links to other headings, etc.

AFAIU, they use sqlite for easier integration with third-party
tools. For example, links between headings can be visualised using
existing web-libraries (see
https://github.com/org-roam/org-roam-ui/blob/main/package.json)

As for the DB size, I am not sure. However, it should scale with the
number of headings. You may think of 10-100k headline DB records and a
multiple of that number for links between headings.

Best,
Ihor

Re: persistent data feature

[Matt Armstrong]

Ihor Radchenko <yantar92@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> Another example is Org Roam (https://www.orgroam.com/), which uses
>>> sqlite to manage its index of Org metadata.
>>
>> Could you give details?
>>
>> I don't know what is Org-roam (and the above web-page describes it in
>> terms that don't speak to me very much).  What does it store in the
>> database?  Why did it choose to use sqlite rather than plain text
>> (e.g. simple personal preference on the part of the author, or was
>> a text file tried at first but proved too inefficient, if so which
>> part)?  What kind of database size is common?
>
> Org-roam stores Org headline metadata like headline title, todo-state,
> scheduling info, properties, tags, links to other headings, etc.
>
> AFAIU, they use sqlite for easier integration with third-party
> tools. For example, links between headings can be visualised using
> existing web-libraries (see
> https://github.com/org-roam/org-roam-ui/blob/main/package.json)
>
> As for the DB size, I am not sure. However, it should scale with the
> number of headings. You may think of 10-100k headline DB records and a
> multiple of that number for links between headings.

The author gave a presentation on org-roam in EmacsConf 2020:
https://emacsconf.org/2020/talks/16/.  It is a Zettelkasten note taking
system for Emacs built on top of Org.

I believe that org-roam uses sqlite primarily for performance, as an
index over many the small org files the packages encourages users to
create.  I suspect that the use case is much like Gnus.

I have a memory (that I can't verify as fact now) of the early version
using awk to extract the needed info from the org files directly, but
this proved to be slow in practice.

I suspect that people have found ways to use the database for other
things is a secondary benefit.

Re: persistent data feature

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> There will, of course, be large amounts of data, and that will be
>> slow.  (People already do this with a number things, and the results
>> are predictably sluggish.)
>
> To help me understand this discussion, I think it would help me to
> have examples of such large databases currently implemented as text
> files, along with an idea of what "large" means in this context (how
> many MBs) and where the "slow"ness manifests itself.

I think the Gnus registry (info "(gnus) The Gnus Registry") is a good
example.  Mine is 3.6 MB but only because I've set
`gnus-registry-max-entries' to 10000 and `gnus-registry-track-extra' to
nil because I've basically only ever used it to be able to refer to an
article's parent.

There should be no reason to cap its number of entries or the number of
things it tracks.  By default it would also track sender, subject, and
recipients of messages.  I guess that would easily quadruple its size
for me.

I also think the Gnus registry is an example of where a DB with a
specific schema and queries would make sense instead of some persistent
variable.  There's no value in having that huge pile of hash-maps in
memory if I just want to, e.g., move the current mail into the folder
where its parent already resides.

Bye,
Tassilo

Re: persistent data feature

[Qiantan Hong]

> I think the Gnus registry (info "(gnus) The Gnus Registry") is a good
> example.  Mine is 3.6 MB but only because I've set
> `gnus-registry-max-entries' to 10000 and `gnus-registry-track-extra' to
> nil because I've basically only ever used it to be able to refer to an
> article's parent.
> 
> There should be no reason to cap its number of entries or the number of
> things it tracks.  By default it would also track sender, subject, and
> recipients of messages.  I guess that would easily quadruple its size
> for me.
I agree the number should not be restricted. I will try to implement
split bucketing in probably the next^2 revision of resist! then.

There’s still a technical question unanswered. If I’m to use multiple
files to store one kv store, is it better to let user supply a directory,
or let user supply a filename and we add suffixes?

> I also think the Gnus registry is an example of where a DB with a
> specific schema and queries would make sense instead of some persistent
> variable.  There's no value in having that huge pile of hash-maps in
> memory if I just want to, e.g., move the current mail into the folder
> where its parent already resides.
I think in your case, even if the db quadruples, it’s still < 20MB in
textual representation and probably much smaller when resided
in the heap. It’s still a pretty long way to go before memory-residence
become problematic.

Re: persistent data feature

[Stefan Monnier]

> I think in your case, even if the db quadruples, it’s still < 20MB in
> textual representation and probably much smaller when resided
> in the heap.

The textual representation can easily be smaller than the
in-heap representation.  A cons-cell typically uses up 16B nowadays, and
a string uses 32B in addition to the bytes of the actual characters.


        Stefan

Re: persistent data feature

[Tassilo Horn]

Qiantan Hong <qhong@mit.edu> writes:

>> There should be no reason to cap its number of entries or the number
>> of things it tracks.  By default it would also track sender, subject,
>> and recipients of messages.  I guess that would easily quadruple its
>> size for me.
> I agree the number should not be restricted. I will try to implement
> split bucketing in probably the next^2 revision of resist! then.
>
> There’s still a technical question unanswered. If I’m to use multiple
> files to store one kv store, is it better to let user supply a
> directory, or let user supply a filename and we add suffixes?

I'd use a directory but don't care too much.

>> I also think the Gnus registry is an example of where a DB with a
>> specific schema and queries would make sense instead of some
>> persistent variable.  There's no value in having that huge pile of
>> hash-maps in memory if I just want to, e.g., move the current mail
>> into the folder where its parent already resides.
> I think in your case, even if the db quadruples, it’s still < 20MB in
> textual representation and probably much smaller when resided in the
> heap. It’s still a pretty long way to go before memory-residence
> become problematic.

Sure, but as said I've restricted the registry to 10000 entries.  I
actually have 100000 mails in my spool and maybe the same number of
newsgroup articles.  Then we're around 300 MB and that's a size I'd
start worrying about.

And my main point is that only a tiny fraction of that data will ever be
used but which of it will be used cannot be known beforehand.  A typical
scenario for the Gnus registry is: given a single message, show the
complete thread.  Unless you have one store per thread, that's a query
which will inevitably require loading large parts if not all of the data
into memory.  Real databases are good in evaluating such queries quickly
without excessive memory usage.

Bye,
Tassilo

Re: persistent data feature

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> There will, of course, be large amounts of data, and that will be slow.
>> (People already do this with a number things, and the results are
>> predictably sluggish.)
>
> To help me understand this discussion, I think it would help me to have
> examples of such large databases currently implemented as text files,
> along with an idea of what "large" means in this context (how many MBs)
> and where the "slow"ness manifests itself.

I can think of two examples of large text databases: elfeed and recent
org-mode's caching (org-persist).

The slowness manifests itself on reading/printing(saving) the database
and from the requirement to load the whole database into memory for
queries.

We are talking about 300Mb for elfeed and up to 20-30Mb per object
(usually below 1Mb though) for org-persist with 100-200Mb total object
size.

Best,
Ihor

Re: persistent data feature

[Stefan Monnier]

>>> There will, of course, be large amounts of data, and that will be slow.
>>> (People already do this with a number things, and the results are
>>> predictably sluggish.)
>>
>> To help me understand this discussion, I think it would help me to have
>> examples of such large databases currently implemented as text files,
>> along with an idea of what "large" means in this context (how many MBs)
>> and where the "slow"ness manifests itself.
>
> I can think of two examples of large text databases: elfeed and recent
> org-mode's caching (org-persist).

I think I'm beginning to see a pattern: most of those uses (same for
org-roam) are for cache-like data, i.e. data that can be recreated from
other data.

I suspect the same holds for gnus-registry, but doesn't for ecomplete.

Cache-like data has the property that you don't need to version it, and
users should never need to manipulate it directly: if there's a problem
with it you should be able to just delete it and start over.

For such cases, there's no great benefit to use a "transparent" text
representation (but it's important to provide some way to rebuild the
cache).


        Stefan

Re: persistent data feature

[Qiantan Hong]

> On Dec 11, 2021, at 5:53 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>>> There will, of course, be large amounts of data, and that will be slow.
>>>> (People already do this with a number things, and the results are
>>>> predictably sluggish.)
>>> 
>>> To help me understand this discussion, I think it would help me to have
>>> examples of such large databases currently implemented as text files,
>>> along with an idea of what "large" means in this context (how many MBs)
>>> and where the "slow"ness manifests itself.
>> 
>> I can think of two examples of large text databases: elfeed and recent
>> org-mode's caching (org-persist).
> 
> I think I'm beginning to see a pattern: most of those uses (same for
> org-roam) are for cache-like data, i.e. data that can be recreated from
> other data.
Interesting.
> 
> I suspect the same holds for gnus-registry, but doesn't for ecomplete.
> 
> Cache-like data has the property that you don't need to version it, and
> users should never need to manipulate it directly: if there's a problem
> with it you should be able to just delete it and start over.
> 
> For such cases, there's no great benefit to use a "transparent" text
> representation (but it's important to provide some way to rebuild the
> cache).
I think a good fit to that would be incremental image.
Is it possible to have Emacs save/load part of the heap
(e.g. reachable from a root object) quickly, potentially
by directly mapping the memory in the load case?
Is pdumper useful for that purpose?

Re: persistent data feature

[Ihor Radchenko]

Qiantan Hong <qhong@mit.edu> writes:

> I think a good fit to that would be incremental image.

Note that not all data can be saved incrementally. Org mode's cache is
currently using AVL-tree with extra links between nodes - it is a
full-scale circular object. I am not sure if there is a universal way to
store and update such data incrementally.

> Is it possible to have Emacs save/load part of the heap
> (e.g. reachable from a root object) quickly, potentially
> by directly mapping the memory in the load case?
> Is pdumper useful for that purpose?

Anything that speeds up read/write would be helpful. From my experience,
reading large cache data for org-persist takes 2-3 seconds (though it is
still 2-3x faster compared to re-building cache from scratch).

Best,
Ihor

Re: persistent data feature

[Alexandre Garreau]

Le dimanĉo, 12-a de decembro 2021, 4-a horo kaj 48:02 CET Ihor Radchenko a 
écrit :
> Qiantan Hong <qhong@mit.edu> writes:
> 
> 
> 
> > I think a good fit to that would be incremental image.
> 
> Note that not all data can be saved incrementally. Org mode's cache is
> currently using AVL-tree with extra links between nodes - it is a
> full-scale circular object. I am not sure if there is a universal way to
> store and update such data incrementally.

Doesn’t elisp’s printer detects cycles and print #0, #1, etc. where 
needed?

Re: persistent data feature

[Qiantan Hong]

>>> I think a good fit to that would be incremental image.
>> 
>> Note that not all data can be saved incrementally. Org mode's cache is
>> currently using AVL-tree with extra links between nodes - it is a
>> full-scale circular object. I am not sure if there is a universal way to
>> store and update such data incrementally.
> 
> Doesn’t elisp’s printer detects cycles and print #0, #1, etc. where 
> needed?
I think there’s some confusion here.

My first “incremental” has a specific meaning in “incremental image”,
it means dumping part of the Lisp heap, compared to “full image”
which dump the whole heap. `prin1` can be viewed as a trivial
implementation of this, but usually this is done much more efficiently
by dumping out the memory representation directly.

I believe the “incremental” Ihor means is to store incremental changes
to a data structure/object graph, as what I did in resist!.el for hash
tables. This is indeed hard to do in general. Either we need to
modify the Elisp VM to add hook that detect changes, or we
need to do graph edit distance algorithm on different snapshots
of objects. It probably doesn’t worth the effort.

As what Ihor (I suppose) had in mind is incremental log operation,
prin1 is unrelated to this context because it prints out the whole object graph.

Re: persistent data feature

[Ihor Radchenko]

Qiantan Hong <qhong@mit.edu> writes:

>> Doesn’t elisp’s printer detects cycles and print #0, #1, etc. where 
>> needed?
> I think there’s some confusion here.
>
> My first “incremental” has a specific meaning in “incremental image”,
> it means dumping part of the Lisp heap, compared to “full image”
> which dump the whole heap. `prin1` can be viewed as a trivial
> implementation of this, but usually this is done much more efficiently
> by dumping out the memory representation directly.

Thanks for the clarification!

> I believe the “incremental” Ihor means is to store incremental changes
> to a data structure/object graph, as what I did in resist!.el for hash
> tables. This is indeed hard to do in general. Either we need to
> modify the Elisp VM to add hook that detect changes, or we
> need to do graph edit distance algorithm on different snapshots
> of objects. It probably doesn’t worth the effort.
>
> As what Ihor (I suppose) had in mind is incremental log operation,
> prin1 is unrelated to this context because it prints out the whole object graph.

Yep. You are right. I was thinking about your resist!.el and also about
textual representation of sqlite databases.

Also, I had several cases when prin1 is not accurate on large circular
objects generating un_read_able output or entering recursion deeper
than max-specpdl-size. Incremental logging could be easier in this
context.

Best,
Ihor

Re: persistent data feature

[Tomas Hlavaty]

On Sat 11 Dec 2021 at 09:30, Ihor Radchenko <yantar92@gmail.com> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> There will, of course, be large amounts of data, and that will be slow.
>>> (People already do this with a number things, and the results are
>>> predictably sluggish.)
>>
>> To help me understand this discussion, I think it would help me to have
>> examples of such large databases currently implemented as text files,
>> along with an idea of what "large" means in this context (how many MBs)
>> and where the "slow"ness manifests itself.
>
> I can think of two examples of large text databases: elfeed and recent
> org-mode's caching (org-persist).
>
> The slowness manifests itself on reading/printing(saving) the database
> and from the requirement to load the whole database into memory for
> queries.
>
> We are talking about 300Mb for elfeed and up to 20-30Mb per object
> (usually below 1Mb though) for org-persist with 100-200Mb total object
> size.

On the other hand, my notmuch (emacs email reader) database is 30GB and
lives externally out of emacs without issues.
Why link sqlite and not xapian?
Where does it stop?

Re: persistent data feature

[Alexandre Garreau]

Le sabato, 11-a de decembro 2021, 16-a horo kaj 38:10 CET Tomas Hlavaty a 
écrit :
> On the other hand, my notmuch (emacs email reader) database is 30GB and
> lives externally out of emacs without issues.

in what format is it? sexps?

> Why link sqlite and not xapian?
> Where does it stop?

Yeah and why not akonadi and then mysql, which it uses.

Akonadi is the reason why I wish so much to switch again from kmail to 
gnus, because it takes 100% of the cpu all the time, and yet makes kmail 
slows and extreeemely buggy most of the time (loosing mail, hanging 
forever to open some mails, some mails gets unreadable and undeletable, 
etc.).  The database is mostly corrupted but since it’s complex I have no 
fucking idea of how to fix it.  The only “database” I’m understanding 
anything to, here, is the list of mail, which is a bunch of recursive 
maildirs… but apparently these are not enough, and I regularely find myself 
to delete all akonadi data and loose a lot of metadata, thinking that I 
may one day reimport all the maildirs into gnus, and regenerate a lot of 
nov databases during one long day…

…but I still don’t switch, because the way kmail stores mail is of course 
at least partially incompatible with gnus, because all these databases are 
so complex…

And I think that a fear of getting a such thing into emacs, under the 
excuse that “for the developer, it works” and that “it brings more 
features” is what makes people afraid from something even as minimal as 
sqlite… for emacs is one of the very few software to be minimal and 
transparent enough to have a better situation than sqlite

Re: persistent data feature

[Tomas Hlavaty]

On Sat 11 Dec 2021 at 19:38, Alexandre Garreau <galex-713@galex-713.eu> wrote:
> Le sabato, 11-a de decembro 2021, 16-a horo kaj 38:10 CET Tomas Hlavaty a 
> écrit :
>> On the other hand, my notmuch (emacs email reader) database is 30GB and
>> lives externally out of emacs without issues.
>
> in what format is it? sexps?

https://getting-started-with-xapian.readthedocs.io/en/latest/concepts/indexing/databases.html

>> Why link sqlite and not xapian?
>> Where does it stop?
>
> Yeah and why not akonadi and then mysql, which it uses.

yeah

> Akonadi is the reason why I wish so much to switch again from kmail to 
> gnus, because

Try notmuch;-)

> …but I still don’t switch, because the way kmail stores mail is of course 
> at least partially incompatible with gnus, because all these databases are 
> so complex…

With notmuch, Emails are stored in maildir.  The notmuch database
contains the search index and user tags.  You can always migrate to
whatever supports maildir.  Or reindex your emails.  Tags can be
exported/imported as simple text.

> And I think that a fear of getting a such thing into emacs, under the 
> excuse that “for the developer, it works” and that “it brings more 
> features” is what makes people afraid from something even as minimal as 
> sqlite… for emacs is one of the very few software to be minimal and 
> transparent enough to have a better situation than sqlite

Databases are complex.
Suitability of sqlite depends on the use-case.
It is a leaky abstraction.
For example, will users have to deal with schema upgrades?
Will they have to deal with sql?
Will they have to deal with sqlite specific issues?
Once sqlite is linked into emacs, it will eventually infest
almost everything and emacs will be stuck with it forever.

Also I find the trend of linking everything into the emacs process
concerning:
$ ldd .emacs-27.2 | wc -l
107

Strangely enough, sqlite is already there:
$ ldd emacs-27.2 | grep sqlite
libsqlite3.so.0

Re: persistent data feature

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Sat, 11 Dec 2021 20:36:29 +0100
> 
> Also I find the trend of linking everything into the emacs process
> concerning:

What's the alternative? to reinvent all the wheels in Emacs's own
code? to use external programs? something else?  Rejecting a trend for
which there are no better alternatives is not smart.

> $ ldd .emacs-27.2 | wc -l
> 107

Did you try comparing this with other similar applications?

Re: persistent data feature

[Alexandre Garreau]

Le sabato, 11-a de decembro 2021 21-a horo kaj 15:03 CET, vous avez écrit 
:
> > From: Tomas Hlavaty <tom@logand.com>
> > Date: Sat, 11 Dec 2021 20:36:29 +0100
> > 
> > Also I find the trend of linking everything into the emacs process
> 
> > concerning:
> What's the alternative? to reinvent all the wheels in Emacs's own
> code? to use external programs? something else?  Rejecting a trend for
> which there are no better alternatives is not smart.
> 
> 
> 
> > $ ldd .emacs-27.2 | wc -l
> > 107
> 
> Did you try comparing this with other similar applications?

under debian stable (please note I have emacs 27.1, and it doesn’t include 
sqlite, for some reason):

$ ldd /usr/bin/nano | wc -l
6
$ ldd /usr/bin/zile | wc -l
9
$ ldd /usr/bin/vim | wc -l
92
$ ldd /usr/bin/vim.tiny | wc -l
9
$ ldd /usr/bin/vim.nox | wc -l
22
$ ldd /usr/bin/vim.gtk3 | wc -l
92
$ ldd /usr/bin/pluma | wc -l
83
$ ldd /usr/bin/gedit | wc -l
85
$ ldd /usr/bin/kwrite | wc -l
107
$ ldd /usr/bin/kate | wc -l
109
$ ldd /usr/bin/codeblocks | wc -l
94

So indeed.

I wanted to reinstall geany, gnome-builder and qtcreator to compare with 
standard IDEs, but all three triggered to install clang…

I think, given not all users of emacs are programmers, using it as an IDE… 
and not all emacsian programmers using its IDE capabilities, it shouldn’t 
be compared to qtcreator or codeblocks, but rather to vim (with which it 
is in legendary competition).  However, gtk vim is not so light in 
comparison (it’s even more heavy than gedit!)

While GNU Zile, the stripped down emacs (with only some lisp and lua 
support, but without packages) is equal or simpler than vim, in terms of 
dependencies.  We could also compare with mit-scheme (6 libraries), 
because it provides Edwin, a scheme emacs.

Re: persistent data feature

[Alexandre Garreau]

Le sabato, 11-a de decembro 2021, 21-a horo kaj 15:03 CET Eli Zaretskii a 
écrit :
> > From: Tomas Hlavaty <tom@logand.com>
> > Date: Sat, 11 Dec 2021 20:36:29 +0100
> > 
> > Also I find the trend of linking everything into the emacs process
> 
> > concerning:
> What's the alternative? to reinvent all the wheels in Emacs's own
> code? to use external programs? something else?  Rejecting a trend for
> which there are no better alternatives is not smart.
> 
> > $ ldd .emacs-27.2 | wc -l
> > 107
> 
> Did you try comparing this with other similar applications?

Actually ldd shows the recursive dependencies (it runs the program) rather 
than the direct dependencies, it would be more fair to use readelf for 
that purpose (and readelf also doesn’t requires you to run the program, so 
it’s a safe habit that prevents security issues):

gal@galex-713:~$ readelf -d /usr/bin/emacs | grep -c NEEDED
44
gal@galex-713:~$ readelf -d /usr/bin/nano | grep -c NEEDED
3
gal@galex-713:~$ readelf -d /usr/bin/nano | grep -c NEEDED
3
gal@galex-713:~$ readelf -d /usr/bin/zile | grep -c NEEDED
5
gal@galex-713:~$ readelf -d /usr/bin/vim.tiny | grep -c NEEDED
4
gal@galex-713:~$ readelf -d /usr/bin/vim.nox | grep -c NEEDED
13
gal@galex-713:~$ readelf -d /usr/bin/vim.gtk3 | grep -c NEEDED
27
gal@galex-713:~$ readelf -d /usr/bin/pluma | grep -c NEEDED
18
gal@galex-713:~$ readelf -d /usr/bin/gedit | grep -c NEEDED
5
gal@galex-713:~$ readelf -d /usr/bin/kwrite | grep -c NEEDED
16
gal@galex-713:~$ readelf -d /usr/bin/kate | grep -c NEEDED
27
gal@galex-713:~$ readelf -d /usr/bin/codeblocks | grep -c NEEDED
15

We can however observe that, by direct usage, emacs is more demanding than 
other similar software.  Even though, in practice, it is pretty similare 
in memory footprint since, indirectly, it requires pretty much the same 
(essentially graphical) libraries as other similar graphical software.

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Also I find the trend of linking everything into the emacs process
  > > concerning:

  > What's the alternative? to reinvent all the wheels in Emacs's own
  > code? to use external programs? something else?  Rejecting a trend for
  > which there are no better alternatives is not smart.

For any given facility, I see these alternatives to consider:
(1) Link it in statically.
(2) Use an external program in a subprocess.
(3) Use a dynamically linked module.
(4) Write it in Lisp.
(5) Decline to support it.

Each of them may be the best choice in some cases.

If we conclude that we are using (1) too readily,
that wouldn't imply we should reject it entirely henceforth,
but would lead us to be more ready to choose the others.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: tom@logand.com, galex-713@galex-713.eu, emacs-devel@gnu.org
> Date: Sun, 12 Dec 2021 22:42:41 -0500
> 
>   > > Also I find the trend of linking everything into the emacs process
>   > > concerning:
> 
>   > What's the alternative? to reinvent all the wheels in Emacs's own
>   > code? to use external programs? something else?  Rejecting a trend for
>   > which there are no better alternatives is not smart.
> 
> For any given facility, I see these alternatives to consider:
> (1) Link it in statically.
> (2) Use an external program in a subprocess.
> (3) Use a dynamically linked module.
> (4) Write it in Lisp.
> (5) Decline to support it.

I think (1) and (3) are the same, for the purpose of this sub-thread,
since the issue raised was how many external libraries we depend on.
(2) is only viable if the amount of stuff flowing back and forth
between Emacs and the program is relatively small, otherwise it's just
bad design and worse performance.  (4) is not relevant because that's
normally the first solution we try, and if we decided that Lisp cannot
handle it well enough, or not at all (like with drawing images on the
Emacs display), we are way past that when we get to the decision
point.  Declining to support a useful capability when there's a
reasonable way of supporting it is silly.

So we are left with (1) and (3), which are usually indistinguishable
(except in weird cases like libgcc).

> If we conclude that we are using (1) too readily,
> that wouldn't imply we should reject it entirely henceforth,
> but would lead us to be more ready to choose the others.

Once again, (1) and (3) are the same.  I think nowadays people prefer
shared libraries for reasons of system maintenance, and its infeasible
to try to force users or distros to use only static linking (nor do I
see any reasons for that, either technical or political, as long as we
are talking about Free Software anyway).

Re: persistent data feature

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Once again, (1) and (3) are the same.  I think nowadays people prefer
> shared libraries for reasons of system maintenance, and its infeasible
> to try to force users or distros to use only static linking (nor do I
> see any reasons for that, either technical or political, as long as we
> are talking about Free Software anyway).

I think RMS may have meant the dynamic module support which exists in
Emacs.

Re: persistent data feature

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: rms@gnu.org,  galex-713@galex-713.eu,  tom@logand.com,  emacs-devel@gnu.org
> Date: Mon, 13 Dec 2021 21:04:18 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Once again, (1) and (3) are the same.  I think nowadays people prefer
> > shared libraries for reasons of system maintenance, and its infeasible
> > to try to force users or distros to use only static linking (nor do I
> > see any reasons for that, either technical or political, as long as we
> > are talking about Free Software anyway).
> 
> I think RMS may have meant the dynamic module support which exists in
> Emacs.

In that case, it's a misunderstanding and a tangent in the context of
this discussion.  We don't use sqlite3 via a dynamic module, and I
personally think it would be wrong for Emacs to provide _any_ core
feature through modules -- doing so just adds complexity for no good
reason.

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think RMS may have meant the dynamic module support which exists in
  > > Emacs.

Yes, that is what I had in mind.

  > In that case, it's a misunderstanding and a tangent in the context of
  > this discussion.  We don't use sqlite3 via a dynamic module, and I
  > personally think it would be wrong for Emacs to provide _any_ core
  > feature through modules -- doing so just adds complexity for no good
  > reason.

If we really want a feature to be in the core, then we don't want to
make it a dynamic module.

However, it can happen that we want to support a feature with C code
but we don't want it to add to the complexity of core Emacs.  Making
it a module could be a way to support it but keep it out of the core.

That would also reduce the number of libraries that Emacs depends on
-- because if a module depends on the library, that means Emacs itself
does not have to depend on it.




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> If we really want a feature to be in the core, then we don't want to
> make it a dynamic module.
>
> However, it can happen that we want to support a feature with C code
> but we don't want it to add to the complexity of core Emacs.  Making
> it a module could be a way to support it but keep it out of the core.
>
> That would also reduce the number of libraries that Emacs depends on
> -- because if a module depends on the library, that means Emacs itself
> does not have to depend on it.

FWIW, Lars' SQLite support was adapted from an existing free dynamic
module.

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > FWIW, Lars' SQLite support was adapted from an existing free dynamic
  > module.

A priori, I don't see a reason why SQLite support should be in the core.
Or a reason why it should NOT be in the core.
That's just a matter of practical benefits and inconveniences,
and I don't have an opinion about that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: luangruo@yahoo.com, galex-713@galex-713.eu, tom@logand.com,
> 	emacs-devel@gnu.org
> Date: Mon, 13 Dec 2021 23:13:25 -0500
> 
> If we really want a feature to be in the core, then we don't want to
> make it a dynamic module.
> 
> However, it can happen that we want to support a feature with C code
> but we don't want it to add to the complexity of core Emacs.

I cannot imagine how such a situation could arise, really.  And in all
my long involvement with Emacs development it didn't happen yet.
Which suggests that it's probably a theoretical possibility, but a
real situation we need to deal with.  And the module interface was
designed to not require _any_ support from Emacs to be able to add a
module, so I'm quite sure this won't happen.

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If we really want a feature to be in the core, then we don't want to
  > > make it a dynamic module.
  > > 
  > > However, it can happen that we want to support a feature with C code
  > > but we don't want it to add to the complexity of core Emacs.

  > I cannot imagine how such a situation could arise, really.  And in all
  > my long involvement with Emacs development it didn't happen yet.

I won't argue that any particular one of those libaries should not be
used by Emacs core.  But, given how the number of dynamic libraries
required by Emacs core has grown, that suggests maybe it would be better
of SOME of them were used via dynamic modules.

I mean to say only that it is something to consider.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021, 6-a horo kaj 16:24 CET Richard 
Stallman a écrit :
>   > > If we really want a feature to be in the core, then we don't want
>   > > to
>   > > make it a dynamic module.
>   > > 
>   > > However, it can happen that we want to support a feature with C
>   > > code
>   > > but we don't want it to add to the complexity of core Emacs.
>   > 
>   > I cannot imagine how such a situation could arise, really.  And in
>   > all
>   > my long involvement with Emacs development it didn't happen yet.
> 
> I won't argue that any particular one of those libaries should not be
> used by Emacs core.  But, given how the number of dynamic libraries
> required by Emacs core has grown, that suggests maybe it would be better
> of SOME of them were used via dynamic modules.
> 
> I mean to say only that it is something to consider.

That would be cool.  That would make “canonical” extensions more akin to 
users’ one, and would reduce the gap between user hacking and standard 
development (akin to the “absence of keywords” of lisp, and to the fact 
users’ extensions can look like core features), encourage hacking, and 
equality of emacs’ hackers.

Plus I like the idea you can install libraries separatly from the end 
software, before or after, and remove them at wish.

Re: persistent data feature

[Stefan Monnier]

> That would be cool.  That would make “canonical” extensions more akin to
> users’ one, and would reduce the gap between user hacking and standard
> development (akin to the “absence of keywords” of lisp, and to the fact
> users’ extensions can look like core features), encourage hacking, and
> equality of emacs’ hackers.
>
> Plus I like the idea you can install libraries separatly from the end
> software, before or after, and remove them at wish.

That was my original intention when I pushed for the addition of
loadable modules.  I suspect we need to significantly improve our
loadable module support before it can become reality.

In the mean time, maybe we could try and make the non-w32 world use
a similar approach to the one used in the w32 code, so support for
sqlite/gnutls/whathaveyou is compiled-in but Emacs still works
properly if the .so is missing at run-time.


        Stefan

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In the mean time, maybe we could try and make the non-w32 world use
  > a similar approach to the one used in the w32 code, so support for
  > sqlite/gnutls/whathaveyou is compiled-in but Emacs still works
  > properly if the .so is missing at run-time.

Is that something we coulo do by changing Emacs?
Or would the change have to be made at lower levels in the system?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[Stefan Monnier]

>   > In the mean time, maybe we could try and make the non-w32 world use
>   > a similar approach to the one used in the w32 code, so support for
>   > sqlite/gnutls/whathaveyou is compiled-in but Emacs still works
>   > properly if the .so is missing at run-time.
> Is that something we coulo do by changing Emacs?

I think so, yes.

> Or would the change have to be made at lower levels in the system?

I hope that's not needed, no.


        Stefan

Re: persistent data feature

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021, 17-a horo kaj 44:09 CET Stefan Monnier 
a écrit :
> > That would be cool.  That would make “canonical” extensions more akin
> > to users’ one, and would reduce the gap between user hacking and
> > standard development (akin to the “absence of keywords” of lisp, and
> > to the fact users’ extensions can look like core features), encourage
> > hacking, and equality of emacs’ hackers.
> > 
> > Plus I like the idea you can install libraries separatly from the end
> > software, before or after, and remove them at wish.
> 
> That was my original intention when I pushed for the addition of
> loadable modules.  I suspect we need to significantly improve our
> loadable module support before it can become reality.
> 
> In the mean time, maybe we could try and make the non-w32 world use
> a similar approach to the one used in the w32 code, so support for
> sqlite/gnutls/whathaveyou is compiled-in but Emacs still works
> properly if the .so is missing at run-time.

oh yes indeed it would be very cool! plus doesn’t gnu and especially emacs 
has some sort of policy saying we shouldn’t support some feature on 
proprietary systems and not on free ones? granted here it’s minor (because 
we already have fully working package managers on free systems while 
windows more need this as it doesn’t, but still, that’s a robustness 
characteristic)

Re: persistent data feature

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>, rms@gnu.org, luangruo@yahoo.com, tom@logand.com, emacs-devel@gnu.org
> Date: Thu, 16 Dec 2021 06:10:30 +0100
> 
> Le merkredo, 15-a de decembro 2021, 17-a horo kaj 44:09 CET Stefan Monnier 
> a écrit :
> > > That would be cool.  That would make “canonical” extensions more akin
> > > to users’ one, and would reduce the gap between user hacking and
> > > standard development (akin to the “absence of keywords” of lisp, and
> > > to the fact users’ extensions can look like core features), encourage
> > > hacking, and equality of emacs’ hackers.
> > > 
> > > Plus I like the idea you can install libraries separatly from the end
> > > software, before or after, and remove them at wish.
> > 
> > That was my original intention when I pushed for the addition of
> > loadable modules.  I suspect we need to significantly improve our
> > loadable module support before it can become reality.
> > 
> > In the mean time, maybe we could try and make the non-w32 world use
> > a similar approach to the one used in the w32 code, so support for
> > sqlite/gnutls/whathaveyou is compiled-in but Emacs still works
> > properly if the .so is missing at run-time.
> 
> oh yes indeed it would be very cool! plus doesn’t gnu and especially emacs 
> has some sort of policy saying we shouldn’t support some feature on 
> proprietary systems and not on free ones? granted here it’s minor (because 
> we already have fully working package managers on free systems while 
> windows more need this as it doesn’t, but still, that’s a robustness 
> characteristic)

I don't object to make the shared-library loading be dynamic, of
course.  It should be easy to do so by utilizing the existing
WINDOWSNT code with minimal changes.

However, please note that doing this on Posix systems is much less
useful than on MS-Windows, for 2 main reasons:

  . unlike MS-Windows, on Posix systems Emacs will start with no
    problems when an optional shared library is missing; it is only
    when some function from the library is called that you'll have
    trouble (Emacs will segfault);

  . unlike on MS-Windows, users of Posix systems who don't build their
    own Emacs generally install a prepackaged binary distribution, and
    that always makes sure the dependencies, in the form of those
    optional shared libraries, are also installed

So the gains are much smaller, and basically, AFAICT, are only of the
aesthetic nature, because the Emacs memory footprint is unaffected by
the optional library either way, unless the user uses stuff like
LD_PRELOAD.

This is why we didn't do on Unix what we do for w32.  It could be that
just arranging for the relevant FOO-available-p function to return nil
on Posix systems would be enough to prevent the crashes when the
library is missing, which is currently the only real, albeit rare,
problem with this.

That said, if someone submits patches for using the dynamic loading on
Unix and GNU systems, I won't object, of course.

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > . unlike MS-Windows, on Posix systems Emacs will start with no
    >   problems when an optional shared library is missing; it is only
    >   when some function from the library is called that you'll have
    >   trouble (Emacs will segfault);

This suggests that if Emacs can find out whether the shared library
is missing, it could signal an error before calling any function
in that library, and we would get exactly the behavior that is sought.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2021-12-14 07:14]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > I think RMS may have meant the dynamic module support which exists in
>   > > Emacs.
> 
> Yes, that is what I had in mind.
> 
>   > In that case, it's a misunderstanding and a tangent in the context of
>   > this discussion.  We don't use sqlite3 via a dynamic module, and I
>   > personally think it would be wrong for Emacs to provide _any_ core
>   > feature through modules -- doing so just adds complexity for no good
>   > reason.
> 
> If we really want a feature to be in the core, then we don't want to
> make it a dynamic module.

Dynamic module already exists, it is here:

pekingduck/emacs-sqlite3-api: SQLite3 API for GNU Emacs 25+
https://github.com/pekingduck/emacs-sqlite3-api



Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: persistent data feature

[Jean Louis]

* Tomas Hlavaty <tom@logand.com> [2021-12-11 22:54]:
> > And I think that a fear of getting a such thing into emacs, under the 
> > excuse that “for the developer, it works” and that “it brings more 
> > features” is what makes people afraid from something even as minimal as 
> > sqlite… for emacs is one of the very few software to be minimal and 
> > transparent enough to have a better situation than sqlite
> 
> Databases are complex.

They are not more complex then the text files of which user has lack
of knowledge what are they about. If user has knowledge of programming
language A, but not programming language B, that B will appear
"complex". It is individual perception.

> For example, will users have to deal with schema upgrades?

They need not think of the background. Just as you use M-x customize
in quite nice way, one can use database from User Interface.

> Will they have to deal with sql?

That depends of programmers who make the User Interface. Me I am using
least SQL in the database management, and I can add new column to
database table by using "A" and then say that column shall contain
integers or dates only and I would call the column "Date Created".

My interface for database editing is similar to below. I can edit,
update, or nullify some entries. One could call all of the table
columns "variables", and in this example there is table "variables"
where one table column is called "variables_name" with visible
representation being "Variable", having "url" as its value:

                             ID   84
                           Note   nil
                       Variable   "url"
                          Value   "http://www.example.com"
                       Template   "Nice 1"
                       Language   nil
                           Area   nil
                           Type   "Standard"

If there is a way to list all customizable variables, I could as well
insert them into the database under the Emacs type, and under specific
Emacs profile. With network databases I could have my profile fully on
network accessible only by my username/password, accessible through
Emacs on any side of the world. That is just hypothetical possibility,
though not so far from realization. I am using database backed
variables since 20 years already for purposes of Website Revision
System. I could as well store Emacs Lisp variables, all those from my
init.el and I seriously think about temporarily re-defining `setq' to
function that will slurp the variable into the database, and then
forget about the init.el

> Will they have to deal with sqlite specific issues?

Users need not even know that there is sqlite backend. One could use
multiple backends while users having always same User Interface.

> Once sqlite is linked into emacs, it will eventually infest
> almost everything and emacs will be stuck with it forever.

Maybe "infest" is meant to mean something bad, though I think of it as
positive. Emacs has to have GDBM built-in since decades, and I still
wonder why it did not have. 

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Why link sqlite and not xapian?

As a general point, we don't have an obligation to make Emacs support
all free database packages that anyone might want to use.  If
supporting sqlite3 "does the job", we don't need to do that job again
with another package.

(At present we know of a grave flaw in sqlite3, and we don't know
of an easy fix.  See the other message.)

However, if xapian is superior, we might want to support xapian
_instead of_ sqlite3.

I know nothing about xapian, though I surmise it is a database
package.  Thus, I have no opinion about that choice.  Probably some of
us know pertinent things and could post about whether xapian has
advantages.

One drawback of sqlite3 is that it dynamically loads extensions.  If
xapian does NOT dynamically load extensions -- or at least offers a
mode of use which does not dynamically load extensions -- that is a
significant advantage, for us.

If xapian is GPL-covered, promoting it is desirable.

Are there any other alternatives that might be preferable to sqlite3?


We should have taken _more time_ to consider what we were going to do
in this area, before doing anything.  There was no hurry so we should
not have rushed this.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: persistent data feature

[LdBeth]

On Mon, 13 Dec 2021 11:42:15 +0800,
Richard Stallman wrote:
> 
> However, if xapian is superior, we might want to support xapian
> _instead of_ sqlite3.
> 
> I know nothing about xapian, though I surmise it is a database
> package.  Thus, I have no opinion about that choice.  Probably some of
> us know pertinent things and could post about whether xapian has

Xapian is a search engine that can build a database out of texts and
performs a quick search through it. It is no intended to be used as a
"general" database like sqlite3.

> If xapian is GPL-covered, promoting it is desirable.

Its homepage, https://xapian.org, claims it is released under GPLv2.

--
LDB

Re: persistent data feature

[Po Lu]

LdBeth <andpuke@foxmail.com> writes:

> Xapian is a search engine that can build a database out of texts and
> performs a quick search through it. It is no intended to be used as a
> "general" database like sqlite3.

> Its homepage, https://xapian.org, claims it is released under GPLv2.

(Potential tangent:) is that GPLv2 or is that GPLv2+?

Re: persistent data feature

[LdBeth]

On Mon, 13 Dec 2021 17:32:39 +0800,
Po Lu wrote:
> 
> LdBeth <andpuke@foxmail.com> writes:
> 
> > Xapian is a search engine that can build a database out of texts and
> > performs a quick search through it. It is no intended to be used as a
> > "general" database like sqlite3.
> 
> > Its homepage, https://xapian.org, claims it is released under GPLv2.
> 
> (Potential tangent:) is that GPLv2 or is that GPLv2+?

Sorry for didn't make that clear, it is GPLv2+, so "any later version"
can be applied.

--
LDB

Re: persistent data feature

[Yuan Fu]

> 
> On the other hand, my notmuch (emacs email reader) database is 30GB and
> lives externally out of emacs without issues.
> Why link sqlite and not xapian?
> Where does it stop?

FWIW I have a (extremely basic) xapian module at 

https://github.com/casouri/xeft

I called it xeft-module, but I probably should name it xapian-lite-module so people know it is not tied to the elisp program xeft.el that uses the xapian module.

Yuan

Re: persistent data feature

[Lars Ingebrigtsen]

Yuan Fu <casouri@gmail.com> writes:

> FWIW I have a (extremely basic) xapian module at 
>
> https://github.com/casouri/xeft

Looks very cool.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: persistent data feature

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> If we provide a feature for storing persistent Lisp data, I contend
>>> that the default option should be to store it in a purely textual file.
>>>
>>> Many users will not store large amounts of data, and a textual file
>>> would be much less risk-prone than any other method.
>>
>> There will, of course, be large amounts of data, and that will be slow.
>> (People already do this with a number things, and the results are
>> predictably sluggish.)
>
> To help me understand this discussion, I think it would help me to have
> examples of such large databases currently implemented as text files,
> along with an idea of what "large" means in this context (how many MBs)
> and where the "slow"ness manifests itself.

FWIW I would also provide a sqlite backend for EBDB (contact
management). Not because the eieio-persistent text files are large (my
3200 contacts take up 1.5M), but because I'd like to let users make use
of that data in other, non-Emacs applications. I would also be looking
at a usage pattern that does not slurp the whole database into memory at
startup. Probably it would build some minimal caches at startup, and
then access the database for reads and writes as necessary.

Re: persistent data feature

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Many users will not store large amounts of data, and a textual file
  > > would be much less risk-prone than any other method.

  > There will, of course, be large amounts of data, and that will be slow.

I can't understand that sentence.  You seem to be claiming that every
user will have a large amount of data to save.  I don't think I will.

  > > If we decide also to implement the feature of sqlite3 database access,
  > > we could offer the option of storing persistent Lisp data in such a
  > > database.  Users should be able to choose one storage format or the
  > > other for each database.

  > You'll get a slower and messier interface.

I don't think it will be either slower or messier
due to offering the choice.  It will need one global variable
to specify which method of storage.  Each method ought to be
as fast as if it were the only method.

                                                If you're (optionally)
  > storing the data in different data files, which ones has precedence?

What does "precedence" mean here?  I think I have lost you.

  > Offering a number of backend storage options is just needlessly
  > complicating things, and we'll end up with a big non-performant mess, as
  > usual in Emacs.

That is too general to be convincing.

  > For this feature to be attractive, there's a bunch of performance issues
  > and features that have to be present,

Without details, that seems to be handwaving.  All I can say is,
you have not demonstrated your claim.

                                          and they will be in my
  > implementation, based on storing using sqlite3.  I won't be adding any
  > other backends for this.

Before including this feature in master we should make sure both back-ends
are working.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Jean Louis]

Someone said previously:

> > that cannot be readily inspected, diffed, backed up,
> > version-controlled, shared, altered, or understood. (Or archived,
> > published, indexed, checksummed, ...)

Myself I inspect almost any database backed values through Emacs, edit
it, update it, delete it. Then I have implemented for my rather simple
purposes a `rcd-vc' package that saves any edited database column
value that I designate to be saved into the database. And I have
implemented for myself `diff' feature to find differences. Not that I
have much time and need for it, but from time to time I find it very
handy. Thus same can be implemented for SQLite.

Emacs: RCD Version Control system with PostgreSQL backend: 
https://hyperscope.link/3/6/7/9/6/Emacs-RCD-Version-Control-system-with-PostgreSQL-backend-36796.html

Databases can be easily archived, its tables, columns, can be
archived, that is what rcd-vc.el is doing within the database. 

Indexing and checksuming and many other features are often built-in
features in the database. 

Access to databases opens wide range of opportunities for Emacs users.


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Daniel Fleischer]

Pip Cet [2021-12-08 Wed 18:36] wrote:

> The proposal is to have Emacs store some user data in some binary
> format that cannot be readily inspected, diffed, backed up,
> version-controlled, shared, altered, or understood. (Or archived,
> published, indexed, checksummed, ...)

It's a binary file, sure, but what exactly prevents your from backing it
up? or putting it under git, sharing it, calculating its checksum or
exploring it via a free tool such as sqlite-browser?

-- 

Daniel Fleischer

Re: sqlite3

[Óscar Fuentes]

Daniel Fleischer <danflscr@gmail.com> writes:

> Pip Cet [2021-12-08 Wed 18:36] wrote:
>
>> The proposal is to have Emacs store some user data in some binary
>> format that cannot be readily inspected, diffed, backed up,
>> version-controlled, shared, altered, or understood. (Or archived,
>> published, indexed, checksummed, ...)
>
> It's a binary file, sure, but what exactly prevents your from backing it
> up? or putting it under git, sharing it, calculating its checksum or
> exploring it via a free tool such as sqlite-browser?

Of course you can do all of the above, but keep in mind that adding,
deleting or editing a row can have arbitrarily large effects on the
binary representation of the database.

And having to install an specific tool just to "explore" the database's
contents (read: have access within the tool's constraints) doesn't look
like very enticing perspective to me.

So you can do with a database a lot of things we usually do with text
files, but "readily" does not apply.

Re: sqlite3

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 09 Dec 2021 07:55:44 +0100
> 
> And having to install an specific tool just to "explore" the database's
> contents (read: have access within the tool's constraints) doesn't look
> like very enticing perspective to me.

Unless that tool comes with the library in the same package, that is.

Re: sqlite3

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> And having to install an specific tool just to "explore" the database's
>> contents (read: have access within the tool's constraints) doesn't look
>> like very enticing perspective to me.
>
> Unless that tool comes with the library in the same package, that is.

You have to know about its existence and how to use it.

Re: sqlite3

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 09 Dec 2021 10:50:56 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> And having to install an specific tool just to "explore" the database's
> >> contents (read: have access within the tool's constraints) doesn't look
> >> like very enticing perspective to me.
> >
> > Unless that tool comes with the library in the same package, that is.
> 
> You have to know about its existence and how to use it.

That information is just one  Internet search away.  Try:

   compare sqlite databases

Re: sqlite3

[Alexandre Garreau]

Le ĵaŭdo, 9-a de decembro 2021, 11-a horo kaj 16:50 CET Eli Zaretskii a 
écrit :
> > From: Óscar Fuentes <ofv@wanadoo.es>
> > Date: Thu, 09 Dec 2021 10:50:56 +0100
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> 
> 
> 
> > > Unless that tool comes with the library in the same package, that
> > > is.
> 
> 
> 
> > You have to know about its existence and how to use it.
> 
> That information is just one  Internet search away.  Try:
> 
>    compare sqlite databases

To me that looks like a bad argument in favor of replacing plain text by 
sqlite, in opposition to the argument “everybody already knows plain text, 
how to read and edit it”.  A lot more convincing answer to me is “make 
that tool autoloaded for the right extension/magic number for sqlite3 
databases, and learn about it with C-h m, just as usual with any mode in 
emacs”, and “present the sqlite buffer as plaintext just as we already do 
for directories with dired, and allow to edit it with text commands just 
as editable dired”.

The “text files everybody knows how to read and edit them” is a real 
philosophy, coming from the unix background.  I can understand its 
limitations, but it shouldn’t be repelled by a mere classical unfriendly 
“google is your friend” (a search engine stays a so big and costly thing, 
that currently there is no decent free-software one, they all are 
proprietary and centralized, and the only correct one either depend on ads 
(google stays the main one, and is considered as a base standard of 
quality (which is hardly attained by many competitors) for many people 
requiring others to websearch stuff by themselves), or are based in the 
patriot-act US (duckduckgo))..

Re: sqlite3

[Jean Louis]

* Óscar Fuentes <ofv@wanadoo.es> [2021-12-09 09:56]:
> Daniel Fleischer <danflscr@gmail.com> writes:
> 
> > Pip Cet [2021-12-08 Wed 18:36] wrote:
> >
> >> The proposal is to have Emacs store some user data in some binary
> >> format that cannot be readily inspected, diffed, backed up,
> >> version-controlled, shared, altered, or understood. (Or archived,
> >> published, indexed, checksummed, ...)
> >
> > It's a binary file, sure, but what exactly prevents your from backing it
> > up? or putting it under git, sharing it, calculating its checksum or
> > exploring it via a free tool such as sqlite-browser?
> 
> Of course you can do all of the above, but keep in mind that adding,
> deleting or editing a row can have arbitrarily large effects on the
> binary representation of the database.
> 
> And having to install an specific tool just to "explore" the database's
> contents (read: have access within the tool's constraints) doesn't look
> like very enticing perspective to me.
> 
> So you can do with a database a lot of things we usually do with text
> files, but "readily" does not apply.

To open text file with editor such as Emacs you need complex
dependencies. On Parabola GNU/Linux-libre, they are listed as: librsvg
gpm giflib libxpm libotf m17n-lib gtk3 hicolor-icon-theme
desktop-file-utils alsa-lib gnutls jansson cairo harfbuzz

Emacs is huge software. Complex.

sqlite3 is 1.6M with these dependencies;

	linux-vdso.so.1 (0x00007ffee8782000)
	libreadline.so.8 => /usr/lib/libreadline.so.8 (0x00007f47f83f7000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f47f82b3000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f47f82ac000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f47f8292000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f47f8271000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f47f80a5000)
	libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x00007f47f8030000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f47f85db000)



-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Lars Ingebrigtsen]

Pip Cet <pipcet@gmail.com> writes:

> The proposal is to have Emacs store some user data in some binary
> format that cannot be readily inspected, diffed, backed up,
> version-controlled, shared, altered, or understood. (Or archived,
> published, indexed, checksummed, ...)

The slope is pretty slippery from .emacs to .newsrc.eld to Gnus registry
storage to .sqlite files.

The .emacs file with the Customize settings is usually something a user
can read and edit and diff with some confidence.

The .newsrc.eld file, which is a couple of huge alists with complicated
nesting, is very hard to read, edit or diff, even thought it's "text".
Putting it in VC is meaningless.

The Gnus registry, which is a dumped hash table, can't meaningfully be
read or edited by anybody unless that person is an expert on hash table
representation and what it all means.  And it's not diffable or VC-able
without special software.

An .sqlite file isn't pretending to be user editable in vi, but there's
a plethora of software out there do do all things you'd expect, and (of
course) Emacs will come with a mode to inspect and edit these files more
free-form.  They will, in the end, be a lot more editable than a hash
table dump: This mode will allow you to say "delete all values that
refer to 'foo' no matter where it's stored in this file", and that's not
something that's anybody implemented for all these "text" formats that
Emacs uses today.

As for archiving, publishing, indexing or checksumming: Doing that with
an .sqlite file is a lot more meaningful than a file containing a hash
table dump.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> The Gnus registry, which is a dumped hash table, can't meaningfully be
> read or edited by anybody unless that person is an expert on hash table
> representation and what it all means.  And it's not diffable or VC-able
> without special software.
If you just want a persistent hash table, 
my resist! package do uses a diffable plain text representation.
Even after adding splited buckets in the future.

Re: sqlite3

[Pip Cet]

On Thu, Dec 9, 2021 at 12:09 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Pip Cet <pipcet@gmail.com> writes:
> > The proposal is to have Emacs store some user data in some binary
> > format that cannot be readily inspected, diffed, backed up,
> > version-controlled, shared, altered, or understood. (Or archived,
> > published, indexed, checksummed, ...)
>
> The slope is pretty slippery from .emacs to .newsrc.eld to Gnus registry
> storage to .sqlite files.

I agree that .newsrc.eld and the Gnus registry, as you describe them,
have problems that need to be fixed. Making those permanently
unfixable by switching to a binary-only format seems like a bad idea
to me, unless there are overwhelming advantages that I'm just failing
to see so far.

> The .emacs file with the Customize settings is usually something a user
> can read and edit and diff with some confidence.

I'm glad you're not proposing to convert that!

> The .newsrc.eld file, which is a couple of huge alists with complicated
> nesting, is very hard to read, edit or diff, even thought it's "text".

Then we need to work on improving its format.

> Putting it in VC is meaningless.

How so? I usually have my home directory under version control, and
git's xdelta implementation should deal with alists just fine (and
produce readable output at sub-line granularity)...

> The Gnus registry, which is a dumped hash table, can't meaningfully be
> read or edited by anybody unless that person is an expert on hash table
> representation and what it all means.

Then we need to improve that. It's not a huge change to allow dumping
hash tables in predictable order (rather than
predictable-unless-the-bucket-count-changed order) and at one tuple
per line; are there additional issues?

> And it's not diffable or VC-able
> without special software.

Possibly so. That still means you're making problems that are
currently coincidental fundamentally unfixable.

> An .sqlite file isn't pretending to be user editable in vi, but there's
> a plethora of software out there do do all things you'd expect, and (of

How much of it is GNU software?

> course) Emacs will come with a mode to inspect and edit these files more
> free-form.

Indeed, I would have considered it a good idea to introduce such a
mode well before proposing (as you might or might not have done) that
random packages start using a new library.

> They will, in the end, be a lot more editable than a hash
> table dump: This mode will allow you to say "delete all values that
> refer to 'foo' no matter where it's stored in this file", and that's not
> something that's anybody implemented for all these "text" formats that
> Emacs uses today.

You're touching on an important point here: if we go the sqlite route,
efforts will be split between improving things for text files
(something that is desperately necessary) and improving things for
sqlite databases.

Again, I'm not saying Emacs is currently perfect (blasphemy!), but at
least it's fixable in principle, without duplicating too much effort.

> As for archiving, publishing, indexing or checksumming: Doing that with
> an .sqlite file is a lot more meaningful than a file containing a hash
> table dump.

(I don't think sqlite (or sqlite3) is considered an acceptable format
by most archives, or publishers. ASCII text is.)

I don't see how indexing an sqlite file, without first converting it
into text format, gives you anything valuable: does sqlite even
guarantee long strings won't be split up arbitrarily?

And a checksum that changes with no underlying changes in the data is
problematic. Usually not a problem with ELisp dumps, but it usually is
with sqlite files.

I must confess my experience with sqlite is not very recent; back
then, it fsync()ed heavily (and unnecessarily), created additional
files that needed to be kept in sync with each other, and often ended
up corrupting the databases anyway. Even if those issues have been
fixed, I feel almost zero consideration has been given to
alternatives.

I must repeat that interacting with existing sqlite files is something
Emacs should be able to do, but creating new ones for Emacs purposes
is quite a different proposal, and by my reading it's what you
actually suggested.

And, merely as a request, I'd still like to hear a further convincing
case for all this being necessary in the first place. So far I've
heard "Gnus" and "emoji favorites". I can't currently use the former
and the latter is inaccessible to me on multiple levels, so I'm left
with my "binary junk in .emacs.d" impression of what you're actually
proposing.

Pip

Re: sqlite3

[Lars Ingebrigtsen]

Pip Cet <pipcet@gmail.com> writes:

> I agree that .newsrc.eld and the Gnus registry, as you describe them,
> have problems that need to be fixed. Making those permanently
> unfixable by switching to a binary-only format seems like a bad idea
> to me, unless there are overwhelming advantages that I'm just failing
> to see so far.

"Need to be fixed"?  They've been like this for three decades now, and
it's amusing how they suddenly "need" to be fixed when they're brought
up as an example for how textual data isn't really a panacea for
editability, diffing, etc.

The Gnus data is edited within Gnus, because editing the data outside of
Gnus will lead to breakage.

>> Putting it in VC is meaningless.
>
> How so? I usually have my home directory under version control, and
> git's xdelta implementation should deal with alists just fine (and
> produce readable output at sub-line granularity)...

It's meaningless because you can't really do anything with that
information.

>> And it's not diffable or VC-able
>> without special software.
>
> Possibly so. That still means you're making problems that are
> currently coincidental fundamentally unfixable.

?

>> An .sqlite file isn't pretending to be user editable in vi, but there's
>> a plethora of software out there do do all things you'd expect, and (of
>
> How much of it is GNU software?

You mean percentage wise?  I don't understand the issue.

>> course) Emacs will come with a mode to inspect and edit these files more
>> free-form.
>
> Indeed, I would have considered it a good idea to introduce such a
> mode well before proposing (as you might or might not have done) that
> random packages start using a new library.

Er...  I should have written a new library to allow editing SQLite files
before starting a discussion about whether Emacs should have in-core
SQLite support.  Check.

> And a checksum that changes with no underlying changes in the data is
> problematic. Usually not a problem with ELisp dumps, but it usually is
> with sqlite files.

There are programs to do all this.

> And, merely as a request, I'd still like to hear a further convincing
> case for all this being necessary in the first place. So far I've
> heard "Gnus" and "emoji favorites". I can't currently use the former
> and the latter is inaccessible to me on multiple levels, so I'm left
> with my "binary junk in .emacs.d" impression of what you're actually
> proposing.

Emacs needs easy, performant access to large data storage, and this
gives us that, and more.  And it's embarrassing that Emacs doesn't have
this already.

(For instance, my imdb mode stashes about 12GB worth of data in sqlite
and is simple and performant.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Pip Cet]

On Fri, Dec 10, 2021 at 12:33 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Pip Cet <pipcet@gmail.com> writes:
> > I agree that .newsrc.eld and the Gnus registry, as you describe them,
> > have problems that need to be fixed. Making those permanently
> > unfixable by switching to a binary-only format seems like a bad idea
> > to me, unless there are overwhelming advantages that I'm just failing
> > to see so far.
>
> "Need to be fixed"?

Yes. Not particularly urgently, since Emacs has so many long-standing issues.

>  They've been like this for three decades now, and
> it's amusing how they suddenly "need" to be fixed when they're brought
> up as an example for how textual data isn't really a panacea for
> editability, diffing, etc.

I agree, of course, that it's not a panacea. It's easy to get text
formats wrong, and when one does, they need to be fixed.

> The Gnus data is edited within Gnus, because editing the data outside of
> Gnus will lead to breakage.

How were you planning, assuming you want to migrate it to sqlite, to
encourage people to follow that suggestion? I think "editing this file
will most likely break things" is something that should be
communicated to users who have somehow started looking at an sqlite
file, and I'm not sure how they'd know.

Again, I never said that all text formats are, by virtue of being
text-based, readily editable, diffable, etc. What I said is that this
particular binary format isn't.

> >> Putting it in VC is meaningless.
> >
> > How so? I usually have my home directory under version control, and
> > git's xdelta implementation should deal with alists just fine (and
> > produce readable output at sub-line granularity)...
>
> It's meaningless because you can't really do anything with that
> information.

Yes, I can. Not necessarily as much as I'd like to :-)

> >> course) Emacs will come with a mode to inspect and edit these files more
> >> free-form.
> >
> > Indeed, I would have considered it a good idea to introduce such a
> > mode well before proposing (as you might or might not have done) that
> > random packages start using a new library.
>
> Er...  I should have written a new library to allow editing SQLite files
> before starting a discussion about whether Emacs should have in-core
> SQLite support.  Check.

I believe others have pointed out quite clearly that two proposals
have become quite mixed up here, at least in my mind. If it's clear to
you you weren't proposing to use sqlite3 as Emacs's persistent
key-value store, I must have misread the first message.

> (For instance, my imdb mode stashes about 12GB worth of data in sqlite
> and is simple and performant.)

Thanks for the example.

Pip

Re: sqlite3

[Lars Ingebrigtsen]

Pip Cet <pipcet@gmail.com> writes:

> How were you planning, assuming you want to migrate it to sqlite, to
> encourage people to follow that suggestion? I think "editing this file
> will most likely break things" is something that should be
> communicated to users who have somehow started looking at an sqlite
> file, and I'm not sure how they'd know.

I don't want to migrate .newsrc.eld to sqlite.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> We don't really require much of anything, it's just that Emacs doesn't
> work very well without some things.  Like gnutls.  This would be in the
> same category, so I'd expect 99.7% of Emacsen would have support for it
> toot sweet.
Implementing cryptography and TLS in Emacs Lisp is too much,
and would probably be hopelessly slow because they need to run
through every byte from the network traffic. There’s no other choice
than relying on either external gnutls-cli or gnutls as a library.

For sqlite3, my point is, in summary
1. As a query language, LISt Processing works equally well,
    and even if lisp starts to not work well, Prolog works better than SQL.
    There’s also the question of introducing two scripting languages into Emacs.
2. As a persistent kv store, it’s massive overkill.
    I postulate that a plain Emacs Lisp log-structured implementation is suffice.

IDK if I read the context correctly, but is it that your main intention currently
is to have a persistent kv store?
If that’s the case I might try to find sometime to hack a Emacs Lisp implementation.

In fact I can probably do better if I implement a real object/Lisp data store
rather than kv-store — aka, you define a root object of the object store,
and every time Emacs loads up the object graph reachable from the root
object is faithfully reconstructed.
Sounds much sweeter than SQL CRUD!

> (Except for those that hate SQL sooooo much that they can't even stand
> the thought of something in Emacs talking SQL even if they can't see
> it.)
There’s hardly anything in Emacs that can’t be seen, until maybe you reach
Intel microcodes, and electrons ;)
Either Lisp or Prolog would probably be more friendly for Lisp hackers,
comparing to SQL strings (flashbacks from being an SQL injection hacker…)


Best,
Qiantan

Re: sqlite3

[Tomas Hlavaty]

On Mon 06 Dec 2021 at 21:15, Qiantan Hong <qhong@mit.edu> wrote:
> For sqlite3, my point is, in summary
> 1. As a query language, LISt Processing works equally well,
>     and even if lisp starts to not work well, Prolog works better than SQL.
>     There’s also the question of introducing two scripting languages into Emacs.
[...]
> In fact I can probably do better if I implement a real object/Lisp data store
> rather than kv-store — aka, you define a root object of the object store,
> and every time Emacs loads up the object graph reachable from the root
> object is faithfully reconstructed.

It could even be a first class Emacs object like persistent symbols in
picolisp:-)

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021, 1-a horo kaj 17:56 CET Tomas Hlavaty a 
écrit :
> On Mon 06 Dec 2021 at 21:15, Qiantan Hong <qhong@mit.edu> wrote:
> > For sqlite3, my point is, in summary
> > 1. As a query language, LISt Processing works equally well,
> > 
> >     and even if lisp starts to not work well, Prolog works better than
> >     SQL.
> >     There’s also the question of introducing two scripting languages
> >     into Emacs.
> [...]
> 
> > In fact I can probably do better if I implement a real object/Lisp
> > data store rather than kv-store — aka, you define a root object of
> > the object store, and every time Emacs loads up the object graph
> > reachable from the root object is faithfully reconstructed.
> 
> It could even be a first class Emacs object like persistent symbols in
> picolisp:-)

what are these? could you link to its documentation?

Re: sqlite3

[Jean Louis]

* Eric Abrahamsen <eric@ericabrahamsen.net> [2021-12-06 08:30]:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > I think many user experience things in Emacs would be better if Emacs
> > remembered more.
> >
> > The bigger systems don't have this problem -- Gnus needs a large
> > .newsrc.eld file, and it maintains that.
> 
> A plea from the heart: Gnus also needs a better data store, and sqlite3
> is the answer to that need. The .newsrc.eld format is not okay.

sqlite3 is local file system database. In the great motion to include
the database one shall consider that database system shall become
available to Emacs Lisp programmers. And in that sense, the motion
shall expand into using the network databases such as
PostgreSQL. Module for it already exists. I am sure that developers
are fine to include it in Emacs or ELPA. I have asked them and we
discussed it. 

Reference:

emacs-libpq @ Github
https://github.com/anse1/emacs-libpq

Please keep the motion to include sqlite3, PostgreSQL, GDBM and other
DBM-like databases, so that Emacs can be built or support such. 

I am working with PostgreSQL all the time. For example I store my
cursors places in the database, for any single editable database
entry. 

It will boost Emacs' capabilities.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Bastien]

Hi Lars,

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.

I for one would love to see this.

I have two specific Org features in mind: migrating Org logging
abilities from in-org-file storage to a proper persistent storage
and storing backlinks for a set of files (say, agenda files).

Thanks for bringing this up!

-- 
 Bastien

Re: sqlite3

[Clément Pit-Claudel]

On 12/5/21 8:51 PM, Lars Ingebrigtsen wrote:
> sqlite3 is supported on more platforms than Emacs is, the interface is
> small and stable, and (best of all) somebody has already created
> interface functions for Emacs (via a module):
> 
>   https://github.com/syohex/emacs-sqlite3

There is also https://github.com/skeeto/emacsql , which builds SQLite3 as part of its installation.  It's already available and in use by the forge package (https://magit.vc/manual/forge.html), and it doesn't depend on modules.  

Re: sqlite3

[Lars Ingebrigtsen]

Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

> There is also https://github.com/skeeto/emacsql , which builds SQLite3
> as part of its installation.  It's already available and in use by the
> forge package (https://magit.vc/manual/forge.html), and it doesn't
> depend on modules.

"It works by maintaining a inferior process running (a "connection") for
interacting with the back-end database."

So it just talks to an sqlite3 executable over a pipe?  That's not
really what we're looking for here...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Jonas Bernoulli]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Clément Pit-Claudel <cpitclaudel@gmail.com> writes:
>
>> There is also https://github.com/skeeto/emacsql , which builds SQLite3
>> as part of its installation.  It's already available and in use by the
>> forge package (https://magit.vc/manual/forge.html), and it doesn't
>> depend on modules.
>
> "It works by maintaining a inferior process running (a "connection") for
> interacting with the back-end database."
>
> So it just talks to an sqlite3 executable over a pipe?  That's not
> really what we're looking for here...

Emacsql provides mainly two things:
1) An abstraction over different sql implementations.
2) A s-expressions dsl to write sql.

But I also think we should use syohex's sqlite3 module directly.  That
being said emacsql can be taught to use the sqlite3 module, in fact I
have already done that in the emacsql-libsqlite3 [1] package.

1: https://github.com/emacscollective/emacsql-libsqlite3

In my experience the sqlite3 module works reliably and I though I would
mention my wrapper around that in case Clément or some other Forge/Epkg
user would be interested in something like that.

     Jonas

Re: sqlite3

[Alfred M. Szmidt]

Before this discussion takes off, one should first take into acount if
it fits adding sqlite to Emacs with the overall guidelines of GNU
Emacs, the GNU project, and the GNU coding standards.

The sqlite non-license (it says public domain) status might be
problematic, since to get a proper license would require the FSF to
buy one explicitly.  Richard or someone would have to check how that
would work out.


The main benefit of dbm is that it is standard on all Unix systems,
GNU included while sqlite isn't.  There is also GNU GDBM which has
extra features over standard DBM, like data consitency and what nots
and is installed as widley as sqlite on GNU/Linux systems.

GNU recutils is not as standard, but it is a GNU project and could be
extended in maners that are useful for GNU emacs.  The nicest thing
about the recutils format is that it is plain text, and it is fast.
The developers maybe can fill in how well it does on bigger datasets;
adding Jose to CC.

Re: sqlite3

[Po Lu]

"Alfred M. Szmidt" <ams@gnu.org> writes:

> The main benefit of dbm is that it is standard on all Unix systems,
> GNU included while sqlite isn't.  There is also GNU GDBM which has
> extra features over standard DBM, like data consitency and what nots
> and is installed as widley as sqlite on GNU/Linux systems.

> GNU recutils is not as standard, but it is a GNU project and could be
> extended in maners that are useful for GNU emacs.  The nicest thing
> about the recutils format is that it is plain text, and it is fast.
> The developers maybe can fill in how well it does on bigger datasets;
> adding Jose to CC.

Exactly, this makes out my point very well.

Thanks!

Re: sqlite3

[Lars Ingebrigtsen]

"Alfred M. Szmidt" <ams@gnu.org> writes:

> The sqlite non-license (it says public domain) status might be
> problematic, since to get a proper license would require the FSF to
> buy one explicitly.  Richard or someone would have to check how that
> would work out.

Fedora is in the process of moving from *db to sqlite3 for the RPM
package format (or changed recently:
https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb), for instance, so I
don't think there's any license problems here.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Alfred M. Szmidt]

   > The sqlite non-license (it says public domain) status might be
   > problematic, since to get a proper license would require the FSF to
   > buy one explicitly.  Richard or someone would have to check how that
   > would work out.

   Fedora is in the process of moving from *db to sqlite3 for the RPM
   package format (or changed recently:
   https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb), for instance, so I
   don't think there's any license problems here.

The sqlite lack of license is not the sole issue, there is a
requirement on copyright assignments / disclaimers, all those things
need to be examined first.

Fedora and GNU also have different goals, different guidelines and
different policies.  If adding something that would be such a
significant dependency that it would be part of GNU Emacs one needs to
take into consideration based on GNU policies, and not on Fedoras.

Re: sqlite3

[Lars Ingebrigtsen]

"Alfred M. Szmidt" <ams@gnu.org> writes:

> The sqlite lack of license is not the sole issue, there is a
> requirement on copyright assignments / disclaimers, all those things
> need to be examined first.

It's a library built from a public domain code base that Emacs links
against.  I don't think there's anything to examine.

(If Emacs were to import the source code into Emacs, it would be a
different thing.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Jose E. Marchesi]

> Before this discussion takes off, one should first take into acount if
> it fits adding sqlite to Emacs with the overall guidelines of GNU
> Emacs, the GNU project, and the GNU coding standards.
>
> The sqlite non-license (it says public domain) status might be
> problematic, since to get a proper license would require the FSF to
> buy one explicitly.  Richard or someone would have to check how that
> would work out.
>
>
> The main benefit of dbm is that it is standard on all Unix systems,
> GNU included while sqlite isn't.  There is also GNU GDBM which has
> extra features over standard DBM, like data consitency and what nots
> and is installed as widley as sqlite on GNU/Linux systems.
>
> GNU recutils is not as standard, but it is a GNU project and could be
> extended in maners that are useful for GNU emacs.  The nicest thing
> about the recutils format is that it is plain text, and it is fast.
> The developers maybe can fill in how well it does on bigger datasets;
> adding Jose to CC.

What is the purpose of integrating a database sytem in Emacs?
To store what kind of data?
What amount of data?
Do you need to maintain data integrity?
Do you need foreign keys?
Do you need multi-field keys?

The reason I'm asking is because:

gdbm is a key-value tuples database.
sqlite is a relational database system.
GNU recutils is a sort of an hybrid between key-value and relational.

And so:

gdbm may or may not be what you need in Emacs.
sqlite is not a replacement for gdbm.
recutils is not a replacement for sqlite.

It seems to me that any discussion in which it is raised the alternative
of using so disparaged systems such as gdbm, sqlite and recutils for
some particular purpose, is in serious need of some clarification on why
a database sytem is needed to begin with and the nature of the data to
be stored there.

Re: sqlite3

[Po Lu]

"Jose E. Marchesi" <jemarch@gnu.org> writes:

> gdbm may or may not be what you need in Emacs.
> sqlite is not a replacement for gdbm.
> recutils is not a replacement for sqlite.

> It seems to me that any discussion in which it is raised the alternative
> of using so disparaged systems such as gdbm, sqlite and recutils for
> some particular purpose, is in serious need of some clarification on why
> a database sytem is needed to begin with and the nature of the data to
> be stored there.

The idea is that people want a way to store key-value pairs quickly (as
in, faster than a hash table serialized with `print').  Hence the idea
to use (g)dbm.

Re: sqlite3

[Jose E. Marchesi]

> "Jose E. Marchesi" <jemarch@gnu.org> writes:
>
>> gdbm may or may not be what you need in Emacs.
>> sqlite is not a replacement for gdbm.
>> recutils is not a replacement for sqlite.
>
>> It seems to me that any discussion in which it is raised the alternative
>> of using so disparaged systems such as gdbm, sqlite and recutils for
>> some particular purpose, is in serious need of some clarification on why
>> a database sytem is needed to begin with and the nature of the data to
>> be stored there.
>
> The idea is that people want a way to store key-value pairs quickly (as
> in, faster than a hash table serialized with `print').  Hence the idea
> to use (g)dbm.

If these are your requirements then I wouldn't recommend you to use
recutils, for several reasons.  Let me mention just a few.

First, recfiles are intended to be edited by hand as well as
programmatically.  This makes them sort of redundant, not very compact,
and also not the fastest format to parse and operate on.

Second, recutils supports relational and data integrity features that
may not be very useful in your case.  If as I suspect you plan to expose
a sort of data structure in Elisp to support key-value pairs (like what,
persistent a-lists backed by a dbm file?)  people using them will
undoubtly want to implement their own data integrity logic, or even
none.  You don't want to use anything that has its own ideas of what a
date looks like or how to compare them, for example.  This also applies
to sqlite and other relational database systems.

Third, recutils field keys are restricted to be something similar to a C
identifier, like in the record below from the GNU poke TODO file:

  Summary: unify `break' and `continue' in the same AST node
  Component: Compiler
  Kind: ENH
  Priority: 3
  Description:
  + The `break' and `continue' statements are similar enough to merit
  + using the same kind of AST node, which would require a discriminant
  + field (kind = break | continue).
  +
  + Likewise, there is some code duplication in the assembler which may be
  + optimized: pkl_asm_break_label_1 vs. pkl_asm_continue_label_1.
  Target: 2.0

it follows that if your pair keys can be arbitrary Elisp values then you
would be forced to use quite unnatural records like this for an a-list
'(("foo" (1 2 3) ("bar" (4 5 6)) (nil (0))):

  Key: "foo"
  Value: (1 2 3)

  Key: "bar"
  Value: (4 5 6)

  Key: nil
  Value: (0)

Fourth, you would also have problems with opaque lisp objects that don't
have a printed representation, in both keys and values.

Fifth, it seems to me that you don't need the notion of a "record"
(recutils) nor a relation (sqlite) in a key-value database.  If you come
with persistent a-lists (or similar) just pluck them into another list,
which may be itself be persistent and also backed by dbm.

And so on...  gdbm seems to provide exactly what you want, using
Lisp_Objects as keys and values.  I would just rejoice and use it :)

Re: sqlite3

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think many user experience things in Emacs would be better if Emacs
> remembered more.
>
> The bigger systems don't have this problem -- Gnus needs a large
> .newsrc.eld file, and it maintains that.
>
> The tiny things don't really have this problem, either: You save options
> with `customize-save-variable', and that fine.
>
> It's the many things that fall between these two extremes that have the
> problem: Where you want to store some state, but figuring it's just too
> much work to figure out Yet Another Storage Format, but the data is too
> messy to stash in somebody's .emacs file via Customize (like lists of
> stuff).
>
> So there's a bunch of stuff that Emacs just forgets when you shut down,
> where it perhaps shouldn't.

I would definitely like to see sqlite in core and avialable by default. I really
love it and think it is a great as application/desktop database. I think it
would open for some nice applications people could write.

> I've brought this up before, but I didn't really have a solution then,
> but I think I do now: sqlite3.
>
> sqlite3 is supported on more platforms than Emacs is, the interface is
> small and stable, and (best of all) somebody has already created
> interface functions for Emacs (via a module):
>
>   https://github.com/syohex/emacs-sqlite3
>
> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.  That is:
>
> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>
> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.

> In addition, if somebody really wants to write SQL stuff (it can be very
> handy for some things), having sqlite3 in there gives us that in
> addition for free.
>
> This comes with questions about how the users are supposed be able to
> clear out the data, for instance, but we could have a `M-x
> list-persistent-data' where the users could blow out whatever they want
> to.

My only remark is that sqlite is usually used for exactly same purpose as what
Emacs already has: customize. Your example:

> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)

looks to me like something one would use customize for.

Also in general since we can read/write lisp objects easily in elisp it is not
hard to serialize/deserialize a hashmap or an assoc list as an application
key-value db. I don't know how many application does this, but I have used it
myself, and it felt trivial to use in my app. Objects that are not serializable
will not be serializable to sql either, without some extra work, so there is not
much win to use sqllite db instead of hashmap. 

Anyway I still think sqlite in core would be nice; I have thought of it several
times myself.

Back to your example, I also wonder if api, as suggested in your example could
be done so it in addition also can automatically take applications prefix as
namespace and variable name as key, so your example could look like: 

(defgroup my-app nil
"My cool app."
:prefix 'my-app-
:group 'applications)

(defcustom my-app-emoji-favorites nil
"List of most used emojis."
:persist t
:type 'list
:group 'my-app)

(setf (persistent-data 'my-app-emoji-favorites) my-app-emoji--favorites)

And if we could also have more declarative api for defvars:

(defvar my-app-foo 'bar
"Foo property of my-app."
:persist t)

Just thoughts.

Re: sqlite3

[Stefan Kangas]

Arthur Miller <arthur.miller@live.com> writes:

> My only remark is that sqlite is usually used for exactly same purpose as what
> Emacs already has: customize. Your example:
>
>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>
> looks to me like something one would use customize for.

Is this historic use of customize not just a result of the lack
something like sqlite3?

FWIW, I think something like `package-selected-packages' is a minor
abuse of customize, and one which to me is rather impractical: when
synching my Init file between different machines, I don't necessarily
want to have the same packages installed on all of them.

For example, on my macOS machine I'd typically install some macOS
specific packages, that I don't want on my GNU/Linux machines.  Another
example is that I often install a package just to test it briefly, and I
definitely don't want that synchronized.  It is also not ideal that a
package is marked in `package-selected-packages', but is not even
installed locally.

Re: sqlite3

[Qiantan Hong]

>> My only remark is that sqlite is usually used for exactly same purpose as what
>> Emacs already has: customize. Your example:
>> 
>>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>> 
>> looks to me like something one would use customize for.
> 
> Is this historic use of customize not just a result of the lack
> something like sqlite3?
Or it could be the other way around. The prominent use of sqlite3
is that other environments (especially the more “static” ones) lack
customize, or the ability to configure the environment within itself
in general, thus relying on a central external store.

I really don’t like the idea of introducing an external UNIX-y 
dependency that significantly overlaps with Emacs Lisp itself,
it’s almost like saying “let’s make Emacs programmable using
JavaScript, because JavaScript is faster, safer, etc etc”.

We don’t need the sql language. In most cases it’s no better than 
MAPCAR-ing and FILTER-ing, and it’s strictly no better than just 
embedding a Prolog interpreter in Emacs Lisp.

What we really need is a faster persistence mechanism.
Anything that does read/write incrementally will be
lightning fast comparing to the current practice of printing/reading
plain s-exp, and is more than enough. 
Gdbm would fit perfectly if it weren’t, allegedly, too unstable.
Alternatively, I think it would be very doable to just implement 
a persistent hash-table with plain Emacs Lisp.
If I’ve got some free time I’ll probably do it.

> I think the more significant benefit is to be able to correctly handle
> cases where several Emacs sessions modify the persistent data.
> I'm thinking of things like `savehist`.
I think the problem of sharing mutable data between Emacsen
is orthogonal to persistence. It could be handled very well
within the images. crdt.el did a good job in sharing mutable
buffer (together with text properties, overlays, marks etc),
and I could add support for sharing key value maps at some point.
We don’t need to outsource it to some C program.

> Also in general since we can read/write lisp objects easily in elisp it is not
> hard to serialize/deserialize a hashmap or an assoc list as an application
> key-value db. I don't know how many application does this, but I have used it
> myself, and it felt trivial to use in my app. Objects that are not serializable
> will not be serializable to sql either, without some extra work, so there is not
> much win to use sqllite db instead of hashmap. 
That’s half of my point — we don’t need much (like a whole SQL implementation)
more than what we already have. I suspect simply dump out the whole
hash table to say gdbm won’t do a lot better than printing s-exp directly though.
The most important optimization is to write it incrementally.

Best,
Qiantan

Re: sqlite3

[Po Lu]

Qiantan Hong <qhong@mit.edu> writes:

> Gdbm would fit perfectly if it weren’t, allegedly, too unstable.

FWIW, I never found gdbm to be unstable.  On the other hand, I've had
various differing experiences with the Unix dbm implementations, but I
doubt any of the negative ones still hold true today.

Re: sqlite3

[Arthur Miller]

Qiantan Hong <qhong@mit.edu> writes:

>>> My only remark is that sqlite is usually used for exactly same purpose as what
>>> Emacs already has: customize. Your example:
>>> 
>>>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>>> 
>>> looks to me like something one would use customize for.
>> 
>> Is this historic use of customize not just a result of the lack
>> something like sqlite3?
> Or it could be the other way around. The prominent use of sqlite3
> is that other environments (especially the more “static” ones) lack
> customize, or the ability to configure the environment within itself
> in general, thus relying on a central external store.

You are on same thoughts as me. Other applications don't have Lisp. What makes
Lisp special is the homoiconicity, which makes serializing to files relatively
easy, in terms of not having to implement a dsl for the file format (init files,
csv, json, xml etc) and having to implement a query language to de-serialize
that format into lisp objects.

Re: sqlite3

[Stefan Kangas]

Qiantan Hong <qhong@mit.edu> writes:

> I really don’t like the idea of introducing an external UNIX-y
> dependency that significantly overlaps with Emacs Lisp itself,
> it’s almost like saying “let’s make Emacs programmable using
> JavaScript, because JavaScript is faster, safer, etc etc”.

I don't see any similarity between Lars' suggestion and the idea of
introducing a new language to rival Emacs Lisp.

> We don’t need the sql language. In most cases it’s no better than
> MAPCAR-ing and FILTER-ing, and it’s strictly no better than just
> embedding a Prolog interpreter in Emacs Lisp.

AFAIU, the idea is to hide away any SQL inside a new sqlite.el that will
provide some ORM.  I don't see this as fundamentally different to
depending on gdbm, or indeed any other store, which presumably also
would not be programmable directly in Emacs Lisp.

Re: sqlite3

[Alexandre Garreau]

Le lundo, 6-a de decembro 2021, 14-a horo kaj 8:48 CET Stefan Kangas a 
écrit :
> Qiantan Hong <qhong@mit.edu> writes:
> 
> 
> 
> > I really don’t like the idea of introducing an external UNIX-y
> > dependency that significantly overlaps with Emacs Lisp itself,
> > it’s almost like saying “let’s make Emacs programmable using
> > JavaScript, because JavaScript is faster, safer, etc etc”.
> 
> I don't see any similarity between Lars' suggestion and the idea of
> introducing a new language to rival Emacs Lisp.

SQL is a full-fleged logical language akin to prolog: the comparison is 
very sound.  Because then will arrive the same tradeoff occurs than in most 
modern webapps: whenever you interface with persistent I/O you start 
thinking “should that algorithm/datastructure be coded in SQL or <native 
language>?”, and the right answer is almost always SQL because that will 
give more freedom to planning, reorganization, optimization, etc.

the situation would be different if there was a powerful and efficient and 
free implementation of sql in lisp (postgres at the beginning almost 
became one but early changed direction toward C, which is more understood 
and causes less disagreement).

> > We don’t need the sql language. In most cases it’s no better than
> > MAPCAR-ing and FILTER-ing, and it’s strictly no better than just
> > embedding a Prolog interpreter in Emacs Lisp.
> 
> AFAIU, the idea is to hide away any SQL inside a new sqlite.el that will
> provide some ORM.  I don't see this as fundamentally different to
> depending on gdbm, or indeed any other store, which presumably also
> would not be programmable directly in Emacs Lisp.

gdbm could be a fully compatible backend for the exact same usage and 
interface as sqlite.  but that would mean purposedly limiting the power 
and expressiveness provided by sqlite.  people would want to circumvent 
that.

Re: sqlite3

[Arthur Miller]

Stefan Kangas <stefankangas@gmail.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> My only remark is that sqlite is usually used for exactly same purpose as what
>> Emacs already has: customize. Your example:
>>
>>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>>
>> looks to me like something one would use customize for.
>
> Is this historic use of customize not just a result of the lack
> something like sqlite3?

I think it is rather the other way around: a historical lack of lisp in other
applications :).

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

>> My only remark is that sqlite is usually used for exactly same purpose as what
>> Emacs already has: customize. Your example:
>>
>>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>>
>> looks to me like something one would use customize for.
>
> Is this historic use of customize not just a result of the lack
> something like sqlite3?

Yeah, I think so.  There's not always an obvious difference between
"configuration" and "app data", but if it's something that changes upon
every invocation (as the emojis in emoji--favorites do), it's more "app
data", and that doesn't belong in Customize.  (The user may have their
.emacs file in a VC, for instance.)

> For example, on my macOS machine I'd typically install some macOS
> specific packages, that I don't want on my GNU/Linux machines.  Another
> example is that I often install a package just to test it briefly, and I
> definitely don't want that synchronized.  It is also not ideal that a
> package is marked in `package-selected-packages', but is not even
> installed locally.

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Stefan Monnier]

> Yeah, I think so.  There's not always an obvious difference between
> "configuration" and "app data", but if it's something that changes upon
> every invocation (as the emojis in emoji--favorites do), it's more "app
> data", and that doesn't belong in Customize.  (The user may have their
> .emacs file in a VC, for instance.)

And on the flip side, customization data doesn't need to be too careful
with locking because the user is not expected to modify its
customization exactly at the same time in two separate processes.


        Stefan

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> And on the flip side, customization data doesn't need to be too careful
> with locking because the user is not expected to modify its
> customization exactly at the same time in two separate processes.

Yup.  Using sqlite solves a lot of these usability issues -- it allows
concurrency for these things, and as opposed to many other binary
formats, it's portable across all architectures, and there's a wide
variety of tools available to examine the data (if you don't want to use
Emacs itself to do it).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

package-selected-packages in Customize [was: sqlite3]

[Joost Kremers]

On Mon, Dec 06 2021, Stefan Kangas wrote:
> FWIW, I think something like `package-selected-packages' is a minor
> abuse of customize, and one which to me is rather impractical: when
> synching my Init file between different machines, I don't necessarily
> want to have the same packages installed on all of them.

For this very reason, I moved all of my settings out of Customize, set
`custom-file` to "~/.emacs.d/lisp/custom.el", which is consequently only used
for `package-selected-packages`, and keep this file out of the git repo.

Sure, I can't use `package-install-selected-packages`, but most of my packages
are installed through `use-package` anyway, so I can live with that. And I can't
use Customize, but that's also just a slight annoyance.



-- 
Joost Kremers
Life has its moments

Re: package-selected-packages in Customize [was: sqlite3]

[Stefan Kangas]

Joost Kremers <joostkremers@fastmail.fm> writes:

> On Mon, Dec 06 2021, Stefan Kangas wrote:
>> FWIW, I think something like `package-selected-packages' is a minor
>> abuse of customize, and one which to me is rather impractical: when
>> synching my Init file between different machines, I don't necessarily
>> want to have the same packages installed on all of them.
>
> For this very reason, I moved all of my settings out of Customize, set
> `custom-file` to "~/.emacs.d/lisp/custom.el", which is consequently only used
> for `package-selected-packages`, and keep this file out of the git repo.
>
> Sure, I can't use `package-install-selected-packages`, but most of my packages
> are installed through `use-package` anyway, so I can live with that. And I can't
> use Customize, but that's also just a slight annoyance.

That sounds less than ideal, indeed.  So this should be a prime
candidate for moving out of customize, and perhaps one early user of the
new persistent variable/sqlite support.

Re: sqlite3

[Byung-Hee HWANG]

Arthur Miller <arthur.miller@live.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> I think many user experience things in Emacs would be better if Emacs
>> remembered more.
>>
>> The bigger systems don't have this problem -- Gnus needs a large
>> .newsrc.eld file, and it maintains that.
>>
>> The tiny things don't really have this problem, either: You save options
>> with `customize-save-variable', and that fine.
>>
>> It's the many things that fall between these two extremes that have the
>> problem: Where you want to store some state, but figuring it's just too
>> much work to figure out Yet Another Storage Format, but the data is too
>> messy to stash in somebody's .emacs file via Customize (like lists of
>> stuff).
>>
>> So there's a bunch of stuff that Emacs just forgets when you shut down,
>> where it perhaps shouldn't.
>
> I would definitely like to see sqlite in core and avialable by default. I really
> love it and think it is a great as application/desktop database. I think it
> would open for some nice applications people could write.
>
>> I've brought this up before, but I didn't really have a solution then,
>> but I think I do now: sqlite3.
>>
>> sqlite3 is supported on more platforms than Emacs is, the interface is
>> small and stable, and (best of all) somebody has already created
>> interface functions for Emacs (via a module):
>>
>>   https://github.com/syohex/emacs-sqlite3
>>
>> I think it'd be good to bring that into core, and then write a small
>> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
>> that we don't have to write SQL all over the place.  That is:
>>
>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>>
>> I.e., what Emacs needs is a persistent key/value store, and this would
>> give us that.
>
>> In addition, if somebody really wants to write SQL stuff (it can be very
>> handy for some things), having sqlite3 in there gives us that in
>> addition for free.
>>
>> This comes with questions about how the users are supposed be able to
>> clear out the data, for instance, but we could have a `M-x
>> list-persistent-data' where the users could blow out whatever they want
>> to.
>
> My only remark is that sqlite is usually used for exactly same purpose as what
> Emacs already has: customize. Your example:
>
>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>
> looks to me like something one would use customize for.
>
> Also in general since we can read/write lisp objects easily in elisp it is not
> hard to serialize/deserialize a hashmap or an assoc list as an application
> key-value db. I don't know how many application does this, but I have used it
> myself, and it felt trivial to use in my app. Objects that are not serializable
> will not be serializable to sql either, without some extra work, so there is not
> much win to use sqllite db instead of hashmap. 
>
> Anyway I still think sqlite in core would be nice; I have thought of it several
> times myself.
>
> Back to your example, I also wonder if api, as suggested in your example could
> be done so it in addition also can automatically take applications prefix as
> namespace and variable name as key, so your example could look like: 
>
> (defgroup my-app nil
> "My cool app."
> :prefix 'my-app-
> :group 'applications)
>
> (defcustom my-app-emoji-favorites nil
> "List of most used emojis."
> :persist t
> :type 'list
> :group 'my-app)
>
> (setf (persistent-data 'my-app-emoji-favorites) my-app-emoji--favorites)
>
> And if we could also have more declarative api for defvars:
>
> (defvar my-app-foo 'bar
> "Foo property of my-app."
> :persist t)
>
> Just thoughts.

+1, very much!!!

Sincerely, Gnus fan Byung-Hee

-- 
^고맙습니다 _和合團結_ 감사합니다_^))//

Re: sqlite3

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> I would definitely like to see sqlite in core and avialable by
> default. I really love it and think it is a great as
> application/desktop database. I think it would open for some nice
> applications people could write.

Regardless of whether or not this feature is implemented, I think it
should be optional, and fallbacks should be in place for systems where
SQLite is unavailable.

For example, I tried compiling the latest SQLite amalgamation release
with DJGPP, and it did not work.  Apparently patches existed to make it
work, but they do not apply to the latest release anymore.

Customize gaining a hard dependency on SQLite3 would be a disaster!

Using a relational database for things like

    (setf (persistent-data :namespace "emoji" :key "favorites")
          emoji--favorites)

Is also overkill, IMO, while something like (g)dbm would be a perfect
match for this kind of task.

Re: sqlite3

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I would definitely like to see sqlite in core and avialable by
>> default. I really love it and think it is a great as
>> application/desktop database. I think it would open for some nice
>> applications people could write.
>
> Regardless of whether or not this feature is implemented, I think it
> should be optional, and fallbacks should be in place for systems where
> SQLite is unavailable.
Yes of course; I agree with you. compile-time option --with-sqlite ...

> For example, I tried compiling the latest SQLite amalgamation release
> with DJGPP, and it did not work.  Apparently patches existed to make it
> work, but they do not apply to the latest release anymore.
>
> Customize gaining a hard dependency on SQLite3 would be a disaster!
>
> Using a relational database for things like
>
>     (setf (persistent-data :namespace "emoji" :key "favorites")
>           emoji--favorites)
>
> Is also overkill, IMO, while something like (g)dbm would be a perfect
> match for this kind of task.

I suggested hashmap serialzied to/from file. I used such approach myself for
key-value pairs. It worked fine for me, it was quite simple to read/write
it. Currently I am testing a thing, and I use just assoc list to read/write it
to a file, and it works fine for me too.

I also think that everyone is uses sqlite for persisting user settings because
they don't have lisp and customize :).

Just because everyone uses sqlite is maybe not the best argument, but
anyway, sqlite is maybe faster when serializing that reading/writing lisp
objects. I don't know, I am not familiar so much with elisp vs sqlite. Also,
there are many uses for relational databse than just persisting user settings. I
am thinking of desktop applications a lá Access/Excel and similar. Sqlite is
great for applications that need data to be portable/movable with application,
or where database system like MySQL/Postgre etc are too much. I don't see it as
just a "setting database" a lá Firefox.

Re: sqlite3

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> I suggested hashmap serialzied to/from file. I used such approach
> myself for key-value pairs. It worked fine for me, it was quite simple
> to read/write it. Currently I am testing a thing, and I use just assoc
> list to read/write it to a file, and it works fine for me too.

I think the ideal choice would either be dbm (and the GNU variant, gdbm,
which improves upon it in every area) or recutils.

Which still raises the other important question of how to serialize
data, and how to do it better than `read' and `print'.

> Just because everyone uses sqlite is maybe not the best argument, but
> anyway, sqlite is maybe faster when serializing that reading/writing
> lisp objects.  I don't know, I am not familiar so much with elisp vs
> sqlite. Also, there are many uses for relational databse than just
> persisting user settings.  I am thinking of desktop applications a lá
> Access/Excel and similar.

What is special about the settings of MS Access or MS Excel that warrant
a relational database for storing them?

Re: sqlite3

[Qiantan Hong]

> Which still raises the other important question of how to serialize
> data, and how to do it better than `read' and `print'.
I agree that this is the real question, and my crude answer would
be to store an append-only log of deltas.
In fact I think if we can do this we don’t even need (g)dbm,
just storing the log in plain s-exp will be performant enough.

The detail gets messier. If we have a hash table whose value
is not mutable, we only need to append a log entry every
persistent-puthash.

What if the values are mutated? I can see two options:
1. A persistent hash table can be declared as :value-mutable t.
   In such case it will make an internal copy of its values
   (using COPY-TREE), and we provide a
   (PERSISTENT-HASH-CHECKPOINT &optional KEY),
   which compares the live value of KEY and “internally cached”
   value of KEY, store the delta if they’re different.
   (If KEY is not provided, do it for every keys)
   
   Cons: potentially doubled memory usage, and potentially
   slow checkpointing.

2. A very radical approach (I consider it probably the “right” way).
    Add a cell-mutated-hook to Emacs Lisp VM so that one
    can attach an “observer” to a slot of a datum in the memory.
    To avoid the performance nightmare, we probably need to have
    two disjoint kinds of watched and unwatched datum. 
    Basically, add an watched counterpart to every Elisp VM type we
    currently have, and implementation of the original unwatched type
    is unchanged so they can hopefully be accessed without any overhead.


Best,
Qiantan

Re: sqlite3

[Qiantan Hong]

> 1. A persistent hash table can be declared as :value-mutable t.
>   In such case it will make an internal copy of its values
>   (using COPY-TREE), and we provide a
>   (PERSISTENT-HASH-CHECKPOINT &optional KEY),
>   which compares the live value of KEY and “internally cached”
>   value of KEY, store the delta if they’re different.
>   (If KEY is not provided, do it for every keys)
Clarification: I mean the live/cached value associated with key
by the hash table, not the value of key itself.

Re: sqlite3

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Mon, Dec 6, 2021 at 8:03 AM Qiantan Hong <qhong@mit.edu> wrote:

> In fact I think if we can do this we don’t even need (g)dbm,
> just storing the log in plain s-exp will be performant enough.
>
> The detail gets messier. If we have a hash table whose value
> is not mutable, we only need to append a log entry every
> persistent-puthash.
>

Potentially relevant, a paper I wrote 8 years ago at a
now defunct startup: "Updating large dynamic objects
<https://docs.google.com/document/d/13g6pZhIf3gw45KG2DysDxXgtVZa0_TU3ndquiOYhyQY/edit?usp=sharing>
".

/john

[-- Attachment #2: Type: text/html, Size: 1326 bytes --]

Re: sqlite3

[Qiantan Hong]

Thanks for the paper!
I think we can safely assume we only need internal objects in
Emacs — people probably won't use Emacs to handle millions
of business record (not fit into main memory) in foreseeable future.

It does remind me that there’s another way to actually implement
incremental persistence in Elisp:

> 1. A persistent hash table can be declared as :value-mutable t.
>   In such case it will make an internal copy of its values ...
> 
> 2. A very radical approach (I consider it probably the “right” way).
>    Add a cell-mutated-hook to Emacs Lisp VM …

3. Provide a function (SYNC-DATUM DATUM STORE),
    and rely on user/package author to call it everytime
    some datum which is also a part of a persistent store
    is mutated.
    This is probably the most flexible solution, with minimal
    overhead, by shifting more responsibility to package authors.

Best,
Qiantan

Mutating existing persistent data (was: sqlite3)

[Stefan Monnier]

>> 1. A persistent hash table can be declared as :value-mutable t.
>>   In such case it will make an internal copy of its values ...
>> 
>> 2. A very radical approach (I consider it probably the “right” way).
>>    Add a cell-mutated-hook to Emacs Lisp VM …
>
> 3. Provide a function (SYNC-DATUM DATUM STORE),
>     and rely on user/package author to call it everytime
>     some datum which is also a part of a persistent store
>     is mutated.
>     This is probably the most flexible solution, with minimal
>     overhead, by shifting more responsibility to package authors.

I don't think it includes enough information for the persistent data to
be updated correctly in the presence of concurrent updates.

You either need the application to provide some atomicity/transaction
boundary information, or you need the application to send changes as
changes rather than as the resulting state (i.e. send "increment x by 2"
rather than "x is now 23").


        Stefan

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> You either need the application to provide some atomicity/transaction
> boundary information, or you need the application to send changes as
> changes rather than as the resulting state (i.e. send "increment x by 2"
> rather than "x is now 23").

So I'm now starting to type in the stuff to make persistent values, and
the name `persistent-value' is of course already taken (by something in
ieieo that I can't even find, as usual).

So here's some synonyms.  Er...  opinions?

  (setf (sticky-value foo) 'bar) 

has a certain ring to it, but...  is it one of the good rings?

    constant
    continual
    endless
    enduring
    incessant
    insistent
    perpetual
    relentless
    repeated
    resolute
    steadfast
    steady
    stubborn
    tenacious
    unrelenting
    unremitting
    assiduous
    bound
    bound and determined
    bulldogged
    dogged
    firm
    fixed
    immovable
    in for long haul
    indefatigable
    interminable
    like bad penny
    never-ending
    obdurate
    obstinate
    perseverant
    persevering
    persisting
    pertinacious
    sticky
    tireless
    unflagging
    unshakable 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So here's some synonyms.  Er...  opinions?
>
>   (setf (sticky-value foo) 'bar) 
>
> has a certain ring to it, but...  is it one of the good rings?

Does this really have to be a generalized variable?

Why not use something like `set-persistent-value' and
`get-persistent-value'?

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Does this really have to be a generalized variable?
>
> Why not use something like `set-persistent-value' and
> `get-persistent-value'?

I think we're moving away from get/set pairs.  We could have gone even
more magical -- reader macros!

Anyway, this is now working, I think, but of course the most difficult
bit, naming, is left to last.  So basically:

(define-sticky-variable foo 'bar)

This defines a sticky wicket foo, I mean object, and `bar' is the
default value if there isn't one already stored.  We access it like
this:

(sticky-value foo)
=> zot

There was apparently a stored value, so we got that back instead of the
initial value.

(setf (sticky-value foo) 'gazonk)
(sticky-value foo)
=> gazonk

These sticky values can be either synchronised or not.  If they're
synchronised, `(sticky-value foo)' will check whether the value has been
updated externally (i.e., from a different Emacs session) and return
that value.  If not, it returns the local value.

(push 'gazonk (sticky-value list-foo))

if `list-foo' is synchronised, will possibly read in the value from the
db first, prepend `gazonk' to it, and then write it back out again.  (So
the other Emacs will get that value next time it accesses (sticky-value
list-foo).

RPC!!!  😱

But we should call this something other than "sticky", because that's
kinda sticky.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think we're moving away from get/set pairs.  We could have gone even
> more magical -- reader macros!

Reader macros are one of the features of Common Lisp that are highly
prone to abuse.  They also make the meaning of simple programs confusing
to the human eye.

It would be a shame for a basic feature of Emacs Lisp to use such a
facility.

Personally, I never saw the benefit of generalized variables in Emacs
Lisp.  I think it would be more practical to stick with the tried and
true method of set and get pairs.

> (push 'gazonk (sticky-value list-foo))

> But we should call this something other than "sticky", because that's
> kinda sticky.

What if Emacs isn't built with sqlite3 support?  Is there a fall back in
place for that situation?

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Reader macros are one of the features of Common Lisp that are highly
> prone to abuse.  They also make the meaning of simple programs confusing
> to the human eye.
>
> It would be a shame for a basic feature of Emacs Lisp to use such a
> facility.

I'm just saying we could have gone even more magical.  Generalised
variables, on the other hand, are pretty vanilla.

> Personally, I never saw the benefit of generalized variables in Emacs
> Lisp.  I think it would be more practical to stick with the tried and
> true method of set and get pairs.

Well, I disagree.

> What if Emacs isn't built with sqlite3 support?  Is there a fall back in
> place for that situation?

Yes, it just stores the data in the object, and nothing happens on disk.
(The same happens when you start emacs with -Q, natch.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yes, it just stores the data in the object, and nothing happens on disk.
> (The same happens when you start emacs with -Q, natch.)

I think the fallback mechanism should also store data to disk, perhaps
by serializing a hash table with `print' and writing it to a file.

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> I think the fallback mechanism should also store data to disk, perhaps
> by serializing a hash table with `print' and writing it to a file.

That would mean writing a lots of itsy bitsy files (i.e., using the file
system as the DB) if this is going to have any kind of meaningful
concurrency.  Otherwise Emacsen will just be overwriting each other's
data aggressively.

And it'll be dog slow.

So I don't see the use case.  Virtually all systems have sqlite, and if
the users have chosen to not build Emacs with it, they don't get the
persistence feature.  That's no different from when they choose to build
without libxml -- they don't get to read web pages in Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> That's no different from when they choose to build without libxml --
> they don't get to read web pages in Emacs.

(I mean, they don't get to use eww in Emacs.  There's other ways to read
web pages in Emacs, of course.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> That would mean writing a lots of itsy bitsy files (i.e., using the file
> system as the DB) if this is going to have any kind of meaningful
> concurrency.  Otherwise Emacsen will just be overwriting each other's
> data aggressively.

That doesn't matter, as it'll be a fallback after all.  Besides, I also
don't see a problem with overwriting files aggressively, as long as
nothing ends up corrupt.

> So I don't see the use case.  Virtually all systems have sqlite, and if
> the users have chosen to not build Emacs with it, they don't get the
> persistence feature.  That's no different from when they choose to build
> without libxml -- they don't get to read web pages in Emacs.

Reading web pages in Emacs is not a vital feature, but this will be an
important feature any Lisp code that wants to persist data to disk will
use.

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Reading web pages in Emacs is not a vital feature, but this will be an
> important feature any Lisp code that wants to persist data to disk will
> use.

And they will have sqlite3 installed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Reading web pages in Emacs is not a vital feature, but this will be an
>> important feature any Lisp code that wants to persist data to disk will
>> use.

> And they will have sqlite3 installed.

The keyword here is "Lisp code", and not "person".  Persisting data to
disk will become a very important feature, the lack of which will make
Emacs unusable for people who do (or can) not build Emacs with sqlite3.

Unlike shr, which is not that important for normal use of Emacs.

Re: Mutating existing persistent data

[Qiantan Hong]

> On Dec 6, 2021, at 10:24 PM, Po Lu <luangruo@yahoo.com> wrote:
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
>>> Reading web pages in Emacs is not a vital feature, but this will be an
>>> important feature any Lisp code that wants to persist data to disk will
>>> use.
> 
>> And they will have sqlite3 installed.
> 
> The keyword here is "Lisp code", and not "person".  Persisting data to
> disk will become a very important feature, the lack of which will make
> Emacs unusable for people who do (or can) not build Emacs with sqlite3.

As for this topic, I don’t think sqlite3 help any much for this specific problem.
The real problem is how to do this for Emacs Lisp, and if we figured this out
any underlying storage will work, either plain text or gdb(m) or sqlite3.

And I don’t think the latter 2 would even provide much performance benefit,
because you will probably print parts of data to string then feed to db anyway.
db doesn’t understand Lisp, it doesn’t solve Lisp problem.

Re: Mutating existing persistent data

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

On Tue, Dec 07, 2021 at 06:27:39AM +0000, Qiantan Hong wrote:
> 
> 
> > On Dec 6, 2021, at 10:24 PM, Po Lu <luangruo@yahoo.com> wrote:
> > 
> > Lars Ingebrigtsen <larsi@gnus.org> writes:
> > 
> >>> Reading web pages in Emacs is not a vital feature, but this will be an
> >>> important feature any Lisp code that wants to persist data to disk will
> >>> use.
> > 
> >> And they will have sqlite3 installed.
> > 
> > The keyword here is "Lisp code", and not "person".  Persisting data to
> > disk will become a very important feature, the lack of which will make
> > Emacs unusable for people who do (or can) not build Emacs with sqlite3.
> 
> As for this topic, I don’t think sqlite3 help any much for this specific problem.
> The real problem is how to do this for Emacs Lisp, and if we figured this out
> any underlying storage will work, either plain text or gdb(m) or sqlite3.
> 
> And I don’t think the latter 2 would even provide much performance benefit,
> because you will probably print parts of data to string then feed to db anyway.
> db doesn’t understand Lisp, it doesn’t solve Lisp problem.

I concur. The interface looks good; why it /has/ to be sqlite3, of all
things... I'd rather avoid such an overspecified dependency. Why not
PostgreSQL, then, or, say, DB2? (no, I'm not serious on that, just to
illustrate my issue).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

<tomas@tuxteam.de> writes:

> I concur. The interface looks good; why it /has/ to be sqlite3, of all
> things... I'd rather avoid such an overspecified dependency. Why not
> PostgreSQL, then, or, say, DB2? (no, I'm not serious on that, just to
> illustrate my issue).

I fail to see how that illustrates anything.

sqlite3 is a C library with excellent performance, robustness and
portability.  There isn't much to recommend other key/value stores over
sqlite3.

That you can also do SQL with it is pretty irrelevant for this use case.

That you can do SQL with it, in addition, is also nice, because that's
useful for other things.

So using sqlite3 gives us everything you'd get with *dbm, and you get
other nice stuff in addition.

I haven't really seen anything new here in this discussion for a while,
and I'm now just re-iterating what I've already said numerous times, so
I won't do that any further.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Qiantan Hong]

> sqlite3 is a C library with excellent performance, robustness and
> portability.  There isn't much to recommend other key/value stores over
> sqlite3.
As I’ve reiterated several times, using it as a persistent Lisp storage
make use of none of the “advantages”, comparing over just plain text.

I’ve sent a pure Elisp implementation in another email, which seems
to be stuck in the cables for a while somehow, it will probably get through
soon.

Re: Mutating existing persistent data

[tomas]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On Tue, Dec 07, 2021 at 08:42:40AM +0100, Lars Ingebrigtsen wrote:
> <tomas@tuxteam.de> writes:
> 
> > I concur. The interface looks good; why it /has/ to be sqlite3, of all
> > things... I'd rather avoid such an overspecified dependency. Why not
> > PostgreSQL, then, or, say, DB2? (no, I'm not serious on that, just to
> > illustrate my issue).
> 
> I fail to see how that illustrates anything.
> 
> sqlite3 is a C library with excellent performance, robustness and
> portability.  There isn't much to recommend other key/value stores over
> sqlite3.

I'd rather like to see sqlite3 as one of several possible backends
instead of a strict requirement, but hey, that's me. I'm also that
strange gui who explicitly disables DBus dependency in the Emacs build.

I'm weird, I know :)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Mutating existing persistent data

[Andreas Schwab]

That looks more like an instance of "if all you have is a hammer,
everything looks like a nail".

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Mutating existing persistent data

[Qiantan Hong]

>> Does this really have to be a generalized variable?
>> 
>> Why not use something like `set-persistent-value' and
>> `get-persistent-value'?
> 
> I think we're moving away from get/set pairs.  We could have gone even
> more magical -- reader macros!

I advocate for the real magic — persistent object storage.

I posted sth about it (and also sth else) in
https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00602.html
Lars, can you comment on it?

Essentially, the interface I’m proposing is 
(make-persistent-store generalized-variable path)

The first time (in the history of universe) it is called,
when path does not exist, it does nothing, but it
initialize the bookkeeping structure.
When it is called later (in potentially a new Emacs 
instance), the full content of generalized-variable is
fully recovered from its last state.

Full content means it will allow arbitrary nested
data, and preserve sharing — it reconstructs an
object graph.

To make the whole thing work, we also need
(persistent-setf generalized-variable value).
Package author should use this instead of
other means of mutation when mutating any
places reachable (aka, “in”) the persistent store.


Best,
Qiantan

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> I posted sth about it (and also sth else) in
> https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00602.html
> Lars, can you comment on it?
>
> Essentially, the interface I’m proposing is 
> (make-persistent-store generalized-variable path)
>
> The first time (in the history of universe) it is called,
> when path does not exist, it does nothing, but it
> initialize the bookkeeping structure.
> When it is called later (in potentially a new Emacs 
> instance), the full content of generalized-variable is
> fully recovered from its last state.

I think you're proposing what I've already implemented.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Qiantan Hong]

> I think you're proposing what I've already implemented.
That’s interesting. If you declare a variable persistent
and the modified a cons deeply nested inside what the
variable points to, are you able to store the mutation
as a single (log operation)?

The only way I can come up with solving this is to
maintain a topological order of the object graph,
if you’ve already done so, that’s impressive.

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> That’s interesting. If you declare a variable persistent
> and the modified a cons deeply nested inside what the
> variable points to, are you able to store the mutation
> as a single (log operation)?

No, I store the whole value.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Qiantan Hong]

>> That’s interesting. If you declare a variable persistent
>> and the modified a cons deeply nested inside what the
>> variable points to, are you able to store the mutation
>> as a single (log operation)?
> 
> No, I store the whole value.
It’s a persistent kv store then.
I think we can implement it painlessly in pure Lisp without sqlite3,
as in  https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00587.html

Maybe provide it as an (always available) backend for your sticky.el?
I also suspect sqlite3 won’t give any noticeable performance benefit
over plain Emacs Lisp, because you need to print s-expr for each
individual Lisp key/values anyway.

Re: Mutating existing persistent data

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>   (setf (sticky-value foo) 'bar) 

Before you go too far into specific implementation, could you also
consider buffer-local persistent values or even generalised key:value
storage?

Org mode uses its own take on persistent storage to cache parsed
representation of individual buffers [1]. Our persistent values have to
be associated with specific files/buffer hashes. It's not just storage
of one global variable-symbol:value pair, but rather a generalised
key:value pair with key constructed from the variable symbol and also
file name + buffer hash.

[1] https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/org-persist.el

Best,
Ihor

Re: Mutating existing persistent data

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 07 Dec 2021 03:53:19 +0100
> Cc: Po Lu <luangruo@yahoo.com>, Qiantan Hong <qhong@mit.edu>,
>  "emacs-devel@gnu.org" <emacs-devel@gnu.org>,
>  Arthur Miller <arthur.miller@live.com>, John Yates <john@yates-sheets.org>
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > You either need the application to provide some atomicity/transaction
> > boundary information, or you need the application to send changes as
> > changes rather than as the resulting state (i.e. send "increment x by 2"
> > rather than "x is now 23").
> 
> So I'm now starting to type in the stuff to make persistent values, and
> the name `persistent-value' is of course already taken (by something in
> ieieo that I can't even find, as usual).
> 
> So here's some synonyms.  Er...  opinions?

Maybe we don't need to give up so easily.  We could use something like
sqlite-persistent-value or db-persistent-value.  eieio itself has
several FOO-persistent-BAR things, so why can't we have one more
variety?

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe we don't need to give up so easily.  We could use something like
> sqlite-persistent-value or db-persistent-value.  eieio itself has
> several FOO-persistent-BAR things, so why can't we have one more
> variety?

That it's a db on the backend is a detail the callers don't really need
to know, so putting that in the name here seems less than optimal.

But perhaps we could go in a totally different direction with the name
since none of the things that rhyme with "persistent" are free or good.
What about...  `state'?

So `define-state-variable' and `(state-value foo)'?  It's a bit on the
vague side, but it's kinda snappy.

Doesn't seem to be taken in-tree at least.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  luangruo@yahoo.com,  qhong@mit.edu,
>   emacs-devel@gnu.org,  arthur.miller@live.com,  john@yates-sheets.org
> Date: Tue, 07 Dec 2021 21:04:37 +0100
> 
> That it's a db on the backend is a detail the callers don't really need
> to know, so putting that in the name here seems less than optimal.

Well, hardly anyone will NOT know that we are using sqlite, so using
that doesn't reveal any secrets.

> What about...  `state'?
> 
> So `define-state-variable' and `(state-value foo)'?  It's a bit on the
> vague side, but it's kinda snappy.

"stateful" sounds better, no?

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> So `define-state-variable' and `(state-value foo)'?  It's a bit on the
>> vague side, but it's kinda snappy.
>
> "stateful" sounds better, no?

All variables are stateful, so I was hoping the slight awkwardness of
"state" might indicate to the person reading such code that there's
something special about a "state variable"...

Uhm...  anybody else got any ideas for a word for a variable that has a
value that survives an Emacs shutdown?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Uhm...  anybody else got any ideas for a word for a variable that has a
> value that survives an Emacs shutdown?

Here's two random ideas:

    define-saved-variable
    define-stored-variable

Re: Mutating existing persistent data

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> Here's two random ideas:
>
>     define-saved-variable
>     define-stored-variable

Hm...  possible...

Joost Kremers <joostkremers@fastmail.fm> writes:

> I noticed you hadn't listed "permanent" yet, or "durable".
>
> But I do think 'persistent' is the usual term to describe this sort of
> thing, so personally I'd find a way to use it.

It's possible that we could make CEDET not use it, I guess?  The way
things are connected up in CEDET makes it a challenge to start to
unravel it, though.

Bob Rogers <rogers-emacs@rgrjr.homedns.org> writes:

> Maybe use "multisession"?  That's a bit of a mouthful, it's more precise
> than anything else I've seen, and some extra semantic weight won't hurt.

"multisession" does convey a lot -- these objects are extra-special in
that they'll discover when they've been changed by a different Emacs (if
enabled), too, and "multisession" also says something about that.

Is there a shorter work that conveys the same thing?  I.e., that the
value might be coming from a different Emacs, and that it's persistent,
too?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mutating existing persistent data

[Bob Rogers]

   From: Lars Ingebrigtsen <larsi@gnus.org>
   Date: Wed, 08 Dec 2021 02:16:32 +0100

   . . .

   Bob Rogers <rogers-emacs@rgrjr.homedns.org> writes:

   > Maybe use "multisession"?  That's a bit of a mouthful, [but] it's
   > more precise than anything else I've seen, and some extra semantic
   > weight won't hurt.

   "multisession" does convey a lot -- these objects are extra-special in
   that they'll discover when they've been changed by a different Emacs (if
   enabled), too, and "multisession" also says something about that.

   Is there a shorter work that conveys the same thing?  I.e., that the
   value might be coming from a different Emacs, and that it's persistent,
   too?

That's a lot to convey in a single word, let alone one that is shorter.
You could consider an acronym, but that risks being obscure.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Mutating existing persistent data

[Joost Kremers]

On Tue, Dec 07 2021, Lars Ingebrigtsen wrote:
> Uhm...  anybody else got any ideas for a word for a variable that has a
> value that survives an Emacs shutdown?

persist-defvar?

Oh, wait, that already exists... 😲 (Well, OK, it's on GNU ELPA, but still.)

I noticed you hadn't listed "permanent" yet, or "durable".

But I do think 'persistent' is the usual term to describe this sort of thing, so
personally I'd find a way to use it.

-- 
Joost Kremers
Life has its moments

Re: Mutating existing persistent data

[Joost Kremers]

On Tue, Dec 07 2021, Joost Kremers wrote:
> On Tue, Dec 07 2021, Lars Ingebrigtsen wrote:
>> Uhm...  anybody else got any ideas for a word for a variable that has a
>> value that survives an Emacs shutdown?
>
> persist-defvar?
>
> Oh, wait, that already exists... 😲 (Well, OK, it's on GNU ELPA, but still.)
>
> I noticed you hadn't listed "permanent" yet, or "durable".
>
> But I do think 'persistent' is the usual term to describe this sort of thing, so
> personally I'd find a way to use it.

Actually, what I wanted to add: If this is to be a persistent storage mechanism
for Emacs, I do think there should be an automatic fall-back option for people
that can't or don't want to use sqlite3, e.g., because they prefer their
persistent storage to be in plain text. Even more so if this persistent storage
is going to be used by 2nd and 3rd party packages (which I assume is the
intention). You don't want to leave unsuspecting users hanging when they install
some package that doesn't work as advertised because it uses a persistent storage
mechanism that doesn't work on the user's installation.

OTOH if no such fall-back is planned, it would IMHO be better to reflect that in
the name of the library and its functions. So preferably `sqlite-defvar` rather
than `persistent-defvar` or something. That would stand a better chance of
alerting package authors to the fact that they cannot count on this
functionality being available on every Emacs install.

Just my €0.02, of course.


-- 
Joost Kremers
Life has its moments

Re: Mutating existing persistent data

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

A defining construct's name should start with `def'.
To define a persistent variable, let's use the word `persistent'.

So let's call it `defvar-persistent'.

I think it is very important to let users specify the use of editable
text files for these values.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Mutating existing persistent data

[Bob Rogers]

   From: Lars Ingebrigtsen <larsi@gnus.org>
   Date: Tue, 07 Dec 2021 21:27:07 +0100

   . . .

   Uhm...  anybody else got any ideas for a word for a variable that has
   a value that survives an Emacs shutdown?

Maybe use "multisession"?  That's a bit of a mouthful, it's more precise
than anything else I've seen, and some extra semantic weight won't hurt.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: sqlite3

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2021-12-06 17:19]:
> Thanks for the paper!
> I think we can safely assume we only need internal objects in
> Emacs — people probably won't use Emacs to handle millions
> of business record (not fit into main memory) in foreseeable future.

Assumption is from your personal view point. I would not rush saying
so. 

Though I am using dynamic module for PostgreSQL database, I am
welcoming inclusion of SQLite into main Emacs for general usage and
access through Emacs Lisp.

I am handling 226,542 people entries through Emacs, 16134 accounts,
1544 various tags, 48 various businesses, 38892 various
hyperdocuments, and I have sent 692,392 mailings to 77,983 people, and
review of it and management goes in breeze. 

With investment of some thousands of dollars, I could easy arrive to
"millions of records". 

And we don't speak of foreseeable future, I use database since 20
years and I have used it over web interface, which I still can do, and
since years I use it through Emacs. We speak of past, not just future.

As related to Dynamic Knowledge Repository, Hyperscope for Emacs, this
below is my statistics of documents. Many people are involved, many
people benefited already from my work through Emacs and database
usage.

┌──────────────────────────────┬───────────────────────┬───────┐
│            Object            │       Sub-type        │ Count │
├──────────────────────────────┼───────────────────────┼───────┤
│ Asciidoctor                  │ Default               │    53 │
│ Asciidoctor                  │ Attachment            │     2 │
│ Asciidoctor                  │ Agreement             │     1 │
│ Asciidoctor with PDF         │ Default               │    15 │
│ Asciidoctor with PDF         │ Agreement             │     7 │
│ Asciidoctor with PDF         │ PDF                   │     6 │
│ Asciidoctor with PDF         │ Task                  │     5 │
│ Asciidoctor with PDF         │ Report                │     3 │
│ Asciidoctor with PDF         │ Partnership Offer     │     2 │
│ Asciidoctor with PDF         │ Attachment            │     2 │
│ Asciidoctor with PDF         │ Quotation             │     1 │
│ Audio                        │ Default               │    61 │
│ Audio                        │ Song                  │    17 │
│ Audio                        │ Voice                 │     1 │
│ AVAILABLE                    │ Default               │     1 │
│ Backlink                     │ Default               │   249 │
│ CBR                          │ Default               │     1 │
│ Common Lisp                  │ Default               │     1 │
│ DB-ID                        │ Default               │     7 │
│ Directory                    │ Default               │    70 │
│ DISEASE                      │ Default               │     5 │
│ DJVU                         │ Default               │    25 │
│ Emacs Lisp                   │ Default               │     4 │
│ Emacs Lisp Hyperlink         │ Default               │    74 │
│ Enriched                     │ Default               │     4 │
│ EPUB                         │ Default               │   119 │
│ EPUB                         │ Recipe                │     1 │
│ File                         │ Default               │   138 │
│ FOLLOW-UP                    │ Default               │    66 │
│ Gnumeric                     │ Spreadsheet           │     3 │
│ HTML Hyperdocument           │ Default               │    24 │
│ HyperScope ID                │ Default               │    13 │
│ Image                        │ Default               │   312 │
│ Image                        │ Food                  │     2 │
│ Image                        │ Recipe                │     1 │
│ Image                        │ Signature             │     1 │
│ Info Node                    │ Default               │     1 │
│ Instruction                  │ Default               │     3 │
│ Instruction                  │ E-mail Snippet        │     1 │
│ Kotl (Hyperbole Outline)     │ Default               │     1 │
│ Launch Program               │ Default               │     6 │
│ Local File                   │ Default               │    25 │
│ Maff                         │ Default               │     2 │
│ Markdown                     │ Default               │  4015 │
│ Markdown                     │ Poem                  │     5 │
│ Markdown (Pandoc, espresso)  │ Default               │    54 │
│ Mbox                         │ Default               │     1 │
│ Media                        │ Default               │    13 │
│ Media                        │ Song                  │     6 │
│ Meeting                      │ Default               │     1 │
│ Message-ID                   │ Default               │     8 │
│ Mobi                         │ Default               │     1 │
│ Mount Point                  │ Default               │     1 │
│ MPV play video at exact time │ Default               │    24 │
│ Note                         │ Default               │  6227 │
│ Note                         │ Report                │    68 │
│ Note                         │ Poem                  │    25 │
│ Note                         │ Request for quotation │    10 │
│ Note                         │ Request For Proposal  │     7 │
│ Note                         │ Quotation             │     6 │
│ Note                         │ Question              │     5 │
│ Note                         │ Cognition             │     4 │
│ Note                         │ Offer                 │     4 │
│ Note                         │ Phone                 │     3 │
│ Note                         │ Proposal              │     3 │
│ Note                         │ E-mail Snippet        │     3 │
│ Note                         │ Progress              │     3 │
│ Note                         │ Template              │     3 │
│ Note                         │ Warning               │     3 │
│ Note                         │ Partnership Offer     │     3 │
│ Note                         │ Dream                 │     2 │
│ Note                         │ Online Account        │     2 │
│ Note                         │ Pricing               │     2 │
│ Note                         │ Call                  │     2 │
│ Note                         │ Recipe                │     2 │
│ Note                         │ Event                 │     1 │
│ Note                         │ PDF                   │     1 │
│ Note                         │ Meeting               │     1 │
│ Note                         │ Contact Information   │     1 │
│ ODS                          │ Default               │     1 │
│ ODT                          │ Default               │     2 │
│ Opportunity                  │ Default               │    37 │
│ Org                          │ Default               │    31 │
│ Org Heading                  │ Default               │     2 │
│ Page in physical book        │ Training              │     1 │
│ PDF                          │ Default               │  2589 │
│ PDF                          │ Book                  │     7 │
│ PDF                          │ Quotation             │     3 │
│ PDF                          │ Report                │     2 │
│ PDF                          │ Recipe                │     1 │
│ PDF                          │ Agreement             │     1 │
│ PDF                          │ Business Profile      │     1 │
│ PDF                          │ PDF                   │     1 │
│ PDF                          │ Contract              │     1 │
│ PDF by Page Nr.              │ Default               │ 14475 │
│ PostgreSQL                   │ Default               │    10 │
│ PostgreSQL                   │ Report                │     6 │
│ QUOTATION                    │ Default               │     3 │
│ Quote                        │ Default               │     6 │
│ Scheme                       │ Default               │     1 │
│ Self                         │ Default               │     3 │
│ Set ➾                        │ Default               │  1706 │
│ Set ➾                        │ Follow-up             │     2 │
│ Set ➾                        │ Recipe                │     1 │
│ Shell Command                │ Default               │     3 │
│ Shell Script                 │ Default               │     1 │
│ SUCCESS                      │ Default               │     1 │
│ Task                         │ Default               │   222 │
│ Task                         │ Call                  │     4 │
│ Task                         │ Travel                │     1 │
│ Task                         │ Training              │     1 │
│ Task                         │ Practical Skill       │     1 │
│ Task                         │ Report                │     1 │
│ Task                         │ Pay                   │     1 │
│ Text                         │ E-mail Signature      │     8 │
│ Text                         │ E-mail Snippet        │     6 │
│ txt2tags                     │ Default               │     1 │
│ URL for Image                │ Default               │    25 │
│ Video                        │ Default               │   128 │
│ Video                        │ Song                  │     1 │
│ WRS Unprocessed Page         │ Default               │     1 │
│ WWW                          │ Default               │  7035 │
│ WWW                          │ PDF                   │    78 │
│ WWW                          │ Recipe                │    20 │
│ WWW                          │ Patent                │    17 │
│ WWW                          │ Video                 │     3 │
│ WWW                          │ Song                  │     3 │
│ WWW                          │ Partnership Offer     │     1 │
│ WWW                          │ Task                  │     1 │
│ WWW                          │ Article               │     1 │
│ WWW                          │ Drug Information      │     1 │
│ WWW                          │ Spreadsheet           │     1 │
│ Xournalpp                    │ Default               │     3 │
│ YouTube Video                │ Default               │   231 │
│ YouTube Video                │ Recipe                │     1 │
│ YouTube Video at exact time  │ Default               │    61 │
└──────────────────────────────┴───────────────────────┴───────┘


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Tim Cross]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I suggested hashmap serialzied to/from file. I used such approach
>> myself for key-value pairs. It worked fine for me, it was quite simple
>> to read/write it. Currently I am testing a thing, and I use just assoc
>> list to read/write it to a file, and it works fine for me too.
>
> I think the ideal choice would either be dbm (and the GNU variant, gdbm,
> which improves upon it in every area) or recutils.
>
> Which still raises the other important question of how to serialize
> data, and how to do it better than `read' and `print'.
>
>> Just because everyone uses sqlite is maybe not the best argument, but
>> anyway, sqlite is maybe faster when serializing that reading/writing
>> lisp objects.  I don't know, I am not familiar so much with elisp vs
>> sqlite. Also, there are many uses for relational databse than just
>> persisting user settings.  I am thinking of desktop applications a lá
>> Access/Excel and similar.
>
> What is special about the settings of MS Access or MS Excel that warrant
> a relational database for storing them?

It probably wouldn't be particularly hard to define a layer which sits
between the Emacs code and the 'database', which could be either sqlite3
or gdbm? users could decide which backend they prefer (either as a
compile time option or possibly even as a user configuration option).

However, I do agree with other comments about simply serialising lisp
objects (i.e. hash maps) probably being sufficient for a majority of
cases. Isn't this essentially what packages like bbdb and applications
like vm (mail) did?

In othe words, I don't know if we are really talking about a wholesale
replacement and move towards sqlite3/gdbm, but instead perhaps providing
an interface to (lets say sqlite3) which could be useful when you do
need a more relational model and limited SQL support.

Re: sqlite3

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I suggested hashmap serialzied to/from file. I used such approach
>> myself for key-value pairs. It worked fine for me, it was quite simple
>> to read/write it. Currently I am testing a thing, and I use just assoc
>> list to read/write it to a file, and it works fine for me too.
>
> I think the ideal choice would either be dbm (and the GNU variant, gdbm,
> which improves upon it in every area) or recutils.
>
> Which still raises the other important question of how to serialize
> data, and how to do it better than `read' and `print'.
>
>> Just because everyone uses sqlite is maybe not the best argument, but
>> anyway, sqlite is maybe faster when serializing that reading/writing
>> lisp objects.  I don't know, I am not familiar so much with elisp vs
>> sqlite. Also, there are many uses for relational databse than just
>> persisting user settings.  I am thinking of desktop applications a lá
>> Access/Excel and similar.
>
> What is special about the settings of MS Access or MS Excel that warrant
> a relational database for storing them?

It is nothing about settings of neither of those. That is not why people use
those applications. They are used to take in and analyze data, collect
research etc.

People use those to create entire applications and to automate office works in
offices, research etc.

Reasons why they choose MS Office varies, but the most important are: economics,
familiarity and ease of use.

For the economics: if you have a big organization, several thousand people,
getting in a new application into the organization can be quite difficult and
costly. In a regional hospital I did some consulting for, anything had to go
through a gigantic IT system with lots of people who all has something important
to say when a decision is made. However, an access db file is just a document,
and MS Office is already used and licensed, so it is very handy to use. I
suggested for one project to replace their research db with sqlite and I
promised to build functionally same gui with TCL, but it was out of question to
install tcl runtime.

The other problem iss familiarity. When you have a personell, in ther
late years, being working there for like 20, 30, 40 years; they are not so very
welcomming to any changes, new tools etc. Adding even a simple button to press
to print out bunch of post addresses on label stickers can be challenging and
need thorough introduction. They simply forgett it was there at all, or where it is,
they have to write a paper how to open the document and lots of such
stuff. Replacing entire application in such system with something else means a
lots of eduction on work hours, hiring consults to held courses etc, so it is
not something big organizations like. 

Also economists love their Excell; I have seen lots of in-house and custom build
tools for analyzing data, repporting, etc. They don't care much about what
technology is used as long as they can do it in Excell because they don't have
time to learn some computer application (or don't care), they don't care about
licensing, software freedom issues etc. Those I have met live just in
completely, completely different world then we, when it comes to computing and
what technology they use.

Re: sqlite3

[Qiantan Hong]

> I suggested hashmap serialzied to/from file. I used such approach myself for
> key-value pairs. It worked fine for me, it was quite simple to read/write
> it. Currently I am testing a thing, and I use just assoc list to read/write it
> to a file, and it works fine for me too.
Did you do it in one-batch, or do you do it incrementally (aka only write the
delta?). AFAIK the former is currently basically the standard practice
(printing/reading an alist or sth similar), and it can get too slow if 
the key value map gets large.


Best,
Qiantan

Re: sqlite3

[Arthur Miller]

Qiantan Hong <qhong@mit.edu> writes:

>> I suggested hashmap serialzied to/from file. I used such approach myself for
>> key-value pairs. It worked fine for me, it was quite simple to read/write
>> it. Currently I am testing a thing, and I use just assoc list to read/write it
>> to a file, and it works fine for me too.
> Did you do it in one-batch, or do you do it incrementally (aka only write the
> delta?). AFAIK the former is currently basically the standard practice
> (printing/reading an alist or sth similar), and it can get too slow if 
> the key value map gets large.
>
I just printed out everything at startup and end of application in a batch. But
that could be definitely done incrementaly in idle timer or something. Also one
could add notify watch so if it changed on disk outisde of emacs, it would be
auto reloaded.

Re: sqlite3

[Yuri Khan]

On Mon, 6 Dec 2021 at 19:39, Arthur Miller <arthur.miller@live.com> wrote:

> I also think that everyone is uses sqlite for persisting user settings because
> they don't have lisp and customize :).

I wanna add a thought here. There are multiple kinds of persistent
data, and they want different backup, versioning and replication
policies.

* Cache. Not backed up, not versioned, and not replicated.
* User configuration. Backed up, versioned, and replicated.
* Machine-specific variable data. Backed up, but probably not
versioned and not much point in replicating.
* User-specific variable data. Backed up, not versioned, replicated.

Emacs treats (mostly) everything as configuration. SQLite treats
everything as variable data. Neither way is a good thing.

Re: sqlite3

[Stefan Monnier]

> I suggested hashmap serialzied to/from file. I used such approach myself for
> key-value pairs. It worked fine for me, it was quite simple to read/write
> it. Currently I am testing a thing, and I use just assoc list to read/write it
> to a file, and it works fine for me too.

The problem with this approach (which is the standard solution currently
in ELisp) is that it works OK for simple cases but breaks down when:
- there's a lot of data.
- the data is modified by several Emacs sessions at the same time.
This second case is what can happen with `ecomplete` or `savehist`.


        Stefan

Re: sqlite3

[Qiantan Hong]

> - the data is modified by several Emacs sessions at the same time.
> This second case is what can happen with `ecomplete` or `savehist`.

It seems that to make it work one always need a single “master” process
that have exclusive access to the file and serves all read/write request.
It can be a database process, but I think an Emacs itself can also perfectly
do the job, (ab)using collaborative editing. Basically all Emacsen
writes to a single collaborative buffer, and the crdt.el server is in charge
of writing out the file. Persistence and concurrent access is thus decoupled.
What do you think?


Best,
Qiantan

Re: sqlite3

[Stefan Monnier]

> It seems that to make it work one always need a single “master” process
> that have exclusive access to the file and serves all read/write request.
> It can be a database process, but I think an Emacs itself can also perfectly
> do the job, (ab)using collaborative editing.

Of course, there are many ways to skin this cat.  I was just pointing
out that the "naive" solution built on read+prin1+write-region doesn't
work well enough if you want it to be robust.  And the solution to the
problem involves a very significant change to the code.


        Stefan

Re: sqlite3

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I suggested hashmap serialzied to/from file. I used such approach myself for
>> key-value pairs. It worked fine for me, it was quite simple to read/write
>> it. Currently I am testing a thing, and I use just assoc list to read/write it
>> to a file, and it works fine for me too.
>
> The problem with this approach (which is the standard solution currently
> in ELisp) is that it works OK for simple cases but breaks down when:
> - there's a lot of data.
> - the data is modified by several Emacs sessions at the same time.
> This second case is what can happen with `ecomplete` or `savehist`.

I wouldn't be surprised, if several processes are involved, if that is what  you
mean with several Emacs sessions at the time, than some locking is needed of
course, and sqlite is a winner there. I believe.

Re: sqlite3

[Arthur Miller]

Arthur Miller <arthur.miller@live.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I suggested hashmap serialzied to/from file. I used such approach myself for
>>> key-value pairs. It worked fine for me, it was quite simple to read/write
>>> it. Currently I am testing a thing, and I use just assoc list to read/write it
>>> to a file, and it works fine for me too.
>>
>> The problem with this approach (which is the standard solution currently
>> in ELisp) is that it works OK for simple cases but breaks down when:
>> - there's a lot of data.
>> - the data is modified by several Emacs sessions at the same time.
>> This second case is what can happen with `ecomplete` or `savehist`.
>
> I wouldn't be surprised, if several processes are involved, if that is what  you
> mean with several Emacs sessions at the time, than some locking is needed of
> course, and sqlite is a winner there. I believe.
By the way, after second thought; sqlite locks entire database. If I remember
well they advice against using sqlite db when concurrent write access is needed.

Does Emacs already has possibility to lock files, or possibly a particular
symbol for read/write access; if not, is such mechanism to much work to add?

And about lots of data; sqlite mmaps db file, if I am not misstaken about, and
Emacs has can mmap files right? How granular and usable is portable dumper? Is
it possible to dump a hashmap or some other object to a file, and read in dump
at runtime?

If those two things would be possible, than there is a fast, multiprocess
key-value store with already existing means.

And I still think sqlite would be useful in core :). For example, easier
importing/exporting with other applications, with some more, 'lispy' interface
if we had it in core.

Re: sqlite3

[Qiantan Hong]

> By the way, after second thought; sqlite locks entire database. If I remember
> well they advice against using sqlite db when concurrent write access is needed.
Seems that its only use is as a persistent hash table then.
I honestly think there’s much easier way to get a persistent hash table
without introducing any dependency. Even just a plain Elisp implementation.

Re: sqlite3

[Arthur Miller]

Qiantan Hong <qhong@mit.edu> writes:

>> By the way, after second thought; sqlite locks entire database. If I remember
>> well they advice against using sqlite db when concurrent write access is needed.
> Seems that its only use is as a persistent hash table then.

Well, it is a relational database for working with structured data and using SQL
in desktop applications. You don't need to persist sqlite db to file
at all. It works perfectly well wiht in-memory databases only. So it is much
more than just key-value database.

You get SQL interpretter/byte-compiler so you can use SQL to work with your
data. But Lars obviously, as I understand from last mails, is thinking of
key-value settings a lá Firefox in Emacs, so I have no idea if he plans to
expose SQL interface or not.

Re: sqlite3

[Qiantan Hong]

> You get SQL interpretter/byte-compiler so you can use SQL to work with your
> data. But Lars obviously, as I understand from last mails, is thinking of
> key-value settings a lá Firefox in Emacs, so I have no idea if he plans to
> expose SQL interface or not.
> 
In that case, it’s unclear to me how much we really get from SQL comparing
to LISt Processing we already have.
If we really benefit from a relational/logical programming language, 
I’d also strongly recommend Prolog over SQL.
Prolog blends better with Lisp (homoiconicity), is strictly more 
expressive (relational calculus is equivalent to a very restricted
subset of Prolog, with no negation, cut, function symbol, and recursion),
and is easily embedded into Lisp.


Best,
Qiantan

Re: sqlite3

[Qiantan Hong]

As for key value store, if you really want a persistent
kv store with O(1) insert, couldn’t we just use something like 
the following super simple (skeleton) code?

(defvar kv-store-table)
(cl-defstruct (kv-store (:constructor make-kv-store-1)) path table)
(defun make-kv-store (path)
  (let* ((kv-store (make-kv-store-1 :path path))
         (kv-store-table (make-hash-table :test 'equal)))
    (when (file-exists-p path)
      (load-file path)
      ;; and some error handling to ignore clear trailing un-balanced form
      ;; just in case Emacs crash in the middle of a kv-put
      )
    (setf (kv-store-table kv-store) kv-store-table)
    kv-store))
(defun compact-kv-store (kv-store)
  ;; dump the full content of kv-store-table at once
  ;; to compress the log and speed up loading
  )
(defun kv-put (key value kv-store)
  (with-temp-buffer
    (print `(puthash ',key ',value kv-store-table) (current-buffer))
    (append-to-file nil nil (kv-store-path kv-store)))
  (puthash key value (kv-store-table kv-store)))
(defun kv-get (key kv-store)
  (gethash key (kv-store-table kv-store)))

Re: sqlite3

[Karl Fogel]

On 06 Dec 2021, Qiantan Hong wrote:
> As for key value store, if you really want a persistent
> kv store with O(1) insert, couldn’t we just use something like 
> the following super simple (skeleton) code?
>
> (defvar kv-store-table)
> (cl-defstruct (kv-store (:constructor make-kv-store-1)) path 
> table)
> (defun make-kv-store (path)
>   (let* ((kv-store (make-kv-store-1 :path path))
>          (kv-store-table (make-hash-table :test 'equal)))
>     (when (file-exists-p path)
>       (load-file path)
>       ;; and some error handling to ignore clear trailing 
>       un-balanced form
>       ;; just in case Emacs crash in the middle of a kv-put
>       )
>     (setf (kv-store-table kv-store) kv-store-table)
>     kv-store))
> (defun compact-kv-store (kv-store)
>   ;; dump the full content of kv-store-table at once
>   ;; to compress the log and speed up loading
>   )
> (defun kv-put (key value kv-store)
>   (with-temp-buffer
>     (print `(puthash ',key ',value kv-store-table) 
>     (current-buffer))
>     (append-to-file nil nil (kv-store-path kv-store)))
>   (puthash key value (kv-store-table kv-store)))
> (defun kv-get (key kv-store)
>   (gethash key (kv-store-table kv-store)))

As a data point:

I wrote a similar generic data-persistence mechanism a while ago, 
because I had exactly the kind of need Lars described in his OP. 
The code has just been living in my .emacs [1].  I never tried to 
package it up as an independent thing, and I'm sure that with more 
& wiser heads we'd easily come up with a better system.  I suspect 
that I'm not the only person who implemented a personal 
persistence mechanism :-).

Just having _one_ persistence mechanism (as opposed to everyone's 
personal improvisation) would be an improvement.  I liked Lars's 
sqlite3 proposal, but it doesn't have to be sqlite3 and I don't 
have a strong enough opinion to argue hard for that versus some 
more Lisp-y solution.

However, whatever the system is, it shouldn't require loading the 
entire data store into memory.  That's a key test that sqlite3 
passes; if we do something else instead, it should also pass that 
test.

Best regards,
-Karl

[1] https://svn.red-bean.com/repos/kfogel/trunk/.emacs, search for
    "A generic cross-session data persistence mechanism."

Re: sqlite3

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> Regardless of whether or not this feature is implemented, I think it
> should be optional, and fallbacks should be in place for systems where
> SQLite is unavailable.

SQLite is everywhere, is it not?  So we add it as a build dependency,
just like we do with other things.

> For example, I tried compiling the latest SQLite amalgamation release
> with DJGPP, and it did not work.  Apparently patches existed to make it
> work, but they do not apply to the latest release anymore.

Is DJGPP relevant outside of the MS-DOS build?

> Customize gaining a hard dependency on SQLite3 would be a disaster!

What disastrous things would happen?

Re: sqlite3

[Po Lu]

Stefan Kangas <stefankangas@gmail.com> writes:

> SQLite is everywhere, is it not?  So we add it as a build dependency,
> just like we do with other things.

Does that imply requiring it for Emacs to build and work?

> Is DJGPP relevant outside of the MS-DOS build?

No, but the MS-DOS build is still relevant.

> What disastrous things would happen?

For example, we would lose the ability to inspect custom files as plain
text.  In many cases, customize has written bad data to my custom file
that prevented Emacs from working, but I could always edit the custom
file manually to get things working again.

This would not be so easy with a binary database.

Re: sqlite3

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> SQLite is everywhere, is it not?  So we add it as a build dependency,
>> just like we do with other things.
>
> Does that imply requiring it for Emacs to build and work?

If we would go this route, I personally wouldn't see it as high priority
to support non-sqlite builds.  But that's me.

>> Is DJGPP relevant outside of the MS-DOS build?
>
> No, but the MS-DOS build is still relevant.

I don't see that, myself.

In any case, AFAIK, the policy is that we provide support for such
platforms to the extent that people volunteer to keep them running.

> For example, we would lose the ability to inspect custom files as plain
> text.  In many cases, customize has written bad data to my custom file
> that prevented Emacs from working, but I could always edit the custom
> file manually to get things working again.
>
> This would not be so easy with a binary database.

As long as we're clear that no one has suggested wholesale replacing
customize with binary data, I agree that this is clearly one of the
drawbacks of SQLite over prin1+write-region.  I don't think it's bad
enough to be catastrophic, however, and there are many upsides as well.

Re: sqlite3

[Jose A. Ortega Ruiz]

On Mon, Dec 06 2021, Po Lu wrote:

[...]

> For example, we would lose the ability to inspect custom files as
> plain text.  In many cases, customize has written bad data to my
> custom file that prevented Emacs from working, but I could always edit
> the custom file manually to get things working again.
>
> This would not be so easy with a binary database.

FWIW, this is also what was worrying me about this idea of moving
customizations to a binary format, be it sqlite or any other.  Sounds to
me exactly like moving from text-based configurations a la Unix to the
Windows Registry, a move that i for one would like to side-step if
possible.

Cheers,
jao
-- 
The fact that we live at the bottom of a deep gravity well, on the
surface of a gas covered planet going around a nuclear fireball 90
million miles away and think this to be normal is obviously some
indication of how skewed our perspective tends to be. -Douglas Adams

Re: sqlite3

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> This comes with questions about how the users are supposed be able to
> clear out the data, for instance, but we could have a `M-x
> list-persistent-data' where the users could blow out whatever they want
> to.

(I'd be in favour of adding support for sqlite3.)

One concern is how to synchronize this between different machines, or
indeed if recommended use should be to do that at all.

Another question we should perhaps make explicit: Will we use one large
sqlite3 file for all user data, or many small ones (e.g. per package)?

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> One concern is how to synchronize this between different machines, or
> indeed if recommended use should be to do that at all.

I think the sqlite format is portable across OS-es and endianness and
stuff?  Let's see...

https://sqlite.org/different.html

----
The SQLite file format is cross-platform. A database file written on one
machine can be copied to and used on a different machine with a
different architecture. Big-endian or little-endian, 32-bit or 64-bit
does not matter. All machines use the same file format. Furthermore, the
developers have pledged to keep the file format stable and backwards
compatible, so newer versions of SQLite can read and write older
database files.
----

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> Another question we should perhaps make explicit: Will we use one large
> sqlite3 file for all user data, or many small ones (e.g. per package)?

I was thinking one large file for the basic key/value storage.  But
packages would be free to make their own files if they want to.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> Another question we should perhaps make explicit: Will we use one large
>> sqlite3 file for all user data, or many small ones (e.g. per package)?
>
> I was thinking one large file for the basic key/value storage.  But
> packages would be free to make their own files if they want to.

Why do you need global storage?

How much settings for any application have so it need to resort to sql instead
of customize?

I see sqlite in core as opportunity to easier work with data produced by other
applications, or to create applications that read in larger sets of data to work
with.

But what you suggest means that now there will be two parallel systems for
storing settings. Are you planing on using sqlite as a backend for customize, or
to remove customize? 

Re: sqlite3

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

> But what you suggest means that now there will be two parallel systems
> for storing settings. Are you planing on using sqlite as a backend for
> customize, or to remove customize?

No, and no, and no.  This won't be used for Customize -- I think I've
said so a number of times now?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> But what you suggest means that now there will be two parallel systems
>> for storing settings. Are you planing on using sqlite as a backend for
>> customize, or to remove customize?
>
> No, and no, and no.  This won't be used for Customize -- I think I've
> said so a number of times now?

I haven't seen you saying that clearly, and I see several people talking about
it, but it's a big thread.

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> sqlite3 is supported on more platforms than Emacs is, the interface is
> small and stable, and (best of all) somebody has already created
> interface functions for Emacs (via a module):
>
>   https://github.com/syohex/emacs-sqlite3

I've now asked the author, Syohei YOSHIDA, if it'd be OK to rewrite this
for inclusion in Emacs core, and the answer was positive, so I've now
done so, and...  it works!  *gasp*

Needs a few more sweeps to clean up all the module-isms.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've now asked the author, Syohei YOSHIDA, if it'd be OK to rewrite this
> for inclusion in Emacs core, and the answer was positive, so I've now
> done so, and...  it works!  *gasp*

It's on the scratch/sqlite branch now, because I wanted to test this on
Macos and Windows...  and...  amazingly enough it seems to work
everywhere without any changes?  That never happens, so I started
wondering whether I was testing this on the wrong system somehow, but
nope.

Don't the Windows things usually require a lot of dll hand-holding, or
am I misremembering which builds need that?  I did a MingW build.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 06 Dec 2021 20:18:01 +0100
> 
> Don't the Windows things usually require a lot of dll hand-holding

Yes, because we don't link against the shared library, to avoid its
being a hard prerequisite.

> I did a MingW build.

Your MinGW build will refuse to start if the SQLite DLL is not
available, which is something we'd like to avoid, because it allows
people to decide which optional libraries they install with an Emacs
binary which someone else compiled.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Your MinGW build will refuse to start if the SQLite DLL is not
> available, which is something we'd like to avoid, because it allows
> people to decide which optional libraries they install with an Emacs
> binary which someone else compiled.

So to test this I should move the sqlite DLL somewhere else (after
building) and see whether it still works, or is there an easier way to
make the DLL loading fail?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Óscar Fuentes]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Your MinGW build will refuse to start if the SQLite DLL is not
>> available, which is something we'd like to avoid, because it allows
>> people to decide which optional libraries they install with an Emacs
>> binary which someone else compiled.
>
> So to test this I should move the sqlite DLL somewhere else (after
> building) and see whether it still works, or is there an easier way to
> make the DLL loading fail?

Just rename or move away the DLL and make sure that there is no another
dll instance on the directories listed on PATH.

However, if sqlite support is supposed to be provided and used by Emacs
core, failing when the dll is absent is a good thing.

Re: sqlite3

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 07 Dec 2021 22:25:05 +0100
> 
> However, if sqlite support is supposed to be provided and used by Emacs
> core, failing when the dll is absent is a good thing.

Not if it's an optional library that supports optional features.

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 07 Dec 2021 21:41:38 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Your MinGW build will refuse to start if the SQLite DLL is not
> > available, which is something we'd like to avoid, because it allows
> > people to decide which optional libraries they install with an Emacs
> > binary which someone else compiled.
> 
> So to test this I should move the sqlite DLL somewhere else (after
> building) and see whether it still works

Move or rename it, yes.

> or is there an easier way to make the DLL loading fail?

You could start a new Command Prompt window, change PATH in that
window so that it doesn't include the directory where the sqlite DLL
lives, then start Emacs from that window.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> You could start a new Command Prompt window, change PATH in that
> window so that it doesn't include the directory where the sqlite DLL
> lives, then start Emacs from that window.

Right.

I've now cargo culted the Windows-specific invocations from xml.c into
sqlite.c, and it compiles...  but invoking any of the sqlite functions
now makes Emacs segfault.  Which isn't quite the result I was hoping
for.  :-/

And running it under gdb is apparently not very helpful, but I've never
run gdb under Windows...

Can you have a look?  It's probably obvious to you what's going wrong.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 08 Dec 2021 08:00:58 +0100
> 
> I've now cargo culted the Windows-specific invocations from xml.c into
> sqlite.c, and it compiles...  but invoking any of the sqlite functions
> now makes Emacs segfault.  Which isn't quite the result I was hoping
> for.  :-/
> 
> And running it under gdb is apparently not very helpful, but I've never
> run gdb under Windows...

The MinGW port of GDB works very well on Windows, I use it all the
time.  What problems did you see?

> Can you have a look?  It's probably obvious to you what's going wrong.

I see two problems:

  . Emacs is linked against the sqlite3 DLL statically, because
    configure.ac unconditionally adds -lsqlite3 to the link command
    line.  This -lsqlite3 should be removed in the Windows build,
    since we will be loading the library at run time by an explicit
    call to the moral equivalent of 'dlopen'.  See how this is done
    in configure.ac for other optional libraries.

  . You didn't add the necessary stuff to dynamic-library-alist in
    w32-win.el.  Without that, Emacs doesn't know which DLL to try to
    load.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> And running it under gdb is apparently not very helpful, but I've never
>> run gdb under Windows...
>
> The MinGW port of GDB works very well on Windows, I use it all the
> time.  What problems did you see?

Emacs crashed under gdb, but "bt" didn't give me anything.  (But after
the latest changes, Emacs doesn't crash any more, so I haven't looked
any further.)

>> Can you have a look?  It's probably obvious to you what's going wrong.
>
> I see two problems:
>
>   . Emacs is linked against the sqlite3 DLL statically, because
>     configure.ac unconditionally adds -lsqlite3 to the link command
>     line.  This -lsqlite3 should be removed in the Windows build,
>     since we will be loading the library at run time by an explicit
>     call to the moral equivalent of 'dlopen'.  See how this is done
>     in configure.ac for other optional libraries.
>
>   . You didn't add the necessary stuff to dynamic-library-alist in
>     w32-win.el.  Without that, Emacs doesn't know which DLL to try to
>     load.

Thanks; with those two changes, the sqlite branch now seems to work fine
dynamically loaded under mingw, but you should probably have a look at
this, as this is the very first time I've done this stuff.

So I'll be merging this part of the branch with the trunk in a couple of
days, but not the persistent storage bit, because I'm still not
completely satisfied with the naming.  But (multisession-value ...)
seems to be the contender for now.

(And the documentation probably needs to be expanded a bit.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 10 Dec 2021 02:19:04 +0100
> 
> > The MinGW port of GDB works very well on Windows, I use it all the
> > time.  What problems did you see?
> 
> Emacs crashed under gdb, but "bt" didn't give me anything.

This could happen if the callstack is smashed.

> >   . Emacs is linked against the sqlite3 DLL statically, because
> >     configure.ac unconditionally adds -lsqlite3 to the link command
> >     line.  This -lsqlite3 should be removed in the Windows build,
> >     since we will be loading the library at run time by an explicit
> >     call to the moral equivalent of 'dlopen'.  See how this is done
> >     in configure.ac for other optional libraries.
> >
> >   . You didn't add the necessary stuff to dynamic-library-alist in
> >     w32-win.el.  Without that, Emacs doesn't know which DLL to try to
> >     load.
> 
> Thanks; with those two changes, the sqlite branch now seems to work fine
> dynamically loaded under mingw, but you should probably have a look at
> this, as this is the very first time I've done this stuff.

Will do, when the code lands on master.

Thanks.

Re: sqlite3

[Eli Zaretskii]

> Date: Fri, 10 Dec 2021 10:38:55 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Thanks; with those two changes, the sqlite branch now seems to work fine
> > dynamically loaded under mingw, but you should probably have a look at
> > this, as this is the very first time I've done this stuff.
> 
> Will do, when the code lands on master.

Now done.

A couple of issues that caught my eye while reading the code:

 . Should we invoke encode_string_utf_8 with 2 last arguments Qnil,
   and signal an error if the result is nil?  That would make sure we
   were passed a valid UTF-8 string.  We could also use non-zero 3rd
   argument, for speed.

 . sqlite-load-extension expects a file name with an extension, but
   that would leas to Lisp code conditioning on system-type to use the
   correct extension.  Should we instead append the extension inside
   the function?  Also, are SQLite extension modules usually installed
   in some known directory, or using some PATH-style list of
   directories?  If they are, we could perhaps use 'load-'-style path
   variable?

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> A couple of issues that caught my eye while reading the code:
>
>  . Should we invoke encode_string_utf_8 with 2 last arguments Qnil,
>    and signal an error if the result is nil?  That would make sure we
>    were passed a valid UTF-8 string.  We could also use non-zero 3rd
>    argument, for speed.

Let's see...  it's these two?

   If HANDLE-8-BIT is Qt, encode eight-bit characters into single bytes
   of the same value, like the usual Emacs encoding does.

   If HANDLE-OVER-UNI is Qt, encode characters beyond the Unicode
   range into the same 4 or 5-byte sequence as used by Emacs
   internally, like the usual Emacs encoding does.

I think sqlite3 will give us whatever bytes have been stuffed into the
file, so I think Qt is the most useful values here.  (Unless I'm missing
something.)  That is, sqlite3 doesn't enforce the charset for the data,
but it does use the charset when interpreting the data (for instance, if
you select for 'ó < ź').

I have not read the documentation in detail on this issue, so if this is
mistaken, somebody please correct me.

>  . sqlite-load-extension expects a file name with an extension, but
>    that would leas to Lisp code conditioning on system-type to use the
>    correct extension.  Should we instead append the extension inside
>    the function?  Also, are SQLite extension modules usually installed
>    in some known directory, or using some PATH-style list of
>    directories?  If they are, we could perhaps use 'load-'-style path
>    variable?

The extension loading thing was added as an afterthought, and is perhaps
not vital in an Emacs context, so perhaps we should reevaluate the
inclusion of that function.

But to answer your question, a module is currently loaded by absolute
name, and is typically "/usr/lib/sqlite3/pcre.so" (on this Debian
system, at least), and comes from the sqlite3-pcre package.  But I think
there's a lot of different extensions floating around out there, so if
we decide to keep the facility, adding something PATH-like might be
nice, and figuring out the .so/DLL thing by itself might also be nice.

So we'd have (sqlite-load-extension db 'pcre), perhaps, instead of the
absolute file name.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 12 Dec 2021 05:21:48 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > A couple of issues that caught my eye while reading the code:
> >
> >  . Should we invoke encode_string_utf_8 with 2 last arguments Qnil,
> >    and signal an error if the result is nil?  That would make sure we
> >    were passed a valid UTF-8 string.  We could also use non-zero 3rd
> >    argument, for speed.
> 
> Let's see...  it's these two?
> 
>    If HANDLE-8-BIT is Qt, encode eight-bit characters into single bytes
>    of the same value, like the usual Emacs encoding does.
> 
>    If HANDLE-OVER-UNI is Qt, encode characters beyond the Unicode
>    range into the same 4 or 5-byte sequence as used by Emacs
>    internally, like the usual Emacs encoding does.
> 
> I think sqlite3 will give us whatever bytes have been stuffed into the
> file, so I think Qt is the most useful values here.  (Unless I'm missing
> something.)  That is, sqlite3 doesn't enforce the charset for the data,
> but it does use the charset when interpreting the data (for instance, if
> you select for 'ó < ź').
> 
> I have not read the documentation in detail on this issue, so if this is
> mistaken, somebody please correct me.

The documentation says "UTF-8", and my reading of that is that
anything outside of that is likely to trigger "undefined behavior".
There's also a question of whether we, the Emacs project, want to
force using only valid UTF-8 in those interfaces.

A related issue is with encoding file names, which are also expected
to be in UTF-8.  I don't know what will happen if a file name includes
eight-bit raw bytes, for example.  At least in the Windows code in
sqlite3, which I did read, the result will probably be a failure to
access the file.

> But to answer your question, a module is currently loaded by absolute
> name, and is typically "/usr/lib/sqlite3/pcre.so" (on this Debian
> system, at least), and comes from the sqlite3-pcre package.  But I think
> there's a lot of different extensions floating around out there, so if
> we decide to keep the facility, adding something PATH-like might be
> nice, and figuring out the .so/DLL thing by itself might also be nice.
> 
> So we'd have (sqlite-load-extension db 'pcre), perhaps, instead of the
> absolute file name.

OK, let's revisit this if and when we decide that
sqlite-load-extension will stay in Emacs.  Right now, it looks there
are several non-trivial issues with it.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The documentation says "UTF-8", and my reading of that is that
> anything outside of that is likely to trigger "undefined behavior".
> There's also a question of whether we, the Emacs project, want to
> force using only valid UTF-8 in those interfaces.

As you can see from the tests, putting binary data into a TEXT field
works fine, so I think the documentation might be underdefined.

(And I think we do want to be able to put binary data into the
database.)

> A related issue is with encoding file names, which are also expected
> to be in UTF-8.  I don't know what will happen if a file name includes
> eight-bit raw bytes, for example.  At least in the Windows code in
> sqlite3, which I did read, the result will probably be a failure to
> access the file.

Yeah, file names are a separate issue, and we should encode them in the
usual ways we do.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 12 Dec 2021 08:48:22 +0100
> 
> > A related issue is with encoding file names, which are also expected
> > to be in UTF-8.  I don't know what will happen if a file name includes
> > eight-bit raw bytes, for example.  At least in the Windows code in
> > sqlite3, which I did read, the result will probably be a failure to
> > access the file.
> 
> Yeah, file names are a separate issue, and we should encode them in the
> usual ways we do.

No, the usual way is to use ENCODE_FILE, but sqlite3 explicitly wants
file names encoded in UTF-8 regardless of the locale's encoding,
AFAIU.

So the question is: do we tolerate eight-bit raw bytes in file names
passed to sqlite3?

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> No, the usual way is to use ENCODE_FILE, but sqlite3 explicitly wants
> file names encoded in UTF-8 regardless of the locale's encoding,
> AFAIU.

Oh, wow.  That's weird.  Is that common?  I can't recall seeing anything
like that with other libraries, but perhaps I'm just misremembering.

> So the question is: do we tolerate eight-bit raw bytes in file names
> passed to sqlite3?

If this is really the case (and the documentation isn't just glossing
over details), then I guess we don't tolerate that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> So the question is: do we tolerate eight-bit raw bytes in file names
>> passed to sqlite3?
>
> If this is really the case (and the documentation isn't just glossing
> over details), then I guess we don't tolerate that?

I did some googling, and it seems like the documentation is correct
here -- the file name passed to sqlite3_open_v2 should indeed be utf-8 only.

(There are other variants of the open* functions that do other things,
but using them have other side effects.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 12 Dec 2021 10:33:56 +0100
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> >> So the question is: do we tolerate eight-bit raw bytes in file names
> >> passed to sqlite3?
> >
> > If this is really the case (and the documentation isn't just glossing
> > over details), then I guess we don't tolerate that?
> 
> I did some googling, and it seems like the documentation is correct
> here -- the file name passed to sqlite3_open_v2 should indeed be utf-8 only.

The documentation says so, yes.  But I looked inside the sqlite3
source, and I'm now confused: it seems like the Unix code path there
just uses the file name passed to the library without any checks or
changes?  The Windows code does convert from UTF-8 to the wchar_t
(a.k.a. UTF-16) encoding that is native to Windows.

So it sounds like we should use ENCODE_FILE after all?

I asked a question about this on the SQLite forum, let's see what the
gurus there tell.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The documentation says so, yes.  But I looked inside the sqlite3
> source, and I'm now confused: it seems like the Unix code path there
> just uses the file name passed to the library without any checks or
> changes?  The Windows code does convert from UTF-8 to the wchar_t
> (a.k.a. UTF-16) encoding that is native to Windows.
>
> So it sounds like we should use ENCODE_FILE after all?

ENCODE_FILE on everything but Windows, but encode as utf-8 on Windows, it
sounds like?  Pretty confusing interface.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 13 Dec 2021 05:05:46 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The documentation says so, yes.  But I looked inside the sqlite3
> > source, and I'm now confused: it seems like the Unix code path there
> > just uses the file name passed to the library without any checks or
> > changes?  The Windows code does convert from UTF-8 to the wchar_t
> > (a.k.a. UTF-16) encoding that is native to Windows.
> >
> > So it sounds like we should use ENCODE_FILE after all?
> 
> ENCODE_FILE on everything but Windows, but encode as utf-8 on Windows, it
> sounds like?  Pretty confusing interface.

ENCODE_FILE already uses UTF-8 on MS-Windows, so there's no problem.

The answers to my question mostly miss the point, but my take from
them is that indeed the UTF-8 requirement for file names is only
relevant for MS-Windows, whereas on Posix hosts sqlite3 just uses the
byte sequence we hand to it.  You can see the answers here:

  https://sqlite.org/forum/forumpost/02798ac965

So I'll be changing the couple of places where we pass file names to
sqlite3 to use ENCODE_FILE.

Re: sqlite3

[Eli Zaretskii]

> Date: Mon, 13 Dec 2021 14:49:03 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> So I'll be changing the couple of places where we pass file names to
> sqlite3 to use ENCODE_FILE.

Now done.

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 06 Dec 2021 20:18:01 +0100
> 
> It's on the scratch/sqlite branch now, because I wanted to test this on
> Macos and Windows...

From what I see there, you completely ignore the text encoding issue.
SQLite needs UTF-8, but our internal representation is a superset of
UTF-8, so we need to decide what to do with the stuff that's beyond
UTF-8.  Does SQLite check this and return an error indication if the
byte sequence we feed it is not a valid UTF-8?  If it does check that,
we can rely on it, but we need to signal an error with a clear
explanation when it detects something like that.  Alternatively, we
should check this ourselves before calling SQLite, and signal an error
on our own.

One other comment is the use of fixnums for integer values.  Since
SQLite can use the full 64-bit width, I guess we need to use bignums
when the value exceeds a fixnum?

Thanks.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>>From what I see there, you completely ignore the text encoding issue.
> SQLite needs UTF-8, but our internal representation is a superset of
> UTF-8, so we need to decide what to do with the stuff that's beyond
> UTF-8.  Does SQLite check this and return an error indication if the
> byte sequence we feed it is not a valid UTF-8?  If it does check that,
> we can rely on it, but we need to signal an error with a clear
> explanation when it detects something like that.  Alternatively, we
> should check this ourselves before calling SQLite, and signal an error
> on our own.

Good catch.  😀  I was scratching my head about that for about ten
minutes -- did I forget to put a "POSSIBLY FIXME" in there?

SQLite does expect UTF-8, but as far as I can tell, it doesn't really
check anything.  So the current code puts our private binary
utf-8-private-plane-encoded things into SQLite -- and gets it back as
is.  (See the sqlite-char test for this stuff.)

That is, 

"insert into test2 (col1, col2) values ('fó\x82o', 3)"

and

"insert into test2 (col1, col2) values ('f\x82o', 3)"

both round-trip without any problems as is.  Perhaps SQlite just sees it
as a byte string?

So I wasn't quite sure what to do here.

> One other comment is the use of fixnums for integer values.  Since
> SQLite can use the full 64-bit width, I guess we need to use bignums
> when the value exceeds a fixnum?

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Perhaps SQlite just sees it as a byte string?

Indeed it does, according to some Q&A's from the sqlite people.

So this happens to work because it's Emacs on both sides, but if
somebody else has put binary data into the database, it'll make Emacs
blow up.  So we do indeed need to encode/decode in src/sqlite.c for
strings.  I'll fix that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 06 Dec 2021 20:51:30 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >>From what I see there, you completely ignore the text encoding issue.
> > SQLite needs UTF-8, but our internal representation is a superset of
> > UTF-8, so we need to decide what to do with the stuff that's beyond
> > UTF-8.  Does SQLite check this and return an error indication if the
> > byte sequence we feed it is not a valid UTF-8?  If it does check that,
> > we can rely on it, but we need to signal an error with a clear
> > explanation when it detects something like that.  Alternatively, we
> > should check this ourselves before calling SQLite, and signal an error
> > on our own.
> 
> Good catch.  😀  I was scratching my head about that for about ten
> minutes -- did I forget to put a "POSSIBLY FIXME" in there?
> 
> SQLite does expect UTF-8, but as far as I can tell, it doesn't really
> check anything.  So the current code puts our private binary
> utf-8-private-plane-encoded things into SQLite -- and gets it back as
> is.  (See the sqlite-char test for this stuff.)

If we want to do better, we need to call encode_string_utf_8, it
already has all the infrastructure for this case, including fast
no-copy operation if everything checks out, and a telltale return
value if not (so we could signal an error).

> That is, 
> 
> "insert into test2 (col1, col2) values ('fó\x82o', 3)"
> 
> and
> 
> "insert into test2 (col1, col2) values ('f\x82o', 3)"
> 
> both round-trip without any problems as is.  Perhaps SQlite just sees it
> as a byte string?

As long as you don't ask it to search inside text, it probably doesn't
care.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If we want to do better, we need to call encode_string_utf_8, it
> already has all the infrastructure for this case, including fast
> no-copy operation if everything checks out, and a telltale return
> value if not (so we could signal an error).

I wasn't sure whether this was true:

   This function is optimized for speed when the input string is
   already a valid UTF-8 sequence, i.e. there are neither 8-bit raw
   bytes nor any UTF-8 sequences longer than 4 bytes in the string's
   contents.

So I went with the slower, more general code_convert_string_norecord,
but perhaps that can be redone.  I have to do some more testing to see
what's actually landing in the database.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 06 Dec 2021 21:20:06 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If we want to do better, we need to call encode_string_utf_8, it
> > already has all the infrastructure for this case, including fast
> > no-copy operation if everything checks out, and a telltale return
> > value if not (so we could signal an error).
> 
> I wasn't sure whether this was true:
> 
>    This function is optimized for speed when the input string is
>    already a valid UTF-8 sequence, i.e. there are neither 8-bit raw
>    bytes nor any UTF-8 sequences longer than 4 bytes in the string's
>    contents.

??? This says that if the string is a valid UTF-8, the function will
work very fast, so what's the problem?

> So I went with the slower, more general code_convert_string_norecord,

I think this is overkill.  encode_string_utf_8 can be called in a way
that it will return the original string if it's valid UTF-8, and
return nil if not (in which case you signal an error).  It was
designed for these situations, so it would be a pity not to use it.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>>    This function is optimized for speed when the input string is
>>    already a valid UTF-8 sequence, i.e. there are neither 8-bit raw
>>    bytes nor any UTF-8 sequences longer than 4 bytes in the string's
>>    contents.
>
> ??? This says that if the string is a valid UTF-8, the function will
> work very fast, so what's the problem?

I'm not sure whether what we're getting from sqlite is valid UTF-8.
(This was in the opposite direction; decode_string_utf_8.)  But the
other direction should be OK so long as we're checking that it's not a
unibyte >128-char string.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 06 Dec 2021 21:35:47 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >>    This function is optimized for speed when the input string is
> >>    already a valid UTF-8 sequence, i.e. there are neither 8-bit raw
> >>    bytes nor any UTF-8 sequences longer than 4 bytes in the string's
> >>    contents.
> >
> > ??? This says that if the string is a valid UTF-8, the function will
> > work very fast, so what's the problem?
> 
> I'm not sure whether what we're getting from sqlite is valid UTF-8.
> (This was in the opposite direction; decode_string_utf_8.)  But the
> other direction should be OK so long as we're checking that it's not a
> unibyte >128-char string.

Yes, I was talking about the other direction, from Emacs to sqlite,
i.e. encoding.  For decoding stuff from sqlite to Emacs, I agree that
code_convert_string_norecord is the right approach, in case someone
put non-UTF-8 bytes in the DB.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, I was talking about the other direction, from Emacs to sqlite,
> i.e. encoding.  For decoding stuff from sqlite to Emacs, I agree that
> code_convert_string_norecord is the right approach, in case someone
> put non-UTF-8 bytes in the DB.

I've added some more tests and stuff, and encode_string_utf_8 does
indeed seem to do the right thing, so I've now pushed that bit.

I'm still wondering about this bit:

      else if (EQ (type, Qinteger))
	/* FIXME: Bignums? */
	ret = sqlite3_bind_int64 (stmt, i + 1, XFIXNUM (value));

On a 32-bit system, this will truncate the value, I think?  What's the
correct incantation here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 07 Dec 2021 06:46:08 +0100
> 
>       else if (EQ (type, Qinteger))
> 	/* FIXME: Bignums? */
> 	ret = sqlite3_bind_int64 (stmt, i + 1, XFIXNUM (value));
> 
> On a 32-bit system, this will truncate the value, I think?  What's the
> correct incantation here?

You mean, what to use instead of XFIXNUM if 'value' is a BIGNUMP?
bignum_to_intmax, I think.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> You mean, what to use instead of XFIXNUM if 'value' is a BIGNUMP?
> bignum_to_intmax, I think.

Thanks; I've now done that, and tests pass (but I don't have access to a
32-bit system to test there).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[dick]

EZ> issue.  SQLite needs UTF-8, but our internal representation is a
EZ> superset of UTF-8

Why you feel the need to raise a 0.001% exception to a project that has
thus far gotten 0.000% off the ground is the very soul of premature
optimization.

As with the interminable forge-versus-email thread, this thread is
worthless without a PoC implementation.

(FWIW, I'm in the store-things-as-ascii-s-expressions camp).

Re: sqlite3

[Eli Zaretskii]

> From: dick <dick.r.chiang@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  emacs-devel@gnu.org
> Date: Mon, 06 Dec 2021 15:02:42 -0500
> 
> EZ> issue.  SQLite needs UTF-8, but our internal representation is a
> EZ> superset of UTF-8
> 
> Why you feel the need to raise a 0.001% exception to a project that has
> thus far gotten 0.000% off the ground is the very soul of premature
> optimization.

It isn't an optimization, it is standard Emacs operation: libraries
that require UTF-8 should be protected from our extensions to UTF-8.

Re: sqlite3

[Alan Mackenzie]

Hello, Lars.

On Mon, Dec 06, 2021 at 02:51:33 +0100, Lars Ingebrigtsen wrote:
> I think many user experience things in Emacs would be better if Emacs
> remembered more.

> The bigger systems don't have this problem -- Gnus needs a large
> .newsrc.eld file, and it maintains that.

> The tiny things don't really have this problem, either: You save options
> with `customize-save-variable', and that fine.

> It's the many things that fall between these two extremes that have the
> problem: Where you want to store some state, but figuring it's just too
> much work to figure out Yet Another Storage Format, but the data is too
> messy to stash in somebody's .emacs file via Customize (like lists of
> stuff).

> So there's a bunch of stuff that Emacs just forgets when you shut down,
> where it perhaps shouldn't.

> I've brought this up before, but I didn't really have a solution then,
> but I think I do now: sqlite3.

YUCK!!

That would break 45 years of Emacs tradition and practice.  In Emacs,
data are stored as PLAIN TEXT.  Even .elc files are pretty much plain
text.

That is, text that you can examine with standard tools, such as grep,
awk, perl, less, ...., and C-s and friends within Emacs.

You seem to be suggesting moving plain text data to an opaque format,
which can only be examined with special purpose tools.

> sqlite3 is supported on more platforms than Emacs is, the interface is
> small and stable, and (best of all) somebody has already created
> interface functions for Emacs (via a module):

>   https://github.com/syohex/emacs-sqlite3

So, in addition to learning ELisp and C, and the tools mentioned above,
one will now have to learn sqlite3 in order to make full use of Emacs?

Firefox changed some years ago to having its store of visited sites as
an opaque format.  That's been entirely a loss for me, since I can no
longer examine and edit it with any tool other than Firefox itself.

Please don't do the same to Emacs.

> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.  That is:

> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)

> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.

Emacs has that already.  It's called an alist.  It becomes persistent
when you print it to a file, not a challenging project.

> In addition, if somebody really wants to write SQL stuff (it can be very
> handy for some things), having sqlite3 in there gives us that in
> addition for free.

If somebody wants to develop an SQL application, that's fine.  But you
seem to be talking about turning Emacs _into_ an SQL application.
That's anything but fine.

> This comes with questions about how the users are supposed be able to
> clear out the data, for instance, but we could have a `M-x
> list-persistent-data' where the users could blow out whatever they want
> to.

Such a data store would be an opaque format.  Please don't.  Just don't.

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: sqlite3

[Qiantan Hong]

> YUCK!!
> 
> That would break 45 years of Emacs tradition and practice.  In Emacs,
> data are stored as PLAIN TEXT.  Even .elc files are pretty much plain
> text.
Or the way I’d like to put it, data are stored as S-EXP.
> That is, text that you can examine with standard tools, such as grep,
> awk, perl, less, ...., and C-s and friends within Emacs.
So you can read, eval, print them, edit them structurally, and
look at the beautiful and highly readable parentheses.
I found myself read C 20 times slower than Lisp, not mentioning
hexademicals!

> Emacs has that already.  It's called an alist.  It becomes persistent
> when you print it to a file, not a challenging project.
Same thoughts. But jokes aside, they do have a valid point,
which is reading/printing a big alist can get slow.
I totally don’t see the need to introduce database to
solve this problem though. I hereby call for a plain Elisp
implementation of a log-structured object(or, CONSes) store.


Best,
Qiantan

Re: sqlite3

[Eric Abrahamsen]

Qiantan Hong <qhong@mit.edu> writes:

>> YUCK!!
>> 
>> That would break 45 years of Emacs tradition and practice.  In Emacs,
>> data are stored as PLAIN TEXT.  Even .elc files are pretty much plain
>> text.
> Or the way I’d like to put it, data are stored as S-EXP.
>> That is, text that you can examine with standard tools, such as grep,
>> awk, perl, less, ...., and C-s and friends within Emacs.
> So you can read, eval, print them, edit them structurally, and
> look at the beautiful and highly readable parentheses.
> I found myself read C 20 times slower than Lisp, not mentioning
> hexademicals!
>
>> Emacs has that already.  It's called an alist.  It becomes persistent
>> when you print it to a file, not a challenging project.
> Same thoughts. But jokes aside, they do have a valid point,
> which is reading/printing a big alist can get slow.
> I totally don’t see the need to introduce database to
> solve this problem though. I hereby call for a plain Elisp
> implementation of a log-structured object(or, CONSes) store.

Scanning the (very long) thread, I don't think anyone is proposing using
sqlite *by default* for any Emacs data storage, least of all the
customization interface. Customize is the perfect poster child for the
advantages of plain-text, Emacs-readable data formats. As others have
said, sqlite is great for very large stores of data, particularly if
data retrieval is performance-sensitive.

I'm excited about this only because, somewhat ironically, only external
packages can require a sqlite library and make use of it -- it's not
available for built-in libraries.

Re: sqlite3

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> So, in addition to learning ELisp and C, and the tools mentioned above,
> one will now have to learn sqlite3 in order to make full use of Emacs?

No, you won't.  The key/value store will be available via a simple
accessor. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 1545 bytes --]

* 2021-12-06 02:51:33+0100, Lars Ingebrigtsen wrote:

> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.

> In addition, if somebody really wants to write SQL stuff (it can be
> very handy for some things), having sqlite3 in there gives us that in
> addition for free.

Excellent! Real SQL and relational database is useful and needed in some
situations but a simple Lispy key-value storage is nice and enough for
most use-cases.

> This comes with questions about how the users are supposed be able to
> clear out the data, for instance, but we could have a `M-x
> list-persistent-data' where the users could blow out whatever they
> want to.

From SQLite's point of view, by default, it only grows database files.
Deleted data in tables are just marked free and the space is filled
later by a new data.

I think Emacs users would want automatic VACUUM so that they never need
to worry about it. (1) One option for that is to set SQLite's

    PRAGMA auto_vacuum = FULL;

before any tables are created in database. (2) Another options is to
maintain a modify counter in the database and issue SQL's VACUUM when
the number of modifications exceeds some limit. It's actually good idea
to run vacuum from time to time anyway, even if "auto_vacuum = FULL". It
fixes possible defragmentation of database.

See: https://sqlite.org/pragma.html#pragma_auto_vacuum

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 434 bytes --]

Re: sqlite3

[Lars Ingebrigtsen]

Teemu Likonen <tlikonen@iki.fi> writes:

> I think Emacs users would want automatic VACUUM so that they never need
> to worry about it. (1) One option for that is to set SQLite's
>
>     PRAGMA auto_vacuum = FULL;
>
> before any tables are created in database.

Thanks; that seems like an easy enough solution, so I've added it to
sticky.el.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]

* 2021-12-07 07:54:23+0100, Lars Ingebrigtsen wrote:

> Teemu Likonen <tlikonen@iki.fi> writes:
>
>> I think Emacs users would want automatic VACUUM so that they never need
>> to worry about it. (1) One option for that is to set SQLite's
>>
>>     PRAGMA auto_vacuum = FULL;
>>
>> before any tables are created in database.
>
> Thanks; that seems like an easy enough solution, so I've added it to
> sticky.el.

That should be enough. In this early stage of development (and without
seeing the code myself) I would also suggest or remind of adding a
metadata table and version number of the database structure, like this:

    CREATE TABLE metadata (key TEXT PRIMARY KEY NOT NULL,
                          intvalue INTEGER,
                          textvalue TEXT);

    INSERT INTO metadata (key, intvalue) VALUES ('version', 1);

In my experience there will _always_ come a need to change the database
structure. A version number metadata makes it easier to perform upgrades
correctly. Other metadata may be needed later too.

In SQL standard, PRIMARY KEY implies NOT NULL, but this is not the case
in SQlite. It has to be added explicitly, thus PRIMARY KEY NOT NULL (see
above).

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 434 bytes --]

Re: sqlite3

[Sergey Organov]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think many user experience things in Emacs would be better if Emacs
> remembered more.
>
> The bigger systems don't have this problem -- Gnus needs a large
> .newsrc.eld file, and it maintains that.
>
> The tiny things don't really have this problem, either: You save options
> with `customize-save-variable', and that fine.
>
> It's the many things that fall between these two extremes that have the
> problem: Where you want to store some state, but figuring it's just too
> much work to figure out Yet Another Storage Format, but the data is too
> messy to stash in somebody's .emacs file via Customize (like lists of
> stuff).
>
> So there's a bunch of stuff that Emacs just forgets when you shut down,
> where it perhaps shouldn't.

Sure thing.

>
> I've brought this up before, but I didn't really have a solution then,
> but I think I do now: sqlite3.

Personally, I'd prefer to keep my configuration/state files in plain
text, editable by Emacs.

Would it be feasible to have common state save/restore interface for
Emacs progrms/packages to use, and then configurable back-end to handle
actual persistence, being sqlite3, dbm, plain (elisp) text, and what
not?

Thanks,
-- Sergey Organov

Re: sqlite3

[Qiantan Hong]

>> I've brought this up before, but I didn't really have a solution then,
>> but I think I do now: sqlite3.
> 
> Personally, I'd prefer to keep my configuration/state files in plain
> text, editable by Emacs.
> 
> Would it be feasible to have common state save/restore interface for
> Emacs progrms/packages to use, and then configurable back-end to handle
> actual persistence, being sqlite3, dbm, plain (elisp) text, and what
> not?
I’ve attached a plain Elisp incremental implementation in
https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00646.html
Interface is simply
make-kv-store (path)
compact-kv-store (kv-store)
kv-put (key value kv-store)
kv-get (key kv-store)

But honestly, I suspect using any db as backend for a persistent lisp
store will have virtually no advantage over plain text.
Then why even introduce them at the first place.

The more exciting thing will be to have a general object store
rather than kv-store (aka, incremental for arbitrary mutation
and preserve sharing). Any relational db will not help that either,
and it is better to be done in pure Lisp.

Best,
Qiantan

Re: sqlite3

[Sergey Organov]

Qiantan Hong <qhong@mit.edu> writes:

>>> I've brought this up before, but I didn't really have a solution then,
>>> but I think I do now: sqlite3.
>> 
>> Personally, I'd prefer to keep my configuration/state files in plain
>> text, editable by Emacs.
>> 
>> Would it be feasible to have common state save/restore interface for
>> Emacs progrms/packages to use, and then configurable back-end to handle
>> actual persistence, being sqlite3, dbm, plain (elisp) text, and what
>> not?
> I’ve attached a plain Elisp incremental implementation in
> https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00646.html
> Interface is simply
> make-kv-store (path)
> compact-kv-store (kv-store)
> kv-put (key value kv-store)
> kv-get (key kv-store)

Nice!

>
> But honestly, I suspect using any db as backend for a persistent lisp
> store will have virtually no advantage over plain text.
> Then why even introduce them at the first place.

That was my secret hope as well, because the obvious need for convenient
universal support for persistent storage has been somehow tightly mixed
with a need for some external DB, the latter being at least non-obvious.

>
> The more exciting thing will be to have a general object store
> rather than kv-store (aka, incremental for arbitrary mutation
> and preserve sharing). Any relational db will not help that either,
> and it is better to be done in pure Lisp.

I believe that depending on (any) DB is not the right way to go for
persistent storage. It rather should be abstracted-out and then DB will
likely won't be used as suitable backend anyway.

Thanks,
-- Sergey Organov

Re: sqlite3

[Po Lu]

Sergey Organov <sorganov@gmail.com> writes:

> I believe that depending on (any) DB is not the right way to go for
> persistent storage. It rather should be abstracted-out and then DB will
> likely won't be used as suitable backend anyway.

I would really like something along the lines of manardb for Emacs Lisp.

We have the advantage of controlling the entire runtime, so it would not
have portability issues across different language implementations (in
the case of manardb, Common Lisp).

But since that is too much work, the second best solution would be to
use *dbm, or even printing hash tables, to achieve a simple key-value
store.

Re: sqlite3

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.  That is:
>
> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>
> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.
>
> In addition, if somebody really wants to write SQL stuff (it can be very
> handy for some things), having sqlite3 in there gives us that in
> addition for free.

What would the use-case for this be?  And would this be part of the same
interface ("persistent-data"), or would they be separate?  I think the
latter would be preferable, because then systems that might not have
SQLite installed -- for whatever reason, despite it's wide
availability -- could fall back on a Elisp implementation.

-- 
	Philip Kaludercic

Re: sqlite3

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

>> I think it'd be good to bring that into core, and then write a small
>> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
>> that we don't have to write SQL all over the place.  That is:
>>
>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>>
>> I.e., what Emacs needs is a persistent key/value store, and this would
>> give us that.
>>
>> In addition, if somebody really wants to write SQL stuff (it can be very
>> handy for some things), having sqlite3 in there gives us that in
>> addition for free.
>
> What would the use-case for this be? And would this be part of the
> same interface ("persistent-data"), or would they be separate?  I
> think the latter would be preferable, because then systems that might
> not have SQLite installed -- for whatever reason, despite it's wide
> availability -- could fall back on a Elisp implementation.

I'm not quite sure what you're asking here?  The use case for having
persistent data or for writing SQL?

Nobody that uses the persistent data interface would have to write SQL.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>> I think it'd be good to bring that into core, and then write a small
>>> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
>>> that we don't have to write SQL all over the place.  That is:
>>>
>>> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>>>
>>> I.e., what Emacs needs is a persistent key/value store, and this would
>>> give us that.
>>>
>>> In addition, if somebody really wants to write SQL stuff (it can be very
>>> handy for some things), having sqlite3 in there gives us that in
>>> addition for free.
>>
>> What would the use-case for this be? And would this be part of the
>> same interface ("persistent-data"), or would they be separate?  I
>> think the latter would be preferable, because then systems that might
>> not have SQLite installed -- for whatever reason, despite it's wide
>> availability -- could fall back on a Elisp implementation.
>
> I'm not quite sure what you're asking here?  The use case for having
> persistent data or for writing SQL?
>
> Nobody that uses the persistent data interface would have to write SQL.

Yes, but do you want the persistent data interface to be bound to
SQLite, or should a separate backed be implemented, if required?

-- 
	Philip Kaludercic

Re: sqlite3

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> Yes, but do you want the persistent data interface to be bound to
> SQLite, or should a separate backed be implemented, if required?

I don't think a separate backend is necessary, but I won't stand in the
way of somebody implementing it.  (I'm sure somebody has also
implemented a GIF decoder in Emacs Lisp because they're against using C
or something.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[tomas]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On Sat, Dec 11, 2021 at 04:19:40AM +0100, Lars Ingebrigtsen wrote:
> Philip Kaludercic <philipk@posteo.net> writes:
> 
> > Yes, but do you want the persistent data interface to be bound to
> > SQLite, or should a separate backed be implemented, if required?
> 
> I don't think a separate backend is necessary, but I won't stand in the
> way of somebody implementing it.

But you, of course, would consider it unnecessary (co-) developing an
interface spec to facilitate that.

>                                  (I'm sure somebody has also
> implemented a GIF decoder in Emacs Lisp because they're against using C
> or something.)

I can't help it: the above feels subtly... loaded.

No. Not happy at all about how things are developing here, sorry to say.
I don't question your intentions at all, but every time I've seen "hey,
let's use $PRODUCT" at the start of some development it hasn't turned
out well. I'd venture to say it is an antipattern.

Cheers
-- t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But you, of course, would consider it unnecessary (co-) developing an
  > interface spec to facilitate that.

We can do that next.

Would you like to show me the sqlite3 interface?  We may as well
see if we can use that one.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Qiantan Hong]

> Would you like to show me the sqlite3 interface?  We may as well
> see if we can use that one.
I believe the sqlite3 interface accepts strings of SQL code, which
makes it unlikely to be used as an interface for general Lisp programs.

> I think we should define a program interface for "save this data
> persistently", hook it up to _both_ formats (textual Lisp and
> sqlite3), then design each piece of code that saves persistent data
> to use whichever representation is selected at the time.
> 
> Eventually we might convert all the existing places that make
> persistent data so that they use this interface and can save the data
> either way.
I’m currently working on this, some discussions happen in thread
[New package: resist!], some off list. Currently my proposal is basically
to provide two packages:
- store.el, which provides a persistent key-value store interface,
   and supports multiple backends.
   the interface:
   store-make, store-get, store-put, store-rem,
   store-push, store-delete
- persist.el (still waiting for response from the author of a package
  of the same name to see if we can use this namespace),
  which provides persistent variables that are saved periodically.
   the interface:
   (persist-make-variable VARIABLE &optional STORE METHOD), 
   (persist-kill-variable VARIABLE)

Re: sqlite3

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Yes, but do you want the persistent data interface to be bound to
>> SQLite, or should a separate backed be implemented, if required?
>
> I don't think a separate backend is necessary, but I won't stand in the
> way of somebody implementing it.  (I'm sure somebody has also
> implemented a GIF decoder in Emacs Lisp because they're against using C
> or something.)

I raise this question from the perspective of the forwards-compatibility
library (compat.el, https://sr.ht/~pkal/compat/).  If there were no
distinction between persistent values and SQLite, then it would become
considerably more difficult to provide compatibility definitions if is
expected that the persistent value library can be mixed together with
raw SQL queries at any time.

-- 
	Philip Kaludercic

Re: sqlite3

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> I raise this question from the perspective of the forwards-compatibility
> library (compat.el, https://sr.ht/~pkal/compat/).  If there were no
> distinction between persistent values and SQLite, then it would become
> considerably more difficult to provide compatibility definitions if is
> expected that the persistent value library can be mixed together with
> raw SQL queries at any time.

I'm not sure I understand the question.  There's no SQL in the
multisession interface.  It looks like this:

  (multisession-value foo)

and

  (setf (multisession-value foo) 'bar)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> I'm not sure I understand the question.  There's no SQL in the
> multisession interface.  It looks like this:
> 
>  (multisession-value foo)
> 
> and
> 
>  (setf (multisession-value foo) 'bar)
This interface seems closer to a key value store than
persistent variable (which I expect would add persistence
functionality to an existing variable).

Compare the above to my interface
(kv-get KEY STORE &optional DFLT)
and (kv-put KEY VAL STORE).

Should we merge our work on this to provide one
single interface?

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> This interface seems closer to a key value store than
> persistent variable (which I expect would add persistence
> functionality to an existing variable).

It's a persistent variable.  See the scratch/sqlite branch.  (I haven't
merged this part yet.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> Qiantan Hong <qhong@mit.edu> writes:
> 
>> This interface seems closer to a key value store than
>> persistent variable (which I expect would add persistence
>> functionality to an existing variable).
> 
> It's a persistent variable.  See the scratch/sqlite branch.  (I haven't
> merged this part yet.)
I believe it’s closer to key value store, at least in my program.
It’s basically 
(defvar multisession-store …) 
(defmacro (multisession-value foo) `(store-get ,foo multisession-store))

The persistent variable interface I provide instead
transparently persist and restore the value cell of a variable,
basically you just (persist-make-variable foo), then other code
just use foo directly with no need to change.

I’ve read through multisession, it seems that all of multisession.el,
the current persist.el on Elpa, and my resist!.el & proposal that
would move into persist.el namespace, have very similar aim.
I believe the best plan so far is to provide one unifying interface
under persist.el namespace, and support multiple backend for
persistent store. 
There’s a summary of the plan in
https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01156.html

According to the plan above, the current multisession.el would
be reworked a bit to provide store-get and store-put interface.
Users can use the store interface directly, which is very similar to 
`(multisession-value …)`, or they can backed their
“transparent” persistant variables using sqlite3 backend.
We should probably also add (setf store-get) so it functions as a place.
Are you willing to proceed in this direction?

Best,
Qiantan

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> I’ve read through multisession, it seems that all of multisession.el,
> the current persist.el on Elpa, and my resist!.el & proposal that
> would move into persist.el namespace, have very similar aim.

[...]

> Are you willing to proceed in this direction?

The interface will stay as it is in multisession.el, I think.

(But the name might change if somebody comes up with a better name for
"multisession" (that's not taken).  "Persistant" is taken, and "persist"
seems awkward.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> On Dec 11, 2021, at 10:50 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> Qiantan Hong <qhong@mit.edu> writes:
> 
>> I’ve read through multisession, it seems that all of multisession.el,
>> the current persist.el on Elpa, and my resist!.el & proposal that
>> would move into persist.el namespace, have very similar aim.
> 
> [...]
> 
>> Are you willing to proceed in this direction?
> 
> The interface will stay as it is in multisession.el, I think.
> 
> (But the name might change if somebody comes up with a better name for
> "multisession" (that's not taken).  "Persistant" is taken, and "persist"
> seems awkward.)
Seems that namespace is a common problem for us.
I’ve discussed with a few other people and am currently
settling on the option to take the persist namespace from
an existing package. But sure, new proposals are welcomed.

I’m thinking of rather not exposing multisession.el directly,
but instead use it as a backend for store.el.
The interface just need to be changed a little bit,
(multisession-value OBJECT) => (store-get OBJECT (STORE sqlite3-store))
(multisession--set-value OBJECT VALUE) => (store-put OBJECT VALUE (STORE sqlite3-store))
Basically, just add the database as an parameter so user can use any databases,
and define the functions as method of store-* instead.

Then, <whatever-name-we-choose-for-persistent-variable>-* will accommodate
my transparent persistent variable interfaces, and backward compatibility for
what’s already in persist.el Elpa package.
The user can configure them to use the sqlite3 backend taken from multisession.el,
or other backends.

If you agree, I could make a patch that implements the above changes.

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> I’m thinking of rather not exposing multisession.el directly,
> but instead use it as a backend for store.el.

[...]

> If you agree, I could make a patch that implements the above changes.

I do not.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

>> If you agree, I could make a patch that implements the above changes.
> 
> I do not.
What’s the problem? I could try to change the plan so that it works for us both.

Without an agreement we would have several incompatible interfaces 
doing very similar things, which I’d like not to see.

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> What’s the problem? I could try to change the plan so that it works
> for us both.

I'm not sure what you mean.  The facility is going to work
(semantically) the way it does on that branch, largely, in Emacs 29.
What you implement in your package is up to you, of course.

(I've now added some documentation that says what those semantics are.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[tomas]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

On Sun, Dec 12, 2021 at 08:00:44AM +0100, Lars Ingebrigtsen wrote:
> Qiantan Hong <qhong@mit.edu> writes:
> 
> > I’m thinking of rather not exposing multisession.el directly,
> > but instead use it as a backend for store.el.
> 
> [...]
> 
> > If you agree, I could make a patch that implements the above changes.
> 
> I do not.

Duly noted, thanks.

-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I’m thinking of rather not exposing multisession.el directly,
  > but instead use it as a backend for store.el.

What is store.el?  Could you show us its interface specs?
Is it written, or is it a proposal?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Alexandre Garreau]

Le lundo, 13-a de decembro 2021, 4-a horo kaj 44:17 CET Richard Stallman a 
écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > I’m thinking of rather not exposing multisession.el directly,
>   > but instead use it as a backend for store.el.
> 
> What is store.el?  Could you show us its interface specs?
> Is it written, or is it a proposal?

still a proposal, of a common abstraction over persistent key-value 
stores, that would have sqlite among backends, but also lisp-implemented 
stores such as the “one-file-per-value in a directory” persist.el currently 
use (most efficient on btree fs like btrfs or zfs), and a logging delta 
store qiantan hong implemented initially on a scratch by reaction to 
sqlite suggestion.

Re: sqlite3

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

>>> I’m thinking of rather not exposing multisession.el directly,
>>> but instead use it as a backend for store.el.
>> 
>> What is store.el?  Could you show us its interface specs?
>> Is it written, or is it a proposal?
> 
> still a proposal, of a common abstraction over persistent key-value 
> stores, that would have sqlite among backends, but also lisp-implemented 
> stores such as the “one-file-per-value in a directory” persist.el currently 
> use (most efficient on btree fs like btrfs or zfs), and a logging delta 
> store qiantan hong implemented initially on a scratch by reaction to 
> sqlite suggestion.
It is now written BTW. I’ve finished the work of extracting
key value store part from resist!.el. I haven’t start extracting
persistent variable part as I haven’t hear back about persist.el
namespace availability yet.

I’ve already included the above mentioned two backends.
Lars’ sqlite3 implementation could be included as the third
with slight modification.

Code is hosted at https://code.librehq.com/qhong/store.el/
For people that don’t want to follow link, I also attach the
current revision below

[-- Attachment #2: store.el --]
[-- Type: application/octet-stream, Size: 15645 bytes --]

;;; store.el --- Persistant Key-Value Store  -*- lexical-binding: t; -*-

;; Copyright (C) 2021 Free Software Foundation, Inc.

;; Author: Qiantan Hong <qhong@alum.mit.edu>
;; Maintainer: Qiantan Hong <qhong@alum.mit.edu>
;; Keywords: persistence database
;; Version: 0.0.1

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:
;; This package implements a persistent key-value store facility.

;; The interface provides the following functions:

;; - Creating and compacting store: `store-create', `store-compact'
;; - Get the path of a store: `store-path'
;; - Put, remove and look up key value pairs: `store-put',
;;   `store-rem', `store-get',
;; - List operations: `store-push', `store-delete'
;; See their docstrings for details.

;; The package supports multiple backends.  All backends should
;; persist all changes immediately into external storage.  It comes
;; with the following built-in backends:

;; - `log': Log-structured incremental S-expr store.  Only increments
;;   are stored, therefore the cost for each persistent operation
;;   should be independent of the total size of the store, and the
;;   cost for each persistent list operation should be independent of
;;   the total size of the list.

;; - `fs': File-system based S-expr store.  Use a file per key-value
;;   pair, with key S-expr as file name and value S-expr as content.

;;; Code:

(require 'cl-lib)
(require 'cl-generic)

;;; Customs
(defgroup persistence nil "Persistence facility."
          :group 'emacs)

(defcustom store-compact-idle-time 30
  "Idle time in seconds to wait before compacting key value stores."
  :type 'number :group 'persistence)

;;; Generic Interface
(cl-defgeneric store-create (path backend &key &allow-other-keys)
  "Create and return a key value store backed by PATH and BACKEND.
If the backing storage doesn't exist, create it.  Otherwise,
content of the returned store is synchronized to the backing
storage.")
(cl-defgeneric store-compact (store &optional nonblock)
  "Compress/vacuum/garbage collect STORE, if applicable.
If the backing storage is already locked, signal `file-locked'
immediately if NONBLOCK is non nil or wait if NONBLOCK is nil."
  ;; We provide a mock implementation for backends that do not support
  ;; compacting.
  nil)
(cl-defgeneric store-path (store)
  "Return the path of STORE's backing storage.
The returned path should create a store backed by the same
storage when passed to `store-create'.")

(cl-defgeneric store-call-with-transaction (store function &key (nonblock nil) (lock t))
  "Call FUNCTION while holding a transaction for STORE.
If LOCK is non nil, grab a lock of the backing storage when
applicable.  If the backing storage is already locked, wait if
NONBLOCK is nil or signal `file-locked' immediately if NONBLOCK
is non nil."
  ;; We provide a mock implementation for backends that do not support
  ;; transaction.
  (funcall function))
(cl-defmacro store-with-transaction
    ((store &key (nonblock nil) (lock t)) &rest body)
  "Run BODY while holding a transaction for STORE.
If LOCK is non nil, grab a lock of the backing storage when
applicable.  If the backing storage is already locked, wait if
NONBLOCK is nil or signal `file-locked' immediately if NONBLOCK
is non nil."
  (declare (indent 1) (debug ([&rest form] body)))
  `(store-call-with-transaction ,store (lambda () ,@body)
                             :lock ,lock :nonblock ,nonblock))

(cl-defgeneric store-put (key value store)
  "Associate KEY with VALUE in STORE.
The operation is immediately persisted.
Return VALUE.")
(cl-defgeneric store-get (key store &optional dflt)
  "Look up KEY in STORE and return its associated value.
If KEY is not found, return DFLT which defaults to nil.")
(cl-defgeneric store-rem (key store)
  "Remove KEY from STORE.
The operation is immediately persisted.
Return nil.")
(gv-define-setter store-get (val key store &optional _dflt)
  ;; TODO: can we detect if the place is used in `push', `cl-pushnew',
  ;; etc and use the potentially more efficient `store-push' instead?
  `(store-put ,key ,val ,store))

(cl-defgeneric store-push (key value store)
  "Add VALUE to the list associated with KEY in STORE.
The operation is immediately persisted.
Return the new list with VALUE added."
  ;; We provide a default implementation for backends not natively
  ;; supporting list operations.  Backends are encouraged to override
  ;; this with more efficient implementation.
  (store-put key (cons value (store-get key store)) store))
(cl-defgeneric store-delete (key value store)
  "Remove VALUE from the list associated with KEY in STORE.
The operation is immediately persisted.
Return the new list with VALUE removed."
  ;; We provide a default implementation for backends not natively
  ;; supporting list operations.  Backends are encouraged to override
  ;; this with more efficient implementation.
  (store-put key (delete value (store-get key store)) store))

;;; Automatic Compacting
(defvar store-need-compacting-list nil
  "List of stores that shall be compacted by `store-compact-demon'.")
(defvar store-compact-idle-timer nil "Idle timer that runs `store-compact-demon'.")
(defun store-compact-demon ()
  (while store-need-compacting-list
    (let ((store (pop store-need-compacting-list)))
      (condition-case nil
          (store-compact store 'nonblock)
        (file-locked
         (message "Giving up compacting %s this time, because it is locked."
                  (store-path store))
         nil)))))
(defun store-compact-demon-summon ()
  (unless store-compact-idle-timer
    (setq store-compact-idle-timer (timer-create))
    (timer-set-function store-compact-idle-timer 'store-compact-demon)
    (timer-set-idle-time store-compact-idle-timer store-compact-idle-time))
  (unless (memq store-compact-idle-timer timer-list)
    (timer-activate-when-idle store-compact-idle-timer t)))

;;; Serialization utility
(defsubst store--print (form)
  (let ((print-length nil) (print-level nil))
    (prin1 form (current-buffer)))
  (insert "\n"))

;;; `log' backend
(defvar store-inhibit-ask-user-about-lock nil)
(defun store-inhibit-ask-user-about-lock-advice (orig-func file opponent)
  (if store-inhibit-ask-user-about-lock
      (signal 'file-locked (list file opponent))
    (funcall orig-func file opponent)))
(advice-add 'ask-user-about-lock :around #'store-inhibit-ask-user-about-lock-advice)
(defvar store-inhibit-ask-user-about-supersession-threat nil)
(defun store-inhibit-ask-user-about-supersession-threat-advice (orig-func file)
  (unless store-inhibit-ask-user-about-supersession-threat
    (funcall orig-func file)))
(advice-add 'ask-user-about-supersession-threat :around #'store-inhibit-ask-user-about-supersession-threat-advice)

(cl-defstruct (store-log (:constructor store-log--make))
  path (table (make-hash-table :test 'equal)) (log-count 0) compact-ratio
  ;; I used to use a `gensym' redirection to exploit thread-safe
  ;; dynamic binding for `store-log-transaction-buffer', but currently
  ;; I dropped the support for recursive and concurrent transaction
  ;; (due to limitation of `lock-file').  When we add it back in the
  ;; future, remember the `gensym' trick.
  transaction-buffer)

(defun store-log--lock (store)
  (setf (buffer-modified-p (store-log-transaction-buffer store)) t) ; otherwise `lock-buffer' would do nothing
  (with-current-buffer (store-log-transaction-buffer store)
    (let ((create-lockfiles t)
          (store-inhibit-ask-user-about-lock t)
          (store-inhibit-ask-user-about-supersession-threat t))
      (lock-buffer))))
(cl-defmethod store-call-with-transaction ((store store-log) function &key (nonblock nil) (lock t))
  (when (store-log-transaction-buffer store)
    (error "Recursive or concurrent transaction is currently not supported"))
  (let ((buffer (generate-new-buffer " *temp log transaction*")))
    (setf (buffer-local-value 'buffer-file-truename buffer) (store-log-path store))
    (unwind-protect
         (progn
           (setf (store-log-transaction-buffer store) buffer)
           (when lock
             (if nonblock
                 (store-log--lock store)
               (while (not (ignore-errors (store-log--lock store) t))
                 (sit-for 0.0 t))))
           (prog1 (funcall function)
             (with-current-buffer buffer
               (write-region nil nil (store-log-path store) t 'silent))))
      (setf (store-log-transaction-buffer store) nil)
      (kill-buffer buffer))))
(cl-defmacro store-log--ensure-transaction-buffer ((store &rest args) &rest body)
  "Make sure BODY is executed inside a transaction buffer for STORE.
If no transaction buffer for STORE is in effect, create a temporary buffer
for appending logs.
ARGS are passed to `store-call-with-transaction'."
  (declare (indent 1) (debug ([&rest form] body)))
  (let ((store-var (gensym "store"))
        (body-sym (gensym "body")))
    `(let ((,store-var ,store))
       (cl-flet ((,body-sym ()
                   (with-current-buffer (store-log-transaction-buffer ,store-var)
                     ,@body)))
         (if (store-log-transaction-buffer ,store-var)
             (,body-sym)
           (store-call-with-transaction ,store-var #',body-sym ,@args))))))

(defun store-log-load (store &optional incremental)
  "Load the content STORE from its backing file.
If INCREMENTAL is nil, load the entire backing file and replace
STORE's content.  Otherwise, INCREMENTAL should be a number
specifying a start offset of the backing file, and the portion
after this offset will be loaded and added to STORE's
content."
  (store-with-transaction (store)
    (unless incremental (clrhash (store-log-table store)))
    (condition-case nil
        (with-temp-buffer
          (insert-file-contents (store-log-path store) nil (or incremental 0))
          (while (< (point) (1- (point-max))) ; exclude trailing newline
            (let ((entry (read (current-buffer))))
              (pcase (car entry)
                ('++ (puthash (cadr entry) (caddr entry) (store-log-table store)))
                ('-- (remhash (cadr entry) (store-log-table store)))
                ('l+ (push (caddr entry) (gethash (cadr entry) (store-log-table store))))
                ('l- (puthash (cadr entry)
                              (delete (caddr entry) (gethash (cadr entry) (store-log-table store)))
                              (store-log-table store))))
              (cl-incf (store-log-log-count store)))))
      (end-of-file
       ;; We might encounter trailing unbalanced form if Emacs
       ;; crashed in the middle of `log-put'.  We compact the file
       ;; and fix unbalanced form as a side effect
       (store-compact store)))))
(cl-defmethod store-create (path (_backend (eql log)) &key (compact-ratio 4.0) &allow-other-keys)
  "Create a key value store backed by file PATH.
If file PATH does not exist, create it and return an empty key value store.
If file PATH exists, load its content into a key value store and return it.

COMPACT-RATIO controls automatic compacting.  Arrange for
compacting if log count exceeds (COMPACT-RATIO * size-of-table)."
  (let* ((store-log (store-log--make :path path :compact-ratio compact-ratio)))
    (when (file-exists-p path)
      (store-log-load store-log))
    store-log))
(cl-defmethod store-path ((store store-log))
  (store-log-path store))

(cl-defmethod store-compact ((store store-log) &optional nonblock)
  (store-log--ensure-transaction-buffer (store :nonblock nonblock)
    (setf (store-log-log-count store) (hash-table-count (store-log-table store)))
    (with-temp-buffer
      (maphash (lambda (key value) (store--print (list '++ key value)))
               (store-log-table store))
      (let ((file-precious-flag t))
        (write-region nil nil (store-log-path store) nil 'silent)))))

(defsubst store-log--inc-log-count (store)
  "Increase log count for STORE, possibly arrange for compacting.
Arrange for compacting if log count exceeds (compact-ratio * size-of-table)."
  (cl-incf (store-log-log-count store))
  (when (and (store-log-compact-ratio store)
             (> (store-log-log-count store)
                (* (store-log-compact-ratio store) (hash-table-count (store-log-table store)))))
    (cl-pushnew store store-need-compacting-list)
    (store-compact-demon-summon)))
(cl-defmethod store-put (key value (store store-log))
  (store-log--ensure-transaction-buffer (store)
    (store--print (list '++ key value))
    (store-log--inc-log-count store)
    (puthash key value (store-log-table store))))
(cl-defmethod store-rem (key (store store-log))
  (store-log--ensure-transaction-buffer (store)
    (store--print (list '-- key))
    (store-log--inc-log-count store)
    (remhash key (store-log-table store))))
(cl-defmethod store-push (key value (store store-log))
  (store-log--ensure-transaction-buffer (store)
    (store--print (list 'l+ key value))
    (store-log--inc-log-count store)
    (push value (gethash key (store-log-table store)))))
(cl-defmethod store-delete (key value (store store-log))
  (store-log--ensure-transaction-buffer (store)
    (store--print (list 'l- key value))
    (store-log--inc-log-count store)
    (puthash key (delete value (gethash key (store-log-table store)))
             (store-log-table store))))
(cl-defmethod store-get (key (store store-log) &optional dflt)
  (gethash key (store-log-table store) dflt))

;; `fs' backend
(cl-defstruct (store-fs (:constructor store-fs--make))
  path)
(cl-defmethod store-create (path (_backend (eql fs)) &key &allow-other-keys)
  (if (file-exists-p path)
      (unless (file-directory-p path)
        (error "`fs' backend requires PATH to be a directory, but %s is not" path))
    (mkdir path))
  ;; We normalize PATH right away, saving future hassles.
  (store-fs--make :path (file-name-as-directory path)))
(cl-defmethod store-path ((store store-fs))
  (store-fs-path store))

(defsubst store-fs--key-to-path (key store)
  ;; TODO: escape illegal characters
  (concat (store-fs-path store) (prin1-to-string key)))
(cl-defmethod store-put (key value (store store-fs))
  (with-temp-buffer
    (store--print value)
    (let ((file-precious-flag t))
      (write-region nil nil
                    (store-fs--key-to-path key store)
                    nil 'quiet)))
  value)
(cl-defmethod store-get (key (store store-fs) &optional dflt)
  (let ((path (store-fs--key-to-path key store)))
    (if (file-exists-p path)
        (with-temp-buffer
          (insert-file-contents path)
          (read (current-buffer)))
      dflt)))
(cl-defmethod store-rem (key (store store-fs))
  (delete-file (store-fs--key-to-path key store))
  nil)
(provide 'store)
;;; store.el ends here

[-- Attachment #3: ATT00001.c --]
[-- Type: text/plain, Size: 22 bytes --]



Best,
Qiantan

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It sounds like you're going in the right direction, making
an interface which can be used for all access to files of Lisp values.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Howard Melman]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> (But the name might change if somebody comes up with a better name for
> "multisession" (that's not taken).  "Persistant" is taken, and "persist"
> seems awkward.)

enduring, surviving, durable, stable, lasting, eternal,
indelible, perennial, continual, incessant?

-- 

Howard

Re: sqlite3

[Lars Ingebrigtsen]

Howard Melman <hmelman@gmail.com> writes:

>> (But the name might change if somebody comes up with a better name for
>> "multisession" (that's not taken).  "Persistant" is taken, and "persist"
>> seems awkward.)
>
> enduring, surviving, durable, stable, lasting, eternal,
> indelible, perennial, continual, incessant?

I think all of those were on the list of synonyms I already posted?
None of those really seem appealing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> I raise this question from the perspective of the forwards-compatibility
>> library (compat.el, https://sr.ht/~pkal/compat/).  If there were no
>> distinction between persistent values and SQLite, then it would become
>> considerably more difficult to provide compatibility definitions if is
>> expected that the persistent value library can be mixed together with
>> raw SQL queries at any time.
>
> I'm not sure I understand the question.  There's no SQL in the
> multisession interface.  It looks like this:
>
>   (multisession-value foo)
>
> and
>
>   (setf (multisession-value foo) 'bar)

I get that, is it defined or undefined behaviour if I mix in a few SQL
queries around various

  (multisession-value foo)

calls?

-- 
	Philip Kaludercic

Re: sqlite3

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> I get that, is it defined or undefined behaviour if I mix in a few SQL
> queries around various
>
>   (multisession-value foo)
>
> calls?

I don't understand what you mean.  SQL queries ... towards which
database?  Doing what?

And somebody talked about doing an implementation of this that wasn't
backed by sqlite, in which case...  nothing?  (But if you deleted the
backing storage to that method, it would affect something?)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> On Dec 12, 2021, at 4:22 AM, larsi@gnus.org wrote:
> 
> Philip Kaludercic <philipk@posteo.net> writes:
> 
> And somebody talked about doing an implementation of this that wasn't
> backed by sqlite, in which case...  nothing?  (But if you deleted the
> backing storage to that method, it would affect something?)
I have a working implementation of s-expr based log structured
key value store. The current persist.el on Elpa has a file system
based implementation, and I plan to also provide it as an backend.

If you delete them, then of course the store is just gone, just like
if you delete the sqlite3 database.

> On Dec 11, 2021, at 12:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> That doesn't imply Emacs should not support sqlite3.  If you happen
>> to store large amounts of persistent data, you might like
>> saving persistent Emacs data that way.
> 
> Which is exactly what is being planned.
> 
>> But it does imply that sqlite3 should not be the default way to store
>> persistent Emacs data.
> 
> Which is exactly what is being planned.  No one has any intention to
> go over all the data Emacs stores between sessions and convert all of
> them to sqlite3.  We only plan to use that where a data base is large,
> and thus using plain text there is unwieldy.

I’ve implemented exactly this, and your multisession.el, with a
little modification, would fit into this plan and functions as a key-value store
backend.

Re: sqlite3

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

* 2021-12-12 14:10:28+0000, Qiantan Hong wrote:

>> Which is exactly what is being planned. No one has any intention to
>> go over all the data Emacs stores between sessions and convert all of
>> them to sqlite3. We only plan to use that where a data base is large,
>> and thus using plain text there is unwieldy.
>
> I’ve implemented exactly this, and your multisession.el, with a little
> modification, would fit into this plan and functions as a key-value
> store backend.

Or maybe another approach: Use multisession.el as the general front end
interface which could dispatch to different key-value storage
implementations, defined in other files like multisession-sqlite.el,
multisession-ihatesql.el, multisession-postgresql.el etc.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 434 bytes --]

Re: sqlite3

[Qiantan Hong]

> On Dec 12, 2021, at 6:39 AM, Teemu Likonen <tlikonen@iki.fi> wrote:
> 
> * 2021-12-12 14:10:28+0000, Qiantan Hong wrote:
> 
>>> Which is exactly what is being planned. No one has any intention to
>>> go over all the data Emacs stores between sessions and convert all of
>>> them to sqlite3. We only plan to use that where a data base is large,
>>> and thus using plain text there is unwieldy.
>> 
>> I’ve implemented exactly this, and your multisession.el, with a little
>> modification, would fit into this plan and functions as a key-value
>> store backend.
> 
> Or maybe another approach: Use multisession.el as the general front end
> interface which could dispatch to different key-value storage
> implementations, defined in other files like multisession-sqlite.el,
> multisession-ihatesql.el, multisession-postgresql.el etc.
I think it’s much easier to use it as a backend, as it just require some
slight modification to do so. It is also closer to a key value store
than persistent variable (I know, it is close to persistent variable,
but persistent variable is also closed to key value store and 
multisession.el is closer to key value store between the two).
Just compare
(multisession-value x) with (store-get x multisession-store).
I also think changing the interface to the latter is useful,
because the user can then use the database of their choice.
It can default to a global one.

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.  That is:
>
> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>
> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.

This is now implemented under the name of `multisession-value' (but not
merged to the trunk yet; I'll probably do that later today, unless
somebody comes up with a better name).

As for the sqlite part of this: My initial benchmarking of this was
wrong.  I thought sqlite3 was going to be a real advantage for this
thing, since I'd benchmarked excellent performance (more than 50K
updates per second, for instance).  But that's only when not committing
after every transaction, which we want to do here, really.

But it turns out that sqlite3 is actually slower for this particular use
case than just writing the data to a file (i.e., using the file system
as the database; one file per value).  So multisession.el now offers two
backends (`files' and `sqlite'), and defaults to `files'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Óscar Fuentes]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> As for the sqlite part of this: My initial benchmarking of this was
> wrong.  I thought sqlite3 was going to be a real advantage for this
> thing, since I'd benchmarked excellent performance (more than 50K
> updates per second, for instance).  But that's only when not committing
> after every transaction, which we want to do here, really.
>
> But it turns out that sqlite3 is actually slower

Surprise! :-)

> for this particular use
> case than just writing the data to a file (i.e., using the file system
> as the database; one file per value).  So multisession.el now offers two
> backends (`files' and `sqlite'), and defaults to `files'.

I'm pretty sure SQlite is providing stronger persistence guarantees than
the file system.

For persisting data to disk, you either have a fast db or a robust db.
Not both.

OTOH, the "one file per value" part above sounds somewhat scary :-)

Re: sqlite3

[Óscar Fuentes]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I thought sqlite3 was going to be a real advantage for this
> thing, since I'd benchmarked excellent performance (more than 50K
> updates per second, for instance).  But that's only when not committing
> after every transaction, which we want to do here, really.

Forgot to mention that modern databases usually offer several strategies
for persisting data. A quick look at SQLite documentation shows that it
can be configured to use a write ahead log, which is faster than the
default rollback journal:

https://sqlite.org/wal.html

Re: sqlite3

[Lars Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Forgot to mention that modern databases usually offer several strategies
> for persisting data. A quick look at SQLite documentation shows that it
> can be configured to use a write ahead log, which is faster than the
> default rollback journal:
>
> https://sqlite.org/wal.html

Yes, I used the WAL in my tests.  (The implementation is still on the
scratch/sqlite branch, for those who are curious.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3 usage for multisession variable storage

[Robin Tarsiger]

Lars Ingebrigtsen wrote:
> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
>> Forgot to mention that modern databases usually offer several strategies
>> for persisting data. A quick look at SQLite documentation shows that it
>> can be configured to use a write ahead log, which is faster than the
>> default rollback journal:
>>
>> https://sqlite.org/wal.html
> 
> Yes, I used the WAL in my tests.  (The implementation is still on the
> scratch/sqlite branch, for those who are curious.)
> 

I'm curious about some of these pragma settings more generally.

   - Why journal_mode WAL rather than TRUNCATE or MEMORY? WAL makes the file
     less easily moveable and introduces the shm requirement, which feels weird
     in an Emacs context. Since Emacs itself often works along the lines of "write
     new file and replace" or "copy to backup and then overwrite", DELETE is
     natural, but TRUNCATE feels slightly better for managed data stores. MEMORY
     would expose the DB to corruption on application crashes, but if you're
     outside a transaction almost all the time, it might be fine... and I don't
     see the files backend currently using backup files (or even locking) anyway,
     yes? (Or is that intended to be added later?)

   - Doesn't that say "bsynchronous" with an extra b? (Unrecognized pragmas are
     normally no-ops.) Also, if you're using synchronous NORMAL rather than OFF
     for SQLite, then presumably make sure write-region-inhibit-fsync is nil while
     comparing...

   - Having auto_vacuum be FULL similarly seems like a waste, adding a bunch of
     extra work to each transaction commit.

   - Given that people have expressed concerns over in-place modification leaving
     unwanted traces of old data, setting secure_delete to ON may be useful here.

   - Setting trusted_schema to OFF is recommended for new applications.

   - You might consider using application_id to make the file identifiable as a
     specific format rather than probing for the existence of the main table.

   - Doesn't the current code in multisession.el not execute the pragmas on subsequent
     opens? Only some of these are persistent.

(As a reminder for anyone following, the documentation for SQLite pragmas is
available at <https://www.sqlite.org/pragma.html>.)

-RTT

Re: sqlite3 usage for multisession variable storage

[Lars Ingebrigtsen]

Robin Tarsiger <rtt@dasyatidae.com> writes:

> I'm curious about some of these pragma settings more generally.
>
>   - Why journal_mode WAL rather than TRUNCATE or MEMORY? WAL makes the file
>     less easily moveable and introduces the shm requirement, which feels weird
>     in an Emacs context. Since Emacs itself often works along the lines of "write
>     new file and replace" or "copy to backup and then overwrite", DELETE is
>     natural, but TRUNCATE feels slightly better for managed data stores. MEMORY
>     would expose the DB to corruption on application crashes, but if you're
>     outside a transaction almost all the time, it might be fine... and I don't
>     see the files backend currently using backup files (or even locking) anyway,
>     yes? (Or is that intended to be added later?)

MEMORY seems inappropriate, since several instances will be using the
file at the same time.  I'm not sure what the TRUNCATE would bring us.

The `files' backup doesn't lock anything, because it just wants whatever
instance manages to write the file last to do so.  But I was pondering
writing to a temp file in the same directory and doing a rename instead.

>   - Doesn't that say "bsynchronous" with an extra b? (Unrecognized pragmas are
>     normally no-ops.) Also, if you're using synchronous NORMAL rather than OFF
>     for SQLite, then presumably make sure write-region-inhibit-fsync is nil while
>     comparing...

Fixed the typo.  I didn't see any noticeable difference between NORMAL
and OFF on this laptop, so I left it at NORMAL.

>   - Having auto_vacuum be FULL similarly seems like a waste, adding a bunch of
>     extra work to each transaction commit.

Likewise, I didn't see any difference between FULL and INCREMENTAL here,
and FULL means less work, so I left it at FULL.

>   - Given that people have expressed concerns over in-place modification leaving
>     unwanted traces of old data, setting secure_delete to ON may be useful here.

I thought the vacuum was supposed to take care of most of that?

>   - Setting trusted_schema to OFF is recommended for new applications.

I'm not sure that's relevant for this usag?

>   - You might consider using application_id to make the file identifiable as a
>     specific format rather than probing for the existence of the main table.

Sure.

>   - Doesn't the current code in multisession.el not execute the pragmas on subsequent
>     opens? Only some of these are persistent.

Good point.  I've now moved the pragmas.  (But that didn't seem to
affect performance, either.)

Thanks for the tips.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3 usage for multisession variable storage

[Robin Tarsiger]

Lars Ingebrigtsen wrote:
> MEMORY seems inappropriate, since several instances will be using the
> file at the same time.  I'm not sure what the TRUNCATE would bring us.

MEMORY doesn't disable the use of exclusive file locking during write
transactions. Several concurrent instances successfully committing or
rolling back in MEMORY mode will have the same consistency constraints.
The difference is in crash recovery; if one instance crashes hard in the
middle of a transaction, it's "restore the file from backup" time.
Whether that's okay depends on your surrounding strategy, but it feels
similar to a subset of the lockfile strategies Emacs already considers
okay.

TRUNCATE is just slightly less operations than DELETE in the case where
you do want to have a full rollback journal and don't care about the
directory looking tidy.

Hmm. What pattern/frequency/relative-frequency of reads/writes is expected
to be common for these variables? And in particular, what about read-
modify-write operations? Will applications tend to expect these to be
atomic (deliberately or otherwise)?

> The `files' backup doesn't lock anything, because it just wants whatever
> instance manages to write the file last to do so.  But I was pondering
> writing to a temp file in the same directory and doing a rename instead.

Yeah, overwrites are still non-atomic in the easiest case. :-)

>>    - Having auto_vacuum be FULL similarly seems like a waste, adding a bunch of
>>      extra work to each transaction commit.
> 
> Likewise, I didn't see any difference between FULL and INCREMENTAL here,
> and FULL means less work, so I left it at FULL.
> 
>>    - Given that people have expressed concerns over in-place modification leaving
>>      unwanted traces of old data, setting secure_delete to ON may be useful here.
> 
> I thought the vacuum was supposed to take care of most of that?

Auto-vacuuming only makes sure free pages are deallocated so that the
database file shrinks as data is removed, even in FULL mode. It does not
necessarily overwrite stale data with zeros, and as mentioned in the
documentation, it can actually make fragmentation worse by being too
aggressive with moving pages around. Normally I would leave it off.

If you want periodic maintenance of the DB file, there may also be
application-level maintenance that would be useful, as I extrapolate it?
Depending on what's stored in these variables, clearing out stale entries
and the like may want to be done at an elisp level. Leaving cleanup+VACUUM
to be done explicitly on timer or user request and then having a hook for
the non-DB-format-related cleanup might be workable? Not sure.

>>    - Setting trusted_schema to OFF is recommended for new applications.
> 
> I'm not sure that's relevant for this usag?

It cheaply removes a little chunk of potentially surprising behaviors. But
you may be right in a sense---it depends on whether elisp assumes it can
trust values read from the store. If so, then the entire DB is just in the
same "may do anything" trusted zone as the rest of .emacs.d, I suppose?

-RTT

Re: sqlite3 usage for multisession variable storage

[Lars Ingebrigtsen]

Robin Tarsiger <rtt@dasyatidae.com> writes:

> The difference is in crash recovery; if one instance crashes hard in the
> middle of a transaction, it's "restore the file from backup" time.
> Whether that's okay depends on your surrounding strategy, but it feels
> similar to a subset of the lockfile strategies Emacs already considers
> okay.

It's the "restore from backup" thing that's not acceptable, which makes
MEMORY unworkable here, I think.

> Hmm. What pattern/frequency/relative-frequency of reads/writes is expected
> to be common for these variables? And in particular, what about read-
> modify-write operations? Will applications tend to expect these to be
> atomic (deliberately or otherwise)?

I expect the writes to typically not happen very often.  But it's nice
that they're fast if somebody has a use case for updating a lot.

> If you want periodic maintenance of the DB file, there may also be
> application-level maintenance that would be useful, as I extrapolate it?
> Depending on what's stored in these variables, clearing out stale entries
> and the like may want to be done at an elisp level. Leaving cleanup+VACUUM
> to be done explicitly on timer or user request and then having a hook for
> the non-DB-format-related cleanup might be workable? Not sure.

There's really no way of knowing whether a value is "stale" -- there's
no central list of which keys are in use.

> It cheaply removes a little chunk of potentially surprising behaviors. But
> you may be right in a sense---it depends on whether elisp assumes it can
> trust values read from the store. If so, then the entire DB is just in the
> same "may do anything" trusted zone as the rest of .emacs.d, I suppose?

Yes, it doesn't seem to be a significant attack vector in this context.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3 usage for multisession variable storage

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 15 Dec 2021 02:34:11 +0100
> Cc: emacs-devel@gnu.org
> 
> Robin Tarsiger <rtt@dasyatidae.com> writes:
> 
> > The difference is in crash recovery; if one instance crashes hard in the
> > middle of a transaction, it's "restore the file from backup" time.
> > Whether that's okay depends on your surrounding strategy, but it feels
> > similar to a subset of the lockfile strategies Emacs already considers
> > okay.
> 
> It's the "restore from backup" thing that's not acceptable, which makes
> MEMORY unworkable here, I think.

Maybe this aspect should be left to the application level (with some
reasonable default in place)?  It sounds like it's a judgment call
that can only be resolved with a specific application in mind.

Re: sqlite3 usage for multisession variable storage

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> It's the "restore from backup" thing that's not acceptable, which makes
>> MEMORY unworkable here, I think.
>
> Maybe this aspect should be left to the application level (with some
> reasonable default in place)?  It sounds like it's a judgment call
> that can only be resolved with a specific application in mind.

Yup.  But this was with the multisession storage in mind -- if lots of
different Emacs things are saving application state to the same .sqlite
file, then we want to ensure pretty good reliability for that file.

Other packages using SQLite should make the decision themselves (by
issuing the pragmas they prefer to have for those .sqlite databases).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> But it turns out that sqlite3 is actually slower for this particular use
> case than just writing the data to a file (i.e., using the file system
> as the database; one file per value).  So multisession.el now offers two
> backends (`files' and `sqlite'), and defaults to `files'.
It seems that you’re doing very similar thing to my store.el.
(Which can be found at 
https://code.librehq.com/qhong/store.el/-/blob/master/store.el )
I still think the interface of store.el is better in terms of
- it let user choose which store to use
- it provides interface of list operations, which can 
  be very useful for Lisp, and can be provided more
  efficient implementation
It also comes with another backend right away.

Can you reconsider merging the work of multisession.el and
store.el, using multisession.el as a backend? It’s a slight
modification to multisession.el, and I don’t really see any downside.

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> Can you reconsider merging the work of multisession.el and
> store.el, using multisession.el as a backend? It’s a slight
> modification to multisession.el, and I don’t really see any downside.

Again, I don't understand what you're asking.  The interface defined by
multisession is the one that Emacs 29 is going to have.  If you want to
create a package with a different interface, that's totally up to you,
of course.

A backend to multisession that handles list-based updates might be nice,
but it's not part of the design specs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

>> Can you reconsider merging the work of multisession.el and
>> store.el, using multisession.el as a backend? It’s a slight
>> modification to multisession.el, and I don’t really see any downside.
> 
> Again, I don't understand what you're asking.  The interface defined by
> multisession is the one that Emacs 29 is going to have.  If you want to
> create a package with a different interface, that's totally up to you,
> of course.
It’s mostly a superset, so if both interface are provided there’re
significant overlap.
> 
> A backend to multisession that handles list-based updates might be nice,
> but it's not part of the design specs.
I’m asking to provide store.el in Emacs 29 as the interface of
persistent data feature instead of multisession.el (which becomes one
implementation).

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> I’m asking to provide store.el in Emacs 29 as the interface of
> persistent data feature instead of multisession.el (which becomes one
> implementation).

I see.  No, I don't think that's going to happen (unless Eli wants to
include it).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

>> I’m asking to provide store.el in Emacs 29 as the interface of
>> persistent data feature instead of multisession.el (which becomes one
>> implementation).
> 
> I see.  No, I don't think that's going to happen (unless Eli wants to
> include it).
Eli, can you help with this?

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 14 Dec 2021 16:42:25 +0100
> Cc: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> Qiantan Hong <qhong@mit.edu> writes:
> 
> > I’m asking to provide store.el in Emacs 29 as the interface of
> > persistent data feature instead of multisession.el (which becomes one
> > implementation).
> 
> I see.  No, I don't think that's going to happen (unless Eli wants to
> include it).

I don't see a reason, no.

Re: sqlite3

[Qiantan Hong]

>>> I’m asking to provide store.el in Emacs 29 as the interface of
>>> persistent data feature instead of multisession.el (which becomes one
>>> implementation).
>> 
>> I see.  No, I don't think that's going to happen (unless Eli wants to
>> include it).
> 
> I don't see a reason, no.
It’s a more general interface, and could be useful for
more programs.

Re: sqlite3

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "larsi@gnus.org" <larsi@gnus.org>,
>         "emacs-devel@gnu.org"
> 	<emacs-devel@gnu.org>
> Date: Tue, 14 Dec 2021 16:44:21 +0000
> Accept-Language: en-US
> 
> >>> I’m asking to provide store.el in Emacs 29 as the interface of
> >>> persistent data feature instead of multisession.el (which becomes one
> >>> implementation).
> >> 
> >> I see.  No, I don't think that's going to happen (unless Eli wants to
> >> include it).
> > 
> > I don't see a reason, no.
> It’s a more general interface, and could be useful for
> more programs.

We don't yet have a good enough understanding of how useful the
current interface is.  Extending and generalizing without any good
idea where we want to go is not something I'd like to do in Emacs.

We just started this experiment, let's see how it unfolds before
considering our next steps.

Re: sqlite3

[Qiantan Hong]

>>>>> I’m asking to provide store.el in Emacs 29 as the interface of
>>>>> persistent data feature instead of multisession.el (which becomes one
>>>>> implementation).
>>>> 
>>>> I see.  No, I don't think that's going to happen (unless Eli wants to
>>>> include it).
>>> 
>>> I don't see a reason, no.
>> It’s a more general interface, and could be useful for
>> more programs.
> 
> We don't yet have a good enough understanding of how useful the
> current interface is.  Extending and generalizing without any good
> idea where we want to go is not something I'd like to do in Emacs.
If we start with the store.el interface, we probably won’t even have
considered something like multisession.el’s interface. FWIW store.el
interface is designed independent from multisession.el, the latter
just happen to fall into a subset of store.el.

The aim of store.el is very clear: it provides a generic persistent key-value
store of Lisp values.

> We just started this experiment, let's see how it unfolds before
> considering our next steps.
Putting the more specialized multisession.el interface in master means 
we will need to provide backward compatibility even if at some point
store.el is added. I don’t see any downside of the other way — providing
just store.el interface from the beginning.

Re: sqlite3

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> Date: Tue, 14 Dec 2021 17:43:18 +0000
> Cc: "larsi@gnus.org" <larsi@gnus.org>,
>  "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> > We don't yet have a good enough understanding of how useful the
> > current interface is.  Extending and generalizing without any good
> > idea where we want to go is not something I'd like to do in Emacs.
> If we start with the store.el interface, we probably won’t even have
> considered something like multisession.el’s interface.

Or we could find out that it's over-generalized, and the additional
features are not used.  That's what having no experience means: you
can err either way.

> > We just started this experiment, let's see how it unfolds before
> > considering our next steps.
> Putting the more specialized multisession.el interface in master means 
> we will need to provide backward compatibility even if at some point
> store.el is added.

Assuming we decide that extending what we have is indeed necessary;
that is not a given.  And if we do decide, I see no catastrophe: we
are doing this all the time, keeping backward compatibility as we go.
There's no need to be afraid of that.

> I don’t see any downside of the other way — providing just store.el
> interface from the beginning.

Well, I do, and I tried to explain my take of this.  It's okay to
agree to disagree, of course.

Re: sqlite3

[Qiantan Hong]

>>> We just started this experiment, let's see how it unfolds before
>>> considering our next steps.
>> Putting the more specialized multisession.el interface in master means 
>> we will need to provide backward compatibility even if at some point
>> store.el is added.
> 
> Assuming we decide that extending what we have is indeed necessary;
> that is not a given.  And if we do decide, I see no catastrophe: we
> are doing this all the time, keeping backward compatibility as we go.
> There's no need to be afraid of that.
Sure. I can see many cases where store.el interface is more suitable
than multisession.el, particularly when a package need… literally a
persistent store… that is basically a persistent hashtable, with key
not necessarily symbols. One example is org-id. Guess I’ll try
to put store.el on Elpa first and see if those package will use it.

As for compatibly, I must be understanding something wrong.
I thought that any interface that went into master must be kept.
Is that not true? Are interface kept only when they went into a release?

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> As for compatibly, I must be understanding something wrong.
> I thought that any interface that went into master must be kept.
> Is that not true? Are interface kept only when they went into a release?

Yes.  The trunk is for development.  Things that go into the trunk may
or may not make it to a release.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "larsi@gnus.org" <larsi@gnus.org>,
>         "emacs-devel@gnu.org"
> 	<emacs-devel@gnu.org>
> Date: Tue, 14 Dec 2021 18:04:35 +0000
> 
> > Assuming we decide that extending what we have is indeed necessary;
> > that is not a given.  And if we do decide, I see no catastrophe: we
> > are doing this all the time, keeping backward compatibility as we go.
> > There's no need to be afraid of that.
> Sure. I can see many cases where store.el interface is more suitable
> than multisession.el, particularly when a package need… literally a
> persistent store… that is basically a persistent hashtable, with key
> not necessarily symbols. One example is org-id. Guess I’ll try
> to put store.el on Elpa first and see if those package will use it.

Yes, letting people use it will give us more useful data to base our
decisions on.

> As for compatibly, I must be understanding something wrong.
> I thought that any interface that went into master must be kept.

That is in general true, and I'm saying that we have good experience
extending existing interfaces without breaking backward compatibility.

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> A backend to multisession that handles list-based updates might be nice,
> but it's not part of the design specs.

(I considered making `files' be able to specify the data type, and a
list could be represented by a number of elements read independently
that could be appended to the file:

foo
bar
(baz)

would be restored as

(foo bar (baz))

but I didn't quite see how that would work with circular and
self-referential lists.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> but I didn't quite see how that would work with circular and
> self-referential lists.)
They don’t really make sense when storing to 
external storage, at least for all the mechanisms
either you or I have. Because our storage mechanism
doesn’t preserve intensional identity (“object identity”),
i.e. if two objects are EQ, we store them into different key,
they might be read back not EQ. We only store extensional
value. In such case, circular list are no different from
an infinite non-circular list, which doesn’t make sense.

Plus it’s not very common in practice to store circular list
in something one may want to make persistent. Mostly
it’s some history variable or so that is push into/delete from,
neither will make it circular.

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> They don’t really make sense when storing to 
> external storage, at least for all the mechanisms
> either you or I have. Because our storage mechanism
> doesn’t preserve intensional identity (“object identity”),
> i.e. if two objects are EQ, we store them into different key,
> they might be read back not EQ. We only store extensional
> value. In such case, circular list are no different from
> an infinite non-circular list, which doesn’t make sense.

We don't preserve EQ-ness of objects, but it's still nice to be able to
store a list where the same object appears a number of time (and get out
a list from the storage where that's still true).

> Plus it’s not very common in practice to store circular list
> in something one may want to make persistent. Mostly
> it’s some history variable or so that is push into/delete from,
> neither will make it circular.

We don't use circular lists a lot, but we do have rings, and it's nice
to be able to store them.

I feel that an append-only list storage thing is a totally separate
thing from persistent variables.  There's many different things that
could be taken into consideration when merging elements from different
instances, for instance.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We don't preserve EQ-ness of objects, but it's still nice to be able to
  > store a list where the same object appears a number of time (and get out
  > a list from the storage where that's still true).

Would you please show an example to illustrate what precisely this
would mean, and why it would be advantageous?

When we want to preserve EQ-ness, we can do that by binding
print-circle to t when converting the Lisp data to text.  When we're
not preserving EQ-ness, when and how does it make a difference in ((a
. b) (a . b)) whether the two elements are EQ?

What I don't see is how some sort of partial preservation of EQ-ness
would be important when we do NOT need full preservation of it.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> When we want to preserve EQ-ness, we can do that by binding
> print-circle to t when converting the Lisp data to text.  When we're
> not preserving EQ-ness, when and how does it make a difference in ((a
> . b) (a . b)) whether the two elements are EQ?

If you're printing a structure, it's nice to have a promise that the
structure you get back has the same structure.  (Even if the structure
itself isn't EQ to the one you printed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > When we want to preserve EQ-ness, we can do that by binding
  > > print-circle to t when converting the Lisp data to text.  When we're
  > > not preserving EQ-ness, when and how does it make a difference in ((a
  > > . b) (a . b)) whether the two elements are EQ?

  > If you're printing a structure, it's nice to have a promise that the
  > structure you get back has the same structure.

That's what I'd call "perserving EQ-ness" (within the structure).
And that is what you would get by binding print-circle to t before
printing the data.

Given that we already offer print-circle, why do we need to worry
further about this issue?  Is there really a need for some other
option intermediate between print-circle = t and print-circle = nil?




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> That's what I'd call "perserving EQ-ness" (within the structure).
> And that is what you would get by binding print-circle to t before
> printing the data.
>
> Given that we already offer print-circle, why do we need to worry
> further about this issue?  Is there really a need for some other
> option intermediate between print-circle = t and print-circle = nil?

The discussion was about Quintan's proposed list logging feature, which
doesn't write out a list in one go, but outputs it element by element.
I was pointing out that this doesn't allow using `print-circle'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> Richard Stallman <rms@gnu.org> writes:
> 
>> That's what I'd call "perserving EQ-ness" (within the structure).
>> And that is what you would get by binding print-circle to t before
>> printing the data.
>> 
>> Given that we already offer print-circle, why do we need to worry
>> further about this issue?  Is there really a need for some other
>> option intermediate between print-circle = t and print-circle = nil?
> 
> The discussion was about Quintan's proposed list logging feature, which
> doesn't write out a list in one go, but outputs it element by element.
> I was pointing out that this doesn't allow using `print-circle'.
I was indeed pointing out there is an intermediate that already happened.
It’s when you do print-circle=t in a key value store.
In such case, EQ-ness within value is preserved, but between different
keys are lost. i.e., if you store the same object into two different keys,
and read them back, potentially the values you get are no longer EQ.

Therefore I conclude that maybe preserving EQness in persistent store
does not make that much sense (otherwise why it can only be preserved
partially?).

I do see practical usefulness of partial EQness preservation.

Re: sqlite3

[LdBeth]

On Sat, 18 Dec 2021 15:53:04 +0800,
Qiantan Hong wrote:
> 
> Therefore I conclude that maybe preserving EQness in persistent store
> does not make that much sense (otherwise why it can only be preserved
> partially?).
> 
> I do see practical usefulness of partial EQness preservation.


It seems no one has yet pointed out "preserving EQness" is actually
hash-consing[1], a technique has been developed in Lisp and now well
known in Functional Programming communities.

It has main features that:

1. Keeps only one copy of (structurally) EQUAL objects and therefore
2. reduce storage usage
3. reduce the time needed to do a EQUAL comparsion to EQ.
4. useful for function memoization

It is definitely useful, but I think we could make it into a separated
package, probably.

[1]: A. .P. Ershov. On Programming of Arithmetic Operations. In the
Communications of the ACM, Volume 118, Number 3, August, 1958, pages
427–430.

Re: sqlite3

[Qiantan Hong]

> It seems no one has yet pointed out "preserving EQness" is actually
> hash-consing[1], a technique has been developed in Lisp and now well
> known in Functional Programming communities.
It is definitely not.

The feature of hash-consing is exactly not preserving EQness,
or, under hash-consing EQ and EQUAL degenerate to the same thing
(Aka, intensional and extensional identity degenerates), effectively
removing the concept of EQ and object identity from the language all 
together.

That’s the reason why hash-consing is widely used in purely functional
programs, where programs only cares about extensional identity.

EQness/object identity/intensional identity is significant under mutation,
and actually affects the semantics.

(defun eq (x y)
  (let ((probe (gensym))
        (save (car x)))
    (rplaca x probe)
    (prog1
        (eq (car y) probe)
      (rplaca x save))))

In layman’s term: even if two objects’ value look the same, they may
or may not be the same object. The difference become apparent when
you poke one, and the other may or may not change.

Hash-consing in a mutable language *does not* preserve EQ-ness, 
the object identity is arbitrarily merged, and the structure of the reference 
graph is not preserved.

Re: sqlite3

[Tomas Hlavaty]

On Sat 18 Dec 2021 at 09:06, Qiantan Hong <qhong@mit.edu> wrote:
> The feature of hash-consing is exactly not preserving EQness,
> or, under hash-consing EQ and EQUAL degenerate to the same thing
> (Aka, intensional and extensional identity degenerates), effectively
> removing the concept of EQ and object identity from the language all 
> together.

It is the other way round:  hash-consing "reduces" EQUAL to EQ.

It does not remove the concept of identity.

> In layman’s term: even if two objects’ value look the same, they may
> or may not be the same object.
> The difference become apparent when
> you poke one, and the other may or may not change.

That is a consequence of having more than one definitions of same-ness.

> Therefore I conclude that maybe preserving EQness in persistent store
> does not make that much sense (otherwise why it can only be preserved
> partially?).

It could preserve EQness if every object, where EQness has to be
preserved, was also stored under different key.

Or there would have to be a way to reference objects stored inside the
structure saved under different key.  (Some kind of extended syntax for
print-circle labels.)

Hash consing (was: Re: sqlite3)

[Po Lu]

LdBeth <andpuke@foxmail.com> writes:

> It seems no one has yet pointed out "preserving EQness" is actually
> hash-consing[1], a technique has been developed in Lisp and now well
> known in Functional Programming communities.
>
> It has main features that:
>
> 1. Keeps only one copy of (structurally) EQUAL objects and therefore
> 2. reduce storage usage
> 3. reduce the time needed to do a EQUAL comparsion to EQ.
> 4. useful for function memoization
>
> It is definitely useful, but I think we could make it into a separated
> package, probably.

I wonder how hash consing would make sense in a language like Emacs Lisp
that has `rplaca' and `rplacd'.

Perhaps we would need some kind of immutable cons data type.

Re: sqlite3

[LdBeth]

It is not about having hashconsing at the Emacs Lisp heap. I think the
context we are talking is save/retrive Lisp objects from a external
store, it maybe SQLite, maybe a text based backend.

Let's restricting it to the objects stored in that store. Apparently
you can not directly call lisp rplaca/rplacd a con cell stored in a
database, you need a certain kinds of read/write protocol, so it is
sane to assume that objects are at some level immutable in the
datastore. (Or maybe you want the object store to be very transparent,
but I guess it would be hard for a language without Metaobject
Protocol) Actually the resists!.el (posted in the other related
thread) do borrowed some concepts from immutable data storage.

Hash-consing involves the implementation detail of comparing a object
of the host language (in this case, Emacs Lisp), to something
represented inside the storage (the text, maybe). The Lisp printer
shipped with Emacs Lisp already does some kinds of comparison when
print-circle is on. It is reasonable to ask for a finer control over
the comparison method being used.

EQUAL is a concrete function, and it is possible to replace it by any
user definable equivalence relation.

On Sat, 18 Dec 2021 17:35:56 +0800,
Po Lu wrote:
> 
> I wonder how hash consing would make sense in a language like Emacs Lisp
> that has `rplaca' and `rplacd'.
> 
> Perhaps we would need some kind of immutable cons data type.


On Sat, 18 Dec 2021 17:06:22 +0800,
Qiantan Hong wrote:
> 
> LDB> It seems no one has yet pointed out "preserving EQness" is actually
> LDB> hash-consing[1], a technique has been developed in Lisp and now well
> LDB> known in Functional Programming communities.
> It is definitely not.
> 
> The feature of hash-consing is exactly not preserving EQness,
> or, under hash-consing EQ and EQUAL degenerate to the same thing
> (Aka, intensional and extensional identity degenerates), effectively
> removing the concept of EQ and object identity from the language all 
> together.
> 
> That’s the reason why hash-consing is widely used in purely functional
> programs, where programs only cares about extensional identity.
> 
> EQness/object identity/intensional identity is significant under mutation,
> and actually affects the semantics.
> 
> (defun eq (x y)
>   (let ((probe (gensym))
>         (save (car x)))
>     (rplaca x probe)
>     (prog1
>         (eq (car y) probe)
>       (rplaca x save))))
> 
> In layman’s term: even if two objects’ value look the same, they may
> or may not be the same object. The difference become apparent when
> you poke one, and the other may or may not change.
> 
> Hash-consing in a mutable language *does not* preserve EQ-ness, 
> the object identity is arbitrarily merged, and the structure of the reference 
> graph is not preserved.

Re: Hash consing (was: Re: sqlite3)

[Philipp]

> Am 18.12.2021 um 10:35 schrieb Po Lu <luangruo@yahoo.com>:
> 
> LdBeth <andpuke@foxmail.com> writes:
> 
>> It seems no one has yet pointed out "preserving EQness" is actually
>> hash-consing[1], a technique has been developed in Lisp and now well
>> known in Functional Programming communities.
>> 
>> It has main features that:
>> 
>> 1. Keeps only one copy of (structurally) EQUAL objects and therefore
>> 2. reduce storage usage
>> 3. reduce the time needed to do a EQUAL comparsion to EQ.
>> 4. useful for function memoization
>> 
>> It is definitely useful, but I think we could make it into a separated
>> package, probably.
> 
> I wonder how hash consing would make sense in a language like Emacs Lisp
> that has `rplaca' and `rplacd'.
> 
> Perhaps we would need some kind of immutable cons data type.
> 

We already have that: for example, the result of evaluating the form '(1 . 2) is an immutable cons cell.  The immutability isn't enforced, though: passing such a cons cell to `rplaca' will cause undefined behavior, but typically not signal an error.

RE: [External] : Re: Hash consing (was: Re: sqlite3)

[Drew Adams]

> > some kind of immutable cons data type.
> We already have that: for example, the result of evaluating the form '(1 .
> 2) is an immutable cons cell.  The immutability isn't enforced, though:
> passing such a cons cell to `rplaca' will cause undefined behavior, but
> typically not signal an error.

Sorry, but I don't understand what you're saying.
What's immutable about it?  What's undefined
about using `setcar' or `rplaca' on it?
 (setq x '(1 . 2))
 (setcar x 3)

RE: [External] : Re: Hash consing (was: Re: sqlite3)

[Tomas Hlavaty]

On Sat 18 Dec 2021 at 16:15, Drew Adams <drew.adams@oracle.com> wrote:
> Sorry, but I don't understand what you're saying.
> What's immutable about it?  What's undefined
> about using `setcar' or `rplaca' on it?
>  (setq x '(1 . 2))
>  (setcar x 3)

In general, don't do that.

In Common Lisp, the consequences are undefined.

In Emacs Lisp, it does not seem to be put that way, see
(describe-function 'quote)

RE: [External] : Re: Hash consing (was: Re: sqlite3)

[Drew Adams]

> > > > some kind of immutable cons data type.
> > >
> > > We already have that:

No, we don't.

> > > for example, the result of evaluating the form
> > > '(1 . 2) is an immutable cons cell.

No, it's not.

> > > The immutability isn't enforced, though:

aka There's no immutability.

> > > passing such a cons cell to `rplaca' will
> > > cause undefined behavior,

Demonstration?  What's undefined about it?

> but typically not signal an error.

> > Sorry, but I don't understand what you're saying.
> > What's immutable about it?  What's undefined
> > about using `setcar' or `rplaca' on it?
> >  (setq x '(1 . 2))
> >  (setcar x 3)
> 
> In general, don't do that.

Right.  Don't mutate that mutable cons.

> In Common Lisp, the consequences are undefined.
> 
> In Emacs Lisp, it does not seem to be put that way,
> see (describe-function 'quote)

I know why one won't want to do it.  It's precisely
because the cons is NOT immutable, and some code
might change it.

Code that mistakenly expects a source code
occurrence of '(1 . 2) to give you a _new_ cons
each time (you think) it's evaluated is bugged,
because it need not, and generally does not, do so.
That's the point of the warning in `C-h f quote'.

There's nothing "immutable" about that cons.
"Don't change this cons", and "don't expect '(...)
to give you a new cons each time", aren't the same
as "it's impossible to change this cons".

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I feel that an append-only list storage thing is a totally separate
  > thing from persistent variables.

Maybe so.  Perhaps we should look at some real use cases before
designing an append-to-list accumulating facility.

One thing that these generally need is a way to discard old elements.
We don't want history lists and logs to grow without bound.
The usual case is that normal operation adds to them, but something
or other truncates then occasionally.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: sqlite3

[Drew Adams]

> > but I didn't quite see how that would work with circular and
> > self-referential lists.)
>
> They don’t really make sense when storing to
> external storage, at least for all the mechanisms
> either you or I have. Because our storage mechanism
> doesn’t preserve intensional identity (“object identity”),
> i.e. if two objects are EQ, we store them into different key,
> they might be read back not EQ. We only store extensional
> value. In such case, circular list are no different from
> an infinite non-circular list, which doesn’t make sense.
> 
> Plus it’s not very common in practice to store circular list
> in something one may want to make persistent. Mostly
> it’s some history variable or so that is push into/delete from,
> neither will make it circular.

FWIW, I do this with bookmarks, with no problem.

But option `bmkp-propertize-bookmark-names-flag'
lets you turn this off (it's non-nil by default):

 Non-nil means to propertize bookmark names to hold full bookmark data.
 This means that you can effectively have more than one bookmark with
 the same name.

In particular, the bookmark name, which is the
car of a bookmark, is a string with a property
that holds the entire bookmark.  The bookmark
is a list with shared (circular) structure.

Emacs has no trouble writing and reading such
lists.

Perhaps you mean something special by "at least
for all the mechanisms either you or I have"?

Re: sqlite3

[Qiantan Hong]

> Emacs has no trouble writing and reading such
> lists.
> 
> Perhaps you mean something special by "at least
> for all the mechanisms either you or I have"?
I’m just saying the key value stores we have don’t 
preserve EQ-ness at the top-level. One might argue
that it is still useful to preserve EQ-ness partly
(only inside each value). That might be the case.

> We don't use circular lists a lot, but we do have rings, and it's nice
> to be able to store them.
I believe ring (as in ring.el) does not necessarily have 
circular structure?

RE: sqlite3

[Drew Adams]

Let's please be clear about who said what -

>da> Emacs has no trouble writing and reading such
>da> lists.
>da>
>da> Perhaps you mean something special by "at least
>da> for all the mechanisms either you or I have"?
>
> I’m just saying the key value stores we have don’t
> preserve EQ-ness at the top-level. One might argue
> that it is still useful to preserve EQ-ness partly
> (only inside each value). That might be the case.


>li> We don't use circular lists a lot, but we do have rings,
>li> and it's nice to be able to store them.
>
> I believe ring (as in ring.el) does not necessarily have
> circular structure?

EQ-ness

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In particular, the bookmark name, which is the
  > car of a bookmark, is a string with a property
  > that holds the entire bookmark.  The bookmark
  > is a list with shared (circular) structure.

  > Emacs has no trouble writing and reading such
  > lists.

Is print-circle non-nil when you write these values?
If not, what do they look like when they come back?
What method is used to make that generate a cyclic list?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: [External] : EQ-ness

[Drew Adams]

BTW, Richard, I often have difficulty telling what
message or sender you're replying to.  It would
help me, at least, if you made that clearer somehow.

In this case, you started a new thread (Subject) by
replying to something I wrote.  I almost didn't
notice that, and so I almost passed by your message
(there are so many to scan or read).  Anyway, just
a suggestion...

> > In particular, the bookmark name, which is the
> > car of a bookmark, is a string with a property
> > that holds the entire bookmark.  The bookmark
> > is a list with shared (circular) structure.
> 
> > Emacs has no trouble writing and reading such
> > lists.
> 
> Is print-circle non-nil when you write these values?
> If not, what do they look like when they come back?
> What method is used to make that generate a cyclic list?

Yes, sorry, guess I should have been clear about
that.  I figured it was nothing special and it
would be clear.

I redefine function `bookmark-write-file', to write
bookmarks using `pp', and with these bindings:

 (print-length  nil)
 (print-level   nil)
 (print-circle  t)
 (print-gensym  t)

[The last two are actually bound to an option, not
`t', so you can choose whether to allow for this
kind of writing.  This means choosing whether to
let the same bookmark name (with different property
values) be usable for multiple bookmarks.  E.g.,
bookmarks named with the same non-dir file-name
parts for files of the same name in different dirs.]

I put the full bookmark, which is a list with car
the bookmark name and cdr the bookmark data, onto the
bookmark name, as text property `bmkp-full-record':

 (put-text-property
   0 (length bname) 'bmkp-full-record bmk bname)

where bmk is the full bookmark.  To get back the
full bookmark from the name:

 (get-text-property 0 'bmkp-full-record bname)

The actual function used to get the full record from
the name has some optional-arg behaviors.  Here's
its doc string:
___

 (bmkp-bookmark-record-from-name BOOKMARK-NAME
                                 &optional NOERROR MEMP ALIST)

Return the full bookmark (record) that corresponds to BOOKMARK-NAME.
BOOKMARK-NAME must be a string.  If it has non-nil text property
`bmkp-full-record' then use that.  Otherwise, look for the first
bookmark in ALIST that has the given name.

Non-nil optional arg NOERROR means return nil if BOOKMARK-NAME does
not name a valid bookmark or is valid but is not in ALIST.  If NOERROR
is nil then raise an error in this case.

Non-nil optional arg MEMP means that if property `bmkp-full-record' is
available then look up its value (the full bookmark) in ALIST, testing
with `eq'.  If that record is not in ALIST, return nil.

Optional arg ALIST defaults to `bookmark-alist'.
___

I use the same "trick", BTW, to allow completion
candidates where the "full" candidate (e.g. in an
alist) is a cons whose car is the name used for
`completing-read' matching and display.

That makes it simple to get the full alist element
from the user's choice (the name that's the element's
car).  This makes it possible to have multiple
choices that have the same display name.  Why, and
how to tell which is which when they have the same
name?  By their position in the sort order, for
example (cycling etc.).

Re: [External] : EQ-ness

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > BTW, Richard, I often have difficulty telling what
  > message or sender you're replying to.

Sorry.

                                           It would
  > help me, at least, if you made that clearer somehow.

What would you suggest?  I started by quoting text -- what would
be clearer than that?

It's good to know that print-circle is working for this purpose.

  > I put the full bookmark, which is a list with car
  > the bookmark name and cdr the bookmark data, onto the
  > bookmark name, as text property `bmkp-full-record':

  >  (put-text-property
  >    0 (length bname) 'bmkp-full-record bmk bname)

Could you please confirm that I understand right?

I think you're saying that you've changed the Emacs bookmark
data structure representation.  Is that right?

Is the standard Emacs bookmark data structure circular?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: [External] : EQ-ness

[Drew Adams]

> From: Richard Stallman <rms@gnu.org>
>
>   > BTW, Richard, I often have difficulty telling what
>   > message or sender you're replying to.
> 
> Sorry.
>                                            It would
>   > help me, at least, if you made that clearer somehow.
> 
> What would you suggest?  I started by quoting text -- what would
> be clearer than that?

Maybe say who the quoted text is from, as I did here?
That's usually not necessary, of course.  Dunno what
might be a general guideline.  Not a big problem,
anyway.

> It's good to know that print-circle is working for this purpose.
> 
>   > I put the full bookmark, which is a list with car
>   > the bookmark name and cdr the bookmark data, onto the
>   > bookmark name, as text property `bmkp-full-record':
> 
>   >  (put-text-property
>   >    0 (length bname) 'bmkp-full-record bmk bname)
> 
> Could you please confirm that I understand right?
> 
> I think you're saying that you've changed the Emacs bookmark
> data structure representation.  Is that right?

No, not the bookmark structure.  Well, it depends
on what is meant...

This just uses a propertized string instead of an
unpropertized string, as the bookmark name.  (And
as I mentioned, this is optional, per a user
option.)  But a property of the string has, as
its value, the entire bookmark (a list).

> Is the standard Emacs bookmark data structure circular?

No.

As for whether this structure (a bookmark entry,
which is a list) is circular: It depends on what
you consider to be circular list structure, or a
circular list.

I've argued previously (not in this thread) that
this _is_ a kind of "circular list" because the
car is a string with a property whose value is
the whole list (the bookmark entry).  Some
disagreed about the terminology.

The self-referentialness here is from the string
pointing (by way of a property) to a list that
contains that same string as its car.  The list
is "circular" in that its car in some way points
to the overall  list itself.

Re: [External] : EQ-ness

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I've argued previously (not in this thread) that
  > this _is_ a kind of "circular list" because the
  > car is a string with a property whose value is
  > the whole list (the bookmark entry).  Some
  > disagreed about the terminology.

I agree with you thjt this is circular structure.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Alexandre Garreau]

Le mardo, 14-a de decembro 2021, 16-a horo kaj 41:29 CET Lars Ingebrigtsen 
a écrit :
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> 
> 
> > A backend to multisession that handles list-based updates might be
> > nice, but it's not part of the design specs.
> 
> (I considered making `files' be able to specify the data type, and a
> list could be represented by a number of elements read independently
> that could be appended to the file:
> 
> foo
> bar
> (baz)
> 
> would be restored as
> 
> (foo bar (baz))
> 
> but I didn't quite see how that would work with circular and
> self-referential lists.)

foo
bar
#1=(baz . #0#)
#1#

Re: sqlite3

[Stefan Monnier]

> but I didn't quite see how that would work with circular and
> self-referential lists.)

AFAIK the general way to avoid these issues is to store/log not the
"data-diff" but the higher-level operation that caused this diff.
E.g. log something like "add X to tree" instead of recording which nodes
in the tree were modified in which way.  This way, the presence or
absence of cycles in the representation of the tree doesn't come into
the picture at all.


        Stefan

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021, 15-a horo kaj 6:52 CET Stefan Monnier 
a écrit :
> > but I didn't quite see how that would work with circular and
> > self-referential lists.)
> 
> AFAIK the general way to avoid these issues is to store/log not the
> "data-diff" but the higher-level operation that caused this diff.
> E.g. log something like "add X to tree" instead of recording which nodes
> in the tree were modified in which way.  This way, the presence or
> absence of cycles in the representation of the tree doesn't come into
> the picture at all.

Looks like Qiantan’s implementation of incremental log-like store.

Re: sqlite3

[Stefan Monnier]

>> > but I didn't quite see how that would work with circular and
>> > self-referential lists.)
>> 
>> AFAIK the general way to avoid these issues is to store/log not the
>> "data-diff" but the higher-level operation that caused this diff.
>> E.g. log something like "add X to tree" instead of recording which nodes
>> in the tree were modified in which way.  This way, the presence or
>> absence of cycles in the representation of the tree doesn't come into
>> the picture at all.
>
> Looks like Qiantan’s implementation of incremental log-like store.

Indeed.  BTW, there's another approach which, instead of looking at the
higher-level API looks at the lower level steps (like "change this slot
to that value" where a value can be a reference), so adding an element
to a doubly linked list would record a set of 4 slot changes.


        Stefan

Re: sqlite3

[Qiantan Hong]

> Indeed.  BTW, there's another approach which, instead of looking at the
> higher-level API looks at the lower level steps (like "change this slot
> to that value" where a value can be a reference), so adding an element
> to a doubly linked list would record a set of 4 slot changes.
I considered this approach very early on and have probably mentioned it
in the thread… let me see… https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00513.html
The problem is that how we let user inform us about those changes.
It either require significant change to the VM, or have to ask user
to use variants of all mutation operation explicitly.


Best,
Qiantan

Re: sqlite3

[Dmitry Gutov]

On 14.12.2021 11:57, Lars Ingebrigtsen wrote:
> As for the sqlite part of this: My initial benchmarking of this was
> wrong.  I thought sqlite3 was going to be a real advantage for this
> thing, since I'd benchmarked excellent performance (more than 50K
> updates per second, for instance).  But that's only when not committing
> after every transaction, which we want to do here, really.

I'm guessing just the ability to avoid committing after every 
transaction (do it on a timer instead, perhaps) might become an 
advantage at some point.

But a "proper" database might give other advantages like a faster search 
in the loaded data (unless it's already "indexed" by using hash tables 
everywhere where they could be used).

Or being able to read the data without loading the whole file into 
memory. Which, for certain scenarios and data sets, might be a bigger 
advantage than faster writes.

> But it turns out that sqlite3 is actually slower for this particular use
> case than just writing the data to a file (i.e., using the file system
> as the database; one file per value).  So multisession.el now offers two
> backends (`files' and `sqlite'), and defaults to `files'.

Does the latter scenario use as many files as you do 'COMMIT' in the 
former scenario?

Re: sqlite3

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> But a "proper" database might give other advantages like a faster
> search in the loaded data (unless it's already "indexed" by using hash
> tables everywhere where they could be used).
>
> Or being able to read the data without loading the whole file into
> memory. Which, for certain scenarios and data sets, might be a bigger
> advantage than faster writes.

Here's a matrix of advantages and disadvantages to three approaches:
sqlite, one-file-per-value, and
one-file-with-a-hash-table-with-several/all-values:
  
                           sqlite  files  hash
Read/write value speed     ⚄       ⚅     ⚀
Read/write value mem       ⚅       ⚄     ⚀
List all values speed      ⚅       ⚀     ⚅
List all values mem        ⚃       ⚁     ⚃
Ease of moving around      ⚄       ⚀     ⚅

>> But it turns out that sqlite3 is actually slower for this particular use
>> case than just writing the data to a file (i.e., using the file system
>> as the database; one file per value).  So multisession.el now offers two
>> backends (`files' and `sqlite'), and defaults to `files'.
>
> Does the latter scenario use as many files as you do 'COMMIT' in the
> former scenario?

No, if you (cl-incf (multisession-value foo)) you'll get one COMMIT per
time, but there'll only be one foo.value file (at a time).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Qiantan Hong]

> Here's a matrix of advantages and disadvantages to three approaches:
> sqlite, one-file-per-value, and
> one-file-with-a-hash-table-with-several/all-values:
> 
>                           sqlite  files  hash
> Read/write value speed     ⚄       ⚅     ⚀
> Read/write value mem       ⚅       ⚄     ⚀
> List all values speed      ⚅       ⚀     ⚅
> List all values mem        ⚃       ⚁     ⚃
> Ease of moving around      ⚄       ⚀     ⚅
I also have an incremental log backend. Maybe I can add it to the
comparison.
                        sqlite  files  hash   log
Read/write value speed     ⚄       ⚅     ⚀    ⚀/⚅
Read/write value mem       ⚅       ⚄     ⚀    ⚀/⚅
List all values speed      ⚅       ⚀     ⚅     ⚃
List all values mem        ⚃       ⚁     ⚃     ⚃
Ease of moving around      ⚄       ⚀     ⚅     ⚅

Re: sqlite3

[Dmitry Gutov]

On 14.12.2021 19:43, Lars Ingebrigtsen wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> But a "proper" database might give other advantages like a faster
>> search in the loaded data (unless it's already "indexed" by using hash
>> tables everywhere where they could be used).
>>
>> Or being able to read the data without loading the whole file into
>> memory. Which, for certain scenarios and data sets, might be a bigger
>> advantage than faster writes.
> 
> Here's a matrix of advantages and disadvantages to three approaches:
> sqlite, one-file-per-value, and
> one-file-with-a-hash-table-with-several/all-values:
>    
>                             sqlite  files  hash
> Read/write value speed     ⚄       ⚅     ⚀
> Read/write value mem       ⚅       ⚄     ⚀
> List all values speed      ⚅       ⚀     ⚅
> List all values mem        ⚃       ⚁     ⚃
> Ease of moving around      ⚄       ⚀     ⚅

I'm not 100% sure how to interpret (is a higher value for "mem" better 
or worse?), but it seems like, at least, for the original scenario of 
having large data sets sqlite might still be optimal.

>>> But it turns out that sqlite3 is actually slower for this particular use
>>> case than just writing the data to a file (i.e., using the file system
>>> as the database; one file per value).  So multisession.el now offers two
>>> backends (`files' and `sqlite'), and defaults to `files'.
>>
>> Does the latter scenario use as many files as you do 'COMMIT' in the
>> former scenario?
> 
> No, if you (cl-incf (multisession-value foo)) you'll get one COMMIT per
> time, but there'll only be one foo.value file (at a time).

OK, but it's still the same number of writes, more or less? IO is the 
slow part of most programs, and when it comes to an SQL database, it 
might have to do an update in multiple places (e.g. the data and the 
index), rather than do one smooth write.

Might also depend on the size of the write (how big the values are).

Speaking of the latter scheme, I might be missing some details, but 
sqlite should provide better atomicity guarantess in the same of being 
interrupted mid-write. Like, if we have one-file-per-value, then the 
total list of keys must live somewhere, and they can get desynchronized.

Re: sqlite3

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

>>                                sqlite  files  hash
>> Read/write value speed     ⚄       ⚅     ⚀
>> Read/write value mem       ⚅       ⚄     ⚀
>> List all values speed      ⚅       ⚀     ⚅
>> List all values mem        ⚃       ⚁     ⚃
>> Ease of moving around      ⚄       ⚀     ⚅
>
> I'm not 100% sure how to interpret (is a higher value for "mem" better
> or worse?), but it seems like, at least, for the original scenario of
> having large data sets sqlite might still be optimal.

⚅ is better than ⚄ etc.

> OK, but it's still the same number of writes, more or less?

Yup.  

> IO is the slow part of most programs, and when it comes to an SQL
> database, it might have to do an update in multiple places (e.g. the
> data and the index), rather than do one smooth write.
>
> Might also depend on the size of the write (how big the values are).

I think the main issue is probably that sqlite tries really hard to
maintain data integrity, so it does fsyncing etc.  (More or less
according to what pragmas you give it.)  It's nice in general, but
probably overkill for this use case.

> Speaking of the latter scheme, I might be missing some details, but
> sqlite should provide better atomicity guarantess in the same of being
> interrupted mid-write. Like, if we have one-file-per-value, then the
> total list of keys must live somewhere, and they can get
> desynchronized.

There is no total list of keys in multisession.el -- you define separate
objects that are each instantiated to disk.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >                            sqlite  files  hash
  > Read/write value speed     ⚄       ⚅     ⚀
  > Read/write value mem       ⚅       ⚄     ⚀
  > List all values speed      ⚅       ⚀     ⚅
  > List all values mem        ⚃       ⚁     ⚃
  > Ease of moving around      ⚄       ⚀     ⚅

The value are all chars my terminal does not display.  Would someone please
resend this using characters that generally work?
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   >                            sqlite  files  hash
>   > Read/write value speed     ⚄       ⚅     ⚀
>   > Read/write value mem       ⚅       ⚄     ⚀
>   > List all values speed      ⚅       ⚀     ⚅
>   > List all values mem        ⚃       ⚁     ⚃
>   > Ease of moving around      ⚄       ⚀     ⚅
>
> The value are all chars my terminal does not display.  Would someone please
> resend this using characters that generally work?

FWIW, they don't display in a way that makes sense on my X display
either.  They are made out of confusing Emoji.

Please resend this in some fashion that most people can understand,
thanks.

Re: sqlite3

[LdBeth]

On Thu, 16 Dec 2021 12:39:42 +0800,
Richard Stallman wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   >                            sqlite  files  hash
>   > Read/write value speed     ⚄       ⚅     ⚀
>   > Read/write value mem       ⚅       ⚄     ⚀
>   > List all values speed      ⚅       ⚀     ⚅
>   > List all values mem        ⚃       ⚁     ⚃
>   > Ease of moving around      ⚄       ⚀     ⚅
> 
> The value are all chars my terminal does not display.  Would someone please
> resend this using characters that generally work?
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
> 
> 
> 
                            [Score: x out of 6]
                            sqlite  files  hash
 Read/write value speed     5       6      1
 Read/write value mem       6       5      1
 List all values speed      6       1      6
 List all values mem        4       2      4
 Ease of moving around      5       1      6

--
LDB

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

   > Read/write value speed     5       6      1
   ...

Thanks.

Does "files" mean "write a separate file for each value
and use the key as its file name?

What method does "hash" mean?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> Does "files" mean "write a separate file for each value
> and use the key as its file name?

Yes, separate file per value. The file name is irrelevant for this
particular comparison.

> What method does "hash" mean?

Saving all the values in a single hash table object that it written to
disk.

Best,
Ihor

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Labeling them as "one per file" and "print all together"
would make each message clearer.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Tomas Hlavaty]

On Tue 14 Dec 2021 at 09:57, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> But it turns out that sqlite3 is actually slower for this particular use
> case than just writing the data to a file (i.e., using the file system
> as the database; one file per value).  So multisession.el now offers two
> backends (`files' and `sqlite'), and defaults to `files'.

So it turned out that sqlite is actually slower
but Emacs is stuck with sqlite dependency anyway?
Is my understanding correct?

Re: sqlite3

[Lars Ingebrigtsen]

Tomas Hlavaty <tom@logand.com> writes:

> So it turned out that sqlite is actually slower
> but Emacs is stuck with sqlite dependency anyway?
> Is my understanding correct?

Are you asking whether Emacs now supports reading, listing and editing
sqlite files?  The answer is yes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Tomas Hlavaty]

On Tue 14 Dec 2021 at 17:59, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>> So it turned out that sqlite is actually slower
>> but Emacs is stuck with sqlite dependency anyway?
>> Is my understanding correct?
>
> Are you asking whether Emacs now supports reading, listing and editing
> sqlite files?  The answer is yes.

No, the issue is not if Emacs supports sqlite.
Emacs supports many things depending on the definition of "support".

The issue is that Emacs now depends on sqlite.
Even though the reason for introducing this dependency
turned out to be wrong.
The arguments for introducing this dependency as a linked library
dismissed other options as wrong and even "not smart".
Now it turned out that all those arguments were wrong.
But the sqlite dependency is going to stay.
So it actually turned out that the only reason for pushing this
dependency was that some people want it but have no good reason
to justify it.
I find that really puzzling.
That is why I ask if my understanding is correct.
Maybe there was some good reason after all, which I missed.
If not, I would expect Emacs to use the proposed superior pure lisp solution
and removing the sqlite dependency completely.
Otherwise it is no wonder that Emacs dependencies are exploding.

Re: sqlite3

[Lars Ingebrigtsen]

Tomas Hlavaty <tom@logand.com> writes:

> The issue is that Emacs now depends on sqlite.

Emacs depends on libsqlite3 the same way it depends on librsvg.

> So it actually turned out that the only reason for pushing this
> dependency was that some people want it but have no good reason
> to justify it.

I think having database support in Emacs is going to be useful, and
there's been a number of things that's been suggested could use it
instead of the current slow mechanisms, like the Gnus registry and
org-roam.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Tomas Hlavaty]

On Tue 14 Dec 2021 at 19:03, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>
>> The issue is that Emacs now depends on sqlite.
>
> Emacs depends on libsqlite3 the same way it depends on librsvg.

At the moment.

And those are very different things with different consequences.

> I think having database support in Emacs is going to be useful,

That seems rather vague reason.

"My" Emacs supported RDBMS since more than a decade ago:

   https://logand.com/sw/dbquery/file/README.html
   https://logand.com/sw/dbquery/file/dbquery.el.html
   https://logand.com/sw/dbquery/files.html

   dbquery is a collection of sql drivers and high level code for
   Emacs-Lisp and Common Lisp with easy to use text based protocol based
   on symbolic expressions.

I guess other people did something similar.  It is not difficult.

But the question is, should a database be in core Emacs?
Should a database be in the Emacs process?

What will be the consequences?

> and there's been a number of things that's been suggested could use it
> instead of the current slow mechanisms, like the Gnus registry and
> org-roam.

The consequences will likely be, that more and more things will not work
without that database.

You suggest two examples.
I can see more exciting developments, e.g.
imagine elfeed does not block Emacs for seconds before quitting;
or imagine elfeed with full text search;
or notmuch email reader using Emacs built in database.

But I am still not sure including sqlite is a good thing.
I imagine that once sqlite is in Emacs, it will be easier to
dump everything into it and call it a day.

Imagine git built on top of sqlite.
Would that have been better?

Is there really not a better solution than depending on sqlite?

I am quite curious, how this will turn out.

Re: sqlite3

[Lele Gaifax]

Tomas Hlavaty <tom@logand.com> writes:

> Imagine git built on top of sqlite.
> Would that have been better?

Well, that's https://fossil-scm.org and from some POVs yes, it's better than
git :-)

ciao, lele.
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it  |                 -- Fortunato Depero, 1929.

Re: sqlite3

[tomas]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]

On Wed, Dec 15, 2021 at 08:46:55AM +0100, Lele Gaifax wrote:
> Tomas Hlavaty <tom@logand.com> writes:
> 
> > Imagine git built on top of sqlite.
> > Would that have been better?
> 
> Well, that's https://fossil-scm.org and from some POVs yes, it's better than
> git :-)

Well, I've known people who think SVN is better than git, then ;-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021, 8-a horo kaj 46:55 CET Lele Gaifax a 
écrit :
> Tomas Hlavaty <tom@logand.com> writes:
> 
> 
> 
> > Imagine git built on top of sqlite.
> > Would that have been better?
> 
> Well, that's https://fossil-scm.org and from some POVs yes, it's better
> than git :-)

I was going to say it but I lost the mail, the threads is so lengthy and 
grows so fast x)

Re: sqlite3

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Tue, 14 Dec 2021 22:55:01 +0100
> Cc: emacs-devel@gnu.org
> 
> But the question is, should a database be in core Emacs?
> Should a database be in the Emacs process?

We decided it could be a good idea, so yes.  Until and unless there's
serious evidence, preferably based on actual experience, that it is a
bad idea.  We are not there yet.

> What will be the consequences?

Good question.  Let's wait and see what are the answers.

> The consequences will likely be, that more and more things will not work
> without that database.

Let's not pretend that we know the consequences, let alone can judge
whether they are good or otherwise, before those consequences actually
are about to happen, let alone already did.

> I am quite curious, how this will turn out.

We all are, I think.  So let's wait and see.  Continued arguing about
this, with little or no facts to show, doesn't help us in this regard.

Re: sqlite3

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Tue, 14 Dec 2021 18:58:37 +0100
> Cc: emacs-devel@gnu.org
> 
> The issue is that Emacs now depends on sqlite.

No, it doesn't.  You can build Emacs without it.

> Even though the reason for introducing this dependency
> turned out to be wrong.

That conclusion is premature, from where I stand.

> The arguments for introducing this dependency as a linked library
> dismissed other options as wrong and even "not smart".

That is at least inaccurate and unfair.  Opinions were heard and
considered.  There's no obligation to agree with everything that's
being said here.

> Now it turned out that all those arguments were wrong.

Now, it didn't.

> But the sqlite dependency is going to stay.

That is still under discussion and study.  Please don't jump to
conclusions, and please don't misrepresent our stand on this matter.

> So it actually turned out that the only reason for pushing this
> dependency was that some people want it but have no good reason
> to justify it.

That is unfair and even hostile.  I don't think you have any reason to
judge us so harshly.

> I find that really puzzling.
> That is why I ask if my understanding is correct.

If the above is your understanding, it is definitely incorrect.

> If not, I would expect Emacs to use the proposed superior pure lisp solution
> and removing the sqlite dependency completely.

No solution is superior in all use cases.  This is software
engineering, after all.

> Otherwise it is no wonder that Emacs dependencies are exploding.

They aren't.

Re: sqlite3

[Tomas Hlavaty]

On Tue 14 Dec 2021 at 20:53, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tomas Hlavaty <tom@logand.com>
> That is unfair and even hostile.  I don't think you have any reason to
> judge us so harshly.

I am sorry if it sounded harsh.

I certainly value your work on Emacs a lot.

Lars' work too, as I was impressed how quickly he understood and
implemented a suggestion proposed some time ago.

>> I find that really puzzling.
>> That is why I ask if my understanding is correct.
>
> If the above is your understanding, it is definitely incorrect.

Thanks for clarification.

It is valuable experience to see reasoning in such a high profile
project like Emacs.

In this case of sqlite, it seemed that good arguments raised by others
and suggested pure lisp implementation were dismissed without good
reason.  It seems to me that e.g. Qiantan raised better arguments.

I guess it must be hard to develop with so many devils advocates :-)

>> If not, I would expect Emacs to use the proposed superior pure lisp solution
>> and removing the sqlite dependency completely.
>
> No solution is superior in all use cases.  This is software
> engineering, after all.

I was addressing the discussed use-cases, not all possible use-cases.

Re: sqlite3

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Tue, 14 Dec 2021 21:21:11 +0100
> 
> > No solution is superior in all use cases.  This is software
> > engineering, after all.
> 
> I was addressing the discussed use-cases, not all possible use-cases.

The use case of accessing existing SQL databases was also discussed.
And there's no reason to believe no one will ever want to build a DB
application in Emacs Lisp, and no reason to leave that arena only to
Python, JS, and other environments that already have SQLite bindings.

So yes, potential use cases do exist where sqlite3 will be a bonus,
even before we consider Emacs-specific features that perhaps could
benefit from that.  I would even say that SQLite support is more
important than xwidgets, for example.

Re: sqlite3

[Qiantan Hong]

> On Dec 14, 2021, at 12:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Tomas Hlavaty <tom@logand.com>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Tue, 14 Dec 2021 21:21:11 +0100
>> 
>>> No solution is superior in all use cases.  This is software
>>> engineering, after all.
>> 
>> I was addressing the discussed use-cases, not all possible use-cases.
> 
> The use case of accessing existing SQL databases was also discussed.
> And there's no reason to believe no one will ever want to build a DB
> application in Emacs Lisp, and no reason to leave that arena only to
> Python, JS, and other environments that already have SQLite bindings.
> 
> So yes, potential use cases do exist where sqlite3 will be a bonus,
> even before we consider Emacs-specific features that perhaps could
> benefit from that.  I would even say that SQLite support is more
> important than xwidgets, for example.
From what I understand, Tomas’ argument was that for the above use case,
linking SQLite3 as a dynamic library to the Emacs core is not strictly better.
A subprocess approach also works very well, as in several existing packages.

There’s basically no way to integrate xwidgets as an independent package/without 
linking into Emacs, on the other hand, because it changes display code.

The above is just to point out some gaps in the reasoning. That doesn’t mean
your conclusion is right or wrong. From my understanding the single strongest
argument for adding SQLite3 support into core is that other part of the core
(say gnus) might want to depend on it.

Is there a good way that makes some core component (optionally) depends
on non-core package? This might also become a problem for experimenting
my store.el, because one PoC would be providing a backend for org-id, but
orgmode is in the core.

Re: sqlite3

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Tomas Hlavaty <tom@logand.com>, "larsi@gnus.org" <larsi@gnus.org>,
>         "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Tue, 14 Dec 2021 20:38:15 +0000
> 
> > The use case of accessing existing SQL databases was also discussed.
> > And there's no reason to believe no one will ever want to build a DB
> > application in Emacs Lisp, and no reason to leave that arena only to
> > Python, JS, and other environments that already have SQLite bindings.
> > 
> > So yes, potential use cases do exist where sqlite3 will be a bonus,
> > even before we consider Emacs-specific features that perhaps could
> > benefit from that.  I would even say that SQLite support is more
> > important than xwidgets, for example.
> From what I understand, Tomas’ argument was that for the above use case,
> linking SQLite3 as a dynamic library to the Emacs core is not strictly better.
> A subprocess approach also works very well, as in several existing packages.

We will have to agree to disagree about this, and if Thomas indeed
meant that, I disagree with him as well.

> The above is just to point out some gaps in the reasoning.

There are no gaps in reasoning, just different judgment of proper
solutions to certain engineering problems.  What is the best solution
is in many cases a judgment call, not a self-evident conclusion based
on reasoning.

> Is there a good way that makes some core component (optionally) depends
> on non-core package?

We can arrange for an option that could be used if people install some
3rd-party package, yes.

Re: sqlite3

[Madhu]

* Eli Zaretskii <83bl1ijb3q.fsf@gnu.org> :
Wrote on Wed, 15 Dec 2021 14:39:37 +0200:

>> From: Qiantan Hong <qhong@mit.edu>
>> Date: Tue, 14 Dec 2021 20:38:15 +0000
>> > The use case of accessing existing SQL databases was also
>> > discussed.  And there's no reason to believe no one will ever want
>> > to build a DB application in Emacs Lisp, and no reason to leave
>> > that arena only to Python, JS, and other environments that already
>> > have SQLite bindings.
>> > So yes, potential use cases do exist where sqlite3 will be a bonus,
>> > even before we consider Emacs-specific features that perhaps could
>> > benefit from that.  I would even say that SQLite support is more
>> > important than xwidgets, for example.
>> From what I understand, Tomas’ argument was that for the above use case,
>> linking SQLite3 as a dynamic library to the Emacs core is not strictly better.
>> A subprocess approach also works very well, as in several existing packages.
>
> We will have to agree to disagree about this, and if Thomas indeed
> meant that, I disagree with him as well.
>
>> The above is just to point out some gaps in the reasoning.
>
> There are no gaps in reasoning, just different judgment of proper
> solutions to certain engineering problems.  What is the best solution
> is in many cases a judgment call, not a self-evident conclusion based
> on reasoning.
>
>> Is there a good way that makes some core component (optionally) depends
>> on non-core package?
>
> We can arrange for an option that could be used if people install some
> 3rd-party package, yes.

I didn't like situation with json. There is lisp/json.el (in elisp) and
there is src/json.c which is incompatible.  In a world of ideal emacs
design (like a decade or two ago) I'd have been able to use the
interface which emacs provides and switch the implementation, (say if
the C version was not available).

This was emacs philosophy as i understood it. advice.el was one
example. The APIs are sound and not dependent on implementation bits and
the implementation could be (and was) switched underneath.  Now all the
funding seems to be in tying APIs that emacs provides to the
implementation.

the network interface was another example.  When Emacs is not compiled
without gnutls it was possible to use openssl s_client to get a network
interface and get the job done.  (This has been obsoleted and requires
work before it can work) - and emacs user is now constrained to either
use gnutls or live without ssl, and this is a social-engineering not an
engineering decision.  It was perfectly possible for ssl to work emacs
in an emacs compiled without gnutls, by falling back to s_client, but
this is no longer possible because of decisions which are calculated to
remove that freedom and tie things to an implementation.

And this loss of freedom is implemented in a subtle way, (by the sort of
arguments which have been presented upthread which invoked gnutls)

Re: sqlite3

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Thu, 16 Dec 2021 15:18:30 +0530
> 
> I didn't like situation with json. There is lisp/json.el (in elisp) and
> there is src/json.c which is incompatible.  In a world of ideal emacs
> design (like a decade or two ago) I'd have been able to use the
> interface which emacs provides and switch the implementation, (say if
> the C version was not available).

I think the idea was that no one will want to stay with json.el
because its performance in important applications is abysmally
inadequate.  So investing efforts in making json.el compatible was
deemed a waste of resources.

> the network interface was another example.  When Emacs is not compiled
> without gnutls it was possible to use openssl s_client to get a network
> interface and get the job done.  (This has been obsoleted and requires
> work before it can work) - and emacs user is now constrained to either
> use gnutls or live without ssl, and this is a social-engineering not an
> engineering decision.

IMNSHO, the solution of using starttls external program for SSL
connections was a terrible design (have an external program deliver
signals to Emacs to communicate with it? really?), and I think its
deprecation had this as one of its main reasons.  The other reason was
portability.

Is availability of GnuTLS really a problem nowadays?  I'd be surprised
if it was a problem of any significance.

> And this loss of freedom is implemented in a subtle way, (by the sort of
> arguments which have been presented upthread which invoked gnutls)

The basic argument in the sub-thread to which you responded was that
we add dependencies to Emacs without a good reason, and that
alternative solutions could be available to cover the same needs.  I
think the two examples you provided do not provide justification for
that argument, as valid technical reasons led us to do what we did in
both cases.  If you perceive the results of those technical decisions
as "loss of freedom", then I guess I'll have to disagree.

Re: sqlite3

[LdBeth]

On Thu, 16 Dec 2021 17:48:30 +0800,
Madhu wrote:
> 
> I didn't like situation with json. There is lisp/json.el (in elisp) and
> there is src/json.c which is incompatible.  In a world of ideal emacs
> design (like a decade or two ago) I'd have been able to use the
> interface which emacs provides and switch the implementation, (say if
> the C version was not available).

The current state of src/json.c is not still as useful as json.el, For
example, it is not possible to deserialize the JSON string to a
hashtable with its value type as alist.

And for `json-serialize', it can only accepts

    (json-serialize '((a . [1 2]) (b . [3 4])))

rather than

    (json-serialize '((a 1 2) (b . (3 4))))

and it is harder add something to a vector than list.

So right now I would still go for the `princ' method for serializing a
hashtable of this kind, with is been used by LSDB,
http://sourceforge.jp/projects/lsdb/

-- 
LDB

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I guess it must be hard to develop with so many devils advocates :-)

I hope anyone tempted to play "devil's advocate" will reconsider and
not do that.  It is never helpful to present arguments you don't
sincerely believe.

See https://gnu.org/philosophy/devils-advocate.html.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, larsi@gnus.org, emacs-devel@gnu.org
> Date: Fri, 17 Dec 2021 23:40:37 -0500
> 
>   > I guess it must be hard to develop with so many devils advocates :-)
> 
> I hope anyone tempted to play "devil's advocate" will reconsider and
> not do that.  It is never helpful to present arguments you don't
> sincerely believe.
> 
> See https://gnu.org/philosophy/devils-advocate.html.

This is a tangent, so I'll try to be very short.

I disagree with Richard on this point, and I disagree with that essay.
IME, hearing devil's advocates is very important for cleansing my
ideas and thoughts, and for making good balanced decisions.  Moreover,
in many rational organizations, the decision-making process requires
the participation of the so-called "red team", which plays the role of
an adversary ("devil") with the purpose of making the decisions sound
and sustainable.  So others have found this useful.

Every technique can be subverted by using it with hostile or cunning
motives, of course, but the ability to discover the hostile intent (or
just those with some agenda) and ignore some of such "devil's
advocates" is part of the necessary process of considering some idea
or decision.

devil's advocate

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I wish I had changed the subject line before, but better late than
never.

  > I disagree with Richard on this point, and I disagree with that essay.
  > IME, hearing devil's advocates is very important for cleansing my
  > ideas and thoughts, and for making good balanced decisions.

I think we are talking about two different practices.

As I explain in the article, people who are "playing devil's advocate"
are not trying to help you make a good decision; they are imitating
someone who is trying to fight you, to see what you would do under
attack.

They don't look for valid arguments; rather, they will say what they
they imagine your worst enemies would say.  That typically includes
fallacies, mistaken assumptions, and distorted goals -- because your
worst enemies would surely use those.

You're talking about people who sincerely want to help, and argue for
a different course of action using valid arguments, based on our real
goals.  That's not playing devil's advocate, they are honestly
disagreeing.

To find such arguments, they need to start with a sincere helpful
attitude.  When people do that, they can be very helpful.
That's totally different from playing devil's advocate.

  > in many rational organizations, the decision-making process requires
  > the participation of the so-called "red team", which plays the role of
  > an adversary ("devil") with the purpose of making the decisions sound
  > and sustainable.

I have never seen this in use.  Is the "red team" charged to look only
for valid and pertinent arguments for changing the plan?  Or is it
supposed to treat the question as a real fight, trying to defeat the
plan by hook or by crook, and cheer if the plan is stopped?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: devil's advocate

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: tom@logand.com, larsi@gnus.org, emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 15:38:20 -0500
> 
> I think we are talking about two different practices.
> 
> As I explain in the article, people who are "playing devil's advocate"
> are not trying to help you make a good decision; they are imitating
> someone who is trying to fight you, to see what you would do under
> attack.

Such an imitation, if it's a good one, can be a valuable tool for
looking at my positions and decisions from a different perspective,
and for making my positions and ideas more clear and correct.

> They don't look for valid arguments; rather, they will say what they
> they imagine your worst enemies would say.

My worst enemies don't necessarily bring up only invalid arguments.
When they do, it is my job to detect the invalid arguments and
disregard them.

> That typically includes fallacies, mistaken assumptions, and
> distorted goals -- because your worst enemies would surely use
> those.

The art of discerning between truth and fallacies is an important
one.  How do I know that every one of my arguments and assumptions
is valid and isn't based on a fallacy?  Confronting those with the
ones provided by an opponent helps me make this distinction, and find
fallacies in my own opinions and ideas.

> You're talking about people who sincerely want to help, and argue for
> a different course of action using valid arguments, based on our real
> goals.  That's not playing devil's advocate, they are honestly
> disagreeing.

I'm not interested in "honesty"; that is almost immaterial.  If the
arguments are good and valid, the honesty of their proponents is
irrelevant.

> To find such arguments, they need to start with a sincere helpful
> attitude.

Opponents and adversaries normally don't want to help me, they want to
convince me that they are right and I'm wrong, or cause me to fail in
achieving my goals.  All I'm interested in is to hear those arguments
and take the useful parts out of them.  And there are always useful
parts, in many cases a lot of them, IME.

> That's totally different from playing devil's advocate.

I will be the one who decides whether it helps me or not.  The intent
of the opponent is not important, only what he/she says is important,
and how I use what he/she says for my purposes.

>   > in many rational organizations, the decision-making process requires
>   > the participation of the so-called "red team", which plays the role of
>   > an adversary ("devil") with the purpose of making the decisions sound
>   > and sustainable.
> 
> I have never seen this in use.  Is the "red team" charged to look only
> for valid and pertinent arguments for changing the plan?  Or is it
> supposed to treat the question as a real fight, trying to defeat the
> plan by hook or by crook

They need to play the adversary, and do it well.  Whether that
includes "by hook or by crook" depends on the domain; for example, in
Cyber warfare it's entirely expected.  The red team is also supposed
to cast doubt on every single assumption that my decisions or plans
are based on, and ask the "what if" questions that perhaps I didn't
ask myself.

> and cheer if the plan is stopped?

That is entirely irrelevant.  Their job is done when they succeed in
pointing out problematic assumptions and weak points in my decisions
or plans, which cause me to modify the plan, prepare plan B, etc.

Re: devil's advocate

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > As I explain in the article, people who are "playing devil's advocate"
  > > are not trying to help you make a good decision; they are imitating
  > > someone who is trying to fight you, to see what you would do under
  > > attack.

  > Such an imitation, if it's a good one, can be a valuable tool for
  > looking at my positions and decisions from a different perspective,
  > and for making my positions and ideas more clear and correct.

If you want to practice responding to enemies, people whose
goal isn't to be helpful, that could be a useful way to do it.

  > > That typically includes fallacies, mistaken assumptions, and
  > > distorted goals -- because your worst enemies would surely use
  > > those.

  > The art of discerning between truth and fallacies is an important
  > one.  How do I know that every one of my arguments and assumptions
  > is valid and isn't based on a fallacy?  Confronting those with the
  > ones provided by an opponent helps me make this distinction, and find
  > fallacies in my own opinions and ideas.

You don't need to encourage devil's advocates to get opportunities to
deal with errors.  Sincere people who are not our enemies do make
errors in reasoning.

In addition, sincere people may recognize your responses as valid --
whereas real enemies, and people mimicking enemies, will generally
cling stubbirnly to the erroneous attacks.

  > Their job is done when they succeed in
  > pointing out problematic assumptions and weak points in my decisions
  > or plans, which cause me to modify the plan, prepare plan B, etc.

It sounds like what you seek is sincere, thoughtful criticism.
An unscrupulous enemy, or someone mimicking an unscrupulous enemy,
will give you less of that.

If you appreciate arguing with enemies, would you please invite them
to do that with you personally?  This list is for trying to _help_ the
development of Emacs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: devil's advocate

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I have never seen this in use.  Is the "red team" charged to look only
  > > for valid and pertinent arguments for changing the plan?  Or is it
  > > supposed to treat the question as a real fight, trying to defeat the
  > > plan by hook or by crook

  > They need to play the adversary, and do it well.  Whether that
  > includes "by hook or by crook" depends on the domain; for example, in
  > Cyber warfare it's entirely expected.

I think you have changed the subject subtly, from design to security,
and extended the issue of devils advocates in an incorrect way.

If pentesters find a security flaw, and they demonstrate it, that is a
real criticism, not a fake one.  The equivalent of devils advocates in
pentesting, if such a thing is possible, would be people who smear you
about presumed security weaknesses without bothering to find them.

Let's move this to emacs-tangents.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Tomas Hlavaty]

On Fri 17 Dec 2021 at 23:40, Richard Stallman <rms@gnu.org> wrote:
>   > I guess it must be hard to develop with so many devils advocates :-)
>
> I hope anyone tempted to play "devil's advocate" will reconsider and
> not do that.  It is never helpful to present arguments you don't
> sincerely believe.
>
> See https://gnu.org/philosophy/devils-advocate.html.

Interesting.  Sorry for using wrong idiom.

Re: sqlite3

[dick]

TH> The issue is that Emacs now depends on sqlite.

The incongruous mentions of hyper-specific "sqlite" in base-level lisp.h
notwithstanding, this is hardly an "issue."

The typical user neither notices nor cares.  The atypical user who would
make a fuss about sqlite3 knows how to build without it.

Re: sqlite3

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Tue, 14 Dec 2021 17:52:31 +0100
> 
> On Tue 14 Dec 2021 at 09:57, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> > But it turns out that sqlite3 is actually slower for this particular use
> > case than just writing the data to a file (i.e., using the file system
> > as the database; one file per value).  So multisession.el now offers two
> > backends (`files' and `sqlite'), and defaults to `files'.
> 
> So it turned out that sqlite is actually slower
> but Emacs is stuck with sqlite dependency anyway?
> Is my understanding correct?

Please don't blow things out of proportion.  No one is stuck with
anything.  Emacs can be compiled without sqlite3, and v29 is still a
very long way before a release, so we have a lot of time to study how
useful is this and decide whether to make it the default etc.  This is
development, so please let us do our job and develop Emacs as best as
we can.  I assure you no one here wants to do any harm to Emacs, and
your opinions and the opinions of others are heard and taken into
consideration, even though we don't always agree.

I don't really understand the attitude.

Re: sqlite3

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

* 2021-12-14 17:52:31+0100, Tomas Hlavaty wrote:

> So it turned out that sqlite is actually slower but Emacs is stuck
> with sqlite dependency anyway? Is my understanding correct?

Don't forget that having a good support for a relational database opens
new possibilities. It may not be the fastest way of storing individual
values but sometimes a relational database and its search capabilities
are necessary.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 434 bytes --]

Re: sqlite3

[Qiantan Hong]

> On Dec 14, 2021, at 10:12 AM, Teemu Likonen <tlikonen@iki.fi> wrote:
> 
> * 2021-12-14 17:52:31+0100, Tomas Hlavaty wrote:
> 
>> So it turned out that sqlite is actually slower but Emacs is stuck
>> with sqlite dependency anyway? Is my understanding correct?
> 
> Don't forget that having a good support for a relational database opens
> new possibilities. It may not be the fastest way of storing individual
> values but sometimes a relational database and its search capabilities
> are necessary.

I think the argument below is still mostly sound:
> On Dec 14, 2021, at 9:58 AM, Tomas Hlavaty <tom@logand.com> wrote:
> 
> No, the issue is not if Emacs supports sqlite.
> Emacs supports many things depending on the definition of "support".
> 
> The issue is that Emacs now depends on sqlite.
> Even though the reason for introducing this dependency
> turned out to be wrong.

There’re lots of other ways to support relational databases in Emacs,
and they already exist outside of Emacs core. If using Emacs as a UI/client
to search/query capability of a underlying SQL database, it’s not likely
that it requires as much data to go through Emacs comparing to say
use database as a store facility for Elisp programs. And it’s totally viable
to just use subprocess mechanism, as in several existing packages
outside of Emacs core.

One may have to add it to Emacs core because some other part of
Emacs core need to use it. It’s not clear whether db should be added
to core or those parts should be moved out of, but it’s irrelevant to
the “possibility” argument. The possibility already exists because good
support for sqlite exists as independent packages.

Re: sqlite3

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

* 2021-12-14 18:29:49+0000, Qiantan Hong wrote:

> There’re lots of other ways to support relational databases in Emacs,
> and they already exist outside of Emacs core.

Relax. SQLite support makes Emacs much cooler. :-) People who don't want
new features can always do

    ./configure --without-anything

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 434 bytes --]

Re: sqlite3

[Tomas Hlavaty]

On Tue 14 Dec 2021 at 21:06, Teemu Likonen <tlikonen@iki.fi> wrote:
> * 2021-12-14 18:29:49+0000, Qiantan Hong wrote:
>
>> There’re lots of other ways to support relational databases in Emacs,
>> and they already exist outside of Emacs core.
>
> Relax. SQLite support makes Emacs much cooler. :-) People who don't want
> new features can always do
>
>     ./configure --without-anything

That's not how it works btw.
Dependencies are "transitive", if that's the right word.
If you do that, you'll loose everything that depends on it.

Re: sqlite3

[Jean Louis]

* Tomas Hlavaty <tom@logand.com> [2021-12-14 19:53]:
> On Tue 14 Dec 2021 at 09:57, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> > But it turns out that sqlite3 is actually slower for this particular use
> > case than just writing the data to a file (i.e., using the file system
> > as the database; one file per value).  So multisession.el now offers two
> > backends (`files' and `sqlite'), and defaults to `files'.
> 
> So it turned out that sqlite is actually slower
> but Emacs is stuck with sqlite dependency anyway?
> Is my understanding correct?

Reference for more research:

35% Faster Than The Filesystem
https://www.sqlite.org/fasterthanfs.html

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[Alexandre Garreau]

Le mardo, 14-a de decembro 2021, 9-a horo kaj 57:51 CET Lars Ingebrigtsen 
a écrit :
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> 
> > I think it'd be good to bring that into core, and then write a small
> > wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> > that we don't have to write SQL all over the place.  That is:
> > 
> > (setf (persistent-data :namespace "emoji" :key "favorites")
> > emoji--favorites)
> > 
> > I.e., what Emacs needs is a persistent key/value store, and this would
> > give us that.
> 
> This is now implemented under the name of `multisession-value' (but not
> merged to the trunk yet; I'll probably do that later today, unless
> somebody comes up with a better name).
> 
> As for the sqlite part of this: My initial benchmarking of this was
> wrong.  I thought sqlite3 was going to be a real advantage for this
> thing, since I'd benchmarked excellent performance (more than 50K
> updates per second, for instance).  But that's only when not committing
> after every transaction, which we want to do here, really.
> 
> But it turns out that sqlite3 is actually slower for this particular use
> case than just writing the data to a file (i.e., using the file system
> as the database; one file per value).  So multisession.el now offers
> two backends (`files' and `sqlite'), and defaults to `files'.

What about having an intermediary abstraction for persistent stores in 
general, and on top of that multisession variables?

What about a all-variables-in-one-file very trivial backend? a gdbm 
backend? would that be possible/allowable if gdbm is only integrated as a 
module?

What fs did you benchmark with? did you try with an old-linked-list-like 
ext4, or a modern btree btrfs?  I think the later might (unportably, 
though) display some advantages that sqlite (portably) provides.

gdbm might provide some others.  I would find benchmarking of all these 
relevant and interesting (and why not of qiantan’s incremental’s store as 
well, if he integrates it)

Re: sqlite3

[Qiantan Hong]

> What about having an intermediary abstraction for persistent stores in 
> general, and on top of that multisession variables?
I think Lars’ multisession variable is basically a strict subset of persistent store,
and can be implemented with almost trivial wrappers if a persistent store
interface is presented. So if we do have a persistent store interface, the
multisession.el interface is unnecessary. A more high-level/transparent persistent
variable interface, instead, is probably more desired.

My plan, store.el + persist.el, follows exactly this direction. It seems that
the Emacs maintainers are unwilling to include them in place of 
what they currently have, however, so guess we have to put it on Elpa
first.

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021 3-a horo kaj 18:53 CET, vous avez écrit 
:
> > What about having an intermediary abstraction for persistent stores in
> > general, and on top of that multisession variables?
> 
> I think Lars’ multisession variable is basically a strict subset of
> persistent store, and can be implemented with almost trivial wrappers
> if a persistent store interface is presented.

No, from what I understood multisession *is* not a store (although it 
*uses* one (or two, now)) but an implementation of persistent variables.

Iiuc it doesn’t expose a stable abstraction layer for a kv-store, right?

> So if we do have a
> persistent store interface, the multisession.el interface is
> unnecessary. A more high-level/transparent persistent variable
> interface, instead, is probably more desired.

Why don’t you hack multisession so that it includes a such layer, between 
the current highest-level interface, the choice between sqlite/files? even 
before that, you could add your incremental-log-store to them, and also, 
why not, at least for benchmark sake, a store where one file contains 
everything such as custom.el

> My plan, store.el + persist.el, follows exactly this direction. It seems
> that the Emacs maintainers are unwilling to include them in place of
> what they currently have, however, so guess we have to put it on Elpa
> first.

apparently they already have an implementation for that, maybe you should 
work with theirs? at least it would advance

I see no reason why ppl would use your implementation, even if it’s more 
richly designed, if it doesn’t provide something that’s very new and not 
already present.  I find your incremental store interesting but not 
appealing, while your abstraction still doesn’t allow using sqlite as one 
of its backends.

Re: sqlite3

[Qiantan Hong]

> No, from what I understood multisession *is* not a store (although it 
> *uses* one (or two, now)) but an implementation of persistent variables.
It’s closer to a kv store than persistent variables. As I mentioned in an
earlier message, kv store and persistent variables themselves are close,
but in multisession.el case, it’s closer to key value store.
(defmacro (multisession-value symbol)
  `(store-get ‘,symbol multisession-global-store))
Persistent variable (in the context of my proposal) is something more transparent,
that works without the need to wrap (some-accessor symbol) at every places.
Namely, they works exactly like normal variables, just the value cell is
preserved across session.
> Iiuc it doesn’t expose a stable abstraction layer for a kv-store, right?
Yes.
>> So if we do have a
>> persistent store interface, the multisession.el interface is
>> unnecessary. A more high-level/transparent persistent variable
>> interface, instead, is probably more desired.
> 
> Why don’t you hack multisession so that it includes a such layer, between 
> the current highest-level interface, the choice between sqlite/files? even 
> before that, you could add your incremental-log-store to them, and also, 
> why not, at least for benchmark sake, a store where one file contains 
> everything such as custom.el
I already have such a layer, if you look at store.el (not sure if you’ve already, but just
let me paste the link here)
https://code.librehq.com/qhong/store.el

>> My plan, store.el + persist.el, follows exactly this direction. It seems
>> that the Emacs maintainers are unwilling to include them in place of
>> what they currently have, however, so guess we have to put it on Elpa
>> first.
> 
> apparently they already have an implementation for that, maybe you should 
> work with theirs? at least it would advance
The problem is not about implementation, it’s about interface.
Lars say the multisession.el interface will not change.
> I see no reason why ppl would use your implementation,
I’m not pushing my implementation, I’m pushing my interface.
From the context I infer that you haven’t look at store.el,
so I guess things will become clear after you look at it.
> even if it’s more 
> richly designed, if it doesn’t provide something that’s very new and not 
> already present.  I find your incremental store interesting but not 
> appealing, while your abstraction still doesn’t allow using sqlite as one 
> of its backends.
The abstraction apparently do allow.

Re: sqlite3

[Qiantan Hong]

> Why don’t you hack multisession so that it includes a such layer, between 
> the current highest-level interface, the choice between sqlite/files? even 
> before that, you could add your incremental-log-store to them, and also, 
> why not, at least for benchmark sake, a store where one file contains 
> everything such as custom.el
I realized some serious problem about file-per-key method, I see Lars
also use it so guess it will help to discuss.
Basically, printing the key as file name does not guarantee distinguishing
the key
1. Even if one escape the key, the printed representation may be larger
  than 255 bytes, or 8bytes in FAT32 (idk if any one still use it). In such case,
  two different key may give the same filename because the prefix is
  the same
2. What’s worse, some FS are case insensitive.

>> AFAIK the general way to avoid these issues is to store/log not the
>> "data-diff" but the higher-level operation that caused this diff.
>> E.g. log something like "add X to tree" instead of recording which nodes
>> in the tree were modified in which way.  This way, the presence or
>> absence of cycles in the representation of the tree doesn't come into
>> the picture at all.
> 
> Looks like Qiantan’s implementation of incremental log-like store.
I think maybe this hinted us that we can get the best out of both
by using filename as a “bucket splitter”, and have an incremental
log store in each file? Then we can be loose about file name normalization
because it’s just an optimization (and make it more readable).
It’s also almost as easy for people to explore as one-file-per-key.

Re: sqlite3

[Lars Ingebrigtsen]

Qiantan Hong <qhong@mit.edu> writes:

> Basically, printing the key as file name does not guarantee distinguishing
> the key
> 1. Even if one escape the key, the printed representation may be larger
>   than 255 bytes, or 8bytes in FAT32 (idk if any one still use it). In
> such case,
>   two different key may give the same filename because the prefix is
>   the same

Well, I don't think we care about FAT, but I think it'd be fine to limit
the key length to 255 bytes.

> 2. What’s worse, some FS are case insensitive.

That's a concern, yes, but again it's not particularly relevant for
persistent variables.  (If you have variables in the same package that
only differs by case, I don't know what to say.)

I did consider just using sha1 of all the keys as the file names, and
then storing the actual key in the file as well, but it seems like
over-engineering (and it makes things less understandable for users
poking around).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Alexandre Garreau]

Le merkredo, 15-a de decembro 2021, 16-a horo kaj 36:19 CET Qiantan Hong a 
écrit :
> > the current highest-level interface, the choice between sqlite/files?
> > even before that, you could add your incremental-log-store to them,
> > and also, why not, at least for benchmark sake, a store where one
> > file contains everything such as custom.el
> 
> I realized some serious problem about file-per-key method, I see Lars
> also use it so guess it will help to discuss.
> Basically, printing the key as file name does not guarantee
> distinguishing the key
> 1. Even if one escape the key, the printed representation may be larger
>   than 255 bytes, or 8bytes in FAT32 (idk if any one still use it). In
> such case, two different key may give the same filename because the
> prefix is the same
> 2. What’s worse, some FS are case insensitive.

elisp symbols are case insensitive as well, it’s a lisp-2, of the kind of 
cl

also the canonical *standard* for naming elisp variables is a subset of 
what is allowed for filenames.  nobody seriously quotes their variable 
names or exceed the 72 columns limit with a single variable name and no 
indentation, so it might be okay to throw an error in edge cases, for 
portability… or maybe just a warning

plus, you could also hash the names, so you get only base 16, or 32, or 
something alike

> >> AFAIK the general way to avoid these issues is to store/log not the
> >> "data-diff" but the higher-level operation that caused this diff.
> >> E.g. log something like "add X to tree" instead of recording which
> >> nodes in the tree were modified in which way.  This way, the
> >> presence or absence of cycles in the representation of the tree
> >> doesn't come into the picture at all.
> 
> 
> 
> > Looks like Qiantan’s implementation of incremental log-like store.
> 
> I think maybe this hinted us that we can get the best out of both
> by using filename as a “bucket splitter”, and have an incremental
> log store in each file? Then we can be loose about file name
> normalization because it’s just an optimization (and make it more
> readable). It’s also almost as easy for people to explore as
> one-file-per-key.

why are you complicating that much???

Re: sqlite3

[Qiantan Hong]

> elisp symbols are case insensitive as well, it’s a lisp-2, of the kind of 
> cl
Nah, Elisp fixed it. Also it’s unrelated to lisp-N

(eq 'test ‘TEST)
nil

> also the canonical *standard* for naming elisp variables is a subset of 
> what is allowed for filenames.  nobody seriously quotes their variable 
> names or exceed the 72 columns limit with a single variable name and no 
> indentation, so it might be okay to throw an error in edge cases, for 
> portability… or maybe just a warning
> 
> plus, you could also hash the names, so you get only base 16, or 32, or 
> something alike
> 
>>>> AFAIK the general way to avoid these issues is to store/log not the
>>>> "data-diff" but the higher-level operation that caused this diff.
>>>> E.g. log something like "add X to tree" instead of recording which
>>>> nodes in the tree were modified in which way.  This way, the
>>>> presence or absence of cycles in the representation of the tree
>>>> doesn't come into the picture at all.
>> 
>> 
>> 
>>> Looks like Qiantan’s implementation of incremental log-like store.
>> 
>> I think maybe this hinted us that we can get the best out of both
>> by using filename as a “bucket splitter”, and have an incremental
>> log store in each file? Then we can be loose about file name
>> normalization because it’s just an optimization (and make it more
>> readable). It’s also almost as easy for people to explore as
>> one-file-per-key.
> 
> why are you complicating that much???
Looks like filename isn’t doing the correct thing, and single
file log has scaling issue. And now a correct and scalable solution
appears, so probably I should do it? Classic MITer don’t care
that much about implementation simplicity after all ;)

Re: sqlite3

[Alexandre Garreau]

Le ĵaŭdo, 16-a de decembro 2021, 6-a horo kaj 9:23 CET Qiantan Hong a 
écrit :
> > elisp symbols are case insensitive as well, it’s a lisp-2, of the kind
> > of 
 cl
> 
> Nah, Elisp fixed it.

> (eq 'test ‘TEST)
> nil
 
Ok, thank you for correction


> Also it’s unrelated to lisp-N

I should have said “traditional lisp”: there’s some kind of division 
between scheme-like and all previous lisp in my mind

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Can you explain the meaning of the term "multisession variables"?
Also, how does it differ from "persistent" variables?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Can you explain the meaning of the term "multisession variables"?
> Also, how does it differ from "persistent" variables?

It's a new name Lars picked for the persistent variable feature, as
`persistent-variable' was taken by some EIEIO related thing.

Thanks.

Re: sqlite3

[Qiantan Hong]

Persistent variable is being used to refer to different thing,
and have certainly caused confusion in the discussion.
I’m consistent in my terminology, at least, and I will try to
distinguish here precisely:

I now distinguish the following 3 things by their interface for
accessing the data:
persistent store (PS), Lars’ multisession variable (LMV),
and Qiantan’s persistent variable (QPV):
(Each of them also have an interface for defining the
storage/variable, but is not very different in that aspect)

PS: is a generalized variable of the form
(store-get KEY STORE)

LMV: is a generalized variable of the form 
(multisession-value MULTISESSION-OBJ)

QPV: is a generalized variable of the form
VAR (aka, at the place of usage, is no different from an
ordinary variable)

In terms of expressivity:

PS cannot be simulated with LMV or QPV.

LMV can be simulated with PS
(defun multisession-value (obj)
   (store-get obj global-store))
(plus a similar definition for setf)

LMV can be simulated with QPV
(defun multisession-value (obj)
   (symbol-value obj))

QPV can be implemented with PS,
but fairly non trivially (and it is how I implement it)
QPV cannot be simulated with LMV.

Finally, my comments: LMV is closer to PS and could be viewed
as a specialization of PS. QPV is different from the other two,
and its major feature is that one could make a variable persistent
without needing to change any code already using that variable
(i.e. QPV interface is transparent).


Best,
Qiantan

Re: sqlite3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But it turns out that sqlite3 is actually slower for this particular use
  > case than just writing the data to a file (i.e., using the file system
  > as the database; one file per value).

Is that terribly wasteful of space?

I have a feeling it must be.  But I don't know how ext file systems
organize disk space, so I could be mistaken.  Does anyone know?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.  That is:
>
> (setf (persistent-data :namespace "emoji" :key "favorites") emoji--favorites)
>
> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.

I've now pushed this to the trunk, and, indeed, I'm using the
emoji--recent thing as the test case.

Things should basically work, but more testing would be nice.  And there
may be changes in how stuff is stored, so nobody should rely on the
data yet.  (I don't expect it to change, but just a heads up.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Things should basically work, but more testing would be nice.

I'm seeing some oddnesses I don't understand in the `files' backend on
Windows.

If I write-region to a file, and then rename the file to "foo.value" in
one Emacs, then another Emacs may or may not get a file-error like this:

    (file-error "Opening input file" "Permission denied" "c:/Users/larsi/AppData/Local/Temp/emacs-test-nJO8TU-multisession/files/sbar/sbar.value")

There's probably something about file locking on Windows that I don't
understand.  (Well, it's a dead certainty that there's a lot about
Windows I don't understand...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 16 Dec 2021 08:10:35 +0100
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Things should basically work, but more testing would be nice.
> 
> I'm seeing some oddnesses I don't understand in the `files' backend on
> Windows.
> 
> If I write-region to a file, and then rename the file to "foo.value" in
> one Emacs, then another Emacs may or may not get a file-error like this:
> 
>     (file-error "Opening input file" "Permission denied" "c:/Users/larsi/AppData/Local/Temp/emacs-test-nJO8TU-multisession/files/sbar/sbar.value")

Is this code on master?  If so, can you show some recipe to reproduce
this?

> There's probably something about file locking on Windows that I don't
> understand.

Who does the file locking in this case, and by what means?

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> If I write-region to a file, and then rename the file to "foo.value" in
>> one Emacs, then another Emacs may or may not get a file-error like this:
>> 
>>     (file-error "Opening input file" "Permission denied"
>> "c:/Users/larsi/AppData/Local/Temp/emacs-test-nJO8TU-multisession/files/sbar/sbar.value")
>
> Is this code on master?  If so, can you show some recipe to reproduce
> this?

It's the multi-test-files-simple code in multisession-tests.el (but you
have to revert ed77d1a8c0 to see it).

>> There's probably something about file locking on Windows that I don't
>> understand.
>
> Who does the file locking in this case, and by what means?

That's what I don't know.

What I think the code does is this:

1) First Emacs 1 writes the file.
2) Then I start another Emacs, and it also writes the file, and then that
Emacs exits.
3) Then Emacs 1 wants to read the file, and it fails with that error.

If Emacs 1 waits a bit, then it's apparently allowed to open that file,
but I'm not quite sure about that.

Which is where I gave up trying to debug this and posted the mail.  😀

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 17 Dec 2021 08:38:44 +0100
> 
> >>     (file-error "Opening input file" "Permission denied"
> >> "c:/Users/larsi/AppData/Local/Temp/emacs-test-nJO8TU-multisession/files/sbar/sbar.value")
> >
> > Is this code on master?  If so, can you show some recipe to reproduce
> > this?
> 
> It's the multi-test-files-simple code in multisession-tests.el (but you
> have to revert ed77d1a8c0 to see it).
> 
> >> There's probably something about file locking on Windows that I don't
> >> understand.
> >
> > Who does the file locking in this case, and by what means?
> 
> That's what I don't know.
> 
> What I think the code does is this:
> 
> 1) First Emacs 1 writes the file.
> 2) Then I start another Emacs, and it also writes the file, and then that
> Emacs exits.
> 3) Then Emacs 1 wants to read the file, and it fails with that error.
> 
> If Emacs 1 waits a bit, then it's apparently allowed to open that file,
> but I'm not quite sure about that.

If Emacs 1 attempts to open and read the file while Emacs 2 still
writes to it, Windows will not let Emacs 1 open the file, and the
failure will be EACCES, as your report shows.  That's because the
default way of opening files which we use in Emacs opens them in
"unshared" mode, so they cannot be shared between different processes.

Which probably means multisession.el should have some mechanism to
deal with this gracefully, like retrying the open/read several times
before giving up.  After all, the window for such races should be
quite short, given that we write very small files.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If Emacs 1 attempts to open and read the file while Emacs 2 still
> writes to it, Windows will not let Emacs 1 open the file, and the
> failure will be EACCES, as your report shows.  That's because the
> default way of opening files which we use in Emacs opens them in
> "unshared" mode, so they cannot be shared between different processes.

But in this case, Emacs 2 should have exited already before we try to
read the file...  I think.  I'll try to debug this more thoroughly;
perhaps I was misinterpreting what I was seeing.

> Which probably means multisession.el should have some mechanism to
> deal with this gracefully, like retrying the open/read several times
> before giving up.  After all, the window for such races should be
> quite short, given that we write very small files.

Yup.  It does this in the sqlite case, so I'll replicate the logic in
the files case, too.  But it seemed very mysterious to me that I was
getting a permission denied after Emacs 2 had exited.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 17 Dec 2021 09:25:51 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If Emacs 1 attempts to open and read the file while Emacs 2 still
> > writes to it, Windows will not let Emacs 1 open the file, and the
> > failure will be EACCES, as your report shows.  That's because the
> > default way of opening files which we use in Emacs opens them in
> > "unshared" mode, so they cannot be shared between different processes.
> 
> But in this case, Emacs 2 should have exited already before we try to
> read the file...  I think.  I'll try to debug this more thoroughly;
> perhaps I was misinterpreting what I was seeing.
> 
> > Which probably means multisession.el should have some mechanism to
> > deal with this gracefully, like retrying the open/read several times
> > before giving up.  After all, the window for such races should be
> > quite short, given that we write very small files.
> 
> Yup.  It does this in the sqlite case, so I'll replicate the logic in
> the files case, too.  But it seemed very mysterious to me that I was
> getting a permission denied after Emacs 2 had exited.

If you are sure the second session already exited, then my hand-waving
doesn't hold water.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If you are sure the second session already exited, then my hand-waving
> doesn't hold water.

I thought I was pretty sure, and...  today I can't reproduce the problem
at all.  😐 It was 100% reproducible yesterday, but not today.  (And I
reverted the change I made to handle the problem.)

OK, I ran the tests a whole bunch of times now, and it failed once, so
the problem is still present, but hard to reproduce.  I'll keep poking
at it, but I probably won't have time to continue today.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 17 Dec 2021 10:31:25 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If you are sure the second session already exited, then my hand-waving
> > doesn't hold water.
> 
> I thought I was pretty sure, and...  today I can't reproduce the problem
> at all.  😐 It was 100% reproducible yesterday, but not today.  (And I
> reverted the change I made to handle the problem.)
> 
> OK, I ran the tests a whole bunch of times now, and it failed once, so
> the problem is still present, but hard to reproduce.  I'll keep poking
> at it, but I probably won't have time to continue today.

Is that multi-test-files-simple?  If so, it almost always fails here
today:

  Test multi-test-files-simple condition:
      (ert-test-failed
       ((should
	 (=
	  (multisession-value sfoo)
	  2))
	:form
	(= 1 2)
	:value nil))

Which sounds like a problem different from what you saw?

Adding (sleep-for 3) after call-process doesn't entirely fix the
problem, although it becomes more rare.

I've installed a change in multisession.el that makes the problem much
more rare by forcing 'fsync' when we close the value file.  The
problem is not yet gone entirely, though: it happens with about 25%
probability here.  I think it could be related to the fact that
'rename-file' is not an atomic operation on MS-Windows. But that's a
wild guess, I have no basis for it.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Which sounds like a problem different from what you saw?

It may stem from the (incomplete) fix I added for the original problem,
though.

> Adding (sleep-for 3) after call-process doesn't entirely fix the
> problem, although it becomes more rare.
>
> I've installed a change in multisession.el that makes the problem much
> more rare by forcing 'fsync' when we close the value file.  The
> problem is not yet gone entirely, though: it happens with about 25%
> probability here.  I think it could be related to the fact that
> 'rename-file' is not an atomic operation on MS-Windows. But that's a
> wild guess, I have no basis for it.

Right.  I'll do some further testing, but the real fix is to add some
backoff logic when the file is locked, which I'll do now.  (And in that
case, I think the `fsync' should be unnecessary.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Right.  I'll do some further testing, but the real fix is to add some
> backoff logic when the file is locked, which I'll do now.  (And in that
> case, I think the `fsync' should be unnecessary.)

Now done, but since I'm no longer seeing the errors, I can't test
whether this fixes the issue, so can you check?

Also -- it would be better to catch just the "Permission denied" error
instead of all file errors in the `condition-case' -- but I guess we
don't signal a more specific error for that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 18 Dec 2021 09:28:07 +0100
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Right.  I'll do some further testing, but the real fix is to add some
> > backoff logic when the file is locked, which I'll do now.  (And in that
> > case, I think the `fsync' should be unnecessary.)
> 
> Now done, but since I'm no longer seeing the errors, I can't test
> whether this fixes the issue, so can you check?

It still fails from time to time here.  I have an SSD disk, maybe
that's a factor, somehow.

> Also -- it would be better to catch just the "Permission denied" error
> instead of all file errors in the `condition-case' -- but I guess we
> don't signal a more specific error for that?

Yes, we do: write-region calls report_file_errno, which includes the
specific error in the error data.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> It still fails from time to time here.  I have an SSD disk, maybe
> that's a factor, somehow.

SSD here, too.

>> Also -- it would be better to catch just the "Permission denied" error
>> instead of all file errors in the `condition-case' -- but I guess we
>> don't signal a more specific error for that?
>
> Yes, we do: write-region calls report_file_errno, which includes the
> specific error in the error data.

You mean in the string "Permission denied"?  Can we rely on that not
being localised?  (Looking at the Windows code, it seems like we're
making the string ourselves, but is that also the case on other systems?)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 11:56:31 +0100
> 
> >> Also -- it would be better to catch just the "Permission denied" error
> >> instead of all file errors in the `condition-case' -- but I guess we
> >> don't signal a more specific error for that?
> >
> > Yes, we do: write-region calls report_file_errno, which includes the
> > specific error in the error data.
> 
> You mean in the string "Permission denied"?  Can we rely on that not
> being localised?  (Looking at the Windows code, it seems like we're
> making the string ourselves, but is that also the case on other systems?)

We could invent a new symbol for EACCES (like 'file-missing' for
ENOENT we already have), and return that instead in this case, instead
of the general 'file-error'.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We could invent a new symbol for EACCES (like 'file-missing' for
> ENOENT we already have), and return that instead in this case, instead
> of the general 'file-error'.

Yes, I think that would be a good idea.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 12:47:18 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We could invent a new symbol for EACCES (like 'file-missing' for
> > ENOENT we already have), and return that instead in this case, instead
> > of the general 'file-error'.
> 
> Yes, I think that would be a good idea.

Is the below good enough?  If so, I will install it.

diff --git a/doc/lispref/errors.texi b/doc/lispref/errors.texi
index f848218..9dd052c 100644
--- a/doc/lispref/errors.texi
+++ b/doc/lispref/errors.texi
@@ -98,6 +98,10 @@ Standard Errors
 @item file-already-exists
 This is a subcategory of @code{file-error}.  @xref{Writing to Files}.
 
+@item permission-denied
+This is a subcategory of @code{file-error}, which occurs when the OS
+doesn't allow Emacs to access a file or a directory for some reason.
+
 @item file-date-error
 This is a subcategory of @code{file-error}.  It occurs when
 @code{copy-file} tries and fails to set the last-modification time of
diff --git a/etc/NEWS b/etc/NEWS
index 862621a..24f3da8 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1186,6 +1186,12 @@ The events 'touchscreen-begin, 'touchscreen-update', and
 'touchscreen-end' have been added to take better advantage of
 touch-capable display panels.
 
++++
+** New error symbol 'permission-denied'.
+This is a subcategory of 'file-error', and is signaled when some file
+operation fails because the OS doesn't allow Emacs to access a file or
+a directory.
+
 \f
 * Changes in Emacs 29.1 on Non-Free Operating Systems
 
diff --git a/src/fileio.c b/src/fileio.c
index a0563cc..03977ae 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -195,7 +195,11 @@ get_file_errno_data (char const *string, Lisp_Object name, int errorno)
   if (errorno == EEXIST)
     return Fcons (Qfile_already_exists, errdata);
   else
-    return Fcons (errorno == ENOENT ? Qfile_missing : Qfile_error,
+    return Fcons (errorno == ENOENT
+		  ? Qfile_missing
+		  : (errorno == EACCES
+		     ? Qpermission_denied
+		     : Qfile_error),
 		  Fcons (build_string (string), errdata));
 }
 
@@ -6380,6 +6384,7 @@ syms_of_fileio (void)
   DEFSYM (Qfile_already_exists, "file-already-exists");
   DEFSYM (Qfile_date_error, "file-date-error");
   DEFSYM (Qfile_missing, "file-missing");
+  DEFSYM (Qpermission_denied, "permission-denied");
   DEFSYM (Qfile_notify_error, "file-notify-error");
   DEFSYM (Qremote_file_error, "remote-file-error");
   DEFSYM (Qexcl, "excl");

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Is the below good enough?  If so, I will install it.

Looks good to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 15:11:25 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Is the below good enough?  If so, I will install it.
> 
> Looks good to me.

Unfortunately, this breaks filelock-tests.el in some strange ways:
even if I replace file-error with permission-denied, the tests fail,
and it seems like should-error doesn't catch the error.  That is, the
failure is not in comparing the error data with the expected value.
Does ERT somehow know all the error symbols or something?

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 15:11:25 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Is the below good enough?  If so, I will install it.
> 
> Looks good to me.

Unfortunately, this breaks filelock-tests.el in some strange ways:
even if I replace file-error with permission-denied, the tests fail,
and it seems like should-error doesn't catch the error.  That is, the
failure is not in comparing the error data with the expected value.
Does ERT somehow know all the error symbols or something?  Or what am
I missing?

Re: sqlite3

[Eli Zaretskii]

> Date: Sun, 19 Dec 2021 17:08:20 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Unfortunately, this breaks filelock-tests.el in some strange ways:
> even if I replace file-error with permission-denied, the tests fail,
> and it seems like should-error doesn't catch the error.  That is, the
> failure is not in comparing the error data with the expected value.
> Does ERT somehow know all the error symbols or something?  Or what am
> I missing?

Found what I was missing.  Will install in a couple of minutes.

Re: sqlite3

[Eli Zaretskii]

> Date: Sun, 19 Dec 2021 17:24:53 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Date: Sun, 19 Dec 2021 17:08:20 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: emacs-devel@gnu.org
> > 
> > Unfortunately, this breaks filelock-tests.el in some strange ways:
> > even if I replace file-error with permission-denied, the tests fail,
> > and it seems like should-error doesn't catch the error.  That is, the
> > failure is not in comparing the error data with the expected value.
> > Does ERT somehow know all the error symbols or something?  Or what am
> > I missing?
> 
> Found what I was missing.  Will install in a couple of minutes.

Done.  Running the tests now to find any that could be affected.

Re: sqlite3

[Eli Zaretskii]

> Date: Sun, 19 Dec 2021 17:54:33 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Date: Sun, 19 Dec 2021 17:24:53 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: emacs-devel@gnu.org
> > 
> > > Date: Sun, 19 Dec 2021 17:08:20 +0200
> > > From: Eli Zaretskii <eliz@gnu.org>
> > > Cc: emacs-devel@gnu.org
> > > 
> > > Unfortunately, this breaks filelock-tests.el in some strange ways:
> > > even if I replace file-error with permission-denied, the tests fail,
> > > and it seems like should-error doesn't catch the error.  That is, the
> > > failure is not in comparing the error data with the expected value.
> > > Does ERT somehow know all the error symbols or something?  Or what am
> > > I missing?
> > 
> > Found what I was missing.  Will install in a couple of minutes.
> 
> Done.  Running the tests now to find any that could be affected.

All done, looks like we are ready to use this new error symbol.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> All done, looks like we are ready to use this new error symbol.

Thanks; I've adjusted the code...  but Savannah seems to be down, so I
can't push or pull?  *sigh*

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 20 Dec 2021 10:30:28 +0100
> Cc: emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > All done, looks like we are ready to use this new error symbol.
> 
> Thanks; I've adjusted the code...  but Savannah seems to be down, so I
> can't push or pull?  *sigh*

The multi-test-files-busy test is sometimes failing:

  insert-file-contents("c:/DOCUME~1/Zaretzky/LOCALS~1/Temp/emacs-test-
  [...]
  Test multi-test-files-busy condition:
      (file-missing "Opening input file" "No such file or directory" "c:/DOCUME~1/Zaretzky/LOCALS~1/Temp/emacs-test-0HFBx5-multisession/files/multisession/multisession--sbar.value")

This is because rename-file is not an atomic operation on MS-Windows:
when the target file already exists, we delete it and then rename the
new file.  So there's a small window between the delete and the rename
that the file doesn't exist under its expected name, and the test
fails.

Does multisession.el know that a value file was already created and
was available at some point in the past?  If so, could it perhaps test
the file for existence before re-reading it, and if it doesn't exist,
retry a few times, sleeping for 10 msec between retries?  (Only on
MS-Windows, of course.)  That might solve the problem.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The multi-test-files-busy test is sometimes failing:

Ah, right.  That's more understandable -- the one I couldn't wrap my
head around was `multi-test-files-simple' failing, because the other
Emacs should have exited and the file should be in an unambiguous state
then the first Emacs reads it.

Have the changes made `multi-test-files-simple' never fail now?

>   insert-file-contents("c:/DOCUME~1/Zaretzky/LOCALS~1/Temp/emacs-test-
>   [...]
>   Test multi-test-files-busy condition:
>       (file-missing "Opening input file" "No such file or directory" "c:/DOCUME~1/Zaretzky/LOCALS~1/Temp/emacs-test-0HFBx5-multisession/files/multisession/multisession--sbar.value")
>
> This is because rename-file is not an atomic operation on MS-Windows:
> when the target file already exists, we delete it and then rename the
> new file.  So there's a small window between the delete and the rename
> that the file doesn't exist under its expected name, and the test
> fails.
>
> Does multisession.el know that a value file was already created and
> was available at some point in the past?  If so, could it perhaps test
> the file for existence before re-reading it, and if it doesn't exist,
> retry a few times, sleeping for 10 msec between retries?  (Only on
> MS-Windows, of course.)  That might solve the problem.

Yes, that sounds like a reasonable approach.  I'll have a go at it, but
I probably won't have time over the next few days, so if you want to
tinker with this instead, that's fine with me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 22 Dec 2021 13:10:40 +0100
> 
> Have the changes made `multi-test-files-simple' never fail now?

Not in my testing so far, but I didn't run the test enough times.  I
will need to revisit.

> Yes, that sounds like a reasonable approach.  I'll have a go at it, but
> I probably won't have time over the next few days, so if you want to
> tinker with this instead, that's fine with me.

I will try, but with the current flood on the mailing lists, I cannot
promise anything.  (Don't you guys have holidays to attend to? ;-)

Re: sqlite3

[Eli Zaretskii]

> Date: Wed, 22 Dec 2021 15:51:22 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: emacs-devel@gnu.org
> > Date: Wed, 22 Dec 2021 13:10:40 +0100
> > 
> > Have the changes made `multi-test-files-simple' never fail now?
> 
> Not in my testing so far, but I didn't run the test enough times.  I
> will need to revisit.
> 
> > Yes, that sounds like a reasonable approach.  I'll have a go at it, but
> > I probably won't have time over the next few days, so if you want to
> > tinker with this instead, that's fine with me.
> 
> I will try, but with the current flood on the mailing lists, I cannot
> promise anything.  (Don't you guys have holidays to attend to? ;-)

OK, I did that now.  multi-test-files-busy no longer fails in my
testing.

multi-test-files-simple, OTOH, still fails at about 20% rate.  And
note that it fails because it somehow reads the old value:

  Test multi-test-files-simple condition:
      (ert-test-failed
       ((should
	 (=
	  (multisession-value multisession--sfoo)
	  2))
	:form
	(= 1 2)
	:value nil))

So I think the problem I'm seeing here is not that Emacs cannot access
the file, it's something else.  Any ideas for digging into this?

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> OK, I did that now.  multi-test-files-busy no longer fails in my
> testing.

Great!

> multi-test-files-simple, OTOH, still fails at about 20% rate.  And
> note that it fails because it somehow reads the old value:
>
>   Test multi-test-files-simple condition:
>       (ert-test-failed
>        ((should
> 	 (=
> 	  (multisession-value multisession--sfoo)
> 	  2))
> 	:form
> 	(= 1 2)
> 	:value nil))
>
> So I think the problem I'm seeing here is not that Emacs cannot access
> the file, it's something else.  Any ideas for digging into this?

If you could get the test to cough up what's in the multisession
directory when this happens, that would be helpful.  The -simple- test
should just write the file in Emacs A, Emacs B starts, reads/writes the
file, and exists, and then Emacs A reads the file.

So it should be "impossible" for that to fail, but apparently Windows is
doing something ... weird?  Anti-virus software?  😀  Or there might be
a bug somewhere in the multisession code.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 26 Dec 2021 12:39:06 +0100
> 
> >   Test multi-test-files-simple condition:
> >       (ert-test-failed
> >        ((should
> > 	 (=
> > 	  (multisession-value multisession--sfoo)
> > 	  2))
> > 	:form
> > 	(= 1 2)
> > 	:value nil))
> >
> > So I think the problem I'm seeing here is not that Emacs cannot access
> > the file, it's something else.  Any ideas for digging into this?
> 
> If you could get the test to cough up what's in the multisession
> directory when this happens, that would be helpful.

I will try.  But ERT is notoriously hard with supporting such
debugging printouts, so maybe I'll need help.

>                                                     The -simple- test
> should just write the file in Emacs A, Emacs B starts, reads/writes the
> file, and exists, and then Emacs A reads the file.
> 
> So it should be "impossible" for that to fail, but apparently Windows is
> doing something ... weird?  Anti-virus software?

No, the problem happens even if I turn off the anti-virus.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I will try.  But ERT is notoriously hard with supporting such
> debugging printouts, so maybe I'll need help.

I was thinking of something simpler, like the following, and compare
when things work and when things fail.

diff --git a/test/lisp/emacs-lisp/multisession-tests.el b/test/lisp/emacs-lisp/multisession-tests.el
index f1dbbb29fd..285a97f884 100644
--- a/test/lisp/emacs-lisp/multisession-tests.el
+++ b/test/lisp/emacs-lisp/multisession-tests.el
@@ -116,6 +116,9 @@ multi-test-files-simple
       (should (= (multisession-value multisession--sfoo) 0))
       (cl-incf (multisession-value multisession--sfoo))
       (should (= (multisession-value multisession--sfoo) 1))
+      (message "before: %s" (directory-files-recursively
+                             multisession-directory
+                             "." t))
       (call-process
        (concat invocation-directory invocation-name)
        nil t nil
@@ -130,6 +133,9 @@ multi-test-files-simple
                         ""
                         :synchronized t)
                       (cl-incf (multisession-value multisession--sfoo))))))
+      (message "after: %s" (directory-files-recursively
+                             multisession-directory
+                             "." t))
       (should (= (multisession-value multisession--sfoo) 2)))))
 
 (ert-deftest multi-test-files-busy ()

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 26 Dec 2021 13:22:14 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I will try.  But ERT is notoriously hard with supporting such
> > debugging printouts, so maybe I'll need help.
> 
> I was thinking of something simpler, like the following, and compare
> when things work and when things fail.

I did the moral equivalent of that, and didn't see any difference: in
both cases, there's a single file multisession--sfoo.value there.

See my other message about the reason for the failures.

Re: sqlite3

[Eli Zaretskii]

> Date: Sun, 26 Dec 2021 13:48:36 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > >   Test multi-test-files-simple condition:
> > >       (ert-test-failed
> > >        ((should
> > > 	 (=
> > > 	  (multisession-value multisession--sfoo)
> > > 	  2))
> > > 	:form
> > > 	(= 1 2)
> > > 	:value nil))
> > >
> > > So I think the problem I'm seeing here is not that Emacs cannot access
> > > the file, it's something else.  Any ideas for digging into this?
> > 
> > If you could get the test to cough up what's in the multisession
> > directory when this happens, that would be helpful.
> 
> I will try.

I think I understand what happens here.  It's a race condition (of
course!).

This code from multisession-backend-value:

     ;; We have a value, but we want to update in case some other
     ;; Emacs instance has updated.
     ((multisession--synchronized object)
      (if (and (file-exists-p file)
               (time-less-p (multisession--cached-sequence object)
                            (file-attribute-modification-time
                             (file-attributes file))))
          (multisession--read-file-value file object)
        ;; Nothing, return the cached value.
        (multisession--cached-value object)))

assumes that the file always exists.  But on Windows, it sometimes
doesn't.

What I see is that if multisession-tests.el looks at the contents of
the value file, it sees the updated value, even though the test fails
because its value is before the update.  Which means the Emacs session
running the test didn't read the file at all.

And note that multisession--read-file-value already tests for the
file's existence, and retries if it doesn't exist.

So I think we should simply restructure the code, and let
multisession--read-file-value do the test and the waiting, and
multisession-backend-value should just return the cached value if
multisession--read-file-value failed to read the file.

Re: sqlite3

[Eli Zaretskii]

> Date: Sun, 26 Dec 2021 14:55:20 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> So I think we should simply restructure the code, and let
> multisession--read-file-value do the test and the waiting, and
> multisession-backend-value should just return the cached value if
> multisession--read-file-value failed to read the file.

For example, with the simple changes below (which also make the sleep
between retries somewhat shorter), the test didn't fail even once in
20 runs:

diff --git a/lisp/emacs-lisp/multisession.el b/lisp/emacs-lisp/multisession.el
index c58a9ab..acd830f 100644
--- a/lisp/emacs-lisp/multisession.el
+++ b/lisp/emacs-lisp/multisession.el
@@ -275,7 +275,7 @@ multisession--read-file-value
           ((permission-denied file-missing)
            (setq i (1+ i)
                  last-error err)
-           (sleep-for (+ 0.1 (/ (float (random 10)) 10))))))
+           (sleep-for (+ 0.01 (* (float (random 10)) 0.01))))))
       (signal (car last-error) (cdr last-error)))))
 
 (defun multisession--object-file-name (object)
@@ -299,10 +299,16 @@ multisession-backend-value
      ;; We have a value, but we want to update in case some other
      ;; Emacs instance has updated.
      ((multisession--synchronized object)
-      (if (and (file-exists-p file)
-               (time-less-p (multisession--cached-sequence object)
-                            (file-attribute-modification-time
-                             (file-attributes file))))
+      ;; On MS-Windows/MS-DOS, we could have race conditions whereby
+      ;; the value file might not exist for short windows of
+      ;; opportunity.  So try reading the file on those systems if it
+      ;; doesn't exist or looks outdated, as our reading method can
+      ;; cope with some of those races.
+      (if (or (and (file-exists-p file)
+                   (time-less-p (multisession--cached-sequence object)
+                                (file-attribute-modification-time
+                                 (file-attributes file))))
+              (memq system-type '(windows-nt ms-dos)))
           (multisession--read-file-value file object)
         ;; Nothing, return the cached value.
         (multisession--cached-value object)))

Re: sqlite3

[Eli Zaretskii]

> Date: Sun, 26 Dec 2021 16:42:52 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> For example, with the simple changes below (which also make the sleep
> between retries somewhat shorter), the test didn't fail even once in
> 20 runs:

Actually, this slightly different version is better:

diff --git a/lisp/emacs-lisp/multisession.el b/lisp/emacs-lisp/multisession.el
index c58a9ab..2dcddc2 100644
--- a/lisp/emacs-lisp/multisession.el
+++ b/lisp/emacs-lisp/multisession.el
@@ -275,7 +275,7 @@ multisession--read-file-value
           ((permission-denied file-missing)
            (setq i (1+ i)
                  last-error err)
-           (sleep-for (+ 0.1 (/ (float (random 10)) 10))))))
+           (sleep-for (+ 0.01 (* (float (random 10)) 0.01))))))
       (signal (car last-error) (cdr last-error)))))
 
 (defun multisession--object-file-name (object)
@@ -299,11 +299,19 @@ multisession-backend-value
      ;; We have a value, but we want to update in case some other
      ;; Emacs instance has updated.
      ((multisession--synchronized object)
-      (if (and (file-exists-p file)
-               (time-less-p (multisession--cached-sequence object)
-                            (file-attribute-modification-time
-                             (file-attributes file))))
-          (multisession--read-file-value file object)
+      ;; On MS-Windows/MS-DOS, we could have race conditions whereby
+      ;; the value file might not exist for short windows of
+      ;; opportunity.  So try reading the file on those systems if it
+      ;; doesn't exist or looks outdated, as our reading method can
+      ;; cope with some of those races.
+      (if (or (and (file-exists-p file)
+                   (time-less-p (multisession--cached-sequence object)
+                                (file-attribute-modification-time
+                                 (file-attributes file))))
+              (memq system-type '(windows-nt ms-dos)))
+          (condition-case nil
+              (multisession--read-file-value file object)
+            (error (multisession--cached-value object)))
         ;; Nothing, return the cached value.
         (multisession--cached-value object)))
      ;; Just return the cached value.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> +      ;; On MS-Windows/MS-DOS, we could have race conditions whereby
> +      ;; the value file might not exist for short windows of
> +      ;; opportunity.  So try reading the file on those systems if it
> +      ;; doesn't exist or looks outdated, as our reading method can
> +      ;; cope with some of those races.
> +      (if (or (and (file-exists-p file)
> +                   (time-less-p (multisession--cached-sequence object)
> +                                (file-attribute-modification-time
> +                                 (file-attributes file))))
> +              (memq system-type '(windows-nt ms-dos)))
> +          (condition-case nil
> +              (multisession--read-file-value file object)
> +            (error (multisession--cached-value object)))

I don't think this is quite correct -- this means that an older version
of the file will be read (and overwrite a newer value) on Windows.

But I don't understand the analysis here -- according to your other
mail, the file always does exist in these tests.  True, there might be a
race condition here, but the file exists before the file-exists-p is
called, and it exists wen the --read-file-value is called, so it's
possible that there's a race condition.

But doesn't it seem more likely that it's the `file-attributes' call
here that returns outdated data on Windows?  That would explain all the
symptoms you're seeing, I think?  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 27 Dec 2021 12:59:12 +0100
> 
> But doesn't it seem more likely that it's the `file-attributes' call
> here that returns outdated data on Windows?  That would explain all the
> symptoms you're seeing, I think?  

Could be.  But how does this affect the solution?

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> But doesn't it seem more likely that it's the `file-attributes' call
>> here that returns outdated data on Windows?  That would explain all the
>> symptoms you're seeing, I think?  
>
> Could be.  But how does this affect the solution?

I think we should confirm whether that's the case or not -- and not
attempt to read the file if it doesn't exist.

That is, I wonder what the timestamp returned here really is, and how it
compares to the cached timestamp.  Could it be that the timestamps are
identical and that's why `time-less-p' is returning nil?  If it's a
matter of the timestamps having too low resolution on (some?) Windows
versions, then perhaps we could have some particular logic for that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 27 Dec 2021 16:07:11 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> But doesn't it seem more likely that it's the `file-attributes' call
> >> here that returns outdated data on Windows?  That would explain all the
> >> symptoms you're seeing, I think?  
> >
> > Could be.  But how does this affect the solution?
> 
> I think we should confirm whether that's the case or not -- and not
> attempt to read the file if it doesn't exist.

If you can show a patch that only reads if the file exists, I can see
if the offending test still succeeds.

> That is, I wonder what the timestamp returned here really is, and how it
> compares to the cached timestamp.  Could it be that the timestamps are
> identical and that's why `time-less-p' is returning nil?  If it's a
> matter of the timestamps having too low resolution on (some?) Windows
> versions, then perhaps we could have some particular logic for that.

The file's time resolution on Windows is 0.1usec (100 nanoseconds).  I
think this should be enough.

Re: sqlite3

[Eli Zaretskii]

> Date: Mon, 27 Dec 2021 17:14:14 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> The file's time resolution on Windows is 0.1usec (100 nanoseconds).  I
> think this should be enough.

Hmmm... but our w32 'stat' emulation only supports time_t times,
i.e. 1-second resolution.  Could that be the reason for the problem?

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Hmmm... but our w32 'stat' emulation only supports time_t times,
> i.e. 1-second resolution.  Could that be the reason for the problem?

Yes, that would explain what we're seeing.  Is there any chance the w32
stat emulation could be fixed to support sub-second resolutions like it
does on other systems?

If not, I think we'll have to change how the values are stored, and add
a "header" section in the files themselves, so you'd have

---
Timestamp: <high res timestamp>

<value>
---

and then multisession has to open the file and look at the timestamp
"header" to see whether it's newer than what we'd got.

But fixing the stat call in w32 would obviously be better (and generally
useful).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 28 Dec 2021 08:17:03 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Hmmm... but our w32 'stat' emulation only supports time_t times,
> > i.e. 1-second resolution.  Could that be the reason for the problem?
> 
> Yes, that would explain what we're seeing.  Is there any chance the w32
> stat emulation could be fixed to support sub-second resolutions like it
> does on other systems?

It could, but it's not a trivial job, and doing it just for this use
case could be overkill.

Besides, Emacs does work with 'stat' which doesn't provide sub-second
resolution, so this is not a w32 specific problem, at least in theory.

> If not, I think we'll have to change how the values are stored, and add
> a "header" section in the files themselves, so you'd have

I think this is a good idea anyway.  Because we could have fast
machines and disks that defeat even the best time resolution.

> ---
> Timestamp: <high res timestamp>

Do we really need time?  That could fail due to clock skew, if the
values are modified from different machines.  Wouldn't just a single
"generation number", that keeps only increasing, be enough?

Re: sqlite3

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Besides, Emacs does work with 'stat' which doesn't provide sub-second
> resolution, so this is not a w32 specific problem, at least in theory.

FWIW, I think st_mtime and st_atime are only accurate to the second on
Haiku as well.  At least that was the conclusion I reached when
debugging a test failure there.  (That test was later fixed, but I don't
know why.)

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Yes, that would explain what we're seeing.  Is there any chance the w32
>> stat emulation could be fixed to support sub-second resolutions like it
>> does on other systems?
>
> It could, but it's not a trivial job, and doing it just for this use
> case could be overkill.

Well, it wouldn't be just for this.  Sub-second file resolution is
generally useful.

>> If not, I think we'll have to change how the values are stored, and add
>> a "header" section in the files themselves, so you'd have
>
> I think this is a good idea anyway.  Because we could have fast
> machines and disks that defeat even the best time resolution.
>
>> ---
>> Timestamp: <high res timestamp>
>
> Do we really need time?  That could fail due to clock skew, if the
> values are modified from different machines.  Wouldn't just a single
> "generation number", that keeps only increasing, be enough?

That would also be fine -- it's what we're using on the sqlite backend
in multisession.

But I'm not very enthusiastic about having a synchronised multisession
variable access reading and parsing a file each time, so I'm leaning
towards just keeping the code as is and adjusting the test on Windows
(and other systems that don't have sub-second resolution).  Synchronised
multisession variables offer no guarantees -- having two Emacsen
updating the variable at the same time is inherently racy, and we're
doing a best effort kind of thing.

So this will be slightly less spiffy on Windows, and that's fine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 28 Dec 2021 15:41:34 +0100
>
> >> Yes, that would explain what we're seeing.  Is there any chance the w32
> >> stat emulation could be fixed to support sub-second resolutions like it
> >> does on other systems?
> >
> > It could, but it's not a trivial job, and doing it just for this use
> > case could be overkill.
> 
> Well, it wouldn't be just for this.  Sub-second file resolution is
> generally useful.

OK, I'll add that to my todo, for some rainy day.

> But I'm not very enthusiastic about having a synchronised multisession
> variable access reading and parsing a file each time, so I'm leaning
> towards just keeping the code as is and adjusting the test on Windows
> (and other systems that don't have sub-second resolution).  Synchronised
> multisession variables offer no guarantees -- having two Emacsen
> updating the variable at the same time is inherently racy, and we're
> doing a best effort kind of thing.
> 
> So this will be slightly less spiffy on Windows, and that's fine.

That'd be fine by me, I think updating the same value simultaneously
from several Emacsen is a pretty corner use case anyway.  We can delay
the solution until someone really needs that, and explains why.  We do
need to say something about the races in the docs, I think.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That'd be fine by me, I think updating the same value simultaneously
> from several Emacsen is a pretty corner use case anyway.

Now done.

> We can delay the solution until someone really needs that, and
> explains why.  We do need to say something about the races in the
> docs, I think.

Do we need to say more than it does?  Which is:

--
But note that this is done without locking, so if many
instances are updating the value at the same time, it's unpredictable
which instance ``wins''.
--

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 29 Dec 2021 16:11:09 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That'd be fine by me, I think updating the same value simultaneously
> > from several Emacsen is a pretty corner use case anyway.
> 
> Now done.

The "simple" test still fails sometimes.  I even enlarged the
sleep-for to 10 sec, and it still fails.

> > We can delay the solution until someone really needs that, and
> > explains why.  We do need to say something about the races in the
> > docs, I think.
> 
> Do we need to say more than it does?  Which is:
> 
> --
> But note that this is done without locking, so if many
> instances are updating the value at the same time, it's unpredictable
> which instance ``wins''.

Sounds good enough, thanks.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The "simple" test still fails sometimes.  I even enlarged the
> sleep-for to 10 sec, and it still fails.

Huh.  Then it sounds like our analysis of the problem is wrong, and it's
not failing on the time-less-p thing?  Or...  can Windows (or our usage
of it) cache the modification times for a long time?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 29 Dec 2021 19:05:08 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The "simple" test still fails sometimes.  I even enlarged the
> > sleep-for to 10 sec, and it still fails.
> 
> Huh.  Then it sounds like our analysis of the problem is wrong, and it's
> not failing on the time-less-p thing?

Maybe.  Though I'm at a loss what could that be.

> Or...  can Windows (or our usage
> of it) cache the modification times for a long time?

Unlikely.

The problem is that its hard to compare time stamps the way we print
them in file-attributes.  But maybe there's way around that, and we
should print the two time stamps being compared in human readable
format.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The problem is that its hard to compare time stamps the way we print
> them in file-attributes.  But maybe there's way around that, and we
> should print the two time stamps being compared in human readable
> format.

Try something like:

diff --git a/test/lisp/emacs-lisp/multisession-tests.el b/test/lisp/emacs-lisp/multisession-tests.el
index 57ca420488..06b73ffb95 100644
--- a/test/lisp/emacs-lisp/multisession-tests.el
+++ b/test/lisp/emacs-lisp/multisession-tests.el
@@ -116,6 +116,14 @@ multi-test-files-simple
       (should (= (multisession-value multisession--sfoo) 0))
       (cl-incf (multisession-value multisession--sfoo))
       (should (= (multisession-value multisession--sfoo) 1))
+      (message
+       "Before: %s"
+       (format-time-string
+        "%FT%T.%N"
+        (file-attribute-modification-time
+         (file-attributes
+          (expand-file-name "files/multisession/multisession--sfoo.value"
+                            multisession-directory)))))
       (call-process
        (concat invocation-directory invocation-name)
        nil t nil
@@ -133,6 +141,14 @@ multi-test-files-simple
       ;; On Windows, we don't have sub-second resolution.
       (when (memq system-type '(windows-nt haiku))
         (sleep-for 2))
+      (message
+       "After: %s"
+       (format-time-string
+        "%FT%T.%N"
+        (file-attribute-modification-time
+         (file-attributes
+          (expand-file-name "files/multisession/multisession--sfoo.value"
+                            multisession-directory)))))
       (should (= (multisession-value multisession--sfoo) 2)))))
 
 (ert-deftest multi-test-files-busy ()


But I'm beginning to wonder whether the problem is something completely
different -- that the problem is that in some cases, Emacs B doesn't
start at all on your system.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 30 Dec 2021 15:41:13 +0100
> Cc: emacs-devel@gnu.org
> 
> But I'm beginning to wonder whether the problem is something completely
> different -- that the problem is that in some cases, Emacs B doesn't
> start at all on your system.

No, it starts okay and actually updates the value file.

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 29 Dec 2021 19:05:08 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The "simple" test still fails sometimes.  I even enlarged the
> > sleep-for to 10 sec, and it still fails.
> 
> Huh.  Then it sounds like our analysis of the problem is wrong, and it's
> not failing on the time-less-p thing?

No, it was failing there, allright.

> Or...  can Windows (or our usage of it) cache the modification times
> for a long time?

I found the problem.  In multisession-tests.el we were waiting on the
wrong side of call-process: the time stamp of the value-file doesn't
change once call-process exits, so waiting there is futile.  Waiting
_before_ the call, OTOH, is what the doctor ordered, because it makes
sure the sub-process updates the value-file after enough time has
passed since the previous update to let 1-sec file-times resolution
get out of our way.

The test runs without any failures now.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I found the problem.  In multisession-tests.el we were waiting on the
> wrong side of call-process: the time stamp of the value-file doesn't
> change once call-process exits, so waiting there is futile.  Waiting
> _before_ the call, OTOH, is what the doctor ordered, because it makes
> sure the sub-process updates the value-file after enough time has
> passed since the previous update to let 1-sec file-times resolution
> get out of our way.

Ah!  Yes, of course...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 18 Dec 2021 08:41:50 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Which sounds like a problem different from what you saw?
> 
> It may stem from the (incomplete) fix I added for the original problem,
> though.
> 
> > Adding (sleep-for 3) after call-process doesn't entirely fix the
> > problem, although it becomes more rare.
> >
> > I've installed a change in multisession.el that makes the problem much
> > more rare by forcing 'fsync' when we close the value file.  The
> > problem is not yet gone entirely, though: it happens with about 25%
> > probability here.  I think it could be related to the fact that
> > 'rename-file' is not an atomic operation on MS-Windows. But that's a
> > wild guess, I have no basis for it.
> 
> Right.  I'll do some further testing, but the real fix is to add some
> backoff logic when the file is locked, which I'll do now.  (And in that
> case, I think the `fsync' should be unnecessary.)

With the current master, the test still fails for me about 20% of the
time.  Seems like slightly less than before your recent changes, but
I'm not sure.

I think fsync should be necessary because we use temporary files, and
those are by default not fsync'ed when we close the descriptor to
which we wrote.  At least on MS-Windows, I think this could mean other
processes might get a stale file contents.

Re: sqlite3

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> With the current master, the test still fails for me about 20% of the
> time.  Seems like slightly less than before your recent changes, but
> I'm not sure.

Hm, that's disappointing.  It's annoying that I can't reproduce the
problem with any sort of reliability -- it only seems to happen, like,
1% of the times.

> I think fsync should be necessary because we use temporary files, and
> those are by default not fsync'ed when we close the descriptor to
> which we wrote.  At least on MS-Windows, I think this could mean other
> processes might get a stale file contents.

Right.  Perhaps we should consider doing this without the rename on
Windows?  (And without fsync on non-Windows.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 11:55:10 +0100
> 
> > I think fsync should be necessary because we use temporary files, and
> > those are by default not fsync'ed when we close the descriptor to
> > which we wrote.  At least on MS-Windows, I think this could mean other
> > processes might get a stale file contents.
> 
> Right.  Perhaps we should consider doing this without the rename on
> Windows?  (And without fsync on non-Windows.)

If you show me a patch, I can try it.

Re: sqlite3

[Juri Linkov]

> If I write-region to a file, and then rename the file to "foo.value"

I wonder why such a non-standard file extension ".value"
while we have ".eld" for Emacs-Lisp-Data.

Re: sqlite3

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

>> If I write-region to a file, and then rename the file to "foo.value"
>
> I wonder why such a non-standard file extension ".value"
> while we have ".eld" for Emacs-Lisp-Data.

Heh.  It's really the "el dingo file", a name for the (ding) Gnus
data file.

But re-parsing it as "emacs lisp data" is good, so perhaps we could
indeed use that here.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: sqlite3

[Jean Louis]

* Lars Ingebrigtsen <larsi@gnus.org> [2021-12-06 04:52]:
> I've brought this up before, but I didn't really have a solution then,
> but I think I do now: sqlite3.

That is great idea to have that built-in into Emacs or as build option
to include it.

Another pending project that wants to enter into ELPA is this module
for PostgreSQL, though developer may be delayed, we discussed it
months ago and they agreed. I am using PostgreSQL all the time.

emacs-libpq @ Github
https://github.com/anse1/emacs-libpq

> sqlite3 is supported on more platforms than Emacs is, the interface is
> small and stable, and (best of all) somebody has already created
> interface functions for Emacs (via a module):
> 
>   https://github.com/syohex/emacs-sqlite3

Not just sqlite3, I would prefer to have the gdbm as well, and Emacs
lisp interface to create, delete, databases and keys and values.

> I think it'd be good to bring that into core, and then write a small
> wrapper library (well, a trivial ORM) for the rest of Emacs to use, so
> that we don't have to write SQL all over the place.

I hope you will leave the option to write SQL as well.

> (setf (persistent-data :namespace "emoji" :key "favorites")
> emoji--favorites)

Excellent idea! Though consider also including various other
databases, such as lmdb, qdbm, gdbm, bdb, tokyocabinet, kyotocabinet
and similar.

As if you only target remembering specific information thatn SQL may
be overkill, and gbdm can be good enough.

> I.e., what Emacs needs is a persistent key/value store, and this would
> give us that.

Great idea! 

> In addition, if somebody really wants to write SQL stuff (it can be
> very handy for some things), having sqlite3 in there gives us that
> in addition for free.

Great! 

Please include possibilities to bind to any kind of databases. I am
using heavily PostgreSQL, though GDBM and other databases are welcome. 

> This comes with questions about how the users are supposed be able to
> clear out the data, for instance, but we could have a `M-x
> list-persistent-data' where the users could blow out whatever they want
> to.

I hope that database included may be useful for users in any other
form, freely, not only for the specific persistent data you proposed.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: sqlite3

[xenodasein--- via Emacs development discussions.]

Late to the party, but here's my take anyway.

Where has this feature been?  Although databases can be considered luxury
software, they are the right way to do persistent storage if available.
They are rigorously researched and ubiquitous for a reason.  "It it is
mostly automated but I still sometimes edit the data file manually" is
not a healthy habit.  There exists of course configuration files that are
meant to be edited manually, but I wouldn't call them persistent storage.
I just hope SQLite's passive-aggressive licensing won't crush our dreams.