Against sqlite3!!! (Was: sqlite3)

Against sqlite3!!! (Was: sqlite3)

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 77 bytes --]

I’ve attached a pure Emacs Lisp implementation of persistent key value store.

[-- Attachment #2: resist!.el --]
[-- Type: application/octet-stream, Size: 1624 bytes --]

;;; resist!.el --- Against SQLite3!  -*- lexical-binding: t; -*-

(defvar kv-store-table)
(defmacro --kv (key value)
  (puthash key value kv-store-table)
  nil)
(cl-defstruct (kv-store (:constructor make-kv-store-1)) path table)
(defun make-kv-store (path)
  (let* ((kv-store (make-kv-store-1 :path path))
         (kv-store-table (make-hash-table :test 'equal))
         need-compactification)
    (when (file-exists-p path)
      (condition-case c
          (load-file path)
        (end-of-file
         ;; We might encounter trailing unbalanced form if Emacs
         ;; crashed in the middle of `kv-put'.  We compact the file
         ;; and fix unbalanced form as a side effect
         (setq need-compactification t)))
      (setf (kv-store-table kv-store) kv-store-table))
    (when need-compactification
      (compact-kv-store kv-store))
    kv-store))
(defsubst kv-put-log (key value kv-store)
  (let ((print-length nil) (print-level nil))
    (prin1 `(--kv ,key ,value) (current-buffer)))
  (insert "\n"))
(defun compact-kv-store (kv-store)
  ;; dump the full content of kv-store-table at once
  ;; to compress the log
  (with-temp-buffer
    (maphash (lambda (key value) (kv-put-log key value kv-store))
             (kv-store-table kv-store))
    (let ((file-precious-flag t))
      (write-file (kv-store-path kv-store)))))
(defun kv-put (key value kv-store)
  (with-temp-buffer
    (kv-put-log key value kv-store)
    (append-to-file nil nil (kv-store-path kv-store)))
  (puthash key value (kv-store-table kv-store)))
(defun kv-get (key kv-store)
  (gethash key (kv-store-table kv-store)))
(provide 'resist!)

Against sqlite3!!! (Was: sqlite3)

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 81 bytes --]

I’ve attached a pure Emacs Lisp implementation of persistent key value store.

[-- Attachment #1.2: Type: text/html, Size: 852 bytes --]

[-- Attachment #2: resist!.el --]
[-- Type: application/octet-stream, Size: 1667 bytes --]

;;; resist!.el --- Against SQLite3!  -*- lexical-binding: t; -*-

(defvar kv-store-table)
(defmacro --kv (key value)
  (puthash key value kv-store-table)
  nil)
(cl-defstruct (kv-store (:constructor make-kv-store-1)) path table)
(defun make-kv-store (path)
  (let* ((kv-store (make-kv-store-1 :path path))
         (kv-store-table (make-hash-table :test 'equal))
         need-compactification)
    (when (file-exists-p path)
      (condition-case c
          (load-file path)
        (end-of-file
         ;; We might encounter trailing unbalanced form if Emacs
         ;; crashed in the middle of `kv-put'.  We compact the file
         ;; and fix unbalanced form as a side effect
         (setq need-compactification t)))
      (setf (kv-store-table kv-store) kv-store-table))
    (when need-compactification
      (compact-kv-store kv-store))
    kv-store))
(defsubst kv-put-log (key value kv-store)
  (let ((print-length nil) (print-level nil))
    (prin1 `(--kv ,key ,value) (current-buffer)))
  (insert "\n"))
(defun compact-kv-store (kv-store)
  ;; dump the full content of kv-store-table at once
  ;; to compress the log
  (with-temp-buffer
    (maphash (lambda (key value) (kv-put-log key value kv-store))
             (kv-store-table kv-store))
    (let ((file-precious-flag t))
      (write-file (kv-store-path kv-store)))))
(defun kv-put (key value kv-store)
  (with-temp-buffer
    (kv-put-log key value kv-store)
    (append-to-file nil nil (kv-store-path kv-store)))
  (puthash key value (kv-store-table kv-store)))
(defun kv-get (key kv-store)
  (gethash key (kv-store-table kv-store)))
(provide 'resist!)

[-- Attachment #3: ATT00001.htm --]
[-- Type: text/html, Size: 332 bytes --]

Re: Against sqlite3!!! (Was: sqlite3)

[Qiantan Hong]

A first unscientific benchmark, people!

(defvar test (make-kv-store "~/Projects/test.eld”))
(loop for i below 10000
      do (kv-put i '(1 3 3) test))

(measure-time
 (make-kv-store "~/Projects/test.eld"))
"0.025423"

Hua, 0.025s for loading 10k entries, by abusing built-in C load directly.

> On Dec 7, 2021, at 12:13 AM, Qiantan Hong <qhong@MIT.EDU> wrote:
> 
>> I’ve attached a pure Emacs Lisp implementation of persistent key value store.
> <resist!.el>
> Seems to stuck for too long, so I’ve resent it — sorry for the spam.

Re: Against sqlite3!!!

[Colin Baxter 😺]

>>>>> "Qiantan" == Qiantan Hong <qhong@mit.edu> writes:

    Qiantan> I’ve attached a pure Emacs Lisp implementation of
    Qiantan> persistent key value store.  

Excellent! Long live text solutions.

Re: Against sqlite3!!!

[Stefan Monnier]

>           (load-file path)

Please don't:
- The function is called `load` (`load-file` is just one of the
  *commands* defined to access the functionality interactively).
- This is a security hole.

`insert-file-contents + read` should hopefully still be fast enough.


        Stefan

Re: Against sqlite3!!! (Was: sqlite3)

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]


Actually, Emacs can serialize/deserialize hash table directly via
prin1-to-string & read

```
(let ((ht (make-hash-table)))
  (puthash "test" "value" ht)
  (format "%S" ht))
```

You can use `read` to "parse" the string returned by that snippet and
get a hash table.

Qiantan Hong <qhong@mit.edu> writes:

>  I’ve attached a pure Emacs Lisp implementation of persistent key value store.
>
> [4. resist!.el --- application/emacs-lisp; resist!.el]...
>
> [5. ATT00001.htm --- text/html; ATT00001.htm]...


-- 
Retrieve my PGP public key:

  gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Against sqlite3!!!

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

> Please don't:
> - The function is called `load` (`load-file` is just one of the
>  *commands* defined to access the functionality interactively).
> - This is a security hole.
> 
> `insert-file-contents + read` should hopefully still be fast enough.
Indeed, I’ve attached an updated version, it runs as fast.
I also added kv-rem

> Actually, Emacs can serialize/deserialize hash table directly via
> prin1-to-string & read

ik, this is the current standard practice.
If you follow the sqlite3 thread you see the problem is saving/loading
(aka printing/reading) whole hash table/alist takes too much time.

The point of my implementation is to do it incrementally everytime
the key value store is mutated.



[-- Attachment #2: resist!.el --]
[-- Type: application/octet-stream, Size: 2199 bytes --]

;;; resist!.el --- Against SQLite3!  -*- lexical-binding: t; -*-

(cl-defstruct (kv-store (:constructor make-kv-store-1)) path table)
(defun make-kv-store (path)
  (let* ((kv-store (make-kv-store-1 :path path))
         (kv-store-table (make-hash-table :test 'equal))
         need-compactification)
    (when (file-exists-p path)
      (condition-case nil
          (with-temp-buffer
            (insert-file-contents path)
            (while (< (point) (1- (point-max))) ; exclude trailing newline
              (let ((entry (read (current-buffer))))
                (pcase (car entry)
                  ('++ (puthash (cadr entry) (caddr entry) kv-store-table))
                  ('-- (remhash (cadr entry) kv-store-table))))))
        (end-of-file
         ;; We might encounter trailing unbalanced form if Emacs
         ;; crashed in the middle of `kv-put'.  We compact the file
         ;; and fix unbalanced form as a side effect
         (setq need-compactification t)))
      (setf (kv-store-table kv-store) kv-store-table))
    (when need-compactification
      (compact-kv-store kv-store))
    kv-store))
(defsubst kv-put-log (key value)
  (let ((print-length nil) (print-level nil))
    (prin1 (list '++ key value) (current-buffer)))
  (insert "\n"))
(defsubst kv-rem-log (key)
  (let ((print-length nil) (print-level nil))
    (prin1 (list '-- key) (current-buffer)))
  (insert "\n"))
(defun compact-kv-store (kv-store)
  ;; dump the full content of kv-store-table at once
  ;; to compress the log
  (with-temp-buffer
    (maphash (lambda (key value) (kv-put-log key value))
             (kv-store-table kv-store))
    (let ((file-precious-flag t))
      (write-file (kv-store-path kv-store)))))
(defun kv-put (key value kv-store)
  (with-temp-buffer
    (kv-put-log key value)
    (append-to-file nil nil (kv-store-path kv-store)))
  (puthash key value (kv-store-table kv-store)))
(defun kv-rem (key)
  (with-temp-buffer
    (kv-rem-log key)
    (append-to-file nil nil (kv-store-path kv-store)))
  (remhash key (kv-store-table kv-store)))
(defun kv-get (key kv-store)
  (gethash key (kv-store-table kv-store)))
(provide 'resist!)

Re: Against sqlite3!!!

[David Engster]

> Actually, Emacs can serialize/deserialize hash table directly via
> prin1-to-string & read
>
> ```
> (let ((ht (make-hash-table)))
>   (puthash "test" "value" ht)
>   (format "%S" ht))
> ```
>
> You can use `read` to "parse" the string returned by that snippet and
> get a hash table.

Yes, and it's slow. The Gnus registry is saved/loaded this way, and this
has annoyed me for years.

-David

Re: Against sqlite3!!!

[Tassilo Horn]

Hi Hong,

I'm not very knowledgable with cl-* stuff but doesn't your
implementation load all key-value pairs at once?  That would be quite a
disadvantage compared to a DB approach where I'd naturally expect that
only the value I'm asking for is loaded.

Also, how would it ensure consistency when I have 2 parallel emacs
sessions (like one for mail/irc and one for programming/editing) where
session 1 modifies the value of key A and the other of key B?  It looks
like the values of the kv-store that gets saved later will win.  In case
that's the emacs session which has modified B, it'll revert A to the
state before the other session modified it, no?

If that were true, I'd say your resist!.el is a non-starter in the
current form.  It should at least load only values explicitly asked for
and only persist/override actually changed values.  A trivial solution
could use one file per key.  Not sure how sensible that is.

Bye,
Tassilo

Re: Against sqlite3!!!

[Qiantan Hong]

> I'm not very knowledgable with cl-* stuff but doesn't your
> implementation load all key-value pairs at once?  That would be quite a
> disadvantage compared to a DB approach where I'd naturally expect that
> only the value I'm asking for is loaded.
It does, but I’ve done some benchmark and it loads 10k entries in 0.02~0.03 
seconds. 100k entries takes <0.5s.
I’d say it should be suffice for most Emacs application I know of.
Nobody is using Emacs for trillions of business records.

On the other hand save operation is fully incremental and don’t
even need to be invoked explicitly.

Being said that, if we really find out loading is not fast enough, I might
come up with some way to load it in segments lazily. I doubt if that will
ever become necessary.

> Also, how would it ensure consistency when I have 2 parallel emacs
> sessions (like one for mail/irc and one for programming/editing) where
> session 1 modifies the value of key A and the other of key B?  It looks
> like the values of the kv-store that gets saved later will win.  In case
> that's the emacs session which has modified B, it'll revert A to the
> state before the other session modified it, no?
Since it records a log of deltas instead of printing the whole data structure,
different key won’t interfere.

Being said that, currently it probably won’t work because UNIX append
is not atomic and will probably be interleaved into nonsense.
There’re various workarounds, lock file being one, but I like
the idea of keeping only one “controller” instance with exclusive
access to the file more.

> If that were true, I'd say your resist!.el is a non-starter in the
> current form.  It should at least load only values explicitly asked for
> and only persist/override actually changed values.  A trivial solution
> could use one file per key.  Not sure how sensible that is.
It does the latter.

As I’ve mentioned, from the benchmark results, the former doesn’t
seem to be a big problem. You’ll do it at most once for every Emacs
instance anyway.


Best,
Qiantan

Re: Against sqlite3!!!

[Arthur Miller]

Qiantan Hong <qhong@mit.edu> writes:

>>                                                   That would be quite a
>> disadvantage compared to a DB approach where I'd naturally expect that
>> only the value I'm asking for is loaded.

Sqlite does not read one key per request, it reads at least a page. So if your
system uses 64k pages, sqlite will read at least 64k at once. Then you also have
system doing disk i/o, caching nodes and so on; so you are never really getting
just "the value you are asking for to be loaded".

> It does, but I’ve done some benchmark and it loads 10k entries in 0.02~0.03 
> seconds. 100k entries takes <0.5s.
> I’d say it should be suffice for most Emacs application I know of.
> Nobody is using Emacs for trillions of business records.
>
> On the other hand save operation is fully incremental and don’t
> even need to be invoked explicitly.
>
> Being said that, if we really find out loading is not fast enough, I might
> come up with some way to load it in segments lazily. I doubt if that will
> ever become necessary.
>
>> Also, how would it ensure consistency when I have 2 parallel emacs
>> sessions (like one for mail/irc and one for programming/editing) where
>> session 1 modifies the value of key A and the other of key B?  It looks
>> like the values of the kv-store that gets saved later will win.  In case
>> that's the emacs session which has modified B, it'll revert A to the
>> state before the other session modified it, no?
> Since it records a log of deltas instead of printing the whole data structure,
> different key won’t interfere.
>
> Being said that, currently it probably won’t work because UNIX append
> is not atomic and will probably be interleaved into nonsense.
> There’re various workarounds, lock file being one, but I like
> the idea of keeping only one “controller” instance with exclusive
> access to the file more.
You can do same as sqlite does: just lock entire file (db). Sqlite locks entire
db while writes are performed. It can be configured to allow concurrent reads
but only if there are no ongoing writes.

Re: Against sqlite3!!!

[Qiantan Hong]

> You can do same as sqlite does: just lock entire file (db). Sqlite locks entire
> db while writes are performed. It can be configured to allow concurrent reads
> but only if there are no ongoing writes.
Indeed, that’s a straightforward solution. Not sure if
the Emacs lock file mechanism is fast enough to run for
every write, though (Emacs doesn’t seem to expose
UNIX flock/fcntl stuff).

There’s also a question of should we try to make write from an Emacs
visible for read from another Emacs.
AFAIC it doesn’t make much sense if we just want to persistent
backing storage. Should we also consider the (ab)use
as an IPC channel?

Best,
Qiantan

Re: Against sqlite3!!!

[Tassilo Horn]

Qiantan Hong <qhong@mit.edu> writes:

>> I'm not very knowledgable with cl-* stuff but doesn't your
>> implementation load all key-value pairs at once?  That would be quite
>> a disadvantage compared to a DB approach where I'd naturally expect
>> that only the value I'm asking for is loaded.
> It does, but I’ve done some benchmark and it loads 10k entries in
> 0.02~0.03 seconds. 100k entries takes <0.5s.

But your values are very only small lists.

> I’d say it should be suffice for most Emacs application I know of.
> Nobody is using Emacs for trillions of business records.

I'd imagine that if such a feature became available, packages would
start using it and store more data that they do now for whatever
reasons.  Like eww/elpher could want to store the list of the last 100
visited URLs.

>> Also, how would it ensure consistency when I have 2 parallel emacs
>> sessions (like one for mail/irc and one for programming/editing) where
>> session 1 modifies the value of key A and the other of key B?  It looks
>> like the values of the kv-store that gets saved later will win.  In case
>> that's the emacs session which has modified B, it'll revert A to the
>> state before the other session modified it, no?
>
> Since it records a log of deltas instead of printing the whole data
> structure, different key won’t interfere.

Ah, allright.  By skimming the code I've first thought the log would
only be for analysis/debugging purposes.

> Being said that, currently it probably won’t work because UNIX append
> is not atomic and will probably be interleaved into nonsense.
> There’re various workarounds, lock file being one, but I like the idea
> of keeping only one “controller” instance with exclusive access to the
> file more.

Interesting, but how would emacs instances interact with that controller
instance?  And would that controller instance simply be the first emacs
instance of a user?  What if the controller blocks because Gnus is
running and currently downloading mails with huge attachments?

>> If that were true, I'd say your resist!.el is a non-starter in the
>> current form.  It should at least load only values explicitly asked for
>> and only persist/override actually changed values.  A trivial solution
>> could use one file per key.  Not sure how sensible that is.
> It does the latter.

No, it uses one file per kv-store but you can have as many kv-stores as
you like, e.g., one per package.  Or do you mean "it only overrides
changed values" with "the latter"?  Indeed, that's true.  I've played
with the code and now understand it.  Nice! :-) (kv-rem is missing a
kv-store arg.)

> As I’ve mentioned, from the benchmark results, the former doesn’t
> seem to be a big problem. You’ll do it at most once for every Emacs
> instance anyway.

Yeah, and since it's no global store in the sense of "every package
feeds its data into the same store", my argument is void.  If my emacs
session doesn't start Gnus, Gnus won't load its own store.

Bye,
Tassilo

Re: Against sqlite3!!!

[Stefan Monnier]

> Since it records a log of deltas instead of printing the whole data structure,
> different key won’t interfere.

> Being said that, currently it probably won’t work because UNIX append
> is not atomic and will probably be interleaved into nonsense.
> There’re various workarounds, lock file being one, but I like
> the idea of keeping only one “controller” instance with exclusive
> access to the file more.

I think allowing several instances to use the file at the same time is
important (I always have 2 sessions active at the same time and I'd like
to be able to use (and share) savehist with both of them).

But that only means having to lock while we're saving, which is a very
short time.  It also requires being able to refresh the in-memory data
once we detect that the ondisk data has been changed.


        Stefan

Re: Against sqlite3!!!

[Lars Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> Yes, and it's slow. The Gnus registry is saved/loaded this way, and this
> has annoyed me for years.

Yup.  The Gnus registry would be well suited to use sqlite directly,
though -- it's basically hand-maintaining a (large) database, and sqlite
is a good fit for that.

That is, I don't think the new normal persistence method would be ideal
for the registry.

But, yes, the Gnus registry is a good demonstration of why serialising
hash tables to disk is unworkable in practice. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Against sqlite3!!!

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> David Engster <deng@randomsample.de> writes:
>
>> Yes, and it's slow. The Gnus registry is saved/loaded this way, and this
>> has annoyed me for years.
>
> Yup.  The Gnus registry would be well suited to use sqlite directly,
> though -- it's basically hand-maintaining a (large) database, and sqlite
> is a good fit for that.
>
> That is, I don't think the new normal persistence method would be ideal
> for the registry.
>
> But, yes, the Gnus registry is a good demonstration of why serialising
> hash tables to disk is unworkable in practice.

Than implement a way for Emacs to dump lisp objects to files faster. It would be
very useful for Emacs in general.

You will still have to serialize your gnus db to sqlite db, and write it to disk
via sqlite, so disk access will still be there.

Have you trye to write it in chunks in idle timer or all in one go? Is that possible?

Re: Against sqlite3!!!

[Qiantan Hong]

> Than implement a way for Emacs to dump lisp objects to files faster. It would be
> very useful for Emacs in general.
I think for this question, the one-and-for-all solution is. to have a fully
incremental persistent object store, i.e. all mutation are stored
incrementally without ever needing to print out the “fully value”
of a Lisp value. What do you think?

resist!.el in its current form basically implemented a special case
of the above, where only mutation to the top level hash table 
is persisted incrementally.

I can’t see a way to implement persistent object store without
some non-trivial memory overhead, though (because each
object has to get an unique id). The best thing I can come up
with has to have a hash table that maps every object in the
store to a numeric ID. Is that too much?

Re: Against sqlite3!!!

[Arthur Miller]

Qiantan Hong <qhong@mit.edu> writes:

>> Than implement a way for Emacs to dump lisp objects to files faster. It would be
>> very useful for Emacs in general.
> I think for this question, the one-and-for-all solution is. to have a fully
> incremental persistent object store, i.e. all mutation are stored
> incrementally without ever needing to print out the “fully value”
> of a Lisp value. What do you think?

I am not sure I understand what you mean. What is "fully value" of a Lisp
value. You mean entire object?

> resist!.el in its current form basically implemented a special case
> of the above, where only mutation to the top level hash table 
> is persisted incrementally.
>
> I can’t see a way to implement persistent object store without
> some non-trivial memory overhead, though (because each
> object has to get an unique id). The best thing I can come up
> with has to have a hash table that maps every object in the
> store to a numeric ID. Is that too much?

Honestly, I have no idea.

I just meant that it is very useful to be able to serialize/deserialize lisp
ojbects, in general, without need to go through some intermediate key/value
database.

I don't know what you are doing to start with; I haven't looked at your
resist.el, but maybe you can cache keys that needs to be flushed to disk in some
'dirty state (a list of kyes) and flush just those keys to a file in idle timer.

Re: Against sqlite3!!!

[Qiantan Hong]

> I am not sure I understand what you mean. What is "fully value" of a Lisp
> value. You mean entire object?
Yes, and this is less than optimal. For example, there’re lots of variables
holding a list and added to/removed from frequently.
A straightforward implementation would need to print/read the whole list
every time.
A cleverer object store would be able to figure out which CONS changed exactly,
and store a single record like 
(rplacd *id-of-the-cons* `(*new-element* . ,(object *id-of-old-tail*)))

> I don't know what you are doing to start with; I haven't looked at your
> resist.el, but maybe you can cache keys that needs to be flushed to disk in some
> 'dirty state (a list of kyes) and flush just those keys to a file in idle timer.
That’s not really relevant. resist!.el persist every kv-put operation immediately
without the need of any explicit save operation.
It might be a good idea to run compact-kv-store to save disk space in idle timer, though.

Re: Against sqlite3!!!

[Yuri Khan]

On Wed, 8 Dec 2021 at 03:19, Tassilo Horn <tsdh@gnu.org> wrote:

> > There’re various workarounds, lock file being one, but I like the idea
> > of keeping only one “controller” instance with exclusive access to the
> > file more.
>
> Interesting, but how would emacs instances interact with that controller
> instance?  And would that controller instance simply be the first emacs
> instance of a user?  What if the controller blocks because Gnus is
> running and currently downloading mails with huge attachments?

This line of thought, if followed naturally, leads to a dedicated
database server process. With multiple concurrent clients, isolation
levels, transaction control, etc. And there are better database
servers than a special Emacs configuration owning an SQLite database
file.

Re: Against sqlite3!!!

[Alexandre Garreau]

Le merkredo, 8-a de decembro 2021, 7-a horo kaj 11:19 CET Arthur Miller a 
écrit :
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> 
> 
> > David Engster <deng@randomsample.de> writes:
> 
> 
> >> Yes, and it's slow. The Gnus registry is saved/loaded this way, and
> >> this has annoyed me for years.
> 
> 
> 
> > Yup.  The Gnus registry would be well suited to use sqlite directly,
> > though -- it's basically hand-maintaining a (large) database, and
> > sqlite is a good fit for that.
> > 
> > That is, I don't think the new normal persistence method would be
> > ideal
> > for the registry.
> > 
> > But, yes, the Gnus registry is a good demonstration of why serialising
> > hash tables to disk is unworkable in practice.
> 
> Than implement a way for Emacs to dump lisp objects to files faster. It
> would be very useful for Emacs in general.
> 
> You will still have to serialize your gnus db to sqlite db, and write it
> to disk via sqlite, so disk access will still be there.

Well, I think, wrt writes, it’s impossible to go faster than just writing 
plaintext.  But sqlite is here to make *reads* faster, not writes.  The 
only way I can see to make writes faster without any form of serialization 
is just garbage collecting related data together, and directly dumping 
memory onto disk, in a native but non-portable format, along with metadata 
about the native format, and specific implementations to decode it from 
other platforms for each memory format.  I’d love to see that.  But I 
think writing is not the bottleneck that we try to improve here.

Re: Against sqlite3!!!

[Qiantan Hong]

> Well, I think, wrt writes, it’s impossible to go faster than just writing 
> plaintext.  But sqlite is here to make *reads* faster, not writes.  The 
> only way I can see to make writes faster without any form of serialization 
> is just garbage collecting related data together, and directly dumping 
> memory onto disk, in a native but non-portable format, along with metadata 
> about the native format, and specific implementations to decode it from 
> other platforms for each memory format.  I’d love to see that.  But I 
> think writing is not the bottleneck that we try to improve here.

I think write still *used to* be a bottleneck in the sense that without
incremental updates, every writes require dumping the whole store,
thus preventing Emacs from remembering things *early and eagerly*.

resist! solves exactly this. Any mutation to the kv-store is immediately
remembered by appending a log, and make-persistent-variable
remember all changes every persistent-variable-idle-time (1 second
seems reasonable).

> Well, I think, wrt writes, it’s impossible to go faster than just writing 
> plaintext.  But sqlite is here to make *reads* faster, not writes. 
Did you mean the initial loading up, or substantial reads?
For the substantial reads, sqlite3 is probably not much faster
than a single gethash.

resist! In its current form do require initially loading the whole store,
but I think it’s unclear whether it is a bottle neck. See
https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00865.html

Re: Against sqlite3!!!

[Qiantan Hong]

> If you start dividing your file into pages and loading one page at a time, 
> you’re starting to reimplement a kv-database, just as gdbm (or some 
> filesystems).  You’d better reuse gdbm,
There is a subtle but IMO significant benefit of resist!, it understands
more than kv-put. In the current version, it also understands
kv-push and kv-delete (aka list operations).

I imagine this can be very useful for Lisp because lots of time people
just add or remove data from a list. Some implementation that doesn’t
understand this would need to dump the whole list again.

> and work on something higher level 
> and more fun like a lisp sql-like query language, inspired from (or even 
> compatible with) sql or sparql or prolog, except each expression would 
> return a lisp value.  That would be very beautiful.
That’s definitely a cool idea. I’ll probably just go for an embedded Prolog 
with S-exp syntax.

It’s probably not my current priority though, because
Emacs packages at the moment still use mostly LISt Processing,
and somehow people still find enough incentive to introduce SQLite3,
which I attribute to the lack of a decent pure Lisp persistent store.
So that’s what resist! aims to solve at this moment.
Maybe if after resist! solved the problem people would still want
SQLite3, I’ll write another package revolt! which implements Prolog.

Best,
Qiantan

Re: Against sqlite3!!!

[Stefan Monnier]

> resist! In its current form do require initially loading the whole store,
> but I think it’s unclear whether it is a bottle neck. See
> https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg00865.html

It depends on the use case.
sqlite is the winner for databases that are not meant to be
memory-resident (usually because they're too large).
It's also the clear winner when you have to read some other
application's sqlite file ;-)


        Stefan

Re: Against sqlite3!!!

[Thierry Volpiatto]

Qiantan Hong <qhong@mit.edu> writes:

>> If you start dividing your file into pages and loading one page at a time, 
>> you’re starting to reimplement a kv-database, just as gdbm (or some 
>> filesystems).  You’d better reuse gdbm,
> There is a subtle but IMO significant benefit of resist!, it understands
> more than kv-put. In the current version, it also understands
> kv-push and kv-delete (aka list operations).
>
> I imagine this can be very useful for Lisp because lots of time people
> just add or remove data from a list. Some implementation that doesn’t
> understand this would need to dump the whole list again.
>
>> and work on something higher level 
>> and more fun like a lisp sql-like query language, inspired from (or even 
>> compatible with) sql or sparql or prolog, except each expression would 
>> return a lisp value.  That would be very beautiful.
> That’s definitely a cool idea. I’ll probably just go for an embedded Prolog 
> with S-exp syntax.
>
> It’s probably not my current priority though, because
> Emacs packages at the moment still use mostly LISt Processing,
> and somehow people still find enough incentive to introduce SQLite3,
> which I attribute to the lack of a decent pure Lisp persistent store.

Emacs-lisp provide eval-when-compile which allows saving elisp objects
in compiled files, I use this since years to save my variables in emacs.
See psession package.

-- 
Thierry

Re: Against sqlite3!!!

[Qiantan Hong]

> Emacs-lisp provide eval-when-compile which allows saving elisp objects
> in compiled files, I use this since years to save my variables in emacs.
> See psession package.
Interesting. Is the representation produced in this way different/more compact
from what you get by printing the object directly?

Re: Against sqlite3!!!

[Qiantan Hong]

>> Emacs-lisp provide eval-when-compile which allows saving elisp objects
>> in compiled files, I use this since years to save my variables in emacs.
>> See psession package.
> Interesting. Is the representation produced in this way different/more compact
> from what you get by printing the object directly?
I tested some simple objects and it seems that the representation in the .elc file
is exactly the same from what PRINT would produce, so it’s probably no faster
than PRINT/READing directly.