Re: sqlite3

[Pip Cet <pipcet@gmail.com>]

On Thu, Dec 9, 2021 at 12:09 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Pip Cet <pipcet@gmail.com> writes:
> > The proposal is to have Emacs store some user data in some binary
> > format that cannot be readily inspected, diffed, backed up,
> > version-controlled, shared, altered, or understood. (Or archived,
> > published, indexed, checksummed, ...)
>
> The slope is pretty slippery from .emacs to .newsrc.eld to Gnus registry
> storage to .sqlite files.

I agree that .newsrc.eld and the Gnus registry, as you describe them,
have problems that need to be fixed. Making those permanently
unfixable by switching to a binary-only format seems like a bad idea
to me, unless there are overwhelming advantages that I'm just failing
to see so far.

> The .emacs file with the Customize settings is usually something a user
> can read and edit and diff with some confidence.

I'm glad you're not proposing to convert that!

> The .newsrc.eld file, which is a couple of huge alists with complicated
> nesting, is very hard to read, edit or diff, even thought it's "text".

Then we need to work on improving its format.

> Putting it in VC is meaningless.

How so? I usually have my home directory under version control, and
git's xdelta implementation should deal with alists just fine (and
produce readable output at sub-line granularity)...

> The Gnus registry, which is a dumped hash table, can't meaningfully be
> read or edited by anybody unless that person is an expert on hash table
> representation and what it all means.

Then we need to improve that. It's not a huge change to allow dumping
hash tables in predictable order (rather than
predictable-unless-the-bucket-count-changed order) and at one tuple
per line; are there additional issues?

> And it's not diffable or VC-able
> without special software.

Possibly so. That still means you're making problems that are
currently coincidental fundamentally unfixable.

> An .sqlite file isn't pretending to be user editable in vi, but there's
> a plethora of software out there do do all things you'd expect, and (of

How much of it is GNU software?

> course) Emacs will come with a mode to inspect and edit these files more
> free-form.

Indeed, I would have considered it a good idea to introduce such a
mode well before proposing (as you might or might not have done) that
random packages start using a new library.

> They will, in the end, be a lot more editable than a hash
> table dump: This mode will allow you to say "delete all values that
> refer to 'foo' no matter where it's stored in this file", and that's not
> something that's anybody implemented for all these "text" formats that
> Emacs uses today.

You're touching on an important point here: if we go the sqlite route,
efforts will be split between improving things for text files
(something that is desperately necessary) and improving things for
sqlite databases.

Again, I'm not saying Emacs is currently perfect (blasphemy!), but at
least it's fixable in principle, without duplicating too much effort.

> As for archiving, publishing, indexing or checksumming: Doing that with
> an .sqlite file is a lot more meaningful than a file containing a hash
> table dump.

(I don't think sqlite (or sqlite3) is considered an acceptable format
by most archives, or publishers. ASCII text is.)

I don't see how indexing an sqlite file, without first converting it
into text format, gives you anything valuable: does sqlite even
guarantee long strings won't be split up arbitrarily?

And a checksum that changes with no underlying changes in the data is
problematic. Usually not a problem with ELisp dumps, but it usually is
with sqlite files.

I must confess my experience with sqlite is not very recent; back
then, it fsync()ed heavily (and unnecessarily), created additional
files that needed to be kept in sync with each other, and often ended
up corrupting the databases anyway. Even if those issues have been
fixed, I feel almost zero consideration has been given to
alternatives.

I must repeat that interacting with existing sqlite files is something
Emacs should be able to do, but creating new ones for Emacs purposes
is quite a different proposal, and by my reading it's what you
actually suggested.

And, merely as a request, I'd still like to hear a further convincing
case for all this being necessary in the first place. So far I've
heard "Gnus" and "emoji favorites". I can't currently use the former
and the latter is inaccessible to me on multiple levels, so I'm left
with my "binary junk in .emacs.d" impression of what you're actually
proposing.

Pip