Re: Unboxed package manager

[Jonas Bernoulli <jonas@bernoul.li>]

Lynn Winebarger <owinebar@gmail.com> writes:

> On Mon, Mar 20, 2023 at 2:11 PM Jonas Bernoulli <jonas@bernoul.li> wrote:
>>
>> Lynn Winebarger <owinebar@gmail.com> writes:
>>
>> > I think I'm going to hack something together starting with advice on the
>> > existing package management and taking some inspiration from the design of
>> > Jonas Bernoulli's epkg and emir packages for tracking installed packages
>> > and component files in a SQLite database.
>>
>> If I were to start over now, I wouldn't write Closql.  At the time it
>> made a lot of sense because I knew nothing about databases and because
>> it allowed me to switch out the old data store ("everything is *its own*
>> file") without changing internal interfaces much.  Moving from files to
>> a database did wonders for performance, so at first I didn't mind the
>> performance impact of the OO interface on top of the database.
>> Meanwhile I have moved away from the OO interface for anything that
>> deals with more than one package at a time, turning many rows into
>> EIEIO objects is a bit costly.
>
> Hmm - I had thought it might be an interesting exercise to get
> acquainted with Emacs's version of CLOS, but maybe not.

That is well worth it.

And Closql is at least interesting; it's just that "turn every row into
an object" is too slow if you have thousands of rows and all you really
want to do is display a table of some of the columns.  (If you do use
Closql, then use the "next" branch, that would already be merged, if I
weren't waiting for two downstreams to act.)

> However, just
> reviewing the way you've organized the package data is probably going
> to be useful.  For example, trying to understand the best way to
> assess whether a particular version of a package is an upgrade.  I
> don't know about the behavior of package.el in master, but
> historically, if there is a version of a package on MELPA and the same
> package is available on GNU or NONGNU ELPA, package.el will treat the
> version on GNU/NONGNU as an upgrade even though the one on MELPA is
> more recent.  I assume it has something to do with the comparison of
> commit hashes or dates with more traditional version numbers, but
> that's just a guess.

I'm afraid I am not doing any of that.  I just track the HEAD of the
most authoritative repository.


>
>>
>> When I switched to SQLite, Emacs had no built-in support (coming in
>> Emacs 29) and there also was no module, so EmacSQL was the natural
>> choice.  I am the maintainer of that now, so I definitely think it
>> serves a purpose, but I do have some reservations.
>>
>> The next release will feature new backends that use the built-in support
>> and a module, but if I were to start now, I probably would go with the
>> built-in support directly.
>>
>> EmacSQL allows writing SQL using vectors instead of concatenating
>> strings, which is nice, but for people just getting started with SQL, it
>> has the disadvantage that you now have to learn two things, SQL and the
>> almost SQL vector syntax, which isn't 100% complete and doesn't map 1:1.
>>
>> The main limitation of EmacSQL is that it stores everything (except
>> NULL) as a string.  This is why I would probably avoid it now, because
>> it limits interoperability with anything that doesn't use EmacSQL.
>
> That's only because it was designed to interact with sqlite through a
> pipe to the shell program, though, right?

No, it was a design decision by the original author, in order to keep
things simple.

> It seems like a method for
> compiling sexpr-type representations of sql queries into statements
> usable with the builtin support would still be useful, and not limited
> in the same way - since the returned values do not require serializing
> as text by the sqlite shell then parsing them in Elisp.

Of course "SQL as vectors" and "store everything as a string" are not
tied to each other per se, but with EmacSQL you get both.

> [ Digression...]
> Just looking at the src/sqlite.c in master, as there is no other
> documentation of the sqlite support I can see, only a simplified form
> of the core API is supported - not unreasonable for an initial
> release.  The only noticeable absence I see, based on a cursory review
> of the sqlite3 API spec, is that a select query cannot be reset.
> Maybe because each db connection is associated with at most one
> prepared statement at a time by the design of the Lisp_Sqlite
> pseudovector?
>
> Other than that, I note that rows are returned as lists rather than
> arrays, which makes the semantics more incompatible with emacsql than
> it really has to be.  Can that be changed before 29 is released?

Why is that a problem?  What would we gain if the DSL and the output
both used vectors or both lists?

> The associated sqlite-mode looks interesting.  I only wonder why it
> doesn't derive from tabulated-list mode instead of directly from
> special.  Tabulated list mode would seem to be made for things like
> database tables.
>
> Lynn

     Jonas