Re: Issuing rollback() due to DESTROY

[Eric Wong <e@80x24.org>]

Felix Lechner <felix.lechner@lease-up.com> wrote:
> Hi Eric,
> 
> I am unable to recover from the Sqlite database error.  I tried .recover
> and .dump, but the result is always the same.  I even deleted the
> database (saving a backup) but the error occurred again.

Hmm...  there maybe some other SQLite-specific recovery knowledge
I'm not up-to-date on that can be done with the sqlite(1)
command.  Perhaps that can be tried...

OTOH, if you delete all the SQLite and Xapian files, you should
be able to run `public-inbox-index' on that directory and it'll
recreate it.

msgmap.sqlite3 is more important if you're using v2, the
uncommon altid feature, or broken messages with multiple
Message-IDs; but it's unlikely important for v1 inboxes.

> The equiment is proportioned generously.  8GB RAM, 16 GB Swap, and the
> dedicated partition has 165 GB free out of 200 GB.

OK.

> Can public-inbox run with an account that is locked and has /var/nologin
> configured as the shell, as long as I use PI_EMERGENCY and PI_CONFIG?

Probably, yes.  I haven't tested it, but it should work as long as it
can write to the inboxes, emergency, and TMPDIR.

<snip>

> > How does that timer run?
> 
> The timer runs every five minutes.  It ryncs a folder, the output of
> which I direct to this mailing list:

Wait, I hope you're not rsync-ing into an inbox directory
containing SQLite files.  That's very bad if a transaction
(from -mda or -watch) is happening at the same time as rsync.

SQLite upstream documents this in https://sqlite.org/howtocorrupt.html
section 1.2: "Backup or restore while a transaction is active"

Perhaps there's other things (faulty HW/drivers, etc...),
but that howtocorrupt doc is a good read.

I've certainly dealt with faulty drive controllers in the past;
and still deal with failing HDD/SSD on a fairly regular basis.
And I don't trust anything important with non-ECC RAM.

I seem to recall btrfs RAID 1 can be problematic in the face of
unclean shutdowns; and I've had problems with ecryptfs auto
unmounting with open files in the past which broke another
Perl + SQLite file storage project I worked on.

> > Is it a particular message that's causing this?
> 
> Not as far as I can tell.  They all ended up in the emergency Maildir.

Good to know :>

> > Also, has this ever worked in the past?
> 
> Yes, for many weeks and every five minutes.  Other derivative lists
> still get updated when I enable the timer:
> 
>     https://patchwise.org/inbox/bugs-to-scan/
> 
> > Does it occasionally still work? (do you see any new messages?)
> 
> No, not for that inbox.

Yeah, nuking all the index files and rerunning public-inbox-index
ought to fix it if you've exhausted all the SQLite repair options.

> > Please, give us as much detail about your setup as possible
> 
> I simply send local mails and my MTA executes -mda. [1]

OK.  I wonder if it was rsync or some dirty shutdown or bad HW
of some sort.

> > Either gnats or mosquitos
> 
> I hope that's been resolved.

Nope.  It's a disaster :<