Issuing rollback() due to DESTROY

Issuing rollback() due to DESTROY

[Felix Lechner]

Hi,

What is the remedy for this error message, please?

    "Issuing rollback() due to DESTROY without explicit disconnect() of
    DBD::SQLite::db handle
    dbname=/srv/patchwise/public-inbox/lists/debbugs-state-changes-rsync.git/msgmap.sqlite3
    at
    /gnu/store/infi869byszi6cvvp6wcd7n431pd4zlh-public-inbox-1.9.0/lib/perl5"

Thank you!

Kind regards
Felix

Re: Issuing rollback() due to DESTROY

[Eric Wong]

Felix Lechner <felix.lechner@lease-up.com> wrote:
> Hi,
> 
> What is the remedy for this error message, please?
> 
>     "Issuing rollback() due to DESTROY without explicit disconnect() of
>     DBD::SQLite::db handle
>     dbname=/srv/patchwise/public-inbox/lists/debbugs-state-changes-rsync.git/msgmap.sqlite3
>     at
>     /gnu/store/infi869byszi6cvvp6wcd7n431pd4zlh-public-inbox-1.9.0/lib/perl5"

Seems like error handling went wrong somewhere.  Could be caused
by another error (OOM, open files limits, disk space, etc...)

What were you doing when this happened?
Were you using public-inbox-mda? public-inbox-watch? public-inbox-learn?

Is it reproducible?

I'm suffering from a nasty cold and insect bites, so maybe others can help.

Thanks.

Re: Issuing rollback() due to DESTROY

[Felix Lechner]

Hi Eric,

On Mon, May 27 2024, Eric Wong wrote:

> Seems like error handling went wrong somewhere.  Could be caused
> by another error (OOM, open files limits, disk space, etc...)

Okay, I'll check, but I don't see anything unusual 12% disk space taken.

> What were you doing when this happened?  Were you using
> public-inbox-mda? public-inbox-watch? public-inbox-learn?

My mail server alerted me to the failure.  It uses public-inbox-mda.

> Is it reproducible?

The error occurred every five minutes for about three hours, until I
deactivated the timer triggering the action.

> I'm suffering from a nasty cold and insect bites

I'm sorry to read that.  What kind of insect bites?

Kind regards
Felix

Re: Issuing rollback() due to DESTROY

[Eric Wong]

Felix Lechner <felix.lechner@lease-up.com> wrote:
> Hi Eric,
> 
> On Mon, May 27 2024, Eric Wong wrote:
> 
> > Seems like error handling went wrong somewhere.  Could be caused
> > by another error (OOM, open files limits, disk space, etc...)
> 
> Okay, I'll check, but I don't see anything unusual 12% disk space taken.

It could also be an unusual spam message, or SpamAssassin failing.

> > What were you doing when this happened?  Were you using
> > public-inbox-mda? public-inbox-watch? public-inbox-learn?
> 
> My mail server alerted me to the failure.  It uses public-inbox-mda.

Anything in stderr?  It's possibly redirected to logs.  This is critical.

You can also try setting ORIGINAL_RECIPIENT in env and feeding
messages to public-inbox-mda as the usual -mda user and see if
you can reproduce it in a terminal.

> > Is it reproducible?
> 
> The error occurred every five minutes for about three hours, until I
> deactivated the timer triggering the action.

Does that correspond to the timer (systemd timer?)

How does that timer run?

Normally -mda was designed to be run as part of the MTA delivery
action, but it can be fired from a terminal, shell script,
etc... provided ORIGINAL_RECIPIENT is set (as typically done by
MTAs (at least postfix).

Is it a particular message that's causing this?

Also, has this ever worked in the past?

Does it occasionally still work? (do you see any new messages?)

Please, give us as much detail about your setup as possible

> > I'm suffering from a nasty cold and insect bites
> 
> I'm sorry to read that.  What kind of insect bites?

Either gnats or mosquitos or similar, not sure.  There seem to
be more of them every year due to rain.
Giant itchy lumps on my arms....
No need to lift weights to bulk up, at least :P

Re: Issuing rollback() due to DESTROY

[Felix Lechner]

Hi Eric,

I am unable to recover from the Sqlite database error.  I tried .recover
and .dump, but the result is always the same.  I even deleted the
database (saving a backup) but the error occurred again.

The equiment is proportioned generously.  8GB RAM, 16 GB Swap, and the
dedicated partition has 165 GB free out of 200 GB.

Can public-inbox run with an account that is locked and has /var/nologin
configured as the shell, as long as I use PI_EMERGENCY and PI_CONFIG?

Other anwers are below.

On Tue, May 28 2024, Eric Wong wrote:

> It could also be an unusual spam message, or SpamAssassin failing.

It's definitely not spam.

> Anything in stderr?  It's possibly redirected to logs.  This is critical.

I do not see anything in the logs or on stderr for -mda (although I see
a few lines for -imapd and -nntpd.

> You can also try setting ORIGINAL_RECIPIENT in env and feeding
> messages to public-inbox-mda as the usual -mda user and see if
> you can reproduce it in a terminal.

When I run it -mda manually (which I have done on other occasion)
nothing happened.  The command returns quickly.  I checked for typos.

> Does that correspond to the timer (systemd timer?)

It's a GNU Shepherd timer on GNU Guix.  The Shepherd is my PID 1 and
takes on a role similar to systemd.

> How does that timer run?

The timer runs every five minutes.  It ryncs a folder, the output of
which I direct to this mailing list:

    https://patchwise.org/inbox/debbugs-state-changes-rsync/

> Is it a particular message that's causing this?

Not as far as I can tell.  They all ended up in the emergency Maildir.

> Also, has this ever worked in the past?

Yes, for many weeks and every five minutes.  Other derivative lists
still get updated when I enable the timer:

    https://patchwise.org/inbox/bugs-to-scan/

> Does it occasionally still work? (do you see any new messages?)

No, not for that inbox.

> Please, give us as much detail about your setup as possible

I simply send local mails and my MTA executes -mda. [1]

> Either gnats or mosquitos

I hope that's been resolved.

Kind regards
Felix

[1] https://codeberg.org/lechner/system-config/src/commit/4d1f42cf1fc2d4ec6e8dd0434e1567e0d0bbfbf6/host/wallace-server/operating-system.scm#L1940

Re: Issuing rollback() due to DESTROY

[Eric Wong]

Felix Lechner <felix.lechner@lease-up.com> wrote:
> Hi Eric,
> 
> I am unable to recover from the Sqlite database error.  I tried .recover
> and .dump, but the result is always the same.  I even deleted the
> database (saving a backup) but the error occurred again.

Hmm...  there maybe some other SQLite-specific recovery knowledge
I'm not up-to-date on that can be done with the sqlite(1)
command.  Perhaps that can be tried...

OTOH, if you delete all the SQLite and Xapian files, you should
be able to run `public-inbox-index' on that directory and it'll
recreate it.

msgmap.sqlite3 is more important if you're using v2, the
uncommon altid feature, or broken messages with multiple
Message-IDs; but it's unlikely important for v1 inboxes.

> The equiment is proportioned generously.  8GB RAM, 16 GB Swap, and the
> dedicated partition has 165 GB free out of 200 GB.

OK.

> Can public-inbox run with an account that is locked and has /var/nologin
> configured as the shell, as long as I use PI_EMERGENCY and PI_CONFIG?

Probably, yes.  I haven't tested it, but it should work as long as it
can write to the inboxes, emergency, and TMPDIR.

<snip>

> > How does that timer run?
> 
> The timer runs every five minutes.  It ryncs a folder, the output of
> which I direct to this mailing list:

Wait, I hope you're not rsync-ing into an inbox directory
containing SQLite files.  That's very bad if a transaction
(from -mda or -watch) is happening at the same time as rsync.

SQLite upstream documents this in https://sqlite.org/howtocorrupt.html
section 1.2: "Backup or restore while a transaction is active"

Perhaps there's other things (faulty HW/drivers, etc...),
but that howtocorrupt doc is a good read.

I've certainly dealt with faulty drive controllers in the past;
and still deal with failing HDD/SSD on a fairly regular basis.
And I don't trust anything important with non-ECC RAM.

I seem to recall btrfs RAID 1 can be problematic in the face of
unclean shutdowns; and I've had problems with ecryptfs auto
unmounting with open files in the past which broke another
Perl + SQLite file storage project I worked on.

> > Is it a particular message that's causing this?
> 
> Not as far as I can tell.  They all ended up in the emergency Maildir.

Good to know :>

> > Also, has this ever worked in the past?
> 
> Yes, for many weeks and every five minutes.  Other derivative lists
> still get updated when I enable the timer:
> 
>     https://patchwise.org/inbox/bugs-to-scan/
> 
> > Does it occasionally still work? (do you see any new messages?)
> 
> No, not for that inbox.

Yeah, nuking all the index files and rerunning public-inbox-index
ought to fix it if you've exhausted all the SQLite repair options.

> > Please, give us as much detail about your setup as possible
> 
> I simply send local mails and my MTA executes -mda. [1]

OK.  I wonder if it was rsync or some dirty shutdown or bad HW
of some sort.

> > Either gnats or mosquitos
> 
> I hope that's been resolved.

Nope.  It's a disaster :<

Re: Issuing rollback() due to DESTROY

[Felix Lechner]

Hi Eric,

On Tue, Jun 18 2024, Eric Wong wrote:

> msgmap.sqlite3 is more important if you're using v2, the
> uncommon altid feature, or broken messages with multiple
> Message-IDs; but it's unlikely important for v1 inboxes.

I use v2.  Should I still be using v1?

> Wait, I hope you're not rsync-ing into an inbox directory
> containing SQLite files.

No.  I am not doing that.  The rsync process sends a email, which is
then filed by my MTA.

> I wonder if it was rsync or some dirty shutdown or bad HW of some
> sort.

Two back-to-back power outages were responsible.

> if you delete all the SQLite and Xapian files, you should be able to
> run `public-inbox-index' on that directory and it'll recreate it.

Thank you for that advice.  Unfortunately, that results in the following
error messages:

$ public-inbox-index public-inbox/lists/debbugs-state-changes-rsync.git/
error: object file /srv/patchwise/public-inbox/lists/debbugs-state-changes-rsync.git/git/0.git/objects/58/7ba7ce7f8dd9c65a711c9d8442c77e76bc6246 is empty
error: object file /srv/patchwise/public-inbox/lists/debbugs-state-changes-rsync.git/git/0.git/objects/58/7ba7ce7f8dd9c65a711c9d8442c77e76bc6246 is empty
fatal: bad object 587ba7ce7f8dd9c65a711c9d8442c77e76bc6246
git log failed: $?=32768 at /gnu/store/infi869byszi6cvvp6wcd7n431pd4zlh-public-inbox-1.9.0/lib/perl5/site_perl/5.36.0/PublicInbox/SearchIdx.pm line 967.
[1]:    (in cleanup) 11336 shard[1] xdb not released
[2]:    (in cleanup) 11337 shard[2] xdb not released
[0]:    (in cleanup) 11335 shard[0] xdb not released

Is there a way to fix the Git repo?

> Nope.  It's a disaster :<

Bug zappers and mosquito nets are commonly available where I live.
Please let me know if I should send links with purchasing information.

Also, thanks for your help so far!

Kind regards
Felix

Re: Issuing rollback() due to DESTROY

[Eric Wong]

Felix Lechner <felix.lechner@lease-up.com> wrote:
> On Tue, Jun 18 2024, Eric Wong wrote:
> > msgmap.sqlite3 is more important if you're using v2, the
> > uncommon altid feature, or broken messages with multiple
> > Message-IDs; but it's unlikely important for v1 inboxes.
> 
> I use v2.  Should I still be using v1?

No, v2 is always better if you have a lot of messages.
v1 was only suited for small archives and I expect debbugs
to be really busy.

> > Wait, I hope you're not rsync-ing into an inbox directory
> > containing SQLite files.
> 
> No.  I am not doing that.  The rsync process sends a email, which is
> then filed by my MTA.

OK...

> > I wonder if it was rsync or some dirty shutdown or bad HW of some
> > sort.
> 
> Two back-to-back power outages were responsible.

Ouch, yeah; and looking below it's even worse...

> > if you delete all the SQLite and Xapian files, you should be able to
> > run `public-inbox-index' on that directory and it'll recreate it.
> 
> Thank you for that advice.  Unfortunately, that results in the following
> error messages:
> 
> $ public-inbox-index public-inbox/lists/debbugs-state-changes-rsync.git/
> error: object file /srv/patchwise/public-inbox/lists/debbugs-state-changes-rsync.git/git/0.git/objects/58/7ba7ce7f8dd9c65a711c9d8442c77e76bc6246 is empty
> error: object file /srv/patchwise/public-inbox/lists/debbugs-state-changes-rsync.git/git/0.git/objects/58/7ba7ce7f8dd9c65a711c9d8442c77e76bc6246 is empty
> fatal: bad object 587ba7ce7f8dd9c65a711c9d8442c77e76bc6246
> git log failed: $?=32768 at /gnu/store/infi869byszi6cvvp6wcd7n431pd4zlh-public-inbox-1.9.0/lib/perl5/site_perl/5.36.0/PublicInbox/SearchIdx.pm line 967.
> [1]:    (in cleanup) 11336 shard[1] xdb not released
> [2]:    (in cleanup) 11337 shard[2] xdb not released
> [0]:    (in cleanup) 11335 shard[0] xdb not released
> 
> Is there a way to fix the Git repo?

Got backups?

If you have the original mails, then you could probably restore
and reimport the blobs into git (possibly starting with a new
inbox and sharing objects).  You might have to brute force the
trees and commits to get repeatable ordering.

You could probably nuke the Xapian + SQLite DBs and reset the
git history to known point which lacks corrupted objects as
a starting point to speed up the process.

But yeah, there's not much that can be done if git storage is
corrupted aside from reimporting :<

> > Nope.  It's a disaster :<
> 
> Bug zappers and mosquito nets are commonly available where I live.
> Please let me know if I should send links with purchasing information.

Thanks, I'm worried about electricity use and possible fire
hazards of zappers.  Just bundling up with thick clothing (they
bite through thin layers!) and wearing goggles from now on
despite the heat :<

Re: Issuing rollback() due to DESTROY

[Felix Lechner]

Hi Eric,

On Fri, Jun 21 2024, Eric Wong wrote:

> Got backups?

I will eventually, for reported defects.  The lists I am playing with
now merely log synchronization activity.

Public Inbox is a great way to keep public logs, by the way!

> But yeah, there's not much that can be done if git storage is
> corrupted aside from reimporting :<

Thanks!  I recreated the whole list folder.  The old messages were not
needed anymore.  I am merely practicing for real emergencies.

> I expect debbugs to be really busy.

Maybe so, but the load is also distributed across tens of thousands of
lists for GNU.org.  One per report number.  It should make future
recoveries easier.  Also, losing NNTP syncronization isn't so bad if
it's for a report the reader was already interested in.

> Just bundling up with thick clothing [...]  and wearing goggles from
> now on despite the heat

Okay.  A doctor once told me that papain, which is commonly available in
grocery stores as meat tenderizer, alleviates the symptoms of insect
bites.  Maybe that is helpful to you.

Kind regards
Felix