/var/guix/db/db.sqlite corruption

/var/guix/db/db.sqlite corruption

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

A while back, I ran out of disk space while running `guix package -i`,
during a build.  I then noticed that `guix gc` would, after outputting
some number of "deleting" lines, fail with an error that's approximately
this (I forgot to make a copy):

  guix gc: error: executing SQLite query: malformed database image

This error appears to come from nix/libstore/sqlite.cc, and the last
portion of the message comes from `sqlite3_errmsg`, so I don't think
this is Guix's fault.

I had already run e2fsck on many occasions since this happened a couple
months back and there are no disk errors.

To work around it, I dumped the DB:

  $ sqlite3 db.sqlite .dump > ~/bak.sql

The final line of that file contains this:

  ROLLBACK; -- due to errors

Understandable.  I replaced that with "COMMIT;", and:

  $ mv db.sqlite{,-bak}
  $ sqlite3 db.sqlite < ~/bak.sql

This produced a database that was 4MiB smaller than the original. :x
But now `guix gc` works.[*]

So my questions are:

  - Is there a way to regenerate the database?
  - What bad things could happen with what I just did?

Thanks, and please nobody reading this message in a list archive in the
future do the above without first reading replies to this thread; I
don't want to be responsible for anything bad that may result! :)

[*]: Actually, I had some other bizarre issues.  After I recreated the
DB, I started getting more generic I/O errors.  There were no errors in
dmesg.  But when I moved the file to a different location (e.g. my home
directory), it worked (via `sqlite3`).  If I moved it back to
`/var/guix/db/db.sqlite`, I/O errors once again.  If I ran `.dump` from
that dir, empty.  If I moved it to my home dir and ran `.dump`, I got
the full dump.  This problem didn't resolve until after a reboot.  I
haven't seen anything like that before, and I don't want to
speculate.  I should have tried flushing the kernel I/O cache before
rebooting to see if that would have fixed it.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: /var/guix/db/db.sqlite corruption

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2110 bytes --]

Hi Mike,

Mike Gerwitz <mtg@gnu.org> writes:

>   guix gc: error: executing SQLite query: malformed database image

I've also seen this happen.  I opened a bug report about it recently:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=36687

I don't have a solution, but I tried a similar work-around with similar
results.  Please see the bug report for details.

>   - Is there a way to regenerate the database?

I don't think it can be regenerated from scratch.  Some information,
like which paths in the store are "valid", cannot be determined just by
examining the contents of the store.

>   - What bad things could happen with what I just did?

I don't know.  Speaking on the side of caution, especially since you
changed the ROLLBACK line to COMMIT, I think that all bets are off.  I
feel a little more comfortable the results of my own attempt, since I
didn't have to make that change (my SQL dump ended with a COMMIT line).
However, it's still concerning that we had to dump the database and
restore it in order to "fix" it.  Eventually I'll probably reinstall
Guix just to be safe, but I take backups of my important data pretty
frequently, so I'm not in a big rush.

> [*]: Actually, I had some other bizarre issues.  After I recreated the
> DB, I started getting more generic I/O errors.  There were no errors in
> dmesg.  But when I moved the file to a different location (e.g. my home
> directory), it worked (via `sqlite3`).  If I moved it back to
> `/var/guix/db/db.sqlite`, I/O errors once again.  If I ran `.dump` from
> that dir, empty.  If I moved it to my home dir and ran `.dump`, I got
> the full dump.  This problem didn't resolve until after a reboot.  I
> haven't seen anything like that before, and I don't want to
> speculate.  I should have tried flushing the kernel I/O cache before
> rebooting to see if that would have fixed it.

Did you remember to stop the guix-daemon and verify that no processes
were accessing the database file when you did all of this?  If not, then
I wouldn't be surprised to see bizarre behavior.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: /var/guix/db/db.sqlite corruption

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 923 bytes --]

On Sat, Aug 03, 2019 at 02:28:01 -0700, Chris Marusich wrote:
> I've also seen this happen.  I opened a bug report about it recently:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=36687

Ah, when I originally researched this issue, you hadn't yet reported
that bug.  I should have checked again.  I'll reply there with my
Message-Id.  Thanks for pointing it out.

> Did you remember to stop the guix-daemon and verify that no processes
> were accessing the database file when you did all of this?  If not, then
> I wouldn't be surprised to see bizarre behavior.

Yeah I had stopped the daemon and was running it manually to pass
`--debug'.  The errors I was getting were not only from Guix---it was
also happening with the sqlite3 command.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]