bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

[Jan (janneke) Nieuwenhuizen]

Hello Guix!

I configured my system with a childhurd service and added

--8<---------------cut here---------------start------------->8---
 (build-machine
  (name "childhurd")
  (system "i586-gnu")
  (host-key "ssh-ed25519 ... root@childhurd")
  (user "root")
  (private-key "/home/janneke/.ssh/id_rsa_childhurd"))
--8<---------------cut here---------------end--------------->8---

to /etc/guix/machines.  Sadly, running guix offload test gives

--8<---------------cut here---------------start------------->8---
$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: 'childhurd' is running GNU Guile 3.0.4
guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
--8<---------------cut here---------------end--------------->8---

So, I looked into the Debian/Hurd sqlite3 package for inspiration and found
debian/patches/20-hurd-locking-style.patch...well, that should fix it, right?

Sadly, this patch seems to break sqlite3 altogether.

--8<---------------cut here---------------start------------->8---
$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
Backtrace:
In ice-9/boot-9.scm:
  1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In unknown file:
          10 (apply-smob/0 #<thunk 7f3c99b1f480>)
In ice-9/boot-9.scm:
    718:2  9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
In ice-9/eval.scm:
    619:8  8 (_ #(#(#<directory (guile-user) 7f3c99743f00>)))
In guix/ui.scm:
  1949:12  7 (run-guix-command _ . _)
    663:2  6 (call-with-error-handling _)
In ice-9/boot-9.scm:
  1736:10  5 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
  1731:15  4 (with-exception-handler #<procedure 7f3c9546d3f0 at ice-9/boot-9.scm:1815:7 (exn)> _ #:unwind? _ # …)
In guix/scripts/offload.scm:
    647:6  3 (check-machine-availability _ _)
In srfi/srfi-1.scm:
   650:11  2 (for-each #<procedure assert-node-has-guix (node name)> (#<<inferior> pid: pipe socket: #<input…>) …)
In guix/scripts/offload.scm:
    582:2  1 (assert-node-has-guix #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500…> …)
In ice-9/boot-9.scm:
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
ERROR:
  1. &inferior-exception:
      arguments: (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: unable to open database file" status: 1>>)
      inferior: #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500> close: #<procedure close-port (_)> version: (0 1 1) packages: #<promise #<procedure 7f3c954a8000 at guix/inferior.scm:161:32 ()>> table: #<promise #<procedure 7f3c954a56f0 at guix/inferior.scm:162:32 ()>>>
      stack: ((#f ("ice-9/boot-9.scm" 1763 13)) (raise-exception ("ice-9/boot-9.scm" 1666 16)) (raise-exception ("ice-9/boot-9.scm" 1668 16)) (#f ("guix/store.scm" 582 42)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (open-connection ("guix/store.scm" 545 2)) (call-with-store ("guix/store.scm" 626 15)) (#f (#f #f #f)) (#f ("guix/repl.scm" 92 21)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (with-exception-handler ("ice-9/boot-9.scm" 1730 15)) (#f ("guix/repl.scm" 119 7)))
--8<---------------cut here---------------end--------------->8---

Maybe we're missing some file_lock patch that Debian has?  Where to look,
glibc, hurd, ...?  Ideas?

Greetings,
Janneke

Jan (janneke) Nieuwenhuizen (3):
  gnu: Add sqlite/hurd with locking fix.
  gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
  gnu: guix: Use sqlite/hurd for locking on the Hurd.

 gnu/local.mk                            |  1 +
 gnu/packages/guile.scm                  |  2 +-
 gnu/packages/package-management.scm     |  2 +-
 gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++
 gnu/packages/sqlite.scm                 |  9 +++++
 5 files changed, 63 insertions(+), 2 deletions(-)
 create mode 100644 gnu/packages/patches/sqlite3-hurd.patch

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix.

[Jan (janneke) Nieuwenhuizen]

* gnu/packages/patches/sqlite3-hurd.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/sqlite.scm (sqlite/hurd): New variable.
---
 gnu/local.mk                            |  1 +
 gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++
 gnu/packages/sqlite.scm                 |  9 +++++
 3 files changed, 61 insertions(+)
 create mode 100644 gnu/packages/patches/sqlite3-hurd.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 5e9dba5ab7..d5451bf2d5 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1358,6 +1358,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/sdl-pango-header-guard.patch		\
   %D%/packages/patches/sdl-pango-matrix_declarations.patch	\
   %D%/packages/patches/sdl-pango-sans-serif.patch		\
+  %D%/packages/patches/sqlite3-hurd.patch			\
   %D%/packages/patches/patchutils-test-perms.patch		\
   %D%/packages/patches/patch-hurd-path-max.patch		\
   %D%/packages/patches/perl-autosplit-default-time.patch	\
diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patch
new file mode 100644
index 0000000000..de87a30cb1
--- /dev/null
+++ b/gnu/packages/patches/sqlite3-hurd.patch
@@ -0,0 +1,51 @@
+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
+Upstream status: Not offered upstream.
+
+diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c
+--- sqlite-autoconf-3310100/sqlite3.c	2020-01-27 21:25:19.000000000 +0100
++++ sqlite-autoconf-3310100-/sqlite3.c	2020-07-01 11:50:13.768333806 +0200
+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode
+ # include <sys/mman.h>
+ #endif
+ 
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ /* # include <sys/ioctl.h> */
+ # include <sys/file.h>
+ # include <sys/param.h>
+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id
+ **
+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off
+ */
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ 
+ /*
+ ** Retry flock() calls that fail with EINTR
+@@ -38586,7 +38586,7 @@ IOMETHODS(
+   0                         /* xShmMap method */
+ )
+ 
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ IOMETHODS(
+   flockIoFinder,            /* Finder function name */
+   flockIoMethods,           /* sqlite3_io_methods object name */
+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){
+     UNIXVFS("unix",          autolockIoFinder ),
+ #elif OS_VXWORKS
+     UNIXVFS("unix",          vxworksIoFinder ),
++#elif defined(__GNU__)
++    UNIXVFS("unix",          flockIoFinder ),
+ #else
+     UNIXVFS("unix",          posixIoFinder ),
+ #endif
+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){
+ #if OS_VXWORKS
+     UNIXVFS("unix-namedsem", semIoFinder ),
+ #endif
+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS
++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)
+     UNIXVFS("unix-posix",    posixIoFinder ),
+ #endif
+ #if SQLITE_ENABLE_LOCKING_STYLE
diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm
index 8468131ddf..d5c44b0d0f 100644
--- a/gnu/packages/sqlite.scm
+++ b/gnu/packages/sqlite.scm
@@ -11,6 +11,7 @@
 ;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -107,3 +108,11 @@ is in the public domain.")
 ;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7.
 (define-public sqlite-with-column-metadata
   (deprecated-package "sqlite-with-column-metadata" sqlite))
+
+(define-public sqlite/hurd
+  ;; TODO move into sqlite on the next rebuild cycle.
+  (package
+    (inherit sqlite)
+    (name "sqlite-for-hurd")
+    (source (origin (inherit (package-source sqlite))
+                    (patches (search-patches "sqlite3-hurd.patch"))))))
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.

[Jan (janneke) Nieuwenhuizen]

This fixes

    guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)

* gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
sqlite.
---
 gnu/packages/guile.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index a32bd33293..c651e2427a 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -665,7 +665,7 @@ Guile's foreign function interface.")
        ("pkg-config" ,pkg-config)))
     (inputs
      `(("guile" ,guile-3.0)
-       ("sqlite" ,sqlite)))
+       ("sqlite" ,sqlite/hurd)))
     (synopsis "Access SQLite databases from Guile")
     (description
      "This package provides Guile bindings to the SQLite database system.")
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.

[Jan (janneke) Nieuwenhuizen]

* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.
---
 gnu/packages/package-management.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 068d46566c..52c5a1e60d 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n")))
          ("zlib" ,zlib)              ;for 'guix publish'
          ("lzlib" ,lzlib)            ;for 'guix publish' and 'guix substitute'
 
-         ("sqlite" ,sqlite)
+         ("sqlite" ,sqlite/hurd)
          ("libgcrypt" ,libgcrypt)
 
          ("guile" ,guile-3.0-latest)
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

[Jan Nieuwenhuizen]

Jan (janneke) Nieuwenhuizen writes:

> Maybe we're missing some file_lock patch that Debian has?  Where to look,
> glibc, hurd, ...?  Ideas?

So...I found a way to reproduce the feature/bug: run "pragma synchronous = normal;"
in two instances.

I created a db.sqlite using

    sh -x tests/guix-pack-localstatedir.sh 

(after removing the cleanup trap).

Then, doing

--8<---------------cut here---------------start------------->8---
$ sqlite3 db.sqlite 
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> SELECT * FROM ValidPaths;
1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||2664
2|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||1351880
3|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||12718808
4|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||432
5|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960
sqlite> pragma synchronous = normal;
sqlite>   C-c C-z^Z
[2]+  Stopped                 sqlite3 db.sqlite
[148]18:55:06 janneke@dundal:~/src/guix/master [env]
$ sqlite3 db.sqlite 
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> pragma synchronous = normal;
sqlite> 
--8<---------------cut here---------------end--------------->8---

works nice on GNU/Linux and on current master Childhurd gives

--8<---------------cut here---------------start------------->8---
$ ssh childhurd
Last login: Wed Jul  1 12:08:17 2020 from 10.0.2.2


  This is the GNU Hurd.  Welcome.

root@childhurd ~# sqlite3 db.sqlite 
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> SELECT * FROM ValidPaths;
1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||2664
2|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||1351880
3|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||12718808
4|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||432
5|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960
sqlite> pragma synchronous = normal;
sqlite>   C-c C-z^Z
[1]+  Stopped                 sqlite3 db.sqlite
root@childhurd ~# sqlite3 db.sqlite 
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> pragma synchronous = normal;
Error: locking protocol
sqlite> 
--8<---------------cut here---------------end--------------->8---

On Debian/Hurd (which I hoped had a patched, fixed sqlite3), it doesn't
work at all

--8<---------------cut here---------------start------------->8---
$ ssh hurd
GNU debian 0.9 GNU-Mach 1.8+git20191117-486/Hurd-0.9 i686-AT386


  This is the GNU Hurd.  Welcome.

The Hurd is not Linux.  Make sure to read
http://www.debian.org/ports/hurd/hurd-install
to check out the few things you _need_ to know.
Also check out the FAQ
http://www.gnu.org/software/hurd/faq.html
or its latest version on
http://darnassus.sceen.net/~hurd-web/faq/

To read a short intro on some nice features of the Hurd, just have a look at
the translator_primer file, for example via 'nano translator_primer'
Last login: Wed Jul  1 12:58:44 2020 from 10.0.2.2
janneke@debian:~$ sqlite3 db.sqlite
SQLite version 3.32.3 2020-06-18 14:00:33
Enter ".help" for usage hints.
sqlite> SELECT * FROM ValidPaths;
Error: unable to open database file
sqlite> 
--8<---------------cut here---------------end--------------->8---

So, still a similar question...is the patch simply broken/bitrotted, or
did some file locking change on the Hurd?

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

[Ludovic Courtès]

Hi!

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

> $ guix offload test
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> guix offload: 'childhurd' is running GNU Guile 3.0.4
> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)

Does sqlite pass its tests on GNU/Hurd?

Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon?
As a stop-gap, we could add a command-line option if that helps.

The “synchronous = normal” mode translates to ‘fsync’ calls, right?  If
you rpctrace sqlite3, do you see ‘file_sync’ calls failing?

My 2¢!

Ludo’.

bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> $ guix offload test
>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
>> guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>> guix offload: 'childhurd' is running GNU Guile 3.0.4
>> guix offload: error: exception occurred on remote host 'localhost':
>> (%exception #<inferior-object #<&store-protocol-error message:
>> "setting synchronous mode: locking protocol" status: 1>>)

> Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon?
> As a stop-gap, we could add a command-line option if that helps.

Tried that, thanks.  No, it does not help.  (But that's good news, see below!)

> The “synchronous = normal” mode translates to ‘fsync’ calls, right?  If
> you rpctrace sqlite3, do you see ‘file_sync’ calls failing?

Tried that before, rpctrace hangs before I see something useful.

> Does sqlite pass its tests on GNU/Hurd?

That's the (or at least a) right question: YES!

> My 2¢!

Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;)

It turns out that Debian's patch (and thus this patch series) is
probably OK: It fixes the locking problem on the Hurd, while exposing
another bug, apparently: "unable to open database file".

It seems there is a compatibility bug/problem/thing with the db.sqlite
that we produce on GNU/Linux.  While an unpatched sqlite3 on the Hurd
can read it, and work with it, the unpatched sqlite has locking
problems.  I found a workaround, though: dumping and loading  the
database file.

Look...

--8<---------------cut here---------------start------------->8---
$ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-orig
db.sqlite                                                                       100%  144KB   5.8MB/s   00:00    
11:30:37 janneke@dundal:~/tmp [env]
$ sqlite3 db.sqlite-orig .dump > db.dump
11:30:45 janneke@dundal:~/tmp [env]
$ sqlite3 -init db.dump db.sqlite-init .quit
-- Loading resources from db.dump
11:30:49 janneke@dundal:~/tmp [env]
$ cmp db.sqlite-orig db.sqlite-init 
db.sqlite-orig db.sqlite-init differ: byte 19, line 1
[1]11:31:11 janneke@dundal:~/tmp [env]
$ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlite
db.sqlite-init                                                                  100%  144KB   7.3MB/s   00:00    
11:31:21 janneke@dundal:~/tmp [env]
$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: 'localhost' is running GNU Guile 3.0.4
sending 1 store item (0 MiB) to 'localhost'...
exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
retrieving 1 store item from 'localhost'...
guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost'
11:31:29 janneke@dundal:~/tmp [env]
--8<---------------cut here---------------end--------------->8---

So...about the compatibility problem.  I tried to diff the db.sqlite-orig
db.sqlite-init binary files: they look completely different.  Not sure
how to handle this workaround, maybe we can insert a two system* calls
somewhere when building the disk image?

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

[Ludovic Courtès]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;)

Heheh.  :-)

> It turns out that Debian's patch (and thus this patch series) is
> probably OK: It fixes the locking problem on the Hurd, while exposing
> another bug, apparently: "unable to open database file".
>
> It seems there is a compatibility bug/problem/thing with the db.sqlite
> that we produce on GNU/Linux.  While an unpatched sqlite3 on the Hurd
> can read it, and work with it, the unpatched sqlite has locking
> problems.  I found a workaround, though: dumping and loading  the
> database file.
>
> Look...
>
> $ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-orig
> db.sqlite                                                                       100%  144KB   5.8MB/s   00:00    
> 11:30:37 janneke@dundal:~/tmp [env]
> $ sqlite3 db.sqlite-orig .dump > db.dump
> 11:30:45 janneke@dundal:~/tmp [env]
> $ sqlite3 -init db.dump db.sqlite-init .quit
> -- Loading resources from db.dump
> 11:30:49 janneke@dundal:~/tmp [env]
> $ cmp db.sqlite-orig db.sqlite-init 
> db.sqlite-orig db.sqlite-init differ: byte 19, line 1
> [1]11:31:11 janneke@dundal:~/tmp [env]
> $ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlite
> db.sqlite-init                                                                  100%  144KB   7.3MB/s   00:00    
> 11:31:21 janneke@dundal:~/tmp [env]
> $ guix offload test
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> guix offload: 'localhost' is running GNU Guile 3.0.4
> sending 1 store item (0 MiB) to 'localhost'...
> exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
> guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
> retrieving 1 store item from 'localhost'...
> guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost'
> 11:31:29 janneke@dundal:~/tmp [env]
>
> So...about the compatibility problem.  I tried to diff the db.sqlite-orig
> db.sqlite-init binary files: they look completely different.  Not sure
> how to handle this workaround, maybe we can insert a two system* calls
> somewhere when building the disk image?

Weird, weird!

Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created
from the same ‘sqlite3 -init’ command on GNU/Linux?  (Perhaps loading
the dump reorders entries or something.)

I think the binary format of the database is supposed to be
architecture-independent and filling it is supposed to be deterministic.

Congrats for getting this far anyway!

Ludo’.

bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

>> It seems there is a compatibility bug/problem/thing with the db.sqlite
>> that we produce on GNU/Linux.  While an unpatched sqlite3 on the Hurd
>> can read it, and work with it, the unpatched sqlite has locking
>> problems.  I found a workaround, though: dumping and loading  the
>> database file.

[..]

>> So...about the compatibility problem.  I tried to diff the db.sqlite-orig
>> db.sqlite-init binary files: they look completely different.  Not sure
>> how to handle this workaround, maybe we can insert a two system* calls
>> somewhere when building the disk image?
>
> Weird, weird!
>
> Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created
> from the same ‘sqlite3 -init’ command on GNU/Linux?

Yeah, they are identical.  The initial dump can only be created atm on
GNU/Linux; the dump can be loaded (obviously) anywhere like so

>> $ sqlite3 -init db.dump db.sqlite-init .quit

and the resulting initial db.sqlite is the same.  Guess we can do that
by hand for now...

> (Perhaps loading the dump reorders entries or something.)

Yes, "or something" certainly! :)  I have no clue...

If/when we decide to pinpoint this bug, what could be a first step?  Who
is creating the database right now, is that the C++ daemon or
guile-sqlite3.  IWBN to have that code create/save a smaller version and
see when reading fails.

> I think the binary format of the database is supposed to be
> architecture-independent and filling it is supposed to be deterministic.

Yes, that works.

> Congrats for getting this far anyway!

Yeah... \o/

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1448 bytes --]

Jan Nieuwenhuizen writes:

Adding one additional patch to this series, hence 4/3 (see attached)
fixes the sqlite problem.

With this over-complete series we can now do offload builds to a
childhurd:

--8<---------------cut here---------------start------------->8---
$ guix build -e '(@@ (gnu packages commencement) rsync-boot0)'
sending 74 store items (365 MiB) to 'localhost'...
exporting path `/gnu/store/1kf05mhh31wl89r3mlx52x5sxrjivi7i-module-import-compiled-guile-builder'
exporting path `/gnu/store/7jxamp3g8wdr6vwrisqfmrncxh8nbfb4-mirrors'
[...]
exporting path `/gnu/store/x8wnf0h0ipibgzadlbmll8bcnhvv1yaq-diffutils-boot0-3.7'
offloading '/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' to 'localhost'...
offloading build of /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv to 'localhost'
[...]
@ build-succeeded /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv -
retrieving 1 store item from 'localhost'...
importing file or directory '/gnu/store/haf6mlm8xa6s2q918s05pijl6ql17mnq-rsync-3.1.3'...
guix offload: error: corrupt input while restoring archive from #<input: string 7ff4794cee00>
guix build: error: build of `/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' failed
[1]10:04:55 janneke@dundal:~/src/guix/master [env]
--8<---------------cut here---------------end--------------->8---

After the build succeeds, the download fails but lets first get this
patch series done...

Janneke


[-- Attachment #2: 0004-database-Do-not-use-journal_model-WAL-for-the-Hurd.patch --]
[-- Type: text/x-patch, Size: 2677 bytes --]

From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001
From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>
Date: Fri, 3 Jul 2020 23:45:20 +0200
Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

This fixes <https://bugs.gnu.org/42151>.

* guix/store/database.scm (call-with-database): When building for the Hurd,
do not set journal_model=WAL.
---
 guix/store/database.scm | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/guix/store/database.scm b/guix/store/database.scm
index a38e4d7e52..da46b0abce 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
 ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,6 +21,7 @@
 (define-module (guix store database)
   #:use-module (sqlite3)
   #:use-module (guix config)
+  #:use-module (guix gexp)
   #:use-module (guix serialization)
   #:use-module (guix store deduplication)
   #:use-module (guix base16)
@@ -27,6 +29,7 @@
   #:use-module (guix build syscalls)
   #:use-module ((guix build utils)
                 #:select (mkdir-p executable-file?))
+  #:use-module (guix utils)
   #:use-module (guix build store-copy)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
@@ -105,9 +108,12 @@ create it and initialize it as a new database."
                      (mkdir-p (dirname file))
                      #t)))
         (db   (sqlite-open file)))
-    ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
-    ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
-    (sqlite-exec db "PRAGMA journal_mode=WAL;")
+    ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
+    (unless (let-system (system target)
+              (equal? target "i586-pc-gnu"))
+      ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
+      ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
+      (sqlite-exec db "PRAGMA journal_mode=WAL;"))
 
     ;; Install a busy handler such that, when the database is locked, sqlite
     ;; retries until 30 seconds have passed, at which point it gives up and
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com


[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

> Jan Nieuwenhuizen writes:
>
> Adding one additional patch to this series, hence 4/3 (see attached)
> fixes the sqlite problem.

For clarity -- patches 1-4 really do fix offloading to the Hurd and this...

> After the build succeeds, the download fails but lets first get this
> patch series done...

...is not true; you gat that when you (I) forgot to add keys in
/etc/guix/

As seen on IRC, possibly Ricardo will an update for the QEMU
command-line (at least for berlin), adding "--cpu base".

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.

[Ludovic Courtès]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

>>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001
> From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>
> Date: Fri, 3 Jul 2020 23:45:20 +0200
> Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
> Content-Transfer-Encoding: 8bit
> Content-Type: text/plain; charset=UTF-8
>
> This fixes <https://bugs.gnu.org/42151>.
>
> * guix/store/database.scm (call-with-database): When building for the Hurd,
> do not set journal_model=WAL.
> ---
>  guix/store/database.scm | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/guix/store/database.scm b/guix/store/database.scm
> index a38e4d7e52..da46b0abce 100644
> --- a/guix/store/database.scm
> +++ b/guix/store/database.scm
> @@ -1,6 +1,7 @@
>  ;;; GNU Guix --- Functional package management for GNU
>  ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
>  ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -20,6 +21,7 @@
>  (define-module (guix store database)
>    #:use-module (sqlite3)
>    #:use-module (guix config)
> +  #:use-module (guix gexp)
>    #:use-module (guix serialization)
>    #:use-module (guix store deduplication)
>    #:use-module (guix base16)
> @@ -27,6 +29,7 @@
>    #:use-module (guix build syscalls)
>    #:use-module ((guix build utils)
>                  #:select (mkdir-p executable-file?))
> +  #:use-module (guix utils)
>    #:use-module (guix build store-copy)
>    #:use-module (srfi srfi-1)
>    #:use-module (srfi srfi-11)
> @@ -105,9 +108,12 @@ create it and initialize it as a new database."
>                       (mkdir-p (dirname file))
>                       #t)))
>          (db   (sqlite-open file)))
> -    ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
> -    ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
> -    (sqlite-exec db "PRAGMA journal_mode=WAL;")
> +    ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
> +    (unless (let-system (system target)
> +              (equal? target "i586-pc-gnu"))
> +      ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
> +      ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
> +      (sqlite-exec db "PRAGMA journal_mode=WAL;"))

I think this has the same effect as:

  (unless #t
    (sqlite-exec db "PRAGMA journal_mode=WAL;"))

because this code is not in a gexp, so ‘let-system’ evaluates to an
object (not #f), and that’s it, it’s never lowered and the ‘equal?’ call
is never made.

You want to disable WAL mode not just when running this code natively on
GNU/Hurd, but also when building a database that will eventually be used
on GNU/Hurd, right?

In that case, I think you’ll have to add, say, a #:wal-mode? parameter
to ‘call-with-database’ (defaulting to #true), and change the caller to
set it appropriately.  The caller, directly or indirectly, is in a gexp,
where you can use the ‘let-system’ expression above.

Does that make sense?

Thanks, and apologies for the delay!

Ludo’.

bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix.

[Ludovic Courtès]

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

> * gnu/packages/patches/sqlite3-hurd.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/packages/sqlite.scm (sqlite/hurd): New variable.

[…]

> +++ b/gnu/packages/patches/sqlite3-hurd.patch
> @@ -0,0 +1,51 @@
> +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
> +Upstream status: Not offered upstream.

The URL is 404, could you fix it?  It would also be great if you could
add a sentence or two explaining what this works around, so that our
future selves know how to deal with this patch.  :-)

Otherwise LGTM, you can push with changes along these lines!

Ludo’.

bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.

[Ludovic Courtès]

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

> This fixes
>
>     guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
>
> * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
> sqlite.
> ---
>  gnu/packages/guile.scm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
> index a32bd33293..c651e2427a 100644
> --- a/gnu/packages/guile.scm
> +++ b/gnu/packages/guile.scm
> @@ -665,7 +665,7 @@ Guile's foreign function interface.")
>         ("pkg-config" ,pkg-config)))
>      (inputs
>       `(("guile" ,guile-3.0)
> -       ("sqlite" ,sqlite)))
> +       ("sqlite" ,sqlite/hurd)))

This should be guarded by ‘if (hurd-target?)’, right?

Ludo’.

bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.

[Ludovic Courtès]

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

> * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.

Same here.

Which makes me thing: how about applying the sqlite/hurd patch in a
build phase of sqlite that would be Hurd-specific?  That way, we’d still
have a single sqlite package.

Thanks,
Ludo’.

bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

[Ludovic Courtès]

Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
> Hi!
>
>>> It seems there is a compatibility bug/problem/thing with the db.sqlite
>>> that we produce on GNU/Linux.  While an unpatched sqlite3 on the Hurd
>>> can read it, and work with it, the unpatched sqlite has locking
>>> problems.  I found a workaround, though: dumping and loading  the
>>> database file.
>
> [..]
>
>>> So...about the compatibility problem.  I tried to diff the db.sqlite-orig
>>> db.sqlite-init binary files: they look completely different.  Not sure
>>> how to handle this workaround, maybe we can insert a two system* calls
>>> somewhere when building the disk image?
>>
>> Weird, weird!
>>
>> Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created
>> from the same ‘sqlite3 -init’ command on GNU/Linux?
>
> Yeah, they are identical.  The initial dump can only be created atm on
> GNU/Linux; the dump can be loaded (obviously) anywhere like so
>
>>> $ sqlite3 -init db.dump db.sqlite-init .quit
>
> and the resulting initial db.sqlite is the same.  Guess we can do that
> by hand for now...
>
>> (Perhaps loading the dump reorders entries or something.)
>
> Yes, "or something" certainly! :)  I have no clue...

Sorry for catching up days later but… did disabling WAL mode fix this
discrepancy?  IOW, without WAL mode, is the file produced on GNU/Linux
bit-identical to that produced on GNU/Hurd?

> If/when we decide to pinpoint this bug, what could be a first step?  Who
> is creating the database right now, is that the C++ daemon or
> guile-sqlite3.  IWBN to have that code create/save a smaller version and
> see when reading fails.

As you found out it’s created by (guix store database) in this case.

Thanks,
Ludo’.

bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>>>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001
>> From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>
>> Date: Fri, 3 Jul 2020 23:45:20 +0200
>> Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
>> Content-Transfer-Encoding: 8bit
>> Content-Type: text/plain; charset=UTF-8
>>
>> This fixes <https://bugs.gnu.org/42151>.
[...]
>> diff --git a/guix/store/database.scm b/guix/store/database.scm
>> index a38e4d7e52..da46b0abce 100644
>> --- a/guix/store/database.scm
>> +++ b/guix/store/database.scm
[...]
>> +    ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
>> +    (unless (let-system (system target)
>> +              (equal? target "i586-pc-gnu"))
>> +      ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
>> +      ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
>> +      (sqlite-exec db "PRAGMA journal_mode=WAL;"))
>
> I think this has the same effect as:
>
>   (unless #t
>     (sqlite-exec db "PRAGMA journal_mode=WAL;"))
>
> because this code is not in a gexp, so ‘let-system’ evaluates to an
> object (not #f), and that’s it, it’s never lowered and the ‘equal?’ call
> is never made.

Oops.  Great catch, thanks!

> You want to disable WAL mode not just when running this code natively on
> GNU/Hurd, but also when building a database that will eventually be used
> on GNU/Hurd, right?

Exactly.

> In that case, I think you’ll have to add, say, a #:wal-mode? parameter
> to ‘call-with-database’ (defaulting to #true), and change the caller to
> set it appropriately.  The caller, directly or indirectly, is in a gexp,
> where you can use the ‘let-system’ expression above.
>
> Does that make sense?

Yes, very much so. [...] After some typing, it turns out that the whole
let-system is gone, we can set use a hard-coded #:wal-mode? #t in
hurd-initialize-root-partition.

> Thanks, and apologies for the delay!

Sure, I'm happy I didn't "just push" ;-)

I'm sending a v2 patch set in a minute.

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH v2 1/4] gnu: Add sqlite/hurd with locking fix.

[Jan (janneke) Nieuwenhuizen]

* gnu/packages/patches/sqlite3-hurd.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/sqlite.scm (sqlite/hurd): Use it in new variable.
---
 gnu/local.mk                            |  1 +
 gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++
 gnu/packages/sqlite.scm                 |  9 +++++
 3 files changed, 61 insertions(+)
 create mode 100644 gnu/packages/patches/sqlite3-hurd.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 5c3b391960..e452004945 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1370,6 +1370,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/sdl-pango-header-guard.patch		\
   %D%/packages/patches/sdl-pango-matrix_declarations.patch	\
   %D%/packages/patches/sdl-pango-sans-serif.patch		\
+  %D%/packages/patches/sqlite3-hurd.patch			\
   %D%/packages/patches/patchutils-test-perms.patch		\
   %D%/packages/patches/patch-hurd-path-max.patch		\
   %D%/packages/patches/perl-autosplit-default-time.patch	\
diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patch
new file mode 100644
index 0000000000..de87a30cb1
--- /dev/null
+++ b/gnu/packages/patches/sqlite3-hurd.patch
@@ -0,0 +1,51 @@
+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
+Upstream status: Not offered upstream.
+
+diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c
+--- sqlite-autoconf-3310100/sqlite3.c	2020-01-27 21:25:19.000000000 +0100
++++ sqlite-autoconf-3310100-/sqlite3.c	2020-07-01 11:50:13.768333806 +0200
+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode
+ # include <sys/mman.h>
+ #endif
+ 
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ /* # include <sys/ioctl.h> */
+ # include <sys/file.h>
+ # include <sys/param.h>
+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id
+ **
+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off
+ */
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ 
+ /*
+ ** Retry flock() calls that fail with EINTR
+@@ -38586,7 +38586,7 @@ IOMETHODS(
+   0                         /* xShmMap method */
+ )
+ 
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ IOMETHODS(
+   flockIoFinder,            /* Finder function name */
+   flockIoMethods,           /* sqlite3_io_methods object name */
+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){
+     UNIXVFS("unix",          autolockIoFinder ),
+ #elif OS_VXWORKS
+     UNIXVFS("unix",          vxworksIoFinder ),
++#elif defined(__GNU__)
++    UNIXVFS("unix",          flockIoFinder ),
+ #else
+     UNIXVFS("unix",          posixIoFinder ),
+ #endif
+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){
+ #if OS_VXWORKS
+     UNIXVFS("unix-namedsem", semIoFinder ),
+ #endif
+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS
++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)
+     UNIXVFS("unix-posix",    posixIoFinder ),
+ #endif
+ #if SQLITE_ENABLE_LOCKING_STYLE
diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm
index 8468131ddf..d5c44b0d0f 100644
--- a/gnu/packages/sqlite.scm
+++ b/gnu/packages/sqlite.scm
@@ -11,6 +11,7 @@
 ;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -107,3 +108,11 @@ is in the public domain.")
 ;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7.
 (define-public sqlite-with-column-metadata
   (deprecated-package "sqlite-with-column-metadata" sqlite))
+
+(define-public sqlite/hurd
+  ;; TODO move into sqlite on the next rebuild cycle.
+  (package
+    (inherit sqlite)
+    (name "sqlite-for-hurd")
+    (source (origin (inherit (package-source sqlite))
+                    (patches (search-patches "sqlite3-hurd.patch"))))))
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH v2 2/4] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.

[Jan (janneke) Nieuwenhuizen]

This fixes

    guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)

* gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
sqlite.
---
 gnu/packages/guile.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index a32bd33293..c651e2427a 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -665,7 +665,7 @@ Guile's foreign function interface.")
        ("pkg-config" ,pkg-config)))
     (inputs
      `(("guile" ,guile-3.0)
-       ("sqlite" ,sqlite)))
+       ("sqlite" ,sqlite/hurd)))
     (synopsis "Access SQLite databases from Guile")
     (description
      "This package provides Guile bindings to the SQLite database system.")
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH v2 3/4] gnu: guix: Use sqlite/hurd for locking on the Hurd.

[Jan (janneke) Nieuwenhuizen]

* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.
---
 gnu/packages/package-management.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 9986976cc6..64c6931011 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n")))
          ("zlib" ,zlib)              ;for 'guix publish'
          ("lzlib" ,lzlib)            ;for 'guix publish' and 'guix substitute'
 
-         ("sqlite" ,sqlite)
+         ("sqlite" ,sqlite/hurd)
          ("libgcrypt" ,libgcrypt)
 
          ("guile" ,guile-3.0-latest)
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH v2 4/4] image: Do not set journal_model=WAL for the Hurd.

[Jan (janneke) Nieuwenhuizen]

This fixes <https://bugs.gnu.org/42151>.

* gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
in call to ...
* gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?
parameter, pass it to ...
(register-closure): ... this, add #:wal-mode? parameter, pass it to ...
* guix/store/database.scm (with-database): ... this, add #:wal-mode?
parameter, pass it to ...
(call-with-database): ... this, add #:wal-mode? parameter; when
set to #f, do not set journal_model=WAL.
---
 gnu/build/image.scm        | 26 ++++++++++++++++----------
 gnu/system/images/hurd.scm |  4 ++--
 guix/store/database.scm    | 29 ++++++++++++++++++++---------
 3 files changed, 38 insertions(+), 21 deletions(-)

diff --git a/gnu/build/image.scm b/gnu/build/image.scm
index e7b0418182..d8efa73f16 100644
--- a/gnu/build/image.scm
+++ b/gnu/build/image.scm
@@ -131,20 +131,23 @@ given CONFIG file."
 (define* (register-closure prefix closure
                            #:key
                            (deduplicate? #t) (reset-timestamps? #t)
-                           (schema (sql-schema)))
+                           (schema (sql-schema))
+                           (wal-mode? #t))
   "Register CLOSURE in PREFIX, where PREFIX is the directory name of the
 target store and CLOSURE is the name of a file containing a reference graph as
 produced by #:references-graphs..  As a side effect, if RESET-TIMESTAMPS? is
 true, reset timestamps on store files and, if DEDUPLICATE? is true,
-deduplicates files common to CLOSURE and the rest of PREFIX."
+deduplicates files common to CLOSURE and the rest of PREFIX.  Pass WAL-MODE?
+to call-with-database."
   (let ((items (call-with-input-file closure read-reference-graph)))
     (parameterize ((sql-schema schema))
       (with-database (store-database-file #:prefix prefix) db
-        (register-items db items
-                        #:prefix prefix
-                        #:deduplicate? deduplicate?
-                        #:reset-timestamps? reset-timestamps?
-                        #:registration-time %epoch)))))
+       #:wal-mode? wal-mode?
+       (register-items db items
+                       #:prefix prefix
+                       #:deduplicate? deduplicate?
+                       #:reset-timestamps? reset-timestamps?
+                       #:registration-time %epoch)))))
 
 (define* (initialize-efi-partition root
                                    #:key
@@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX."
                                     (register-closures? #t)
                                     system-directory
                                     make-device-nodes
+                                    (wal-mode? #t)
                                     #:allow-other-keys)
   "Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to
 install the bootloader configuration.
 
 If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store.  If
 DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the
-rest of the store when registering the closures. SYSTEM-DIRECTORY is the name
-of the directory of the 'system' derivation."
+rest of the store when registering the closures.  SYSTEM-DIRECTORY is the name
+of the directory of the 'system' derivation.  Pass WAL-MODE? to
+register-closure."
   (populate-root-file-system system-directory root)
   (populate-store references-graphs root)
 
@@ -184,7 +189,8 @@ of the directory of the 'system' derivation."
                 (register-closure root
                                   closure
                                   #:reset-timestamps? #t
-                                  #:deduplicate? deduplicate?))
+                                  #:deduplicate? deduplicate?
+                                  #:wal-mode? wal-mode?))
               references-graphs))
 
   (when bootloader-installer
diff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scm
index 31942e7386..4e1db37104 100644
--- a/gnu/system/images/hurd.scm
+++ b/gnu/system/images/hurd.scm
@@ -61,8 +61,8 @@
   #~(lambda* (#:rest args)
       (apply initialize-root-partition
              (append args
-                     (list #:make-device-nodes
-                           make-hurd-device-nodes)))))
+                     (list #:make-device-nodes make-hurd-device-nodes
+                           #:wal-mode? #f)))))
 
 (define hurd-disk-image
   (image
diff --git a/guix/store/database.scm b/guix/store/database.scm
index a38e4d7e52..50b66ce282 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
 ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,6 +21,7 @@
 (define-module (guix store database)
   #:use-module (sqlite3)
   #:use-module (guix config)
+  #:use-module (guix gexp)
   #:use-module (guix serialization)
   #:use-module (guix store deduplication)
   #:use-module (guix base16)
@@ -27,6 +29,7 @@
   #:use-module (guix build syscalls)
   #:use-module ((guix build utils)
                 #:select (mkdir-p executable-file?))
+  #:use-module (guix utils)
   #:use-module (guix build store-copy)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
@@ -97,17 +100,20 @@ as specified by SQL-SCHEMA."
 
   (sqlite-exec db (call-with-input-file schema get-string-all)))
 
-(define (call-with-database file proc)
+(define* (call-with-database file proc #:key (wal-mode? #t))
   "Pass PROC a database record corresponding to FILE.  If FILE doesn't exist,
-create it and initialize it as a new database."
+create it and initialize it as a new database.  Unless WAL-MODE? is set to #f,
+set journal_mode=WAL."
   (let ((new? (and (not (file-exists? file))
                    (begin
                      (mkdir-p (dirname file))
                      #t)))
         (db   (sqlite-open file)))
-    ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
-    ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
-    (sqlite-exec db "PRAGMA journal_mode=WAL;")
+    ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
+    (when wal-mode?
+      ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
+      ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
+      (sqlite-exec db "PRAGMA journal_mode=WAL;"))
 
     ;; Install a busy handler such that, when the database is locked, sqlite
     ;; retries until 30 seconds have passed, at which point it gives up and
@@ -200,10 +206,15 @@ prior to returning."
   ;; Default location of the store database.
   (string-append %store-database-directory "/db.sqlite"))
 
-(define-syntax-rule (with-database file db exp ...)
-  "Open DB from FILE and close it when the dynamic extent of EXP... is left.
-If FILE doesn't exist, create it and initialize it as a new database."
-  (call-with-database file (lambda (db) exp ...)))
+(define-syntax with-database
+  (syntax-rules ()
+    "Open DB from FILE and close it when the dynamic extent of EXP... is left.
+If FILE doesn't exist, create it and initialize it as a new database.  Pass
+#:wal-mode? to call-with-database."
+    ((_ file db #:wal-mode? wal-mode? exp ...)
+     (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?))
+    ((_ file db exp ...)
+     (call-with-database file (lambda (db) exp ...)))))
 
 (define (sqlite-finalize stmt)
   ;; As of guile-sqlite3 0.1.0, cached statements aren't reset when
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

>> * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.
>
> Same here.
>
> Which makes me thing: how about applying the sqlite/hurd patch in a
> build phase of sqlite that would be Hurd-specific?  That way, we’d still
> have a single sqlite package.

Ah, yes that sounds better.  We don't need sqlite/hurd "sqlite-for-hurd"
and all that.  I'll give that a go! 

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> * gnu/packages/patches/sqlite3-hurd.patch: New file.
>> * gnu/local.mk (dist_patch_DATA): Add it.
>> * gnu/packages/sqlite.scm (sqlite/hurd): New variable.
>
> […]
>
>> +++ b/gnu/packages/patches/sqlite3-hurd.patch
>> @@ -0,0 +1,51 @@
>> +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
>> +Upstream status: Not offered upstream.
>
> The URL is 404, could you fix it?

That's weird...Changing it (in completely rewritten patch) to

    https://sources.debian.org/patches/sqlite3/3.32.3-1/20-hurd-locking-style.patch/

> It would also be great if you could
> add a sentence or two explaining what this works around, so that our
> future selves know how to deal with this patch.  :-)

Ah, sure!

> Otherwise LGTM, you can push with changes along these lines!

As it's so much changed, I'll send it by one more time :-)

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> This fixes
>>
>>     guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
>>
>> * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
>> sqlite.
>> ---
>>  gnu/packages/guile.scm | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
>> index a32bd33293..c651e2427a 100644
>> --- a/gnu/packages/guile.scm
>> +++ b/gnu/packages/guile.scm
>> @@ -665,7 +665,7 @@ Guile's foreign function interface.")
>>         ("pkg-config" ,pkg-config)))
>>      (inputs
>>       `(("guile" ,guile-3.0)
>> -       ("sqlite" ,sqlite)))
>> +       ("sqlite" ,sqlite/hurd)))
>
> This should be guarded by ‘if (hurd-target?)’, right?

Yeah...but all changed now...look out for v3.  Thanks!

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH v3 1/2] gnu: sqlite: Add locking-mode fix for the Hurd.

[Jan (janneke) Nieuwenhuizen]

* gnu/packages/patches/sqlite-hurd.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when building
for the Hurd.
[arguments]: Apply it when building for the Hurd.
---
 gnu/local.mk                           |  1 +
 gnu/packages/patches/sqlite-hurd.patch | 58 ++++++++++++++++++++++++++
 gnu/packages/sqlite.scm                | 19 +++++++++
 3 files changed, 78 insertions(+)
 create mode 100644 gnu/packages/patches/sqlite-hurd.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 5c3b391960..42c9d0f379 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1370,6 +1370,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/sdl-pango-header-guard.patch		\
   %D%/packages/patches/sdl-pango-matrix_declarations.patch	\
   %D%/packages/patches/sdl-pango-sans-serif.patch		\
+  %D%/packages/patches/sqlite-hurd.patch			\
   %D%/packages/patches/patchutils-test-perms.patch		\
   %D%/packages/patches/patch-hurd-path-max.patch		\
   %D%/packages/patches/perl-autosplit-default-time.patch	\
diff --git a/gnu/packages/patches/sqlite-hurd.patch b/gnu/packages/patches/sqlite-hurd.patch
new file mode 100644
index 0000000000..d80a2c5be8
--- /dev/null
+++ b/gnu/packages/patches/sqlite-hurd.patch
@@ -0,0 +1,58 @@
+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.32.3-1/20-hurd-locking-style.patch
+Upstream status: Not upstreamed.
+
+This patch is needed to get offloading to work.
+
+Sqlite can use simple file locking mode, but that does not work for the Hurd;
+a second sqlite process fails with a "locking protocol" error.
+
+See also: https://bugs.debian.org/529734.
+
+diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c
+--- sqlite-autoconf-3310100/sqlite3.c	2020-01-27 21:25:19.000000000 +0100
++++ sqlite-autoconf-3310100-/sqlite3.c	2020-07-01 11:50:13.768333806 +0200
+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode
+ # include <sys/mman.h>
+ #endif
+ 
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ /* # include <sys/ioctl.h> */
+ # include <sys/file.h>
+ # include <sys/param.h>
+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id
+ **
+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off
+ */
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ 
+ /*
+ ** Retry flock() calls that fail with EINTR
+@@ -38586,7 +38586,7 @@ IOMETHODS(
+   0                         /* xShmMap method */
+ )
+ 
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ IOMETHODS(
+   flockIoFinder,            /* Finder function name */
+   flockIoMethods,           /* sqlite3_io_methods object name */
+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){
+     UNIXVFS("unix",          autolockIoFinder ),
+ #elif OS_VXWORKS
+     UNIXVFS("unix",          vxworksIoFinder ),
++#elif defined(__GNU__)
++    UNIXVFS("unix",          flockIoFinder ),
+ #else
+     UNIXVFS("unix",          posixIoFinder ),
+ #endif
+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){
+ #if OS_VXWORKS
+     UNIXVFS("unix-namedsem", semIoFinder ),
+ #endif
+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS
++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)
+     UNIXVFS("unix-posix",    posixIoFinder ),
+ #endif
+ #if SQLITE_ENABLE_LOCKING_STYLE
diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm
index 8468131ddf..eeb77749d8 100644
--- a/gnu/packages/sqlite.scm
+++ b/gnu/packages/sqlite.scm
@@ -11,6 +11,7 @@
 ;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -29,6 +30,7 @@
 
 (define-module (gnu packages sqlite)
   #:use-module (gnu packages)
+  #:use-module (gnu packages hurd)
   #:use-module (gnu packages readline)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix packages)
@@ -65,6 +67,11 @@
               "1bj936svd8i5g25xd1bj52hj4zca01fgl3sqkj86z9q5pkz4wa32"))))
    (build-system gnu-build-system)
    (inputs `(("readline" ,readline)))
+   (native-inputs (if (hurd-target?)
+                      ;; TODO move into origin on the next rebuild cycle.
+                      `(("hurd-locking-mode.patch"
+                         ,@(search-patches "sqlite-hurd.patch")))
+                      '()))
    (outputs '("out" "static"))
    (arguments
     `(#:configure-flags
@@ -79,6 +86,18 @@
                            ;; Column metadata is required by GNU Jami and Qt, et.al.
                            "-DSQLITE_ENABLE_COLUMN_METADATA"))
       #:phases (modify-phases %standard-phases
+                 ;; TODO: remove in the next rebuild cycle
+                 ,@(if (hurd-target?)
+                       `((add-after 'unpack 'patch-sqlite/hurd
+                           (lambda* (#:key inputs native-inputs
+                                     #:allow-other-keys)
+                             (let ((patch (assoc-ref
+                                           (if ,(%current-target-system)
+                                               native-inputs
+                                               inputs)
+                                           "hurd-locking-mode.patch")))
+                               (invoke "patch" "-p1" "--force" "-i" patch)))))
+                       '())
                  (add-after 'install 'move-static-library
                    (lambda* (#:key outputs #:allow-other-keys)
                      (let* ((out    (assoc-ref outputs "out"))
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd.

[Jan (janneke) Nieuwenhuizen]

This fixes <https://bugs.gnu.org/42151>.

* gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
in call to ...
* gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?
parameter, pass it to ...
(register-closure): ... this, add #:wal-mode? parameter, pass it to ...
* guix/store/database.scm (with-database): ... this, add #:wal-mode?
parameter, pass it to ...
(call-with-database): ... this, add #:wal-mode? parameter; when
set to #f, do not set journal_model=WAL.
---
 gnu/build/image.scm        | 26 ++++++++++++++++----------
 gnu/system/images/hurd.scm |  4 ++--
 guix/store/database.scm    | 29 ++++++++++++++++++++---------
 3 files changed, 38 insertions(+), 21 deletions(-)

diff --git a/gnu/build/image.scm b/gnu/build/image.scm
index e7b0418182..d8efa73f16 100644
--- a/gnu/build/image.scm
+++ b/gnu/build/image.scm
@@ -131,20 +131,23 @@ given CONFIG file."
 (define* (register-closure prefix closure
                            #:key
                            (deduplicate? #t) (reset-timestamps? #t)
-                           (schema (sql-schema)))
+                           (schema (sql-schema))
+                           (wal-mode? #t))
   "Register CLOSURE in PREFIX, where PREFIX is the directory name of the
 target store and CLOSURE is the name of a file containing a reference graph as
 produced by #:references-graphs..  As a side effect, if RESET-TIMESTAMPS? is
 true, reset timestamps on store files and, if DEDUPLICATE? is true,
-deduplicates files common to CLOSURE and the rest of PREFIX."
+deduplicates files common to CLOSURE and the rest of PREFIX.  Pass WAL-MODE?
+to call-with-database."
   (let ((items (call-with-input-file closure read-reference-graph)))
     (parameterize ((sql-schema schema))
       (with-database (store-database-file #:prefix prefix) db
-        (register-items db items
-                        #:prefix prefix
-                        #:deduplicate? deduplicate?
-                        #:reset-timestamps? reset-timestamps?
-                        #:registration-time %epoch)))))
+       #:wal-mode? wal-mode?
+       (register-items db items
+                       #:prefix prefix
+                       #:deduplicate? deduplicate?
+                       #:reset-timestamps? reset-timestamps?
+                       #:registration-time %epoch)))))
 
 (define* (initialize-efi-partition root
                                    #:key
@@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX."
                                     (register-closures? #t)
                                     system-directory
                                     make-device-nodes
+                                    (wal-mode? #t)
                                     #:allow-other-keys)
   "Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to
 install the bootloader configuration.
 
 If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store.  If
 DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the
-rest of the store when registering the closures. SYSTEM-DIRECTORY is the name
-of the directory of the 'system' derivation."
+rest of the store when registering the closures.  SYSTEM-DIRECTORY is the name
+of the directory of the 'system' derivation.  Pass WAL-MODE? to
+register-closure."
   (populate-root-file-system system-directory root)
   (populate-store references-graphs root)
 
@@ -184,7 +189,8 @@ of the directory of the 'system' derivation."
                 (register-closure root
                                   closure
                                   #:reset-timestamps? #t
-                                  #:deduplicate? deduplicate?))
+                                  #:deduplicate? deduplicate?
+                                  #:wal-mode? wal-mode?))
               references-graphs))
 
   (when bootloader-installer
diff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scm
index 31942e7386..4e1db37104 100644
--- a/gnu/system/images/hurd.scm
+++ b/gnu/system/images/hurd.scm
@@ -61,8 +61,8 @@
   #~(lambda* (#:rest args)
       (apply initialize-root-partition
              (append args
-                     (list #:make-device-nodes
-                           make-hurd-device-nodes)))))
+                     (list #:make-device-nodes make-hurd-device-nodes
+                           #:wal-mode? #f)))))
 
 (define hurd-disk-image
   (image
diff --git a/guix/store/database.scm b/guix/store/database.scm
index a38e4d7e52..50b66ce282 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
 ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,6 +21,7 @@
 (define-module (guix store database)
   #:use-module (sqlite3)
   #:use-module (guix config)
+  #:use-module (guix gexp)
   #:use-module (guix serialization)
   #:use-module (guix store deduplication)
   #:use-module (guix base16)
@@ -27,6 +29,7 @@
   #:use-module (guix build syscalls)
   #:use-module ((guix build utils)
                 #:select (mkdir-p executable-file?))
+  #:use-module (guix utils)
   #:use-module (guix build store-copy)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
@@ -97,17 +100,20 @@ as specified by SQL-SCHEMA."
 
   (sqlite-exec db (call-with-input-file schema get-string-all)))
 
-(define (call-with-database file proc)
+(define* (call-with-database file proc #:key (wal-mode? #t))
   "Pass PROC a database record corresponding to FILE.  If FILE doesn't exist,
-create it and initialize it as a new database."
+create it and initialize it as a new database.  Unless WAL-MODE? is set to #f,
+set journal_mode=WAL."
   (let ((new? (and (not (file-exists? file))
                    (begin
                      (mkdir-p (dirname file))
                      #t)))
         (db   (sqlite-open file)))
-    ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
-    ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
-    (sqlite-exec db "PRAGMA journal_mode=WAL;")
+    ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
+    (when wal-mode?
+      ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
+      ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
+      (sqlite-exec db "PRAGMA journal_mode=WAL;"))
 
     ;; Install a busy handler such that, when the database is locked, sqlite
     ;; retries until 30 seconds have passed, at which point it gives up and
@@ -200,10 +206,15 @@ prior to returning."
   ;; Default location of the store database.
   (string-append %store-database-directory "/db.sqlite"))
 
-(define-syntax-rule (with-database file db exp ...)
-  "Open DB from FILE and close it when the dynamic extent of EXP... is left.
-If FILE doesn't exist, create it and initialize it as a new database."
-  (call-with-database file (lambda (db) exp ...)))
+(define-syntax with-database
+  (syntax-rules ()
+    "Open DB from FILE and close it when the dynamic extent of EXP... is left.
+If FILE doesn't exist, create it and initialize it as a new database.  Pass
+#:wal-mode? to call-with-database."
+    ((_ file db #:wal-mode? wal-mode? exp ...)
+     (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?))
+    ((_ file db exp ...)
+     (call-with-database file (lambda (db) exp ...)))))
 
 (define (sqlite-finalize stmt)
   ;; As of guile-sqlite3 0.1.0, cached statements aren't reset when
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#42151: [PATCH v3 1/2] gnu: sqlite: Add locking-mode fix for the Hurd.

[Ludovic Courtès]

Hi!

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

> * gnu/packages/patches/sqlite-hurd.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when building
> for the Hurd.
> [arguments]: Apply it when building for the Hurd.

LGTM, thank you!

bug#42151: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd.

[Ludovic Courtès]

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

> This fixes <https://bugs.gnu.org/42151>.
>
> * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
> in call to ...
> * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?
> parameter, pass it to ...
> (register-closure): ... this, add #:wal-mode? parameter, pass it to ...
> * guix/store/database.scm (with-database): ... this, add #:wal-mode?
> parameter, pass it to ...
> (call-with-database): ... this, add #:wal-mode? parameter; when
> set to #f, do not set journal_model=WAL.

[...]

> +++ b/gnu/system/images/hurd.scm
> @@ -61,8 +61,8 @@
>    #~(lambda* (#:rest args)
>        (apply initialize-root-partition
>               (append args
> -                     (list #:make-device-nodes
> -                           make-hurd-device-nodes)))))
> +                     (list #:make-device-nodes make-hurd-device-nodes
> +                           #:wal-mode? #f)))))

IWBN to have an XXX comment here stating why we disable WAL mode on
GNU/Hurd, possibly referencing this issue.  That way, we can eventually
revisit the issue and hopefully get rid of this workaround.

Apart from that the patch LGTM, thanks!

Ludo’.

bug#42151: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> This fixes <https://bugs.gnu.org/42151>.
>>
>> * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
>> in call to ...
[...]
>> +                     (list #:make-device-nodes make-hurd-device-nodes
>> +                           #:wal-mode? #f)))))
>
> IWBN to have an XXX comment here stating why we disable WAL mode on
> GNU/Hurd, possibly referencing this issue.  That way, we can eventually
> revisit the issue and hopefully get rid of this workaround.

Sure, done!

> Apart from that the patch LGTM, thanks!

Yay!  Pushed series to master as 4b9eecd322e566783369795ebea63a479b51f486.

Let the offloadings commence!

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com