From: ludo@gnu.org (Ludovic Courtès)
To: "Clément Lassieur" <clement@lassieur.org>
Cc: 32234@debbugs.gnu.org
Subject: bug#32234: [PATCH 2/2] database: Serialize all database accesses in a thread.
Date: Sun, 19 Aug 2018 16:06:03 +0200 [thread overview]
Message-ID: <87zhxih2no.fsf@gnu.org> (raw)
In-Reply-To: <20180806192736.1747-2-clement@lassieur.org> ("Clément Lassieur"'s message of "Mon, 6 Aug 2018 21:27:36 +0200")
Hi Clément!
Clément Lassieur <clement@lassieur.org> skribis:
> Fixes <https://bugs.gnu.org/32234>.
>
> * bin/cuirass.in (main): Keep only one WITH-DATABASE call around all fibers.
> Remove all DB arguments.
> * src/cuirass/base.scm (evaluate, update-build-statuses!, spawn-builds,
> handle-build-event, build-packages): Remove all DB arguments.
> (clear-build-queue, cancel-old-builds): Wrap in WITH-DB-CRITICAL-SECTION,
> remove all DB arguments.
> (restart-builds): Remove the NON-BLOCKING call, remove all DB arguments.
> (process-specs): Remove all DB arguments, remove the WITH-DATABASE call.
> * src/cuirass/database.scm (%db-channel): New parameter.
> (with-db-critical-section): New macro.
> (db-add-input, db-add-specification, db-get-inputs, db-get-specifications,
> db-add-evaluation, db-add-build, db-update-build-status!, db-get-outputs,
> db-get-builds, db-get-build, db-get-pending-derivations, db-get-stamp,
> db-add-stamp, db-get-evaluations, db-get-evaluations-build-summary,
> db-get-evaluations-id-min, db-get-evaluations-id-max, db-get-builds-min,
> db-get-builds-max): Wrap in WITH-DB-CRITICAL-SECTION, remove all DB arguments.
> (with-database): Wrap BODY in PARAMETERIZE form that sets %DB-CHANNEL to the
> channel returned by MAKE-CRITICAL-SECTION.
> * src/cuirass/http.scm (handle-build-request, handle-builds-request): Remove
> all DB arguments.
> (url-handler): Remove all DB arguments, remove the DB-CHANNEL state, remove
> the WITH-CRITICAL-SECTION calls.
> (run-cuirass-server): Remove the DB arguments, remove the
> MAKE-CRITICAL-SECTION call.
> * src/cuirass/utils.scm (make-critical-section): Replace SPAWN-FIBER with
> CALL-WITH-NEW-THREAD. Wrap body in PARAMETERIZE form that clears
> CURRENT-FIBER.
> * tests/database.scm (with-temporary-database, db-add-specification,
> db-add-build, db-update-build-status!, db-get-builds,
> db-get-pending-derivations): Remove the DB arguments.
> (db-init): Set the %DB-CHANNEL parameter to the channel returned by
> MAKE-CRITICAL-SECTION, and return #t.
> (database): Set %DB-CHANNEL to #f during cleanup.
> * tests/http.scm (db-init): Set the %DB-CHANNEL parameter to the channel
> returned by MAKE-CRITICAL-SECTION, and return #t.
> (cuirass-run, fill-db): Remove the DB arguments.
> (http): Set %DB-CHANNEL to #f during cleanup.
Excellent, thanks for working on this! This looks great to me, and I
think the pros outweigh the cons. Did you check on a big database how
well it performs?
One comment:
> -(define* (handle-build-event db event)
> +(define* (handle-build-event event)
> "Handle EVENT, a build event sexp as produced by 'build-event-output-port',
> -updating DB accordingly."
> +updating the database accordingly."
[...]
> +(define %db-channel
> + (make-parameter #f))
> +
> +(define-syntax-rule (with-db-critical-section db exp ...)
> + "Evaluate EXP... in the critical section corresponding to %DB-CHANNEL.
> +DB is bound to the argument of that critical section: the database
> +connection."
> + (call-with-critical-section (%db-channel)
> + (lambda (db) exp ...)))
> +
I think I find it nicer to keep the ‘db’ parameter everywhere (except
that it’s now a channel instead of an actual database) rather than using
this global variable.
WDYT?
Really happy we have a solution to this problem!
Thank you,
Ludo’.
next prev parent reply other threads:[~2018-08-19 14:07 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-21 9:57 bug#32234: Cuirass: The SQLite built in busy handler might block the Fibers scheduler Clément Lassieur
2018-07-23 9:18 ` Ludovic Courtès
2018-07-23 13:42 ` Clément Lassieur
2018-07-23 14:57 ` Ludovic Courtès
2018-07-23 16:18 ` Clément Lassieur
2018-07-23 20:11 ` Ludovic Courtès
2018-08-06 19:27 ` bug#32234: [PATCH 1/2] utils: Avoid deadlock when WITH-CRITICAL-SECTION calls are nested Clément Lassieur
2018-08-06 19:27 ` bug#32234: [PATCH 2/2] database: Serialize all database accesses in a thread Clément Lassieur
2018-08-06 19:35 ` Clément Lassieur
2018-08-19 14:06 ` Ludovic Courtès [this message]
2018-08-26 14:07 ` Clément Lassieur
2018-08-26 21:16 ` Ludovic Courtès
2018-08-27 13:47 ` Clément Lassieur
2018-08-27 12:41 ` Danny Milosavljevic
2018-08-27 13:18 ` Clément Lassieur
2018-08-27 14:23 ` Danny Milosavljevic
2018-08-27 15:05 ` Clément Lassieur
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87zhxih2no.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=32234@debbugs.gnu.org \
--cc=clement@lassieur.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).