From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#32234: [PATCH 2/2] database: Serialize all database accesses in a thread. Date: Sun, 19 Aug 2018 16:06:03 +0200 Message-ID: <87zhxih2no.fsf@gnu.org> References: <20180806192736.1747-1-clement@lassieur.org> <20180806192736.1747-2-clement@lassieur.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42859) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1frOMB-0006dR-Dg for bug-guix@gnu.org; Sun, 19 Aug 2018 10:07:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1frOMA-0004xJ-G0 for bug-guix@gnu.org; Sun, 19 Aug 2018 10:07:03 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:49700) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1frOMA-0004xD-B8 for bug-guix@gnu.org; Sun, 19 Aug 2018 10:07:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1frOMA-00089i-4t for bug-guix@gnu.org; Sun, 19 Aug 2018 10:07:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <20180806192736.1747-2-clement@lassieur.org> ("=?UTF-8?Q?Cl=C3=A9ment?= Lassieur"'s message of "Mon, 6 Aug 2018 21:27:36 +0200") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: =?UTF-8?Q?Cl=C3=A9ment?= Lassieur Cc: 32234@debbugs.gnu.org Hi Cl=C3=A9ment! Cl=C3=A9ment Lassieur skribis: > Fixes . > > * bin/cuirass.in (main): Keep only one WITH-DATABASE call around all fibe= rs. > Remove all DB arguments. > * src/cuirass/base.scm (evaluate, update-build-statuses!, spawn-builds, > handle-build-event, build-packages): Remove all DB arguments. > (clear-build-queue, cancel-old-builds): Wrap in WITH-DB-CRITICAL-SECTION, > remove all DB arguments. > (restart-builds): Remove the NON-BLOCKING call, remove all DB arguments. > (process-specs): Remove all DB arguments, remove the WITH-DATABASE call. > * src/cuirass/database.scm (%db-channel): New parameter. > (with-db-critical-section): New macro. > (db-add-input, db-add-specification, db-get-inputs, db-get-specifications, > db-add-evaluation, db-add-build, db-update-build-status!, db-get-outputs, > db-get-builds, db-get-build, db-get-pending-derivations, db-get-stamp, > db-add-stamp, db-get-evaluations, db-get-evaluations-build-summary, > db-get-evaluations-id-min, db-get-evaluations-id-max, db-get-builds-min, > db-get-builds-max): Wrap in WITH-DB-CRITICAL-SECTION, remove all DB argum= ents. > (with-database): Wrap BODY in PARAMETERIZE form that sets %DB-CHANNEL to = the > channel returned by MAKE-CRITICAL-SECTION. > * src/cuirass/http.scm (handle-build-request, handle-builds-request): Rem= ove > all DB arguments. > (url-handler): Remove all DB arguments, remove the DB-CHANNEL state, remo= ve > the WITH-CRITICAL-SECTION calls. > (run-cuirass-server): Remove the DB arguments, remove the > MAKE-CRITICAL-SECTION call. > * src/cuirass/utils.scm (make-critical-section): Replace SPAWN-FIBER with > CALL-WITH-NEW-THREAD. Wrap body in PARAMETERIZE form that clears > CURRENT-FIBER. > * tests/database.scm (with-temporary-database, db-add-specification, > db-add-build, db-update-build-status!, db-get-builds, > db-get-pending-derivations): Remove the DB arguments. > (db-init): Set the %DB-CHANNEL parameter to the channel returned by > MAKE-CRITICAL-SECTION, and return #t. > (database): Set %DB-CHANNEL to #f during cleanup. > * tests/http.scm (db-init): Set the %DB-CHANNEL parameter to the channel > returned by MAKE-CRITICAL-SECTION, and return #t. > (cuirass-run, fill-db): Remove the DB arguments. > (http): Set %DB-CHANNEL to #f during cleanup. Excellent, thanks for working on this! This looks great to me, and I think the pros outweigh the cons. Did you check on a big database how well it performs? One comment: > -(define* (handle-build-event db event) > +(define* (handle-build-event event) > "Handle EVENT, a build event sexp as produced by 'build-event-output-p= ort', > -updating DB accordingly." > +updating the database accordingly." [...] > +(define %db-channel > + (make-parameter #f)) > + > +(define-syntax-rule (with-db-critical-section db exp ...) > + "Evaluate EXP... in the critical section corresponding to %DB-CHANNEL. > +DB is bound to the argument of that critical section: the database > +connection." > + (call-with-critical-section (%db-channel) > + (lambda (db) exp ...))) > + I think I find it nicer to keep the =E2=80=98db=E2=80=99 parameter everywhe= re (except that it=E2=80=99s now a channel instead of an actual database) rather than = using this global variable. WDYT? Really happy we have a solution to this problem! Thank you, Ludo=E2=80=99.