From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#32190: Cuirass doesn't check if two subsequent jobs yield the same derivation Date: Tue, 24 Jul 2018 12:05:35 +0200 Message-ID: <877ell54pc.fsf@gnu.org> References: <87k1ptirr0.fsf@lassieur.org> <87efg1ijdp.fsf@lassieur.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:49116) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhuCk-0006oO-6v for bug-guix@gnu.org; Tue, 24 Jul 2018 06:06:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhuCg-0000V0-77 for bug-guix@gnu.org; Tue, 24 Jul 2018 06:06:06 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:49577) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhuCg-0000Uw-1k for bug-guix@gnu.org; Tue, 24 Jul 2018 06:06:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1fhuCf-0003EL-QR for bug-guix@gnu.org; Tue, 24 Jul 2018 06:06:01 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87efg1ijdp.fsf@lassieur.org> ("=?UTF-8?Q?Cl=C3=A9ment?= Lassieur"'s message of "Wed, 18 Jul 2018 00:32:02 +0200") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: =?UTF-8?Q?Cl=C3=A9ment?= Lassieur Cc: 32190@debbugs.gnu.org Hi Cl=C3=A9ment, Cl=C3=A9ment Lassieur skribis: > Consider the following table: > > CREATE TABLE Derivations ( > derivation TEXT NOT NULL, > evaluation INTEGER NOT NULL, > job_name TEXT NOT NULL, > system TEXT NOT NULL, > nix_name TEXT NOT NULL, > PRIMARY KEY (derivation, evaluation), > FOREIGN KEY (evaluation) REFERENCES Evaluations (id) > ); > > > And the following code: > > (define (db-add-derivation db job) > "Store a derivation result in database DB and return its ID." > (catch 'sqlite-error > (lambda () > (sqlite-exec db "\ > INSERT INTO Derivations (derivation, job_name, system, nix_name, evaluati= on)\ > VALUES (" > (assq-ref job #:derivation) ", " > (assq-ref job #:job-name) ", " > (assq-ref job #:system) ", " > (assq-ref job #:nix-name) ", " > (assq-ref job #:eval-id) ");") > (last-insert-rowid db)) > (lambda (key who code message . rest) > ;; If we get a unique-constraint-failed error, that means we have > ;; already inserted the same (derivation,eval-id) tuple. That happ= ens > ;; when several jobs produce the same derivation, and we can ignore= it. > (if (=3D code SQLITE_CONSTRAINT_PRIMARYKEY) > (sqlite-exec db "SELECT * FROM Derivations WHERE derivation=3D" > (assq-ref job #:derivation) ";") > (apply throw key who code rest))))) > > I think the above constraint can't happen because by definition a new > job (for the same job_name) is produced at each evaluation. So eval-id > will be incremented every time. I added it at the time because it did happen. In a given eval, there can be two jobs producing the same derivation (for instance a job called =E2=80=9Cgcc=E2=80=9D produces xyz-gcc-5.5.0.drv, and a job called =E2=80= =9Cgcc-5.5.0=E2=80=9D produces the very same xyz-gcc-5.5.0.drv.) > Also, the docs (and a comment in schema.sql) says: > > Builds are not in a one to one relationship with derivations in > order to keep track of non deterministic compilations. > > But I think it doesn't make sense, because Guix won't try to build twice > the same thing unless '--check' is used (which obviously isn't the > case). The rationale (that was back in Mathieu=E2=80=99s GSoC) was that sometimes,= you can have several builds logs for one derivation. In Hydra this happens if a build fails for some non-deterministic reason and then you click on =E2=80=9CRestart=E2=80=9D in the hope that it=E2=80=99ll succeed this time.= ;-) In this situation Hydra keeps both build logs IIRC. Anyway, I lean towards keeping only one build log, at least for now, which is what guix-daemon does. > So not only we have a huge Derivations table full of identical items, > but we also ask Guix to build them and we store the results in the > Builds table... > > Maybe the solution is to replace the (derivation, evaluation) primary > key with (derivation), and only build the newly added derivations. > WDYT? I agree, we don=E2=80=99t need all these identical items, it makes no sense. You can go ahead and clean that up! ;-) Thank you, Ludo=E2=80=99.