From: ludo@gnu.org (Ludovic Courtès)
To: "Clément Lassieur" <clement@lassieur.org>
Cc: 32190@debbugs.gnu.org
Subject: bug#32190: Cuirass doesn't check if two subsequent jobs yield the same derivation
Date: Tue, 24 Jul 2018 12:05:35 +0200 [thread overview]
Message-ID: <877ell54pc.fsf@gnu.org> (raw)
In-Reply-To: <87efg1ijdp.fsf@lassieur.org> ("Clément Lassieur"'s message of "Wed, 18 Jul 2018 00:32:02 +0200")
Hi Clément,
Clément Lassieur <clement@lassieur.org> skribis:
> Consider the following table:
>
> CREATE TABLE Derivations (
> derivation TEXT NOT NULL,
> evaluation INTEGER NOT NULL,
> job_name TEXT NOT NULL,
> system TEXT NOT NULL,
> nix_name TEXT NOT NULL,
> PRIMARY KEY (derivation, evaluation),
> FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
> );
>
>
> And the following code:
>
> (define (db-add-derivation db job)
> "Store a derivation result in database DB and return its ID."
> (catch 'sqlite-error
> (lambda ()
> (sqlite-exec db "\
> INSERT INTO Derivations (derivation, job_name, system, nix_name, evaluation)\
> VALUES ("
> (assq-ref job #:derivation) ", "
> (assq-ref job #:job-name) ", "
> (assq-ref job #:system) ", "
> (assq-ref job #:nix-name) ", "
> (assq-ref job #:eval-id) ");")
> (last-insert-rowid db))
> (lambda (key who code message . rest)
> ;; If we get a unique-constraint-failed error, that means we have
> ;; already inserted the same (derivation,eval-id) tuple. That happens
> ;; when several jobs produce the same derivation, and we can ignore it.
> (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
> (sqlite-exec db "SELECT * FROM Derivations WHERE derivation="
> (assq-ref job #:derivation) ";")
> (apply throw key who code rest)))))
>
> I think the above constraint can't happen because by definition a new
> job (for the same job_name) is produced at each evaluation. So eval-id
> will be incremented every time.
I added it at the time because it did happen. In a given eval, there
can be two jobs producing the same derivation (for instance a job called
“gcc” produces xyz-gcc-5.5.0.drv, and a job called “gcc-5.5.0” produces
the very same xyz-gcc-5.5.0.drv.)
> Also, the docs (and a comment in schema.sql) says:
>
> Builds are not in a one to one relationship with derivations in
> order to keep track of non deterministic compilations.
>
> But I think it doesn't make sense, because Guix won't try to build twice
> the same thing unless '--check' is used (which obviously isn't the
> case).
The rationale (that was back in Mathieu’s GSoC) was that sometimes, you
can have several builds logs for one derivation. In Hydra this happens
if a build fails for some non-deterministic reason and then you click on
“Restart” in the hope that it’ll succeed this time. ;-) In this
situation Hydra keeps both build logs IIRC.
Anyway, I lean towards keeping only one build log, at least for now,
which is what guix-daemon does.
> So not only we have a huge Derivations table full of identical items,
> but we also ask Guix to build them and we store the results in the
> Builds table...
>
> Maybe the solution is to replace the (derivation, evaluation) primary
> key with (derivation), and only build the newly added derivations.
> WDYT?
I agree, we don’t need all these identical items, it makes no sense.
You can go ahead and clean that up! ;-)
Thank you,
Ludo’.
next prev parent reply other threads:[~2018-07-24 10:06 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-05-03 15:44 GSoC: Adding a web interface similar to the Hydra web interface Tatiana Sholokhova
2018-05-04 2:01 ` Maxim Cournoyer
2018-05-04 12:55 ` Ludovic Courtès
2018-05-05 10:50 ` Ricardo Wurmus
2018-05-08 7:26 ` Danny Milosavljevic
2018-05-09 9:56 ` Ricardo Wurmus
2018-05-09 17:21 ` Ricardo Wurmus
2018-05-13 18:45 ` Tatiana Sholokhova
2018-05-13 19:30 ` Gábor Boskovits
2018-05-13 19:33 ` Tonton
2018-05-13 19:54 ` Danny Milosavljevic
2018-05-14 3:34 ` Chris Marusich
2018-05-14 4:20 ` Ricardo Wurmus
2018-05-17 22:31 ` Tatiana Sholokhova
2018-05-18 20:35 ` Ricardo Wurmus
2018-05-21 21:52 ` Tatiana Sholokhova
2018-05-22 5:33 ` Ricardo Wurmus
2018-05-23 21:06 ` Tatiana Sholokhova
2018-05-24 6:03 ` Ricardo Wurmus
[not found] ` <CAMSS15DThnLO+YEVaBmJ9ozMeu4mO1rHAdXHgZ8K+Csu40pORQ@mail.gmail.com>
2018-05-28 10:39 ` Ricardo Wurmus
2018-06-02 15:03 ` Ricardo Wurmus
2018-06-03 15:50 ` Tatiana Sholokhova
2018-06-03 19:40 ` Ricardo Wurmus
2018-06-04 22:14 ` Tatiana Sholokhova
2018-06-05 20:40 ` Ricardo Wurmus
2018-06-06 18:02 ` Danny Milosavljevic
2018-06-10 14:36 ` Tatiana Sholokhova
2018-06-11 10:19 ` Ricardo Wurmus
2018-06-11 11:23 ` Ludovic Courtès
2018-06-12 16:35 ` Danny Milosavljevic
2018-06-12 21:52 ` Ricardo Wurmus
2018-06-12 22:43 ` Tatiana Sholokhova
2018-06-13 6:39 ` Gábor Boskovits
2018-06-13 8:27 ` Danny Milosavljevic
2018-06-13 13:58 ` Joshua Branson
2018-06-13 14:22 ` Gábor Boskovits
2018-06-13 15:07 ` Joshua Branson
2018-06-25 10:46 ` Gábor Boskovits
2018-06-25 12:12 ` Tatiana Sholokhova
2018-06-27 19:56 ` Ludovic Courtès
2018-07-04 20:54 ` Tatiana Sholokhova
2018-07-04 21:47 ` Jelle Licht
2018-07-05 8:27 ` Danny Milosavljevic
2018-07-06 9:58 ` Gábor Boskovits
2018-07-08 19:48 ` Tatiana Sholokhova
2018-07-08 21:09 ` Danny Milosavljevic
2018-07-29 12:01 ` Clément Lassieur
2018-07-29 13:25 ` Gábor Boskovits
2018-07-29 14:41 ` Clément Lassieur
2018-07-08 21:19 ` Gábor Boskovits
2018-07-18 10:37 ` Clément Lassieur
2018-07-19 20:10 ` Tatiana Sholokhova
2018-07-19 21:47 ` Amirouche Boubekki
2018-07-18 10:19 ` Clément Lassieur
2018-07-17 19:31 ` Clément Lassieur
2018-07-17 22:32 ` bug#32190: Cuirass doesn't check if two subsequent jobs yield the same derivation Clément Lassieur
2018-07-24 10:05 ` Ludovic Courtès [this message]
2018-08-04 16:03 ` bug#32190: [PATCH] database: Merge Derivations into Builds table Clément Lassieur
2018-08-04 16:09 ` Clément Lassieur
2018-08-08 12:13 ` Clément Lassieur
2018-08-14 16:57 ` Clément Lassieur
2018-08-14 19:04 ` Ricardo Wurmus
2018-08-15 18:57 ` Clément Lassieur
2018-08-16 21:00 ` Clément Lassieur
2018-05-29 16:07 ` GSoC: Adding a web interface similar to the Hydra web interface Ludovic Courtès
2018-05-29 16:17 ` Gábor Boskovits
2018-07-18 9:34 ` Clément Lassieur
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=877ell54pc.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=32190@debbugs.gnu.org \
--cc=clement@lassieur.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.