From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id aIZ7NgFsiGAu4AAAgWs5BA (envelope-from ) for ; Tue, 27 Apr 2021 21:54:41 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id MKwtMgFsiGDkdgAAB5/wlQ (envelope-from ) for ; Tue, 27 Apr 2021 19:54:41 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 46FC218E27 for ; Tue, 27 Apr 2021 21:54:41 +0200 (CEST) Received: from localhost ([::1]:53576 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lbTn5-0006bu-Mc for larch@yhetil.org; Tue, 27 Apr 2021 15:54:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50024) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lbTmq-0006bk-1V for guix-devel@gnu.org; Tue, 27 Apr 2021 15:54:25 -0400 Received: from mout02.posteo.de ([185.67.36.66]:40543) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lbTmj-0007AU-DF for guix-devel@gnu.org; Tue, 27 Apr 2021 15:54:23 -0400 Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id 63E082400FE for ; Tue, 27 Apr 2021 21:54:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1619553251; bh=e6qgazFQe+6vcwFRD0p6jpcrbPU7OvEf8y4CuGPLEc4=; h=Date:From:To:Cc:Subject:From; b=BMR0ExtC48G+YfSdcHIbpJOq0509RHOcLUF4pRGIjHCjVITxOos86UbJmp1VE1GKU UAGPfHky5xz0ZFAwF3e10OBFdbwk/1cqsE/kX/QqGkFniWZW8/UL1KDjsgQoiB+AXk 65QnKhdqm4XD3IFilkt5frvvZqw90tb+BbWewvTIJbbLRsivKedOjnG4RJvHNOqegf vjeQ+48oIKPvv3M8hHuHI8TZUFH536vFIuo3RbducuyLbHbQw6/Wtja3nCUhfuNxf7 Z2Lmo8sEw9H+P2vZ0KvOm2H2F5aeIcg5wIqbwFsJxxmMqOc3JVQsqNt507vxwt7Xod XCWaONk6PeBsg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4FVCBm3Q4pz6tmL; Tue, 27 Apr 2021 21:54:08 +0200 (CEST) Date: Tue, 27 Apr 2021 19:53:55 +0000 From: Luciana Lima Brito To: Christopher Baines Subject: Re: Outreachy - Guix Data Service: questions about improving the data for derivation comparisons. Message-ID: <20210427165355.0abb1f77@lubrito> In-Reply-To: <877dknfu2i.fsf@cbaines.net> References: <20210421152914.461bbdbb@lubrito> <87bla6iwli.fsf@cbaines.net> <20210422170040.37941083@lubrito> <875z0ehyl3.fsf@cbaines.net> <20210422180208.28473e41@lubrito> <87zgxqggw9.fsf@cbaines.net> <20210423181551.01c42455@lubrito> <87bla4hdto.fsf@cbaines.net> <20210425171507.6a259017@lubrito> <87o8e1foly.fsf@cbaines.net> <20210426161103.0647ee3f@lubrito> <87im48g2s1.fsf@cbaines.net> <20210427101001.0242f90e@lubrito> <20210427153335.1e23f35e@lubrito> <877dknfu2i.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/DwggP_75lcHD_52njEZwd_E" Received-SPF: pass client-ip=185.67.36.66; envelope-from=lubrito@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619553281; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=1spx/XQkChJ5eqS08L90WDEUfNpeWzBAuryLERI9hYE=; b=ReDbZpQDfVCDMtVZGyCKYtmuCt94VJJXtQlheQy8vTqzIVZUVm8ZWOr/hTFWKCG6zDKuvE CooXMpVmKKAQNAWoeQr98K/kVVHkKcegoVUkW2eb65iPrIK57j9q3QNYlJDYIOTVvNksVf o44dqB1tLPegU13dyohVvte3F7rrGrp1H4JxuvUXZqH9FvN7P1rlhqFd5tSQ9bVTaF+MZo TjTkLvJZ7GA3Dy85JkEghPFw5keYElHFiTNcvwZoM2nLL8PbeYPi4qLnN8d1I5ANTc18RD K7dE3zBEBk79hUF/BbHRRt9QERZzSjK3GM4xET+fLYqRn97Ac1oOD6cJJOJTRw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619553281; a=rsa-sha256; cv=none; b=B+QufFlAAXCH1SmloYYjlXMD3bucc/Lh4hTJaL5sozZYaQ9V6rrxi8Wte2dAeR7pCgvv2K yh1qkabZ5QfR8Fv2HmDBBsa01dAPCG6vffNCeCzRdyaUztzLH3sLqc95VuZ/OMRctDZa3X kGuFo5UJFUuSF/3483XZdZKkLLvioNhx5NSdLqdMvYikDb6mNzfoEkQo/2EXKcxXDr/OeA QPJgqPI+wbYF604ZR9ezOzl2/qZbgUql3RBBqrN1w+4hhyUzyUw4nQJvFjBPujHwUxPwX+ JcworlR1NDKN6SLt6+gTK00Ci+nr8ww1MP3QgHFSWN1+9wT1rhmZQyLKgKzdTQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=BMR0ExtC; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.65 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=BMR0ExtC; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 46FC218E27 X-Spam-Score: -2.65 X-Migadu-Scanner: scn0.migadu.com X-TUID: o3cYSQlAgaIg --MP_/DwggP_75lcHD_52njEZwd_E Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline On Tue, 27 Apr 2021 19:42:13 +0100 Christopher Baines wrote: > I'd go with this approach, applying the comments I made about the > match-lambda bit above in the email I sent a few minutes ago. I applied all you instructed me. See if it looks better now. -- Best Regards, Luciana Lima Brito MSc. in Computer Science --MP_/DwggP_75lcHD_52njEZwd_E Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0001-Change-handling-of-queried-data-for-derivations-comp.patch >From 93ef30e06d7e10fd3a140c6f2a729d40540b05a8 Mon Sep 17 00:00:00 2001 From: Luciana Brito Date: Sun, 25 Apr 2021 15:17:33 -0300 Subject: [PATCH] Change handling of queried data for derivations comparison. comparison.scm: return query data for derivation comparison as an alist, instead of list. html.scm: Access derivation differences data using assq-ref. controller.scm: remove mapping for outputs/inputs/sources. utils.scm: add group-to-alist/vector function. --- guix-data-service/comparison.scm | 81 +++++++++--------- guix-data-service/model/utils.scm | 8 ++ guix-data-service/web/compare/controller.scm | 88 +++----------------- guix-data-service/web/compare/html.scm | 74 ++++++++-------- 4 files changed, 97 insertions(+), 154 deletions(-) diff --git a/guix-data-service/comparison.scm b/guix-data-service/comparison.scm index e5e1955..46f5377 100644 --- a/guix-data-service/comparison.scm +++ b/guix-data-service/comparison.scm @@ -74,19 +74,20 @@ 'value)) `((outputs - . ,(group-to-alist + . ,(group-to-alist/vector group-by-last-element (derivation-outputs-differences-data conn (first base-derivation) (first target-derivation)))) (inputs - . ,(group-to-alist + . ,(group-to-alist/vector group-by-last-element (derivation-inputs-differences-data conn (first base-derivation) (first target-derivation)))) + (sources - . ,(group-to-alist + . ,(group-to-alist/vector group-by-last-element (derivation-sources-differences-data conn (first base-derivation) @@ -107,9 +108,9 @@ (target . ,target-builder)))) (arguments . ,(if (eq? base-args target-args) - `((common . ,base-args)) - `((base . ,base-args) - (target . ,target-args)))) + `((common . ,(list->vector base-args))) + `((base . ,(list->vector base-args)) + (target . ,(list->vector target-args))))) (environment-variables . ,(map (lambda (key) (let ((base-value (fetch-value base-env-vars key)) @@ -158,19 +159,23 @@ GROUP BY 1, 2, 3, 4, 5")) (let ((parsed-derivation-ids (map string->number (parse-postgresql-array-string derivation_ids)))) - (list output-name - path - hash-algorithm - hash - recursive - (append (if (memq base-derivation-id - parsed-derivation-ids) - '(base) - '()) - (if (memq target-derivation-id - parsed-derivation-ids) - '(target) - '())))))) + `((output-name . ,output-name) + (path . ,path) + ,@(if (string? hash-algorithm) + `((hash-algorithm . ,hash-algorithm)) + `((hash-algorithm . null))) + ,@(if (string? hash) + `((hash . ,hash)) + `((hash . null))) + (recursive . ,(string=? recursive "t")) + ,(append (if (memq base-derivation-id + parsed-derivation-ids) + '(base) + '()) + (if (memq target-derivation-id + parsed-derivation-ids) + '(target) + '())))))) (exec-query conn query))) (define (derivation-inputs-differences-data conn @@ -202,16 +207,16 @@ INNER JOIN derivations ON derivation_outputs.derivation_id = derivations.id (let ((parsed-derivation-ids (map string->number (parse-postgresql-array-string derivation_ids)))) - (list derivation_file_name - derivation_output_name - (append (if (memq base-derivation-id - parsed-derivation-ids) - '(base) - '()) - (if (memq target-derivation-id - parsed-derivation-ids) - '(target) - '())))))) + `((derivation_file_name . ,derivation_file_name) + (derivation_output_name . ,derivation_output_name) + ,(append (if (memq base-derivation-id + parsed-derivation-ids) + '(base) + '()) + (if (memq target-derivation-id + parsed-derivation-ids) + '(target) + '())))))) (exec-query conn query))) (define (derivation-sources-differences-data conn @@ -235,15 +240,15 @@ GROUP BY derivation_source_files.store_path")) (let ((parsed-derivation-ids (map string->number (parse-postgresql-array-string derivation_ids)))) - (list store_path - (append (if (memq base-derivation-id - parsed-derivation-ids) - '(base) - '()) - (if (memq target-derivation-id - parsed-derivation-ids) - '(target) - '())))))) + `((store_path . ,store_path) + ,(append (if (memq base-derivation-id + parsed-derivation-ids) + '(base) + '()) + (if (memq target-derivation-id + parsed-derivation-ids) + '(target) + '())))))) (exec-query conn query))) (define* (package-derivation-differences-data conn diff --git a/guix-data-service/model/utils.scm b/guix-data-service/model/utils.scm index 13947bd..b11cee5 100644 --- a/guix-data-service/model/utils.scm +++ b/guix-data-service/model/utils.scm @@ -33,6 +33,7 @@ deduplicate-strings group-list-by-first-n-fields group-to-alist + group-to-alist/vector insert-missing-data-and-return-all-ids)) (define NULL '()) @@ -114,6 +115,13 @@ '() lst)) +(define (group-to-alist/vector process lst) + (map + (match-lambda + ((label . items) + (cons label (list->vector items)))) + (group-to-alist process lst))) + (define (table-schema conn table-name) (let ((results (exec-query diff --git a/guix-data-service/web/compare/controller.scm b/guix-data-service/web/compare/controller.scm index 895bb40..8445185 100644 --- a/guix-data-service/web/compare/controller.scm +++ b/guix-data-service/web/compare/controller.scm @@ -588,82 +588,18 @@ '(application/json text/html) mime-types) ((application/json) - (let ((outputs - (map - (lambda (label items) - (cons label - (list->vector - (map - (match-lambda - ((name path hash-alg hash recursive) - `((name . ,name) - (path . ,path) - ,@(if (string? hash-alg) - `((hash-algorithm . ,hash-alg)) - '()) - ,@(if (string? hash) - `((hash . ,hash)) - '()) - (recursive . ,(string=? recursive "t"))))) - (or items '()))))) - '(base target common) - (let ((output-groups (assq-ref data 'outputs))) - (list (assq-ref output-groups 'base) - (assq-ref output-groups 'target) - (assq-ref output-groups 'common))))) - - (inputs - (map - (lambda (label items) - (cons label - (list->vector - (map - (match-lambda - ((derivation output) - `((derivation . ,derivation) - (output . ,output)))) - (or items '()))))) - '(base target common) - (let ((input-groups (assq-ref data 'inputs))) - (list (assq-ref input-groups 'base) - (assq-ref input-groups 'target) - (assq-ref input-groups 'common))))) - - (sources - (map - (lambda (label items) - (cons label - (list->vector - (map - (match-lambda - ((derivation) - `((derivation . ,derivation)))) - (or items '()))))) - '(base target common) - (let ((source-groups (assq-ref data 'sources))) - (list (assq-ref source-groups 'base) - (assq-ref source-groups 'target) - (assq-ref source-groups 'common))))) - - (arguments - (map - (match-lambda - ((label args ...) - `(,label . ,(list->vector args)))) - (assq-ref data 'arguments)))) - - (render-json - `((base . ((derivation . ,base-derivation))) - (target . ((derivation . ,target-derivation))) - (outputs . ,outputs) - (inputs . ,inputs) - (sources . ,sources) - (system . ,(assq-ref data 'system)) - (builder . ,(assq-ref data 'builder)) - (arguments . ,arguments) - (environment-variables . ,(assq-ref - data 'environment-variables))) - #:extra-headers http-headers-for-unchanging-content))) + (render-json + `((base . ((derivation . ,base-derivation))) + (target . ((derivation . ,target-derivation))) + (outputs . ,(assq-ref data 'outputs)) + (inputs . ,(assq-ref data 'inputs)) + (sources . ,(assq-ref data 'sources)) + (system . ,(assq-ref data 'system)) + (builder . ,(assq-ref data 'builder)) + (arguments . ,(assq-ref data 'arguments)) + (environment-variables . ,(assq-ref + data 'environment-variables))) + #:extra-headers http-headers-for-unchanging-content)) (else (render-html #:sxml (compare/derivation diff --git a/guix-data-service/web/compare/html.scm b/guix-data-service/web/compare/html.scm index 5b5fe0a..e1ff15c 100644 --- a/guix-data-service/web/compare/html.scm +++ b/guix-data-service/web/compare/html.scm @@ -487,27 +487,23 @@ (th "Hash") (th "Recursive"))) (tbody - ,@(let ((base-outputs (assq-ref outputs 'base)) - (target-outputs (assq-ref outputs 'target)) - (common-outputs (assq-ref outputs 'common))) - (append-map - (lambda (label items) - (map - (match-lambda - ((name path hash-algorithm hash recursive) - `(tr - (td ,label) - (td ,name) - (td (a (@ (href ,path)) - ,(display-store-item path))) - (td ,hash-algorithm) - (td ,hash) - (td ,recursive)))) - (or items '()))) - (list base target "Common") - (list (assq-ref outputs 'base) - (assq-ref outputs 'target) - (assq-ref outputs 'common)))))))) + ,@(append-map + (lambda (label items) + (map + (lambda (alist) + `(tr + (td ,label) + (td ,(assq-ref alist 'output-name)) + (td (a (@ (href ,(assq-ref alist 'path))) + ,(display-store-item (assq-ref alist 'path)))) + (td ,(assq-ref alist 'hash-algorithm)) + (td ,(assq-ref alist 'hash)) + (td ,(assq-ref alist 'recursive)))) + (or (and=> items vector->list) '()))) + (list base target "Common") + (list (assq-ref outputs 'base) + (assq-ref outputs 'target) + (assq-ref outputs 'common))))))) (h2 "Inputs") ,@(let ((inputs (assq-ref data 'inputs))) `((table @@ -521,14 +517,13 @@ ,@(append-map (lambda (label items) (map - (match-lambda - ((derivation outputs) - `(tr - (td ,label) - (td (a (@ (href ,derivation)) - ,(display-store-item derivation))) - (td ,outputs)))) - (or items '()))) + (lambda (alist) + `(tr + (td ,label) + (td (a (@ (href ,(assq-ref alist 'derivation_file_name))) + ,(display-store-item (assq-ref alist 'derivation_file_name)))) + (td ,(assq-ref alist 'derivation_output_name)))) + (or (and=> items vector->list) '()))) (list base target) (list (assq-ref inputs 'base) (assq-ref inputs 'target))))))) @@ -545,13 +540,12 @@ ,@(append-map (lambda (label items) (map - (match-lambda - ((file) - `(tr - (td ,label) - (td (a (@ (href ,file)) - ,(display-store-item file)))))) - (or items '()))) + (lambda (alist) + `(tr + (td ,label) + (td (a (@ (href ,(assq-ref alist 'store_path))) + ,(display-store-item (assq-ref alist 'store_path)))))) + (or (and=> items vector->list) '()))) (list base target "Common") (list (assq-ref sources 'base) (assq-ref sources 'target) @@ -615,8 +609,8 @@ (td (ol ,@(map (lambda (arg) `(li ,(display-possible-store-item arg))) - (or common-args - base-args))))) + (or (and=> common-args vector->list) + (vector->list base-args)))))) (tr (td ,target) (td ,(display-possible-store-item @@ -625,8 +619,8 @@ (td (ol ,@(map (lambda (arg) `(li ,(display-possible-store-item arg))) - (or common-args - target-args)))))))))))) + (or (and=> common-args vector->list) + (vector->list target-args))))))))))))) (h2 "Environment variables") ,(let ((environment-variables (assq-ref data 'environment-variables))) `(table -- 2.30.2 --MP_/DwggP_75lcHD_52njEZwd_E--