From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id MAIEBjNo5WAClgAAgWs5BA (envelope-from ) for ; Wed, 07 Jul 2021 10:39:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id MyOLATNo5WAxOwAAbx9fmQ (envelope-from ) for ; Wed, 07 Jul 2021 08:39:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 00C491BC36 for ; Wed, 7 Jul 2021 10:39:14 +0200 (CEST) Received: from localhost ([::1]:60854 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m135M-0006gM-Tq for larch@yhetil.org; Wed, 07 Jul 2021 04:39:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57082) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m135B-0006gE-44 for guix-devel@gnu.org; Wed, 07 Jul 2021 04:39:01 -0400 Received: from imta-37.everyone.net ([216.200.145.37]:56982 helo=imta-38.everyone.net) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m1358-00064C-IR for guix-devel@gnu.org; Wed, 07 Jul 2021 04:39:00 -0400 Received: from pps.filterd (omta004.sj2.proofpoint.com [127.0.0.1]) by imta-38.everyone.net (8.16.0.43/8.16.0.43) with SMTP id 1678cuA4022333; Wed, 7 Jul 2021 01:38:56 -0700 X-Eon-Originating-Account: 0NK6zxqkqWkqCPoJatmrj2Pna0Np82Pc-kRWKGm2mMQ X-Eon-Dm: m0116953.ppops.net Received: by m0116953.mta.everyone.net (EON-AUTHRELAY2 - 5a81cfec) id m0116953.60d3a861.1039cf; Wed, 7 Jul 2021 01:38:52 -0700 X-Eon-Sig: AQMHrIJg5WgcbkhsBgIAAAAC,9aa23834857d70bfbf550d61fd29e94a X-Eip: mhtm6evntkJiAwdBpH02s7Ij_kZDH5AfmGQbJ2XMLbQ Date: Wed, 7 Jul 2021 10:38:44 +0200 From: Bengt Richter To: Nathan Benedetto =?utf-8?Q?Proen=C3=A7a?= Subject: Re: Use of %texlive-revision and %texlive-tag in tex.scm Message-ID: <20210707083844.GA2258@LionPure> References: <87o8bgoo1p.fsf@archlinux.i-did-not-set--mail-host-address--so-tickle-me> <20210706094306.GA3432@LionPure> <87o8bfi9es.fsf@archlinux.i-did-not-set--mail-host-address--so-tickle-me> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87o8bfi9es.fsf@archlinux.i-did-not-set--mail-host-address--so-tickle-me> User-Agent: Mutt/1.10.1 (2018-07-13) X-Proofpoint-GUID: ons1hW89h7cEj08h2ZDblV82fCE141oU X-Proofpoint-ORIG-GUID: ons1hW89h7cEj08h2ZDblV82fCE141oU X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-07_05:2021-07-06, 2021-07-07 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 impostorscore=0 bulkscore=0 adultscore=0 lowpriorityscore=0 clxscore=1034 suspectscore=0 mlxlogscore=982 malwarescore=0 priorityscore=1501 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107070051 Received-SPF: pass client-ip=216.200.145.37; envelope-from=bokr@oz.net; helo=imta-38.everyone.net X-Spam_score_int: -15 X-Spam_score: -1.6 X-Spam_bar: - X-Spam_report: (-1.6 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bengt Richter Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1625647154; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=3sBo88Hsx/mYDmHTVicBB2HhtdsQO87lutrafTK6mCI=; b=CacjZQh+EwCrPQOoGfEYMsoD9W2oJfgLpEUlfkdK2U2biytu1eoY5cBC+oTZ7zyTzYwrlR VAVwxzcAkQBCUyKq3A70rHUTjIZMippRrCOXujQFdqvi0cv1hu1M0ub2H58fJo5f9Q/oKD wYBspxgkhf0BostGFsDcIYbr/rgbBBPX8WYRYosb3+d332dI0zs6f3+szp7lUyCqmOjlkG e0HoR830ziQOHAMAJaV2N5mWWWP3zxxQ9UckEge807Et+s5wSpgjHGsPUgEaaX8MhHdb/+ XRytSh73YLeC7vQBb6+IvZXRFn64tZ3+xwbhb33EJWwb3Tl6IndbqBmx4olPiA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1625647154; a=rsa-sha256; cv=none; b=Cksp2EKqhsWhi6D+8rGNRTIzMj4b5qTbkzxR7UjO0jAWdwBQ++90dlkVOLqXHDo0iXn/t1 3iz4ZVyOvPk0/ZbasArJdK4qZVD3WB8czHsja33zv58rQq+POaQjSuijHNT3olhiuM7HQo ZXNdvxzr+A/Ut3rsQ16hmv2JFJIJ3jMyfH6+szceMEutkmFeRLWaiSs/lqD9f+IR2CZSaO LPFBAFVinO9eQe0tF48CqsaMSSsbifF5e6yMK0R5sl9mt9TMTDUwmpEnlhlaCaQAaIUSjT 5HiVZQHLuBc2ZBDPLm6ML9R4XVk/8keZTVvvLL9tc0HYBUaTirJdPNqOfbVYjg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -1.91 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 00C491BC36 X-Spam-Score: -1.91 X-Migadu-Scanner: scn0.migadu.com X-TUID: uqWQRFFH3aiL On +2021-07-06 15:28:43 -0300, Nathan Benedetto Proença wrote: > Bengt Richter writes: > > > Hi Nathan, > > Nice writeup! > > Thank you! > > > On +2021-07-05 11:03:46 -0300, Nathan Benedetto Proença wrote: > >> Hello! > >> > >> I am trying to upgrade the package texlive, first for me, but hopefully > >> for a patch, and I have a question regarding Guix policies. > >> > >> As you can see on > >> > >> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/texlive.scm?id=f7692617b624a01615cf412773c4dad9a2deeb68 > >> > >> the file guix/build-system/texlive.scm exposes two variables: > >> > >> (define %texlive-tag "texlive-2019.3") > >> (define %texlive-revision 51265) > >> > >> These variables are used throughout gnu/packages/tex.scm, as you can see > >> on > >> > >> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tex.scm?id=f7692617b624a01615cf412773c4dad9a2deeb68 > >> > >> An example is the following code: > >> > >> (define hyph-utf8-scripts > >> (origin > >> (method svn-fetch) > >> (uri (texlive-ref "generic" "hyph-utf8")) > >> (file-name (string-append "hyph-utf8-scripts-" > >> (number->string %texlive-revision) > >> "-checkout")) > >> (sha256 > >> (base32 > >> "0lk7shx768sxvgr85y8bnmmnj8x4bbkgpxrz3z8jp8avi33prw83")))) > >> > >> Grep tells me there are 290+ occurrences of `%texlive-revision`. > >> What is the purpose of these variables? > >> > >> You see, they give me the impression that Guix is really concerned about > >> upgrading *all* of texlive at once. > >> These variables tell me I should go to the file texlive.scm and bump the > >> tag and revision, and then handle all the broken hashes. > >> > >> Hence, it seems to me that any attempt to upgrade the texlive package > >> would have to be done in a separate branch, which would only be merged > >> into master when all the packages are upgraded. > >> > >> Is this the case? > >> And if so, why? > >> > >> I have the impression that if such "monolithic" upgrade is not a goal, > >> and "partial" our "per-package" upgrades are desirable, there may be > >> better solutions. > >> > >> For example, we could add keyword arguments to texlive-ref and > >> texlive-origin, so the code above becomes something like this > >> > >> (define hyph-utf8-scripts > >> (origin > >> (method svn-fetch) > >> (uri (texlive-ref "generic" "hyph-utf8" > >> #:texlive-tag "texlive-2019.3" > >> #:texlive-revision 51265)) > >> (file-name "hyph-utf8-scripts-51625-checkout") > >> (sha256 > >> (base32 > >> "0lk7shx768sxvgr85y8bnmmnj8x4bbkgpxrz3z8jp8avi33prw83")))) > >> > >> This would work right now, and we could eventually remove every use of > >> %texlive-revision and %texlive-tag, so they become implementation > >> details of the build-system texlive.scm; a fallback version. > >> And further down the road we may even decide to remove this fallback, > >> and make developers be explicit about their tags and revisions; this > >> could amount to a refactor which makes the keyword arguments into > >> required arguments, for example. > >> > >> I also like the second version of the code because the hash already > >> pinpoints the tag and revision: both texlive-ref and texlive-origin use > >> these variables to find the correct files to download. > >> This just makes this dependency explicit. > >> > >> In any case, as this may be a choice between shipping stable and > >> up-to-date packages, and as I am new to contributing to Guix, I found > >> fitting to ask. > >> > >> Thanks in advance! > >> Nathan > >> > > > > I am wondering about guaranteeing generic behaviour by > > guaranteeing program source and toolchain source hash > > equivalences vs ignoring sources and guaranteeing end > > results by testing results. > > It seems to me that you are talking about my email regarding using > hashing in Scheme refactoring, right? > This one: > > https://lists.gnu.org/archive/html/guix-devel/2021-07/msg00023.html > > I will assume this is the case, even though I we can actually simply > talk about what you wrote. > > First thoughts: I think that what we really want to guarantee is > "correctness", and that both hashes or tests are mere proxies for this. > > I see them as useful in distinct moments of package development. > > For example, lets say that I want to partially upgrade TeXLive (which I > am already convinced is not a good idea, TBH, but serves as an > example). > To be able to do so, I may think that I want to refactor some function > to pinpoint the version I will download. > I can then make a single commit which updates the signature and all the > call sites of the function. > I can then actually upgrade the package I care about in a second commit, > and do some "proper testing" in it, like producing some pdf with latex. > > My interest in checking hashes in the first commit is to get some > confidence that this step, which may change hundred of places, did not > introduce a new bug, so I can focus any debugging on the hopefully more > local change made in the second commit. > > > I.e., if you want to print the sum of x and y passed as > > strings to a program, output as a string to stdout, it > > doesn't matter (other than optimization and debuggability) > > what language the program was written in, so long as it was > > compiled into a form that execve and co can launch and the > > end result is the same. > > > > As part of testing, maybe strace could be used to generate > > some kind of canonical kernel transaction trace that could > > be used to compare behaviours for equivalency of executing > > different-language programs? > > > > This would be a radical change in the approach to > > reproducibility, maybe dynamically selecting from a > > whitelist of trusted/tested substitutable executables with > > hash names in /gnu but not necessarily (though not > > excluding) binaries produced with guix source guarantees. > > > > Seems like guix is turing-complete enough to provide this > > kind of substitutable foreign functions already, so might > > this be a way to avoid mass recompilations? > > > > Or is this already available, but not so much used? > > > > I am not sure where to contibute thoughts like these, where > > they would be of interest rather than distracting. (Pls > > excuse the noise, if that's what this is to you). > > I am not sure I follow, although I do not regard this as noise. > > You are saying that one could use strace as a check that some program > behaves the same as it did before, similar to the use that I suggested > for hashing, right? > > I do not understand how this would be used, and how this avoids mass > recompilations. > I mean, to have binaries to inspect with strace I must build the code > anyway, right? > Am I missing something? > I had in mind expanding on and polishing something like $ strace -s 72 ./bash-hello-strace |& grep ^write write(1, "Hello strace!\n", 14Hello strace! $ strace -s 72 ./guile-hello-strace.scm |& grep ^write write(1, "Hello strace!\n", 14Hello strace! Where $ cat -n ./guile-hello-strace.scm |gxsnip --8<---------------cut here---------------start------------->8--- 1 #! /usr/bin/guile -s 2 !# 3 (display "Hello strace!\n") --8<---------------cut here---------------end--------------->8--- and $ cat -n ./bash-hello-strace |gxsnip --8<---------------cut here---------------start------------->8--- 1 #! /usr/bin/bash 2 echo 'Hello strace!' --8<---------------cut here---------------end--------------->8--- I had in mind expanding on and polishing something like $ strace -s 72 ./bash-hello-strace |& grep ^write write(1, "Hello strace!\n", 14Hello strace! $ strace -s 72 ./guile-hello-strace.scm |& grep ^write write(1, "Hello strace!\n", 14Hello strace! Where $ cat -n ./guile-hello-strace.scm |gxsnip --8<---------------cut here---------------start------------->8--- 1 #! /usr/bin/guile -s 2 !# 3 (display "Hello strace!\n") --8<---------------cut here---------------end--------------->8--- and $ cat -n ./bash-hello-strace |gxsnip --8<---------------cut here---------------start------------->8--- 1 #! /usr/bin/bash 2 echo 'Hello strace!' --8<---------------cut here---------------end--------------->8--- I had in mind expanding on and polishing something like $ strace ./guile-hello-strace.scm |& grep ^write write(1, "Hello strace!\n", 14Hello strace! $ strace ./bash-hello-strace |& grep ^write write(1, "Hello strace!\n", 14Hello strace! Where $ cat -n ./guile-hello-strace.scm |gxsnip --8<---------------cut here---------------start------------->8--- 1 #! /usr/bin/guile -s 2 !# 3 (display "Hello strace!\n") --8<---------------cut here---------------end--------------->8--- and $ cat -n ./bash-hello-strace |gxsnip --8<---------------cut here---------------start------------->8--- 1 #! /usr/bin/bash 2 echo 'Hello strace!' --8<---------------cut here---------------end--------------->8--- Thus one could write something like this, compiling only it in whatever environment, so that updates to the tool chain which might otherwise have to be recompiled can be ignored. -- Regards Bengt Richter