Re: Use of %texlive-revision and %texlive-tag in tex.scm

[Bengt Richter <bokr@bokr.com>]

On +2021-07-06 15:28:43 -0300, Nathan Benedetto Proença wrote:
> Bengt Richter <bokr@bokr.com> writes:
> 
> > Hi Nathan,
> > Nice writeup!
> 
> Thank you!
> 
> > On +2021-07-05 11:03:46 -0300, Nathan Benedetto Proença wrote:
> >> Hello!
> >> 
> >> I am trying to upgrade the package texlive, first for me, but hopefully
> >> for a patch, and I have a question regarding Guix policies.
> >> 
> >> As you can see on
> >> 
> >>    https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/texlive.scm?id=f7692617b624a01615cf412773c4dad9a2deeb68
> >> 
> >> the file guix/build-system/texlive.scm exposes two variables:
> >> 
> >>     (define %texlive-tag "texlive-2019.3")
> >>     (define %texlive-revision 51265)
> >> 
> >> These variables are used throughout gnu/packages/tex.scm, as you can see
> >> on
> >> 
> >>    https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tex.scm?id=f7692617b624a01615cf412773c4dad9a2deeb68
> >> 
> >> An example is the following code:
> >> 
> >>   (define hyph-utf8-scripts
> >>     (origin
> >>       (method svn-fetch)
> >>       (uri (texlive-ref "generic" "hyph-utf8"))
> >>       (file-name (string-append "hyph-utf8-scripts-"
> >>                                 (number->string %texlive-revision)
> >>                                 "-checkout"))
> >>       (sha256
> >>        (base32
> >>         "0lk7shx768sxvgr85y8bnmmnj8x4bbkgpxrz3z8jp8avi33prw83"))))
> >> 
> >> Grep tells me there are 290+ occurrences of `%texlive-revision`.
> >> What is the purpose of these variables?
> >> 
> >> You see, they give me the impression that Guix is really concerned about
> >> upgrading *all* of texlive at once.
> >> These variables tell me I should go to the file texlive.scm and bump the
> >> tag and revision, and then handle all the broken hashes.
> >> 
> >> Hence, it seems to me that any attempt to upgrade the texlive package
> >> would have to be done in a separate branch, which would only be merged
> >> into master when all the packages are upgraded.
> >> 
> >> Is this the case?
> >> And if so, why?
> >> 
> >> I have the impression that if such "monolithic" upgrade is not a goal,
> >> and "partial" our "per-package" upgrades are desirable, there may be
> >> better solutions.
> >> 
> >> For example, we could add keyword arguments to texlive-ref and
> >> texlive-origin, so the code above becomes something like this
> >> 
> >>   (define hyph-utf8-scripts
> >>     (origin
> >>       (method svn-fetch)
> >>       (uri (texlive-ref "generic" "hyph-utf8"
> >>                         #:texlive-tag "texlive-2019.3"
> >>                         #:texlive-revision 51265))
> >>       (file-name "hyph-utf8-scripts-51625-checkout")
> >>       (sha256
> >>        (base32
> >>         "0lk7shx768sxvgr85y8bnmmnj8x4bbkgpxrz3z8jp8avi33prw83"))))
> >> 
> >> This would work right now, and we could eventually remove every use of
> >> %texlive-revision and %texlive-tag, so they become implementation
> >> details of the build-system texlive.scm; a fallback version.
> >> And further down the road we may even decide to remove this fallback,
> >> and make developers be explicit about their tags and revisions; this
> >> could amount to a refactor which makes the keyword arguments into
> >> required arguments, for example.
> >> 
> >> I also like the second version of the code because the hash already
> >> pinpoints the tag and revision: both texlive-ref and texlive-origin use
> >> these variables to find the correct files to download.
> >> This just makes this dependency explicit.
> >> 
> >> In any case, as this may be a choice between shipping stable and
> >> up-to-date packages, and as I am new to contributing to Guix, I found
> >> fitting to ask.
> >> 
> >> Thanks in advance!
> >> Nathan
> >> 
> >
> > I am wondering about guaranteeing generic behaviour by
> > guaranteeing program source and toolchain source hash
> > equivalences vs ignoring sources and guaranteeing end
> > results by testing results.
> 
> It seems to me that you are talking about my email regarding using
> hashing in Scheme refactoring, right?
> This one:
> 
>      https://lists.gnu.org/archive/html/guix-devel/2021-07/msg00023.html
> 
> I will assume this is the case, even though I we can actually simply
> talk about what you wrote.
> 
> First thoughts: I think that what we really want to guarantee is
> "correctness", and that both hashes or tests are mere proxies for this.
> 
> I see them as useful in distinct moments of package development.
> 
> For example, lets say that I want to partially upgrade TeXLive (which I
> am already convinced is not a good idea, TBH, but serves as an
> example).
> To be able to do so, I may think that I want to refactor some function
> to pinpoint the version I will download.
> I can then make a single commit which updates the signature and all the
> call sites of the function.
> I can then actually upgrade the package I care about in a second commit,
> and do some "proper testing" in it, like producing some pdf with latex.
> 
> My interest in checking hashes in the first commit is to get some
> confidence that this step, which may change hundred of places, did not
> introduce a new bug, so I can focus any debugging on the hopefully more
> local change made in the second commit.
> 
> > I.e., if you want to print the sum of x and y passed as
> > strings to a program, output as a string to stdout, it
> > doesn't matter (other than optimization and debuggability)
> > what language the program was written in, so long as it was
> > compiled into a form that execve and co can launch and the
> > end result is the same.
> >
> > As part of testing, maybe strace could be used to generate
> > some kind of canonical kernel transaction trace that could
> > be used to compare behaviours for equivalency of executing
> > different-language programs?
> >
> > This would be a radical change in the approach to
> > reproducibility, maybe dynamically selecting from a
> > whitelist of trusted/tested substitutable executables with
> > hash names in /gnu but not necessarily (though not
> > excluding) binaries produced with guix source guarantees.
> >
> > Seems like guix is turing-complete enough to provide this
> > kind of substitutable foreign functions already, so might
> > this be a way to avoid mass recompilations?
> >
> > Or is this already available, but not so much used?
> >
> > I am not sure where to contibute thoughts like these, where
> > they would be of interest rather than distracting. (Pls
> > excuse the noise, if that's what this is to you).
> 
> I am not sure I follow, although I do not regard this as noise.
> 
> You are saying that one could use strace as a check that some program
> behaves the same as it did before, similar to the use that I suggested
> for hashing, right?
> 
> I do not understand how this would be used, and how this avoids mass
> recompilations.
> I mean, to have binaries to inspect with strace I must build the code
> anyway, right?
> Am I missing something?
>

I had in mind expanding on and polishing something like

$ strace -s 72 ./bash-hello-strace |& grep ^write
write(1, "Hello strace!\n", 14Hello strace!


$ strace -s 72 ./guile-hello-strace.scm |& grep ^write
write(1, "Hello strace!\n", 14Hello strace!

Where
$ cat -n ./guile-hello-strace.scm |gxsnip
--8<---------------cut here---------------start------------->8---
     1	#! /usr/bin/guile -s
     2	!#
     3	(display "Hello strace!\n")
--8<---------------cut here---------------end--------------->8---

and
$ cat -n ./bash-hello-strace |gxsnip
--8<---------------cut here---------------start------------->8---
     1	#! /usr/bin/bash
     2	echo 'Hello strace!'
--8<---------------cut here---------------end--------------->8---


I had in mind expanding on and polishing something like

$ strace -s 72 ./bash-hello-strace |& grep ^write
write(1, "Hello strace!\n", 14Hello strace!


$ strace -s 72 ./guile-hello-strace.scm |& grep ^write
write(1, "Hello strace!\n", 14Hello strace!

Where
$ cat -n ./guile-hello-strace.scm |gxsnip
--8<---------------cut here---------------start------------->8---
     1	#! /usr/bin/guile -s
     2	!#
     3	(display "Hello strace!\n")
--8<---------------cut here---------------end--------------->8---

and
$ cat -n ./bash-hello-strace |gxsnip
--8<---------------cut here---------------start------------->8---
     1	#! /usr/bin/bash
     2	echo 'Hello strace!'
--8<---------------cut here---------------end--------------->8---


I had in mind expanding on and polishing something like

$ strace ./guile-hello-strace.scm |& grep ^write
write(1, "Hello strace!\n", 14Hello strace!

$ strace ./bash-hello-strace |& grep ^write
write(1, "Hello strace!\n", 14Hello strace!

Where
$ cat -n ./guile-hello-strace.scm |gxsnip
--8<---------------cut here---------------start------------->8---
     1	#! /usr/bin/guile -s
     2	!#
     3	(display "Hello strace!\n")
--8<---------------cut here---------------end--------------->8---

and
$ cat -n ./bash-hello-strace |gxsnip
--8<---------------cut here---------------start------------->8---
     1	#! /usr/bin/bash
     2	echo 'Hello strace!'
--8<---------------cut here---------------end--------------->8---

Thus one could write something like this, compiling only it in whatever environment,
so that updates to the tool chain which might otherwise have to be recompiled can be ignored.

--
Regards
Bengt Richter