Re: Use of %texlive-revision and %texlive-tag in tex.scm

["Nathan Benedetto Proença" <nathan@vieiraproenca.com>]

Bengt Richter <bokr@bokr.com> writes:

> Hi Nathan,
> Nice writeup!

Thank you!

> On +2021-07-05 11:03:46 -0300, Nathan Benedetto Proença wrote:
>> Hello!
>> 
>> I am trying to upgrade the package texlive, first for me, but hopefully
>> for a patch, and I have a question regarding Guix policies.
>> 
>> As you can see on
>> 
>>    https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/texlive.scm?id=f7692617b624a01615cf412773c4dad9a2deeb68
>> 
>> the file guix/build-system/texlive.scm exposes two variables:
>> 
>>     (define %texlive-tag "texlive-2019.3")
>>     (define %texlive-revision 51265)
>> 
>> These variables are used throughout gnu/packages/tex.scm, as you can see
>> on
>> 
>>    https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tex.scm?id=f7692617b624a01615cf412773c4dad9a2deeb68
>> 
>> An example is the following code:
>> 
>>   (define hyph-utf8-scripts
>>     (origin
>>       (method svn-fetch)
>>       (uri (texlive-ref "generic" "hyph-utf8"))
>>       (file-name (string-append "hyph-utf8-scripts-"
>>                                 (number->string %texlive-revision)
>>                                 "-checkout"))
>>       (sha256
>>        (base32
>>         "0lk7shx768sxvgr85y8bnmmnj8x4bbkgpxrz3z8jp8avi33prw83"))))
>> 
>> Grep tells me there are 290+ occurrences of `%texlive-revision`.
>> What is the purpose of these variables?
>> 
>> You see, they give me the impression that Guix is really concerned about
>> upgrading *all* of texlive at once.
>> These variables tell me I should go to the file texlive.scm and bump the
>> tag and revision, and then handle all the broken hashes.
>> 
>> Hence, it seems to me that any attempt to upgrade the texlive package
>> would have to be done in a separate branch, which would only be merged
>> into master when all the packages are upgraded.
>> 
>> Is this the case?
>> And if so, why?
>> 
>> I have the impression that if such "monolithic" upgrade is not a goal,
>> and "partial" our "per-package" upgrades are desirable, there may be
>> better solutions.
>> 
>> For example, we could add keyword arguments to texlive-ref and
>> texlive-origin, so the code above becomes something like this
>> 
>>   (define hyph-utf8-scripts
>>     (origin
>>       (method svn-fetch)
>>       (uri (texlive-ref "generic" "hyph-utf8"
>>                         #:texlive-tag "texlive-2019.3"
>>                         #:texlive-revision 51265))
>>       (file-name "hyph-utf8-scripts-51625-checkout")
>>       (sha256
>>        (base32
>>         "0lk7shx768sxvgr85y8bnmmnj8x4bbkgpxrz3z8jp8avi33prw83"))))
>> 
>> This would work right now, and we could eventually remove every use of
>> %texlive-revision and %texlive-tag, so they become implementation
>> details of the build-system texlive.scm; a fallback version.
>> And further down the road we may even decide to remove this fallback,
>> and make developers be explicit about their tags and revisions; this
>> could amount to a refactor which makes the keyword arguments into
>> required arguments, for example.
>> 
>> I also like the second version of the code because the hash already
>> pinpoints the tag and revision: both texlive-ref and texlive-origin use
>> these variables to find the correct files to download.
>> This just makes this dependency explicit.
>> 
>> In any case, as this may be a choice between shipping stable and
>> up-to-date packages, and as I am new to contributing to Guix, I found
>> fitting to ask.
>> 
>> Thanks in advance!
>> Nathan
>> 
>
> I am wondering about guaranteeing generic behaviour by
> guaranteeing program source and toolchain source hash
> equivalences vs ignoring sources and guaranteeing end
> results by testing results.

It seems to me that you are talking about my email regarding using
hashing in Scheme refactoring, right?
This one:

     https://lists.gnu.org/archive/html/guix-devel/2021-07/msg00023.html

I will assume this is the case, even though I we can actually simply
talk about what you wrote.

First thoughts: I think that what we really want to guarantee is
"correctness", and that both hashes or tests are mere proxies for this.

I see them as useful in distinct moments of package development.

For example, lets say that I want to partially upgrade TeXLive (which I
am already convinced is not a good idea, TBH, but serves as an
example).
To be able to do so, I may think that I want to refactor some function
to pinpoint the version I will download.
I can then make a single commit which updates the signature and all the
call sites of the function.
I can then actually upgrade the package I care about in a second commit,
and do some "proper testing" in it, like producing some pdf with latex.

My interest in checking hashes in the first commit is to get some
confidence that this step, which may change hundred of places, did not
introduce a new bug, so I can focus any debugging on the hopefully more
local change made in the second commit.

> I.e., if you want to print the sum of x and y passed as
> strings to a program, output as a string to stdout, it
> doesn't matter (other than optimization and debuggability)
> what language the program was written in, so long as it was
> compiled into a form that execve and co can launch and the
> end result is the same.
>
> As part of testing, maybe strace could be used to generate
> some kind of canonical kernel transaction trace that could
> be used to compare behaviours for equivalency of executing
> different-language programs?
>
> This would be a radical change in the approach to
> reproducibility, maybe dynamically selecting from a
> whitelist of trusted/tested substitutable executables with
> hash names in /gnu but not necessarily (though not
> excluding) binaries produced with guix source guarantees.
>
> Seems like guix is turing-complete enough to provide this
> kind of substitutable foreign functions already, so might
> this be a way to avoid mass recompilations?
>
> Or is this already available, but not so much used?
>
> I am not sure where to contibute thoughts like these, where
> they would be of interest rather than distracting. (Pls
> excuse the noise, if that's what this is to you).

I am not sure I follow, although I do not regard this as noise.

You are saying that one could use strace as a check that some program
behaves the same as it did before, similar to the use that I suggested
for hashing, right?

I do not understand how this would be used, and how this avoids mass
recompilations.
I mean, to have binaries to inspect with strace I must build the code
anyway, right?
Am I missing something?

> -- 
> Regards,
> Bengt Richter