Question about packaging TexLive

Question about packaging TexLive

[Simon Tournier]

Hi,

CC: TeX team

For context, TeX (broad meaning) is deep in the graph, thus its
preservation is important when considering time-travel.  And because TeX
packages relies on Subversion, it raises some questions – for the
interesting reader, see [1].

Here, I would like to open a discussion about the way we are packaging
TeX, quoting Nicolas from [2]:

        There are two approaches going on here. `simple-texlive-package' tries
        to provide a tessellation of the full texlive, i.e., if you install
        (assuming they are defined in Guix) every standalone texlive package,
        you will get the exact full texlive distribution, whereas
        `texlive-ref' + `texlive-build-system' tries to generate a minimal (a
        bit too minimal considering missing documentation) working system.
        I think the old way is error prone.

        So, do we (not only the two of us, obviously) agree on the
        `simple-texlive-package' approach? If so, what should we do about
        "source"?

        2: https://issues.guix.gnu.org/issue/62570#10

Other said, any objection to go from this pattern:

--8<---------------cut here---------------start------------->8---
(define-public texlive-foo
  (package
    (name "texlive-foo")
    (version (number->string %texlive-revision))
    (source (origin
              (method svn-multi-fetch)
              (uri (svn-multi-reference
                    (url (string-append "svn://www.tug.org/texlive/tags/"
                                        %texlive-tag "/Master/texmf-dist"))
                    (locations '("/metafont/"
                                 "/fonts/source/public/modes/"))
                    (revision %texlive-revision)))
              (file-name (string-append name "-" version "-checkout"))
              (sha256
               (base32
                "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"))))
    (build-system gnu-build-system)
    (arguments
     `(#:tests? #f ; no test target
       #:phases
[...]
--8<---------------cut here---------------end--------------->8---

to this pattern:

--8<---------------cut here---------------start------------->8---
(define-public texlive-foo
  (package
    (inherit (simple-texlive-package
              "texlive-foo"
              (list "/tex/generic/foo")
              (base32
               "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
              #:trivial? #t))
--8<---------------cut here---------------end--------------->8---

It is about ~35 packages, if I count correctly.

WDYT?

Cheers,
simon

1: https://issues.guix.gnu.org/43442#20

Re: Question about packaging TexLive

[Nicolas Goaziou]

Hello,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Other said, any objection to go from this pattern:
>
> --8<---------------cut here---------------start------------->8---
> (define-public texlive-foo
>   (package
>     (name "texlive-foo")
>     (version (number->string %texlive-revision))
>     (source (origin
>               (method svn-multi-fetch)
>               (uri (svn-multi-reference
>                     (url (string-append "svn://www.tug.org/texlive/tags/"
>                                         %texlive-tag "/Master/texmf-dist"))
>                     (locations '("/metafont/"
>                                  "/fonts/source/public/modes/"))
>                     (revision %texlive-revision)))
>               (file-name (string-append name "-" version "-checkout"))
>               (sha256
>                (base32
>                 "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"))))
>     (build-system gnu-build-system)
>     (arguments
>      `(#:tests? #f ; no test target
>        #:phases
> [...]
> --8<---------------cut here---------------end--------------->8---
>
>
> to this pattern:
>
> --8<---------------cut here---------------start------------->8---
> (define-public texlive-foo
>   (package
>     (inherit (simple-texlive-package
>               "texlive-foo"
>               (list "/tex/generic/foo")
>               (base32
>                "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
>               #:trivial? #t))
> --8<---------------cut here---------------end--------------->8---
>
> It is about ~35 packages, if I count correctly.
>
> WDYT?

I think this is necessary but not sufficient.

1. "#:trivial? #t" means we're giving up generating ".sty" files from
   source;

2. Some affected packages are also missing propagated inputs (e.g.,
   texlive-latex-graphics for texlive-latex-fncychap);

3. Some affected package also need to be renamed
   (texlive-latex-fncychap, texlive-latex-framed).

AFAIU, this changes will probably ease time travelling, but the massive
rebuild it will entail will have to be repeated for the other steps.

So maybe we could do every step above in one go?

Regards,
-- 
Nicolas Goaziou

Re: Question about packaging TexLive

[Ricardo Wurmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
>> Other said, any objection to go from this pattern:
>>
>> --8<---------------cut here---------------start------------->8---
>> (define-public texlive-foo
>>   (package
>>     (name "texlive-foo")
>>     (version (number->string %texlive-revision))
>>     (source (origin
>>               (method svn-multi-fetch)
>>               (uri (svn-multi-reference
>>                     (url (string-append "svn://www.tug.org/texlive/tags/"
>>                                         %texlive-tag "/Master/texmf-dist"))
>>                     (locations '("/metafont/"
>>                                  "/fonts/source/public/modes/"))
>>                     (revision %texlive-revision)))
>>               (file-name (string-append name "-" version "-checkout"))
>>               (sha256
>>                (base32
>>                 "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"))))
>>     (build-system gnu-build-system)
>>     (arguments
>>      `(#:tests? #f ; no test target
>>        #:phases
>> [...]
>> --8<---------------cut here---------------end--------------->8---
>>
>>
>> to this pattern:
>>
>> --8<---------------cut here---------------start------------->8---
>> (define-public texlive-foo
>>   (package
>>     (inherit (simple-texlive-package
>>               "texlive-foo"
>>               (list "/tex/generic/foo")
>>               (base32
>>                "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
>>               #:trivial? #t))
>> --8<---------------cut here---------------end--------------->8---
>>
>> It is about ~35 packages, if I count correctly.
>>
>> WDYT?
>
> I think this is necessary but not sufficient.
>
> 1. "#:trivial? #t" means we're giving up generating ".sty" files from
>    source;

Trivial should only ever be used for packages where there aren’t any
sources to build from.

> 2. Some affected packages are also missing propagated inputs (e.g.,
>    texlive-latex-graphics for texlive-latex-fncychap);

Correct.  The old importer didn’t know about dependencies.  The newer
importer uses tlpdb to get a list of dependencies.

> 3. Some affected package also need to be renamed
>    (texlive-latex-fncychap, texlive-latex-framed).
>
> AFAIU, this changes will probably ease time travelling, but the massive
> rebuild it will entail will have to be repeated for the other steps.
>
> So maybe we could do every step above in one go?

Using svn-multi-reference is the old way; we should be using
simple-texlive-package for all packages wherever possible.

Yes, we still have a lot of old package names.  They all should be
renamed to the new style.

All packages must be checked for completeness.  All new packages I’ve
added have been checked.  All old packages must be checked before
renaming them.  The way to check them is to build them and then check
the outputs with something like this:

--8<---------------cut here---------------start------------->8---
,use (guix import texlive)
,pp (files-differ? "/gnu/store/…-texlive-amsfonts-fixed-59745/share/" "amsfonts")
--8<---------------cut here---------------end--------------->8---

This compares outputs with tlpdb.

Note that simple-texlive-package is far from perfect.  We have many
tex packages that require excessive workarounds to ensure that files end
up in the correct location and that superfluous files are removed.
simple-texlive-package should eventually take care of all of this, but I
haven’t been able to make time to work on it.

All this work should be done on a separate feature branch.

-- 
Ricardo

Re: Question about packaging TexLive

[Nicolas Goaziou]

Ricardo Wurmus <rekado@elephly.net> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> 1. "#:trivial? #t" means we're giving up generating ".sty" files from
>>    source;
>
> Trivial should only ever be used for packages where there aren’t any
> sources to build from.

We are on the same length.

However, some packages do use "#:trivial #t" even though they should
not, e.g., `texlive-latex-marginfix'. My point is that we should not
make "#:trivial #t" the default, assuming that was a possibility
stemming from Simon's question.

>> 2. Some affected packages are also missing propagated inputs (e.g.,
>>    texlive-latex-graphics for texlive-latex-fncychap);
>
> Correct.  The old importer didn’t know about dependencies.  The newer
> importer uses tlpdb to get a list of dependencies.

AFAIU, this is not sufficient. All dependencies do not appear in
"texlive.tlpbd". For example, `texlive-halloweenmath' has no "depend"
entry in the tlpbd. Yet, it requires both texlive-amsmath and
texlive-pict2e (i.e., "halloweenmath.sty" contains
"\RequirePackage{amsmath,pict2e}").

So, unless I'm mistaken, most dependencies have to be resolved manually.

> All packages must be checked for completeness.  All new packages I’ve
> added have been checked.  All old packages must be checked before
> renaming them.  The way to check them is to build them and then check
> the outputs with something like this:
>
> --8<---------------cut here---------------start------------->8---
> ,use (guix import texlive)
> ,pp (files-differ? "/gnu/store/…-texlive-amsfonts-fixed-59745/share/" "amsfonts")
> --8<---------------cut here---------------end--------------->8---
>
> This compares outputs with tlpdb.

I use this valuable tool, but this is somewhat tedious. Could this
become a test in the linter?

> Note that simple-texlive-package is far from perfect.  We have many
> tex packages that require excessive workarounds to ensure that files end
> up in the correct location and that superfluous files are removed.
> simple-texlive-package should eventually take care of all of this, but I
> haven’t been able to make time to work on it.

We may need to update more Texlive packages to get a better grasp of the
changes required in `simple-texlive-packages'. When doing such updates,
I mostly copy and paste definition from previously updated packages,
which is a good thing. However, some translations seem complicated.

> All this work should be done on a separate feature branch.

Agreed.

Regards,

Re: Question about packaging TexLive

[Andreas Enge]

Am Wed, Apr 19, 2023 at 01:47:16PM +0200 schrieb Nicolas Goaziou:
> AFAIU, this is not sufficient. All dependencies do not appear in
> "texlive.tlpbd". For example, `texlive-halloweenmath' has no "depend"
> entry in the tlpbd. Yet, it requires both texlive-amsmath and
> texlive-pict2e (i.e., "halloweenmath.sty" contains
> "\RequirePackage{amsmath,pict2e}").
> So, unless I'm mistaken, most dependencies have to be resolved manually.

Could the result be upstreamed as bug reports?

Andreas

Re: Question about packaging TexLive

[Ricardo Wurmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>>> 2. Some affected packages are also missing propagated inputs (e.g.,
>>>    texlive-latex-graphics for texlive-latex-fncychap);
>>
>> Correct.  The old importer didn’t know about dependencies.  The newer
>> importer uses tlpdb to get a list of dependencies.
>
> AFAIU, this is not sufficient. All dependencies do not appear in
> "texlive.tlpbd". For example, `texlive-halloweenmath' has no "depend"
> entry in the tlpbd. Yet, it requires both texlive-amsmath and
> texlive-pict2e (i.e., "halloweenmath.sty" contains
> "\RequirePackage{amsmath,pict2e}").
>
> So, unless I'm mistaken, most dependencies have to be resolved manually.

Yes.  My point was that tlpdb contains some dependency information,
which is completely absent in those packages produced with the earlier
importer.

We should change the importer to scan the directory for RequirePackage
and the like.

>> All packages must be checked for completeness.  All new packages I’ve
>> added have been checked.  All old packages must be checked before
>> renaming them.  The way to check them is to build them and then check
>> the outputs with something like this:
>>
>> --8<---------------cut here---------------start------------->8---
>> ,use (guix import texlive)
>> ,pp (files-differ? "/gnu/store/…-texlive-amsfonts-fixed-59745/share/" "amsfonts")
>> --8<---------------cut here---------------end--------------->8---
>>
>> This compares outputs with tlpdb.
>
> I use this valuable tool, but this is somewhat tedious. Could this
> become a test in the linter?

I don’t see why it could not.

>> Note that simple-texlive-package is far from perfect.  We have many
>> tex packages that require excessive workarounds to ensure that files end
>> up in the correct location and that superfluous files are removed.
>> simple-texlive-package should eventually take care of all of this, but I
>> haven’t been able to make time to work on it.
>
> We may need to update more Texlive packages to get a better grasp of the
> changes required in `simple-texlive-packages'.

A good example for the required changes is texlive-totpages or
texlive-soulutf8.  All these overrides to #:tex-directory and #:phases
are pretty common.  We usually need to add a chdir phase to change
directories to the sources and override the copy-files phase, because
otherwise it copies garbage.

This is the least I’d expect simple-texlive-packages to take care of.

-- 
Ricardo