Re: documentation in TeX Live collections

Re: documentation in TeX Live collections

[Nicolas Goaziou]

Hello,

Thanks for that detailed report!

Emmanuel Beffara <manu@beffara.org> writes:

> There has been a lot of movement around TeX Live recently and it is very nice
> to see. We now have a usable modular installation and a great number of
> available packages and collections. However, I don't understand what is the
> proper way to reach documentation in the current system.
>
> I installed `texlive-scheme-medium` in my home profile managed by `guix home`,
> everything works including `texdoc` (although it always says « Info: Running
> Texdoc not installed in the current TEXMFMAIN. » for some reason), but there
> is essentially no documentation installed:
>
>     $ texdoc inputenc
>     Info: Running Texdoc not installed in the current TEXMFMAIN.
>     You don't appear to have any local documentation installed.
>
>     There may be online documentation available for "inputenc" at
> 	https://texdoc.org/serve/inputenc/0
>     This documentation may be for a different version than you have installed.
>
>     Would you like to search online? (y/N)
>
> Indeed the `doc` folder is nearly empty:
>
>     $ ls $GUIX_TEXMF/doc
>     bibtex8/  bibtexu/  chktex/
>
> Apparently, all individual packages have a specific "doc" output but
> collections and schemes do not and they don't have them as inputs either. So
> we end up with an installation with no documentation

It sounds like a good default to me. I doubt anyone reads all
documentation for every TeX Live package they install.

> (apart from the three above, which is surprising).

They are brought by `texlive-bin', which has no "doc" output.

> I tried to explicity include documentation in a sub-shell but this changes
> nothing:
>
>     $ guix shell texlive-latex:doc -- texdoc inputenc

Note that you also need to install texlive-texdoc (or some collection/scheme including it).

>     Info: Running Texdoc not installed in the current TEXMFMAIN.
>     You don't appear to have any local documentation installed.
>
>     There may be online documentation available for "inputenc" at
> 	https://texdoc.org/serve/inputenc/0
>     This documentation may be for a different version than you have installed.
>
>     Would you like to search online? (y/N)
>
> Including the TeX Live scheme in the same shell makes things worse:
>
>     $ guix shell texlive-scheme-medium texlive-latex:doc -- texdoc inputenc
>     Info: Running Texdoc not installed in the current TEXMFMAIN.
>     texdoc error: No texlive.tlpdb nor shipped tlpdb data found.
>
> Inspecting GUIX_TEXMF in this shell reveals that it now contains two paths,
> one of which does contain the right documentation:
>
>     $ guix shell texlive-scheme-medium texlive-latex:doc
>     $ env | grep TEX
>     GUIX_TEXMF=/gnu/store/fg1z0jgkj0r4v8i3rmpg0c1vfirbg1ac-profile/share/texmf-dist:/home/manu/.guix-home/profile/share/texmf-dist
>     $ ls /gnu/store/fg1z0jgkj0r4v8i3rmpg0c1vfirbg1ac-profile/share/texmf-dist/doc
>     bibtex8  bibtexu  chktex  latex
>     $ export GUIX_TEXMF=${GUIX_TEXMF%:*}
>     $ texdoc inputenc
>     ... inputenc.pdf is displayed! ...
>
> Apparently the fact that this GUIX_TEXMF variable contains several paths is
> problematic for texdoc.

Would the following definition for texlive-texdoc solve both issues
mentioned above? (the warning and the error.)

--8<---------------cut here---------------start------------->8---
(define-public texlive-texdoc
  (package
    (name "texlive-texdoc")
    (version (number->string %texlive-revision))
    (source (texlive-origin
             name version
             (list "doc/man/man1/texdoc.1"
                   "doc/man/man1/texdoc.man1.pdf"
                   "doc/support/texdoc/" "scripts/texdoc/"
                   "texdoc/")
             (base32
              "19mvh7pm2332f6c8nzgcbscm9vcz0apwfgm0m55ycibssc2fb3ww")))
    (outputs '("out" "doc"))
    (build-system texlive-build-system)
    (arguments
     (list #:link-scripts #~(list "texdoc.tlu")
           #:phases
           #~(modify-phases %standard-phases
               ;; Prevent "Info: Running Texdoc not installed in the current
               ;; TEXMFMAIN" warning by skipping an unnecessary test.
               (add-after 'unpack 'fix-script
                 (lambda _
                   (substitute* "scripts/texdoc/texdoc.tlu"
                     (("if texmf ~= nil") "if false"))))
               ;; Teach `texdoc' how to handle multiple directories in
               ;; GUIX_TEXMF environment variable.
               (add-after 'link-scripts 'wrap-programs
                 (lambda _
                   (wrap-program (string-append #$output "/bin/texdoc")
                     '("GUIX_TEXMF" = ("${GUIX_TEXMF%:*}"))))))))
    (propagated-inputs (list texlive-kpathsea))
    (home-page "https://ctan.org/pkg/texdoc")
    (synopsis "Documentation access for TeX Live")
    (description
     "@command{texdoc} is a Lua script providing easy access to the
documentation in TeX Live: PDF, DVI, plain text files, and more.  Viewing and
other configuration can be extensively customized.")
    (license license:gpl3+)))
--8<---------------cut here---------------end--------------->8---

> As an attempt to work around this, I tried to add texlive-latex:doc to my home
> profile definition and it did make that documentation available to texdoc.
> Moreover, for some reason, ALL documentation was downloaded:
>
>     $ guix home reconfigure home.scm
>     ...
>      texlive-cm-66594-doc  2KiB
>      texlive-etex-66594-doc  189KiB
>      texlive-hyphen-complete-66594-doc  783KiB
>      texlive-kpathsea-66594-doc  1022KiB
>      texlive-pdftex-66594  4.2MiB
>     ...
>     $ ls -d /gnu/store/*-texlive-*-doc/ | wc
>        1105    1105   82506
>
> Apparently something has triggered the download of documentation for all
> packages `texlive-scheme-medium` depends on but only the one I explicitly
> requested is made available in the profile (which is expected). All these
> other documentation were downloaded but not used and `guix gc` actually
> deletes them all!

I noticed that, too, but I don't have any explanation for it at the
moment.

For example,

  ./pre-inst-env guix shell texlive-scheme-basic texlive-texdoc texlive-babel:doc

is enough to trigger a massive download of "doc" outputs.

> So what would be the proper way to install `texlive-scheme-medium` in a home
> profile with the documentation of the packages it includes ?

If that's a common request, we could add a `texlive-collection-foo-doc'
package that would propagate all "doc" outputs from all packages
included in `texlive-collection-foo'.

However, I'm a bit reluctant to add more artificial packages (i.e., not
known to TeX Live distribution). Also, it might be as simple to do it in
one's own manifest.

I'm Cc'ing guix-devel ML.

Regards,
-- 
Nicolas Goaziou

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hello,

De Nicolas Goaziou le 27/08/2023 à 12:13:
> > Apparently, all individual packages have a specific "doc" output but
> > collections and schemes do not and they don't have them as inputs either. So
> > we end up with an installation with no documentation
> 
> It sounds like a good default to me. I doubt anyone reads all
> documentation for every TeX Live package they install.

Yet, as far as I know, most packages in Guix (apart from texlive-* ones) come
with their documentation, so it feels somewhat inconsistent. But I agree that
not including the docs in the main outputs can make sense, especially given
the volume it represents. Anyway, given that there is extensive documentation
in TeX Live, it seems only natural to have it easily accessible.

> > I tried to explicity include documentation in a sub-shell but this changes
> > nothing:
> >
> >     $ guix shell texlive-latex:doc -- texdoc inputenc
> 
> Note that you also need to install texlive-texdoc (or some collection/scheme including it).

Indeed, but this command was run in a profile where texlive-scheme-medium is
present, so texdoc was already available. Adding texlive-texdoc to the guix
shell command does not change anything.

> > Apparently the fact that this GUIX_TEXMF variable contains several paths is
> > problematic for texdoc.
> 
> Would the following definition for texlive-texdoc solve both issues
> mentioned above? (the warning and the error.)
[...]
>                (add-after 'link-scripts 'wrap-programs
>                  (lambda _
>                    (wrap-program (string-append #$output "/bin/texdoc")
>                      '("GUIX_TEXMF" = ("${GUIX_TEXMF%:*}"))))))))

It would certainly remove the warning but it would make only the first path
usable by texdoc, while other tools seem to support having several paths in
GUIX_TEXMF. Besides, I don't understand how GUIX_TEXMF is taken into account
by the various tools. Web2c and co don't know them, so there must be some
wrapping or patching somewhere in the package definitions?

> > Apparently something has triggered the download of documentation for all
> > packages `texlive-scheme-medium` depends on but only the one I explicitly
> > requested is made available in the profile (which is expected). All these
> > other documentation were downloaded but not used and `guix gc` actually
> > deletes them all!
> 
> I noticed that, too, but I don't have any explanation for it at the
> moment.

Is there a way to diagnose that kind of thing? I stumbled on a similar
behaviour in other contexts and was unable to investigate it (in my case, big
debug versions of Qt libraries are often downloaded, although I neved
requested any debugging version of anything).

> > So what would be the proper way to install `texlive-scheme-medium` in a home
> > profile with the documentation of the packages it includes ?
> 
> If that's a common request, we could add a `texlive-collection-foo-doc'
> package that would propagate all "doc" outputs from all packages
> included in `texlive-collection-foo'.
> 
> However, I'm a bit reluctant to add more artificial packages (i.e., not
> known to TeX Live distribution). Also, it might be as simple to do it in
> one's own manifest.

I think it would make much more sense to have "doc" outputs also for
collections and schemes. It would be consistent with the structure of
individual packages and would not require artificial packages.

Having individual package documentations in one's manifests is of course
doable but it is contradictory with the approach of collections.

-- 
Emmanuel Beffara

Re: documentation in TeX Live collections

[Nicolas Goaziou]

Hello,

Emmanuel Beffara <manu@beffara.org> writes:

> Yet, as far as I know, most packages in Guix (apart from texlive-* ones) come
> with their documentation, so it feels somewhat inconsistent.

Every texlive-* package comes with its documentation, in a separate
output. "doc" output are not uncommon at all in Guix. Therefore,
I disagree with the inconsistency you're talking about.

> not including the docs in the main outputs can make sense, especially given
> the volume it represents. Anyway, given that there is extensive documentation
> in TeX Live, it seems only natural to have it easily accessible.

Barring the `texdoc' issue, documentation is easily accessible; you just
need to install the "doc" output of the package you're interested in.

>> Would the following definition for texlive-texdoc solve both issues
>> mentioned above? (the warning and the error.)
> [...]
>>                (add-after 'link-scripts 'wrap-programs
>>                  (lambda _
>>                    (wrap-program (string-append #$output "/bin/texdoc")
>>                      '("GUIX_TEXMF" = ("${GUIX_TEXMF%:*}"))))))))
>
> It would certainly remove the warning but it would make only the first path
> usable by texdoc, while other tools seem to support having several paths in
> GUIX_TEXMF. Besides, I don't understand how GUIX_TEXMF is taken into account
> by the various tools. Web2c and co don't know them, so there must be some
> wrapping or patching somewhere in the package definitions?

It seems some programs do handle it fine, e.g., tlmgr, but not all of
them (e.g., texdoc or chktex). In any case, I don't know enough about
this part of the code to answer this.

> Is there a way to diagnose that kind of thing? I stumbled on a similar
> behaviour in other contexts and was unable to investigate it (in my case, big
> debug versions of Qt libraries are often downloaded, although I neved
> requested any debugging version of anything).

Usually, there is `guix graph --path package1 package2', which explains
why package2 is installed along with package1. I couldn't get any
meaningful result in this case.

>> If that's a common request, we could add a `texlive-collection-foo-doc'
>> package that would propagate all "doc" outputs from all packages
>> included in `texlive-collection-foo'.
>> 
>> However, I'm a bit reluctant to add more artificial packages (i.e., not
>> known to TeX Live distribution). Also, it might be as simple to do it in
>> one's own manifest.
>
> I think it would make much more sense to have "doc" outputs also for
> collections and schemes. It would be consistent with the structure of
> individual packages and would not require artificial packages.

I disagree. Collections are meta-packages. There is no documentation,
nor content, attached to them. Moreover Guix meta-packages do nothing
special about the documentation of packages they propagate. This would
be inconsistent.

> Having individual package documentations in one's manifests is of course
> doable but it is contradictory with the approach of collections.

How so?

In any case, I suggest to write a proper bug report for this. Hopefully,
someone with better understanding about the implications of GUIX_TEXMF
will be able to solve this.

Regards,
-- 
Nicolas Goaziou

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hello,

Thanks for the instructive feedback!

De Nicolas Goaziou le 28/08/2023 à 13:01:
> Every texlive-* package comes with its documentation, in a separate
> output. "doc" output are not uncommon at all in Guix. Therefore,
> I disagree with the inconsistency you're talking about.

Ok, I admit I didn't investigate much before asserting that! I just observed
that most of the tools I use come with their man or info pages in the main
output and extrapolated from that.

> > I think it would make much more sense to have "doc" outputs also for
> > collections and schemes. It would be consistent with the structure of
> > individual packages and would not require artificial packages.
> 
> I disagree. Collections are meta-packages. There is no documentation,
> nor content, attached to them. Moreover Guix meta-packages do nothing
> special about the documentation of packages they propagate. This would
> be inconsistent.

I don't understand how "out" and "doc" are different in this respect. The
"out" output of a collection meta-package has no content of its own and it
only serves to gather the "out" outputs of its inputs. Similarly, the "doc"
output would have no content of its own and only gather the "doc" outputs of
its inputs. How is that inconsistent?

There may be something I misunderstand about how Guix packages work here.

> > Having individual package documentations in one's manifests is of course
> > doable but it is contradictory with the approach of collections.
> 
> How so?

The point of a collection is to bring a meaningful set of packages on a
general topic without having to worry about its specific contents: the exact
list of packages may evolve from one version to another, it may contain many
things that are indirectly required, etc. I think it makes sense to be able to
request such an collection of packages with its documentation, and that should
not involve listing all the packages individually.

I realise it could also be possible to program that with something like

    (map (lambda (p) (list p "doc"))
         (filter (lambda (p) (member "doc" (package-outputs p)))
                 (map cadr (package-transitive-inputs
                             texlive-scheme-medium))))

in the definition of a manifest but it feels a bit low-level. But it is nice
that doing this kind of thing possible, by the way!

> In any case, I suggest to write a proper bug report for this. Hopefully,
> someone with better understanding about the implications of GUIX_TEXMF
> will be able to solve this.

I can do that for the texdoc behaviour.

-- 
Emmanuel

Re: documentation in TeX Live collections

[Nicolas Goaziou]

Emmanuel Beffara <manu@beffara.org> writes:

> I don't understand how "out" and "doc" are different in this respect. The
> "out" output of a collection meta-package has no content of its own and it
> only serves to gather the "out" outputs of its inputs. Similarly, the "doc"
> output would have no content of its own and only gather the "doc" outputs of
> its inputs. How is that inconsistent?
>
> There may be something I misunderstand about how Guix packages work
> here.

Outputs are used to split files to be installed after building
a package. Since meta-packages do not build anything, there is nothing
to install, and therefore, to split. The default output is enough.

I imagine it would be possible to bend that concept, and, for example,
create a tree of symlinks, pointing to the documentation of the various
propagated packages, that would ultimately be moved to a "doc" output.
AFAICT, however, no package in Guix does this.

Another data point to consider: `texlive-collection-foo' and
hypothetical `texlive-collection-foo:doc' would require to propagate two
different sets of packages, which may be an argument in favor of
creating two different packages in the first place.

Please note that I have no strong opinion on that subject anyway. I hope
experienced TeX Live users can chime in.

>> In any case, I suggest to write a proper bug report for this. Hopefully,
>> someone with better understanding about the implications of GUIX_TEXMF
>> will be able to solve this.
>
> I can do that for the texdoc behaviour.

Great! Thank you.

-- 
Nicolas Goaziou

Re: documentation in TeX Live collections

[Nicolas Goaziou]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Emmanuel Beffara <manu@beffara.org> writes:

>>> In any case, I suggest to write a proper bug report for this. Hopefully,
>>> someone with better understanding about the implications of GUIX_TEXMF
>>> will be able to solve this.
>>
>> I can do that for the texdoc behaviour.

By the way, I think I fixed `texdoc' utility. Would you care testing it
after a fresh `guix pull'?

Re: documentation in TeX Live collections

[Andreas Enge]

Hello,

Am Mon, Aug 28, 2023 at 06:54:35PM +0200 schrieb Nicolas Goaziou:
> Emmanuel Beffara <manu@beffara.org> writes:
> > I don't understand how "out" and "doc" are different in this respect. The
> > "out" output of a collection meta-package has no content of its own and it
> > only serves to gather the "out" outputs of its inputs. Similarly, the "doc"
> > output would have no content of its own and only gather the "doc" outputs of
> > its inputs. How is that inconsistent?
> >
> Outputs are used to split files to be installed after building
> a package. Since meta-packages do not build anything, there is nothing
> to install, and therefore, to split. The default output is enough.

if I understand things correctly, we would like the following behaviour
for propagated inputs in the texlive context:
We have these metapackages with propagated inputs; all of these inputs
have "out" and "doc". Then we would like to automatically create "out"
and "doc" for the metapackage, into which the corresponding "out" and
"doc" of their "ingredients" are propagated.

Well, more precisely, the metapackages are empty, so it is a bit fuzzy
what I mean by "into which" above.

We would like the following:
- If a user installs metapackage:out, they get all the ingredient:out
  in their profile.
- If a user installs metapackage:doc, they get all the ingredient:doc
  in their profile.
I am quite certain this is not how propagated inputs work, and I do not
know whether their behaviour could be changed in this way.

The documentation is a bit unclear:
   https://guix.gnu.org/de/manual/devel/en/guix.html#package_002dpropagated_002dinputs
"propagated-inputs is similar to inputs, but the specified packages will be automatically installed to profiles"
What is a "package" in this context? I think it means all outputs of
a package. But then we should already have all the documentation with
the metapackages, right? And indeed, when installing texlive-scheme-medium
into my profile, I have lots of downloads such as
 texlive-tex-ini-files-66594  3KiB                                   452KiB/s 00:00 ▕██████████████████▏ 100.0%
 texlive-tex-ini-files-66594-doc  1KiB                               257KiB/s 00:00 ▕██████████████████▏ 100.0%
(every package twice with its -doc).
So as a first observation, separating the doc output serves no purpose:
it will be downloaded anyway, and actually forms the bulk of the whole
texmf-dist. The above package is not typical in this respect, here is
another one:
 texlive-upmendex-66594  77B                                          33KiB/s 00:00 ▕██████████████████▏ 100.0%
 texlive-upmendex-66594-doc  945KiB                                  2.0MiB/s 00:00 ▕██████████████████▏ 100.0%

But strangely, $HOME/.guix-profile/share/texmf-dist/doc is just a pointer to
   /gnu/store/a184f1m1mbwkccxyi86dn4mdamay6lw5-texlive-bin-20230313/share/texmf-dist/doc

However, the doc output of texlive-tex-ini-files has a share/temxf-dist/doc
with a subdirectory generic/, which thus does not appear in the profile.

See also
   https://issues.guix.gnu.org/65550

I do not really understand what is happening. All outputs are downloaded,
but only the out outputs are propagated?

If this is true, then I think it would make sense to not split into two
outputs, but to always include the documentation in the texlive packages.

Andreas

PS: Something else that is strange: I end up with
    $HOME/.guix-profile/share/texmf (without the -dist suffix) that points to
    /gnu/store/lzq5fd5b2l3341s0da5a1vzhxc1li3yb-asymptote-2.86/share/texmf

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hi,
 
De Nicolas Goaziou le 28/08/2023 à 20:01:
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> >>> In any case, I suggest to write a proper bug report for this. Hopefully,
> >>> someone with better understanding about the implications of GUIX_TEXMF
> >>> will be able to solve this.
> >>
> >> I can do that for the texdoc behaviour.
> 
> By the way, I think I fixed `texdoc' utility. Would you care testing it
> after a fresh `guix pull'?

It seems to work fine: now texdoc does find all documentation in the various
items in GUIX_TEXMF. So there seems to be no need for a texdoc bug report
anymore, thanks! 

The only subtlety is that any call to guix shell with extra texlive packages
should also include texlive-bin (directly or not) so that the proper hooks are
called, otherwise GUIX_TEXMF is not updated. This goes beyond the point of
documentation, of course.

-- 
Emmanuel

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hellon

De Andreas Enge le 28/08/2023 à 20:05:
> if I understand things correctly, we would like the following behaviour
> for propagated inputs in the texlive context:
> We have these metapackages with propagated inputs; all of these inputs
> have "out" and "doc". Then we would like to automatically create "out"
> and "doc" for the metapackage, into which the corresponding "out" and
> "doc" of their "ingredients" are propagated.
> 
> Well, more precisely, the metapackages are empty, so it is a bit fuzzy
> what I mean by "into which" above.
> 
> We would like the following:
> - If a user installs metapackage:out, they get all the ingredient:out
>   in their profile.
> - If a user installs metapackage:doc, they get all the ingredient:doc
>   in their profile.
> I am quite certain this is not how propagated inputs work, and I do not
> know whether their behaviour could be changed in this way.

Indeed, that is what I imagine. Actually, it does not feel specific to the
texlive context and could apply to propagated inputs in general. 

[...]
> See also
>    https://issues.guix.gnu.org/65550

Interesting! The case of development outputs referred to in this issue
suggests similar concerns.

> I do not really understand what is happening. All outputs are downloaded,
> but only the out outputs are propagated?

It does look like that: all outputs of the propagated inputs are downloaded
but only the "out" outputs are actually propagated to the profile. And indeed,
after guix gc, the other outputs are removed from the store. Is that the
intended behaviour?

> If this is true, then I think it would make sense to not split into two
> outputs, but to always include the documentation in the texlive packages.

I'm not sure about that. Splitting the documentation to a different output
does make sense in itself (for considerations of space, mostly). This thing
about propagating only "out" outputs looks more like an issue with package
definitions, or even the package model if propagating other outputs happens to
be unsupported.

-- 
Emmanuel