Manual PDF and translation (modular texlive?)

Manual PDF and translation (modular texlive?)

[zimoun]

Dear,

Currently it is not easy to produce the PDF of the manual.  For
reference, see [1].  There are 2 issues:

 a) the ’texlive’ package.
 b) the fonts about Russian or Chinese.

About the a), it is really painful to download the *big* texlive package
to be able to compile TeX.  Especially when we have modular texlive
packages.  However, it is not clear to me which packages I have to use.
Any help is welcome. :-)

About the b), I am maybe overlooking but if I read correctly, it is not
documented.  What are the packages for the correct rendering of these
both languages.


With these, it becomes easy to have a manifest file and then:

  make pdf

will work without wizard-ness.

Thank you in advance.
All the best,
simon

1: <http://issues.guix.gnu.org/issue/26604#6>

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Currently it is not easy to produce the PDF of the manual.  For
> reference, see [1].  There are 2 issues:
>
>  a) the ’texlive’ package.
>  b) the fonts about Russian or Chinese.
>
> About the a), it is really painful to download the *big* texlive package
> to be able to compile TeX.  Especially when we have modular texlive
> packages.  However, it is not clear to me which packages I have to use.
> Any help is welcome. :-)

I tried this:

    guix environment --pure guix --ad-hoc -e '(begin (use-modules (gnu packages tex))(texlive-union (list texlive-epsf texlive-fonts-ec texlive-amsfonts)))'

This allows me to build doc/guix.pdf, but it chokes on the French and
German versions (at least).  Frustratingly, texi2dvi swallows all errors
and just prints:

    pdftex exited with bad status, quitting.

Thanks, I guess.

So I ran this in strace to see what pdftex actually complained about,
and of course it’s mangled accented characters that it cannot find in
the generated fonts.  Here’s an excerpt:

--8<---------------cut here---------------start------------->8---
31695 read(3, "333)x433.62, glue set 4.13036\n.@glue(@leftskip) 86.72375\n.@texttt \"\n.@texttt /\n.@texttt d\n.@texttt e\n.etc.\n\n[177] l.12161: Undefined cross reference `Syst^^c3^^a8mes de fichiers-snt'. l.12161: Undefined cross reference `Syst^^c3^^a8mes de fichiers-snt'. l.12161: Undefined cross reference `Syst^^c3^^a8mes de fichiers-pg'.\nMissing character: There is no ^^c3 in font cmr10!\nMissing character: There is no ^^a9 in font cmr10!\n l.12193: Undefined cross reference `Pr^^c3^^a9parer l'installation-snt'. l.12193: Undefined cross reference `Pr^^c3^^a9parer l'installation-snt'. l.12193: Undefined cross reference `Pr^^c3^^a9parer l'installation-pg'. l.12206: Undefined cross reference `Syst^^c3^^a8mes de fichiers-snt'. l.12206: Undefined cross reference `Syst^^c3^^a8mes de fichiers-snt'. l.12206: Undefined cross reference `Syst^^c3^^a8mes de fichiers-pg'. [178]\nOverfull \\hbox (32.18782pt too wide) in paragraph at lines 12228--12228\n []                          @texttt \"video\"   ;p^^Seriph^^Seriques r^^Seseaux comme les webc"..., 98304) = 86500
--8<---------------cut here---------------end--------------->8---

I’m very bad at fixing TeX font problems and encoding problems.  You can
tell by looking at the generated output when using texlive-union,
because it seems to be generating fonts anew even though we should
already have them all.

Any help with the TeX Live stuff would be appreciated!  It really needs
some love, and I can no longer give it; we have drifted apart.

I know of some unfixed problems that just need work (essentially just
looking at the generated files in the “texlive-{font,latex,tex,generic}-*”
packages and ensuring that they contain all files they should according
to texlive.tlpdb), but then there are also some issues that I don’t
understand enough to fix.

I’ll be glad to assist any one who would like to lead the effort to fix
our modular TeX Live packages once (or twice) and for all.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> zimoun <zimon.toutoune@gmail.com> writes:
>
>> Currently it is not easy to produce the PDF of the manual.  For
>> reference, see [1].  There are 2 issues:
>>
>>  a) the ’texlive’ package.
>>  b) the fonts about Russian or Chinese.
>>
>> About the a), it is really painful to download the *big* texlive package
>> to be able to compile TeX.  Especially when we have modular texlive
>> packages.  However, it is not clear to me which packages I have to use.
>> Any help is welcome. :-)
>
> I tried this:
>
>     guix environment --pure guix --ad-hoc -e '(begin (use-modules (gnu packages tex))(texlive-union (list texlive-epsf texlive-fonts-ec texlive-amsfonts)))'
>
> This allows me to build doc/guix.pdf, but it chokes on the French and
> German versions (at least).  Frustratingly, texi2dvi swallows all errors
> and just prints:
>
>     pdftex exited with bad status, quitting.
>
> Thanks, I guess.

I can now build the French and the German manuals in this environment:

    guix environment --pure guix --ad-hoc -e '(begin (use-modules (gnu packages tex))(texlive-union (list texlive-epsf texlive-fonts-ec texlive-amsfonts texlive-tex-texinfo)))'

Unfortunately, the accents and umlauts in the table of contents are
still wrong, but they are fine everywhere else.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[zimoun]

Hi Ricardo,

Thank you for looking at this.

On Fri, 16 Oct 2020 at 22:10, Ricardo Wurmus <rekado@elephly.net> wrote:

> I can now build the French and the German manuals in this environment:
>
>     guix environment --pure guix --ad-hoc -e '(begin (use-modules (gnu packages tex))(texlive-union (list texlive-epsf texlive-fonts-ec texlive-amsfonts texlive-tex-texinfo)))'
>
> Unfortunately, the accents and umlauts in the table of contents are
> still wrong, but they are fine everywhere else.

Working on fixing this –– at least the French one ;-) –– I notice that:

   guix environment -C --ad-hoc texinfo -- texi2dvi --help

raises an error about ’sed’ is missing.  Adding ’sed’, it still raises
an error about ’cat’ (coreutils, right?).  These 2 dependencies should
be added as ’propagated-inputs’, right?



Well, back to the real issue.  I do not know if the issue comes from TeX
configuration or from ’makeinfo’.  For example,

--8<---------------cut here---------------start------------->8---
mkdir -p /tmp/test && cd /tmp/test

cp ~/guix/doc/{guix.fr,os-*,fdl-1.3,contributing.fr,version-fr}.texi .
cp -R ~/guix/doc/images images
cp ~/guix/doc/environment-gdb.scm .
cp ~/guix/doc/package-hello.{scm,json} .

guix environment guix -C \
     --ad-hoc texlive-base texlive-fonts-ec texlive-epsf texlive-tex-texinfo \
     -- makeinfo --pdf guix.fr.texi
--8<---------------cut here---------------end--------------->8---

lead to incorrect Table of Contents but the rest is good (at least what
I checked).  It seems as ’makeinfo’ incorrectly deal with the unicode in
the master menu.  On the other hand,

  guix environment guix -C --ad-hoc texlive -- makeinfo --pdf guix.fr.pdf

works correctly.  Hum?  I do not know…


All the best,
simon

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Well, back to the real issue.  I do not know if the issue comes from TeX
> configuration or from ’makeinfo’.  For example,
>
> --8<---------------cut here---------------start------------->8---
> mkdir -p /tmp/test && cd /tmp/test
>
> cp ~/guix/doc/{guix.fr,os-*,fdl-1.3,contributing.fr,version-fr}.texi .
> cp -R ~/guix/doc/images images
> cp ~/guix/doc/environment-gdb.scm .
> cp ~/guix/doc/package-hello.{scm,json} .
>
> guix environment guix -C \
>      --ad-hoc texlive-base texlive-fonts-ec texlive-epsf texlive-tex-texinfo \
>      -- makeinfo --pdf guix.fr.texi
> --8<---------------cut here---------------end--------------->8---
>
> lead to incorrect Table of Contents but the rest is good (at least what
> I checked).  It seems as ’makeinfo’ incorrectly deal with the unicode in
> the master menu.  On the other hand,
>
>   guix environment guix -C --ad-hoc texlive -- makeinfo --pdf guix.fr.pdf
>
> works correctly.  Hum?  I do not know…

It’s almost certainly due to configuration of the modular texlive.
We’re generating a whole bunch of font map files and encoding files
already, but something somewhere must still be missing.

I embarked on a journey to fix this and am now deep into upgrading and
adding TeX Live packages…  The first step in fixing this is to ensure
that all texlive-* packages contain the files that they should.  The old
texlive-* packages consisted simply of a single directory from the big
TeX Live SVN repository.  Since the addition of multiple SVN fetch and
simple-texlive-package using it I have added a whole bunch of new
texlive-* packages that contain all files as recorded in texlive.tlpdb.

This needs to be done for all remaining old packages (you can tell they
are old by the naming scheme: texlive-{latex,luatex,generic,fonts…}-*
instead of just texlive-[name]).

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[zimoun]

Hi Ricardo,

On Sat, 17 Oct 2020 at 23:55, Ricardo Wurmus <rekado@elephly.net> wrote:

> It’s almost certainly due to configuration of the modular texlive.
> We’re generating a whole bunch of font map files and encoding files
> already, but something somewhere must still be missing.
>
> I embarked on a journey to fix this and am now deep into upgrading and
> adding TeX Live packages…  The first step in fixing this is to ensure
> that all texlive-* packages contain the files that they should.  The old
> texlive-* packages consisted simply of a single directory from the big
> TeX Live SVN repository.  Since the addition of multiple SVN fetch and
> simple-texlive-package using it I have added a whole bunch of new
> texlive-* packages that contain all files as recorded in texlive.tlpdb.

Are you planning to commit these in the wip-texlive branch?
Is it "reasonable" to merge before the release?


> This needs to be done for all remaining old packages (you can tell they
> are old by the naming scheme: texlive-{latex,luatex,generic,fonts…}-*
> instead of just texlive-[name]).

I do not think I will have time to help in the coming days (before the
release).  But I can try to at least fix the ones required by the
manual, if you have not done yet.


Cheers,
simon

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Ricardo,
>
> On Sat, 17 Oct 2020 at 23:55, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>> It’s almost certainly due to configuration of the modular texlive.
>> We’re generating a whole bunch of font map files and encoding files
>> already, but something somewhere must still be missing.
>>
>> I embarked on a journey to fix this and am now deep into upgrading and
>> adding TeX Live packages…  The first step in fixing this is to ensure
>> that all texlive-* packages contain the files that they should.  The old
>> texlive-* packages consisted simply of a single directory from the big
>> TeX Live SVN repository.  Since the addition of multiple SVN fetch and
>> simple-texlive-package using it I have added a whole bunch of new
>> texlive-* packages that contain all files as recorded in texlive.tlpdb.
>
> Are you planning to commit these in the wip-texlive branch?

Perhaps.  I’m still in the middle of figuring things out and
experimenting.  At some point I’ll probably re-create the wip-texlive
branch, though.

> Is it "reasonable" to merge before the release?

No.  Changes to TeX would lead to a full rebuild.


-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> I can now build the French and the German manuals in this environment:
>
>     guix environment --pure guix --ad-hoc -e '(begin (use-modules (gnu packages tex))(texlive-union (list texlive-epsf texlive-fonts-ec texlive-amsfonts texlive-tex-texinfo)))'
>
> Unfortunately, the accents and umlauts in the table of contents are
> still wrong, but they are fine everywhere else.

Yeah, ‘doc/build.scm’ has this comment:

  ;; FIXME: This union works, except for the table of contents of non-English
  ;; manuals, which contains escape sequences like "^^ca^^fe" instead of
  ;; accented letters.
  ;;
  ;; (define texlive
  ;;   (texlive-union (list texlive-tex-texinfo
  ;;                        texlive-generic-epsf
  ;;                        texlive-fonts-ec)))

What’s interesting is that it breaks accents in the table of contents,
but not elsewhere.

Ludo’.

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> I can now build the French and the German manuals in this environment:
>>
>>     guix environment --pure guix --ad-hoc -e '(begin (use-modules (gnu packages tex))(texlive-union (list texlive-epsf texlive-fonts-ec texlive-amsfonts texlive-tex-texinfo)))'
>>
>> Unfortunately, the accents and umlauts in the table of contents are
>> still wrong, but they are fine everywhere else.
>
> Yeah, ‘doc/build.scm’ has this comment:
>
>   ;; FIXME: This union works, except for the table of contents of non-English
>   ;; manuals, which contains escape sequences like "^^ca^^fe" instead of
>   ;; accented letters.
>   ;;
>   ;; (define texlive
>   ;;   (texlive-union (list texlive-tex-texinfo
>   ;;                        texlive-generic-epsf
>   ;;                        texlive-fonts-ec)))
>
> What’s interesting is that it breaks accents in the table of contents,
> but not elsewhere.

These double caret sequences are representations of multi-byte
characters.  “^^c3^^b6”, for example, is a lowercase a with umlaut.

The TeX log file contains a whole bunch of these messages:

    l.139: Unicode char @u8:^^e5^^8f^^82 not defined for Texinfo

Then later things like this:

    Missing character: There is no ^^c3 in font cmr10!
    Missing character: There is no ^^9f in font cmr10!
    Missing character: There is no ^^c3 in font cmr10!
    Missing character: There is no ^^9f in font cmr10!
    Missing character: There is no ^^c3 in font cmr10!
    Missing character: There is no ^^a4 in font cmr10!

I’m not sure this is correct, because it seems to me that “^^c3” is only
part of a longer multi-byte sequence, but this error indicates that
individual bytes are looked up in the font.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>> What’s interesting is that it breaks accents in the table of contents,
>> but not elsewhere.
>
> These double caret sequences are representations of multi-byte
> characters.  “^^c3^^b6”, for example, is a lowercase a with umlaut.
>
> The TeX log file contains a whole bunch of these messages:
>
>     l.139: Unicode char @u8:^^e5^^8f^^82 not defined for Texinfo
>
> Then later things like this:
>
>     Missing character: There is no ^^c3 in font cmr10!
>     Missing character: There is no ^^9f in font cmr10!
>     Missing character: There is no ^^c3 in font cmr10!
>     Missing character: There is no ^^9f in font cmr10!
>     Missing character: There is no ^^c3 in font cmr10!
>     Missing character: There is no ^^a4 in font cmr10!
>
> I’m not sure this is correct, because it seems to me that “^^c3” is only
> part of a longer multi-byte sequence, but this error indicates that
> individual bytes are looked up in the font.

With the full “texlive” package I also see “not defined for Texinfo” in
the logs, but the characters use octal notation instead of double caret
notation.  The generated guix.de.toc contains the correct characters
with umlauts, while the .toc file generated with the modular TeX Live
contains caret-notated characters.

I’ll try to figure out why that is.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>>> What’s interesting is that it breaks accents in the table of contents,
>>> but not elsewhere.
>>
>> These double caret sequences are representations of multi-byte
>> characters.  “^^c3^^b6”, for example, is a lowercase a with umlaut.
>>
>> The TeX log file contains a whole bunch of these messages:
>>
>>     l.139: Unicode char @u8:^^e5^^8f^^82 not defined for Texinfo
>>
>> Then later things like this:
>>
>>     Missing character: There is no ^^c3 in font cmr10!
>>     Missing character: There is no ^^9f in font cmr10!
>>     Missing character: There is no ^^c3 in font cmr10!
>>     Missing character: There is no ^^9f in font cmr10!
>>     Missing character: There is no ^^c3 in font cmr10!
>>     Missing character: There is no ^^a4 in font cmr10!
>>
>> I’m not sure this is correct, because it seems to me that “^^c3” is only
>> part of a longer multi-byte sequence, but this error indicates that
>> individual bytes are looked up in the font.
>
> With the full “texlive” package I also see “not defined for Texinfo” in
> the logs, but the characters use octal notation instead of double caret
> notation.  The generated guix.de.toc contains the correct characters
> with umlauts, while the .toc file generated with the modular TeX Live
> contains caret-notated characters.
>
> I’ll try to figure out why that is.

The reason is that the generated guix.de.toc file is ASCII-encoded in
the modular case but UTF-8 encoded in the monolithic case.  Why is that?

texinfo.tex enables byte-I/O for engines that do not have native UTF-8
support; it uses native UTF-8 for LuaTeX and XeTeX only.  Sure enough,
with

    PDFTEX=xetex make doc/guix.de.pdf

the TOC looks actually fine!  LuaTeX is broken due to a botched upgrade
(I’m working on a fix), so I haven’t tested it.

Two things are weird here:

1) texi2dvi still fails, because apparently “xetex” didn’t return a good
status code; the PDF was built fine, though.

2) we aren’t using XeTeX or LuaTeX with the monolithic “texlive”
package, so why does pdfTeX behave differently here?  I see in the logs
that the date of the format file differs — does this indicate that our
pdfTeX format file is wrong?  I will compare the two files.

Another observation: the pdftex.map file in the monolithic “texlive”
package is huge and mentions a great many fonts; in the modular TeX Live
this is generated for fonts that are actually available.  It’s not
impossible that this font map needs more entries, but perhaps everything
is fine already.  I just can’t say for sure.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> 2) we aren’t using XeTeX or LuaTeX with the monolithic “texlive”
> package, so why does pdfTeX behave differently here?  I see in the logs
> that the date of the format file differs — does this indicate that our
> pdfTeX format file is wrong?  I will compare the two files.

The comparison with diffoscope wasn’t very helpful because the file
embeds countless store directory names and differs everywhere.

So I just copied the known-good pdftex.fmt from the monolithic package
and added it to the union: the TOC looks fine!  (The build with “make
doc/guix.de.pdf” still fails for unknown reasons, but it produces a fine
PDF file.)

So: what’s wrong with our pdftex.fmt?  It’s not clear how the file is
generated in the TeX Live repository.  People are probably expected to
just copy it, but we’re generating it from source with the following
code in “texlive-latex-base”:

--8<---------------cut here---------------start------------->8---
…

                 ;; XXX: We can't build all formats at this point, nor are they
                 ;; part of the LaTeX base, so we disable them.  Actually, we
                 ;; should be running this all in a profile hook, so that only
                 ;; selected formats and hyphenation patterns are included, but it
                 ;; takes long and TeX Live isn't designed to be modular like
                 ;; that.  Everything operates on a shared directory, which we
                 ;; would only have at profile generation time.
                 (let ((disabled-formats
                        '("aleph aleph" "lamed aleph" "uptex uptex" "euptex euptex"
                          "eptex eptex" "ptex ptex" "pdfxmltex pdftex" "platex eptex"
                          "csplain pdftex" "mf mf-nowin" "mex pdftex" "pdfmex pdftex"
                          "luacsplain luatex"
                          "cont-en xetex" "cont-en pdftex" "pdfcsplain xetex"
                          "pdfcsplain pdftex" "pdfcsplain luatex" "cslatex pdftex"
                          "mptopdf pdftex" "uplatex euptex" "jadetex pdftex"
                          "amstex pdftex" "pdfcslatex pdftex" "lollipop tex"
                          "xmltex pdftex" "pdfjadetex pdftex" "eplain pdftex"
                          "texsis pdftex" "mltex pdftex" "utf8mex pdftex")))
                   (mkdir "web2c")
                   (install-file (string-append
                                  (assoc-ref inputs "texlive-kpathsea")
                                  "/share/texmf-dist/web2c/fmtutil.cnf")
                                 "web2c")
                   (make-file-writable "web2c/fmtutil.cnf")
                   (substitute* "web2c/fmtutil.cnf"
                     (((string-append "^(" (string-join disabled-formats "|") ")") m)
                      (string-append "#! " m))))
                 (invoke "fmtutil-sys" "--all"
                         "--fmtdir=web2c"
                         (string-append "--cnffile=web2c/fmtutil.cnf"))
                 ;; We don't actually want to install it.
                 (delete-file "web2c/fmtutil.cnf")
                 #t))
--8<---------------cut here---------------end--------------->8---

I suspect that the build environment doesn’t have locales set up so the
format dumping tool assumes that we want to us Latin-1 encoding for
everything.  (Roughly speaking, the fmt files are like dumped Lisp
images as I understand it.)

I’ll try to tweak the texlive-latex-base package and see if I can get it
to generate the pdftex.fmt correctly.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[zimoun]

Hi Ricardo,

Thank you so much for the detailed step by step progress and the deep
investigation.

On Thu, 22 Oct 2020 at 21:51, Ricardo Wurmus <rekado@elephly.net> wrote:

>                  ;; XXX: We can't build all formats at this point, nor are they
>                  ;; part of the LaTeX base, so we disable them.  Actually, we
>                  ;; should be running this all in a profile hook, so that only
>                  ;; selected formats and hyphenation patterns are included, but it
>                  ;; takes long and TeX Live isn't designed to be modular like
>                  ;; that.  Everything operates on a shared directory, which we
>                  ;; would only have at profile generation time.
>                  (let ((disabled-formats
>                         '("aleph aleph" "lamed aleph" "uptex uptex" "euptex euptex"
>                           "eptex eptex" "ptex ptex" "pdfxmltex pdftex" "platex eptex"
>                           "csplain pdftex" "mf mf-nowin" "mex pdftex" "pdfmex pdftex"
>                           "luacsplain luatex"
>                           "cont-en xetex" "cont-en pdftex" "pdfcsplain xetex"
>                           "pdfcsplain pdftex" "pdfcsplain luatex" "cslatex pdftex"
>                           "mptopdf pdftex" "uplatex euptex" "jadetex pdftex"
>                           "amstex pdftex" "pdfcslatex pdftex" "lollipop tex"
>                           "xmltex pdftex" "pdfjadetex pdftex" "eplain pdftex"
>                           "texsis pdftex" "mltex pdftex" "utf8mex pdftex")))

[...]

> I suspect that the build environment doesn’t have locales set up so the
> format dumping tool assumes that we want to us Latin-1 encoding for
> everything.  (Roughly speaking, the fmt files are like dumped Lisp
> images as I understand it.)

I am totally naive here.  Is it not one of these disabled formats
which should not be?


Cheers,
simon

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> On Thu, 22 Oct 2020 at 21:51, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>>                  ;; XXX: We can't build all formats at this point, nor are they
>>                  ;; part of the LaTeX base, so we disable them.  Actually, we
>>                  ;; should be running this all in a profile hook, so that only
>>                  ;; selected formats and hyphenation patterns are included, but it
>>                  ;; takes long and TeX Live isn't designed to be modular like
>>                  ;; that.  Everything operates on a shared directory, which we
>>                  ;; would only have at profile generation time.
>>                  (let ((disabled-formats
>>                         '("aleph aleph" "lamed aleph" "uptex uptex" "euptex euptex"
>>                           "eptex eptex" "ptex ptex" "pdfxmltex pdftex" "platex eptex"
>>                           "csplain pdftex" "mf mf-nowin" "mex pdftex" "pdfmex pdftex"
>>                           "luacsplain luatex"
>>                           "cont-en xetex" "cont-en pdftex" "pdfcsplain xetex"
>>                           "pdfcsplain pdftex" "pdfcsplain luatex" "cslatex pdftex"
>>                           "mptopdf pdftex" "uplatex euptex" "jadetex pdftex"
>>                           "amstex pdftex" "pdfcslatex pdftex" "lollipop tex"
>>                           "xmltex pdftex" "pdfjadetex pdftex" "eplain pdftex"
>>                           "texsis pdftex" "mltex pdftex" "utf8mex pdftex")))
>
> [...]
>
>> I suspect that the build environment doesn’t have locales set up so the
>> format dumping tool assumes that we want to us Latin-1 encoding for
>> everything.  (Roughly speaking, the fmt files are like dumped Lisp
>> images as I understand it.)
>
> I am totally naive here.  Is it not one of these disabled formats
> which should not be?

No, the pdftex formats are:

--8<---------------cut here---------------start------------->8---
# from pdftex:
pdftex pdftex language.def -translate-file=cp227.tcx *pdfetex.ini
etex pdftex language.def -translate-file=cp227.tcx *etex.ini
pdfetex pdftex language.def -translate-file=cp227.tcx *pdfetex.ini
--8<---------------cut here---------------end--------------->8---

That’s copied from fmtutil.cfg.  The first word on each line is the
format name, the second is the TeX engine.  In the list of disabled
formats you see a whole bunch of formats that use the pdftex engine, but
they do not result in pdftex.fmt.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> I suspect that the build environment doesn’t have locales set up so the
> format dumping tool assumes that we want to us Latin-1 encoding for
> everything.

It’s not that, but I have another clue.

This is the relevant line from fmtutil.cfg:

    pdftex pdftex language.def -translate-file=cp227.tcx *pdfetex.ini

Here’s output from building pdftex.fmt (from the build log of
texlive-latex-base):

--8<---------------cut here---------------start------------->8---
fmtutil [INFO]: --- remaking pdftex with pdftex
fmtutil: running `pdftex -ini   -jobname=pdftex -progname=pdftex -translate-file=cp227.tcx *pdfetex.ini' ...
warning: Could not open char translation file `cp227.tcx'.
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019) (INITEX)
 restricted \write18 enabled.
entering extended mode
(/gnu/store/4k3na5c6wpbp87lrq7aj8bbppxl805jm-texlive-tex-plain-51265/share/texmf-dist/tex/plain/config/pdfetex.ini (/gnu/store/01fpijdjxffhq3rn8jdj2bdfmhqa56ad-texlive-tex-ini-files-51265/share/texmf-dist/tex/generic/tex-ini-files/pdftexconfig.tex) (/gnu/store/4k3na5c6wpbp87lrq7aj8bbppxl805jm-texlive-tex-plain-51265/share/texmf-dist/tex/plain/etex/etex.src (/gnu/store/4k3na5c6wpbp87lrq7aj8bbppxl805jm-texlive-tex-plain-51265/share/texmf-dist/tex/plain/base/plain.tex Preloading the plain format: codes, registers, parameters, fonts, more fonts, macros, math definitions, output routines, hyphenation (/gnu/store/awsj27jah4yiwyxz53ghdpss7v4kyspi-texlive-hyphen-base-51265/share/texmf-dist/tex/generic/hyphen/hyphen.tex [skipping from \patterns to end-of-file...])) (/gnu/store/4k3na5c6wpbp87lrq7aj8bbppxl805jm-texlive-tex-plain-51265/share/texmf-dist/tex/plain/etex/etexdefs.lib Skipping module "grouptypes"; Loading module "interactionmodes"; Skipping module "nodetypes";
Skipping module "iftypes";)
(/gnu/store/awsj27jah4yiwyxz53ghdpss7v4kyspi-texlive-hyphen-base-51265/share/texmf-dist/tex/generic/config/language.def
(/gnu/store/awsj27jah4yiwyxz53ghdpss7v4kyspi-texlive-hyphen-base-51265/share/texmf-dist/tex/generic/hyphen/hyphen.tex)
(/gnu/store/8lsza6zyrcp7iivlpb5zfzprmfsc214r-texlive-dehyph-exptl-51265/share/texmf-dist/tex/generic/dehyph-exptl/dehypht-x-2019-04-04.tex
dehyph-exptl: using an 8-bit TeX engine.
[…]
Beginning to dump on file pdftex.fmt
 (preloaded format=pdftex 2020.10.22)
5456 strings of total length 140996
13424 memory locations dumped; current usage is 203&10315
1681 multiletter control sequences
[…]
fmtutil [INFO]: /tmp/guix-build-texlive-latex-base-51265.drv-0/source/web2c/pdftex/pdftex.fmt installed.
--8<---------------cut here---------------end--------------->8---

Two things here:

“warning: Could not open char translation file `cp227.tcx'.” – Oh?  It’s
in the union, but maybe a search path in the configuration file is
misconfigured.

“dehyph-exptl: using an 8-bit TeX engine.” — hmm.  Probably correct,
because pdftex doesn’t natively support multibyte characters.  But maybe
a problem.

So the next step for me is to figure out how I can make pdftex find
cp227.tcx.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[zimoun]

On Thu, 22 Oct 2020 at 22:26, Ricardo Wurmus <rekado@elephly.net> wrote:

> > I am totally naive here.  Is it not one of these disabled formats
> > which should not be?
>
> No, the pdftex formats are:

Ok.  I am really dumb here.  That was just because the 'pdftex' engine
appears a couple of times, which is not the case of xetex ou luatex.

Re: Manual PDF and translation (modular texlive?)

[zimoun]

On Thu, 22 Oct 2020 at 22:35, Ricardo Wurmus <rekado@elephly.net> wrote:

> “warning: Could not open char translation file `cp227.tcx'.” – Oh?  It’s
> in the union, but maybe a search path in the configuration file is
> misconfigured.

Oh!  Looks like a good clue.

Re: Manual PDF and translation (modular texlive?)

[zimoun]

On Thu, 22 Oct 2020 at 22:35, Ricardo Wurmus <rekado@elephly.net> wrote:

> So the next step for me is to figure out how I can make pdftex find
> cp227.tcx.

Adding 'texlive-kpathsea' in the default packages of 'texlive-base' or
in the 'texlive-union', is it not enough?

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> On Thu, 22 Oct 2020 at 22:35, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>> So the next step for me is to figure out how I can make pdftex find
>> cp227.tcx.
>
> Adding 'texlive-kpathsea' in the default packages of 'texlive-base' or
> in the 'texlive-union', is it not enough?

I don’t think so.  This file is only used while generating the format
file.  But I’ll give it a try anyway :)

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]


Ricardo Wurmus <rekado@elephly.net> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> I suspect that the build environment doesn’t have locales set up so the
>> format dumping tool assumes that we want to us Latin-1 encoding for
>> everything.
>
> It’s not that, but I have another clue.
>
> This is the relevant line from fmtutil.cfg:
>
>     pdftex pdftex language.def -translate-file=cp227.tcx *pdfetex.ini

This patch fixes it:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-texlive-latex-base-Use-character-translation-fil.patch --]
[-- Type: text/x-patch, Size: 1340 bytes --]

From a1e4825c71308b40a0a250cdddbabf2cf534c5a0 Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <rekado@elephly.net>
Date: Sun, 25 Oct 2020 12:01:09 +0100
Subject: [PATCH] gnu: texlive-latex-base: Use character translation file.

* gnu/packages/tex.scm (texlive-latex-base)[argumenst]: Patch fmtutil.cnf to
ensure that the character translation file cp227.tcx is used during format
file generation.
---
 gnu/packages/tex.scm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/tex.scm b/gnu/packages/tex.scm
index 1cffb52410..95ec9d6ada 100644
--- a/gnu/packages/tex.scm
+++ b/gnu/packages/tex.scm
@@ -2470,7 +2470,10 @@ formats.")
                    (make-file-writable "web2c/fmtutil.cnf")
                    (substitute* "web2c/fmtutil.cnf"
                      (((string-append "^(" (string-join disabled-formats "|") ")") m)
-                      (string-append "#! " m))))
+                      (string-append "#! " m))
+                     (("translate-file=cp227")
+                      (format #f "translate-file=~a/share/texmf-dist/web2c/cp227"
+                              (assoc-ref inputs "texlive-kpathsea")))))
                  (invoke "fmtutil-sys" "--all"
                          "--fmtdir=web2c"
                          (string-append "--cnffile=web2c/fmtutil.cnf"))
-- 
2.28.0


[-- Attachment #3: Type: text/plain, Size: 828 bytes --]


With this patch the German PDF manual has no more encoding problems.
This should go to core-updates or a new texlive-upgrade branch, because
it results in countless rebuilds.

There are still remaining problems when using make doc/guix.de.pdf:

* kpathsea causes mktexpk to be run for a number of fonts that already
  exist in the union (e.g. ecrm1200).  Why aren’t they found?  Perhaps
  that’s because TeX Live only includes the font metrics (.tfm), but not
  the .pk files.  I wonder why the .pk files are used at all.

* despite successfully generating the PDF file in the temporary build
  directory, make aborts:

  /gnu/store/mw4llmn2l617gf5zakfk1l154f19lxbm-profile/bin/texi2dvi: pdftex exited with bad status, quitting.
  make: *** [Makefile:4375: doc/guix.de.pdf] Error 1

Why?

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> This patch fixes it:

[...]

> +                     (("translate-file=cp227")
> +                      (format #f "translate-file=~a/share/texmf-dist/web2c/cp227"
> +                              (assoc-ref inputs "texlive-kpathsea")))))

Woow, thumbs up.  Not something one could have guessed!

> With this patch the German PDF manual has no more encoding problems.
> This should go to core-updates or a new texlive-upgrade branch, because
> it results in countless rebuilds.

OK.

> There are still remaining problems when using make doc/guix.de.pdf:
>
> * kpathsea causes mktexpk to be run for a number of fonts that already
>   exist in the union (e.g. ecrm1200).  Why aren’t they found?  Perhaps
>   that’s because TeX Live only includes the font metrics (.tfm), but not
>   the .pk files.  I wonder why the .pk files are used at all.
>
> * despite successfully generating the PDF file in the temporary build
>   directory, make aborts:
>
>   /gnu/store/mw4llmn2l617gf5zakfk1l154f19lxbm-profile/bin/texi2dvi: pdftex exited with bad status, quitting.
>   make: *** [Makefile:4375: doc/guix.de.pdf] Error 1

‘texi2dvi’ swallows error messages; I’m not sure it can be made more
verbose, but I remember debugging it through strace (!) where I could
see the underlying messages, which sometimes contain hints if you’re
able to read between the lines.

Kudos on the progress you made!

Thanks,
Ludo’.

Re: Manual PDF and translation (modular texlive?)

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> This patch fixes it:
>
> [...]
>
>> +                     (("translate-file=cp227")
>> +                      (format #f "translate-file=~a/share/texmf-dist/web2c/cp227"
>> +                              (assoc-ref inputs "texlive-kpathsea")))))
>
> Woow, thumbs up.  Not something one could have guessed!

Hah, when I originally added the code to build the fmt files this
problem did stick out to me, but I thought it would be solved eventually
by modifying search paths in the configuration file.

It’s odd that the file cannot be found when it is already part of the
texlive-union and the search paths include all sub-directories.  But I
don’t want to investigate further when patching gets us to the correct
output faster :)

>> There are still remaining problems when using make doc/guix.de.pdf:
>>
>> * kpathsea causes mktexpk to be run for a number of fonts that already
>>   exist in the union (e.g. ecrm1200).  Why aren’t they found?  Perhaps
>>   that’s because TeX Live only includes the font metrics (.tfm), but not
>>   the .pk files.  I wonder why the .pk files are used at all.
>>
>> * despite successfully generating the PDF file in the temporary build
>>   directory, make aborts:
>>
>>   /gnu/store/mw4llmn2l617gf5zakfk1l154f19lxbm-profile/bin/texi2dvi: pdftex exited with bad status, quitting.
>>   make: *** [Makefile:4375: doc/guix.de.pdf] Error 1
>
> ‘texi2dvi’ swallows error messages; I’m not sure it can be made more
> verbose, but I remember debugging it through strace (!) where I could
> see the underlying messages, which sometimes contain hints if you’re
> able to read between the lines.

I used strace as well before I found that the log files are kept in the
temporary build directory under doc/guix.de.t2p.  Later I found that
“make V=1 doc/guix.de.pdf” shows the TeX output, which is also helpful.
Still, it’s very noisy and it’s not clear which of the many messages
correspond to the fatal error.  Oh well.

The .pk files are bitmap fonts that are generated when T1 fonts are not
found.  Adding texlive-cm-super to the union avoids generation of bitmap
fonts for the CM fonts (which are the default when no other fonts are
found), so that solves most of the problems and allows me to remove all
of those (setenv "HOME" "/tmp") hacks in packages that use the
texlive-union.  That’s now implemented on the wip-texlive branch.

The French manual can be built in PDF format, so I’m suspecting a
problem with the German texi sources that turns fatal when building the
PDF.

-- 
Ricardo

Re: Manual PDF and translation (modular texlive?)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

Hello Ricardo,

I've a huge backlog of messages from guix, I've read this thread today
and I just want to say: thank you so much for your work on this!!!

I don't (still) know if this work on modular texlive is done or not,
anyway you solved a lot of issues and - as we can see - texlive
workflows are very very hard to debug.

Reproducible TeX documents with modular packages is a really great great
feature.

[...]

Thanks! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]