[core-updates] texlive 2021 bootstrap problem

[core-updates] texlive 2021 bootstrap problem

[Thiago Jung Bauermann]

Hello,

In an attempt to verify whether bug 48064¹ (”texlive-* packages fail to 
build non-deterministically”) is fixed by the current TeX Live version, I’ve 
been working on updating it to version 2021.1².

Unfortunately I ran into a bootstrapping issue: texlive-latex-base now 
depends on texlive-latex-l3kernel, otherwise its build fails with:

LaTeX2e <2020-10-01> patch level 4
! LaTeX requires expl3.

The problem is that if I add texlive-latex-l3kernel to the propagated-
inputs list of texlive-latex-base, then a dependency loop is formed since 
texlive-latex-l3kernel uses the texlive-build-system, which adds texlive-
latex-base as an implicit input. And if I understand thing correctly, 
l3kernel does indeed need some form of TeX or LaTeX in order to run
`source/latex/l3kernel/l3.ins`.

I tried to “cheat” and added the build argument
`#:texlive-latex-base ,texlive-bin` to texlive-latex-l3kernel (since 
texlive-bin doesn’t depend on texlive-latex-base), but then I get an error 
about not being able to find `luatex.fmt`.

Looking around the web, I found this answer:

> To answer your question about deactivation ... the L3 programming layer 
> is part of the core LaTeX format since release 2019-10-01 and it can't be 
> deactivated. At the moment it is there to speed up processing of packages 
> written in expl3, e.g., xparse, fontspec, and many others, but going 
> forward core parts of LaTeX will make direct use of it.
> 
> Deactivation is only possible by using a release prior to 2019-10 (even 
> rollback via "latexrelease" will load).
— https://tex.stackexchange.com/a/538716

Does anyone have an idea how to break this dependency loop? Do we need to 
have a package definition for a pre-2019-10 version of LaTeX just to build 
texlive-latex-l3kernel, and then use that as a dependency for the actual 
texlive-latex-base package?

The only alternative I can think of is trying to build a minimal version of 
2021.1 LaTeX which somehow doesn’t use expl3 so that it can be used to 
build texlive-latex-l3kernel. But that is above my very superficial 
understanding of TeX and LaTeX. And the quote I pasted above doesn’t look 
very encouraging.

If anyone wants to have a look at what I have so far, I pushed my patch to 
branch “bug-48064-texlive-2021” at https://gitlab.com/bauermann/guix .

-- 
Thanks,
Thiago


¹ https://issues.guix.gnu.org/48064
² Well, almost. In order to have a stable URL to fetch from, I’m using the 
latest version available in the historic repository, which is slightly 
older than 2021.1.

Re: [core-updates] texlive 2021 bootstrap problem

[Thiago Jung Bauermann]

Hello,

Em terça-feira, 15 de junho de 2021, às 16:00:51 -03, Thiago Jung Bauermann 
escreveu:
> Does anyone have an idea how to break this dependency loop? Do we need to
> have a package definition for a pre-2019-10 version of LaTeX just to
> build texlive-latex-l3kernel, and then use that as a dependency for the
> actual texlive-latex-base package?

I’m pursuing this option. I took the definition of texlive-latex-base from 
the master branch (which doesn’t depend on LaTeX3) and added it to my 
branch as texlive-latex-no-l3 (with a few minor changes to get it to build 
with texlive 2021), and used it as a dependency to build texlive-latex-
l3kernel.

This worked! Now texlive-latex-l3kernel builds correctly, and the build of 
texlive-latex-base is attempted (instead of just causing Guix to max out 
its heap while trying to process a dependency loop, as it was doing 
before). But the build fails when generating some latex formats – a problem 
I’m still investigating.

But this is progress nevertheless. :-)

> If anyone wants to have a look at what I have so far, I pushed my patch
> to branch “bug-48064-texlive-2021” at https://gitlab.com/bauermann/guix

I just force-pushed this branch with my current version of the patches.

-- 
Thanks,
Thiago

Re: [core-updates] texlive 2021 bootstrap problem

[Maxim Cournoyer]

Hi Thiago,

Thiago Jung Bauermann <bauermann@kolabnow.com> writes:

> Hello,
>
> Em terça-feira, 15 de junho de 2021, às 16:00:51 -03, Thiago Jung Bauermann 
> escreveu:
>> Does anyone have an idea how to break this dependency loop? Do we need to
>> have a package definition for a pre-2019-10 version of LaTeX just to
>> build texlive-latex-l3kernel, and then use that as a dependency for the
>> actual texlive-latex-base package?
>
> I’m pursuing this option. I took the definition of texlive-latex-base from 
> the master branch (which doesn’t depend on LaTeX3) and added it to my 
> branch as texlive-latex-no-l3 (with a few minor changes to get it to build 
> with texlive 2021), and used it as a dependency to build texlive-latex-
> l3kernel.
>
> This worked! Now texlive-latex-l3kernel builds correctly, and the build of 
> texlive-latex-base is attempted (instead of just causing Guix to max out 
> its heap while trying to process a dependency loop, as it was doing 
> before). But the build fails when generating some latex formats – a problem 
> I’m still investigating.
>
> But this is progress nevertheless. :-)

Thumbs up for working and figuring things out for a texlive 2021
upgrade!

Thank you,

Maxim

Re: [core-updates] texlive 2021 bootstrap problem

[Thiago Jung Bauermann]

Hello Maxim,

Em quinta-feira, 24 de junho de 2021, às 15:26:55 -03, Maxim Cournoyer 
escreveu:
> Hi Thiago,
> 
> Thiago Jung Bauermann <bauermann@kolabnow.com> writes:
> > Hello,
> > 
> > Em terça-feira, 15 de junho de 2021, às 16:00:51 -03, Thiago Jung
> > Bauermann> 
> > escreveu:
> >> Does anyone have an idea how to break this dependency loop? Do we need
> >> to have a package definition for a pre-2019-10 version of LaTeX just
> >> to build texlive-latex-l3kernel, and then use that as a dependency
> >> for the actual texlive-latex-base package?
> > 
> > I’m pursuing this option. I took the definition of texlive-latex-base
> > from the master branch (which doesn’t depend on LaTeX3) and added it
> > to my branch as texlive-latex-no-l3 (with a few minor changes to get
> > it to build with texlive 2021), and used it as a dependency to build
> > texlive-latex- l3kernel.
> > 
> > This worked! Now texlive-latex-l3kernel builds correctly, and the build
> > of texlive-latex-base is attempted (instead of just causing Guix to
> > max out its heap while trying to process a dependency loop, as it was
> > doing before). But the build fails when generating some latex formats
> > – a problem I’m still investigating.
> > 
> > But this is progress nevertheless. :-)
> 
> Thumbs up for working and figuring things out for a texlive 2021
> upgrade!

Thank you! I was just going to report on recent developments.
It turns out that using an older LaTeX to build texlive-latex-l3kernel 
isn’t necessary.

I decided to ask on the TeX Live mailing list about the bootstrap problem¹ 
(not sure why this didn’t occur to me earlier). It turns out it’s not a 
real circular dependency, it just appears so because of Guix packaging.

It’s possible to build texlive-latex-l3kernel with IniTeX (i.e., by calling 
`pdftex -ini` or `tex -ini`) and docstrip, without any format loaded.

So I made some changes to texlive-build-system to that, and it worked. By 
which I mean: the texlive-latex-l3kernel package builds, and the texlive-
latex-base build is attempted and fails in the same way as it was failing
when I sent the last email, when generating latex formats. I think I’m 
missing some file(s) with definitions for LaTeX macros, but I don’t know yet.

I pushed a new branch “bug-48064-texlive-2021-l3kernel-initex” to my GitLab 
repo at https://gitlab.com/bauermann/guix.git with two patches implementing 
the new approach.

-- 
Thanks,
Thiago

¹ https://tug.org/pipermail/tex-live/2021-June/047156.html

Re: [core-updates] texlive 2021 bootstrap problem

[Maxim Cournoyer]

Hello Thiago,

Thiago Jung Bauermann <bauermann@kolabnow.com> writes:

[...]

> I decided to ask on the TeX Live mailing list about the bootstrap problem¹ 
> (not sure why this didn’t occur to me earlier). It turns out it’s not a 
> real circular dependency, it just appears so because of Guix packaging.
>
> It’s possible to build texlive-latex-l3kernel with IniTeX (i.e., by calling 
> `pdftex -ini` or `tex -ini`) and docstrip, without any format loaded.

It was worth it to ask :-).

> So I made some changes to texlive-build-system to that, and it worked. By 
> which I mean: the texlive-latex-l3kernel package builds, and the texlive-
> latex-base build is attempted and fails in the same way as it was failing
> when I sent the last email, when generating latex formats. I think I’m 
> missing some file(s) with definitions for LaTeX macros, but I don’t know yet.
>
> I pushed a new branch “bug-48064-texlive-2021-l3kernel-initex” to my GitLab 
> repo at https://gitlab.com/bauermann/guix.git with two patches implementing 
> the new approach.

I had a cursory look at the commits; they look good to me!  And the
commit messages are very useful in understanding the hows and whys.
Make sure to send them to guix-patches@gnu.org after you are satisfied
with the result!

Happy hacking,

Maxim

Re: [core-updates] texlive 2021 bootstrap problem

[Thiago Jung Bauermann]

Hi Maxim,

Em sexta-feira, 25 de junho de 2021, às 00:34:23 -03, Maxim Cournoyer 
escreveu:
> Thiago Jung Bauermann <bauermann@kolabnow.com> writes:
> > It’s possible to build texlive-latex-l3kernel with IniTeX (i.e., by
> > calling `pdftex -ini` or `tex -ini`) and docstrip, without any format
> > loaded.
> It was worth it to ask :-).

Indeed!

> > So I made some changes to texlive-build-system to that, and it worked.
> > By which I mean: the texlive-latex-l3kernel package builds, and the
> > texlive- latex-base build is attempted and fails in the same way as it
> > was failing when I sent the last email, when generating latex formats.
> > I think I’m missing some file(s) with definitions for LaTeX macros,
> > but I don’t know yet.
> > 
> > I pushed a new branch “bug-48064-texlive-2021-l3kernel-initex” to my
> > GitLab repo at https://gitlab.com/bauermann/guix.git with two patches
> > implementing the new approach.
> 
> I had a cursory look at the commits; they look good to me!  And the
> commit messages are very useful in understanding the hows and whys.
> Make sure to send them to guix-patches@gnu.org after you are satisfied
> with the result!

Thank you for having a look! I will certainly submit themt o Guix when I 
manage to get a working build of TeX Live 2021.

-- 
Thanks,
Thiago

Re: [core-updates] texlive 2021 bootstrap problem

[Thiago Jung Bauermann]

Hello,

TeX Live situation update:

Em sexta-feira, 25 de junho de 2021, às 02:00:37 -03, Thiago Jung Bauermann 
escreveu:
> Em sexta-feira, 25 de junho de 2021, às 00:34:23 -03, Maxim Cournoyer
> escreveu:
> 
> > I had a cursory look at the commits; they look good to me!  And the
> > commit messages are very useful in understanding the hows and whys.
> > Make sure to send them to guix-patches@gnu.org after you are satisfied
> > with the result!
> 
> Thank you for having a look! I will certainly submit themt o Guix when I
> manage to get a working build of TeX Live 2021.

I was able to build the main TeX Live packages with  source tarball version 
20210325 – the latest on the historic TeX Live FTP site – and subversion 
tag texlive-2021.3 – the latest tag in the TeX Live repo.

Unfortunately issue 48064¹ still happens. On the bright side, I found a 
workaround for it that also works with TeX Live 2020 currently in core-
updates I will submit the patch with the workaround as soon as testing 
finishes, hopefuly later today.

I will also separately submit patches updating TeX Live to 2021 after I 
build all of the texlive packages, hopefuly in a couple of days.

-- 
Thanks,
Thiago

¹ https://issues.guix.gnu.org/48064

Re: [core-updates] texlive 2021 bootstrap problem

[Maxim Cournoyer]

Hello Thiago,

Thiago Jung Bauermann <bauermann@kolabnow.com> writes:

> Hello,
>
> TeX Live situation update:
>
> Em sexta-feira, 25 de junho de 2021, às 02:00:37 -03, Thiago Jung Bauermann 
> escreveu:
>> Em sexta-feira, 25 de junho de 2021, às 00:34:23 -03, Maxim Cournoyer
>> escreveu:
>> 
>> > I had a cursory look at the commits; they look good to me!  And the
>> > commit messages are very useful in understanding the hows and whys.
>> > Make sure to send them to guix-patches@gnu.org after you are satisfied
>> > with the result!
>> 
>> Thank you for having a look! I will certainly submit themt o Guix when I
>> manage to get a working build of TeX Live 2021.
>
> I was able to build the main TeX Live packages with  source tarball version 
> 20210325 – the latest on the historic TeX Live FTP site – and subversion 
> tag texlive-2021.3 – the latest tag in the TeX Live repo.
>
> Unfortunately issue 48064¹ still happens. On the bright side, I found a 
> workaround for it that also works with TeX Live 2020 currently in core-
> updates I will submit the patch with the workaround as soon as testing 
> finishes, hopefuly later today.
>
> I will also separately submit patches updating TeX Live to 2021 after I 
> build all of the texlive packages, hopefuly in a couple of days.

Sounds good!  Thanks for the update about your progress.

Maxim

Re: [core-updates] texlive 2021 bootstrap problem

[Thiago Jung Bauermann]

Hello,

TeX Live situation update:

Em quarta-feira, 30 de junho de 2021, às 13:21:56 -03, Maxim Cournoyer 
escreveu:
> Thiago Jung Bauermann <bauermann@kolabnow.com> writes:
> > I was able to build the main TeX Live packages with  source tarball
> > version 20210325 – the latest on the historic TeX Live FTP site – and
> > subversion tag texlive-2021.3 – the latest tag in the TeX Live repo.
> > 
> > Unfortunately issue 48064¹ still happens. On the bright side, I found a
> > workaround for it that also works with TeX Live 2020 currently in core-
> > updates I will submit the patch with the workaround as soon as testing
> > finishes, hopefuly later today.

Unfortunately the patch I was testing which just changed some packages to 
build with pdfTex didn’t work and I ended up sending a patch which simply 
switches all packages to build with pdfTeX instead of LuaTeX. It’s 
available on the issue page¹. It didn’t show up on the guix-packages 
mailing list – not sure why.

Ludo reported the problem to the LuaTeX development mailing list (thanks!) 
and a developer there already mentioned that he would work on it ASAP.

> > I will also separately submit patches updating TeX Live to 2021 after I
> > build all of the texlive packages, hopefuly in a couple of days.

I just finished working on the TeX Live 2021 update. The patches are 
available on issue 49408².

It updates all packages matching “texlive*” but doesn’t add any new one. If 
I should add new packages, please let me know. In that case I would also 
need criteria to know which new packages to add. :-)

At this point I don’t see anything else related to TeX I could be doing. 
Feel free to let me know if there’s anything.

> Sounds good!  Thanks for the update about your progress.

Thanks for your feedback!

-- 
Thanks,
Thiago


¹ https://issues.guix.gnu.org/48064#15
² http://issues.guix.gnu.org/49408