exporting to PDF multi-lingual document

exporting to PDF multi-lingual document

[Andrés Ramírez]

Hi.

When exporting this document the Japanese characters do not appear in
the output.

--8<---------------cut here---------------start------------->8---
#+TITLE: Doc multi-lingual
#+options: author:nil email:nil toc:nil num:nil
#+LATEX_HEADER: \usepackage[french,japanese]{babel}
#+LaTeX_HEADER: \pagenumbering{gobble}
* Español
Hola
* English
Hi
* French and Japanese {JPN characters does not appear when rendering the
PDF}
Ceci c'est en Français
ダウンロード
Ceci c'est en Français
--8<---------------cut here---------------end--------------->8---

Could You address me to the proper sintax for the japanese characters to
appear in the output?.

Best Regards
Andrés Ramírez

Re: exporting to PDF multi-lingual document

[Ihor Radchenko]

Andrés Ramírez <rrandresf@hotmail.com> writes:

> When exporting this document the Japanese characters do not appear in
> the output.
> ...
> Could You address me to the proper sintax for the japanese characters to
> appear in the output?.

We discussed a similar issue previously in https://list.orgmode.org/orgmode/87h6z7jq4o.fsf@localhost/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: exporting to PDF multi-lingual document

[andrés ramírez]

Hi. Ihor.

>>>>> "Ihor" == Ihor Radchenko <yantar92@posteo.net> writes:

[...]


    Ihor> We discussed a similar issue previously in
    Ihor> https://list.orgmode.org/orgmode/87h6z7jq4o.fsf@localhost/

I have dowloaded de whole thread. And I got something that shows
'quotation marks' in place of Japanese characteres.

I think the problem is the font (probably a different font is
needed). So let me know If You know any hints:
--8<---------------cut here---------------start------------->8---
#+TITLE: Doc multi-lingual
#+options: author:nil email:nil toc:nil num:nil date:nil
#+LATEX_COMPILER: lualatex
#+LANGUAGE: jp
#+LaTeX_Header: \usepackage[AUTO]{polyglossia}
#+LaTeX_Header: \usepackage{fontspec}
#+LaTeX_Header: \setmainfont{FreeSerif}
* Español
Hola
* English
Hi
* French 
Ceci c'est en Français
* Japanese
ダウンロード
--8<---------------cut here---------------end--------------->8---

Best Regards
Andrés Ramírez

Re: exporting to PDF multi-lingual document

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

Hi,

Andrés Ramírez <rrandresf@hotmail.com> writes:
> Could You address me to the proper sintax for the japanese characters to
> appear in the output?.

I think you have to solve this on the LaTeX-side: including packages
that support the required unicode.

For a part of that I usually use uniinput, for example this:
https://hg.sr.ht/~arnebab/vorlesung-verteilte-systeme/browse/vorlesung-1-p2p/uniinput.sty

To solve the problem, first export to LaTeX instead of PDF and then
process that LaTeX file directly with pdflatex. Use one of the guides to
write japanese with LaTeX — some examples:

https://tex.stackexchange.com/questions/15516/how-to-write-japanese-with-latex
https://www.overleaf.com/learn/latex/Japanese
https://www.preining.info/blog/2014/12/writing-japanese-in-latex-part-1-introduction/
https://ctan.math.illinois.edu/macros/latex/contrib/babel-contrib/japanese/japanese.pdf

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: exporting to PDF multi-lingual document

[Ihor Radchenko]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> I think you have to solve this on the LaTeX-side: including packages
> that support the required unicode.

Well. Org does provide support for non-English languages. For example,
see org-latex-language-alist and org-latex-guess-babel-language.

It would be nice if Org could handle LaTeX language property, without
users doing all the LaTeX workarounds on their own. (It is not like this
problem is new or uncommon in LaTeX community).

We just need to figure out a good universal preamble that can be
generated according to the #+LANGUAGE settings.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: exporting to PDF multi-lingual document

[andrés ramírez]

Hi. Arne.

>>>>> "Arne" == Arne Babenhauserheide <arne_bab@web.de> writes:


[...]


    Arne> I think you have to solve this on the LaTeX-side: including packages that support the
    Arne> required unicode.

[...]


Thanks. That helped. It ended like this:
--8<---------------cut here---------------start------------->8---
#+TITLE: Doc multi-lingual
#+options: author:nil email:nil toc:nil num:nil date:nil
#+LATEX_HEADER: \usepackage{CJKutf8}
* Español
Hola
* English
Hi
* French 
Ceci c'est en Français
* Japanese
@@latex:\begin{CJK}{UTF8}{min}@@
ダウンロード
@@latex:\end{CJK}@@
--8<---------------cut here---------------end--------------->8---

Best Regards
Andrés Ramírez

Re: exporting to PDF multi-lingual document

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]


andrés ramírez <rrandresf@hotmail.com> writes:

>     Arne> I think you have to solve this on the LaTeX-side: including packages that support the
>     Arne> required unicode.
> Thanks. That helped. It ended like this:

I’m glad to hear that! Thank you for posting your solution.

> #+LATEX_HEADER: \usepackage{CJKutf8}
> * Japanese
> @@latex:\begin{CJK}{UTF8}{min}@@
> ダウンロード
> @@latex:\end{CJK}@@

This still seems like a lot of manual deciding whether something is
Japanese or not. I’m not sure whether there’s an easier solution,
though.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: exporting to PDF multi-lingual document

[Stefan Nobis]

Hi.

I got curious and it seems if you are willing to use lualatex (which
is, I think, currently the most actively developed engine and nowadays
a really good choice), the following solution should work:

--8<---------------cut here---------------start------------->8---

#+TITLE: Doc multi-lingual
#+options: author:nil email:nil toc:nil num:nil date:nil
#+LATEX_COMPILER: lualatex
#+LANGUAGE: jp
#+LaTeX_Header: \usepackage{luatexja}

* Español
Hola
* English
Hi
* French
Ceci c'est en Français
* Japanese
ダウンロード

--8<---------------cut here---------------end--------------->8---

The crucial parts are using lualatex and loading the package
"luatexja". It seems, that the package luatexja is currently one of
the best options to produce Japanese output and it is actively
developed. There is also quite a bit of documentation (see e.g. with
"texdoc luatexja" on the CLI or https://www.ctan.org/pkg/luatexja).

I have no experience with Japanese output and I'm unable to read
Japanese, so I could not judge the correctness or quality. But the
above approach seems to work quite well and it might be possible to
support this solution from the Org side (because only a single package
has to be loaded). The requirements seem to be OK (roughly TeXlive
2020 or newer). The package automatically sets up required fonts (but
it is still possible to override the default font settings) and even
with the default configuration it is possible to use Japanese fonts
inside math mode.

-- 
Until the next mail...,
Stefan.

Re: exporting to PDF multi-lingual document

[Ihor Radchenko]

Stefan Nobis <stefan-ml@snobis.de> writes:

> I have no experience with Japanese output and I'm unable to read
> Japanese, so I could not judge the correctness or quality. But the
> above approach seems to work quite well and it might be possible to
> support this solution from the Org side (because only a single package
> has to be loaded). The requirements seem to be OK (roughly TeXlive
> 2020 or newer). The package automatically sets up required fonts (but
> it is still possible to override the default font settings) and even
> with the default configuration it is possible to use Japanese fonts
> inside math mode.

The downside of lualatex is that it is slower:
https://list.orgmode.org/orgmode/87a69j9c6s.fsf@localhost/

Yet, better multi-language support would be welcome as an option.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: exporting to PDF multi-lingual document

[Stefan Nobis]

Ihor Radchenko <yantar92@posteo.net> writes:

> The downside of lualatex is that it is slower:
> https://list.orgmode.org/orgmode/87a69j9c6s.fsf@localhost/

Yes, for sure. But I have the impression that newer versions of luatex
have become a bit faster (maybe it's just a subjective impression that
I want to be true, because I don't want to regret the complete switch
to luatex). :)

On the other hand: Luatex is much more flexible and has better support
for modern fonts and modern font render engines.

It depends what one assumes to be the most common use case for Org. If
it's many rather short and simple documents then pdftex should suffice
most of the time and the fast compile speed will matter. If we want to
support as many different situations out of the box, I think luatex
would be the better/easier way - at the cost of slower compilation
speeds.

Another approach: Use pdftex for simple/short documents and make
selection of luatex as backend engine per document as easy as
possible. Additionally, we might try to detect situations in which
luatex would be the better choice (e.g. certain languages). But this
would certainly be more work and lead to more complex code.

-- 
Until the next mail...,
Stefan.

Re: exporting to PDF multi-lingual document

[Ihor Radchenko]

Stefan Nobis <stefan-ml@snobis.de> writes:

> On the other hand: Luatex is much more flexible and has better support
> for modern fonts and modern font render engines.

So does xelatex, AFAIU.

> It depends what one assumes to be the most common use case for Org. If
> it's many rather short and simple documents then pdftex should suffice
> most of the time and the fast compile speed will matter. If we want to
> support as many different situations out of the box, I think luatex
> would be the better/easier way - at the cost of slower compilation
> speeds.

The problem with luatex (and xelatex) is export to .tex files. Org is
sometimes used when writing research papers and journals generally
demand pdflatex, not luatex.

Also, some people explicitly add LaTeX headers to their documents -
headers that may or may not be compatible with LuaTeX.

What could be done is having #+LATEX_COMPILER set to something like
'auto by default - use luatex for some non-English #+LANGUAGE settings,
but fall back when headers are customized.

> Additionally, we might try to detect situations in which
> luatex would be the better choice (e.g. certain languages). But this
> would certainly be more work and lead to more complex code.

The most immediate problem to solve is figuring out a good universal
preamble that will work with UTF8 Org sources that can contain arbitrary
(known) languages.

The code will be more complex, but Timothy's work on conditional
preamble should make things much easier. So, complexity should not stop
us - the necessary infrastructure will be in place sooner or later.
https://list.orgmode.org/orgmode/875yc95rxp.fsf@tec.tecosaur.net/

> Another approach: Use pdftex for simple/short documents and make
> selection of luatex as backend engine per document as easy as
> possible.

It is already easy - #+LATEX_COMPILER

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: exporting to PDF multi-lingual document

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]


Stefan Nobis <stefan-ml@snobis.de> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> The downside of lualatex is that it is slower:
>> https://list.orgmode.org/orgmode/87a69j9c6s.fsf@localhost/
>
> Yes, for sure. But I have the impression that newer versions of luatex
> have become a bit faster (maybe it's just a subjective impression that
> I want to be true, because I don't want to regret the complete switch
> to luatex). :)
>
> On the other hand: Luatex is much more flexible and has better support
> for modern fonts and modern font render engines.
>
> It depends what one assumes to be the most common use case for Org. If
> it's many rather short and simple documents then pdftex should suffice
> most of the time and the fast compile speed will matter. If we want to

Isn’t the speed more relevant for large documents? Small documents don’t
need the speed that much, but for my 300 page roleplaying book I already
feel the compile times a lot (with pdflatex).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: exporting to PDF multi-lingual document

[Stefan Nobis]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> Isn’t the speed more relevant for large documents?

I always had the impression that luatex has mostly a rather fixed
overhead. Maybe that's because many of my larger documents make quite
some use of TikZ and most time may be spend there. Therefore I assumed
the fixed overhead does not matter that much for larger documents. But
I did no real benchmarks or performance comparisons and it might
depend of the kind of document and packages used.

-- 
Until the next mail...,
Stefan.

Re: exporting to PDF multi-lingual document

[Stefan Nobis]

Ihor Radchenko <yantar92@posteo.net> writes:

> The downside of lualatex is that it is slower:
> https://list.orgmode.org/orgmode/87a69j9c6s.fsf@localhost/

BTW: One of the discussion points in that thread has been the problem
of default fonts. With pdflatex the default font is CM (Computer
Modern) and if the appropriate packages are installed, you
automatically get cyrillic letters. That's why Max is a fan of this
solution.

With lualatex the default font is LM (Latin Modern; quite similar to
CM, but some metrics differ). And AFAIK LM is still worse in the range
of supported Unicode characters.

But only recently I discovered a package called "fontsetup". And this
package seems to provide a big step towards the goal discussed in the
above mentioned thread: Provide default fonts for most common
situations. The goal of fontsetup is more about easy switching between
different fonts, e.g. with the option "cambria" set everything up such
that the Microsoft Cambria font family is used and fall back to proper
fonts where Cambria has gaps. But that's also something that would be
nice for Org. Maybe we should have a closer look at this (I have no
real world experience with this package, only played a little bit with
it).

-- 
Until the next mail...,
Stefan.

Re: exporting to PDF multi-lingual document

[Ihor Radchenko]

Stefan Nobis <stefan-ml@snobis.de> writes:

> But only recently I discovered a package called "fontsetup". And this
> package seems to provide a big step towards the goal discussed in the
> above mentioned thread: Provide default fonts for most common
> situations. The goal of fontsetup is more about easy switching between
> different fonts, e.g. with the option "cambria" set everything up such
> that the Microsoft Cambria font family is used and fall back to proper
> fonts where Cambria has gaps. But that's also something that would be
> nice for Org. Maybe we should have a closer look at this (I have no
> real world experience with this package, only played a little bit with
> it).

This sounds similar to
https://list.orgmode.org/orgmode/87pmiblo2v.fsf@gmail.com/

AFAIR, the basic problem is that we cannot guarantee that any given UTF
font is installed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: exporting to PDF multi-lingual document

[Stefan Nobis]

Ihor Radchenko <yantar92@posteo.net> writes:

> AFAIR, the basic problem is that we cannot guarantee that any given
> UTF font is installed.

Yes, indeed. Probably the best route is to make tweaking as easy as
possible. There are still too few (if any) really comprehensive (and
good looking) Unicode fonts available.

-- 
Until the next mail...,
Stefan.

Re: exporting to PDF multi-lingual document

[Ihor Radchenko]

Stefan Nobis <stefan-ml@snobis.de> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> AFAIR, the basic problem is that we cannot guarantee that any given
>> UTF font is installed.
>
> Yes, indeed. Probably the best route is to make tweaking as easy as
> possible. There are still too few (if any) really comprehensive (and
> good looking) Unicode fonts available.

We might suggest installing some Libre fonts (if there are good ones)
when we detect relevant compilation errors. Though ideally all the
necessary fonts should be provided by TeXLive. I am not sure why they
are not yet there in 2023.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: exporting to PDF multi-lingual document

[Stefan Nobis]

Ihor Radchenko <yantar92@posteo.net> writes:

> Though ideally all the necessary fonts should be provided by
> TeXLive. I am not sure why they are not yet there in 2023.

First problem: Is there any free/libre font out there that has a
really broad and comprehensive coverage of Unicode glyphs? I really
don't know, but from time to time I hear about fonts that I assumed to
have good Unicode coverage that this is not the case (e.g. TeX Gyre).
I think, it is just an enormous amount of work to produce a complete
Unicode font (even more so, if it should look nice).

The other problem: Not everyone does a full installation of TeX Live.
So even if good fonts are available, they are not always installed on
every system (but this part could be mitigated by documentation and/or
error messages and hints).

But I'm no Unicode or font expert (I only need latin letters). I'm
just impressed what is already possible and about all the work done by
volunteers. And sometimes I'm astonished by how much work is still to
do for a complete and smooth Unicode experience. :)

-- 
Until the next mail...,
Stefan.

Re: exporting to PDF multi-lingual document

[Thomas Dye]

Google’s Noto font might have the best coverage and is open source. 

Tom

> On Jul 2, 2023, at 7:48 PM, Stefan Nobis <stefan-ml@snobis.de> wrote:
> 
> Ihor Radchenko <yantar92@posteo.net> writes:
> 
>> Though ideally all the necessary fonts should be provided by
>> TeXLive. I am not sure why they are not yet there in 2023.
> 
> First problem: Is there any free/libre font out there that has a
> really broad and comprehensive coverage of Unicode glyphs? I really
> don't know, but from time to time I hear about fonts that I assumed to
> have good Unicode coverage that this is not the case (e.g. TeX Gyre).
> I think, it is just an enormous amount of work to produce a complete
> Unicode font (even more so, if it should look nice).
> 
> The other problem: Not everyone does a full installation of TeX Live.
> So even if good fonts are available, they are not always installed on
> every system (but this part could be mitigated by documentation and/or
> error messages and hints).
> 
> But I'm no Unicode or font expert (I only need latin letters). I'm
> just impressed what is already possible and about all the work done by
> volunteers. And sometimes I'm astonished by how much work is still to
> do for a complete and smooth Unicode experience. :)
> 
> -- 
> Until the next mail...,
> Stefan.
> 

Re: exporting to PDF multi-lingual document

[Ihor Radchenko]

Stefan Nobis <stefan-ml@snobis.de> writes:

> The other problem: Not everyone does a full installation of TeX Live.
> So even if good fonts are available, they are not always installed on
> every system (but this part could be mitigated by documentation and/or
> error messages and hints).

Fair point.
I guess that a viable option could be (1) Having a list of fontsets that
will be tried on by one. (2) If none available, we can check the font
Emacs itself uses to render the buffer text and use them.

For (2), we have pretty good odds that the necessary fonts are already
installed and used by Emacs.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: exporting to PDF multi-lingual document

[andrés ramírez]

Hi. Stefan.

>>>>> "Stefan" == Stefan Nobis <stefan-ml@snobis.de> writes:

    Stefan> Hi.  I got curious and it seems if you are willing to use lualatex (which is, I think,
    Stefan> currently the most actively developed engine and nowadays a really good choice), the
    Stefan> following solution should work:


[...]

I have tried the snippet. But It shows just Japanese and the other
paragraphs are not present on the PDF.

Best Regards
Andrés Ramírez

Re: exporting to PDF multi-lingual document

[Stefan Nobis]

andrés ramírez <rrandresf@hotmail.com> writes:

> I have tried the snippet. But It shows just Japanese and the other
> paragraphs are not present on the PDF.

That's quite strange. I just re-tested with emacs -Q (Emacs 28.2 with
integrated Org 9.5.5). It generates the following tex file:

--8<---------------cut here---------------start------------->8---

% Created 2023-07-02 Sun 16:49
% Intended LaTeX compiler: lualatex
\documentclass[11pt]{article}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{luatexja}
\date{}
\title{Doc multi-lingual}
\hypersetup{
 pdftitle={Doc multi-lingual},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 28.2 (Org mode 9.5.5)}, 
 pdflang={Jp}}
\begin{document}

\maketitle

\section*{Español}
\label{sec:orgfec75bb}
Hola
\section*{English}
\label{sec:org36763e7}
Hi
\section*{French}
\label{sec:org8e5a1dd}
Ceci c'est en Français
\section*{Japanese}
\label{sec:org70cd53d}
ダウンロード
\end{document}

--8<---------------cut here---------------end--------------->8---

When I compile this file with lualatex, everything looks as expected
to me. BTW: I'm using TeX Live 2023 with all recent updates.

Also, from the LaTeX perspective I see nothing that should prevent the
latin letters from being shown in the final PDF. Here are the fonts
used:

--8<---------------cut here---------------start------------->8---

% pdffonts foo.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
BRVOXH+LMRoman17-Regular             CID Type 0C       Identity-H       yes yes yes      8  0
XVVEIR+LMRoman12-Bold                CID Type 0C       Identity-H       yes yes yes     10  0
HIWCUH+LMRoman10-Regular             CID Type 0C       Identity-H       yes yes yes     11  0
EPCPZB+HaranoAjiMincho-Regular       CID Type 0C       Identity-H       yes yes yes     15  0

--8<---------------cut here---------------end--------------->8---

Maybe there are some other configurations on the Emacs or LaTeX side
on your system that changes some of the defaults?

--
Until the next mail...,
Stefan.

Re: exporting to PDF multi-lingual document

[andrés ramírez]

Hi. Stefan.
>>>>> "Stefan" == Stefan Nobis <stefan-ml@snobis.de> writes:


[...]


    Stefan> That's quite strange. I just re-tested with emacs -Q (Emacs 28.2 with integrated Org
    Stefan> 9.5.5). It generates the following tex file:

[...]


    Stefan> Maybe there are some other configurations on the Emacs or LaTeX side on your system that
    Stefan> changes some of the defaults?

I tested your tex file with lualatex from command line, and It gave an
error about the font lmroman10-regular. So after installing
'otf-latin-modern' package. The pdf rendered with all the paragraphs.

Thanks.
Best Regards
ps: lualatex is really slow compared with pdflatex. But It works which
is the important part