[patch] fix ox-latex async export bug

[patch] fix ox-latex async export bug

[Rasmus]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Hi there,

When I try to export documents asynchronously with ox-latex, I always get
a bug in the “org-export-processFOO” files. The last sexp is always something like this: 

    (funcall '#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_83> "test.tex")

where the “#” and “’” are mixed around. This happens even with a very
simple ‘org-export-async-init-file’ just loading ELPA Org.

This was previously reported here:

    https://lists.gnu.org/archive/html/emacs-orgmode/2021-06/msg00422.html

Nicolas’ fix (replicated in this patch) seems to fix it.
I don’t really understand why this bug happens to be honest.

This happens both on my GNU/Linux system (Emacs v29) and my work Windows
PC (Emacs v28 or v29) [all pre-releases, obviously].

Thanks,
Rasmus

-- 
If you can mix business and politics wonderful things can happen!

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-latex-export-to-pdf-Avoid-using-lambda-function.patch --]
[-- Type: text/x-patch, Size: 986 bytes --]

From be15ab50ef4cefc579f87766b21eaed7514379d1 Mon Sep 17 00:00:00 2001
From: Rasmus Pank Roulund <X260@pank.eu>
Date: Sun, 28 Nov 2021 16:34:37 +0100
Subject: [PATCH] org-latex-export-to-pdf: Avoid using lambda function.

Fix "invalid-read-syntax '#'" when exporting asynchronously with
ox-latex.

See also previous bug report: https://lists.gnu.org/archive/html/emacs-orgmode/2021-06/msg00422.html
---
 lisp/ox-latex.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index 409d92164..8187119ec 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -3708,7 +3708,7 @@ Return PDF file's name."
   (let ((outfile (org-export-output-file-name ".tex" subtreep)))
     (org-export-to-file 'latex outfile
       async subtreep visible-only body-only ext-plist
-      (lambda (file) (org-latex-compile file)))))
+      #'org-latex-compile)))
 
 (defun org-latex-compile (texfile &optional snippet)
   "Compile a TeX file.
-- 
2.34.1

Re: [patch] fix ox-latex async export bug

[Nicolas Goaziou]

Hello,

Rasmus <rasmus@gmx.us> writes:

> When I try to export documents asynchronously with ox-latex, I always get
> a bug in the “org-export-processFOO” files. The last sexp is always something like this: 
>
>     (funcall '#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_83> "test.tex")
>
> where the “#” and “’” are mixed around. This happens even with a very
> simple ‘org-export-async-init-file’ just loading ELPA Org.
>
> This was previously reported here:
>
>     https://lists.gnu.org/archive/html/emacs-orgmode/2021-06/msg00422.html
>
> Nicolas’ fix (replicated in this patch) seems to fix it.

Applied. Thank you.

> I don’t really understand why this bug happens to be honest.

The patch is already an improvement, but the beast is still lurking,
indeed.

Regards,
-- 
Nicolas Goaziou

Re: [patch] fix ox-latex async export bug

[Tim Cross]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Rasmus <rasmus@gmx.us> writes:
>
>> When I try to export documents asynchronously with ox-latex, I always get
>> a bug in the “org-export-processFOO” files. The last sexp is always something like this: 
>>
>>     (funcall '#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_83> "test.tex")
>>
>> where the “#” and “’” are mixed around. This happens even with a very
>> simple ‘org-export-async-init-file’ just loading ELPA Org.
>>
>> This was previously reported here:
>>
>>     https://lists.gnu.org/archive/html/emacs-orgmode/2021-06/msg00422.html
>>
>> Nicolas’ fix (replicated in this patch) seems to fix it.
>
> Applied. Thank you.
>
>> I don’t really understand why this bug happens to be honest.
>
> The patch is already an improvement, but the beast is still lurking,
> indeed.
>

Just a shot in the dark which might be completely misleading, but ...

I noticed a thread recently on emacs-devel which talked about one of the
problems with async calls in Emacs is that they cannot handle user input
correctly. All seems to work fine provided the async process being
generated doesn't try to wait for user input at some point. I'm
wondering if this could be a problem when exporting to latex if the
underlying latex process encounters errors and is waiting for user input
before it can continue (which happens if there are problems in the tex
source latex is trying to process)?

It might be worth checking next time you encounter an error if you can
run latex on the generated *.tex file and see if it waits for user
input?

Re: [patch] fix ox-latex async export bug

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

Hi Tim,

> I’m wondering if this could be a problem when exporting to latex if the
>underlying latex process encounters errors and is waiting for user input before
>it can continue (which happens if there are problems in the tex source latex is
>trying to process)?
>
> It might be worth checking next time you encounter an error if you can
> run latex on the generated *.tex file and see if it waits for user
> input?

If you look at the default value of `org-latex-pdf-process', it has the flag
`-interaction=nonstopmode' which should mean the process never pauses and waits
for user input.

All the best,
Timothy

Re: [patch] fix ox-latex async export bug

[Sébastien Miquel]

[-- Attachment #1.1: Type: text/plain, Size: 435 bytes --]

Hi,

Nicolas Goaziou writes:
>> I don’t really understand why this bug happens to be honest.
> The patch is already an improvement, but the beast is still lurking,
> indeed.
This is most likely due to native compilation which compiles the
unquoted lambda. Once compiled, it (presumably) fails to be passed to
the external emacs process.

Attached is a patch that applies the same fix where affected.

Regards,

-- 
Sébastien Miquel

[-- Attachment #1.2: Type: text/html, Size: 1102 bytes --]

[-- Attachment #2: 0001-ox-Fix-async-export-with-native-compilation.patch --]
[-- Type: text/x-patch, Size: 3364 bytes --]

From 35ae093113d9a04a99b55f0747848b373a7463f3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?S=C3=A9bastien=20Miquel?= <sebastien.miquel@posteo.eu>
Date: Mon, 29 Nov 2021 09:54:33 +0100
Subject: [PATCH] ox: Fix async export with native compilation

* lisp/ox-beamer.el (org-beamer-export-to-pdf):
* lisp/ox-icalendar.el (org-icalendar-export-to-ics):
* lisp/ox-koma-letter.el (org-koma-letter-export-to-pdf):
* lisp/ox-man.el (org-man-export-to-pdf):
* lisp/ox-texinfo.el (org-texinfo-export-to-info): Quote lambda.

Quote or name lambdas to prevent their compilation into anonymous
functions which cannot be passed to the external async emacs process.
---
 lisp/ox-beamer.el      | 2 +-
 lisp/ox-icalendar.el   | 4 ++--
 lisp/ox-koma-letter.el | 2 +-
 lisp/ox-man.el         | 2 +-
 lisp/ox-texinfo.el     | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/lisp/ox-beamer.el b/lisp/ox-beamer.el
index c9a67f891..3bfcd01d4 100644
--- a/lisp/ox-beamer.el
+++ b/lisp/ox-beamer.el
@@ -1059,7 +1059,7 @@ Return PDF file's name."
   (let ((file (org-export-output-file-name ".tex" subtreep)))
     (org-export-to-file 'beamer file
       async subtreep visible-only body-only ext-plist
-      (lambda (file) (org-latex-compile file)))))
+      #'org-latex-compile)))
 
 ;;;###autoload
 (defun org-beamer-select-environment ()
diff --git a/lisp/ox-icalendar.el b/lisp/ox-icalendar.el
index 0a682c7c8..68c5679ea 100644
--- a/lisp/ox-icalendar.el
+++ b/lisp/ox-icalendar.el
@@ -888,8 +888,8 @@ Return ICS file name."
     (org-export-to-file 'icalendar outfile
       async subtreep visible-only body-only
       '(:ascii-charset utf-8 :ascii-links-to-notes nil)
-      (lambda (file)
-	(run-hook-with-args 'org-icalendar-after-save-hook file) nil))))
+      '(lambda (file)
+	 (run-hook-with-args 'org-icalendar-after-save-hook file) nil))))
 
 ;;;###autoload
 (defun org-icalendar-export-agenda-files (&optional async)
diff --git a/lisp/ox-koma-letter.el b/lisp/ox-koma-letter.el
index 6a895a6a2..978e4e41f 100644
--- a/lisp/ox-koma-letter.el
+++ b/lisp/ox-koma-letter.el
@@ -982,7 +982,7 @@ Return PDF file's name."
         (org-koma-letter-special-contents))
     (org-export-to-file 'koma-letter file
       async subtreep visible-only body-only ext-plist
-      (lambda (file) (org-latex-compile file)))))
+      #'org-latex-compile)))
 
 
 (provide 'ox-koma-letter)
diff --git a/lisp/ox-man.el b/lisp/ox-man.el
index 6d3476cda..9a1f00f35 100644
--- a/lisp/ox-man.el
+++ b/lisp/ox-man.el
@@ -1117,7 +1117,7 @@ Return PDF file's name."
   (let ((outfile (org-export-output-file-name ".man" subtreep)))
     (org-export-to-file 'man outfile
       async subtreep visible-only body-only ext-plist
-      (lambda (file) (org-latex-compile file)))))
+      #'org-latex-compile)))
 
 (defun org-man-compile (file)
   "Compile a Groff file.
diff --git a/lisp/ox-texinfo.el b/lisp/ox-texinfo.el
index b0125894a..734c8a4f3 100644
--- a/lisp/ox-texinfo.el
+++ b/lisp/ox-texinfo.el
@@ -1701,7 +1701,7 @@ Return INFO file's name."
 	(org-export-coding-system org-texinfo-coding-system))
     (org-export-to-file 'texinfo outfile
       async subtreep visible-only body-only ext-plist
-      (lambda (file) (org-texinfo-compile file)))))
+      #'org-texinfo-compile)))
 
 ;;;###autoload
 (defun org-texinfo-publish-to-texinfo (plist filename pub-dir)
-- 
2.34.1

Re: [patch] fix ox-latex async export bug

[Nicolas Goaziou]

Hello,

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> Attached is a patch that applies the same fix where affected.

Thank you. It mostly looks good, but I have one nit.

> -      (lambda (file)
> -	(run-hook-with-args 'org-icalendar-after-save-hook file) nil))))
> +      '(lambda (file)
> +	 (run-hook-with-args 'org-icalendar-after-save-hook file) nil))))

This is not really the same fix. You're quoting a lambda, which should
not be required if the problem disappeared. IOW, the true fix probably
belong in the `org-export-async-start' function.

WDYT?

Regards,
-- 
Nicolas Goaziou

Re: [patch] fix ox-latex async export bug

[Sébastien Miquel]

Hi,

Nicolas Goaziou writes:
> This is not really the same fix.

Indeed, but I tested it and it does work.

> You're quoting a lambda, which should
> not be required if the problem disappeared. IOW, the true fix probably
> belong in the `org-export-async-start' function.

What happens is as follows
  1. This lambda is passed as the =post-process= variable of
     =org-export-to-file=.
  2. Which converts it to code using =`(funcall ',post-process)=
  3. Which =org-export-async-start= writes to a file, to be executed by
     the external emacs process.

I think native compilation compiles the lamdba in
=org-latex-export-to-pdf= and that there is no way to get back this
original lambda (the code) from within =org-export-to-file= or
=org-export-async-start=. Quoting the lambda prevents this
compilation.

My understanding of elisp and the native compilation business is quite
superficial, so this may be wrong.

Regards,

-- 
Sébastien Miquel

Re: [patch] fix ox-latex async export bug

[Rasmus Pank Roulund]

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> Hi,
>
> Nicolas Goaziou writes:
>>> I don’t really understand why this bug happens to be honest.
>> The patch is already an improvement, but the beast is still lurking,
>> indeed.
> This is most likely due to native compilation which compiles the
> unquoted lambda. Once compiled, it (presumably) fails to be passed to
> the external emacs process.

Ah, interesting.  Is this a bug or a feature?

FWIW, I definitely use native compilation on my private laptop and almost
certainly on my work (Windows) laptop as well.

Thanks,
Rasmus

-- 
If you can mix business and politics wonderful things can happen!

Re: [patch] fix ox-latex async export bug

[Rasmus]

Timothy <tecosaur@gmail.com> writes:

> Hi Tim,
>
>> I’m wondering if this could be a problem when exporting to latex if the
>>underlying latex process encounters errors and is waiting for user
>> input before
>>it can continue (which happens if there are problems in the tex
>> source latex is
>>trying to process)?
>>
>> It might be worth checking next time you encounter an error if you can
>> run latex on the generated *.tex file and see if it waits for user
>> input?
>
> If you look at the default value of `org-latex-pdf-process', it has the flag
> `-interaction=nonstopmode' which should mean the process never pauses
> and waits
> for user input.

Indeed.  I run LaTeX in noninteractive mode. (And by now I rarely make mistakes... ;)

Rasmus

-- 
However beautiful the theory, one should occasionally look at the evidence

Re: [patch] fix ox-latex async export bug

[Sébastien Miquel]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

Hi,

Rasmus Pank Roulund writes:
>> This is most likely due to native compilation which compiles the
>> unquoted lambda. Once compiled, it (presumably) fails to be passed to
>> the external emacs process.
> Ah, interesting.  Is this a bug or a feature?

I think the bug's with org-mode. As I explain elsewhere in the thread,
this lambda is to be treated as data, and written to a file to be
executed by the external emacs instance. It should always have been
quoted (naming it happens to work also).

Regards,

-- 
Sébastien Miquel

[-- Attachment #2: Type: text/html, Size: 1341 bytes --]

Re: [patch] fix ox-latex async export bug

[Rasmus]

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> Rasmus Pank Roulund writes:
>>> This is most likely due to native compilation which compiles the
>>> unquoted lambda. Once compiled, it (presumably) fails to be passed to
>>> the external emacs process.
>> Ah, interesting.  Is this a bug or a feature?
>
> I think the bug's with org-mode. As I explain elsewhere in the thread,
> this lambda is to be treated as data, and written to a file to be
> executed by the external emacs instance. It should always have been
> quoted (naming it happens to work also).

Thanks. That makes sense.  I did wonder how those files worked (as the
lambdas are gibberish) when I was debugging the problem.

Rasmus

-- 
Bang bang

Re: [patch] fix ox-latex async export bug

[Nicolas Goaziou]

Hello,

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> I think native compilation compiles the lamdba in
> =org-latex-export-to-pdf= and that there is no way to get back this
> original lambda (the code) from within =org-export-to-file= or
> =org-export-async-start=. Quoting the lambda prevents this
> compilation.

OK. I applied your patch and modified some docstrings accordingly. Thank
you.

Regards,
-- 
Nicolas Goaziou