bug#28806: syntax highlighting in ox-odt and emacs26+ broken

bug#28806: syntax highlighting in ox-odt and emacs26+ broken

[Jay Kamat]

Hi!

I think 'francisv' from the #org-mode irc channel found a interesting
bug in emacs or org mode. I don't know enough about emacs internals to
figure out exactly what's going on, so I'll provide as much information
as I'm able to find out. I'm not entirely sure if this is an error in
org or in emacs, but since it's a complicated interaction between them
(and exists in the bundled version of org) I'm reporting it here.

orgmode syntax highlighting when exporting via ox-odt in emacs 26 seems
to be broken. This is not reproducible under emacs 25 at all. In
addition, this seems to be dependent on which emacs is compiling org to
bytecode.

Steps to Reproduce:
1. Compile Emacs *26* from source (emacs-26 branch)
2. Load an org file with the following (built-in org):

#+BEGIN_SRC c
int main() {
    return 0;
}
#+END_SRC

3. Export to odt via C-c C-e o o 
4. Open file in libreoffice

Expected Results

Syntax highlighting for the code block as in emacs25

Actual Results

Code seems to be colored in a single color, interestingly, this is *not*
black.

(I can provide screenshots if that would be helpful)

After poking around, I found a bit more information on how to reproduce
it:

Environments with working syntax export:
1. Emacs25, always (unless unrelated errors are present)
2. Emacs26, when running org compiled with emacs 25 (from elpa/repo)
3. Emacs26, when running org 'interpreted' (not compiled to bytecode)

Environments with broken syntax export:
1. Emacs26 when compiled from source, using builtin org
2. Emacs26 when org is compiled with Emacs 26 (from elpa/repo)

I tested all emacs versions on org 0b83168465, changing which version
org was compiled with by modifying the EMACS variable in local.mk in
org. I ran everything in ~emacs -Q~.

I attempted to bisect emacs to find which change in emacs is causing
this, bug I wasn't able to compile some older versions of emacs due to
internal bytecode errors preventing a clean compilation. I was able to
find that it was '14 revisions away from 5f3379b338' but I don't know if
that's helpful.

From what I can tell (which is probably incorrect), this seems like
either:

1. A bug in the new bytecode 
2. Some backwards-incompatible change to macros in emacs 26 that org
didn't pick up on.

In the latter case, the change should be added to the NEWS file as a
backwards incompatible change.

A stackexchange question about this is here:
https://emacs.stackexchange.com/questions/36023/how-to-export-babel-source-blocks-with-syntax-highlighting-to-odt-in-org-mode-in

I posted in the org mode mailing list about this to try to get some
initial feedback from orgmode before posting here:
http://lists.gnu.org/archive/html/emacs-orgmode/2017-10/msg00161.html

Please let me know if anyone has any questions or can't reproduce, and
apologies if anything is wrong about this report (I'm fairly new
here). 

I'm willing to put in time to fix this if needed, if someone can point
me in the right direction.

Thanks,
-Jay

bug#28806: syntax highlighting in ox-odt and emacs26+ broken

[Eli Zaretskii]

> From: Jay Kamat <jaygkamat@gmail.com>
> Date: Fri, 13 Oct 2017 02:29:23 -0400
> 
> 1. Compile Emacs *26* from source (emacs-26 branch)
> 2. Load an org file with the following (built-in org):
> 
> #+BEGIN_SRC c
> int main() {
>     return 0;
> }
> #+END_SRC
> 
> 3. Export to odt via C-c C-e o o 
> 4. Open file in libreoffice
> 
> Expected Results
> 
> Syntax highlighting for the code block as in emacs25
> 
> Actual Results
> 
> Code seems to be colored in a single color, interestingly, this is *not*
> black.
> 
> (I can provide screenshots if that would be helpful)
> 
> After poking around, I found a bit more information on how to reproduce
> it:
> 
> Environments with working syntax export:
> 1. Emacs25, always (unless unrelated errors are present)
> 2. Emacs26, when running org compiled with emacs 25 (from elpa/repo)
> 3. Emacs26, when running org 'interpreted' (not compiled to bytecode)
> 
> Environments with broken syntax export:
> 1. Emacs26 when compiled from source, using builtin org
> 2. Emacs26 when org is compiled with Emacs 26 (from elpa/repo)
> 
> I tested all emacs versions on org 0b83168465, changing which version
> org was compiled with by modifying the EMACS variable in local.mk in
> org. I ran everything in ~emacs -Q~.
> 
> I attempted to bisect emacs to find which change in emacs is causing
> this, bug I wasn't able to compile some older versions of emacs due to
> internal bytecode errors preventing a clean compilation. I was able to
> find that it was '14 revisions away from 5f3379b338' but I don't know if
> that's helpful.
> 
> >From what I can tell (which is probably incorrect), this seems like
> either:
> 
> 1. A bug in the new bytecode 
> 2. Some backwards-incompatible change to macros in emacs 26 that org
> didn't pick up on.
> 
> In the latter case, the change should be added to the NEWS file as a
> backwards incompatible change.
> 
> A stackexchange question about this is here:
> https://emacs.stackexchange.com/questions/36023/how-to-export-babel-source-blocks-with-syntax-highlighting-to-odt-in-org-mode-in
> 
> I posted in the org mode mailing list about this to try to get some
> initial feedback from orgmode before posting here:
> http://lists.gnu.org/archive/html/emacs-orgmode/2017-10/msg00161.html

Thanks.

Having read all of the references and discussions you've provided, I
see no evidence that this is an Emacs issue, as opposed to an Org
issue.  I think the Org developers should take a look at this first,
and only if they provide clear evidence that the problem is due to
Emacs, should the problem come here.

bug#28806: syntax highlighting in ox-odt and emacs26+ broken

[Eli Zaretskii]

> From: pierre.techoueyres@free.fr (Pierre Téchoueyres)
> Cc: Jay Kamat <jaygkamat@gmail.com>,  28806@debbugs.gnu.org
> Date: Fri, 09 Feb 2018 00:03:31 +0100
> 
> > Having read all of the references and discussions you've provided, I
> > see no evidence that this is an Emacs issue, as opposed to an Org
> > issue.  I think the Org developers should take a look at this first,
> > and only if they provide clear evidence that the problem is due to
> > Emacs, should the problem come here.
> >
> 
> Hi Eli,
> I don't know if you noticed it but the fact that the uncompiled version
> is working as expected or that an compiled one with an older emacs
> (version 25 for example) tend to indicate a problem on the emacs side.
> 
> I can also add that forcing reevaluation of
> `org-odt-do-format-code' solve the problem.
> 
> Can this be considered as an evidence that the problem is on the emacs
> side ?

No, I don't think so.  All of the above can be the result of some Org
specific problem.  That's why I suggested that the Org developers look
into this issue first.