Bug: syntax highlighting in ox-odt and emacs26+ broken [9.1.2 (9.1.2-elpa @ /home/jay/Code/tmp/emacs/lisp/org/)]

Bug: syntax highlighting in ox-odt and emacs26+ broken [9.1.2 (9.1.2-elpa @ /home/jay/Code/tmp/emacs/lisp/org/)]

[Jay Kamat]

Hi Everyone,

I think 'francisv' from the #org-mode irc channel found a interesting
bug in emacs or org mode. I don't know enough about emacs to figure out
exactly what's going on, so I'll provide as much information as I'm able
to find out.

orgmode syntax highlighting when exporting to ox-odt in emacs 26 seems
to be broken. This is not reproducible under emacs 25 at all. In
addition, this seems to be dependent on which emacs is compiling org
bytecode.

Steps to Reproduce:
1. Compile Emacs *26* from source
2. Load an org file with the following (built-in org):

#+BEGIN_SRC c
int main() {
    return 0;
}
#+END_SRC

3. Export to odt via C-c C-e o o 
4. Open file in libreoffice

Expected Results

Syntax highlighting for the code block as in emacs25

Actual Results

Code seems to be colored in a single color, interestingly, this is *not*
black.

(I can provide screenshots if that would be helpful)

After poking around, I found a bit more information on how to reproduce
it:

Environments with working syntax export:
1. Emacs25, always (unless unrelated errors are present)
2. Emacs26, when running org compiled with emacs 25 (from elpa/repo)
3. Emacs26, when running org 'interpreted' (not compiled to bytecode)

Environments with broken syntax export:
1. Emacs26 when compiled from source, using builtin org
2. Emacs26 when org is compiled with Emacs 26 (from elpa/repo)

I tested all emacs versions on org 0b83168465, changing which version
org was compiled with by modifying the EMACS variable in local.mk in
org. I also ran everything in ~emacs -Q~.

I attempted to bisect emacs to find which change in emacs is causing
this, bug I wasn't able to compile some older versions of emacs. I was
able to find that it was '14 revisions away from 5f3379b338' but I don't
know if that's helpful.

A stackexchange question about this is here:
https://emacs.stackexchange.com/questions/36023/how-to-export-babel-source-blocks-with-syntax-highlighting-to-odt-in-org-mode-in

This is probably not a bug in org mode (but rather in emacs 26) but I
wanted to post it here first to see if anyone had any idea what could be
causing this. If this dosen't seem like an org bug, I'm happy to
redirect it to emacs bug reports.

Please let me know if anyone has any questions, and apologies if
anything is wrong about this report.

Emacs  : GNU Emacs 26.0.60 (build 5, x86_64-pc-linux-gnu, GTK+ Version 3.22.11)
 of 2017-10-07
Package: Org mode version 9.1.2 (9.1.2-elpa @ /home/jay/Code/tmp/emacs/lisp/org/)

Thanks,
-Jay

Re: Bug: syntax highlighting in ox-odt and emacs26+ broken [9.1.2 (9.1.2-elpa @ /home/jay/Code/tmp/emacs/lisp/org/)]

[Tim Cross]

I'm not an org developer, but I did notice a message on the emacs-devel
list a few days ago stating that Emacs 26 has updated to use org 9.1.1.

As Emacs 26 is a moving target and this problem does not appear in the
latest stable version i.e. 25.3, I would agree this is most likely due
to a change in emacs 26 and may indicate either a bug in Emacs 26 or an
incompatible change in Emacs 26 which may require changes to
org. However, I think it would be unwise to try and address the issue in
org until Emacs 26 is closer to a release as there is too high a chance
that any change we make to org will be undone by further refinements in
Emacs 26.

My recommendation would be to confirm if the problem exists in the org
bundled with Emacs 26 (9.1.1) and if it is, report this on the emacs
devel list, making it clear the issue does not exist with emacs
25.3. This will at least allow Emacs devel to note that something which
it is changing is either buggy and needs more work or is an incompatible
change for org (and possibly other modes) which will need to be
added to the changes/News file and addressed by mode authors prior to 26
being released.

Tim

Jay Kamat writes:

> Hi Everyone,
>
> I think 'francisv' from the #org-mode irc channel found a interesting
> bug in emacs or org mode. I don't know enough about emacs to figure out
> exactly what's going on, so I'll provide as much information as I'm able
> to find out.
>
> orgmode syntax highlighting when exporting to ox-odt in emacs 26 seems
> to be broken. This is not reproducible under emacs 25 at all. In
> addition, this seems to be dependent on which emacs is compiling org
> bytecode.
>
> Steps to Reproduce:
> 1. Compile Emacs *26* from source
> 2. Load an org file with the following (built-in org):
>
> #+BEGIN_SRC c
> int main() {
>     return 0;
> }
> #+END_SRC
>
> 3. Export to odt via C-c C-e o o 
> 4. Open file in libreoffice
>
> Expected Results
>
> Syntax highlighting for the code block as in emacs25
>
> Actual Results
>
> Code seems to be colored in a single color, interestingly, this is *not*
> black.
>
> (I can provide screenshots if that would be helpful)
>
> After poking around, I found a bit more information on how to reproduce
> it:
>
> Environments with working syntax export:
> 1. Emacs25, always (unless unrelated errors are present)
> 2. Emacs26, when running org compiled with emacs 25 (from elpa/repo)
> 3. Emacs26, when running org 'interpreted' (not compiled to bytecode)
>
> Environments with broken syntax export:
> 1. Emacs26 when compiled from source, using builtin org
> 2. Emacs26 when org is compiled with Emacs 26 (from elpa/repo)
>
> I tested all emacs versions on org 0b83168465, changing which version
> org was compiled with by modifying the EMACS variable in local.mk in
> org. I also ran everything in ~emacs -Q~.
>
> I attempted to bisect emacs to find which change in emacs is causing
> this, bug I wasn't able to compile some older versions of emacs. I was
> able to find that it was '14 revisions away from 5f3379b338' but I don't
> know if that's helpful.
>
> A stackexchange question about this is here:
> https://emacs.stackexchange.com/questions/36023/how-to-export-babel-source-blocks-with-syntax-highlighting-to-odt-in-org-mode-in
>
> This is probably not a bug in org mode (but rather in emacs 26) but I
> wanted to post it here first to see if anyone had any idea what could be
> causing this. If this dosen't seem like an org bug, I'm happy to
> redirect it to emacs bug reports.
>
> Please let me know if anyone has any questions, and apologies if
> anything is wrong about this report.
>
> Emacs  : GNU Emacs 26.0.60 (build 5, x86_64-pc-linux-gnu, GTK+ Version 3.22.11)
>  of 2017-10-07
> Package: Org mode version 9.1.2 (9.1.2-elpa @ /home/jay/Code/tmp/emacs/lisp/org/)
>
> Thanks,
> -Jay


-- 
Tim Cross

Re: Bug: syntax highlighting in ox-odt and emacs26+ broken [9.1.2 (9.1.2-elpa @ /home/jay/Code/tmp/emacs/lisp/org/)]

[Jay Kamat]

Hi Tim,

> My recommendation would be to confirm if the problem exists in the org
> bundled with Emacs 26 (9.1.1) and if it is, report this on the emacs
> devel list, making it clear the issue does not exist with emacs
> 25.3. This will at least allow Emacs devel to note that something which
> it is changing is either buggy and needs more work or is an incompatible
> change for org (and possibly other modes) which will need to be
> added to the changes/News file and addressed by mode authors prior to 26
> being released.

This bug does indeed exist in the org bundled with emacs. Reporting this
to emacs-devel sounds like a good plan, and I'll do so in a couple days
to let anyone on the org mailing list have a chance to correct me if I'm
doing anything wrong on the org side. But I do agree this looks like a
bug in emacs core (even though I wansn't able to find the cause).

-Jay

Re: Bug: syntax highlighting in ox-odt and emacs26+ broken [9.1.2 (9.1.2-elpa @ /home/jay/Code/tmp/emacs/lisp/org/)]

[Jay Kamat]

I submitted a bug report to emacs core, and Eli says:

> Having read all of the references and discussions you've provided, I
> see no evidence that this is an Emacs issue, as opposed to an Org
> issue.  I think the Org developers should take a look at this first,
> and only if they provide clear evidence that the problem is due to
> Emacs, should the problem come here.

http://lists.gnu.org/archive/html/emacs-orgmode/2017-10/msg00163.html

I tried to take a look at this, but this problem seems more subtle than
I originally thought. The `org-odt-src-block' function seems to be
returning the exact same thing in both scenarios, so the problem
(unfortunately) must be related to the highest level (where the odt is
stitched together). I'll try to keep looking at this, but I would
appreciate if someone else could take a look. I would also be happy to
accept any pointers relating to the ox-odt build process.

-Jay

Jay Kamat <jaygkamat@gmail.com> writes:

> Hi Tim,
>
>> My recommendation would be to confirm if the problem exists in the org
>> bundled with Emacs 26 (9.1.1) and if it is, report this on the emacs
>> devel list, making it clear the issue does not exist with emacs
>> 25.3. This will at least allow Emacs devel to note that something which
>> it is changing is either buggy and needs more work or is an incompatible
>> change for org (and possibly other modes) which will need to be
>> added to the changes/News file and addressed by mode authors prior to 26
>> being released.
>
> This bug does indeed exist in the org bundled with emacs. Reporting this
> to emacs-devel sounds like a good plan, and I'll do so in a couple days
> to let anyone on the org mailing list have a chance to correct me if I'm
> doing anything wrong on the org side. But I do agree this looks like a
> bug in emacs core (even though I wansn't able to find the cause).
>
> -Jay