[BUG] ox-html does not export captions of source blocks without language

[BUG] ox-html does not export captions of source blocks without language

[Johan Bolmsjö]

** Description

The caption "Caption 1" is not exported by ox-html in the following
source block.

#+caption: Caption 1
#+begin_src
foo bar baz
#+end_src

The caption "Caption 2" is exported by ox-html in the following source
block.

#+caption: Caption 2
#+begin_src sh
  echo foo bar baz
#+end_src

** Expectation

The caption "Caption 1" is exported just as "Caption 2" is. The plain
text exporter ox-ascii exports both "Caption 1" and "Caption 2".

** Version Info

- GNU Emacs 28.2 (build 3, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw scroll bars) of 2022-11-19
- Org mode version 9.6 (9.6-gf49ee9 @ /home/johan/.emacs.d/straight/build/org/)

Re: [BUG] ox-html does not export captions of source blocks without language

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Wed, Dec 14, 2022, 4:44 PM Johan Bolmsjö <org-mode@johan.bitmaster.se>
wrote:

> ** Description
>
> The caption "Caption 1" is not exported by ox-html in the following
> source block.
>
> #+caption: Caption 1
> #+begin_src
> foo bar baz
> #+end_src
>

The begin_src block always requires a "language" as far as I know. If you
don't want to specify a language, you can use #+begin_example instead.

You can even do "#+begin_src text".

It's a different issue as to why you are seeing a different behavior
between the two exporters.

>

[-- Attachment #2: Type: text/html, Size: 1188 bytes --]

Re: [BUG] ox-html does not export captions of source blocks without language

[Johan Bolmsjö]

On Wed, Dec 14, 2022, at 22:51, Kaushal Modi wrote:
> On Wed, Dec 14, 2022, 4:44 PM Johan Bolmsjö <org-mode@johan.bitmaster.se> wrote:
>> The caption "Caption 1" is not exported by ox-html in the following
>> source block.
>> 
>> #+caption: Caption 1
>> #+begin_src
>> foo bar baz
>> #+end_src
> 
> The begin_src block always requires a "language" as far as I know. If you don't want to specify a language, you can use #+begin_example instead. 
> 
> You can even do "#+begin_src text".
> 
> It's a different issue as to why you are seeing a different behavior between the two exporters. 

Okay, the manual confirms that the language is mandatory.
https://orgmode.org/manual/Structure-of-Code-Blocks.html

"#+begin_src text" exports the caption with both ox-html and ox-ascii.
"#+begin_example" export the caption with neither ox-html nor ox-ascii.

So the only discrepancy is how the two exporters handles this non-conforming source block.

[Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Ihor Radchenko]

Johan Bolmsjö <org-mode@johan.bitmaster.se> writes:

> #+caption: Caption 1
> #+begin_src
> foo bar baz
> #+end_src

There is an inconsistency here between Org parser and
https://orgmode.org/worg/org-syntax.html + manual.

The actual parser does allow empty lang in src blocks, setting :lang
element property to nil. Should we stop doing this and treat such src
blocks as paragraphs? Or should we allow empty lang and instead adapt
the exporters to treat empty lang correctly?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

On Thu, Dec 15, 2022, 4:32 AM Ihor Radchenko <yantar92@posteo.net> wrote:

> Johan Bolmsjö <org-mode@johan.bitmaster.se> writes:
>
> > #+caption: Caption 1
> > #+begin_src
> > foo bar baz
> > #+end_src
>
> There is an inconsistency here between Org parser and
> https://orgmode.org/worg/org-syntax.html + manual.
>
> The actual parser does allow empty lang in src blocks, setting :lang
> element property to nil.



Should we stop doing this and treat such src
> blocks as paragraphs?


I think that this would cause more of a surprise to the user when something
in a source block exports as a plain paragraph instead of in a <pre> block
(for HTML exports).

Or should we allow empty lang and instead adapt
> the exporters to treat empty lang correctly?
>

I vote for this one. Then

#+begin_src
foo
#+end_src

will be analogous to this in Markdown:

```
foo
```

[-- Attachment #2: Type: text/html, Size: 2327 bytes --]

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Max Nikulin]

On 15/12/2022 16:31, Ihor Radchenko wrote:
> 
> The actual parser does allow empty lang in src blocks, setting :lang
> element property to nil. Should we stop doing this and treat such src
> blocks as paragraphs? Or should we allow empty lang and instead adapt
> the exporters to treat empty lang correctly?

Source blocks without language may be treated as #+begin_example blocks. 
I believe, a warning should be issued in such case.

I do not see a reason why caption is not allowed for examples.

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]

Hi Ihor,

> There is an inconsistency here between Org parser and
> <https://orgmode.org/worg/org-syntax.html> + manual.
>
> The actual parser does allow empty lang in src blocks, setting :lang
> element property to nil. Should we stop doing this and treat such src
> blocks as paragraphs? Or should we allow empty lang and instead adapt
> the exporters to treat empty lang correctly?

My 2c: a nil lang seems like the “less wrong” option.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

> On 15/12/2022 16:31, Ihor Radchenko wrote:
>> The actual parser does allow empty lang in src blocks, setting :lang
>> element property to nil. Should we stop doing this and treat such src
>> blocks as paragraphs? Or should we allow empty lang and instead adapt
>> the exporters to treat empty lang correctly?
>
> Source blocks without language may be treated as #+begin_example blocks. I believe, a
> warning should be issued in such case.
>
> I do not see a reason why caption is not allowed for examples.

Yes, I think I agree. Semantically, a src block without a language is
really just an example block - it cannot be executed or evaluated and is
essentially reduced to a example block.

I think having a warning is also a good idea as it will alert users when
they have inadvertently forgotten to add the lang.

I don't see any reason not to allow captions for examples either.

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> I don't see any reason not to allow captions for examples either.

In LaTeX, example blocks are exported as verbatim. I am not sure if we
can even put captions into verbatim.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1: Type: text/plain, Size: 385 bytes --]

No, please. For two reasons:

1.- In HTML, you have <code>
2.- This could trickle to other backends where it makes sense

Thanks, /PA
-- 
Fragen sind nicht da um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #2: Type: text/html, Size: 694 bytes --]

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Tom Gillespie]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

Hi Ihor,
   Chiming in here with a slight variant on what
others have said. Best!
Tom

I don't think this should be handled at the syntactic
layer at all. The empty string block language should
be syntactically valid with any special behavior
needed being handled later.

Linters could treat it as a warning/error though,
but the parsing is made significantly easier if
the empty string is present allowing the grammar
to be fully closed and regular.

Thus, I don't think we need to make this a syntactic
error or pun a src block without a lang to another type.

I think we can add an implementation for when the
block language is the empty string. This keeps
the grammar regular by removing a special case.

I assume that internally the empty string block lang
would mostly call the example block codepaths,
except that it should probably issue a warning or fail
if someone tries to org-src-edit the block so that we
can alert them that they are missing the lang.

Treating src blocks missing a lang as paragraphs is
incorrect because according to the syntax spec they
are syntactically still blocks (greater or lesser depending
on your inclinations).

I think the general principle we want to follow here is
that a block (or any entity in general) should not lose
its type because some part of its syntax is malformed
(I have made similar arguments about property drawers).

That is, if something starts with #+begin_NAME stuff
and there is a corresponding #+end_NAME, then it
is a block.

The choice of how a src block without a lang should
behave is a bit more complex as there are multiple
consumers of src blocks that make different assumptions.

As mentioned above. I think that if a block is missing the lang
we could think of it instead as the null language. If we have the
:var language because someone has other contents on the line
they have a well formed src block, but will get a different error
because there is currently no known language ":var".

[-- Attachment #2: Type: text/html, Size: 2576 bytes --]

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Source blocks without language may be treated as #+begin_example blocks. 
> I believe, a warning should be issued in such case.

M-x org-lint already does it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs?

[Max Nikulin]

On 15/12/2022 22:07, Ihor Radchenko wrote:
> Tim Cross writes:
>> I don't see any reason not to allow captions for examples either.
> 
> In LaTeX, example blocks are exported as verbatim. I am not sure if we
> can even put captions into verbatim.

The default backend for source blocks is verbatim as well:

---- >8 ---- org ----
#+caption: Minimal C program
#+begin_src c
   void main() {
     return 0;
   }
#+end_src
---- 8< ---- org ----
---- >8 ---- latex ----
\begin{verbatim}
void main() {
   return 0;
}
\end{verbatim}
\captionof{figure}{Minimal C program}
---- 8< ---- latex ----

So I do not see a problem to support captions for example blocks.

Perhaps page break may happen before the caption, but it is another issue.

You might remember my vote for the symmetry "#+begin_example lang" as 
"#+begin_src lang :eval never :noweb no" and Tom's one against it in the 
following thread:
https://list.orgmode.org/874k0vud2l.fsf@localhost/
Re: [DISCUSSION] Refactoring fontification system. Wed, 08 Jun 2022 
10:02:58 +0800

> Max Nikulin writes:
>> Source blocks without language may be treated as #+begin_example blocks. 
>> I believe, a warning should be issued in such case.
> 
> M-x org-lint already does it.

 From time to time I write about warnings during export, tangle, etc. I 
would not mind if you just ignore it. From my point of view, the 
following should be convenient, however I am unsure if it may be 
implemented in non-disturbing way. Exporter may issue warnings, they are 
collected and presented to user in a temporary window with buffer names 
and line numbers as in compilation-minor-mode. The user may see that 
something goes wrong, but if their does not switch to the warnings 
window then it disappears. It would be great if exporters and lint share 
code generating warnings. Another issue that mature development tools 
usually allows to suppress particular warning for a specific line. It 
may be a problem taking into account specific of Org syntax.

As to `org-lint', I believe, it is hard to discover for new users and 
there should be a reason to run it. It easy to forget about it even when 
some problem is faced. It is mentioned in the manual, but
- https://orgmode.org/manual/Feedback.html does not request to run 
org-lint before reporting a bug
- `org-submit-bug-report' neither suggest `org-lint' nor runs it as a 
check. Likely it should be suppressed (with appropriate report) for 
large buffers.

P.S. I have accidentally noticed a suspicious line. I have not tried if 
it is real bug, but in `org-latex-make-preamble' I expect a property 
obtained from the INFO argument, not global variable:

lisp/ox-latex.el:2014:     (when (and (eq org-latex-src-block-backend 
'engraved)

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

> Treating src blocks missing a lang as paragraphs is
> incorrect because according to the syntax spec they
> are syntactically still blocks (greater or lesser depending
> on your inclinations).
>
> I think the general principle we want to follow here is
> that a block (or any entity in general) should not lose
> its type because some part of its syntax is malformed
> (I have made similar arguments about property drawers).
>

I think you might be right here.

The big characteristic of source blocks and example blocks which make
them different from paragraphs is the line breaks and indentation. If
they are treated as paragraphs, won't this information be lost?

[BUG] org-latex-src-block-backend is directly used as variable instead of querying export option (was: [Syntax discussion] Should we treat src blocks without LANG as paragraphs?)

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> P.S. I have accidentally noticed a suspicious line. I have not tried if 
> it is real bug, but in `org-latex-make-preamble' I expect a property 
> obtained from the INFO argument, not global variable:
>
> lisp/ox-latex.el:2014:     (when (and (eq org-latex-src-block-backend 
> 'engraved)

Confirmed.
Timothy?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] org-latex-src-block-backend is directly used as variable instead of querying export option (was: [Syntax discussion] Should we treat src blocks without LANG as paragraphs?)

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

Hi Ihor and Max,

>> P.S. I have accidentally noticed a suspicious line. I have not tried if
>> it is real bug, but in `org-latex-make-preamble’ I expect a property
>> obtained from the INFO argument, not global variable:
>>
>> lisp/ox-latex.el:2014:     (when (and (eq org-latex-src-block-backend
>> ’engraved)
>
> Confirmed.
> Timothy?

I can’t find this in my personal dev branch, but I can see it at
<https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/ox-latex.el#n2032>.
It seems I resolved this in
<https://git.tecosaur.net/tec/org-mode/commit/865eb1d32f>.

Hmmm, the generated preamble commits seem fairly solid now from my usage, so I’m
not sure if it’s worth fixing this separately or presenting the generated
preamble patch for review on this ML.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: [BUG] org-latex-src-block-backend is directly used as variable instead of querying export option (was: [Syntax discussion] Should we treat src blocks without LANG as paragraphs?)

[Ihor Radchenko]

Timothy <orgmode@tec.tecosaur.net> writes:

> I can’t find this in my personal dev branch, but I can see it at
> <https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/ox-latex.el#n2032>.
> It seems I resolved this in
> <https://git.tecosaur.net/tec/org-mode/commit/865eb1d32f>.
>
> Hmmm, the generated preamble commits seem fairly solid now from my usage, so I’m
> not sure if it’s worth fixing this separately or presenting the generated
> preamble patch for review on this ML.

Considering that `org-latex-src-block-backend' cannot be modified using
in-buffer keywords or options, it is relatively safe to keep things as
they are. The only potential breakage will happen if users try to modify
options in export filters directly inside INFO.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ox-html does not export captions of source blocks without language

[Ihor Radchenko]

Johan Bolmsjö <org-mode@johan.bitmaster.se> writes:

> ** Description
>
> The caption "Caption 1" is not exported by ox-html in the following
> source block.
>
> #+caption: Caption 1
> #+begin_src
> foo bar baz
> #+end_src
>
> The caption "Caption 2" is exported by ox-html in the following source
> block.
>
> #+caption: Caption 2
> #+begin_src sh
>   echo foo bar baz
> #+end_src
>
> ** Expectation
>
> The caption "Caption 1" is exported just as "Caption 2" is. The plain
> text exporter ox-ascii exports both "Caption 1" and "Caption 2".

Here is the plan to resolve this issue:

1. We update the manual allowing src blocks to have empty language spec
2. We update org-syntax document
3. We change org-html-src-block to add caption to src blocks without lang
4. We _do not_ treat such src blocks as example blocks.
   It is because at least ox-html, ox-latex, and ox-ascii never add
   captions to example blocks. Changing this default may possible, but
   require further discussion. The existing code already uses separate
   implementations for example blocks and src blocks with nil lang.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

[FR] Present list of errors in separate buffer when running org-export, similar to what org-lint does (was: [Syntax discussion] Should we treat src blocks without LANG as paragraphs?)

[Ihor Radchenko]

As Max proposed, it may be a good idea to extend the concept of org-lint
to export.

We may unify the LaTeX errors, some errors/warnings potentially signalled
by export backend, and maybe some warnings from org-lint during the
export process. These errors can ideally be displayed just like in
org-lint, with ability to jump to the problematic LoCs in
Org/intermediate file.

This is a medium task and one will need to:
1. Generalize org-lint interactive interface (list of errors) to work
   with other code. Extract it into a separate library.
2. Accumulate export issues into a list to be displayed. For example,
   broken links can be displayed
3. Make functions like `org-latex-compile' accumulate LaTeX errors as
   well. (Optionally) associate LaTeX buffer lines with Org source.
4. Display the list of errors/warnings at the end of successful or
   failed export.

Each step if fairly simple, except optional part in 3.
Patches welcome!

Max Nikulin <manikulin@gmail.com> writes:

> ... Exporter may issue warnings, they are 
> collected and presented to user in a temporary window with buffer names 
> and line numbers as in compilation-minor-mode. The user may see that 
> something goes wrong, but if their does not switch to the warnings 
> window then it disappears. It would be great if exporters and lint share 
> code generating warnings. Another issue that mature development tools 
> usually allows to suppress particular warning for a specific line. It 
> may be a problem taking into account specific of Org syntax.
>
> As to `org-lint', I believe, it is hard to discover for new users and 
> there should be a reason to run it. It easy to forget about it even when 
> some problem is faced. It is mentioned in the manual, but
> - https://orgmode.org/manual/Feedback.html does not request to run 
> org-lint before reporting a bug
> - `org-submit-bug-report' neither suggest `org-lint' nor runs it as a 
> check. Likely it should be suppressed (with appropriate report) for 
> large buffers.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Present list of errors in separate buffer when running org-export, similar to what org-lint does (was: [Syntax discussion] Should we treat src blocks without LANG as paragraphs?)

[Tim Cross]

Ihor Radchenko <yantar92@posteo.net> writes:

> As Max proposed, it may be a good idea to extend the concept of org-lint
> to export.
>
> We may unify the LaTeX errors, some errors/warnings potentially signalled
> by export backend, and maybe some warnings from org-lint during the
> export process. These errors can ideally be displayed just like in
> org-lint, with ability to jump to the problematic LoCs in
> Org/intermediate file.
>
> This is a medium task and one will need to:
> 1. Generalize org-lint interactive interface (list of errors) to work
>    with other code. Extract it into a separate library.
> 2. Accumulate export issues into a list to be displayed. For example,
>    broken links can be displayed
> 3. Make functions like `org-latex-compile' accumulate LaTeX errors as
>    well. (Optionally) associate LaTeX buffer lines with Org source.
> 4. Display the list of errors/warnings at the end of successful or
>    failed export.
>
> Each step if fairly simple, except optional part in 3.
> Patches welcome!
>

Anything which provides additional information to assist the user in
identifying issues in their document is a plus in my view. 

Re: [BUG] ox-html does not export captions of source blocks without language

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> Here is the plan to resolve this issue:
>
> 1. We update the manual allowing src blocks to have empty language spec

See the attached patch.

> 2. We update org-syntax document

It turned out to be unnecessary.  org-syntax document already declares
block DATA to be optional:

#+begin_name [DATA]
#+end_name

See https://orgmode.org/worg/org-syntax.html#Blocks

Code blocks fall within a subset of this rule.

> 3. We change org-html-src-block to add caption to src blocks without lang

See the attached patch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-manual.org-Clarify-that-LANGUAGE-may-be-omitted-.patch --]
[-- Type: text/x-patch, Size: 2704 bytes --]

From a7c5aa3431cc1946aa7f8055c39e18e5afc4cef4 Mon Sep 17 00:00:00 2001
Message-Id: <a7c5aa3431cc1946aa7f8055c39e18e5afc4cef4.1673863743.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Mon, 16 Jan 2023 12:59:47 +0300
Subject: [PATCH 1/2] org-manual.org: Clarify that LANGUAGE may be omitted in
 code blocks

* doc/org-manual.org (Structure of Code Blocks):
(Editing Source Code): Clarify that <language> is optional.  Link to
possible consequences of <language> being omitted.
---
 doc/org-manual.org | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 4466af8e4..c241e170f 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -17313,9 +17313,16 @@ ** Structure of Code Blocks
 - =<language>= ::
 
   #+cindex: language, in code blocks
-  Mandatory.  It is the identifier of the source code language in the
+  Optional.  It is the identifier of the source code language in the
   block.  See [[*Languages]], for identifiers of supported languages.
 
+  When =<language>= identifier is omitted, the block also cannot
+  have =<switches>= and =<header arguments>=.
+
+  Language identifier is also used to fontify code blocks in Org
+  buffers, when ~org-src-fontify-natively~ is set to non-~nil~.  See
+  [[*Editing Source Code]].
+
 - =<switches>= ::
 
   #+cindex: switches, in code blocks
@@ -18951,6 +18958,9 @@ ** Editing Source Code
   header line, then the edit buffer uses that major mode.  Use this
   variable to arbitrarily map language identifiers to major modes.
 
+  When language identifier is omitted in the src block, Org mode's
+  behavior is undefined.
+
 - ~org-src-window-setup~ ::
 
   #+vindex: org-src-window-setup
@@ -18976,10 +18986,13 @@ ** Editing Source Code
 
 #+vindex: org-src-fontify-natively
 #+vindex: org-src-block-faces
-Set ~org-src-fontify-natively~ to non-~nil~ to turn on native code
-fontification in the /Org/ buffer.  Fontification of code blocks can
-give visual separation of text and code on the display page.  To
-further customize the appearance of ~org-block~ for specific
+Fontification of code blocks can give visual separation of text and
+code on the display page.  Set ~org-src-fontify-natively~ to non-~nil~
+to turn on native code fontification in the /Org/ buffer.  The
+fontification follows the major mode used to edit the code block (see
+~org-src-lang-modes~ above).
+
+To further customize the appearance of ~org-block~ for specific
 languages, customize ~org-src-block-faces~.  The following example
 shades the background of regular blocks, and colors source blocks only
 for Python and Emacs Lisp languages.
-- 
2.39.0


[-- Attachment #3: 0002-org-html-src-block-Treat-code-blocks-without-LANG-eq.patch --]
[-- Type: text/x-patch, Size: 3530 bytes --]

From 8c832d374066bbba430dc21a6b4fb098361c44a9 Mon Sep 17 00:00:00 2001
Message-Id: <8c832d374066bbba430dc21a6b4fb098361c44a9.1673863743.git.yantar92@posteo.net>
In-Reply-To: <a7c5aa3431cc1946aa7f8055c39e18e5afc4cef4.1673863743.git.yantar92@posteo.net>
References: <a7c5aa3431cc1946aa7f8055c39e18e5afc4cef4.1673863743.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Mon, 16 Jan 2023 13:04:01 +0300
Subject: [PATCH 2/2] org-html-src-block: Treat code blocks without LANG
 equally
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ox-html.el (org-html-src-block): Do not treat src blocks
without LANG as example blocks.  Instead, export them using "nil"
language.  This way, such src blocks will get captions, unlike example
blocks.

The new behavior is consistent with ox-latex and ox-ascii.

Reported-by: Johan Bolmsjö <org-mode@johan.bitmaster.se>
Link: https://orgmode.org/list/87zgb90win.fsf@localhost
---
 lisp/ox-html.el | 52 ++++++++++++++++++++++++-------------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 7b79c57d4..5e58ccba3 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -3616,32 +3616,32 @@ (defun org-html-src-block (src-block _contents info)
 	   (klipsify  (and  (plist-get info :html-klipsify-src)
                             (member lang '("javascript" "js"
 					   "ruby" "scheme" "clojure" "php" "html")))))
-      (if (not lang) (format "<pre class=\"example\"%s>\n%s</pre>" label code)
-	(format "<div class=\"org-src-container\">\n%s%s\n</div>"
-		;; Build caption.
-		(let ((caption (org-export-get-caption src-block)))
-		  (if (not caption) ""
-		    (let ((listing-number
-			   (format
-			    "<span class=\"listing-number\">%s </span>"
-			    (format
-			     (org-html--translate "Listing %d:" info)
-			     (org-export-get-ordinal
-			      src-block info nil #'org-html--has-caption-p)))))
-		      (format "<label class=\"org-src-name\">%s%s</label>"
-			      listing-number
-			      (org-trim (org-export-data caption info))))))
-		;; Contents.
-		(if klipsify
-		    (format "<pre><code class=\"src src-%s\"%s%s>%s</code></pre>"
-			    lang
-			    label
-			    (if (string= lang "html")
-				" data-editor-type=\"html\""
-			      "")
-			    code)
-		  (format "<pre class=\"src src-%s\"%s>%s</pre>"
-                          lang label code)))))))
+      (format "<div class=\"org-src-container\">\n%s%s\n</div>"
+	      ;; Build caption.
+	      (let ((caption (org-export-get-caption src-block)))
+		(if (not caption) ""
+		  (let ((listing-number
+			 (format
+			  "<span class=\"listing-number\">%s </span>"
+			  (format
+			   (org-html--translate "Listing %d:" info)
+			   (org-export-get-ordinal
+			    src-block info nil #'org-html--has-caption-p)))))
+		    (format "<label class=\"org-src-name\">%s%s</label>"
+			    listing-number
+			    (org-trim (org-export-data caption info))))))
+	      ;; Contents.
+	      (if klipsify
+		  (format "<pre><code class=\"src src-%s\"%s%s>%s</code></pre>"
+			  lang ; lang being nil is OK.
+			  label
+			  (if (string= lang "html")
+			      " data-editor-type=\"html\""
+			    "")
+			  code)
+		(format "<pre class=\"src src-%s\"%s>%s</pre>"
+                        ;; Lang being nil is OK.
+                        lang label code))))))
 
 ;;;; Statistics Cookie
 
-- 
2.39.0


[-- Attachment #4: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

[ANN] orgtbl-fit

[tbanelwebmin]

Hi the List!

The new orgtbl-fit package has just been released on Melpa. It
does regression fitting on Org Mode tables.

Example. We suspect that `obs' depends on `x' and `y'.

|  x | y |  obs |
|----+---+------|
| 32 | 7 | 38.3 |
| 18 | 3 | 11.4 |
| 43 | 9 | 47.3 |
| 11 | 2 |  8.9 |
| 35 | 8 | 45.1 |

Let us put the cursor on the `obs' column, and type
M-x orgtbl-fit

Two columns are added
- predicted obs column
- difference between obs and predicted

|  x | y |  obs | Best Fit | Fit Diff |
|----+---+------+----------+----------|
| 32 | 7 | 38.3 |     38.2 |     -0.1 |
| 18 | 3 | 11.4 |     11.6 |      0.2 |
| 43 | 9 | 47.3 |     47.2 |     -0.1 |
| 11 | 2 |  8.9 |      8.7 |     -0.2 |
| 35 | 8 | 45.1 |     45.3 |      0.2 |
#+TBLFM: $4=-0.289267886829 - 1.06613976706*$1 + 10.3668885192*$2; 
%.1f::$5=$4-$3; %.1f

So we discovered that
obs = -0.29 -1.07*x +10.37*y

Behind the scene, the calcFunc-fit function from Emacs-Calc is called.

Install through Melpa:
M-x package-install orgtbl-fit

Source and documentation here:
https://github.com/tbanel/orgtblfit/blob/master/README.org

Have fun
Thierry

Re: [BUG] ox-html does not export captions of source blocks without language

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Here is the plan to resolve this issue:
>>
>> 1. We update the manual allowing src blocks to have empty language spec
>
> See the attached patch.

Applied onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=d98ca046cc7633dd9c06236d1221e97c876795f5
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=77a1cfb9a4c7e62e5c5bcf81fe2c74090caea37a

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin <tbanelwebmin@free.fr> writes:

> The new orgtbl-fit package has just been released on Melpa. It
> does regression fitting on Org Mode tables.
>
> Example. We suspect that `obs' depends on `x' and `y'.
> ...
>
> Let us put the cursor on the `obs' column, and type
> M-x orgtbl-fit
>
> Two columns are added
> - predicted obs column
> - difference between obs and predicted
>
> |  x | y |  obs | Best Fit | Fit Diff |
> |----+---+------+----------+----------|
> | 32 | 7 | 38.3 |     38.2 |     -0.1 |
> | 18 | 3 | 11.4 |     11.6 |      0.2 |
> | 43 | 9 | 47.3 |     47.2 |     -0.1 |
> | 11 | 2 |  8.9 |      8.7 |     -0.2 |
> | 35 | 8 | 45.1 |     45.3 |      0.2 |
> #+TBLFM: $4=-0.289267886829 - 1.06613976706*$1 + 10.3668885192*$2; 
> %.1f::$5=$4-$3; %.1f

Are there situations when this package is actually useful for data
analysis? (I am usually using gnuplot for fitting)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ANN] orgtbl-fit

[tbanelwebmin]

Hi Ihor & the List

GnuPlot is a great software. If you feel confortable with it, continue 
using it! If others are used to R or Python, that is fine too.

Orgtbl-fit may be useful if:

  * - You want a pure Emacs process, without external dependencies.
  * - You know that Emacs-Calc can fit your data, but you are not
    familiar with it.
  * - Your data is already available as an Org Mode table.
  * - You don't want to write a script (just point at the target column
    and type M-x orgtbl-fit, that's all).

Actually, orgtbl-fit is a bridge between Org Mode tables and Calc.

By the way, Org Mode table spreadsheet capabilities are also a bridge 
with Calc.

Examples & documentation can be read here:
https://github.com/tbanel/orgtblfit/blob/main/README.org

Have fun!


On 1/24/23 20:55, Ihor Radchenko wrote:
> tbanelwebmin <tbanelwebmin@free.fr> writes:
>
>> The new orgtbl-fit package has just been released on Melpa. It
>> does regression fitting on Org Mode tables.
>>
>> Example. We suspect that `obs' depends on `x' and `y'.
>> ...
>>
>> Let us put the cursor on the `obs' column, and type
>> M-x orgtbl-fit
>>
>> Two columns are added
>> - predicted obs column
>> - difference between obs and predicted
>>
>> |  x | y |  obs | Best Fit | Fit Diff |
>> |----+---+------+----------+----------|
>> | 32 | 7 | 38.3 |     38.2 |     -0.1 |
>> | 18 | 3 | 11.4 |     11.6 |      0.2 |
>> | 43 | 9 | 47.3 |     47.2 |     -0.1 |
>> | 11 | 2 |  8.9 |      8.7 |     -0.2 |
>> | 35 | 8 | 45.1 |     45.3 |      0.2 |
>> #+TBLFM: $4=-0.289267886829 - 1.06613976706*$1 + 10.3668885192*$2;
>> %.1f::$5=$4-$3; %.1f
> Are there situations when this package is actually useful for data
> analysis? (I am usually using gnuplot for fitting)
>

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin <tbanelwebmin@free.fr> writes:

> Actually, orgtbl-fit is a bridge between Org Mode tables and Calc.
>
> By the way, Org Mode table spreadsheet capabilities are also a bridge 
> with Calc.
>
> Examples & documentation can be read here:
> https://github.com/tbanel/orgtblfit/blob/main/README.org

Interesting.
Could it be somehow integrated with TBLFM formulas?
I imagine something like

? +?*year +?*passengers +?*(year-2016)*passengers

, when set as a column value in table formula, to be auto-updated with
actual coefficients upon re-calculating the table.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ANN] orgtbl-fit

[tbanelwebmin]

On 1/26/23 11:35, Ihor Radchenko wrote:
> tbanelwebmin <tbanelwebmin@free.fr> writes:
>
>> Actually, orgtbl-fit is a bridge between Org Mode tables and Calc.
>>
>> By the way, Org Mode table spreadsheet capabilities are also a bridge
>> with Calc.
>>
>> Examples & documentation can be read here:
>> https://github.com/tbanel/orgtblfit/blob/main/README.org
> Interesting.
> Could it be somehow integrated with TBLFM formulas?
> I imagine something like
>
> ? +?*year +?*passengers +?*(year-2016)*passengers
>
> , when set as a column value in table formula, to be auto-updated with
> actual coefficients upon re-calculating the table.
>

Hey! That's an awesome idea.


Expanding on the idea
---------------------

We need to specify the target column ("consumption" in this example). 
Therefore, the formula could be something like that:

$4 = fit (consumption = ? +?*year +?*passengers +?*(year-2016)*passengers)

It would benefit from other spreadsheet features, like constants and 
remote references.

On the development side, the TBLFM handling is already quite a big chunk 
of code. We must take care that such an additional feature do not add 
complexity and maintenance burden.


Orgtbl-fit as-is
----------------

It is also possible to include orgtbl-fit as-is into Org Mode core. It 
would sit side-by-side with the core without changing anything in its 
code and its unit-tests.


Data-analysis toolkit
---------------------

 From a higher perspective, we could give a consistent data-analysis 
toolkit to Org Mode (and call it org-data-analysis.el).

It would start with fitting, clustering & aggregation. Then, new 
algorithms would be added upon user requests.

Of course, there should be an interest among Org Mode users for such a 
toolkit.

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin <tbanelwebmin@free.fr> writes:

>>> Examples & documentation can be read here:
>>> https://github.com/tbanel/orgtblfit/blob/main/README.org
>> Interesting.
>> Could it be somehow integrated with TBLFM formulas?
>> I imagine something like
>>
>> ? +?*year +?*passengers +?*(year-2016)*passengers
>>
>> , when set as a column value in table formula, to be auto-updated with
>> actual coefficients upon re-calculating the table.
>> ...
> We need to specify the target column ("consumption" in this example). 
> Therefore, the formula could be something like that:
>
> $4 = fit (consumption = ? +?*year +?*passengers +?*(year-2016)*passengers)
> It would benefit from other spreadsheet features, like constants and 
> remote references.

Makes sense.

> On the development side, the TBLFM handling is already quite a big chunk 
> of code. We must take care that such an additional feature do not add 
> complexity and maintenance burden.

From point of view of org-table.el, adding fitting functionality is
mostly delegating the work to GNU Calc. Org side is just parsing the
formula and transforming it to appropriate Calc function call.

So, given that the parsing is extended cleanly, I do not think that
maintenance burden will increase all that much. It may even benefit from
someone taking a fresh look and possibly refactoring org-table TBLFM
parser.

> Orgtbl-fit as-is
> ----------------
>
> It is also possible to include orgtbl-fit as-is into Org Mode core. It 
> would sit side-by-side with the core without changing anything in its 
> code and its unit-tests.

IMHO, it is not sufficiently integrated with org-table.el facilities in
its current state. I'm afraid that we will have code duplication if
include orgtbl-fit as is.

BTW, the dollar replacement is something org-table can benefit from---a
number of people are confused because "$" is treated specially by Calc.

> Data-analysis toolkit
> ---------------------
>
>  From a higher perspective, we could give a consistent data-analysis 
> toolkit to Org Mode (and call it org-data-analysis.el).
>
> It would start with fitting, clustering & aggregation. Then, new 
> algorithms would be added upon user requests.
>
> Of course, there should be an interest among Org Mode users for such a 
> toolkit.

We can, but it should be first and foremost added to GNU Calc. On Org
side, we just need appropriate integration. Maintaining generic data
analysis code in Org is out of Org's scope.

Contributing to GNU Calc will also benefit GNU Calc users who don't use
Org mode.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ANN] orgtbl-fit

[tbanelwebmin]

[-- Attachment #1: Type: text/html, Size: 5190 bytes --]

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin <tbanelwebmin@free.fr> writes:

>     BTW, the dollar replacement is something org-table can benefit from---a
>     number of people are confused because "$" is treated specially by Calc.
>
> I'm not sure what you mean. The dollar in spreadsheet formulas? Like:
> #+TBLFM: $6=$5+1

Which means that I misread the sources. I was referring to
 | 2$ | 3$| #ERROR |
 #+tblfm: $3=$2+$1

Error in the above is because Calc handles "$" specially.

>     We can, but it should be first and foremost added to GNU Calc. On Org
>     side, we just need appropriate integration. Maintaining generic data
>     analysis code in Org is out of Org's scope.
>
> Absolutely
>
> Although the latest Calc release seams to be 2.02f, dating back in January 1992. Has it reached perfection 31
> years ago?

No. It became a part of Emacs, AFAIU.
New (small) things are being added to Calc as a part of Emacs development.

From NEWS.29:

** Calc

+++
*** New user option 'calc-kill-line-numbering'.
Set it to nil to exclude line numbering from kills and copies.

From NEWS.28:

** Calc

*** The behavior when doing forward-delete has been changed.
Previously, using the 'C-d' command would delete the final number in
the input field, no matter where point was.  This has been changed to
work more traditionally, with 'C-d' deleting the next character.
Likewise, point isn't moved to the end of the string before inserting
digits.

*** Setting the word size to zero disables word clipping.
The word size normally clips the results of certain bit-oriented
operations such as shifts and bitwise XOR.  A word size of zero, set
by 'b w', makes the operation have effect on the whole argument values
and the result is not truncated in any way.

*** The '/' operator now has higher precedence in (La)TeX input mode.
It no longer has lower precedence than '+' and '-'.

*** New user option 'calc-make-windows-dedicated'.
When this user option is non-nil, Calc will mark its windows as
dedicated.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ANN] orgtbl-fit

[tbanelwebmin]

On 3/3/23 16:13, Ihor Radchenko wrote:
>
>>      BTW, the dollar replacement is something org-table can benefit from---a
>>      number of people are confused because "$" is treated specially by Calc.
>>
>> I'm not sure what you mean. The dollar in spreadsheet formulas? Like:
>> #+TBLFM: $6=$5+1
> Which means that I misread the sources. I was referring to
>   | 2$ | 3$| #ERROR |
>   #+tblfm: $3=$2+$1
>
> Error in the above is because Calc handles "$" specially.

Org or Calc was waiting for something after the $, as in the last row of 
this table:

|     |    | sum    | error message                   |
|-----+----+--------+---------------------------------|
| 2$  | 3$ | #ERROR | $'s not allowed in this context |
| 2_  | 3_ | #ERROR | Expected a number               |
| 2€  | 3€ | #ERROR | Expected `)'                    |
| 2㍐ | 3¥ | #ERROR | Expected `)'                    |
| 2£  | 3£ | #ERROR | Expected `)'                    |
| 2*  | 3+ | #ERROR | Expected a number               |
| 2   | 3  | 5      |                                 |
| $2  | 3  | 6      |                                 |
| 2$2 | 3  | 9      |                                 |
  #+tblfm: $3=$2+$1

>
>>      We can, but it should be first and foremost added to GNU Calc. On Org
>>      side, we just need appropriate integration. Maintaining generic data
>>      analysis code in Org is out of Org's scope.
>>
>> Absolutely
>>
>> Although the latest Calc release seams to be 2.02f, dating back in January 1992. Has it reached perfection 31
>> years ago?
> No. It became a part of Emacs, AFAIU.
> New (small) things are being added to Calc as a part of Emacs development.
>

Good!
So, what you said makes sense: adding features to Calc, and then giving 
access to them from Org.