[BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)]

[BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)]

[Daniel Ortmann]

Remember to cover the basics, that is, what you expected to happen and
what in fact did happen. You don't know how to make a good report? See

https://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org mailing list.
------------------------------------------------------------------------

I have several small sh source blocks with ':results none'. The source
is executed for effect and the last line of the source is:
... | tee /target/path/to/file.csv

⛔ Error (org-babel): Error reading results: (user-error "Region is 
longer than ‘org-table-convert-region-max-lines’ (999) lines; not 
converting")

I expected that :results none would simply drop the output rather than
complaining about it.

Even though it says "error", the results seem to be correct.

Emacs : GNU Emacs 29.0.50 (build 8, x86_64-pc-linux-gnu, GTK+ Version 
3.22.30, cairo version 1.15.12)
of 2022-11-17
Package: Org mode version 9.6-pre (release_9.5.5-1118-g70cee1 @ 
/home/dortmann/src/git-org-mode/lisp/)

Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)]

[Ihor Radchenko]

Daniel Ortmann <daniel.ortmann@oracle.com> writes:

> I have several small sh source blocks with ':results none'. The source
> is executed for effect and the last line of the source is:
> ... | tee /target/path/to/file.csv
>
> ⛔ Error (org-babel): Error reading results: (user-error "Region is 
> longer than ‘org-table-convert-region-max-lines’ (999) lines; not 
> converting")
>
> I expected that :results none would simply drop the output rather than
> complaining about it.
>
> Even though it says "error", the results seem to be correct.

This is because blocks with :results none may still be used as input for
other source blocks as reference or noweb reference. So, the results are
computed, but not inserted or otherwise indicated.

Could you please provide a reproducer so that we fix the issue with
table conversion?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)]

[Daniel Ortmann]

[-- Attachment #1.1: Type: text/plain, Size: 1141 bytes --]

Please see attached which has the following code which reproduces the issue:

#+begin_src sh :shebang #!/bin/bash :results none
for (( i=1500 ; i>0 ; i-=1 ))
do
     head -c 6 /dev/urandom | uuencode -m -
done |
tee /dev/null
#+end_src


On 11/18/22 02:45, Ihor Radchenko wrote:
> Daniel Ortmann<daniel.ortmann@oracle.com>  writes:
>
>> I have several small sh source blocks with ':results none'. The source
>> is executed for effect and the last line of the source is:
>> ... | tee /target/path/to/file.csv
>>
>> ⛔ Error (org-babel): Error reading results: (user-error "Region is
>> longer than ‘org-table-convert-region-max-lines’ (999) lines; not
>> converting")
>>
>> I expected that :results none would simply drop the output rather than
>> complaining about it.
>>
>> Even though it says "error", the results seem to be correct.
> This is because blocks with :results none may still be used as input for
> other source blocks as reference or noweb reference. So, the results are
> computed, but not inserted or otherwise indicated.
>
> Could you please provide a reproducer so that we fix the issue with
> table conversion?
>

[-- Attachment #1.2: Type: text/html, Size: 1841 bytes --]

[-- Attachment #2: reproduce.org --]
[-- Type: application/vnd.lotus-organizer, Size: 376 bytes --]

Re: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)]

[Ihor Radchenko]

Daniel Ortmann <daniel.ortmann@oracle.com> writes:

> Please see attached which has the following code which reproduces the issue:
>
> #+begin_src sh :shebang #!/bin/bash :results none
> for (( i=1500 ; i>0 ; i-=1 ))
> do
>      head -c 6 /dev/urandom | uuencode -m -
> done |
> tee /dev/null
> #+end_src

Thanks!

I was able to reproduce. However, this particular error appears to be
intentional.

`org-table-convert-region-max-lines' has been introduced explicitly for
such scenarios because table conversion may take significant time when
the output is large.

I think we can do 2 things to address the problem:

1. Tweak the error wording. Probably to something like
   Babel result processing: Region is longer than `org-table-convert-region-max-lines' (%s) lines; not converting

2. Maybe introduce something like :results ignore to discard the results
   completely.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)]

[Daniel Ortmann]

I am happy with whatever you decide.  :-)

On 11/20/22 20:34, Ihor Radchenko wrote:
> Daniel Ortmann <daniel.ortmann@oracle.com> writes:
>
>> Please see attached which has the following code which reproduces the issue:
>>
>> #+begin_src sh :shebang #!/bin/bash :results none
>> for (( i=1500 ; i>0 ; i-=1 ))
>> do
>>       head -c 6 /dev/urandom | uuencode -m -
>> done |
>> tee /dev/null
>> #+end_src
> Thanks!
>
> I was able to reproduce. However, this particular error appears to be
> intentional.
>
> `org-table-convert-region-max-lines' has been introduced explicitly for
> such scenarios because table conversion may take significant time when
> the output is large.
>
> I think we can do 2 things to address the problem:
>
> 1. Tweak the error wording. Probably to something like
>     Babel result processing: Region is longer than `org-table-convert-region-max-lines' (%s) lines; not converting
>
> 2. Maybe introduce something like :results ignore to discard the results
>     completely.
>

[PATCH] Add new :results ignore header argument (was: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)])

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 206 bytes --]

Daniel Ortmann <daniel.ortmann@oracle.com> writes:

> I am happy with whatever you decide.  :-)

Then, here is a tentative patch introducing new :results ignore header
argument.

Any objections?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-babel-Add-new-results-ignore-header-argument.patch --]
[-- Type: text/x-patch, Size: 4422 bytes --]

From ea48cb04301588688f736e9461ebb7824224fe44 Mon Sep 17 00:00:00 2001
Message-Id: <ea48cb04301588688f736e9461ebb7824224fe44.1669083088.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 22 Nov 2022 10:06:24 +0800
Subject: [PATCH] org-babel: Add new :results ignore header argument

* lisp/ob-core.el (org-babel-result-cond): Unconditionally return nil
and suppress all the processing for :results ignore.
(org-babel-common-header-args-w-values):
(org-babel-sha1-hash): Add the new value to know :results value list.
* doc/org-manual.org (Handling):
* etc/ORG-NEWS (New =:results ignore= header argument): Document the
new value.

Reported-by: Daniel Ortmann <daniel.ortmann@oracle.com>
Link: https://orgmode.org/list/87tu2tjary.fsf@localhost
---
 doc/org-manual.org |  8 ++++++++
 etc/ORG-NEWS       |  8 ++++++++
 lisp/ob-core.el    | 31 ++++++++++++++++---------------
 3 files changed, 32 insertions(+), 15 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 4f877d371..194cabe99 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -18482,6 +18482,14 @@ *** Handling
   can still be used when referenced from another code block.
   Usage example: =:results none=.
 
+- =ignore= ::
+
+  Ignore the results completely.  This option is similar to =none=,
+  but no processing is performed on the return value.  Calling the
+  code block programatically (see [[*How to evaluate source code]]) or by
+  reference (see [[*Passing arguments]] and [[*Noweb Reference Syntax]]) will
+  always yield nil.
+
 - =append= ::
 
   Append results to the Org buffer.  Latest results are at the bottom.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 1f79f8ae5..d92cb139a 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -373,6 +373,14 @@ value of ~org-babel-clojure-backend~. For example:
 (range 2)
 #+end_src
 
+*** New =:results ignore= header argument
+
+Unlike =:results none=, the return value of code blocks called with
+=:results ignore= header argument is always ~nil~.  Org does not
+attempt to analyze the results and simply returns nil.  This can be
+useful when the code block is used for side effects only but generates
+large outputs that may be slow to analyze for Org.
+
 ** New options
 *** A new option for custom setting ~org-refile-use-outline-path~ to show document title in refile targets
 
diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1259909a0..89d3e104c 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -425,7 +425,7 @@ (defconst org-babel-common-header-args-w-values
     (prologue   . :any)
     (results	. ((file list vector table scalar verbatim)
 		   (raw html latex org code pp drawer link graphics)
-		   (replace silent none append prepend)
+		   (replace silent none ignore append prepend)
 		   (output value)))
     (rownames	. ((no yes)))
     (sep	. :any)
@@ -1345,7 +1345,7 @@ (defun org-babel-sha1-hash (&optional info context)
 		(lambda (a b) (string< (car a) (car b)))))
     (let* ((rm (lambda (lst)
 		 (dolist (p '("replace" "silent" "none"
-			      "append" "prepend"))
+			      "ignore" "append" "prepend"))
 		   (setq lst (remove p lst)))
 		 lst))
 	   (norm (lambda (arg)
@@ -3282,19 +3282,20 @@ (defmacro org-babel-result-cond (result-params scalar-form &rest table-forms)
   (declare (indent 1) (debug t))
   (org-with-gensyms (params)
     `(let ((,params ,result-params))
-       (if (or (member "scalar" ,params)
-	       (member "verbatim" ,params)
-	       (member "html" ,params)
-	       (member "code" ,params)
-	       (member "pp" ,params)
-	       (member "file" ,params)
-	       (and (or (member "output" ,params)
-			(member "raw"    ,params)
-			(member "org"    ,params)
-			(member "drawer" ,params))
-		    (not (member "table" ,params))))
-	   ,scalar-form
-	 ,@table-forms))))
+       (unless (member "ignore" ,params)
+         (if (or (member "scalar" ,params)
+	         (member "verbatim" ,params)
+	         (member "html" ,params)
+	         (member "code" ,params)
+	         (member "pp" ,params)
+	         (member "file" ,params)
+	         (and (or (member "output" ,params)
+			  (member "raw"    ,params)
+			  (member "org"    ,params)
+			  (member "drawer" ,params))
+		      (not (member "table" ,params))))
+	     ,scalar-form
+	   ,@table-forms)))))
 
 (defmacro org-babel-temp-directory ()
   "Return temporary directory suitable for `default-directory'."
-- 
2.35.1


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Add new :results ignore header argument (was: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)])

[Daniel Ortmann]

No objections.

On 11/21/22 20:12, Ihor Radchenko wrote:
> Daniel Ortmann <daniel.ortmann@oracle.com> writes:
>
>> I am happy with whatever you decide.  :-)
> Then, here is a tentative patch introducing new :results ignore header
> argument.
>
> Any objections?
>
>
>

Re: [PATCH] Add new :results ignore header argument (was: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)])

[Daniel Ortmann]

Works great!

On 11/22/22 00:02, Daniel Ortmann wrote:
> No objections.
>
> On 11/21/22 20:12, Ihor Radchenko wrote:
>> Daniel Ortmann <daniel.ortmann@oracle.com> writes:
>>
>>> I am happy with whatever you decide. :-)
>> Then, here is a tentative patch introducing new :results ignore header
>> argument.
>>
>> Any objections?
>>
>>
>>
>
>

Re: [PATCH] Add new :results ignore header argument (was: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)])

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> Any objections?

With the existing imprecise names, "silent" (that talks) and "none"
(that exists), we should try for maximum precision to avoid further
confusion.  Looking at your documentation, the word "discard" conveys
the meaning better than "ignore".  It says that we actually "throw the
result away" for all intents and purposes, this time.  Other than
discard, I found scrap, drop, ditch, swallow.

Rudy
-- 
"Strange as it may sound, the power of mathematics rests on its evasion
of all unnecessary thought and on its wonderful saving of mental
operations."
-- Ernst Mach, 1838-1916

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: [PATCH] Add new :results ignore header argument (was: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)])

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

Rudolf Adamkovič <salutis@me.com> writes:

>> Any objections?
>
> With the existing imprecise names, "silent" (that talks) and "none"
> (that exists), we should try for maximum precision to avoid further
> confusion.  Looking at your documentation, the word "discard" conveys
> the meaning better than "ignore".  It says that we actually "throw the
> result away" for all intents and purposes, this time.  Other than
> discard, I found scrap, drop, ditch, swallow.

Good idea. I also like "discard" better.
See the updated patch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-org-babel-Add-new-results-discard-header-argument.patch --]
[-- Type: text/x-patch, Size: 5762 bytes --]

From c262579e8f1ef9ffecfc8e051fd5ea2539b4525e Mon Sep 17 00:00:00 2001
Message-Id: <c262579e8f1ef9ffecfc8e051fd5ea2539b4525e.1669428691.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 22 Nov 2022 10:06:24 +0800
Subject: [PATCH v2] org-babel: Add new :results discard header argument

* lisp/ob-core.el (org-babel-result-cond): Unconditionally return nil
and suppress all the processing for :results discard.
(org-babel-common-header-args-w-values):
(org-babel-sha1-hash): Add the new value to know :results value list.
* doc/org-manual.org (Handling):
* etc/ORG-NEWS (New =:results discard= header argument): Document the
new value.

Reported-by: Daniel Ortmann <daniel.ortmann@oracle.com>
Link: https://orgmode.org/list/87tu2tjary.fsf@localhost
---
 doc/org-manual.org |  8 ++++++++
 etc/ORG-NEWS       |  8 ++++++++
 lisp/ob-core.el    | 43 ++++++++++++++++++++++---------------------
 3 files changed, 38 insertions(+), 21 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 760c1ee86..67b8b4dee 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -18487,6 +18487,14 @@ *** Handling
   can still be used when referenced from another code block.
   Usage example: =:results none=.
 
+- =discard= ::
+
+  Ignore the results completely.  This option is similar to =none=,
+  but no processing is performed on the return value.  Calling the
+  code block programatically (see [[*How to evaluate source code]]) or by
+  reference (see [[*Passing arguments]] and [[*Noweb Reference Syntax]]) will
+  always yield nil.
+
 - =append= ::
 
   Append results to the Org buffer.  Latest results are at the bottom.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 63ff5d749..fd569106a 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -373,6 +373,14 @@ value of ~org-babel-clojure-backend~. For example:
 (range 2)
 #+end_src
 
+*** New =:results discard= header argument
+
+Unlike =:results none=, the return value of code blocks called with
+=:results discard= header argument is always ~nil~.  Org does not
+attempt to analyze the results and simply returns nil.  This can be
+useful when the code block is used for side effects only but generates
+large outputs that may be slow to analyze for Org.
+
 ** New options
 *** A new option for custom setting ~org-refile-use-outline-path~ to show document title in refile targets
 
diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 3a07c10d5..5f679a5e9 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -425,7 +425,7 @@ (defconst org-babel-common-header-args-w-values
     (prologue   . :any)
     (results	. ((file list vector table scalar verbatim)
 		   (raw html latex org code pp drawer link graphics)
-		   (replace silent none append prepend)
+		   (replace silent none discard append prepend)
 		   (output value)))
     (rownames	. ((no yes)))
     (sep	. :any)
@@ -1345,7 +1345,7 @@ (defun org-babel-sha1-hash (&optional info context)
 		(lambda (a b) (string< (car a) (car b)))))
     (let* ((rm (lambda (lst)
 		 (dolist (p '("replace" "silent" "none"
-			      "append" "prepend"))
+			      "discard" "append" "prepend"))
 		   (setq lst (remove p lst)))
 		 lst))
 	   (norm (lambda (arg)
@@ -1353,8 +1353,8 @@ (defun org-babel-sha1-hash (&optional info context)
 				(copy-sequence (cdr arg))
 			      (cdr arg))))
 		     (when (and v (not (and (sequencep v)
-					    (not (consp v))
-					    (= (length v) 0))))
+					  (not (consp v))
+					  (= (length v) 0))))
 		       (cond
 			((and (listp v) ; lists are sorted
 			      (member (car arg) '(:result-params)))
@@ -1382,10 +1382,10 @@ (defun org-babel-sha1-hash (&optional info context)
                          (mapconcat
                           #'identity
                           (delq nil (mapcar (lambda (arg)
-                                              (let ((normalized (funcall norm arg)))
-                                                (when normalized
-                                                  (format "%S" normalized))))
-                                            (nth 2 info))) ":")
+                                            (let ((normalized (funcall norm arg)))
+                                              (when normalized
+                                                (format "%S" normalized))))
+                                          (nth 2 info))) ":")
                          expanded))
              (hash (sha1 it)))
         (when (called-interactively-p 'interactive) (message hash))
@@ -3289,19 +3289,20 @@ (defmacro org-babel-result-cond (result-params scalar-form &rest table-forms)
   (declare (indent 1) (debug t))
   (org-with-gensyms (params)
     `(let ((,params ,result-params))
-       (if (or (member "scalar" ,params)
-	       (member "verbatim" ,params)
-	       (member "html" ,params)
-	       (member "code" ,params)
-	       (member "pp" ,params)
-	       (member "file" ,params)
-	       (and (or (member "output" ,params)
-			(member "raw"    ,params)
-			(member "org"    ,params)
-			(member "drawer" ,params))
-		    (not (member "table" ,params))))
-	   ,scalar-form
-	 ,@table-forms))))
+       (unless (member "discard" ,params)
+         (if (or (member "scalar" ,params)
+	         (member "verbatim" ,params)
+	         (member "html" ,params)
+	         (member "code" ,params)
+	         (member "pp" ,params)
+	         (member "file" ,params)
+	         (and (or (member "output" ,params)
+			  (member "raw"    ,params)
+			  (member "org"    ,params)
+			  (member "drawer" ,params))
+		      (not (member "table" ,params))))
+	     ,scalar-form
+	   ,@table-forms)))))
 
 (defmacro org-babel-temp-directory ()
   "Return temporary directory suitable for `default-directory'."
-- 
2.35.1


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Add new :results ignore header argument (was: [External] : Re: [BUG] executing sh source block with ':results none' encounters error region is longer than org-table-convert-region-max-lines [9.6-pre (release_9.5.5-1118-g70cee1 @ /home/dortmann/src/git-org-mode/lisp/)])

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> See the updated patch.

Applied onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=eed4708b6

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>