Re: [PATCH] ob-lua: Support all types and multiple values in results

["Rudolf Adamkovič" <rudolf@adamkovic.org>]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

> I will continue working on this patch soon.

All right, here I am, at last!

In the attached patchset, I address all the problems discussed in this
thread, plus some more I found:

1. Escape double quote characters in results [as discussed]
2. Sort tabular results [to fix flaky tests]
3. Remove half-baked session support [as discussed]
4. Improve documentation strings and commentary [bonus]

Rudy


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-lua-Escape-double-quote-characters-in-results.patch --]
[-- Type: text/x-patch, Size: 1635 bytes --]

From 6dcca348337c05d33d62da6eeb1f4193b73f4d6e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Rudolf=20Adamkovi=C4=8D?= <rudolf@adamkovic.org>
Date: Fri, 14 Jun 2024 16:55:24 +0200
Subject: [PATCH 1/4] ob-lua: Escape double quote characters in results

* lisp/ob-lua.el (org-babel-lua-wrapper-method): Escape all double
quote characters in Lua before giving them to Org Babel.
* testing/lisp/test-ob-lua.el (test-ob-lua/escaping-quotes): Test
automatic escaping of double quote characters.

Reported-by: Max Nikulin <manikulin@gmail.com>
Link: https://list.orgmode.org/216278dc-075c-47d1-94dc-f5bde8346b3c@gmail.com/
---
 lisp/ob-lua.el              | 2 +-
 testing/lisp/test-ob-lua.el | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el
index 041abfabc..0168bc070 100644
--- a/lisp/ob-lua.el
+++ b/lisp/ob-lua.el
@@ -283,7 +283,7 @@ function dump(it, indent)
       end
       return result
    else
-      return tostring(it)
+      return string.gsub(tostring(it), '\"', '\\\"')
    end
 end
 
diff --git a/testing/lisp/test-ob-lua.el b/testing/lisp/test-ob-lua.el
index 0a60c68ca..ff69f5411 100644
--- a/testing/lisp/test-ob-lua.el
+++ b/testing/lisp/test-ob-lua.el
@@ -176,6 +176,12 @@ return x
             (org-test-with-temp-text "src_lua{return 1, 2, 3}"
               (org-babel-execute-src-block))))))
 
+(ert-deftest test-ob-lua/escaping-quotes ()
+  (should
+   (equal "A \" B"
+          (org-test-with-temp-text "src_lua{return 'A \" B'}"
+            (org-babel-execute-src-block)))))
+
 (provide 'test-ob-lua)
 
 ;;; test-ob-lua.el ends here
-- 
2.39.3 (Apple Git-146)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-lua-Sort-tabular-results.patch --]
[-- Type: text/x-patch, Size: 1252 bytes --]

From 3a2fc3f052ea2d36f51ae4fd9d0078fb8a08e522 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Rudolf=20Adamkovi=C4=8D?= <rudolf@adamkovic.org>
Date: Fri, 14 Jun 2024 16:59:37 +0200
Subject: [PATCH 2/4] ob-lua: Sort tabular results

`test-ob-lua/colnames-yes-header-argument-pp' fails intermittently due
to tabular results being returned from Lua to Emacs Lisp in different
order than expected.  We fix the problem by sorting all tabular
results before returning them.

* lisp/ob-lua.el (org-babel-lua-wrapper-method): Sort all tabular
results, recursively, before returning them from Lua to Emacs Lisp.
---
 lisp/ob-lua.el | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el
index 0168bc070..5790073b6 100644
--- a/lisp/ob-lua.el
+++ b/lisp/ob-lua.el
@@ -270,7 +270,13 @@ function dump(it, indent)
       if #indent ~= 0 then
          result = result .. '\\n'
       end
-      for key, value in pairs(it) do
+      local keys = {}
+      for key in pairs(it) do
+        table.insert(keys, key)
+      end
+      table.sort(keys)
+      for _, key in ipairs(keys) do
+         local value = it[key]
          result = result
             .. indent
             .. dump(key)
-- 
2.39.3 (Apple Git-146)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-ob-lua-Remove-half-baked-session-support.patch --]
[-- Type: text/x-patch, Size: 12106 bytes --]

From da489d577fa714f6ce00fec5beb464f86823c4fe Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Rudolf=20Adamkovi=C4=8D?= <rudolf@adamkovic.org>
Date: Fri, 14 Jun 2024 17:36:19 +0200
Subject: [PATCH 3/4] ob-lua: Remove half-baked session support

* lisp/ob-lua.el (org-babel-execute:lua):
(org-babel-prep-session:lua):
(org-babel-load-session:lua):
(org-babel-lua-session-buffer):
(org-babel-lua-with-earmuffs):
(org-babel-session-buffer:lua):
(org-babel-lua-without-earmuffs):
(lua-default-interpreter):
(lua-which-bufname):
(lua-shell-buffer-name):
(org-babel-lua-initiate-session-by-key):
(org-babel-lua-initiate-session):
(org-babel-lua-evaluate):
(org-babel-lua-evaluate-external-process):
(org-babel-lua-evaluate-session):
(org-babel-lua-read-string): Remove all unfinished Lua and Emacs Lisp
code related to sessions and emit a user error when a Lua session is
requested.
* testing/lisp/test-ob-lua.el (test-ob-lua/no-sessions): Test the user
error emitted when requesting a Lua session.

Link: https://list.orgmode.org/87y192etb4.fsf@localhost/
---
 lisp/ob-lua.el              | 205 +-----------------------------------
 testing/lisp/test-ob-lua.el |   9 ++
 2 files changed, 14 insertions(+), 200 deletions(-)

diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el
index 5790073b6..f36c726df 100644
--- a/lisp/ob-lua.el
+++ b/lisp/ob-lua.el
@@ -25,15 +25,6 @@
 
 ;; Org-Babel support for evaluating lua source code.
 
-;; Requirements:
-;; for session support, lua-mode is needed.
-;; lua-mode is not part of GNU Emacs/orgmode, but can be obtained
-;; from marmalade or melpa.
-;; The source repository is here:
-;; https://github.com/immerrr/lua-mode
-
-;; However, sessions are not yet working.
-
 ;;; Code:
 
 (require 'org-macs)
@@ -90,11 +81,10 @@ This will typically be `lua-mode'."
 (defun org-babel-execute:lua (body params)
   "Execute Lua BODY according to PARAMS.
 This function is called by `org-babel-execute-src-block'."
-  (let* ((session (org-babel-lua-initiate-session
-		   (cdr (assq :session params))))
+  (let* ((session (cdr (assq :session params)))
          (result-params (cdr (assq :result-params params)))
          (result-type (cdr (assq :result-type params)))
-	 (return-val (when (and (eq result-type 'value) (not session))
+	 (return-val (when (eq result-type 'value)
 		       (cdr (assq :return params))))
 	 (preamble (cdr (assq :preamble params)))
          (full-body
@@ -102,7 +92,9 @@ This function is called by `org-babel-execute-src-block'."
 	   (concat body (if return-val (format "\nreturn %s" return-val) ""))
 	   params (org-babel-variable-assignments:lua params)))
          (result (org-babel-lua-evaluate
-		  session full-body result-type result-params preamble)))
+		  full-body result-type result-params preamble)))
+    (when (and session (not (equal session "none")))
+      (user-error "Sessions not supported for Lua"))
     (org-babel-reassemble-table
      result
      (org-babel-pick-name (cdr (assq :colname-names params))
@@ -110,28 +102,6 @@ This function is called by `org-babel-execute-src-block'."
      (org-babel-pick-name (cdr (assq :rowname-names params))
 			  (cdr (assq :rownames params))))))
 
-(defun org-babel-prep-session:lua (session params)
-  "Prepare SESSION according to the header arguments in PARAMS.
-VARS contains resolved variable references."
-  (let* ((session (org-babel-lua-initiate-session session))
-	 (var-lines
-	  (org-babel-variable-assignments:lua params)))
-    (org-babel-comint-in-buffer session
-      (mapc (lambda (var)
-              (end-of-line 1) (insert var) (comint-send-input)
-              (org-babel-comint-wait-for-output session))
-	    var-lines))
-    session))
-
-(defun org-babel-load-session:lua (session body params)
-  "Load BODY into SESSION."
-  (save-window-excursion
-    (let ((buffer (org-babel-prep-session:lua session params)))
-      (with-current-buffer buffer
-        (goto-char (process-mark (get-buffer-process (current-buffer))))
-        (insert (org-babel-chomp body)))
-      buffer)))
-
 ;; helper functions
 
 (defun org-babel-variable-assignments:lua (params)
@@ -178,76 +148,6 @@ Emacs-lisp table, otherwise return the results as a string."
 
 (defvar org-babel-lua-buffers '((:default . "*Lua*")))
 
-(defun org-babel-lua-session-buffer (session)
-  "Return the buffer associated with SESSION."
-  (cdr (assoc session org-babel-lua-buffers)))
-
-(defun org-babel-lua-with-earmuffs (session)
-  "Return buffer name for SESSION, as *SESSION*."
-  (let ((name (if (stringp session) session (format "%s" session))))
-    (if (and (string= "*" (substring name 0 1))
-	     (string= "*" (substring name (- (length name) 1))))
-	name
-      (format "*%s*" name))))
-
-(defun org-babel-session-buffer:lua (session &optional _)
-  "Return session buffer name for SESSION."
-  (or (org-babel-lua-session-buffer session)
-      (org-babel-lua-with-earmuffs session)))
-
-(defun org-babel-lua-without-earmuffs (session)
-"Remove stars around *SESSION*, leaving SESSION."
-  (let ((name (if (stringp session) session (format "%s" session))))
-    (if (and (string= "*" (substring name 0 1))
-	     (string= "*" (substring name (- (length name) 1))))
-	(substring name 1 (- (length name) 1))
-      name)))
-
-(defvar lua-default-interpreter)
-(defvar lua-which-bufname)
-(defvar lua-shell-buffer-name)
-(defun org-babel-lua-initiate-session-by-key (&optional session)
-  "Initiate a lua session.
-If there is not a current inferior-process-buffer in SESSION
-then create.  Return the initialized session."
-  ;; (require org-babel-lua-mode)
-  (save-window-excursion
-    (let* ((session (if session (intern session) :default))
-           (lua-buffer (org-babel-lua-session-buffer session))
-	   ;; (cmd (if (member system-type '(cygwin windows-nt ms-dos))
-	   ;; 	    (concat org-babel-lua-command " -i")
-	   ;; 	  org-babel-lua-command))
-	   )
-      (cond
-       ((and (eq 'lua-mode org-babel-lua-mode)
-             (fboundp 'lua-start-process)) ; lua-mode.el
-        ;; Make sure that lua-which-bufname is initialized, as otherwise
-        ;; it will be overwritten the first time a Lua buffer is
-        ;; created.
-        ;;(lua-toggle-shells lua-default-interpreter)
-        ;; `lua-shell' creates a buffer whose name is the value of
-        ;; `lua-which-bufname' with '*'s at the beginning and end
-        (let* ((bufname (if (and lua-buffer (buffer-live-p lua-buffer))
-                            (replace-regexp-in-string ;; zap surrounding *
-                             "^\\*\\([^*]+\\)\\*$" "\\1" (buffer-name lua-buffer))
-                          (concat "Lua-" (symbol-name session))))
-               (lua-which-bufname bufname))
-          (lua-start-process)
-          (setq lua-buffer (org-babel-lua-with-earmuffs bufname))))
-       (t
-	(error "No function available for running an inferior Lua")))
-      (setq org-babel-lua-buffers
-            (cons (cons session lua-buffer)
-                  (assq-delete-all session org-babel-lua-buffers)))
-      session)))
-
-(defun org-babel-lua-initiate-session (&optional session _params)
-  "Create a session named SESSION according to PARAMS."
-  (unless (string= session "none")
-    (error "Sessions currently not supported, work in progress")
-    (org-babel-lua-session-buffer
-     (org-babel-lua-initiate-session-by-key session))))
-
 (defvar org-babel-lua-eoe-indicator "--eoe"
   "A string to indicate that evaluation has completed.")
 
@@ -306,19 +206,6 @@ output:write(combine(main()))
 output:close()")
 
 (defun org-babel-lua-evaluate
-    (session body &optional result-type result-params preamble)
-  "Evaluate BODY in SESSION as Lua code.
-RESULT-TYPE and RESULT-PARAMS are passed to
-`org-babel-lua-evaluate-session' or
-`org-babel-lua-evaluate-external-process'.
-PREAMBLE is passed to `org-babel-lua-evaluate-external-process'."
-  (if session
-      (org-babel-lua-evaluate-session
-       session body result-type result-params)
-    (org-babel-lua-evaluate-external-process
-     body result-type result-params preamble)))
-
-(defun org-babel-lua-evaluate-external-process
     (body &optional result-type result-params preamble)
   "Evaluate BODY in external Lua process.
 If RESULT-TYPE equals `output' then return standard output as a
@@ -354,88 +241,6 @@ PREAMBLE string is appended to BODY."
       raw
       (org-babel-lua-table-or-string (org-trim raw)))))
 
-(defun org-babel-lua-evaluate-session
-    (session body &optional result-type result-params)
-  "Pass BODY to the Lua process in SESSION.
-If RESULT-TYPE equals `output' then return standard output as a
-string.  If RESULT-TYPE equals `value' then return the value of the
-last statement in BODY, as elisp."
-  (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0.005)))
-	 (dump-last-value
-	  (lambda
-	    (tmp-file pp)
-	    (mapc
-	     (lambda (statement) (insert statement) (funcall send-wait))
-	     (if pp
-		 (list
-		  "-- table to string
-function t2s(t, indent)
-   if indent == nil then
-      indent = \"\"
-   end
-   if type(t) == \"table\" then
-      ts = \"\"
-      for k,v in pairs(t) do
-         if type(v) == \"table\" then
-            ts = ts .. indent .. t2s(k,indent .. \"  \") .. \" = \\n\" ..
-               t2s(v, indent .. \"  \")
-         else
-            ts = ts .. indent .. t2s(k,indent .. \"  \") .. \" = \" ..
-               t2s(v, indent .. \"  \") .. \"\\n\"
-         end
-      end
-      return ts
-   else
-      return tostring(t)
-   end
-end
-"
-		  (concat "fd:write(_))
-fd:close()"
-			  (org-babel-process-file-name tmp-file 'noquote)))
-	       (list (format "fd=io.open(\"%s\", \"w\")
-fd:write( _ )
-fd:close()"
-			     (org-babel-process-file-name tmp-file
-                                                          'noquote)))))))
-	 (input-body (lambda (body)
-		       (mapc (lambda (line) (insert line) (funcall send-wait))
-			     (split-string body "[\r\n]"))
-		       (funcall send-wait)))
-         (results
-          (pcase result-type
-            (`output
-             (mapconcat
-              #'org-trim
-              (butlast
-               (org-babel-comint-with-output
-                   (session org-babel-lua-eoe-indicator t body)
-                 (funcall input-body body)
-                 (funcall send-wait) (funcall send-wait)
-                 (insert org-babel-lua-eoe-indicator)
-                 (funcall send-wait))
-               2) "\n"))
-            (`value
-             (let ((tmp-file (org-babel-temp-file "lua-")))
-               (org-babel-comint-with-output
-                   (session org-babel-lua-eoe-indicator nil body)
-                 (let ((comint-process-echoes nil))
-                   (funcall input-body body)
-                   (funcall dump-last-value tmp-file
-                            (member "pp" result-params))
-                   (funcall send-wait) (funcall send-wait)
-                   (insert org-babel-lua-eoe-indicator)
-                   (funcall send-wait)))
-               (org-babel-eval-read-file tmp-file))))))
-    (unless (string= (substring org-babel-lua-eoe-indicator 1 -1) results)
-      (org-babel-result-cond result-params
-	results
-        (org-babel-lua-table-or-string results)))))
-
-(defun org-babel-lua-read-string (string)
-  "Strip single quotes from around Lua STRING."
-  (org-unbracket-string "'" "'" string))
-
 (provide 'ob-lua)
 
 ;;; ob-lua.el ends here
diff --git a/testing/lisp/test-ob-lua.el b/testing/lisp/test-ob-lua.el
index ff69f5411..58a120e25 100644
--- a/testing/lisp/test-ob-lua.el
+++ b/testing/lisp/test-ob-lua.el
@@ -182,6 +182,15 @@ return x
           (org-test-with-temp-text "src_lua{return 'A \" B'}"
             (org-babel-execute-src-block)))))
 
+(ert-deftest test-ob-lua/no-sessions ()
+  (should
+   (equal
+    '(user-error "Sessions not supported for Lua")
+    (should-error
+     (org-test-with-temp-text "src_lua[:session 1]{}"
+       (org-babel-execute-src-block))
+     :type 'user-error))))
+
 (provide 'test-ob-lua)
 
 ;;; test-ob-lua.el ends here
-- 
2.39.3 (Apple Git-146)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-ob-lua-Improve-documentation-strings-and-commentary.patch --]
[-- Type: text/x-patch, Size: 3275 bytes --]

From 01ce86a80530eb70ea0213a9896e4dbb1aadca63 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Rudolf=20Adamkovi=C4=8D?= <rudolf@adamkovic.org>
Date: Fri, 14 Jun 2024 17:47:17 +0200
Subject: [PATCH 4/4] ob-lua: Improve documentation strings and commentary

* lisp/ob-lua.el (org-babel-lua-mode):
(org-babel-lua-hline-to):
(org-babel-lua-None-to):
(org-babel-lua-var-to-lua):
(org-babel-lua-table-or-string):
(org-babel-lua-evaluate-external-process): Harmonize spelling,
capitalization, and quoting.
---
 lisp/ob-lua.el | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el
index f36c726df..826273e1f 100644
--- a/lisp/ob-lua.el
+++ b/lisp/ob-lua.el
@@ -23,7 +23,7 @@
 
 ;;; Commentary:
 
-;; Org-Babel support for evaluating lua source code.
+;; Org Babel support for evaluating Lua source code.
 
 ;;; Code:
 
@@ -51,7 +51,7 @@
   :type 'string)
 
 (defcustom org-babel-lua-mode 'lua-mode
-  "Preferred lua mode for use in running lua interactively.
+  "Preferred Lua mode for use in running Lua interactively.
 This will typically be `lua-mode'."
   :group 'org-babel
   :version "26.1"
@@ -59,14 +59,14 @@ This will typically be `lua-mode'."
   :type 'symbol)
 
 (defcustom org-babel-lua-hline-to "None"
-  "Replace hlines in incoming tables with this when translating to lua."
+  "Replace `hlines' in incoming tables with this when translating to Lua."
   :group 'org-babel
   :version "26.1"
   :package-version '(Org . "8.3")
   :type 'string)
 
 (defcustom org-babel-lua-None-to 'hline
-  "Replace `None' in lua tables with this before returning."
+  "Replace `None' in Lua tables with this before returning."
   :group 'org-babel
   :version "26.1"
   :package-version '(Org . "8.3")
@@ -115,8 +115,8 @@ The variable definitions are defining in PARAMS."
    (org-babel--get-vars params)))
 
 (defun org-babel-lua-var-to-lua (var)
-  "Convert an elisp value to a lua variable.
-Convert an elisp value, VAR, into a string of lua source code
+  "Convert an Emacs Lisp value to a Lua variable.
+Convert an Emacs Lisp value, VAR, into a string of Lua source code
 specifying a variable of the same value."
   (if (listp var)
       (if (and (= 1 (length var)) (not (listp (car var))))
@@ -136,9 +136,9 @@ specifying a variable of the same value."
        (if (stringp var) (substring-no-properties var) var)))))
 
 (defun org-babel-lua-table-or-string (results)
-  "Convert RESULTS into an appropriate elisp value.
+  "Convert RESULTS into an appropriate Emacs Lisp value.
 If the results look like a list or tuple, then convert them into an
-Emacs-lisp table, otherwise return the results as a string."
+Emacs Lisp table, otherwise return the results as a string."
   (let ((res (org-babel-script-escape results)))
     (if (listp res)
         (mapcar (lambda (el) (if (eq el 'None)
@@ -210,7 +210,7 @@ output:close()")
   "Evaluate BODY in external Lua process.
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
-last statement in BODY, as elisp.
+last statement in BODY, as Emacs Lisp.
 RESULT-PARAMS list all the :result header arg parameters.
 PREAMBLE string is appended to BODY."
   (let ((raw
-- 
2.39.3 (Apple Git-146)


[-- Attachment #6: Type: text/plain, Size: 311 bytes --]

-- 
"Programming reliably -- must be an activity of an undeniably
mathematical nature […] You see, mathematics is about thinking, and
doing mathematics is always trying to think as well as possible."
--- Edsger W. Dijkstra, 1981

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org