Please review: rewrite sql-interactive-remove-continuation-prompt

Please review: rewrite sql-interactive-remove-continuation-prompt

[Filipp Gunbin]

Hi,

After several attempts to make multiline queries work for Apache
Cassandra, I gave up and almost rewrote
sql-interactive-remove-continuation-prompt.  It's now more streamline
and self-contained.

I was coding the logic described in docstring below.  If I didn't miss
something, then I'd like to install this and see what breaks.  If I did,
then what did I miss?

I tested this on multiline queries with Postgres and Cassandra, I don't
have access to other DBs right now (most importantly, Oracle and MySQL /
Maria).  I tested sql-send-region (C-c C-r from sql-mode),
sql-accumulate-and-indent (C-j then RET from sql-interactive-mode) and
just typing into the interpreter.

Previously we were anchoring the prompt search regexps to start/end of
string (` and '), but now just search for regexps as they are.  Current
regexps in sql.el all anchor to the beginning of line (^, not `).  The
change is indended, as prompts may appear anywhere in the output.

Thanks.
Filipp


diff --git a/lisp/progmodes/sql.el b/lisp/progmodes/sql.el
index 5e5f5e13fe..979b743a65 100644
--- a/lisp/progmodes/sql.el
+++ b/lisp/progmodes/sql.el
@@ -3648,94 +3648,69 @@ sql-input-sender
 
 (defvar sql-preoutput-hold nil)
 
-(defun sql-starts-with-prompt-re ()
-  "Anchor the prompt expression at the beginning of the output line.
-Remove the start of line regexp."
-  (concat "\\`" comint-prompt-regexp))
-
-(defun sql-ends-with-prompt-re ()
-  "Anchor the prompt expression at the end of the output line.
-Match a SQL prompt or a password prompt."
-  (concat "\\(?:\\(?:" sql-prompt-regexp "\\)\\|"
-          "\\(?:" comint-password-prompt-regexp "\\)\\)\\'"))
-
 (defun sql-interactive-remove-continuation-prompt (oline)
   "Strip out continuation prompts out of the OLINE.
 
 Added to the `comint-preoutput-filter-functions' hook in a SQL
-interactive buffer.  If `sql-output-newline-count' is greater than
-zero, then an output line matching the continuation prompt is filtered
-out.  If the count is zero, then a newline is inserted into the output
-to force the output from the query to appear on a new line.
-
-The complication to this filter is that the continuation prompts
-may arrive in multiple chunks.  If they do, then the function
-saves any unfiltered output in a buffer and prepends that buffer
-to the next chunk to properly match the broken-up prompt.
-
-If the filter gets confused, it should reset and stop filtering
-to avoid deleting non-prompt output."
-
-  ;; continue gathering lines of text iff
-  ;;  + we know what a prompt looks like, and
-  ;;  + there is held text, or
-  ;;  + there are continuation prompt yet to come, or
-  ;;  + not just a prompt string
+interactive buffer.  The complication to this filter is that the
+continuation prompts may arrive in multiple chunks.  If they do,
+then the function saves any unfiltered output in a buffer and
+prepends that buffer to the next chunk to properly match the
+broken-up prompt.
+
+The filter goes into play only if something is already
+accumulated, or we're waiting for continuation
+prompts (`sql-output-newline-count' is positive).  In this case:
+- Accumulate process output into `sql-preoutput-hold'.
+- Remove any complete prompts / continuation prompts that we're waiting
+  for.
+- In case we're expecting more prompts - return all currently
+  accumulated _complete_ lines, leaving the rest for the next
+  invocation.  They will appear in the output immediately.  This way we
+  don't accumulate large chunks of data for no reason.
+- If we found all expected prompts - just return all current accumulated
+  data."
   (when (and comint-prompt-regexp
-             (or (> (length (or sql-preoutput-hold "")) 0)
-                 (> (or sql-output-newline-count 0) 0)
-                 (not (or (string-match sql-prompt-regexp oline)
-                          (and sql-prompt-cont-regexp
-                               (string-match sql-prompt-cont-regexp oline))))))
-
+             ;; We either already have something held, or expect
+             ;; prompts
+             (or sql-preoutput-hold
+                 (and sql-output-newline-count
+                      (> sql-output-newline-count 0))))
     (save-match-data
-      (let (prompt-found last-nl)
-
-        ;; Add this text to what's left from the last pass
-        (setq oline (concat sql-preoutput-hold oline)
-              sql-preoutput-hold "")
-
-        ;; If we are looking for multiple prompts
-        (when (and (integerp sql-output-newline-count)
-                   (>= sql-output-newline-count 1))
-          ;; Loop thru each starting prompt and remove it
-          (let ((start-re (sql-starts-with-prompt-re)))
-            (while (and (not (string= oline ""))
-                      (> sql-output-newline-count 0)
-                      (string-match start-re oline))
-              (setq oline (replace-match "" nil nil oline)
-                    sql-output-newline-count (1- sql-output-newline-count)
-                    prompt-found t)))
-
-          ;; If we've found all the expected prompts, stop looking
-          (if (= sql-output-newline-count 0)
-              (setq sql-output-newline-count nil)
-
-            ;; Still more possible prompts, leave them for the next pass
-            (setq sql-preoutput-hold oline
-                  oline "")))
-
-        ;; If no prompts were found, stop looking
-        (unless prompt-found
-          (setq sql-output-newline-count nil
-                oline (concat oline sql-preoutput-hold)
-                sql-preoutput-hold ""))
-
-        ;; Break up output by physical lines if we haven't hit the final prompt
-        (let ((end-re (sql-ends-with-prompt-re)))
-          (unless (and (not (string= oline ""))
-                       (string-match end-re oline)
-                       (>= (match-end 0) (length oline)))
-            ;; Find everything upto the last nl
-            (setq last-nl 0)
-            (while (string-match "\n" oline last-nl)
-              (setq last-nl (match-end 0)))
-            ;; Hold after the last nl, return upto last nl
-            (setq sql-preoutput-hold (concat (substring oline last-nl)
-                                             sql-preoutput-hold)
-                  oline (substring oline 0 last-nl)))))))
+      ;; Add this text to what's left from the last pass
+      (setq oline (concat sql-preoutput-hold oline)
+            sql-preoutput-hold nil)
+
+      ;; If we are looking for prompts
+      (when (and sql-output-newline-count
+                 (> sql-output-newline-count 0))
+        ;; Loop thru each starting prompt and remove it
+        (while (and (not (string-empty-p oline))
+                    (> sql-output-newline-count 0)
+                    (string-match comint-prompt-regexp oline))
+          (setq oline (replace-match "" nil nil oline)
+                sql-output-newline-count (1- sql-output-newline-count)))
+
+        ;; If we've found all the expected prompts, stop looking
+        (if (= sql-output-newline-count 0)
+            (setq sql-output-newline-count nil)
+          ;; Still more possible prompts, leave them for the next pass
+          (setq sql-preoutput-hold oline
+                oline "")))
+
+      ;; Lines that are now complete may be passed further
+      (when sql-preoutput-hold
+        (let ((last-nl 0))
+          (while (string-match "\n" sql-preoutput-hold last-nl)
+            (setq last-nl (match-end 0)))
+          ;; Return up to last nl, hold after the last nl
+          (setq oline (substring sql-preoutput-hold 0 last-nl)
+                sql-preoutput-hold (substring sql-preoutput-hold last-nl))
+          (when (string-empty-p sql-preoutput-hold)
+            (setq sql-preoutput-hold nil))))))
   oline)
 
+
 ;;; Sending the region to the SQLi buffer.
 (defvar sql-debug-send nil
   "Display text sent to SQL process pragmatically.")

Re: Please review: rewrite sql-interactive-remove-continuation-prompt

[Filipp Gunbin]

On 30/04/2022 00:51 +0300, Filipp Gunbin wrote:

> I tested this on multiline queries with Postgres and Cassandra, 

Just to note: I used the yet unreleased version (latest commit) of my
sql-cassandra package from ELPA, updated for this patch.

Re: Please review: rewrite sql-interactive-remove-continuation-prompt

[Lars Ingebrigtsen]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Previously we were anchoring the prompt search regexps to start/end of
> string (` and '), but now just search for regexps as they are.  Current
> regexps in sql.el all anchor to the beginning of line (^, not `).  The
> change is indended, as prompts may appear anywhere in the output.

I think that makes change, but before pushing, can you add some test
cases for these things?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Please review: rewrite sql-interactive-remove-continuation-prompt

[Filipp Gunbin]

On 30/04/2022 13:53 +0200, Lars Ingebrigtsen wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> Previously we were anchoring the prompt search regexps to start/end of
>> string (` and '), but now just search for regexps as they are.  Current
>> regexps in sql.el all anchor to the beginning of line (^, not `).  The
>> change is indended, as prompts may appear anywhere in the output.
>
> I think that makes change, but before pushing, can you add some test
> cases for these things?

Sure, added tests.

I've also CC'ed Michael, maybe he has something to say.

Thanks.


commit dfb2707195aea71470f615649057dbd4f9671639
Author: Filipp Gunbin <fgunbin@fastmail.fm>
Date:   Sat Apr 30 00:49:04 2022 +0300

    Rewrite sql-interactive-remove-continuation-prompt
    
    * lisp/progmodes/sql.el (sql-starts-with-prompt-re): Remove.
    (sql-ends-with-prompt-re): Remove
    (sql-interactive-remove-continuation-prompt): Delete prompts from
    anywhere in the process output, not just at the beginning of current
    string.  Streamline logic, describe it in docstring.
    * test/lisp/progmodes/sql-tests.el: Add tests

diff --git a/lisp/progmodes/sql.el b/lisp/progmodes/sql.el
index 5e5f5e13fe..979b743a65 100644
--- a/lisp/progmodes/sql.el
+++ b/lisp/progmodes/sql.el
@@ -3648,94 +3648,69 @@ sql-input-sender
 
 (defvar sql-preoutput-hold nil)
 
-(defun sql-starts-with-prompt-re ()
-  "Anchor the prompt expression at the beginning of the output line.
-Remove the start of line regexp."
-  (concat "\\`" comint-prompt-regexp))
-
-(defun sql-ends-with-prompt-re ()
-  "Anchor the prompt expression at the end of the output line.
-Match a SQL prompt or a password prompt."
-  (concat "\\(?:\\(?:" sql-prompt-regexp "\\)\\|"
-          "\\(?:" comint-password-prompt-regexp "\\)\\)\\'"))
-
 (defun sql-interactive-remove-continuation-prompt (oline)
   "Strip out continuation prompts out of the OLINE.
 
 Added to the `comint-preoutput-filter-functions' hook in a SQL
-interactive buffer.  If `sql-output-newline-count' is greater than
-zero, then an output line matching the continuation prompt is filtered
-out.  If the count is zero, then a newline is inserted into the output
-to force the output from the query to appear on a new line.
-
-The complication to this filter is that the continuation prompts
-may arrive in multiple chunks.  If they do, then the function
-saves any unfiltered output in a buffer and prepends that buffer
-to the next chunk to properly match the broken-up prompt.
-
-If the filter gets confused, it should reset and stop filtering
-to avoid deleting non-prompt output."
-
-  ;; continue gathering lines of text iff
-  ;;  + we know what a prompt looks like, and
-  ;;  + there is held text, or
-  ;;  + there are continuation prompt yet to come, or
-  ;;  + not just a prompt string
+interactive buffer.  The complication to this filter is that the
+continuation prompts may arrive in multiple chunks.  If they do,
+then the function saves any unfiltered output in a buffer and
+prepends that buffer to the next chunk to properly match the
+broken-up prompt.
+
+The filter goes into play only if something is already
+accumulated, or we're waiting for continuation
+prompts (`sql-output-newline-count' is positive).  In this case:
+- Accumulate process output into `sql-preoutput-hold'.
+- Remove any complete prompts / continuation prompts that we're waiting
+  for.
+- In case we're expecting more prompts - return all currently
+  accumulated _complete_ lines, leaving the rest for the next
+  invocation.  They will appear in the output immediately.  This way we
+  don't accumulate large chunks of data for no reason.
+- If we found all expected prompts - just return all current accumulated
+  data."
   (when (and comint-prompt-regexp
-             (or (> (length (or sql-preoutput-hold "")) 0)
-                 (> (or sql-output-newline-count 0) 0)
-                 (not (or (string-match sql-prompt-regexp oline)
-                          (and sql-prompt-cont-regexp
-                               (string-match sql-prompt-cont-regexp oline))))))
-
+             ;; We either already have something held, or expect
+             ;; prompts
+             (or sql-preoutput-hold
+                 (and sql-output-newline-count
+                      (> sql-output-newline-count 0))))
     (save-match-data
-      (let (prompt-found last-nl)
-
-        ;; Add this text to what's left from the last pass
-        (setq oline (concat sql-preoutput-hold oline)
-              sql-preoutput-hold "")
-
-        ;; If we are looking for multiple prompts
-        (when (and (integerp sql-output-newline-count)
-                   (>= sql-output-newline-count 1))
-          ;; Loop thru each starting prompt and remove it
-          (let ((start-re (sql-starts-with-prompt-re)))
-            (while (and (not (string= oline ""))
-                      (> sql-output-newline-count 0)
-                      (string-match start-re oline))
-              (setq oline (replace-match "" nil nil oline)
-                    sql-output-newline-count (1- sql-output-newline-count)
-                    prompt-found t)))
-
-          ;; If we've found all the expected prompts, stop looking
-          (if (= sql-output-newline-count 0)
-              (setq sql-output-newline-count nil)
-
-            ;; Still more possible prompts, leave them for the next pass
-            (setq sql-preoutput-hold oline
-                  oline "")))
-
-        ;; If no prompts were found, stop looking
-        (unless prompt-found
-          (setq sql-output-newline-count nil
-                oline (concat oline sql-preoutput-hold)
-                sql-preoutput-hold ""))
-
-        ;; Break up output by physical lines if we haven't hit the final prompt
-        (let ((end-re (sql-ends-with-prompt-re)))
-          (unless (and (not (string= oline ""))
-                       (string-match end-re oline)
-                       (>= (match-end 0) (length oline)))
-            ;; Find everything upto the last nl
-            (setq last-nl 0)
-            (while (string-match "\n" oline last-nl)
-              (setq last-nl (match-end 0)))
-            ;; Hold after the last nl, return upto last nl
-            (setq sql-preoutput-hold (concat (substring oline last-nl)
-                                             sql-preoutput-hold)
-                  oline (substring oline 0 last-nl)))))))
+      ;; Add this text to what's left from the last pass
+      (setq oline (concat sql-preoutput-hold oline)
+            sql-preoutput-hold nil)
+
+      ;; If we are looking for prompts
+      (when (and sql-output-newline-count
+                 (> sql-output-newline-count 0))
+        ;; Loop thru each starting prompt and remove it
+        (while (and (not (string-empty-p oline))
+                    (> sql-output-newline-count 0)
+                    (string-match comint-prompt-regexp oline))
+          (setq oline (replace-match "" nil nil oline)
+                sql-output-newline-count (1- sql-output-newline-count)))
+
+        ;; If we've found all the expected prompts, stop looking
+        (if (= sql-output-newline-count 0)
+            (setq sql-output-newline-count nil)
+          ;; Still more possible prompts, leave them for the next pass
+          (setq sql-preoutput-hold oline
+                oline "")))
+
+      ;; Lines that are now complete may be passed further
+      (when sql-preoutput-hold
+        (let ((last-nl 0))
+          (while (string-match "\n" sql-preoutput-hold last-nl)
+            (setq last-nl (match-end 0)))
+          ;; Return up to last nl, hold after the last nl
+          (setq oline (substring sql-preoutput-hold 0 last-nl)
+                sql-preoutput-hold (substring sql-preoutput-hold last-nl))
+          (when (string-empty-p sql-preoutput-hold)
+            (setq sql-preoutput-hold nil))))))
   oline)
 
+
 ;;; Sending the region to the SQLi buffer.
 (defvar sql-debug-send nil
   "Display text sent to SQL process pragmatically.")
diff --git a/test/lisp/progmodes/sql-tests.el b/test/lisp/progmodes/sql-tests.el
index 7e36d845e2..c644d115df 100644
--- a/test/lisp/progmodes/sql-tests.el
+++ b/test/lisp/progmodes/sql-tests.el
@@ -425,5 +425,85 @@ sql-tests-comint-automatic-password
   (let ((sql-password "password"))
     (should (equal "password" (sql-comint-automatic-password "")))))
 
+
+\f
+;; Tests for sql-interactive-remove-continuation-prompt
+
+(defmacro sql-tests-remove-cont-prompts-harness (&rest body)
+  "Set-up and tear-down for tests of
+`sql-interactive-remove-continuation-prompt'."
+  (declare (indent 0))
+  `(let ((comint-prompt-regexp "^ +\\.\\{3\\} ")
+         (sql-output-newline-count nil)
+         (sql-preoutput-hold nil))
+     ,@body
+     (should (null sql-output-newline-count))
+     (should (null sql-preoutput-hold))))
+
+(ert-deftest sql-tests-remove-cont-prompts-pass-through ()
+  "Test that `sql-interactive-remove-continuation-prompt' just
+passes the output line through when it doesn't expect prompts."
+  (sql-tests-remove-cont-prompts-harness
+   (should
+    (equal " ... "
+           (sql-interactive-remove-continuation-prompt
+            " ... ")))))
+
+(ert-deftest sql-tests-remove-cont-prompts-anchored-successive ()
+  "Test that `sql-interactive-remove-continuation-prompt' is able
+to delete multiple prompts (anchored to bol) even if they appear
+in a single line, but not more than `sql-output-newline-count'."
+  (sql-tests-remove-cont-prompts-harness
+   (setq sql-output-newline-count 2)
+   (should
+    (equal
+     ;; 2 of 3 prompts are deleted
+     "some output ... more output...\n\
+ ... \n\
+output after prompt"
+     (sql-interactive-remove-continuation-prompt
+      "some output ... more output...\n\
+ ...  ...  ... \n\
+output after prompt")))))
+
+(ert-deftest sql-tests-remove-cont-prompts-collect-chunked-output ()
+  "Test that `sql-interactive-remove-continuation-prompt' properly
+collects output when output arrives in chunks, with prompts
+intermixed."
+  (sql-tests-remove-cont-prompts-harness
+   (setq sql-output-newline-count 2)
+
+   ;; Part of first prompt gets held.  Complete line is passed
+   ;; through.
+   (should (equal "line1\n"
+                  (sql-interactive-remove-continuation-prompt
+                   "line1\n ..")))
+   (should (equal " .." sql-preoutput-hold))
+   (should (equal 2 sql-output-newline-count))
+
+   ;; First prompt is complete - remove it.  Hold part of line2.
+   (should (equal ""
+                  (sql-interactive-remove-continuation-prompt ". li")))
+   (should (equal "li" sql-preoutput-hold))
+   (should (equal 1 sql-output-newline-count))
+
+   ;; Remove second prompt.  Flush output & don't hold / process any
+   ;; output further on.
+   (should (equal "line2\nli"
+                  (sql-interactive-remove-continuation-prompt "ne2\n ... li")))
+   (should (null sql-preoutput-hold))
+   (should (null sql-output-newline-count))
+   (should (equal "line3\n ... "
+                  (sql-interactive-remove-continuation-prompt "line3\n ... ")))))
+
+(ert-deftest sql-tests-remove-cont-prompts-flush-held ()
+  "Test that when we don't wait for prompts,
+ `sql-interactive-remove-continuation-prompt' just 'flushes' held
+ output, with no prompt processing."
+  (sql-tests-remove-cont-prompts-harness
+   (setq sql-preoutput-hold "line1\n ..")
+   (should (equal "line1\n ... line2 .."
+                  (sql-interactive-remove-continuation-prompt ". line2 ..")))))
+
 (provide 'sql-tests)
 ;;; sql-tests.el ends here

Re: Please review: rewrite sql-interactive-remove-continuation-prompt

[Lars Ingebrigtsen]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Sure, added tests.

Thanks; pushed to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Please review: rewrite sql-interactive-remove-continuation-prompt

[Filipp Gunbin]

On 03/05/2022 12:35 +0200, Lars Ingebrigtsen wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> Sure, added tests.
>
> Thanks; pushed to Emacs 29.

Thanks.  Should any bugs arise (like with DBs I haven't tested with),
please assign them to me.

Filipp

Re: Please review: rewrite sql-interactive-remove-continuation-prompt

[Lars Ingebrigtsen]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Thanks.  Should any bugs arise (like with DBs I haven't tested with),
> please assign them to me.

I'll try to remember.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no