Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Friedrich Beckmann]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]

Hi Stefan, hi Vasilij,

thanks for your response regarding PSPP mode for emacs in GNU Elpa. I think we can simply copy pspp-mode.el to elpa.git and do the necessary modifications there. The keyword list comes from the SPSS command syntax which is also described here:

https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index <https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index>

SPSS is a commercial statistics software from IBM and PSPP is a GNU free software replacement for SPSS.

Regards

Friedrich

> Anfang der weitergeleiteten Nachricht:
> 
> Von: Stefan Monnier <monnier@iro.umontreal.ca>
> Betreff: Aw: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting
> Datum: 30. Juni 2020 um 16:16:21 MESZ
> An: Friedrich Beckmann <friedrich.beckmann@gmx.de>
> 
> Sorry for not answering earlier, your mail sneaked through my net.
> 
> IIUC the package's copyright is already assigned to the FSF, so we can
> add it to GNU ELPA, indeed.
> 
> Would you be interested in moving it to live in elpa.git instead of
> pspp.git (that would make things a bit easier, avoiding the need to
> merge pspp-mode.el changes from pspp.git to elpa.git every once in
> a while).
> 
> Also, do you happen to know where the long list of identifiers in
> `pspp-font-lock-keywords` come from?  It would be good to add a comment
> describing how they were generated and from which dataset, so as to know
> how/when to update them.
> 
> 
>        Stefan
> 
> 
> Friedrich Beckmann [2020-06-20 21:53:25] wrote:
> 
>> Hi,
>> 
>> I contribute to the GNU PSPP software and maintain the pspp debian
>> package. The package contains a pspp-mode.el file which provides
>> the pspp-mode for syntax highlighting spss syntax files. Currently
>> the file is installed via dh-elpa in the debian package but I think
>> it should be in the normal gnu elpa distribution.
>> 
>> The file is available here:
>> 
>> http://git.savannah.gnu.org/cgit/pspp.git/tree/pspp-mode.el
>> 
>> Can you integrate that in gnu elpa?
>> 
>> Regards
>> 
>> Friedrich
> 


[-- Attachment #2: Type: text/html, Size: 4913 bytes --]

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[John Darrington]

I've updated pspp-mode.el in the pspp repository so that it has the
format that lisp hackers like.

I've also added a long overdue change to make the keywords case
insenstive.

J'

On Tue, Jun 30, 2020 at 11:55:46PM +0200, Friedrich Beckmann wrote:
     Hi Stefan, hi Vasilij,
     
     thanks for your response regarding PSPP mode for emacs in GNU Elpa. I think we can simply copy pspp-mode.el to elpa.git and do the necessary modifications there. The keyword list comes from the SPSS command syntax which is also described here:
     
     https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index <https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index>
     
     SPSS is a commercial statistics software from IBM and PSPP is a GNU free software replacement for SPSS.
     
     Regards
     
     Friedrich
     
     > Anfang der weitergeleiteten Nachricht:
     > 
     > Von: Stefan Monnier <monnier@iro.umontreal.ca>
     > Betreff: Aw: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting
     > Datum: 30. Juni 2020 um 16:16:21 MESZ
     > An: Friedrich Beckmann <friedrich.beckmann@gmx.de>
     > 
     > Sorry for not answering earlier, your mail sneaked through my net.
     > 
     > IIUC the package's copyright is already assigned to the FSF, so we can
     > add it to GNU ELPA, indeed.
     > 
     > Would you be interested in moving it to live in elpa.git instead of
     > pspp.git (that would make things a bit easier, avoiding the need to
     > merge pspp-mode.el changes from pspp.git to elpa.git every once in
     > a while).
     > 
     > Also, do you happen to know where the long list of identifiers in
     > `pspp-font-lock-keywords` come from?  It would be good to add a comment
     > describing how they were generated and from which dataset, so as to know
     > how/when to update them.
     > 
     > 
     >        Stefan
     > 
     > 
     > Friedrich Beckmann [2020-06-20 21:53:25] wrote:
     > 
     >> Hi,
     >> 
     >> I contribute to the GNU PSPP software and maintain the pspp debian
     >> package. The package contains a pspp-mode.el file which provides
     >> the pspp-mode for syntax highlighting spss syntax files. Currently
     >> the file is installed via dh-elpa in the debian package but I think
     >> it should be in the normal gnu elpa distribution.
     >> 
     >> The file is available here:
     >> 
     >> http://git.savannah.gnu.org/cgit/pspp.git/tree/pspp-mode.el
     >> 
     >> Can you integrate that in gnu elpa?
     >> 
     >> Regards
     >> 
     >> Friedrich
     > 
     

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Stefan Monnier]

> thanks for your response regarding PSPP mode for emacs in GNU ELPA.
> I think we can simply copy pspp-mode.el to elpa.git and do the necessary
> modifications there.

Having the development/maintenance of `pspp-mode.el` take place in
`elpa.git` is indeed the better option from where I stand, but
would `pspp-mode.el` then be removed from the PSPP Git repository (to
avoid a risk of the two versions going out of sync)?

> The keyword list comes from the SPSS command syntax which is also
> described here:
>
> https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index
> <https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index>

So I presume the corresponding Elisp code was hand-written?


        Stefan

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[John Darrington]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On Sat, Jul 04, 2020 at 09:12:33AM -0400, Stefan Monnier wrote:
     > thanks for your response regarding PSPP mode for emacs in GNU ELPA.
     > I think we can simply copy pspp-mode.el to elpa.git and do the necessary
     > modifications there.
     
     Having the development/maintenance of `pspp-mode.el` take place in
     `elpa.git` is indeed the better option from where I stand, but
     would `pspp-mode.el` then be removed from the PSPP Git repository (to
     avoid a risk of the two versions going out of sync)?


Probably it would be - but I don't think a decision has been made yet.

     
     > The keyword list comes from the SPSS command syntax which is also
     > described here:
     >
     > https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index
     > <https://www.gnu.org/software/pspp/manual/html_node/Command-Index.html#Command-Index>
     
     So I presume the corresponding Elisp code was hand-written?

Yes.   That list doesn't change very often.  Perhaps we should check
that it is up to date now.

J'

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Stefan Monnier]

Hi John,

So pspp-mode.el is now in GNU ELPA: https://elpa.gnu.org/packages/pspp-mode.html

> Probably it would be - but I don't think a decision has been made yet.

Fair enough.  But to avoid divergence, it would be good to do it.

> Yes.   That list doesn't change very often.  Perhaps we should check
> that it is up to date now.

I was looking at the code a bit and am about to install some changes to
it, but I bumped into some questions about the PSPP syntax of comments.

The patch below does:

    * pspp-mode.el: Prefer `setq` to `set '`.
    (pspp-mode-hook): Let define-derived-mode declare it for us.
    (pspp-mode-map): Don't bind `C-j` since this is not specific to PSPP
    but is a user preference.  Nowadays `electric-indent-mode` is used
    instead anyway.  Also, use a local var for the temp.
    (pspp--downcase-list, pspp--upcase-list, pspp--updown-list):
    Use `mapcar` instead of an inefficient recursion.
    (pspp-indent-line): Comment out unused var `verbatim`.
    (pspp-comment-start-line-p): Fix incomplete escaping.
    (pspp-mode-syntax-table): Don't use `w` for non-word symbol constituents
    since it breaks the expected behavior of forward-word.
    Use a short non-prefixed name for the local var.
    Tweak the syntax-table for comments.
    (pspp--syntax-propertize): New var.
    (pspp-font-lock-keywords): Use [:alnum:] and [:alpha:].
    (pspp-mode): Use define-derived-mode.

But the main issue is the comment syntax.  I'm trying to handle them
right using `syntax-propertize`, but the patch can't handle the
t-test-sps I found in PSPP's Git.  I don't understand what the real
syntax should be.  The doc seems to suggest that comments start with `*`
or `comment` (in the position of the beginning of a command?)
and end with a `.` at an end of line or with an empty line, but in
t-test.sps I see:

    * Females have gender 0
    * Create 8 female cases
    loop #i = 1 to 8.

where the comment does not seem to be terminated (neither by a `.` nor
by an empty line).  What am I missing?


        Stefan


diff --git a/pspp-mode.el b/pspp-mode.el
index ca9bbc931..157475234 100644
--- a/pspp-mode.el
+++ b/pspp-mode.el
@@ -1,4 +1,4 @@
-;;; pspp-mode.el --- Major mode for editing PSPP files
+;;; pspp-mode.el --- Major mode for editing PSPP files  -*- lexical-binding: t; -*-
 
 ;; Copyright (C) 2005,2018,2020 Free Software Foundation, Inc.
 ;; Author: Scott Andrew Borton <scott@pp.htv.fi>
@@ -25,13 +25,10 @@
 ;; along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 ;;; Code:
-(defvar pspp-mode-hook nil)
-
 
 (defvar pspp-mode-map
-  (let ((pspp-mode-map (make-keymap)))
-    (define-key pspp-mode-map "\C-j" 'newline-and-indent)
-    pspp-mode-map)
+  (let ((map (make-sparse-keymap)))
+    map)
   "Keymap for PSPP major mode")
 
 
@@ -48,37 +45,33 @@
       (while (not (or
                    (or (bobp) pspp-end-of-block-found)
                    pspp-start-of-block-found))
-        (set 'pspp-end-of-block-found
-             (looking-at "^[ \t]*\\(END\\|end\\)[\t ]+\\(DATA\\|data\\)\."))
-        (set 'pspp-start-of-block-found
-             (looking-at "^[ \t]*\\(BEGIN\\|begin\\)[\t ]+\\(DATA\\|data\\)"))
+        (setq pspp-end-of-block-found
+              (looking-at "^[ \t]*\\(END\\|end\\)[\t ]+\\(DATA\\|data\\)\."))
+        (setq pspp-start-of-block-found
+              (looking-at "^[ \t]*\\(BEGIN\\|begin\\)[\t ]+\\(DATA\\|data\\)"))
         (forward-line -1))
 
       (and pspp-start-of-block-found (not pspp-end-of-block-found)))))
 
 
-(defconst pspp-indent-width
+(defconst pspp-indent-width             ;FIXME: Should be a defcustom.
   2
   "size of indent")
 
 
 (defun pspp--downcase-list (l)
-  "Takes a list of strings and returns that list with all elements downcased"
-  (if l
-      (cons (downcase (car l)) (pspp--downcase-list (cdr l)))
-    nil))
+  "Take a list of strings and return that list with all elements downcased."
+  (mapcar #'downcase l))
 
 
 (defun pspp--upcase-list (l)
-  "Takes a list of strings and returns that list with all elements upcased"
-  (if l
-      (cons (upcase (car l)) (pspp--upcase-list (cdr l)))
-    nil))
+  "Take a list of strings and return that list with all elements upcased."
+  (mapcar #'upcase l))
 
 
 (defun pspp--updown-list (l)
-  "Takes a list of strings and returns that list with all elements upcased
-and downcased"
+  "Take a list of strings and return that list with all elements upcased
+and downcased."
   (append (pspp--upcase-list l) (pspp--downcase-list l)))
 
 
@@ -87,9 +80,10 @@ and downcased"
           (regexp-opt (pspp--updown-list '("DO"
                                      "BEGIN"
                                      "LOOP"
-                                     "INPUT")) t)
+                                     "INPUT"))
+                      t)
           "[\t ]+")
-  "constructs which cause indentation")
+  "Constructs which cause indentation.")
 
 
 (defconst pspp-unindenters
@@ -98,16 +92,17 @@ and downcased"
                                      "DATA"
                                      "LOOP"
                                      "REPEAT"
-                                     "INPUT")) t)
+                                     "INPUT"))
+                      t)
           "[\t ]*")
   ;; Note that "END CASE" and "END FILE" do not unindent.
-  "constructs which cause end of indentation")
+  "Constructs which cause end of indentation.")
 
 
 (defun pspp-indent-line ()
   "Indent current line as PSPP code."
   (beginning-of-line)
-  (let ((verbatim nil)
+  (let (;; (verbatim nil)
         (the-indent 0)    ; Default indent to column 0
         (case-fold-search t))
     (if (bobp)
@@ -127,7 +122,7 @@ and downcased"
                     (setq within-command t))))))
       ;; If we're not at the start of a new command, then add an indent.
       (if within-command
-          (set 'the-indent (+ 1 the-indent))))
+          (setq the-indent (+ 1 the-indent))))
     ;; Set the indentation according to the DO - END blocks
     (save-excursion
       (beginning-of-line)
@@ -137,37 +132,38 @@ and downcased"
             (cond ((save-excursion
                      (forward-line -1)
                      (looking-at pspp-indenters))
-                   (set 'the-indent (+ the-indent 1)))
+                   (setq the-indent (+ the-indent 1)))
 
                   ((looking-at pspp-unindenters)
-                   (set 'the-indent (- the-indent 1)))))
+                   (setq the-indent (- the-indent 1)))))
         (forward-line -1)))
 
     (save-excursion
       (beginning-of-line)
       (if (looking-at "^[\t ]*ELSE")
-          (set 'the-indent (- the-indent 1))))
+          (setq the-indent (- the-indent 1))))
 
     ;; Stuff in the data-blocks should be untouched
     (if (not (pspp-data-block-p)) (indent-line-to (* pspp-indent-width the-indent)))))
 
 
 (defun pspp-comment-start-line-p ()
-  "Returns t if the current line is the first line of a comment, nil otherwise"
+  "Return t if the current line is the first line of a comment, nil otherwise"
   (beginning-of-line)
-  (or (looking-at "^\*")
+  (or (looking-at "^\\*")
       (looking-at "^[\t ]*comment[\t ]")
       (looking-at "^[\t ]*COMMENT[\t ]")))
 
 
 (defun pspp-comment-end-line-p ()
-  "Returns t if the current line is the candidate for the last line of a comment, nil otherwise"
+  "Return t if the current line is the candidate for the last line of a comment, nil otherwise"
   (beginning-of-line)
   (looking-at ".*\\.[\t ]*$"))
 
 
 (defun pspp-comment-p ()
-  "Returns t if point is in a comment.  Nil otherwise."
+  "Return t if point is in a comment.  Nil otherwise."
+  ;; FIXME: Use `syntax-ppss'?
   (if (pspp-data-block-p)
       nil
     (let ((pspp-comment-start-found nil)
@@ -180,16 +176,16 @@ and downcased"
                     (not pspp-comment-start-found)
                     (not pspp-comment-end-found))
           (beginning-of-line)
-          (if (pspp-comment-start-line-p) (set 'pspp-comment-start-found t))
+          (if (pspp-comment-start-line-p) (setq pspp-comment-start-found t))
           (if (bobp)
-              (set 'pspp-comment-end-found nil)
+              (setq pspp-comment-end-found nil)
             (save-excursion
               (forward-line -1)
-              (if (pspp-comment-end-line-p) (set 'pspp-comment-end-found t))))
-          (set 'lines (forward-line -1))))
+              (if (pspp-comment-end-line-p) (setq pspp-comment-end-found t))))
+          (setq lines (forward-line -1))))
 
       (save-excursion
-        (set 'pspp-single-line-comment (and
+        (setq pspp-single-line-comment (and
                                         (pspp-comment-start-line-p)
                                         (pspp-comment-end-line-p))))
 
@@ -198,30 +194,49 @@ and downcased"
 
 
 (defvar pspp-mode-syntax-table
-  (let ((x-pspp-mode-syntax-table (make-syntax-table)))
+  (let ((st (make-syntax-table)))
 
     ;; Special chars allowed in variables
-    (modify-syntax-entry ?#  "w" x-pspp-mode-syntax-table)
-    (modify-syntax-entry ?@  "w" x-pspp-mode-syntax-table)
-    (modify-syntax-entry ?$  "w" x-pspp-mode-syntax-table)
+    (modify-syntax-entry ?#  "_" st)
+    (modify-syntax-entry ?@  "_" st)
+    (modify-syntax-entry ?$  "_" st)
 
     ;; Comment syntax
-    ;;  This is incomplete, because:
-    ;;  a) Comments can also be given by COMMENT
-    ;;  b) The sequence .\n* is interpreted incorrectly.
-
-    (modify-syntax-entry ?*  ". 2" x-pspp-mode-syntax-table)
-    (modify-syntax-entry ?.  ". 3" x-pspp-mode-syntax-table)
-    (modify-syntax-entry ?\n  "- 41" x-pspp-mode-syntax-table)
+    ;; See `pspp--syntax-propertize' for the details.
+    (modify-syntax-entry ?*  "<" st)
+    (modify-syntax-entry ?.  ". 3" st)
+    (modify-syntax-entry ?\n  "  34" st)
 
     ;; String delimiters
-    (modify-syntax-entry ?'  "\"" x-pspp-mode-syntax-table)
-    (modify-syntax-entry ?\"  "\"" x-pspp-mode-syntax-table)
-
-    x-pspp-mode-syntax-table)
-
-  "Syntax table for pspp-mode")
-
+    (modify-syntax-entry ?'  "\"" st)
+    (modify-syntax-entry ?\"  "\"" st)
+
+    st)
+
+  "Syntax table for pspp-mode.")
+
+(defconst pspp--syntax-propertize
+  (syntax-propertize-rules
+   ("\\*"
+    (0 (unless (save-excursion
+                 (goto-char (match-beginning 0))
+                 (skip-chars-backward " \t\n")
+                 (memq (char-before) '(nil ?\.)))
+         (string-to-syntax "."))))
+   ("\\_<\\([Cc]\\)[Oo][Mm][Mm][Ee][Nn][Tt]\\_>"
+    (1 (when (save-excursion
+               (goto-char (match-beginning 0))
+               (skip-chars-backward " \t\n")
+               (memq (char-before) '(nil ?\.)))
+         (string-to-syntax "<"))))
+   ;; PSPP, like Pascal, uses '' and "" rather than \' and \" to escape quotes.
+   ("''\\|\"\"" (0 (if (save-excursion
+                         (nth 3 (syntax-ppss (match-beginning 0))))
+                       (string-to-syntax ".")
+                     ;; In case of 3 or more quotes in a row, only advance
+                     ;; one quote at a time.
+                     (forward-char -1)
+                     nil)))))
 
 (defconst pspp-font-lock-keywords
   (list (cons
@@ -627,29 +642,28 @@ and downcased"
                                "YRMODA"))
                              t) "\\>")  'font-lock-function-name-face)
 
-        '( "\\<[#$@a-zA-Z][a-zA-Z0-9_]*\\>" . font-lock-variable-name-face))
+        ;; FIXME: The doc at
+        ;; https://www.gnu.org/software/pspp/manual/html_node/Tokens.html
+        ;; does not include `$' in the allowed first chars and it includes
+        ;; `. $ # @' additionally to `_' in the subsequent chars.
+        '( "\\<[#$@[:alpha:]][[:alnum:]_]*\\>" . font-lock-variable-name-face))
   "Highlighting expressions for PSPP mode.")
 
 
 ;;;###autoload
-(defun pspp-mode ()
-  (interactive)
-  (kill-all-local-variables)
-  (use-local-map pspp-mode-map)
-  (set-syntax-table pspp-mode-syntax-table)
-
+(define-derived-mode pspp-mode prog-mode "PSPP"
+  "Major mode to edit PSPP files."
   (set (make-local-variable 'font-lock-keywords-case-fold-search) t)
   (set (make-local-variable 'font-lock-defaults) '(pspp-font-lock-keywords))
 
-  (set (make-local-variable 'indent-line-function) 'pspp-indent-line)
+  (set (make-local-variable 'indent-line-function) #'pspp-indent-line)
   (set (make-local-variable 'comment-start) "* ")
   (set (make-local-variable 'compile-command)
        (concat "pspp "
                buffer-file-name))
 
-  (setq major-mode 'pspp-mode)
-  (setq mode-name "PSPP")
-  (run-hooks 'pspp-mode-hook))
+  (set (make-local-variable 'syntax-propertize-function)
+       pspp--syntax-propertize))
 
 (provide 'pspp-mode)
 

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[John Darrington]

On Sat, Jul 04, 2020 at 06:45:56PM -0400, Stefan Monnier wrote:
     Hi John,
     
     
     But the main issue is the comment syntax.  I'm trying to handle them
     right using `syntax-propertize`, but the patch can't handle the
     t-test-sps I found in PSPP's Git.  I don't understand what the real
     syntax should be.  The doc seems to suggest that comments start with `*`
     or `comment` (in the position of the beginning of a command?)
     and end with a `.` at an end of line or with an empty line, but in
     t-test.sps I see:
     
         * Females have gender 0
         * Create 8 female cases
         loop #i = 1 to 8.
     
     where the comment does not seem to be terminated (neither by a `.` nor
     by an empty line).  What am I missing?
     
I think the docs are misleading here.  As I understand the rules:

 COMMENT at the start of a line (optionally preceded by whitespace) begins a comment.
 * at the start of a line (optionally preceded by whitespace) begins a comment.
 In general, comments are terminated by the end of a line.

Maybe Ben could say for sure?

Unfortunately, the spss syntax is poorly formed, poorly documented and highly irregular.
It was kindof bodged together in the 1960s and we've been stuck with it since.

J'

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Stefan Monnier]

>  COMMENT at the start of a line (optionally preceded by whitespace) begins a comment.
>  * at the start of a line (optionally preceded by whitespace) begins a comment.
>  In general, comments are terminated by the end of a line.

That would mean that the doc needs to be fixed, since it says at
https://www.gnu.org/software/pspp/manual/html_node/COMMENT.html:

    COMMENT can extend over any number of lines.  Don’t forget to terminate
    it with a dot or a blank line.

I don't fully understand what `segmenter_parse_comment_1__` does in
`src/language/lexer/segment.c` but it seems to suggest that the doc is
right, yet the `t-test.sps` file strongly suggests that the reality
is different.

Could someone enlighten me as to the actual syntax of comments for .sps
files (maybe it's not defined in `src/language/lexer/segment.c` but in
some other file)?


        Stefan

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Ben Pfaff]

On Sun, Jul 5, 2020 at 7:59 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> >  COMMENT at the start of a line (optionally preceded by whitespace) begins a comment.
> >  * at the start of a line (optionally preceded by whitespace) begins a comment.
> >  In general, comments are terminated by the end of a line.
>
> That would mean that the doc needs to be fixed, since it says at
> https://www.gnu.org/software/pspp/manual/html_node/COMMENT.html:
>
>     COMMENT can extend over any number of lines.  Don’t forget to terminate
>     it with a dot or a blank line.
>
> I don't fully understand what `segmenter_parse_comment_1__` does in
> `src/language/lexer/segment.c` but it seems to suggest that the doc is
> right, yet the `t-test.sps` file strongly suggests that the reality
> is different.
>
> Could someone enlighten me as to the actual syntax of comments for .sps
> files (maybe it's not defined in `src/language/lexer/segment.c` but in
> some other file)?

I think that my parallel contribution to the thread here explains this,
but if not then please do ask for clarification.

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Ben Pfaff]

On Sat, Jul 4, 2020 at 3:46 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> But the main issue is the comment syntax.  I'm trying to handle them
> right using `syntax-propertize`, but the patch can't handle the
> t-test-sps I found in PSPP's Git.  I don't understand what the real
> syntax should be.  The doc seems to suggest that comments start with `*`
> or `comment` (in the position of the beginning of a command?)
> and end with a `.` at an end of line or with an empty line, but in
> t-test.sps I see:
>
>     * Females have gender 0
>     * Create 8 female cases
>     loop #i = 1 to 8.
>
> where the comment does not seem to be terminated (neither by a `.` nor
> by an empty line).  What am I missing?

SPSS syntax is bizarre. It has two modes: "batch" and "interactive" mode.
A syntax file might be written in either syntax. There's no way to easily guess
which one.

In "interactive" syntax, a command ends with a line that ends with . or
with a blank line.

In "batch" syntax, a command mainly ends with a line that is not indented.
I think that a blank line also works in this syntax. I've forgotten some of
the details (although they should be documented).

PSPP can be configured to interpret files in either syntax mode. It's a pain
requiring users to do this, so by default it tries to guess on a line-by-line
basis: if a line is not indented and if it begins with keywords that are an
SPSS command name, then it assumes that it is the beginning of a command
and that the previous command ended. This is nasty also, but it works
well most of the time regardless of what syntax mode the file was written
in.

I think that's why the file above gets treated the way it does.

Maybe PSPP should treat comment commands specially. That would
be yet another special case in the lexer/parser, but maybe it is warranted.

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Stefan Monnier]

Hi Ben,

Thanks for chiming in,

> SPSS syntax is bizarre. It has two modes: "batch" and "interactive" mode.

Aha!

> A syntax file might be written in either syntax.  There's no way to
> easily guess which one.
>
> In "interactive" syntax, a command ends with a line that ends with . or
> with a blank line.

So that's the syntax that's documented in the Texinfo, and that's the
syntax handled by `segmenter_parse_comment_1__`?

> PSPP can be configured to interpret files in either syntax mode. It's a pain
> requiring users to do this, so by default it tries to guess on a line-by-line
> basis: if a line is not indented and if it begins with keywords that are an
> SPSS command name, then it assumes that it is the beginning of a command
> and that the previous command ended. This is nasty also, but it works
> well most of the time regardless of what syntax mode the file was written
> in.

Thanks.
I guess `pspp-mode.el` should try to reproduce the same DWIMish behavior.

Yet another question: do comments (either "*" or "COMMENT") always start
in column 0 or can they start indented or even on the same line as some
other command, as in:

    input program.  * let's get started.

> I think that's why the file above gets treated the way it does.
>
> Maybe PSPP should treat comment commands specially. That would
> be yet another special case in the lexer/parser, but maybe it is warranted.

Of course, I prefer it when comments are handled in the lexer rather
than in the parser, so they can occur anywhere between tokens, but
I expect that this boat has sailed ;-)


        Stefan

Re: Fwd: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Ben Pfaff]

On Sun, Jul 5, 2020 at 3:01 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > SPSS syntax is bizarre. It has two modes: "batch" and "interactive" mode.
>
> Aha!
>
> > A syntax file might be written in either syntax.  There's no way to
> > easily guess which one.
> >
> > In "interactive" syntax, a command ends with a line that ends with . or
> > with a blank line.
>
> So that's the syntax that's documented in the Texinfo, and that's the
> syntax handled by `segmenter_parse_comment_1__`?

I think that these modes are documented in the Texinfo, at least here:
https://www.gnu.org/software/pspp/manual/html_node/Syntax-Variants.html
Maybe there is some other place that the two forms are not
acknowledged. If so, then I guess that I should fix that.

> > PSPP can be configured to interpret files in either syntax mode. It's a pain
> > requiring users to do this, so by default it tries to guess on a line-by-line
> > basis: if a line is not indented and if it begins with keywords that are an
> > SPSS command name, then it assumes that it is the beginning of a command
> > and that the previous command ended. This is nasty also, but it works
> > well most of the time regardless of what syntax mode the file was written
> > in.
>
> Thanks.
> I guess `pspp-mode.el` should try to reproduce the same DWIMish behavior.
>
> Yet another question: do comments (either "*" or "COMMENT") always start
> in column 0 or can they start indented or even on the same line as some
> other command, as in:
>
>     input program.  * let's get started.

SPSS syntax doesn't allow for more than one command per line.  That's
because all the ways of ending a command, in both modes, involve a
newline. Above, for example, the '.' does not end a command because it
is not the last non-blank character on the line. Instead, it would be a
syntax error because '.' by itself isn't valid syntax for the INPUT
PROGRAM command.

Comments that begin with * or COMMENT can start in any column,
though, since there can be white space at the beginning of the line.
But they have to follow the rules for starting SPSS commands. It's
sort of like the : command in the Bourne shell: everything after the
: keyword is ignored, but it still has to follow the Bourne shell syntax
rules in ways that things that come after # do not.

> > I think that's why the file above gets treated the way it does.
> >
> > Maybe PSPP should treat comment commands specially. That would
> > be yet another special case in the lexer/parser, but maybe it is warranted.
>
> Of course, I prefer it when comments are handled in the lexer rather
> than in the parser, so they can occur anywhere between tokens, but
> I expect that this boat has sailed ;-)

I know this syntax is insane. I didn't design it, I just implemented it.
I think it's the way it is because it was designed for IBM 80-column
cards back in the 1960s. SPSS is that old.

However, SPSS has *another* comment syntax as well. It looks like
I didn't document this one, although PSPP supports it. These comments
start with /* and end with */ or at the end of the line, whichever comes
first. I'll make a note to add this to the documentation.

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Friedrich Beckmann]

Hi,

i think the benefit of having the pspp-mode in GNU ELPA is that I can load it in emacs via the standard elpa way, i.e. M-x package-list—packages and then select pspp-mode - which is working right now because Stefan included pspp-mode.el in the GNU
ELPA repository.

Until now I included the pspp-mode.el in the pspp debian package where it is then installed as an „external“ elpa package. The benefit is that once I install pspp, then the pspp-mode is autoloaded in emacs right away, i.e. when I open a .sps file, then the mode is immediately working. I think it is nice that I do not even need to know ELPA and the mode works in emacs right away when I install the debian pspp package. So maybe that is a reason to keep pspp-mode.el in the distribution. An alternative way would be  to find a way that installing the pspp package will „activate“ the pspp-mode.el from GNU ELPA.

Friedrich

> Am 04.07.2020 um 17:15 schrieb John Darrington <john@darrington.wattle.id.au>:
> 
> On Sat, Jul 04, 2020 at 09:12:33AM -0400, Stefan Monnier wrote:
>> thanks for your response regarding PSPP mode for emacs in GNU ELPA.
>> I think we can simply copy pspp-mode.el to elpa.git and do the necessary
>> modifications there.
> 
>     Having the development/maintenance of `pspp-mode.el` take place in
>     `elpa.git` is indeed the better option from where I stand, but
>     would `pspp-mode.el` then be removed from the PSPP Git repository (to
>     avoid a risk of the two versions going out of sync)?
> 
> 
> Probably it would be - but I don't think a decision has been made yet.
> 

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[John Darrington]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

I have removed pspp-mode.el from the PSPP repository.

If debian or other packagers want to have pspp-mode automatically
installed, then I guess it would be as simple as putting

(package-install 'pspp-mode nil) 

in the site-emacs file.

J'

On Sun, Jul 05, 2020 at 11:44:23AM +0200, Friedrich Beckmann wrote:
     
     Until now I included the pspp-mode.el in the pspp debian package where it is then installed as an ???external??? elpa package. The benefit is that once I install pspp, then the pspp-mode is autoloaded in emacs right away, i.e. when I open a .sps file, then the mode is immediately working. I think it is nice that I do not even need to know ELPA and the mode works in emacs right away when I install the debian pspp package. So maybe that is a reason to keep pspp-mode.el in the distribution. An alternative way would be  to find a way that installing the pspp package will ???activate??? the pspp-mode.el from GNU ELPA.
     

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Sean Whitton]

Hello,

On Sun 05 Jul 2020 at 11:44AM +02, Friedrich Beckmann wrote:

> i think the benefit of having the pspp-mode in GNU ELPA is that I can load it in emacs via the standard elpa way, i.e. M-x package-list—packages and then select pspp-mode - which is working right now because Stefan included pspp-mode.el in the GNU
> ELPA repository.
>
> Until now I included the pspp-mode.el in the pspp debian package where it is then installed as an „external“ elpa package. The benefit is that once I install pspp, then the pspp-mode is autoloaded in emacs right away, i.e. when I open a .sps file, then the mode is immediately working. I think it is nice that I do not even need to know ELPA and the mode works in emacs right away when I install the debian pspp package. So maybe that is a reason to keep pspp-mode.el in the distribution. An alternative way would be  to find a way that installing the pspp package will „activate“ the pspp-mode.el from GNU ELPA.

There could easily be a separate package in Debian containing just
pspp-mode.el and tracking GNU ELPA as upstream.

Docs are here:
https://manpages.debian.org/buster/dh-make-elpa/dh-make-elpa.1.en.html

-- 
Sean Whitton

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Friedrich Beckmann]

Hi Sean,

thanks for your idea with the separate debian package for the pspp-mode. Do you think there is also a way to do this:

>>  An alternative way would be  to find a way that installing the pspp package will „activate“ the pspp-mode.el from GNU ELPA.

i.e. no upstream code tracking, no additional debian package - just a trigger that will activate this mode via GNU ELPA when I install the pspp application.

Regards

Friedrich

> Am 05.07.2020 um 18:51 schrieb Sean Whitton <spwhitton@spwhitton.name>:
> 
> There could easily be a separate package in Debian containing just
> pspp-mode.el and tracking GNU ELPA as upstream.
> 

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[John Darrington]

On Sun, Jul 05, 2020 at 09:51:12AM -0700, Sean Whitton wrote:
     Hello,
     
     There could easily be a separate package in Debian containing just
     pspp-mode.el and tracking GNU ELPA as upstream.
     
     Docs are here:
     https://manpages.debian.org/buster/dh-make-elpa/dh-make-elpa.1.en.html
     
Sure.  But from a user's perspective I don't see the advantage in that.
I mean what is the difference between typing "apt-get install pspp-mode"
and "M-x package-install 'pspp-mode" ?

J'

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[tomas]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

On Sun, Jul 05, 2020 at 08:35:02PM +0200, John Darrington wrote:
> On Sun, Jul 05, 2020 at 09:51:12AM -0700, Sean Whitton wrote:
>      Hello,
>      
>      There could easily be a separate package in Debian containing just
>      pspp-mode.el and tracking GNU ELPA as upstream.
>      
>      Docs are here:
>      https://manpages.debian.org/buster/dh-make-elpa/dh-make-elpa.1.en.html
>      
> Sure.  But from a user's perspective I don't see the advantage in that.
> I mean what is the difference between typing "apt-get install pspp-mode"
> and "M-x package-install 'pspp-mode" ?

If your system has more than one user (EEEK!), then there /is/ at
least one difference ;-)

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Sean Whitton]

Hello John,

On Sun 05 Jul 2020 at 08:35PM +02, John Darrington wrote:

> Sure.  But from a user's perspective I don't see the advantage in that.
> I mean what is the difference between typing "apt-get install pspp-mode"
> and "M-x package-install 'pspp-mode" ?

From the perspective of those of us working on packaging Emacs addons in
Debian, significant advantages are

1) the way addons work doesn't change within a Debian stable release,
   just like Emacs itself doesn't change within a Debian stable release

2) addons get installed using Debian's mechanisms for a secure software
   supply chain, rather than relying on https alone.

-- 
Sean Whitton

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Dmitry Alexandrov]

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

Sean Whitton <spwhitton@spwhitton.name> wrote:
> On Sun 05 Jul 2020 at 08:35PM +02, John Darrington wrote:
>> Sure.  But from a user's perspective I don't see the advantage in that.  I mean what is the difference between typing "apt-get install pspp-mode" and "M-x package-install 'pspp-mode" ?
>
> From the perspective of those of us working on packaging Emacs addons in Debian, significant advantages are
>
> 1) the way addons work doesn't change within a Debian stable release,
>    just like Emacs itself doesn't change within a Debian stable release
>
> 2) addons get installed using Debian's mechanisms for a secure software
>    supply chain, rather than relying on https alone.

3) they are automatically available to  #!/usr/bin/emacs --script  scripts; and pollute stderr at every run:

	$ ./hello.el
	Loading /etc/emacs/site-start.d/00debian.el (source)...
	Loading /etc/emacs/site-start.d/50autoconf.el (source)...
	Loading /etc/emacs/site-start.d/50cmake-data.el (source)...
	Loading /etc/emacs/site-start.d/50dictem.el (source)...
	Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...
	Loading debian-ispell...
	Loading /var/cache/dictionaries-common/emacsen-ispell-default.el (source)...
	Loading /var/cache/dictionaries-common/emacsen-ispell-dicts.el (source)...
	Loading /etc/emacs/site-start.d/50global.el (source)...
	Loading /etc/emacs/site-start.d/50mu4e.el (source)...
	Hello, Debian!

stimulating a user to learn nice tricks that allow to pass more than single argument to interpreter in order to get rid of them:

	#!/bin/sh
	":"; exec emacs --no-site-file --no-site-lisp --script "$0" -- "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Sean Whitton]

Hello Dmitry,

On Mon 06 Jul 2020 at 08:53AM +03, Dmitry Alexandrov wrote:

> Sean Whitton <spwhitton@spwhitton.name> wrote:
>> On Sun 05 Jul 2020 at 08:35PM +02, John Darrington wrote:
>>> Sure.  But from a user's perspective I don't see the advantage in that.  I mean what is the difference between typing "apt-get install pspp-mode" and "M-x package-install 'pspp-mode" ?
>>
>> From the perspective of those of us working on packaging Emacs addons in Debian, significant advantages are
>>
>> 1) the way addons work doesn't change within a Debian stable release,
>>    just like Emacs itself doesn't change within a Debian stable release
>>
>> 2) addons get installed using Debian's mechanisms for a secure software
>>    supply chain, rather than relying on https alone.
>
> 3) they are automatically available to  #!/usr/bin/emacs --script  scripts; and pollute stderr at every run:
>
> 	$ ./hello.el
> 	Loading /etc/emacs/site-start.d/00debian.el (source)...
> 	Loading /etc/emacs/site-start.d/50autoconf.el (source)...
> 	Loading /etc/emacs/site-start.d/50cmake-data.el (source)...
> 	Loading /etc/emacs/site-start.d/50dictem.el (source)...
> 	Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...
> 	Loading debian-ispell...
> 	Loading /var/cache/dictionaries-common/emacsen-ispell-default.el (source)...
> 	Loading /var/cache/dictionaries-common/emacsen-ispell-dicts.el (source)...
> 	Loading /etc/emacs/site-start.d/50global.el (source)...
> 	Loading /etc/emacs/site-start.d/50mu4e.el (source)...
> 	Hello, Debian!

This only happens with old style addons.  Once we have finished
migrating all addons in the archive to our newer tooling you won't see
this.

-- 
Sean Whitton

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Dmitry Alexandrov]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

Sean Whitton <spwhitton@spwhitton.name> wrote:
> On Mon 06 Jul 2020 at 08:53AM +03, Dmitry Alexandrov wrote:
>> Sean Whitton <spwhitton@spwhitton.name> wrote:
>>> On Sun 05 Jul 2020 at 08:35PM +02, John Darrington wrote:
>>>> Sure.  But from a user's perspective I don't see the advantage in that.  I mean what is the difference between typing "apt-get install pspp-mode" and "M-x package-install 'pspp-mode" ?
>>>
>>> From the perspective of those of us working on packaging Emacs addons in Debian, significant advantages are
>>>
>>> 1) the way addons work doesn't change within a Debian stable release, just like Emacs itself doesn't change within a Debian stable release
>>>
>>> 2) addons get installed using Debian's mechanisms for a secure software supply chain, rather than relying on https alone.
>>
>> 3) they are automatically available to  #!/usr/bin/emacs --script  scripts; and pollute stderr at every run:

> This only happens with old style addons.  Once we have finished migrating all addons in the archive to our newer tooling you won't see this.

The first part of the advantage will be still valid, though, right?  That is, they are prepended to ‘load-path’ even with -q / --batch / --script, while package.el-managed packages do not.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: [ELPA] New package: pspp-mode.el for PSPP/SPSS syntax highlighting

[Sean Whitton]

Hello,

On Mon 06 Jul 2020 at 09:38AM +03, Dmitry Alexandrov wrote:

> Sean Whitton <spwhitton@spwhitton.name> wrote:
>> On Mon 06 Jul 2020 at 08:53AM +03, Dmitry Alexandrov wrote:
>>> Sean Whitton <spwhitton@spwhitton.name> wrote:
>>>> On Sun 05 Jul 2020 at 08:35PM +02, John Darrington wrote:
>>>>> Sure.  But from a user's perspective I don't see the advantage in that.  I mean what is the difference between typing "apt-get install pspp-mode" and "M-x package-install 'pspp-mode" ?
>>>>
>>>> From the perspective of those of us working on packaging Emacs addons in Debian, significant advantages are
>>>>
>>>> 1) the way addons work doesn't change within a Debian stable release, just like Emacs itself doesn't change within a Debian stable release
>>>>
>>>> 2) addons get installed using Debian's mechanisms for a secure software supply chain, rather than relying on https alone.
>>>
>>> 3) they are automatically available to  #!/usr/bin/emacs --script  scripts; and pollute stderr at every run:
>
>> This only happens with old style addons.  Once we have finished migrating all addons in the archive to our newer tooling you won't see this.
>
> The first part of the advantage will be still valid, though, right?  That is, they are prepended to ‘load-path’ even with -q / --batch / --script, while package.el-managed packages do not.

Actually, they are added to package-directory-alist and then
package-initialize adds them to the load-path.

-- 
Sean Whitton