Re: [PATCH] add language/wisp to Guile?

[Christine Lemmer-Webber <cwebber@dustycloud.org>]

Haven't fully caught up on this thread, but as a side note I have a
mostly-finished implementation of a Wisp parser which takes a very
different approach than Arne's, and was more understandable to me
personally, a bit more functional and recursive-descent style.

I could make it available if anyone is curious.

Would love to see Wisp in Guile proper!


"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> [[PGP Signed Part:Undecided]]
> Hi,
>
> Since (language wisp)¹ has been rock stable for years now and is used in
> the Guix Workflow Language and supported in the Chickadee and the
> Tsukundere game engines, I thought it coud be a good time to merge Wisp
> into Guile itself.
>
> So I prepared a patch that adds language/wisp, some texinfo for
> SRFI-119, and some tests.
>
>
> Why add Wisp?
>
> For Wisp: it is then available directly wherever Guile is available.
>           This will make it much easier for people to follow tutorials.
>
> For Guile:
>
> - Wisp has proven to be good at enabling people to get an
>   entrance to Scheme² without pulling them out of the community.
>
> - It has also been shown to enable people who are used to other
>   programming languages to get a quick start at tools written in Guile.
>
> - And it provides access to the full capabilities of Guile with minimal
>   maintenance effort, because it is just the thinnest possible layer
>   around Scheme. The last required change was in 2020 while I used it
>   continuously.
>
>
> The attached patch provides just the wisp reader, but not the
> wisp->scheme transformer, because the latter has known broken edge-cases
> (and who needs the transformer can get it from the wisp repo and execute
> it directly with a Guile that then already supports wisp without any
> path adaptions).
>
>
> So I’d like to ask: can we merge Wisp as supported language into Guile?
>
>
> Best wishes,
> Arne
>
>
> ¹: https://www.draketo.de/software/wisp
>
> ²: »Wisp allows people to see code how Lispers perceive it. Its
>     structure becomes apparent.« — Ricardo Wurmus in IRC
>
> From 4d4759f9fc67b01c40bde41b93e3998f7d64eabd Mon Sep 17 00:00:00 2001
> From: Arne Babenhauserheide <arne_bab@web.de>
> Date: Fri, 3 Feb 2023 22:20:04 +0100
> Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation
>
> * doc/ref/srfi-modules.texi (srfi-119): add node
> * module/language/wisp.scm: New file.
> * module/language/wisp/spec.scm: New file.
> * test-suite/tests/srfi-119.test: New file.
> ---
>  doc/ref/srfi-modules.texi      |  30 ++
>  module/language/wisp.scm       | 796 +++++++++++++++++++++++++++++++++
>  module/language/wisp/spec.scm  | 107 +++++
>  test-suite/tests/srfi-119.test |  81 ++++
>  4 files changed, 1014 insertions(+)
>  create mode 100644 module/language/wisp.scm
>  create mode 100644 module/language/wisp/spec.scm
>  create mode 100644 test-suite/tests/srfi-119.test
>
> diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi
> index 0ef136215..759e293ec 100644
> --- a/doc/ref/srfi-modules.texi
> +++ b/doc/ref/srfi-modules.texi
> @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page
>  * SRFI-98::                     Accessing environment variables.
>  * SRFI-105::                    Curly-infix expressions.
>  * SRFI-111::                    Boxes.
> +* SRFI-119::                    Wisp: simpler indentation-sensitive scheme.
>  * SRFI-171::                    Transducers
>  @end menu
>  
> @@ -5662,6 +5663,34 @@ Return the current contents of @var{box}.
>  Set the contents of @var{box} to @var{value}.
>  @end deffn
>  
> +@node SRFI-119
> +@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme.
> +@cindex SRFI-119
> +@cindex wisp
> +
> +The languages shipped in Guile include SRFI-119 (wisp), an encoding of
> +Scheme that allows replacing parentheses with equivalent indentation and
> +inline colons. See
> +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification
> +of SRFI-119}.  Some examples:
> +
> +@example
> +display "Hello World!"         @result{}  (display "Hello World!")
> +@end example
> +
> +@example
> +define : factorial n           @result{}  (define (factorial n)
> +    if : zero? n               @result{}      (if (zero? n)
> +       . 1                     @result{}          1
> +       * n : factorial @{n - 1@} @result{}    (* n (factorial @{n - 1@}))))
> +@end example
> +
> +To execute a file with wisp code, select the language and filename
> +extension @code{.w} vie @code{guile --language=wisp -x .w}.
> +
> +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated.
> +
> +
>  @node SRFI-171
>  @subsection Transducers
>  @cindex SRFI-171
> @@ -5705,6 +5734,7 @@ left-to-right, due to how transducers are initiated.
>  * SRFI-171 Helpers::                  Utilities for writing your own transducers
>  @end menu
>  
> +
>  @node SRFI-171 General Discussion
>  @subsubsection SRFI-171 General Discussion
>  @cindex transducers discussion
> diff --git a/module/language/wisp.scm b/module/language/wisp.scm
> new file mode 100644
> index 000000000..ba24f54c5
> --- /dev/null
> +++ b/module/language/wisp.scm
> @@ -0,0 +1,796 @@
> +;;; Wisp
> +
> +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc.
> +;; Copyright (C) 2014--2023 Arne Babenhauserheide.
> +
> +;;;; This library is free software; you can redistribute it and/or
> +;;;; modify it under the terms of the GNU Lesser General Public
> +;;;; License as published by the Free Software Foundation; either
> +;;;; version 3 of the License, or (at your option) any later version.
> +;;;;
> +;;;; This library is distributed in the hope that it will be useful,
> +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +;;;; Lesser General Public License for more details.
> +;;;;
> +;;;; You should have received a copy of the GNU Lesser General Public
> +;;;; License along with this library; if not, write to the Free Software
> +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> +
> +;;; Commentary:
> +
> +;; Scheme-only implementation of a wisp-preprocessor which output a
> +;; scheme code tree to feed to a scheme interpreter instead of a
> +;; preprocessed file.
> +
> +;; Limitations:
> +;; - only unescapes up to 12 leading underscores at line start (\____________)
> +;; - in some cases the source line information is missing in backtraces.
> +;;   check for set-source-property!
> +
> +;;; Code:
> +
> +(define-module (language wisp)
> +   #:export (wisp-scheme-read-chunk wisp-scheme-read-all 
> +               wisp-scheme-read-file-chunk wisp-scheme-read-file
> +               wisp-scheme-read-string))
> +
> +; use curly-infix by default
> +(read-enable 'curly-infix)
> +
> +(use-modules
> +  (srfi srfi-1)
> +  (srfi srfi-11 ); for let-values
> +  (ice-9 rw ); for write-string/partial
> +  (ice-9 match))
> +
> +
> +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...)
> +(define (line-indent line)
> +         (car line))
> +
> +(define (line-real-indent line)
> +         "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)."
> +         (let (( indent (line-indent line)))
> +             (if (= -1 indent)
> +               0
> +               indent)))
> +
> +(define (line-code line)
> +         (let ((code (cdr line)))
> +             ; propagate source properties
> +             (when (not (null? code))
> +                    (set-source-properties! code (source-properties line)))
> +             code))
> +
> +; literal values I need
> +(define readcolon 
> +       (string->symbol ":"))
> +
> +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd")
> +; define an intermediate dot replacement with UUID to avoid clashes.
> +(define repr-dot ; .
> +       (string->symbol (string-append "REPR-DOT-" wisp-uuid)))
> +
> +; allow using reader additions as the first element on a line to prefix the list
> +(define repr-quote ; '
> +       (string->symbol (string-append "REPR-QUOTE-" wisp-uuid)))
> +(define repr-unquote ; ,
> +       (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid)))
> +(define repr-quasiquote ; `
> +       (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid)))
> +(define repr-unquote-splicing ; ,@
> +       (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid)))
> +
> +(define repr-syntax ; #'
> +       (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid)))
> +(define repr-unsyntax ; #,
> +       (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid)))
> +(define repr-quasisyntax ; #`
> +       (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid)))
> +(define repr-unsyntax-splicing ; #,@
> +       (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid)))
> +
> +; TODO: wrap the reader to return the repr of the syntax reader
> +; additions
> +
> +(define (match-charlist-to-repr charlist)
> +         (let
> +           ((chlist (reverse charlist)))
> +           (cond
> +             ((equal? chlist (list #\.))
> +               repr-dot)
> +             ((equal? chlist (list #\'))
> +               repr-quote)
> +             ((equal? chlist (list #\,))
> +               repr-unquote)
> +             ((equal? chlist (list #\`))
> +               repr-quasiquote)
> +             ((equal? chlist (list #\, #\@ ))
> +               repr-unquote-splicing)
> +             ((equal? chlist (list #\# #\' ))
> +               repr-syntax)
> +             ((equal? chlist (list #\# #\, ))
> +               repr-unsyntax)
> +             ((equal? chlist (list #\# #\` ))
> +               repr-quasisyntax)
> +             ((equal? chlist (list #\# #\, #\@ ))
> +               repr-unsyntax-splicing)
> +             (else
> +               #f))))
> +
> +(define (wisp-read port)
> +       "wrap read to catch list prefixes."
> +       (let ((prefix-maxlen 4))
> +         (let longpeek
> +           ((peeked '())
> +             (repr-symbol #f))
> +           (cond
> +             ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port)) )
> +               (if repr-symbol ; found a special symbol, return it.
> +                  ; TODO: Somehow store source-properties. The commented-out code below does not work.
> +                  ; catch #t
> +                  ;     lambda ()
> +                  ;         write : source-properties symbol-or-symbols
> +                  ;         set-source-property! symbol-or-symbols 'filename : port-filename port
> +                  ;         set-source-property! symbol-or-symbols 'line : 1+ : port-line port
> +                  ;         set-source-property! symbol-or-symbols 'column : port-column port
> +                  ;         write : source-properties symbol-or-symbols
> +                  ;     lambda : key . arguments
> +                  ;         . #f
> +                  repr-symbol
> +                  (let unpeek
> +                    ((remaining peeked))
> +                    (cond
> +                      ((equal? '() remaining )
> +                        (read port )); let read to the work
> +                      (else
> +                        (unread-char (car remaining) port)
> +                        (unpeek (cdr remaining)))))))
> +             (else
> +               (let*
> +                 ((next-char (read-char port))
> +                   (peeked (cons next-char peeked)))
> +                 (longpeek
> +                   peeked
> +                   (match-charlist-to-repr peeked))))))))
> +
> +
> +
> +(define (line-continues? line)
> +         (equal? repr-dot (car (line-code line))))
> +
> +(define (line-only-colon? line)
> +         (and
> +           (equal? ":" (car (line-code line)))
> +           (null? (cdr (line-code line)))))
> +
> +(define (line-empty-code? line)
> +         (null? (line-code line)))
> +
> +(define (line-empty? line)
> +         (and
> +           ; if indent is -1, we stripped a comment, so the line was not really empty.
> +           (= 0 (line-indent line))
> +           (line-empty-code? line)))
> +
> +(define (line-strip-continuation line   )
> +         (if (line-continues? line)
> +              (append
> +                (list
> +                  (line-indent line))
> +                (cdr (line-code line)))
> +              line))
> +
> +(define (line-strip-indentation-marker line)
> +         "Strip the indentation markers from the beginning of the line"
> +         (cdr line))
> +
> +(define (indent-level-reduction indentation-levels level select-fun)
> +         "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN"
> +         (let loop
> +           ((newlevels indentation-levels)
> +             (diff 0))
> +           (cond
> +             ((= level (car newlevels))
> +               (select-fun (list diff indentation-levels)))
> +             ((< level (car newlevels))
> +               (loop
> +                 (cdr newlevels)
> +                 (1+ diff)))
> +             (else
> +               (throw 'wisp-syntax-error "Level ~A not found in the indentation-levels ~A.")))))
> +
> +(define (indent-level-difference indentation-levels level)
> +         "Find how many indentation levels need to be popped off to find the given level."
> +         (indent-level-reduction indentation-levels level
> +           (lambda (x ); get the count
> +                    (car x))))
> +
> +(define (indent-reduce-to-level indentation-levels level)
> +         "Find how many indentation levels need to be popped off to find the given level."
> +         (indent-level-reduction indentation-levels level
> +           (lambda (x ); get the levels
> +                    (car (cdr x)))))
> +
> +(define (chunk-ends-with-period currentsymbols next-char)
> +       "Check whether indent-and-symbols ends with a period, indicating the end of a chunk."
> +       (and (not (null? currentsymbols))
> +             (equal? #\newline next-char)
> +             (equal? repr-dot
> +                    (list-ref currentsymbols (- (length currentsymbols) 1)))))
> +
> +(define (wisp-scheme-read-chunk-lines port)
> +         (let loop
> +           ((indent-and-symbols (list )); '((5 "(foobar)" "\"yobble\"")(3 "#t"))
> +             (inindent #t)
> +             (inunderscoreindent (equal? #\_ (peek-char port)))
> +             (incomment #f)
> +             (currentindent 0)
> +             (currentsymbols '())
> +             (emptylines 0))
> +           (cond
> +            ((>= emptylines 2 ); the chunk end has to be checked
> +                              ; before we look for new chars in the
> +                              ; port to make execution in the REPL
> +                              ; after two empty lines work
> +                              ; (otherwise it shows one more line).
> +             indent-and-symbols)
> +            (else
> +             (let ((next-char (peek-char port)))
> +               (cond
> +                 ((eof-object? next-char)
> +                   (append indent-and-symbols (list (append (list currentindent) currentsymbols))))
> +                 ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char))))
> +                  (append indent-and-symbols )); top-level form ends chunk
> +                 ((chunk-ends-with-period currentsymbols next-char)
> +                   ; the line ends with a period. This is forbidden in
> +                   ; SRFI-119. Use it to end the line in the REPL without
> +                   ; showing continuation dots (...).
> +                   (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1)))))
> +                 ((and inindent (equal? #\space next-char))
> +                   (read-char port ); remove char
> +                   (loop
> +                     indent-and-symbols
> +                     #t ; inindent
> +                     #f ; inunderscoreindent
> +                     #f ; incomment
> +                     (1+ currentindent)
> +                     currentsymbols
> +                     emptylines))
> +                 ((and inunderscoreindent (equal? #\_ next-char))
> +                   (read-char port ); remove char
> +                   (loop
> +                     indent-and-symbols
> +                     #t ; inindent
> +                     #t ; inunderscoreindent
> +                     #f ; incomment
> +                     (1+ currentindent)
> +                     currentsymbols
> +                     emptylines))
> +                 ; any char but whitespace *after* underscoreindent is
> +                 ; an error. This is stricter than the current wisp
> +                 ; syntax definition. TODO: Fix the definition. Better
> +                 ; start too strict. FIXME: breaks on lines with only
> +                 ; underscores which should be empty lines.
> +                 ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char))))
> +                   (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols)))
> +                 ((equal? #\newline next-char)
> +                   (read-char port ); remove the newline
> +                   ; The following two lines would break the REPL by requiring one char too many.
> +                   ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port
> +                   ;      read-char port ; remove a full \n\r. Damn special cases...
> +                   (let* ; distinguish pure whitespace lines and lines
> +                        ; with comment by giving the former zero
> +                        ; indent. Lines with a comment at zero indent
> +                        ; get indent -1 for the same reason - meaning
> +                        ; not actually empty.
> +                     (
> +                       (indent
> +                         (cond
> +                           (incomment
> +                             (if (= 0 currentindent ); specialcase
> +                               -1
> +                               currentindent ))
> +                           ((not (null? currentsymbols )); pure whitespace
> +                             currentindent)
> +                           (else
> +                             0)))
> +                       (parsedline (append (list indent) currentsymbols))
> +                       (emptylines
> +                         (if (not (line-empty? parsedline))
> +                            0 
> +                            (1+ emptylines))))
> +                     (when (not (= 0 (length parsedline)))
> +                         ; set the source properties to parsedline so we can try to add them later.
> +                         (set-source-property! parsedline 'filename (port-filename port))
> +                         (set-source-property! parsedline 'line (port-line port)))
> +                     ; TODO: If the line is empty. Either do it here and do not add it, just
> +                     ; increment the empty line counter, or strip it later. Replace indent
> +                     ; -1 by indent 0 afterwards.
> +                     (loop
> +                       (append indent-and-symbols (list parsedline))
> +                       #t ; inindent
> +                       (if (<= 2 emptylines)
> +                         #f ; chunk ends here
> +                         (equal? #\_ (peek-char port ))); are we in underscore indent?
> +                       #f ; incomment
> +                       0
> +                       '()
> +                       emptylines)))
> +                 ((equal? #t incomment)
> +                   (read-char port ); remove one comment character
> +                   (loop
> +                     indent-and-symbols
> +                     #f ; inindent 
> +                     #f ; inunderscoreindent 
> +                     #t ; incomment
> +                     currentindent
> +                     currentsymbols
> +                     emptylines))
> +                 ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char) ); remove whitespace when not in indent
> +                   (read-char port ); remove char
> +                   (loop
> +                     indent-and-symbols
> +                     #f ; inindent
> +                     #f ; inunderscoreindent
> +                     #f ; incomment
> +                     currentindent
> +                     currentsymbols
> +                     emptylines))
> +                          ; | cludge to appease the former wisp parser
> +                          ; | used for bootstrapping which has a
> +                          ; v problem with the literal comment char
> +                 ((equal? (string-ref ";" 0) next-char)
> +                   (loop
> +                     indent-and-symbols
> +                     #f ; inindent
> +                     #f ; inunderscoreindent
> +                     #t ; incomment
> +                     currentindent
> +                     currentsymbols
> +                     emptylines))
> +                 (else ; use the reader
> +                   (loop
> +                     indent-and-symbols
> +                     #f ; inindent
> +                     #f ; inunderscoreindent
> +                     #f ; incomment
> +                     currentindent
> +                     ; this also takes care of the hashbang and leading comments.
> +                     (append currentsymbols (list (wisp-read port)))
> +                     emptylines))))))))
> +
> +
> +(define (line-code-replace-inline-colons line)
> +         "Replace inline colons by opening parens which close at the end of the line"
> +         ; format #t "replace inline colons for line ~A\n" line
> +         (let loop
> +           ((processed '())
> +              (unprocessed line))
> +           (cond
> +             ((null? unprocessed)
> +               ; format #t "inline-colons processed line: ~A\n" processed
> +               processed)
> +             ; replace : . with nothing
> +             ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed))))
> +               (loop
> +                 (append processed
> +                   (loop '() (cdr (cdr unprocessed))))
> +                 '()))
> +             ((equal? readcolon (car unprocessed))
> +               (loop
> +                 ; FIXME: This should turn unprocessed into a list. 
> +                 (append processed
> +                   (list (loop '() (cdr unprocessed))))
> +                 '()))
> +             (else
> +               (loop
> +                 (append processed
> +                   (list (car unprocessed)))
> +                 (cdr unprocessed))))))
> +
> +(define (line-replace-inline-colons line)
> +         (cons
> +           (line-indent line)
> +           (line-code-replace-inline-colons (line-code line))))
> +
> +(define (line-strip-lone-colon line)
> +         "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons."
> +         (if
> +           (equal?
> +             (line-code line)
> +             (list readcolon))
> +           (list (line-indent line))
> +           line))
> +
> +(define (line-finalize line)
> +       "Process all wisp-specific information in a line and strip it"
> +       (let
> +         (
> +           (l
> +             (line-code-replace-inline-colons
> +               (line-strip-indentation-marker
> +                 (line-strip-lone-colon
> +                   (line-strip-continuation line))))))
> +         (when (not (null? (source-properties line)))
> +                (catch #t
> +                  (lambda ()
> +                    (set-source-properties! l (source-properties line)))
> +                  (lambda (key . arguments)
> +                    #f)))
> +         l))
> +
> +(define (wisp-add-source-properties-from source target)
> +       "Copy the source properties from source into the target and return the target."
> +       (catch #t
> +           (lambda ()
> +               (set-source-properties! target (source-properties source)))
> +           (lambda (key . arguments)
> +               #f))
> +       target)
> +
> +(define (wisp-propagate-source-properties code)
> +       "Propagate the source properties from the sourrounding list into every part of the code."
> +       (let loop
> +         ((processed '())
> +           (unprocessed code))
> +         (cond
> +           ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed)))
> +             unprocessed)
> +           ((and (pair? unprocessed) (not (list? unprocessed)))
> +             (cons
> +               (wisp-propagate-source-properties (car unprocessed))
> +               (wisp-propagate-source-properties (cdr unprocessed))))
> +           ((null? unprocessed)
> +             processed)
> +           (else
> +             (let ((line (car unprocessed)))
> +               (if (null? (source-properties unprocessed))
> +                   (wisp-add-source-properties-from line unprocessed)
> +                   (wisp-add-source-properties-from unprocessed line))
> +               (loop
> +                 (append processed (list (wisp-propagate-source-properties line)))
> +                 (cdr unprocessed)))))))
> +
> +(define* (wisp-scheme-indentation-to-parens lines)
> +         "Add parentheses to lines and remove the indentation markers"
> +         (when
> +           (and
> +             (not (null? lines))
> +             (not (line-empty-code? (car lines)))
> +             (not (= 0 (line-real-indent (car lines ))))); -1 is a line with a comment
> +           (if (= 1 (line-real-indent (car lines)))
> +             ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands
> +             (set! lines
> +                  (cons
> +                      (cons 0 (cdr (car lines)))
> +                      (cdr lines)))
> +             (throw 'wisp-syntax-error
> +               (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A"
> +                 (car lines)))))
> +         (let loop
> +           ((processed '())
> +             (unprocessed lines)
> +             (indentation-levels '(0)))
> +           (let*
> +             (
> +               (current-line
> +                 (if (<= 1 (length unprocessed))
> +                      (car unprocessed)
> +                      (list 0 ))); empty code
> +               (next-line
> +                 (if (<= 2 (length unprocessed))
> +                      (car (cdr unprocessed))
> +                      (list 0 ))); empty code
> +               (current-indentation
> +                      (car indentation-levels))
> +               (current-line-indentation (line-real-indent current-line)))
> +             ; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n"
> +             ;     . processed current-line next-line unprocessed indentation-levels current-indentation
> +             (cond
> +                 ; the real end: this is reported to the outside world.
> +               ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels)))
> +                 ; display "done\n"
> +                 ; reverse the processed lines, because I use cons.
> +                 processed)
> +               ; the recursion end-condition
> +               ((and (null? unprocessed))
> +                 ; display "last step\n"
> +                 ; this is the last step. Nothing more to do except
> +                 ; for rolling up the indentation levels.  return the
> +                 ; new processed and unprocessed lists: this is a
> +                 ; side-recursion
> +                 (values processed unprocessed))
> +               ((null? indentation-levels)
> +                 ; display "indentation-levels null\n"
> +                 (throw 'wisp-programming-error "The indentation-levels are null but the current-line is null: Something killed the indentation-levels."))
> +               (else ; now we come to the line-comparisons and indentation-counting.
> +                   (cond
> +                     ((line-empty-code? current-line)
> +                       ; display "current-line empty\n"
> +                       ; We cannot process indentation without
> +                       ; code. Just switch to the next line. This should
> +                       ; only happen at the start of the recursion.
> +                       ; TODO: Somehow preserve the line-numbers.
> +                       (loop
> +                         processed
> +                         (cdr unprocessed)
> +                         indentation-levels))
> +                     ((and (line-empty-code? next-line) (<= 2 (length unprocessed )))
> +                       ; display "next-line empty\n"
> +                       ; TODO: Somehow preserve the line-numbers.
> +                       ; take out the next-line from unprocessed.
> +                       (loop
> +                         processed
> +                         (cons current-line
> +                           (cdr (cdr unprocessed)))
> +                         indentation-levels))
> +                     ((> current-indentation current-line-indentation)
> +                       ; display "current-indent > next-line\n"
> +                       ; this just steps back one level via the side-recursion.
> +                       (let ((previous-indentation (car (cdr indentation-levels))))
> +                         (if (<= current-line-indentation previous-indentation)
> +                            (values processed unprocessed)
> +                            (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning.
> +                                (let ((linenumber (- (length lines) (length unprocessed))))
> +                                    (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line)))
> +                                (loop
> +                                  processed
> +                                  unprocessed
> +                                  (cons ; recursion via the indentation-levels
> +                                    current-line-indentation
> +                                    (cdr indentation-levels)))))))
> +                     ((= current-indentation current-line-indentation)
> +                       ; display "current-indent = next-line\n"
> +                       (let
> +                         ((line (line-finalize current-line))
> +                           (next-line-indentation (line-real-indent next-line)))
> +                         (cond
> +                           ((>= current-line-indentation next-line-indentation)
> +                             ; simple recursiive step to the next line
> +                             ; display "current-line-indent >= next-line-indent\n"
> +                             (loop
> +                               (append processed
> +                                 (if (line-continues? current-line)
> +                                      line
> +                                      (wisp-add-source-properties-from line (list line))))
> +                               (cdr unprocessed ); recursion here
> +                               indentation-levels))
> +                           ((< current-line-indentation next-line-indentation)
> +                             ; display "current-line-indent < next-line-indent\n"
> +                             ; format #t "line: ~A\n" line
> +                             ; side-recursion via a sublist
> +                             (let-values
> +                               (
> +                                 ((sub-processed sub-unprocessed)
> +                                   (loop
> +                                     line
> +                                     (cdr unprocessed ); recursion here
> +                                     indentation-levels)))
> +                               ; format #t "side-recursion:\n  sub-processed: ~A\n  processed: ~A\n\n" sub-processed processed
> +                               (loop
> +                                 (append processed (list sub-processed))
> +                                 sub-unprocessed ; simply use the recursion from the sub-recursion
> +                                 indentation-levels))))))
> +                     ((< current-indentation current-line-indentation)
> +                       ; display "current-indent < next-line\n"
> +                       (loop
> +                         processed
> +                         unprocessed
> +                         (cons ; recursion via the indentation-levels
> +                           current-line-indentation
> +                           indentation-levels)))
> +                     (else
> +                       (throw 'wisp-not-implemented
> +                             (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A."
> +                               current-line next-line processed)))))))))
> +
> +
> +(define (wisp-scheme-replace-inline-colons lines)
> +         "Replace inline colons by opening parens which close at the end of the line"
> +         (let loop
> +           ((processed '())
> +             (unprocessed lines))
> +           (if (null? unprocessed)
> +                processed
> +                (loop
> +                  (append processed (list (line-replace-inline-colons (car unprocessed))))
> +                  (cdr unprocessed)))))
> +
> +
> +(define (wisp-scheme-strip-indentation-markers lines)
> +         "Strip the indentation markers from the beginning of the lines"
> +         (let loop
> +           ((processed '())
> +             (unprocessed lines))
> +           (if (null? unprocessed)
> +                processed
> +                (loop
> +                  (append processed (cdr (car unprocessed)))
> +                  (cdr unprocessed)))))
> +
> +(define (wisp-unescape-underscore-and-colon code)
> +         "replace \\_ and \\: by _ and :"
> +         (match code
> +             ((a ...)
> +               (map wisp-unescape-underscore-and-colon a))
> +             ('\_
> +               '_)
> +             ('\__
> +               '__)
> +             ('\___
> +               '___)
> +             ('\____
> +               '____)
> +             ('\_____
> +               '_____)
> +             ('\______
> +               '______)
> +             ('\_______
> +               '_______)
> +             ('\________
> +               '________)
> +             ('\_________
> +               '_________)
> +             ('\__________
> +               '__________)
> +             ('\___________
> +               '___________)
> +             ('\____________
> +               '____________)
> +             ('\:
> +               ':)
> +             (a
> +               a)))
> +
> +
> +(define (wisp-replace-empty-eof code)
> +         "replace ((#<eof>)) by ()"
> +         ; FIXME: Actually this is a hack which fixes a bug when the
> +         ; parser hits files with only hashbang and comments.
> +         (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code))))
> +              (list)
> +              code))
> +
> +
> +(define (wisp-replace-paren-quotation-repr code)
> +         "Replace lists starting with a quotation symbol by
> +         quoted lists."
> +         (match code
> +             (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'quote (map wisp-replace-paren-quotation-repr a)))
> +             ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list 
> +                (append
> +                        (map wisp-replace-paren-quotation-repr a)
> +                        (list (list 'quote (map wisp-replace-paren-quotation-repr b)))))
> +             (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a))))
> +             (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'unquote (map wisp-replace-paren-quotation-repr a)))
> +             ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b )
> +                (append
> +                        (map wisp-replace-paren-quotation-repr a)
> +                        (list (list 'unquote (map wisp-replace-paren-quotation-repr b)))))
> +             (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'quasiquote (map wisp-replace-paren-quotation-repr a)))
> +             ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list 
> +                (append
> +                        (map wisp-replace-paren-quotation-repr a)
> +                        (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b)))))
> +             (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a)))
> +             (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'syntax (map wisp-replace-paren-quotation-repr a)))
> +             (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'unsyntax (map wisp-replace-paren-quotation-repr a)))
> +             (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'quasisyntax (map wisp-replace-paren-quotation-repr a)))
> +             (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
> +                (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a)))
> +             ;; literal array as start of a line: # (a b) c -> (#(a b) c)
> +             ((#\# a ...)
> +               (with-input-from-string ;; hack to defer to read
> +                   (string-append "#"
> +                       (with-output-to-string
> +                           (λ ()
> +                             (write (map wisp-replace-paren-quotation-repr a)
> +                                     (current-output-port)))))
> +                   read))
> +             ((a ...)
> +               (map wisp-replace-paren-quotation-repr a))
> +             (a
> +               a)))
> +
> +(define (wisp-make-improper code)
> +         "Turn (a #{.}# b) into the correct (a . b).
> +
> +read called on a single dot creates a variable named #{.}# (|.|
> +in r7rs). Due to parsing the indentation before the list
> +structure is known, the reader cannot create improper lists
> +when it reads a dot. So we have to take another pass over the
> +code to recreate the improper lists.
> +
> +Match is awesome!"
> +         (let
> +           (
> +             (improper
> +               (match code
> +                  ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c)
> +                    (append (map wisp-make-improper a) 
> +                      (cons (wisp-make-improper b) (wisp-make-improper c))))
> +                  ((a ...)
> +                    (map wisp-make-improper a))
> +                  (a
> +                    a))))
> +           (define (syntax-error li msg)
> +                   (throw 'wisp-syntax-error (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li)))
> +           (if #t
> +            improper
> +            (let check
> +             ((tocheck improper))
> +             (match tocheck
> +               ; lists with only one member
> +               (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd)
> +                 (syntax-error tocheck "list with the period as only member"))
> +               ; list with remaining dot.
> +               ((a ...)
> +                 (if (and (member repr-dot a))
> +                      (syntax-error tocheck "leftover period in list")
> +                      (map check a)))
> +               ; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why?
> +               (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c)
> +                 (syntax-error tocheck "dot as first element in already improper pair"))
> +               ; simple pair, other way round
> +               ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd)
> +                 (syntax-error tocheck "dot as last element in already improper pair"))
> +               ; more complex pairs
> +               ((? pair? a)
> +                 (let 
> +                   ((head (drop-right a 1))
> +                     (tail (last-pair a)))
> +                   (cond
> +                    ((equal? repr-dot (car tail))
> +                      (syntax-error tocheck "equal? repr-dot : car tail"))
> +                    ((equal? repr-dot (cdr tail))
> +                      (syntax-error tocheck "equal? repr-dot : cdr tail"))
> +                    ((member repr-dot head)
> +                      (syntax-error tocheck "member repr-dot head"))
> +                    (else
> +                      a))))
> +               (a
> +                 a))))))
> +
> +(define (wisp-scheme-read-chunk port)
> +         "Read and parse one chunk of wisp-code"
> +         (let (( lines (wisp-scheme-read-chunk-lines port)))
> +              (wisp-make-improper
> +                (wisp-replace-empty-eof
> +                  (wisp-unescape-underscore-and-colon
> +                    (wisp-replace-paren-quotation-repr
> +                      (wisp-propagate-source-properties
> +                        (wisp-scheme-indentation-to-parens lines))))))))
> +
> +(define (wisp-scheme-read-all port)
> +         "Read all chunks from the given port"
> +         (let loop
> +           ((tokens '()))
> +           (cond
> +             ((eof-object? (peek-char port))
> +               tokens)
> +             (else
> +               (loop
> +                 (append tokens (wisp-scheme-read-chunk port)))))))
> +
> +(define (wisp-scheme-read-file path)
> +         (call-with-input-file path wisp-scheme-read-all))
> +
> +(define (wisp-scheme-read-file-chunk path)
> +         (call-with-input-file path wisp-scheme-read-chunk))
> +
> +(define (wisp-scheme-read-string str)
> +         (call-with-input-string str wisp-scheme-read-all))
> +
> +(define (wisp-scheme-read-string-chunk str)
> +         (call-with-input-string str wisp-scheme-read-chunk))
> +
> diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm
> new file mode 100644
> index 000000000..d5ea7abce
> --- /dev/null
> +++ b/module/language/wisp/spec.scm
> @@ -0,0 +1,107 @@
> +;; Language interface for Wisp in Guile
> +
> +;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm
> +
> +;;; Copyright (C) 2005-2014 by David A. Wheeler and Alan Manuel K. Gloria
> +;;; Copyright (C) Arne Babenhauserheide (2014--2023).
> +
> +;;; Permission is hereby granted, free of charge, to any person
> +;;; obtaining a copy of this software and associated documentation
> +;;; files (the "Software"), to deal in the Software without
> +;;; restriction, including without limitation the rights to use, copy,
> +;;; modify, merge, publish, distribute, sublicense, and/or sell copies
> +;;; of the Software, and to permit persons to whom the Software is
> +;;; furnished to do so, subject to the following conditions:
> +;;;
> +;;; The above copyright notice and this permission notice shall be
> +;;; included in all copies or substantial portions of the Software.
> +;;;
> +;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> +;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> +;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> +;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> +;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> +;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> +;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> +;;; SOFTWARE.
> +
> +; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm
> +(define-module (language wisp spec)
> +  #:use-module (language wisp)
> +  #:use-module (system base compile)
> +  #:use-module (system base language)
> +  #:use-module (language scheme compile-tree-il)
> +  #:use-module (language scheme decompile-tree-il)
> +  #:export (wisp))
> +
> +; Set locale to something which supports unicode. Required to avoid using fluids.
> +(catch #t
> +      (lambda ()
> +        (setlocale LC_ALL ""))
> +      (lambda (key . parameters)
> +        (let ((locale-fallback "en_US.UTF-8"))
> +          (format (current-error-port)
> +              (string-join
> +                  (list ";;; Warning: setlocale LC_ALL \"\" failed with ~A: ~A"
> +                     "switching to explicit ~A locale. Please setup your locale."
> +                     "If this fails, you might need glibc support for unicode locales.\n")
> +                   "\n;;;          ")
> +              key parameters locale-fallback)
> +          (catch #t
> +            (lambda ()
> +              (setlocale LC_ALL locale-fallback))
> +            (lambda (key . parameters)
> +              (format (current-error-port)
> +                  (string-join
> +                      (list ";;; Warning: fallback setlocale LC_ALL ~A failed with ~A: ~A"
> +                         "Not switching to Unicode."
> +                         "You might need glibc support for unicode locales.\n")
> +                       "\n;;;          ")
> +                  locale-fallback key parameters))))))
> +
> +;;;
> +;;; Language definition
> +;;;
> +
> +(define wisp-pending-sexps (list))
> +
> +(define (read-one-wisp-sexp port env)
> +         ;; allow using "# foo" as #(foo).
> +         (read-hash-extend #\# (λ (chr port) #\#))
> +         (cond
> +            ((eof-object? (peek-char port))
> +              (read-char port )); return eof: we’re done
> +            (else
> +              (let ((chunk (wisp-scheme-read-chunk port)))
> +                (cond
> +                  ((not (null? chunk))
> +                    (car chunk))
> +                  (else
> +                    #f))))))
> +
> +(define-language wisp
> +  #:title "Wisp Scheme Syntax. See SRFI-119 for details."
> +  ; . #:reader read-one-wisp-sexp
> +  #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ;
> +  #:compilers `((tree-il . ,compile-tree-il))
> +  #:decompilers `((tree-il . ,decompile-tree-il))
> +  #:evaluator (lambda (x module) (primitive-eval x))
> +  #:printer write ; TODO: backtransform to wisp? Use source-properties?
> +  #:make-default-environment
> +  (lambda ()
> +    ;; Ideally we'd duplicate the whole module hierarchy so that `set!',
> +    ;; `fluid-set!', etc. don't have any effect in the current environment.
> +    (let ((m (make-fresh-user-module)))
> +      ;; Provide a separate `current-reader' fluid so that
> +      ;; compile-time changes to `current-reader' are
> +      ;; limited to the current compilation unit.
> +      (module-define! m 'current-reader (make-fluid))
> +      ;; Default to `simple-format', as is the case until
> +      ;; (ice-9 format) is loaded. This allows
> +      ;; compile-time warnings to be emitted when using
> +      ;; unsupported options.
> +      (module-set! m 'format simple-format)
> +      m)))
> +
> +
> +
> diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test
> new file mode 100644
> index 000000000..a888df41d
> --- /dev/null
> +++ b/test-suite/tests/srfi-119.test
> @@ -0,0 +1,81 @@
> +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*-
> +;;;;
> +;;;; Copyright (C) 2023 Free Software Foundation, Inc.
> +;;;;
> +;;;; This library is free software; you can redistribute it and/or
> +;;;; modify it under the terms of the GNU Lesser General Public
> +;;;; License as published by the Free Software Foundation; either
> +;;;; version 3 of the License, or (at your option) any later version.
> +;;;;
> +;;;; This library is distributed in the hope that it will be useful,
> +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +;;;; Lesser General Public License for more details.
> +;;;;
> +;;;; You should have received a copy of the GNU Lesser General Public
> +;;;; License along with this library; if not, write to the Free Software
> +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> +
> +(define-module (test-srfi-119)
> +  #:use-module (test-suite lib)
> +  #:use-module (srfi srfi-1)
> +  #:use-module (language wisp))
> +
> +(define (read-string s)
> +  (with-input-from-string s read))
> +
> +(define (with-read-options opts thunk)
> +  (let ((saved-options (read-options)))
> +    (dynamic-wind
> +        (lambda ()
> +          (read-options opts))
> +        thunk
> +        (lambda ()
> +          (read-options saved-options)))))
> +
> +(define (wisp->list str)
> +        (wisp-scheme-read-string str))
> +
> +(with-test-prefix "wisp-read-simple"
> +  (pass-if (equal? (wisp->list "<= n 5")    '((<= n 5))))
> +  (pass-if (equal? (wisp->list ". 5") '(5)))
> +  (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3))))))
> +(with-test-prefix "wisp-read-complex"
> +  (pass-if (equal? (wisp->list "
> +a b c d e
> +  . f g h
> +  . i j k
> +
> +concat \"I want \"
> +    getwish from me
> +    . \" - \" username
> +") '(
> +(a b c d e
> +  f g h
> +  i j k)
> +
> +(concat "I want "
> +    (getwish from me)
> +    " - " username))))
> +
> +  (pass-if (equal? (wisp->list "
> +define : a b c
> +_ d e
> +___ f
> +___ g h
> +__  . i
> +
> +define : _
> +_  display \"hello\n\"
> +
> +\\_") '(
> +(define (a b c)
> +  (d e
> +    (f)
> +    (g h)
> +    i))
> +
> +(define (_)
> +   (display "hello\n"))
> +
> +(_)))))
> -- 
> 2.39.1