* [PATCH] add language/wisp to Guile? @ 2023-02-03 21:26 Dr. Arne Babenhauserheide 2023-02-04 15:08 ` Maxime Devos 2023-09-30 13:17 ` Christine Lemmer-Webber 0 siblings, 2 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-03 21:26 UTC (permalink / raw) To: guile-devel [-- Attachment #1.1: Type: text/plain, Size: 1597 bytes --] Hi, Since (language wisp)¹ has been rock stable for years now and is used in the Guix Workflow Language and supported in the Chickadee and the Tsukundere game engines, I thought it coud be a good time to merge Wisp into Guile itself. So I prepared a patch that adds language/wisp, some texinfo for SRFI-119, and some tests. Why add Wisp? For Wisp: it is then available directly wherever Guile is available. This will make it much easier for people to follow tutorials. For Guile: - Wisp has proven to be good at enabling people to get an entrance to Scheme² without pulling them out of the community. - It has also been shown to enable people who are used to other programming languages to get a quick start at tools written in Guile. - And it provides access to the full capabilities of Guile with minimal maintenance effort, because it is just the thinnest possible layer around Scheme. The last required change was in 2020 while I used it continuously. The attached patch provides just the wisp reader, but not the wisp->scheme transformer, because the latter has known broken edge-cases (and who needs the transformer can get it from the wisp repo and execute it directly with a Guile that then already supports wisp without any path adaptions). So I’d like to ask: can we merge Wisp as supported language into Guile? Best wishes, Arne ¹: https://www.draketo.de/software/wisp ²: »Wisp allows people to see code how Lispers perceive it. Its structure becomes apparent.« — Ricardo Wurmus in IRC [-- Attachment #1.2: 0001-Add-language-wisp-wisp-tests-and-srfi-119-documentat.patch --] [-- Type: text/x-patch, Size: 48201 bytes --] From 4d4759f9fc67b01c40bde41b93e3998f7d64eabd Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 3 Feb 2023 22:20:04 +0100 Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation * doc/ref/srfi-modules.texi (srfi-119): add node * module/language/wisp.scm: New file. * module/language/wisp/spec.scm: New file. * test-suite/tests/srfi-119.test: New file. --- doc/ref/srfi-modules.texi | 30 ++ module/language/wisp.scm | 796 +++++++++++++++++++++++++++++++++ module/language/wisp/spec.scm | 107 +++++ test-suite/tests/srfi-119.test | 81 ++++ 4 files changed, 1014 insertions(+) create mode 100644 module/language/wisp.scm create mode 100644 module/language/wisp/spec.scm create mode 100644 test-suite/tests/srfi-119.test diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 0ef136215..759e293ec 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. +* SRFI-119:: Wisp: simpler indentation-sensitive scheme. * SRFI-171:: Transducers @end menu @@ -5662,6 +5663,34 @@ Return the current contents of @var{box}. Set the contents of @var{box} to @var{value}. @end deffn +@node SRFI-119 +@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme. +@cindex SRFI-119 +@cindex wisp + +The languages shipped in Guile include SRFI-119 (wisp), an encoding of +Scheme that allows replacing parentheses with equivalent indentation and +inline colons. See +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification +of SRFI-119}. Some examples: + +@example +display "Hello World!" @result{} (display "Hello World!") +@end example + +@example +define : factorial n @result{} (define (factorial n) + if : zero? n @result{} (if (zero? n) + . 1 @result{} 1 + * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) +@end example + +To execute a file with wisp code, select the language and filename +extension @code{.w} vie @code{guile --language=wisp -x .w}. + +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. + + @node SRFI-171 @subsection Transducers @cindex SRFI-171 @@ -5705,6 +5734,7 @@ left-to-right, due to how transducers are initiated. * SRFI-171 Helpers:: Utilities for writing your own transducers @end menu + @node SRFI-171 General Discussion @subsubsection SRFI-171 General Discussion @cindex transducers discussion diff --git a/module/language/wisp.scm b/module/language/wisp.scm new file mode 100644 index 000000000..ba24f54c5 --- /dev/null +++ b/module/language/wisp.scm @@ -0,0 +1,796 @@ +;;; Wisp + +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. +;; Copyright (C) 2014--2023 Arne Babenhauserheide. + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;;; Commentary: + +;; Scheme-only implementation of a wisp-preprocessor which output a +;; scheme code tree to feed to a scheme interpreter instead of a +;; preprocessed file. + +;; Limitations: +;; - only unescapes up to 12 leading underscores at line start (\____________) +;; - in some cases the source line information is missing in backtraces. +;; check for set-source-property! + +;;; Code: + +(define-module (language wisp) + #:export (wisp-scheme-read-chunk wisp-scheme-read-all + wisp-scheme-read-file-chunk wisp-scheme-read-file + wisp-scheme-read-string)) + +; use curly-infix by default +(read-enable 'curly-infix) + +(use-modules + (srfi srfi-1) + (srfi srfi-11 ); for let-values + (ice-9 rw ); for write-string/partial + (ice-9 match)) + + +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define (line-indent line) + (car line)) + +(define (line-real-indent line) + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." + (let (( indent (line-indent line))) + (if (= -1 indent) + 0 + indent))) + +(define (line-code line) + (let ((code (cdr line))) + ; propagate source properties + (when (not (null? code)) + (set-source-properties! code (source-properties line))) + code)) + +; literal values I need +(define readcolon + (string->symbol ":")) + +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") +; define an intermediate dot replacement with UUID to avoid clashes. +(define repr-dot ; . + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) + +; allow using reader additions as the first element on a line to prefix the list +(define repr-quote ; ' + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) +(define repr-unquote ; , + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) +(define repr-quasiquote ; ` + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) +(define repr-unquote-splicing ; ,@ + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) + +(define repr-syntax ; #' + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) +(define repr-unsyntax ; #, + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) +(define repr-quasisyntax ; #` + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) +(define repr-unsyntax-splicing ; #,@ + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) + +; TODO: wrap the reader to return the repr of the syntax reader +; additions + +(define (match-charlist-to-repr charlist) + (let + ((chlist (reverse charlist))) + (cond + ((equal? chlist (list #\.)) + repr-dot) + ((equal? chlist (list #\')) + repr-quote) + ((equal? chlist (list #\,)) + repr-unquote) + ((equal? chlist (list #\`)) + repr-quasiquote) + ((equal? chlist (list #\, #\@ )) + repr-unquote-splicing) + ((equal? chlist (list #\# #\' )) + repr-syntax) + ((equal? chlist (list #\# #\, )) + repr-unsyntax) + ((equal? chlist (list #\# #\` )) + repr-quasisyntax) + ((equal? chlist (list #\# #\, #\@ )) + repr-unsyntax-splicing) + (else + #f)))) + +(define (wisp-read port) + "wrap read to catch list prefixes." + (let ((prefix-maxlen 4)) + (let longpeek + ((peeked '()) + (repr-symbol #f)) + (cond + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port)) ) + (if repr-symbol ; found a special symbol, return it. + ; TODO: Somehow store source-properties. The commented-out code below does not work. + ; catch #t + ; lambda () + ; write : source-properties symbol-or-symbols + ; set-source-property! symbol-or-symbols 'filename : port-filename port + ; set-source-property! symbol-or-symbols 'line : 1+ : port-line port + ; set-source-property! symbol-or-symbols 'column : port-column port + ; write : source-properties symbol-or-symbols + ; lambda : key . arguments + ; . #f + repr-symbol + (let unpeek + ((remaining peeked)) + (cond + ((equal? '() remaining ) + (read port )); let read to the work + (else + (unread-char (car remaining) port) + (unpeek (cdr remaining))))))) + (else + (let* + ((next-char (read-char port)) + (peeked (cons next-char peeked))) + (longpeek + peeked + (match-charlist-to-repr peeked)))))))) + + + +(define (line-continues? line) + (equal? repr-dot (car (line-code line)))) + +(define (line-only-colon? line) + (and + (equal? ":" (car (line-code line))) + (null? (cdr (line-code line))))) + +(define (line-empty-code? line) + (null? (line-code line))) + +(define (line-empty? line) + (and + ; if indent is -1, we stripped a comment, so the line was not really empty. + (= 0 (line-indent line)) + (line-empty-code? line))) + +(define (line-strip-continuation line ) + (if (line-continues? line) + (append + (list + (line-indent line)) + (cdr (line-code line))) + line)) + +(define (line-strip-indentation-marker line) + "Strip the indentation markers from the beginning of the line" + (cdr line)) + +(define (indent-level-reduction indentation-levels level select-fun) + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" + (let loop + ((newlevels indentation-levels) + (diff 0)) + (cond + ((= level (car newlevels)) + (select-fun (list diff indentation-levels))) + ((< level (car newlevels)) + (loop + (cdr newlevels) + (1+ diff))) + (else + (throw 'wisp-syntax-error "Level ~A not found in the indentation-levels ~A."))))) + +(define (indent-level-difference indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x ); get the count + (car x)))) + +(define (indent-reduce-to-level indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x ); get the levels + (car (cdr x))))) + +(define (chunk-ends-with-period currentsymbols next-char) + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." + (and (not (null? currentsymbols)) + (equal? #\newline next-char) + (equal? repr-dot + (list-ref currentsymbols (- (length currentsymbols) 1))))) + +(define (wisp-scheme-read-chunk-lines port) + (let loop + ((indent-and-symbols (list )); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (inindent #t) + (inunderscoreindent (equal? #\_ (peek-char port))) + (incomment #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) + (cond + ((>= emptylines 2 ); the chunk end has to be checked + ; before we look for new chars in the + ; port to make execution in the REPL + ; after two empty lines work + ; (otherwise it shows one more line). + indent-and-symbols) + (else + (let ((next-char (peek-char port))) + (cond + ((eof-object? next-char) + (append indent-and-symbols (list (append (list currentindent) currentsymbols)))) + ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) + (append indent-and-symbols )); top-level form ends chunk + ((chunk-ends-with-period currentsymbols next-char) + ; the line ends with a period. This is forbidden in + ; SRFI-119. Use it to end the line in the REPL without + ; showing continuation dots (...). + (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1))))) + ((and inindent (equal? #\space next-char)) + (read-char port ); remove char + (loop + indent-and-symbols + #t ; inindent + #f ; inunderscoreindent + #f ; incomment + (1+ currentindent) + currentsymbols + emptylines)) + ((and inunderscoreindent (equal? #\_ next-char)) + (read-char port ); remove char + (loop + indent-and-symbols + #t ; inindent + #t ; inunderscoreindent + #f ; incomment + (1+ currentindent) + currentsymbols + emptylines)) + ; any char but whitespace *after* underscoreindent is + ; an error. This is stricter than the current wisp + ; syntax definition. TODO: Fix the definition. Better + ; start too strict. FIXME: breaks on lines with only + ; underscores which should be empty lines. + ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) + ((equal? #\newline next-char) + (read-char port ); remove the newline + ; The following two lines would break the REPL by requiring one char too many. + ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port + ; read-char port ; remove a full \n\r. Damn special cases... + (let* ; distinguish pure whitespace lines and lines + ; with comment by giving the former zero + ; indent. Lines with a comment at zero indent + ; get indent -1 for the same reason - meaning + ; not actually empty. + ( + (indent + (cond + (incomment + (if (= 0 currentindent ); specialcase + -1 + currentindent )) + ((not (null? currentsymbols )); pure whitespace + currentindent) + (else + 0))) + (parsedline (append (list indent) currentsymbols)) + (emptylines + (if (not (line-empty? parsedline)) + 0 + (1+ emptylines)))) + (when (not (= 0 (length parsedline))) + ; set the source properties to parsedline so we can try to add them later. + (set-source-property! parsedline 'filename (port-filename port)) + (set-source-property! parsedline 'line (port-line port))) + ; TODO: If the line is empty. Either do it here and do not add it, just + ; increment the empty line counter, or strip it later. Replace indent + ; -1 by indent 0 afterwards. + (loop + (append indent-and-symbols (list parsedline)) + #t ; inindent + (if (<= 2 emptylines) + #f ; chunk ends here + (equal? #\_ (peek-char port ))); are we in underscore indent? + #f ; incomment + 0 + '() + emptylines))) + ((equal? #t incomment) + (read-char port ); remove one comment character + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #t ; incomment + currentindent + currentsymbols + emptylines)) + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char) ); remove whitespace when not in indent + (read-char port ); remove char + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #f ; incomment + currentindent + currentsymbols + emptylines)) + ; | cludge to appease the former wisp parser + ; | used for bootstrapping which has a + ; v problem with the literal comment char + ((equal? (string-ref ";" 0) next-char) + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #t ; incomment + currentindent + currentsymbols + emptylines)) + (else ; use the reader + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #f ; incomment + currentindent + ; this also takes care of the hashbang and leading comments. + (append currentsymbols (list (wisp-read port))) + emptylines)))))))) + + +(define (line-code-replace-inline-colons line) + "Replace inline colons by opening parens which close at the end of the line" + ; format #t "replace inline colons for line ~A\n" line + (let loop + ((processed '()) + (unprocessed line)) + (cond + ((null? unprocessed) + ; format #t "inline-colons processed line: ~A\n" processed + processed) + ; replace : . with nothing + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) + (loop + (append processed + (loop '() (cdr (cdr unprocessed)))) + '())) + ((equal? readcolon (car unprocessed)) + (loop + ; FIXME: This should turn unprocessed into a list. + (append processed + (list (loop '() (cdr unprocessed)))) + '())) + (else + (loop + (append processed + (list (car unprocessed))) + (cdr unprocessed)))))) + +(define (line-replace-inline-colons line) + (cons + (line-indent line) + (line-code-replace-inline-colons (line-code line)))) + +(define (line-strip-lone-colon line) + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." + (if + (equal? + (line-code line) + (list readcolon)) + (list (line-indent line)) + line)) + +(define (line-finalize line) + "Process all wisp-specific information in a line and strip it" + (let + ( + (l + (line-code-replace-inline-colons + (line-strip-indentation-marker + (line-strip-lone-colon + (line-strip-continuation line)))))) + (when (not (null? (source-properties line))) + (catch #t + (lambda () + (set-source-properties! l (source-properties line))) + (lambda (key . arguments) + #f))) + l)) + +(define (wisp-add-source-properties-from source target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) + +(define (wisp-propagate-source-properties code) + "Propagate the source properties from the sourrounding list into every part of the code." + (let loop + ((processed '()) + (unprocessed code)) + (cond + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) + unprocessed) + ((and (pair? unprocessed) (not (list? unprocessed))) + (cons + (wisp-propagate-source-properties (car unprocessed)) + (wisp-propagate-source-properties (cdr unprocessed)))) + ((null? unprocessed) + processed) + (else + (let ((line (car unprocessed))) + (if (null? (source-properties unprocessed)) + (wisp-add-source-properties-from line unprocessed) + (wisp-add-source-properties-from unprocessed line)) + (loop + (append processed (list (wisp-propagate-source-properties line))) + (cdr unprocessed))))))) + +(define* (wisp-scheme-indentation-to-parens lines) + "Add parentheses to lines and remove the indentation markers" + (when + (and + (not (null? lines)) + (not (line-empty-code? (car lines))) + (not (= 0 (line-real-indent (car lines ))))); -1 is a line with a comment + (if (= 1 (line-real-indent (car lines))) + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands + (set! lines + (cons + (cons 0 (cdr (car lines))) + (cdr lines))) + (throw 'wisp-syntax-error + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" + (car lines))))) + (let loop + ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* + ( + (current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (list 0 ))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (list 0 ))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) + ; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ; . processed current-line next-line unprocessed indentation-levels current-indentation + (cond + ; the real end: this is reported to the outside world. + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) + ; display "done\n" + ; reverse the processed lines, because I use cons. + processed) + ; the recursion end-condition + ((and (null? unprocessed)) + ; display "last step\n" + ; this is the last step. Nothing more to do except + ; for rolling up the indentation levels. return the + ; new processed and unprocessed lists: this is a + ; side-recursion + (values processed unprocessed)) + ((null? indentation-levels) + ; display "indentation-levels null\n" + (throw 'wisp-programming-error "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")) + (else ; now we come to the line-comparisons and indentation-counting. + (cond + ((line-empty-code? current-line) + ; display "current-line empty\n" + ; We cannot process indentation without + ; code. Just switch to the next line. This should + ; only happen at the start of the recursion. + ; TODO: Somehow preserve the line-numbers. + (loop + processed + (cdr unprocessed) + indentation-levels)) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed ))) + ; display "next-line empty\n" + ; TODO: Somehow preserve the line-numbers. + ; take out the next-line from unprocessed. + (loop + processed + (cons current-line + (cdr (cdr unprocessed))) + indentation-levels)) + ((> current-indentation current-line-indentation) + ; display "current-indent > next-line\n" + ; this just steps back one level via the side-recursion. + (let ((previous-indentation (car (cdr indentation-levels)))) + (if (<= current-line-indentation previous-indentation) + (values processed unprocessed) + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. + (let ((linenumber (- (length lines) (length unprocessed)))) + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + (cdr indentation-levels))))))) + ((= current-indentation current-line-indentation) + ; display "current-indent = next-line\n" + (let + ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) + (cond + ((>= current-line-indentation next-line-indentation) + ; simple recursiive step to the next line + ; display "current-line-indent >= next-line-indent\n" + (loop + (append processed + (if (line-continues? current-line) + line + (wisp-add-source-properties-from line (list line)))) + (cdr unprocessed ); recursion here + indentation-levels)) + ((< current-line-indentation next-line-indentation) + ; display "current-line-indent < next-line-indent\n" + ; format #t "line: ~A\n" line + ; side-recursion via a sublist + (let-values + ( + ((sub-processed sub-unprocessed) + (loop + line + (cdr unprocessed ); recursion here + indentation-levels))) + ; format #t "side-recursion:\n sub-processed: ~A\n processed: ~A\n\n" sub-processed processed + (loop + (append processed (list sub-processed)) + sub-unprocessed ; simply use the recursion from the sub-recursion + indentation-levels)))))) + ((< current-indentation current-line-indentation) + ; display "current-indent < next-line\n" + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + indentation-levels))) + (else + (throw 'wisp-not-implemented + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))) + + +(define (wisp-scheme-replace-inline-colons lines) + "Replace inline colons by opening parens which close at the end of the line" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (list (line-replace-inline-colons (car unprocessed)))) + (cdr unprocessed))))) + + +(define (wisp-scheme-strip-indentation-markers lines) + "Strip the indentation markers from the beginning of the lines" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (cdr (car unprocessed))) + (cdr unprocessed))))) + +(define (wisp-unescape-underscore-and-colon code) + "replace \\_ and \\: by _ and :" + (match code + ((a ...) + (map wisp-unescape-underscore-and-colon a)) + ('\_ + '_) + ('\__ + '__) + ('\___ + '___) + ('\____ + '____) + ('\_____ + '_____) + ('\______ + '______) + ('\_______ + '_______) + ('\________ + '________) + ('\_________ + '_________) + ('\__________ + '__________) + ('\___________ + '___________) + ('\____________ + '____________) + ('\: + ':) + (a + a))) + + +(define (wisp-replace-empty-eof code) + "replace ((#<eof>)) by ()" + ; FIXME: Actually this is a hack which fixes a bug when the + ; parser hits files with only hashbang and comments. + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) + (list) + code)) + + +(define (wisp-replace-paren-quotation-repr code) + "Replace lists starting with a quotation symbol by + quoted lists." + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a))) + +(define (wisp-make-improper code) + "Turn (a #{.}# b) into the correct (a . b). + +read called on a single dot creates a variable named #{.}# (|.| +in r7rs). Due to parsing the indentation before the list +structure is known, the reader cannot create improper lists +when it reads a dot. So we have to take another pass over the +code to recreate the improper lists. + +Match is awesome!" + (let + ( + (improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (append (map wisp-make-improper a) + (cons (wisp-make-improper b) (wisp-make-improper c)))) + ((a ...) + (map wisp-make-improper a)) + (a + a)))) + (define (syntax-error li msg) + (throw 'wisp-syntax-error (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))) + (if #t + improper + (let check + ((tocheck improper)) + (match tocheck + ; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ; more complex pairs + ((? pair? a) + (let + ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a + a)))))) + +(define (wisp-scheme-read-chunk port) + "Read and parse one chunk of wisp-code" + (let (( lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines)))))))) + +(define (wisp-scheme-read-all port) + "Read all chunks from the given port" + (let loop + ((tokens '())) + (cond + ((eof-object? (peek-char port)) + tokens) + (else + (loop + (append tokens (wisp-scheme-read-chunk port))))))) + +(define (wisp-scheme-read-file path) + (call-with-input-file path wisp-scheme-read-all)) + +(define (wisp-scheme-read-file-chunk path) + (call-with-input-file path wisp-scheme-read-chunk)) + +(define (wisp-scheme-read-string str) + (call-with-input-string str wisp-scheme-read-all)) + +(define (wisp-scheme-read-string-chunk str) + (call-with-input-string str wisp-scheme-read-chunk)) + diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm new file mode 100644 index 000000000..d5ea7abce --- /dev/null +++ b/module/language/wisp/spec.scm @@ -0,0 +1,107 @@ +;; Language interface for Wisp in Guile + +;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm + +;;; Copyright (C) 2005-2014 by David A. Wheeler and Alan Manuel K. Gloria +;;; Copyright (C) Arne Babenhauserheide (2014--2023). + +;;; Permission is hereby granted, free of charge, to any person +;;; obtaining a copy of this software and associated documentation +;;; files (the "Software"), to deal in the Software without +;;; restriction, including without limitation the rights to use, copy, +;;; modify, merge, publish, distribute, sublicense, and/or sell copies +;;; of the Software, and to permit persons to whom the Software is +;;; furnished to do so, subject to the following conditions: +;;; +;;; The above copyright notice and this permission notice shall be +;;; included in all copies or substantial portions of the Software. +;;; +;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +;;; SOFTWARE. + +; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm +(define-module (language wisp spec) + #:use-module (language wisp) + #:use-module (system base compile) + #:use-module (system base language) + #:use-module (language scheme compile-tree-il) + #:use-module (language scheme decompile-tree-il) + #:export (wisp)) + +; Set locale to something which supports unicode. Required to avoid using fluids. +(catch #t + (lambda () + (setlocale LC_ALL "")) + (lambda (key . parameters) + (let ((locale-fallback "en_US.UTF-8")) + (format (current-error-port) + (string-join + (list ";;; Warning: setlocale LC_ALL \"\" failed with ~A: ~A" + "switching to explicit ~A locale. Please setup your locale." + "If this fails, you might need glibc support for unicode locales.\n") + "\n;;; ") + key parameters locale-fallback) + (catch #t + (lambda () + (setlocale LC_ALL locale-fallback)) + (lambda (key . parameters) + (format (current-error-port) + (string-join + (list ";;; Warning: fallback setlocale LC_ALL ~A failed with ~A: ~A" + "Not switching to Unicode." + "You might need glibc support for unicode locales.\n") + "\n;;; ") + locale-fallback key parameters)))))) + +;;; +;;; Language definition +;;; + +(define wisp-pending-sexps (list)) + +(define (read-one-wisp-sexp port env) + ;; allow using "# foo" as #(foo). + (read-hash-extend #\# (λ (chr port) #\#)) + (cond + ((eof-object? (peek-char port)) + (read-char port )); return eof: we’re done + (else + (let ((chunk (wisp-scheme-read-chunk port))) + (cond + ((not (null? chunk)) + (car chunk)) + (else + #f)))))) + +(define-language wisp + #:title "Wisp Scheme Syntax. See SRFI-119 for details." + ; . #:reader read-one-wisp-sexp + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; + #:compilers `((tree-il . ,compile-tree-il)) + #:decompilers `((tree-il . ,decompile-tree-il)) + #:evaluator (lambda (x module) (primitive-eval x)) + #:printer write ; TODO: backtransform to wisp? Use source-properties? + #:make-default-environment + (lambda () + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', + ;; `fluid-set!', etc. don't have any effect in the current environment. + (let ((m (make-fresh-user-module))) + ;; Provide a separate `current-reader' fluid so that + ;; compile-time changes to `current-reader' are + ;; limited to the current compilation unit. + (module-define! m 'current-reader (make-fluid)) + ;; Default to `simple-format', as is the case until + ;; (ice-9 format) is loaded. This allows + ;; compile-time warnings to be emitted when using + ;; unsupported options. + (module-set! m 'format simple-format) + m))) + + + diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test new file mode 100644 index 000000000..a888df41d --- /dev/null +++ b/test-suite/tests/srfi-119.test @@ -0,0 +1,81 @@ +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*- +;;;; +;;;; Copyright (C) 2023 Free Software Foundation, Inc. +;;;; +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +(define-module (test-srfi-119) + #:use-module (test-suite lib) + #:use-module (srfi srfi-1) + #:use-module (language wisp)) + +(define (read-string s) + (with-input-from-string s read)) + +(define (with-read-options opts thunk) + (let ((saved-options (read-options))) + (dynamic-wind + (lambda () + (read-options opts)) + thunk + (lambda () + (read-options saved-options))))) + +(define (wisp->list str) + (wisp-scheme-read-string str)) + +(with-test-prefix "wisp-read-simple" + (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) + (pass-if (equal? (wisp->list ". 5") '(5))) + (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) +(with-test-prefix "wisp-read-complex" + (pass-if (equal? (wisp->list " +a b c d e + . f g h + . i j k + +concat \"I want \" + getwish from me + . \" - \" username +") '( +(a b c d e + f g h + i j k) + +(concat "I want " + (getwish from me) + " - " username)))) + + (pass-if (equal? (wisp->list " +define : a b c +_ d e +___ f +___ g h +__ . i + +define : _ +_ display \"hello\n\" + +\\_") '( +(define (a b c) + (d e + (f) + (g h) + i)) + +(define (_) + (display "hello\n")) + +(_))))) -- 2.39.1 [-- Attachment #1.3: Type: text/plain, Size: 81 bytes --] -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-03 21:26 [PATCH] add language/wisp to Guile? Dr. Arne Babenhauserheide @ 2023-02-04 15:08 ` Maxime Devos 2023-02-04 15:46 ` Dr. Arne Babenhauserheide 2023-02-16 8:03 ` Dr. Arne Babenhauserheide 2023-09-30 13:17 ` Christine Lemmer-Webber 1 sibling, 2 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-04 15:08 UTC (permalink / raw) To: Dr. Arne Babenhauserheide, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 4872 bytes --] > Why add Wisp? > > For Wisp: it is then available directly wherever Guile is available. > This will make it much easier for people to follow tutorials. I'm not convinced of this argument, because package managers exist, but ... > For Guile: > > - Wisp has proven to be good at enabling people to get an > entrance to Scheme² without pulling them out of the community. > - [...] ... all good points, and the implementation of Wisp is tiny anyway. For an additional reason: Wisp is a SRFI (Scheme Requests for Implementation) and Guile is a Scheme implementation. > So I’d like to ask: can we merge Wisp as supported language into Guile? From some conversations elsewhere, I got the impression that (use-modules (foo)) will search for foo.scm and not in foo.w. I think you'll need to tweak the loading mechanism to also look for foo.w instead of only foo.scm, if not done already. Also, I think that when foo.go exists, but foo.scm doesn't, then Guile refuses to load foo.scm, though I'm less sure of that. If this is the case, I propose removing the requirement that the source code is available, or alternatively keep the 'source code available' requirement and also accept 'foo.w', if not done already. > +; Set locale to something which supports unicode. Required to avoid > using fluids. > +(catch #t * Why avoid fluids? * Assuming for sake of argument that fluids are to be avoided, what is the point of setting the locale to something supporting Unicode? As-is, it now becomes impossible to use 'gettext' to translate software to non-English locales when the software imports (language wisp), which seems unfortunate to me. If you elaborate on what your goal here is, maybe I have an alternative solution. > + ;; allow using "# foo" as #(foo). > + (read-hash-extend #\# (λ (chr port) #\#)) That's a rather Wisp-specific extension, but it appears you are extending things globally. Instead, I propose extending it temporarily, with the undocumented '%read-hash-procedures' fluid. > + (let > + ( > + (l Lonely parenthesis. + (not (= 0 (line-real-indent (car lines ))))); -1 is a line with a comment Superfluous space after 'lines'. > + ; simple recursiive step to the next line I think the convention is ';;', OTOH there exist multiple conventions. +(define (wisp-scheme-replace-inline-colons lines) + "Replace inline colons by opening parens which close at the end of the line" Too much space; convention is two spaces. (Similar styles issues in other places.) "guix style" might be useful. > +(define (wisp-replace-paren-quotation-repr code) > + "Replace lists starting with a quotation symbol by > + quoted lists." > + (match code > + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'quote (map wisp-replace-paren-quotation-repr a))) > [...] > +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") > +; define an intermediate dot replacement with UUID to avoid clashes. > +(define repr-dot ; . > + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) There is a risk of collision -- e.g., suppose that someone translates your implementation of Wisp into Wisp. I imagine there might be a risk of misinterpreting the 'REPR-QUOTE-...' in wisp-replace-parent-quotation-repr, though I haven't tried it out. As such, assuming this actually works, I propose using uninterned symbols instead, e.g.: (define repr-dot (make-symbol "REPR-DOT")). If this change is done, you might need to replace + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) (unverified -- I think removing this is unneeded but I don't understand this REPR-... stuff well enough). Also, I wonder if you could just do something like (apply vector (map wisp-replace-paren-quotation-repr a)) instead of this 'hack to defer to read' thing. This seems simpler to me and equivalent. (AFAIK, these REPR-... symbols are never written to a port or turned into syntax, so I think that uninterned symbols would work here.) (Aside from the REPR-... thing, I'm assuming (language wisp) is alright -- the SRFI is in 'final' status and it has been stable for years now, after all.) Greetings, Maxime [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-04 15:08 ` Maxime Devos @ 2023-02-04 15:46 ` Dr. Arne Babenhauserheide 2023-02-04 19:09 ` Maxime Devos 2023-02-16 8:03 ` Dr. Arne Babenhauserheide 1 sibling, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-04 15:46 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 7161 bytes --] Thank you for your review! Maxime Devos <maximedevos@telenet.be> writes: >> Why add Wisp? >> For Wisp: it is then available directly wherever Guile is available. >> This will make it much easier for people to follow tutorials. > > I'm not convinced of this argument, because package managers exist, but ... > >> For Guile: >> - Wisp has proven to be good at enabling people to get an >> entrance to Scheme² without pulling them out of the community. >> - [...] > > ... all good points, and the implementation of Wisp is tiny anyway. > For an additional reason: Wisp is a SRFI (Scheme Requests for > Implementation) and Guile is a Scheme implementation. That’s a good point — I should really have written it :-) >> So I’d like to ask: can we merge Wisp as supported language into Guile? > > From some conversations elsewhere, I got the impression that > > (use-modules (foo)) > > will search for foo.scm and not in foo.w. I think you'll need to > tweak the loading mechanism to also look for foo.w instead of only > foo.scm, if not done already. This needs an addition to the extensions via guile -x .w — I wrote that in the documentation. I didn’t want to do that unconditionally, because detecting a wisp file as scheme import would cause errors. Is there a way to only extend the loading mechanism to detect .w when language is changed to wisp? readable uses (set! %load-extensions (cons ".sscm" %load-extensions)) Would that be the correct way of doing this? > Also, I think that when foo.go exists, but foo.scm doesn't, then Guile > refuses to load foo.scm, though I'm less sure of that. If this is the > case, I propose removing the requirement that the source code is > available, or alternatively keep the 'source code available' > requirement and also accept 'foo.w', if not done already. I think accepting any extension supported by any language in Guile would be better. >> +; Set locale to something which supports unicode. Required to avoid >> using fluids. >> +(catch #t > > * Why avoid fluids? I’m not sure anymore. It has been years since I wrote that code … I think it was because I did not understand what that would mean for the program. And I actually still don’t know … Hoow would I do that instead with fluids? > * Assuming for sake of argument that fluids are to be avoided, > what is the point of setting the locale to something supporting > Unicode? I had problems with reading unicode symbols. Things like define (Σ . args) : apply + args > As-is, it now becomes impossible to use 'gettext' to translate > software to non-English locales when the software imports (language > wisp), which seems unfortunate to me. That is very much not what I want. > If you elaborate on what your > goal here is, maybe I have an alternative solution. This is to ensure that Wisp are always read as Unicode. Since it uses regular (read) as part of parsing, it must affect (read), too. >> + ;; allow using "# foo" as #(foo). >> + (read-hash-extend #\# (λ (chr port) #\#)) > > That's a rather Wisp-specific extension, but it appears you are > extending things globally. Instead, I propose extending it > temporarily, with the undocumented '%read-hash-procedures' fluid. > >> + (let >> + ( >> + (l > > Lonely parenthesis. Thank you! Will be fixed :-) > + (not (= 0 (line-real-indent (car lines ))))); -1 is a > line with a comment > > Superfluous space after 'lines'. > >> + ; simple recursiive step to the next line > > I think the convention is ';;', OTOH there exist multiple conventions. > > +(define (wisp-scheme-replace-inline-colons lines) > + "Replace inline colons by opening parens which close at the > end of the line" > > Too much space; convention is two spaces. > (Similar styles issues in other places.) > "guix style" might be useful. I’ll do that … >> +(define (wisp-replace-paren-quotation-repr code) >> + "Replace lists starting with a quotation symbol by >> + quoted lists." >> + (match code >> + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) >> + (list 'quote (map wisp-replace-paren-quotation-repr a))) >> [...] >> +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") >> +; define an intermediate dot replacement with UUID to avoid clashes. >> +(define repr-dot ; . >> + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) > > There is a risk of collision -- e.g., suppose that someone translates > your implementation of Wisp into Wisp. I imagine there might be a > risk of misinterpreting the 'REPR-QUOTE-...' in > wisp-replace-parent-quotation-repr, though I haven't tried it out. This is actually auto-translated from wisp via wisp2lisp :-) > As such, assuming this actually works, I propose using uninterned > symbols instead, e.g.: > > (define repr-dot (make-symbol "REPR-DOT")). That looks better — does uninterned symbol mean it can’t be mis-interpreted? Can I (match l ...) on uninterned symbols? They are used to match on precisely these symbols later. Can I write it into a string and then read it back? When I see them, I have to turn them into a different representation that I can then write back into the string and allow it to be read by the normal reader. > If this change is done, you might need to replace > > + ;; literal array as start of a line: # (a b) c -> (#(a b) c) > + ((#\# a ...) > + (with-input-from-string ;; hack to defer to read > + (string-append "#" > + (with-output-to-string > + (λ () > + (write (map > wisp-replace-paren-quotation-repr a) > + (current-output-port))))) > + read)) > > > (unverified -- I think removing this is unneeded but I don't > understand this REPR-... stuff well enough). The REPR supports the syntactic sugar like '(...) for (quote ...) by turning (' ...) into '(...). Also it is needed to turn ((. a b c)) into (a b c). However the literal array is used to make it possible to define procedure properties which need a literal array. > Also, I wonder if you could just do something like > > (apply vector (map wisp-replace-paren-quotation-repr a)) > > instead of this 'hack to defer to read' thing. This seems simpler to > me and equivalent. That looks much cleaner. Thank you! > (AFAIK, these REPR-... symbols are never written to a port or turned > into syntax, so I think that uninterned symbols would work here.) They are unread into a string. > (Aside from the REPR-... thing, I'm assuming (language wisp) is > alright -- the SRFI is in 'final' status and it has been stable for > years now, after all.) Thank you! Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-04 15:46 ` Dr. Arne Babenhauserheide @ 2023-02-04 19:09 ` Maxime Devos 2023-02-04 21:35 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-04 19:09 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 12311 bytes --] On 04-02-2023 16:46, Dr. Arne Babenhauserheide wrote: > [...] >>> So I’d like to ask: can we merge Wisp as supported language into Guile? >> >> From some conversations elsewhere, I got the impression that >> >> (use-modules (foo)) >> >> will search for foo.scm and not in foo.w. I think you'll need to >> tweak the loading mechanism to also look for foo.w instead of only >> foo.scm, if not done already. > > This needs an addition to the extensions via guile -x .w — I wrote that > in the documentation. I didn’t want to do that unconditionally, because > detecting a wisp file as scheme import would cause errors. If done carefully, I don't think this situations would happen. More precisely: * .w would be in the file extensions list. * Instead of a list, it would actually be a map from extensions to languages: .scm -> scheme .w -> wisp With this change, (use-modules (foo)) will load 'foo.scm' as Scheme and 'foo.w' as Wisp. (Assuming that foo.go is out-of-date or doesn't exist.) (For backwards compatibility, I think %load-extensions needs to remain a list of strings, but a %extension-language variable could be defined.) * "guile --language=whatever foo" loads foo as whatever, regardless of the extension of 'foo' (if a specific language is requested, then the user knows best). * "guile foo" without --language will look up the extension of foo in the extension map. If an entry exists, it would use the corresponding language. If no entry exists, it would use a default language (scheme). With these changes, I don't think that Wisp code would be detected as Scheme or the other way around. > Is there a way to only extend the loading mechanism to detect .w when > language is changed to wisp? Regardless of whether it's technically possible, that sounds insufficient to me. Suppose someone writes a library 'Foo' in Wisp. Suppose I write a library 'Bar' in parenthese-y Scheme, that happens to use the Foo library as a dependency. Then when compiling Bar or running its tests, it will be done in the Scheme language, and additionally assuming that compiled .go are available for Foo, then the language will never be changed to Wisp, and hence .w will never be added to %load-extensions. As such, the Makefile.am or equivalent of Foo would need to be converted to Wisp, or '-x w' would need to be added. I don't care what language the library Foo is written in, and my library Bar isn't written in Wisp so it seems unreasonable to have to add -x w. (It wouldn't be too much trouble, but still not something that should have to be done _in Bar_, as the Wispyness of Foo is just an implementation detail of Foo, not Bar.) Worse, adding the Wispy library Foo of the parenthese-y library Bar would be an incompatible change, as parenthese-y dependents of Foo would need to add '-x w' in places whereas they didn't to previously. It's easily resolvable, but I think it would be very annoying as well. > readable uses This sentence appears to be incomplete; I might have misinterpreted it below (I don't know what you mean with 'readable' -- its an adjective and you are using it as a noun?). > (set! %load-extensions (cons ".sscm" %load-extensions)) > > Would that be the correct way of doing this? I assume you meant ".w" instead of ".sscm". I don't quite see how this would be an answer to: Is there a way to only extend the loading mechanism to detect .w when language is changed to wisp? More precisely, I'm missing how it addresses 'only ... when the language is changed to wisp'. FWIW, it appears to be an answer to the following unasked question: How to make Guile accept "foo.go" when "foo.w" exists and is up-to-date. >> Also, I think that when foo.go exists, but foo.scm doesn't, then Guile >> refuses to load foo.scm, though I'm less sure of that. If this is the >> case, I propose removing the requirement that the source code is >> available, or alternatively keep the 'source code available' >> requirement and also accept 'foo.w', if not done already. > > I think accepting any extension supported by any language in Guile would > be better. This sounds like the second proposal ('alternatively ...'), but the way it is written, you appear to proposing it as a third proposal. Is this the case? (I mean, after this patch, Wisp is a supported language, so it seems equivalent to me.) >>> +; Set locale to something which supports unicode. Required to avoid >>> using fluids. >>> +(catch #t >> >> * Why avoid fluids? > > I’m not sure anymore. It has been years since I wrote that code … > > I think it was because I did not understand what that would mean for the > program. And I actually still don’t know … > > Hoow would I do that instead with fluids? > >> * Assuming for sake of argument that fluids are to be avoided, >> what is the point of setting the locale to something supporting >> Unicode? > > I had problems with reading unicode symbols. Things like > define (Σ . args) : apply + args > [...]> > This is to ensure that Wisp are always read as Unicode. Since it uses > regular (read) as part of parsing, it must affect (read), too. OK. So, Wisp files are supposed to be UTF-8, no matter the locale? AFAICT, the SRFI-119 document does not mention this UTF-8 (or UTF-16, or ...) requirement anywhere, this seems like an omission in <https://srfi.schemers.org/srfi-119/srfi-119.html> to me. First, I would like to point out the following part of ‘(guile)The Top of a Script File’: • If this source code file is not ASCII or ISO-8859-1 encoded, a coding declaration such as ‘coding: utf-8’ should appear in a comment somewhere in the first five lines of the file: see *note Character Encoding of Source Files::. oing by this, it is already possible to ask Guile to read the Scheme files as UTF-8; presumably the relevant bits could be copied over to Wisp. (I don't know if this applies to non-script files, but I'd assume so.) It's not 'UTF-8 by default', but it can be 'close enough', and doing 'always UTF-8 even if coding: something-else' would be inconsistent with the Scheme language, so I ask you to consider whether it's worth (and perhaps the answer is 'yes'). (OTOH, (guile)Character Encoding says 'In the absence of any hints, UTF-8 is assumed.' which appears to suffice for you, but it also contradicts "If this source file is not ASCII or ISO-8859-1 encodes, ...", so I don't know what precisely is going on here.) If you aren't going for the 'coding: ...' stuff or porting the encoding autodetection from Scheme to Wisp, here's an alternative solution: Keep in mind that encodings are a per-port property -- the locale might have a default encoding, and ports by default take the encoding from %default-port-encoding or the locale (I think), but you can override the port encoding: -- Scheme Procedure: set-port-encoding! port enc -- C Function: scm_set_port_encoding_x (port, enc) Sets the character encoding that will be used to interpret I/O to PORT. ENC is a string containing the name of an encoding. Valid encoding names are those defined by IANA (http://www.iana.org/assignments/character-sets), for example ‘"UTF-8"’ or ‘"ISO-8859-1"’. As such, I propose calling set-port-encoding! right in the beginning of read-one-wisp-sexp. Also, unrelated, I now noticed some dead code you can remove: +(define wisp-pending-sexps (list)) > [...] >>> +(define (wisp-replace-paren-quotation-repr code) >>> + "Replace lists starting with a quotation symbol by >>> + quoted lists." >>> + (match code >>> + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) >>> + (list 'quote (map wisp-replace-paren-quotation-repr a))) >>> [...] >>> +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") >>> +; define an intermediate dot replacement with UUID to avoid clashes. >>> +(define repr-dot ; . >>> + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) >> >> There is a risk of collision -- e.g., suppose that someone translates >> your implementation of Wisp into Wisp. I imagine there might be a >> risk of misinterpreting the 'REPR-QUOTE-...' in >> wisp-replace-parent-quotation-repr, though I haven't tried it out. > > This is actually auto-translated from wisp via wisp2lisp :-) > >> As such, assuming this actually works, I propose using uninterned >> symbols instead, e.g.: >> >> (define repr-dot (make-symbol "REPR-DOT")). > > That looks better — does uninterned symbol mean it can’t be > mis-interpreted? Yes. This is because 'read' only reads interned symbols; uninterned symbols are unreadable: scheme@(guile-user)> (make-symbol "foo") $1 = #<uninterned-symbol foo 7f17efab7240> scheme@(guile-user)> #<uninterned-symbol foo 7f17efab7240> While reading expression: #<unknown port>:2:3: Unknown # object: "#<" Also: (eq? (make-symbol "stuff") 'stuff) -> #false. > Can I (match l ...) on uninterned symbols? They are used to match on > precisely these symbols later. Yes, but it's going to look differently and more verbose: (define interned-symbol1 (make-symbol "foo1")) (define interned-symbol2 (make-symbol "foo2")) (match symbol ((? (lambda (x) (eq? x interned-symbol1))) stuff1) ((? (lambda (x) (eq? x interned-symbol2))) stuff2) [...]) -- basically, replace 'stuff by (? (lambda (x) ...)). > Can I write it into a string and then read it back? No. If you could, then uninterned symbols wouldn't be uninterned anymore, but rather a separation of symbols in two kinds that pretty much behave the same, and then you would again have a (very low) risk of a collision: > When I see them, I have to turn them into a different representation > that I can then write back into the string and allow it to be read by > the normal reader. That's the case for the old code, but AFAIK it is only done in the following ... > >> If this change is done, you might need to replace >> >> + ;; literal array as start of a line: # (a b) c -> (#(a b) c) >> + ((#\# a ...) >> + (with-input-from-string ;; hack to defer to read >> + (string-append "#" >> + (with-output-to-string >> + (λ () >> + (write (map >> wisp-replace-paren-quotation-repr a) >> + (current-output-port))))) >> + read)) >> >> (unverified -- I think removing this is unneeded but I don't >> understand this REPR-... stuff well enough). ..., for which I proposed a replacement, so do you still need to turn it in a string & back? > > The REPR supports the syntactic sugar like '(...) for (quote ...) by turning > (' ...) into '(...). > > Also it is needed to turn ((. a b c)) into (a b c). > > However the literal array is used to make it possible to define > procedure properties which need a literal array. > >> Also, I wonder if you could just do something like >> >> (apply vector (map wisp-replace-paren-quotation-repr a)) >> >> instead of this 'hack to defer to read' thing. This seems simpler to >> me and equivalent. > > That looks much cleaner. Thank you! This sounds positive, but it is unclear to me if I have found a solution, because of your negative "However the literal array is used to make it possible to define procedure properties which need a literal array." comment. Do I need to look into solving the 'literal array and procedure properties' stuff, or does the (apply vector (map ...)) suffice as-is? (If there is 'literal array and procedure properties' stuff to be solved, you will need to elaborate on what you mean, because arrays aren't procedures and procedures aren't arrays -- maybe you meant 'object properties'?) Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-04 19:09 ` Maxime Devos @ 2023-02-04 21:35 ` Dr. Arne Babenhauserheide 2023-02-05 15:08 ` Maxime Devos 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-04 21:35 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 10358 bytes --] Maxime Devos <maximedevos@telenet.be> writes: >> This needs an addition to the extensions via guile -x .w — I wrote >> that >> in the documentation. I didn’t want to do that unconditionally, because >> detecting a wisp file as scheme import would cause errors. > > If done carefully, I don't think this situations would happen. > More precisely: > > * .w would be in the file extensions list. > > * Instead of a list, it would actually be a map from extensions to > languages: > > .scm -> scheme > .w -> wisp > > With this change, (use-modules (foo)) will load 'foo.scm' as Scheme > and 'foo.w' as Wisp. (Assuming that foo.go is out-of-date or > doesn't exist.) > > (For backwards compatibility, I think %load-extensions needs to > remain a list of strings, but a %extension-language variable could > be defined.) > > * "guile --language=whatever foo" loads foo as whatever, regardless > of the extension of 'foo' (if a specific language is requested, > then the user knows best). > > * "guile foo" without --language will look up the extension of foo in > the extension map. If an entry exists, it would use the > corresponding language. If no entry exists, it would use > a default language (scheme). This sounds good, though a bit more complex than I think it should be. I think this should stick to only load Scheme if no language is detected to keep Scheme the default language for Guile — and also to avoid stumbling over files that just take that extension. Checking more files could slow down startup and I think having multiple languages fully equal would risk splintering the development community. Guile is first and foremost Scheme and fast startup time is essential. More complicated is what should be done if a *.go file is detected during import. There I could see Guile check if a file with any supported extension is up to date. >> Is there a way to only extend the loading mechanism to detect .w when >> language is changed to wisp? > I don't care what language the library Foo is written in, and my > library Bar isn't written in Wisp so it seems unreasonable to have to > add -x w. I think you’re right with that. For any already compiled library, the language should not matter. >> readable uses > > This sentence appears to be incomplete; I might have misinterpreted it > below (I don't know what you mean with 'readable' -- its an adjective > and you are using it as a noun?). readable is a noun, yes: the readable lisp project. >> (set! %load-extensions (cons ".sscm" %load-extensions)) >> Would that be the correct way of doing this? > FWIW, it appears to be an answer to the following unasked question: > > How to make Guile accept "foo.go" when "foo.w" exists and is > up-to-date. Yes, I think that is the most important question. If that is solved, guile provides a multi-language environment in which only the build tools of the libraries themselves have to know the languages used. > This sounds like the second proposal ('alternatively ...'), but the > way it is written, you appear to proposing it as a third proposal. Is > this the case? It only differs in details (keeping Scheme more central and only checking for non-scheme languages if a *.go file is detected). > (I mean, after this patch, Wisp is a supported language, so it seems > equivalent to me.) Pretty close, yes. >>>> +; Set locale to something which supports unicode. Required to avoid >>>> using fluids. >>>> +(catch #t >>> >>> * Why avoid fluids? >> I’m not sure anymore. It has been years since I wrote that code … >> I think it was because I did not understand what that would mean for >> the >> program. And I actually still don’t know … >> Hoow would I do that instead with fluids? >> >>> * Assuming for sake of argument that fluids are to be avoided, >>> what is the point of setting the locale to something supporting >>> Unicode? >> I had problems with reading unicode symbols. Things like >> define (Σ . args) : apply + args >> [...]> >> This is to ensure that Wisp are always read as Unicode. Since it uses >> regular (read) as part of parsing, it must affect (read), too. > > OK. So, Wisp files are supposed to be UTF-8, no matter the locale? > AFAICT, the SRFI-119 document does not mention this UTF-8 (or UTF-16, > or ...) requirement anywhere, this seems like an omission in > <https://srfi.schemers.org/srfi-119/srfi-119.html> to me. That’s an omission, yes … but since it was omitted (by me …), you’re right. Forcing UTF-8 is actually not the way. > First, I would like to point out the following part of > ‘(guile)The Top of a Script File’: > > • If this source code file is not ASCII or ISO-8859-1 encoded, a > coding declaration such as ‘coding: utf-8’ should appear in a > comment somewhere in the first five lines of the file: see *note > Character Encoding of Source Files::. … > (OTOH, (guile)Character Encoding says 'In the absence of any hints, > UTF-8 is assumed.' which appears to suffice for you, but it also > contradicts "If this source file is not ASCII or ISO-8859-1 encodes, > ...", so I don't know what precisely is going on here.) I think this inconsistency calls for calling in old timers who know why this is there. Maybe one of these is just a leftover? > Keep in mind that encodings are a per-port property -- the locale > might have a default encoding, and ports by default take the encoding > from %default-port-encoding or the locale (I think), but you can > override the port encoding: > > -- Scheme Procedure: set-port-encoding! port enc > -- C Function: scm_set_port_encoding_x (port, enc) > Sets the character encoding that will be used to interpret I/O to > PORT. ENC is a string containing the name of an encoding. Valid > encoding names are those defined by IANA > (http://www.iana.org/assignments/character-sets), for example > ‘"UTF-8"’ or ‘"ISO-8859-1"’. > > As such, I propose calling set-port-encoding! right in the beginning > of read-one-wisp-sexp. This sounds like the best way forward on the short term. > Also, unrelated, I now noticed some dead code you can remove: > > +(define wisp-pending-sexps (list)) You’re right, that was only needed in a previous iteration of wisp (last used more than 3 years ago, IIRC). Thank you! >>> (define repr-dot (make-symbol "REPR-DOT")). >> That looks better — does uninterned symbol mean it can’t be >> mis-interpreted? > > Yes. This is because 'read' only reads interned symbols; uninterned > symbols are unreadable: … >> Can I write it into a string and then read it back? > > No. If you could, then uninterned symbols wouldn't be uninterned > anymore, but rather a separation of symbols in two kinds that pretty > much behave the same, and then you would again have a (very low) risk > of a collision: This sounds like I cannot go that way, because there’s a necessary pre-processing step in wisp-read via (match-charlist-to-repr peeked): (define (wisp-read port) "wrap read to catch list prefixes." (let ((prefix-maxlen 4)) (let longpeek ((peeked '()) (repr-symbol #f)) (cond ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) (if repr-symbol ; found a special symbol, return it. repr-symbol (let unpeek ((remaining peeked)) (cond ((equal? '() remaining) (read port)); let read to the work (else (unread-char (car remaining) port) (unpeek (cdr remaining))))))) (else (let* ((next-char (read-char port)) (peeked (cons next-char peeked))) (longpeek peeked (match-charlist-to-repr peeked)))))))) This actually needs to be able to write the replacement symbols back into the port. > ..., for which I proposed a replacement, so do you still need to turn > it in a string & back? Sadly yes. Otherwise the normal reader will play tricks on the code, because it does not know where a symbol needs to be interpreted differently (i.e. where ` needs to be treated as `() even though that’s not in the string). >> The REPR supports the syntactic sugar like '(...) for (quote ...) by >> turning >> (' ...) into '(...). >> Also it is needed to turn ((. a b c)) into (a b c). >> However the literal array is used to make it possible to define >> procedure properties which need a literal array. >> >>> Also, I wonder if you could just do something like >>> >>> (apply vector (map wisp-replace-paren-quotation-repr a)) >>> >>> instead of this 'hack to defer to read' thing. This seems simpler to >>> me and equivalent. >> That looks much cleaner. Thank you! > > This sounds positive, but it is unclear to me if I have found a > solution, because of your negative "However the literal array is used > to make it possible to define procedure properties which need a > literal array." comment. > > Do I need to look into solving the 'literal array and procedure > properties' stuff, or does the (apply vector (map ...)) suffice as-is? > > (If there is 'literal array and procedure properties' stuff to be > solved, you will need to elaborate on what you mean, because arrays > aren't procedures and procedures aren't arrays -- maybe you meant > 'object properties'?) I meant this: (define (foo) #((bar . baz)) #f) (procedure-properties foo) => ((name . foo) (bar . baz)) I use that for doctests: (define (A) #((tests (test-eqv 'A (A)) (test-assert #t))) 'A) (define %this-module (current-module)) (define (main args) (doctests-testmod %this-module)) Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-04 21:35 ` Dr. Arne Babenhauserheide @ 2023-02-05 15:08 ` Maxime Devos 2023-02-14 8:32 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-05 15:08 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 6790 bytes --] On 04-02-2023 22:35, Dr. Arne Babenhauserheide wrote: > > Maxime Devos <maximedevos@telenet.be> writes: > >>> This needs an addition to the extensions via guile -x .w — I wrote >>> that >>> in the documentation. I didn’t want to do that unconditionally, because >>> detecting a wisp file as scheme import would cause errors. >> >> If done carefully, I don't think this situations would happen. >> More precisely: >> >> * .w would be in the file extensions list. >> >> * Instead of a list, it would actually be a map from extensions to >> languages: >> >> .scm -> scheme >> .w -> wisp >> >> With this change, (use-modules (foo)) will load 'foo.scm' as Scheme >> and 'foo.w' as Wisp. (Assuming that foo.go is out-of-date or >> doesn't exist.) >> >> (For backwards compatibility, I think %load-extensions needs to >> remain a list of strings, but a %extension-language variable could >> be defined.) >> >> * "guile --language=whatever foo" loads foo as whatever, regardless >> of the extension of 'foo' (if a specific language is requested, >> then the user knows best). >> >> * "guile foo" without --language will look up the extension of foo in >> the extension map. If an entry exists, it would use the >> corresponding language. If no entry exists, it would use >> a default language (scheme). > > This sounds good, though a bit more complex than I think it should be. > > I think this should stick to only load Scheme if no language is detected > to keep Scheme the default language for Guile To my knowledge, this is the case in my proposal. Detecting the language is done via the file extension, and if no known mapping exists, it defaults to Scheme. > — and also to avoid > stumbling over files that just take that extension. While I suppose it is theoretically possible someone will write a .w file that contains Scheme code instead of Wisp, I'm not convinced by this argument. It sounds very unlikely, and also a 'don't do that, then' situation. > Checking more files > could slow down startup and I think having multiple languages fully > equal would risk splintering the development community. > > Guile is first and foremost Scheme and fast startup time is essential. > > More complicated is what should be done if a *.go file is detected > during import. There I could see Guile check if a file with any > supported extension is up to date. Maybe the .go could contain some information on what the corresponding source code file name is, and Guile could read the .go without checking first checking for up-to-dateness. (But only reading; not loading yet!) (There is already debugging information with such information, but to my understanding that's for individual procedures, not the .go as a whole, and by using stuff like 'include' or macros there can be multiple source files.) Once read, it should be easy to look up the source code file name from the .go and then verify whether the .go is up to date, and proceed with actually loading the .go (as in, put stuff in the module system, run top-level code, ...). IIUC, that would be fully backwards compatible, and not cause any non-negligible slowdowns. I also have an alternative proposal, more complicated and backwards-incompatible -- I wouldn't recommend it, but for completeness: * when doing (use-module (foo)) and foo.go exists in the $GUILE_LOAD_COMPILED_PATH, load it, and don't bother checking whether foo.scm, foo.w or foo.whatever exists or is up-to-date. (If not done already in Guile.) That should solve the 'I don't care what language the library Foo is written in, and my library Bar isn't written in Wisp so it seems unreasonable to have to add -x w.’, and would also avoid the need for a 'extension -> language map' thing. It should also be a little faster than what we had before. That's for "make install", "apt-get install", "guix install" ...-like uses of compiled .go -- let's call them 'installed .go'. It won't work for ~/.cache/guile/ccache/3.0-LE-8-4 (‘cached .go’) as for that it's actually important to check up-to-dateness because, well, cache. Additionally, to support compiling software that is already installed, there needs to be an option to treat certain modules with the 'cache' behaviour even if not in the actual ~/.cache, maybe with some '--local-module=(stuff ...)' option (name pending). This would be backwards-incompatible, but it could be done. >>> readable uses >> >> This sentence appears to be incomplete; I might have misinterpreted it >> below (I don't know what you mean with 'readable' -- its an adjective >> and you are using it as a noun?). > > readable is a noun, yes: the readable lisp project. Looks like you meant this: <https://readable.sourceforge.io/>. >>> Can I write it into a string and then read it back? >> >> No. [...] > > This sounds like I cannot go that way, because there’s a necessary > pre-processing step in wisp-read via (match-charlist-to-repr peeked): > [...] > This actually needs to be able to write the replacement symbols back > into the port. > >> ..., for which I proposed a replacement, so do you still need to turn >> it in a string & back? > > Sadly yes. Otherwise the normal reader will play tricks on the code, > because it does not know where a symbol needs to be interpreted > differently (i.e. where ` needs to be treated as `() even though that’s > not in the string). OK, too bad. Looks like the REPR-... stuff is to stay for now. > I meant this: > > (define (foo) > #((bar . baz)) > #f) > (procedure-properties foo) > => ((name . foo) (bar . baz)) > > I use that for doctests: [...] I didn't know that these literal vectors get turned into procedure properties ... looks interesting. Also, about wisp-unescape-underscore-and-colon and the 'only unescapes up to 12 leading underscores at line start (\____________)' limitation: I have found a solution: you can use a combination of string-every, symbol->string, string->symbol, substring and string-ref: (cond ((list? code) (map wisp-... code)) ((eq? code '\:) ':) ;; Look for symbols like \____ and remove the \. ((symbol? code) (let ((as-string (symbol->string code))) (if (and (>= (string-length as-string) 2) ; at least a single underscore (char=? (string-ref as-string 0) #\\) (string-every #\_ (substring as-string 1))) (string->symbol (substring as-string 1)) code))) (#true code)) Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-05 15:08 ` Maxime Devos @ 2023-02-14 8:32 ` Dr. Arne Babenhauserheide 2023-02-14 21:24 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-14 8:32 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1.1: Type: text/plain, Size: 6155 bytes --] Maxime Devos <maximedevos@telenet.be> writes: > On 04-02-2023 22:35, Dr. Arne Babenhauserheide wrote: >> Maxime Devos <maximedevos@telenet.be> writes: >> >>>> This needs an addition to the extensions via guile -x .w — I wrote >>>> that >>>> in the documentation. I didn’t want to do that unconditionally, because >>>> detecting a wisp file as scheme import would cause errors. >>> >>> If done carefully, I don't think this situations would happen. >>> More precisely: >>> >>> * .w would be in the file extensions list. >>> >>> * Instead of a list, it would actually be a map from extensions to >>> languages: >>> >>> .scm -> scheme >>> .w -> wisp >>> >>> With this change, (use-modules (foo)) will load 'foo.scm' as Scheme >>> and 'foo.w' as Wisp. (Assuming that foo.go is out-of-date or >>> doesn't exist.) >>> >>> (For backwards compatibility, I think %load-extensions needs to >>> remain a list of strings, but a %extension-language variable could >>> be defined.) >>> >>> * "guile --language=whatever foo" loads foo as whatever, regardless >>> of the extension of 'foo' (if a specific language is requested, >>> then the user knows best). >>> >>> * "guile foo" without --language will look up the extension of foo in >>> the extension map. If an entry exists, it would use the >>> corresponding language. If no entry exists, it would use >>> a default language (scheme). >> This sounds good, though a bit more complex than I think it should >> be. >> I think this should stick to only load Scheme if no language is >> detected >> to keep Scheme the default language for Guile > > To my knowledge, this is the case in my proposal. Detecting the > language is done via the file extension, and if no known mapping > exists, it defaults to Scheme. I’ve been thinking about this for a while and I expect that this will take quite a bit of discussion, because it is a change that would affect every language already shipped by Guile, and it is a change that needs strategic decisions. I like the idea but would like to separate out changes to how languages are treated. >> — and also to avoid >> stumbling over files that just take that extension. > > While I suppose it is theoretically possible someone will write a .w > file that contains Scheme code instead of Wisp, I'm not convinced by > this argument. It sounds very unlikely, and also a 'don't do that, > then' situation. I don’t mean Scheme files, but rather some other tool using the extension for another file format. > Maybe the .go could contain some information on what the corresponding > source code file name is, and Guile could read the .go without > checking first checking for up-to-dateness. (But only reading; not > loading yet!) That would be nice, but would require doing changes in a critical core part of Guile. It would change this addition from a risk-free added feature to a risky core change. > Once read, it should be easy to look up the source code file name from > the .go and then verify whether the .go is up to date, and proceed > with actually loading the .go (as in, put stuff in the module system, > run top-level code, ...). That’s what I’l like to see, yes. > IIUC, that would be fully backwards compatible, and not cause any > non-negligible slowdowns. It could actually speed up loading, because there are fewer locations for caching bytecode files that locations for source-files. > * when doing (use-module (foo)) and foo.go exists in the > $GUILE_LOAD_COMPILED_PATH, load it, and don't > bother checking whether foo.scm, foo.w or foo.whatever exists > or is up-to-date. (If not done already in Guile.) > > That should solve the 'I don't care what language the library Foo is > written in, and my library Bar isn't written in Wisp so it seems > unreasonable to have to add -x w.’, and would also avoid the need for > a 'extension -> language map' thing. It should also be a little > faster than what we had before. That would also enable shipping pre-compiled software without sourcecode, so there may be strategic reasons to avoid it. Always providing the sourcecode also makes compliance with automatic copyleft licenses automatic. >>>> readable uses >>> >>> This sentence appears to be incomplete; I might have misinterpreted it >>> below (I don't know what you mean with 'readable' -- its an adjective >>> and you are using it as a noun?). >> readable is a noun, yes: the readable lisp project. > > Looks like you meant this: <https://readable.sourceforge.io/>. >>>> Can I write it into a string and then read it back? >>> >>> No. [...] >> This sounds like I cannot go that way, because there’s a necessary >> pre-processing step in wisp-read via (match-charlist-to-repr peeked): >> [...] > This actually needs to be able to write the replacement symbols back >> into the port. >> >>> ..., for which I proposed a replacement, so do you still need to turn >>> it in a string & back? >> Sadly yes. Otherwise the normal reader will play tricks on the code, >> because it does not know where a symbol needs to be interpreted >> differently (i.e. where ` needs to be treated as `() even though that’s >> not in the string). > > OK, too bad. Looks like the REPR-... stuff is to stay for now. OK — thank you for your try to find a better way! > (cond ((list? code) (map wisp-... code)) > ((eq? code '\:) ':) > ;; Look for symbols like \____ and remove the \. > ((symbol? code) > (let ((as-string (symbol->string code))) > (if (and (>= (string-length as-string) 2) ; at least a single > underscore > (char=? (string-ref as-string 0) #\\) > (string-every #\_ (substring as-string 1))) > (string->symbol (substring as-string 1)) > code))) > (#true code)) This looks great — thank you! I’m attaching my changes to the original patch and a new squashed patch. [-- Attachment #1.2: 0002-Stylistic-changes-to-language-wisp.patch --] [-- Type: text/x-patch, Size: 16537 bytes --] From 0755a9bcb5f359c39cc5e91ff33e03e8b897b93d Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 14 Feb 2023 09:41:24 +0100 Subject: [PATCH 2/3] Stylistic changes to language/wisp * module/language/wisp.scm: fix whitespace and remove commented-out code * module/language/wisp/spec.scm: remove unused variable Acked-by: Arne Babenhauserheide <arne_bab@web.de> --- module/language/wisp.scm | 101 ++++++++++++++-------------------- module/language/wisp/spec.scm | 2 - 2 files changed, 42 insertions(+), 61 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index ba24f54c5..bec91e274 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -40,8 +40,8 @@ (use-modules (srfi srfi-1) - (srfi srfi-11 ); for let-values - (ice-9 rw ); for write-string/partial + (srfi srfi-11); for let-values + (ice-9 rw); for write-string/partial (ice-9 match)) @@ -106,15 +106,15 @@ repr-unquote) ((equal? chlist (list #\`)) repr-quasiquote) - ((equal? chlist (list #\, #\@ )) + ((equal? chlist (list #\, #\@)) repr-unquote-splicing) - ((equal? chlist (list #\# #\' )) + ((equal? chlist (list #\# #\')) repr-syntax) - ((equal? chlist (list #\# #\, )) + ((equal? chlist (list #\# #\,)) repr-unsyntax) - ((equal? chlist (list #\# #\` )) + ((equal? chlist (list #\# #\`)) repr-quasisyntax) - ((equal? chlist (list #\# #\, #\@ )) + ((equal? chlist (list #\# #\, #\@)) repr-unsyntax-splicing) (else #f)))) @@ -124,33 +124,23 @@ (let ((prefix-maxlen 4)) (let longpeek ((peeked '()) - (repr-symbol #f)) + (repr-symbol #f)) (cond - ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port)) ) + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) (if repr-symbol ; found a special symbol, return it. - ; TODO: Somehow store source-properties. The commented-out code below does not work. - ; catch #t - ; lambda () - ; write : source-properties symbol-or-symbols - ; set-source-property! symbol-or-symbols 'filename : port-filename port - ; set-source-property! symbol-or-symbols 'line : 1+ : port-line port - ; set-source-property! symbol-or-symbols 'column : port-column port - ; write : source-properties symbol-or-symbols - ; lambda : key . arguments - ; . #f repr-symbol (let unpeek ((remaining peeked)) (cond - ((equal? '() remaining ) - (read port )); let read to the work + ((equal? '() remaining) + (read port)); let read to the work (else (unread-char (car remaining) port) (unpeek (cdr remaining))))))) (else (let* ((next-char (read-char port)) - (peeked (cons next-char peeked))) + (peeked (cons next-char peeked))) (longpeek peeked (match-charlist-to-repr peeked)))))))) @@ -174,7 +164,7 @@ (= 0 (line-indent line)) (line-empty-code? line))) -(define (line-strip-continuation line ) +(define (line-strip-continuation line) (if (line-continues? line) (append (list @@ -204,13 +194,13 @@ (define (indent-level-difference indentation-levels level) "Find how many indentation levels need to be popped off to find the given level." (indent-level-reduction indentation-levels level - (lambda (x ); get the count + (lambda (x); get the count (car x)))) (define (indent-reduce-to-level indentation-levels level) "Find how many indentation levels need to be popped off to find the given level." (indent-level-reduction indentation-levels level - (lambda (x ); get the levels + (lambda (x); get the levels (car (cdr x))))) (define (chunk-ends-with-period currentsymbols next-char) @@ -222,7 +212,7 @@ (define (wisp-scheme-read-chunk-lines port) (let loop - ((indent-and-symbols (list )); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) (inindent #t) (inunderscoreindent (equal? #\_ (peek-char port))) (incomment #f) @@ -230,7 +220,7 @@ (currentsymbols '()) (emptylines 0)) (cond - ((>= emptylines 2 ); the chunk end has to be checked + ((>= emptylines 2); the chunk end has to be checked ; before we look for new chars in the ; port to make execution in the REPL ; after two empty lines work @@ -242,14 +232,14 @@ ((eof-object? next-char) (append indent-and-symbols (list (append (list currentindent) currentsymbols)))) ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) - (append indent-and-symbols )); top-level form ends chunk + (append indent-and-symbols)); top-level form ends chunk ((chunk-ends-with-period currentsymbols next-char) ; the line ends with a period. This is forbidden in ; SRFI-119. Use it to end the line in the REPL without ; showing continuation dots (...). (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1))))) ((and inindent (equal? #\space next-char)) - (read-char port ); remove char + (read-char port); remove char (loop indent-and-symbols #t ; inindent @@ -259,7 +249,7 @@ currentsymbols emptylines)) ((and inunderscoreindent (equal? #\_ next-char)) - (read-char port ); remove char + (read-char port); remove char (loop indent-and-symbols #t ; inindent @@ -276,7 +266,7 @@ ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) ((equal? #\newline next-char) - (read-char port ); remove the newline + (read-char port); remove the newline ; The following two lines would break the REPL by requiring one char too many. ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port ; read-char port ; remove a full \n\r. Damn special cases... @@ -285,14 +275,13 @@ ; indent. Lines with a comment at zero indent ; get indent -1 for the same reason - meaning ; not actually empty. - ( - (indent + ((indent (cond (incomment - (if (= 0 currentindent ); specialcase + (if (= 0 currentindent); specialcase -1 - currentindent )) - ((not (null? currentsymbols )); pure whitespace + currentindent)) + ((not (null? currentsymbols)); pure whitespace currentindent) (else 0))) @@ -313,13 +302,13 @@ #t ; inindent (if (<= 2 emptylines) #f ; chunk ends here - (equal? #\_ (peek-char port ))); are we in underscore indent? + (equal? #\_ (peek-char port))); are we in underscore indent? #f ; incomment 0 '() emptylines))) ((equal? #t incomment) - (read-char port ); remove one comment character + (read-char port); remove one comment character (loop indent-and-symbols #f ; inindent @@ -328,8 +317,8 @@ currentindent currentsymbols emptylines)) - ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char) ); remove whitespace when not in indent - (read-char port ); remove char + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent + (read-char port); remove char (loop indent-and-symbols #f ; inindent @@ -406,10 +395,7 @@ (define (line-finalize line) "Process all wisp-specific information in a line and strip it" - (let - ( - (l - (line-code-replace-inline-colons + (let ((l (line-code-replace-inline-colons (line-strip-indentation-marker (line-strip-lone-colon (line-strip-continuation line)))))) @@ -459,7 +445,7 @@ (and (not (null? lines)) (not (line-empty-code? (car lines))) - (not (= 0 (line-real-indent (car lines ))))); -1 is a line with a comment + (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment (if (= 1 (line-real-indent (car lines))) ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands (set! lines @@ -474,15 +460,14 @@ (unprocessed lines) (indentation-levels '(0))) (let* - ( - (current-line + ((current-line (if (<= 1 (length unprocessed)) (car unprocessed) - (list 0 ))); empty code + (list 0))); empty code (next-line (if (<= 2 (length unprocessed)) (car (cdr unprocessed)) - (list 0 ))); empty code + (list 0))); empty code (current-indentation (car indentation-levels)) (current-line-indentation (line-real-indent current-line))) @@ -517,7 +502,7 @@ processed (cdr unprocessed) indentation-levels)) - ((and (line-empty-code? next-line) (<= 2 (length unprocessed ))) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) ; display "next-line empty\n" ; TODO: Somehow preserve the line-numbers. ; take out the next-line from unprocessed. @@ -555,18 +540,17 @@ (if (line-continues? current-line) line (wisp-add-source-properties-from line (list line)))) - (cdr unprocessed ); recursion here + (cdr unprocessed); recursion here indentation-levels)) ((< current-line-indentation next-line-indentation) ; display "current-line-indent < next-line-indent\n" ; format #t "line: ~A\n" line ; side-recursion via a sublist (let-values - ( - ((sub-processed sub-unprocessed) + (((sub-processed sub-unprocessed) (loop line - (cdr unprocessed ); recursion here + (cdr unprocessed); recursion here indentation-levels))) ; format #t "side-recursion:\n sub-processed: ~A\n processed: ~A\n\n" sub-processed processed (loop @@ -660,7 +644,7 @@ (match code (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) (list 'quote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list (append (map wisp-replace-paren-quotation-repr a) (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) @@ -668,13 +652,13 @@ (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) (list 'unquote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) (append (map wisp-replace-paren-quotation-repr a) (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list (append (map wisp-replace-paren-quotation-repr a) (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) @@ -713,8 +697,7 @@ code to recreate the improper lists. Match is awesome!" (let - ( - (improper + ((improper (match code ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) (append (map wisp-make-improper a) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index d5ea7abce..155292d9a 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -63,8 +63,6 @@ ;;; Language definition ;;; -(define wisp-pending-sexps (list)) - (define (read-one-wisp-sexp port env) ;; allow using "# foo" as #(foo). (read-hash-extend #\# (λ (chr port) #\#)) -- 2.39.1 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.3: 0003-Remove-limit-of-_-prefix-in-language-wisp-thanks-to-.patch --] [-- Type: text/x-patch, Size: 2679 bytes --] From d8e688c567e70831d2d916154eb5fa245651a4ca Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 14 Feb 2023 09:49:38 +0100 Subject: [PATCH 3/3] =?UTF-8?q?Remove=20limit=20of=20=5F-prefix=20in=20lan?= =?UTF-8?q?guage/wisp=20=E2=80=94=20thanks=20to=20Maxime=20Devos!?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * module/language/wisp.scm: replace match by general cond Acked-by: Arne Babenhauserheide <arne_bab@web.de> --- module/language/wisp.scm | 45 +++++++++++----------------------------- 1 file changed, 12 insertions(+), 33 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index bec91e274..1ce3ced1c 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -24,7 +24,6 @@ ;; preprocessed file. ;; Limitations: -;; - only unescapes up to 12 leading underscores at line start (\____________) ;; - in some cases the source line information is missing in backtraces. ;; check for set-source-property! @@ -595,38 +594,18 @@ (cdr unprocessed))))) (define (wisp-unescape-underscore-and-colon code) - "replace \\_ and \\: by _ and :" - (match code - ((a ...) - (map wisp-unescape-underscore-and-colon a)) - ('\_ - '_) - ('\__ - '__) - ('\___ - '___) - ('\____ - '____) - ('\_____ - '_____) - ('\______ - '______) - ('\_______ - '_______) - ('\________ - '________) - ('\_________ - '_________) - ('\__________ - '__________) - ('\___________ - '___________) - ('\____________ - '____________) - ('\: - ':) - (a - a))) + "replace \\_ and \\: by _ and :" + (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code))) (define (wisp-replace-empty-eof code) -- 2.39.1 [-- Attachment #1.4: 0001-Add-language-wisp-wisp-tests-and-srfi-119-documentat-squashed.patch --] [-- Type: text/x-patch, Size: 46919 bytes --] From bb38072818ff754ba67533380982e993ffc0ff52 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 3 Feb 2023 22:20:04 +0100 Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation * doc/ref/srfi-modules.texi (srfi-119): add node * module/language/wisp.scm: New file. * module/language/wisp/spec.scm: New file. * test-suite/tests/srfi-119.test: New file. --- doc/ref/srfi-modules.texi | 30 ++ module/language/wisp.scm | 758 +++++++++++++++++++++++++++++++++ module/language/wisp/spec.scm | 105 +++++ test-suite/tests/srfi-119.test | 81 ++++ 4 files changed, 974 insertions(+) create mode 100644 module/language/wisp.scm create mode 100644 module/language/wisp/spec.scm create mode 100644 test-suite/tests/srfi-119.test diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 0ef136215..759e293ec 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. +* SRFI-119:: Wisp: simpler indentation-sensitive scheme. * SRFI-171:: Transducers @end menu @@ -5662,6 +5663,34 @@ Return the current contents of @var{box}. Set the contents of @var{box} to @var{value}. @end deffn +@node SRFI-119 +@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme. +@cindex SRFI-119 +@cindex wisp + +The languages shipped in Guile include SRFI-119 (wisp), an encoding of +Scheme that allows replacing parentheses with equivalent indentation and +inline colons. See +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification +of SRFI-119}. Some examples: + +@example +display "Hello World!" @result{} (display "Hello World!") +@end example + +@example +define : factorial n @result{} (define (factorial n) + if : zero? n @result{} (if (zero? n) + . 1 @result{} 1 + * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) +@end example + +To execute a file with wisp code, select the language and filename +extension @code{.w} vie @code{guile --language=wisp -x .w}. + +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. + + @node SRFI-171 @subsection Transducers @cindex SRFI-171 @@ -5705,6 +5734,7 @@ left-to-right, due to how transducers are initiated. * SRFI-171 Helpers:: Utilities for writing your own transducers @end menu + @node SRFI-171 General Discussion @subsubsection SRFI-171 General Discussion @cindex transducers discussion diff --git a/module/language/wisp.scm b/module/language/wisp.scm new file mode 100644 index 000000000..1ce3ced1c --- /dev/null +++ b/module/language/wisp.scm @@ -0,0 +1,758 @@ +;;; Wisp + +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. +;; Copyright (C) 2014--2023 Arne Babenhauserheide. + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;;; Commentary: + +;; Scheme-only implementation of a wisp-preprocessor which output a +;; scheme code tree to feed to a scheme interpreter instead of a +;; preprocessed file. + +;; Limitations: +;; - in some cases the source line information is missing in backtraces. +;; check for set-source-property! + +;;; Code: + +(define-module (language wisp) + #:export (wisp-scheme-read-chunk wisp-scheme-read-all + wisp-scheme-read-file-chunk wisp-scheme-read-file + wisp-scheme-read-string)) + +; use curly-infix by default +(read-enable 'curly-infix) + +(use-modules + (srfi srfi-1) + (srfi srfi-11); for let-values + (ice-9 rw); for write-string/partial + (ice-9 match)) + + +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define (line-indent line) + (car line)) + +(define (line-real-indent line) + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." + (let (( indent (line-indent line))) + (if (= -1 indent) + 0 + indent))) + +(define (line-code line) + (let ((code (cdr line))) + ; propagate source properties + (when (not (null? code)) + (set-source-properties! code (source-properties line))) + code)) + +; literal values I need +(define readcolon + (string->symbol ":")) + +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") +; define an intermediate dot replacement with UUID to avoid clashes. +(define repr-dot ; . + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) + +; allow using reader additions as the first element on a line to prefix the list +(define repr-quote ; ' + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) +(define repr-unquote ; , + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) +(define repr-quasiquote ; ` + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) +(define repr-unquote-splicing ; ,@ + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) + +(define repr-syntax ; #' + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) +(define repr-unsyntax ; #, + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) +(define repr-quasisyntax ; #` + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) +(define repr-unsyntax-splicing ; #,@ + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) + +; TODO: wrap the reader to return the repr of the syntax reader +; additions + +(define (match-charlist-to-repr charlist) + (let + ((chlist (reverse charlist))) + (cond + ((equal? chlist (list #\.)) + repr-dot) + ((equal? chlist (list #\')) + repr-quote) + ((equal? chlist (list #\,)) + repr-unquote) + ((equal? chlist (list #\`)) + repr-quasiquote) + ((equal? chlist (list #\, #\@)) + repr-unquote-splicing) + ((equal? chlist (list #\# #\')) + repr-syntax) + ((equal? chlist (list #\# #\,)) + repr-unsyntax) + ((equal? chlist (list #\# #\`)) + repr-quasisyntax) + ((equal? chlist (list #\# #\, #\@)) + repr-unsyntax-splicing) + (else + #f)))) + +(define (wisp-read port) + "wrap read to catch list prefixes." + (let ((prefix-maxlen 4)) + (let longpeek + ((peeked '()) + (repr-symbol #f)) + (cond + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) + (if repr-symbol ; found a special symbol, return it. + repr-symbol + (let unpeek + ((remaining peeked)) + (cond + ((equal? '() remaining) + (read port)); let read to the work + (else + (unread-char (car remaining) port) + (unpeek (cdr remaining))))))) + (else + (let* + ((next-char (read-char port)) + (peeked (cons next-char peeked))) + (longpeek + peeked + (match-charlist-to-repr peeked)))))))) + + + +(define (line-continues? line) + (equal? repr-dot (car (line-code line)))) + +(define (line-only-colon? line) + (and + (equal? ":" (car (line-code line))) + (null? (cdr (line-code line))))) + +(define (line-empty-code? line) + (null? (line-code line))) + +(define (line-empty? line) + (and + ; if indent is -1, we stripped a comment, so the line was not really empty. + (= 0 (line-indent line)) + (line-empty-code? line))) + +(define (line-strip-continuation line) + (if (line-continues? line) + (append + (list + (line-indent line)) + (cdr (line-code line))) + line)) + +(define (line-strip-indentation-marker line) + "Strip the indentation markers from the beginning of the line" + (cdr line)) + +(define (indent-level-reduction indentation-levels level select-fun) + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" + (let loop + ((newlevels indentation-levels) + (diff 0)) + (cond + ((= level (car newlevels)) + (select-fun (list diff indentation-levels))) + ((< level (car newlevels)) + (loop + (cdr newlevels) + (1+ diff))) + (else + (throw 'wisp-syntax-error "Level ~A not found in the indentation-levels ~A."))))) + +(define (indent-level-difference indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the count + (car x)))) + +(define (indent-reduce-to-level indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the levels + (car (cdr x))))) + +(define (chunk-ends-with-period currentsymbols next-char) + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." + (and (not (null? currentsymbols)) + (equal? #\newline next-char) + (equal? repr-dot + (list-ref currentsymbols (- (length currentsymbols) 1))))) + +(define (wisp-scheme-read-chunk-lines port) + (let loop + ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (inindent #t) + (inunderscoreindent (equal? #\_ (peek-char port))) + (incomment #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) + (cond + ((>= emptylines 2); the chunk end has to be checked + ; before we look for new chars in the + ; port to make execution in the REPL + ; after two empty lines work + ; (otherwise it shows one more line). + indent-and-symbols) + (else + (let ((next-char (peek-char port))) + (cond + ((eof-object? next-char) + (append indent-and-symbols (list (append (list currentindent) currentsymbols)))) + ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) + (append indent-and-symbols)); top-level form ends chunk + ((chunk-ends-with-period currentsymbols next-char) + ; the line ends with a period. This is forbidden in + ; SRFI-119. Use it to end the line in the REPL without + ; showing continuation dots (...). + (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1))))) + ((and inindent (equal? #\space next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; inindent + #f ; inunderscoreindent + #f ; incomment + (1+ currentindent) + currentsymbols + emptylines)) + ((and inunderscoreindent (equal? #\_ next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; inindent + #t ; inunderscoreindent + #f ; incomment + (1+ currentindent) + currentsymbols + emptylines)) + ; any char but whitespace *after* underscoreindent is + ; an error. This is stricter than the current wisp + ; syntax definition. TODO: Fix the definition. Better + ; start too strict. FIXME: breaks on lines with only + ; underscores which should be empty lines. + ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) + ((equal? #\newline next-char) + (read-char port); remove the newline + ; The following two lines would break the REPL by requiring one char too many. + ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port + ; read-char port ; remove a full \n\r. Damn special cases... + (let* ; distinguish pure whitespace lines and lines + ; with comment by giving the former zero + ; indent. Lines with a comment at zero indent + ; get indent -1 for the same reason - meaning + ; not actually empty. + ((indent + (cond + (incomment + (if (= 0 currentindent); specialcase + -1 + currentindent)) + ((not (null? currentsymbols)); pure whitespace + currentindent) + (else + 0))) + (parsedline (append (list indent) currentsymbols)) + (emptylines + (if (not (line-empty? parsedline)) + 0 + (1+ emptylines)))) + (when (not (= 0 (length parsedline))) + ; set the source properties to parsedline so we can try to add them later. + (set-source-property! parsedline 'filename (port-filename port)) + (set-source-property! parsedline 'line (port-line port))) + ; TODO: If the line is empty. Either do it here and do not add it, just + ; increment the empty line counter, or strip it later. Replace indent + ; -1 by indent 0 afterwards. + (loop + (append indent-and-symbols (list parsedline)) + #t ; inindent + (if (<= 2 emptylines) + #f ; chunk ends here + (equal? #\_ (peek-char port))); are we in underscore indent? + #f ; incomment + 0 + '() + emptylines))) + ((equal? #t incomment) + (read-char port); remove one comment character + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #t ; incomment + currentindent + currentsymbols + emptylines)) + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent + (read-char port); remove char + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #f ; incomment + currentindent + currentsymbols + emptylines)) + ; | cludge to appease the former wisp parser + ; | used for bootstrapping which has a + ; v problem with the literal comment char + ((equal? (string-ref ";" 0) next-char) + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #t ; incomment + currentindent + currentsymbols + emptylines)) + (else ; use the reader + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #f ; incomment + currentindent + ; this also takes care of the hashbang and leading comments. + (append currentsymbols (list (wisp-read port))) + emptylines)))))))) + + +(define (line-code-replace-inline-colons line) + "Replace inline colons by opening parens which close at the end of the line" + ; format #t "replace inline colons for line ~A\n" line + (let loop + ((processed '()) + (unprocessed line)) + (cond + ((null? unprocessed) + ; format #t "inline-colons processed line: ~A\n" processed + processed) + ; replace : . with nothing + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) + (loop + (append processed + (loop '() (cdr (cdr unprocessed)))) + '())) + ((equal? readcolon (car unprocessed)) + (loop + ; FIXME: This should turn unprocessed into a list. + (append processed + (list (loop '() (cdr unprocessed)))) + '())) + (else + (loop + (append processed + (list (car unprocessed))) + (cdr unprocessed)))))) + +(define (line-replace-inline-colons line) + (cons + (line-indent line) + (line-code-replace-inline-colons (line-code line)))) + +(define (line-strip-lone-colon line) + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." + (if + (equal? + (line-code line) + (list readcolon)) + (list (line-indent line)) + line)) + +(define (line-finalize line) + "Process all wisp-specific information in a line and strip it" + (let ((l (line-code-replace-inline-colons + (line-strip-indentation-marker + (line-strip-lone-colon + (line-strip-continuation line)))))) + (when (not (null? (source-properties line))) + (catch #t + (lambda () + (set-source-properties! l (source-properties line))) + (lambda (key . arguments) + #f))) + l)) + +(define (wisp-add-source-properties-from source target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) + +(define (wisp-propagate-source-properties code) + "Propagate the source properties from the sourrounding list into every part of the code." + (let loop + ((processed '()) + (unprocessed code)) + (cond + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) + unprocessed) + ((and (pair? unprocessed) (not (list? unprocessed))) + (cons + (wisp-propagate-source-properties (car unprocessed)) + (wisp-propagate-source-properties (cdr unprocessed)))) + ((null? unprocessed) + processed) + (else + (let ((line (car unprocessed))) + (if (null? (source-properties unprocessed)) + (wisp-add-source-properties-from line unprocessed) + (wisp-add-source-properties-from unprocessed line)) + (loop + (append processed (list (wisp-propagate-source-properties line))) + (cdr unprocessed))))))) + +(define* (wisp-scheme-indentation-to-parens lines) + "Add parentheses to lines and remove the indentation markers" + (when + (and + (not (null? lines)) + (not (line-empty-code? (car lines))) + (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment + (if (= 1 (line-real-indent (car lines))) + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands + (set! lines + (cons + (cons 0 (cdr (car lines))) + (cdr lines))) + (throw 'wisp-syntax-error + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" + (car lines))))) + (let loop + ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* + ((current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (list 0))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (list 0))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) + ; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ; . processed current-line next-line unprocessed indentation-levels current-indentation + (cond + ; the real end: this is reported to the outside world. + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) + ; display "done\n" + ; reverse the processed lines, because I use cons. + processed) + ; the recursion end-condition + ((and (null? unprocessed)) + ; display "last step\n" + ; this is the last step. Nothing more to do except + ; for rolling up the indentation levels. return the + ; new processed and unprocessed lists: this is a + ; side-recursion + (values processed unprocessed)) + ((null? indentation-levels) + ; display "indentation-levels null\n" + (throw 'wisp-programming-error "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")) + (else ; now we come to the line-comparisons and indentation-counting. + (cond + ((line-empty-code? current-line) + ; display "current-line empty\n" + ; We cannot process indentation without + ; code. Just switch to the next line. This should + ; only happen at the start of the recursion. + ; TODO: Somehow preserve the line-numbers. + (loop + processed + (cdr unprocessed) + indentation-levels)) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) + ; display "next-line empty\n" + ; TODO: Somehow preserve the line-numbers. + ; take out the next-line from unprocessed. + (loop + processed + (cons current-line + (cdr (cdr unprocessed))) + indentation-levels)) + ((> current-indentation current-line-indentation) + ; display "current-indent > next-line\n" + ; this just steps back one level via the side-recursion. + (let ((previous-indentation (car (cdr indentation-levels)))) + (if (<= current-line-indentation previous-indentation) + (values processed unprocessed) + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. + (let ((linenumber (- (length lines) (length unprocessed)))) + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + (cdr indentation-levels))))))) + ((= current-indentation current-line-indentation) + ; display "current-indent = next-line\n" + (let + ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) + (cond + ((>= current-line-indentation next-line-indentation) + ; simple recursiive step to the next line + ; display "current-line-indent >= next-line-indent\n" + (loop + (append processed + (if (line-continues? current-line) + line + (wisp-add-source-properties-from line (list line)))) + (cdr unprocessed); recursion here + indentation-levels)) + ((< current-line-indentation next-line-indentation) + ; display "current-line-indent < next-line-indent\n" + ; format #t "line: ~A\n" line + ; side-recursion via a sublist + (let-values + (((sub-processed sub-unprocessed) + (loop + line + (cdr unprocessed); recursion here + indentation-levels))) + ; format #t "side-recursion:\n sub-processed: ~A\n processed: ~A\n\n" sub-processed processed + (loop + (append processed (list sub-processed)) + sub-unprocessed ; simply use the recursion from the sub-recursion + indentation-levels)))))) + ((< current-indentation current-line-indentation) + ; display "current-indent < next-line\n" + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + indentation-levels))) + (else + (throw 'wisp-not-implemented + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))) + + +(define (wisp-scheme-replace-inline-colons lines) + "Replace inline colons by opening parens which close at the end of the line" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (list (line-replace-inline-colons (car unprocessed)))) + (cdr unprocessed))))) + + +(define (wisp-scheme-strip-indentation-markers lines) + "Strip the indentation markers from the beginning of the lines" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (cdr (car unprocessed))) + (cdr unprocessed))))) + +(define (wisp-unescape-underscore-and-colon code) + "replace \\_ and \\: by _ and :" + (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code))) + + +(define (wisp-replace-empty-eof code) + "replace ((#<eof>)) by ()" + ; FIXME: Actually this is a hack which fixes a bug when the + ; parser hits files with only hashbang and comments. + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) + (list) + code)) + + +(define (wisp-replace-paren-quotation-repr code) + "Replace lists starting with a quotation symbol by + quoted lists." + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a))) + +(define (wisp-make-improper code) + "Turn (a #{.}# b) into the correct (a . b). + +read called on a single dot creates a variable named #{.}# (|.| +in r7rs). Due to parsing the indentation before the list +structure is known, the reader cannot create improper lists +when it reads a dot. So we have to take another pass over the +code to recreate the improper lists. + +Match is awesome!" + (let + ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (append (map wisp-make-improper a) + (cons (wisp-make-improper b) (wisp-make-improper c)))) + ((a ...) + (map wisp-make-improper a)) + (a + a)))) + (define (syntax-error li msg) + (throw 'wisp-syntax-error (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))) + (if #t + improper + (let check + ((tocheck improper)) + (match tocheck + ; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ; more complex pairs + ((? pair? a) + (let + ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a + a)))))) + +(define (wisp-scheme-read-chunk port) + "Read and parse one chunk of wisp-code" + (let (( lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines)))))))) + +(define (wisp-scheme-read-all port) + "Read all chunks from the given port" + (let loop + ((tokens '())) + (cond + ((eof-object? (peek-char port)) + tokens) + (else + (loop + (append tokens (wisp-scheme-read-chunk port))))))) + +(define (wisp-scheme-read-file path) + (call-with-input-file path wisp-scheme-read-all)) + +(define (wisp-scheme-read-file-chunk path) + (call-with-input-file path wisp-scheme-read-chunk)) + +(define (wisp-scheme-read-string str) + (call-with-input-string str wisp-scheme-read-all)) + +(define (wisp-scheme-read-string-chunk str) + (call-with-input-string str wisp-scheme-read-chunk)) + diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm new file mode 100644 index 000000000..155292d9a --- /dev/null +++ b/module/language/wisp/spec.scm @@ -0,0 +1,105 @@ +;; Language interface for Wisp in Guile + +;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm + +;;; Copyright (C) 2005-2014 by David A. Wheeler and Alan Manuel K. Gloria +;;; Copyright (C) Arne Babenhauserheide (2014--2023). + +;;; Permission is hereby granted, free of charge, to any person +;;; obtaining a copy of this software and associated documentation +;;; files (the "Software"), to deal in the Software without +;;; restriction, including without limitation the rights to use, copy, +;;; modify, merge, publish, distribute, sublicense, and/or sell copies +;;; of the Software, and to permit persons to whom the Software is +;;; furnished to do so, subject to the following conditions: +;;; +;;; The above copyright notice and this permission notice shall be +;;; included in all copies or substantial portions of the Software. +;;; +;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +;;; SOFTWARE. + +; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm +(define-module (language wisp spec) + #:use-module (language wisp) + #:use-module (system base compile) + #:use-module (system base language) + #:use-module (language scheme compile-tree-il) + #:use-module (language scheme decompile-tree-il) + #:export (wisp)) + +; Set locale to something which supports unicode. Required to avoid using fluids. +(catch #t + (lambda () + (setlocale LC_ALL "")) + (lambda (key . parameters) + (let ((locale-fallback "en_US.UTF-8")) + (format (current-error-port) + (string-join + (list ";;; Warning: setlocale LC_ALL \"\" failed with ~A: ~A" + "switching to explicit ~A locale. Please setup your locale." + "If this fails, you might need glibc support for unicode locales.\n") + "\n;;; ") + key parameters locale-fallback) + (catch #t + (lambda () + (setlocale LC_ALL locale-fallback)) + (lambda (key . parameters) + (format (current-error-port) + (string-join + (list ";;; Warning: fallback setlocale LC_ALL ~A failed with ~A: ~A" + "Not switching to Unicode." + "You might need glibc support for unicode locales.\n") + "\n;;; ") + locale-fallback key parameters)))))) + +;;; +;;; Language definition +;;; + +(define (read-one-wisp-sexp port env) + ;; allow using "# foo" as #(foo). + (read-hash-extend #\# (λ (chr port) #\#)) + (cond + ((eof-object? (peek-char port)) + (read-char port )); return eof: we’re done + (else + (let ((chunk (wisp-scheme-read-chunk port))) + (cond + ((not (null? chunk)) + (car chunk)) + (else + #f)))))) + +(define-language wisp + #:title "Wisp Scheme Syntax. See SRFI-119 for details." + ; . #:reader read-one-wisp-sexp + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; + #:compilers `((tree-il . ,compile-tree-il)) + #:decompilers `((tree-il . ,decompile-tree-il)) + #:evaluator (lambda (x module) (primitive-eval x)) + #:printer write ; TODO: backtransform to wisp? Use source-properties? + #:make-default-environment + (lambda () + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', + ;; `fluid-set!', etc. don't have any effect in the current environment. + (let ((m (make-fresh-user-module))) + ;; Provide a separate `current-reader' fluid so that + ;; compile-time changes to `current-reader' are + ;; limited to the current compilation unit. + (module-define! m 'current-reader (make-fluid)) + ;; Default to `simple-format', as is the case until + ;; (ice-9 format) is loaded. This allows + ;; compile-time warnings to be emitted when using + ;; unsupported options. + (module-set! m 'format simple-format) + m))) + + + diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test new file mode 100644 index 000000000..a888df41d --- /dev/null +++ b/test-suite/tests/srfi-119.test @@ -0,0 +1,81 @@ +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*- +;;;; +;;;; Copyright (C) 2023 Free Software Foundation, Inc. +;;;; +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +(define-module (test-srfi-119) + #:use-module (test-suite lib) + #:use-module (srfi srfi-1) + #:use-module (language wisp)) + +(define (read-string s) + (with-input-from-string s read)) + +(define (with-read-options opts thunk) + (let ((saved-options (read-options))) + (dynamic-wind + (lambda () + (read-options opts)) + thunk + (lambda () + (read-options saved-options))))) + +(define (wisp->list str) + (wisp-scheme-read-string str)) + +(with-test-prefix "wisp-read-simple" + (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) + (pass-if (equal? (wisp->list ". 5") '(5))) + (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) +(with-test-prefix "wisp-read-complex" + (pass-if (equal? (wisp->list " +a b c d e + . f g h + . i j k + +concat \"I want \" + getwish from me + . \" - \" username +") '( +(a b c d e + f g h + i j k) + +(concat "I want " + (getwish from me) + " - " username)))) + + (pass-if (equal? (wisp->list " +define : a b c +_ d e +___ f +___ g h +__ . i + +define : _ +_ display \"hello\n\" + +\\_") '( +(define (a b c) + (d e + (f) + (g h) + i)) + +(define (_) + (display "hello\n")) + +(_))))) -- 2.39.1 [-- Attachment #1.5: Type: text/plain, Size: 103 bytes --] Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-14 8:32 ` Dr. Arne Babenhauserheide @ 2023-02-14 21:24 ` Dr. Arne Babenhauserheide 2023-02-14 23:01 ` Maxime Devos 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-14 21:24 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 214 bytes --] PS: So what’s still missing here is to avoid setting the locale. Do you happen to have a hint how to actually do this right? -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-14 21:24 ` Dr. Arne Babenhauserheide @ 2023-02-14 23:01 ` Maxime Devos 2023-02-15 1:46 ` Matt Wette 2023-02-15 8:36 ` Dr. Arne Babenhauserheide 0 siblings, 2 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-14 23:01 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 4096 bytes --] > [...] > That would be nice, but would require doing changes in a critical core > part of Guile. It would change this addition from a risk-free added > feature to a risky core change. I maintain that a new language shouldn't be merged until the Scheme-specific load path stuff is fixed/extended to work for non-Scheme things (e.g. Wisp) too -- if this requires somewhat risky (*) changes to core parts, then that just means we'll have to do some risky stuff, then. I also expect that Guile maintainers will have the opposite opinion (i.e., ‘fixing the load path stuff isn't necessary for merging a new language implementation’). (*) FWIW I disagree on the 'risky' assessment -- it seems like a ‘if it runs, it will work’ thing to me. That it modifies a core part of Guile, makes it less risky IMO, as it would automatically be more tested. Aside from the (*) and the 'I also expect [...],', I don't have anything new to say about this, so I'll stop here. > [...] > That would also enable shipping pre-compiled software without > sourcecode, That can already be done -- besides legalities, nothing stops people from putting [^] or [^] .scm files in $GUILE_LOAD_PATH and putting .go in $GUILE_LOAD_COMPILED_PATH. [^]: Redacted to not give people ideas on how to circumvent stuff. I can elaborate by non-public e-mail if you like. > so there may be strategic reasons to avoid it. Always > providing the sourcecode also makes compliance with automatic copyleft > licenses automatic. Mm, yes, I guess. If only people weren't careless and didn't try to circumvent copyleft, then things would be easier ... On 14-02-2023 22:24, Dr. Arne Babenhauserheide wrote: > PS: So what’s still missing here is to avoid setting the locale. Do you > happen to have a hint how to actually do this right? I think you might have forgotten about this: > -- Scheme Procedure: set-port-encoding! port enc > -- C Function: scm_set_port_encoding_x (port, enc) > Sets the character encoding that will be used to interpret I/O to > PORT. ENC is a string containing the name of an encoding. Valid > encoding names are those defined by IANA > (http://www.iana.org/assignments/character-sets), for example > ‘"UTF-8"’ or ‘"ISO-8859-1"’. > > As such, I propose calling set-port-encoding! right in the beginning of read-one-wisp-sexp. More concretely, replace (define (read-one-wisp-sexp port env) ;; allow using "# foo" as #(foo). (read-hash-extend #\# (λ (chr port) #\#)) (cond ((eof-object? (peek-char port)) (read-char port )); return eof: we’re done (else (let ((chunk (wisp-scheme-read-chunk port))) (cond ((not (null? chunk)) (car chunk)) (else #f)))))) by (define (read-one-wisp-sexp port env) ;; Allow using "# foo" as #(foo). ;; Don't use the globally-acting read-hash-extend, because this ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme. ;; Instead, use fluids to temporarily add the extension. (define %read-hash-procedures/parameter (fluid->parameter %read-hash-procedures)) (parameterize ((%read-hash-procedures/parameter `((#\# ,(λ (chr port) #\#)) ,@(%read-hash-procedures/parameter)))) ;; Read Wisp files as UTF-8, to support non-ASCII characters. ;; TODO: would be nice to support ';; coding: whatever' lines ;; like in parenthese-y Scheme. (set-port-encoding! port "UTF-8") (if (eof-object? (peek-char port)) (read-char port) ; return eof: we’re done (let ((chunk (wisp-scheme-read-chunk port))) (and (not (null? chunk)) ; <---- XXX: maybe (pair? chunk) (car chunk)))))) (untested). (I've also done the read-hash-extend stuff and simplified the 'cond' expressions.) Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-14 23:01 ` Maxime Devos @ 2023-02-15 1:46 ` Matt Wette 2023-02-16 21:38 ` Dr. Arne Babenhauserheide 2023-02-15 8:36 ` Dr. Arne Babenhauserheide 1 sibling, 1 reply; 83+ messages in thread From: Matt Wette @ 2023-02-15 1:46 UTC (permalink / raw) To: guile-devel You may be interested in the load-lang patch I generated a few years ago to allow file-extension based loading, in addition to '#land elisp" type hooks. https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch Matt ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-15 1:46 ` Matt Wette @ 2023-02-16 21:38 ` Dr. Arne Babenhauserheide 2023-02-17 1:26 ` Matt Wette 2023-02-17 23:06 ` Maxime Devos 0 siblings, 2 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-16 21:38 UTC (permalink / raw) To: Matt Wette; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 528 bytes --] Matt Wette <matt.wette@gmail.com> writes: > You may be interested in the load-lang patch I generated a few years ago > to allow file-extension based loading, in addition to '#lang elisp" > type hooks. > > https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch @Maxime: Is this something you’d be interested in championing? @Matt: Who needs to ack your patch for it to go into the repo? Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-16 21:38 ` Dr. Arne Babenhauserheide @ 2023-02-17 1:26 ` Matt Wette 2023-02-23 11:36 ` Ludovic Courtès 2023-02-17 23:06 ` Maxime Devos 1 sibling, 1 reply; 83+ messages in thread From: Matt Wette @ 2023-02-17 1:26 UTC (permalink / raw) To: guile-devel On 2/16/23 1:38 PM, Dr. Arne Babenhauserheide wrote: > Matt Wette <matt.wette@gmail.com> writes: > >> You may be interested in the load-lang patch I generated a few years ago >> to allow file-extension based loading, in addition to '#lang elisp" >> type hooks. >> >> https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch > @Maxime: Is this something you’d be interested in championing? > > @Matt: Who needs to ack your patch for it to go into the repo? > If by repo you mean main branch, I'm guessing this is a Ludo/Andy call. Maybe someone (you?) with write priv's could make a wip-branch for it? Matt ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-17 1:26 ` Matt Wette @ 2023-02-23 11:36 ` Ludovic Courtès 2023-02-23 17:48 ` Dr. Arne Babenhauserheide 2023-02-23 18:42 ` Maxime Devos 0 siblings, 2 replies; 83+ messages in thread From: Ludovic Courtès @ 2023-02-23 11:36 UTC (permalink / raw) To: guile-devel Hi! Sorry for the late reply. FWIW, I think it might be best to keep Wisp as a separate package: that allows it to evolve independently of Guile (and possibly more quickly :-)), and it might simplify maintenance in some way. Adding #lang support in Guile would be nice. As discussed on IRC, it can be experimented with in a WIP branch. We’ll then have to discuss when to incorporate it. My gut feeling is that it may have to wait until the next stable series (3.2.x), as this is quite a core change, but let’s see what people think. Thanks! Ludo’. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 11:36 ` Ludovic Courtès @ 2023-02-23 17:48 ` Dr. Arne Babenhauserheide 2023-02-23 18:42 ` Maxime Devos 1 sibling, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-23 17:48 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 4049 bytes --] Hi, Ludovic Courtès <ludo@gnu.org> writes: > FWIW, I think it might be best to keep Wisp as a separate package: that > allows it to evolve independently of Guile (and possibly more quickly > :-)), and it might simplify maintenance in some way. While this reasoning is true for most languages, I don’t think it applies to wisp: wisp will not evolve faster independently of Guile, because it is pretty much finished, and it should not need much maintenance (and if some should be needed, I can do that directly in Guile). It is such a thin layer that (different from other languages) there is no need to take care of changing syntax to follow or new semantics. It is defined such that its implementation can actually be finished. Every change in Scheme can automatically also be used with Wisp. While ecmascript, elisp, lokke, and python-on-guile must be kept up to date to stay useful, Wisp does not need that. The only larger changes in the past 4 years were in editor-support and some scripts, and these can stay separate, just like emacs-geiser-guile is separate from Guile. I propose adding Wisp now, because while I have been using Wisp for many projects in the past years, the changes to wisp itself since declaring 1.0 at FOSDEM 2019 were minimal, and the biggest were due to the review of Maxime here: wisp 1.0.10 (2023-02-16): - only extend the reader for ## while reading wisp. Thanks to Maxime Devos for help to use fluids! wisp 1.0.9 (2023-02-16): - remove limitation of the number of prefix underscores (_). Thanks to Maxime Devos for a much cleaner algorithm! - only set *port* encoding to UTF-8, do not change encoding for the application. Thanks to Maxime Devos! wisp 1.0.8 (2022-12-09): - wisp2lisp can now process stdin when called with - as filename. And it has help output. wisp 1.0.7 (2021-12-20): - fix: a lisp-style comment in the bash-cript had broken the wisp REPL wisp 1.0.6 (2021-11-30): - allow (and ignore!) a single space indentation for the first line of a chunk to support meta-commands - ensure that (language wisp) is compiled in the wisp REPL wisp 1.0.5 (2021-05-02): - explicitly allow using wisp as language under the expat-license for easier embedding in Guile-using games like Tsukundere Shitsumon: https://gitlab.com/leoprikler/tsukundere-shitsumon/ wisp 1.0.4 (2021-02-08): - add one more setlocale fallback: If it cannot use unicode, wisp now proceeds with degraded operation rather than failing outright. wisp 1.0.3 (2020-09-15): - provide wisp script that wraps guile --language=wisp -x .w - add Guile 3.0 to supported versions - fix documentation: wisp allows up to 12 underscores - You can create wisp-projects with conf via `conf new -l wisp PROJNAME`. See https://hg.sr.ht/~arnebab/conf - wisp moved to sourcehut: https://hg.sr.ht/~arnebab/wisp wisp 1.0.2 (2019-04-09): - guild compile is missing the load path wisp 1.0.1 (2019-03-23): - fix install directory, thanks to James-Adam Renquinha Henri and Ludovic Courtès who both discovered a mistake in the paths: correct module path is /usr/share/guile/site/<version>/…, but I used /usr/share/guile/<version>/site - simplify install logic and pre-compile installed modules. - add beautiful make help wisp 1.0 (2019-02-08): - add FOSDEM 2019 slides: docs/fosdem2019.org - As presented at FOSDEM, wisp the language is complete. Tooling, documentation, and porting are still work in progress. I plan to then only install language/wisp from the wisp-repo when the local Guile does not provide wisp. The reasoning for adding wisp to Guile is at the start of the thread: https://lists.gnu.org/archive/html/guile-devel/2023-02/msg00012.html > Adding #lang support in Guile would be nice. As discussed on IRC, it > can be experimented with in a WIP branch. I now created the branch wip-load-lang with the two patches by Matt. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 11:36 ` Ludovic Courtès 2023-02-23 17:48 ` Dr. Arne Babenhauserheide @ 2023-02-23 18:42 ` Maxime Devos 2023-02-24 15:45 ` Ludovic Courtès 1 sibling, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-23 18:42 UTC (permalink / raw) To: Ludovic Courtès, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 1776 bytes --] On 23-02-2023 12:36, Ludovic Courtès wrote: > Hi! > > Sorry for the late reply. > > FWIW, I think it might be best to keep Wisp as a separate package: that > allows it to evolve independently of Guile (and possibly more quickly > :-)), and it might simplify maintenance in some way. To my understanding, Wisp is pretty much finished -- it is standardised as surfie 119: <https://srfi.schemers.org/srfi-119/>, which is in final status. As such, there is no room for evolving (beside bugfixes, perhaps). The maintenance aspect (and also the evolving) is addressed in the cover letter: > - And it provides access to the full capabilities of Guile with minimal > maintenance effort, because it is just the thinnest possible layer > around Scheme. The last required change was in 2020 while I used it > continuously. There also were several other points in the cover letter for keeping Wisp as a _non-separate_ package. Why should Wisp be a separate package when other SRFIs are made part of Guile? Your point about maintenance and evolving applies equally to other SRFIs. > Adding #lang support in Guile would be nice. As discussed on IRC, it > can be experimented with in a WIP branch. Have you seen my messages on how the "#lang" construct is problematic for some languages, and how alternatives like "[comment delimiter] -*- stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be equally simple (*) and not have any downsides (**). (*) The port encoding detection supports "-*- coding: whatever -*-", presumably that functionality could be reused. (**) For compatibility with Racket, it's not like we couldn't implement both "#lang" and "-*- stuff: language -*-". Greetingss, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 18:42 ` Maxime Devos @ 2023-02-24 15:45 ` Ludovic Courtès 2023-02-24 16:34 ` Dr. Arne Babenhauserheide 2023-02-24 23:48 ` [PATCH] add language/wisp to Guile? Maxime Devos 0 siblings, 2 replies; 83+ messages in thread From: Ludovic Courtès @ 2023-02-24 15:45 UTC (permalink / raw) To: guile-devel Hello! Maxime Devos <maximedevos@telenet.be> skribis: > Why should Wisp be a separate package when other SRFIs are made part > of Guile? Your point about maintenance and evolving applies equally > to other SRFIs. That’s a good point. Making it available as (srfi srfi-119) would make sense I guess. I need to take a closer look… >> Adding #lang support in Guile would be nice. As discussed on IRC, it >> can be experimented with in a WIP branch. > > Have you seen my messages on how the "#lang" construct is problematic > for some languages, and how alternatives like "[comment delimiter] -*- > stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be > equally simple (*) and not have any downsides (**). > > (*) The port encoding detection supports "-*- coding: whatever -*-", > presumably that functionality could be reused. > > (**) For compatibility with Racket, it's not like we couldn't > implement both "#lang" and "-*- stuff: language -*-". I haven’t seen your messages yet, I just wanted to express support of the general idea. For years, we have discussed #lang support; I know Andy is enthusiastic about it, and while I was initially reluctant, I’ve come to appreciate the idea. What you point out is worth considering, but note that Guile already supports #!r6rs for instance. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-24 15:45 ` Ludovic Courtès @ 2023-02-24 16:34 ` Dr. Arne Babenhauserheide 2023-03-08 10:34 ` Dr. Arne Babenhauserheide 2023-02-24 23:48 ` [PATCH] add language/wisp to Guile? Maxime Devos 1 sibling, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-24 16:34 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 693 bytes --] Hi, Ludovic Courtès <ludo@gnu.org> writes: >> Why should Wisp be a separate package when other SRFIs are made part >> of Guile? Your point about maintenance and evolving applies equally >> to other SRFIs. > > That’s a good point. Making it available as (srfi srfi-119) would make > sense I guess. I need to take a closer look… That’s where the documentation and tests are located: - test-suite/tests/srfi-119.test - doc/ref/srfi-modules.texi::5666:@node SRFI-119 The language implementation is in (language wisp) because that’s in the language search path. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-24 16:34 ` Dr. Arne Babenhauserheide @ 2023-03-08 10:34 ` Dr. Arne Babenhauserheide 2023-05-01 9:54 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-03-08 10:34 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 1177 bytes --] Hi, "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > Ludovic Courtès <ludo@gnu.org> writes: >>> Why should Wisp be a separate package when other SRFIs are made part >>> of Guile? Your point about maintenance and evolving applies equally >>> to other SRFIs. >> >> That’s a good point. Making it available as (srfi srfi-119) would make >> sense I guess. I need to take a closer look… > > That’s where the documentation and tests are located: > > - test-suite/tests/srfi-119.test > - doc/ref/srfi-modules.texi::5666:@node SRFI-119 > > The language implementation is in (language wisp) because that’s in the > language search path. Given the complexities in changing the way languages are handled (the required discussions, as we’ve seen in the not yet resolved discussion), would you be OK with keeping the question about adding support for SRFI-119 to Guile separate from the general discussion about language handling? Are there improvements needed besides the ones I did thanks to the review by Maxime or is this good to go? Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-03-08 10:34 ` Dr. Arne Babenhauserheide @ 2023-05-01 9:54 ` Dr. Arne Babenhauserheide 2023-06-10 16:40 ` Ludovic Courtès 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-05-01 9:54 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1.1: Type: text/plain, Size: 1180 bytes --] > "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: >> Ludovic Courtès <ludo@gnu.org> writes: >>>> Why should Wisp be a separate package when other SRFIs are made part >>>> of Guile? Your point about maintenance and evolving applies equally >>>> to other SRFIs. >>> >>> That’s a good point. Making it available as (srfi srfi-119) would make >>> sense I guess. I need to take a closer look… >> >> That’s where the documentation and tests are located: >> >> - test-suite/tests/srfi-119.test >> - doc/ref/srfi-modules.texi::5666:@node SRFI-119 >> >> The language implementation is in (language wisp) because that’s in the >> language search path. > > Given the complexities in changing the way languages are handled (the > required discussions, as we’ve seen in the not yet resolved discussion), > would you be OK with keeping the question about adding support for > SRFI-119 to Guile separate from the general discussion about language > handling? > > Are there improvements needed besides the ones I did thanks to the > review by Maxime or is this good to go? I created a new (squashed) version of the patch to simplify reviewing: [-- Attachment #1.2: 0001-Add-language-wisp-wisp-tests-and-srfi-119-documentat.patch --] [-- Type: text/x-patch, Size: 46310 bytes --] From c7a50f632293cf88f241d45d1fd52fa2f58ce772 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 3 Feb 2023 22:20:04 +0100 Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation * doc/ref/srfi-modules.texi (srfi-119): add node * module/language/wisp.scm: New file. * module/language/wisp/spec.scm: New file. * test-suite/tests/srfi-119.test: New file. --- doc/ref/srfi-modules.texi | 30 ++ module/language/wisp.scm | 761 +++++++++++++++++++++++++++++++++ module/language/wisp/spec.scm | 85 ++++ test-suite/tests/srfi-119.test | 81 ++++ 4 files changed, 957 insertions(+) create mode 100644 module/language/wisp.scm create mode 100644 module/language/wisp/spec.scm create mode 100644 test-suite/tests/srfi-119.test diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 0ef136215..759e293ec 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. +* SRFI-119:: Wisp: simpler indentation-sensitive scheme. * SRFI-171:: Transducers @end menu @@ -5662,6 +5663,34 @@ Return the current contents of @var{box}. Set the contents of @var{box} to @var{value}. @end deffn +@node SRFI-119 +@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme. +@cindex SRFI-119 +@cindex wisp + +The languages shipped in Guile include SRFI-119 (wisp), an encoding of +Scheme that allows replacing parentheses with equivalent indentation and +inline colons. See +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification +of SRFI-119}. Some examples: + +@example +display "Hello World!" @result{} (display "Hello World!") +@end example + +@example +define : factorial n @result{} (define (factorial n) + if : zero? n @result{} (if (zero? n) + . 1 @result{} 1 + * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) +@end example + +To execute a file with wisp code, select the language and filename +extension @code{.w} vie @code{guile --language=wisp -x .w}. + +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. + + @node SRFI-171 @subsection Transducers @cindex SRFI-171 @@ -5705,6 +5734,7 @@ left-to-right, due to how transducers are initiated. * SRFI-171 Helpers:: Utilities for writing your own transducers @end menu + @node SRFI-171 General Discussion @subsubsection SRFI-171 General Discussion @cindex transducers discussion diff --git a/module/language/wisp.scm b/module/language/wisp.scm new file mode 100644 index 000000000..7a12e126a --- /dev/null +++ b/module/language/wisp.scm @@ -0,0 +1,761 @@ +;;; Wisp + +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;;; Commentary: + +;; Scheme-only implementation of a wisp-preprocessor which output a +;; scheme code tree to feed to a scheme interpreter instead of a +;; preprocessed file. + +;; Limitations: +;; - in some cases the source line information is missing in backtraces. +;; check for set-source-property! + +;;; Code: + +(define-module (language wisp) + #:export (wisp-scheme-read-chunk wisp-scheme-read-all + wisp-scheme-read-file-chunk wisp-scheme-read-file + wisp-scheme-read-string)) + +; use curly-infix by default +(read-enable 'curly-infix) + +(use-modules + (srfi srfi-1) + (srfi srfi-11); for let-values + (ice-9 rw); for write-string/partial + (ice-9 match)) + + +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define (line-indent line) + (car line)) + +(define (line-real-indent line) + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." + (let (( indent (line-indent line))) + (if (= -1 indent) + 0 + indent))) + +(define (line-code line) + (let ((code (cdr line))) + ; propagate source properties + (when (not (null? code)) + (set-source-properties! code (source-properties line))) + code)) + +; literal values I need +(define readcolon + (string->symbol ":")) + +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") +; define an intermediate dot replacement with UUID to avoid clashes. +(define repr-dot ; . + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) + +; allow using reader additions as the first element on a line to prefix the list +(define repr-quote ; ' + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) +(define repr-unquote ; , + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) +(define repr-quasiquote ; ` + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) +(define repr-unquote-splicing ; ,@ + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) + +(define repr-syntax ; #' + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) +(define repr-unsyntax ; #, + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) +(define repr-quasisyntax ; #` + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) +(define repr-unsyntax-splicing ; #,@ + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) + +; TODO: wrap the reader to return the repr of the syntax reader +; additions + +(define (match-charlist-to-repr charlist) + (let + ((chlist (reverse charlist))) + (cond + ((equal? chlist (list #\.)) + repr-dot) + ((equal? chlist (list #\')) + repr-quote) + ((equal? chlist (list #\,)) + repr-unquote) + ((equal? chlist (list #\`)) + repr-quasiquote) + ((equal? chlist (list #\, #\@)) + repr-unquote-splicing) + ((equal? chlist (list #\# #\')) + repr-syntax) + ((equal? chlist (list #\# #\,)) + repr-unsyntax) + ((equal? chlist (list #\# #\`)) + repr-quasisyntax) + ((equal? chlist (list #\# #\, #\@)) + repr-unsyntax-splicing) + (else + #f)))) + +(define (wisp-read port) + "wrap read to catch list prefixes." + (let ((prefix-maxlen 4)) + (let longpeek + ((peeked '()) + (repr-symbol #f)) + (cond + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) + (if repr-symbol ; found a special symbol, return it. + repr-symbol + (let unpeek + ((remaining peeked)) + (cond + ((equal? '() remaining) + (read port)); let read to the work + (else + (unread-char (car remaining) port) + (unpeek (cdr remaining))))))) + (else + (let* + ((next-char (read-char port)) + (peeked (cons next-char peeked))) + (longpeek + peeked + (match-charlist-to-repr peeked)))))))) + + + +(define (line-continues? line) + (equal? repr-dot (car (line-code line)))) + +(define (line-only-colon? line) + (and + (equal? ":" (car (line-code line))) + (null? (cdr (line-code line))))) + +(define (line-empty-code? line) + (null? (line-code line))) + +(define (line-empty? line) + (and + ; if indent is -1, we stripped a comment, so the line was not really empty. + (= 0 (line-indent line)) + (line-empty-code? line))) + +(define (line-strip-continuation line) + (if (line-continues? line) + (append + (list + (line-indent line)) + (cdr (line-code line))) + line)) + +(define (line-strip-indentation-marker line) + "Strip the indentation markers from the beginning of the line" + (cdr line)) + +(define (indent-level-reduction indentation-levels level select-fun) + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" + (let loop + ((newlevels indentation-levels) + (diff 0)) + (cond + ((= level (car newlevels)) + (select-fun (list diff indentation-levels))) + ((< level (car newlevels)) + (loop + (cdr newlevels) + (1+ diff))) + (else + (throw 'wisp-syntax-error "Level ~A not found in the indentation-levels ~A."))))) + +(define (indent-level-difference indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the count + (car x)))) + +(define (indent-reduce-to-level indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the levels + (car (cdr x))))) + +(define (chunk-ends-with-period currentsymbols next-char) + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." + (and (not (null? currentsymbols)) + (equal? #\newline next-char) + (equal? repr-dot + (list-ref currentsymbols (- (length currentsymbols) 1))))) + +(define (wisp-scheme-read-chunk-lines port) + (let loop + ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (inindent #t) + (inunderscoreindent (equal? #\_ (peek-char port))) + (incomment #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) + (cond + ((>= emptylines 2); the chunk end has to be checked + ; before we look for new chars in the + ; port to make execution in the REPL + ; after two empty lines work + ; (otherwise it shows one more line). + indent-and-symbols) + (else + (let ((next-char (peek-char port))) + (cond + ((eof-object? next-char) + (append indent-and-symbols (list (append (list currentindent) currentsymbols)))) + ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) + (append indent-and-symbols)); top-level form ends chunk + ((chunk-ends-with-period currentsymbols next-char) + ; the line ends with a period. This is forbidden in + ; SRFI-119. Use it to end the line in the REPL without + ; showing continuation dots (...). + (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1))))) + ((and inindent (equal? #\space next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; inindent + #f ; inunderscoreindent + #f ; incomment + (1+ currentindent) + currentsymbols + emptylines)) + ((and inunderscoreindent (equal? #\_ next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; inindent + #t ; inunderscoreindent + #f ; incomment + (1+ currentindent) + currentsymbols + emptylines)) + ; any char but whitespace *after* underscoreindent is + ; an error. This is stricter than the current wisp + ; syntax definition. TODO: Fix the definition. Better + ; start too strict. FIXME: breaks on lines with only + ; underscores which should be empty lines. + ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) + ((equal? #\newline next-char) + (read-char port); remove the newline + ; The following two lines would break the REPL by requiring one char too many. + ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port + ; read-char port ; remove a full \n\r. Damn special cases... + (let* ; distinguish pure whitespace lines and lines + ; with comment by giving the former zero + ; indent. Lines with a comment at zero indent + ; get indent -1 for the same reason - meaning + ; not actually empty. + ((indent + (cond + (incomment + (if (= 0 currentindent); specialcase + -1 + currentindent)) + ((not (null? currentsymbols)); pure whitespace + currentindent) + (else + 0))) + (parsedline (append (list indent) currentsymbols)) + (emptylines + (if (not (line-empty? parsedline)) + 0 + (1+ emptylines)))) + (when (not (= 0 (length parsedline))) + ; set the source properties to parsedline so we can try to add them later. + (set-source-property! parsedline 'filename (port-filename port)) + (set-source-property! parsedline 'line (port-line port))) + ; TODO: If the line is empty. Either do it here and do not add it, just + ; increment the empty line counter, or strip it later. Replace indent + ; -1 by indent 0 afterwards. + (loop + (append indent-and-symbols (list parsedline)) + #t ; inindent + (if (<= 2 emptylines) + #f ; chunk ends here + (equal? #\_ (peek-char port))); are we in underscore indent? + #f ; incomment + 0 + '() + emptylines))) + ((equal? #t incomment) + (read-char port); remove one comment character + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #t ; incomment + currentindent + currentsymbols + emptylines)) + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent + (read-char port); remove char + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #f ; incomment + currentindent + currentsymbols + emptylines)) + ; | cludge to appease the former wisp parser + ; | used for bootstrapping which has a + ; v problem with the literal comment char + ((equal? (string-ref ";" 0) next-char) + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #t ; incomment + currentindent + currentsymbols + emptylines)) + (else ; use the reader + (loop + indent-and-symbols + #f ; inindent + #f ; inunderscoreindent + #f ; incomment + currentindent + ; this also takes care of the hashbang and leading comments. + (append currentsymbols (list (wisp-read port))) + emptylines)))))))) + + +(define (line-code-replace-inline-colons line) + "Replace inline colons by opening parens which close at the end of the line" + ; format #t "replace inline colons for line ~A\n" line + (let loop + ((processed '()) + (unprocessed line)) + (cond + ((null? unprocessed) + ; format #t "inline-colons processed line: ~A\n" processed + processed) + ; replace : . with nothing + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) + (loop + (append processed + (loop '() (cdr (cdr unprocessed)))) + '())) + ((equal? readcolon (car unprocessed)) + (loop + ; FIXME: This should turn unprocessed into a list. + (append processed + (list (loop '() (cdr unprocessed)))) + '())) + (else + (loop + (append processed + (list (car unprocessed))) + (cdr unprocessed)))))) + +(define (line-replace-inline-colons line) + (cons + (line-indent line) + (line-code-replace-inline-colons (line-code line)))) + +(define (line-strip-lone-colon line) + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." + (if + (equal? + (line-code line) + (list readcolon)) + (list (line-indent line)) + line)) + +(define (line-finalize line) + "Process all wisp-specific information in a line and strip it" + (let ((l (line-code-replace-inline-colons + (line-strip-indentation-marker + (line-strip-lone-colon + (line-strip-continuation line)))))) + (when (not (null? (source-properties line))) + (catch #t + (lambda () + (set-source-properties! l (source-properties line))) + (lambda (key . arguments) + #f))) + l)) + +(define (wisp-add-source-properties-from source target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) + +(define (wisp-propagate-source-properties code) + "Propagate the source properties from the sourrounding list into every part of the code." + (let loop + ((processed '()) + (unprocessed code)) + (cond + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) + unprocessed) + ((and (pair? unprocessed) (not (list? unprocessed))) + (cons + (wisp-propagate-source-properties (car unprocessed)) + (wisp-propagate-source-properties (cdr unprocessed)))) + ((null? unprocessed) + processed) + (else + (let ((line (car unprocessed))) + (if (null? (source-properties unprocessed)) + (wisp-add-source-properties-from line unprocessed) + (wisp-add-source-properties-from unprocessed line)) + (loop + (append processed (list (wisp-propagate-source-properties line))) + (cdr unprocessed))))))) + +(define* (wisp-scheme-indentation-to-parens lines) + "Add parentheses to lines and remove the indentation markers" + (when + (and + (not (null? lines)) + (not (line-empty-code? (car lines))) + (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment + (if (= 1 (line-real-indent (car lines))) + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands + (set! lines + (cons + (cons 0 (cdr (car lines))) + (cdr lines))) + (throw 'wisp-syntax-error + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" + (car lines))))) + (let loop + ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* + ((current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (list 0))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (list 0))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) + ; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ; . processed current-line next-line unprocessed indentation-levels current-indentation + (cond + ; the real end: this is reported to the outside world. + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) + ; display "done\n" + ; reverse the processed lines, because I use cons. + processed) + ; the recursion end-condition + ((and (null? unprocessed)) + ; display "last step\n" + ; this is the last step. Nothing more to do except + ; for rolling up the indentation levels. return the + ; new processed and unprocessed lists: this is a + ; side-recursion + (values processed unprocessed)) + ((null? indentation-levels) + ; display "indentation-levels null\n" + (throw 'wisp-programming-error "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")) + (else ; now we come to the line-comparisons and indentation-counting. + (cond + ((line-empty-code? current-line) + ; display "current-line empty\n" + ; We cannot process indentation without + ; code. Just switch to the next line. This should + ; only happen at the start of the recursion. + ; TODO: Somehow preserve the line-numbers. + (loop + processed + (cdr unprocessed) + indentation-levels)) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) + ; display "next-line empty\n" + ; TODO: Somehow preserve the line-numbers. + ; take out the next-line from unprocessed. + (loop + processed + (cons current-line + (cdr (cdr unprocessed))) + indentation-levels)) + ((> current-indentation current-line-indentation) + ; display "current-indent > next-line\n" + ; this just steps back one level via the side-recursion. + (let ((previous-indentation (car (cdr indentation-levels)))) + (if (<= current-line-indentation previous-indentation) + (values processed unprocessed) + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. + (let ((linenumber (- (length lines) (length unprocessed)))) + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + (cdr indentation-levels))))))) + ((= current-indentation current-line-indentation) + ; display "current-indent = next-line\n" + (let + ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) + (cond + ((>= current-line-indentation next-line-indentation) + ; simple recursiive step to the next line + ; display "current-line-indent >= next-line-indent\n" + (loop + (append processed + (if (line-continues? current-line) + line + (wisp-add-source-properties-from line (list line)))) + (cdr unprocessed); recursion here + indentation-levels)) + ((< current-line-indentation next-line-indentation) + ; display "current-line-indent < next-line-indent\n" + ; format #t "line: ~A\n" line + ; side-recursion via a sublist + (let-values + (((sub-processed sub-unprocessed) + (loop + line + (cdr unprocessed); recursion here + indentation-levels))) + ; format #t "side-recursion:\n sub-processed: ~A\n processed: ~A\n\n" sub-processed processed + (loop + (append processed (list sub-processed)) + sub-unprocessed ; simply use the recursion from the sub-recursion + indentation-levels)))))) + ((< current-indentation current-line-indentation) + ; display "current-indent < next-line\n" + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + indentation-levels))) + (else + (throw 'wisp-not-implemented + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))) + + +(define (wisp-scheme-replace-inline-colons lines) + "Replace inline colons by opening parens which close at the end of the line" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (list (line-replace-inline-colons (car unprocessed)))) + (cdr unprocessed))))) + + +(define (wisp-scheme-strip-indentation-markers lines) + "Strip the indentation markers from the beginning of the lines" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (cdr (car unprocessed))) + (cdr unprocessed))))) + +(define (wisp-unescape-underscore-and-colon code) + "replace \\_ and \\: by _ and :" + (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code))) + + +(define (wisp-replace-empty-eof code) + "replace ((#<eof>)) by ()" + ; FIXME: Actually this is a hack which fixes a bug when the + ; parser hits files with only hashbang and comments. + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) + (list) + code)) + + +(define (wisp-replace-paren-quotation-repr code) + "Replace lists starting with a quotation symbol by + quoted lists." + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a))) + +(define (wisp-make-improper code) + "Turn (a #{.}# b) into the correct (a . b). + +read called on a single dot creates a variable named #{.}# (|.| +in r7rs). Due to parsing the indentation before the list +structure is known, the reader cannot create improper lists +when it reads a dot. So we have to take another pass over the +code to recreate the improper lists. + +Match is awesome!" + (let + ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (append (map wisp-make-improper a) + (cons (wisp-make-improper b) (wisp-make-improper c)))) + ((a ...) + (map wisp-make-improper a)) + (a + a)))) + (define (syntax-error li msg) + (throw 'wisp-syntax-error (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))) + (if #t + improper + (let check + ((tocheck improper)) + (match tocheck + ; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ; more complex pairs + ((? pair? a) + (let + ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a + a)))))) + +(define (wisp-scheme-read-chunk port) + "Read and parse one chunk of wisp-code" + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\#)) + (let ((lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines))))))))) + +(define (wisp-scheme-read-all port) + "Read all chunks from the given port" + (let loop + ((tokens '())) + (cond + ((eof-object? (peek-char port)) + tokens) + (else + (loop + (append tokens (wisp-scheme-read-chunk port))))))) + +(define (wisp-scheme-read-file path) + (call-with-input-file path wisp-scheme-read-all)) + +(define (wisp-scheme-read-file-chunk path) + (call-with-input-file path wisp-scheme-read-chunk)) + +(define (wisp-scheme-read-string str) + (call-with-input-string str wisp-scheme-read-all)) + +(define (wisp-scheme-read-string-chunk str) + (call-with-input-string str wisp-scheme-read-chunk)) + diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm new file mode 100644 index 000000000..477036c71 --- /dev/null +++ b/module/language/wisp/spec.scm @@ -0,0 +1,85 @@ +;; Language interface for Wisp in Guile + +;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm + +;;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria +;;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;; Permission is hereby granted, free of charge, to any person +;;; obtaining a copy of this software and associated documentation +;;; files (the "Software"), to deal in the Software without +;;; restriction, including without limitation the rights to use, copy, +;;; modify, merge, publish, distribute, sublicense, and/or sell copies +;;; of the Software, and to permit persons to whom the Software is +;;; furnished to do so, subject to the following conditions: +;;; +;;; The above copyright notice and this permission notice shall be +;;; included in all copies or substantial portions of the Software. +;;; +;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +;;; SOFTWARE. + +; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm +(define-module (language wisp spec) + #:use-module (language wisp) + #:use-module (system base compile) + #:use-module (system base language) + #:use-module (language scheme compile-tree-il) + #:use-module (language scheme decompile-tree-il) + #:export (wisp)) + +;;; +;;; Language definition +;;; + + +(define (read-one-wisp-sexp port env) + ;; Allow using "# foo" as #(foo). + ;; Don't use the globally-acting read-hash-extend, because this + ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme. + ;; Instead, use fluids to temporarily add the extension. + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\# )) + ;; Read Wisp files as UTF-8, to support non-ASCII characters. + ;; TODO: would be nice to support ';; coding: whatever' lines + ;; like in parenthese-y Scheme. + (set-port-encoding! port "UTF-8") + (if (eof-object? (peek-char port)) + (read-char port) ; return eof: we’re done + (let ((chunk (wisp-scheme-read-chunk port))) + (and (not (null? chunk)) ; <---- XXX: maybe (pair? chunk) + (car chunk)))))) + +(define-language wisp + #:title "Wisp Scheme Syntax. See SRFI-119 for details." + ; . #:reader read-one-wisp-sexp + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; + #:compilers `((tree-il . ,compile-tree-il)) + #:decompilers `((tree-il . ,decompile-tree-il)) + #:evaluator (lambda (x module) (primitive-eval x)) + #:printer write ; TODO: backtransform to wisp? Use source-properties? + #:make-default-environment + (lambda () + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', + ;; `fluid-set!', etc. don't have any effect in the current environment. + (let ((m (make-fresh-user-module))) + ;; Provide a separate `current-reader' fluid so that + ;; compile-time changes to `current-reader' are + ;; limited to the current compilation unit. + (module-define! m 'current-reader (make-fluid)) + ;; Default to `simple-format', as is the case until + ;; (ice-9 format) is loaded. This allows + ;; compile-time warnings to be emitted when using + ;; unsupported options. + (module-set! m 'format simple-format) + m))) + + + diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test new file mode 100644 index 000000000..a888df41d --- /dev/null +++ b/test-suite/tests/srfi-119.test @@ -0,0 +1,81 @@ +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*- +;;;; +;;;; Copyright (C) 2023 Free Software Foundation, Inc. +;;;; +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +(define-module (test-srfi-119) + #:use-module (test-suite lib) + #:use-module (srfi srfi-1) + #:use-module (language wisp)) + +(define (read-string s) + (with-input-from-string s read)) + +(define (with-read-options opts thunk) + (let ((saved-options (read-options))) + (dynamic-wind + (lambda () + (read-options opts)) + thunk + (lambda () + (read-options saved-options))))) + +(define (wisp->list str) + (wisp-scheme-read-string str)) + +(with-test-prefix "wisp-read-simple" + (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) + (pass-if (equal? (wisp->list ". 5") '(5))) + (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) +(with-test-prefix "wisp-read-complex" + (pass-if (equal? (wisp->list " +a b c d e + . f g h + . i j k + +concat \"I want \" + getwish from me + . \" - \" username +") '( +(a b c d e + f g h + i j k) + +(concat "I want " + (getwish from me) + " - " username)))) + + (pass-if (equal? (wisp->list " +define : a b c +_ d e +___ f +___ g h +__ . i + +define : _ +_ display \"hello\n\" + +\\_") '( +(define (a b c) + (d e + (f) + (g h) + i)) + +(define (_) + (display "hello\n")) + +(_))))) -- 2.39.2 [-- Attachment #1.3: Type: text/plain, Size: 103 bytes --] Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-05-01 9:54 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Dr. Arne Babenhauserheide @ 2023-06-10 16:40 ` Ludovic Courtès 2023-06-12 10:22 ` Maxime Devos 0 siblings, 1 reply; 83+ messages in thread From: Ludovic Courtès @ 2023-06-10 16:40 UTC (permalink / raw) To: guile-devel; +Cc: Dr. Arne Babenhauserheide Hi! "Dr. Arne Babenhauserheide" <arne_bab@web.de> skribis: >>> Ludovic Courtès <ludo@gnu.org> writes: [...] >> Given the complexities in changing the way languages are handled (the >> required discussions, as we’ve seen in the not yet resolved discussion), >> would you be OK with keeping the question about adding support for >> SRFI-119 to Guile separate from the general discussion about language >> handling? >> >> Are there improvements needed besides the ones I did thanks to the >> review by Maxime or is this good to go? > > I created a new (squashed) version of the patch to simplify reviewing: It does, thank you! Overall I’m confident the code has been battle-tested (I suppose there are minimal changes compared to Wisp, right?), so I’ll comment mostly on cosmetic issues. > From c7a50f632293cf88f241d45d1fd52fa2f58ce772 Mon Sep 17 00:00:00 2001 > From: Arne Babenhauserheide <arne_bab@web.de> > Date: Fri, 3 Feb 2023 22:20:04 +0100 > Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation > > * doc/ref/srfi-modules.texi (srfi-119): add node > * module/language/wisp.scm: New file. > * module/language/wisp/spec.scm: New file. > * test-suite/tests/srfi-119.test: New file. Please add the new files to the relevant ‘Makefile.am’ files. > +* SRFI-119:: Wisp: simpler indentation-sensitive scheme. Please capitalize “Scheme” (in general we like to pay attention to typography, spelling, and language in the manual.) > +@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme. > +@cindex SRFI-119 > +@cindex wisp Same: “Scheme”, “Wisp”. > +The languages shipped in Guile include SRFI-119 (wisp), an encoding of s/(wisp)/, also referred to as @dfn{Wisp} (for ``Whitespace …'')/ > +Scheme that allows replacing parentheses with equivalent indentation and > +inline colons. See Two spaces after end-of-sentence periods, to facilitate navigation in Emacs. > +++ b/module/language/wisp.scm > @@ -0,0 +1,761 @@ > +;;; Wisp > + > +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. > +;; Copyright (C) 2014--2023 Arne Babenhauserheide. > +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> I see you don’t have a copyright assignment on file for Guile. If you wish, you can assign copyright to the FSF: https://lists.gnu.org/archive/html/guile-devel/2022-10/msg00008.html Let us know what you’d like to do. > +(define-module (language wisp) > + #:export (wisp-scheme-read-chunk wisp-scheme-read-all > + wisp-scheme-read-file-chunk wisp-scheme-read-file > + wisp-scheme-read-string)) Please remove tabs from this file and indent it “the usual way”. You can do that by passing it through Emacs and using ‘M-x indent-region’, or possibly using ‘guix style -f’. > +; use curly-infix by default Use two leading colons for comments, except for margin comments. > +(read-enable 'curly-infix) This needs to be: (eval-when (expand load eval) (read-enable 'curly-infix)) > +(use-modules > + (srfi srfi-1) > + (srfi srfi-11); for let-values > + (ice-9 rw); for write-string/partial > + (ice-9 match)) Please make them #:use-module clauses in the ‘define-module’ form above. > +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) > +(define (line-indent line) > + (car line)) > + > +(define (line-real-indent line) > + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." > + (let (( indent (line-indent line))) > + (if (= -1 indent) > + 0 > + indent))) > + > +(define (line-code line) > + (let ((code (cdr line))) > + ; propagate source properties > + (when (not (null? code)) > + (set-source-properties! code (source-properties line))) > + code)) Instead of using lists or pairs for “lines”, I suggest defining a proper record type, like: (define-record-type <input-line> (input-line indent content) input-line? (indent input-line-indent) (content input-line-content)) This will make the code easier to read and type-clear. > +; literal values I need > +(define readcolon > + (string->symbol ":")) > + > +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") > +; define an intermediate dot replacement with UUID to avoid clashes. > +(define repr-dot ; . > + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) > + > +; allow using reader additions as the first element on a line to prefix the list > +(define repr-quote ; ' > + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) > +(define repr-unquote ; , > + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) > +(define repr-quasiquote ; ` > + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) > +(define repr-unquote-splicing ; ,@ > + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) > + > +(define repr-syntax ; #' > + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) > +(define repr-unsyntax ; #, > + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) > +(define repr-quasisyntax ; #` > + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) > +(define repr-unsyntax-splicing ; #,@ > + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) Instead of using these UUIDs, I suggest using known unique objects, “unique” in the sense of ‘eq?’. So it can be: (define repr-dot (make-symbol "repr-dot")) ;uninterned symbol > +(define (wisp-scheme-read-chunk-lines port) > + (let loop > + ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) > + (inindent #t) > + (inunderscoreindent (equal? #\_ (peek-char port))) > + (incomment #f) > + (currentindent 0) > + (currentsymbols '()) > + (emptylines 0)) I’d encourage following the usual naming convention, so ‘in-indent?’, ‘in-comment?’, etc. > + ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) > + (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) I suggest using exception objects or SRFI-35 error conditions (they’re interchangeable) carrying information about the king of parsing error and its location. That way, the user interface, such as the compiler, can produce helpful error messages (and internationalized, too). > +(define (wisp-replace-paren-quotation-repr code) > + "Replace lists starting with a quotation symbol by > + quoted lists." > + (match code > + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'quote (map wisp-replace-paren-quotation-repr a))) You need to write these clauses like: (match code (((? wisp-quote?) body ...) …)) where: (define (wisp-quote? obj) (eq? obj repr-quote)) On the use of source properties: they’re not ideal (they rely on a side hash table that’s quite memory-hungry) and it would be nice to consider also implementing the “annotated read” interface (info "(guile) Annotated Scheme Read"). That can come in a subsequent patch, though. > +++ b/module/language/wisp/spec.scm > @@ -0,0 +1,85 @@ > +;; Language interface for Wisp in Guile > + > +;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm > + > +;;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria > +;;; Copyright (C) 2014--2023 Arne Babenhauserheide. > +;;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> > + > +;;; Permission is hereby granted, free of charge, to any person > +;;; obtaining a copy of this software and associated documentation > +;;; files (the "Software"), to deal in the Software without > +;;; restriction, including without limitation the rights to use, copy, > +;;; modify, merge, publish, distribute, sublicense, and/or sell copies > +;;; of the Software, and to permit persons to whom the Software is > +;;; furnished to do so, subject to the following conditions: > +;;; > +;;; The above copyright notice and this permission notice shall be > +;;; included in all copies or substantial portions of the Software. > +;;; > +;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > +;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > +;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND > +;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS > +;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN > +;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN > +;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > +;;; SOFTWARE. Can you make it LGPLv3+? It’s a small file anyway. > +(define-language wisp > + #:title "Wisp Scheme Syntax. See SRFI-119 for details." > + ; . #:reader read-one-wisp-sexp > + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; > + #:compilers `((tree-il . ,compile-tree-il)) > + #:decompilers `((tree-il . ,decompile-tree-il)) > + #:evaluator (lambda (x module) (primitive-eval x)) > + #:printer write ; TODO: backtransform to wisp? Use source-properties? > + #:make-default-environment > + (lambda () > + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', > + ;; `fluid-set!', etc. don't have any effect in the current environment. > + (let ((m (make-fresh-user-module))) > + ;; Provide a separate `current-reader' fluid so that > + ;; compile-time changes to `current-reader' are > + ;; limited to the current compilation unit. > + (module-define! m 'current-reader (make-fluid)) > + ;; Default to `simple-format', as is the case until > + ;; (ice-9 format) is loaded. This allows > + ;; compile-time warnings to be emitted when using > + ;; unsupported options. > + (module-set! m 'format simple-format) I’d remove this last statement, I think we should avoid fiddling with bindings here. > +(define-module (test-srfi-119) > + #:use-module (test-suite lib) > + #:use-module (srfi srfi-1) > + #:use-module (language wisp)) The test suite looks rather short; it would be nice if it would cover more code. In particular, I think source location info is important for front-ends like this, because it determines the quality of error messages and thus its ease of use. Perhaps you could add test in that direction? Overall I feel we should be able to merge this rather soon. Do ping me or Andy when needed! Thanks, Ludo’. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-06-10 16:40 ` Ludovic Courtès @ 2023-06-12 10:22 ` Maxime Devos 2023-08-10 6:28 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-06-12 10:22 UTC (permalink / raw) To: Ludovic Courtès, guile-devel; +Cc: Dr. Arne Babenhauserheide [-- Attachment #1.1.1: Type: text/plain, Size: 1658 bytes --] Op 10-06-2023 om 18:40 schreef Ludovic Courtès: > Instead of using these UUIDs, I suggest using known unique objects, > “unique” in the sense of ‘eq?’. > > So it can be: > > (define repr-dot (make-symbol "repr-dot")) ;uninterned symbol > IIRC, in a previous reply, this was already asked, and the response was that this doesn't actually work because of reasons. >> + ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) >> + (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) > > I suggest using exception objects or SRFI-35 error conditions (they’re > interchangeable) carrying information about the king of parsing error > and its location. That way, the user interface, such as the compiler, > can produce helpful error messages (and internationalized, too). Agreed. > [...] > Can you make it LGPLv3+? It’s a small file anyway. Only if David A. Wheeler, Alan Manuel K. Gloria and Maxime Devos agrees. Otherwise, you still need to include the license text: > [...] > +;;; The above copyright notice and this permission notice shall be > +;;; included in all copies or substantial portions of the Software. > [...] and having both this license text and the LGPLv3+ stuff in the same file is messy (technically possible if done carefully, but messy). As its a small file anyway, I don't think it's worth it unless everyone agrees to make in LGPLv3+. As far as I'm concerned, I agree. Best regards, Maxime Devos. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-06-12 10:22 ` Maxime Devos @ 2023-08-10 6:28 ` Dr. Arne Babenhauserheide 2023-08-14 20:11 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-08-10 6:28 UTC (permalink / raw) To: Maxime Devos; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 625 bytes --] Maxime Devos <maximedevos@telenet.be> writes: >> [...] >> Can you make it LGPLv3+? It’s a small file anyway. > > Only if David A. Wheeler, Alan Manuel K. Gloria and Maxime Devos > agrees. Otherwise, you still need to include the license text: I just asked David and Alan, and when I checked the sources I realized that this was unnecessary: the file was LGPL originally and I once asked for permission to relicense to MIT for SRFI-119. So yes, thanks to permission of Maxime Devos the file can be LGPL. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-08-10 6:28 ` Dr. Arne Babenhauserheide @ 2023-08-14 20:11 ` Dr. Arne Babenhauserheide 2023-08-14 20:30 ` Dr. Arne Babenhauserheide ` (2 more replies) 0 siblings, 3 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-08-14 20:11 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Maxime Devos, Ludovic Courtès, guile-devel [-- Attachment #1.1: Type: text/plain, Size: 7661 bytes --] Hi, I did the changes for the review. It took a while (sorry for that) but it got cleaner as a result (thank you!) Firstoff: I’d like to assign my copyright to the FSF. I’ll sign the papers once I receive them. Also I have an Employer disclaimer of rights on paper for Emacs already, so that should not cause trouble. Attached is a new squashed patch. I’ll send another email with only the commits for the individual changes on top of the original squashed patch to avoid tripping up tools that extract diffs. Changes: - [X] LGPL - [X] Please add the new files to the relevant ‘Makefile.am’ files. - [X] Note the changes in Makefiles in the commit - [X] Please capitalize “Scheme” and “Wisp” (in general we like to pay attention to typography, spelling, and language in the manual.) - [X] s/(wisp)/, also referred to as @dfn{Wisp} (for ``Whitespace …'')/ - [X] Two spaces after end-of-sentence periods, to facilitate navigation in Emacs. - [X] indent “the usual way” - [X] comments always with ;; except for margin comments - [X] (read-enable 'curly-infix) This needs to be: (eval-when (expand load eval) (read-enable 'curly-infix)) - [X] Please make them #:use-module clauses in the ‘define-module’ form - [X] I’d encourage following the usual naming convention, so ‘in-indent?’, ‘in-comment?’, etc. - [X] use exception objects or SRFI-35 error conditions instead of throw with symbols - [X] add a test for source location info A change I did not do: - [ ] +Use record-types for the lines+. Reason: I decided not to do this because it currently needs to propagate the source properties when retrieving the code, so this is not a good match for a record type (it may become one with an annotated reader, but I’d like to shift that to a later change: > implementing the “annotated read” interface (info "(guile) > Annotated Scheme Read"). That can come in a subsequent patch, though. Ludovic Courtès <ludo@gnu.org> writes: > Overall I’m confident the code has been battle-tested (I suppose there > are minimal changes compared to Wisp, right?), so I’ll comment mostly on > cosmetic issues. Yes, the code comes directly from Wisp. The original was simply auto-translated. Naturally that’s no guarantee that there are no bugs, but there should be few enough that this can be used in production. >> From c7a50f632293cf88f241d45d1fd52fa2f58ce772 Mon Sep 17 00:00:00 2001 >> From: Arne Babenhauserheide <arne_bab@web.de> >> Date: Fri, 3 Feb 2023 22:20:04 +0100 >> Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation >> >> * doc/ref/srfi-modules.texi (srfi-119): add node >> * module/language/wisp.scm: New file. >> * module/language/wisp/spec.scm: New file. >> * test-suite/tests/srfi-119.test: New file. > > Please add the new files to the relevant ‘Makefile.am’ files. Added. >> +* SRFI-119:: Wisp: simpler indentation-sensitive scheme. > > Please capitalize “Scheme” (in general we like to pay attention to > typography, spelling, and language in the manual.) > >> +The languages shipped in Guile include SRFI-119 (wisp), an encoding of > > s/(wisp)/, also referred to as @dfn{Wisp} (for ``Whitespace …'')/ > >> +Scheme that allows replacing parentheses with equivalent indentation and >> +inline colons. See > > Two spaces after end-of-sentence periods, to facilitate navigation in > Emacs. Adjusted. >> +(define-module (language wisp) >> + #:export (wisp-scheme-read-chunk wisp-scheme-read-all >> + wisp-scheme-read-file-chunk wisp-scheme-read-file >> + wisp-scheme-read-string)) > > Please remove tabs from this file and indent it “the usual way”. You > can do that by passing it through Emacs and using ‘M-x indent-region’, > or possibly using ‘guix style -f’. I used M-x indent-region. > Use two leading colons for comments, except for margin comments. 👍 >> +(read-enable 'curly-infix) > > This needs to be: > > (eval-when (expand load eval) > (read-enable 'curly-infix)) Adjusted — thank you (when to use eval-when was always a bit of a mystery to me …). > Please make them #:use-module clauses in the ‘define-module’ form above. Done. > Instead of using lists or pairs for “lines”, I suggest defining a proper > record type, like: > … > This will make the code easier to read and type-clear. Instead of this, I created a make-line procedure to make this easy to refactor later when the code no longer needs adaptions when retrieving the code part. > I’d encourage following the usual naming convention, so ‘in-indent?’, > ‘in-comment?’, etc. Done. >> + ((and inunderscoreindent (and (not (equal? #\space next-char)) >> (not (equal? #\newline next-char)))) >> + (throw 'wisp-syntax-error "initial underscores without following >> whitespace at beginning of the line after" (last >> indent-and-symbols))) > > I suggest using exception objects or SRFI-35 error conditions (they’re > interchangeable) carrying information about the king of parsing error > and its location. That way, the user interface, such as the compiler, > can produce helpful error messages (and internationalized, too). Done: I used the convenience transformation, i.e. (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "Level ~A not found in the indentation-levels ~A." level indentation-levels)))) >> +++ b/module/language/wisp/spec.scm > Can you make it LGPLv3+? It’s a small file anyway. I investigated and found that it actually was LGPLv3+ in the beginning and I got permission to relicense to MIT for the SRFI, so with permission of Maxime (who did later changes) this is now LGPLv3+ again. >> + (let ((m (make-fresh-user-module))) >> + ;; Provide a separate `current-reader' fluid so that >> + ;; compile-time changes to `current-reader' are >> + ;; limited to the current compilation unit. >> + (module-define! m 'current-reader (make-fluid)) >> + ;; Default to `simple-format', as is the case until >> + ;; (ice-9 format) is loaded. This allows >> + ;; compile-time warnings to be emitted when using >> + ;; unsupported options. >> + (module-set! m 'format simple-format) > > I’d remove this last statement, I think we should avoid fiddling with > bindings here. Removed it now. >> +(define-module (test-srfi-119) >> + #:use-module (test-suite lib) >> + #:use-module (srfi srfi-1) >> + #:use-module (language wisp)) > > The test suite looks rather short; it would be nice if it would cover > more code. I have a pretty extensive test-suite in wisp itself but I didn’t think it proper to just paste it into Guile. If you’d prefer that, I can still add it all, though. > In particular, I think source location info is important for front-ends > like this, because it determines the quality of error messages and thus > its ease of use. Perhaps you could add test in that direction? I added a test and found shortcomings thanks to it. After fixing these, the current code has more robust source-property handling (that hadn’t yet been tested further than ensuring that error messages give proper source positions — which they did). > Overall I feel we should be able to merge this rather soon. Do ping me > or Andy when needed! Thank you! The following is the complete squashed patch. [-- Attachment #1.2: squashed patch --] [-- Type: text/x-patch, Size: 43501 bytes --] From 5857f8b961562d1ad2ae201401d5343233422eff Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 3 Feb 2023 22:20:04 +0100 Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation * doc/ref/srfi-modules.texi (srfi-119): add node * module/language/wisp.scm: New file. * module/language/wisp/spec.scm: New file. * test-suite/tests/srfi-119.test: New file. * am/bootstrap.am (SOURCES): add language/wisp.scm and language/wisp/spec.scm * test-suite/Makefile.am (SCM_TESTS): add tests/srfi-119.test --- am/bootstrap.am | 3 + doc/ref/srfi-modules.texi | 31 ++ module/language/wisp.scm | 784 +++++++++++++++++++++++++++++++++ module/language/wisp/spec.scm | 73 +++ test-suite/Makefile.am | 3 +- test-suite/tests/srfi-119.test | 89 ++++ 6 files changed, 982 insertions(+), 1 deletion(-) create mode 100644 module/language/wisp.scm create mode 100644 module/language/wisp/spec.scm create mode 100644 test-suite/tests/srfi-119.test diff --git a/am/bootstrap.am b/am/bootstrap.am index ff0d1799e..80a8dcdde 100644 --- a/am/bootstrap.am +++ b/am/bootstrap.am @@ -393,6 +393,9 @@ SOURCES = \ \ system/syntax.scm \ \ + language/wisp.scm \ + language/wisp/spec.scm \ + \ system/xref.scm \ \ sxml/apply-templates.scm \ diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 0cdf56923..5b82f8070 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. +* SRFI-119:: Wisp: simpler indentation-sensitive Scheme. * SRFI-171:: Transducers @end menu @@ -5662,6 +5663,35 @@ Return the current contents of @var{box}. Set the contents of @var{box} to @var{value}. @end deffn +@node SRFI-119 +@subsection SRFI-119 Wisp: simpler indentation-sensitive Scheme. +@cindex SRFI-119 +@cindex wisp + +The languages shipped in Guile include SRFI-119, also referred to as +@dfn{Wisp} (for ``Whitespace to Lisp''), an encoding of Scheme that +allows replacing parentheses with equivalent indentation and inline +colons. See +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification +of SRFI-119}. Some examples: + +@example +display "Hello World!" @result{} (display "Hello World!") +@end example + +@example +define : factorial n @result{} (define (factorial n) + if : zero? n @result{} (if (zero? n) + . 1 @result{} 1 + * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) +@end example + +To execute a file with wisp code, select the language and filename +extension @code{.w} vie @code{guile --language=wisp -x .w}. + +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. + + @node SRFI-171 @subsection Transducers @cindex SRFI-171 @@ -5705,6 +5735,7 @@ left-to-right, due to how transducers are initiated. * SRFI-171 Helpers:: Utilities for writing your own transducers @end menu + @node SRFI-171 General Discussion @subsubsection SRFI-171 General Discussion @cindex transducers discussion diff --git a/module/language/wisp.scm b/module/language/wisp.scm new file mode 100644 index 000000000..b4e885eec --- /dev/null +++ b/module/language/wisp.scm @@ -0,0 +1,784 @@ +;;; Wisp + +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;;; Commentary: + +;; Scheme-only implementation of a wisp-preprocessor which output a +;; scheme code tree to feed to a scheme interpreter instead of a +;; preprocessed file. + +;; Limitations: +;; - in some cases the source line information is missing in backtraces. +;; check for set-source-property! + +;;; Code: + +(define-module (language wisp) + #:export (wisp-scheme-read-chunk wisp-scheme-read-all + wisp-scheme-read-file-chunk wisp-scheme-read-file + wisp-scheme-read-string) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11); for let-values + #:use-module (srfi srfi-9); for records + #:use-module (ice-9 rw); for write-string/partial + #:use-module (ice-9 match)) + +;; use curly-infix by default +(eval-when (expand load eval) + (read-enable 'curly-infix)) + + +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define make-line list) + +(define (line-indent line) + (car line)) + +(define (line-real-indent line) + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." + (let ((indent (line-indent line))) + (if (= -1 indent) + 0 + indent))) + +(define (line-code line) + "Strip the indentation markers from the beginning of the line and preserve source-properties" + (let ((code (cdr line))) + ;; propagate source properties + (when (not (null? code)) + (set-source-properties! code (source-properties line))) + code)) + +;; literal values I need +(define readcolon + (string->symbol ":")) + +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") +;; define an intermediate dot replacement with UUID to avoid clashes. +(define repr-dot ; . + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) + +;; allow using reader additions as the first element on a line to prefix the list +(define repr-quote ; ' + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) +(define repr-unquote ; , + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) +(define repr-quasiquote ; ` + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) +(define repr-unquote-splicing ; ,@ + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) + +(define repr-syntax ; #' + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) +(define repr-unsyntax ; #, + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) +(define repr-quasisyntax ; #` + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) +(define repr-unsyntax-splicing ; #,@ + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) + +;; TODO: wrap the reader to return the repr of the syntax reader +;; additions + +(define (match-charlist-to-repr charlist) + (let + ((chlist (reverse charlist))) + (cond + ((equal? chlist (list #\.)) + repr-dot) + ((equal? chlist (list #\')) + repr-quote) + ((equal? chlist (list #\,)) + repr-unquote) + ((equal? chlist (list #\`)) + repr-quasiquote) + ((equal? chlist (list #\, #\@)) + repr-unquote-splicing) + ((equal? chlist (list #\# #\')) + repr-syntax) + ((equal? chlist (list #\# #\,)) + repr-unsyntax) + ((equal? chlist (list #\# #\`)) + repr-quasisyntax) + ((equal? chlist (list #\# #\, #\@)) + repr-unsyntax-splicing) + (else + #f)))) + +(define (wisp-read port) + "wrap read to catch list prefixes." + (let ((prefix-maxlen 4)) + (let longpeek + ((peeked '()) + (repr-symbol #f)) + (cond + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) + (if repr-symbol ; found a special symbol, return it. + repr-symbol + (let unpeek + ((remaining peeked)) + (cond + ((equal? '() remaining) + (read port)); let read to the work + (else + (unread-char (car remaining) port) + (unpeek (cdr remaining))))))) + (else + (let* + ((next-char (read-char port)) + (peeked (cons next-char peeked))) + (longpeek + peeked + (match-charlist-to-repr peeked)))))))) + + + +(define (line-continues? line) + (equal? repr-dot (car (line-code line)))) + +(define (line-only-colon? line) + (and + (equal? ":" (car (line-code line))) + (null? (cdr (line-code line))))) + +(define (line-empty-code? line) + (null? (line-code line))) + +(define (line-empty? line) + (and + ;; if indent is -1, we stripped a comment, so the line was not really empty. + (= 0 (line-indent line)) + (line-empty-code? line))) + +(define (line-strip-continuation line) + (if (line-continues? line) + (apply make-line + (line-indent line) + (cdr (line-code line))) + line)) + +(define (line-strip-indentation-marker line) + "Strip the indentation markers from the beginning of the line for line-finalize without propagating source-properties (those are propagated in a second step)" + (cdr line)) + +(define (indent-level-reduction indentation-levels level select-fun) + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" + (let loop + ((newlevels indentation-levels) + (diff 0)) + (cond + ((= level (car newlevels)) + (select-fun (list diff indentation-levels))) + ((< level (car newlevels)) + (loop + (cdr newlevels) + (1+ diff))) + (else + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "Level ~A not found in the indentation-levels ~A." level indentation-levels)))))))) + +(define (indent-level-difference indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the count + (car x)))) + +(define (indent-reduce-to-level indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the levels + (car (cdr x))))) + +(define (chunk-ends-with-period currentsymbols next-char) + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." + (and (not (null? currentsymbols)) + (equal? #\newline next-char) + (equal? repr-dot + (list-ref currentsymbols (- (length currentsymbols) 1))))) + + +(define (wisp-scheme-read-chunk-lines port) + (let loop + ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (in-indent? #t) + (in-underscoreindent? (equal? #\_ (peek-char port))) + (in-comment? #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) + (cond + ((>= emptylines 2) + ;; the chunk end has to be checked + ;; before we look for new chars in the + ;; port to make execution in the REPL + ;; after two empty lines work + ;; (otherwise it shows one more line). + indent-and-symbols) + (else + (let ((next-char (peek-char port))) + (cond + ((eof-object? next-char) + (let ((line (apply make-line currentindent currentsymbols))) + (set-source-property! line 'filename (port-filename port)) + (set-source-property! line 'line (port-line port)) + (append indent-and-symbols (list line)))) + ((and in-indent? (zero? currentindent) (not in-comment?) (not (null? indent-and-symbols)) (not in-underscoreindent?) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) + (append indent-and-symbols)); top-level form ends chunk + ((chunk-ends-with-period currentsymbols next-char) + ;; the line ends with a period. This is forbidden in + ;; SRFI-119. Use it to end the line in the REPL without + ;; showing continuation dots (...). + (append indent-and-symbols (list (apply make-line currentindent (drop-right currentsymbols 1))))) + ((and in-indent? (equal? #\space next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ((and in-underscoreindent? (equal? #\_ next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #t ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ;; any char but whitespace *after* underscoreindent is + ;; an error. This is stricter than the current wisp + ;; syntax definition. TODO: Fix the definition. Better + ;; start too strict. FIXME: breaks on lines with only + ;; underscores which should be empty lines. + ((and in-underscoreindent? (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))))) + ((equal? #\newline next-char) + (read-char port); remove the newline + (let* + ;; distinguish pure whitespace lines and lines + ;; with comment by giving the former zero + ;; indent. Lines with a comment at zero indent + ;; get indent -1 for the same reason - meaning + ;; not actually empty. + ((indent + (cond + (in-comment? + (if (= 0 currentindent); specialcase + -1 + currentindent)) + ((not (null? currentsymbols)); pure whitespace + currentindent) + (else + 0))) + (parsedline (apply make-line indent currentsymbols)) + (emptylines + (if (not (line-empty? parsedline)) + 0 + (1+ emptylines)))) + (when (not (= 0 (length (line-code parsedline)))) + ;; set the source properties to parsedline so we can try to add them later. + (set-source-property! parsedline 'filename (port-filename port)) + (set-source-property! parsedline 'line (port-line port))) + ;; TODO: If the line is empty. Either do it here and do not add it, just + ;; increment the empty line counter, or strip it later. Replace indent + ;; -1 by indent 0 afterwards. + (loop + (append indent-and-symbols (list parsedline)) + #t ; in-indent? + (if (<= 2 emptylines) + #f ; chunk ends here + (equal? #\_ (peek-char port))); are we in underscore indent? + #f ; in-comment? + 0 + '() + emptylines))) + ((equal? #t in-comment?) + (read-char port); remove one comment character + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent + (read-char port); remove char + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + currentsymbols + emptylines)) + ;; | cludge to appease the former wisp parser + ;; | used for bootstrapping which has a + ;; v problem with the literal comment char + ((equal? (string-ref ";" 0) next-char) + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + (else ; use the reader + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + ;; this also takes care of the hashbang and leading comments. + (append currentsymbols (list (wisp-read port))) + emptylines)))))))) + + +(define (line-code-replace-inline-colons line) + "Replace inline colons by opening parens which close at the end of the line" + ;; format #t "replace inline colons for line ~A\n" line + (let loop + ((processed '()) + (unprocessed line)) + (cond + ((null? unprocessed) + ;; format #t "inline-colons processed line: ~A\n" processed + processed) + ;; replace : . with nothing + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) + (loop + (append processed + (loop '() (cdr (cdr unprocessed)))) + '())) + ((equal? readcolon (car unprocessed)) + (loop + (append processed + (list (loop '() (cdr unprocessed)))) + '())) + (else + (loop + (append processed + (list (car unprocessed))) + (cdr unprocessed)))))) + +(define (line-replace-inline-colons line) + (cons + (line-indent line) + (line-code-replace-inline-colons (line-code line)))) + +(define (line-strip-lone-colon line) + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." + (if (equal? (line-code line) (list readcolon)) + (make-line (line-indent line)) + line)) + +(define (line-finalize line) + "Process all wisp-specific information in a line and strip it" + (let ((l (line-code-replace-inline-colons + (line-strip-indentation-marker + (line-strip-lone-colon + (line-strip-continuation line)))))) + (when (not (null? (source-properties line))) + (catch #t + (lambda () + (set-source-properties! l (source-properties line))) + (lambda (key . arguments) + #f))) + l)) + +(define (wisp-add-source-properties-from source target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) + +(define (wisp-add-source-properties-from/when-required source target) + "Copy the source properties if target has none." + (if (null? (source-properties target)) + (wisp-add-source-properties-from source target) + target)) + +(define (wisp-propagate-source-properties code) + "Propagate the source properties from the sourrounding list into every part of the code." + (let loop + ((processed '()) + (unprocessed code)) + (cond + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) + unprocessed) + ((and (pair? unprocessed) (not (list? unprocessed))) + (cons + (wisp-propagate-source-properties (car unprocessed)) + (wisp-propagate-source-properties (cdr unprocessed)))) + ((null? unprocessed) + processed) + (else + (let ((line (car unprocessed))) + (wisp-add-source-properties-from/when-required line unprocessed) + (wisp-add-source-properties-from/when-required code unprocessed) + (wisp-add-source-properties-from/when-required unprocessed line) + (wisp-add-source-properties-from/when-required unprocessed code) + (let ((processed (append processed (list (wisp-propagate-source-properties line))))) + ;; must propagate from line, because unprocessed and code can be null, then they cannot keep source-properties. + (wisp-add-source-properties-from/when-required line processed) + (loop processed + (cdr unprocessed)))))))) + +(define* (wisp-scheme-indentation-to-parens lines) + "Add parentheses to lines and remove the indentation markers" + (when + (and + (not (null? lines)) + (not (line-empty-code? (car lines))) + (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment + (if (= 1 (line-real-indent (car lines))) + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands + (set! lines + (cons + (cons 0 (cdr (car lines))) + (cdr lines))) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" + (car lines))))))) + (let loop + ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* + ((current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (make-line 0))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (make-line 0))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) + ;; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ;; . processed current-line next-line unprocessed indentation-levels current-indentation + (cond + ;; the real end: this is reported to the outside world. + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) + ;; reverse the processed lines, because I use cons. + processed) + ;; the recursion end-condition + ((and (null? unprocessed)) + ;; this is the last step. Nothing more to do except + ;; for rolling up the indentation levels. return the + ;; new processed and unprocessed lists: this is a + ;; side-recursion + (values processed unprocessed)) + ((null? indentation-levels) + (raise-exception + (make-exception-from-throw + 'wisp-programming-error + (list + "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")))) + (else ; now we come to the line-comparisons and indentation-counting. + (cond + ((line-empty-code? current-line) + ;; We cannot process indentation without + ;; code. Just switch to the next line. This should + ;; only happen at the start of the recursion. + (loop + processed + (cdr unprocessed) + indentation-levels)) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) + ;; take out the next-line from unprocessed. + (loop + processed + (cons current-line + (cdr (cdr unprocessed))) + indentation-levels)) + ((> current-indentation current-line-indentation) + ;; this just steps back one level via the side-recursion. + (let ((previous-indentation (car (cdr indentation-levels)))) + (if (<= current-line-indentation previous-indentation) + (values processed unprocessed) + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. + (let ((linenumber (- (length lines) (length unprocessed)))) + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + (cdr indentation-levels))))))) + ((= current-indentation current-line-indentation) + (let + ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) + (cond + ((>= current-line-indentation next-line-indentation) + ;; simple recursiive step to the next line + (loop + (append processed + (if (line-continues? current-line) + line + (wisp-add-source-properties-from line (list line)))) + (cdr unprocessed); recursion here + indentation-levels)) + ((< current-line-indentation next-line-indentation) + ;; side-recursion via a sublist + (let-values + (((sub-processed sub-unprocessed) + (loop + line + (cdr unprocessed); recursion here + indentation-levels))) + (loop + (append processed (list sub-processed)) + sub-unprocessed ; simply use the recursion from the sub-recursion + indentation-levels)))))) + ((< current-indentation current-line-indentation) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + indentation-levels))) + (else + (raise-exception + (make-exception-from-throw + 'wisp-not-implemented + (list + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))))) + + +(define (wisp-scheme-replace-inline-colons lines) + "Replace inline colons by opening parens which close at the end of the line" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (list (line-replace-inline-colons (car unprocessed)))) + (cdr unprocessed))))) + + +(define (wisp-scheme-strip-indentation-markers lines) + "Strip the indentation markers from the beginning of the lines" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (cdr (car unprocessed))) + (cdr unprocessed))))) + +(define (wisp-unescape-underscore-and-colon code) + "replace \\_ and \\: by _ and :" + (wisp-add-source-properties-from/when-required + code + (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code)))) + + +(define (wisp-replace-empty-eof code) + "replace ((#<eof>)) by ()" + ;; This is a hack which fixes a bug when the + ;; parser hits files with only hashbang and comments. + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) + (wisp-add-source-properties-from code (list)) + code)) + + +(define (wisp-replace-paren-quotation-repr code) + "Replace lists starting with a quotation symbol by + quoted lists." + (wisp-add-source-properties-from/when-required + code + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a)))) + +(define (wisp-make-improper code) + "Turn (a #{.}# b) into the correct (a . b). + +read called on a single dot creates a variable named #{.}# (|.| +in r7rs). Due to parsing the indentation before the list +structure is known, the reader cannot create improper lists +when it reads a dot. So we have to take another pass over the +code to recreate the improper lists. + +Match is awesome!" + (define is-proper? #t) + ;; local alias + (define (add-prop/req form) + (wisp-add-source-properties-from/when-required code form)) + (wisp-add-source-properties-from/when-required + code + (let + ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (set! is-proper? #f) + (wisp-add-source-properties-from/when-required + code + (append (map wisp-make-improper (map add-prop/req a)) + (cons (wisp-make-improper (add-prop/req b)) + (wisp-make-improper (add-prop/req c)))))) + ((a ...) + (add-prop/req + (map wisp-make-improper (map add-prop/req a)))) + (a + a)))) + (define (syntax-error li msg) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) + (if is-proper? + improper + (let check + ((tocheck improper)) + (match tocheck + ;; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ;; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ;; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ;; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ;; more complex pairs + ((? pair? a) + (let + ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a + a))))))) + +(define (wisp-scheme-read-chunk port) + "Read and parse one chunk of wisp-code" + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\#)) + (let ((lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines))))))))) + +(define (wisp-scheme-read-all port) + "Read all chunks from the given port" + (let loop + ((tokens '())) + (cond + ((eof-object? (peek-char port)) + tokens) + (else + (loop + (append tokens (wisp-scheme-read-chunk port))))))) + +(define (wisp-scheme-read-file path) + (call-with-input-file path wisp-scheme-read-all)) + +(define (wisp-scheme-read-file-chunk path) + (call-with-input-file path wisp-scheme-read-chunk)) + +(define (wisp-scheme-read-string str) + (call-with-input-string str wisp-scheme-read-all)) + +(define (wisp-scheme-read-string-chunk str) + (call-with-input-string str wisp-scheme-read-chunk)) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm new file mode 100644 index 000000000..1efd3e8b2 --- /dev/null +++ b/module/language/wisp/spec.scm @@ -0,0 +1,73 @@ +;;; Language interface for Wisp in Guile + +;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/?p=nacre:guile-sweet.git;a=blob;f=sweet/spec.scm;hb=ae306867e371cb4b56e00bb60a50d9a0b8353109 + +(define-module (language wisp spec) + #:use-module (language wisp) + #:use-module (system base compile) + #:use-module (system base language) + #:use-module (language scheme compile-tree-il) + #:use-module (language scheme decompile-tree-il) + #:export (wisp)) + +;;; +;;; Language definition +;;; + + +(define (read-one-wisp-sexp port env) + ;; Allow using "# foo" as #(foo). + ;; Don't use the globally-acting read-hash-extend, because this + ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme. + ;; Instead, use fluids to temporarily add the extension. + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\# )) + ;; Read Wisp files as UTF-8, to support non-ASCII characters. + ;; TODO: would be nice to support ';; coding: whatever' lines + ;; like in parenthese-y Scheme. + (set-port-encoding! port "UTF-8") + (if (eof-object? (peek-char port)) + (read-char port) ; return eof: we’re done + (let ((chunk (wisp-scheme-read-chunk port))) + (and (not (null? chunk)) ; <---- XXX: maybe (pair? chunk) + (car chunk)))))) + +(define-language wisp + #:title "Wisp Scheme Syntax. See SRFI-119 for details" + ; . #:reader read-one-wisp-sexp + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; + #:compilers `((tree-il . ,compile-tree-il)) + #:decompilers `((tree-il . ,decompile-tree-il)) + #:evaluator (lambda (x module) (primitive-eval x)) + #:printer write ; TODO: backtransform to wisp? Use source-properties? + #:make-default-environment + (lambda () + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', + ;; `fluid-set!', etc. don't have any effect in the current environment. + (let ((m (make-fresh-user-module))) + ;; Provide a separate `current-reader' fluid so that + ;; compile-time changes to `current-reader' are + ;; limited to the current compilation unit. + (module-define! m 'current-reader (make-fluid)) + m))) + + + diff --git a/test-suite/Makefile.am b/test-suite/Makefile.am index 81e63bce2..247d97746 100644 --- a/test-suite/Makefile.am +++ b/test-suite/Makefile.am @@ -162,7 +162,8 @@ SCM_TESTS = tests/00-initial-env.test \ tests/srfi-98.test \ tests/srfi-105.test \ tests/srfi-111.test \ - tests/srfi-171.test \ + tests/srfi-119.test \ + tests/srfi-171.test \ tests/srfi-4.test \ tests/srfi-9.test \ tests/statprof.test \ diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test new file mode 100644 index 000000000..f4a19a0a7 --- /dev/null +++ b/test-suite/tests/srfi-119.test @@ -0,0 +1,89 @@ +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*- +;;;; +;;;; Copyright (C) 2023 Free Software Foundation, Inc. +;;;; +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +(define-module (test-srfi-119) + #:use-module (test-suite lib) + #:use-module (srfi srfi-1) + #:use-module (language wisp)) + +(define (read-string s) + (with-input-from-string s read)) + +(define (with-read-options opts thunk) + (let ((saved-options (read-options))) + (dynamic-wind + (lambda () + (read-options opts)) + thunk + (lambda () + (read-options saved-options))))) + +(define (wisp->list str) + (wisp-scheme-read-string str)) + +(with-test-prefix "wisp-read-simple" + (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) + (pass-if (equal? (wisp->list ". 5") '(5))) + (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) +(with-test-prefix "wisp-read-complex" + (pass-if (equal? (wisp->list " +a b c d e + . f g h + . i j k + +concat \"I want \" + getwish from me + . \" - \" username +") '( +(a b c d e + f g h + i j k) + +(concat "I want " + (getwish from me) + " - " username)))) + + (pass-if (equal? (wisp->list " +define : a b c +_ d e +___ f +___ g h +__ . i + +define : _ +_ display \"hello\n\" + +\\_") '( +(define (a b c) + (d e + (f) + (g h) + i)) + +(define (_) + (display "hello\n")) + +(_)))) + + ;; nesting with pairs + (pass-if (equal? (wisp->list "1 . 2\n3 4\n 5 . 6") + '((1 . 2)(3 4 (5 . 6)))))) + +(with-test-prefix "wisp-source-properties" + (pass-if (not (find null? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6"))))) + (pass-if (not (find null? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))))) -- 2.41.0 [-- Attachment #1.3: Type: text/plain, Size: 101 bytes --] Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-08-14 20:11 ` Dr. Arne Babenhauserheide @ 2023-08-14 20:30 ` Dr. Arne Babenhauserheide 2023-08-14 22:43 ` Dr. Arne Babenhauserheide 2023-08-18 10:29 ` Ludovic Courtès 2 siblings, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-08-14 20:30 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Maxime Devos, Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 132490 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > Attached is a new squashed patch. I’ll send another email with only the > commits for the individual changes on top of the original squashed patch > to avoid tripping up tools that extract diffs. This is the promised email with just the changes compared to the original squashed patch :-) I tried to create atomic changes, but the indentation change mixed a few together that I did not manage to separate (I did that indentation change too early and didn’t commit in time — I’m sorry for that). To minimize the impact I added a last change including just a diff without whitespace changes (-w). It starts with DIFF_WITHOUT_WHITESPACE I hope this simplifies reviewing for you! Best wishes, Arne From ec1d873871040a7bf99cc8f0ab940e09fd76977b Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Mon, 12 Jun 2023 01:22:56 +0200 Subject: [PATCH 02/11] SRFI-119: add new files to Makefile.am and bootstrap.am * am/bootstrap.am (SOURCES): add language/wisp.scm and language/wisp/spec.scm * test-suite/Makefile.am (SCM_TESTS) add tests/srfi-119.test --- am/bootstrap.am | 3 +++ test-suite/Makefile.am | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/am/bootstrap.am b/am/bootstrap.am index ff0d1799e..80a8dcdde 100644 --- a/am/bootstrap.am +++ b/am/bootstrap.am @@ -393,6 +393,9 @@ SOURCES = \ \ system/syntax.scm \ \ + language/wisp.scm \ + language/wisp/spec.scm \ + \ system/xref.scm \ \ sxml/apply-templates.scm \ diff --git a/test-suite/Makefile.am b/test-suite/Makefile.am index 81e63bce2..247d97746 100644 --- a/test-suite/Makefile.am +++ b/test-suite/Makefile.am @@ -162,7 +162,8 @@ SCM_TESTS = tests/00-initial-env.test \ tests/srfi-98.test \ tests/srfi-105.test \ tests/srfi-111.test \ - tests/srfi-171.test \ + tests/srfi-119.test \ + tests/srfi-171.test \ tests/srfi-4.test \ tests/srfi-9.test \ tests/statprof.test \ -- 2.41.0 From c07a1643ca4df87a552abd32cc00d80741ef8e17 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Thu, 10 Aug 2023 08:32:17 +0200 Subject: [PATCH 03/11] SRFI-119: change license of language/wisp/spec to LGPLv3+ * module/language/wisp/spec.scm: changed license. This was changed from LGPLv3+ to MIT for inclusion in SRFI-119 and was now reverted back to LGPLv3+. Permission granted by Maxime Devos who had done changes in the MIT version. --- module/language/wisp/spec.scm | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index 477036c71..821033432 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -6,25 +6,20 @@ ;;; Copyright (C) 2014--2023 Arne Babenhauserheide. ;;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> -;;; Permission is hereby granted, free of charge, to any person -;;; obtaining a copy of this software and associated documentation -;;; files (the "Software"), to deal in the Software without -;;; restriction, including without limitation the rights to use, copy, -;;; modify, merge, publish, distribute, sublicense, and/or sell copies -;;; of the Software, and to permit persons to whom the Software is -;;; furnished to do so, subject to the following conditions: -;;; -;;; The above copyright notice and this permission notice shall be -;;; included in all copies or substantial portions of the Software. -;;; -;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS -;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN -;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -;;; SOFTWARE. +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + ; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm (define-module (language wisp spec) -- 2.41.0 From 76bd2f42d3a568453981ce8f60f6b06bfc23ccf5 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sat, 12 Aug 2023 22:06:50 +0200 Subject: [PATCH 04/11] Polish srfi-119 documentation * doc/ref/srfi-modules.texi (srfi-119): fix capitalization, improve wording, and use two spaces after period for navigation in Emacs. --- doc/ref/srfi-modules.texi | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index e9e64bea8..5b82f8070 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -64,7 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. -* SRFI-119:: Wisp: simpler indentation-sensitive scheme. +* SRFI-119:: Wisp: simpler indentation-sensitive Scheme. * SRFI-171:: Transducers @end menu @@ -5664,13 +5664,14 @@ Set the contents of @var{box} to @var{value}. @end deffn @node SRFI-119 -@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme. +@subsection SRFI-119 Wisp: simpler indentation-sensitive Scheme. @cindex SRFI-119 @cindex wisp -The languages shipped in Guile include SRFI-119 (wisp), an encoding of -Scheme that allows replacing parentheses with equivalent indentation and -inline colons. See +The languages shipped in Guile include SRFI-119, also referred to as +@dfn{Wisp} (for ``Whitespace to Lisp''), an encoding of Scheme that +allows replacing parentheses with equivalent indentation and inline +colons. See @uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification of SRFI-119}. Some examples: -- 2.41.0 From 3cc2679d3e2f11c67d52d9c54e8ec66030697006 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sat, 12 Aug 2023 22:08:54 +0200 Subject: [PATCH 05/11] SRFI-119 spec: fix leading comments * module/language/wisp/spec.scm (comments): fix capitalization, improve wording, and use two spaces after period for navigation in Emacs. --- module/language/wisp/spec.scm | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index 821033432..fde08b429 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -1,10 +1,8 @@ -;; Language interface for Wisp in Guile +;;; Language interface for Wisp in Guile -;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm - -;;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria -;;; Copyright (C) 2014--2023 Arne Babenhauserheide. -;;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> +;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> ;;;; This library is free software; you can redistribute it and/or ;;;; modify it under the terms of the GNU Lesser General Public @@ -20,8 +18,8 @@ ;;;; License along with this library; if not, write to the Free Software ;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +;; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/?p=nacre:guile-sweet.git;a=blob;f=sweet/spec.scm;hb=ae306867e371cb4b56e00bb60a50d9a0b8353109 -; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm (define-module (language wisp spec) #:use-module (language wisp) #:use-module (system base compile) -- 2.41.0 From 8c88bdea77ba33ced9e449165c6ad939f3f8c388 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sat, 12 Aug 2023 22:10:03 +0200 Subject: [PATCH 06/11] SRFI-119 (wisp): review fixes * module/language/wisp.scm (comments): use ;; for all non-margin comments * module/language/wisp.scm (indentation): auto-indent cleanly * module/language/wisp.scm (indent-level-reduction, indent-level-reduction, wisp-scheme-indentation-to-parens, wisp-make-improper): use raise-exception instead of raw throw * module/language/wisp.scm (wisp-scheme-read-chunk-lines): use conventional variable naming in-indent? instead of inindent, also in-underscoreindent? in-comment? * module/language/wisp.scm (top-level): guard read-enable curly-infix with eval-when * module/language/wisp.scm (make-line): new function, alias of list. Change: used as apply make-line indent code (instead of append) --- module/language/wisp.scm | 1225 +++++++++++++++++++------------------- 1 file changed, 604 insertions(+), 621 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index 7a12e126a..acc1f0725 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -31,643 +31,627 @@ ;;; Code: (define-module (language wisp) - #:export (wisp-scheme-read-chunk wisp-scheme-read-all - wisp-scheme-read-file-chunk wisp-scheme-read-file - wisp-scheme-read-string)) + #:export (wisp-scheme-read-chunk wisp-scheme-read-all + wisp-scheme-read-file-chunk wisp-scheme-read-file + wisp-scheme-read-string) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11); for let-values + #:use-module (srfi srfi-9); for records + #:use-module (ice-9 rw); for write-string/partial + #:use-module (ice-9 match)) -; use curly-infix by default -(read-enable 'curly-infix) - -(use-modules - (srfi srfi-1) - (srfi srfi-11); for let-values - (ice-9 rw); for write-string/partial - (ice-9 match)) +;; use curly-infix by default +(eval-when (expand load eval) + (read-enable 'curly-infix)) ;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define make-line list) + (define (line-indent line) - (car line)) + (car line)) (define (line-real-indent line) - "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." - (let (( indent (line-indent line))) - (if (= -1 indent) - 0 - indent))) + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." + (let ((indent (line-indent line))) + (if (= -1 indent) + 0 + indent))) (define (line-code line) - (let ((code (cdr line))) - ; propagate source properties - (when (not (null? code)) - (set-source-properties! code (source-properties line))) - code)) + "Strip the indentation markers from the beginning of the line and preserve source-properties" + (let ((code (cdr line))) + ;; propagate source properties + (when (not (null? code)) + (set-source-properties! code (source-properties line))) + code)) -; literal values I need -(define readcolon - (string->symbol ":")) +;; literal values I need +(define readcolon + (string->symbol ":")) (define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") -; define an intermediate dot replacement with UUID to avoid clashes. +;; define an intermediate dot replacement with UUID to avoid clashes. (define repr-dot ; . - (string->symbol (string-append "REPR-DOT-" wisp-uuid))) + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) -; allow using reader additions as the first element on a line to prefix the list +;; allow using reader additions as the first element on a line to prefix the list (define repr-quote ; ' - (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) (define repr-unquote ; , - (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) (define repr-quasiquote ; ` - (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) (define repr-unquote-splicing ; ,@ - (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) (define repr-syntax ; #' - (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) (define repr-unsyntax ; #, - (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) (define repr-quasisyntax ; #` - (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) (define repr-unsyntax-splicing ; #,@ - (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) -; TODO: wrap the reader to return the repr of the syntax reader -; additions +;; TODO: wrap the reader to return the repr of the syntax reader +;; additions (define (match-charlist-to-repr charlist) - (let - ((chlist (reverse charlist))) - (cond - ((equal? chlist (list #\.)) - repr-dot) - ((equal? chlist (list #\')) - repr-quote) - ((equal? chlist (list #\,)) - repr-unquote) - ((equal? chlist (list #\`)) - repr-quasiquote) - ((equal? chlist (list #\, #\@)) - repr-unquote-splicing) - ((equal? chlist (list #\# #\')) - repr-syntax) - ((equal? chlist (list #\# #\,)) - repr-unsyntax) - ((equal? chlist (list #\# #\`)) - repr-quasisyntax) - ((equal? chlist (list #\# #\, #\@)) - repr-unsyntax-splicing) - (else - #f)))) + (let + ((chlist (reverse charlist))) + (cond + ((equal? chlist (list #\.)) + repr-dot) + ((equal? chlist (list #\')) + repr-quote) + ((equal? chlist (list #\,)) + repr-unquote) + ((equal? chlist (list #\`)) + repr-quasiquote) + ((equal? chlist (list #\, #\@)) + repr-unquote-splicing) + ((equal? chlist (list #\# #\')) + repr-syntax) + ((equal? chlist (list #\# #\,)) + repr-unsyntax) + ((equal? chlist (list #\# #\`)) + repr-quasisyntax) + ((equal? chlist (list #\# #\, #\@)) + repr-unsyntax-splicing) + (else + #f)))) (define (wisp-read port) - "wrap read to catch list prefixes." - (let ((prefix-maxlen 4)) - (let longpeek - ((peeked '()) - (repr-symbol #f)) - (cond - ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) - (if repr-symbol ; found a special symbol, return it. - repr-symbol - (let unpeek - ((remaining peeked)) - (cond - ((equal? '() remaining) - (read port)); let read to the work - (else - (unread-char (car remaining) port) - (unpeek (cdr remaining))))))) - (else - (let* - ((next-char (read-char port)) - (peeked (cons next-char peeked))) - (longpeek - peeked - (match-charlist-to-repr peeked)))))))) + "wrap read to catch list prefixes." + (let ((prefix-maxlen 4)) + (let longpeek + ((peeked '()) + (repr-symbol #f)) + (cond + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) + (if repr-symbol ; found a special symbol, return it. + repr-symbol + (let unpeek + ((remaining peeked)) + (cond + ((equal? '() remaining) + (read port)); let read to the work + (else + (unread-char (car remaining) port) + (unpeek (cdr remaining))))))) + (else + (let* + ((next-char (read-char port)) + (peeked (cons next-char peeked))) + (longpeek + peeked + (match-charlist-to-repr peeked)))))))) (define (line-continues? line) - (equal? repr-dot (car (line-code line)))) + (equal? repr-dot (car (line-code line)))) (define (line-only-colon? line) - (and - (equal? ":" (car (line-code line))) - (null? (cdr (line-code line))))) + (and + (equal? ":" (car (line-code line))) + (null? (cdr (line-code line))))) (define (line-empty-code? line) - (null? (line-code line))) + (null? (line-code line))) (define (line-empty? line) - (and - ; if indent is -1, we stripped a comment, so the line was not really empty. - (= 0 (line-indent line)) - (line-empty-code? line))) + (and + ;; if indent is -1, we stripped a comment, so the line was not really empty. + (= 0 (line-indent line)) + (line-empty-code? line))) (define (line-strip-continuation line) - (if (line-continues? line) - (append - (list - (line-indent line)) - (cdr (line-code line))) - line)) + (if (line-continues? line) + (apply make-line + (line-indent line) + (cdr (line-code line))) + line)) (define (line-strip-indentation-marker line) - "Strip the indentation markers from the beginning of the line" - (cdr line)) + "Strip the indentation markers from the beginning of the line for line-finalize without propagating source-properties (those are propagated in a second step)" + (cdr line)) (define (indent-level-reduction indentation-levels level select-fun) - "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" - (let loop - ((newlevels indentation-levels) - (diff 0)) - (cond - ((= level (car newlevels)) - (select-fun (list diff indentation-levels))) - ((< level (car newlevels)) - (loop - (cdr newlevels) - (1+ diff))) - (else - (throw 'wisp-syntax-error "Level ~A not found in the indentation-levels ~A."))))) + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" + (let loop + ((newlevels indentation-levels) + (diff 0)) + (cond + ((= level (car newlevels)) + (select-fun (list diff indentation-levels))) + ((< level (car newlevels)) + (loop + (cdr newlevels) + (1+ diff))) + (else + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "Level ~A not found in the indentation-levels ~A." level indentation-levels)))))))) (define (indent-level-difference indentation-levels level) - "Find how many indentation levels need to be popped off to find the given level." - (indent-level-reduction indentation-levels level - (lambda (x); get the count - (car x)))) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the count + (car x)))) (define (indent-reduce-to-level indentation-levels level) - "Find how many indentation levels need to be popped off to find the given level." - (indent-level-reduction indentation-levels level - (lambda (x); get the levels - (car (cdr x))))) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the levels + (car (cdr x))))) (define (chunk-ends-with-period currentsymbols next-char) - "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." - (and (not (null? currentsymbols)) - (equal? #\newline next-char) - (equal? repr-dot - (list-ref currentsymbols (- (length currentsymbols) 1))))) + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." + (and (not (null? currentsymbols)) + (equal? #\newline next-char) + (equal? repr-dot + (list-ref currentsymbols (- (length currentsymbols) 1))))) + (define (wisp-scheme-read-chunk-lines port) - (let loop - ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) - (inindent #t) - (inunderscoreindent (equal? #\_ (peek-char port))) - (incomment #f) - (currentindent 0) - (currentsymbols '()) - (emptylines 0)) - (cond - ((>= emptylines 2); the chunk end has to be checked - ; before we look for new chars in the - ; port to make execution in the REPL - ; after two empty lines work - ; (otherwise it shows one more line). - indent-and-symbols) - (else - (let ((next-char (peek-char port))) - (cond - ((eof-object? next-char) - (append indent-and-symbols (list (append (list currentindent) currentsymbols)))) - ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) - (append indent-and-symbols)); top-level form ends chunk - ((chunk-ends-with-period currentsymbols next-char) - ; the line ends with a period. This is forbidden in - ; SRFI-119. Use it to end the line in the REPL without - ; showing continuation dots (...). - (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1))))) - ((and inindent (equal? #\space next-char)) - (read-char port); remove char - (loop - indent-and-symbols - #t ; inindent - #f ; inunderscoreindent - #f ; incomment - (1+ currentindent) - currentsymbols - emptylines)) - ((and inunderscoreindent (equal? #\_ next-char)) - (read-char port); remove char - (loop - indent-and-symbols - #t ; inindent - #t ; inunderscoreindent - #f ; incomment - (1+ currentindent) - currentsymbols - emptylines)) - ; any char but whitespace *after* underscoreindent is - ; an error. This is stricter than the current wisp - ; syntax definition. TODO: Fix the definition. Better - ; start too strict. FIXME: breaks on lines with only - ; underscores which should be empty lines. - ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) - (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) - ((equal? #\newline next-char) - (read-char port); remove the newline - ; The following two lines would break the REPL by requiring one char too many. - ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port - ; read-char port ; remove a full \n\r. Damn special cases... - (let* ; distinguish pure whitespace lines and lines - ; with comment by giving the former zero - ; indent. Lines with a comment at zero indent - ; get indent -1 for the same reason - meaning - ; not actually empty. - ((indent - (cond - (incomment - (if (= 0 currentindent); specialcase - -1 - currentindent)) - ((not (null? currentsymbols)); pure whitespace - currentindent) - (else - 0))) - (parsedline (append (list indent) currentsymbols)) - (emptylines - (if (not (line-empty? parsedline)) - 0 - (1+ emptylines)))) - (when (not (= 0 (length parsedline))) - ; set the source properties to parsedline so we can try to add them later. - (set-source-property! parsedline 'filename (port-filename port)) - (set-source-property! parsedline 'line (port-line port))) - ; TODO: If the line is empty. Either do it here and do not add it, just - ; increment the empty line counter, or strip it later. Replace indent - ; -1 by indent 0 afterwards. - (loop - (append indent-and-symbols (list parsedline)) - #t ; inindent - (if (<= 2 emptylines) - #f ; chunk ends here - (equal? #\_ (peek-char port))); are we in underscore indent? - #f ; incomment - 0 - '() - emptylines))) - ((equal? #t incomment) - (read-char port); remove one comment character - (loop - indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #t ; incomment - currentindent - currentsymbols - emptylines)) - ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent - (read-char port); remove char - (loop - indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #f ; incomment - currentindent - currentsymbols - emptylines)) - ; | cludge to appease the former wisp parser - ; | used for bootstrapping which has a - ; v problem with the literal comment char - ((equal? (string-ref ";" 0) next-char) - (loop - indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #t ; incomment - currentindent - currentsymbols - emptylines)) - (else ; use the reader - (loop - indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #f ; incomment - currentindent - ; this also takes care of the hashbang and leading comments. - (append currentsymbols (list (wisp-read port))) - emptylines)))))))) + (let loop + ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (in-indent? #t) + (in-underscoreindent? (equal? #\_ (peek-char port))) + (in-comment? #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) + (cond + ((>= emptylines 2) + ;; the chunk end has to be checked + ;; before we look for new chars in the + ;; port to make execution in the REPL + ;; after two empty lines work + ;; (otherwise it shows one more line). + indent-and-symbols) + (else + (let ((next-char (peek-char port))) + (cond + ((eof-object? next-char) + (append indent-and-symbols (list (apply make-line currentindent currentsymbols)))) + ((and in-indent? (zero? currentindent) (not in-comment?) (not (null? indent-and-symbols)) (not in-underscoreindent?) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) + (append indent-and-symbols)); top-level form ends chunk + ((chunk-ends-with-period currentsymbols next-char) + ;; the line ends with a period. This is forbidden in + ;; SRFI-119. Use it to end the line in the REPL without + ;; showing continuation dots (...). + (append indent-and-symbols (list (apply make-line currentindent (drop-right currentsymbols 1))))) + ((and in-indent? (equal? #\space next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ((and in-underscoreindent? (equal? #\_ next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #t ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ;; any char but whitespace *after* underscoreindent is + ;; an error. This is stricter than the current wisp + ;; syntax definition. TODO: Fix the definition. Better + ;; start too strict. FIXME: breaks on lines with only + ;; underscores which should be empty lines. + ((and in-underscoreindent? (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))))) + ((equal? #\newline next-char) + (read-char port); remove the newline + (let* + ;; distinguish pure whitespace lines and lines + ;; with comment by giving the former zero + ;; indent. Lines with a comment at zero indent + ;; get indent -1 for the same reason - meaning + ;; not actually empty. + ((indent + (cond + (in-comment? + (if (= 0 currentindent); specialcase + -1 + currentindent)) + ((not (null? currentsymbols)); pure whitespace + currentindent) + (else + 0))) + (parsedline (apply make-line indent currentsymbols)) + (emptylines + (if (not (line-empty? parsedline)) + 0 + (1+ emptylines)))) + (when (not (= 0 (length (line-code parsedline)))) + ;; set the source properties to parsedline so we can try to add them later. + (set-source-property! parsedline 'filename (port-filename port)) + (set-source-property! parsedline 'line (port-line port))) + ;; TODO: If the line is empty. Either do it here and do not add it, just + ;; increment the empty line counter, or strip it later. Replace indent + ;; -1 by indent 0 afterwards. + (loop + (append indent-and-symbols (list parsedline)) + #t ; in-indent? + (if (<= 2 emptylines) + #f ; chunk ends here + (equal? #\_ (peek-char port))); are we in underscore indent? + #f ; in-comment? + 0 + '() + emptylines))) + ((equal? #t in-comment?) + (read-char port); remove one comment character + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent + (read-char port); remove char + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + currentsymbols + emptylines)) + ;; | cludge to appease the former wisp parser + ;; | used for bootstrapping which has a + ;; v problem with the literal comment char + ((equal? (string-ref ";" 0) next-char) + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + (else ; use the reader + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + ;; this also takes care of the hashbang and leading comments. + (append currentsymbols (list (wisp-read port))) + emptylines)))))))) (define (line-code-replace-inline-colons line) - "Replace inline colons by opening parens which close at the end of the line" - ; format #t "replace inline colons for line ~A\n" line - (let loop - ((processed '()) - (unprocessed line)) - (cond - ((null? unprocessed) - ; format #t "inline-colons processed line: ~A\n" processed - processed) - ; replace : . with nothing - ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) - (loop - (append processed - (loop '() (cdr (cdr unprocessed)))) - '())) - ((equal? readcolon (car unprocessed)) - (loop - ; FIXME: This should turn unprocessed into a list. - (append processed - (list (loop '() (cdr unprocessed)))) - '())) - (else - (loop - (append processed - (list (car unprocessed))) - (cdr unprocessed)))))) + "Replace inline colons by opening parens which close at the end of the line" + ;; format #t "replace inline colons for line ~A\n" line + (let loop + ((processed '()) + (unprocessed line)) + (cond + ((null? unprocessed) + ;; format #t "inline-colons processed line: ~A\n" processed + processed) + ;; replace : . with nothing + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) + (loop + (append processed + (loop '() (cdr (cdr unprocessed)))) + '())) + ((equal? readcolon (car unprocessed)) + (loop + (append processed + (list (loop '() (cdr unprocessed)))) + '())) + (else + (loop + (append processed + (list (car unprocessed))) + (cdr unprocessed)))))) (define (line-replace-inline-colons line) - (cons - (line-indent line) - (line-code-replace-inline-colons (line-code line)))) + (cons + (line-indent line) + (line-code-replace-inline-colons (line-code line)))) (define (line-strip-lone-colon line) - "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." - (if - (equal? - (line-code line) - (list readcolon)) - (list (line-indent line)) - line)) + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." + (if (equal? (line-code line) (list readcolon)) + (make-line (line-indent line)) + line)) (define (line-finalize line) - "Process all wisp-specific information in a line and strip it" - (let ((l (line-code-replace-inline-colons - (line-strip-indentation-marker - (line-strip-lone-colon - (line-strip-continuation line)))))) - (when (not (null? (source-properties line))) - (catch #t - (lambda () - (set-source-properties! l (source-properties line))) - (lambda (key . arguments) - #f))) - l)) + "Process all wisp-specific information in a line and strip it" + (let ((l (line-code-replace-inline-colons + (line-strip-indentation-marker + (line-strip-lone-colon + (line-strip-continuation line)))))) + (when (not (null? (source-properties line))) + (catch #t + (lambda () + (set-source-properties! l (source-properties line))) + (lambda (key . arguments) + #f))) + l)) (define (wisp-add-source-properties-from source target) - "Copy the source properties from source into the target and return the target." - (catch #t - (lambda () - (set-source-properties! target (source-properties source))) - (lambda (key . arguments) - #f)) - target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) (define (wisp-propagate-source-properties code) - "Propagate the source properties from the sourrounding list into every part of the code." - (let loop - ((processed '()) - (unprocessed code)) - (cond - ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) - unprocessed) - ((and (pair? unprocessed) (not (list? unprocessed))) - (cons - (wisp-propagate-source-properties (car unprocessed)) - (wisp-propagate-source-properties (cdr unprocessed)))) - ((null? unprocessed) - processed) - (else - (let ((line (car unprocessed))) - (if (null? (source-properties unprocessed)) - (wisp-add-source-properties-from line unprocessed) - (wisp-add-source-properties-from unprocessed line)) - (loop - (append processed (list (wisp-propagate-source-properties line))) - (cdr unprocessed))))))) + "Propagate the source properties from the sourrounding list into every part of the code." + (let loop + ((processed '()) + (unprocessed code)) + (cond + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) + unprocessed) + ((and (pair? unprocessed) (not (list? unprocessed))) + (cons + (wisp-propagate-source-properties (car unprocessed)) + (wisp-propagate-source-properties (cdr unprocessed)))) + ((null? unprocessed) + processed) + (else + (let ((line (car unprocessed))) + (if (null? (source-properties unprocessed)) + (wisp-add-source-properties-from line unprocessed) + (wisp-add-source-properties-from unprocessed line)) + (loop + (append processed (list (wisp-propagate-source-properties line))) + (cdr unprocessed))))))) (define* (wisp-scheme-indentation-to-parens lines) - "Add parentheses to lines and remove the indentation markers" - (when - (and - (not (null? lines)) - (not (line-empty-code? (car lines))) - (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment - (if (= 1 (line-real-indent (car lines))) - ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands - (set! lines - (cons - (cons 0 (cdr (car lines))) - (cdr lines))) - (throw 'wisp-syntax-error + "Add parentheses to lines and remove the indentation markers" + (when + (and + (not (null? lines)) + (not (line-empty-code? (car lines))) + (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment + (if (= 1 (line-real-indent (car lines))) + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands + (set! lines + (cons + (cons 0 (cdr (car lines))) + (cdr lines))) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" - (car lines))))) - (let loop - ((processed '()) - (unprocessed lines) - (indentation-levels '(0))) - (let* - ((current-line - (if (<= 1 (length unprocessed)) - (car unprocessed) - (list 0))); empty code - (next-line - (if (<= 2 (length unprocessed)) - (car (cdr unprocessed)) - (list 0))); empty code - (current-indentation - (car indentation-levels)) - (current-line-indentation (line-real-indent current-line))) - ; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" - ; . processed current-line next-line unprocessed indentation-levels current-indentation - (cond - ; the real end: this is reported to the outside world. - ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) - ; display "done\n" - ; reverse the processed lines, because I use cons. - processed) - ; the recursion end-condition - ((and (null? unprocessed)) - ; display "last step\n" - ; this is the last step. Nothing more to do except - ; for rolling up the indentation levels. return the - ; new processed and unprocessed lists: this is a - ; side-recursion - (values processed unprocessed)) - ((null? indentation-levels) - ; display "indentation-levels null\n" - (throw 'wisp-programming-error "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")) - (else ; now we come to the line-comparisons and indentation-counting. - (cond - ((line-empty-code? current-line) - ; display "current-line empty\n" - ; We cannot process indentation without - ; code. Just switch to the next line. This should - ; only happen at the start of the recursion. - ; TODO: Somehow preserve the line-numbers. - (loop - processed - (cdr unprocessed) - indentation-levels)) - ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) - ; display "next-line empty\n" - ; TODO: Somehow preserve the line-numbers. - ; take out the next-line from unprocessed. - (loop - processed - (cons current-line - (cdr (cdr unprocessed))) - indentation-levels)) - ((> current-indentation current-line-indentation) - ; display "current-indent > next-line\n" - ; this just steps back one level via the side-recursion. - (let ((previous-indentation (car (cdr indentation-levels)))) - (if (<= current-line-indentation previous-indentation) - (values processed unprocessed) - (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. - (let ((linenumber (- (length lines) (length unprocessed)))) - (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) - (loop - processed - unprocessed - (cons ; recursion via the indentation-levels - current-line-indentation - (cdr indentation-levels))))))) - ((= current-indentation current-line-indentation) - ; display "current-indent = next-line\n" - (let - ((line (line-finalize current-line)) - (next-line-indentation (line-real-indent next-line))) - (cond - ((>= current-line-indentation next-line-indentation) - ; simple recursiive step to the next line - ; display "current-line-indent >= next-line-indent\n" - (loop - (append processed - (if (line-continues? current-line) - line - (wisp-add-source-properties-from line (list line)))) - (cdr unprocessed); recursion here - indentation-levels)) - ((< current-line-indentation next-line-indentation) - ; display "current-line-indent < next-line-indent\n" - ; format #t "line: ~A\n" line - ; side-recursion via a sublist - (let-values - (((sub-processed sub-unprocessed) - (loop - line - (cdr unprocessed); recursion here - indentation-levels))) - ; format #t "side-recursion:\n sub-processed: ~A\n processed: ~A\n\n" sub-processed processed - (loop - (append processed (list sub-processed)) - sub-unprocessed ; simply use the recursion from the sub-recursion - indentation-levels)))))) - ((< current-indentation current-line-indentation) - ; display "current-indent < next-line\n" - (loop - processed - unprocessed - (cons ; recursion via the indentation-levels - current-line-indentation - indentation-levels))) - (else - (throw 'wisp-not-implemented - (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." - current-line next-line processed))))))))) + (car lines))))))) + (let loop + ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* + ((current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (make-line 0))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (make-line 0))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) + ;; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ;; . processed current-line next-line unprocessed indentation-levels current-indentation + (cond + ;; the real end: this is reported to the outside world. + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) + ;; reverse the processed lines, because I use cons. + processed) + ;; the recursion end-condition + ((and (null? unprocessed)) + ;; this is the last step. Nothing more to do except + ;; for rolling up the indentation levels. return the + ;; new processed and unprocessed lists: this is a + ;; side-recursion + (values processed unprocessed)) + ((null? indentation-levels) + (raise-exception (make-exception-from-throw 'wisp-programming-error (list "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")))) + (else ; now we come to the line-comparisons and indentation-counting. + (cond + ((line-empty-code? current-line) + ;; We cannot process indentation without + ;; code. Just switch to the next line. This should + ;; only happen at the start of the recursion. + (loop + processed + (cdr unprocessed) + indentation-levels)) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) + ;; take out the next-line from unprocessed. + (loop + processed + (cons current-line + (cdr (cdr unprocessed))) + indentation-levels)) + ((> current-indentation current-line-indentation) + ;; this just steps back one level via the side-recursion. + (let ((previous-indentation (car (cdr indentation-levels)))) + (if (<= current-line-indentation previous-indentation) + (values processed unprocessed) + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. + (let ((linenumber (- (length lines) (length unprocessed)))) + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + (cdr indentation-levels))))))) + ((= current-indentation current-line-indentation) + (let + ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) + (cond + ((>= current-line-indentation next-line-indentation) + ;; simple recursiive step to the next line + (loop + (append processed + (if (line-continues? current-line) + line + (wisp-add-source-properties-from line (list line)))) + (cdr unprocessed); recursion here + indentation-levels)) + ((< current-line-indentation next-line-indentation) + ;; side-recursion via a sublist + (let-values + (((sub-processed sub-unprocessed) + (loop + line + (cdr unprocessed); recursion here + indentation-levels))) + (loop + (append processed (list sub-processed)) + sub-unprocessed ; simply use the recursion from the sub-recursion + indentation-levels)))))) + ((< current-indentation current-line-indentation) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + indentation-levels))) + (else + (raise-exception (make-exception-from-throw 'wisp-not-implemented (list + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))))) (define (wisp-scheme-replace-inline-colons lines) - "Replace inline colons by opening parens which close at the end of the line" - (let loop - ((processed '()) - (unprocessed lines)) - (if (null? unprocessed) - processed - (loop - (append processed (list (line-replace-inline-colons (car unprocessed)))) - (cdr unprocessed))))) + "Replace inline colons by opening parens which close at the end of the line" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (list (line-replace-inline-colons (car unprocessed)))) + (cdr unprocessed))))) (define (wisp-scheme-strip-indentation-markers lines) - "Strip the indentation markers from the beginning of the lines" - (let loop - ((processed '()) - (unprocessed lines)) - (if (null? unprocessed) - processed - (loop - (append processed (cdr (car unprocessed))) - (cdr unprocessed))))) + "Strip the indentation markers from the beginning of the lines" + (let loop + ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (cdr (car unprocessed))) + (cdr unprocessed))))) (define (wisp-unescape-underscore-and-colon code) "replace \\_ and \\: by _ and :" (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) - ((eq? code '\:) ':) - ;; Look for symbols like \____ and remove the \. - ((symbol? code) - (let ((as-string (symbol->string code))) - (if (and (>= (string-length as-string) 2) ; at least a single underscore - (char=? (string-ref as-string 0) #\\) - (string-every #\_ (substring as-string 1))) - (string->symbol (substring as-string 1)) - code))) - (#t code))) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code))) (define (wisp-replace-empty-eof code) - "replace ((#<eof>)) by ()" - ; FIXME: Actually this is a hack which fixes a bug when the - ; parser hits files with only hashbang and comments. - (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) - (list) - code)) + "replace ((#<eof>)) by ()" + ;; This is a hack which fixes a bug when the + ;; parser hits files with only hashbang and comments. + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) + (list) + code)) (define (wisp-replace-paren-quotation-repr code) - "Replace lists starting with a quotation symbol by + "Replace lists starting with a quotation symbol by quoted lists." - (match code - (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list - (append - (map wisp-replace-paren-quotation-repr a) - (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) - (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) - (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unquote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) - (append - (map wisp-replace-paren-quotation-repr a) - (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) - (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list - (append - (map wisp-replace-paren-quotation-repr a) - (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) - (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) - (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'syntax (map wisp-replace-paren-quotation-repr a))) - (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) - (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) - (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) - ;; literal array as start of a line: # (a b) c -> (#(a b) c) - ((#\# a ...) - (with-input-from-string ;; hack to defer to read - (string-append "#" - (with-output-to-string - (λ () - (write (map wisp-replace-paren-quotation-repr a) - (current-output-port))))) - read)) - ((a ...) - (map wisp-replace-paren-quotation-repr a)) - (a - a))) + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a))) (define (wisp-make-improper code) - "Turn (a #{.}# b) into the correct (a . b). + "Turn (a #{.}# b) into the correct (a . b). read called on a single dot creates a variable named #{.}# (|.| in r7rs). Due to parsing the indentation before the list @@ -676,86 +660,85 @@ when it reads a dot. So we have to take another pass over the code to recreate the improper lists. Match is awesome!" - (let - ((improper - (match code - ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) - (append (map wisp-make-improper a) - (cons (wisp-make-improper b) (wisp-make-improper c)))) - ((a ...) - (map wisp-make-improper a)) - (a - a)))) - (define (syntax-error li msg) - (throw 'wisp-syntax-error (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))) - (if #t - improper - (let check - ((tocheck improper)) - (match tocheck - ; lists with only one member - (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) - (syntax-error tocheck "list with the period as only member")) - ; list with remaining dot. - ((a ...) - (if (and (member repr-dot a)) - (syntax-error tocheck "leftover period in list") - (map check a))) - ; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? - (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) - (syntax-error tocheck "dot as first element in already improper pair")) - ; simple pair, other way round - ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) - (syntax-error tocheck "dot as last element in already improper pair")) - ; more complex pairs - ((? pair? a) - (let - ((head (drop-right a 1)) - (tail (last-pair a))) - (cond - ((equal? repr-dot (car tail)) - (syntax-error tocheck "equal? repr-dot : car tail")) - ((equal? repr-dot (cdr tail)) - (syntax-error tocheck "equal? repr-dot : cdr tail")) - ((member repr-dot head) - (syntax-error tocheck "member repr-dot head")) - (else - a)))) - (a - a)))))) + (let + ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (append (map wisp-make-improper a) + (cons (wisp-make-improper b) (wisp-make-improper c)))) + ((a ...) + (map wisp-make-improper a)) + (a + a)))) + (define (syntax-error li msg) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) + (if #t + improper + (let check + ((tocheck improper)) + (match tocheck + ;; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ;; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ;; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ;; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ;; more complex pairs + ((? pair? a) + (let + ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a + a)))))) (define (wisp-scheme-read-chunk port) - "Read and parse one chunk of wisp-code" - (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) - (read-hash-extend #\# (lambda args #\#)) - (let ((lines (wisp-scheme-read-chunk-lines port))) - (wisp-make-improper - (wisp-replace-empty-eof - (wisp-unescape-underscore-and-colon - (wisp-replace-paren-quotation-repr - (wisp-propagate-source-properties - (wisp-scheme-indentation-to-parens lines))))))))) + "Read and parse one chunk of wisp-code" + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\#)) + (let ((lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines))))))))) (define (wisp-scheme-read-all port) - "Read all chunks from the given port" - (let loop - ((tokens '())) - (cond - ((eof-object? (peek-char port)) - tokens) - (else - (loop - (append tokens (wisp-scheme-read-chunk port))))))) + "Read all chunks from the given port" + (let loop + ((tokens '())) + (cond + ((eof-object? (peek-char port)) + tokens) + (else + (loop + (append tokens (wisp-scheme-read-chunk port))))))) (define (wisp-scheme-read-file path) - (call-with-input-file path wisp-scheme-read-all)) + (call-with-input-file path wisp-scheme-read-all)) (define (wisp-scheme-read-file-chunk path) - (call-with-input-file path wisp-scheme-read-chunk)) + (call-with-input-file path wisp-scheme-read-chunk)) (define (wisp-scheme-read-string str) - (call-with-input-string str wisp-scheme-read-all)) + (call-with-input-string str wisp-scheme-read-all)) (define (wisp-scheme-read-string-chunk str) - (call-with-input-string str wisp-scheme-read-chunk)) - + (call-with-input-string str wisp-scheme-read-chunk)) -- 2.41.0 From 8bacc9f43c3c5ffe1634a4e3fadda90ce4bdebcc Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sat, 12 Aug 2023 23:13:26 +0200 Subject: [PATCH 07/11] SRFI-119 (wisp): change lang enter message * module/language/wisp/spec.scm (define-language): no period at end, because the actual message in the REPL adds a ! --- module/language/wisp/spec.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index fde08b429..e6dbf1764 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -51,7 +51,7 @@ (car chunk)))))) (define-language wisp - #:title "Wisp Scheme Syntax. See SRFI-119 for details." + #:title "Wisp Scheme Syntax. See SRFI-119 for details" ; . #:reader read-one-wisp-sexp #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; #:compilers `((tree-il . ,compile-tree-il)) -- 2.41.0 From e79472e185027316b8195c8fbe502bd0c819fcaa Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sat, 12 Aug 2023 23:17:57 +0200 Subject: [PATCH 08/11] SRFI-119 (wisp): simplify for review * module/language/wisp/spec.scm (define-language): do not set simple-format as formatter for the reader --- module/language/wisp/spec.scm | 5 ----- 1 file changed, 5 deletions(-) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index e6dbf1764..1efd3e8b2 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -67,11 +67,6 @@ ;; compile-time changes to `current-reader' are ;; limited to the current compilation unit. (module-define! m 'current-reader (make-fluid)) - ;; Default to `simple-format', as is the case until - ;; (ice-9 format) is loaded. This allows - ;; compile-time warnings to be emitted when using - ;; unsupported options. - (module-set! m 'format simple-format) m))) -- 2.41.0 From 5cd837c71683dd5de51837f859450dcdbc99d83e Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sun, 13 Aug 2023 11:35:23 +0200 Subject: [PATCH 09/11] SRFI-119 (wisp): Add source-property tests * module/language/wisp.scm (wisp-add-source-properties-from/when-required): add the source properties from source to target if target has no source-properties. * module/language/wisp.scm (wisp-propagate-source-properties): fix source property propagation * module/language/wisp.scm (wisp-scheme-read-chunk-lines wisp-unescape-underscore-and-colon wisp-unescape-underscore-and-colon wisp-replace-paren-quotation-repr wisp-make-improper): preserve source properties * test-suite/tests/srfi-119.test (top-level): add testgroup wisp-source-properties --- module/language/wisp.scm | 181 +++++++++++++++++++-------------- test-suite/tests/srfi-119.test | 10 +- 2 files changed, 114 insertions(+), 77 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index acc1f0725..812a8bad0 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -234,7 +234,10 @@ (let ((next-char (peek-char port))) (cond ((eof-object? next-char) - (append indent-and-symbols (list (apply make-line currentindent currentsymbols)))) + (let ((line (apply make-line currentindent currentsymbols))) + (set-source-property! line 'filename (port-filename port)) + (set-source-property! line 'line (port-line port)) + (append indent-and-symbols (list line)))) ((and in-indent? (zero? currentindent) (not in-comment?) (not (null? indent-and-symbols)) (not in-underscoreindent?) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) (append indent-and-symbols)); top-level form ends chunk ((chunk-ends-with-period currentsymbols next-char) @@ -414,6 +417,12 @@ #f)) target) +(define (wisp-add-source-properties-from/when-required source target) + "Copy the source properties if target has none." + (if (null? (source-properties target)) + (wisp-add-source-properties-from source target) + target)) + (define (wisp-propagate-source-properties code) "Propagate the source properties from the sourrounding list into every part of the code." (let loop @@ -430,12 +439,15 @@ processed) (else (let ((line (car unprocessed))) - (if (null? (source-properties unprocessed)) - (wisp-add-source-properties-from line unprocessed) - (wisp-add-source-properties-from unprocessed line)) - (loop - (append processed (list (wisp-propagate-source-properties line))) - (cdr unprocessed))))))) + (wisp-add-source-properties-from/when-required line unprocessed) + (wisp-add-source-properties-from/when-required code unprocessed) + (wisp-add-source-properties-from/when-required unprocessed line) + (wisp-add-source-properties-from/when-required unprocessed code) + (let ((processed (append processed (list (wisp-propagate-source-properties line))))) + ;; must propagate from line, because unprocessed and code can be null, then they cannot keep source-properties. + (wisp-add-source-properties-from/when-required line processed) + (loop processed + (cdr unprocessed)))))))) (define* (wisp-scheme-indentation-to-parens lines) "Add parentheses to lines and remove the indentation markers" @@ -580,17 +592,19 @@ (define (wisp-unescape-underscore-and-colon code) "replace \\_ and \\: by _ and :" - (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) - ((eq? code '\:) ':) - ;; Look for symbols like \____ and remove the \. - ((symbol? code) - (let ((as-string (symbol->string code))) - (if (and (>= (string-length as-string) 2) ; at least a single underscore - (char=? (string-ref as-string 0) #\\) - (string-every #\_ (substring as-string 1))) - (string->symbol (substring as-string 1)) - code))) - (#t code))) + (wisp-add-source-properties-from/when-required + code + (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code)))) (define (wisp-replace-empty-eof code) @@ -598,57 +612,59 @@ ;; This is a hack which fixes a bug when the ;; parser hits files with only hashbang and comments. (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) - (list) + (wisp-add-source-properties-from code (list)) code)) (define (wisp-replace-paren-quotation-repr code) "Replace lists starting with a quotation symbol by quoted lists." - (match code - (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list - (append - (map wisp-replace-paren-quotation-repr a) - (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) - (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) - (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unquote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) - (append - (map wisp-replace-paren-quotation-repr a) - (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) - (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) - ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list - (append - (map wisp-replace-paren-quotation-repr a) - (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) - (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) - (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'syntax (map wisp-replace-paren-quotation-repr a))) - (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) - (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) - (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) - (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) - ;; literal array as start of a line: # (a b) c -> (#(a b) c) - ((#\# a ...) - (with-input-from-string ;; hack to defer to read - (string-append "#" - (with-output-to-string - (λ () - (write (map wisp-replace-paren-quotation-repr a) - (current-output-port))))) - read)) - ((a ...) - (map wisp-replace-paren-quotation-repr a)) - (a - a))) + (wisp-add-source-properties-from/when-required + code + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a)))) (define (wisp-make-improper code) "Turn (a #{.}# b) into the correct (a . b). @@ -660,18 +676,6 @@ when it reads a dot. So we have to take another pass over the code to recreate the improper lists. Match is awesome!" - (let - ((improper - (match code - ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) - (append (map wisp-make-improper a) - (cons (wisp-make-improper b) (wisp-make-improper c)))) - ((a ...) - (map wisp-make-improper a)) - (a - a)))) - (define (syntax-error li msg) - (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) (if #t improper (let check @@ -706,7 +710,32 @@ Match is awesome!" (else a)))) (a - a)))))) + ;; local alias + (define (add-prop/req form) + (wisp-add-source-properties-from/when-required code form)) + (wisp-add-source-properties-from/when-required + code + (let + ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (set! is-proper? #f) + (wisp-add-source-properties-from/when-required + code + (append (map wisp-make-improper (map add-prop/req a)) + (cons (wisp-make-improper (add-prop/req b)) + (wisp-make-improper (add-prop/req c)))))) + ((a ...) + (add-prop/req + (map wisp-make-improper (map add-prop/req a)))) + (a + a)))) + (define (syntax-error li msg) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) + a))))))) (define (wisp-scheme-read-chunk port) "Read and parse one chunk of wisp-code" diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test index a888df41d..f4a19a0a7 100644 --- a/test-suite/tests/srfi-119.test +++ b/test-suite/tests/srfi-119.test @@ -78,4 +78,12 @@ _ display \"hello\n\" (define (_) (display "hello\n")) -(_))))) +(_)))) + + ;; nesting with pairs + (pass-if (equal? (wisp->list "1 . 2\n3 4\n 5 . 6") + '((1 . 2)(3 4 (5 . 6)))))) + +(with-test-prefix "wisp-source-properties" + (pass-if (not (find null? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6"))))) + (pass-if (not (find null? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))))) -- 2.41.0 From 7332c2c5bdc0a55a4c51fd8595b004ef768b675a Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sun, 13 Aug 2023 11:39:19 +0200 Subject: [PATCH 10/11] SRFI-119 (wisp): improve indentation * module/language/wisp.scm (indentation): restructure so auto-format creates less indentation. --- module/language/wisp.scm | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index 812a8bad0..506b37ea5 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -170,8 +170,8 @@ (define (line-strip-continuation line) (if (line-continues? line) (apply make-line - (line-indent line) - (cdr (line-code line))) + (line-indent line) + (cdr (line-code line))) line)) (define (line-strip-indentation-marker line) @@ -462,9 +462,12 @@ (cons (cons 0 (cdr (car lines))) (cdr lines))) - (raise-exception (make-exception-from-throw 'wisp-syntax-error (list - (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" - (car lines))))))) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" + (car lines))))))) (let loop ((processed '()) (unprocessed lines) @@ -496,7 +499,11 @@ ;; side-recursion (values processed unprocessed)) ((null? indentation-levels) - (raise-exception (make-exception-from-throw 'wisp-programming-error (list "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")))) + (raise-exception + (make-exception-from-throw + 'wisp-programming-error + (list + "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")))) (else ; now we come to the line-comparisons and indentation-counting. (cond ((line-empty-code? current-line) @@ -562,9 +569,12 @@ current-line-indentation indentation-levels))) (else - (raise-exception (make-exception-from-throw 'wisp-not-implemented (list - (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." - current-line next-line processed))))))))))) + (raise-exception + (make-exception-from-throw + 'wisp-not-implemented + (list + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))))) (define (wisp-scheme-replace-inline-colons lines) -- 2.41.0 From fa76d6d2937da46e445f8cf2eaaa82c14fc5ec4f Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Sun, 13 Aug 2023 11:40:20 +0200 Subject: [PATCH 11/11] SRFI-119 (wisp): stricter syntax checks * module/language/wisp.scm (wisp-make-improper): run the syntax validation for illegal improper lists. --- module/language/wisp.scm | 69 ++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 34 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index 506b37ea5..b4e885eec 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -686,40 +686,7 @@ when it reads a dot. So we have to take another pass over the code to recreate the improper lists. Match is awesome!" - (if #t - improper - (let check - ((tocheck improper)) - (match tocheck - ;; lists with only one member - (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) - (syntax-error tocheck "list with the period as only member")) - ;; list with remaining dot. - ((a ...) - (if (and (member repr-dot a)) - (syntax-error tocheck "leftover period in list") - (map check a))) - ;; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? - (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) - (syntax-error tocheck "dot as first element in already improper pair")) - ;; simple pair, other way round - ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) - (syntax-error tocheck "dot as last element in already improper pair")) - ;; more complex pairs - ((? pair? a) - (let - ((head (drop-right a 1)) - (tail (last-pair a))) - (cond - ((equal? repr-dot (car tail)) - (syntax-error tocheck "equal? repr-dot : car tail")) - ((equal? repr-dot (cdr tail)) - (syntax-error tocheck "equal? repr-dot : cdr tail")) - ((member repr-dot head) - (syntax-error tocheck "member repr-dot head")) - (else - a)))) - (a + (define is-proper? #t) ;; local alias (define (add-prop/req form) (wisp-add-source-properties-from/when-required code form)) @@ -745,6 +712,40 @@ Match is awesome!" (make-exception-from-throw 'wisp-syntax-error (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) + (if is-proper? + improper + (let check + ((tocheck improper)) + (match tocheck + ;; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ;; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ;; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ;; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ;; more complex pairs + ((? pair? a) + (let + ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a a))))))) (define (wisp-scheme-read-chunk port) -- 2.41.0 DIFF_WITHOUT_WHITESPACE 6 files changed, 214 insertions(+), 190 deletions(-) am/bootstrap.am | 3 + doc/ref/srfi-modules.texi | 11 +- module/language/wisp.scm | 329 ++++++++++++++++++++++------------------- module/language/wisp/spec.scm | 50 +++---- test-suite/Makefile.am | 1 + test-suite/tests/srfi-119.test | 10 +- modified am/bootstrap.am @@ -393,6 +393,9 @@ SOURCES = \ \ system/syntax.scm \ \ + language/wisp.scm \ + language/wisp/spec.scm \ + \ system/xref.scm \ \ sxml/apply-templates.scm \ modified doc/ref/srfi-modules.texi @@ -64,7 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. -* SRFI-119:: Wisp: simpler indentation-sensitive scheme. +* SRFI-119:: Wisp: simpler indentation-sensitive Scheme. * SRFI-171:: Transducers @end menu @@ -5664,13 +5664,14 @@ Set the contents of @var{box} to @var{value}. @end deffn @node SRFI-119 -@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme. +@subsection SRFI-119 Wisp: simpler indentation-sensitive Scheme. @cindex SRFI-119 @cindex wisp -The languages shipped in Guile include SRFI-119 (wisp), an encoding of -Scheme that allows replacing parentheses with equivalent indentation and -inline colons. See +The languages shipped in Guile include SRFI-119, also referred to as +@dfn{Wisp} (for ``Whitespace to Lisp''), an encoding of Scheme that +allows replacing parentheses with equivalent indentation and inline +colons. See @uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification of SRFI-119}. Some examples: modified module/language/wisp.scm @@ -33,19 +33,21 @@ (define-module (language wisp) #:export (wisp-scheme-read-chunk wisp-scheme-read-all wisp-scheme-read-file-chunk wisp-scheme-read-file - wisp-scheme-read-string)) + wisp-scheme-read-string) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11); for let-values + #:use-module (srfi srfi-9); for records + #:use-module (ice-9 rw); for write-string/partial + #:use-module (ice-9 match)) -; use curly-infix by default -(read-enable 'curly-infix) - -(use-modules - (srfi srfi-1) - (srfi srfi-11); for let-values - (ice-9 rw); for write-string/partial - (ice-9 match)) +;; use curly-infix by default +(eval-when (expand load eval) + (read-enable 'curly-infix)) ;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define make-line list) + (define (line-indent line) (car line)) @@ -57,22 +59,23 @@ indent))) (define (line-code line) + "Strip the indentation markers from the beginning of the line and preserve source-properties" (let ((code (cdr line))) - ; propagate source properties + ;; propagate source properties (when (not (null? code)) (set-source-properties! code (source-properties line))) code)) -; literal values I need +;; literal values I need (define readcolon (string->symbol ":")) (define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") -; define an intermediate dot replacement with UUID to avoid clashes. +;; define an intermediate dot replacement with UUID to avoid clashes. (define repr-dot ; . (string->symbol (string-append "REPR-DOT-" wisp-uuid))) -; allow using reader additions as the first element on a line to prefix the list +;; allow using reader additions as the first element on a line to prefix the list (define repr-quote ; ' (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) (define repr-unquote ; , @@ -91,8 +94,8 @@ (define repr-unsyntax-splicing ; #,@ (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) -; TODO: wrap the reader to return the repr of the syntax reader -; additions +;; TODO: wrap the reader to return the repr of the syntax reader +;; additions (define (match-charlist-to-repr charlist) (let @@ -160,20 +163,19 @@ (define (line-empty? line) (and - ; if indent is -1, we stripped a comment, so the line was not really empty. + ;; if indent is -1, we stripped a comment, so the line was not really empty. (= 0 (line-indent line)) (line-empty-code? line))) (define (line-strip-continuation line) (if (line-continues? line) - (append - (list - (line-indent line)) + (apply make-line + (line-indent line) (cdr (line-code line))) line)) (define (line-strip-indentation-marker line) - "Strip the indentation markers from the beginning of the line" + "Strip the indentation markers from the beginning of the line for line-finalize without propagating source-properties (those are propagated in a second step)" (cdr line)) (define (indent-level-reduction indentation-levels level select-fun) @@ -189,7 +191,7 @@ (cdr newlevels) (1+ diff))) (else - (throw 'wisp-syntax-error "Level ~A not found in the indentation-levels ~A."))))) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "Level ~A not found in the indentation-levels ~A." level indentation-levels)))))))) (define (indent-level-difference indentation-levels level) "Find how many indentation levels need to be popped off to find the given level." @@ -210,74 +212,77 @@ (equal? repr-dot (list-ref currentsymbols (- (length currentsymbols) 1))))) + (define (wisp-scheme-read-chunk-lines port) (let loop ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) - (inindent #t) - (inunderscoreindent (equal? #\_ (peek-char port))) - (incomment #f) + (in-indent? #t) + (in-underscoreindent? (equal? #\_ (peek-char port))) + (in-comment? #f) (currentindent 0) (currentsymbols '()) (emptylines 0)) (cond - ((>= emptylines 2); the chunk end has to be checked - ; before we look for new chars in the - ; port to make execution in the REPL - ; after two empty lines work - ; (otherwise it shows one more line). + ((>= emptylines 2) + ;; the chunk end has to be checked + ;; before we look for new chars in the + ;; port to make execution in the REPL + ;; after two empty lines work + ;; (otherwise it shows one more line). indent-and-symbols) (else (let ((next-char (peek-char port))) (cond ((eof-object? next-char) - (append indent-and-symbols (list (append (list currentindent) currentsymbols)))) - ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) + (let ((line (apply make-line currentindent currentsymbols))) + (set-source-property! line 'filename (port-filename port)) + (set-source-property! line 'line (port-line port)) + (append indent-and-symbols (list line)))) + ((and in-indent? (zero? currentindent) (not in-comment?) (not (null? indent-and-symbols)) (not in-underscoreindent?) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) (append indent-and-symbols)); top-level form ends chunk ((chunk-ends-with-period currentsymbols next-char) - ; the line ends with a period. This is forbidden in - ; SRFI-119. Use it to end the line in the REPL without - ; showing continuation dots (...). - (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1))))) - ((and inindent (equal? #\space next-char)) + ;; the line ends with a period. This is forbidden in + ;; SRFI-119. Use it to end the line in the REPL without + ;; showing continuation dots (...). + (append indent-and-symbols (list (apply make-line currentindent (drop-right currentsymbols 1))))) + ((and in-indent? (equal? #\space next-char)) (read-char port); remove char (loop indent-and-symbols - #t ; inindent - #f ; inunderscoreindent - #f ; incomment + #t ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? (1+ currentindent) currentsymbols emptylines)) - ((and inunderscoreindent (equal? #\_ next-char)) + ((and in-underscoreindent? (equal? #\_ next-char)) (read-char port); remove char (loop indent-and-symbols - #t ; inindent - #t ; inunderscoreindent - #f ; incomment + #t ; in-indent? + #t ; in-underscoreindent? + #f ; in-comment? (1+ currentindent) currentsymbols emptylines)) - ; any char but whitespace *after* underscoreindent is - ; an error. This is stricter than the current wisp - ; syntax definition. TODO: Fix the definition. Better - ; start too strict. FIXME: breaks on lines with only - ; underscores which should be empty lines. - ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) - (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) + ;; any char but whitespace *after* underscoreindent is + ;; an error. This is stricter than the current wisp + ;; syntax definition. TODO: Fix the definition. Better + ;; start too strict. FIXME: breaks on lines with only + ;; underscores which should be empty lines. + ((and in-underscoreindent? (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))))) ((equal? #\newline next-char) (read-char port); remove the newline - ; The following two lines would break the REPL by requiring one char too many. - ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port - ; read-char port ; remove a full \n\r. Damn special cases... - (let* ; distinguish pure whitespace lines and lines - ; with comment by giving the former zero - ; indent. Lines with a comment at zero indent - ; get indent -1 for the same reason - meaning - ; not actually empty. + (let* + ;; distinguish pure whitespace lines and lines + ;; with comment by giving the former zero + ;; indent. Lines with a comment at zero indent + ;; get indent -1 for the same reason - meaning + ;; not actually empty. ((indent (cond - (incomment + (in-comment? (if (= 0 currentindent); specialcase -1 currentindent)) @@ -285,35 +290,35 @@ currentindent) (else 0))) - (parsedline (append (list indent) currentsymbols)) + (parsedline (apply make-line indent currentsymbols)) (emptylines (if (not (line-empty? parsedline)) 0 (1+ emptylines)))) - (when (not (= 0 (length parsedline))) - ; set the source properties to parsedline so we can try to add them later. + (when (not (= 0 (length (line-code parsedline)))) + ;; set the source properties to parsedline so we can try to add them later. (set-source-property! parsedline 'filename (port-filename port)) (set-source-property! parsedline 'line (port-line port))) - ; TODO: If the line is empty. Either do it here and do not add it, just - ; increment the empty line counter, or strip it later. Replace indent - ; -1 by indent 0 afterwards. + ;; TODO: If the line is empty. Either do it here and do not add it, just + ;; increment the empty line counter, or strip it later. Replace indent + ;; -1 by indent 0 afterwards. (loop (append indent-and-symbols (list parsedline)) - #t ; inindent + #t ; in-indent? (if (<= 2 emptylines) #f ; chunk ends here (equal? #\_ (peek-char port))); are we in underscore indent? - #f ; incomment + #f ; in-comment? 0 '() emptylines))) - ((equal? #t incomment) + ((equal? #t in-comment?) (read-char port); remove one comment character (loop indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #t ; incomment + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? currentindent currentsymbols emptylines)) @@ -321,47 +326,47 @@ (read-char port); remove char (loop indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #f ; incomment + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? currentindent currentsymbols emptylines)) - ; | cludge to appease the former wisp parser - ; | used for bootstrapping which has a - ; v problem with the literal comment char + ;; | cludge to appease the former wisp parser + ;; | used for bootstrapping which has a + ;; v problem with the literal comment char ((equal? (string-ref ";" 0) next-char) (loop indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #t ; incomment + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? currentindent currentsymbols emptylines)) (else ; use the reader (loop indent-and-symbols - #f ; inindent - #f ; inunderscoreindent - #f ; incomment + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? currentindent - ; this also takes care of the hashbang and leading comments. + ;; this also takes care of the hashbang and leading comments. (append currentsymbols (list (wisp-read port))) emptylines)))))))) (define (line-code-replace-inline-colons line) "Replace inline colons by opening parens which close at the end of the line" - ; format #t "replace inline colons for line ~A\n" line + ;; format #t "replace inline colons for line ~A\n" line (let loop ((processed '()) (unprocessed line)) (cond ((null? unprocessed) - ; format #t "inline-colons processed line: ~A\n" processed + ;; format #t "inline-colons processed line: ~A\n" processed processed) - ; replace : . with nothing + ;; replace : . with nothing ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) (loop (append processed @@ -369,7 +374,6 @@ '())) ((equal? readcolon (car unprocessed)) (loop - ; FIXME: This should turn unprocessed into a list. (append processed (list (loop '() (cdr unprocessed)))) '())) @@ -386,11 +390,8 @@ (define (line-strip-lone-colon line) "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." - (if - (equal? - (line-code line) - (list readcolon)) - (list (line-indent line)) + (if (equal? (line-code line) (list readcolon)) + (make-line (line-indent line)) line)) (define (line-finalize line) @@ -416,6 +417,12 @@ #f)) target) +(define (wisp-add-source-properties-from/when-required source target) + "Copy the source properties if target has none." + (if (null? (source-properties target)) + (wisp-add-source-properties-from source target) + target)) + (define (wisp-propagate-source-properties code) "Propagate the source properties from the sourrounding list into every part of the code." (let loop @@ -432,12 +439,15 @@ processed) (else (let ((line (car unprocessed))) - (if (null? (source-properties unprocessed)) - (wisp-add-source-properties-from line unprocessed) - (wisp-add-source-properties-from unprocessed line)) - (loop - (append processed (list (wisp-propagate-source-properties line))) - (cdr unprocessed))))))) + (wisp-add-source-properties-from/when-required line unprocessed) + (wisp-add-source-properties-from/when-required code unprocessed) + (wisp-add-source-properties-from/when-required unprocessed line) + (wisp-add-source-properties-from/when-required unprocessed code) + (let ((processed (append processed (list (wisp-propagate-source-properties line))))) + ;; must propagate from line, because unprocessed and code can be null, then they cannot keep source-properties. + (wisp-add-source-properties-from/when-required line processed) + (loop processed + (cdr unprocessed)))))))) (define* (wisp-scheme-indentation-to-parens lines) "Add parentheses to lines and remove the indentation markers" @@ -452,9 +462,12 @@ (cons (cons 0 (cdr (car lines))) (cdr lines))) - (throw 'wisp-syntax-error + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" - (car lines))))) + (car lines))))))) (let loop ((processed '()) (unprocessed lines) @@ -463,57 +476,53 @@ ((current-line (if (<= 1 (length unprocessed)) (car unprocessed) - (list 0))); empty code + (make-line 0))); empty code (next-line (if (<= 2 (length unprocessed)) (car (cdr unprocessed)) - (list 0))); empty code + (make-line 0))); empty code (current-indentation (car indentation-levels)) (current-line-indentation (line-real-indent current-line))) - ; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" - ; . processed current-line next-line unprocessed indentation-levels current-indentation + ;; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ;; . processed current-line next-line unprocessed indentation-levels current-indentation (cond - ; the real end: this is reported to the outside world. + ;; the real end: this is reported to the outside world. ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) - ; display "done\n" - ; reverse the processed lines, because I use cons. + ;; reverse the processed lines, because I use cons. processed) - ; the recursion end-condition + ;; the recursion end-condition ((and (null? unprocessed)) - ; display "last step\n" - ; this is the last step. Nothing more to do except - ; for rolling up the indentation levels. return the - ; new processed and unprocessed lists: this is a - ; side-recursion + ;; this is the last step. Nothing more to do except + ;; for rolling up the indentation levels. return the + ;; new processed and unprocessed lists: this is a + ;; side-recursion (values processed unprocessed)) ((null? indentation-levels) - ; display "indentation-levels null\n" - (throw 'wisp-programming-error "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")) + (raise-exception + (make-exception-from-throw + 'wisp-programming-error + (list + "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")))) (else ; now we come to the line-comparisons and indentation-counting. (cond ((line-empty-code? current-line) - ; display "current-line empty\n" - ; We cannot process indentation without - ; code. Just switch to the next line. This should - ; only happen at the start of the recursion. - ; TODO: Somehow preserve the line-numbers. + ;; We cannot process indentation without + ;; code. Just switch to the next line. This should + ;; only happen at the start of the recursion. (loop processed (cdr unprocessed) indentation-levels)) ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) - ; display "next-line empty\n" - ; TODO: Somehow preserve the line-numbers. - ; take out the next-line from unprocessed. + ;; take out the next-line from unprocessed. (loop processed (cons current-line (cdr (cdr unprocessed))) indentation-levels)) ((> current-indentation current-line-indentation) - ; display "current-indent > next-line\n" - ; this just steps back one level via the side-recursion. + ;; this just steps back one level via the side-recursion. (let ((previous-indentation (car (cdr indentation-levels)))) (if (<= current-line-indentation previous-indentation) (values processed unprocessed) @@ -527,14 +536,12 @@ current-line-indentation (cdr indentation-levels))))))) ((= current-indentation current-line-indentation) - ; display "current-indent = next-line\n" (let ((line (line-finalize current-line)) (next-line-indentation (line-real-indent next-line))) (cond ((>= current-line-indentation next-line-indentation) - ; simple recursiive step to the next line - ; display "current-line-indent >= next-line-indent\n" + ;; simple recursiive step to the next line (loop (append processed (if (line-continues? current-line) @@ -543,22 +550,18 @@ (cdr unprocessed); recursion here indentation-levels)) ((< current-line-indentation next-line-indentation) - ; display "current-line-indent < next-line-indent\n" - ; format #t "line: ~A\n" line - ; side-recursion via a sublist + ;; side-recursion via a sublist (let-values (((sub-processed sub-unprocessed) (loop line (cdr unprocessed); recursion here indentation-levels))) - ; format #t "side-recursion:\n sub-processed: ~A\n processed: ~A\n\n" sub-processed processed (loop (append processed (list sub-processed)) sub-unprocessed ; simply use the recursion from the sub-recursion indentation-levels)))))) ((< current-indentation current-line-indentation) - ; display "current-indent < next-line\n" (loop processed unprocessed @@ -566,9 +569,12 @@ current-line-indentation indentation-levels))) (else - (throw 'wisp-not-implemented + (raise-exception + (make-exception-from-throw + 'wisp-not-implemented + (list (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." - current-line next-line processed))))))))) + current-line next-line processed))))))))))) (define (wisp-scheme-replace-inline-colons lines) @@ -596,6 +602,8 @@ (define (wisp-unescape-underscore-and-colon code) "replace \\_ and \\: by _ and :" + (wisp-add-source-properties-from/when-required + code (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) ((eq? code '\:) ':) ;; Look for symbols like \____ and remove the \. @@ -606,21 +614,23 @@ (string-every #\_ (substring as-string 1))) (string->symbol (substring as-string 1)) code))) - (#t code))) + (#t code)))) (define (wisp-replace-empty-eof code) "replace ((#<eof>)) by ()" - ; FIXME: Actually this is a hack which fixes a bug when the - ; parser hits files with only hashbang and comments. + ;; This is a hack which fixes a bug when the + ;; parser hits files with only hashbang and comments. (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) - (list) + (wisp-add-source-properties-from code (list)) code)) (define (wisp-replace-paren-quotation-repr code) "Replace lists starting with a quotation symbol by quoted lists." + (wisp-add-source-properties-from/when-required + code (match code (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) (list 'quote (map wisp-replace-paren-quotation-repr a))) @@ -664,7 +674,7 @@ ((a ...) (map wisp-replace-paren-quotation-repr a)) (a - a))) + a)))) (define (wisp-make-improper code) "Turn (a #{.}# b) into the correct (a . b). @@ -676,38 +686,52 @@ when it reads a dot. So we have to take another pass over the code to recreate the improper lists. Match is awesome!" + (define is-proper? #t) + ;; local alias + (define (add-prop/req form) + (wisp-add-source-properties-from/when-required code form)) + (wisp-add-source-properties-from/when-required + code (let ((improper (match code ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) - (append (map wisp-make-improper a) - (cons (wisp-make-improper b) (wisp-make-improper c)))) + (set! is-proper? #f) + (wisp-add-source-properties-from/when-required + code + (append (map wisp-make-improper (map add-prop/req a)) + (cons (wisp-make-improper (add-prop/req b)) + (wisp-make-improper (add-prop/req c)))))) ((a ...) - (map wisp-make-improper a)) + (add-prop/req + (map wisp-make-improper (map add-prop/req a)))) (a a)))) (define (syntax-error li msg) - (throw 'wisp-syntax-error (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))) - (if #t + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) + (if is-proper? improper (let check ((tocheck improper)) (match tocheck - ; lists with only one member + ;; lists with only one member (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) (syntax-error tocheck "list with the period as only member")) - ; list with remaining dot. + ;; list with remaining dot. ((a ...) (if (and (member repr-dot a)) (syntax-error tocheck "leftover period in list") (map check a))) - ; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + ;; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) (syntax-error tocheck "dot as first element in already improper pair")) - ; simple pair, other way round + ;; simple pair, other way round ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) (syntax-error tocheck "dot as last element in already improper pair")) - ; more complex pairs + ;; more complex pairs ((? pair? a) (let ((head (drop-right a 1)) @@ -722,7 +746,7 @@ Match is awesome!" (else a)))) (a - a)))))) + a))))))) (define (wisp-scheme-read-chunk port) "Read and parse one chunk of wisp-code" @@ -758,4 +782,3 @@ Match is awesome!" (define (wisp-scheme-read-string-chunk str) (call-with-input-string str wisp-scheme-read-chunk)) - modified module/language/wisp/spec.scm @@ -1,32 +1,25 @@ -;; Language interface for Wisp in Guile +;;; Language interface for Wisp in Guile -;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm +;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> -;;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria -;;; Copyright (C) 2014--2023 Arne Babenhauserheide. -;;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA -;;; Permission is hereby granted, free of charge, to any person -;;; obtaining a copy of this software and associated documentation -;;; files (the "Software"), to deal in the Software without -;;; restriction, including without limitation the rights to use, copy, -;;; modify, merge, publish, distribute, sublicense, and/or sell copies -;;; of the Software, and to permit persons to whom the Software is -;;; furnished to do so, subject to the following conditions: -;;; -;;; The above copyright notice and this permission notice shall be -;;; included in all copies or substantial portions of the Software. -;;; -;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS -;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN -;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -;;; SOFTWARE. +;; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/?p=nacre:guile-sweet.git;a=blob;f=sweet/spec.scm;hb=ae306867e371cb4b56e00bb60a50d9a0b8353109 -; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm (define-module (language wisp spec) #:use-module (language wisp) #:use-module (system base compile) @@ -58,7 +51,7 @@ (car chunk)))))) (define-language wisp - #:title "Wisp Scheme Syntax. See SRFI-119 for details." + #:title "Wisp Scheme Syntax. See SRFI-119 for details" ; . #:reader read-one-wisp-sexp #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; #:compilers `((tree-il . ,compile-tree-il)) @@ -74,11 +67,6 @@ ;; compile-time changes to `current-reader' are ;; limited to the current compilation unit. (module-define! m 'current-reader (make-fluid)) - ;; Default to `simple-format', as is the case until - ;; (ice-9 format) is loaded. This allows - ;; compile-time warnings to be emitted when using - ;; unsupported options. - (module-set! m 'format simple-format) m))) modified test-suite/Makefile.am @@ -162,6 +162,7 @@ SCM_TESTS = tests/00-initial-env.test \ tests/srfi-98.test \ tests/srfi-105.test \ tests/srfi-111.test \ + tests/srfi-119.test \ tests/srfi-171.test \ tests/srfi-4.test \ tests/srfi-9.test \ modified test-suite/tests/srfi-119.test @@ -78,4 +78,12 @@ _ display \"hello\n\" (define (_) (display "hello\n")) -(_))))) +(_)))) + + ;; nesting with pairs + (pass-if (equal? (wisp->list "1 . 2\n3 4\n 5 . 6") + '((1 . 2)(3 4 (5 . 6)))))) + +(with-test-prefix "wisp-source-properties" + (pass-if (not (find null? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6"))))) + (pass-if (not (find null? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))))) -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-08-14 20:11 ` Dr. Arne Babenhauserheide 2023-08-14 20:30 ` Dr. Arne Babenhauserheide @ 2023-08-14 22:43 ` Dr. Arne Babenhauserheide 2023-08-18 10:29 ` Ludovic Courtès 2 siblings, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-08-14 22:43 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Maxime Devos, Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 3997 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi > index 0cdf56923..5b82f8070 100644 > --- a/doc/ref/srfi-modules.texi > +++ b/doc/ref/srfi-modules.texi > +To execute a file with wisp code, select the language and filename > +extension @code{.w} vie @code{guile --language=wisp -x .w}. Arg, missed one capitalization error here. I have a local fix, but don’t want to re-send the whole squashed change for that … From 6ac896f1bbb5968eca0e4e7b95c5ee9ebf7d5dda Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 15 Aug 2023 00:43:53 +0200 Subject: [PATCH] Fix: capitalize Wisp * doc/ref/srfi-modules.texi (srfi-119): capitalize Wisp --- doc/ref/srfi-modules.texi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 5b82f8070..0ffc01252 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -5686,7 +5686,7 @@ define : factorial n @result{} (define (factorial n) * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) @end example -To execute a file with wisp code, select the language and filename +To execute a file with Wisp code, select the language and filename extension @code{.w} vie @code{guile --language=wisp -x .w}. In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. -- 2.41.0 > +;; Scheme-only implementation of a wisp-preprocessor which output a > +;; scheme code tree to feed to a scheme interpreter instead of a > +;; preprocessed file. Same here, but for Scheme: From 78287cf3b584e3c0dd481d1a4d84735165ad5e3a Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 15 Aug 2023 00:46:53 +0200 Subject: [PATCH] Fix: capitalize Scheme * modules/language/wisp.scm (comments): capitalize Scheme --- module/language/wisp.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index b4e885eec..f3127c9d3 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -21,7 +21,7 @@ ;;; Commentary: ;; Scheme-only implementation of a wisp-preprocessor which output a -;; scheme code tree to feed to a scheme interpreter instead of a +;; Scheme code tree to feed to a Scheme interpreter instead of a ;; preprocessed file. ;; Limitations: -- 2.41.0 > diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm > new file mode 100644 > index 000000000..1efd3e8b2 > --- /dev/null > +++ b/module/language/wisp/spec.scm > … > + #:printer write ; TODO: backtransform to wisp? Use source-properties? and here … From 9e797f17d5f001ca81b5c7e0a6b31399fec7ce6d Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 15 Aug 2023 00:56:07 +0200 Subject: [PATCH] Fix: capitalize Wisp * modules/language/wisp/spec.scm (define-language): capitalize Wisp --- module/language/wisp/spec.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index 1efd3e8b2..5f8feca9a 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -57,7 +57,7 @@ #:compilers `((tree-il . ,compile-tree-il)) #:decompilers `((tree-il . ,decompile-tree-il)) #:evaluator (lambda (x module) (primitive-eval x)) - #:printer write ; TODO: backtransform to wisp? Use source-properties? + #:printer write ; TODO: backtransform to Wisp? Use source-properties? #:make-default-environment (lambda () ;; Ideally we'd duplicate the whole module hierarchy so that `set!', -- 2.41.0 That’s when I read my own patch after submission … but now that’s fixed. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-08-14 20:11 ` Dr. Arne Babenhauserheide 2023-08-14 20:30 ` Dr. Arne Babenhauserheide 2023-08-14 22:43 ` Dr. Arne Babenhauserheide @ 2023-08-18 10:29 ` Ludovic Courtès 2023-08-18 12:16 ` Dr. Arne Babenhauserheide 2 siblings, 1 reply; 83+ messages in thread From: Ludovic Courtès @ 2023-08-18 10:29 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel Hello Arne, "Dr. Arne Babenhauserheide" <arne_bab@web.de> skribis: > I did the changes for the review. It took a while (sorry for that) but > it got cleaner as a result (thank you!) > > Firstoff: I’d like to assign my copyright to the FSF. I’ll sign the > papers once I receive them. Also I have an Employer disclaimer of rights > on paper for Emacs already, so that should not cause trouble. OK. I assumed you already emailed assign@gnu.org the relevant form, right? Let us know how it goes. > Changes: > > - [X] LGPL > - [X] Please add the new files to the relevant ‘Makefile.am’ files. > - [X] Note the changes in Makefiles in the commit > - [X] Please capitalize “Scheme” and “Wisp” (in general we like to pay attention to typography, spelling, and language in the manual.) > - [X] s/(wisp)/, also referred to as @dfn{Wisp} (for ``Whitespace …'')/ > - [X] Two spaces after end-of-sentence periods, to facilitate navigation in Emacs. > - [X] indent “the usual way” > - [X] comments always with ;; except for margin comments > - [X] (read-enable 'curly-infix) > This needs to be: > (eval-when (expand load eval) > (read-enable 'curly-infix)) > - [X] Please make them #:use-module clauses in the ‘define-module’ form > - [X] I’d encourage following the usual naming convention, so > ‘in-indent?’, ‘in-comment?’, etc. > - [X] use exception objects or SRFI-35 error conditions instead of throw with symbols > - [X] add a test for source location info Awesome. > A change I did not do: > > - [ ] +Use record-types for the lines+. Reason: I decided not to do > this because it currently needs to propagate the source properties > when retrieving the code, so this is not a good match for a record > type (it may become one with an annotated reader, but I’d like to > shift that to a later change: What about having a ‘location’ field in that record type? Would that work for you? Or, alternatively, add source properties just on the relevant part of the list. Having lots of ‘car’ and ‘cdr’ in the code to access the various “fields” of the line hinders readability and prevents proper type-checking and error-reporting. As I wrote back in June, source properties are not ideal and explicitly storing location info, as is done in syntax objects, is preferable: https://lists.gnu.org/archive/html/guile-devel/2023-06/msg00008.html Some comments: > From 5857f8b961562d1ad2ae201401d5343233422eff Mon Sep 17 00:00:00 2001 > From: Arne Babenhauserheide <arne_bab@web.de> > Date: Fri, 3 Feb 2023 22:20:04 +0100 > Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation > > * doc/ref/srfi-modules.texi (srfi-119): add node > * module/language/wisp.scm: New file. > * module/language/wisp/spec.scm: New file. > * test-suite/tests/srfi-119.test: New file. > * am/bootstrap.am (SOURCES): add language/wisp.scm and language/wisp/spec.scm > * test-suite/Makefile.am (SCM_TESTS): add tests/srfi-119.test [...] > +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") > +;; define an intermediate dot replacement with UUID to avoid clashes. > +(define repr-dot ; . > + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) As I wrote in June, please remove those UUIDs and use uninterned symbols instead (which cannot be serialized but can be compared with ‘eq?’, which is all we need.) > +(define (match-charlist-to-repr charlist) > + (let > + ((chlist (reverse charlist))) > + (cond > + ((equal? chlist (list #\.)) > + repr-dot) > + ((equal? chlist (list #\')) > + repr-quote) This would probably be more readable with ‘match’ instead of ‘cond’. Also, as mentioned regarding the naming convention, please write ‘char-list’ or ‘chars’ rather than ‘chlist’. > +(define (wisp-read port) > + "wrap read to catch list prefixes." ^ Capital letter. Also maybe mention PORT and the return value. > + (cond > + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) > + (if repr-symbol ; found a special symbol, return it. > + repr-symbol > + (let unpeek > + ((remaining peeked)) Please avoid long lines and write ‘let’ on a line as in: (let ((x 2)) …) or: (let loop ((x 1)) …) (This is the style commonly used elsewhere in Guile.) > +(define (line-continues? line) > + (equal? repr-dot (car (line-code line)))) > + > +(define (line-only-colon? line) > + (and > + (equal? ":" (car (line-code line))) > + (null? (cdr (line-code line))))) > + > +(define (line-empty-code? line) > + (null? (line-code line))) I think this illustrates excessive use of lists, car, and cdr, and its impact on readability. > +(define (with-read-options opts thunk) > + (let ((saved-options (read-options))) > + (dynamic-wind > + (lambda () > + (read-options opts)) > + thunk > + (lambda () > + (read-options saved-options))))) > + > +(define (wisp->list str) > + (wisp-scheme-read-string str)) Please reindent (M-q) these two procedures. :-) > +(with-test-prefix "wisp-read-simple" > + (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) > + (pass-if (equal? (wisp->list ". 5") '(5))) > + (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) Prefer ‘pass-if-equal’: (pass-if-equal '((<= n 5)) (wisp->list "<= n 5")) That gives better reporting in ‘check-guile.log’. > +(with-test-prefix "wisp-source-properties" > + (pass-if (not (find null? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6"))))) > + (pass-if (not (find null? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))))) Maybe avoid the double negation by writing: (every pair? (map …)). However, why not check explicitly the line/column numbers? It seems to me like this would be more appropriate: we want to make sure the Wisp parser gets it right so users can get accurate error reporting. That’s it! I’m hope I’m not adding too much to what I already wrote in the previous review. Let me know! Thank you, Ludo’. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-08-18 10:29 ` Ludovic Courtès @ 2023-08-18 12:16 ` Dr. Arne Babenhauserheide 2023-08-18 17:50 ` Dr. Arne Babenhauserheide 2023-09-08 17:46 ` Dr. Arne Babenhauserheide 0 siblings, 2 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-08-18 12:16 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guile-devel [-- Attachment #1.1: Type: text/plain, Size: 7879 bytes --] Hello Ludo, Ludovic Courtès <ludo@gnu.org> writes: > OK. I assumed you already emailed assign@gnu.org the relevant form, > right? Let us know how it goes. If I read it right, the docs https://www.gnu.org/prep/maintain/html_node/Copyright-Papers.html say that I need to ask someone to get the papers I can then sign. Can I get the papers from somewhere or does someone have to send them to me? >> A change I did not do: >> >> - [ ] +Use record-types for the lines+. Reason: I decided not to do >> this because it currently needs to propagate the source properties >> when retrieving the code, so this is not a good match for a record >> type (it may become one with an annotated reader, but I’d like to >> shift that to a later change: > > What about having a ‘location’ field in that record type? Would that > work for you? Or, alternatively, add source properties just on the > relevant part of the list. I did part of the refactoring to a record type but it became less clear to read than working on the list, because the special cases to handle increased, so I reverted the change. It really didn’t get cleaner. car and cdr are mostly used in the variable length code-part of the line. > As I wrote back in June, source properties are not ideal and explicitly > storing location info, as is done in syntax objects, is preferable: > > https://lists.gnu.org/archive/html/guile-devel/2023-06/msg00008.html Once I get that sorted out, that would be the moment I’d switch to records (then they would need no added special casing), but I would prefer to take your offer to do that change in a followup patch. > Some comments: > >> From 5857f8b961562d1ad2ae201401d5343233422eff Mon Sep 17 00:00:00 2001 >> From: Arne Babenhauserheide <arne_bab@web.de> >> Date: Fri, 3 Feb 2023 22:20:04 +0100 >> Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation >> >> * doc/ref/srfi-modules.texi (srfi-119): add node >> * module/language/wisp.scm: New file. >> * module/language/wisp/spec.scm: New file. >> * test-suite/tests/srfi-119.test: New file. >> * am/bootstrap.am (SOURCES): add language/wisp.scm and language/wisp/spec.scm >> * test-suite/Makefile.am (SCM_TESTS): add tests/srfi-119.test > > [...] > >> +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") >> +;; define an intermediate dot replacement with UUID to avoid clashes. >> +(define repr-dot ; . >> + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) > > As I wrote in June, please remove those UUIDs and use uninterned symbols > instead (which cannot be serialized but can be compared with ‘eq?’, > which is all we need.) I already tried this when Maxime suggested it, but it did not work. I no longer remember in which part of the code it broke, but it broke, so I reverted to using the uuids. >> +(define (match-charlist-to-repr charlist) >> + (let >> + ((chlist (reverse charlist))) >> + (cond >> + ((equal? chlist (list #\.)) >> + repr-dot) >> + ((equal? chlist (list #\')) >> + repr-quote) > > This would probably be more readable with ‘match’ instead of ‘cond’. Match was too clever here. But changed formatting already gets close, and a helper ges further: (define (equal-rest? chars . args) (equal? chars args)) (define (match-charlist-to-repr char-list) (let ((chars (reverse char-list))) (cond ((equal-rest? chars #\.) repr-dot) ((equal-rest? chars #\') repr-quote) ((equal-rest? chars #\,) repr-unquote) ((equal-rest? chars #\`) repr-quasiquote) ((equal-rest? chars #\, #\@) repr-unquote-splicing) ((equal-rest? chars #\# #\') repr-syntax) ((equal-rest? chars #\# #\,) repr-unsyntax) ((equal-rest? chars #\# #\`) repr-quasisyntax) ((equal-rest? chars #\# #\, #\@) repr-unsyntax-splicing) (else #f)))) > Also, as mentioned regarding the naming convention, please write > ‘char-list’ or ‘chars’ rather than ‘chlist’. Fixed — thank you! >> +(define (wisp-read port) >> + "wrap read to catch list prefixes." > ^ > Capital letter. Also maybe mention PORT and the return value. Good point — the return value is pretty important here. New doc: "Wrap read to catch list prefixes: read one or several chars from PORT and return read symbols or replacement-symbols as representation for special forms." >> + (cond >> + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) >> + (if repr-symbol ; found a special symbol, return it. >> + repr-symbol >> + (let unpeek >> + ((remaining peeked)) > > Please avoid long lines and write ‘let’ on a line as in: > > (let ((x 2)) > …) > > or: > > (let loop ((x 1)) > …) > > (This is the style commonly used elsewhere in Guile.) Much nicer — thank you! (let ((prefix-maxlen 4)) (let longpeek ((peeked '()) (repr-symbol #f)) (cond ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) >> +(define (line-continues? line) >> + (equal? repr-dot (car (line-code line)))) >> + >> +(define (line-only-colon? line) >> + (and >> + (equal? ":" (car (line-code line))) >> + (null? (cdr (line-code line))))) >> + >> +(define (line-empty-code? line) >> + (null? (line-code line))) > > I think this illustrates excessive use of lists, car, and cdr, and its > impact on readability. This is a place where the line has variable length, so I can’t make it nicer with a record type without adding more complexity. >> +(define (with-read-options opts thunk) >> + (let ((saved-options (read-options))) >> + (dynamic-wind >> + (lambda () >> + (read-options opts)) >> + thunk >> + (lambda () >> + (read-options saved-options))))) >> + >> +(define (wisp->list str) >> + (wisp-scheme-read-string str)) > > Please reindent (M-q) these two procedures. :-) Ah, missed running indent-region on these. Thank you! (now I re-indented the whole test file) >> +(with-test-prefix "wisp-read-simple" >> + (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) >> + (pass-if (equal? (wisp->list ". 5") '(5))) >> + (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) > > Prefer ‘pass-if-equal’: > > (pass-if-equal '((<= n 5)) > (wisp->list "<= n 5")) > > That gives better reporting in ‘check-guile.log’. That’s much nicer! Thank you! >> +(with-test-prefix "wisp-source-properties" >> + (pass-if (not (find null? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6"))))) >> + (pass-if (not (find null? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))))) > > Maybe avoid the double negation by writing: (every pair? (map …)). That’s beautiful! I had thought about a way to do this but didn’t think about pair?. But actually what we need is that the source-properties are the same from Wisp and from Scheme. And that’s actually something else than what I did (though I thought that I did it), so this needed a few changes. Previously it reported as line the *end* of a form, but Guile usually reports the beginning of a form. I now changed the parser to also report the beginning of a form. > That’s it! I’m hope I’m not adding too much to what I already wrote in > the previous review. Let me know! I hope the new patch is ready to merge. I’m attaching the new squashed patch again here and will add the patches for the review changes to a second email. [-- Attachment #1.2: 0001-Add-language-wisp-Wisp-tests-and-SRFI-119-documentat.patch --] [-- Type: text/x-patch, Size: 44573 bytes --] From 6d7e47cbf07dccf98c644e926319f4688e2ac83a Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 3 Feb 2023 22:20:04 +0100 Subject: [PATCH] Add language/wisp, Wisp tests, and SRFI-119 documentation * doc/ref/srfi-modules.texi (srfi-119): add node * module/language/wisp.scm: New file. * module/language/wisp/spec.scm: New file. * test-suite/tests/srfi-119.test: New file. * am/bootstrap.am (SOURCES): add language/wisp.scm and language/wisp/spec.scm * test-suite/Makefile.am (SCM_TESTS) add tests/srfi-119.test --- am/bootstrap.am | 3 + doc/ref/srfi-modules.texi | 31 ++ module/language/wisp.scm | 776 +++++++++++++++++++++++++++++++++ module/language/wisp/spec.scm | 70 +++ test-suite/Makefile.am | 3 +- test-suite/tests/srfi-119.test | 108 +++++ 6 files changed, 990 insertions(+), 1 deletion(-) create mode 100644 module/language/wisp.scm create mode 100644 module/language/wisp/spec.scm create mode 100644 test-suite/tests/srfi-119.test diff --git a/am/bootstrap.am b/am/bootstrap.am index ff0d1799e..80a8dcdde 100644 --- a/am/bootstrap.am +++ b/am/bootstrap.am @@ -393,6 +393,9 @@ SOURCES = \ \ system/syntax.scm \ \ + language/wisp.scm \ + language/wisp/spec.scm \ + \ system/xref.scm \ \ sxml/apply-templates.scm \ diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 0cdf56923..0ffc01252 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. +* SRFI-119:: Wisp: simpler indentation-sensitive Scheme. * SRFI-171:: Transducers @end menu @@ -5662,6 +5663,35 @@ Return the current contents of @var{box}. Set the contents of @var{box} to @var{value}. @end deffn +@node SRFI-119 +@subsection SRFI-119 Wisp: simpler indentation-sensitive Scheme. +@cindex SRFI-119 +@cindex wisp + +The languages shipped in Guile include SRFI-119, also referred to as +@dfn{Wisp} (for ``Whitespace to Lisp''), an encoding of Scheme that +allows replacing parentheses with equivalent indentation and inline +colons. See +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification +of SRFI-119}. Some examples: + +@example +display "Hello World!" @result{} (display "Hello World!") +@end example + +@example +define : factorial n @result{} (define (factorial n) + if : zero? n @result{} (if (zero? n) + . 1 @result{} 1 + * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) +@end example + +To execute a file with Wisp code, select the language and filename +extension @code{.w} vie @code{guile --language=wisp -x .w}. + +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. + + @node SRFI-171 @subsection Transducers @cindex SRFI-171 @@ -5705,6 +5735,7 @@ left-to-right, due to how transducers are initiated. * SRFI-171 Helpers:: Utilities for writing your own transducers @end menu + @node SRFI-171 General Discussion @subsubsection SRFI-171 General Discussion @cindex transducers discussion diff --git a/module/language/wisp.scm b/module/language/wisp.scm new file mode 100644 index 000000000..dae9642ae --- /dev/null +++ b/module/language/wisp.scm @@ -0,0 +1,776 @@ +;;; Wisp + +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;;; Commentary: + +;; Scheme-only implementation of a wisp-preprocessor which output a +;; Scheme code tree to feed to a Scheme interpreter instead of a +;; preprocessed file. + +;; Limitations: +;; - in some cases the source line information is missing in backtraces. +;; check for set-source-property! + +;;; Code: + +(define-module (language wisp) + #:export (wisp-scheme-read-chunk wisp-scheme-read-all + wisp-scheme-read-file-chunk wisp-scheme-read-file + wisp-scheme-read-string) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11); for let-values + #:use-module (srfi srfi-9); for records + #:use-module (ice-9 rw); for write-string/partial + #:use-module (ice-9 match)) + +;; use curly-infix by default +(eval-when (expand load eval) + (read-enable 'curly-infix)) + + +;; Helpers to preserver source properties + +(define (wisp-add-source-properties-from source target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) + +(define (wisp-add-source-properties-from/when-required source target) + "Copy the source properties if target has none." + (if (null? (source-properties target)) + (wisp-add-source-properties-from source target) + target)) + + +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define make-line list) + +(define (line-indent line) + (car line)) + +(define (line-real-indent line) + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." + (let ((indent (line-indent line))) + (if (= -1 indent) + 0 + indent))) + +(define (line-code line) + "Strip the indentation markers from the beginning of the line and preserve source-properties" + (let ((code (cdr line))) + ;; propagate source properties + (when (not (null? code)) + (wisp-add-source-properties-from/when-required line code)) + code)) + +;; literal values I need +(define readcolon + (string->symbol ":")) + +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") +;; define an intermediate dot replacement with UUID to avoid clashes. +(define repr-dot ; . + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) + +;; allow using reader additions as the first element on a line to prefix the list +(define repr-quote ; ' + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) +(define repr-unquote ; , + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) +(define repr-quasiquote ; ` + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) +(define repr-unquote-splicing ; ,@ + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) + +(define repr-syntax ; #' + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) +(define repr-unsyntax ; #, + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) +(define repr-quasisyntax ; #` + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) +(define repr-unsyntax-splicing ; #,@ + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) + +;; TODO: wrap the reader to return the repr of the syntax reader +;; additions + +(define (equal-rest? chars . args) + (equal? chars args)) + +(define (match-charlist-to-repr char-list) + (let ((chars (reverse char-list))) + (cond + ((equal-rest? chars #\.) repr-dot) + ((equal-rest? chars #\') repr-quote) + ((equal-rest? chars #\,) repr-unquote) + ((equal-rest? chars #\`) repr-quasiquote) + ((equal-rest? chars #\, #\@) repr-unquote-splicing) + ((equal-rest? chars #\# #\') repr-syntax) + ((equal-rest? chars #\# #\,) repr-unsyntax) + ((equal-rest? chars #\# #\`) repr-quasisyntax) + ((equal-rest? chars #\# #\, #\@) repr-unsyntax-splicing) + (else #f)))) + +(define (wisp-read port) + "Wrap read to catch list prefixes: read one or several chars from PORT and return read symbols or replacement-symbols as representation for special forms." + (let ((prefix-maxlen 4)) + (let longpeek ((peeked '()) (repr-symbol #f)) + (cond + ((or (< prefix-maxlen (length peeked)) + (eof-object? (peek-char port)) + (equal? #\space (peek-char port)) + (equal? #\newline (peek-char port))) + (if repr-symbol ; found a special symbol, return it. + repr-symbol + (let unpeek ((remaining peeked)) + (cond + ((equal? '() remaining) + (read port)); let read to the work + (else + (unread-char (car remaining) port) + (unpeek (cdr remaining))))))) + (else + (let* ((next-char (read-char port)) + (peeked (cons next-char peeked))) + (longpeek + peeked + (match-charlist-to-repr peeked)))))))) + + + +(define (line-continues? line) + (equal? repr-dot (car (line-code line)))) + +(define (line-only-colon? line) + (and + (equal? ":" (car (line-code line))) + (null? (cdr (line-code line))))) + +(define (line-empty-code? line) + (null? (line-code line))) + +(define (line-empty? line) + (and + ;; if indent is -1, we stripped a comment, so the line was not really empty. + (= 0 (line-indent line)) + (line-empty-code? line))) + +(define (line-strip-continuation line) + (if (line-continues? line) + (apply make-line + (line-indent line) + (cdr (line-code line))) + line)) + +(define (line-strip-indentation-marker line) + "Strip the indentation markers from the beginning of the line for line-finalize without propagating source-properties (those are propagated in a second step)" + (cdr line)) + +(define (indent-level-reduction indentation-levels level select-fun) + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" + (let loop ((newlevels indentation-levels) + (diff 0)) + (cond + ((= level (car newlevels)) + (select-fun (list diff indentation-levels))) + ((< level (car newlevels)) + (loop + (cdr newlevels) + (1+ diff))) + (else + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "Level ~A not found in the indentation-levels ~A." level indentation-levels)))))))) + +(define (indent-level-difference indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the count + (car x)))) + +(define (indent-reduce-to-level indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the levels + (car (cdr x))))) + +(define (chunk-ends-with-period currentsymbols next-char) + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." + (and (not (null? currentsymbols)) + (equal? #\newline next-char) + (equal? repr-dot + (list-ref currentsymbols (- (length currentsymbols) 1))))) + + +(define (wisp-scheme-read-chunk-lines port) + ;; the line number for this chunk is the line number when starting to read it + ;; a top-level form stops processing, so we only need to retrieve this here. + (define line-number (port-line port)) + (let loop ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (in-indent? #t) + (in-underscoreindent? (equal? #\_ (peek-char port))) + (in-comment? #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) + (cond + ((>= emptylines 2) + ;; the chunk end has to be checked + ;; before we look for new chars in the + ;; port to make execution in the REPL + ;; after two empty lines work + ;; (otherwise it shows one more line). + indent-and-symbols) + (else + (let ((next-char (peek-char port))) + (cond + ((eof-object? next-char) + (let ((line (apply make-line currentindent currentsymbols))) + (set-source-property! line 'filename (port-filename port)) + (set-source-property! line 'line line-number) + (append indent-and-symbols (list line)))) + ((and in-indent? + (zero? currentindent) + (not in-comment?) + (not (null? indent-and-symbols)) + (not in-underscoreindent?) + (not (or (equal? #\space next-char) + (equal? #\newline next-char) + (equal? (string-ref ";" 0) next-char)))) + (append indent-and-symbols)); top-level form ends chunk + ((chunk-ends-with-period currentsymbols next-char) + ;; the line ends with a period. This is forbidden in + ;; SRFI-119. Use it to end the line in the REPL without + ;; showing continuation dots (...). + (append indent-and-symbols (list (apply make-line currentindent (drop-right currentsymbols 1))))) + ((and in-indent? (equal? #\space next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ((and in-underscoreindent? (equal? #\_ next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #t ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ;; any char but whitespace *after* underscoreindent is + ;; an error. This is stricter than the current wisp + ;; syntax definition. + ;; TODO: Fix the definition. Better start too strict. + ;; FIXME: breaks on lines with only underscores which should be + ;; empty lines. + ((and in-underscoreindent? (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))))) + ((equal? #\newline next-char) + (read-char port); remove the newline + (let* + ;; distinguish pure whitespace lines and lines + ;; with comment by giving the former zero + ;; indent. Lines with a comment at zero indent + ;; get indent -1 for the same reason - meaning + ;; not actually empty. + ((indent + (cond + (in-comment? + (if (= 0 currentindent); specialcase + -1 + currentindent)) + ((not (null? currentsymbols)); pure whitespace + currentindent) + (else + 0))) + (parsedline (apply make-line indent currentsymbols)) + (emptylines + (if (not (line-empty? parsedline)) + 0 + (1+ emptylines)))) + (when (not (= 0 (length (line-code parsedline)))) + ;; set the source properties to parsedline so we can try to add them later. + (set-source-property! parsedline 'filename (port-filename port)) + (set-source-property! parsedline 'line line-number)) + ;; TODO: If the line is empty. Either do it here and do not add it, just + ;; increment the empty line counter, or strip it later. Replace indent + ;; -1 by indent 0 afterwards. + (loop + (append indent-and-symbols (list parsedline)) + #t ; in-indent? + (if (<= 2 emptylines) + #f ; chunk ends here + (equal? #\_ (peek-char port))); are we in underscore indent? + #f ; in-comment? + 0 + '() + emptylines))) + ((equal? #t in-comment?) + (read-char port); remove one comment character + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent + (read-char port); remove char + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + currentsymbols + emptylines)) + ;; | cludge to appease the former wisp parser + ;; | used for bootstrapping which has a + ;; v problem with the literal comment char + ((equal? (string-ref ";" 0) next-char) + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + (else ; use the reader + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + ;; this also takes care of the hashbang and leading comments. + (append currentsymbols (list (wisp-read port))) + emptylines)))))))) + + +(define (line-code-replace-inline-colons line) + "Replace inline colons by opening parens which close at the end of the line" + ;; format #t "replace inline colons for line ~A\n" line + (let loop ((processed '()) + (unprocessed line)) + (cond + ((null? unprocessed) + ;; format #t "inline-colons processed line: ~A\n" processed + processed) + ;; replace : . with nothing + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) + (loop + (append processed + (loop '() (cdr (cdr unprocessed)))) + '())) + ((equal? readcolon (car unprocessed)) + (loop + (append processed + (list (loop '() (cdr unprocessed)))) + '())) + (else + (loop + (append processed + (list (car unprocessed))) + (cdr unprocessed)))))) + +(define (line-replace-inline-colons line) + (cons + (line-indent line) + (line-code-replace-inline-colons (line-code line)))) + +(define (line-strip-lone-colon line) + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." + (if (equal? (line-code line) (list readcolon)) + (make-line (line-indent line)) + line)) + +(define (line-finalize line) + "Process all wisp-specific information in a line and strip it" + (let ((l (line-code-replace-inline-colons + (line-strip-indentation-marker + (line-strip-lone-colon + (line-strip-continuation line)))))) + (when (not (null? (source-properties line))) + (catch #t + (lambda () + (set-source-properties! l (source-properties line))) + (lambda (key . arguments) + #f))) + l)) + +(define (wisp-propagate-source-properties code) + "Propagate the source properties from the sourrounding list into every part of the code." + (let loop ((processed '()) + (unprocessed code)) + (cond + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) + unprocessed) + ((and (pair? unprocessed) (not (list? unprocessed))) + (cons + (wisp-propagate-source-properties (car unprocessed)) + (wisp-propagate-source-properties (cdr unprocessed)))) + ((null? unprocessed) + processed) + (else + (let ((line (car unprocessed))) + (wisp-add-source-properties-from/when-required line unprocessed) + (wisp-add-source-properties-from/when-required code unprocessed) + (wisp-add-source-properties-from/when-required unprocessed line) + (wisp-add-source-properties-from/when-required unprocessed code) + (let ((processed (append processed (list (wisp-propagate-source-properties line))))) + ;; must propagate from line, because unprocessed and code can be null, then they cannot keep source-properties. + (wisp-add-source-properties-from/when-required line processed) + (loop processed + (cdr unprocessed)))))))) + +(define* (wisp-scheme-indentation-to-parens lines) + "Add parentheses to lines and remove the indentation markers" + (when + (and + (not (null? lines)) + (not (line-empty-code? (car lines))) + (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment + (if (= 1 (line-real-indent (car lines))) + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands + (set! lines + (cons + (cons 0 (cdr (car lines))) + (cdr lines))) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" + (car lines))))))) + (let loop ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* ((current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (make-line 0))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (make-line 0))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) + ;; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ;; . processed current-line next-line unprocessed indentation-levels current-indentation + (cond + ;; the real end: this is reported to the outside world. + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) + ;; reverse the processed lines, because I use cons. + processed) + ;; the recursion end-condition + ((and (null? unprocessed)) + ;; this is the last step. Nothing more to do except + ;; for rolling up the indentation levels. return the + ;; new processed and unprocessed lists: this is a + ;; side-recursion + (values processed unprocessed)) + ((null? indentation-levels) + (raise-exception + (make-exception-from-throw + 'wisp-programming-error + (list + "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")))) + (else ; now we come to the line-comparisons and indentation-counting. + (cond + ((line-empty-code? current-line) + ;; We cannot process indentation without + ;; code. Just switch to the next line. This should + ;; only happen at the start of the recursion. + (loop + processed + (cdr unprocessed) + indentation-levels)) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) + ;; take out the next-line from unprocessed. + (loop + processed + (cons current-line + (cdr (cdr unprocessed))) + indentation-levels)) + ((> current-indentation current-line-indentation) + ;; this just steps back one level via the side-recursion. + (let ((previous-indentation (car (cdr indentation-levels)))) + (if (<= current-line-indentation previous-indentation) + (values processed unprocessed) + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. + (let ((linenumber (- (length lines) (length unprocessed)))) + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + (cdr indentation-levels))))))) + ((= current-indentation current-line-indentation) + (let ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) + (cond + ((>= current-line-indentation next-line-indentation) + ;; simple recursiive step to the next line + (loop + (append processed + (if (line-continues? current-line) + line + (wisp-add-source-properties-from line (list line)))) + (cdr unprocessed); recursion here + indentation-levels)) + ((< current-line-indentation next-line-indentation) + ;; side-recursion via a sublist + (let-values + (((sub-processed sub-unprocessed) + (loop + line + (cdr unprocessed); recursion here + indentation-levels))) + (loop + (append processed (list sub-processed)) + sub-unprocessed ; simply use the recursion from the sub-recursion + indentation-levels)))))) + ((< current-indentation current-line-indentation) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + indentation-levels))) + (else + (raise-exception + (make-exception-from-throw + 'wisp-not-implemented + (list + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))))) + + +(define (wisp-scheme-replace-inline-colons lines) + "Replace inline colons by opening parens which close at the end of the line" + (let loop ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (list (line-replace-inline-colons (car unprocessed)))) + (cdr unprocessed))))) + + +(define (wisp-scheme-strip-indentation-markers lines) + "Strip the indentation markers from the beginning of the lines" + (let loop ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (cdr (car unprocessed))) + (cdr unprocessed))))) + +(define (wisp-unescape-underscore-and-colon code) + "replace \\_ and \\: by _ and :" + (wisp-add-source-properties-from/when-required + code + (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code)))) + + +(define (wisp-replace-empty-eof code) + "replace ((#<eof>)) by ()" + ;; This is a hack which fixes a bug when the + ;; parser hits files with only hashbang and comments. + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) + (wisp-add-source-properties-from code (list)) + code)) + + +(define (wisp-replace-paren-quotation-repr code) + "Replace lists starting with a quotation symbol by + quoted lists." + (wisp-add-source-properties-from/when-required + code + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a)))) + +(define (wisp-make-improper code) + "Turn (a #{.}# b) into the correct (a . b). + +read called on a single dot creates a variable named #{.}# (|.| +in r7rs). Due to parsing the indentation before the list +structure is known, the reader cannot create improper lists +when it reads a dot. So we have to take another pass over the +code to recreate the improper lists. + +Match is awesome!" + (define is-proper? #t) + ;; local alias + (define (add-prop/req form) + (wisp-add-source-properties-from/when-required code form)) + (wisp-add-source-properties-from/when-required + code + (let ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (set! is-proper? #f) + (wisp-add-source-properties-from/when-required + code + (append (map wisp-make-improper (map add-prop/req a)) + (cons (wisp-make-improper (add-prop/req b)) + (wisp-make-improper (add-prop/req c)))))) + ((a ...) + (add-prop/req + (map wisp-make-improper (map add-prop/req a)))) + (a + a)))) + (define (syntax-error li msg) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) + (if is-proper? + improper + (let check ((tocheck improper)) + (match tocheck + ;; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ;; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ;; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ;; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ;; more complex pairs + ((? pair? a) + (let ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a + a))))))) + +(define (wisp-scheme-read-chunk port) + "Read and parse one chunk of wisp-code" + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\#)) + (let ((lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines))))))))) + +(define (wisp-scheme-read-all port) + "Read all chunks from the given port" + (let loop ((tokens '())) + (cond + ((eof-object? (peek-char port)) + tokens) + (else + (loop + (append tokens (wisp-scheme-read-chunk port))))))) + +(define (wisp-scheme-read-file path) + (call-with-input-file path wisp-scheme-read-all)) + +(define (wisp-scheme-read-file-chunk path) + (call-with-input-file path wisp-scheme-read-chunk)) + +(define (wisp-scheme-read-string str) + (call-with-input-string str wisp-scheme-read-all)) + +(define (wisp-scheme-read-string-chunk str) + (call-with-input-string str wisp-scheme-read-chunk)) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm new file mode 100644 index 000000000..f7fd794e0 --- /dev/null +++ b/module/language/wisp/spec.scm @@ -0,0 +1,70 @@ +;;; Language interface for Wisp in Guile + +;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/?p=nacre:guile-sweet.git;a=blob;f=sweet/spec.scm;hb=ae306867e371cb4b56e00bb60a50d9a0b8353109 + +(define-module (language wisp spec) + #:use-module (language wisp) + #:use-module (system base compile) + #:use-module (system base language) + #:use-module (language scheme compile-tree-il) + #:use-module (language scheme decompile-tree-il) + #:export (wisp)) + +;;; +;;; Language definition +;;; + + +(define (read-one-wisp-sexp port env) + ;; Allow using "# foo" as #(foo). + ;; Don't use the globally-acting read-hash-extend, because this + ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme. + ;; Instead, use fluids to temporarily add the extension. + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\# )) + ;; Read Wisp files as UTF-8, to support non-ASCII characters. + ;; TODO: would be nice to support ';; coding: whatever' lines + ;; like in parenthese-y Scheme. + (set-port-encoding! port "UTF-8") + (if (eof-object? (peek-char port)) + (read-char port) ; return eof: we’re done + (let ((chunk (wisp-scheme-read-chunk port))) + (and (not (null? chunk)) ; <---- XXX: maybe (pair? chunk) + (car chunk)))))) + +(define-language wisp + #:title "Wisp Scheme Syntax. See SRFI-119 for details" + ;; . #:reader read-one-wisp-sexp + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; + #:compilers `((tree-il . ,compile-tree-il)) + #:decompilers `((tree-il . ,decompile-tree-il)) + #:evaluator (lambda (x module) (primitive-eval x)) + #:printer write ; TODO: backtransform to Wisp? Use source-properties? + #:make-default-environment + (lambda () + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', + ;; `fluid-set!', etc. don't have any effect in the current environment. + (let ((m (make-fresh-user-module))) + ;; Provide a separate `current-reader' fluid so that + ;; compile-time changes to `current-reader' are + ;; limited to the current compilation unit. + (module-define! m 'current-reader (make-fluid)) + m))) diff --git a/test-suite/Makefile.am b/test-suite/Makefile.am index 81e63bce2..247d97746 100644 --- a/test-suite/Makefile.am +++ b/test-suite/Makefile.am @@ -162,7 +162,8 @@ SCM_TESTS = tests/00-initial-env.test \ tests/srfi-98.test \ tests/srfi-105.test \ tests/srfi-111.test \ - tests/srfi-171.test \ + tests/srfi-119.test \ + tests/srfi-171.test \ tests/srfi-4.test \ tests/srfi-9.test \ tests/statprof.test \ diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test new file mode 100644 index 000000000..60e1e0377 --- /dev/null +++ b/test-suite/tests/srfi-119.test @@ -0,0 +1,108 @@ +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*- +;;;; +;;;; Copyright (C) 2023 Free Software Foundation, Inc. +;;;; +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +(define-module (test-srfi-119) + #:use-module (test-suite lib) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) ;; cut + #:use-module (language wisp)) + +(define (read-string s) + (with-input-from-string s read)) + +(define (with-read-options opts thunk) + (let ((saved-options (read-options))) + (dynamic-wind + (lambda () + (read-options opts)) + thunk + (lambda () + (read-options saved-options))))) + +(define (wisp->list str) + (wisp-scheme-read-string str)) + +(define (scheme->list str) + (with-input-from-string str + (λ () + (let loop ((result '())) + (if (eof-object? (peek-char)) + (reverse! result) + (loop (cons (read) result))))))) + +(with-test-prefix "wisp-read-simple" + (pass-if-equal '((<= n 5)) + (wisp->list "<= n 5")) + (pass-if-equal '(5) + (wisp->list ". 5")) + (pass-if-equal '((+ 1 (* 2 3))) + (wisp->list "+ 1 : * 2 3"))) +(with-test-prefix "wisp-read-complex" + (pass-if-equal '( + (a b c d e + f g h + i j k) + + (concat "I want " + (getwish from me) + " - " username)) (wisp->list " +a b c d e + . f g h + . i j k + +concat \"I want \" + getwish from me + . \" - \" username +")) + + (pass-if-equal + '( + (define (a b c) + (d e + (f) + (g h) + i)) + + (define (_) + (display "hello\n")) + + (_)) (wisp->list " +define : a b c +_ d e +___ f +___ g h +__ . i + +define : _ +_ display \"hello\n\" + +\\_")) + + ;; nesting with pairs + (pass-if-equal '((1 . 2)(3 4 (5 . 6))) + (wisp->list "1 . 2\n3 4\n 5 . 6"))) + +(with-test-prefix "wisp-source-properties" + ;; has properties + (pass-if (every pair? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6")))) + (pass-if (every pair? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))) + ;; has the same properties + (pass-if-equal + (map source-properties (scheme->list "(1 . 2)\n(3 4\n (5 . 6))\n(1 4)\n\n(7 8)")) + (map (cut cons '(filename . #f) <>) + (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6\n1 4\n\n7 8"))))) -- 2.41.0 [-- Attachment #1.3: Type: text/plain, Size: 103 bytes --] Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-08-18 12:16 ` Dr. Arne Babenhauserheide @ 2023-08-18 17:50 ` Dr. Arne Babenhauserheide 2023-09-08 17:46 ` Dr. Arne Babenhauserheide 1 sibling, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-08-18 17:50 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1.1: Type: text/plain, Size: 283 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > I’m attaching the new squashed patch again here and will add the patches > for the review changes to a second email. Attached are the promised patches of the additional review changes. Thank you for your review! [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.2: 0012-SRFI-119-Wisp-Fix-capitalize-Wisp.patch --] [-- Type: text/x-patch, Size: 1013 bytes --] From 3d9b452137911e1948586657edb1ea614d8a70c0 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 15 Aug 2023 00:43:53 +0200 Subject: [PATCH 12/21] SRFI-119 (Wisp): Fix: capitalize Wisp * doc/ref/srfi-modules.texi (srfi-119): capitalize Wisp --- doc/ref/srfi-modules.texi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 5b82f8070..0ffc01252 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -5686,7 +5686,7 @@ define : factorial n @result{} (define (factorial n) * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) @end example -To execute a file with wisp code, select the language and filename +To execute a file with Wisp code, select the language and filename extension @code{.w} vie @code{guile --language=wisp -x .w}. In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.3: 0013-SRFI-119-Wisp-Fix-capitalize-Scheme.patch --] [-- Type: text/x-patch, Size: 841 bytes --] From d8585c6380cbdba2ad0f6c56aaf6637826cd5b93 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 15 Aug 2023 00:46:53 +0200 Subject: [PATCH 13/21] SRFI-119 (Wisp): Fix: capitalize Scheme * modules/language/wisp.scm (comments): capitalize Scheme --- module/language/wisp.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index b4e885eec..f3127c9d3 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -21,7 +21,7 @@ ;;; Commentary: ;; Scheme-only implementation of a wisp-preprocessor which output a -;; scheme code tree to feed to a scheme interpreter instead of a +;; Scheme code tree to feed to a Scheme interpreter instead of a ;; preprocessed file. ;; Limitations: -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.4: 0014-SRFI-119-Wisp-Fix-capitalize-Wisp.patch --] [-- Type: text/x-patch, Size: 1028 bytes --] From 44344fa738cb51b034bb03791a6e1ee828390a42 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Tue, 15 Aug 2023 00:56:07 +0200 Subject: [PATCH 14/21] SRFI-119 (Wisp): Fix: capitalize Wisp * modules/language/wisp/spec.scm (define-language): capitalize Wisp --- module/language/wisp/spec.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index 1efd3e8b2..5f8feca9a 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -57,7 +57,7 @@ #:compilers `((tree-il . ,compile-tree-il)) #:decompilers `((tree-il . ,decompile-tree-il)) #:evaluator (lambda (x module) (primitive-eval x)) - #:printer write ; TODO: backtransform to wisp? Use source-properties? + #:printer write ; TODO: backtransform to Wisp? Use source-properties? #:make-default-environment (lambda () ;; Ideally we'd duplicate the whole module hierarchy so that `set!', -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.5: 0015-SRFI-119-Wisp-cleanup-char-list-cond.patch --] [-- Type: text/x-patch, Size: 2104 bytes --] From 16967e979262f7f3d86e194295a1a3f5a7f68cd0 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 18 Aug 2023 19:06:23 +0200 Subject: [PATCH 15/21] SRFI-119 (Wisp): cleanup char-list cond * module/language/wisp.scm (match-charlist-to-repr): use helper and re-indent --- module/language/wisp.scm | 38 +++++++++++++++----------------------- 1 file changed, 15 insertions(+), 23 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index f3127c9d3..3ac128df2 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -97,30 +97,22 @@ ;; TODO: wrap the reader to return the repr of the syntax reader ;; additions -(define (match-charlist-to-repr charlist) - (let - ((chlist (reverse charlist))) +(define (equal-rest? chars . args) + (equal? chars args)) + +(define (match-charlist-to-repr char-list) + (let ((chars (reverse char-list))) (cond - ((equal? chlist (list #\.)) - repr-dot) - ((equal? chlist (list #\')) - repr-quote) - ((equal? chlist (list #\,)) - repr-unquote) - ((equal? chlist (list #\`)) - repr-quasiquote) - ((equal? chlist (list #\, #\@)) - repr-unquote-splicing) - ((equal? chlist (list #\# #\')) - repr-syntax) - ((equal? chlist (list #\# #\,)) - repr-unsyntax) - ((equal? chlist (list #\# #\`)) - repr-quasisyntax) - ((equal? chlist (list #\# #\, #\@)) - repr-unsyntax-splicing) - (else - #f)))) + ((equal-rest? chars #\.) repr-dot) + ((equal-rest? chars #\') repr-quote) + ((equal-rest? chars #\,) repr-unquote) + ((equal-rest? chars #\`) repr-quasiquote) + ((equal-rest? chars #\, #\@) repr-unquote-splicing) + ((equal-rest? chars #\# #\') repr-syntax) + ((equal-rest? chars #\# #\,) repr-unsyntax) + ((equal-rest? chars #\# #\`) repr-quasisyntax) + ((equal-rest? chars #\# #\, #\@) repr-unsyntax-splicing) + (else #f)))) (define (wisp-read port) "wrap read to catch list prefixes." -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.6: 0016-SRFI-119-Wisp-improve-docstring.patch --] [-- Type: text/x-patch, Size: 891 bytes --] From a74e63f65e6f02c9aeff76bf1d6a93043fa95c45 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 18 Aug 2023 19:10:07 +0200 Subject: [PATCH 16/21] SRFI-119 (Wisp): improve docstring * module/language/wisp.scm (wisp-read): improve docstring --- module/language/wisp.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index 3ac128df2..96429218d 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -115,7 +115,7 @@ (else #f)))) (define (wisp-read port) - "wrap read to catch list prefixes." + "Wrap read to catch list prefixes: read one or several chars from PORT and return read symbols or replacement-symbols as representation for special forms." (let ((prefix-maxlen 4)) (let longpeek ((peeked '()) -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.7: 0017-SRFI-119-Wisp-improve-let-and-let-formatting.patch --] [-- Type: text/x-patch, Size: 10900 bytes --] From 361c00fc77a3cd8621be47a37fca18265ae59310 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 18 Aug 2023 19:14:07 +0200 Subject: [PATCH 17/21] SRFI-119 (Wisp): improve let and let* formatting * module/language/wisp.scm (wisp-read, wisp-scheme-read-chunk-lines): clean up let and let* arguments --- module/language/wisp.scm | 133 +++++++++++++++++++-------------------- 1 file changed, 64 insertions(+), 69 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index 96429218d..3b14eba54 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -117,15 +117,15 @@ (define (wisp-read port) "Wrap read to catch list prefixes: read one or several chars from PORT and return read symbols or replacement-symbols as representation for special forms." (let ((prefix-maxlen 4)) - (let longpeek - ((peeked '()) - (repr-symbol #f)) + (let longpeek ((peeked '()) (repr-symbol #f)) (cond - ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port))) + ((or (< prefix-maxlen (length peeked)) + (eof-object? (peek-char port)) + (equal? #\space (peek-char port)) + (equal? #\newline (peek-char port))) (if repr-symbol ; found a special symbol, return it. repr-symbol - (let unpeek - ((remaining peeked)) + (let unpeek ((remaining peeked)) (cond ((equal? '() remaining) (read port)); let read to the work @@ -133,9 +133,8 @@ (unread-char (car remaining) port) (unpeek (cdr remaining))))))) (else - (let* - ((next-char (read-char port)) - (peeked (cons next-char peeked))) + (let* ((next-char (read-char port)) + (peeked (cons next-char peeked))) (longpeek peeked (match-charlist-to-repr peeked)))))))) @@ -172,9 +171,8 @@ (define (indent-level-reduction indentation-levels level select-fun) "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" - (let loop - ((newlevels indentation-levels) - (diff 0)) + (let loop ((newlevels indentation-levels) + (diff 0)) (cond ((= level (car newlevels)) (select-fun (list diff indentation-levels))) @@ -230,7 +228,14 @@ (set-source-property! line 'filename (port-filename port)) (set-source-property! line 'line (port-line port)) (append indent-and-symbols (list line)))) - ((and in-indent? (zero? currentindent) (not in-comment?) (not (null? indent-and-symbols)) (not in-underscoreindent?) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) + ((and in-indent? + (zero? currentindent) + (not in-comment?) + (not (null? indent-and-symbols)) + (not in-underscoreindent?) + (not (or (equal? #\space next-char) + (equal? #\newline next-char) + (equal? (string-ref ";" 0) next-char)))) (append indent-and-symbols)); top-level form ends chunk ((chunk-ends-with-period currentsymbols next-char) ;; the line ends with a period. This is forbidden in @@ -259,9 +264,10 @@ emptylines)) ;; any char but whitespace *after* underscoreindent is ;; an error. This is stricter than the current wisp - ;; syntax definition. TODO: Fix the definition. Better - ;; start too strict. FIXME: breaks on lines with only - ;; underscores which should be empty lines. + ;; syntax definition. + ;; TODO: Fix the definition. Better start too strict. + ;; FIXME: breaks on lines with only underscores which should be + ;; empty lines. ((and in-underscoreindent? (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) (raise-exception (make-exception-from-throw 'wisp-syntax-error (list "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))))) ((equal? #\newline next-char) @@ -351,9 +357,8 @@ (define (line-code-replace-inline-colons line) "Replace inline colons by opening parens which close at the end of the line" ;; format #t "replace inline colons for line ~A\n" line - (let loop - ((processed '()) - (unprocessed line)) + (let loop ((processed '()) + (unprocessed line)) (cond ((null? unprocessed) ;; format #t "inline-colons processed line: ~A\n" processed @@ -417,9 +422,8 @@ (define (wisp-propagate-source-properties code) "Propagate the source properties from the sourrounding list into every part of the code." - (let loop - ((processed '()) - (unprocessed code)) + (let loop ((processed '()) + (unprocessed code)) (cond ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) unprocessed) @@ -460,22 +464,20 @@ (list (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" (car lines))))))) - (let loop - ((processed '()) - (unprocessed lines) - (indentation-levels '(0))) - (let* - ((current-line - (if (<= 1 (length unprocessed)) - (car unprocessed) - (make-line 0))); empty code - (next-line - (if (<= 2 (length unprocessed)) - (car (cdr unprocessed)) - (make-line 0))); empty code - (current-indentation - (car indentation-levels)) - (current-line-indentation (line-real-indent current-line))) + (let loop ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* ((current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (make-line 0))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (make-line 0))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) ;; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" ;; . processed current-line next-line unprocessed indentation-levels current-indentation (cond @@ -528,9 +530,8 @@ current-line-indentation (cdr indentation-levels))))))) ((= current-indentation current-line-indentation) - (let - ((line (line-finalize current-line)) - (next-line-indentation (line-real-indent next-line))) + (let ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) (cond ((>= current-line-indentation next-line-indentation) ;; simple recursiive step to the next line @@ -571,9 +572,8 @@ (define (wisp-scheme-replace-inline-colons lines) "Replace inline colons by opening parens which close at the end of the line" - (let loop - ((processed '()) - (unprocessed lines)) + (let loop ((processed '()) + (unprocessed lines)) (if (null? unprocessed) processed (loop @@ -583,9 +583,8 @@ (define (wisp-scheme-strip-indentation-markers lines) "Strip the indentation markers from the beginning of the lines" - (let loop - ((processed '()) - (unprocessed lines)) + (let loop ((processed '()) + (unprocessed lines)) (if (null? unprocessed) processed (loop @@ -684,21 +683,20 @@ Match is awesome!" (wisp-add-source-properties-from/when-required code form)) (wisp-add-source-properties-from/when-required code - (let - ((improper - (match code - ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) - (set! is-proper? #f) - (wisp-add-source-properties-from/when-required - code - (append (map wisp-make-improper (map add-prop/req a)) - (cons (wisp-make-improper (add-prop/req b)) - (wisp-make-improper (add-prop/req c)))))) - ((a ...) - (add-prop/req - (map wisp-make-improper (map add-prop/req a)))) - (a - a)))) + (let ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (set! is-proper? #f) + (wisp-add-source-properties-from/when-required + code + (append (map wisp-make-improper (map add-prop/req a)) + (cons (wisp-make-improper (add-prop/req b)) + (wisp-make-improper (add-prop/req c)))))) + ((a ...) + (add-prop/req + (map wisp-make-improper (map add-prop/req a)))) + (a + a)))) (define (syntax-error li msg) (raise-exception (make-exception-from-throw @@ -706,8 +704,7 @@ Match is awesome!" (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) (if is-proper? improper - (let check - ((tocheck improper)) + (let check ((tocheck improper)) (match tocheck ;; lists with only one member (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) @@ -725,9 +722,8 @@ Match is awesome!" (syntax-error tocheck "dot as last element in already improper pair")) ;; more complex pairs ((? pair? a) - (let - ((head (drop-right a 1)) - (tail (last-pair a))) + (let ((head (drop-right a 1)) + (tail (last-pair a))) (cond ((equal? repr-dot (car tail)) (syntax-error tocheck "equal? repr-dot : car tail")) @@ -754,8 +750,7 @@ Match is awesome!" (define (wisp-scheme-read-all port) "Read all chunks from the given port" - (let loop - ((tokens '())) + (let loop ((tokens '())) (cond ((eof-object? (peek-char port)) tokens) -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.8: 0018-SRFI-119-Wisp-Fix-comment-syntax-and-trailing-whites.patch --] [-- Type: text/x-patch, Size: 1185 bytes --] From 0ca2a934c96d657c996d8b2f0241cc7e38ae2a0e Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 18 Aug 2023 19:15:20 +0200 Subject: [PATCH 18/21] SRFI-119 (Wisp): Fix comment syntax and trailing whitespace * module/language/wisp/spec.scm (define-language): comment with ;;, strip trailing lines --- module/language/wisp/spec.scm | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index 5f8feca9a..f7fd794e0 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -52,7 +52,7 @@ (define-language wisp #:title "Wisp Scheme Syntax. See SRFI-119 for details" - ; . #:reader read-one-wisp-sexp + ;; . #:reader read-one-wisp-sexp #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; #:compilers `((tree-il . ,compile-tree-il)) #:decompilers `((tree-il . ,decompile-tree-il)) @@ -68,6 +68,3 @@ ;; limited to the current compilation unit. (module-define! m 'current-reader (make-fluid)) m))) - - - -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.9: 0019-SRFI-119-Wisp-reindent-test.patch --] [-- Type: text/x-patch, Size: 1206 bytes --] From 7de03c8ce421e809afb95823037d655aa9a47fd2 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 18 Aug 2023 19:17:25 +0200 Subject: [PATCH 19/21] SRFI-119 (Wisp): reindent test * test-suite/tests/srfi-119.test (with-read-options, wisp->list): M-x indent-region --- test-suite/tests/srfi-119.test | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test index f4a19a0a7..6fe87f2b2 100644 --- a/test-suite/tests/srfi-119.test +++ b/test-suite/tests/srfi-119.test @@ -27,14 +27,14 @@ (define (with-read-options opts thunk) (let ((saved-options (read-options))) (dynamic-wind - (lambda () - (read-options opts)) - thunk - (lambda () - (read-options saved-options))))) + (lambda () + (read-options opts)) + thunk + (lambda () + (read-options saved-options))))) (define (wisp->list str) - (wisp-scheme-read-string str)) + (wisp-scheme-read-string str)) (with-test-prefix "wisp-read-simple" (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) -- 2.41.0 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.10: 0020-SRFI-119-Wisp-use-pass-if-equal-instead-of-pass-if-e.patch --] [-- Type: text/x-patch, Size: 2557 bytes --] From 8cd856e060840277b1a8b30892d6ef4f55fe5c7a Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 18 Aug 2023 19:19:40 +0200 Subject: [PATCH 20/21] SRFI-119 (Wisp): use pass-if-equal instead of pass-if (equal? ...) * test-suite/tests/srfi-119.test (wisp-read-simple, wisp-read-complex): use pass-if-equal and invert conditions to improve error messages --- test-suite/tests/srfi-119.test | 54 ++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test index 6fe87f2b2..64ccc2ff6 100644 --- a/test-suite/tests/srfi-119.test +++ b/test-suite/tests/srfi-119.test @@ -37,11 +37,21 @@ (wisp-scheme-read-string str)) (with-test-prefix "wisp-read-simple" - (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) - (pass-if (equal? (wisp->list ". 5") '(5))) - (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) + (pass-if-equal '((<= n 5)) + (wisp->list "<= n 5")) + (pass-if-equal '(5) + (wisp->list ". 5")) + (pass-if-equal '((+ 1 (* 2 3))) + (wisp->list "+ 1 : * 2 3"))) (with-test-prefix "wisp-read-complex" - (pass-if (equal? (wisp->list " + (pass-if-equal '( + (a b c d e + f g h + i j k) + + (concat "I want " + (getwish from me) + " - " username)) (wisp->list " a b c d e . f g h . i j k @@ -49,16 +59,20 @@ a b c d e concat \"I want \" getwish from me . \" - \" username -") '( -(a b c d e - f g h - i j k) +")) + + (pass-if-equal + '( + (define (a b c) + (d e + (f) + (g h) + i)) -(concat "I want " - (getwish from me) - " - " username)))) + (define (_) + (display "hello\n")) - (pass-if (equal? (wisp->list " + (_)) (wisp->list " define : a b c _ d e ___ f @@ -68,21 +82,11 @@ __ . i define : _ _ display \"hello\n\" -\\_") '( -(define (a b c) - (d e - (f) - (g h) - i)) - -(define (_) - (display "hello\n")) - -(_)))) +\\_")) ;; nesting with pairs - (pass-if (equal? (wisp->list "1 . 2\n3 4\n 5 . 6") - '((1 . 2)(3 4 (5 . 6)))))) + (pass-if-equal '((1 . 2)(3 4 (5 . 6))) + (wisp->list "1 . 2\n3 4\n 5 . 6"))) (with-test-prefix "wisp-source-properties" (pass-if (not (find null? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6"))))) -- 2.41.0 [-- Attachment #1.11: 0021-SRFI-119-Wisp-add-tests-for-equality-of-source-prope.patch --] [-- Type: text/x-patch, Size: 6683 bytes --] From e120fc39aca45d55ede90b4200b7b9e39bc83e1e Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Fri, 18 Aug 2023 19:25:59 +0200 Subject: [PATCH 21/21] SRFI-119 (Wisp): add tests for equality of source-properties and fix them * test-suite/tests/srfi-119.test (scheme->list): new procedure * test-suite/tests/srfi-119.test (wisp-source-properties): use pass-if (every pair? ...) for the existance test. Use scheme->list to compare source-properties from regular Scheme read and wisp read. * module/language/wisp.scm (line-code): replace custom logic with wisp-add-source-properties-from/when-required * module/language/wisp.scm (wisp-scheme-read-chunk-lines): set the line-number from the start of the chunk as source-property instead of the line number from the end of the chunk. --- module/language/wisp.scm | 57 ++++++++++++++++++---------------- test-suite/tests/srfi-119.test | 19 ++++++++++-- 2 files changed, 48 insertions(+), 28 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index 3b14eba54..dae9642ae 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -45,6 +45,24 @@ (read-enable 'curly-infix)) +;; Helpers to preserver source properties + +(define (wisp-add-source-properties-from source target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) + +(define (wisp-add-source-properties-from/when-required source target) + "Copy the source properties if target has none." + (if (null? (source-properties target)) + (wisp-add-source-properties-from source target) + target)) + + ;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) (define make-line list) @@ -63,7 +81,7 @@ (let ((code (cdr line))) ;; propagate source properties (when (not (null? code)) - (set-source-properties! code (source-properties line))) + (wisp-add-source-properties-from/when-required line code)) code)) ;; literal values I need @@ -204,14 +222,16 @@ (define (wisp-scheme-read-chunk-lines port) - (let loop - ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) - (in-indent? #t) - (in-underscoreindent? (equal? #\_ (peek-char port))) - (in-comment? #f) - (currentindent 0) - (currentsymbols '()) - (emptylines 0)) + ;; the line number for this chunk is the line number when starting to read it + ;; a top-level form stops processing, so we only need to retrieve this here. + (define line-number (port-line port)) + (let loop ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (in-indent? #t) + (in-underscoreindent? (equal? #\_ (peek-char port))) + (in-comment? #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) (cond ((>= emptylines 2) ;; the chunk end has to be checked @@ -226,7 +246,7 @@ ((eof-object? next-char) (let ((line (apply make-line currentindent currentsymbols))) (set-source-property! line 'filename (port-filename port)) - (set-source-property! line 'line (port-line port)) + (set-source-property! line 'line line-number) (append indent-and-symbols (list line)))) ((and in-indent? (zero? currentindent) @@ -296,7 +316,7 @@ (when (not (= 0 (length (line-code parsedline)))) ;; set the source properties to parsedline so we can try to add them later. (set-source-property! parsedline 'filename (port-filename port)) - (set-source-property! parsedline 'line (port-line port))) + (set-source-property! parsedline 'line line-number)) ;; TODO: If the line is empty. Either do it here and do not add it, just ;; increment the empty line counter, or strip it later. Replace indent ;; -1 by indent 0 afterwards. @@ -405,21 +425,6 @@ #f))) l)) -(define (wisp-add-source-properties-from source target) - "Copy the source properties from source into the target and return the target." - (catch #t - (lambda () - (set-source-properties! target (source-properties source))) - (lambda (key . arguments) - #f)) - target) - -(define (wisp-add-source-properties-from/when-required source target) - "Copy the source properties if target has none." - (if (null? (source-properties target)) - (wisp-add-source-properties-from source target) - target)) - (define (wisp-propagate-source-properties code) "Propagate the source properties from the sourrounding list into every part of the code." (let loop ((processed '()) diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test index 64ccc2ff6..60e1e0377 100644 --- a/test-suite/tests/srfi-119.test +++ b/test-suite/tests/srfi-119.test @@ -19,6 +19,7 @@ (define-module (test-srfi-119) #:use-module (test-suite lib) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) ;; cut #:use-module (language wisp)) (define (read-string s) @@ -36,6 +37,14 @@ (define (wisp->list str) (wisp-scheme-read-string str)) +(define (scheme->list str) + (with-input-from-string str + (λ () + (let loop ((result '())) + (if (eof-object? (peek-char)) + (reverse! result) + (loop (cons (read) result))))))) + (with-test-prefix "wisp-read-simple" (pass-if-equal '((<= n 5)) (wisp->list "<= n 5")) @@ -89,5 +98,11 @@ _ display \"hello\n\" (wisp->list "1 . 2\n3 4\n 5 . 6"))) (with-test-prefix "wisp-source-properties" - (pass-if (not (find null? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6"))))) - (pass-if (not (find null? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))))) + ;; has properties + (pass-if (every pair? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6")))) + (pass-if (every pair? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))) + ;; has the same properties + (pass-if-equal + (map source-properties (scheme->list "(1 . 2)\n(3 4\n (5 . 6))\n(1 4)\n\n(7 8)")) + (map (cut cons '(filename . #f) <>) + (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6\n1 4\n\n7 8"))))) -- 2.41.0 [-- Attachment #1.12: Type: text/plain, Size: 101 bytes --] Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-08-18 12:16 ` Dr. Arne Babenhauserheide 2023-08-18 17:50 ` Dr. Arne Babenhauserheide @ 2023-09-08 17:46 ` Dr. Arne Babenhauserheide 2023-10-05 14:10 ` Dr. Arne Babenhauserheide 1 sibling, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-09-08 17:46 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 702 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: >> OK. I assumed you already emailed assign@gnu.org the relevant form, >> right? Let us know how it goes. > > If I read it right, the docs > https://www.gnu.org/prep/maintain/html_node/Copyright-Papers.html > say that I need to ask someone to get the papers I can then sign. > > Can I get the papers from somewhere or does someone have to send them to > me? I now sent a request to be sent the papers to sign to assign@gnu.org using a template I also got for Emacs, so I’m sending these directly for Emacs, ELPA, and Guile. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-09-08 17:46 ` Dr. Arne Babenhauserheide @ 2023-10-05 14:10 ` Dr. Arne Babenhauserheide 2023-10-10 23:04 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-10-05 14:10 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 705 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: >>> OK. I assumed you already emailed assign@gnu.org the relevant form, >>> right? Let us know how it goes. > I now sent a request to be sent the papers to sign to assign@gnu.org > using a template I also got for Emacs, so I’m sending these directly for > Emacs, ELPA, and Guile. I now assigned my copyright in GNU Guile to FSF and received the version of the copyright assigment form with two signatures, so copyright assigment for contributing Wisp to Guile is completed. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-10-05 14:10 ` Dr. Arne Babenhauserheide @ 2023-10-10 23:04 ` Dr. Arne Babenhauserheide 2023-10-27 22:05 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-10-10 23:04 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 453 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > >> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: >>>> OK. I assumed you already emailed assign@gnu.org the relevant form, >>>> right? Let us know how it goes. >> I now sent a request to be sent the papers to sign to assign@gnu.org > I now assigned my copyright in GNU Guile to FSF Is the patch now good to go? Best wishes, Arne [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-10-10 23:04 ` Dr. Arne Babenhauserheide @ 2023-10-27 22:05 ` Dr. Arne Babenhauserheide 2024-01-09 7:05 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-10-27 22:05 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 527 bytes --] ArneBab writes: > ArneBab writes: >> I now assigned my copyright in GNU Guile to FSF > Is the patch now good to go? Clarification: I don’t actually need someone to push this (I can push). I just need the go to actually do it (rebase the branch then merge it), because I don’t want to push my own changes (especially not larger ones) without a go from a maintainer. Is adding SRFI-119 to Guile good to go? Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2023-10-27 22:05 ` Dr. Arne Babenhauserheide @ 2024-01-09 7:05 ` Dr. Arne Babenhauserheide 2024-01-19 8:21 ` Dr. Arne Babenhauserheide 2024-01-19 12:10 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Christina O'Donnell 0 siblings, 2 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-01-09 7:05 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 224 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > Is adding SRFI-119 to Guile good to go? It’s a new year — any chance for one more look whether adding SRFI-119 in Guile is ok to merge? Best wishes, Arne [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-09 7:05 ` Dr. Arne Babenhauserheide @ 2024-01-19 8:21 ` Dr. Arne Babenhauserheide 2024-03-11 1:16 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) Dr. Arne Babenhauserheide 2024-01-19 12:10 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Christina O'Donnell 1 sibling, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-01-19 8:21 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel, Andy Wingo [-- Attachment #1: Type: text/plain, Size: 1288 bytes --] Hi, I just got the most beautiful feedback on Wisp as a Scheme primer, so I would like to nag about inclusion of SRFI-119 into Guile again: »I tend to use [Wisp] as a Scheme primer for colleagues that are used to Python but want to explore the realms of functional programming without … having to break with known syntax and conventions … it makes Scheme way more “approachable” while not drifting too much apart. Before using Wisp as an introduction, I got occasional feedback that parens where seen as confusing/somewhat cluttery in terms of readability; Wisp looks as clean, plain and readable as Python code; which makes it relatively easy to awake interest for Scheme in folks that are accustomed to Python already.« — Wilko https://emacs.ch/@thees/111720771253976695 Inclusion in Guile would make it much easier for them to just try Wisp on any of the supported platforms, so I think this is a strong indicator that inclusion of Wisp would help spread Guile Scheme. => Is the squashed patch ok to merge? Link to the patch in the archives: https://lists.gnu.org/archive/html/guile-devel/2023-08/msg00011.html Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) 2024-01-19 8:21 ` Dr. Arne Babenhauserheide @ 2024-03-11 1:16 ` Dr. Arne Babenhauserheide 2024-06-01 9:57 ` Ludovic Courtès 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-03-11 1:16 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Ludovic Courtès, guile-devel, Andy Wingo [-- Attachment #1.1: Type: text/plain, Size: 517 bytes --] Hi, I just added the remaining tests from the Wisp reader testsuite to the patch, squashed and rebased it again. The updated patch is attached. As far as I know, all review comments are addressed and I hope this is ready to be merged. Test suite result: Totals for this test run: passes: 36 failures: 0 unexpected passes: 0 expected failures: 0 unresolved test cases: 0 untested test cases: 0 unsupported test cases: 0 errors: 0 The squashed patch: [-- Attachment #1.2: 0001-Add-language-wisp-Wisp-tests-and-SRFI-119-documentat.patch --] [-- Type: text/x-patch, Size: 61345 bytes --] From 81e7cbbade4fd01e092a2c39a3af90e4abf7f684 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Mon, 11 Mar 2024 06:34:52 +0100 Subject: [PATCH] Add language/wisp, Wisp tests, and SRFI-119 documentation * doc/ref/srfi-modules.texi (srfi-119): add node * module/language/wisp.scm: New file. * module/language/wisp/spec.scm: New file. * test-suite/tests/srfi-119.test: New file. --- am/bootstrap.am | 3 + doc/ref/srfi-modules.texi | 31 + module/language/wisp.scm | 776 ++++++++++++++++++++++++ module/language/wisp/spec.scm | 70 +++ test-suite/Makefile.am | 3 +- test-suite/tests/srfi-119.test | 1040 ++++++++++++++++++++++++++++++++ 6 files changed, 1922 insertions(+), 1 deletion(-) create mode 100644 module/language/wisp.scm create mode 100644 module/language/wisp/spec.scm create mode 100644 test-suite/tests/srfi-119.test diff --git a/am/bootstrap.am b/am/bootstrap.am index a71946958..7c4b5d8a8 100644 --- a/am/bootstrap.am +++ b/am/bootstrap.am @@ -395,6 +395,9 @@ SOURCES = \ \ system/syntax.scm \ \ + language/wisp.scm \ + language/wisp/spec.scm \ + \ system/xref.scm \ \ sxml/apply-templates.scm \ diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 09b591e89..329e74958 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-98:: Accessing environment variables. * SRFI-105:: Curly-infix expressions. * SRFI-111:: Boxes. +* SRFI-119:: Wisp: simpler indentation-sensitive Scheme. * SRFI-171:: Transducers @end menu @@ -5662,6 +5663,35 @@ Return the current contents of @var{box}. Set the contents of @var{box} to @var{value}. @end deffn +@node SRFI-119 +@subsection SRFI-119 Wisp: simpler indentation-sensitive Scheme. +@cindex SRFI-119 +@cindex wisp + +The languages shipped in Guile include SRFI-119, also referred to as +@dfn{Wisp} (for ``Whitespace to Lisp''), an encoding of Scheme that +allows replacing parentheses with equivalent indentation and inline +colons. See +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification +of SRFI-119}. Some examples: + +@example +display "Hello World!" @result{} (display "Hello World!") +@end example + +@example +define : factorial n @result{} (define (factorial n) + if : zero? n @result{} (if (zero? n) + . 1 @result{} 1 + * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) +@end example + +To execute a file with Wisp code, select the language and filename +extension @code{.w} vie @code{guile --language=wisp -x .w}. + +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. + + @node SRFI-171 @subsection Transducers @cindex SRFI-171 @@ -5705,6 +5735,7 @@ left-to-right, due to how transducers are initiated. * SRFI-171 Helpers:: Utilities for writing your own transducers @end menu + @node SRFI-171 General Discussion @subsubsection SRFI-171 General Discussion @cindex transducers discussion diff --git a/module/language/wisp.scm b/module/language/wisp.scm new file mode 100644 index 000000000..dae9642ae --- /dev/null +++ b/module/language/wisp.scm @@ -0,0 +1,776 @@ +;;; Wisp + +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;;; Commentary: + +;; Scheme-only implementation of a wisp-preprocessor which output a +;; Scheme code tree to feed to a Scheme interpreter instead of a +;; preprocessed file. + +;; Limitations: +;; - in some cases the source line information is missing in backtraces. +;; check for set-source-property! + +;;; Code: + +(define-module (language wisp) + #:export (wisp-scheme-read-chunk wisp-scheme-read-all + wisp-scheme-read-file-chunk wisp-scheme-read-file + wisp-scheme-read-string) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11); for let-values + #:use-module (srfi srfi-9); for records + #:use-module (ice-9 rw); for write-string/partial + #:use-module (ice-9 match)) + +;; use curly-infix by default +(eval-when (expand load eval) + (read-enable 'curly-infix)) + + +;; Helpers to preserver source properties + +(define (wisp-add-source-properties-from source target) + "Copy the source properties from source into the target and return the target." + (catch #t + (lambda () + (set-source-properties! target (source-properties source))) + (lambda (key . arguments) + #f)) + target) + +(define (wisp-add-source-properties-from/when-required source target) + "Copy the source properties if target has none." + (if (null? (source-properties target)) + (wisp-add-source-properties-from source target) + target)) + + +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) +(define make-line list) + +(define (line-indent line) + (car line)) + +(define (line-real-indent line) + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." + (let ((indent (line-indent line))) + (if (= -1 indent) + 0 + indent))) + +(define (line-code line) + "Strip the indentation markers from the beginning of the line and preserve source-properties" + (let ((code (cdr line))) + ;; propagate source properties + (when (not (null? code)) + (wisp-add-source-properties-from/when-required line code)) + code)) + +;; literal values I need +(define readcolon + (string->symbol ":")) + +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") +;; define an intermediate dot replacement with UUID to avoid clashes. +(define repr-dot ; . + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) + +;; allow using reader additions as the first element on a line to prefix the list +(define repr-quote ; ' + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) +(define repr-unquote ; , + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) +(define repr-quasiquote ; ` + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) +(define repr-unquote-splicing ; ,@ + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) + +(define repr-syntax ; #' + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) +(define repr-unsyntax ; #, + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) +(define repr-quasisyntax ; #` + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) +(define repr-unsyntax-splicing ; #,@ + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) + +;; TODO: wrap the reader to return the repr of the syntax reader +;; additions + +(define (equal-rest? chars . args) + (equal? chars args)) + +(define (match-charlist-to-repr char-list) + (let ((chars (reverse char-list))) + (cond + ((equal-rest? chars #\.) repr-dot) + ((equal-rest? chars #\') repr-quote) + ((equal-rest? chars #\,) repr-unquote) + ((equal-rest? chars #\`) repr-quasiquote) + ((equal-rest? chars #\, #\@) repr-unquote-splicing) + ((equal-rest? chars #\# #\') repr-syntax) + ((equal-rest? chars #\# #\,) repr-unsyntax) + ((equal-rest? chars #\# #\`) repr-quasisyntax) + ((equal-rest? chars #\# #\, #\@) repr-unsyntax-splicing) + (else #f)))) + +(define (wisp-read port) + "Wrap read to catch list prefixes: read one or several chars from PORT and return read symbols or replacement-symbols as representation for special forms." + (let ((prefix-maxlen 4)) + (let longpeek ((peeked '()) (repr-symbol #f)) + (cond + ((or (< prefix-maxlen (length peeked)) + (eof-object? (peek-char port)) + (equal? #\space (peek-char port)) + (equal? #\newline (peek-char port))) + (if repr-symbol ; found a special symbol, return it. + repr-symbol + (let unpeek ((remaining peeked)) + (cond + ((equal? '() remaining) + (read port)); let read to the work + (else + (unread-char (car remaining) port) + (unpeek (cdr remaining))))))) + (else + (let* ((next-char (read-char port)) + (peeked (cons next-char peeked))) + (longpeek + peeked + (match-charlist-to-repr peeked)))))))) + + + +(define (line-continues? line) + (equal? repr-dot (car (line-code line)))) + +(define (line-only-colon? line) + (and + (equal? ":" (car (line-code line))) + (null? (cdr (line-code line))))) + +(define (line-empty-code? line) + (null? (line-code line))) + +(define (line-empty? line) + (and + ;; if indent is -1, we stripped a comment, so the line was not really empty. + (= 0 (line-indent line)) + (line-empty-code? line))) + +(define (line-strip-continuation line) + (if (line-continues? line) + (apply make-line + (line-indent line) + (cdr (line-code line))) + line)) + +(define (line-strip-indentation-marker line) + "Strip the indentation markers from the beginning of the line for line-finalize without propagating source-properties (those are propagated in a second step)" + (cdr line)) + +(define (indent-level-reduction indentation-levels level select-fun) + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" + (let loop ((newlevels indentation-levels) + (diff 0)) + (cond + ((= level (car newlevels)) + (select-fun (list diff indentation-levels))) + ((< level (car newlevels)) + (loop + (cdr newlevels) + (1+ diff))) + (else + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list (format #f "Level ~A not found in the indentation-levels ~A." level indentation-levels)))))))) + +(define (indent-level-difference indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the count + (car x)))) + +(define (indent-reduce-to-level indentation-levels level) + "Find how many indentation levels need to be popped off to find the given level." + (indent-level-reduction indentation-levels level + (lambda (x); get the levels + (car (cdr x))))) + +(define (chunk-ends-with-period currentsymbols next-char) + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." + (and (not (null? currentsymbols)) + (equal? #\newline next-char) + (equal? repr-dot + (list-ref currentsymbols (- (length currentsymbols) 1))))) + + +(define (wisp-scheme-read-chunk-lines port) + ;; the line number for this chunk is the line number when starting to read it + ;; a top-level form stops processing, so we only need to retrieve this here. + (define line-number (port-line port)) + (let loop ((indent-and-symbols (list)); '((5 "(foobar)" "\"yobble\"")(3 "#t")) + (in-indent? #t) + (in-underscoreindent? (equal? #\_ (peek-char port))) + (in-comment? #f) + (currentindent 0) + (currentsymbols '()) + (emptylines 0)) + (cond + ((>= emptylines 2) + ;; the chunk end has to be checked + ;; before we look for new chars in the + ;; port to make execution in the REPL + ;; after two empty lines work + ;; (otherwise it shows one more line). + indent-and-symbols) + (else + (let ((next-char (peek-char port))) + (cond + ((eof-object? next-char) + (let ((line (apply make-line currentindent currentsymbols))) + (set-source-property! line 'filename (port-filename port)) + (set-source-property! line 'line line-number) + (append indent-and-symbols (list line)))) + ((and in-indent? + (zero? currentindent) + (not in-comment?) + (not (null? indent-and-symbols)) + (not in-underscoreindent?) + (not (or (equal? #\space next-char) + (equal? #\newline next-char) + (equal? (string-ref ";" 0) next-char)))) + (append indent-and-symbols)); top-level form ends chunk + ((chunk-ends-with-period currentsymbols next-char) + ;; the line ends with a period. This is forbidden in + ;; SRFI-119. Use it to end the line in the REPL without + ;; showing continuation dots (...). + (append indent-and-symbols (list (apply make-line currentindent (drop-right currentsymbols 1))))) + ((and in-indent? (equal? #\space next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ((and in-underscoreindent? (equal? #\_ next-char)) + (read-char port); remove char + (loop + indent-and-symbols + #t ; in-indent? + #t ; in-underscoreindent? + #f ; in-comment? + (1+ currentindent) + currentsymbols + emptylines)) + ;; any char but whitespace *after* underscoreindent is + ;; an error. This is stricter than the current wisp + ;; syntax definition. + ;; TODO: Fix the definition. Better start too strict. + ;; FIXME: breaks on lines with only underscores which should be + ;; empty lines. + ((and in-underscoreindent? (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) + (raise-exception (make-exception-from-throw 'wisp-syntax-error (list "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))))) + ((equal? #\newline next-char) + (read-char port); remove the newline + (let* + ;; distinguish pure whitespace lines and lines + ;; with comment by giving the former zero + ;; indent. Lines with a comment at zero indent + ;; get indent -1 for the same reason - meaning + ;; not actually empty. + ((indent + (cond + (in-comment? + (if (= 0 currentindent); specialcase + -1 + currentindent)) + ((not (null? currentsymbols)); pure whitespace + currentindent) + (else + 0))) + (parsedline (apply make-line indent currentsymbols)) + (emptylines + (if (not (line-empty? parsedline)) + 0 + (1+ emptylines)))) + (when (not (= 0 (length (line-code parsedline)))) + ;; set the source properties to parsedline so we can try to add them later. + (set-source-property! parsedline 'filename (port-filename port)) + (set-source-property! parsedline 'line line-number)) + ;; TODO: If the line is empty. Either do it here and do not add it, just + ;; increment the empty line counter, or strip it later. Replace indent + ;; -1 by indent 0 afterwards. + (loop + (append indent-and-symbols (list parsedline)) + #t ; in-indent? + (if (<= 2 emptylines) + #f ; chunk ends here + (equal? #\_ (peek-char port))); are we in underscore indent? + #f ; in-comment? + 0 + '() + emptylines))) + ((equal? #t in-comment?) + (read-char port); remove one comment character + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char)); remove whitespace when not in indent + (read-char port); remove char + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + currentsymbols + emptylines)) + ;; | cludge to appease the former wisp parser + ;; | used for bootstrapping which has a + ;; v problem with the literal comment char + ((equal? (string-ref ";" 0) next-char) + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #t ; in-comment? + currentindent + currentsymbols + emptylines)) + (else ; use the reader + (loop + indent-and-symbols + #f ; in-indent? + #f ; in-underscoreindent? + #f ; in-comment? + currentindent + ;; this also takes care of the hashbang and leading comments. + (append currentsymbols (list (wisp-read port))) + emptylines)))))))) + + +(define (line-code-replace-inline-colons line) + "Replace inline colons by opening parens which close at the end of the line" + ;; format #t "replace inline colons for line ~A\n" line + (let loop ((processed '()) + (unprocessed line)) + (cond + ((null? unprocessed) + ;; format #t "inline-colons processed line: ~A\n" processed + processed) + ;; replace : . with nothing + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) + (loop + (append processed + (loop '() (cdr (cdr unprocessed)))) + '())) + ((equal? readcolon (car unprocessed)) + (loop + (append processed + (list (loop '() (cdr unprocessed)))) + '())) + (else + (loop + (append processed + (list (car unprocessed))) + (cdr unprocessed)))))) + +(define (line-replace-inline-colons line) + (cons + (line-indent line) + (line-code-replace-inline-colons (line-code line)))) + +(define (line-strip-lone-colon line) + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." + (if (equal? (line-code line) (list readcolon)) + (make-line (line-indent line)) + line)) + +(define (line-finalize line) + "Process all wisp-specific information in a line and strip it" + (let ((l (line-code-replace-inline-colons + (line-strip-indentation-marker + (line-strip-lone-colon + (line-strip-continuation line)))))) + (when (not (null? (source-properties line))) + (catch #t + (lambda () + (set-source-properties! l (source-properties line))) + (lambda (key . arguments) + #f))) + l)) + +(define (wisp-propagate-source-properties code) + "Propagate the source properties from the sourrounding list into every part of the code." + (let loop ((processed '()) + (unprocessed code)) + (cond + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) + unprocessed) + ((and (pair? unprocessed) (not (list? unprocessed))) + (cons + (wisp-propagate-source-properties (car unprocessed)) + (wisp-propagate-source-properties (cdr unprocessed)))) + ((null? unprocessed) + processed) + (else + (let ((line (car unprocessed))) + (wisp-add-source-properties-from/when-required line unprocessed) + (wisp-add-source-properties-from/when-required code unprocessed) + (wisp-add-source-properties-from/when-required unprocessed line) + (wisp-add-source-properties-from/when-required unprocessed code) + (let ((processed (append processed (list (wisp-propagate-source-properties line))))) + ;; must propagate from line, because unprocessed and code can be null, then they cannot keep source-properties. + (wisp-add-source-properties-from/when-required line processed) + (loop processed + (cdr unprocessed)))))))) + +(define* (wisp-scheme-indentation-to-parens lines) + "Add parentheses to lines and remove the indentation markers" + (when + (and + (not (null? lines)) + (not (line-empty-code? (car lines))) + (not (= 0 (line-real-indent (car lines))))); -1 is a line with a comment + (if (= 1 (line-real-indent (car lines))) + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands + (set! lines + (cons + (cons 0 (cdr (car lines))) + (cdr lines))) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" + (car lines))))))) + (let loop ((processed '()) + (unprocessed lines) + (indentation-levels '(0))) + (let* ((current-line + (if (<= 1 (length unprocessed)) + (car unprocessed) + (make-line 0))); empty code + (next-line + (if (<= 2 (length unprocessed)) + (car (cdr unprocessed)) + (make-line 0))); empty code + (current-indentation + (car indentation-levels)) + (current-line-indentation (line-real-indent current-line))) + ;; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" + ;; . processed current-line next-line unprocessed indentation-levels current-indentation + (cond + ;; the real end: this is reported to the outside world. + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) + ;; reverse the processed lines, because I use cons. + processed) + ;; the recursion end-condition + ((and (null? unprocessed)) + ;; this is the last step. Nothing more to do except + ;; for rolling up the indentation levels. return the + ;; new processed and unprocessed lists: this is a + ;; side-recursion + (values processed unprocessed)) + ((null? indentation-levels) + (raise-exception + (make-exception-from-throw + 'wisp-programming-error + (list + "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")))) + (else ; now we come to the line-comparisons and indentation-counting. + (cond + ((line-empty-code? current-line) + ;; We cannot process indentation without + ;; code. Just switch to the next line. This should + ;; only happen at the start of the recursion. + (loop + processed + (cdr unprocessed) + indentation-levels)) + ((and (line-empty-code? next-line) (<= 2 (length unprocessed))) + ;; take out the next-line from unprocessed. + (loop + processed + (cons current-line + (cdr (cdr unprocessed))) + indentation-levels)) + ((> current-indentation current-line-indentation) + ;; this just steps back one level via the side-recursion. + (let ((previous-indentation (car (cdr indentation-levels)))) + (if (<= current-line-indentation previous-indentation) + (values processed unprocessed) + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. + (let ((linenumber (- (length lines) (length unprocessed)))) + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + (cdr indentation-levels))))))) + ((= current-indentation current-line-indentation) + (let ((line (line-finalize current-line)) + (next-line-indentation (line-real-indent next-line))) + (cond + ((>= current-line-indentation next-line-indentation) + ;; simple recursiive step to the next line + (loop + (append processed + (if (line-continues? current-line) + line + (wisp-add-source-properties-from line (list line)))) + (cdr unprocessed); recursion here + indentation-levels)) + ((< current-line-indentation next-line-indentation) + ;; side-recursion via a sublist + (let-values + (((sub-processed sub-unprocessed) + (loop + line + (cdr unprocessed); recursion here + indentation-levels))) + (loop + (append processed (list sub-processed)) + sub-unprocessed ; simply use the recursion from the sub-recursion + indentation-levels)))))) + ((< current-indentation current-line-indentation) + (loop + processed + unprocessed + (cons ; recursion via the indentation-levels + current-line-indentation + indentation-levels))) + (else + (raise-exception + (make-exception-from-throw + 'wisp-not-implemented + (list + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." + current-line next-line processed))))))))))) + + +(define (wisp-scheme-replace-inline-colons lines) + "Replace inline colons by opening parens which close at the end of the line" + (let loop ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (list (line-replace-inline-colons (car unprocessed)))) + (cdr unprocessed))))) + + +(define (wisp-scheme-strip-indentation-markers lines) + "Strip the indentation markers from the beginning of the lines" + (let loop ((processed '()) + (unprocessed lines)) + (if (null? unprocessed) + processed + (loop + (append processed (cdr (car unprocessed))) + (cdr unprocessed))))) + +(define (wisp-unescape-underscore-and-colon code) + "replace \\_ and \\: by _ and :" + (wisp-add-source-properties-from/when-required + code + (cond ((list? code) (map wisp-unescape-underscore-and-colon code)) + ((eq? code '\:) ':) + ;; Look for symbols like \____ and remove the \. + ((symbol? code) + (let ((as-string (symbol->string code))) + (if (and (>= (string-length as-string) 2) ; at least a single underscore + (char=? (string-ref as-string 0) #\\) + (string-every #\_ (substring as-string 1))) + (string->symbol (substring as-string 1)) + code))) + (#t code)))) + + +(define (wisp-replace-empty-eof code) + "replace ((#<eof>)) by ()" + ;; This is a hack which fixes a bug when the + ;; parser hits files with only hashbang and comments. + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) + (wisp-add-source-properties-from code (list)) + code)) + + +(define (wisp-replace-paren-quotation-repr code) + "Replace lists starting with a quotation symbol by + quoted lists." + (wisp-add-source-properties-from/when-required + code + (match code + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b) + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b); this is the quoted empty list + (append + (map wisp-replace-paren-quotation-repr a) + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'syntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) + ;; literal array as start of a line: # (a b) c -> (#(a b) c) + ((#\# a ...) + (with-input-from-string ;; hack to defer to read + (string-append "#" + (with-output-to-string + (λ () + (write (map wisp-replace-paren-quotation-repr a) + (current-output-port))))) + read)) + ((a ...) + (map wisp-replace-paren-quotation-repr a)) + (a + a)))) + +(define (wisp-make-improper code) + "Turn (a #{.}# b) into the correct (a . b). + +read called on a single dot creates a variable named #{.}# (|.| +in r7rs). Due to parsing the indentation before the list +structure is known, the reader cannot create improper lists +when it reads a dot. So we have to take another pass over the +code to recreate the improper lists. + +Match is awesome!" + (define is-proper? #t) + ;; local alias + (define (add-prop/req form) + (wisp-add-source-properties-from/when-required code form)) + (wisp-add-source-properties-from/when-required + code + (let ((improper + (match code + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) + (set! is-proper? #f) + (wisp-add-source-properties-from/when-required + code + (append (map wisp-make-improper (map add-prop/req a)) + (cons (wisp-make-improper (add-prop/req b)) + (wisp-make-improper (add-prop/req c)))))) + ((a ...) + (add-prop/req + (map wisp-make-improper (map add-prop/req a)))) + (a + a)))) + (define (syntax-error li msg) + (raise-exception + (make-exception-from-throw + 'wisp-syntax-error + (list (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))))) + (if is-proper? + improper + (let check ((tocheck improper)) + (match tocheck + ;; lists with only one member + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "list with the period as only member")) + ;; list with remaining dot. + ((a ...) + (if (and (member repr-dot a)) + (syntax-error tocheck "leftover period in list") + (map check a))) + ;; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) + (syntax-error tocheck "dot as first element in already improper pair")) + ;; simple pair, other way round + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) + (syntax-error tocheck "dot as last element in already improper pair")) + ;; more complex pairs + ((? pair? a) + (let ((head (drop-right a 1)) + (tail (last-pair a))) + (cond + ((equal? repr-dot (car tail)) + (syntax-error tocheck "equal? repr-dot : car tail")) + ((equal? repr-dot (cdr tail)) + (syntax-error tocheck "equal? repr-dot : cdr tail")) + ((member repr-dot head) + (syntax-error tocheck "member repr-dot head")) + (else + a)))) + (a + a))))))) + +(define (wisp-scheme-read-chunk port) + "Read and parse one chunk of wisp-code" + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\#)) + (let ((lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines))))))))) + +(define (wisp-scheme-read-all port) + "Read all chunks from the given port" + (let loop ((tokens '())) + (cond + ((eof-object? (peek-char port)) + tokens) + (else + (loop + (append tokens (wisp-scheme-read-chunk port))))))) + +(define (wisp-scheme-read-file path) + (call-with-input-file path wisp-scheme-read-all)) + +(define (wisp-scheme-read-file-chunk path) + (call-with-input-file path wisp-scheme-read-chunk)) + +(define (wisp-scheme-read-string str) + (call-with-input-string str wisp-scheme-read-all)) + +(define (wisp-scheme-read-string-chunk str) + (call-with-input-string str wisp-scheme-read-chunk)) diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm new file mode 100644 index 000000000..f7fd794e0 --- /dev/null +++ b/module/language/wisp/spec.scm @@ -0,0 +1,70 @@ +;;; Language interface for Wisp in Guile + +;; Copyright (C) 2005--2014 by David A. Wheeler and Alan Manuel K. Gloria +;; Copyright (C) 2014--2023 Arne Babenhauserheide. +;; Copyright (C) 2023 Maxime Devos <maximedevos@telenet.be> + +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +;; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/?p=nacre:guile-sweet.git;a=blob;f=sweet/spec.scm;hb=ae306867e371cb4b56e00bb60a50d9a0b8353109 + +(define-module (language wisp spec) + #:use-module (language wisp) + #:use-module (system base compile) + #:use-module (system base language) + #:use-module (language scheme compile-tree-il) + #:use-module (language scheme decompile-tree-il) + #:export (wisp)) + +;;; +;;; Language definition +;;; + + +(define (read-one-wisp-sexp port env) + ;; Allow using "# foo" as #(foo). + ;; Don't use the globally-acting read-hash-extend, because this + ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme. + ;; Instead, use fluids to temporarily add the extension. + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\# )) + ;; Read Wisp files as UTF-8, to support non-ASCII characters. + ;; TODO: would be nice to support ';; coding: whatever' lines + ;; like in parenthese-y Scheme. + (set-port-encoding! port "UTF-8") + (if (eof-object? (peek-char port)) + (read-char port) ; return eof: we’re done + (let ((chunk (wisp-scheme-read-chunk port))) + (and (not (null? chunk)) ; <---- XXX: maybe (pair? chunk) + (car chunk)))))) + +(define-language wisp + #:title "Wisp Scheme Syntax. See SRFI-119 for details" + ;; . #:reader read-one-wisp-sexp + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; + #:compilers `((tree-il . ,compile-tree-il)) + #:decompilers `((tree-il . ,decompile-tree-il)) + #:evaluator (lambda (x module) (primitive-eval x)) + #:printer write ; TODO: backtransform to Wisp? Use source-properties? + #:make-default-environment + (lambda () + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', + ;; `fluid-set!', etc. don't have any effect in the current environment. + (let ((m (make-fresh-user-module))) + ;; Provide a separate `current-reader' fluid so that + ;; compile-time changes to `current-reader' are + ;; limited to the current compilation unit. + (module-define! m 'current-reader (make-fluid)) + m))) diff --git a/test-suite/Makefile.am b/test-suite/Makefile.am index 81e63bce2..247d97746 100644 --- a/test-suite/Makefile.am +++ b/test-suite/Makefile.am @@ -162,7 +162,8 @@ SCM_TESTS = tests/00-initial-env.test \ tests/srfi-98.test \ tests/srfi-105.test \ tests/srfi-111.test \ - tests/srfi-171.test \ + tests/srfi-119.test \ + tests/srfi-171.test \ tests/srfi-4.test \ tests/srfi-9.test \ tests/statprof.test \ diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test new file mode 100644 index 000000000..5390a25b3 --- /dev/null +++ b/test-suite/tests/srfi-119.test @@ -0,0 +1,1040 @@ +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*- +;;;; +;;;; Copyright (C) 2023 Free Software Foundation, Inc. +;;;; +;;;; This library is free software; you can redistribute it and/or +;;;; modify it under the terms of the GNU Lesser General Public +;;;; License as published by the Free Software Foundation; either +;;;; version 3 of the License, or (at your option) any later version. +;;;; +;;;; This library is distributed in the hope that it will be useful, +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;;; Lesser General Public License for more details. +;;;; +;;;; You should have received a copy of the GNU Lesser General Public +;;;; License along with this library; if not, write to the Free Software +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +(define-module (test-srfi-119) + #:use-module (test-suite lib) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) ;; cut + #:use-module (language wisp)) + +(define (read-string s) + (with-input-from-string s read)) + +(define (with-read-options opts thunk) + (let ((saved-options (read-options))) + (dynamic-wind + (lambda () + (read-options opts)) + thunk + (lambda () + (read-options saved-options))))) + +(define (wisp->list str) + (wisp-scheme-read-string str)) + +(define (scheme->list str) + (with-input-from-string str + (λ () + (let loop ((result '())) + (if (eof-object? (peek-char)) + (reverse! result) + (loop (cons (read) result))))))) + +(with-test-prefix "wisp-read-simple" + (pass-if-equal '((<= n 5)) + (wisp->list "<= n 5")) + (pass-if-equal '(5) + (wisp->list ". 5")) + (pass-if-equal '((+ 1 (* 2 3))) + (wisp->list "+ 1 : * 2 3"))) +(with-test-prefix "wisp-read-complex" + (pass-if-equal '( + (a b c d e + f g h + i j k) + + (concat "I want " + (getwish from me) + " - " username)) (wisp->list " +a b c d e + . f g h + . i j k + +concat \"I want \" + getwish from me + . \" - \" username +")) + + (pass-if-equal + '( + (define (a b c) + (d e + (f) + (g h) + i)) + + (define (_) + (display "hello\n")) + + (_)) (wisp->list " +define : a b c +_ d e +___ f +___ g h +__ . i + +define : _ +_ display \"hello\n\" + +\\_")) + + ;; nesting with pairs + (pass-if-equal '((1 . 2)(3 4 (5 . 6))) + (wisp->list "1 . 2\n3 4\n 5 . 6"))) + +(with-test-prefix "wisp-source-properties" + ;; has properties + (pass-if (every pair? (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6")))) + (pass-if (every pair? (map source-properties (wisp->list "1 2\n3 4\n 5 6")))) + ;; has the same properties + (pass-if-equal + (map source-properties (scheme->list "(1 . 2)\n(3 4\n (5 . 6))\n(1 4)\n\n(7 8)")) + (map (cut cons '(filename . #f) <>) + (map source-properties (wisp->list "1 . 2\n3 4\n 5 . 6\n1 4\n\n7 8"))))) + +(with-test-prefix "btest" + (pass-if-equal '((display "b") +(newline) + + +) (wisp->list " +display \"b\" +newline +"))) + +(with-test-prefix "continuation" + (pass-if-equal '((a b c d e + f g h + i j k) + +(concat "I want " + (getwish from me) + " - " username) + + + +) (wisp->list " +a b c d e + . f g h + . i j k + +concat \"I want \" + getwish from me + . \" - \" username + +"))) + +(with-test-prefix "dotted-pair" + (pass-if-equal '((use-modules ((ice-9 popen) #:select ((open-input-pipe . oip)))) + + + ) (wisp->list " +use-modules : (ice-9 popen) #:select ((open-input-pipe . oip)) +"))) + +(with-test-prefix "example" + (pass-if-equal '((defun a (b c) + (let + ( + (d "i am a string +do not break me!") + ( + ; comment: 0 + (f) +; comment : 1 + `(g )); comment " : " 2 + ( + (h (I am in brackets: + do not : change "me")) + i))) + ,('j k) + + l + +; comment + + (a c)) + +(defun b (:n o) + "second defun : with a docstring!" + (message "I am here") + t) + +(defun c (e f) + ((g)) + ( + (h + (i)) + (j)) + '(()) + (k) + l + (m)) + +(defun _ (:) + + :) + +(_ b) + +(defun d () + (let + ((a b) + (c d)))) + +(a (((c)))) + +(let + ((a b) + (c))) + +(let ((a b))) + +a + + + +) (wisp->list " +defun a (b c) + let + : + d \"i am a string +do not break me!\" + : + ; comment: 0 + f +; comment : 1 + ` g ; comment \" : \" 2 + : + h (I am in brackets: + do not : change \"me\") + . i + , 'j k + + . l + +; comment + + a c + +defun b : :n o + . \"second defun : with a docstring!\" + message \"I am here\" + . t + +defun c : e f + : g + : + h + i + j + ' : + k + . l + . : m + +defun _ : \\: +__ +__ . \\: + +\\_ b + +defun d : + let + : a b + c d + +a : : : c + +let + : a b + c + +let : : a b + +. a + +"))) + +(with-test-prefix "factorial" + (pass-if-equal '(;; short version +; note: once you use one inline colon, all the following forms on that +; line will get closed at the end of the line + +(define (factorial n) + (if (zero? n) + 1 + (* n (factorial (- n 1))))) + +(display (factorial 5 )) + + +;; more vertical space, less colons +(define (factorial n) + (if (zero? n) + 1 + (* n + (factorial + (- n 1))))) + +(display (factorial 5 )) + + + +) (wisp->list " +;; short version +; note: once you use one inline colon, all the following forms on that +; line will get closed at the end of the line + +define : factorial n + if : zero? n + . 1 + * n : factorial : - n 1 + +display : factorial 5 + + +;; more vertical space, less colons +define : factorial n + if : zero? n + . 1 + * n + factorial + - n 1 + +display : factorial 5 + +"))) + +(with-test-prefix "fast-sum" + (pass-if-equal '((use-modules (srfi srfi-1)) + +; only for the nice test +#!curly-infix + +(define-syntax fast-sum + (syntax-rules (iota) + ((fast-sum (iota count start)) + (+ 1 + (apply - + (map (lambda (x) (/ {x * {x + 1} } 2)) + (list {count + {start - 1}} start))))) + ((fast-sum e) + (apply + e)))) + + +) (wisp->list " +use-modules : srfi srfi-1 + +; only for the nice test +. #!curly-infix + +define-syntax fast-sum + syntax-rules : iota + : fast-sum : iota count start + + 1 + apply - + map : lambda (x) : / {x * {x + 1} } 2 + list {count + {start - 1}} start + : fast-sum e + apply + e +"))) + +(with-test-prefix "flexible-parameter-list" + (pass-if-equal '(; Test using a . as first parameter on a line by prefixing it with a second . +(define + (a i + . b) + (unless (>= i (length b)) + (display (number->string (length b ))) + (display (list-ref b i)) + (newline) + (apply a ( + i 1 ) b))) + + +(a 0 "123" "345" "567") + + +) (wisp->list " +; Test using a . as first parameter on a line by prefixing it with a second . +define + a i + . . b + unless : >= i : length b + display : number->string : length b + display : list-ref b i + newline + apply a ( + i 1 ) b + + +a 0 \"123\" \"345\" \"567\" +"))) + +(with-test-prefix "hello" + (pass-if-equal '((define (hello who) + ;; include the newline + (format #t "~A ~A!\n" + "Hello" who)) +(hello "Wisp") + +) (wisp->list " +define : hello who + ;; include the newline + format #t \"~A ~A!\\n\" + . \"Hello\" who +hello \"Wisp\" +"))) + +(with-test-prefix "mtest" + (pass-if-equal '(#!/home/arne/wisp/wisp-multiline.sh !# + +(display 1) + + +) (wisp->list " +#!/home/arne/wisp/wisp-multiline.sh !# + +display 1 +"))) + +(with-test-prefix "multiline-string" + (pass-if-equal '((display " + This is a + \"multi-line\" + string. + ") + + ) (wisp->list " +display \" + This is a + \\\"multi-line\\\" + string. + \" +"))) + +(with-test-prefix "namedlet" + (pass-if-equal '(#!/home/arne/wisp/wisp-multiline.sh +; !# +(define (hello who) + (display who)) + +(let hello + ((who 0)) + (if (= who 5) + (display who) + (hello (+ 1 who)))) + + +) (wisp->list " +#!/home/arne/wisp/wisp-multiline.sh +; !# +define : hello who + display who + +let hello + : who 0 + if : = who 5 + display who + hello : + 1 who +"))) + +;; the following is no error, but produces a warning because indentation is inconsistent. +(with-test-prefix "partial-indent" + (pass-if-equal '((write + (list + (+ 1 2) + (+ 2 3))) +(newline) + +(write + (list + (+ 1 2 + (+ 3 4)) + (+ 2 3))) +(newline) + + +) (wisp->list " +write + list + + 1 2 + + 2 3 +newline + +write + list + + 1 2 + + 3 4 + + 2 3 +newline +"))) + +(with-test-prefix "quotecolon" + (pass-if-equal '(#!/home/arne/wisp/wisp-multiline.sh +; !# +(define a 1 ); test whether ' : correctly gets turned into '( +; and whether brackets in commments are treated correctly. + +(define a '(1 2 3)) + +(define + (a b) + (c)) + +(define a (quasiquote ,(+ 2 2))) +) (wisp->list " +#!/home/arne/wisp/wisp-multiline.sh +; !# +define a 1 ; test whether ' : correctly gets turned into '( +; and whether brackets in commments are treated correctly. + +define a ' : 1 2 3 + +define + a b + c + +define a : quasiquote , : + 2 2 +"))) + +(with-test-prefix "range" + (pass-if-equal '((import (rnrs)) + +(define range + (case-lambda + ((n ); one-argument syntax + (range 0 n 1)) + ((n0 n ); two-argument syntax + (range n0 n 1)) + ((n0 n s ); three-argument syntax + (assert + (and + (for-all number? (list n0 n s)) + (not (zero? s)))) + (let ((cmp (if (positive? s) >= <= ))) + (let loop + ((i n0 ) + (acc '())) + (if + (cmp i n ) + (reverse acc) + (loop (+ i s) (cons i acc)))))))) + +(display (apply string-append "" (map number->string (range 5)))) +(newline) + +) (wisp->list " +import : rnrs + +define range + case-lambda + : n ; one-argument syntax + range 0 n 1 + : n0 n ; two-argument syntax + range n0 n 1 + : n0 n s ; three-argument syntax + assert + and + for-all number? : list n0 n s + not : zero? s + let : : cmp : if (positive? s) >= <= + let loop + : i n0 + acc '() + if + cmp i n + reverse acc + loop (+ i s) (cons i acc) + +display : apply string-append \"\" : map number->string : range 5 +newline"))) + +(with-test-prefix "readable-tests" + (pass-if-equal '((define (fibfast n) + (if (< n 2)) + n + (fibup n 2 1 0 )) + +(define (fibup maxnum count n-1 n-2) + (if (= maxnum count) + (+ n-1 n-2) + (fibup maxnum + (+ count 1 ) + (+ n-1 n-2 ) + n-1))) + +(define (factorial n) + (if (<= n 1) + 1 + (* n + (factorial (- n 1))))) + +(define (gcd x y) + (if (= y 0)) + x + (gcd y + (rem x y))) + +(define (add-if-all-numbers lst) + (call/cc + (lambda (exit) + (let loop + ( + (lst lst ) + (sum 0)) + (if (null? lst) + sum + (if (not (number? (car lst))) + (exit #f) + (+ (car lst) + (loop (cdr lst))))))))) + +) (wisp->list " +define : fibfast n + if : < n 2 + . n + fibup n 2 1 0 + +define : fibup maxnum count n-1 n-2 + if : = maxnum count + + n-1 n-2 + fibup maxnum + + count 1 + + n-1 n-2 + . n-1 + +define : factorial n + if : <= n 1 + . 1 + * n + factorial : - n 1 + +define (gcd x y) + if (= y 0) + . x + gcd y + rem x y + +define : add-if-all-numbers lst + call/cc + lambda : exit + let loop + : + lst lst + sum 0 + if : null? lst + . sum + if : not : number? : car lst + exit #f + + : car lst + loop : cdr lst"))) + +(with-test-prefix "receive" + (pass-if-equal '((import (ice-9 receive) (srfi srfi-1)) +(write + (receive (car cdr) + (car+cdr '(car . cdr)) + car)) +) (wisp->list " +import (ice-9 receive) (srfi srfi-1) +write + receive : car cdr + car+cdr '(car . cdr) + . car +"))) + +(with-test-prefix "self-referencial" + (pass-if-equal '(; http://stackoverflow.com/questions/23167464/scheme-self-reference-lambda-macro +; because this is as cool as things get +(define-syntax slambda + (lambda (x) + (syntax-case x () + ((slambda formals body0 body1 ...) + (with-syntax + ((self (datum->syntax #'slambda 'self))) + #'(letrec ((self (lambda formals body0 body1 ...))) + self)))))) + + + +( + (slambda (x) (+ x 1)) + 10) + +((slambda () self)) + + + +) (wisp->list " +; http://stackoverflow.com/questions/23167464/scheme-self-reference-lambda-macro +; because this is as cool as things get +define-syntax slambda + lambda : x + syntax-case x : + : slambda formals body0 body1 ... + with-syntax + : self : datum->syntax #'slambda 'self + #' letrec : : self : lambda formals body0 body1 ... + . self + + + +: + slambda (x) : + x 1 + . 10 + +: slambda () self + +"))) + +(with-test-prefix "shebang" + (pass-if-equal '(#!/usr/bin/wisp.py # !# +; This tests shebang lines + + + ) (wisp->list " +#!/usr/bin/wisp.py # !# +; This tests shebang lines +"))) + +(with-test-prefix "strangecomments" + (pass-if-equal '((use-modules (wisp-scheme)) +; works +(display + (call-with-input-string "foo ; bar\n ; nop \n\n; nup\n; nup \n \n\n\n foo : moo \"\n\" \n___ . goo . hoo" wisp-scheme-read-chunk)) +(newline) +(display + (call-with-input-string "foo \n___ . goo . hoo" wisp-scheme-read-chunk)) +(newline) + +) (wisp->list " +use-modules : wisp-scheme +; works +display + call-with-input-string \"foo ; bar\\n ; nop \\n\\n; nup\\n; nup \\n \\n\\n\\n foo : moo \\\"\\n\\\" \\n___ . goo . hoo\" wisp-scheme-read-chunk +newline +display + call-with-input-string \"foo \\n___ . goo . hoo\" wisp-scheme-read-chunk +newline +"))) + +(with-test-prefix "sublist" +(pass-if-equal '(; sublists allow to start single line function calls with a colon ( : ). + +(defun a (b c) + (let ((e . f)) + g)) + + +) (wisp->list " +; sublists allow to start single line function calls with a colon ( : ). +; +defun a : b c + let : : e . f + . g + "))) + +(with-test-prefix "sxml" +(pass-if-equal '((use-modules (sxml simple)) +(use-modules (ice-9 match)) + +; define a template +(define template + (quote + (html + (head (title "test")) + (body + (h1 "test") + (message "the header") + (p "it " (em "works!") + (br) + (" it actually works!")))))) + +; transform it +(define template2 + (let loop + ((l template)) + (match l + (('message a ...) + `(p (@ (style "margin-left: 2em")) + (strong ,(map loop a)))) + ((a ...) + (map loop a )) + (a + a)))) + +; write xml to the output port +(sxml->xml template2) + +(newline) + + +) (wisp->list " + use-modules : sxml simple +use-modules : ice-9 match + +; define a template +define template + quote + html + head : title \"test\" + body + h1 \"test\" + message \"the header\" + p \"it \" : em \"works!\" + br + \" it actually works!\" + +; transform it +define template2 + let loop + : l template + match l + : 'message a ... + ` p : @ : style \"margin-left: 2em\" + strong ,(map loop a) + : a ... + map loop a + a + . a + +; write xml to the output port +sxml->xml template2 + +newline +"))) + +(with-test-prefix "syntax-colon" +(pass-if-equal '((let + ( + (a 1) + (b 2)) + (let + ( + ( + c 3)) + (format #t "a: ~A, b: ~A, c: ~A" + a b c))) + +((a)) + +(define (hello) + (display "hello\n")) + +(let + ((a 1) + (b 2)) + (format #t "a: ~A, b: ~A" + a b)) + +(let ((a '()))) + +(let + ( ; foo + (a + '()))) + +( + (a)) + +(define (:) + (hello)) + +(:) + + +) (wisp->list " +let + : + a 1 + b 2 + let + : + : + . c 3 + format #t \"a: ~A, b: ~A, c: ~A\" + . a b c + +: a + +define : hello + display \"hello\\n\" + +let + : a 1 + b 2 + format #t \"a: ~A, b: ~A\" + . a b + +let : : a ' : + +let + : ; foo + a + ' + +: + a + +define : \\: + hello + +\\: +"))) + +(with-test-prefix "syntax-dot" +(pass-if-equal '((define (foo) + "bar") + +(define (bar) + '(1 + . 2 )); pair + +(display (foo)) +(newline) +(display (bar)) +(newline) + + +) (wisp->list " +define : foo + . \"bar\" + +define : bar + ' 1 + . . 2 ; pair + +display : foo +newline +display : bar +newline +"))) + +(with-test-prefix "syntax-empty" +(pass-if-equal '() (wisp->list " +"))) + +(with-test-prefix "syntax-indent" +(pass-if-equal '((define + (hello who) + (format #t "Hello ~A\n" who)) + +(define + (let + ( + (a 1) + (b 2) + (c 3)) + (format #t "a: ~A, b: ~A, c: ~A" + (+ a 2) + b c))) + + + +) (wisp->list "define + hello who + format #t \"Hello ~A\\n\" who + +define + let + : + a 1 + b 2 + c 3 + format #t \"a: ~A, b: ~A, c: ~A\" + + a 2 + . b c + + "))) + +(with-test-prefix "syntax-strings-parens" +(pass-if-equal '(; Test linebreaks in strings and brackets + +"flubbub + +flabbab" + +(hrug (nadda +madda gadda "shoktom + mee" " sep +ka" + hadda) + (gom)) + +(flu) + +(sum [foo +bar] barz {1 + [* 2 2]}) + +(mara { +li ++ +lo - (mabba) +}) +) (wisp->list " +; Test linebreaks in strings and brackets + +. \"flubbub + +flabbab\" + +hrug (nadda +madda gadda \"shoktom + mee\" \" sep +ka\" + hadda) + gom + +flu + +sum [foo +bar] barz {1 + [* 2 2]} + +mara { +li ++ +lo - (mabba) +} +"))) + +(with-test-prefix "syntax-underscore" +(pass-if-equal '((define (a b c) + (d e + (f) + (g h) + i)) + +(define (_) + (display "hello\n")) + +(_) + + +) (wisp->list " +define : a b c +_ d e +___ f +___ g h +__ . i + +define : _ +_ display \"hello\\n\" + +\\_ +"))) + -- 2.41.0 [-- Attachment #1.3: Type: text/plain, Size: 1382 bytes --] "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > I just got the most beautiful feedback on Wisp as a Scheme primer, so I > would like to nag about inclusion of SRFI-119 into Guile again: > > »I tend to use [Wisp] as a Scheme primer for colleagues that are used > to Python but want to explore the realms of functional programming > without … having to break with known syntax and conventions … it makes > Scheme way more “approachable” while not drifting too much apart. > > Before using Wisp as an introduction, I got occasional feedback that > parens where seen as confusing/somewhat cluttery in terms of > readability; Wisp looks as clean, plain and readable as Python code; > which makes it relatively easy to awake interest for Scheme in folks > that are accustomed to Python already.« — Wilko > https://emacs.ch/@thees/111720771253976695 > > > Inclusion in Guile would make it much easier for them to just try Wisp > on any of the supported platforms, so I think this is a strong indicator > that inclusion of Wisp would help spread Guile Scheme. => Is the squashed patch ok to merge? Since it’s already rebased and the copyright assignment is in place, I only need a “go for it” to push it. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) 2024-03-11 1:16 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) Dr. Arne Babenhauserheide @ 2024-06-01 9:57 ` Ludovic Courtès 2024-06-01 15:06 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Ludovic Courtès @ 2024-06-01 9:57 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel Hi Arne, "Dr. Arne Babenhauserheide" <arne_bab@web.de> skribis: > From 81e7cbbade4fd01e092a2c39a3af90e4abf7f684 Mon Sep 17 00:00:00 2001 > From: Arne Babenhauserheide <arne_bab@web.de> > Date: Mon, 11 Mar 2024 06:34:52 +0100 > Subject: [PATCH] Add language/wisp, Wisp tests, and SRFI-119 documentation > > * doc/ref/srfi-modules.texi (srfi-119): add node > * module/language/wisp.scm: New file. > * module/language/wisp/spec.scm: New file. > * test-suite/tests/srfi-119.test: New file. I have the pleasure to inform you that I have finally pushed this! :-) Apologies for taking so long, and thank you for being patient. Some of the suggestions I made earlier¹ were still not implemented though: 1. Using uninterned symbols rather than UUIDs. 2. Using a record type for lines instead of tuples. 3. Avoiding source properties. I took the liberty to implement #1 in commit 27feb2bfd38087cf03989673da0fc74ed795307d. Tests pass but please let me know if you notice something wrong! It’d be great if you could look at #2 and #3 along the lines of what I suggested earlier. This time we should be able to move forward more quickly. :-) Thanks! Ludo’. ¹ https://lists.gnu.org/archive/html/guile-devel/2023-08/msg00009.html ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) 2024-06-01 9:57 ` Ludovic Courtès @ 2024-06-01 15:06 ` Dr. Arne Babenhauserheide 2024-06-07 5:44 ` Damien Mattei 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-06-01 15:06 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 1160 bytes --] Hi Ludo’, Ludovic Courtès <ludo@gnu.org> writes: > I have the pleasure to inform you that I have finally pushed this! :-) > > Apologies for taking so long, and thank you for being patient. Thank you for reviewing and pushing it! > Some of the suggestions I made earlier¹ were still not implemented > though: > > 1. Using uninterned symbols rather than UUIDs. > > 2. Using a record type for lines instead of tuples. > > 3. Avoiding source properties. > > I took the liberty to implement #1 in commit > 27feb2bfd38087cf03989673da0fc74ed795307d. Tests pass but please let me > know if you notice something wrong! Wow, you got it working! I tried repeatedly and didn’t manage (always failed on edge-cases). Creating the λ inside the match looks pretty readable. Thank you! > It’d be great if you could look at #2 and #3 along the lines of what I > suggested earlier. This time we should be able to move forward more > quickly. :-) I’ll look into them again. It’s great to be moving forward! Thank you! - Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) 2024-06-01 15:06 ` Dr. Arne Babenhauserheide @ 2024-06-07 5:44 ` Damien Mattei 2024-06-10 8:13 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch withmore " Maxime Devos 2024-06-13 20:29 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more " Dr. Arne Babenhauserheide 0 siblings, 2 replies; 83+ messages in thread From: Damien Mattei @ 2024-06-07 5:44 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 1542 bytes --] Hello, where is the doc of Wisp? i did not yet install it, as i understand it will be included in next release of Guile? what is the exact meaning of : and . ? any simple examples? regards, Damien On Sat, Jun 1, 2024 at 5:07 PM Dr. Arne Babenhauserheide <arne_bab@web.de> wrote: > Hi Ludo’, > > Ludovic Courtès <ludo@gnu.org> writes: > > I have the pleasure to inform you that I have finally pushed this! :-) > > > > Apologies for taking so long, and thank you for being patient. > > Thank you for reviewing and pushing it! > > > Some of the suggestions I made earlier¹ were still not implemented > > though: > > > > 1. Using uninterned symbols rather than UUIDs. > > > > 2. Using a record type for lines instead of tuples. > > > > 3. Avoiding source properties. > > > > I took the liberty to implement #1 in commit > > 27feb2bfd38087cf03989673da0fc74ed795307d. Tests pass but please let me > > know if you notice something wrong! > > Wow, you got it working! > > I tried repeatedly and didn’t manage (always failed on edge-cases). > > Creating the λ inside the match looks pretty readable. > > Thank you! > > > It’d be great if you could look at #2 and #3 along the lines of what I > > suggested earlier. This time we should be able to move forward more > > quickly. :-) > > I’ll look into them again. > > It’s great to be moving forward! > > Thank you! > > - Arne > -- > Unpolitisch sein > heißt politisch sein, > ohne es zu merken. > draketo.de > [-- Attachment #2: Type: text/html, Size: 2637 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* RE: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch withmore tests, squashed) 2024-06-07 5:44 ` Damien Mattei @ 2024-06-10 8:13 ` Maxime Devos 2024-06-10 23:03 ` Damien Mattei 2024-06-13 20:29 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more " Dr. Arne Babenhauserheide 1 sibling, 1 reply; 83+ messages in thread From: Maxime Devos @ 2024-06-10 8:13 UTC (permalink / raw) To: Damien Mattei, Dr. Arne Babenhauserheide; +Cc: guile-devel@gnu.org [-- Attachment #1: Type: text/plain, Size: 223 bytes --] >where is the doc of Wisp? i did not yet install it, as i understand it will be included in next release of Guile? Going by the the commit message, it now is in: > * doc/ref/srfi-modules.texi (srfi-119): add node [-- Attachment #2: Type: text/html, Size: 1553 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch withmore tests, squashed) 2024-06-10 8:13 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch withmore " Maxime Devos @ 2024-06-10 23:03 ` Damien Mattei 0 siblings, 0 replies; 83+ messages in thread From: Damien Mattei @ 2024-06-10 23:03 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel@gnu.org [-- Attachment #1: Type: text/plain, Size: 387 bytes --] thank you ,Arne give me some links too. Damien On Mon, Jun 10, 2024 at 10:13 AM Maxime Devos <maximedevos@telenet.be> wrote: > > > >where is the doc of Wisp? i did not yet install it, as i understand it > will be included in next release of Guile? > > > > Going by the the commit message, it now is in: > > > > > * doc/ref/srfi-modules.texi (srfi-119): add node > [-- Attachment #2: Type: text/html, Size: 1295 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) 2024-06-07 5:44 ` Damien Mattei 2024-06-10 8:13 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch withmore " Maxime Devos @ 2024-06-13 20:29 ` Dr. Arne Babenhauserheide 1 sibling, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-06-13 20:29 UTC (permalink / raw) To: Damien Mattei; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 3258 bytes --] Hi, I realized that I did not actually send the reply to the list. Damien Mattei <damien.mattei@gmail.com> writes: > where is the doc of Wisp? i did not yet install it, as i understand it will be included in next release of Guile? The documentation is either in the SRFI or on the wisp website: - https://srfi.schemers.org/srfi-119/srfi-119.html - https://www.draketo.de/software/wisp - https://www.draketo.de/software/wisp#after-updates If that’s useful, I’ll gladly send a patch to add it to the Guile reference manual. > what is the exact meaning of : and . ? These control how parentheses are added to turn wisp with indentation into regular Scheme. In general every non-empty line opens a paren-pair which closes before the next line with lower or equal indentation: hello world is (hello world) hello append world "!" newline is (hello (append world "!")) (newline) If you start the line with ., it does not start with a paren. + 1 . 2 3 is (+ 1 2 3) If you need double-parentheses, you mark one line with : (only whitespace, one :, and comments) let : a 1 b 2 + a b is (let ( (a 1) (b 2)) (+ a b)) If : is in a line with other non-whitespace non-comments, it starts an inline paren-pair that ends at the end of the line. define : hello world display "Hello " display world is (define (hello world) (display "Hello ") (display world)) That can also make let look nicer: let : a 1 b 2 + a b is (let ((a 1) (b 2)) (+ a b)) > any simple examples? display "Hello World!" ↦ (display "Hello World!") define : factorial n ↦ (define (factorial n) if : zero? n (if (zero? n) . 1 1 * n : factorial {n - 1} (* n (factorial {n - 1})))) > is it possible in Wisp to use normal scheme expression? can we mix the > scheme code and wisp code? you can use normal scheme in wisp. Within parentheses, indentation-processing is then disabled (so you can simply copy-paste scheme into wisp). For example this is valid Wisp: define (hello world) display (string-append "Hello " world "!") newline hello "Wisp" The reason is that the goal of wisp is to live within a Scheme world, not replace Scheme. So the design assumes that the dominant language will be Scheme. The hardest part I see with such embedding is to stay simple. Every syntax adaption is a risk. That’s the main reason why Wisp split off from readable (srfi-110): readable added more and more special cases and lost the simplicity I see as one of the big strength of Scheme. Wisp in contrast is intentionally limited to the minimum set of additions needed to create a usable indentation-based Scheme. For example the dot-prefix (.) does not take away existing syntax because the dot is used in a position where it is illegal in regular Scheme: (. 1 2 3) is a syntax error. But in many Schemes, (= 1 (. 1)). Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-09 7:05 ` Dr. Arne Babenhauserheide 2024-01-19 8:21 ` Dr. Arne Babenhauserheide @ 2024-01-19 12:10 ` Christina O'Donnell 2024-01-19 21:37 ` Ricardo Wurmus 2024-01-19 23:56 ` Dr. Arne Babenhauserheide 1 sibling, 2 replies; 83+ messages in thread From: Christina O'Donnell @ 2024-01-19 12:10 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel, Ludovic Courtès Hi Arne, On 09/01/2024 07:05, Dr. Arne Babenhauserheide wrote: > It’s a new year — any chance for one more look whether adding SRFI-119 > in Guile is ok to merge? As a disclaimer, I'm a Scheme newbie, but I think my opinion may have /some/ value. On its own, Wisp seems like a better syntax for tooling that looks at files as a list of lines, rather than as a tree of S-expressions. For example, the diff tool looks for changes in lines, so when adding a value to the end of a list it will always show 1 extra line removed and then added again with one less paren. For example (from a record definition): (device-tree-support? bootloader-configuration-device-tree-support? - (default #t))) ;boolean + (default #t)) ;boolean + (extra-initrd bootloader-configuration-extra-initrd + (default #f))) ;string | #f Whereas in Wisp syntax, you'd simply get: device-tree-support? bootloader-configuration-device-tree-support? default #t ;boolean + extra-initrd bootloader-configuration-extra-initrd + default #f ;string | #f Likewise, commenting code in scheme is a pain because code-commenting is done by line. R6RS comments (#;) aren't implemented in Guile to my knowledge. So parenthesis aware commenting is left up to the editor to implement. I could say the same about editors like Vim and many other editors that see the world as a list of lines. However it could cause some fragmentation of the community as peoples editors are set up for Lisp and not Wisp or vice versa. Though, I think that could be mostly resolved if there was a script that could convert Wisp to Lisp and back. Ideally such that on a large code base there's very few instances where Lisp -> Wisp -> Lisp produces changes the code even by white-space. But, I don't know whether even then you'd find much interest from the other maintainers. Kind regards, - Christina ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-19 12:10 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Christina O'Donnell @ 2024-01-19 21:37 ` Ricardo Wurmus 2024-01-19 21:47 ` Christina O'Donnell 2024-01-19 23:56 ` Dr. Arne Babenhauserheide 1 sibling, 1 reply; 83+ messages in thread From: Ricardo Wurmus @ 2024-01-19 21:37 UTC (permalink / raw) To: Christina O'Donnell Cc: Dr. Arne Babenhauserheide, Ludovic Courtès, guile-devel Christina O'Donnell <cdo@mutix.org> writes: > Likewise, commenting code in scheme is a pain because code-commenting > is done by > line. R6RS comments (#;) aren't implemented in Guile to my knowledge. So > parenthesis aware commenting is left up to the editor to implement. #; works in Guile. -- Ricardo ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-19 21:37 ` Ricardo Wurmus @ 2024-01-19 21:47 ` Christina O'Donnell 2024-01-20 11:01 ` Damien Mattei 0 siblings, 1 reply; 83+ messages in thread From: Christina O'Donnell @ 2024-01-19 21:47 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guile-devel Hi Ricardo, > #; works in Guile. Ah, that's my bad. - Christina ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-19 21:47 ` Christina O'Donnell @ 2024-01-20 11:01 ` Damien Mattei 2024-01-20 19:18 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Damien Mattei @ 2024-01-20 11:01 UTC (permalink / raw) To: Christina O'Donnell; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 1117 bytes --] hello Christina, i just add a #; support in Scheme+ : https://github.com/damien-mattei/Scheme-PLUS-for-Guile Scheme+ is an extension syntax to Scheme. It goes in the opposite direction of Wisp or Rhombus (based on Racket) by keeping the same number of parenthesis, but they are just differents : ( ), { }, [ ] and it allow the use of infix expressions. It is my idea, i think Scheme musk keep a strong identity with parenthesis. Parenthesis have advantage on indentation (and inconvenient i admit). For example if you comment a single line of indented code (python for example) that was used to define a loop you have to reindent all the body code in the loop. If you want to move a block of indented code you then have to re-indent it specifically for the new place too. Anyway scheme and scheme+ ,C ,C++ code are still indented for readness even if parentheses would be enough to validate the syntax. Regards, Damien On Fri, Jan 19, 2024 at 10:47 PM Christina O'Donnell <cdo@mutix.org> wrote: > Hi Ricardo, > > > #; works in Guile. > > Ah, that's my bad. > > - Christina > > [-- Attachment #2: Type: text/html, Size: 2042 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-20 11:01 ` Damien Mattei @ 2024-01-20 19:18 ` Dr. Arne Babenhauserheide 2024-01-20 22:59 ` Damien Mattei 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-01-20 19:18 UTC (permalink / raw) To: Damien Mattei; +Cc: Christina O'Donnell, guile-devel [-- Attachment #1: Type: text/plain, Size: 3692 bytes --] Damien Mattei <damien.mattei@gmail.com> writes: > i just add a #; support in Scheme+ : https://github.com/damien-mattei/Scheme-PLUS-for-Guile > Scheme+ is an extension syntax to Scheme. It goes in the opposite direction of Wisp or Rhombus (based on Racket) Note that Wisp and Rhombus differ: Wisp is just Scheme that allows reducing parentheses where they can be inferred from indentation. In intentionally avoids doing anything complex, so it keeps compatibility with future improvements of Scheme. If you want to improve beyond what Wisp offers, the best way is usually to build appropriate control structures that can also be used in regular Scheme. > by keeping the same > number of parenthesis, but they are just differents : ( ), { }, [ ] > and it allow the use of infix expressions. Is your implementation of {} compatible with SRFI-105? (curly infix) Can it be used to provide proper operator precedence? (that’s currently not offered by SRFI-105 and would be quite an improvement) > Parenthesis have advantage on indentation (and inconvenient i admit). For example if you comment a single line of indented code (python for > example) that was used to define a loop you have to reindent all the body code in the loop. If you want to move a block of indented code you > then have to re-indent it specifically for the new place too. To some degree yes, though rectangle-editing in Emacs usually makes that painless. > Anyway scheme and scheme+ ,C ,C++ code are still indented for readness even if parentheses would be enough to validate the syntax. One point of wisp is that having a parenthesis as first element on almost every line makes it harder for people to understand the code as long as they have not yet trained their eyes to fade parens. Indentation is what’s easier to humans to understand at a glance, so if you want to prevent mistakes, you have to indent strictly by the parens. Wisp follows Python in the reasoning that if the parens *always* strictly define the indentation, you can just edit the indentation in the first place without the "detour" over the parens. An alternate way of getting something similar is parinfer which strictly couples parens and indentation while editing: https://shaunlebron.github.io/parinfer/ There are also cases where parentheses are clearer, even for newcomers. For these, Wisp defers to pure paren-processing inside parentheses (this also enables simple copy-pasting of regular Scheme code). Combining rainbow parens (or other ways to fade parens) with parinfer gives similar reading experience as with wisp, but only for those in specialized editors, so it doesn’t help newcomers. On the other hand, a mode like lispy provides expert users with a level of convenience during editing parethesized code which clearly outclasses editing Wisp. So these show clear reasons why Wisp is most used in the niche of newcomers: - it looks easier without specialized modes, including patches - it makes it easier to do simple changes (you don’t have to keep parentheses balanced) - it makes it *harder* to build advanced tooling which does in-place editing of the AST and simply reflows the code, because switching between styles (i.e. adding linebreaks when something is inserted in a line) requires more decisions when you use wisp. I like it that way, because it encourages contributing code to the regular Scheme codebase, and not building a separate set of Wisp modules. That reduces the risk of splitting the community. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-20 19:18 ` Dr. Arne Babenhauserheide @ 2024-01-20 22:59 ` Damien Mattei 2024-01-20 23:22 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Damien Mattei @ 2024-01-20 22:59 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 1089 bytes --] On Sat, Jan 20, 2024 at 9:07 PM Dr. Arne Babenhauserheide <arne_bab@web.de> wrote: > > Damien Mattei <damien.mattei@gmail.com> writes: > > > > by keeping the same > > number of parenthesis, but they are just differents : ( ), { }, [ ] > > and it allow the use of infix expressions. > > Is your implementation of {} compatible with SRFI-105? (curly infix) > yes it is based on SRFI-105 > Can it be used to provide proper operator precedence? (that’s currently > not offered by SRFI-105 and would be quite an improvement) > yes there is operator precedence , an expression like : {{x - v * t} / (sqrt {1 - v ** 2 / c ** 2})} is converted in this one: (/ (- x (* v t)) (sqrt (- 1 (/ (** v 2) (** c 2))))) the operator precedence are predefined in a list ordered by priority in the source code: (define infix-operators-lst '( (expt **) (* / %) (+ -) (<< >>) (& ∣) (< > = ≠ <= >= <>) (and) (or) (<- -> ← → <v v> ⇜ ⇝) (<+ +> ⥆ ⥅) ) ) Best regards, Damien [-- Attachment #2: Type: text/html, Size: 2670 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-20 22:59 ` Damien Mattei @ 2024-01-20 23:22 ` Dr. Arne Babenhauserheide 2024-01-21 23:21 ` Damien Mattei 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-01-20 23:22 UTC (permalink / raw) To: Damien Mattei; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 892 bytes --] Damien Mattei <damien.mattei@gmail.com> writes: > On Sat, Jan 20, 2024 at 9:07 PM Dr. Arne Babenhauserheide <arne_bab@web.de> wrote: > Damien Mattei <damien.mattei@gmail.com> writes: > > by keeping the same > > number of parenthesis, but they are just differents : ( ), { }, [ ] > > and it allow the use of infix expressions. > > Is your implementation of {} compatible with SRFI-105? (curly infix) > > yes it is based on SRFI-105 That’s very cool! While I’m doubtful about your other changes in scheme+, this is an improvement that I’d love to see merged into Guile. > the operator precedence are predefined in a list ordered by priority in the source code: > > (define infix-operators-lst How does it treat procedures which are not in that list? Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-20 23:22 ` Dr. Arne Babenhauserheide @ 2024-01-21 23:21 ` Damien Mattei 0 siblings, 0 replies; 83+ messages in thread From: Damien Mattei @ 2024-01-21 23:21 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 2033 bytes --] On Sun, Jan 21, 2024 at 12:24 AM Dr. Arne Babenhauserheide <arne_bab@web.de> wrote: > > > > Is your implementation of {} compatible with SRFI-105? (curly infix) > > > > yes it is based on SRFI-105 > > That’s very cool! > > While I’m doubtful about your other changes in scheme+, this is an > improvement that I’d love to see merged into Guile. > it is available for many scheme, as a module, if there is a packet manager,for racket it is quite simple to install,anyway it is 100% scheme, so for guile it should be installed like another module, i do not know if it could be integrated in guile. > > > the operator precedence are predefined in a list ordered by priority in > the source code: > > > > (define infix-operators-lst > > How does it treat procedures which are not in that list? > first , operator precedence is based on this list: https://runestone.academy/ns/books/published/fopp/Conditionals/PrecedenceofOperators.html but i do not know what answer, which procedure are you thinking about? the list of operator is quite exhaustive already. If new operator was to add it is as simple as update the list. Note that changing operator precedence is the first time i hear about it, as far as i know, not possible in C++,Java, searching for Python , it find that: https://stackoverflow.com/questions/11811051/change-operator-precedence-in-python but the true answer seems NO: https://www.tutorialspoint.com/Can-we-change-operator-precedence-in-Python I had two methods for operator precedence, it was done at run-time so if you change rules it is immediately apply for the next computation. Now i added optimization at compilation , by parsing the source file before and all infix operator precedence is done before compilation, so it will no more possible to change the precedence rules. Note that in conjunction with operator overload too this is more useful but this could be complex to manage and implement. Not sure it is a good idea. Best regards, Damien [-- Attachment #2: Type: text/html, Size: 3978 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) 2024-01-19 12:10 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Christina O'Donnell 2024-01-19 21:37 ` Ricardo Wurmus @ 2024-01-19 23:56 ` Dr. Arne Babenhauserheide 1 sibling, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2024-01-19 23:56 UTC (permalink / raw) To: Christina O'Donnell; +Cc: Ludovic Courtès, guile-devel [-- Attachment #1: Type: text/plain, Size: 3497 bytes --] Christina O'Donnell <cdo@mutix.org> writes: > On 09/01/2024 07:05, Dr. Arne Babenhauserheide wrote: >> It’s a new year — any chance for one more look whether adding SRFI-119 >> in Guile is ok to merge? > > As a disclaimer, I'm a Scheme newbie, but I think my opinion may have /some/ > value. > On its own, Wisp seems like a better syntax for tooling that looks at > files as a list of lines, rather than as a tree of S-expressions. For > example, the diff tool looks for changes in lines, so when adding a > value to the end of a list it will always show 1 extra line removed > and then added again with one less paren. For example (from a record > definition): > > (device-tree-support? bootloader-configuration-device-tree-support? > - (default #t))) ;boolean > + (default #t)) ;boolean > + (extra-initrd bootloader-configuration-extra-initrd > + (default #f))) ;string | #f Looking at that may actually explain why many non-lisp-languages put closing parens (braces: }) on their own line. Some are even starting to do that for function arguments by allowing a comma after the last parameter. > However it could cause some fragmentation of the community as peoples > editors are set up for Lisp and not Wisp or vice versa. Though, I > think that could be mostly resolved if there was a script that could > convert Wisp to Lisp and back. Ideally such that on a large code base > there's very few instances where Lisp -> Wisp -> Lisp produces changes > the code even by white-space. But, I don't know whether even then > you'd find much interest from the other maintainers. there’s currently only a script to go from wisp to lisp (called wisp2lisp, shipped with wisp), but not the other way round — for three reasons: 1. that was the original implementation: just transform wisp to lisp, then run it as lisp. 2. the transformation from lisp to wisp is somewhat more complex. You need to decide whether to use inline-parentheses, a colon, or indentation. Though if the goal is to back-transform code transformed from wisp, that would be easier. 3. wisp2lisp is a contingency plan: if wisp turns out not to be right for you, you can simply migrate onwards to Scheme with a simple command. Part of this is that Wisp is not intended to take over Scheme. There is already so much Scheme code and Scheme tooling and knowledge about Scheme, and ongoing improvements in Scheme, that Wisp will work best by being a part of Scheme. To share them with non-Wisp Scheme, extract common parts, transform them to Scheme with wisp2lisp, and make these the canonical source. That way you get more tooling, better linters, compatibility to many more Scheme implementations, and far less required maintenance for Wisp. The more specialized tooling wisp would get, the harder it would be to actually work with all of Scheme and benefit from all of its improvements, because that tooling would have to play constant catch-up with the evolution of Scheme. So I think the sweet spot is to keep Wisp small, lean, and simple. This has worked well for the past decade, and it looks like the best way forward. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-24 15:45 ` Ludovic Courtès 2023-02-24 16:34 ` Dr. Arne Babenhauserheide @ 2023-02-24 23:48 ` Maxime Devos 2023-02-24 23:51 ` Maxime Devos 1 sibling, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-24 23:48 UTC (permalink / raw) To: Ludovic Courtès, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 1491 bytes --] On 24-02-2023 16:45, Ludovic Courtès wrote: >>> Adding #lang support in Guile would be nice. As discussed on IRC, it >>> can be experimented with in a WIP branch. >> Have you seen my messages on how the "#lang" construct is problematic >> for some languages, and how alternatives like "[comment delimiter] -*- >> stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be >> equally simple (*) and not have any downsides (**). >> >> (*) The port encoding detection supports "-*- coding: whatever -*-", >> presumably that functionality could be reused. >> >> (**) For compatibility with Racket, it's not like we couldn't >> implement both "#lang" and "-*- stuff: language -*-". > I haven’t seen your messages yet, I just wanted to express support of > the general idea. For years, we have discussed #lang support; I know > Andy is enthusiastic about it, and while I was initially reluctant, I’ve > come to appreciate the idea. > > What you point out is worth considering, but note that Guile already > supports #!r6rs for instance. While I initially thought that #!r6rs was problematic, it was pointed out to me that #!r6rs is actually in the R6RS (unlike #lang, and even if #lang was in the R6RS, it isn't in the de-jure or de-factor standards of many other languages). As such, #!r6rs is a different (unproblematic) situation compared to #lang -- #lang is not the same as #!r6rs from a compatibility perspective. Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-24 23:48 ` [PATCH] add language/wisp to Guile? Maxime Devos @ 2023-02-24 23:51 ` Maxime Devos 2023-02-25 0:15 ` Matt Wette 0 siblings, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-24 23:51 UTC (permalink / raw) To: Ludovic Courtès, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 403 bytes --] On 25-02-2023 00:48, Maxime Devos wrote: >>> (**) For compatibility with Racket, it's not like we couldn't >>> implement both "#lang" and "-*- stuff: language -*-". TBC, I mean ‘only support #lang' for values of 'lang' that Racket supports’, or alternatively ‘support #lang for compatibility, but recommend -*- ... -*- in the Guile manual.’. Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-24 23:51 ` Maxime Devos @ 2023-02-25 0:15 ` Matt Wette 2023-02-25 10:42 ` Maxime Devos 0 siblings, 1 reply; 83+ messages in thread From: Matt Wette @ 2023-02-25 0:15 UTC (permalink / raw) To: guile-devel On 2/24/23 3:51 PM, Maxime Devos wrote: > On 25-02-2023 00:48, Maxime Devos wrote: >>>> (**) For compatibility with Racket, it's not like we couldn't >>>> implement both "#lang" and "-*- stuff: language -*-". > > TBC, I mean ‘only support #lang' for values of 'lang' that Racket > supports’, or alternatively ‘support #lang for compatibility, but > recommend -*- ... -*- in the Guile manual.’. The point of this is to tell the compiler what language it needs to parse, if not the default (Scheme). Using `#lang pascal' or `#!lang pascal' allows the compiler to figure this out in the first few characters read. Using `-*- lang: pascal -*-' might require the compiler to read a lot of text (not knowing if what text is comment or not) to determine if language has been specified. For the `-*-' based design, is there a maximum number of characters to be read before deciding? Matt ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-25 0:15 ` Matt Wette @ 2023-02-25 10:42 ` Maxime Devos 0 siblings, 0 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-25 10:42 UTC (permalink / raw) To: Matt Wette, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 2402 bytes --] On 25-02-2023 01:15, Matt Wette wrote: > On 2/24/23 3:51 PM, Maxime Devos wrote: >> On 25-02-2023 00:48, Maxime Devos wrote: >>>>> (**) For compatibility with Racket, it's not like we couldn't >>>>> implement both "#lang" and "-*- stuff: language -*-". >> >> TBC, I mean ‘only support #lang' for values of 'lang' that Racket >> supports’, or alternatively ‘support #lang for compatibility, but >> recommend -*- ... -*- in the Guile manual.’. > > The point of this is to tell the compiler what language it needs to parse, > if not the default (Scheme). Using `#lang pascal' or `#!lang pascal' > allows > the compiler to figure this out in the first few characters read. Using > `-*- lang: pascal -*-' might require the compiler to read a lot of text > (not knowing > if what text is comment or not) to determine if language has been > specified. > For the `-*-' based design, is there a maximum number of characters to be > read before deciding? Guile already searches for '-*- ... -*-' lines, as mentioned in a previous message: > (*) The port encoding detection supports "-*- coding: whatever -*-", > presumably that functionality could be reused. As such, it shouldn't incur an overly large cost to support '-*- something: modula -*-' too. On the character limit: according to Emacs documentation, there is a line limit: the '-*- ... -*-' must be on the first line. (See: ‘(emacs)Specifying File Variables’.) This seems the same situation like '#lang' to me; not ‘a lot of text’. A more explicit character limit of 'first few hundred bytes' is mentioned in see the documentation of 'file-encoding': -- Scheme Procedure: file-encoding port -- C Function: scm_file_encoding (port) Attempt to scan the **first few hundred bytes** from the PORT for hints about its character encoding. Return a string containing the encoding name or ‘#f’ if the encoding cannot be determined. The port is rewound. Currently, the only supported method is to look for an Emacs-like character coding declaration (*note how Emacs recognizes file encoding: (emacs)Recognize Coding.). The coding declaration is of the form ‘coding: XXXXX’ and must appear in a Scheme comment. Additional heuristics may be added in the future. Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-16 21:38 ` Dr. Arne Babenhauserheide 2023-02-17 1:26 ` Matt Wette @ 2023-02-17 23:06 ` Maxime Devos 2023-02-18 3:50 ` Philip McGrath 1 sibling, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-17 23:06 UTC (permalink / raw) To: Dr. Arne Babenhauserheide, Matt Wette; +Cc: guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 2013 bytes --] On 16-02-2023 22:38, Dr. Arne Babenhauserheide wrote: > > Matt Wette <matt.wette@gmail.com> writes: > >> You may be interested in the load-lang patch I generated a few years ago >> to allow file-extension based loading, in addition to '#lang elisp" >> type hooks. >> >> https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch > > @Maxime: Is this something you’d be interested in championing? For the '#lang whatever stuff', no: The '#lang whatever' stuff makes Scheme (*) files unportable between implementations, as '#lang scheme' is not a valid comment -- there exist Schemes beyond Guile and Racket. If it were changed to recognising '-*- mode: scheme -*-' or '-*- language: scheme -*-' or such, it would be better IMO, but insufficient, because (^). (*) Same argument applies for some, but not all, other non-Scheme languages too. I'm assuming you don't meant the '%file-extension-map' stuff, because of your previous ‘[...] and also to avoid stumbling over files that just take that extension’ response to the proposal for such a thing. Even if you meant that, (^) also applies. (^) it doesn't integrate with the module system -- more concretely, (use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments would solve that, but we agree that that would be unreasonable in many situations. (Alternatively one could place ECMAScript code in a file with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but ... no.) Aside from the '#lang ...' -> '-*- language: scheme -*-' stuff, I think the idea behind the patch is good (**) -- it solves the problem it aims to solve, AFAICT. However, this problem is not the module system problem that Wisp currently has. As such, I suppose you could say that I would 'champion' the patch on its own, but not champion it in relation to '[PATCH] add language/wisp to Guile'. (**) It needs some documentation though. Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-17 23:06 ` Maxime Devos @ 2023-02-18 3:50 ` Philip McGrath 2023-02-18 15:58 ` Maxime Devos 2023-02-23 7:59 ` [PATCH] add language/wisp to Guile? Maxime Devos 0 siblings, 2 replies; 83+ messages in thread From: Philip McGrath @ 2023-02-18 3:50 UTC (permalink / raw) To: guile-devel Hi, On Fri, Feb 17, 2023, at 6:06 PM, Maxime Devos wrote: > On 16-02-2023 22:38, Dr. Arne Babenhauserheide wrote: >> >> Matt Wette <matt.wette@gmail.com> writes: >> >>> You may be interested in the load-lang patch I generated a few years ago >>> to allow file-extension based loading, in addition to '#lang elisp" >>> type hooks. >>> >>> https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch >> >> @Maxime: Is this something you’d be interested in championing? > > For the '#lang whatever stuff', no: > > The '#lang whatever' stuff makes Scheme (*) files unportable between > implementations, as '#lang scheme' is not a valid comment -- there exist > Schemes beyond Guile and Racket. If it were changed to recognising > '-*- mode: scheme -*-' or '-*- language: scheme -*-' or such, it would > be better IMO, but insufficient, because (^). > I haven't read the patch or this thread closely, but R6RS has an answer to any concerns about compatibility with `#lang`. At the beginning of Chapter 4, "Lexical and Datum Syntax" (<http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-7.html#node_chap_4>) the report specifies: > An implementation must not extend the lexical or datum syntax in any way, with one exception: it need not treat the syntax `#!<identifier>`, for any <identifier> (see section 4.2.4) that is not `r6rs`, as a syntax violation, and it may use specific `#!`-prefixed identifiers as flags indicating that subsequent input contains extensions to the standard lexical or datum syntax. The syntax `#!r6rs` may be used to signify that the input afterward is written with the lexical syntax and datum syntax described by this report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3. Chez Scheme uses such comments to support extensions to lexical syntax, as documented in <https://cisco.github.io/ChezScheme/csug9.5/intro.html#./intro:h1>: > The Chez Scheme lexical extensions described above are disabled in an input stream after an `#!r6rs` comment directive has been seen, unless a `#!chezscheme` comment directive has been seen since. Each library loaded implicitly via import and each RNRS top-level program loaded via the `--program` command-line option, the `scheme-script` command, or the `load-program` procedure is treated as if it begins implicitly with an `#!r6rs` comment directive. > The case of symbol and character names is normally significant, as required by the Revised6 Report. Names are folded, as if by string-foldcase, following a `#!fold-case` comment directive in the same input stream unless a `#!no-fold-case` has been seen since. Names are also folded if neither directive has been seen and the parameter `case-sensitive` has been set to `#f`. In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or `_`." (See <https://docs.racket-lang.org/reference/reader.html#%28part._parse-reader%29>.) This does not conflict with Racket's support for script shebangs: "A `#!` (which is `#!` followed by a space) or `#!/` starts a line comment that can be continued to the next line by ending a line with `\`. This form of comment normally appears at the beginning of a Unix script file." (See <https://docs.racket-lang.org/reference/reader.html#%28part._parse-comment%29>.) Furthermore, the lexical syntax for the rest of the file is entirely under control of the specified language. Most languages parameterize the reader to reject further uses of `#lang` or its `#!` alias. Some "meta-languages" chain-load another language but parameterize it in some way (e.g. <https://docs.racket-lang.org/exact-decimal-lang/>). The `#!r6rs` language, of course, handles `#!` exactly as specified by R6RS, with no extensions. (Guile does not handle `#!r6rs` properly, presumably because of the legacy `#!`/`!#` block comments. I think this should be a surmountable obstacle, though, especially since Guile does support standard `#|`/`|#` block comments.) > > (^) it doesn't integrate with the module system -- more concretely, > (use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments > would solve that, but we agree that that would be unreasonable in many > situations. (Alternatively one could place ECMAScript code in a file > with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but > ... no.) > Racket has a mechanism to enable additional source file extensions without needing explicit command-line arguments by defining `module-suffixes` or `doc-modules-suffixes` in a metadata module that is consulted when the collection is "set up": https://docs.racket-lang.org/raco/setup-info.html However, this mechanism is not widely used. Overall, the experience of the Racket community strongly suggests that a file should say what language it is written in. Furthermore, that language is a property of the code, not of its runtime environment, so environment variables, command-line options, and similar extralinguistic mechanism are a particularly poor fit for controlling it. File extensions are not the worst possible mechanisms, but they have similar problems: code written in an unsaved editor or a blog post may not have a file extension. (For more on this theme, see the corresponding point of the Racket Manifesto: <https://cs.brown.edu/~sk/Publications/Papers/Published/fffkbmt-racket-manifesto/paper.pdf>) Actually writing the language into the source code has proven to work well. To end with an argument from authority, this is from Andy Wingo's "lessons learned from guile, the ancient & spry" (<https://wingolog.org/archives/2020/02/07/lessons-learned-from-guile-the-ancient-spry>): > On the change side, we need parallel installability for entire languages. Racket did a great job facilitating this with #lang and we should just adopt that. -Philip ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-18 3:50 ` Philip McGrath @ 2023-02-18 15:58 ` Maxime Devos 2023-02-18 19:56 ` Matt Wette 2023-02-26 7:45 ` Philip McGrath 2023-02-23 7:59 ` [PATCH] add language/wisp to Guile? Maxime Devos 1 sibling, 2 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-18 15:58 UTC (permalink / raw) To: Philip McGrath, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 9006 bytes --] On 18-02-2023 04:50, Philip McGrath wrote: > I haven't read the patch or this thread closely, I'll assume you have read it non-closely. > but R6RS has an answer to any concerns about compatibility with `#lang`. At the beginning of Chapter 4, "Lexical and Datum Syntax" (<http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-7.html#node_chap_4>) the report specifies: > >> An implementation must not extend the lexical or datum syntax in any way, with one exception: it need not treat the syntax `#!<identifier>`, for any <identifier> (see section 4.2.4) that is not `r6rs`, as a syntax violation, and it may use specific `#!`-prefixed identifiers as flags indicating that subsequent input contains extensions to the standard lexical or datum syntax. The syntax `#!r6rs` may be used to signify that the input afterward is written with the lexical syntax and datum syntax described by this report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3. That is for '#!lang', not '#lang'. R6RS allows the former, but the patch does the latter. As such, R6RS does not have an answer about incompatibility with `#lang', unless you count ‘it's incompatible’ as an answer. > Chez Scheme uses such comments to support extensions to lexical syntax, as documented in <https://cisco.github.io/ChezScheme/csug9.5/intro.html#./intro:h1>: > >> The Chez Scheme lexical extensions described above are disabled in an input stream after an `#!r6rs` comment directive has been seen, unless a `#!chezscheme` comment directive has been seen since. Each library loaded implicitly via import and each RNRS top-level program loaded via the `--program` command-line option, the `scheme-script` command, or the `load-program` procedure is treated as if it begins implicitly with an `#!r6rs` comment directive. [...] Again, that's '#!whatever', not '#lang' -- Chez does the former, not the latter. > In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or `_`." (See <https://docs.racket-lang.org/reference/reader.html#%28part._parse-reader%29>.) [...] > (Guile does not handle `#!r6rs` properly, presumably because of the legacy `#!`/`!#` block comments. I think this should be a surmountable obstacle, though, especially since Guile does support standard `#|`/`|#` block comments.) ‘#! ... !#’ comments aren't legacy; they exist to allow putting the shebang in the first line of a script, and to pass additional arguments to the Guile interpreter (see: (guile)The Top of a Script File) (*). As such, you can't just replace them with #| ... |# (unless you patch the kernel to recognise "#| ..." as a shebang line). (*) Maybe they exist for other purposes too. Furthermore, according to the kernel, #!r6rs would mean that the script needs to be interpreted by a program named 'r6rs', but 'guile' is named 'guile', not 'r6rs'. (I assume this is in POSIX somewhere, though I couldn't find it.) (This is an incompatibility between R6RS and any system that has shebangs.) >> >> (^) it doesn't integrate with the module system -- more concretely, >> (use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments >> would solve that, but we agree that that would be unreasonable in many >> situations. (Alternatively one could place ECMAScript code in a file >> with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but >> ... no.) >> > > Racket has a mechanism to enable additional source file extensions without needing explicit command-line arguments by defining `module-suffixes` or `doc-modules-suffixes` in a metadata module that is consulted when the collection is "set up": https://docs.racket-lang.org/raco/setup-info.html However, this mechanism is not widely used. I guess this is an improvement over the runtime 'guile -x extension'. However, if I'm understanding 'setup-info.html' correctly, the downside is that you now need a separate file containing compilation settings. I have previously proposed a mechanism that makes the '-x' + '--language' a compile-time thing (i.e., embed the source file extension in the compiled .go; see previous e-mails in this thread), without having to make a separate file containing compilation settings. How is Racket's method an improvement over my proposal? > Overall, the experience of the Racket community strongly suggests that a file should say what language it is written in. Furthermore, that language is a property of the code, not of its runtime environment, so environment variables, command-line options, and similar extralinguistic mechanism are a particularly poor fit for controlling it. Agreed on the 'no environment variables' thing, disagreed on the 'no command-line options'. In the past e-mails in this thread, there was agreement on the ‘embed the source file extension in the compiled .go or something like that; and add -x extension stuff _when compiling_ (not runtime!) the software that uses the extension’. Do you any particular issues with that proposal? AFAICT, it solves everything and is somewhat more straightforward that Racket. > File extensions are not the worst possible mechanisms, but they have similar problems: code written in an unsaved editor or a blog post may not have a file extension. With the proposal I wrote, it remains possible to override any 'file extension -> language' mapping. It's not in any way incompatible with "-*- lang: whatever -*-"-like comments. Additionally, Guile can only load files that exist (i.e, 'saved'); Guile is not an editor or blog reader, so these do not appear problems for Guile to me. If the editor needs to determine the language for syntax highlighting or such, then there exist constructs like ';; -*- mode: scheme -*-' that are valid Scheme, but that's not a Guile matter. > (For more on this theme, see the corresponding point of the Racket Manifesto: <https://cs.brown.edu/~sk/Publications/Papers/Published/fffkbmt-racket-manifesto/paper.pdf>) Actually writing the language into the source code has proven to work well. What is the corresponding point? I'm not finding any search results for 'file extension' or 'file name', and I'm not finding any relevant search results for 'editor'. Could you give me a page reference and a relevant quote? > To end with an argument from authority, this is from Andy Wingo's "lessons learned from guile, the ancient & spry" (<https://wingolog.org/archives/2020/02/07/lessons-learned-from-guile-the-ancient-spry>): > >> On the change side, we need parallel installability for entire languages. Racket did a great job facilitating this with #lang and we should just adopt that. It was never in dispute that 'we need parallel installability'. This ‘parallel installability’ is also addressed by my proposal. For the second part: ‘Racket did a great job facilitating this with #lang and we should just adopt that’: * I previously pointed out some problems with that proposal -- i.e., '#lang whatever' is bogus Scheme / Wisp / ..., and 'the module system won't find it, because of the unexpected file extensions'. * I gave an alternate proposal (i.e., the 'embed source file extension in the compiled .go, and add something like '-x extension' and '--language ...' at compilation time), which doesn't have the pointed out problems. * Andy Wingo hasn't addressed the pointed out problems with the Racket solution adapted to Guile. * Andy Wingo hasn't considered any alternative proposals in that blog post. Why are you taking Andy Wingo's proposal as an argument by authority, when I have already pointed out the problems with that proposal and proposed a different solution without those problems? ‘Arguments by authority’ stop being good when someone explains how the authority is wrong. As you appear to consider arguments by authority to be valid arguments even in the existence of evidence that the authority is _wrong_, here's an argument from authority that ‘arguments from authority’ can be refuted: > (From: https://en.wikipedia.org/wiki/Argument_from_authority) > Scientific knowledge is best established by evidence and experiment rather than argued through authority[18][19][20] as authority has no place in science.[19][21] Carl Sagan wrote of arguments from authority: "One of the great commandments of science is, 'Mistrust arguments from authority.' ... Too many such arguments have proved too painfully wrong. Authorities must prove their contentions like everybody else."[22] However, countering this it has been argued that science is fundamentally dependent on arguments from authority to progress because "they allow science to avoid forever revisiting the same ground".[23] Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-18 15:58 ` Maxime Devos @ 2023-02-18 19:56 ` Matt Wette 2023-02-21 12:09 ` Dr. Arne Babenhauserheide 2023-02-26 7:45 ` Philip McGrath 1 sibling, 1 reply; 83+ messages in thread From: Matt Wette @ 2023-02-18 19:56 UTC (permalink / raw) To: guile-devel On 2/18/23 7:58 AM, Maxime Devos wrote: > On 18-02-2023 04:50, Philip McGrath wrote: >> I haven't read the patch or this thread closely, > > I'll assume you have read it non-closely. > >> but R6RS has an answer to any concerns about compatibility with >> `#lang`. At the beginning of Chapter 4, "Lexical and Datum Syntax" >> (<http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-7.html#node_chap_4>) >> the report specifies: >> >>> An implementation must not extend the lexical or datum syntax in >>> any way, with one exception: it need not treat the syntax >>> `#!<identifier>`, for any <identifier> (see section 4.2.4) that is >>> not `r6rs`, as a syntax violation, and it may use specific >>> `#!`-prefixed identifiers as flags indicating that subsequent input >>> contains extensions to the standard lexical or datum syntax. The >>> syntax `#!r6rs` may be used to signify that the input afterward is >>> written with the lexical syntax and datum syntax described by this >>> report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3. > > That is for '#!lang', not '#lang'. R6RS allows the former, but the > patch does the latter. As such, R6RS does not have an answer about > incompatibility with `#lang', unless you count ‘it's incompatible’ as > an answer. I just looked on the web and it appears that #! as first line is a comment in JavaScript also. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar#hashbang_comments ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-18 19:56 ` Matt Wette @ 2023-02-21 12:09 ` Dr. Arne Babenhauserheide 0 siblings, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-21 12:09 UTC (permalink / raw) To: Matt Wette; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 191 bytes --] Hi Matt, Please tell me once you know for which patch exactly you need a WIP-branch. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-18 15:58 ` Maxime Devos 2023-02-18 19:56 ` Matt Wette @ 2023-02-26 7:45 ` Philip McGrath 2023-02-26 15:42 ` Maxime Devos 2023-10-02 14:59 ` Christine Lemmer-Webber 1 sibling, 2 replies; 83+ messages in thread From: Philip McGrath @ 2023-02-26 7:45 UTC (permalink / raw) To: Maxime Devos, Ludovic Courtès, Matt Wette, guile-devel Cc: Christine Lemmer-Webber Hi, On Sat, Feb 18, 2023, at 10:58 AM, Maxime Devos wrote: > On 18-02-2023 04:50, Philip McGrath wrote: >> I haven't read the patch or this thread closely, > > I'll assume you have read it non-closely. > >> but R6RS has an answer to any concerns about compatibility with `#lang`. At the beginning of Chapter 4, "Lexical and Datum Syntax" (<http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-7.html#node_chap_4>) the report specifies: >> >>> An implementation must not extend the lexical or datum syntax in any way, with one exception: it need not treat the syntax `#!<identifier>`, for any <identifier> (see section 4.2.4) that is not `r6rs`, as a syntax violation, and it may use specific `#!`-prefixed identifiers as flags indicating that subsequent input contains extensions to the standard lexical or datum syntax. The syntax `#!r6rs` may be used to signify that the input afterward is written with the lexical syntax and datum syntax described by this report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3. > > That is for '#!lang', not '#lang'. R6RS allows the former, but the > patch does the latter. As such, R6RS does not have an answer about > incompatibility with `#lang', unless you count ‘it's incompatible’ as an > answer. > Let me try to be more concrete. If you want a portable, RnRS-standardized lexical syntax for `#lang`, use `#!<identifier>`, and systems that understand `#lang` will treat it (in appropriate contexts) as an alias for `#lang `. Alternatively, you could embrace that Guile (like every other Scheme system I'm aware of) starts by default in a mode with implementation-specific extensions. Indeed, R6RS Appendix A specifically recognizes that "the default mode offered by a Scheme implementation may be non-conformant, and such a Scheme implementation may require special settings or declarations to enter the report-conformant mode" [1]. Then you could just write `#lang` and worry about the non-portable block comments some other day. This is what I would personally prefer. >> In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or `_`." (See <https://docs.racket-lang.org/reference/reader.html#%28part._parse-reader%29>.) [...] > (Guile does not handle `#!r6rs` properly, presumably because of the > legacy `#!`/`!#` block comments. I think this should be a surmountable > obstacle, though, especially since Guile does support standard `#|`/`|#` > block comments.) > > ‘#! ... !#’ comments aren't legacy; they exist to allow putting the > shebang in the first line of a script, and to pass additional arguments > to the Guile interpreter (see: (guile)The Top of a Script File) (*). As > such, you can't just replace them with #| ... |# (unless you patch the > kernel to recognise "#| ..." as a shebang line). > > (*) Maybe they exist for other purposes too. According to "(guile)Block Comments", the `#!...!#` syntax existed before Guile 2.0 added support for `#|...|#` comments from SRFI 30 and R6RS. > > Furthermore, according to the kernel, #!r6rs would mean that the script > needs to be interpreted by a program named 'r6rs', but 'guile' is named > 'guile', not 'r6rs'. (I assume this is in POSIX somewhere, though I > couldn't find it.) > > (This is an incompatibility between R6RS and any system that has shebangs.) > This is not an incompatibility, because the `#!r6rs` lexeme (or `#!<identifier>`, more generally) is not the shebang line for the script. R6RS Appendix D [2] gives this example of a Scheme script: ``` #!/usr/bin/env scheme-script #!r6rs (import (rnrs base) (rnrs io ports) (rnrs programs)) (put-bytes (standard-output-port) (call-with-port (open-file-input-port (cadr (command-line))) get-bytes-all)) ``` The appendix says that, "if the first line of a script begins with `#!/` or `#!<space>`, implementations should ignore it on all platforms, even if it does not conform to the recommended syntax". Admittedly this is not handled as consistently as I would prefer: I wish they had just standardized `#!/` and `#! ` as special comment syntax, as Racket does, and clarified the interaction with `#!<identifier>`. But Matt points out that JavaScript also has very similar special treatment for a single initial shebang comment. Lua has a similar mechanism: my vague recollection is that many languages do. >>> >>> (^) it doesn't integrate with the module system -- more concretely, >>> (use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments >>> would solve that, but we agree that that would be unreasonable in many >>> situations. (Alternatively one could place ECMAScript code in a file >>> with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but >>> ... no.) Generally I would use `.scm` (or `.rkt`), and certainly I would do so if there isn't some well-established other extension. If you are just using the file, you shouldn't necessarily have to care what language it's implemented in internally. In particular, I don't think the `#lang` concept should be conflated with editor configuration like `'-*- mode: ecmascript -*-`. As an example, consider these two Racket programs: ``` #!datalog parent(anchises, aeneas). parent(aeneas, ascanius). ancestor(A, B) :- parent(A, B). ancestor(A, B) :- parent(A, C), ancestor(C, B). ancestor(A, ascanius)? ``` ``` #lang algol60 begin comment Credit to Rosetta Code; integer procedure fibonacci(n); value n; integer n; begin integer i, fn, fn1, fn2; fn2 := 1; fn1 := 0; fn := 0; for i := 1 step 1 until n do begin fn := fn1 + fn2; fn2 := fn1; fn1 := fn end; fibonacci := fn end; integer i; for i := 0 step 1 until 20 do printnln(fibonacci(i)) end ``` While I'm sure there are Emacs modes available for Datalog and Algol 60, and some people might want to use them for these programs, I would probably want to edit them both in racket-mode: because racket-mode supports the `#lang` protocol, it can obtain the syntax highlighting, indentation, and other support defined by each language, while also retaining the global features that all `#lang`-based languages get "for free", like a tool to rename variables that respects the actual model of scope. This is one of the value propositions of the `#lang` system. >> >> Racket has a mechanism to enable additional source file extensions without needing explicit command-line arguments by defining `module-suffixes` or `doc-modules-suffixes` in a metadata module that is consulted when the collection is "set up": https://docs.racket-lang.org/raco/setup-info.html However, this mechanism is not widely used. > > I guess this is an improvement over the runtime 'guile -x extension'. > However, if I'm understanding 'setup-info.html' correctly, the downside > is that you now need a separate file containing compilation settings. > > I have previously proposed a mechanism that makes the '-x' + > '--language' a compile-time thing (i.e., embed the source file extension > in the compiled .go; see previous e-mails in this thread), without > having to make a separate file containing compilation settings. > > How is Racket's method an improvement over my proposal? > My focus in this thread is explaining and advocating for `#lang`. I see the whole business with file extensions as basically orthogonal to `#lang`, and my opinions about it are much less strong, but I'll try to answer your question. I think it would make sense for `.go` files to record the file extension of their corresponding source files: Racket's `.zo` files do likewise. I don't object to a command-line option *at compile-time* (as you said) to enable additional file extensions, and I agree that there isn't a huge difference between that and an approach with a separate configuration file, though I do find the configuration-file approach somewhat more declarative, which I prefer. What I was really trying to argue here is that the file extension should not determine the meaning of the program it contains: more on that below. >> Overall, the experience of the Racket community strongly suggests that a file should say what language it is written in. Furthermore, that language is a property of the code, not of its runtime environment, so environment variables, command-line options, and similar extralinguistic mechanism are a particularly poor fit for controlling it. > > Agreed on the 'no environment variables' thing, disagreed on the 'no > command-line options'. In the past e-mails in this thread, there was > agreement on the ‘embed the source file extension in the compiled .go or > something like that; and add -x extension stuff _when compiling_ (not > runtime!) the software that uses the extension’. > > Do you any particular issues with that proposal? AFAICT, it solves > everything and is somewhat more straightforward that Racket. > I don't have particular issues with a compile-time command-line option to determine which files to compile. I do object to using command-line options or file extensions to determine what language a file is written in. >> File extensions are not the worst possible mechanisms, but they have similar problems: code written in an unsaved editor or a blog post may not have a file extension. > > With the proposal I wrote, it remains possible to override any 'file > extension -> language' mapping. It's not in any way incompatible with > "-*- lang: whatever -*-"-like comments. > > Additionally, Guile can only load files that exist (i.e, 'saved'); Guile > is not an editor or blog reader, so these do not appear problems for > Guile to me. > While it's true that the only files Guile can load are "files that exist", it's not true that "Guile can only load files": consider procedures like `eval-string`, `compile`, and, ultimately, `read-syntax`. AFAICT, to the extent that Guile's current implementations of such procedures support multiple languages, they rely on out-of-band configuration, like an optional `#:language` argument, which is just as extra-linguistic as relying on command-line options, environment variables, or file extensions. What I'm trying to advocate is that programs should say in-band, as part of their source code, what language they are written in. > If the editor needs to determine the language for syntax highlighting or > such, then there exist constructs like ';; -*- mode: scheme -*-' that > are valid Scheme, but that's not a Guile matter. > See above for why the `#!language/wisp` option is perfectly valid R6RS Scheme and for some of my concerns about overloading editor configuration to determine the semantics of programs. More broadly, everyone who reads a piece of source code, including humans as well as editors and the `guile` executable, needs to know what language it's written in to hope to understand it. >> (For more on this theme, see the corresponding point of the Racket Manifesto: <https://cs.brown.edu/~sk/Publications/Papers/Published/fffkbmt-racket-manifesto/paper.pdf>) Actually writing the language into the source code has proven to work well. > > What is the corresponding point? I'm not finding any search results for > 'file extension' or 'file name', and I'm not finding any relevant search > results for 'editor'. Could you give me a page reference and a relevant > quote? > I was trying to refer to section 5, "Racket Internalizes Extra-Linguistic Mechanisms", which begins on p. 121 (p. 9 of the PDF). Admittedly, the connection between the main set of examples they discuss and this conversation is non-obvious. Maybe the most relevant quote is the last paragraph of that section, on p. 123 (PDF p. 11): "Finally, Racket also internalizes other aspects of its context. Dating back to the beginning, Racket programs can programmatically link modules and classes. In conventional languages, programmers must resort to extra-linguistic tools to abstract over such linguistic constructs; only ML-style languages and some scripting languages make modules and classes programmable, too." (Internal citations omitted.) >> To end with an argument from authority, this is from Andy Wingo's "lessons learned from guile, the ancient & spry" (<https://wingolog.org/archives/2020/02/07/lessons-learned-from-guile-the-ancient-spry>): >> Sorry, this was meant to be tongue-in-cheek, and it seems that didn't come across. "Argument from authority" is often considered a category of logical fallacy, and ending with a quote is sometimes considered to be bad style or to weaken a piece of persuasive writing. > * I previously pointed out some problems with that proposal > -- i.e., '#lang whatever' is bogus Scheme / Wisp / ..., I hope I've explained why something like `#!language/wisp` is perfectly within the bounds of R6RS. Also, given that Guile already starts with non-standard extensions enabled by default, I don't see any reason not to also support `#lang language/wisp`. In particular, the spelling of `#lang` proceeds directly from the Scheme tradition. This is from the R6RS Rationale document, chapter 4, "Lexical Syntax", section 3, "Future Extensions" [3]: >>>> The `#` is the prefix of several different kinds of syntactic entities: vectors, bytevectors, syntactic abbreviations related to syntax construction, nested comments, characters, `#!r6rs`, and implementation-specific extensions to the syntax that start with `#!`. In each case, the character following the `#` specifies what kind of syntactic datum follows. In the case of bytevectors, the syntax anticipates several different kinds of homogeneous vectors, even though R6RS specifies only one. The `u8` after the `#v` identifies the components of the vector as unsigned 8-bit entities or octets. > and > 'the module system won't find it, because of the unexpected > file extensions'. > This is indeed something that needs to be addressed, but it seems like a very solvable problem. Using the extension ".scm" for everything would be one trivial solution. Something like your proposal to enable file extensions based on a compile-time option could likewise be part of a solution. In general, I'll say that, while using Guile, I've often missed Racket's more flexible constructs for importing modules. I especially miss `(require "foo/bar.rkt")`, which imports a module at a path relative to the module where the `require` form appears: it makes it easy to organize small programs into multiple files without having to mess with a load path. More messages have come since I started writing this reply, so I'll try to address them, too. On Thu, Feb 23, 2023, at 1:04 PM, Maxime Devos wrote: > On 23-02-2023 09:51, Dr. Arne Babenhauserheide wrote: >>> Thinking a bit more about it, it should be possible to special-case >>> Guile's interpretation of "#!" such that "#!r6rs" doesn't require a >>> closing "!#". (Technically backwards-incompatible, but I don't think >>> people are writing #!r6rs ...!# in the wild.) >> Do you need the closing !# if you restrict yourself to the first line? > > I thought so at first, but doing a little experiment, it appears you > don't need to: > > $ guile > scheme@(guile-user)> #!r6rs > (display "hi") (newline) > > (output: hi) > > Apparently Guile already has required behaviour. > All the `#!r6rs` examples I've tried since I got Ludo’'s mail have worked, but I remember some not working as I'd expected in the past. I'll see if I can come up with any problematic examples again. On Thu, Feb 23, 2023, at 1:42 PM, Maxime Devos wrote: > Have you seen my messages on how the "#lang" construct is problematic > for some languages, and how alternatives like "[comment delimiter] -*- > stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be > equally simple (*) and not have any downsides (**). > > (*) The port encoding detection supports "-*- coding: whatever -*-", > presumably that functionality could be reused. > IMO, the use of "-*- coding: whatever -*-" to detect encoding is an ugly hack and should not be extended further. I tried to raise some objections above to conflating editor configuration with syntax saying what a file's language is. More broadly, I find "magic comments" highly objectionable. The whole point of comments is to be able to communicate freely to human readers without affecting the interpreter/compiler/evaluator. Introducing magic comments means must constantly think about whether what you are writing for humans might change the meaning of your program. Magic comments *without knowing a priori what is a comment* are even worse: now, you have to beware of accidental "magic" in ALL of the lexical syntax of your program. (Consider that something like `(define (-*- mode: c++ -*-) 14)` is perfectly good Scheme.) (It's not really relevant for the `#lang`-like case, but something I find especially ironic about encoding "magic comments" or, say, `<?xml version="1.0" encoding="UTF-8"?>`, is that suddenly if you encode the Unicode text in some other encoding it becomes a lie.) On Fri, Feb 24, 2023, at 6:51 PM, Maxime Devos wrote: > On 25-02-2023 00:48, Maxime Devos wrote: >>>> (**) For compatibility with Racket, it's not like we couldn't >>>> implement both "#lang" and "-*- stuff: language -*-". > > TBC, I mean ‘only support #lang' for values of 'lang' that Racket > supports’ If I understand what you're proposing here, I don't think it's a viable option. The fundamental purpose of the `#lang` construct (however you spell it) is to provide an open, extensible protocol for defining languages. Thus, "values of 'lang' that Racket supports" are unbounded, provided that a module has been installed where the language specification says to look. From The Racket Reference [4]: >>>> The `#lang` reader form is similar to `#reader`, but more constrained: the `#lang` must be followed by a single space (ASCII 32), and then a non-empty sequence of alphanumeric ASCII, `+`, `-`, `_`, and/or `/` characters terminated by whitespace or an end-of-file. The sequence must not start or end with `/`. A sequence `#lang ‹name›` is equivalent to either `#reader (submod ‹name› reader)` or `#reader ‹name›/lang/reader`, where the former is tried first guarded by a `module-declared?` check (but after filtering by `current-reader-guard`, so both are passed to the value of `current-reader-guard` if the latter is used). Note that the terminating whitespace (if any) is not consumed before the external reading procedure is called. >>>> >>>> Finally, `#!` is an alias for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or `_`. Use of this alias is discouraged except as needed to construct programs that conform to certain grammars, such as that of R6RS [Sperber07]. (The rationale for the constraints, which Racketeers generally tend to chafe against, is that the syntax of `#lang‹name›` is the one and only thing that `#lang` doesn't give us a way to compatibly change. We can quickly get to a less constrained syntax by using a chaining "meta-language": see `#lang s-exp` and `#lang reader` on that page for two of many examples.) I expect reading this would raise more questions, because that page gives lots of details on Racket's `#lang` protocol. Do I really expect Guile to implement all of those details? If not, in what sense is what I'm advocating actually compatible with `#lang`? I am definitely **not** suggesting that Guile implement all the details of Racket's `#lang` implementation. What I do strongly advocate is that you design Guile's support for `#lang` (or `#!`) to leave open a pathway for compatibility in the future. I think the best way to explain how that would work is to take as an extended example Zuo, the tiny Scheme-like language created last year to replace the build scripts for Racket and Racket's branch of Chez Scheme. Zuo was initially prototyped in Racket as a `#lang` language. Since the goal was to use Zuo to build Racket, the primary implementation is an interpreter implemented in a single file of C code, avoiding bootstrapping issues. There isn't a working Zuo implementation as a Racket at the moment. (There's a shim implementation, and there's some work in progress, as people have time and interest, to get a real implementation working again.) Zuo is based on `#lang`, but its protocol [5][6] is quite different than Racket's. Nevertheless, as I will explain, they are compatible. The C code in fact implements not `#lang zuo` or even `#lang zuo/base` but `#lang zuo/kernel`: the rest of `#lang zuo` is implemented in Zuo, building up to `#lang zuo` through a series of internal languages. A module written in `#lang zuo/kernel` is a single expression which produces an immutable symbol-keyed hash table, which is Zup's core representation of a module. When Zuo encounters `#lang whatever`, it looks up the symbol `'read-and-eval` in the hash table representing the module `whatever`: the result should be a procedure that, given a Zuo string (a Scheme bytevector) with the source of the module, returns a hash table to be used as the module's representation. An implementation of `#lang zuo/kernel` in Racket would bridge this protocol with Racket's `#lang` by synthesizing `reader` submodules implementing the procedures the Racket protocol expects by wrapping the procedure mapped to `'read-and-eval` in the Zuo-level hash table. The wrappers would propagate themselves, so a language implemented in a language implemented in `#lang zuo/kernel` would likewise be automatically bridged, and so on ad infinitum. Racket's submodules [7] make this work especially elegantly. In Guile, my experience with the tower of languages is limited, but AIUI many of the existing facilities are like `lookup-language`[8] in expecting language X to be implemented by a language object bound to X in the module `(language X spec)`. I'd suggest that Guile support `#lang language/X` (or `#!language/X`, if you prefer to spell it that way) by likewise looking up X in the `(language X spec)` module. One day, compatibility could be achieved by adding trivial bridge (sub)modules: for an illustration of how trivial this can be, see [8], a one-line module that makes SRFI 11 available as `(import (srfi :11))` for R6RS by wrapping its historical PLT Scheme location, `(require srfi/11)`. I would NOT suggest supporting arbitrary things after `#lang`, because one part of planning for compatibility is avoiding future namespace collisions. Happily, `language/` is not otherwise in use in the Racket world, so I suggest that Guile claim it. I don't think this should be overly restrictive: if it seems worth-while to support languages from other modules, you could implement the "chaining meta-language" approach I mentioned above: imagine something like `#!language/other (@ (some other module) exported-language)`, where the `other` export of `(language other spec)` is responsible for reading the next datum and using it to obtain the language object to be used for the rest of the module. (Other kinds of potential namespace collisions are easier to manage: for example, we could imagine that `(use-modules (foo bar baz))` might not access the same module as `(require foo/bar/baz)`. This is in a way an example of where it makes sense to be constrained in the syntax of `#lang` itself and let `#lang` unlock endless possibilities.) I've sort of alluded above to my pipe dream of a grand unified future for Racket-and-Guile-on-Chez, Guile-and-Racket-on-the-Guile-VM, and endless other possibilities. I wrote about it in more detail on the guix-devel list at [10]. (These thoughts were inspired by conversations with Christine Lemmer-Webber, though she bears no responsibility for my zany imaginings.) Finally, I looked into the history of `#!` in R6RS a bit, and I'll leave a few pointers here for posterity. Will Clinger's 2015 Scheme Workshop paper [11] says in section 3.1 that "Kent Dybvig suggested the `#!r6rs` flag in May 2006", Clinger "formally proposed addition of Dybvig’s suggestion" [12], and, "less than six weeks later," `#!r6rs` was "in the R6RS editors’ status report". (I am not persuaded by all of the arguments about `#!r6rs` in that paper: in particular, the analysis doesn't seem to account for R6RS Appendix A [1].) As best as I can tell, the suggestion from Kent Dybvig is [13]: On Wed May 10 15:40:13 EDT 2006, Kent Dybvig wrote: > We already have (as of last week's meeting) a syntax for dealing with > implementation-dependent lexical exceptions, which is to allow for > #!<symbol-like-thing>, e.g.: > > #!mzsceheme > #!larceny > ... > > Perhaps we can plan on using the same tool for future extensions to the > syntax: > > #!r7rs > > We can even require #!r6rs to appear at the top of a library now, or at > least allow it to be included. > > This is a lot more concise than a MIME content-type line. > > Kent I haven't tracked down any older writing about `#!<symbol-like-thing>` for "implementation-dependent lexical exceptions": it may have been a conference call. -Philip [1]: http://www.r6rs.org/final/html/r6rs-app/r6rs-app-Z-H-3.html#node_chap_A [2]: http://www.r6rs.org/final/html/r6rs-app/r6rs-app-Z-H-6.html#node_chap_D [3]: http://www.r6rs.org/final/html/r6rs-rationale/r6rs-rationale-Z-H-6.html#node_chap_4 [4]: https://docs.racket-lang.org/reference/reader.html#%28part._parse-reader%29 [5]: https://docs.racket-lang.org/zuo/Zuo_Overview.html#%28part._.Zuo_.Implementation_and_.Macros%29 [6]: https://docs.racket-lang.org/zuo/Zuo_Overview.html#%28part._module-protocol%29 [7]: https://www-old.cs.utah.edu/plt/publications/gpce13-f-color.pdf [8]: https://www.gnu.org/software/guile/manual/html_node/Compiler-Tower.html#index-lookup_002dlanguage [9]: https://github.com/racket/srfi/blob/25eb1c0e1ab8a1fa227750aa7f0689a2c531f8c8/srfi-lib/srfi/%253a11.rkt [10]: https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00010.html [11]: https://andykeep.com/SchemeWorkshop2015/papers/sfpw1-2015-clinger.pdf [12]: http://www.r6rs.org/r6rs-editors/2006-May/001251.html [13]: http://www.r6rs.org/r6rs-editors/2006-May/001248.html ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-26 7:45 ` Philip McGrath @ 2023-02-26 15:42 ` Maxime Devos 2023-02-26 16:14 ` Dr. Arne Babenhauserheide 2023-02-26 17:58 ` Matt Wette 2023-10-02 14:59 ` Christine Lemmer-Webber 1 sibling, 2 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-26 15:42 UTC (permalink / raw) To: Philip McGrath, Ludovic Courtès, Matt Wette, guile-devel Cc: Christine Lemmer-Webber [-- Attachment #1.1.1: Type: text/plain, Size: 37164 bytes --] Op 26-02-2023 om 08:45 schreef Philip McGrath: > Hi, > > On Sat, Feb 18, 2023, at 10:58 AM, Maxime Devos wrote: >> On 18-02-2023 04:50, Philip McGrath wrote: >>> I haven't read the patch or this thread closely, >> >> I'll assume you have read it non-closely. >> >>> but R6RS has an answer to any concerns about compatibility with `#lang`. At the beginning of Chapter 4, "Lexical and Datum Syntax" (<http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-7.html#node_chap_4>) the report specifies: >>> >>>> An implementation must not extend the lexical or datum syntax in any way, with one exception: it need not treat the syntax `#!<identifier>`, for any <identifier> (see section 4.2.4) that is not `r6rs`, as a syntax violation, and it may use specific `#!`-prefixed identifiers as flags indicating that subsequent input contains extensions to the standard lexical or datum syntax. The syntax `#!r6rs` may be used to signify that the input afterward is written with the lexical syntax and datum syntax described by this report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3. >> >> That is for '#!lang', not '#lang'. R6RS allows the former, but the >> patch does the latter. As such, R6RS does not have an answer about >> incompatibility with `#lang', unless you count ‘it's incompatible’ as an >> answer. >> > > Let me try to be more concrete. > > If you want a portable, RnRS-standardized lexical syntax for `#lang`, use `#!<identifier>`, and systems that understand `#lang` will treat it (in appropriate contexts) as an alias for `#lang `. RnRS only standardises #!r6rs, not #!<identifier>. Even if RnRS standardised #!<identifier> for values of <identifier> that aren't rnrs, the RnRS only holds sway for Scheme, and one of the main points of Guile's language system is to support more than only Scheme. > > Alternatively, you could embrace that Guile (like every other Scheme system I'm aware of) starts by default in a mode with implementation-specific extensions. Indeed, R6RS Appendix A specifically recognizes that "the default mode offered by a Scheme implementation may be non-conformant, and such a Scheme implementation may require special settings or declarations to enter the report-conformant mode" [1]. Then you could just write `#lang` and worry about the non-portable block comments some other day. This is what I would personally prefer. Emphasis on 'non-conformant'. The appendix states that Scheme implementations don't need to be R6RS by default; it doesn't state that things non-conformant things are conformant with R6RS. Remember that this part of the discussion started with: ‘The '#lang whatever' stuff makes Scheme (*) files unportable between implementations, as '#lang scheme' is not a valid comment’. The R6RS might permit non-R6RS implementations, but this does not make non-R6RS constructs like '#lang scheme' portable. >>> In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or `_`." (See <https://docs.racket-lang.org/reference/reader.html#%28part._parse-reader%29>.) [...] > (Guile does not handle `#!r6rs` properly, presumably because of the >> legacy `#!`/`!#` block comments. I think this should be a surmountable >> obstacle, though, especially since Guile does support standard `#|`/`|#` >> block comments.) >> >> ‘#! ... !#’ comments aren't legacy; they exist to allow putting the >> shebang in the first line of a script, and to pass additional arguments >> to the Guile interpreter (see: (guile)The Top of a Script File) (*). As >> such, you can't just replace them with #| ... |# (unless you patch the >> kernel to recognise "#| ..." as a shebang line). >> >> (*) Maybe they exist for other purposes too. > > According to "(guile)Block Comments", the `#!...!#` syntax existed before Guile 2.0 added support for `#|...|#` comments from SRFI 30 and R6RS. I agree, and I don't follow what your point is here. >> Furthermore, according to the kernel, #!r6rs would mean that the script >> needs to be interpreted by a program named 'r6rs', but 'guile' is named >> 'guile', not 'r6rs'. (I assume this is in POSIX somewhere, though I >> couldn't find it.) >> >> (This is an incompatibility between R6RS and any system that has shebangs.) >> > > This is not an incompatibility, because the `#!r6rs` lexeme (or `#!<identifier>`, more generally) is not the shebang line for the script. R6RS Appendix D [2] gives this example of a Scheme script: > > ``` > #!/usr/bin/env scheme-script > #!r6rs > (import (rnrs base) > (rnrs io ports) > (rnrs programs)) > (put-bytes (standard-output-port) > (call-with-port > (open-file-input-port > (cadr (command-line))) > get-bytes-all)) > ``` OK, didn't notice that appendix. Only covers Scheme, though. > -- > The appendix says that, "if the first line of a script begins with `#!/` or `#!<space>`, implementations should ignore it on all platforms, even if it does not conform to the recommended syntax". Admittedly this is not handled as consistently as I would prefer: I wish they had just standardized `#!/` and `#! ` as special comment syntax, as Racket does, and clarified the interaction with `#!<identifier>`. But Matt points out that JavaScript also has very similar special treatment for a single initial shebang comment. Lua has a similar mechanism: my vague recollection is that many languages do. I do not follow what your point is here -- I only (falsely) claimed that POSIX and R6RS are incompatible w.r.t. shebangs and "#!"; I did not make such claims for other languages -- some other languages don't even have "#!" (e.g. BASIC). >>>> >>>> (^) it doesn't integrate with the module system -- more concretely, >>>> (use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments >>>> would solve that, but we agree that that would be unreasonable in many >>>> situations. (Alternatively one could place ECMAScript code in a file >>>> with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but >>>> ... no.) > > Generally I would use `.scm` (or `.rkt`), and certainly I would do so if there isn't some well-established other extension. If you are just using the file, you shouldn't necessarily have to care what language it's implemented in internally. Maybe you would, but Guile shouldn't require people to change the extension of source files to something invalid, as I pointed out with the ECMAScript example. .scm means Scheme, not ECMAScript. As such, support for non-.scm file extensions is required. > In particular, I don't think the `#lang` concept should be conflated with editor configuration like `'-*- mode: ecmascript -*-`. > [...] Then don't do that, and use non-editor configuration like '-*- programming-language: ecmascript -*-' instead. While Emacs is the main user of '-*- ... -*-' lines, there is nothing stopping use from adding a few variables like e.g. 'programming-language' (*) that Emacs doesn't assign a meaning to. (*) I don't actually know if Emacs assigns a meaning to this variable or not. Some other word might perhaps be needed. For convenience, I would recommend supporting '-*- mode: ... -*-' too, such that non-Scheme source files can sometimes be loaded without making any Guile-specific changes to the source files. If whoever writes or reads the source file wants to use another Emacs mode, or if it the mode is ambiguous because it covers multiple languages, there is nothing stopping them from setting both 'mode: ...' and 'programming-language: ...': % -*- language: datalog; mode: racket -*- [...] As an example, consider these two Racket programs: > > ``` > #!datalog > parent(anchises, aeneas). > parent(aeneas, ascanius). > ancestor(A, B) :- parent(A, B). > ancestor(A, B) :- parent(A, C), ancestor(C, B). > ancestor(A, ascanius)? > ``` > > ``` > #lang algol60 > begin > comment Credit to Rosetta Code; > integer procedure fibonacci(n); value n; integer n; > begin > integer i, fn, fn1, fn2; > fn2 := 1; > fn1 := 0; > fn := 0; > for i := 1 step 1 until n do begin > fn := fn1 + fn2; > fn2 := fn1; > fn1 := fn > end; > fibonacci := fn > end; > > integer i; > for i := 0 step 1 until 20 do printnln(fibonacci(i)) > end > ``` > > While I'm sure there are Emacs modes available for Datalog and Algol 60, and some people might want to use them for these programs, I would probably want to edit them both in racket-mode: because racket-mode supports the `#lang` protocol, it can obtain the syntax highlighting, indentation, and other support defined by each language, while also retaining the global features that all `#lang`-based languages get "for free", like a tool to rename variables that respects the actual model of scope. This is one of the value propositions of the `#lang` system. As pointed out by my previous example, this is solved by '-*- ... -*-' too. > >>> >>> Racket has a mechanism to enable additional source file extensions without needing explicit command-line arguments by defining `module-suffixes` or `doc-modules-suffixes` in a metadata module that is consulted when the collection is "set up": https://docs.racket-lang.org/raco/setup-info.html However, this mechanism is not widely used. >> >> I guess this is an improvement over the runtime 'guile -x extension'. >> However, if I'm understanding 'setup-info.html' correctly, the downside >> is that you now need a separate file containing compilation settings. >> >> I have previously proposed a mechanism that makes the '-x' + >> '--language' a compile-time thing (i.e., embed the source file extension >> in the compiled .go; see previous e-mails in this thread), without >> having to make a separate file containing compilation settings. >> >> How is Racket's method an improvement over my proposal? >> > > My focus in this thread is explaining and advocating for `#lang`. I see the whole business with file extensions as basically orthogonal to `#lang`, and my opinions about it are much less strong, but I'll try to answer your question. I think it would make sense for `.go` files to record the file extension of their corresponding source files: Racket's `.zo` files do likewise. I don't object to a command-line option *at compile-time* (as you said) to enable additional file extensions, and I agree that there isn't a huge difference between that and an approach with a separate configuration file, though I do find the configuration-file approach somewhat more declarative, which I prefer. '--language whatever' appears pretty declarative to me, as in it declares that the language is 'whatever'. > What I was really trying to argue here is that the file extension should not determine the meaning of the program it contains: more on that below. That's what the '--language whatever' compilation argument is for: it overrides the 'guess by file extension' fallback. >>> Overall, the experience of the Racket community strongly suggests that a file should say what language it is written in. Furthermore, that language is a property of the code, not of its runtime environment, so environment variables, command-line options, and similar extralinguistic mechanism are a particularly poor fit for controlling it. >> >> Agreed on the 'no environment variables' thing, disagreed on the 'no >> command-line options'. In the past e-mails in this thread, there was >> agreement on the ‘embed the source file extension in the compiled .go or >> something like that; and add -x extension stuff _when compiling_ (not >> runtime!) the software that uses the extension’. >> >> Do you any particular issues with that proposal? AFAICT, it solves >> everything and is somewhat more straightforward that Racket. >> > > I don't have particular issues with a compile-time command-line option to determine which files to compile. I do object to using command-line options or file extensions to determine what language a file is written in. > >>> File extensions are not the worst possible mechanisms, but they have similar problems: code written in an unsaved editor or a blog post may not have a file extension. >> >> With the proposal I wrote, it remains possible to override any 'file >> extension -> language' mapping. It's not in any way incompatible with >> "-*- lang: whatever -*-"-like comments. >> >> Additionally, Guile can only load files that exist (i.e, 'saved'); Guile >> is not an editor or blog reader, so these do not appear problems for >> Guile to me. >> > > While it's true that the only files Guile can load are "files that exist", it's not true that "Guile can only load files": consider procedures like `eval-string`, `compile`, and, ultimately, `read-syntax`. * read-syntax is for reading S-expressions -- it is only for Scheme, other languages are out-of-scope for that procedure. As such, read-syntax appears irrelevant here to me. * For 'compile' and 'eval-string', I'd like to that when the point out that they have a "#:from" and #:lang" argument to set the language, as you appear to know going by your responses below. As such, even if Guile had an integrated editor, that editor can pass the language to Guile's compiler. I mean, if the editor is good, it has syntax highlighting, and to do syntax highlighting it needs to know the language, so it knows the language anyway (e.g. maybe it has separate "Write new Scheme" and "Write new ECMAScript" buttons, or maybe it has a 'mode: scheme' and 'mode: ecmascript' like Emacs and being an editor, it then knows how to convert that editor configuration into #:from/#:lang). * What I meant with 'Guile can only load files that exist', is that the files it loads are only those that exist. I did not mean that no loadable non-file things exist. The point here, is that if you wrote a blog post that defines the (foo) module and you enter (use-modules (foo)) in a Guile REPL, it isn't going to surf to your blog to download the (foo) module. As Guile doesn't even know about your blog post, it has no use for any file extension or language declaration that your blog post about (foo) might or might not have. > AFAICT, to the extent that Guile's current implementations of such procedures support multiple languages, they rely on out-of-band configuration, like an optional `#:language` argument, which is just as extra-linguistic as relying on command-line options, environment variables, or file extensions. First, I never proposed relying on environment variables. I oppose using environment variables for these things. Why are you mentioning environment variables, when this has never been proposed? Second, the implicit argument here appears to be 'extra-linguistic is bad, so we shouldn't do these extra-linguistic' things. But what's the problem with being 'extra-linguistic'? Some stuff like environment variables are plain bad here (no disagreement here), file extensions are bad to rely on but acceptable and convenient as a fallback. Third, I am not proposing to rely on command line options and file extension -- I only propose _using_ them, not _relying_ on them -- if someone wants to implement an in-band (intra-linguistic?) override like '-*- ... -*-'/#lang for file-extension based detection, they can do that -- my '-*- ... -*-' is just a proposed improvement over "#lang'. Fourth, TBC, I'd like to point out that '-*- ... -*-' is equally 'intra/extra-linguistic' as '#!lang' (see my response to 'magic comments' later), though I do know that's not the point you appear to be making right here. > What I'm trying to advocate is that programs should say in-band, as part of their source code, what language they are written in. That's done by '-*- ... -*-' too, and I haven't noticed any argumentation for ‘programs should say in-band what language they are written in’. Also, there is a gap between the following five statements, which you appear to sometimes be conflating: (A) Programs should say in-band what language they are written in. (B) ‘Guile should use in-band information to determine what language a program is written in.’ (C) ‘Guile should use out-of-band information to determine what language a program is written in.’ (D) ‘Guile should exclusively use out-of-band information to determine what language a program is written in.’ (E) ‘Guile should exclusively use in-of-band information to determine what language a program is written in.’ I disagree with (A), because often it's perfectly clear from context (out-of-band) what language it is. Take for example Guile itself. Being Guile, of course everything under 'modules/' is Scheme code. Adding '#!r6rs' or '-*- language: scheme -*-' lines to every .scm isn't incorrect, but is rather silly. Likewise, I have written a Scheme library called 'Scheme-GNUnet'. From the name alone, it is clear that it's Scheme. More generally, usually it's pretty clear (for a human) which language it is by just looking at the code, and if not, probably the README mentions which language the software uses. I don't dispute (B), but neither do I find it particularly important given that adding a '--language=whatever' argument is trivial. I would like to point out that (A) does not imply (B) -- it is possible to consider it good practice to mention the language in-band, without any language implementations actually using this information. More to the point, to me (A) appears irrelevant to this thread. Sure, perhaps it's a good practice, but Guile is not a programmer; Guile is a language implementation. (A) is only relevant insofar Guile would make use of this in-band information. > What I'm trying to advocate is that programs should say in-band, as > part of their source code, what language they are written in. This would be advocating for (A). But as mentioned above, (A) is irrelevant by itself, and it doesn't imply (B). It is also false -- you weren't advocating for (A), but for (B) -- (A) is just a means to (B) in your argumentation structure. Quoting one of your first messages: > To end with an argument from authority, this is from Andy Wingo's "lessons learned from guile, the ancient & spry" (<https://wingolog.org/archives/2020/02/07/lessons-learned-from-guile-the-ancient-spry>): > >> On the change side, we need parallel installability for entire languages. Racket did a great job facilitating this with #lang and we should just adopt that. You are also advocating for 'E/not (C)': > I do object to using command-line options or file extensions to determine what language a file is written in. You also appear to be thinking that I'm advocating for '(D)' -- while I agree with (D) (using a non-universal language construct (*) like '#lang' to determine the language something is written in, is rather circular), I'm not arguing for it. (*) Again, #lang is rather Racket-specific, whereas comments are mostly universal. >> If the editor needs to determine the language for syntax highlighting or >> such, then there exist constructs like ';; -*- mode: scheme -*-' that >> are valid Scheme, but that's not a Guile matter. >> > > See above for why the `#!language/wisp` option is perfectly valid R6RS Scheme Wisp isn't R6RS. Wisp code needs to be valid Wisp, not valid R6RS Scheme. There also exist languages beyond Wisp and Scheme. > and for some of my concerns about overloading editor configuration to determine the semantics of programs. See above replies. > More broadly, everyone who reads a piece of source code, including humans as well as editors and the `guile` executable, needs to know what language it's written in to hope to understand it. For programmers, this is covered by: * looking at the code -- even without any explicit in-band information like ';; -*- ... -*-' comments or "#lang", or out-of-band information like file extensions, a README or Makefile with compilation, it usually is pretty clear what language it is in. * usually source code is in files, which usually has file extensions. Usually there's a good map file extension->language, e.g. .scm files only contain Scheme, .js only contains ECMAScript, ... For editors, this is covered by: * Editor configuration like '-*- mode: scheme -*-'. * Language-specific declarations like #lang, #!r6rs, '-*- programming-language: scheme -*-' * File extensions. * If the editor guessed wrong, likely the syntax highlighting is wrong etc., so the programmer gives a hint to the editor (e.g. by adding a -*- mode: scheme -*- line, or #!r6rs, ...) For the Guile executable, this is covered by: * --language=.../#:from/#:lang arguments. * -*- ... -*- / #!r6rs lines (but not #lang except when needed for compatibility with Racket, otherwise Guile would create incompatibilities.) * File extensions. * Default to Scheme. * If guessing wrong, there will almost surely be some parsing error, in which case the programmer will intervene by modifying a single line in the Makefile or such to add "--language=" line, or if they per se want to spend much more time than needed, add "-*- programming-language: whatever -*-" comments to every single source file. >>> (For more on this theme, see the corresponding point of the Racket Manifesto: <https://cs.brown.edu/~sk/Publications/Papers/Published/fffkbmt-racket-manifesto/paper.pdf>) Actually writing the language into the source code has proven to work well. >> >> What is the corresponding point? I'm not finding any search results for >> 'file extension' or 'file name', and I'm not finding any relevant search >> results for 'editor'. Could you give me a page reference and a relevant >> quote? >> > > I was trying to refer to section 5, "Racket Internalizes Extra-Linguistic Mechanisms", which begins on p. 121 (p. 9 of the PDF). Admittedly, the connection between the main set of examples they discuss and this conversation is non-obvious. Maybe the most relevant quote is the last paragraph of that section, on p. 123 (PDF p. 11): "Finally, Racket also internalizes other aspects of its context. Dating back to the beginning, Racket programs can programmatically link modules and classes. In conventional languages, programmers must resort to extra-linguistic tools to abstract over such linguistic constructs; only ML-style languages and some scripting languages make modules and classes programmable, too." (Internal citations omitted.) This e-mail thread is about determining the language, not classes and modules. Trying to decode this vague paragraph, the relevant bit here appears ‘must resort to _extra-linguistic_ tools to abstract over such _linguistic constructs_’. As such, I assume that 'extra-linguistic' refers to file extensions (and other things, but it's the file extensions that are relevant here). Using this guess to unvaguify the phrasing, I get: ‘Programmers must resort to use file extensions to indicate which language a programmer is written in.’ However, that this is a bad thing appears to be the point that you were making in the first place, for which you gave the PDF as a source, so this doesn't explain anything. >>> To end with an argument from authority, this is from Andy Wingo's "lessons learned from guile, the ancient & spry" (<https://wingolog.org/archives/2020/02/07/lessons-learned-from-guile-the-ancient-spry>): >>> > > Sorry, this was meant to be tongue-in-cheek, and it seems that didn't come across. "Argument from authority" is often considered a category of logical fallacy, and ending with a quote is sometimes considered to be bad style or to weaken a piece of persuasive writing. > >> * I previously pointed out some problems with that proposal >> -- i.e., '#lang whatever' is bogus Scheme / Wisp / ..., > > I hope I've explained why something like `#!language/wisp` is perfectly within the bounds of R6RS. No, because Wisp is not R6RS -- R6RS is only relevant insofar the Wisp standard delegates to R6RS. (TBC I'm not claiming that #!language/wisp is invalid Wisp, I'm only claiming that your argumentation has holes here.) Also, you forgot the '...' in 'Scheme / Wisp / ...' -- while R6RS is somewhat relevant to Wisp, there exist languages over which the R6RS has no sway, e.g. BASIC. > Also, given that Guile already starts with non-standard extensions enabled by default, I don't see any reason not to also support `#lang language/wisp`. Here is a reason for not adding non-standard extensions, from a previous reply of mine: > The '#lang whatever' stuff makes Scheme (*) files unportable between implementations, as '#lang scheme' is not a valid comment -- there exist Schemes beyond Guile and Racket. If it were changed to recognising > '-*- mode: scheme -*-' or '-*- language: scheme -*-' or such, it would be better IMO, but insufficient, because (^). > > (*) Same argument applies for some, but not all, other non-Scheme languages too. That Guile might have made some mistakes with non-standard enabled-by-default language extensions in the past, does not mean that it should make more mistakes in the present. > In particular, the spelling of `#lang` proceeds directly from the Scheme tradition. This is from the R6RS Rationale document, chapter 4, "Lexical Syntax", section 3, "Future Extensions" [3]: [...] Again, the Scheme tradition holds no sway over non-Scheme languages (except for situations like Wisp, perhaps), e.g. Pascal and BASIC. Guile does not limit itself to Scheme languages, e.g. it has some support for elisp, brainfuck and python (see: python-on-guile). >> and >> 'the module system won't find it, because of the unexpected >> file extensions'. >> > > This is indeed something that needs to be addressed, but it seems like a very solvable problem. Using the extension ".scm" for everything would be one trivial solution. Something like your proposal to enable file extensions based on a compile-time option could likewise be part of a solution. The problem with the 'use .scm for everything' solution is that you would need to use .scm for everything, even non-Scheme files, and even when the source code comes from a project that uses a non-Guile implementation and as such uses very different extensions, e.g. '.js'. > In general, I'll say that, while using Guile, I've often missed Racket's more flexible constructs for importing modules. I especially miss `(require "foo/bar.rkt")`, which imports a module at a path relative to the module where the `require` form appears: it makes it easy to organize small programs into multiple files without having to mess with a load path. I fail to see the relevancy of this comment. Also, 'include' already doe something pretty close to this; presumably 'use-modules' could be modified to accept a #:relative-source-file-name argument: (define-module (baz)) ; /project/baz.scm ;; -> /project/foo/bar.rkt (use-modules ((foo bar) #:relative-source-file-name "foo/bar.rkt")) > On Thu, Feb 23, 2023, at 1:42 PM, Maxime Devos wrote: >> Have you seen my messages on how the "#lang" construct is problematic >> for some languages, and how alternatives like "[comment delimiter] -*- >> stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be >> equally simple (*) and not have any downsides (**). >> >> (*) The port encoding detection supports "-*- coding: whatever -*-", >> presumably that functionality could be reused. >> > > IMO, the use of "-*- coding: whatever -*-" to detect encoding is an ugly hack and should not be extended further. > > I tried to raise some objections above to conflating editor configuration with syntax saying what a file's language is. > > More broadly, I find "magic comments" highly objectionable. The whole point of comments is to be able to communicate freely to human readers without affecting the interpreter/compiler/evaluator. Introducing magic comments means must constantly think about whether what you are writing for humans might change the meaning of your program. Magic comments *without knowing a priori what is a comment* are even worse: now, you have to beware of accidental "magic" in ALL of the lexical syntax of your program. (Consider that something like `(define (-*- mode: c++ -*-) 14)` is perfectly good Scheme.) I object to the second claim -- while I can't account for aliens given the lack of them, I find it pointless to restrict the purpose of comments to human animals. The third and penultimate claim are false. If implemented correctly in Guile, only the first language declaration counts, it's not 'ALL of the lexical syntax of your program'. You previously claimed that programs should contain in-band information on which language something is written in. If this is followed, your example would actually look like: ;; -*- programming-language: scheme -*- ;; ^ or mode: c++, or #!r6rs, or an out-of-band --language=..., ... (define (-*- mode: c++ -*-) 14) As the relevant '-*- ...: scheme -*-' precedes the irrelevant '-*- mode: c++ -*-', it's the relevant one that is picked up by Guile, not the irrelevant one. As such, as long as the programmer uses the '--language=' compilation option in the Makefile, or puts a 'real' language declaration in the beginning of the source file (as a 'magic comment', or #!r6rs, or #lang as far as required for compatibility with Racket), things will work out. Even if the programmer doesn't do any of that, it's still unproblematic, because of error messages at compilation / interpretation time -- different languages tend to have incompatible syntax, if you pass a Scheme program to a C++ parser you'll just get a stream of syntax errors. Surely, the programmer will pass the code to the compiler or interpreter at some point, right? Otherwise, the programming was pointless. Likewise, test suites (ought to) exist, which would catch these problems even if they weren't written to catch these problems. (If they don't exist, then the programmer has much worse problems than a super implausible '(define (-*- mode: c++ -*- 14)' situation.) > > (It's not really relevant for the `#lang`-like case, but something I find especially ironic about encoding "magic comments" or, say, `<?xml version="1.0" encoding="UTF-8"?>`, is that suddenly if you encode the Unicode text in some other encoding it becomes a lie.) That sounds exactly the same situation as with #lang to me (and, as such, relevant). If you take a Scheme file #scheme ; ^ equivalent of <?xml version="1.0" encoding="UTF-8?>" [...] ; <- Scheme code and then convert it to Wisp, but forget to adjust the "#lang": #scheme ; ^ equivalent of <?xml version="1.0" encoding="something-else"?> [...]; <-- Wisp code then you'll get a bunch of syntax errors. > > On Fri, Feb 24, 2023, at 6:51 PM, Maxime Devos wrote: >> On 25-02-2023 00:48, Maxime Devos wrote: >>>>> (**) For compatibility with Racket, it's not like we couldn't >>>>> implement both "#lang" and "-*- stuff: language -*-". >> >> TBC, I mean ‘only support #lang' for values of 'lang' that Racket >> supports’ > > If I understand what you're proposing here, I don't think it's a viable option. > > The fundamental purpose of the `#lang` construct (however you spell it) is to provide an open, extensible protocol for defining languages. Thus, "values of 'lang' that Racket supports" are unbounded, provided that a module has been installed where the language specification says to look. From The Racket Reference [4]: The problem, as I wrote several times previously in different words, is that this 'open, extensible protocol' is not a standard protocol shared between languages. No language that precede the existence of Racket acknowledges this protocol in its specification of its syntax, and, like I said before, if the language doesn't have "#" comments, then #lang is also contrary to the syntax of the language. Like I wrote about R6RS: Racket only holds sway over Racket; it has no authority on the syntax of, say, BASIC and Pascal. Also, being unbounded in not a problem, because unbounded!=infinite. At any point in time, Racket itself only supports a finite number of 'values of 'lang'', and at any point at time there are only a finite number of external modules that implement certain 'lang'. As such, at any version of Guile, Guile could have a finite list of 'lang' where it recognises the Racket-specific extension #lang extension which is incompatible with non-Racket, non-Guile implementations. > [...] > I am definitely **not** suggesting that Guile implement all the details of Racket's `#lang` implementation. What I do strongly advocate is that you design Guile's support for `#lang` (or `#!`) to leave open a pathway for compatibility in the future. [...] The problem with this advocating, is that I agree with you here (except for 'you design' (*)), so why are you repeating this again? I wrote something among the lines ‘For __compatibility__ with Racket, __#lang should be recognised for values of 'lang' that are recognised by Racket__, but not for other languages’ (emphasis added). (*) Sure, someone could implement this compatibility, whatever, but we don't need this compatibility for Wisp. For Wisp, the more general and less problematic 'embed source file name in .go, + --language/file extension guessing' suffices. It's also rather pushy -- _you_ are demanding that _I_ paper over a source of incompatibility _introduced by others_ (Racket) (and furthermore _I_ consider that source of incompatibility _bad_), in the ML of a _volunteer project_, in a discussion that's ultimately about Wisp, not Racket, where _I_ (**) already have voluntarily designed a solution for Wisp? (**) And others maybe, I don't recall how much can be attributed to whom. > [...] > (Other kinds of potential namespace collisions are easier to manage: for example, we could imagine that `(use-modules (foo bar baz))` might not access the same module as `(require foo/bar/baz)`. [...] This is interesting but seems completely orthogonal; this e-mail thread is about detecting which language something is in, and finding source files with non-.scm modules, not about making the module system non-global. > [...] > I've sort of alluded above to my pipe dream of a grand unified future for Racket-and-Guile-on-Chez, Guile-and-Racket-on-the-Guile-VM, and endless other possibilities. I wrote about it in more detail on the guix-devel list at [10]. (These thoughts were inspired by conversations with Christine Lemmer-Webber, though she bears no responsibility for my zany imaginings.) OK, but what has this to do with this e-mail thread? This e-mail thread is about supporting additional languages, not about emulating Racket on top of Guile somehow (or perhaps you count Racket's dialect of Scheme as a language of its own to be implemented in Guile?). > Finally, I looked into the history of `#!` in R6RS a bit, and I'll leave a few pointers here for posterity. Will Clinger's 2015 Scheme Workshop paper [11] says in section 3.1 that "Kent Dybvig suggested the `#!r6rs` flag in May 2006", Clinger "formally proposed addition of Dybvig’s suggestion" [12], and, "less than six weeks later," `#!r6rs` was "in the R6RS editors’ status report". (I am not persuaded by all of the arguments about `#!r6rs` in that paper: in particular, the analysis doesn't seem to account for R6RS Appendix A [1].) As best as I can tell, the suggestion from Kent Dybvig is [13]: Again, how is RnRS relevant to _non-Scheme_ languages? Besides the 'shebangs actually are r6rs', I am disappointed by this discussion -- you keep repeating irrelevant points or points that were already addressed. (Again, R6RS and Racket are simply _irrelevant_ to non-Scheme languages that did not originate from Racket, and you are not giving arguments for them actually being relevant somehow.) As this line of discussion has proven to just be a pointless time sink, I will not read or respond to further replies by you in this line of discussion. Greetings, Maxime [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-26 15:42 ` Maxime Devos @ 2023-02-26 16:14 ` Dr. Arne Babenhauserheide 2023-02-26 17:58 ` Matt Wette 1 sibling, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-26 16:14 UTC (permalink / raw) To: Maxime Devos Cc: Philip McGrath, Ludovic Courtès, Matt Wette, Christine Lemmer-Webber, guile-devel [-- Attachment #1: Type: text/plain, Size: 1765 bytes --] Maxime Devos <maximedevos@telenet.be> writes: > Op 26-02-2023 om 08:45 schreef Philip McGrath: >> What I'm trying to advocate is that programs should say in-band, as part of their source code, what language they are written in. In-band is not the same as in-language. I agree that it should be part of the source code — at least for executable files - but this is already possible. I’ve been using shell-indirection for many years that ensures that files for the different languages are recognized and compiled with the correct language. It does not look as clean, but it allows solving a lot of problems. It ensures that when language/wisp.scm is not available, it first gets compiled as scheme, and then the wisp files are used as wisp, as in the file dryads-wake.w: #!/usr/bin/env bash # -*- wisp -*- # ensure that (language wisp) is pre-compiled if ! guile -L $(dirname $(realpath "$0")) -C $(dirname $(realpath "$0")) --language=wisp -c '' 2>/dev/null; then guile -L $(dirname $(realpath "$0")) -C $(dirname $(realpath "$0")) -c '(import (language wisp spec))' >/dev/null 2>&1 fi # run dryads-wake as module to ensure it is used pre-compiled exec -a "$0" guile -L $(dirname $(realpath "$0")) -C $(dirname $(realpath "$0")) --language=wisp -x .w -e '(dryads-wake)' -c '' "${@}" ; !# ; … code follows … This is an extra-linguistic feature, but it defines in the source-code what language is used for the different parts. And it has its own kind of elegance. Not in isolated language design, but in leveraging different parts of the GNU system to provide more capabilities than available from any of its parts. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-26 15:42 ` Maxime Devos 2023-02-26 16:14 ` Dr. Arne Babenhauserheide @ 2023-02-26 17:58 ` Matt Wette 2023-02-26 18:03 ` Dr. Arne Babenhauserheide 1 sibling, 1 reply; 83+ messages in thread From: Matt Wette @ 2023-02-26 17:58 UTC (permalink / raw) To: guile-devel With respect to file extensions, guile does not use file extension: You can name a file containing Scheme code "foo.js" and "guile foo.js" will execute it. The code in the wip-load-lang branch provides file-extension support: 1) Currently emacscript, with "js" extension, elisp, with "el" extension, and Scheme, with "scm" extension, are supported. 2) Additional languages can be added by calling add-lang-extension, from (system base compile): (add-lang-extension "m" 'nx-mlang) Matt ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-26 17:58 ` Matt Wette @ 2023-02-26 18:03 ` Dr. Arne Babenhauserheide 2023-02-26 18:20 ` Matt Wette 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-26 18:03 UTC (permalink / raw) To: Matt Wette; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 544 bytes --] Matt Wette <matt.wette@gmail.com> writes: > With respect to file extensions, guile does not use file extension: > You can name a file containing Scheme code "foo.js" and "guile foo.js" > will execute it. The module-system uses file extensions: If you (define-module (foo) #:export (main)) (define (main args) (display 'foo)) in foo.js, then guile -L . -e '(foo)' -c '' won’t find it. But it will find it in foo.scm. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-26 18:03 ` Dr. Arne Babenhauserheide @ 2023-02-26 18:20 ` Matt Wette 2023-02-26 21:39 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Matt Wette @ 2023-02-26 18:20 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel On 2/26/23 10:03 AM, Dr. Arne Babenhauserheide wrote: > Matt Wette <matt.wette@gmail.com> writes: > >> With respect to file extensions, guile does not use file extension: >> You can name a file containing Scheme code "foo.js" and "guile foo.js" >> will execute it. > The module-system uses file extensions: If you > > (define-module (foo) #:export (main)) > (define (main args) (display 'foo)) > > in foo.js, then > > guile -L . -e '(foo)' -c '' > > won’t find it. But it will find it in foo.scm. > Guile does not use file extensions consistently, then? ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-26 18:20 ` Matt Wette @ 2023-02-26 21:39 ` Dr. Arne Babenhauserheide 0 siblings, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-26 21:39 UTC (permalink / raw) To: Matt Wette; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 931 bytes --] Matt Wette <matt.wette@gmail.com> writes: > On 2/26/23 10:03 AM, Dr. Arne Babenhauserheide wrote: >> Matt Wette <matt.wette@gmail.com> writes: >> >>> You can name a file containing Scheme code "foo.js" and "guile foo.js" >> guile -L . -e '(foo)' -c '' >> >> won’t find it. But it will find it in foo.scm. >> > > Guile does not use file extensions consistently, then? Those are two different use-cases. With guile foo.js, guile does not have to search for the file: it just uses the one you passed. With -e '(foo)' it has to search for a matching module in the known / defined load paths. Note also guile -L . -e '(foo)' -c '' has lower startup time than guile foo.scm. If I understand it correctly that’s because it simply mmaps the compiled *.go file while guile foo.scm always reads the file. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-26 7:45 ` Philip McGrath 2023-02-26 15:42 ` Maxime Devos @ 2023-10-02 14:59 ` Christine Lemmer-Webber 2023-10-02 21:46 ` guile support for multiple languages [was: [PATCH] add language/wisp to Guile?] Matt Wette 1 sibling, 1 reply; 83+ messages in thread From: Christine Lemmer-Webber @ 2023-10-02 14:59 UTC (permalink / raw) To: Philip McGrath Cc: Maxime Devos, Ludovic Courtès, Matt Wette, guile-devel "Philip McGrath" <philip@philipmcgrath.com> writes: > I've sort of alluded above to my pipe dream of a grand unified future > for Racket-and-Guile-on-Chez, Guile-and-Racket-on-the-Guile-VM, and > endless other possibilities. I wrote about it in more detail on the > guix-devel list at [10]. (These thoughts were inspired by > conversations with Christine Lemmer-Webber, though she bears no > responsibility for my zany imaginings.) It's a long email so I'm only quoting the part that mentions me. ;) In general I think the hash-lang idea in Racket is a neat one. File extensions are also possible I suppose, but ultimately, when the program boots, there has to be some sort of way of finding the "current configuration" of languages. Note that #langs have an ambient authority problem, if we had a more ocap'y system, but... we're far from that, any module can do a mess of things, so I suppose if we're relying on the module system with ambient authority already, the same situation may apply. One thing that's worth noting is that languages do *two things* (or three): they provide a reader, and they provide an execution model (including an initial set of bindings). It would be nice if we could separate those two things as much as possible, when it is possible. I think readers are the least interesting part of language design usually. But obviously I support Wisp here, so... ;) - Christine ^ permalink raw reply [flat|nested] 83+ messages in thread
* guile support for multiple languages [was: [PATCH] add language/wisp to Guile?] 2023-10-02 14:59 ` Christine Lemmer-Webber @ 2023-10-02 21:46 ` Matt Wette 0 siblings, 0 replies; 83+ messages in thread From: Matt Wette @ 2023-10-02 21:46 UTC (permalink / raw) To: Christine Lemmer-Webber, Philip McGrath Cc: Maxime Devos, Ludovic Courtès, guile-devel On 10/2/23 7:59 AM, Christine Lemmer-Webber wrote: > "Philip McGrath" <philip@philipmcgrath.com> writes: > >> I've sort of alluded above to my pipe dream of a grand unified future >> for Racket-and-Guile-on-Chez, Guile-and-Racket-on-the-Guile-VM, and >> endless other possibilities. I wrote about it in more detail on the >> guix-devel list at [10]. (These thoughts were inspired by >> conversations with Christine Lemmer-Webber, though she bears no >> responsibility for my zany imaginings.) > It's a long email so I'm only quoting the part that mentions me. ;) > > In general I think the hash-lang idea in Racket is a neat one. File > extensions are also possible I suppose, but ultimately, when the program > boots, there has to be some sort of way of finding the "current > configuration" of languages. > > Note that #langs have an ambient authority problem, if we had a more > ocap'y system, but... we're far from that, any module can do a mess of > things, so I suppose if we're relying on the module system with ambient > authority already, the same situation may apply. > > One thing that's worth noting is that languages do *two things* (or > three): they provide a reader, and they provide an execution model > (including an initial set of bindings). It would be nice if we could > separate those two things as much as possible, when it is possible. > I think readers are the least interesting part of language design > usually. But obviously I support Wisp here, so... ;) > > - Christine Maybe I missed something, but I believe guile does separate these. Languages are supported with the "spec.scm" files in which are specified #:reader, #:evaluator and #:make-default-environment. Matt ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-18 3:50 ` Philip McGrath 2023-02-18 15:58 ` Maxime Devos @ 2023-02-23 7:59 ` Maxime Devos 2023-02-23 8:51 ` Dr. Arne Babenhauserheide 1 sibling, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-23 7:59 UTC (permalink / raw) To: Philip McGrath, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 1803 bytes --] >> In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or `_`." (See <https://docs.racket-lang.org/reference/reader.html#%28part._parse-reader%29>.) [...] > (Guile does not handle `#!r6rs` properly, presumably because of the legacy `#!`/`!#` block comments. I think this should be a surmountable obstacle, though, especially since Guile does support standard `#|`/`|#` block comments.) > > ‘#! ... !#’ comments aren't legacy; they exist to allow putting the shebang in the first line of a script, and to pass additional arguments to the Guile interpreter (see: (guile)The Top of a Script File) (*). As such, you can't just replace them with #| ... |# (unless you patch the kernel to recognise "#| ..." as a shebang line). [...] > > Furthermore, according to the kernel, #!r6rs would mean that the script needs to be interpreted by a program named 'r6rs', but 'guile' is named 'guile', not 'r6rs'. (I assume this is in POSIX somewhere, though I couldn't find it.) > > (This is an incompatibility between R6RS and any system that has shebangs.) Thinking a bit more about it, it should be possible to special-case Guile's interpretation of "#!" such that "#!r6rs" doesn't require a closing "!#". (Technically backwards-incompatible, but I don't think people are writing #!r6rs ...!# in the wild.) Still doesn't really address the problem though, as Scheme scripts (or scripts in another language) may need to start with a shebang and "#!lang" or "#lang" is not a valid comment in all languages. (E.g., I don't think it's valid Pascal, though I only have read some Pascal code, I haven't looked at the specification.) Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 931 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 7:59 ` [PATCH] add language/wisp to Guile? Maxime Devos @ 2023-02-23 8:51 ` Dr. Arne Babenhauserheide 2023-02-23 18:04 ` Maxime Devos 0 siblings, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-23 8:51 UTC (permalink / raw) To: Maxime Devos; +Cc: Philip McGrath, guile-devel [-- Attachment #1: Type: text/plain, Size: 2051 bytes --] Maxime Devos <maximedevos@telenet.be> writes: >> ‘#! ... !#’ comments aren't legacy; they exist to allow putting the >> shebang in the first line of a script, and to pass additional >> arguments to the Guile interpreter (see: (guile)The Top of a Script >> File) (*). This is awesome, by the way. It’s what allowed me to write wisp scripts that just work without having wisp shipped by starting as bash script, pre-compiling the language files, and then exec'ing guile with the right arguments that interprets the file as module and runs the code inside. >> Furthermore, according to the kernel, #!r6rs would mean that the >> script needs to be interpreted by a program named 'r6rs', but >> 'guile' is named 'guile', not 'r6rs'. (I assume this is in POSIX >> somewhere, though I couldn't find it.) We could fix that by installing a binary named r6rs. > Thinking a bit more about it, it should be possible to special-case > Guile's interpretation of "#!" such that "#!r6rs" doesn't require a > closing "!#". (Technically backwards-incompatible, but I don't think > people are writing #!r6rs ...!# in the wild.) Do you need the closing !# if you restrict yourself to the first line? > Still doesn't really address the problem though, as Scheme scripts (or > scripts in another language) may need to start with a shebang and > "#!lang" or "#lang" is not a valid comment in all languages. (E.g., I > don't think it's valid Pascal, though I only have read some Pascal > code, I haven't looked at the specification.) I think itmust be ignored in all languages that work as scripts in POSIX. So I would expect that support for ignoring #!... in the first line is very widespread. Also since the language implementation is in Guile, this could simply be added for Guile. That may prevent using this file from other implementations of the language, but it should work well enough as a first step. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 8:51 ` Dr. Arne Babenhauserheide @ 2023-02-23 18:04 ` Maxime Devos 2023-02-23 18:22 ` Maxime Devos ` (2 more replies) 0 siblings, 3 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-23 18:04 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Philip McGrath, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 1595 bytes --] On 23-02-2023 09:51, Dr. Arne Babenhauserheide wrote: >> Thinking a bit more about it, it should be possible to special-case >> Guile's interpretation of "#!" such that "#!r6rs" doesn't require a >> closing "!#". (Technically backwards-incompatible, but I don't think >> people are writing #!r6rs ...!# in the wild.) > Do you need the closing !# if you restrict yourself to the first line? I thought so at first, but doing a little experiment, it appears you don't need to: $ guile scheme@(guile-user)> #!r6rs (display "hi") (newline) (output: hi) Apparently Guile already has required behaviour. >>> Still doesn't really address the problem though, as Scheme scripts (or >>> scripts in another language) may need to start with a shebang and >>> "#!lang" or "#lang" is not a valid comment in all languages. (E.g., I >>> don't think it's valid Pascal, though I only have read some Pascal >>> code, I haven't looked at the specification.) >> I think itmust be ignored in all languages that work as scripts in >> POSIX. So I would expect that support for ignoring #!... in the first >> line is very widespread. The problem is that not all languages were made with POSIX-style scripts in mind, e.g. Pascal, BASIC and Java (*). Greetings, Maxime. (*) Java actually allows "#!", but only in 'Shebang' files (see: https://openjdk.org/jeps/330#Shebang_files). It remains invalid to put a '#!java' line in files with a class definition that is supposed to be found by Java's class loaders and compiler (in Guile terms, the source code of a module). [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 18:04 ` Maxime Devos @ 2023-02-23 18:22 ` Maxime Devos 2023-02-23 18:36 ` Maxime Devos 2023-02-23 18:37 ` Maxime Devos 2 siblings, 0 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-23 18:22 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Philip McGrath, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 2231 bytes --] On 23-02-2023 19:04, Maxime Devos wrote: >>>> Still doesn't really address the problem though, as Scheme scripts (or >>>> scripts in another language) may need to start with a shebang and >>>> "#!lang" or "#lang" is not a valid comment in all languages. (E.g., I >>>> don't think it's valid Pascal, though I only have read some Pascal >>>> code, I haven't looked at the specification.) >>> I think itmust be ignored in all languages that work as scripts in >>> POSIX. So I would expect that support for ignoring #!... in the first >>> line is very widespread. > > The problem is that not all languages were made with POSIX-style scripts > in mind, e.g. Pascal, BASIC and Java (*). I forgot about the following: >> Also since the language implementation is in Guile, this could simply be >> added for Guile. That may prevent using this file from other >> implementations of the language, but it should work well enough as a >> first step. I disagree, because there is an alternative solution that should be straightforward and avoids the downside of making an extension to the language that could confuse other implementations. Quoting myself: > The '#lang whatever' stuff makes Scheme (*) files unportable between implementations, as '#lang scheme' is not a valid comment -- there exist Schemes beyond Guile and Racket. If it were changed to recognising > '-*- mode: scheme -*-' or '-*- language: scheme -*-' or such, it would be better IMO, [...] E.g.: ;; -*- insert-bikeshed-here: scheme -*- // -*- insert-bikeshed-here: c++ -*- # -*- insert-bikeshed-here: ecmascript -*- /* -*- insert-bikeshed-here: c -*- */ {*** *- insert-bikeshed-here: pascal -*- ***} REM -*- insert-bikeshed-here: basic -*- I suppose the decision could be made to add support for "#lang" for compatibility with Racket when 'lang' is some language Racket supports, despite the fact that "#lang" is invalid in some of those languages, but IMO we should avoid adding new extensions that confuse ‘native’ implementations when something compatible like "-*- ... -*-" can be done instead. Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 18:04 ` Maxime Devos 2023-02-23 18:22 ` Maxime Devos @ 2023-02-23 18:36 ` Maxime Devos 2023-02-23 18:37 ` Maxime Devos 2 siblings, 0 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-23 18:36 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Philip McGrath, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 436 bytes --] On 23-02-2023 19:04, Maxime Devos wrote: > (*) Java actually allows "#!", but only in 'Shebang' files (see: > https://openjdk.org/jeps/330#Shebang_files). It remains invalid to put > a '#!java' line in files with a class definition that is supposed to be > found by Java's class loaders and compiler (in Guile terms, the source > code of a module). (Also, to my understanding, "#lang" is unconditionally invalid.) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-23 18:04 ` Maxime Devos 2023-02-23 18:22 ` Maxime Devos 2023-02-23 18:36 ` Maxime Devos @ 2023-02-23 18:37 ` Maxime Devos 2 siblings, 0 replies; 83+ messages in thread From: Maxime Devos @ 2023-02-23 18:37 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: Philip McGrath, guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 539 bytes --] On 23-02-2023 19:04, Maxime Devos wrote: > (*) Java actually allows "#!", but only in 'Shebang' files (see: > https://openjdk.org/jeps/330#Shebang_files). It remains invalid to put > a '#!java' line in files with a class definition that is supposed to be > found by Java's class loaders and compiler (in Guile terms, the source > code of a module). (Also, #java is unconditionally invalid, to my understanding.) (I pressed the 'Send button' a little to early when composing the previous version of this e-mail.) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-14 23:01 ` Maxime Devos 2023-02-15 1:46 ` Matt Wette @ 2023-02-15 8:36 ` Dr. Arne Babenhauserheide 2023-02-15 20:13 ` Maxime Devos 1 sibling, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-15 8:36 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 4556 bytes --] Maxime Devos <maximedevos@telenet.be> writes: >> [...] >> That would be nice, but would require doing changes in a critical core >> part of Guile. It would change this addition from a risk-free added >> feature to a risky core change. > > I maintain that a new language shouldn't be merged until the > Scheme-specific load path stuff is fixed/extended to work for > non-Scheme things (e.g. Wisp) too -- if this requires somewhat risky > (*) changes to core parts, then that just means we'll have to do some > risky stuff, then. > > I also expect that Guile maintainers will have the opposite opinion > (i.e., ‘fixing the load path stuff isn't necessary for merging a new > language implementation’). > > (*) FWIW I disagree on the 'risky' assessment -- it seems like a ‘if > it runs, it will work’ thing to me. That it modifies a core part of > Guile, makes it less risky IMO, as it would automatically be more > tested. > > Aside from the (*) and the 'I also expect [...],', I don't have > anything new to say about this, so I'll stop here. Thank you for your suggestions and contributions so far. They help me a lot! >> [...] >> That would also enable shipping pre-compiled software without >> sourcecode, > > That can already be done -- besides legalities, nothing stops people > from putting [^] or [^] .scm files in $GUILE_LOAD_PATH and putting .go > in $GUILE_LOAD_COMPILED_PATH. > > [^]: Redacted to not give people ideas on how to circumvent stuff. > I can elaborate by non-public e-mail if you like. Thank you! (for redacting) — I hope I’ll never need that :-) > On 14-02-2023 22:24, Dr. Arne Babenhauserheide wrote: >> PS: So what’s still missing here is to avoid setting the locale. Do you >> happen to have a hint how to actually do this right? > > I think you might have forgotten about this: > >> -- Scheme Procedure: set-port-encoding! port enc >> -- C Function: scm_set_port_encoding_x (port, enc) >> Sets the character encoding that will be used to interpret I/O to >> PORT. ENC is a string containing the name of an encoding. Valid >> encoding names are those defined by IANA >> (http://www.iana.org/assignments/character-sets), for example >> ‘"UTF-8"’ or ‘"ISO-8859-1"’. >> As such, I propose calling set-port-encoding! right in the beginning >> of read-one-wisp-sexp. Yikes, yes. I shouldn’t spend so much time thinking about implications when I haven’t yet applied all the clear and uncontested improvements. Thank you! > More concretely, replace > > (define (read-one-wisp-sexp port env) > ;; allow using "# foo" as #(foo). > (read-hash-extend #\# (λ (chr port) #\#)) > (cond > ((eof-object? (peek-char port)) > (read-char port )); return eof: we’re done > (else > (let ((chunk (wisp-scheme-read-chunk port))) > (cond > ((not (null? chunk)) > (car chunk)) > (else > #f)))))) > > by > > (define (read-one-wisp-sexp port env) > ;; Allow using "# foo" as #(foo). > ;; Don't use the globally-acting read-hash-extend, because this > ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme. > ;; Instead, use fluids to temporarily add the extension. > (define %read-hash-procedures/parameter > (fluid->parameter %read-hash-procedures)) > (parameterize ((%read-hash-procedures/parameter > `((#\# ,(λ (chr port) #\#)) > ,@(%read-hash-procedures/parameter)))) > ;; Read Wisp files as UTF-8, to support non-ASCII characters. > ;; TODO: would be nice to support ';; coding: whatever' lines > ;; like in parenthese-y Scheme. > (set-port-encoding! port "UTF-8") > (if (eof-object? (peek-char port)) > (read-char port) ; return eof: we’re done > (let ((chunk (wisp-scheme-read-chunk port))) > (and (not (null? chunk)) ; <---- XXX: maybe (pair? chunk) > (car chunk)))))) > > (untested). > > (I've also done the read-hash-extend stuff and simplified the 'cond' > expressions.) Thank you again for that! Which begs an important question: How would you like to be attributed? I plan to also merge this back to the wisp repo and I’d like to attribute you there, too. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-15 8:36 ` Dr. Arne Babenhauserheide @ 2023-02-15 20:13 ` Maxime Devos 2023-02-16 7:01 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-15 20:13 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 509 bytes --] > [...] > Which begs an important question: How would you like to be attributed? I > plan to also merge this back to the wisp repo and I’d like to attribute > you there, too. You could add a ";; Copyright © 2023 Maxime Devos <maximedevos@telenet.be>" line next to yours in the file that contains the read-one-wisp-sexp I modified, a line like 'https://hg.sr.ht/~arnebab/wisp/browse/NEWS?rev=tip#L63' in the NEWS and an entry in 'Specific Contributions' in AUTHORS.in. Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-15 20:13 ` Maxime Devos @ 2023-02-16 7:01 ` Dr. Arne Babenhauserheide 0 siblings, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-16 7:01 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 851 bytes --] Maxime Devos <maximedevos@telenet.be> writes: >> [...] >> Which begs an important question: How would you like to be attributed? I >> plan to also merge this back to the wisp repo and I’d like to attribute >> you there, too. > > You could add a ";; Copyright © 2023 Maxime Devos > <maximedevos@telenet.be>" line next to yours in the file that contains > the read-one-wisp-sexp I modified, a line like > 'https://hg.sr.ht/~arnebab/wisp/browse/NEWS?rev=tip#L63' in the NEWS > and an entry in 'Specific Contributions' in AUTHORS.in. I implemented that now — thank you! In the wisp-repo I added the copyright line in both the source file for wisp.scm and spec.scm, because I see the unlimited underscores as significant, too. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-04 15:08 ` Maxime Devos 2023-02-04 15:46 ` Dr. Arne Babenhauserheide @ 2023-02-16 8:03 ` Dr. Arne Babenhauserheide 2023-02-16 11:30 ` Maxime Devos 1 sibling, 1 reply; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-16 8:03 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1: Type: text/plain, Size: 1029 bytes --] Maxime Devos <maximedevos@telenet.be> writes: >> + ;; allow using "# foo" as #(foo). >> + (read-hash-extend #\# (λ (chr port) #\#)) > > That's a rather Wisp-specific extension, but it appears you are > extending things globally. Instead, I propose extending it > temporarily, with the undocumented '%read-hash-procedures' fluid. I tried the spec-example, but it didn’t work for me. Do you mean using it similar to this test? (pass-if "R6RS/SRFI-30 block comment syntax overridden" ;; To be compatible with 1.8 and earlier, we should be able to override ;; this syntax. (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) (read-hash-extend #\| (lambda args 'not)) (fold (lambda (x y result) (and result (eq? x y))) #t (read-string "(this is #| a comment)") `(this is not a comment)))) Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-16 8:03 ` Dr. Arne Babenhauserheide @ 2023-02-16 11:30 ` Maxime Devos 2023-02-16 21:35 ` Dr. Arne Babenhauserheide 0 siblings, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-02-16 11:30 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 1437 bytes --] On 16-02-2023 09:03, Dr. Arne Babenhauserheide wrote: > > Maxime Devos <maximedevos@telenet.be> writes: >>> + ;; allow using "# foo" as #(foo). >>> + (read-hash-extend #\# (λ (chr port) #\#)) >> >> That's a rather Wisp-specific extension, but it appears you are >> extending things globally. Instead, I propose extending it >> temporarily, with the undocumented '%read-hash-procedures' fluid. > > I tried the spec-example, but it didn’t work for me. Assuming that with 'spec-example', you meant (parameterize ((%read-hash-procedures/parameter `((#\# ,(λ (chr port) #\#)) ,@(%read-hash-procedures/parameter)))) [...]): I forgot to place a '.' between #\# and ,(λ ...). > Do you mean using it similar to this test? > > (pass-if "R6RS/SRFI-30 block comment syntax overridden" > ;; To be compatible with 1.8 and earlier, we should be able to override > ;; this syntax. > (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) > (read-hash-extend #\| (lambda args 'not)) > (fold (lambda (x y result) > (and result (eq? x y))) > #t > (read-string "(this is #| a comment)") > `(this is not a comment)))) That appears to me a valid (and slightly simpler and more robust) way of doing things, yes. Greetings, Maxime. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-16 11:30 ` Maxime Devos @ 2023-02-16 21:35 ` Dr. Arne Babenhauserheide 0 siblings, 0 replies; 83+ messages in thread From: Dr. Arne Babenhauserheide @ 2023-02-16 21:35 UTC (permalink / raw) To: Maxime Devos; +Cc: guile-devel [-- Attachment #1.1: Type: text/plain, Size: 961 bytes --] Maxime Devos <maximedevos@telenet.be> writes: > On 16-02-2023 09:03, Dr. Arne Babenhauserheide wrote: >> Do you mean using it similar to this test? >> (pass-if "R6RS/SRFI-30 block comment syntax overridden" >> ;; To be compatible with 1.8 and earlier, we should be able to override >> ;; this syntax. >> (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) >> (read-hash-extend #\| (lambda args 'not)) >> (fold (lambda (x y result) >> (and result (eq? x y))) >> #t >> (read-string "(this is #| a comment)") >> `(this is not a comment)))) > > That appears to me a valid (and slightly simpler and more robust) way > of doing things, yes. I finally got it working with passing tests — thank you! The reason that it did not work was that in the wisp repo there are tests that use wisp.scm directly without going through spec.scm. [-- Attachment #1.2: 0001-Only-extend-the-reader-while-reading-wisp.-Thanks-to.patch --] [-- Type: text/x-patch, Size: 2927 bytes --] From 12aa7314ad85f442f8bfe85839127bf1929be2ba Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide <arne_bab@web.de> Date: Thu, 16 Feb 2023 22:34:00 +0100 Subject: [PATCH] Only extend the reader while reading wisp. Thanks to Maxime Devos! * module/language/wisp/spec.scm (read-one-wisp-sexp): extend hash only in fluid * module/language/wisp.scm (wisp-scheme-read-chunk): extend hash in fluid --- module/language/wisp.scm | 18 ++++++++++-------- module/language/wisp/spec.scm | 8 ++------ 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/module/language/wisp.scm b/module/language/wisp.scm index 65ef8e65a..7a12e126a 100644 --- a/module/language/wisp.scm +++ b/module/language/wisp.scm @@ -725,14 +725,16 @@ Match is awesome!" a)))))) (define (wisp-scheme-read-chunk port) - "Read and parse one chunk of wisp-code" - (let (( lines (wisp-scheme-read-chunk-lines port))) - (wisp-make-improper - (wisp-replace-empty-eof - (wisp-unescape-underscore-and-colon - (wisp-replace-paren-quotation-repr - (wisp-propagate-source-properties - (wisp-scheme-indentation-to-parens lines)))))))) + "Read and parse one chunk of wisp-code" + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\#)) + (let ((lines (wisp-scheme-read-chunk-lines port))) + (wisp-make-improper + (wisp-replace-empty-eof + (wisp-unescape-underscore-and-colon + (wisp-replace-paren-quotation-repr + (wisp-propagate-source-properties + (wisp-scheme-indentation-to-parens lines))))))))) (define (wisp-scheme-read-all port) "Read all chunks from the given port" diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm index 3ba248aa9..477036c71 100644 --- a/module/language/wisp/spec.scm +++ b/module/language/wisp/spec.scm @@ -45,12 +45,8 @@ ;; Don't use the globally-acting read-hash-extend, because this ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme. ;; Instead, use fluids to temporarily add the extension. - (read-hash-extend #\# (λ (chr port) #\#)) - (define %read-hash-procedures/parameter - (fluid->parameter %read-hash-procedures)) - (parameterize ((%read-hash-procedures/parameter - `((#\# ,(λ (chr port) #\#)) - ,@(%read-hash-procedures/parameter)))) + (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures))) + (read-hash-extend #\# (lambda args #\# )) ;; Read Wisp files as UTF-8, to support non-ASCII characters. ;; TODO: would be nice to support ';; coding: whatever' lines ;; like in parenthese-y Scheme. -- 2.39.1 [-- Attachment #1.3: Type: text/plain, Size: 101 bytes --] Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-02-03 21:26 [PATCH] add language/wisp to Guile? Dr. Arne Babenhauserheide 2023-02-04 15:08 ` Maxime Devos @ 2023-09-30 13:17 ` Christine Lemmer-Webber 2023-09-30 20:09 ` Maxime Devos 1 sibling, 1 reply; 83+ messages in thread From: Christine Lemmer-Webber @ 2023-09-30 13:17 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-devel Haven't fully caught up on this thread, but as a side note I have a mostly-finished implementation of a Wisp parser which takes a very different approach than Arne's, and was more understandable to me personally, a bit more functional and recursive-descent style. I could make it available if anyone is curious. Would love to see Wisp in Guile proper! "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes: > [[PGP Signed Part:Undecided]] > Hi, > > Since (language wisp)¹ has been rock stable for years now and is used in > the Guix Workflow Language and supported in the Chickadee and the > Tsukundere game engines, I thought it coud be a good time to merge Wisp > into Guile itself. > > So I prepared a patch that adds language/wisp, some texinfo for > SRFI-119, and some tests. > > > Why add Wisp? > > For Wisp: it is then available directly wherever Guile is available. > This will make it much easier for people to follow tutorials. > > For Guile: > > - Wisp has proven to be good at enabling people to get an > entrance to Scheme² without pulling them out of the community. > > - It has also been shown to enable people who are used to other > programming languages to get a quick start at tools written in Guile. > > - And it provides access to the full capabilities of Guile with minimal > maintenance effort, because it is just the thinnest possible layer > around Scheme. The last required change was in 2020 while I used it > continuously. > > > The attached patch provides just the wisp reader, but not the > wisp->scheme transformer, because the latter has known broken edge-cases > (and who needs the transformer can get it from the wisp repo and execute > it directly with a Guile that then already supports wisp without any > path adaptions). > > > So I’d like to ask: can we merge Wisp as supported language into Guile? > > > Best wishes, > Arne > > > ¹: https://www.draketo.de/software/wisp > > ²: »Wisp allows people to see code how Lispers perceive it. Its > structure becomes apparent.« — Ricardo Wurmus in IRC > > From 4d4759f9fc67b01c40bde41b93e3998f7d64eabd Mon Sep 17 00:00:00 2001 > From: Arne Babenhauserheide <arne_bab@web.de> > Date: Fri, 3 Feb 2023 22:20:04 +0100 > Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation > > * doc/ref/srfi-modules.texi (srfi-119): add node > * module/language/wisp.scm: New file. > * module/language/wisp/spec.scm: New file. > * test-suite/tests/srfi-119.test: New file. > --- > doc/ref/srfi-modules.texi | 30 ++ > module/language/wisp.scm | 796 +++++++++++++++++++++++++++++++++ > module/language/wisp/spec.scm | 107 +++++ > test-suite/tests/srfi-119.test | 81 ++++ > 4 files changed, 1014 insertions(+) > create mode 100644 module/language/wisp.scm > create mode 100644 module/language/wisp/spec.scm > create mode 100644 test-suite/tests/srfi-119.test > > diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi > index 0ef136215..759e293ec 100644 > --- a/doc/ref/srfi-modules.texi > +++ b/doc/ref/srfi-modules.texi > @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page > * SRFI-98:: Accessing environment variables. > * SRFI-105:: Curly-infix expressions. > * SRFI-111:: Boxes. > +* SRFI-119:: Wisp: simpler indentation-sensitive scheme. > * SRFI-171:: Transducers > @end menu > > @@ -5662,6 +5663,34 @@ Return the current contents of @var{box}. > Set the contents of @var{box} to @var{value}. > @end deffn > > +@node SRFI-119 > +@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme. > +@cindex SRFI-119 > +@cindex wisp > + > +The languages shipped in Guile include SRFI-119 (wisp), an encoding of > +Scheme that allows replacing parentheses with equivalent indentation and > +inline colons. See > +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification > +of SRFI-119}. Some examples: > + > +@example > +display "Hello World!" @result{} (display "Hello World!") > +@end example > + > +@example > +define : factorial n @result{} (define (factorial n) > + if : zero? n @result{} (if (zero? n) > + . 1 @result{} 1 > + * n : factorial @{n - 1@} @result{} (* n (factorial @{n - 1@})))) > +@end example > + > +To execute a file with wisp code, select the language and filename > +extension @code{.w} vie @code{guile --language=wisp -x .w}. > + > +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated. > + > + > @node SRFI-171 > @subsection Transducers > @cindex SRFI-171 > @@ -5705,6 +5734,7 @@ left-to-right, due to how transducers are initiated. > * SRFI-171 Helpers:: Utilities for writing your own transducers > @end menu > > + > @node SRFI-171 General Discussion > @subsubsection SRFI-171 General Discussion > @cindex transducers discussion > diff --git a/module/language/wisp.scm b/module/language/wisp.scm > new file mode 100644 > index 000000000..ba24f54c5 > --- /dev/null > +++ b/module/language/wisp.scm > @@ -0,0 +1,796 @@ > +;;; Wisp > + > +;; Copyright (C) 2013, 2017, 2018, 2020 Free Software Foundation, Inc. > +;; Copyright (C) 2014--2023 Arne Babenhauserheide. > + > +;;;; This library is free software; you can redistribute it and/or > +;;;; modify it under the terms of the GNU Lesser General Public > +;;;; License as published by the Free Software Foundation; either > +;;;; version 3 of the License, or (at your option) any later version. > +;;;; > +;;;; This library is distributed in the hope that it will be useful, > +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of > +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > +;;;; Lesser General Public License for more details. > +;;;; > +;;;; You should have received a copy of the GNU Lesser General Public > +;;;; License along with this library; if not, write to the Free Software > +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + > +;;; Commentary: > + > +;; Scheme-only implementation of a wisp-preprocessor which output a > +;; scheme code tree to feed to a scheme interpreter instead of a > +;; preprocessed file. > + > +;; Limitations: > +;; - only unescapes up to 12 leading underscores at line start (\____________) > +;; - in some cases the source line information is missing in backtraces. > +;; check for set-source-property! > + > +;;; Code: > + > +(define-module (language wisp) > + #:export (wisp-scheme-read-chunk wisp-scheme-read-all > + wisp-scheme-read-file-chunk wisp-scheme-read-file > + wisp-scheme-read-string)) > + > +; use curly-infix by default > +(read-enable 'curly-infix) > + > +(use-modules > + (srfi srfi-1) > + (srfi srfi-11 ); for let-values > + (ice-9 rw ); for write-string/partial > + (ice-9 match)) > + > + > +;; Helper functions for the indent-and-symbols data structure: '((indent token token ...) ...) > +(define (line-indent line) > + (car line)) > + > +(define (line-real-indent line) > + "Get the indentation without the comment-marker for unindented lines (-1 is treated as 0)." > + (let (( indent (line-indent line))) > + (if (= -1 indent) > + 0 > + indent))) > + > +(define (line-code line) > + (let ((code (cdr line))) > + ; propagate source properties > + (when (not (null? code)) > + (set-source-properties! code (source-properties line))) > + code)) > + > +; literal values I need > +(define readcolon > + (string->symbol ":")) > + > +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd") > +; define an intermediate dot replacement with UUID to avoid clashes. > +(define repr-dot ; . > + (string->symbol (string-append "REPR-DOT-" wisp-uuid))) > + > +; allow using reader additions as the first element on a line to prefix the list > +(define repr-quote ; ' > + (string->symbol (string-append "REPR-QUOTE-" wisp-uuid))) > +(define repr-unquote ; , > + (string->symbol (string-append "REPR-UNQUOTE-" wisp-uuid))) > +(define repr-quasiquote ; ` > + (string->symbol (string-append "REPR-QUASIQUOTE-" wisp-uuid))) > +(define repr-unquote-splicing ; ,@ > + (string->symbol (string-append "REPR-UNQUOTESPLICING-" wisp-uuid))) > + > +(define repr-syntax ; #' > + (string->symbol (string-append "REPR-SYNTAX-" wisp-uuid))) > +(define repr-unsyntax ; #, > + (string->symbol (string-append "REPR-UNSYNTAX-" wisp-uuid))) > +(define repr-quasisyntax ; #` > + (string->symbol (string-append "REPR-QUASISYNTAX-" wisp-uuid))) > +(define repr-unsyntax-splicing ; #,@ > + (string->symbol (string-append "REPR-UNSYNTAXSPLICING-" wisp-uuid))) > + > +; TODO: wrap the reader to return the repr of the syntax reader > +; additions > + > +(define (match-charlist-to-repr charlist) > + (let > + ((chlist (reverse charlist))) > + (cond > + ((equal? chlist (list #\.)) > + repr-dot) > + ((equal? chlist (list #\')) > + repr-quote) > + ((equal? chlist (list #\,)) > + repr-unquote) > + ((equal? chlist (list #\`)) > + repr-quasiquote) > + ((equal? chlist (list #\, #\@ )) > + repr-unquote-splicing) > + ((equal? chlist (list #\# #\' )) > + repr-syntax) > + ((equal? chlist (list #\# #\, )) > + repr-unsyntax) > + ((equal? chlist (list #\# #\` )) > + repr-quasisyntax) > + ((equal? chlist (list #\# #\, #\@ )) > + repr-unsyntax-splicing) > + (else > + #f)))) > + > +(define (wisp-read port) > + "wrap read to catch list prefixes." > + (let ((prefix-maxlen 4)) > + (let longpeek > + ((peeked '()) > + (repr-symbol #f)) > + (cond > + ((or (< prefix-maxlen (length peeked)) (eof-object? (peek-char port)) (equal? #\space (peek-char port)) (equal? #\newline (peek-char port)) ) > + (if repr-symbol ; found a special symbol, return it. > + ; TODO: Somehow store source-properties. The commented-out code below does not work. > + ; catch #t > + ; lambda () > + ; write : source-properties symbol-or-symbols > + ; set-source-property! symbol-or-symbols 'filename : port-filename port > + ; set-source-property! symbol-or-symbols 'line : 1+ : port-line port > + ; set-source-property! symbol-or-symbols 'column : port-column port > + ; write : source-properties symbol-or-symbols > + ; lambda : key . arguments > + ; . #f > + repr-symbol > + (let unpeek > + ((remaining peeked)) > + (cond > + ((equal? '() remaining ) > + (read port )); let read to the work > + (else > + (unread-char (car remaining) port) > + (unpeek (cdr remaining))))))) > + (else > + (let* > + ((next-char (read-char port)) > + (peeked (cons next-char peeked))) > + (longpeek > + peeked > + (match-charlist-to-repr peeked)))))))) > + > + > + > +(define (line-continues? line) > + (equal? repr-dot (car (line-code line)))) > + > +(define (line-only-colon? line) > + (and > + (equal? ":" (car (line-code line))) > + (null? (cdr (line-code line))))) > + > +(define (line-empty-code? line) > + (null? (line-code line))) > + > +(define (line-empty? line) > + (and > + ; if indent is -1, we stripped a comment, so the line was not really empty. > + (= 0 (line-indent line)) > + (line-empty-code? line))) > + > +(define (line-strip-continuation line ) > + (if (line-continues? line) > + (append > + (list > + (line-indent line)) > + (cdr (line-code line))) > + line)) > + > +(define (line-strip-indentation-marker line) > + "Strip the indentation markers from the beginning of the line" > + (cdr line)) > + > +(define (indent-level-reduction indentation-levels level select-fun) > + "Reduce the INDENTATION-LEVELS to the given LEVEL and return the value selected by SELECT-FUN" > + (let loop > + ((newlevels indentation-levels) > + (diff 0)) > + (cond > + ((= level (car newlevels)) > + (select-fun (list diff indentation-levels))) > + ((< level (car newlevels)) > + (loop > + (cdr newlevels) > + (1+ diff))) > + (else > + (throw 'wisp-syntax-error "Level ~A not found in the indentation-levels ~A."))))) > + > +(define (indent-level-difference indentation-levels level) > + "Find how many indentation levels need to be popped off to find the given level." > + (indent-level-reduction indentation-levels level > + (lambda (x ); get the count > + (car x)))) > + > +(define (indent-reduce-to-level indentation-levels level) > + "Find how many indentation levels need to be popped off to find the given level." > + (indent-level-reduction indentation-levels level > + (lambda (x ); get the levels > + (car (cdr x))))) > + > +(define (chunk-ends-with-period currentsymbols next-char) > + "Check whether indent-and-symbols ends with a period, indicating the end of a chunk." > + (and (not (null? currentsymbols)) > + (equal? #\newline next-char) > + (equal? repr-dot > + (list-ref currentsymbols (- (length currentsymbols) 1))))) > + > +(define (wisp-scheme-read-chunk-lines port) > + (let loop > + ((indent-and-symbols (list )); '((5 "(foobar)" "\"yobble\"")(3 "#t")) > + (inindent #t) > + (inunderscoreindent (equal? #\_ (peek-char port))) > + (incomment #f) > + (currentindent 0) > + (currentsymbols '()) > + (emptylines 0)) > + (cond > + ((>= emptylines 2 ); the chunk end has to be checked > + ; before we look for new chars in the > + ; port to make execution in the REPL > + ; after two empty lines work > + ; (otherwise it shows one more line). > + indent-and-symbols) > + (else > + (let ((next-char (peek-char port))) > + (cond > + ((eof-object? next-char) > + (append indent-and-symbols (list (append (list currentindent) currentsymbols)))) > + ((and inindent (zero? currentindent) (not incomment) (not (null? indent-and-symbols)) (not inunderscoreindent) (not (or (equal? #\space next-char) (equal? #\newline next-char) (equal? (string-ref ";" 0) next-char)))) > + (append indent-and-symbols )); top-level form ends chunk > + ((chunk-ends-with-period currentsymbols next-char) > + ; the line ends with a period. This is forbidden in > + ; SRFI-119. Use it to end the line in the REPL without > + ; showing continuation dots (...). > + (append indent-and-symbols (list (append (list currentindent) (drop-right currentsymbols 1))))) > + ((and inindent (equal? #\space next-char)) > + (read-char port ); remove char > + (loop > + indent-and-symbols > + #t ; inindent > + #f ; inunderscoreindent > + #f ; incomment > + (1+ currentindent) > + currentsymbols > + emptylines)) > + ((and inunderscoreindent (equal? #\_ next-char)) > + (read-char port ); remove char > + (loop > + indent-and-symbols > + #t ; inindent > + #t ; inunderscoreindent > + #f ; incomment > + (1+ currentindent) > + currentsymbols > + emptylines)) > + ; any char but whitespace *after* underscoreindent is > + ; an error. This is stricter than the current wisp > + ; syntax definition. TODO: Fix the definition. Better > + ; start too strict. FIXME: breaks on lines with only > + ; underscores which should be empty lines. > + ((and inunderscoreindent (and (not (equal? #\space next-char)) (not (equal? #\newline next-char)))) > + (throw 'wisp-syntax-error "initial underscores without following whitespace at beginning of the line after" (last indent-and-symbols))) > + ((equal? #\newline next-char) > + (read-char port ); remove the newline > + ; The following two lines would break the REPL by requiring one char too many. > + ; if : and (equal? #\newline next-char) : equal? #\return : peek-char port > + ; read-char port ; remove a full \n\r. Damn special cases... > + (let* ; distinguish pure whitespace lines and lines > + ; with comment by giving the former zero > + ; indent. Lines with a comment at zero indent > + ; get indent -1 for the same reason - meaning > + ; not actually empty. > + ( > + (indent > + (cond > + (incomment > + (if (= 0 currentindent ); specialcase > + -1 > + currentindent )) > + ((not (null? currentsymbols )); pure whitespace > + currentindent) > + (else > + 0))) > + (parsedline (append (list indent) currentsymbols)) > + (emptylines > + (if (not (line-empty? parsedline)) > + 0 > + (1+ emptylines)))) > + (when (not (= 0 (length parsedline))) > + ; set the source properties to parsedline so we can try to add them later. > + (set-source-property! parsedline 'filename (port-filename port)) > + (set-source-property! parsedline 'line (port-line port))) > + ; TODO: If the line is empty. Either do it here and do not add it, just > + ; increment the empty line counter, or strip it later. Replace indent > + ; -1 by indent 0 afterwards. > + (loop > + (append indent-and-symbols (list parsedline)) > + #t ; inindent > + (if (<= 2 emptylines) > + #f ; chunk ends here > + (equal? #\_ (peek-char port ))); are we in underscore indent? > + #f ; incomment > + 0 > + '() > + emptylines))) > + ((equal? #t incomment) > + (read-char port ); remove one comment character > + (loop > + indent-and-symbols > + #f ; inindent > + #f ; inunderscoreindent > + #t ; incomment > + currentindent > + currentsymbols > + emptylines)) > + ((or (equal? #\space next-char) (equal? #\tab next-char) (equal? #\return next-char) ); remove whitespace when not in indent > + (read-char port ); remove char > + (loop > + indent-and-symbols > + #f ; inindent > + #f ; inunderscoreindent > + #f ; incomment > + currentindent > + currentsymbols > + emptylines)) > + ; | cludge to appease the former wisp parser > + ; | used for bootstrapping which has a > + ; v problem with the literal comment char > + ((equal? (string-ref ";" 0) next-char) > + (loop > + indent-and-symbols > + #f ; inindent > + #f ; inunderscoreindent > + #t ; incomment > + currentindent > + currentsymbols > + emptylines)) > + (else ; use the reader > + (loop > + indent-and-symbols > + #f ; inindent > + #f ; inunderscoreindent > + #f ; incomment > + currentindent > + ; this also takes care of the hashbang and leading comments. > + (append currentsymbols (list (wisp-read port))) > + emptylines)))))))) > + > + > +(define (line-code-replace-inline-colons line) > + "Replace inline colons by opening parens which close at the end of the line" > + ; format #t "replace inline colons for line ~A\n" line > + (let loop > + ((processed '()) > + (unprocessed line)) > + (cond > + ((null? unprocessed) > + ; format #t "inline-colons processed line: ~A\n" processed > + processed) > + ; replace : . with nothing > + ((and (<= 2 (length unprocessed)) (equal? readcolon (car unprocessed)) (equal? repr-dot (car (cdr unprocessed)))) > + (loop > + (append processed > + (loop '() (cdr (cdr unprocessed)))) > + '())) > + ((equal? readcolon (car unprocessed)) > + (loop > + ; FIXME: This should turn unprocessed into a list. > + (append processed > + (list (loop '() (cdr unprocessed)))) > + '())) > + (else > + (loop > + (append processed > + (list (car unprocessed))) > + (cdr unprocessed)))))) > + > +(define (line-replace-inline-colons line) > + (cons > + (line-indent line) > + (line-code-replace-inline-colons (line-code line)))) > + > +(define (line-strip-lone-colon line) > + "A line consisting only of a colon is just a marked indentation level. We need to kill the colon before replacing inline colons." > + (if > + (equal? > + (line-code line) > + (list readcolon)) > + (list (line-indent line)) > + line)) > + > +(define (line-finalize line) > + "Process all wisp-specific information in a line and strip it" > + (let > + ( > + (l > + (line-code-replace-inline-colons > + (line-strip-indentation-marker > + (line-strip-lone-colon > + (line-strip-continuation line)))))) > + (when (not (null? (source-properties line))) > + (catch #t > + (lambda () > + (set-source-properties! l (source-properties line))) > + (lambda (key . arguments) > + #f))) > + l)) > + > +(define (wisp-add-source-properties-from source target) > + "Copy the source properties from source into the target and return the target." > + (catch #t > + (lambda () > + (set-source-properties! target (source-properties source))) > + (lambda (key . arguments) > + #f)) > + target) > + > +(define (wisp-propagate-source-properties code) > + "Propagate the source properties from the sourrounding list into every part of the code." > + (let loop > + ((processed '()) > + (unprocessed code)) > + (cond > + ((and (null? processed) (not (pair? unprocessed)) (not (list? unprocessed))) > + unprocessed) > + ((and (pair? unprocessed) (not (list? unprocessed))) > + (cons > + (wisp-propagate-source-properties (car unprocessed)) > + (wisp-propagate-source-properties (cdr unprocessed)))) > + ((null? unprocessed) > + processed) > + (else > + (let ((line (car unprocessed))) > + (if (null? (source-properties unprocessed)) > + (wisp-add-source-properties-from line unprocessed) > + (wisp-add-source-properties-from unprocessed line)) > + (loop > + (append processed (list (wisp-propagate-source-properties line))) > + (cdr unprocessed))))))) > + > +(define* (wisp-scheme-indentation-to-parens lines) > + "Add parentheses to lines and remove the indentation markers" > + (when > + (and > + (not (null? lines)) > + (not (line-empty-code? (car lines))) > + (not (= 0 (line-real-indent (car lines ))))); -1 is a line with a comment > + (if (= 1 (line-real-indent (car lines))) > + ;; accept a single space as indentation of the first line (and ignore the indentation) to support meta commands > + (set! lines > + (cons > + (cons 0 (cdr (car lines))) > + (cdr lines))) > + (throw 'wisp-syntax-error > + (format #f "The first symbol in a chunk must start at zero indentation. Indentation and line: ~A" > + (car lines))))) > + (let loop > + ((processed '()) > + (unprocessed lines) > + (indentation-levels '(0))) > + (let* > + ( > + (current-line > + (if (<= 1 (length unprocessed)) > + (car unprocessed) > + (list 0 ))); empty code > + (next-line > + (if (<= 2 (length unprocessed)) > + (car (cdr unprocessed)) > + (list 0 ))); empty code > + (current-indentation > + (car indentation-levels)) > + (current-line-indentation (line-real-indent current-line))) > + ; format #t "processed: ~A\ncurrent-line: ~A\nnext-line: ~A\nunprocessed: ~A\nindentation-levels: ~A\ncurrent-indentation: ~A\n\n" > + ; . processed current-line next-line unprocessed indentation-levels current-indentation > + (cond > + ; the real end: this is reported to the outside world. > + ((and (null? unprocessed) (not (null? indentation-levels)) (null? (cdr indentation-levels))) > + ; display "done\n" > + ; reverse the processed lines, because I use cons. > + processed) > + ; the recursion end-condition > + ((and (null? unprocessed)) > + ; display "last step\n" > + ; this is the last step. Nothing more to do except > + ; for rolling up the indentation levels. return the > + ; new processed and unprocessed lists: this is a > + ; side-recursion > + (values processed unprocessed)) > + ((null? indentation-levels) > + ; display "indentation-levels null\n" > + (throw 'wisp-programming-error "The indentation-levels are null but the current-line is null: Something killed the indentation-levels.")) > + (else ; now we come to the line-comparisons and indentation-counting. > + (cond > + ((line-empty-code? current-line) > + ; display "current-line empty\n" > + ; We cannot process indentation without > + ; code. Just switch to the next line. This should > + ; only happen at the start of the recursion. > + ; TODO: Somehow preserve the line-numbers. > + (loop > + processed > + (cdr unprocessed) > + indentation-levels)) > + ((and (line-empty-code? next-line) (<= 2 (length unprocessed ))) > + ; display "next-line empty\n" > + ; TODO: Somehow preserve the line-numbers. > + ; take out the next-line from unprocessed. > + (loop > + processed > + (cons current-line > + (cdr (cdr unprocessed))) > + indentation-levels)) > + ((> current-indentation current-line-indentation) > + ; display "current-indent > next-line\n" > + ; this just steps back one level via the side-recursion. > + (let ((previous-indentation (car (cdr indentation-levels)))) > + (if (<= current-line-indentation previous-indentation) > + (values processed unprocessed) > + (begin ;; not yet used level! TODO: maybe throw an error here instead of a warning. > + (let ((linenumber (- (length lines) (length unprocessed)))) > + (format (current-error-port) ";;; WARNING:~A: used lower but undefined indentation level (line ~A of the current chunk: ~S). This makes refactoring much more error-prone, therefore it might become an error in a later version of Wisp.\n" (source-property current-line 'line) linenumber (cdr current-line))) > + (loop > + processed > + unprocessed > + (cons ; recursion via the indentation-levels > + current-line-indentation > + (cdr indentation-levels))))))) > + ((= current-indentation current-line-indentation) > + ; display "current-indent = next-line\n" > + (let > + ((line (line-finalize current-line)) > + (next-line-indentation (line-real-indent next-line))) > + (cond > + ((>= current-line-indentation next-line-indentation) > + ; simple recursiive step to the next line > + ; display "current-line-indent >= next-line-indent\n" > + (loop > + (append processed > + (if (line-continues? current-line) > + line > + (wisp-add-source-properties-from line (list line)))) > + (cdr unprocessed ); recursion here > + indentation-levels)) > + ((< current-line-indentation next-line-indentation) > + ; display "current-line-indent < next-line-indent\n" > + ; format #t "line: ~A\n" line > + ; side-recursion via a sublist > + (let-values > + ( > + ((sub-processed sub-unprocessed) > + (loop > + line > + (cdr unprocessed ); recursion here > + indentation-levels))) > + ; format #t "side-recursion:\n sub-processed: ~A\n processed: ~A\n\n" sub-processed processed > + (loop > + (append processed (list sub-processed)) > + sub-unprocessed ; simply use the recursion from the sub-recursion > + indentation-levels)))))) > + ((< current-indentation current-line-indentation) > + ; display "current-indent < next-line\n" > + (loop > + processed > + unprocessed > + (cons ; recursion via the indentation-levels > + current-line-indentation > + indentation-levels))) > + (else > + (throw 'wisp-not-implemented > + (format #f "Need to implement further line comparison: current: ~A, next: ~A, processed: ~A." > + current-line next-line processed))))))))) > + > + > +(define (wisp-scheme-replace-inline-colons lines) > + "Replace inline colons by opening parens which close at the end of the line" > + (let loop > + ((processed '()) > + (unprocessed lines)) > + (if (null? unprocessed) > + processed > + (loop > + (append processed (list (line-replace-inline-colons (car unprocessed)))) > + (cdr unprocessed))))) > + > + > +(define (wisp-scheme-strip-indentation-markers lines) > + "Strip the indentation markers from the beginning of the lines" > + (let loop > + ((processed '()) > + (unprocessed lines)) > + (if (null? unprocessed) > + processed > + (loop > + (append processed (cdr (car unprocessed))) > + (cdr unprocessed))))) > + > +(define (wisp-unescape-underscore-and-colon code) > + "replace \\_ and \\: by _ and :" > + (match code > + ((a ...) > + (map wisp-unescape-underscore-and-colon a)) > + ('\_ > + '_) > + ('\__ > + '__) > + ('\___ > + '___) > + ('\____ > + '____) > + ('\_____ > + '_____) > + ('\______ > + '______) > + ('\_______ > + '_______) > + ('\________ > + '________) > + ('\_________ > + '_________) > + ('\__________ > + '__________) > + ('\___________ > + '___________) > + ('\____________ > + '____________) > + ('\: > + ':) > + (a > + a))) > + > + > +(define (wisp-replace-empty-eof code) > + "replace ((#<eof>)) by ()" > + ; FIXME: Actually this is a hack which fixes a bug when the > + ; parser hits files with only hashbang and comments. > + (if (and (not (null? code)) (pair? (car code)) (eof-object? (car (car code))) (null? (cdr code)) (null? (cdr (car code)))) > + (list) > + code)) > + > + > +(define (wisp-replace-paren-quotation-repr code) > + "Replace lists starting with a quotation symbol by > + quoted lists." > + (match code > + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'quote (map wisp-replace-paren-quotation-repr a))) > + ((a ... 'REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list > + (append > + (map wisp-replace-paren-quotation-repr a) > + (list (list 'quote (map wisp-replace-paren-quotation-repr b))))) > + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'quasiquote (list 'unquote (map wisp-replace-paren-quotation-repr a)))) > + (('REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'unquote (map wisp-replace-paren-quotation-repr a))) > + ((a ... 'REPR-UNQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ) > + (append > + (map wisp-replace-paren-quotation-repr a) > + (list (list 'unquote (map wisp-replace-paren-quotation-repr b))))) > + (('REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'quasiquote (map wisp-replace-paren-quotation-repr a))) > + ((a ... 'REPR-QUASIQUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd b ); this is the quoted empty list > + (append > + (map wisp-replace-paren-quotation-repr a) > + (list (list 'quasiquote (map wisp-replace-paren-quotation-repr b))))) > + (('REPR-UNQUOTESPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'unquote-splicing (map wisp-replace-paren-quotation-repr a))) > + (('REPR-SYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'syntax (map wisp-replace-paren-quotation-repr a))) > + (('REPR-UNSYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'unsyntax (map wisp-replace-paren-quotation-repr a))) > + (('REPR-QUASISYNTAX-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'quasisyntax (map wisp-replace-paren-quotation-repr a))) > + (('REPR-UNSYNTAXSPLICING-e749c73d-c826-47e2-a798-c16c13cb89dd a ...) > + (list 'unsyntax-splicing (map wisp-replace-paren-quotation-repr a))) > + ;; literal array as start of a line: # (a b) c -> (#(a b) c) > + ((#\# a ...) > + (with-input-from-string ;; hack to defer to read > + (string-append "#" > + (with-output-to-string > + (λ () > + (write (map wisp-replace-paren-quotation-repr a) > + (current-output-port))))) > + read)) > + ((a ...) > + (map wisp-replace-paren-quotation-repr a)) > + (a > + a))) > + > +(define (wisp-make-improper code) > + "Turn (a #{.}# b) into the correct (a . b). > + > +read called on a single dot creates a variable named #{.}# (|.| > +in r7rs). Due to parsing the indentation before the list > +structure is known, the reader cannot create improper lists > +when it reads a dot. So we have to take another pass over the > +code to recreate the improper lists. > + > +Match is awesome!" > + (let > + ( > + (improper > + (match code > + ((a ... b 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd c) > + (append (map wisp-make-improper a) > + (cons (wisp-make-improper b) (wisp-make-improper c)))) > + ((a ...) > + (map wisp-make-improper a)) > + (a > + a)))) > + (define (syntax-error li msg) > + (throw 'wisp-syntax-error (format #f "incorrect dot-syntax #{.}# in code: ~A: ~A" msg li))) > + (if #t > + improper > + (let check > + ((tocheck improper)) > + (match tocheck > + ; lists with only one member > + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) > + (syntax-error tocheck "list with the period as only member")) > + ; list with remaining dot. > + ((a ...) > + (if (and (member repr-dot a)) > + (syntax-error tocheck "leftover period in list") > + (map check a))) > + ; simple pair - this and the next do not work when parsed from wisp-scheme itself. Why? > + (('REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd . c) > + (syntax-error tocheck "dot as first element in already improper pair")) > + ; simple pair, other way round > + ((a . 'REPR-DOT-e749c73d-c826-47e2-a798-c16c13cb89dd) > + (syntax-error tocheck "dot as last element in already improper pair")) > + ; more complex pairs > + ((? pair? a) > + (let > + ((head (drop-right a 1)) > + (tail (last-pair a))) > + (cond > + ((equal? repr-dot (car tail)) > + (syntax-error tocheck "equal? repr-dot : car tail")) > + ((equal? repr-dot (cdr tail)) > + (syntax-error tocheck "equal? repr-dot : cdr tail")) > + ((member repr-dot head) > + (syntax-error tocheck "member repr-dot head")) > + (else > + a)))) > + (a > + a)))))) > + > +(define (wisp-scheme-read-chunk port) > + "Read and parse one chunk of wisp-code" > + (let (( lines (wisp-scheme-read-chunk-lines port))) > + (wisp-make-improper > + (wisp-replace-empty-eof > + (wisp-unescape-underscore-and-colon > + (wisp-replace-paren-quotation-repr > + (wisp-propagate-source-properties > + (wisp-scheme-indentation-to-parens lines)))))))) > + > +(define (wisp-scheme-read-all port) > + "Read all chunks from the given port" > + (let loop > + ((tokens '())) > + (cond > + ((eof-object? (peek-char port)) > + tokens) > + (else > + (loop > + (append tokens (wisp-scheme-read-chunk port))))))) > + > +(define (wisp-scheme-read-file path) > + (call-with-input-file path wisp-scheme-read-all)) > + > +(define (wisp-scheme-read-file-chunk path) > + (call-with-input-file path wisp-scheme-read-chunk)) > + > +(define (wisp-scheme-read-string str) > + (call-with-input-string str wisp-scheme-read-all)) > + > +(define (wisp-scheme-read-string-chunk str) > + (call-with-input-string str wisp-scheme-read-chunk)) > + > diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm > new file mode 100644 > index 000000000..d5ea7abce > --- /dev/null > +++ b/module/language/wisp/spec.scm > @@ -0,0 +1,107 @@ > +;; Language interface for Wisp in Guile > + > +;;; adapted from guile-sweet: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/common.scm > + > +;;; Copyright (C) 2005-2014 by David A. Wheeler and Alan Manuel K. Gloria > +;;; Copyright (C) Arne Babenhauserheide (2014--2023). > + > +;;; Permission is hereby granted, free of charge, to any person > +;;; obtaining a copy of this software and associated documentation > +;;; files (the "Software"), to deal in the Software without > +;;; restriction, including without limitation the rights to use, copy, > +;;; modify, merge, publish, distribute, sublicense, and/or sell copies > +;;; of the Software, and to permit persons to whom the Software is > +;;; furnished to do so, subject to the following conditions: > +;;; > +;;; The above copyright notice and this permission notice shall be > +;;; included in all copies or substantial portions of the Software. > +;;; > +;;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > +;;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > +;;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND > +;;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS > +;;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN > +;;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN > +;;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > +;;; SOFTWARE. > + > +; adapted from spec.scm: https://gitorious.org/nacre/guile-sweet/source/ae306867e371cb4b56e00bb60a50d9a0b8353109:sweet/spec.scm > +(define-module (language wisp spec) > + #:use-module (language wisp) > + #:use-module (system base compile) > + #:use-module (system base language) > + #:use-module (language scheme compile-tree-il) > + #:use-module (language scheme decompile-tree-il) > + #:export (wisp)) > + > +; Set locale to something which supports unicode. Required to avoid using fluids. > +(catch #t > + (lambda () > + (setlocale LC_ALL "")) > + (lambda (key . parameters) > + (let ((locale-fallback "en_US.UTF-8")) > + (format (current-error-port) > + (string-join > + (list ";;; Warning: setlocale LC_ALL \"\" failed with ~A: ~A" > + "switching to explicit ~A locale. Please setup your locale." > + "If this fails, you might need glibc support for unicode locales.\n") > + "\n;;; ") > + key parameters locale-fallback) > + (catch #t > + (lambda () > + (setlocale LC_ALL locale-fallback)) > + (lambda (key . parameters) > + (format (current-error-port) > + (string-join > + (list ";;; Warning: fallback setlocale LC_ALL ~A failed with ~A: ~A" > + "Not switching to Unicode." > + "You might need glibc support for unicode locales.\n") > + "\n;;; ") > + locale-fallback key parameters)))))) > + > +;;; > +;;; Language definition > +;;; > + > +(define wisp-pending-sexps (list)) > + > +(define (read-one-wisp-sexp port env) > + ;; allow using "# foo" as #(foo). > + (read-hash-extend #\# (λ (chr port) #\#)) > + (cond > + ((eof-object? (peek-char port)) > + (read-char port )); return eof: we’re done > + (else > + (let ((chunk (wisp-scheme-read-chunk port))) > + (cond > + ((not (null? chunk)) > + (car chunk)) > + (else > + #f)))))) > + > +(define-language wisp > + #:title "Wisp Scheme Syntax. See SRFI-119 for details." > + ; . #:reader read-one-wisp-sexp > + #:reader read-one-wisp-sexp ; : lambda (port env) : let ((x (read-one-wisp-sexp port env))) (display x)(newline) x ; > + #:compilers `((tree-il . ,compile-tree-il)) > + #:decompilers `((tree-il . ,decompile-tree-il)) > + #:evaluator (lambda (x module) (primitive-eval x)) > + #:printer write ; TODO: backtransform to wisp? Use source-properties? > + #:make-default-environment > + (lambda () > + ;; Ideally we'd duplicate the whole module hierarchy so that `set!', > + ;; `fluid-set!', etc. don't have any effect in the current environment. > + (let ((m (make-fresh-user-module))) > + ;; Provide a separate `current-reader' fluid so that > + ;; compile-time changes to `current-reader' are > + ;; limited to the current compilation unit. > + (module-define! m 'current-reader (make-fluid)) > + ;; Default to `simple-format', as is the case until > + ;; (ice-9 format) is loaded. This allows > + ;; compile-time warnings to be emitted when using > + ;; unsupported options. > + (module-set! m 'format simple-format) > + m))) > + > + > + > diff --git a/test-suite/tests/srfi-119.test b/test-suite/tests/srfi-119.test > new file mode 100644 > index 000000000..a888df41d > --- /dev/null > +++ b/test-suite/tests/srfi-119.test > @@ -0,0 +1,81 @@ > +;;;; srfi-119.test --- Test suite for Guile's SRFI-119 reader. -*- scheme -*- > +;;;; > +;;;; Copyright (C) 2023 Free Software Foundation, Inc. > +;;;; > +;;;; This library is free software; you can redistribute it and/or > +;;;; modify it under the terms of the GNU Lesser General Public > +;;;; License as published by the Free Software Foundation; either > +;;;; version 3 of the License, or (at your option) any later version. > +;;;; > +;;;; This library is distributed in the hope that it will be useful, > +;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of > +;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > +;;;; Lesser General Public License for more details. > +;;;; > +;;;; You should have received a copy of the GNU Lesser General Public > +;;;; License along with this library; if not, write to the Free Software > +;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + > +(define-module (test-srfi-119) > + #:use-module (test-suite lib) > + #:use-module (srfi srfi-1) > + #:use-module (language wisp)) > + > +(define (read-string s) > + (with-input-from-string s read)) > + > +(define (with-read-options opts thunk) > + (let ((saved-options (read-options))) > + (dynamic-wind > + (lambda () > + (read-options opts)) > + thunk > + (lambda () > + (read-options saved-options))))) > + > +(define (wisp->list str) > + (wisp-scheme-read-string str)) > + > +(with-test-prefix "wisp-read-simple" > + (pass-if (equal? (wisp->list "<= n 5") '((<= n 5)))) > + (pass-if (equal? (wisp->list ". 5") '(5))) > + (pass-if (equal? (wisp->list "+ 1 : * 2 3") '((+ 1 (* 2 3)))))) > +(with-test-prefix "wisp-read-complex" > + (pass-if (equal? (wisp->list " > +a b c d e > + . f g h > + . i j k > + > +concat \"I want \" > + getwish from me > + . \" - \" username > +") '( > +(a b c d e > + f g h > + i j k) > + > +(concat "I want " > + (getwish from me) > + " - " username)))) > + > + (pass-if (equal? (wisp->list " > +define : a b c > +_ d e > +___ f > +___ g h > +__ . i > + > +define : _ > +_ display \"hello\n\" > + > +\\_") '( > +(define (a b c) > + (d e > + (f) > + (g h) > + i)) > + > +(define (_) > + (display "hello\n")) > + > +(_))))) > -- > 2.39.1 ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-09-30 13:17 ` Christine Lemmer-Webber @ 2023-09-30 20:09 ` Maxime Devos 2023-10-02 14:48 ` Christine Lemmer-Webber 0 siblings, 1 reply; 83+ messages in thread From: Maxime Devos @ 2023-09-30 20:09 UTC (permalink / raw) To: Christine Lemmer-Webber, Dr. Arne Babenhauserheide; +Cc: guile-devel [-- Attachment #1.1.1: Type: text/plain, Size: 885 bytes --] Op 30-09-2023 om 15:17 schreef Christine Lemmer-Webber: > Haven't fully caught up on this thread, but as a side note I have a > mostly-finished implementation of a Wisp parser which takes a very > different approach than Arne's, and was more understandable to me > personally, a bit more functional and recursive-descent style. > > I could make it available if anyone is curious. > > Would love to see Wisp in Guile proper! I think I technically can't count as ‘curious’ here, as I don't think I'll actually read this other implementation, but I would be interested in it nonetheless (to potentially replace Arne's implementation later, once it's complete), because it sounds like you might have avoided the REPR (*) bug thing from Arne's implementation. (*) Extremely unlikely to be a problem in practice, but still a bug. Best regards, Maxime Devos. [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 929 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH] add language/wisp to Guile? 2023-09-30 20:09 ` Maxime Devos @ 2023-10-02 14:48 ` Christine Lemmer-Webber 0 siblings, 0 replies; 83+ messages in thread From: Christine Lemmer-Webber @ 2023-10-02 14:48 UTC (permalink / raw) To: Maxime Devos; +Cc: Dr. Arne Babenhauserheide, guile-devel Maxime Devos <maximedevos@telenet.be> writes: > [[PGP Signed Part:Undecided]] > Op 30-09-2023 om 15:17 schreef Christine Lemmer-Webber: >> Haven't fully caught up on this thread, but as a side note I have a >> mostly-finished implementation of a Wisp parser which takes a very >> different approach than Arne's, and was more understandable to me >> personally, a bit more functional and recursive-descent style. >> I could make it available if anyone is curious. >> Would love to see Wisp in Guile proper! > > I think I technically can't count as ‘curious’ here, as I don't think > I'll actually read this other implementation, but I would be > interested in it nonetheless (to potentially replace Arne's > implementation later, once it's complete), because it sounds like you > might have avoided the REPR (*) bug thing from Arne's implementation. > > (*) Extremely unlikely to be a problem in practice, but still a bug. > > Best regards, > Maxime Devos. Well it looks like I did make it available already, I just forgot, and didn't advertise it much: https://gitlab.com/dustyweb/rewisp/-/blob/main/rewisp.scm?ref_type=heads It really isn't very complete, but here's some of the example data I was using: (define wisp-fac-noinfix-repeat "\ define : factorial n ; foo __ if : zero? n ____ . 1 ____ * n : factorial : - n 1 define : factorial n __ if : zero? n ____ . 1 ____ * n : factorial : - n 1") scheme@(rewisp)> ,pp (parse-lines->sexp (call-with-input-string wisp-fac-noinfix-repeat read-wisp-lines)) $6 = ((define (factorial n) (if (zero? n) 1 (* n (factorial (- n 1))))) (define (factorial n) (if (zero? n) 1 (* n (factorial (- n 1)))))) What's kind of interesting is to look at it before the parse-lines->sexp step though: scheme@(rewisp)> ,pp (call-with-input-string wisp-fac-noinfix-repeat read-wisp-lines) $7 = (#<<line> indent: 0 args: (#<syntax:unknown file:1:0 define> (#<syntax:unknown file:1:9 factorial> #<syntax:unknown file:1:19 n>))> #<<line> indent: 4 args: (#<syntax:unknown file:2:4 if> (#<syntax:unknown file:2:9 zero?> #<syntax:unknown file:2:15 n>))> #<<line> indent: 7 args: (#<<DOT> source-loc: #(#f 2 7)> #<syntax:unknown file:3:9 1>)> #<<line> indent: 7 args: (#<syntax:unknown file:4:7 *> #<syntax:unknown file:4:9 n> (#<syntax:unknown file:4:13 factorial> (#<syntax:unknown file:4:25 -> #<syntax:unknown file:4:27 n> #<syntax:unknown file:4:29 1>)))> #f #<<line> indent: 0 args: (#<syntax:unknown file:6:0 define> (#<syntax:unknown file:6:9 factorial> #<syntax:unknown file:6:19 n>))> #<<line> indent: 4 args: (#<syntax:unknown file:7:4 if> (#<syntax:unknown file:7:9 zero?> #<syntax:unknown file:7:15 n>))> #<<line> indent: 7 args: (#<<DOT> source-loc: #(#f 7 7)> #<syntax:unknown file:8:9 1>)> #<<line> indent: 7 args: (#<syntax:unknown file:9:7 *> #<syntax:unknown file:9:9 n> (#<syntax:unknown file:9:13 factorial> (#<syntax:unknown file:9:25 -> #<syntax:unknown file:9:27 n> #<syntax:unknown file:9:29 1>)))>) So what it does is rewisp builds up a set of lines and parses each line individually, noting its indentation level and the "arguments" that appear in it. It then has a separate step to assemble it into a sexp structure by examining the indentation level and whether or not it sees the special <DOT> record. Guile's syntax records are used so that this would be able to work correctly with the rest of Guile's tools. So, that's it... first, read each line one by one, don't try to figure out its relationship to the other lines, and we have a flat structure. Next, turn that flat structure into a nested structure. I thought the design was pretty good. - Christine ^ permalink raw reply [flat|nested] 83+ messages in thread
end of thread, other threads:[~2024-06-13 20:29 UTC | newest] Thread overview: 83+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-02-03 21:26 [PATCH] add language/wisp to Guile? Dr. Arne Babenhauserheide 2023-02-04 15:08 ` Maxime Devos 2023-02-04 15:46 ` Dr. Arne Babenhauserheide 2023-02-04 19:09 ` Maxime Devos 2023-02-04 21:35 ` Dr. Arne Babenhauserheide 2023-02-05 15:08 ` Maxime Devos 2023-02-14 8:32 ` Dr. Arne Babenhauserheide 2023-02-14 21:24 ` Dr. Arne Babenhauserheide 2023-02-14 23:01 ` Maxime Devos 2023-02-15 1:46 ` Matt Wette 2023-02-16 21:38 ` Dr. Arne Babenhauserheide 2023-02-17 1:26 ` Matt Wette 2023-02-23 11:36 ` Ludovic Courtès 2023-02-23 17:48 ` Dr. Arne Babenhauserheide 2023-02-23 18:42 ` Maxime Devos 2023-02-24 15:45 ` Ludovic Courtès 2023-02-24 16:34 ` Dr. Arne Babenhauserheide 2023-03-08 10:34 ` Dr. Arne Babenhauserheide 2023-05-01 9:54 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Dr. Arne Babenhauserheide 2023-06-10 16:40 ` Ludovic Courtès 2023-06-12 10:22 ` Maxime Devos 2023-08-10 6:28 ` Dr. Arne Babenhauserheide 2023-08-14 20:11 ` Dr. Arne Babenhauserheide 2023-08-14 20:30 ` Dr. Arne Babenhauserheide 2023-08-14 22:43 ` Dr. Arne Babenhauserheide 2023-08-18 10:29 ` Ludovic Courtès 2023-08-18 12:16 ` Dr. Arne Babenhauserheide 2023-08-18 17:50 ` Dr. Arne Babenhauserheide 2023-09-08 17:46 ` Dr. Arne Babenhauserheide 2023-10-05 14:10 ` Dr. Arne Babenhauserheide 2023-10-10 23:04 ` Dr. Arne Babenhauserheide 2023-10-27 22:05 ` Dr. Arne Babenhauserheide 2024-01-09 7:05 ` Dr. Arne Babenhauserheide 2024-01-19 8:21 ` Dr. Arne Babenhauserheide 2024-03-11 1:16 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more tests, squashed) Dr. Arne Babenhauserheide 2024-06-01 9:57 ` Ludovic Courtès 2024-06-01 15:06 ` Dr. Arne Babenhauserheide 2024-06-07 5:44 ` Damien Mattei 2024-06-10 8:13 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch withmore " Maxime Devos 2024-06-10 23:03 ` Damien Mattei 2024-06-13 20:29 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch with more " Dr. Arne Babenhauserheide 2024-01-19 12:10 ` [PATCH] add SRFI-119 / language/wisp to Guile? (new patch, squashed) Christina O'Donnell 2024-01-19 21:37 ` Ricardo Wurmus 2024-01-19 21:47 ` Christina O'Donnell 2024-01-20 11:01 ` Damien Mattei 2024-01-20 19:18 ` Dr. Arne Babenhauserheide 2024-01-20 22:59 ` Damien Mattei 2024-01-20 23:22 ` Dr. Arne Babenhauserheide 2024-01-21 23:21 ` Damien Mattei 2024-01-19 23:56 ` Dr. Arne Babenhauserheide 2023-02-24 23:48 ` [PATCH] add language/wisp to Guile? Maxime Devos 2023-02-24 23:51 ` Maxime Devos 2023-02-25 0:15 ` Matt Wette 2023-02-25 10:42 ` Maxime Devos 2023-02-17 23:06 ` Maxime Devos 2023-02-18 3:50 ` Philip McGrath 2023-02-18 15:58 ` Maxime Devos 2023-02-18 19:56 ` Matt Wette 2023-02-21 12:09 ` Dr. Arne Babenhauserheide 2023-02-26 7:45 ` Philip McGrath 2023-02-26 15:42 ` Maxime Devos 2023-02-26 16:14 ` Dr. Arne Babenhauserheide 2023-02-26 17:58 ` Matt Wette 2023-02-26 18:03 ` Dr. Arne Babenhauserheide 2023-02-26 18:20 ` Matt Wette 2023-02-26 21:39 ` Dr. Arne Babenhauserheide 2023-10-02 14:59 ` Christine Lemmer-Webber 2023-10-02 21:46 ` guile support for multiple languages [was: [PATCH] add language/wisp to Guile?] Matt Wette 2023-02-23 7:59 ` [PATCH] add language/wisp to Guile? Maxime Devos 2023-02-23 8:51 ` Dr. Arne Babenhauserheide 2023-02-23 18:04 ` Maxime Devos 2023-02-23 18:22 ` Maxime Devos 2023-02-23 18:36 ` Maxime Devos 2023-02-23 18:37 ` Maxime Devos 2023-02-15 8:36 ` Dr. Arne Babenhauserheide 2023-02-15 20:13 ` Maxime Devos 2023-02-16 7:01 ` Dr. Arne Babenhauserheide 2023-02-16 8:03 ` Dr. Arne Babenhauserheide 2023-02-16 11:30 ` Maxime Devos 2023-02-16 21:35 ` Dr. Arne Babenhauserheide 2023-09-30 13:17 ` Christine Lemmer-Webber 2023-09-30 20:09 ` Maxime Devos 2023-10-02 14:48 ` Christine Lemmer-Webber
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).