Christmas wish: Literate Elisp

Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 7643 bytes --]

Hello,

I have a question/proposal about Elisp and literate programming.
Well, more of a proposal then a question, but here it is:

From Wikipedia: "Literate programming is a programming paradigm
introduced by Donald Knuth in which a computer program is given
an explanation of its logic in a natural language, such as English,
interspersed with snippets of macros and traditional source code,
from which compilable source code can be generated."

Emacs already supports a form of literal programming in form of
org mode and babel where we can insert code for programming
languages in-between #+BEGIN_SRC and #+END_SRC markers,
which is super nice and cool feature.

However I got a thought that LISPs (lisp-like languages), have natural
code markers, since all code is enclosed with parenthesis. Thus one
could see '(' and ')' as code markers in literate-programming style,
more of as Knuth proposed. In other words, LISP (or at least Elisp)
does not need special markers to denote start and end of code. Unlike
Haskell, there is no need to use '\begin_code' or '>' to differentiate
code from text (comments).

My proposal is to slightly change Elisp parser to treat lines that start
with any other printable character but '(' as a start of comment and to
simply ignore the line, just as it treats ';' as a comment. Code blocks
would still be parsed as they are now, and ';' would still mean a comment,
wherever it is encountered, it is just that anything that does not
belong in a code-block (lists) is a comment. For example consider this mail,
if this would be thrown into parser all lines to this point would be simply
ignored since they don't start with '(' or are white spaces.

(my-elisp-fun
        (progn
                (while (very-cool)
                       (do-something-very-literate-here))))

Then I could have Elisp code in between and continue to write this mail
and later on just use this as a source code. Wouldn't that be a step
toward true and more cool literate programming language? Below is another
snippet of imaginary Elisp. If we could do this, then this email would be
a working literate Elisp, where those two snippets are code and text of
this mail is just ignored.

(my-other-fun
        (progn
                ; this-is-some-other-fun
                (to-hopefully-demonstrate-the-idea)))

What would this achieve

More then highly increased coolness factor, it would be a small quality
of life improvement. For example this would actually make it slightly
easier to use org-mode for Elisp programming. For example for us that
use org-mode to structure Emacs init file, we could just throw in our org
file directly, instead of using babel to entangle it into Elisp file first.
If every printable character but '(' starts a comment line, then everything
in org-file but Elisp code would be simply ignored, and only Elisp executed.

If we think other way around, it would also let us use pure Elisp for literate
programming without org-mode whatsoever, albeit it would be a cool feature to
use org-headings and similar to structure the code. It might make code more
structured and thus more readable. When I think in terms of Elisp as a starting
point rather then in terms of org-mode as a starting point, that could result in
adding org-mode organizational features directly to Elisp. One could even mark
say not-implemented functions as todo items, use calendar etc.

We could also entangle other languages within pure Elisp code without using org
mode whatsoever. Either within some code markers for processing them out to
separate files, or without code markers just as documentation or whatever. I
don't have some better example of use-case at the moment.

I don't mean that it is incredibly slow to entangle files, but it would be
slightly more efficient to process Elisp entangled in org mode. I also don't
think it is hard to type ';' at the beginning of a line to start a comment line.
But it is a small convenience and thus quality of life improvement that probably
does not need much changes to a parser but has quite a dramatic effect on how
source code looks in human eye (at least mine, if you don't mind that I count
myself as a part of the species :-)). It would let us use org-mode as a standard
Elisp source code format, which might be just a perceived convenience rather
then some real extra useful thing that does not exist yet.

Some thoughts about implementation

I think that in terms of cost effectiveness with implementation in mind, it
probably isn't that much work to implement this, but honestly I have no idea.
I believe it can't be much work, but I am not sure so I should really put an
exclamation mark to word probably in paragraph above. Feel free to educate me
about cost of making it work. I was looking myself in C source to see if I could
test this myself before I post here, but I couldn't find where you have implemented
parser. I am sorry, but I am that bad :-(.

Essentially when parsing literate Elisp, if I may call it so, what
parser has to do is to simply not flag random printable characters, on lines
that does not belong to code-blocks, as errors in source code. Instead just treat
them as if it has seen a ';'.

It means there are just two classes of printable characters: '(' that opens
a code block, and everything else that opens a comment block. Well almost.
Parsing code blocks would not need to be changed at all, and ';' in code blocks
would still mean that rest of line is a comment, and all code with comments
would still continue to work as it does now. It would only affect new code that
is written in this style. However new Elisp code wouldn't be backward
compatible with old versions of Emacs.

As extra, one could keep current Elisp parser and make new one and use as
in Haskell, en extra 'l' in suffix to denote a literate program, '.lel'. Though
it kind-a looks fun, I don't think it wouldn't be needed, I don't think this
change would need different parser, to ensure backward compatibility. I don't
think that is very important since if we would write Elisp that needs to run on
older versions, we can just let be to write it in literal form.

Drawbacks

As I can think of, it would maybe make spotting errors slightly harder, for
example I could type a random character before an opening parenthesis and
comment out entire line, but those kind of errors are easily spotted on first code
run. Another drawback would be probably syntax highlighting. It could probably
become much harder to detect comments in code since there is no ';' to mark a
comment-line. Maybe I am wrong about this one, it is just a fast thought.

Final thought

I have no idea if somebody else has already thought about this and found that it
can't work. It seems like a very straight-forward and simple thing so I am
probably not the first one to think the thought, and there is probably some
reason why it is not done that I am not aware of. In that case just ignore this
email. It was just a thought I got yesterday. There might be something I am not
aware of that makes this impossible, I am not that familiar with Emacs source to
try to implement this myself unfortunately. It is just an idea, a thought for
discussion, but it would be cool if it can work.

I am sorry for very long email, I hope you have at least somewhat enjoyed my
rather ramblings and Christmas wishes, and please ask Santa to excuse my
English, I am not native English speaker, it is just my 3rd language if it is in
any defense or my horrible writing.


[-- Attachment #2: Type: text/html, Size: 11030 bytes --]

Re: Christmas wish: Literate Elisp

[Stefan Monnier]

> My proposal is to slightly change Elisp parser to treat lines that start
> with any other printable character but '(' as a start of comment and to
> simply ignore the line, just as it treats ';' as a comment.

The `sexpresso` Haskell package follows the same idea ;-)

As for using it in Elisp: I don't think there's anything stopping anyone
from making such a `literate-elisp-mode` and even arrange for `load` to
handle such a file (just like there is already a package that lets
`load` work directly on .org files).

I'd welcome such a package in GNU ELPA.


        Stefan

Re: Christmas wish: Literate Elisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The idea sounds basically plausible, but note that simply rejecting
all lines that start with anything other than ( plus whitespace
will give the wrong results.  Lisp code includes lines that start
with other things.  Look at any file of Lisp code and you will see
many that start with letters, digits or quotes.  The proposal
needs to be corrected for this.

But I think that correction won't be difficult.

Indented comments should be recognized and formatted differently
but not exactly the same as interpolated text.

Perhaps triple-semicolon should be the marker for interpolated text.
That way, it would not require any change in Lisp parsing or in Lisp Mode.

-- 
Dr Richard Stallman
Founder, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

To be honest my contact with Haskell broke once the university course, some 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy packages. I just recalled that it was possible to invert text and code in Haskell. But cool. Didn't know there is a package to let load org files directly either. What I do in init is use org-babel to entangle my init file (org-babel-load-file (expand-file-name "~/.emacs.d/lisp/init.org")). What is the name of the package? I have tried to google it and looked into elpa packages with list-package but I don't see it. I am just curious how is it done, as I was thinking or they refactor the code as in org-babel into corresponding lisp file.

Thanks for the answers.
________________________________
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Skickat: den 12 december 2019 18:29
Till: arthur miller <arthur.miller@live.com>
Kopia: emacs-devel <emacs-devel@gnu.org>; archambv@iro.umontreal.ca <archambv@iro.umontreal.ca>
Ämne: Re: Christmas wish: Literate Elisp

> My proposal is to slightly change Elisp parser to treat lines that start
> with any other printable character but '(' as a start of comment and to
> simply ignore the line, just as it treats ';' as a comment.

The `sexpresso` Haskell package follows the same idea ;-)

As for using it in Elisp: I don't think there's anything stopping anyone
from making such a `literate-elisp-mode` and even arrange for `load` to
handle such a file (just like there is already a package that lets
`load` work directly on .org files).

I'd welcome such a package in GNU ELPA.


        Stefan


[-- Attachment #2: Type: text/html, Size: 2520 bytes --]

Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2835 bytes --]

"The idea sounds basically plausible, but note that simply rejecting
all lines that start with anything other than ( plus whitespace
will give the wrong results.  Lisp code includes lines that start
with other things.  Look at any file of Lisp code and you will see
many that start with letters, digits or quotes.  The proposal
needs to be corrected for this."

To be honest I am not sure if I am a bit not understood here, or just
me not being knowledgable enough about lisp (could be both 🙂). I
will try to clarify.

What I propose is to treat lines that does not belong to a code block ().
I don't mean every line that does not start with '('. there can be many
lines in a code block between an '(' and final matching ')'. Once '(' is
encountered, parsing should be just as it is now, without difference.
';' would still be the comment indicator, so no change needed there.
Just for the text that does not belong to any matching pair of (), so to
say in-between "block final"  ')' and "block opening"  '('. I am sorry if
my terminology is not very lisp-like, I am used to think more in terms
of a C compiler.

I am not super-versatile with elisp to be honest, so I will have to
ask: is it possible to have elisp that isn't in-between '(' and ')'.
Is it possible to have symbols or literals in elisp outside ()? I
am sorry if I am total noob, but I am not used to see those in
sources, is it just rare to have them, or do I missunderand? I am
looking around in some elisp but I can't find any.

________________________________
Från: Richard Stallman <rms@gnu.org>
Skickat: den 14 december 2019 05:16
Till: arthur miller <arthur.miller@live.com>
Kopia: emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: Christmas wish: Literate Elisp

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The idea sounds basically plausible, but note that simply rejecting
all lines that start with anything other than ( plus whitespace
will give the wrong results.  Lisp code includes lines that start
with other things.  Look at any file of Lisp code and you will see
many that start with letters, digits or quotes.  The proposal
needs to be corrected for this.

But I think that correction won't be difficult.

Indented comments should be recognized and formatted differently
but not exactly the same as interpolated text.

Perhaps triple-semicolon should be the marker for interpolated text.
That way, it would not require any change in Lisp parsing or in Lisp Mode.

--
Dr Richard Stallman
Founder, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



[-- Attachment #2: Type: text/html, Size: 8042 bytes --]

Re: Sv: Christmas wish: Literate Elisp

[Stefan Monnier]

arthur miller [2019-12-14 04:40:30] wrote:

> To be honest my contact with Haskell broke once the university course, some
> 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy
> packages.  I just recalled that it was possible to invert text and code in
> Haskell.  But cool.

The fact that it's written is Haskell is just incidental.

> Didn't know there is a package to let load org files directly either.

Hmm... The closest I can find is https://github.com/jingtaozf/literate-elisp/
but it doesn't quite match what I think I was referring to.


        Stefan


> Thanks for the answers.
> ________________________________
> Från: Stefan Monnier <monnier@iro.umontreal.ca>
> Skickat: den 12 december 2019 18:29
> Till: arthur miller <arthur.miller@live.com>
> Kopia: emacs-devel <emacs-devel@gnu.org>; archambv@iro.umontreal.ca <archambv@iro.umontreal.ca>
> Ämne: Re: Christmas wish: Literate Elisp
>
>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.
>
>
>         Stefan

RE: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2408 bytes --]

Thanks, I have also found literste-li when I searched, but thought it didn't looked  as suggested .

I have played a bit yesterday,  for about an hour and managed to find readevalloop in lread.c in sources. I managed to get a partial hack to work in some cases, but in other I get parse errors. I am not sure I understand exactly how the parser hangs together yet, if I can do everything in that function or I have to change some of other functions too. I will have more time tonight evening.



Skickat från min Samsung Galaxy-smartphone.



-------- Originalmeddelande --------
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Datum: 2019-12-14 15:08 (GMT+01:00)
Till: arthur miller <arthur.miller@live.com>
Kopia: emacs-devel <emacs-devel@gnu.org>, archambv@iro.umontreal.ca
Ämne: Re: Sv: Christmas wish: Literate Elisp

arthur miller [2019-12-14 04:40:30] wrote:

> To be honest my contact with Haskell broke once the university course, some
> 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy
> packages.  I just recalled that it was possible to invert text and code in
> Haskell.  But cool.

The fact that it's written is Haskell is just incidental.

> Didn't know there is a package to let load org files directly either.

Hmm... The closest I can find is https://github.com/jingtaozf/literate-elisp/
but it doesn't quite match what I think I was referring to.


        Stefan


> Thanks for the answers.
> ________________________________
> Från: Stefan Monnier <monnier@iro.umontreal.ca>
> Skickat: den 12 december 2019 18:29
> Till: arthur miller <arthur.miller@live.com>
> Kopia: emacs-devel <emacs-devel@gnu.org>; archambv@iro.umontreal.ca <archambv@iro.umontreal.ca>
> Ämne: Re: Christmas wish: Literate Elisp
>
>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.
>
>
>         Stefan


[-- Attachment #2: Type: text/html, Size: 3734 bytes --]

Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1.1: Type: text/plain, Size: 2782 bytes --]

Hello again,

I was able to make readevalloop do what I want. It was rather trivial to
implement this (once I realized how it works 🙂). Emacs built fine and I was
able to test wtih eval-buffer and eval-regionwhich worked as intended.

It was just 4 code lines, I just copied codeto parse comment and change the
condition in if-statement:

(message "Hello, World!")

(message "Hello Again!")

In code blocs the ';' is still a comment delimiter:

(message
 ;; Here is a line comment
 "I am a bit chatty today!")

That's it for today folks!

(message "Bye bye cruel world!")

This email was composed in scratch buffer and executed during writing with
eval-buffer for testing.

However, byte compiler is not happy about this. It emits warnings
for every word out of code blocks as references to free variables. I am not sure
where to look to patch it, maybe another day.
________________________________
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Skickat: den 14 december 2019 15:08
Till: arthur miller <arthur.miller@live.com>
Kopia: emacs-devel <emacs-devel@gnu.org>; archambv@iro.umontreal.ca <archambv@iro.umontreal.ca>
Ämne: Re: Sv: Christmas wish: Literate Elisp

arthur miller [2019-12-14 04:40:30] wrote:

> To be honest my contact with Haskell broke once the university course, some
> 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy
> packages.  I just recalled that it was possible to invert text and code in
> Haskell.  But cool.

The fact that it's written is Haskell is just incidental.

> Didn't know there is a package to let load org files directly either.

Hmm... The closest I can find is https://github.com/jingtaozf/literate-elisp/
but it doesn't quite match what I think I was referring to.


        Stefan


> Thanks for the answers.
> ________________________________
> Från: Stefan Monnier <monnier@iro.umontreal.ca>
> Skickat: den 12 december 2019 18:29
> Till: arthur miller <arthur.miller@live.com>
> Kopia: emacs-devel <emacs-devel@gnu.org>; archambv@iro.umontreal.ca <archambv@iro.umontreal.ca>
> Ämne: Re: Christmas wish: Literate Elisp
>
>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.
>
>
>         Stefan


[-- Attachment #1.2: Type: text/html, Size: 4373 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: lread.patch --]
[-- Type: text/x-patch; name="lread.patch", Size: 389 bytes --]

--- lread.c	2019-12-16 12:52:05.739490741 +0100
+++ mylread.c	2019-12-16 12:50:46.572077138 +0100
@@ -2053,6 +2053,12 @@
 	  || c == NO_BREAK_SPACE)
 	goto read_next;
 
+      if (c != '(' && c != '#')
+	{
+	  while ((c = READCHAR) != '\n' && c != -1);
+	  goto read_next;
+	}
+
       if (! HASH_TABLE_P (read_objects_map)
 	  || XHASH_TABLE (read_objects_map)->count)
 	read_objects_map

Re: Sv: Sv: Christmas wish: Literate Elisp

[Stefan Monnier]

>> As for using it in Elisp: I don't think there's anything stopping anyone
>> from making such a `literate-elisp-mode` and even arrange for `load` to
>> handle such a file (just like there is already a package that lets
>> `load` work directly on .org files).
> I was able to make readevalloop do what I want. It was rather trivial to

Note that in my comment above I meant it in existing Emacsen,
i.e. without any modification to the C code.
The idea is to make use of existing hooks such as
`load-source-file-function` or `file-name-handler-alist`.

I'm not sure what hook could be used to do the same with
byte-compilation, but in the worst case, an advice should do the trick.


        Stefan

Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]

Aha 😊

Sorry, I didn’t realized what you ment. I am not sure I know that much of elisp,
but I may try to look at, would be lots of learning for me 😊.

Skickades från E-post<https://go.microsoft.com/fwlink/?LinkId=550986> för Windows 10

Från: Stefan Monnier<mailto:monnier@iro.umontreal.ca>
Skickat: den 16 december 2019 14:41
Till: arthur miller<mailto:arthur.miller@live.com>
Kopia: emacs-devel<mailto:emacs-devel@gnu.org>; archambv@iro.umontreal.ca<mailto:archambv@iro.umontreal.ca>
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

>> As for using it in Elisp: I don't think there's anything stopping anyone
>> from making such a `literate-elisp-mode` and even arrange for `load` to
>> handle such a file (just like there is already a package that lets
>> `load` work directly on .org files).
> I was able to make readevalloop do what I want. It was rather trivial to

Note that in my comment above I meant it in existing Emacsen,
i.e. without any modification to the C code.
The idea is to make use of existing hooks such as
`load-source-file-function` or `file-name-handler-alist`.

I'm not sure what hook could be used to do the same with
byte-compilation, but in the worst case, an advice should do the trick.


        Stefan


[-- Attachment #2: Type: text/html, Size: 3263 bytes --]

Re: Sv: Sv: Christmas wish: Literate Elisp

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

Arthur,

Have you happened to have looked at Outshine (
https://github.com/alphapapa/outshine)?

It enhances outline-minor-mode and lets it work with standard org-syntax
while in comment blocks (and defines hotkeys to swap into an edit buffer
that makes it look like a normal Org-Mode buffer).

By keeping the literate syntax in comments you don't break the existing
parser.

Regards,
Jon

On Mon, Dec 16, 2019 at 9:03 AM arthur miller <arthur.miller@live.com>
wrote:

> Aha 😊
>
> Sorry, I didn’t realized what you ment. I am not sure I know that much of
> elisp,
>
> but I may try to look at, would be lots of learning for me 😊.
>
>
>
> Skickades från E-post <https://go.microsoft.com/fwlink/?LinkId=550986>
> för Windows 10
>
>
>
> *Från: *Stefan Monnier <monnier@iro.umontreal.ca>
> *Skickat: *den 16 december 2019 14:41
> *Till: *arthur miller <arthur.miller@live.com>
> *Kopia: *emacs-devel <emacs-devel@gnu.org>; archambv@iro.umontreal.ca
> *Ämne: *Re: Sv: Sv: Christmas wish: Literate Elisp
>
>
>
> >> As for using it in Elisp: I don't think there's anything stopping anyone
> >> from making such a `literate-elisp-mode` and even arrange for `load` to
> >> handle such a file (just like there is already a package that lets
> >> `load` work directly on .org files).
> > I was able to make readevalloop do what I want. It was rather trivial to
>
> Note that in my comment above I meant it in existing Emacsen,
> i.e. without any modification to the C code.
> The idea is to make use of existing hooks such as
> `load-source-file-function` or `file-name-handler-alist`.
>
> I'm not sure what hook could be used to do the same with
> byte-compilation, but in the worst case, an advice should do the trick.
>
>
>         Stefan
>
>
>

[-- Attachment #2: Type: text/html, Size: 3294 bytes --]

RE: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 3097 bytes --]

Of course, I am aware of outline modes, but my proposal is more about literate programming then outlining. Being able to use org outlining in elisp is just an extra icing on the cake that comes out implicitly as a bi product for free. Otherwise of course one can use outline code for folding.

It is true, that keeping directives in comments does not break the existing parser, but then it is not so much of literate programming, is it?

If we would just use directives in comments then we already have org + babel. But I think 4 extra lines of C code to enable this feature in eval-loop was a trivial price to pay :-).

Now I just wonder how to change the bytecode compiler. Unfortunately it seems to be much more involved process. I don't know  if I can do it, since I am  not so acquainted with Emacs internals, so it would be cool if someone more knowledgeable can at least hint me if not help.


Skickat från min Samsung Galaxy-smartphone.



-------- Originalmeddelande --------
Från: Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com>
Datum: 2019-12-16 17:07 (GMT+01:00)
Till: arthur miller <arthur.miller@live.com>
Kopia: Stefan Monnier <monnier@iro.umontreal.ca>, archambv@iro.umontreal.ca, emacs-devel <emacs-devel@gnu.org>
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

Arthur,

Have you happened to have looked at Outshine (https://github.com/alphapapa/outshine)?

It enhances outline-minor-mode and lets it work with standard org-syntax while in comment blocks (and defines hotkeys to swap into an edit buffer that makes it look like a normal Org-Mode buffer).

By keeping the literate syntax in comments you don't break the existing parser.

Regards,
Jon

On Mon, Dec 16, 2019 at 9:03 AM arthur miller <arthur.miller@live.com<mailto:arthur.miller@live.com>> wrote:
Aha 😊

Sorry, I didn’t realized what you ment. I am not sure I know that much of elisp,
but I may try to look at, would be lots of learning for me 😊.

Skickades från E-post<https://go.microsoft.com/fwlink/?LinkId=550986> för Windows 10

Från: Stefan Monnier<mailto:monnier@iro.umontreal.ca>
Skickat: den 16 december 2019 14:41
Till: arthur miller<mailto:arthur.miller@live.com>
Kopia: emacs-devel<mailto:emacs-devel@gnu.org>; archambv@iro.umontreal.ca<mailto:archambv@iro.umontreal.ca>
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

>> As for using it in Elisp: I don't think there's anything stopping anyone
>> from making such a `literate-elisp-mode` and even arrange for `load` to
>> handle such a file (just like there is already a package that lets
>> `load` work directly on .org files).
> I was able to make readevalloop do what I want. It was rather trivial to

Note that in my comment above I meant it in existing Emacsen,
i.e. without any modification to the C code.
The idea is to make use of existing hooks such as
`load-source-file-function` or `file-name-handler-alist`.

I'm not sure what hook could be used to do the same with
byte-compilation, but in the worst case, an advice should do the trick.


        Stefan


[-- Attachment #2: Type: text/html, Size: 5392 bytes --]

Re: Sv: Sv: Christmas wish: Literate Elisp

[Adam Porter]

arthur miller <arthur.miller@live.com> writes:

> It is true, that keeping directives in comments does not break the
> existing parser, but then it is not so much of literate programming,
> is it?

Literate programming is not a matter of syntax.  For example, this
summary in the Wikipedia article seems to describe it well:

  Literate programming is a programming paradigm introduced by Donald
  Knuth in which a computer program is given an explanation of its logic
  in a natural language, such as English, interspersed with snippets of
  macros and traditional source code, from which compilable source code
  can be generated.

  The literate programming paradigm, as conceived by Knuth, represents a
  move away from writing computer programs in the manner and order
  imposed by the computer, and instead enables programmers to develop
  programs in the order demanded by the logic and flow of their
  thoughts. Literate programs are written as an uninterrupted exposition
  of logic in an ordinary human language, much like the text of an
  essay, in which macros are included to hide abstractions and
  traditional source code.

  Literate programming (LP) tools are used to obtain two representations
  from a literate source file: one suitable for further compilation or
  execution by a computer, the "tangled" code, and another for viewing
  as formatted documentation, which is said to be "woven" from the
  literate source. While the first generation of literate programming
  tools were computer language-specific, the later ones are
  language-agnostic and exist above the programming languages.

None of that requires un-prefixed comment syntax.  Consider examples of
actual, full-scale literate programming projects, which are written in a
variety of languages and source code formats.

> If we would just use directives in comments then we already have org +
> babel. But I think 4 extra lines of C code to enable this feature in
> eval-loop was a trivial price to pay :-).

Changing the canonical parser of a widely used language would have
effects reaching far beyond the parser's native software.  Consider all
other software which parses Elisp, e.g. syntax highlighting tools
outside of Emacs, which would not correctly highlight these un-prefixed
comments you propose.  As well, consider other implementations, like
Guile's, which would also have to be adjusted.  Then consider all the
code linting tools which would likely need fixing.  It's not merely a
matter of 4 lines of code in an Emacs source file.

As has been mentioned, benefits such as outlining are easily achieved
with existing tools, some of which are even built-in to Emacs
(e.g. outline-minor-mode for simple outlining in Elisp files, and
org-mode for prose-first, noweb-style source files).

Are you really so concerned about omitting a semicolon at the beginning
of top-level comment lines that you want to change the Elisp parser?

Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 5997 bytes --]

Hello, sorry for a bit late answer, but I was very busy Monday/Tuesday.

To make this more readable I will try to summerize here as one-piece text
for easier reading.

As you mention yourself from,Wikipedia, literate programming is about moving
away from programming for computers to programming for humans. Using comments
to comment-away text ment for humans or to mark lines between comments and
code is indeed writing for the computer, not the human. Humans tend to be
distracted by unnecessary clutter, and managing stuff just for the sake of
machine does take a little more effort then not doing it.

Being programming-language agnostic though is probably not possible
without having some kind of special tool, or special syntax due to some
languages being probably very hard to distinguish from ordinary text. For
example I believe that C/C++ or even Bash or Makefiles are very hard if
not impossible to process that way without some very complicated parser. I
can also imagine case where Elisp is impossible to distinguish from text,
(for example this line is valid text but would fail in parser), or whatever
we would put into a parenthesis on a separate line that does not belong to code.

Anyway, due to elisp looks, we have those "natural markers", ( and ) which
we can use as delimiters between code and text.

For the question of being concerned about omitting ';' to change the parser.
Well yes :-). Small syntactic sugar change but it reduces that visual clutter.
It is one step in that direction of writing code for humans and not for the
computer. See it as experiment. I thought it would be a cool thing to have. It
is just an idea. The change is not in an intrusive way. All your elisp is
still very same. It just adds an extra feature pretty much for free. If it would
be an intrusive change that requires your old elisp to change, than I wouldn't
suggest it. By the way, one could also add a variable to turn off that feature.

As for the tools, since old elisp code does not change, all your tools would
still continue to work on all your ordinary elisp, and you could still use ';'
just as you do now. What would change is that you would be able to write some
elisp as you can't now, and only on that code the tools would probably get
confused. I have no idea how hard would it be to change font-locking and
indentation checkers, maybe hard, maybe not at all. I can't answer that since I
am not acquianted with how those tools are implemented (lots of regex as I have
heard). But in worst case, you can always fall back on writing ordinary elisp.

As a note, one will still have to use ';' in directives, like ;;;###autoload
or ;;; -*- lexical-binding: t; -*- . Otherwise it would need lots more work
to get it to work, which I am not sure is worth.

Från: Adam Porter<mailto:adam@alphapapa.net>
Skickat: den 17 december 2019 12:07
Till: emacs-devel@gnu.org<mailto:emacs-devel@gnu.org>
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

arthur miller <arthur.miller@live.com> writes:

> It is true, that keeping directives in comments does not break the
> existing parser, but then it is not so much of literate programming,
> is it?

Literate programming is not a matter of syntax.  For example, this
summary in the Wikipedia article seems to describe it well:

  Literate programming is a programming paradigm introduced by Donald
  Knuth in which a computer program is given an explanation of its logic
  in a natural language, such as English, interspersed with snippets of
  macros and traditional source code, from which compilable source code
  can be generated.

  The literate programming paradigm, as conceived by Knuth, represents a
  move away from writing computer programs in the manner and order
  imposed by the computer, and instead enables programmers to develop
  programs in the order demanded by the logic and flow of their
  thoughts. Literate programs are written as an uninterrupted exposition
  of logic in an ordinary human language, much like the text of an
  essay, in which macros are included to hide abstractions and
  traditional source code.

  Literate programming (LP) tools are used to obtain two representations
  from a literate source file: one suitable for further compilation or
  execution by a computer, the "tangled" code, and another for viewing
  as formatted documentation, which is said to be "woven" from the
  literate source. While the first generation of literate programming
  tools were computer language-specific, the later ones are
  language-agnostic and exist above the programming languages.

None of that requires un-prefixed comment syntax.  Consider examples of
actual, full-scale literate programming projects, which are written in a
variety of languages and source code formats.

> If we would just use directives in comments then we already have org +
> babel. But I think 4 extra lines of C code to enable this feature in
> eval-loop was a trivial price to pay :-).

Changing the canonical parser of a widely used language would have
effects reaching far beyond the parser's native software.  Consider all
other software which parses Elisp, e.g. syntax highlighting tools
outside of Emacs, which would not correctly highlight these un-prefixed
comments you propose.  As well, consider other implementations, like
Guile's, which would also have to be adjusted.  Then consider all the
code linting tools which would likely need fixing.  It's not merely a
matter of 4 lines of code in an Emacs source file.

As has been mentioned, benefits such as outlining are easily achieved
with existing tools, some of which are even built-in to Emacs
(e.g. outline-minor-mode for simple outlining in Elisp files, and
org-mode for prose-first, noweb-style source files).

Are you really so concerned about omitting a semicolon at the beginning
of top-level comment lines that you want to change the Elisp parser?



[-- Attachment #2: Type: text/html, Size: 9182 bytes --]

Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

[Adam Porter]

arthur miller <arthur.miller@live.com> writes:

> As you mention yourself from,Wikipedia, literate programming is about moving
> away from programming for computers to programming for humans. Using comments
> to comment-away text ment for humans or to mark lines between comments and
> code is indeed writing for the computer, not the human. Humans tend to be
> distracted by unnecessary clutter, and managing stuff just for the sake of
> machine does take a little more effort then not doing it.

; I am not more distracted while reading this line of text
than I am while reading this one.

On the contrary, the semicolon prefix (and fontification according to my
preferences in Elisp buffers) makes it easy for me to see what is code
and what is prose.

> I can also imagine case where Elisp is impossible to distinguish from text,
> (for example this line is valid text but would fail in parser), or whatever
> we would put into a parenthesis on a separate line that does not
> belong to code.

That line beginning with a paren parses as valid Elisp.  Whether it
evaluates depends on the context.

> Anyway, due to elisp looks, we have those "natural markers", ( and ) which
> we can use as delimiters between code and text.

This is the problem: as you have shown, parentheses commonly appear in
prose, even at the beginning of lines.  Punctuation, such as semicolons,
do not
; or, at least, should not.

> For the question of being concerned about omitting ';' to change the parser.
> Well yes :-). Small syntactic sugar change but it reduces that visual
> clutter.

You call it clutter; I call it clearly delineating code and prose.  What
would look cluttered to me would be prose (which may include commented-out
code) interspersed with
(insert "code, which may include or generate prose.")

> It is one step in that direction of writing code for humans and not for the
> computer.

Until we can write programs in human language, we must write both.  For
humans' sake, the two need to be clearly separated.  If you don't like
comment prefix syntax, you can use Org files and their source blocks, so
your prose and your code can both start at column 0.  You can even
choose a face for org-meta-line that makes #+BEGIN_SRC lines virtually
(or literally!) disappear.

> See it as experiment. I thought it would be a cool thing to have. It
> is just an idea. The change is not in an intrusive way. All your elisp
> is still very same.  It just adds an extra feature pretty much for
> free. If it would be an intrusive change that requires your old elisp
> to change, than I wouldn't suggest it. By the way, one could also add
> a variable to turn off that feature.

No, in fact, some of my Elisp would have to change, because it would no
longer be sufficient to look at comment prefixes to know whether a line
is code or comment.  Such a change would be very expensive in terms of
the necessary human work that would ripple outward.  And your proposed
variable would not obviate the need to account for both states in my
code.

And since parens can appear at the start of lines in prose, how would
anyone but a human know whether a line is code or prose?

> As for the tools, since old elisp code does not change, all your tools
> would still continue to work on all your ordinary elisp, and you could
> still use ';' just as you do now. What would change is that you would
> be able to write some elisp as you can't now, and only on that code
> the tools would probably get confused. I have no idea how hard would
> it be to change font-locking and indentation checkers, maybe hard,
> maybe not at all. I can't answer that since I am not acquianted with
> how those tools are implemented (lots of regex as I have heard).

> But in worst case, you can always fall back on writing ordinary elisp.

My tools which handle Elisp would have no such luxury, because they must
handle all valid Elisp.

> I have no idea how hard would it be to change font-locking and
> indentation checkers, maybe hard, maybe not at all.

Please, if you won't take my word for it, try it and see.

Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 7893 bytes --]

> On the contrary, the semicolon prefix (and fontification according to my
> preferences in Elisp buffers) makes it easy for me to see what is code
> and what is prose.

Aren’t starting ’(’ and ’)’ + identation enough?

> No, in fact, some of my Elisp would have to change, because it would no
> onger be sufficient to look at comment prefixes to know whether a line
> s code or comment.  Such a change would be very expensive in terms of
> the necessary human work that would ripple outward.  And your proposed
> variable would not obviate the need to account for both states in my
> code.

In which way? Can you alaborate more please? You would not need to write
anything special. If you don’t wish to write literal code, then just don’t. Prefix
your comment lines with ’;’ as currently and don’t care about it, Everything will
work as it does now. I don’t see a problem there. If I am wrong please explain.

> This is the problem: as you have shown, parentheses commonly appear in
> prose, even at the beginning of lines.  Punctuation, such as semicolons,
> do not
> ; or, at least, should not

Yes, and it is always possible to construct some obscure case where things break.
Look for example at C or C++ and all undefined behaviours in the standard. I mean,
sure, there are always soem obscurities, but do they matter? If one writes an
article or a book or some prosa and uses literal programming, and there is one
explicit case not allowed to use, how important is than to use exactly that
constrcut? I mean considering cons and pros, should one abandon the entire
idea because of one minor obscurity?

> Until we can write programs in human language, we must write both.  For
> humans' sake, the two need to be clearly separated.  If you don't like
> comment prefix syntax, you can use Org files and their source blocks, so
> your prose and your code can both start at column 0.  You can even
> choose a face for org-meta-line that makes #+BEGIN_SRC lines virtually
> (or literally!) disappear.

Well as I see literal programming it is to make it easier for humans to mix code and
prosa. What I proposedis just one small step further in that direction. I don’t see how
org makes it any different? In org you still have to use both ’;’ and #+BEGIN_SRC  -
#+END_SRC markers. So it is even more clutter to eye. It is not about starting at column 0,
It is about being able to simple write and mix code and text. If it is good or bad idea is up
to individual preference. I personally find it a cool thing, you seem to think it violates old
ways, so obviously we wont agree on that one. But thank you for the discussion, I surely
appreciate the input, even if I don’t agree on the main conclusion. I gladly read more!

As a small reflection about us humans being slentrians, I have an illustration.
Caves are still as good for living as they were 20 000 years ago. We can for sure
make many cases against living in houses such as not being natural, can crash on it’s
inhabitants, cost resources to construct etc etc. But yet, pros of liiving in houses outweigh
far the cons, and not many people today prefer to live in a cave. Sorry maybe ti is a too
contrived illustration. Anyway, if you are good with ’;’ just continue to use it. But if a
change like this does not seem to cost lots in development terms, then why opposing if
somebody else find it cool. It is just that is more elegant and in a sense cool thing to be
able to do it without special tools as some markers.

Från: Adam Porter<mailto:adam@alphapapa.net>
Skickat: den 18 december 2019 19:50
Till: emacs-devel@gnu.org<mailto:emacs-devel@gnu.org>
Ämne: Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

arthur miller <arthur.miller@live.com> writes:

> As you mention yourself from,Wikipedia, literate programming is about moving
> away from programming for computers to programming for humans. Using comments
> to comment-away text ment for humans or to mark lines between comments and
> code is indeed writing for the computer, not the human. Humans tend to be
> distracted by unnecessary clutter, and managing stuff just for the sake of
> machine does take a little more effort then not doing it.

; I am not more distracted while reading this line of text
than I am while reading this one.

On the contrary, the semicolon prefix (and fontification according to my
preferences in Elisp buffers) makes it easy for me to see what is code
and what is prose.

> I can also imagine case where Elisp is impossible to distinguish from text,
> (for example this line is valid text but would fail in parser), or whatever
> we would put into a parenthesis on a separate line that does not
> belong to code.

That line beginning with a paren parses as valid Elisp.  Whether it
evaluates depends on the context.

> Anyway, due to elisp looks, we have those "natural markers", ( and ) which
> we can use as delimiters between code and text.

This is the problem: as you have shown, parentheses commonly appear in
prose, even at the beginning of lines.  Punctuation, such as semicolons,
do not
; or, at least, should not.

> For the question of being concerned about omitting ';' to change the parser.
> Well yes :-). Small syntactic sugar change but it reduces that visual
> clutter.

You call it clutter; I call it clearly delineating code and prose.  What
would look cluttered to me would be prose (which may include commented-out
code) interspersed with
(insert "code, which may include or generate prose.")

> It is one step in that direction of writing code for humans and not for the
> computer.

Until we can write programs in human language, we must write both.  For
humans' sake, the two need to be clearly separated.  If you don't like
comment prefix syntax, you can use Org files and their source blocks, so
your prose and your code can both start at column 0.  You can even
choose a face for org-meta-line that makes #+BEGIN_SRC lines virtually
(or literally!) disappear.

> See it as experiment. I thought it would be a cool thing to have. It
> is just an idea. The change is not in an intrusive way. All your elisp
> is still very same.  It just adds an extra feature pretty much for
> free. If it would be an intrusive change that requires your old elisp
> to change, than I wouldn't suggest it. By the way, one could also add
> a variable to turn off that feature.

No, in fact, some of my Elisp would have to change, because it would no
longer be sufficient to look at comment prefixes to know whether a line
is code or comment.  Such a change would be very expensive in terms of
the necessary human work that would ripple outward.  And your proposed
variable would not obviate the need to account for both states in my
code.

And since parens can appear at the start of lines in prose, how would
anyone but a human know whether a line is code or prose?

> As for the tools, since old elisp code does not change, all your tools
> would still continue to work on all your ordinary elisp, and you could
> still use ';' just as you do now. What would change is that you would
> be able to write some elisp as you can't now, and only on that code
> the tools would probably get confused. I have no idea how hard would
> it be to change font-locking and indentation checkers, maybe hard,
> maybe not at all. I can't answer that since I am not acquianted with
> how those tools are implemented (lots of regex as I have heard).

> But in worst case, you can always fall back on writing ordinary elisp.

My tools which handle Elisp would have no such luxury, because they must
handle all valid Elisp.

> I have no idea how hard would it be to change font-locking and
> indentation checkers, maybe hard, maybe not at all.

Please, if you won't take my word for it, try it and see.



[-- Attachment #2: Type: text/html, Size: 13297 bytes --]

Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

[Stefan Monnier]

> ; I am not more distracted while reading this line of text
> than I am while reading this one.

Thre is no discussion of changing the existing syntax/semantic of
Elisp here.  Arthur is proposing a new file format/syntax as an
alternative to the use of weave/tangle or Org-mode.

You don't have to agree with each other.


        Stefan

Christmas wish: Literate Elisp (Intro)

[VanL]

>> ; I am not more distracted while reading this line of text
>> than I am while reading this one.
>
> Thre is no discussion of changing the existing syntax/semantic of
> Elisp here.  Arthur is proposing a new file format/syntax as an
> alternative to the use of weave/tangle or Org-mode.
>
> You don't have to agree with each other.

Arthur's vision is a beautiful thing to my eyes.  Lawyers and
programmers can read and write, thoughtfully and clearly, together
collaboratively, at the Creation before the two crashs.

(message
        "Before you go, have you heard the comedian tell the joke about
         the lawyer, the scientist, and the engineer going to Mars?")

-- 
VanL., 🐞
  əə0@ 一 二 三 言 語 𝔖 元 示 証 明 海 自 己 漢 本 人 Gnus/Emacs (berkeley-unix)

Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

[Adam Porter]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Thre is no discussion of changing the existing syntax/semantic of
> Elisp here.  Arthur is proposing a new file format/syntax as an
> alternative to the use of weave/tangle or Org-mode.

That's not my understanding of what Arthur is proposing.  You did
propose an alternative file format in earlier messages when you wrote:

> As for using it in Elisp: I don't think there's anything stopping
> anyone from making such a `literate-elisp-mode` and even arrange for
> `load` to handle such a file (just like there is already a package
> that lets `load` work directly on .org files).  I'd welcome such a
> package in GNU ELPA.

And:

> Note that in my comment above I meant it in existing Emacsen,
> i.e. without any modification to the C code.  The idea is to make use
> of existing hooks such as `load-source-file-function` or
> `file-name-handler-alist`.  I'm not sure what hook could be used to do
> the same with byte-compilation, but in the worst case, an advice
> should do the trick.

But Arthur has been proposing to alter the Emacs Elisp parser itself,
e.g. when he wrote:

> My proposal is to slightly change Elisp parser to treat lines that
> start with any other printable character but '(' as a start of comment
> and to simply ignore the line, just as it treats ';' as a comment

And:

> I was able to make readevalloop do what I want. It was rather trivial
> to implement this (once I realized how it works 🙂). Emacs built fine
> and I was able to test wtih eval-buffer and eval-regionwhich worked as
> intended.

> It was just 4 code lines, I just copied codeto parse comment and
> change the condition in if-statement:

> However, byte compiler is not happy about this. It emits warnings for
> every word out of code blocks as references to free variables. I am
> not sure where to look to patch it, maybe another day.

And: 

> If we would just use directives in comments then we already have org +
> babel. But I think 4 extra lines of C code to enable this feature in
> eval-loop was a trivial price to pay :-).

> Now I just wonder how to change the bytecode compiler. Unfortunately
> it seems to be much more involved process. I don't know if I can do
> it, since I am not so acquainted with Emacs internals, so it would be
> cool if someone more knowledgeable can at least hint me if not help.

I've looked again at all messages in the thread.  Am I missing
something?

Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[Adam Porter]

arthur miller <arthur.miller@live.com> writes:

>> On the contrary, the semicolon prefix (and fontification according to
>> my preferences in Elisp buffers) makes it easy for me to see what is
>> code and what is prose.
>
> Aren’t starting ’(’ and ’)’ + identation enough?

I explained in the message you quoted why they are not enough.  Please
read it carefully.

>> No, in fact, some of my Elisp would have to change, because it would
>> no onger be sufficient to look at comment prefixes to know whether a
>> line s code or comment.  Such a change would be very expensive in
>> terms of the necessary human work that would ripple outward.  And
>> your proposed variable would not obviate the need to account for both
>> states in my code.
>
> In which way? Can you alaborate more please? You would not need to
> write anything special. If you don’t wish to write literal code, then
> just don’t. Prefix your comment lines with ’;’ as currently and don’t
> care about it, Everything will work as it does now. I don’t see a
> problem there. If I am wrong please explain.

You don't seem to understand.  I am talking about existing code outside
of emacs.git which parses Elisp, which would not be compatible with your
proposed changes to Elisp syntax.

>> This is the problem: as you have shown, parentheses commonly appear in
>> prose, even at the beginning of lines.  Punctuation, such as semicolons,
>> do not
>> ; or, at least, should not
>
> Yes, and it is always possible to construct some obscure case where
> things break.  Look for example at C or C++ and all undefined
> behaviours in the standard. I mean, sure, there are always soem
> obscurities, but do they matter? If one writes an article or a book or
> some prosa and uses literal programming, and there is one explicit
> case not allowed to use, how important is than to use exactly that
> constrcut? I mean considering cons and pros, should one abandon the
> entire idea because of one minor obscurity?

Your own message showed how it happens in prose.  In fact, it's common
enough and ambiguous enough that Elisp already forbids open-parens at
column 0 in strings and requires them to be escaped.  How is this a
minor obscurity?

Don't you realize how important it is that syntax in a file format be
unambiguous?

> Well as I see literal programming it is to make it easier for humans
> to mix code and prosa. What I proposedis just one small step further
> in that direction.

It is also a step toward ambiguity, churn, and bugs in other people's
code, which you would not have to fix.

> I don’t see how org makes it any different? In org you still have to
> use both ’;’ and #+BEGIN_SRC - #+END_SRC markers.

Have you used Org mode?  Have you looked at a literate program written
in an Org file?  The point is that top-level comments are written
between code blocks, without comment prefixes.

> So it is even more clutter to eye.

As I said in the message you're replying to:

  You can even choose a face for org-meta-line that makes #+BEGIN_SRC
  lines virtually (or literally!) disappear.

> It is about being able to simple write and mix code and text. If it is
> good or bad idea is up to individual preference. I personally find it
> a cool thing, you seem to think it violates old ways,

I have said nothing about old ways.  I explained exactly what the
problems with the proposal are.  Please read my message carefully.

> As a small reflection about us humans being slentrians, I have an
> illustration.  Caves are still as good for living as they were 20 000
> years ago. We can for sure make many cases against living in houses
> such as not being natural, can crash on it’s inhabitants, cost
> resources to construct etc etc. But yet, pros of liiving in houses
> outweigh far the cons, and not many people today prefer to live in a
> cave. Sorry maybe ti is a too contrived illustration. Anyway, if you
> are good with ’;’ just continue to use it. But if a change like this
> does not seem to cost lots in development terms, then why opposing if
> somebody else find it cool. It is just that is more elegant and in a
> sense cool thing to be able to do it without special tools as some
> markers.

It's not cool to imply that citing technical problems with your proposal
is like being a cave-dweller.

Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 3216 bytes --]

Yes Adam you are correct, but altering parser does not necessary mean
that elisp will change in a way that will force you to change your
existing code or coding practice. I proposed it in a way that will simply
add an extra feature, which you don’t need to use if you don’t like it. It
is trivial to make it by default ”off” by introducing new variable one can
set in init file to enable it (or disable it, whichever is better for default).

Hope it makes it a bit more clear what I suggested.

Från: Adam Porter<mailto:adam@alphapapa.net>
Skickat: den 19 december 2019 00:00
Till: emacs-devel@gnu.org<mailto:emacs-devel@gnu.org>
Ämne: Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Thre is no discussion of changing the existing syntax/semantic of
> Elisp here.  Arthur is proposing a new file format/syntax as an
> alternative to the use of weave/tangle or Org-mode.

That's not my understanding of what Arthur is proposing.  You did
propose an alternative file format in earlier messages when you wrote:

> As for using it in Elisp: I don't think there's anything stopping
> anyone from making such a `literate-elisp-mode` and even arrange for
> `load` to handle such a file (just like there is already a package
> that lets `load` work directly on .org files).  I'd welcome such a
> package in GNU ELPA.

And:

> Note that in my comment above I meant it in existing Emacsen,
> i.e. without any modification to the C code.  The idea is to make use
> of existing hooks such as `load-source-file-function` or
> `file-name-handler-alist`.  I'm not sure what hook could be used to do
> the same with byte-compilation, but in the worst case, an advice
> should do the trick.

But Arthur has been proposing to alter the Emacs Elisp parser itself,
e.g. when he wrote:

> My proposal is to slightly change Elisp parser to treat lines that
> start with any other printable character but '(' as a start of comment
> and to simply ignore the line, just as it treats ';' as a comment

And:

> I was able to make readevalloop do what I want. It was rather trivial
> to implement this (once I realized how it works 🙂). Emacs built fine
> and I was able to test wtih eval-buffer and eval-regionwhich worked as
> intended.

> It was just 4 code lines, I just copied codeto parse comment and
> change the condition in if-statement:

> However, byte compiler is not happy about this. It emits warnings for
> every word out of code blocks as references to free variables. I am
> not sure where to look to patch it, maybe another day.

And:

> If we would just use directives in comments then we already have org +
> babel. But I think 4 extra lines of C code to enable this feature in
> eval-loop was a trivial price to pay :-).

> Now I just wonder how to change the bytecode compiler. Unfortunately
> it seems to be much more involved process. I don't know if I can do
> it, since I am not so acquainted with Emacs internals, so it would be
> cool if someone more knowledgeable can at least hint me if not help.

I've looked again at all messages in the thread.  Am I missing
something?



[-- Attachment #2: Type: text/html, Size: 5443 bytes --]

Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[Adam Porter]

arthur miller <arthur.miller@live.com> writes:

> Yes Adam you are correct, but altering parser does not necessary mean 
>
> that elisp will change in a way that will force you to change your
>
> existing code or coding practice. I proposed it in a way that will simply 
>
> add an extra feature, which you don’t need to use if you don’t like it. It
>
> is trivial to make it by default ”off” by introducing new variable one can
>
> set in init file to enable it (or disable it, whichever is better for default).
>
> Hope it makes it a bit more clear what I suggested.

(Please do not double-space your messages.)

I have tried to explain this issue as clearly as I can.  I will ask once
more: Do you understand that Elisp code written in the way you propose
would not be compatible with existing tools which parse Elisp?  And that
such tools would require modification to parse such code correctly?

Stefan suggested ways to implement your idea as an alternative, literate
syntax, in a separate file format, by writing it in Elisp, using advice
and/or configuration variables, so that modification of the parser in C
would not be required, and the existing Elisp syntax and parser would
remain unchanged.  That is a great idea.  Why don't you want to do that?

Sv: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 6392 bytes --]

I didn’t say you are a cave dweller. I am sorry if you perceive my
illustration that way. That was just a very first image that came
into my mind. Sorry if you find it somehow offending.  I was just
trying to say that a case can be made against literally anything.
One has to compare cons and pros, and see if pros outweigh the cons.

I don’t want to repeat myself, but I am sure that as long as one don’t
use the new feature all your external tools will continue to work as
they do now. If you don’t write literal code then your code would look
like just as it does now, so no confusion there. If it is disabled by
default then I see no way it could confuse any existing tool. I might
be wrong of course, but please give me technical reasons.
Tools need to be updated just if they will be used with new syntax. So
in general, if you have a tool that can’t deal with ”literal syntax”,
than don’t use the literal syntax or the tool. Is that a problem?

About literal programs without special markers being confusing:
honestly I believe that most humans find it quite easy to understand
the difference between text (prose for humans) and code (prose for
computers). I don't think humans need explicit markers as comments
or similar. Comments are written for computers to distinguish between
code and things they should ignore so that we humans can communicate
with each other in our code. I don’t know, something like that. I can
totally understand if this proposal is not implementable due to
technical reasons, or that this really does not give that much of a
value to be worth enabling. But I don’t buy the argument that humans
need some special markers to understand the difference between code
and text. The whole Idea with literal programming is to minimize that
difference.

Från: Adam Porter<mailto:adam@alphapapa.net>
Skickat: den 19 december 2019 00:18
Till: emacs-devel@gnu.org<mailto:emacs-devel@gnu.org>
Ämne: Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

arthur miller <arthur.miller@live.com> writes:

>> On the contrary, the semicolon prefix (and fontification according to
>> my preferences in Elisp buffers) makes it easy for me to see what is
>> code and what is prose.
>
> Aren’t starting ’(’ and ’)’ + identation enough?

I explained in the message you quoted why they are not enough.  Please
read it carefully.

>> No, in fact, some of my Elisp would have to change, because it would
>> no onger be sufficient to look at comment prefixes to know whether a
>> line s code or comment.  Such a change would be very expensive in
>> terms of the necessary human work that would ripple outward.  And
>> your proposed variable would not obviate the need to account for both
>> states in my code.
>
> In which way? Can you alaborate more please? You would not need to
> write anything special. If you don’t wish to write literal code, then
> just don’t. Prefix your comment lines with ’;’ as currently and don’t
> care about it, Everything will work as it does now. I don’t see a
> problem there. If I am wrong please explain.

You don't seem to understand.  I am talking about existing code outside
of emacs.git which parses Elisp, which would not be compatible with your
proposed changes to Elisp syntax.

>> This is the problem: as you have shown, parentheses commonly appear in
>> prose, even at the beginning of lines.  Punctuation, such as semicolons,
>> do not
>> ; or, at least, should not
>
> Yes, and it is always possible to construct some obscure case where
> things break.  Look for example at C or C++ and all undefined
> behaviours in the standard. I mean, sure, there are always soem
> obscurities, but do they matter? If one writes an article or a book or
> some prosa and uses literal programming, and there is one explicit
> case not allowed to use, how important is than to use exactly that
> constrcut? I mean considering cons and pros, should one abandon the
> entire idea because of one minor obscurity?

Your own message showed how it happens in prose.  In fact, it's common
enough and ambiguous enough that Elisp already forbids open-parens at
column 0 in strings and requires them to be escaped.  How is this a
minor obscurity?

Don't you realize how important it is that syntax in a file format be
unambiguous?

> Well as I see literal programming it is to make it easier for humans
> to mix code and prosa. What I proposedis just one small step further
> in that direction.

It is also a step toward ambiguity, churn, and bugs in other people's
code, which you would not have to fix.

> I don’t see how org makes it any different? In org you still have to
> use both ’;’ and #+BEGIN_SRC - #+END_SRC markers.

Have you used Org mode?  Have you looked at a literate program written
in an Org file?  The point is that top-level comments are written
between code blocks, without comment prefixes.

> So it is even more clutter to eye.

As I said in the message you're replying to:

  You can even choose a face for org-meta-line that makes #+BEGIN_SRC
  lines virtually (or literally!) disappear.

> It is about being able to simple write and mix code and text. If it is
> good or bad idea is up to individual preference. I personally find it
> a cool thing, you seem to think it violates old ways,

I have said nothing about old ways.  I explained exactly what the
problems with the proposal are.  Please read my message carefully.

> As a small reflection about us humans being slentrians, I have an
> illustration.  Caves are still as good for living as they were 20 000
> years ago. We can for sure make many cases against living in houses
> such as not being natural, can crash on it’s inhabitants, cost
> resources to construct etc etc. But yet, pros of liiving in houses
> outweigh far the cons, and not many people today prefer to live in a
> cave. Sorry maybe ti is a too contrived illustration. Anyway, if you
> are good with ’;’ just continue to use it. But if a change like this
> does not seem to cost lots in development terms, then why opposing if
> somebody else find it cool. It is just that is more elegant and in a
> sense cool thing to be able to do it without special tools as some
> markers.

It's not cool to imply that citing technical problems with your proposal
is like being a cave-dweller.



[-- Attachment #2: Type: text/html, Size: 9395 bytes --]

Sv: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

I actually answered in my previous mail. You are wrong about
that this proposal would broke your tools.

Those 4 lines of C, would cost me many more lines of elisp, as least
as much as I can elisp (rather as little). I am currently fighting with
byte compiler which is mostly elisp and it does not go so well 😊. By
the way those 4 lines will be literally invisible (not entered) if one puts
a variable around them as I suggested.

For the record, even if I wrote an elisp package that implements this in
pure elisp, and somebody decides to write ”literal code” your existings
tools would still be broken. I don’t see how the implementation choice
would change that matter nor do I see why is it so big matter for you?

Från: Adam Porter<mailto:adam@alphapapa.net>
Skickat: den 19 december 2019 00:53
Till: emacs-devel@gnu.org<mailto:emacs-devel@gnu.org>
Ämne: Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

arthur miller <arthur.miller@live.com> writes:

> Yes Adam you are correct, but altering parser does not necessary mean
>
> that elisp will change in a way that will force you to change your
>
> existing code or coding practice. I proposed it in a way that will simply
>
> add an extra feature, which you don’t need to use if you don’t like it. It
>
> is trivial to make it by default ”off” by introducing new variable one can
>
> set in init file to enable it (or disable it, whichever is better for default).
>
> Hope it makes it a bit more clear what I suggested.

(Please do not double-space your messages.)

I have tried to explain this issue as clearly as I can.  I will ask once
more: Do you understand that Elisp code written in the way you propose
would not be compatible with existing tools which parse Elisp?  And that
such tools would require modification to parse such code correctly?

Stefan suggested ways to implement your idea as an alternative, literate
syntax, in a separate file format, by writing it in Elisp, using advice
and/or configuration variables, so that modification of the parser in C
would not be required, and the existing Elisp syntax and parser would
remain unchanged.  That is a great idea.  Why don't you want to do that?




[-- Attachment #2: Type: text/html, Size: 4347 bytes --]

Sv: Christmas wish: Literate Elisp (Intro)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

😊

Skickades från E-post<https://go.microsoft.com/fwlink/?LinkId=550986> för Windows 10

________________________________
Från: Emacs-devel <emacs-devel-bounces+arthur.miller=live.com@gnu.org> för VanL <van@scratch.space>
Skickat: Wednesday, December 18, 2019 11:43:39 PM
Till: emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Christmas wish: Literate Elisp (Intro)


>> ; I am not more distracted while reading this line of text
>> than I am while reading this one.
>
> Thre is no discussion of changing the existing syntax/semantic of
> Elisp here.  Arthur is proposing a new file format/syntax as an
> alternative to the use of weave/tangle or Org-mode.
>
> You don't have to agree with each other.

Arthur's vision is a beautiful thing to my eyes.  Lawyers and
programmers can read and write, thoughtfully and clearly, together
collaboratively, at the Creation before the two crashs.

(message
        "Before you go, have you heard the comedian tell the joke about
         the lawyer, the scientist, and the engineer going to Mars?")

--
VanL., 🐞
  əə0@ 一 二 三 言 語 𝔖 元 示 証 明 海 自 己 漢 本 人 Gnus/Emacs (berkeley-unix)



[-- Attachment #2: Type: text/html, Size: 3189 bytes --]

Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[chad]

[-- Attachment #1: Type: text/plain, Size: 3505 bytes --]

It seems that there is a communications breakdown here. At the risk of
poking the bear, I'm going to try to see if I can help resolve it.

The place where you are adding 4 lines of C code does not change the syntax
of elisp as a whole; it merely changes the way that particular version of
emacs parses the new format. There is a large body of existing software
which will be totally unaware of your changes. That body is large enough
that it is doubtful that it can, as a practical matter, be reasonably
changed to not break on the syntax changes that you are proposing.

On the other hand, there are ways to accomplish what seems (at least to
some of us) to be your stated goals without requiring a change to the
syntax of elisp. Using that approach instead, that large body of existing
software would continue to work, and, over time, might eventually be
expanded to add extra value to the additions.

(This principle is often described as "be conservative in what you send,
and liberal in what you accept".)

I hope that helps.
~Chad

On Wed, Dec 18, 2019 at 4:03 PM arthur miller <arthur.miller@live.com>
wrote:

> I actually answered in my previous mail. You are wrong about
>
> that this proposal would broke your tools.
>
> Those 4 lines of C, would cost me many more lines of elisp, as least
>
> as much as I can elisp (rather as little). I am currently fighting with
>
> byte compiler which is mostly elisp and it does not go so well 😊. By
>
> the way those 4 lines will be literally invisible (not entered) if one puts
>
> a variable around them as I suggested.
>
>
>
> For the record, even if I wrote an elisp package that implements this in
>
> pure elisp, and somebody decides to write ”literal code” your existings
>
> tools would still be broken. I don’t see how the implementation choice
>
> would change that matter nor do I see why is it so big matter for you?
>
>
>
> *Från: *Adam Porter <adam@alphapapa.net>
> *Skickat: *den 19 december 2019 00:53
> *Till: *emacs-devel@gnu.org
> *Ämne: *Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp
>
>
>
> arthur miller <arthur.miller@live.com> writes:
>
> > Yes Adam you are correct, but altering parser does not necessary mean
> >
> > that elisp will change in a way that will force you to change your
> >
> > existing code or coding practice. I proposed it in a way that will
> simply
> >
> > add an extra feature, which you don’t need to use if you don’t like it.
> It
> >
> > is trivial to make it by default ”off” by introducing new variable one
> can
> >
> > set in init file to enable it (or disable it, whichever is better for
> default).
> >
> > Hope it makes it a bit more clear what I suggested.
>
> (Please do not double-space your messages.)
>
> I have tried to explain this issue as clearly as I can.  I will ask once
> more: Do you understand that Elisp code written in the way you propose
> would not be compatible with existing tools which parse Elisp?  And that
> such tools would require modification to parse such code correctly?
>
> Stefan suggested ways to implement your idea as an alternative, literate
> syntax, in a separate file format, by writing it in Elisp, using advice
> and/or configuration variables, so that modification of the parser in C
> would not be required, and the existing Elisp syntax and parser would
> remain unchanged.  That is a great idea.  Why don't you want to do that?
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 4960 bytes --]

Re: Christmas wish: Literate Elisp

[Jean-Christophe Helary]

> On Dec 19, 2019, at 9:42, chad <yandros@gmail.com> wrote:
> 
> There is a large body of existing software which will be totally unaware of your changes.

Although I think the premise of your comment is absolutely valid, I'm not so sure about the "*large* body of existing software".


Jean-Christophe Helary
-----------------------------------------------
http://mac4translators.blogspot.com @brandelune

Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1.1: Type: text/plain, Size: 2538 bytes --]

Here is a little prototype to test my idea with literal Elisp.

I have patched read-eval loop as stated in previous mail, in lread.c.
It is probably possible to implement that in Elisp, but I don't have
enough knowledge of either Emacs internals nor Elisp to do this in some
short time. It would took me a lot of time to look up all the
things I would need. Anyway, C code turned to be rather trivial, and also
completely by-passable if so desired, so I have introduced a user-customizable
variable 'emacs-lisp-allow-literal-comments', which is by default nil.

I wasn't sure in which elisp file to introduce this variable, so I have used
'simple.el' found in lisp directory of Emacs source distribution. That file
seems to have some other user customizable variables that affect elisp so I
thought it might be appropriate place.

For the byte compiler I have patched bytecomp.el, in rather brutish way, but it
seems to work. It wasn't very difficult either, but I think I have done it rather
ugly. Someone might wish to refactor that code. Anyway, it is less then
twenty lines of code, and it is by default bypassed as well. The variable
that controls it is also user customizable and found  in same file,
named 'byte-comp-allow-literal-comments'.

I have attached also a small trivial elisp file for illustration purpose.

It is just a test of an idea, and small prototype to show that it might work.
It needs more thorough testing and can probably be implemented in some better
way.

I have tested on GNU/Linux and Windows. Emacs was able to compile it's own
elisp as well as external packages I use.

As a note, the change in C is completely backwards compatible. No logical
change to elisp parser happens when 'emacs-lisp-allow-literal-comments'
variable is nil.

________________________________
Från: Emacs-devel <emacs-devel-bounces+arthur.miller=live.com@gnu.org> för Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
Skickat: den 19 december 2019 02:50
Till: emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: Christmas wish: Literate Elisp



> On Dec 19, 2019, at 9:42, chad <yandros@gmail.com> wrote:
>
> There is a large body of existing software which will be totally unaware of your changes.

Although I think the premise of your comment is absolutely valid, I'm not so sure about the "*large* body of existing software".


Jean-Christophe Helary
-----------------------------------------------
http://mac4translators.blogspot.com @brandelune




[-- Attachment #1.2: Type: text/html, Size: 4297 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: simple.patch --]
[-- Type: text/x-patch; name="simple.patch", Size: 479 bytes --]

--- simple.el	2019-12-20 01:06:16.849343961 +0100
+++ mysimple.el	2019-12-20 01:03:05.212467674 +0100
@@ -127,6 +127,14 @@
   :safe #'booleanp
   :version "27.1")
 
+(defcustom emacs-lisp-allow-literal-comments nil
+  "If not nil, elisp will allow you to type in top-level comments
+   without using semicolon."
+  :group 'lisp
+  :type 'boolean
+  :safe #'booleanp
+  :version "27.1")
+
 (defvar next-error-highlight-timer nil)
 
 (defvar next-error-overlay-arrow-position nil)

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: lread.patch --]
[-- Type: text/x-patch; name="lread.patch", Size: 906 bytes --]

--- lread.c	2019-12-20 01:07:33.264089317 +0100
+++ mylread.c	2019-12-20 01:02:38.605308850 +0100
@@ -1955,6 +1955,8 @@
   /* True on the first time around.  */
   bool first_sexp = 1;
   Lisp_Object macroexpand = intern ("internal-macroexpand-for-load");
+  Lisp_Object litcode = intern ("emacs-lisp-allow-literal-comments");
+  Lisp_Object allow_literal_comments = find_symbol_value(litcode);
 
   if (NILP (Ffboundp (macroexpand))
       || (STRINGP (sourcename) && suffix_p (sourcename, ".elc")))
@@ -2053,6 +2055,15 @@
 	  || c == NO_BREAK_SPACE)
 	goto read_next;
 
+      if (EQ (allow_literal_comments, Qt))
+        {
+          if (c != '(' && c != '#')
+            {
+              while ((c = READCHAR) != '\n' && c != -1);
+              goto read_next;
+            }
+        }
+
       if (! HASH_TABLE_P (read_objects_map)
 	  || XHASH_TABLE (read_objects_map)->count)
 	read_objects_map

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: bytecomp.patch --]
[-- Type: text/x-patch; name="bytecomp.patch", Size: 3207 bytes --]

--- bytecomp.el	2019-12-20 01:05:44.575414147 +0100
+++ mybytecomp.el	2019-12-20 01:03:24.409489389 +0100
@@ -154,6 +154,14 @@
   :type '(choice (const nil) function)
   :version "23.2")
 
+(defcustom byte-compile-allow-literal-comments nil
+  "If not nil, byte compiler will allow you to type
+   top-level comments without using semicolon in emacs lisp."
+  :group 'bytecomp
+  :type 'boolean
+  :safe #'booleanp
+  :version "27.1")
+
 ;; This enables file name handlers such as jka-compr
 ;; to remove parts of the file name that should not be copied
 ;; through to the output file name.
@@ -2079,8 +2087,8 @@
 	(print-level nil)
 	;; Prevent edebug from interfering when we compile
 	;; and put the output into a file.
-;; 	(edebug-all-defs nil)
-;; 	(edebug-all-forms nil)
+        ;; 	(edebug-all-defs nil)
+        ;; 	(edebug-all-forms nil)
 	;; Simulate entry to byte-compile-top-level
         (byte-compile-jump-tables nil)
         (byte-compile-constants nil)
@@ -2127,19 +2135,35 @@
 			       (= (following-char) ?\;))
 		   (forward-line 1))
 		 (not (eobp)))
-	  (setq byte-compile-read-position (point)
+
+          (setq byte-compile-read-position (point)
 		byte-compile-last-position byte-compile-read-position)
-          (let* ((lread--unescaped-character-literals nil)
-                 (form (read inbuffer))
-                 (warning (byte-run--unescaped-character-literals-warning)))
-            (when warning (byte-compile-warn "%s" warning))
-	    (byte-compile-toplevel-file-form form)))
-	;; Compile pending forms at end of file.
-	(byte-compile-flush-pending)
-	;; Make warnings about unresolved functions
-	;; give the end of the file as their position.
-	(setq byte-compile-last-position (point-max))
-	(byte-compile-warn-about-unresolved-functions))
+
+	  (if byte-compile-allow-literal-comments
+              (progn
+                (if (or (= (following-char) ?\()
+                        (= (following-char) ?\#)
+                        (= (following-char) ?\'))
+                    (progn
+                      (let* ((lread--unescaped-character-literals nil)
+                             (form (read inbuffer))
+                             (warning (byte-run--unescaped-character-literals-warning)))
+                        (when warning (byte-compile-warn "%s" warning))
+	                (byte-compile-toplevel-file-form form)))
+                  (forward-line 1)))
+            (progn
+              (let* ((lread--unescaped-character-literals nil)
+                     (form (read inbuffer))
+                     (warning (byte-run--unescaped-character-literals-warning)))
+                (when warning (byte-compile-warn "%s" warning))
+	        (byte-compile-toplevel-file-form form)))))
+
+        ;; Compile pending forms at end of file.
+        (byte-compile-flush-pending)
+        ;; Make warnings about unresolved functions
+        ;; give the end of the file as their position.
+        (setq byte-compile-last-position (point-max))
+        (byte-compile-warn-about-unresolved-functions))
       ;; Fix up the header at the front of the output
       ;; if the buffer contains multibyte characters.
       (and byte-compile-current-file

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: chatty.el --]
[-- Type: text/x-emacs-lisp; name="chatty.el", Size: 524 bytes --]

Elisp now alows comments without ;.

(setq emacs-lisp-allow-literal-comments nil)
(setq byte-compile-allow-literal-comments nil)

(message "Hello, World!")

It is of course possible to intertwene code
with comments anywhere in the file.

(message "Hello Again!")

Of course we can also comment-out code
just as before:

;;(message "I am silent")
In code blocs the ';' is still a comment delimiter:
(message
 ;; Here is a line commne
 "I am a bit chatty today!")

That's it for today folks!

(message "Bye bye cruel world!")

Re: Sv: Christmas wish: Literate Elisp

[Stefan Monnier]

Of course, this decision is not up to me, but I'll point out that
I think I don't think we should change the C code to or the
byte-compiler to directly support this new format.

I'd welcome changes that add hooks to make it possible/easier to support
such a format (and other formats like Org-mode), OTOH.


        Stefan


arthur miller [2019-12-20 00:55:29] wrote:

> Here is a little prototype to test my idea with literal Elisp.
>
> I have patched read-eval loop as stated in previous mail, in lread.c.
> It is probably possible to implement that in Elisp, but I don't have
> enough knowledge of either Emacs internals nor Elisp to do this in some
> short time. It would took me a lot of time to look up all the
> things I would need. Anyway, C code turned to be rather trivial, and also
> completely by-passable if so desired, so I have introduced a user-customizable
> variable 'emacs-lisp-allow-literal-comments', which is by default nil.
>
> I wasn't sure in which elisp file to introduce this variable, so I have used
> 'simple.el' found in lisp directory of Emacs source distribution. That file
> seems to have some other user customizable variables that affect elisp so I
> thought it might be appropriate place.
>
> For the byte compiler I have patched bytecomp.el, in rather brutish way, but it
> seems to work. It wasn't very difficult either, but I think I have done it rather
> ugly. Someone might wish to refactor that code. Anyway, it is less then
> twenty lines of code, and it is by default bypassed as well. The variable
> that controls it is also user customizable and found  in same file,
> named 'byte-comp-allow-literal-comments'.
>
> I have attached also a small trivial elisp file for illustration purpose.
>
> It is just a test of an idea, and small prototype to show that it might work.
> It needs more thorough testing and can probably be implemented in some better
> way.
>
> I have tested on GNU/Linux and Windows. Emacs was able to compile it's own
> elisp as well as external packages I use.
>
> As a note, the change in C is completely backwards compatible. No logical
> change to elisp parser happens when 'emacs-lisp-allow-literal-comments'
> variable is nil.
>
> ________________________________
> Från: Emacs-devel <emacs-devel-bounces+arthur.miller=live.com@gnu.org> för
> Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
> Skickat: den 19 december 2019 02:50
> Till: emacs-devel@gnu.org <emacs-devel@gnu.org>
> Ämne: Re: Christmas wish: Literate Elisp
>
>
>
>> On Dec 19, 2019, at 9:42, chad <yandros@gmail.com> wrote:
>>
>> There is a large body of existing software which will be totally unaware of your changes.
>
> Although I think the premise of your comment is absolutely valid, I'm not so
> sure about the "*large* body of existing software".
>
>
> Jean-Christophe Helary
> -----------------------------------------------
> http://mac4translators.blogspot.com @brandelune

Re: Sv: Christmas wish: Literate Elisp

[Stefan Monnier]

> Of course, this decision is not up to me, but I'll point out that
> I think I don't think we should change the C code to or the
  ^^^^^^^^^^^^^^^^^^^^^
Do I think?


        Stefan

Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 5181 bytes --]

I really understand the concern, but I must ask how would one change the
elisp itself without re-implementing much or some of of elisp parser if it would
be done externally?

The read-eval loop is not exposed to elisp, so one would need to implement
own? Or am I missunderstanding? As I understand the elisps read-eval loop
is part of C runtime core, so if one would implement another parser on top of
it, it would mean some kind of preprocessor to preprocess elisp before it is
handled into elisp parser. Thisis only for read-eval loop.

I believe it would mean lots of work, and maybe less efficiency since it might
impose some extra copying/reading? I don't know, as I noted before I am not
that very wellacquinted with Emacs internals so I can't tell for sure. Personally
I don't care in which way it is implemented as long as the result is the same.

Just as a note about the C change and hooks and runtime support: the flag I
used lets me completely bypass the change to the "old" parser. Skipping
comment lines is hard coded into read-eval loop, so I have just hacked around
it, but if one would introduce some kind of modifiable read-eval loop with hooks
or I don't know what then it would probably need more C changes than waht
I did. Byte compiler would also need some change since comments are
hardcoded there as well. Also if one not introduce any change in byte compiler
and read eval loop, it might mean one would preprocess elisp source in some
hook, which might need extra loop through source file which is less effecient.

At least that is how I am reading the code, but I just started to hack it, so I might
be missing something. If one added a buffer-local flag instead of as I did a global
flag, then this could be that support in runtime to make it possible/easier to
support this. I don't know, I might be wrong, just as I see the code as of the
moment.
________________________________
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Skickat: den 20 december 2019 16:00
Till: arthur miller <arthur.miller@live.com>
Kopia: emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: Sv: Christmas wish: Literate Elisp

Of course, this decision is not up to me, but I'll point out that
I think I don't think we should change the C code to or the
byte-compiler to directly support this new format.

I'd welcome changes that add hooks to make it possible/easier to support
such a format (and other formats like Org-mode), OTOH.


        Stefan


arthur miller [2019-12-20 00:55:29] wrote:

> Here is a little prototype to test my idea with literal Elisp.
>
> I have patched read-eval loop as stated in previous mail, in lread.c.
> It is probably possible to implement that in Elisp, but I don't have
> enough knowledge of either Emacs internals nor Elisp to do this in some
> short time. It would took me a lot of time to look up all the
> things I would need. Anyway, C code turned to be rather trivial, and also
> completely by-passable if so desired, so I have introduced a user-customizable
> variable 'emacs-lisp-allow-literal-comments', which is by default nil.
>
> I wasn't sure in which elisp file to introduce this variable, so I have used
> 'simple.el' found in lisp directory of Emacs source distribution. That file
> seems to have some other user customizable variables that affect elisp so I
> thought it might be appropriate place.
>
> For the byte compiler I have patched bytecomp.el, in rather brutish way, but it
> seems to work. It wasn't very difficult either, but I think I have done it rather
> ugly. Someone might wish to refactor that code. Anyway, it is less then
> twenty lines of code, and it is by default bypassed as well. The variable
> that controls it is also user customizable and found  in same file,
> named 'byte-comp-allow-literal-comments'.
>
> I have attached also a small trivial elisp file for illustration purpose.
>
> It is just a test of an idea, and small prototype to show that it might work.
> It needs more thorough testing and can probably be implemented in some better
> way.
>
> I have tested on GNU/Linux and Windows. Emacs was able to compile it's own
> elisp as well as external packages I use.
>
> As a note, the change in C is completely backwards compatible. No logical
> change to elisp parser happens when 'emacs-lisp-allow-literal-comments'
> variable is nil.
>
> ________________________________
> Från: Emacs-devel <emacs-devel-bounces+arthur.miller=live.com@gnu.org> för
> Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
> Skickat: den 19 december 2019 02:50
> Till: emacs-devel@gnu.org <emacs-devel@gnu.org>
> Ämne: Re: Christmas wish: Literate Elisp
>
>
>
>> On Dec 19, 2019, at 9:42, chad <yandros@gmail.com> wrote:
>>
>> There is a large body of existing software which will be totally unaware of your changes.
>
> Although I think the premise of your comment is absolutely valid, I'm not so
> sure about the "*large* body of existing software".
>
>
> Jean-Christophe Helary
> -----------------------------------------------
> http://mac4translators.blogspot.com @brandelune


[-- Attachment #2: Type: text/html, Size: 10579 bytes --]

Re: Christmas wish: Literate Elisp

[Unknown]

arthur miller <arthur.miller@live.com> 작성:

> I don’t want to repeat myself, but I am sure that as long as one don’t
> use the new feature all your external tools will continue to work as
> they do now. If you don’t write literal code then your code would look
> like just as it does now, so no confusion there. If it is disabled by
> default then I see no way it could confuse any existing tool. I might
> be wrong of course, but please give me technical reasons.
> Tools need to be updated just if they will be used with new syntax. So
> in general, if you have a tool that can’t deal with ”literal syntax”,
> than don’t use the literal syntax or the tool. Is that a problem?

Yeah, it is, mostly because code written be people who believes that  
writing prose interleaved with Elisp without any explicit markers (like  
you) will break the existing tools.

If one decided to push their prose-with-elisp code to GitHub, GitHub’s  
Elisp highlighter would be confused. That’s just one example with the  
problem. You might totally be fine with that: you know that it’s not the  
usual syntax… but other people would find out & be upset about that.

I bet there will be elisp-mundging code in Emacs that aren’t hooked onto  
the reader… which means that those code will be also broken.

> About literal programs without special markers being confusing:
> honestly I believe that most humans find it quite easy to understand
> the difference between text (prose for humans) and code (prose for
> computers). I don't think humans need explicit markers as comments
> or similar. Comments are written for computers to distinguish between
> code and things they should ignore so that we humans can communicate
> with each other in our code. I don’t know, something like that. I can
> totally understand if this proposal is not implementable due to
> technical reasons, or that this really does not give that much of a
> value to be worth enabling. But I don’t buy the argument that humans
> need some special markers to understand the difference between code
> and text. The whole Idea with literal programming is to minimize that
> difference.

Well, maybe the whole idea of literal programming is to minimize the  
difference, but elisp isn’t for literal programming right?

If you really want this, I would nudge you to make a Elisp dialect  
optimized for literal programming implemented by a preprocessor in the  
front of the Elisp reader.
I really can’t see why this is beneficial to anyone (except maybe for you).

> Från: Adam Porter
> Skickat: den 19 december 2019 00:18
> Till: emacs-devel@gnu.org
> Ämne: Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp
>
> arthur miller <arthur.miller@live.com> writes:
>
> >> On the contrary, the semicolon prefix (and fontification according to
> >> my preferences in Elisp buffers) makes it easy for me to see what is
> >> code and what is prose.
> >
> > Aren’t starting ’(’ and ’)’ + identation enough?
>
> I explained in the message you quoted why they are not enough.  Please
> read it carefully.
>
> >> No, in fact, some of my Elisp would have to change, because it would
> >> no onger be sufficient to look at comment prefixes to know whether a
> >> line s code or comment.  Such a change would be very expensive in
> >> terms of the necessary human work that would ripple outward.  And
> >> your proposed variable would not obviate the need to account for both
> >> states in my code.
> >
> > In which way? Can you alaborate more please? You would not need to
> > write anything special. If you don’t wish to write literal code, then
> > just don’t. Prefix your comment lines with ’;’ as currently and don’t
> > care about it, Everything will work as it does now. I don’t see a
> > problem there. If I am wrong please explain.
>
> You don't seem to understand.  I am talking about existing code outside
> of emacs.git which parses Elisp, which would not be compatible with your
> proposed changes to Elisp syntax.
>
> >> This is the problem: as you have shown, parentheses commonly appear in
> >> prose, even at the beginning of lines.  Punctuation, such as semicolons,
> >> do not
> >> ; or, at least, should not
> >
> > Yes, and it is always possible to construct some obscure case where
> > things break.  Look for example at C or C++ and all undefined
> > behaviours in the standard. I mean, sure, there are always soem
> > obscurities, but do they matter? If one writes an article or a book or
> > some prosa and uses literal programming, and there is one explicit
> > case not allowed to use, how important is than to use exactly that
> > constrcut? I mean considering cons and pros, should one abandon the
> > entire idea because of one minor obscurity?
>
> Your own message showed how it happens in prose.  In fact, it's common
> enough and ambiguous enough that Elisp already forbids open-parens at
> column 0 in strings and requires them to be escaped.  How is this a
> minor obscurity?
>
> Don't you realize how important it is that syntax in a file format be
> unambiguous?
>
> > Well as I see literal programming it is to make it easier for humans
> > to mix code and prosa. What I proposedis just one small step further
> > in that direction.
>
> It is also a step toward ambiguity, churn, and bugs in other people's
> code, which you would not have to fix.
>
> > I don’t see how org makes it any different? In org you still have to
> > use both ’;’ and #+BEGIN_SRC - #+END_SRC markers.
>
> Have you used Org mode?  Have you looked at a literate program written
> in an Org file?  The point is that top-level comments are written
> between code blocks, without comment prefixes.
>
> > So it is even more clutter to eye.
>
> As I said in the message you're replying to:
>
>   You can even choose a face for org-meta-line that makes #+BEGIN_SRC
>   lines virtually (or literally!) disappear.
>
> > It is about being able to simple write and mix code and text. If it is
> > good or bad idea is up to individual preference. I personally find it
> > a cool thing, you seem to think it violates old ways,
>
> I have said nothing about old ways.  I explained exactly what the
> problems with the proposal are.  Please read my message carefully.
>
> > As a small reflection about us humans being slentrians, I have an
> > illustration.  Caves are still as good for living as they were 20 000
> > years ago. We can for sure make many cases against living in houses
> > such as not being natural, can crash on it’s inhabitants, cost
> > resources to construct etc etc. But yet, pros of liiving in houses
> > outweigh far the cons, and not many people today prefer to live in a
> > cave. Sorry maybe ti is a too contrived illustration. Anyway, if you
> > are good with ’;’ just continue to use it. But if a change like this
> > does not seem to cost lots in development terms, then why opposing if
> > somebody else find it cool. It is just that is more elegant and in a
> > sense cool thing to be able to do it without special tools as some
> > markers.
>
> It's not cool to imply that citing technical problems with your proposal
> is like being a cave-dweller.

Re: Sv: Sv: Christmas wish: Literate Elisp

[Stefan Monnier]

> I really understand the concern, but I must ask how would one change the
> elisp itself without re-implementing much or some of of elisp parser if it would
> be done externally?

I recommend you take a look at the existing function used on
load-source-file-function (i.e. load-with-code-conversion): it loads the
source file into a buffer and then calls `eval-buffer`.

So all it would take is for you to erase the text (aka non-code) part of
the source code before passing it to `eval-buffer` (but of course, only
do it when that source is using your new format).

Or instead of `eval-buffer` you might "manually" skip the text parts,
then use `read` + `eval` on the code parts.  This would probably take
a bit more effort in order to correctly handle (defvar foo).


        Stefan

Re: Sv: Sv: Christmas wish: Literate Elisp

[Eduardo Ochs]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

I was expecting that Arthur would come up with a preprocessor written
in (I guess) 30 lines of Elisp... a lot of people would like to play
with it, add tests, modify it, and so on - and we could use it to
learn how to create our own preprocessors.

A preprocessor written as patch to the C source of Emacs feels useless
to me.

  [[]],
    Eduardo Ochs
    http://angg.twu.net/#eev
    http://angg.twu.net/emacsconf2019.html

On Fri, 20 Dec 2019 at 13:18, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > I really understand the concern, but I must ask how would one change the
> > elisp itself without re-implementing much or some of of elisp parser if
> it would
> > be done externally?
>
> I recommend you take a look at the existing function used on
> load-source-file-function (i.e. load-with-code-conversion): it loads the
> source file into a buffer and then calls `eval-buffer`.
>
> So all it would take is for you to erase the text (aka non-code) part of
> the source code before passing it to `eval-buffer` (but of course, only
> do it when that source is using your new format).
>
> Or instead of `eval-buffer` you might "manually" skip the text parts,
> then use `read` + `eval` on the code parts.  This would probably take
> a bit more effort in order to correctly handle (defvar foo).
>
>
>         Stefan
>
>
>

[-- Attachment #2: Type: text/html, Size: 1859 bytes --]

Re: Christmas wish: Literate Elisp

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.


I'm very late to this thread, for which apologies.

My own package, lentic, could achieve this. It's gives you two
visualisations of the same file; so one with comment characters, and one
without. You can then do the literate part in one mode and the coding
part in another.

It already supports org-mode delination markers; it could be made to
support brackets as a delineator too.

Phil

Re: Christmas wish: Literate Elisp

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 6212 bytes --]

I've been watching this thread and resisting adding to it, but feel now
might be when I can contribute something useful.

This whole discussion seems to centre around some views that literate
programming for lisp is a positive way forward and therefore, making
changes to both C and Elisp parsing is a good idea if it means we can move
to a 'modern' literate programming model with greater ease.

As someone who has done literate programming for non-trivial projects (i.e.
something other than my .emacs file), I think this assumption needs to be
challenged. Literate programming has been around for a long time, but has
failed to gain traction. Why? Is it due to the existing tools or is it
something more fundamental?

I have been a fan of Donald Knuth's work for a long time. When I first came
across literate programming in the mid-90s, I thought it sounded really
interesting and a potentially great approach to writing code. A few of us
adopted this approach in our project and embarked on this new path of
discovery. Unfortunately, the reality just did not hold up to the promise.
Some of this was due to limitations in tooling (relating to the 'weave' and
'tangle' tools of the time), but a big part really just failed because
people don't move between prose and code that easily and we don't maintain
the prose as well as the code. In the end, you just have lots of
out-of-date prose mixed in with your code, which is often more misleading
than no prose at all.

The tools at the time were particularly poor when it came to working within
a REPL type environment. This was partly because they were a two step
process (tangling) and you could only go in one direction (getting chagnes
made in the REPL back into your source was distracting and often forgotten,
so work was frequently lost). Dealing with scope, namespaces etc was a pain
in a system which relied on 'tangling' of source files in order to get the
final output and nearly all supporting utilities (code formatters,
highlighting, linters etc) were broken.  Like others who have posted in
this thread, we thought it could all be improved if you could eliminate the
weaving/tangling and be free to just write prose within your code, so we
tried to do this with the lisp we were using (sorry, can't remember what
lisp flavor it was). At first glance, it looked very easy to do - a few
tweaks to the parser, some new reader macros and we should be fine. On one
level, it did fix some of the issues, but we did find lots of unforeseen
edge cases and it added sufficient additional complexity to the whole
system that in the end, we decided it just didn't deliver the benefits we
had hoped for.  At the end of the day, it wasn't the tools or the
implementation which killed it - it was simply that literate programming
simply didn't fit with how people like to code. It was one of those things
which sounds nice in theory, but for most, fails in practice.

We did find the approach works better for some languages than others. For
example, languages like C, where you have a code, compile, run, test cycle,
it wasn't oo bad. However, for languages where you often included a more
REPL driven style of devleopment, where lots of devleopment is done as
experimentation/refinement at the repl, the paradigm added very little and
more often than not, just got in the way.  We did find it was useful for
smaller tasks and configuration management, but for larger projects,
especially with teams of developers, it didn't stand up.

So, my advice, for what it is worth is

Implement a solution which does NOT require any modifications to either the
elisp parser or C code and use it to develop some non-trivial elisp
packages. You could probably base it on org-mode. My suspicion is that you
will find that after some time, it really doesn't deliver what you hoped.
However, if I'm wrong, it will provide the necessary experience and
familiarity to help guide a more refined and targeted model for working
with literate programming and elisp.

Don't under estimate the effort or impact making changes to either the
elisp parser or C sources will have. There has been a number of comments in
this thread which makes it sound like you would just need to 'tweak' the
parser and everything will work. That 'tweaking' is far more complicated
than it might seem on the surface. One of the main benefits of lisp
dialects is there simple syntax and the idea of code as data. Any changes
made to either the syntax or parsing of elisp is going to impact on lots of
things, many of which none of us have even thought about yet.

My feeling is that if literate programming is a great way forward, this
will be evident in the number of people who start using an elisp literate
programming mode. If this does turn out to be the case, then we can use the
experiences from many people who are using that mode to see what can be
done to develop the paradigm further. This may involve low level C or elsip
parser or syntax changes, but they will be based on experience and demand.

On Sat, 21 Dec 2019 at 03:56, Phillip Lord <phillip.lord@russet.org.uk>
wrote:

>
>
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> >> My proposal is to slightly change Elisp parser to treat lines that start
> >> with any other printable character but '(' as a start of comment and to
> >> simply ignore the line, just as it treats ';' as a comment.
> >
> > The `sexpresso` Haskell package follows the same idea ;-)
> >
> > As for using it in Elisp: I don't think there's anything stopping anyone
> > from making such a `literate-elisp-mode` and even arrange for `load` to
> > handle such a file (just like there is already a package that lets
> > `load` work directly on .org files).
> >
> > I'd welcome such a package in GNU ELPA.
>
>
> I'm very late to this thread, for which apologies.
>
> My own package, lentic, could achieve this. It's gives you two
> visualisations of the same file; so one with comment characters, and one
> without. You can then do the literate part in one mode and the coding
> part in another.
>
> It already supports org-mode delination markers; it could be made to
> support brackets as a delineator too.
>
> Phil
>
>

-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 7283 bytes --]

Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2455 bytes --]

> I was expecting that Arthur would come up with a preprocessor written
> in (I guess) 30 lines of Elisp...

Interesting 🙂. If you have red my previous mails to this list you
might have noticed that I am not an elisp guru. Stefan has
outlined two possible ways how this could be implemented in
Elisp, if you follow his advice I am sure you will have a working
solution you can present to us. I don't care how many lines of code
it will take.

> a lot of people would like to play
> with it, add tests, modify it, and so on

You still can my friend, and I encourage you to do and test if the
idea itself is valid. That was the purpose why I wrote patch. I clearly
stated it is just a prototype of an idea in order to test the soundness of
the idea. It takes two minutes to patch 3 files and rebuild Emacs. if you
wish to play with it. It is not like it cost some enormous mount of money
or labour.

> and we could use it to
> earn how to create our own preprocessors.

There are many sources in Emacs source code or elsewhere you can look
at if you need learning material.

>A preprocessor written as patch to the C source of Emacs feels useless
>to me.

The patch is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

🙂

  [[]],
    Eduardo Ochs
    http://angg.twu.net/#eev
    http://angg.twu.net/emacsconf2019.html

On Fri, 20 Dec 2019 at 13:18, Stefan Monnier <monnier@iro.umontreal.ca<mailto:monnier@iro.umontreal.ca>> wrote:
> I really understand the concern, but I must ask how would one change the
> elisp itself without re-implementing much or some of of elisp parser if it would
> be done externally?

I recommend you take a look at the existing function used on
load-source-file-function (i.e. load-with-code-conversion): it loads the
source file into a buffer and then calls `eval-buffer`.

So all it would take is for you to erase the text (aka non-code) part of
the source code before passing it to `eval-buffer` (but of course, only
do it when that source is using your new format).

Or instead of `eval-buffer` you might "manually" skip the text parts,
then use `read` + `eval` on the code parts.  This would probably take
a bit more effort in order to correctly handle (defvar foo).


        Stefan



[-- Attachment #2: Type: text/html, Size: 4352 bytes --]

Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 9998 bytes --]

Very interesting and nice read, thank you for taking your time Tim.

I have no idea if literate programmign is very good or not.I certainly
don't do it much. I never intended to present it as some kind of modern,
silver bullet or something. I hope I didn't sound like that. I am aware
that it is not the new idea either.

Personally I really like ORG-mode and ipossibility to mix languages
in one and same file. I was just looking at something I was working
with and thought it had so much noise around stuff that matters. I
wondered if it could be simplified. Then I got the idea that elisp could
actually be written without special markers, since in elisp we put
all code between () (with some exceptions). As I stated in my very first
mail where I asked if we could have this in Emacs, I am not sure myself
if this is somethign very usefull or just a cool gimmick. I personally can
see some nice use cases, but it might not be of value to somebody else.
I can maybe make a video where I play with it a bit after the hollidays it
it is interesting.

Yes, you are right about tools, checkers and so on. If we call it "literal elisp"
then one thing should be clear: literal elisp and elisp are not same. All
elisp is literal elisp, but all literal elsip would not be elisp. Relevance of
this should be obvious to anyone, all current tools work with elisp, not
literal elisp. So if you open a buffer and type comments without ';' in
front, of course, identation, font locking etc will be confused. This should
be self evident fact. Literal elisp would be a superset of elisp (with only
one extra feature - different handling of comments). This is regardless
of implementation language, just as elisp would be same language
regardless if interpreter is written in C or Rust or Elisp for that matter.

While yes, I have hacked C code, it does not mean I have altered how
Elisp works. Elisp is still the very same, so even with the patch all Elisp
should still work and parse the very same in Emacs and all external
tools as it does since the beginning of the time. Look at the patch, apply
it, run it. There is even boolean flag that eliminates that code at run time
if desired. I don't know how your lisp and your patches worked, not all
Lisps are created equal. It is interesting experience you share, and I do
understand what you mean and I agree with you. I am very conservative
myself when it comes to making changes, what is not broken should not
be repaired. I am also very pragmatic. I don't do changes just for changes
sake, I do it only if it does something useful.

Also, don't misstake me, I don't insist on change be done in C. I have done
those changes the way I did because I am new to Emacs internals hacking.
I am not new to programming, but I have no idea how they do in Emacs code,
what API calls are there and so on. I also have limited with time. So I have
hacked where Emacs self help functionality and debugger has thrown me in.

It turned to be some elisp and some C. I am sure it is possible to do this in
different ways, and if somebody is willing to implement some other way,
please go ahead. Of those two ideas that Stefan outlined, I am not willing
to implement them. First one is super brutish and inneficient, the second
one is sane but involves too much work. I don't have knowledge and time
to put into it. But I am sure it can be done.

________________________________
Från: Tim Cross <theophilusx@gmail.com>
Skickat: den 21 december 2019 02:16
Till: Phillip Lord <phillip.lord@russet.org.uk>
Kopia: arthur miller <arthur.miller@live.com>; emacs-devel <emacs-devel@gnu.org>
Ämne: Re: Christmas wish: Literate Elisp

I've been watching this thread and resisting adding to it, but feel now might be when I can contribute something useful.

This whole discussion seems to centre around some views that literate programming for lisp is a positive way forward and therefore, making changes to both C and Elisp parsing is a good idea if it means we can move to a 'modern' literate programming model with greater ease.

As someone who has done literate programming for non-trivial projects (i.e. something other than my .emacs file), I think this assumption needs to be challenged. Literate programming has been around for a long time, but has failed to gain traction. Why? Is it due to the existing tools or is it something more fundamental?

I have been a fan of Donald Knuth's work for a long time. When I first came across literate programming in the mid-90s, I thought it sounded really interesting and a potentially great approach to writing code. A few of us adopted this approach in our project and embarked on this new path of discovery. Unfortunately, the reality just did not hold up to the promise. Some of this was due to limitations in tooling (relating to the 'weave' and 'tangle' tools of the time), but a big part really just failed because people don't move between prose and code that easily and we don't maintain the prose as well as the code. In the end, you just have lots of out-of-date prose mixed in with your code, which is often more misleading than no prose at all.

The tools at the time were particularly poor when it came to working within a REPL type environment. This was partly because they were a two step process (tangling) and you could only go in one direction (getting chagnes made in the REPL back into your source was distracting and often forgotten, so work was frequently lost). Dealing with scope, namespaces etc was a pain in a system which relied on 'tangling' of source files in order to get the final output and nearly all supporting utilities (code formatters, highlighting, linters etc) were broken.  Like others who have posted in this thread, we thought it could all be improved if you could eliminate the weaving/tangling and be free to just write prose within your code, so we tried to do this with the lisp we were using (sorry, can't remember what lisp flavor it was). At first glance, it looked very easy to do - a few tweaks to the parser, some new reader macros and we should be fine. On one level, it did fix some of the issues, but we did find lots of unforeseen edge cases and it added sufficient additional complexity to the whole system that in the end, we decided it just didn't deliver the benefits we had hoped for.  At the end of the day, it wasn't the tools or the implementation which killed it - it was simply that literate programming simply didn't fit with how people like to code. It was one of those things which sounds nice in theory, but for most, fails in practice.

We did find the approach works better for some languages than others. For example, languages like C, where you have a code, compile, run, test cycle, it wasn't oo bad. However, for languages where you often included a more REPL driven style of devleopment, where lots of devleopment is done as experimentation/refinement at the repl, the paradigm added very little and more often than not, just got in the way.  We did find it was useful for smaller tasks and configuration management, but for larger projects, especially with teams of developers, it didn't stand up.

So, my advice, for what it is worth is

Implement a solution which does NOT require any modifications to either the elisp parser or C code and use it to develop some non-trivial elisp packages. You could probably base it on org-mode. My suspicion is that you will find that after some time, it really doesn't deliver what you hoped. However, if I'm wrong, it will provide the necessary experience and familiarity to help guide a more refined and targeted model for working with literate programming and elisp.

Don't under estimate the effort or impact making changes to either the elisp parser or C sources will have. There has been a number of comments in this thread which makes it sound like you would just need to 'tweak' the parser and everything will work. That 'tweaking' is far more complicated than it might seem on the surface. One of the main benefits of lisp dialects is there simple syntax and the idea of code as data. Any changes made to either the syntax or parsing of elisp is going to impact on lots of things, many of which none of us have even thought about yet.

My feeling is that if literate programming is a great way forward, this will be evident in the number of people who start using an elisp literate programming mode. If this does turn out to be the case, then we can use the experiences from many people who are using that mode to see what can be done to develop the paradigm further. This may involve low level C or elsip parser or syntax changes, but they will be based on experience and demand.

On Sat, 21 Dec 2019 at 03:56, Phillip Lord <phillip.lord@russet.org.uk<mailto:phillip.lord@russet.org.uk>> wrote:


Stefan Monnier <monnier@iro.umontreal.ca<mailto:monnier@iro.umontreal.ca>> writes:

>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.


I'm very late to this thread, for which apologies.

My own package, lentic, could achieve this. It's gives you two
visualisations of the same file; so one with comment characters, and one
without. You can then do the literate part in one mode and the coding
part in another.

It already supports org-mode delination markers; it could be made to
support brackets as a delineator too.

Phil



--
regards,

Tim

--
Tim Cross


[-- Attachment #2: Type: text/html, Size: 19717 bytes --]

Re: Sv: Sv: Christmas wish: Literate Elisp

[Eduardo Ochs]

[-- Attachment #1: Type: text/plain, Size: 2885 bytes --]

Sorry, I should have written a longer message, with most, or all,
parts of it written in the first person...

In my experience hanging out at the #emacs channel at Freenode and
asking for help or hints on elisp is fun, and it's easy to test ideas
interactively with C-x C-e... asking for help on the C side of Emacs
is not so fun - patching the C source is considered arcane knowledge
(<- my impression!) and fewer people will be able to discuss that and
to help... also, I don't know how to recompile Emacs after a change in
a .c and restart it in less that 30 seconds (slow machine here - I use
an old ThinkPad with LibreBoot) - in fact I don't think I've ever
patched the C source before! Here's the script that I used to test
Arthur's patch:

  # Download the attachments of this message:
  #   https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00556.html
  # in /tmp/:
  cd /tmp/
  wget -O bytecomp.patch
https://lists.gnu.org/archive/html/emacs-devel/2019-12/txtkIKP0OSag7.txt
  wget -O lread.patch
https://lists.gnu.org/archive/html/emacs-devel/2019-12/txttFA8rLAthl.txt
  wget -O chatty.el
https://lists.gnu.org/archive/html/emacs-devel/2019-12/txt6f3pEXMWNI.txt

  # Apply the patches in /tmp/:
  cd /tmp/
  cp -v ~/bigsrc/emacs/src/lread.c                 /tmp/
  cp -v ~/bigsrc/emacs/lisp/emacs-lisp/bytecomp.el /tmp/
  # (find-man "1 patch")
  patch lread.c     lread.patch
  patch bytecomp.el bytecomp.patch

  # If they applied cleanly, copy the patched files back:
  cp -v /tmp/lread.c     ~/bigsrc/emacs/src/lread.c
  cp -v /tmp/bytecomp.el ~/bigsrc/emacs/lisp/emacs-lisp/bytecomp.el

  # Rebuild Emacs.
  # I usually skip some of the steps below - my way of executing
  # these scripts line by line from Emacs is explained here:
  #   http://angg.twu.net/emacsconf2019.html
  #   http://angg.twu.net/eev-intros/find-eev-quick-intro.html#6

  rm -Rfv  ~/bigsrc/emacs/
  mkdir -p ~/bigsrc/emacs/
  cd       ~/bigsrc/
  git clone git://git.sv.gnu.org/emacs

  cd       ~/bigsrc/emacs/
  time ./autogen.sh   2>&1 | tee oa
  time ./configure    2>&1 | tee oc
  time make bootstrap 2>&1 | tee omb
  time make           2>&1 | tee om

  # Run the new Emacs:
  ~/bigsrc/emacs/src/emacs


Cheers,
  Eduardo Ochs
  http://angg.twu.net/#eev


On Fri, 20 Dec 2019 at 22:18, arthur miller <arthur.miller@live.com> wrote:

> > I was expecting that Arthur would come up with a preprocessor written
> > in (I guess) 30 lines of Elisp...
>
> Interesting 🙂. If you have red my previous mails to this list you
> might have noticed that I am not an elisp guru. Stefan has
> outlined two possible ways how this could be implemented in
> Elisp, if you follow his advice I am sure you will have a working
> solution you can present to us. I don't care how many lines of code
> it will take.
>
> (...)
>

[-- Attachment #2: Type: text/html, Size: 4798 bytes --]

Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2862 bytes --]


> In my experience hanging out at the #emacs channel at Freenode and
> asking for help or hints on elisp is fun, and it's easy to test ideas
> interactively with C-x C-e...

Indeed, completely agree!

> I don't know how to recompile Emacs after a change in
> a .c and restart it in less that 30 seconds

I don't know either 🙂. It take me about 2 - 3 minutes every time to recompile.
I have my emacs src from github and I just run make -j4 after a change. I don't
run full build. I then run emacs from src folder with -Q flag and test file as
argument.

>  wget -O bytecomp.patch https://lists.gnu.org/archive/html/emacs-devel/2019-12/txtkIKP0OSag7.txt
>  wget -O lread.patch    https://lists.gnu.org/archive/html/emacs-devel/2019-12/txttFA8rLAthl.txt
> wget -O chatty.el      https://lists.gnu.org/archive/html/emacs-devel/2019-12/txt6f3pEXMWNI.txt

You will also need simple.patch. Contains a variable to turn off/on parser in eval so you
can just eval-buffer or eval-region.

Cool if you test! I would be glad to hear opinions on the idea of literal elisp. (Please I am aware of opinions on
implementation 🙂).


  # Apply the patches in /tmp/:
  cd /tmp/
  cp -v ~/bigsrc/emacs/src/lread.c                 /tmp/
  cp -v ~/bigsrc/emacs/lisp/emacs-lisp/bytecomp.el /tmp/
  # (find-man "1 patch")
  patch lread.c     lread.patch
  patch bytecomp.el bytecomp.patch

  # If they applied cleanly, copy the patched files back:
  cp -v /tmp/lread.c     ~/bigsrc/emacs/src/lread.c
  cp -v /tmp/bytecomp.el ~/bigsrc/emacs/lisp/emacs-lisp/bytecomp.el

  # Rebuild Emacs.
  # I usually skip some of the steps below - my way of executing
  # these scripts line by line from Emacs is explained here:
  #   http://angg.twu.net/emacsconf2019.html
  #   http://angg.twu.net/eev-intros/find-eev-quick-intro.html#6

  rm -Rfv  ~/bigsrc/emacs/
  mkdir -p ~/bigsrc/emacs/
  cd       ~/bigsrc/
  git clone git://git.sv.gnu.org/emacs<http://git.sv.gnu.org/emacs>

  cd       ~/bigsrc/emacs/
  time ./autogen.sh   2>&1 | tee oa
  time ./configure    2>&1 | tee oc
  time make bootstrap 2>&1 | tee omb
  time make           2>&1 | tee om

  # Run the new Emacs:
  ~/bigsrc/emacs/src/emacs


Cheers,
  Eduardo Ochs
  http://angg.twu.net/#eev


On Fri, 20 Dec 2019 at 22:18, arthur miller <arthur.miller@live.com<mailto:arthur.miller@live.com>> wrote:
> I was expecting that Arthur would come up with a preprocessor written
> in (I guess) 30 lines of Elisp...

Interesting 🙂. If you have red my previous mails to this list you
might have noticed that I am not an elisp guru. Stefan has
outlined two possible ways how this could be implemented in
Elisp, if you follow his advice I am sure you will have a working
solution you can present to us. I don't care how many lines of code
it will take.

(...)

[-- Attachment #2: Type: text/html, Size: 5858 bytes --]

Re: Christmas wish: Literate Elisp

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 12160 bytes --]

I certainly don't want to discourage you from experimenting and trying out
ideas - this is what makes Emacs great. What I would encourage is, in this
case, using org-mode to develop something non-trivial (possibly your
literate elsip idea) as it will give you the elisp experience you need and
more importantly, a taste of a form of literate programming, which will
both highlight the limitations of the current implementation (ie.e.
org-mode) and possibly, the literate programming approach itself. The key
point is that it needs to be non-trivial. Most of the limitations I found
with literate programming really only become evident in larger projects.
For smaller things, it works great and it is how I maintain my own init.el
file. It is only when you do something more complex that weaknesses become
evident.

I would also highly recommend re-visiting Stepan's posts after you have
gained some more elisp experience. Stephan is one of the most experienced
and knowledgeable contributors on this list. His advice and guidance are
valuable and should be considered carefully. While it may not be obvious
why he is making a recommendation, you can be fairly confident it is based
on experience and deep knowledge of Emacs. Spending the time to analyze his
suggestions and looking into them in detail is likely to save you a lot of
time going down dead ends or having to re-design/re-factor things as you
discover them the hard way.

I hope you don't become too discouraged and do work at implementing your
idea. At the very least, you may be able to find ways to improve or enhance
org-mode, which is always a valuable contribution.

On Sat, 21 Dec 2019 at 15:24, arthur miller <arthur.miller@live.com> wrote:

> Very interesting and nice read, thank you for taking your time Tim.
>
> I have no idea if literate programmign is very good or not.I certainly
> don't do it much. I never intended to present it as some kind of modern,
> silver bullet or something. I hope I didn't sound like that. I am aware
> that it is not the new idea either.
>
> Personally I really like ORG-mode and ipossibility to mix languages
> in one and same file. I was just looking at something I was working
> with and thought it had so much noise around stuff that matters. I
> wondered if it could be simplified. Then I got the idea that elisp could
> actually be written without special markers, since in elisp we put
> all code between () (with some exceptions). As I stated in my very first
> mail where I asked if we could have this in Emacs, I am not sure myself
> if this is somethign very usefull or just a cool gimmick. I personally can
> see some nice use cases, but it might not be of value to somebody else.
> I can maybe make a video where I play with it a bit after the hollidays it
> it is interesting.
>
> Yes, you are right about tools, checkers and so on. If we call it "literal
> elisp"
> then one thing should be clear: literal elisp and elisp are not same. All
> elisp is literal elisp, but all literal elsip would not be elisp.
> Relevance of
> this should be obvious to anyone, all current tools work with elisp, not
> literal elisp. So if you open a buffer and type comments without ';' in
> front, of course, identation, font locking etc will be confused. This
> should
> be self evident fact. Literal elisp would be a superset of elisp (with only
> one extra feature - different handling of comments). This is regardless
> of implementation language, just as elisp would be same language
> regardless if interpreter is written in C or Rust or Elisp for that
> matter.
>
> While yes, I have hacked C code, it does not mean I have altered how
> Elisp works. Elisp is still the very same, so even with the patch all Elisp
> should still work and parse the very same in Emacs and all external
> tools as it does since the beginning of the time. Look at the patch, apply
> it, run it. There is even boolean flag that eliminates that code at run
> time
> if desired. I don't know how your lisp and your patches worked, not all
> Lisps are created equal. It is interesting experience you share, and I do
> understand what you mean and I agree with you. I am very conservative
> myself when it comes to making changes, what is not broken should not
> be repaired. I am also very pragmatic. I don't do changes just for
> changes
> sake, I do it only if it does something useful.
>
> Also, don't misstake me, I don't insist on change be done in C. I have done
> those changes the way I did because I am new to Emacs internals hacking.
> I am not new to programming, but I have no idea how they do in Emacs code,
> what API calls are there and so on. I also have limited with time. So I
> have
> hacked where Emacs self help functionality and debugger has thrown me in.
>
> It turned to be some elisp and some C. I am sure it is possible to do this
> in
> different ways, and if somebody is willing to implement some other way,
> please go ahead. Of those two ideas that Stefan outlined, I am not willing
> to implement them. First one is super brutish and inneficient, the second
> one is sane but involves too much work. I don't have knowledge and time
> to put into it. But I am sure it can be done.
>
> ------------------------------
> *Från:* Tim Cross <theophilusx@gmail.com>
> *Skickat:* den 21 december 2019 02:16
> *Till:* Phillip Lord <phillip.lord@russet.org.uk>
> *Kopia:* arthur miller <arthur.miller@live.com>; emacs-devel <
> emacs-devel@gnu.org>
> *Ämne:* Re: Christmas wish: Literate Elisp
>
> I've been watching this thread and resisting adding to it, but feel now
> might be when I can contribute something useful.
>
> This whole discussion seems to centre around some views that literate
> programming for lisp is a positive way forward and therefore, making
> changes to both C and Elisp parsing is a good idea if it means we can move
> to a 'modern' literate programming model with greater ease.
>
> As someone who has done literate programming for non-trivial projects
> (i.e. something other than my .emacs file), I think this assumption needs
> to be challenged. Literate programming has been around for a long time, but
> has failed to gain traction. Why? Is it due to the existing tools or is it
> something more fundamental?
>
> I have been a fan of Donald Knuth's work for a long time. When I first
> came across literate programming in the mid-90s, I thought it sounded
> really interesting and a potentially great approach to writing code. A few
> of us adopted this approach in our project and embarked on this new path of
> discovery. Unfortunately, the reality just did not hold up to the promise.
> Some of this was due to limitations in tooling (relating to the 'weave' and
> 'tangle' tools of the time), but a big part really just failed because
> people don't move between prose and code that easily and we don't maintain
> the prose as well as the code. In the end, you just have lots of
> out-of-date prose mixed in with your code, which is often more misleading
> than no prose at all.
>
> The tools at the time were particularly poor when it came to working
> within a REPL type environment. This was partly because they were a two
> step process (tangling) and you could only go in one direction (getting
> chagnes made in the REPL back into your source was distracting and often
> forgotten, so work was frequently lost). Dealing with scope, namespaces etc
> was a pain in a system which relied on 'tangling' of source files in order
> to get the final output and nearly all supporting utilities (code
> formatters, highlighting, linters etc) were broken.  Like others who have
> posted in this thread, we thought it could all be improved if you could
> eliminate the weaving/tangling and be free to just write prose within your
> code, so we tried to do this with the lisp we were using (sorry, can't
> remember what lisp flavor it was). At first glance, it looked very easy to
> do - a few tweaks to the parser, some new reader macros and we should be
> fine. On one level, it did fix some of the issues, but we did find lots of
> unforeseen edge cases and it added sufficient additional complexity to the
> whole system that in the end, we decided it just didn't deliver the
> benefits we had hoped for.  At the end of the day, it wasn't the tools or
> the implementation which killed it - it was simply that literate
> programming simply didn't fit with how people like to code. It was one of
> those things which sounds nice in theory, but for most, fails in practice.
>
> We did find the approach works better for some languages than others. For
> example, languages like C, where you have a code, compile, run, test cycle,
> it wasn't oo bad. However, for languages where you often included a more
> REPL driven style of devleopment, where lots of devleopment is done as
> experimentation/refinement at the repl, the paradigm added very little and
> more often than not, just got in the way.  We did find it was useful for
> smaller tasks and configuration management, but for larger projects,
> especially with teams of developers, it didn't stand up.
>
> So, my advice, for what it is worth is
>
> Implement a solution which does NOT require any modifications to either
> the elisp parser or C code and use it to develop some non-trivial elisp
> packages. You could probably base it on org-mode. My suspicion is that you
> will find that after some time, it really doesn't deliver what you hoped.
> However, if I'm wrong, it will provide the necessary experience and
> familiarity to help guide a more refined and targeted model for working
> with literate programming and elisp.
>
> Don't under estimate the effort or impact making changes to either the
> elisp parser or C sources will have. There has been a number of comments in
> this thread which makes it sound like you would just need to 'tweak' the
> parser and everything will work. That 'tweaking' is far more complicated
> than it might seem on the surface. One of the main benefits of lisp
> dialects is there simple syntax and the idea of code as data. Any changes
> made to either the syntax or parsing of elisp is going to impact on lots of
> things, many of which none of us have even thought about yet.
>
> My feeling is that if literate programming is a great way forward, this
> will be evident in the number of people who start using an elisp literate
> programming mode. If this does turn out to be the case, then we can use the
> experiences from many people who are using that mode to see what can be
> done to develop the paradigm further. This may involve low level C or elsip
> parser or syntax changes, but they will be based on experience and demand.
>
> On Sat, 21 Dec 2019 at 03:56, Phillip Lord <phillip.lord@russet.org.uk>
> wrote:
>
>
>
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> >> My proposal is to slightly change Elisp parser to treat lines that start
> >> with any other printable character but '(' as a start of comment and to
> >> simply ignore the line, just as it treats ';' as a comment.
> >
> > The `sexpresso` Haskell package follows the same idea ;-)
> >
> > As for using it in Elisp: I don't think there's anything stopping anyone
> > from making such a `literate-elisp-mode` and even arrange for `load` to
> > handle such a file (just like there is already a package that lets
> > `load` work directly on .org files).
> >
> > I'd welcome such a package in GNU ELPA.
>
>
> I'm very late to this thread, for which apologies.
>
> My own package, lentic, could achieve this. It's gives you two
> visualisations of the same file; so one with comment characters, and one
> without. You can then do the literate part in one mode and the coding
> part in another.
>
> It already supports org-mode delination markers; it could be made to
> support brackets as a delineator too.
>
> Phil
>
>
>
> --
> regards,
>
> Tim
>
> --
> Tim Cross
>
>

-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 21330 bytes --]

Re: Christmas wish: Literate Elisp

[VanL]

Tim Cross <theophilusx@gmail.com> writes:

> It is only when you do something more complex that weaknesses become
> evident.

> On Sat, 21 Dec 2019 at 15:24, arthur miller <arthur.miller@live.com> wrote:
>
>  I have no idea if literate programmign is very good or not.I certainly 
>  don't do it much.

>  I am not sure myself if this is somethign very usefull or just a cool
>  gimmick. I personally can see some nice use cases,

>  I am very conservative myself when it comes to making changes, what
>  is not broken should not be repaired. I am also very pragmatic. I
>  don't do changes just for changes sake, I do it only if it does
>  something useful.

On Reddit¹, UselessCodeMonkey mentions two of the best development
teams at writing software, rated five stars.  Learnings from their
practices is worth the while to apply if they're not proprietary and
offlimits.

Footnotes: 
¹  https://www.reddit.com/r/spacex/comments/edavxw/boeing_starliner_suffers_offnominal_insertion/

-- 
VanL.,
 əə0@ 🐞 7 6 4 5
 一 二 三 言 語 𝔖 元 示 証 明 海 自 己 漢 本 人 Gnus/Emacs (berkeley-unix)

Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2890 bytes --]

🙂 👍

I have certainly no access to a spaceship 🙁. Unfortunately! If
anyone is friends with Elon and can borough one I would
certainly love to test my patch to control a spaceship with it!

Jokes aside, I have seen those threads myself and have read
about Boeings misstakes already. I am also aware about Ariane
5 and Xenobia (which I dived last year) and so on.

Yes Tim, Stefan is great guy here, I completely agree with you,
I have also been watching this list for years, even if I never
post here (I did it once before), as well as Eli Z, RMS
and lots of other here. I value their opinion a lot, and I value
everybody else's constructive input, such as yours.

When it comes to experimenting, that was exactly what I did.
I came with idea and I hacked it to see if it is possible. I never
suggested it was finnished patch to be included in a final
product.  I didn't even suggest I would be the one to implement
it, I am not an Emacs dev and I certainly have not much
knowledge about what and how they do. Anyway patch is
simple and there for experimentation, anyone can build
a side copy of Emacs and test this if they are curious. It didn't
blow up my computer, and while I can't promise, I certainly do
hope it won't blow up other peoples computers either, but be
careful! 🙂.

Anyway, if it is gonna be an Elisp solution or C or whatever
language or none solution at all, is left as a homework for
interested developer(s).



________________________________
Från: Emacs-devel <emacs-devel-bounces+arthur.miller=live.com@gnu.org> för VanL <van@scratch.space>
Skickat: den 21 december 2019 10:39
Till: emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: Christmas wish: Literate Elisp

Tim Cross <theophilusx@gmail.com> writes:

> It is only when you do something more complex that weaknesses become
> evident.

> On Sat, 21 Dec 2019 at 15:24, arthur miller <arthur.miller@live.com> wrote:
>
>  I have no idea if literate programmign is very good or not.I certainly
>  don't do it much.

>  I am not sure myself if this is somethign very usefull or just a cool
>  gimmick. I personally can see some nice use cases,

>  I am very conservative myself when it comes to making changes, what
>  is not broken should not be repaired. I am also very pragmatic. I
>  don't do changes just for changes sake, I do it only if it does
>  something useful.

On Reddit¹, UselessCodeMonkey mentions two of the best development
teams at writing software, rated five stars.  Learnings from their
practices is worth the while to apply if they're not proprietary and
offlimits.

Footnotes:
¹  https://www.reddit.com/r/spacex/comments/edavxw/boeing_starliner_suffers_offnominal_insertion/

--
VanL.,
 əə0@ 🐞 7 6 4 5
 一 二 三 言 語 𝔖 元 示 証 明 海 自 己 漢 本 人 Gnus/Emacs (berkeley-unix)



[-- Attachment #2: Type: text/html, Size: 7836 bytes --]

Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 3740 bytes --]

As info; my bytecomp patch has bug, don't use it
(just don't bytecompile) when you test, last version
won't produce .elc file. I don't have time to fix it today.

C patch works fine (with eval-buffer or eval-region ).

/a
________________________________
Från: arthur miller <arthur.miller@live.com>
Skickat: den 21 december 2019 07:02
Till: Eduardo Ochs <eduardoochs@gmail.com>
Ämne: Sv: Sv: Sv: Christmas wish: Literate Elisp

Here are all 4 files again with simple.patch for lisp/simple.el.

Hope it works for you.

cheers
/a
________________________________
Från: Eduardo Ochs <eduardoochs@gmail.com>
Skickat: den 21 december 2019 06:24
Till: arthur miller <arthur.miller@live.com>
Kopia: emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

Sorry, I should have written a longer message, with most, or all,
parts of it written in the first person...

In my experience hanging out at the #emacs channel at Freenode and
asking for help or hints on elisp is fun, and it's easy to test ideas
interactively with C-x C-e... asking for help on the C side of Emacs
is not so fun - patching the C source is considered arcane knowledge
(<- my impression!) and fewer people will be able to discuss that and
to help... also, I don't know how to recompile Emacs after a change in
a .c and restart it in less that 30 seconds (slow machine here - I use
an old ThinkPad with LibreBoot) - in fact I don't think I've ever
patched the C source before! Here's the script that I used to test
Arthur's patch:

  # Download the attachments of this message:
  #   https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00556.html
  # in /tmp/:
  cd /tmp/
  wget -O bytecomp.patch https://lists.gnu.org/archive/html/emacs-devel/2019-12/txtkIKP0OSag7.txt
  wget -O lread.patch    https://lists.gnu.org/archive/html/emacs-devel/2019-12/txttFA8rLAthl.txt
  wget -O chatty.el      https://lists.gnu.org/archive/html/emacs-devel/2019-12/txt6f3pEXMWNI.txt

  # Apply the patches in /tmp/:
  cd /tmp/
  cp -v ~/bigsrc/emacs/src/lread.c                 /tmp/
  cp -v ~/bigsrc/emacs/lisp/emacs-lisp/bytecomp.el /tmp/
  # (find-man "1 patch")
  patch lread.c     lread.patch
  patch bytecomp.el bytecomp.patch

  # If they applied cleanly, copy the patched files back:
  cp -v /tmp/lread.c     ~/bigsrc/emacs/src/lread.c
  cp -v /tmp/bytecomp.el ~/bigsrc/emacs/lisp/emacs-lisp/bytecomp.el

  # Rebuild Emacs.
  # I usually skip some of the steps below - my way of executing
  # these scripts line by line from Emacs is explained here:
  #   http://angg.twu.net/emacsconf2019.html
  #   http://angg.twu.net/eev-intros/find-eev-quick-intro.html#6

  rm -Rfv  ~/bigsrc/emacs/
  mkdir -p ~/bigsrc/emacs/
  cd       ~/bigsrc/
  git clone git://git.sv.gnu.org/emacs<http://git.sv.gnu.org/emacs>

  cd       ~/bigsrc/emacs/
  time ./autogen.sh   2>&1 | tee oa
  time ./configure    2>&1 | tee oc
  time make bootstrap 2>&1 | tee omb
  time make           2>&1 | tee om

  # Run the new Emacs:
  ~/bigsrc/emacs/src/emacs


Cheers,
  Eduardo Ochs
  http://angg.twu.net/#eev


On Fri, 20 Dec 2019 at 22:18, arthur miller <arthur.miller@live.com<mailto:arthur.miller@live.com>> wrote:
> I was expecting that Arthur would come up with a preprocessor written
> in (I guess) 30 lines of Elisp...

Interesting 🙂. If you have red my previous mails to this list you
might have noticed that I am not an elisp guru. Stefan has
outlined two possible ways how this could be implemented in
Elisp, if you follow his advice I am sure you will have a working
solution you can present to us. I don't care how many lines of code
it will take.

(...)

[-- Attachment #2: Type: text/html, Size: 8494 bytes --]

Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

[arthur miller]

[-- Attachment #1.1: Type: text/plain, Size: 3101 bytes --]

My 2nd patch was wrong as I noticed too late yesterady, sorry for that.
Unfortunately I didn't have time to fix it. Here is one that works (at least
in my emacs), if any one would like to test the idea and play with idea of
using literal elisp. It is not only about being able to omit ';' in comments.

The idea is that normally in elisp, compiler skips comment-lines and
evaluate everything else. Here compiler evaluates code-lines and skips
everythign else. While it might seems like a same thing, it is not! There is
a certain inversion and slight difference because of that apparently subtle
difference. For instance, sine but code is ignored, this lets one use
org-mode as a kind of annotation tool or organisational tool for the
code. Normally the parser would choke and produced  error
messages if it was feeded with org headings and so n.

Another possibility is to maybe be able use elisp as a template or
annotation language or code generator for some other languages. For
example one could write a C code and use elisp macros or code
generators which expand to C after eval-buffer, kind -of what org-mode
does, but it would maybe make it easier or open some more possibilites
if one combines org-mode with directly evaluable elisp. Don't know, haven't
had time yet.

Anyway, I hope you can give it a try and discuss the idea not the
implementeation, implementation is just a sketch to give you something
to play with. It is unnecessary to theorycraft essays about all kind of
space problems before one tries it, in my opinion.

Cheers
________________________________
Från: arthur miller <arthur.miller@live.com>
Skickat: den 21 december 2019 17:03
Till: Stefan Monnier <monnier@iro.umontreal.ca>
Ämne: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp


> I personally would oppose these few-lines of new code in lread.c at
> least until this new feature is properly supported in the rest of the
> Elisp-manipulating tools we have (starting with the byte-compiler, but
> including checkdoc, autoload, ...) and until there's some clear evidence
> that it's useful (i.e. used by more than one person).

Completely agree on that one.

Unfortunately the time and effort and my struggle to just patch
bytecompiler.el to do this tells me I am not the right person to jump on
implementing "the skip" approach you suggest. Maybe some code, or
at least approach used in bytecompiler and eval can be even reused for
this, but to me personally it would take too much time and effort.

The other approach of copying file to temp buffer and modifying the file
is certainly less jobb and easier to implement. I also agree that it is suitable
for small toys and demo examples. It couldn't be difficult to implement, one
would need to keep track of balanced paranthesis + some few other small
things which isn't very difficult. While I can certainly hack elisp to a degree,
I am still not the right person to implement something you would distribute in
Emacs anyway, so if somebod else is interesting to implement this approach
it would be great.



[-- Attachment #1.2: Type: text/html, Size: 8056 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: lread.patch --]
[-- Type: text/x-patch; name="lread.patch", Size: 1067 bytes --]

--- lread.c	2019-12-20 01:07:33.264089317 +0100
+++ mylread.c	2019-12-22 07:35:31.479305940 +0100
@@ -1955,6 +1955,8 @@
   /* True on the first time around.  */
   bool first_sexp = 1;
   Lisp_Object macroexpand = intern ("internal-macroexpand-for-load");
+  Lisp_Object litcode = intern ("emacs-lisp-allow-literal-comments");
+  Lisp_Object allow_literal_comments = find_symbol_value(litcode);
 
   if (NILP (Ffboundp (macroexpand))
       || (STRINGP (sourcename) && suffix_p (sourcename, ".elc")))
@@ -2053,6 +2055,20 @@
 	  || c == NO_BREAK_SPACE)
 	goto read_next;
 
+      if (EQ (allow_literal_comments, Qt))
+        {
+          if (c != '(')
+            {
+              if(c == '#' || c == '\'' || c == '\`') c = READCHAR;
+              if(c == '(') UNREAD(c);
+              else
+                {
+                  while ((c = READCHAR) != '\n' && c != -1);
+                  goto read_next;
+                }
+            }
+        }
+
       if (! HASH_TABLE_P (read_objects_map)
 	  || XHASH_TABLE (read_objects_map)->count)
 	read_objects_map

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: bytecomp.patch --]
[-- Type: text/x-patch; name="bytecomp.patch", Size: 3106 bytes --]

--- bytecomp.el	2019-12-20 01:05:44.575414147 +0100
+++ mybytecomp.el	2019-12-22 07:35:59.509753178 +0100
@@ -154,6 +154,14 @@
   :type '(choice (const nil) function)
   :version "23.2")
 
+(defcustom byte-compile-allow-literal-comments nil
+  "If not nil, byte compiler will allow you to type
+   top-level comments without using semicolon in emacs lisp."
+  :group 'bytecomp
+  :type 'boolean
+  :safe #'booleanp
+  :version "27.1")
+
 ;; This enables file name handlers such as jka-compr
 ;; to remove parts of the file name that should not be copied
 ;; through to the output file name.
@@ -2079,8 +2087,8 @@
 	(print-level nil)
 	;; Prevent edebug from interfering when we compile
 	;; and put the output into a file.
-;; 	(edebug-all-defs nil)
-;; 	(edebug-all-forms nil)
+        ;; 	(edebug-all-defs nil)
+        ;; 	(edebug-all-forms nil)
 	;; Simulate entry to byte-compile-top-level
         (byte-compile-jump-tables nil)
         (byte-compile-constants nil)
@@ -2127,19 +2135,35 @@
 			       (= (following-char) ?\;))
 		   (forward-line 1))
 		 (not (eobp)))
-	  (setq byte-compile-read-position (point)
+
+          (setq byte-compile-read-position (point)
 		byte-compile-last-position byte-compile-read-position)
-          (let* ((lread--unescaped-character-literals nil)
-                 (form (read inbuffer))
-                 (warning (byte-run--unescaped-character-literals-warning)))
-            (when warning (byte-compile-warn "%s" warning))
-	    (byte-compile-toplevel-file-form form)))
-	;; Compile pending forms at end of file.
-	(byte-compile-flush-pending)
-	;; Make warnings about unresolved functions
-	;; give the end of the file as their position.
-	(setq byte-compile-last-position (point-max))
-	(byte-compile-warn-about-unresolved-functions))
+
+	  (if byte-compile-allow-literal-comments
+              (progn
+                (if (= (following-char) ?\()
+                    (progn
+
+                      (let* ((lread--unescaped-character-literals nil)
+                             (form (read inbuffer))
+                             (warning (byte-run--unescaped-character-literals-warning)))
+                        (when warning (byte-compile-warn "%s" warning))
+	                (byte-compile-toplevel-file-form form)))
+                  (forward-line 1)))
+            (progn
+
+              (let* ((lread--unescaped-character-literals nil)
+                     (form (read inbuffer))
+                     (warning (byte-run--unescaped-character-literals-warning)))
+                (when warning (byte-compile-warn "%s" warning))
+	        (byte-compile-toplevel-file-form form)))))
+
+        ;; Compile pending forms at end of file.
+        (byte-compile-flush-pending)
+        ;; Make warnings about unresolved functions
+        ;; give the end of the file as their position.
+        (setq byte-compile-last-position (point-max))
+        (byte-compile-warn-about-unresolved-functions))
       ;; Fix up the header at the front of the output
       ;; if the buffer contains multibyte characters.
       (and byte-compile-current-file

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: simple.patch --]
[-- Type: text/x-patch; name="simple.patch", Size: 479 bytes --]

--- simple.el	2019-12-20 01:06:16.849343961 +0100
+++ mysimple.el	2019-12-20 01:03:05.212467674 +0100
@@ -127,6 +127,14 @@
   :safe #'booleanp
   :version "27.1")
 
+(defcustom emacs-lisp-allow-literal-comments nil
+  "If not nil, elisp will allow you to type in top-level comments
+   without using semicolon."
+  :group 'lisp
+  :type 'boolean
+  :safe #'booleanp
+  :version "27.1")
+
 (defvar next-error-highlight-timer nil)
 
 (defvar next-error-overlay-arrow-position nil)

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: chatty.el --]
[-- Type: text/x-emacs-lisp; name="chatty.el", Size: 720 bytes --]

;; -*- mode: org; -*-

Elisp now alows comments without ;.

' can also use start literal comments with
# also works

* Some elisp here
(setq emacs-lisp-allow-literal-comments t)
(setq byte-compile-allow-literal-comments t)
(message "I can now run simple elisp on my own!")

* More stuff here
#+BEGIN_SRC emacs-lisp
(message "Hello, World!")
#+END_SRC

It is of course possible to intertwene code
with comments anywhere in the file.

(message "Hello Again!")

Of course we can also comment-out code
just as before:

;;(message "I am silent")
In code blocs the ';' is still a comment delimiter:
(message
 ;; Here is a line comment
 "I am a bit chatty today!")

That's it for today folks!

(message "Bye bye cruel world!")