[Pre-PATCH] Add new :lexical header argument

[Pre-PATCH] Add new :lexical header argument

[Evgenii Klimov]

[-- Attachment #1: Type: text/plain, Size: 1053 bytes --]

Hi, here I propose new header argument to enable scope change
(lexical/dynamic binding) of the tangled file.  We have :shebang header
argument and the new one behaves similarly.

If you like the idea I think we should discuss the following:
- should we allow to set it for non-elisp blocks?  e.g. :shebang is not
  restricted and `elisp-enable-lexical-binding' is smart enough to use
  the comment symbol appropriate to the context. 
- should it be :lexical or :scope.  :lexical is long to type, but boolean
  and clear; scope is shorter, but the user will have to type
  "dynamic/lexical" or "dyn/lex".
- should we add the third option of the argument to explicitly forbid
  lexical binding?  E.g. if we a have source block that rely on the
  dynamic binding but it could be used in another block with lexical
  binding via noweb. I'm not sure that it's worth the effort.
- which default value to choose?  Currently if you tangle it would be
  dynamic, but lexical binding is the future.

After all I'll add tests and update Changelog/Docs if needed.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-tangle.el-Add-new-lexical-header-argument.patch --]
[-- Type: text/x-diff, Size: 3158 bytes --]

From 3ad56137cb709182dc1bc242cd80a1474078cb95 Mon Sep 17 00:00:00 2001
From: Evgenii Klimov <eugene.dev@lipklim.org>
Date: Thu, 13 Jul 2023 18:16:08 +0100
Subject: [PATCH] ob-tangle.el: Add new :lexical header argument

* lisp/ob-tangle.el (org-babel-tangle-use-lexical-binding): Add
new customization variable.
(org-babel-tangle): Add ability to enable lexical binding in
tangled file.

* lisp/ob-core.el (org-babel-common-header-args-w-values): Add new
:lexical header argument.

Previously one could achieve this manually, but 1) had to keep in mind
to hold this source block top-most, 2) couldn't use comment header
argument, as `lexical-binding' variable must be set in the first line
of a file.
---
 lisp/ob-core.el   |  1 +
 lisp/ob-tangle.el | 15 +++++++++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 47410b53f..c1801a677 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -437,6 +437,7 @@ then run `org-babel-switch-to-session'."
     (sep	. :any)
     (session	. :any)
     (shebang	. :any)
+    (lexical    . ((yes no)))
     (tangle	. ((tangle yes no :any)))
     (tangle-mode . ((#o755 #o555 #o444 :any)))
     (var	. :any)
diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 1274d0db7..916dd8489 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -76,6 +76,11 @@ then the name of the language is used."
   :group 'org-babel-tangle
   :type 'boolean)
 
+(defcustom org-babel-tangle-use-lexical-binding nil
+  "Enable lexical binding in tangled file by default."
+  :group 'org-babel-tangle
+  :type 'boolean)
+
 (defcustom org-babel-post-tangle-hook nil
   "Hook run in code files tangled by `org-babel-tangle'."
   :group 'org-babel-tangle
@@ -262,7 +267,7 @@ expression."
 	     (when file-name
                (let ((lspecs (cdr by-fn))
 		     (fnd (file-name-directory file-name))
-		     modes make-dir she-banged lang)
+		     modes make-dir she-banged scoped lang)
 	         ;; drop source-blocks to file
 	         ;; We avoid append-to-file as it does not work with tramp.
 	         (with-temp-buffer
@@ -273,6 +278,9 @@ expression."
 			     (get-spec (lambda (name) (cdr (assq name (nth 4 spec)))))
 			     (she-bang (let ((sheb (funcall get-spec :shebang)))
 				         (when (> (length sheb) 0) sheb)))
+                             (lexicalp (or (string-equal (funcall get-spec :lexical)
+                                                         "yes")
+                                           org-babel-tangle-use-lexical-binding))
 			     (tangle-mode (funcall get-spec :tangle-mode)))
 		        (unless (string-equal block-lang lang)
 			  (setq lang block-lang)
@@ -294,7 +302,10 @@ expression."
 		        (when (and she-bang (not she-banged))
 			  (insert (concat she-bang "\n"))
 			  (setq she-banged t))
-		        (org-babel-spec-to-string spec)
+                        (org-babel-spec-to-string spec)
+		        (when (and lexicalp (not scoped))
+			  (save-excursion (elisp-enable-lexical-binding))
+			  (setq scoped t))
 		        (setq block-counter (+ 1 block-counter))))
 		    lspecs)
 		   (when make-dir
-- 
2.34.1

Re: [Pre-PATCH] Add new :lexical header argument

[Ihor Radchenko]

Evgenii Klimov <eugene.dev@lipklim.org> writes:

> * lisp/ob-core.el (org-babel-common-header-args-w-values): Add new
> :lexical header argument.

A short note: ob-emacs-lisp already defines :lexical header arg. See
org-babel-header-args:emacs-lisp.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

[Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Evgenii Klimov via General discussions about Org-mode.]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]


Ihor Radchenko <yantar92@posteo.net> writes:

> Evgenii Klimov <eugene.dev@lipklim.org> writes:
>
>> * lisp/ob-core.el (org-babel-common-header-args-w-values): Add new
>> :lexical header argument.
>
> A short note: ob-emacs-lisp already defines :lexical header arg. See
> org-babel-header-args:emacs-lisp.

Thanks for pointing it out. Updated the patch to reflect your note and
documented the change.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-ob-tangle.el-Add-the-capability-to-specify-lexica.patch --]
[-- Type: text/x-diff, Size: 2998 bytes --]

From 7e75b55ccb4e82d9341b6b308b558e5d01df5128 Mon Sep 17 00:00:00 2001
From: Evgenii Klimov <eugene.dev@lipklim.org>
Date: Thu, 13 Jul 2023 18:16:08 +0100
Subject: [PATCH v2] ob-tangle.el: Add the capability to specify lexical scope
 in tangled files

* lisp/ob-tangle.el (org-babel-tangle): Add the ability to enable
lexical binding in Elisp tangled file.

* etc/ORG-NEWS (=:lexical= header argument now influences tangled
files): Document this change.

Previously one could achieve this manually, but 1) had to keep in mind
to hold this source block top-most, 2) couldn't use comment header
argument, as `lexical-binding' variable must be set in the first line
of a file.
---
 etc/ORG-NEWS      | 7 +++++++
 lisp/ob-tangle.el | 9 +++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index a4725ae8c..c632c4814 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -284,6 +284,13 @@ setting the ~STYLE~ property for each sub-task.
 The change is breaking when ~org-use-property-inheritance~ is set to ~t~.
 
 ** New and changed options
+*** =:lexical= header argument now influences tangled files
+
+When extracting an Elisp src block, the target's file
+~lexical-binding~ variable is set according to the src block's
+=:lexical= parameter.  If at least one block is set to lexical scope,
+then the whole file will be with lexical scope.
+
 *** Commands affected by ~org-fold-catch-invisible-edits~ can now be customized
 
 New user option ~org-fold-catch-invisible-edits-commands~ controls
diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 1274d0db7..10fe5dcf5 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -262,7 +262,7 @@ expression."
 	     (when file-name
                (let ((lspecs (cdr by-fn))
 		     (fnd (file-name-directory file-name))
-		     modes make-dir she-banged lang)
+		     modes make-dir she-banged scoped lang)
 	         ;; drop source-blocks to file
 	         ;; We avoid append-to-file as it does not work with tramp.
 	         (with-temp-buffer
@@ -273,6 +273,8 @@ expression."
 			     (get-spec (lambda (name) (cdr (assq name (nth 4 spec)))))
 			     (she-bang (let ((sheb (funcall get-spec :shebang)))
 				         (when (> (length sheb) 0) sheb)))
+                             (lexicalp (org-babel-emacs-lisp-lexical
+                                        (funcall get-spec :lexical)))
 			     (tangle-mode (funcall get-spec :tangle-mode)))
 		        (unless (string-equal block-lang lang)
 			  (setq lang block-lang)
@@ -294,7 +296,10 @@ expression."
 		        (when (and she-bang (not she-banged))
 			  (insert (concat she-bang "\n"))
 			  (setq she-banged t))
-		        (org-babel-spec-to-string spec)
+                        (org-babel-spec-to-string spec)
+		        (when (and lexicalp (not scoped))
+			  (save-excursion (elisp-enable-lexical-binding))
+			  (setq scoped t))
 		        (setq block-counter (+ 1 block-counter))))
 		    lspecs)
 		   (when make-dir
-- 
2.34.1

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Evgenii Klimov <eugene.dev@lipklim.org> writes:

> diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
> index a4725ae8c..c632c4814 100644
> --- a/etc/ORG-NEWS
> +++ b/etc/ORG-NEWS
> @@ -284,6 +284,13 @@ setting the ~STYLE~ property for each sub-task.
>  The change is breaking when ~org-use-property-inheritance~ is set to ~t~.
>  
>  ** New and changed options
> +*** =:lexical= header argument now influences tangled files
> +
> +When extracting an Elisp src block, the target's file
> +~lexical-binding~ variable is set according to the src block's
> +=:lexical= parameter.  If at least one block is set to lexical scope,
> +then the whole file will be with lexical scope.

I am not sure if I like this approach.
I have 2 problems with the patch:

1. Previous users of :lexical header arg might be surprised.
   Though it is an OK breaking change since people who used :lexical
   argument and expected it to be ignored in the tangled file probably
   did something wrong.
   
2. More importantly, we are adding ob-emacs-lisp-specific handling into
   generic ob-tangle.

What about generalizing the idea and providing a way to set Emacs
buffer-local variables in the tangled files?

Can be something like:

#+begin_src elisp :file-locals (lexical-binding t eval (line-number-mode))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

Hi Ihor,

> 2. More importantly, we are adding ob-emacs-lisp-specific handling into
>    generic ob-tangle.
>
> What about generalizing the idea and providing a way to set Emacs
> buffer-local variables in the tangled files?

Looking at this patch earlier, I had the same concern. I think it’s worth being
paranoid about a proliferation of overly-specialised ob-tangle arguments.

It occurs to me that this use case could also perhaps be satisfied by file-local
variables? If we presume that mixing tangling to lexically-bound and
non-lexically bound elisp files is a corner case we don’t care that much about,
a `org-babel-elisp-lexical' variable could be used to set the behaviour, and
modified using file-local variable forms as usual.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Timothy <orgmode@tec.tecosaur.net> writes:

> It occurs to me that this use case could also perhaps be satisfied by file-local
> variables? If we presume that mixing tangling to lexically-bound and
> non-lexically bound elisp files is a corner case we don’t care that much about,
> a `org-babel-elisp-lexical' variable could be used to set the behaviour, and
> modified using file-local variable forms as usual.

I do not think that setting `lexical-binding' file-local variable in an
Org file makes much sense. I am sure that we can do better.

I can see that there are people in favour of :lexical feature.
So, we can probably add it, but it should not be in ob-core.
Instead, we should provide an infrastructure allowing ob-* backends to
modify tangling. Maybe some kind of special function that will override
default `org-babel-spec-to-string' per backend?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

Hi Ihor,

> I do not think that setting `lexical-binding’ file-local variable in an
> Org file makes much sense. I am sure that we can do better.

I suppose part of the question is what sort of way we treat it? In my mind,
considering the current way lexical scope is seen in Emacs (used everywhere in
modern elisp, not on-by-default because that would break compat, is my
impression). I think it would make sense if this eventually ends up with a
“globally enabled everywhere by default” setting, which is why I think a
defcustom may be the best fit.

> I can see that there are people in favour of :lexical feature.
> So, we can probably add it, but it should not be in ob-core.
> Instead, we should provide an infrastructure allowing ob-* backends to
> modify tangling. Maybe some kind of special function that will override
> default `org-babel-spec-to-string’ per backend?

Mmm, something generic for backend-specific modification sounds like a good idea
to me.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Timothy <orgmode@tec.tecosaur.net> writes:

>> I do not think that setting `lexical-binding’ file-local variable in an
>> Org file makes much sense. I am sure that we can do better.
>
> I suppose part of the question is what sort of way we treat it? In my mind,
> considering the current way lexical scope is seen in Emacs (used everywhere in
> modern elisp, not on-by-default because that would break compat, is my
> impression). I think it would make sense if this eventually ends up with a
> “globally enabled everywhere by default” setting, which is why I think a
> defcustom may be the best fit.

Do you mean to enable :lexical t in all the elisp source blocks?
It is already possible (and done) using `org-babel-default-header-args:emacs-lisp'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> What about generalizing the idea and providing a way to set Emacs
> buffer-local variables in the tangled files?
>
> Can be something like:
>
> #+begin_src elisp :file-locals (lexical-binding t eval (line-number-mode))

Over one month have passed since the last message in this thread.

Evgenii, are you still interested to continue working on this?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Evgenii Klimov]

Hi Ihor

On August 23, 2023 12:44:32 PM GMT+01:00, Ihor Radchenko <yantar92@posteo.net> wrote:
>Ihor Radchenko <yantar92@posteo.net> writes:
>
>> What about generalizing the idea and providing a way to set Emacs
>> buffer-local variables in the tangled files?
>>
>> Can be something like:
>>
>> #+begin_src elisp :file-locals (lexical-binding t eval (line-number-mode))
>
>Over one month have passed since the last message in this thread.
>
>Evgenii, are you still interested to continue working on this?
>

I am busy for the next month and even after that generalizing the idea would take too much effort and time, since I'm not an advanced programmer. So probably the answer is no.