[BUG] Eval sets incorrect runtime metainformation

[BUG] Eval sets incorrect runtime metainformation

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 3605 bytes --]

TLDR: primitive-eval ignores information provided by reader.

There are two identical snippets of code, which differ only in keyword
argument `compile?` value.  When compile? is set to #t, metainformation
for procedure is correct, but when it is set to #f resulting data is
different :)

--8<---------------cut here---------------start------------->8---
(define-module (2024-06-25-eval-string-metadata))

(use-modules (system vm program)
             (ice-9 eval-string))

(eval-string "(define (test-fn) 'hey)"
             #:module (current-module)
             #:file "hello.scm"
             #:line 1
             #:column 1
             #:compile? #f)

(format #t "~a\n" (program-sources test-fn))
;; ((0 ice-9/eval.scm 329 . 13) (12 ice-9/eval.scm 330 . 21) (44 ice-9/eval.scm 330 . 15))

(eval-string "(define (test-fn) 'hey)"
             #:module (current-module)
             #:file "hello.scm"
             #:line 1
             #:column 1
             #:compile? #t)

(format #t "~a\n" (program-sources test-fn))
;; ((0 hello.scm 1 . 1) (12 hello.scm 1 . 19))

(exit 0)
--8<---------------cut here---------------end--------------->8---


To track down the issue, I simplified the code a bit and removed
unecessary wrappers, the resulting snippet looks like this:

--8<---------------cut here---------------start------------->8---
(use-modules (system base language)
             (system vm program))

(call-with-input-string
 "(define (hello) 'hey)"
 (lambda (port)
   (set-port-filename! port "test.scm")
   (set-port-line! port 100)
   (set-port-column! port 0)

   (let ((reader (language-reader (lookup-language (current-language))))
         (eval (language-evaluator (lookup-language (current-language)))))
     (eval (pk (reader port (current-module))) (current-module)))))

(format #t "~a\n" (program-sources hello))
;;; (#<syntax:test.scm:101:0 (#<syntax:test.scm:101:1 define> #<syntax:test.scm:101:8 (#<syntax:test.scm:101:9 hello>)> #<syntax:test.scm:101:16 (quote #<syntax:test.scm:101:17 hey>)>)>)
;; ((0 ice-9/eval.scm 329 . 13) (12 ice-9/eval.scm 330 . 21) (44 ice-9/eval.scm 330 . 15))
--8<---------------cut here---------------end--------------->8---

The reader keeps all metainformation obtained from port, so the issue is
in the eval phase.

The problem is in language-evaluator (primitive-eval in our case), which
just ignores all the info from reader.

--8<---------------cut here---------------start------------->8---
(use-modules (system vm program))
(primitive-eval '(define (hello) 'hey))
(format #t "~a\n" (program-sources hello))
;; ((0 ice-9/eval.scm 329 . 13) (12 ice-9/eval.scm 330 . 21) (44 ice-9/eval.scm 330 . 15))
--8<---------------cut here---------------end--------------->8---

It leads to several shortcomings: the inability to utilize the built-in
eval for implementing interactive development tooling (IDEs with proper
eval and goto definition functionality, interactive test runners and
debuggers) and do other programmatic interactions with the running Guile
process.  I want to bring more attention to this issue, because it can
have serious negative impact on the guile ecosystem.

The another related problem is that the metainformation is stored in
prodecures properties, but not in variables, which makes it impossible
to implement a proper goto definition in general case.

Would be glad to cooperate on improving all of those points 👆 and would
be twice as glad to get some guidance or hear a word of encouragement.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [BUG] Eval sets incorrect runtime metainformation

[Andy Wingo]

Hello,

On Tue 25 Jun 2024 17:07, Andrew Tropin <andrew@trop.in> writes:

> (use-modules (system vm program)
>              (ice-9 eval-string))
>
> (eval-string "(define (test-fn) 'hey)"
>              #:file "hello.scm"
>              #:line 1
>              #:column 1
>              #:compile? #f)
>
> (format #t "~a\n" (program-sources test-fn))
> ;; ((0 ice-9/eval.scm 329 . 13) (12 ice-9/eval.scm 330 . 21) (44 ice-9/eval.scm 330 . 15))

What you are seeing here is that in general the debugging experience is
different for interpreted and compiled procedures.  For example, you
will not be able to set a breakpoint in interpreted code, because the
code for the closure that is part of `eval` corresponds to potentially
many different functions.  program-sources will only work usefully on
compiled procedures.
https://www.gnu.org/software/guile/manual/html_node/Compiled-Procedures.html.

I would suggest that if you are working on a rich IDE, that you pass
#:compile? #t.  Nothing else will work as you like.

That said, the evaluator does attach so-called "meta-data" information
to procedures, such as the procedure name.
https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
If you know that you are making a procedure you can insert some
meta-data for use by your run-time, in an initial vector alist.  See
https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
But that's limited and doesn't take macros, etc into account.

Regards,

Andy

RE: [BUG] Eval sets incorrect runtime metainformation

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]

> (use-modules (system vm program)
>              (ice-9 eval-string))
>
> (eval-string "(define (test-fn) 'hey)"
>              #:file "hello.scm"
>              #:line 1
>              #:column 1
>              #:compile? #f)
>
> (format #t "~a\n" (program-sources test-fn))
> ;; ((0 ice-9/eval.scm 329 . 13) (12 ice-9/eval.scm 330 . 21) (44 ice-9/eval.scm 330 . 15))

>What you are seeing here is that in general the debugging experience is
different for interpreted and compiled procedures.  For example, you
will not be able to set a breakpoint in interpreted code, because the
code for the closure that is part of `eval` corresponds to potentially
many different functions.  program-sources will only work usefully on
compiled procedures.
>https://www.gnu.org/software/guile/manual/html_node/Compiled-Procedures.html.

IIRC, the question wasn’t about debugging in general, it was about source locations in particular. Surely program-sources (or, in this case, procedure-source maybe?) (why are the procedures in this family even named program-whatever, this prevents doing the same for interpreted code later) could be adjusted to also work for ‘eval’. For example, ‘eval’ could set the ‘source’ (*) procedure property when a closure is made.

Likewise for arity and procedure name.

(*) Looking at https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html, it appears the documentation doesn’t actually document what the ‘source’ property is for – It could be interpreted in multiple ways currently.

>I would suggest that if you are working on a rich IDE, that you pass
#:compile? #t.  Nothing else will work as you like.

Something else will work as well: adjusting ‘eval’ to set procedure properties or something like that.

>That said, the evaluator does attach so-called "meta-data" information
to procedures, such as the procedure name.
https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
If you know that you are making a procedure you can insert some
meta-data for use by your run-time, in an initial vector alist.  See
https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
But that's limited and doesn't take macros, etc into account.

That’s the job of ‘eval’, not the user of ‘eval’.

Best regards,
Maxime Devos.

[-- Attachment #2: Type: text/html, Size: 4785 bytes --]

RE: [BUG] Eval sets incorrect runtime metainformation

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 4109 bytes --]

On 2024-06-26 11:36, Maxime Devos wrote:

>>> (use-modules (system vm program)
>>>              (ice-9 eval-string))
>>>
>>> (eval-string "(define (test-fn) 'hey)"
>>>              #:file "hello.scm"
>>>              #:line 1
>>>              #:column 1
>>>              #:compile? #f)
>>>
>>> (format #t "~a\n" (program-sources test-fn))
>>> ;; ((0 ice-9/eval.scm 329 . 13) (12 ice-9/eval.scm 330 . 21) (44 ice-9/eval.scm 330 . 15))
>
>>What you are seeing here is that in general the debugging experience is
>> different for interpreted and compiled procedures.  For example, you
>> will not be able to set a breakpoint in interpreted code, because the
>> code for the closure that is part of `eval` corresponds to potentially
>> many different functions.  program-sources will only work usefully on
>> compiled procedures.
>>https://www.gnu.org/software/guile/manual/html_node/Compiled-Procedures.html.
>
> IIRC, the question wasn’t about debugging in general, it was about
> source locations in particular. Surely program-sources (or, in this
> case, procedure-source maybe?) (why are the procedures in this family
> even named program-whatever, this prevents doing the same for
> interpreted code later) could be adjusted to also work for ‘eval’. For
> example, ‘eval’ could set the ‘source’ (*) procedure property when a
> closure is made.
>
> Likewise for arity and procedure name.
>
> (*) Looking at
> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html,
> it appears the documentation doesn’t actually document what the
> ‘source’ property is for – It could be interpreted in multiple ways
> currently.
>
>>I would suggest that if you are working on a rich IDE, that you pass
>> #:compile? #t.  Nothing else will work as you like.

I do pass (use compile directly to be more precise) :) but there is
another very important problem related to this approach:
https://yhetil.org/guile-devel/20240621164008.eEg72C00D1xd29F01Eg7DF@baptiste.telenet-ops.be/T/#t

It also leads to problem with define-once, which sets the value to
#<unspecified> in most cases, when "evaluated" with compile.

>
> Something else will work as well: adjusting ‘eval’ to set procedure
> properties or something like that.

Agree with Maxime here, and that is what I propose in the original
message.

>> That said, the evaluator does attach so-called "meta-data" information
>> to procedures, such as the procedure name.
>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>> If you know that you are making a procedure you can insert some
>> meta-data for use by your run-time, in an initial vector alist.  See
>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>> But that's limited and doesn't take macros, etc into account.

Yep, that's is why I wrote:

--8<---------------cut here---------------start------------->8---
The another related problem is that the metainformation is stored in
prodecures properties, but not in variables, which makes it impossible
to implement a proper goto definition in general case.
--8<---------------cut here---------------end--------------->8---

How hard is to make a generic metainformation, which is attachable to
macros, symbols and other objects of the language, not only to
procedures?

>> That said, the evaluator does attach so-called "meta-data" information
>> to procedures, such as the procedure name.
>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>> If you know that you are making a procedure you can insert some
>> meta-data for use by your run-time, in an initial vector alist.  See
>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>> But that's limited and doesn't take macros, etc into account.
>
> That’s the job of ‘eval’, not the user of ‘eval’.

Think the same, eval should respect information provided by reader and
attach it to corresponding objects.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [BUG] Eval sets incorrect runtime metainformation

[Andy Wingo]

On Wed 26 Jun 2024 11:36, Maxime Devos <maximedevos@telenet.be> writes:

> IIRC, the question wasn’t about debugging in general, it was about
> source locations in particular. Surely program-sources (or, in this
> case, procedure-source maybe?) (why are the procedures in this family
> even named program-whatever, this prevents doing the same for
> interpreted code later) could be adjusted to also work for ‘eval’. For
> example, ‘eval’ could set the ‘source’ (*) procedure property when a
> closure is made.

I think it's really valuable to imagine how things should be but if you
are going to argue they should be different, you should first try to
understand how they are.

`program-sources` is a mapping from bytecode offsets to source
locations.  For compiled procedures we can make this mapping because
each bytecode position has a single source.  For interpreted procedures,
what you end up getting is the bytecode-to-source mapping *for eval*,
not for the code being interpreted.

Is it a great thing that there is a debugging (I use the term on purpose
to mean all kinds of run-time reflection etc) difference between eval
and compile?  No, of course not.  I would rather there not be a
difference and not have to document something that is at best
extraneous.  There are differing pressures on eval: for bootstrap times
(and macro expansion time) you want it to have the least amount of
overhead possible, whereas for debugging you want to attach meta-data
that isn't strictly needed at run-time.  Attaching that meta-data has
memory and time overheads.

If we are looking to get the source location *just of the interpreted
closure* -- that is possible; see eval.scm:581, there you would attach
some other properties.  You would have to define a different debugging
interface that looks for source location information in a way different
from program-sources.  For me it's not worth it but I encourage you to
experiment with (ice-9 eval); it's just another Scheme program.  (You
would need to take a different approach to memoization, in order to pass
through source location information.)

Andy

Re: [BUG] Eval sets incorrect runtime metainformation

[Philip McGrath]

[-- Attachment #1: Type: text/plain, Size: 2827 bytes --]

On 6/26/24 07:41, Andrew Tropin wrote:>
>>> That said, the evaluator does attach so-called "meta-data" information
>>> to procedures, such as the procedure name.
>>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>>> If you know that you are making a procedure you can insert some
>>> meta-data for use by your run-time, in an initial vector alist.  See
>>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>>> But that's limited and doesn't take macros, etc into account.
> 
> Yep, that's is why I wrote:
> 
> --8<---------------cut here---------------start------------->8---
> The another related problem is that the metainformation is stored in
> prodecures properties, but not in variables, which makes it impossible
> to implement a proper goto definition in general case.
> --8<---------------cut here---------------end--------------->8---
> 
> How hard is to make a generic metainformation, which is attachable to
> macros, symbols and other objects of the language, not only to
> procedures?
> 

Instead of run-time reflection on values, I think an IDE implementing 
jump to definition should use source location and binding information 
from syntax objects. That's how DrRacket does it.

The attached DrRacket screenshot (of [1]), in which I've tacked a bunch 
of binding arrows, shows how local definitions and complex 
macro-introduced binding structures are supported. It also works across 
modules, even with renaming: in the program `#lang racket (define one 
1)`, right-clicking on `define` and choosing "Jump to Binding 
Occurrence" will highlight `racket`, because the initial import of the 
racket language establishes the applicable binding of define, whereas 
choosing "Open Defining File" will open the racket/private/kw module, 
and then choosing "Jump to Definition (in Other File)" will jump to the 
definition of `new-define` on line 1171 of kw.rkt, which is renamed 
elsewhere to become the `define` of `#lang racket`.

Since syntax objects from the expander encode all the details of scope, 
this works mostly automatically. DrRacket makes this functionality 
available as a library, so navigation features can also be used from 
Emacs with racket-xp-mode [2] or other editors with the Language Server 
Protocol [3].

That said, if you do want to associate extra information with values at 
runtime, Guile has object properties [4], or of course you could always 
keep your own hash table on the side.

Philip

[1]: 
https://github.com/racket/racket/blob/6105332045595f8324985d7a34acc68fa5a61dcf/pkgs/racket-test/tests/json/indent.rkt#L244-L276
[2]: https://www.racket-mode.com/#racket_002dxp_002dmode
[3]: https://github.com/jeapostrophe/racket-langserver/
[4]: 
https://www.gnu.org/software/guile/manual/html_node/Object-Properties.html

[-- Attachment #2: Binding arrows from define-main for indent-test-data-cli 2024-06-26.png --]
[-- Type: image/png, Size: 571391 bytes --]

Re: [BUG] Eval sets incorrect runtime metainformation

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 3667 bytes --]

On 2024-06-26 18:06, Philip McGrath wrote:

> On 6/26/24 07:41, Andrew Tropin wrote:>
>>>> That said, the evaluator does attach so-called "meta-data" information
>>>> to procedures, such as the procedure name.
>>>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>>>> If you know that you are making a procedure you can insert some
>>>> meta-data for use by your run-time, in an initial vector alist.  See
>>>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>>>> But that's limited and doesn't take macros, etc into account.
>> 
>> Yep, that's is why I wrote:
>> 
>> --8<---------------cut here---------------start------------->8---
>> The another related problem is that the metainformation is stored in
>> prodecures properties, but not in variables, which makes it impossible
>> to implement a proper goto definition in general case.
>> --8<---------------cut here---------------end--------------->8---
>> 
>> How hard is to make a generic metainformation, which is attachable to
>> macros, symbols and other objects of the language, not only to
>> procedures?
>> 
>
> Instead of run-time reflection on values, I think an IDE implementing 
> jump to definition should use source location and binding information 
> from syntax objects. That's how DrRacket does it.

If you could share a snippet of code or link to related library or docs,
it could help to understand better the idea.  I still relatively newbie
in questions related to snyntax objects and overall scheme's expand.

>
> The attached DrRacket screenshot (of [1]), in which I've tacked a bunch 
> of binding arrows, shows how local definitions and complex 
> macro-introduced binding structures are supported. It also works across 
> modules, even with renaming: in the program `#lang racket (define one 
> 1)`, right-clicking on `define` and choosing "Jump to Binding 
> Occurrence" will highlight `racket`, because the initial import of the 
> racket language establishes the applicable binding of define, whereas 
> choosing "Open Defining File" will open the racket/private/kw module, 
> and then choosing "Jump to Definition (in Other File)" will jump to the 
> definition of `new-define` on line 1171 of kw.rkt, which is renamed 
> elsewhere to become the `define` of `#lang racket`.
>
> Since syntax objects from the expander encode all the details of scope, 
> this works mostly automatically. DrRacket makes this functionality 
> available as a library, so navigation features can also be used from 
> Emacs with racket-xp-mode [2] or other editors with the Language Server 
> Protocol [3].
>
> That said, if you do want to associate extra information with values at
> runtime, Guile has object properties [4], or of course you could always 
> keep your own hash table on the side.

Thank you for pointing to object properties!

I guess we can implement our own eval, to attach location information to
corresponding objects, I just expected it to be a part of the guile,
because we already have this information for procedures anyway.  The
another problem with our own eval is that information for already loaded
code won't be enriched with our custom stuff :/

>
> Philip
>
> [1]: 
> https://github.com/racket/racket/blob/6105332045595f8324985d7a34acc68fa5a61dcf/pkgs/racket-test/tests/json/indent.rkt#L244-L276
> [2]: https://www.racket-mode.com/#racket_002dxp_002dmode
> [3]: https://github.com/jeapostrophe/racket-langserver/
> [4]: 
> https://www.gnu.org/software/guile/manual/html_node/Object-Properties.html
> x

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [BUG] Eval sets incorrect runtime metainformation

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]

On 2024-06-26 18:04, Andy Wingo wrote:

> On Wed 26 Jun 2024 11:36, Maxime Devos <maximedevos@telenet.be> writes:
>
>> IIRC, the question wasn’t about debugging in general, it was about
>> source locations in particular. Surely program-sources (or, in this
>> case, procedure-source maybe?) (why are the procedures in this family
>> even named program-whatever, this prevents doing the same for
>> interpreted code later) could be adjusted to also work for ‘eval’. For
>> example, ‘eval’ could set the ‘source’ (*) procedure property when a
>> closure is made.
>
> I think it's really valuable to imagine how things should be but if you
> are going to argue they should be different, you should first try to
> understand how they are.
>
> `program-sources` is a mapping from bytecode offsets to source
> locations.  For compiled procedures we can make this mapping because
> each bytecode position has a single source.  For interpreted procedures,
> what you end up getting is the bytecode-to-source mapping *for eval*,
> not for the code being interpreted.
>
> Is it a great thing that there is a debugging (I use the term on purpose
> to mean all kinds of run-time reflection etc) difference between eval
> and compile?  No, of course not.  I would rather there not be a
> difference and not have to document something that is at best
> extraneous.  There are differing pressures on eval: for bootstrap times
> (and macro expansion time) you want it to have the least amount of
> overhead possible, whereas for debugging you want to attach meta-data
> that isn't strictly needed at run-time.  Attaching that meta-data has
> memory and time overheads.

Does it mean, we could want another eval implementation, which is a bit
havier, but will preserve additional meta-data?

not-that-primitive-and-a-bit-fancier-eval :)

>
> If we are looking to get the source location *just of the interpreted
> closure* -- that is possible; see eval.scm:581, there you would attach
> some other properties.  You would have to define a different debugging
> interface that looks for source location information in a way different
> from program-sources.  For me it's not worth it but I encourage you to
> experiment with (ice-9 eval); it's just another Scheme program.  (You
> would need to take a different approach to memoization, in order to pass
> through source location information.)
>
> Andy

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [BUG] Eval sets incorrect runtime metainformation

[Philip McGrath]

On 6/28/24 09:20, Andrew Tropin wrote:
> On 2024-06-26 18:06, Philip McGrath wrote:
> 
>> On 6/26/24 07:41, Andrew Tropin wrote:>
>>>>> That said, the evaluator does attach so-called "meta-data" information
>>>>> to procedures, such as the procedure name.
>>>>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>>>>> If you know that you are making a procedure you can insert some
>>>>> meta-data for use by your run-time, in an initial vector alist.  See
>>>>> https://www.gnu.org/software/guile/manual/html_node/Procedure-Properties.html.
>>>>> But that's limited and doesn't take macros, etc into account.
>>>
>>> Yep, that's is why I wrote:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> The another related problem is that the metainformation is stored in
>>> prodecures properties, but not in variables, which makes it impossible
>>> to implement a proper goto definition in general case.
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> How hard is to make a generic metainformation, which is attachable to
>>> macros, symbols and other objects of the language, not only to
>>> procedures?
>>>
>>
>> Instead of run-time reflection on values, I think an IDE implementing
>> jump to definition should use source location and binding information
>> from syntax objects. That's how DrRacket does it.
> 
> If you could share a snippet of code or link to related library or docs,
> it could help to understand better the idea.

I'm not intimately familiar with the implementation, but a key library 
is drracket/check-syntax, documented at [1], which is implemented in the 
[2] Git repository (mostly in the drracket-tool-text-lib directory, but 
with tests etc. elsewhere). The source for the Emacs usage is at [3]. 
You can probably get more useful advice on 
<https://racket.discourse.group>, especially from Robby Findler and Greg 
Hendershot. (Note that, while it requires one-time registration, 
Racket's Discourse instance is configured to be usable as a mailing 
list, including starting new threads by email.)

[1]: 
https://docs.racket-lang.org/drracket-tools/Accessing_Check_Syntax_Programmatically.html
[2]: https://github.com/racket/drracket
[3]: https://github.com/greghendershott/racket-mode/

> I still relatively newbie
> in questions related to snyntax objects and overall scheme's expand.
> 

The Scheme reports leave it to implementations to decide how to 
implement syntax objects and expansion more generally, so you will need 
some Guile-specific code. The procedures documented in [4] seem like a 
good place to start, especially `syntax-source`, `syntax-module`, and 
`syntax-local-binding`. I expect Guile has all of the functionality you 
would need, though some of it might be internal or awkward to use. It is 
possible, though, that you might need or want to enhance the expander: 
Racket's expander is especially feature-rich in this area, in part 
because Racket (MzScheme) was originally created to write DrRacket 
(DrScheme), so this kind of tool support has been a consideration from 
the beginning.

[4]: 
https://www.gnu.org/software/guile/manual/html_node/Syntax-Transformer-Helpers.html

Philip

RE: [BUG] Eval sets incorrect runtime metainformation

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2541 bytes --]

>Instead of run-time reflection on values, I think an IDE implementing 
jump to definition should use source location and binding information 
from syntax objects. That's how DrRacket does it.

>The attached DrRacket screenshot (of [1]), in which I've tacked a bunch 
of binding arrows, shows how local definitions and complex 
macro-introduced binding structures are supported. It also works across 
modules, even with renaming: in the program `#lang racket (define one 
1)`, right-clicking on `define` and choosing "Jump to Binding 
Occurrence" will highlight `racket`, because the initial import of the 
racket language establishes the applicable binding of define, whereas 
choosing "Open Defining File" will open the racket/private/kw module, 
and then choosing "Jump to Definition (in Other File)" will jump to the 
definition of `new-define` on line 1171 of kw.rkt, which is renamed 
elsewhere to become the `define` of `#lang racket`.

>Since syntax objects from the expander encode all the details of scope, 
this works mostly automatically. DrRacket makes this functionality 
available as a library, so navigation features can also be used from 
Emacs with racket-xp-mode [2] or other editors with the Language Server 
Protocol [3].

I suppose this could work, but it seems the wrong layer of abstraction and a bit round-about to me, it also only works on languages that use Scheme-style syntax. Instead, I’d propose compiling the relevant code to Tree-IL (without optimisations, should work with most languages).  (For Scheme, this is essentially macro expansion, but the resulting object won’t technically be a syntax object.)

In Tree-IL, all syntax has been expanded (in tree form), which eliminates a lot of complications. Yet, source location information remains available (albeit not documented ...)! And the original variable names remain available (together with an unshadow-ified version, so for local variables you can move upwards to find the corresponding definition and in particular its source location.

A large upside is that this should work for most languages (not only Scheme) and is relatively straightforward to implement, a small downside is that information on definitions like (let-syntax ((f [...])) ...) aren’t available – you only see the return of (f stuff), not ‘f’ itself.

(But this downside applies to the ‘syntax’ route as well, unless you are doing complicated (but possible!) DIY partial macro expansion shenanigans.)

Best regards,
Maxime Devos.

[-- Attachment #2: Type: text/html, Size: 4342 bytes --]

Re: [BUG] Eval sets incorrect runtime metainformation

[Philip McGrath]

On 6/29/24 19:05, Maxime Devos wrote:
> > Instead of run-time reflection on values, I think an IDE implementing
> > jump to definition should use source location and binding information
> > from syntax objects. That's how DrRacket does it.
> > ...
> > Since syntax objects from the expander encode all the details of scope,
> > this works mostly automatically. DrRacket makes this functionality
> > available as a library, so navigation features can also be used from
> > Emacs with racket-xp-mode [2] or other editors with the Language Server
> > Protocol [3].
> 
> I suppose this could work, but it seems the wrong layer of abstraction 
> and a bit round-about to me, it also only works on languages that use 
> Scheme-style syntax. Instead, I’d propose compiling the relevant code to 
> Tree-IL (without optimisations, should work with most languages).  (For 
> Scheme, this is essentially macro expansion, but the resulting object 
> won’t technically be a syntax object.)
> 
> In Tree-IL, all syntax has been expanded (in tree form), which 
> eliminates a lot of complications. Yet, source location information 
> remains available (albeit not documented ...)! And the original variable 
> names remain available (together with an unshadow-ified version, so for 
> local variables you can move upwards to find the corresponding 
> definition and in particular its source location.
> 
> A large upside is that this should work for most languages (not only 
> Scheme) and is relatively straightforward to implement,

I hadn't realized that other languages in Guile might compile to Tree-IL 
directly instead of generating syntax objects. Is that common and/or 
encouraged? It seems like it would require the new language's compiler 
to do a lot of work that could otherwise be delegated to Guile, and it 
would make it difficult to implement parts of the new language using macros.

> a small downside 
> is that information on definitions like (let-syntax ((f [...])) ...) 
> aren’t available – you only see the return of (f stuff), not ‘f’ itself.
> 
> (But this downside applies to the ‘syntax’ route as well, unless you are 
> doing complicated (but possible!) DIY partial macro expansion shenanigans.)
> 

This and other features, like correctly tracking the binding of `else` 
in `cond`, require cooperation from the macro expander. The explanation 
of origin tracking in 
<https://docs.racket-lang.org/reference/stxprops.html> might be one 
place to start reading about how Racket supports this (but I am not an 
expert!). Similar expander features can also enable other tools, like 
DrRacket's macro stepper: 
<https://docs.racket-lang.org/macro-debugger/index.html>

Philip

RE: [BUG] Eval sets incorrect runtime metainformation

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

>I hadn't realized that other languages in Guile might compile to Tree-IL 
directly instead of generating syntax objects. Is that common and/or 
encouraged?

I haven’t really checked, but probably? For anything that’s not Schemy in its macros, it strips a pointless layer of abstraction, and Tree-IL is straightforward to work with.

>It seems like it would require the new language's compiler 
to do a lot of work that could otherwise be delegated to Guile, and it 
would make it difficult to implement parts of the new language using macros.

I don’t see what this lot of extra work this would be (unless the language has macros that are Schemy). But yes, if implementing terms in macros, it seems pretty inevitable to use syntax objects, in which case I suppose you might compile to Scheme instead of Tree-IL.

For the mentioned IDE, it wouldn’t really matter what’s happening precisely, as long are there is a path from ‘language -> Tree-Il’.

Best regards,
Maxime Devos.

[-- Attachment #2: Type: text/html, Size: 2406 bytes --]

Re: [BUG] Eval sets incorrect runtime metainformation

[Rob Browning]

Philip McGrath <philip@philipmcgrath.com> writes:

> I hadn't realized that other languages in Guile might compile to Tree-IL 
> directly instead of generating syntax objects. Is that common and/or 
> encouraged? It seems like it would require the new language's compiler 
> to do a lot of work that could otherwise be delegated to Guile, and it 
> would make it difficult to implement parts of the new language using macros.

For what it's worth, Lokke (a Clojure dialect) does both.  For the most
part it compiles by producing Scheme code from its reader that's then
rewritten by macros.  But it also includes a pass or two at the Tree-IL
level to handle some things that would be difficult to get right
otherwise (while also preserving the fairly tight integration with
Scheme that's intended).

For example, they handle adding support for "keywords as functions" (in
a generalized way), and resolving Clojure's namespaced symbols to Guile
module refs.

For anyone interested in the current arrangement (the README also covers
a bunch of limitations, differences, etc.):
https://codeberg.org/lokke/lokke/src/branch/main/DESIGN.md

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: [BUG] Eval sets incorrect runtime metainformation

[Matt Wette]

On 6/30/24 3:27 PM, Philip McGrath wrote:
>
> I hadn't realized that other languages in Guile might compile to 
> Tree-IL directly instead of generating syntax objects. Is that common 
> and/or encouraged? It seems like it would require the new language's 
> compiler to do a lot of work that could otherwise be delegated to 
> Guile, and it would make it difficult to implement parts of the new 
> language using macros.
>

I've written a LALR parser generator (based on the algorithm in the 
dragon book) and have produced
a few sample parsers and compilers.  I always compile to tree-il. See 
the javascript and mlang
subdirs under 
https://git.savannah.nongnu.org/cgit/nyacc.git/tree/examples/nyacc/lang

Matt