procedure-source availability

procedure-source availability

[Panicz Maciej Godek]

Hi,
I remember, that guile 1.8 had a meta-procedure called
'procedure-source' that returned what it said it would.

As I'm trying to use it now (with 2.0), it always returns #f.
Is there any way to convince guile to memoize the source
code of at least some of the procedures defined by user,
so they would be available through 'procedure-source',
or is 'procedure-source' completely redundant and abandoned?

best regards
M.

Re: procedure-source availability

[Panicz Maciej Godek]

I've found a bug report on savannah, which noted the lack of
``procedure-source'' for guile 1.9.11
http://savannah.gnu.org/bugs/?30469
I wonder if any further decisions were made regarding the maintenance
of ``procedure-source''. I would find it really useful for my
application if the sexp containing the lambda expression that was used
to generate the procedure was stored in it's property list, and I
don't understand why this isn't currently done. (Is it because of the
performance issues?)
Best regards
MG

Re: procedure-source availability

[Mark H Weaver]

On 09/30/2012 04:11 PM, Panicz Maciej Godek wrote:
> I've found a bug report on savannah, which noted the lack of
> ``procedure-source'' for guile 1.9.11
> http://savannah.gnu.org/bugs/?30469
> I wonder if any further decisions were made regarding the maintenance
> of ``procedure-source''. I would find it really useful for my
> application if the sexp containing the lambda expression that was used
> to generate the procedure was stored in it's property list, and I
> don't understand why this isn't currently done.

Out of curiosity, why do you need this?

The reason I ask is that, in the presence of macros, it is not clear 
what you mean by the "procedure-source".

Do you mean before macro expansion?  If so, what use it is to you, since 
it may contain user macros that your code cannot understand? 
Furthermore, many procedures simply don't exist before macro expansion. 
  For example, a record type definition expands to several procedure 
definitions that do not exist in the original source.

If you mean after macro expansion (or perhaps in some intermediate state 
of macro expansion, e.g. at the point where the macro expander first 
encounters the lambda expression), then a normal sexp with the original 
identifier names is no longer sufficient.  In psyntax, these 
intermediate forms are represented as syntax objects, but at the very 
least you'd need to rename the identifiers to prevent unintended 
variable capture.

It's a very thorny issue, and I suspect that's the reason that 
'procedure-source' no longer works.  If we are going to reimplement that 
functionality, then we first need to figure out precisely what it 
_should_ do in the general case.  Which brings me back to the question: 
"Why do you need this?"

     Regards,
       Mark

Re: procedure-source availability

[Panicz Maciej Godek]

>> I've found a bug report on savannah, which noted the lack of
>> ``procedure-source'' for guile 1.9.11
>> http://savannah.gnu.org/bugs/?30469
>> I wonder if any further decisions were made regarding the maintenance
>> of ``procedure-source''. I would find it really useful for my
>> application if the sexp containing the lambda expression that was used
>> to generate the procedure was stored in it's property list, and I
>> don't understand why this isn't currently done.
>
>
> Out of curiosity, why do you need this?

The short answer is: to have more options to explore and play around with.
The long answer is that I have been designing a framework for rapid
GUI development (and more), sort of a REPL among GUIs. The sources are
available on bitbucket's mercurial, if you want to see:
hg clone https://bitbucket.org/panicz/slayer
(apart from guile 2.0, they require SDL, SDL_image and SDL_ttf to build)
One of the key features is to save the current design of the GUI to
file. Initially I was thinking about serialization or dumping memory
images (like the $_GLOBAL variable in PHP, or like images in
smalltalk), but since I didn't find it to be a real option (since
guile seemed to be lacking those features), I decided to write a
function that outputs the scheme code needed to reconstruct the
current state of the environment (or its sufficient approximation).
Although closures would still be problematic, it would still be a good
step ahead, and even the problem with the closures would perhaps find
a solution eventually -- I thought to myself.

> The reason I ask is that, in the presence of macros, it is not clear what
> you mean by the "procedure-source".
>
> Do you mean before macro expansion?  If so, what use it is to you, since it
> may contain user macros that your code cannot understand? Furthermore, many
> procedures simply don't exist before macro expansion.  For example, a record
> type definition expands to several procedure definitions that do not exist
> in the original source.
>
> If you mean after macro expansion (or perhaps in some intermediate state of
> macro expansion, e.g. at the point where the macro expander first encounters
> the lambda expression), then a normal sexp with the original identifier
> names is no longer sufficient.  In psyntax, these intermediate forms are
> represented as syntax objects, but at the very least you'd need to rename
> the identifiers to prevent unintended variable capture.
>
> It's a very thorny issue, and I suspect that's the reason that
> 'procedure-source' no longer works.  If we are going to reimplement that
> functionality, then we first need to figure out precisely what it _should_
> do in the general case.  Which brings me back to the question: "Why do you
> need this?"

It would certainly seem more elegant if the user had the access to the
original source (and if the unexpanded macros are a problem, the
function could also store the code or a reference to the code of all
the macros that were used at the time of its creation). But if it
comes to my application, the expanded code would surely suffice -- the
only thing I care about is that the fixed point of the current state
is preserved. (Note that the name ``procedure-source'' would then
become confusing, and something like ``procedure-code'' or
``procedure-expanded-source'' would seem more appropriate)

Best regards
M

Re: procedure-source availability

[Ludovic Courtès]

Hi,

Panicz Maciej Godek <godek.maciek@gmail.com> skribis:

> The short answer is: to have more options to explore and play around with.
> The long answer is that I have been designing a framework for rapid
> GUI development (and more), sort of a REPL among GUIs. The sources are
> available on bitbucket's mercurial, if you want to see:
> hg clone https://bitbucket.org/panicz/slayer

Would it be an option for the GUI to open the source file at the right
location, when the user wants to see the source?

Thanks,
Ludo’.

Re: procedure-source availability

[Panicz Maciej Godek]

>> The short answer is: to have more options to explore and play around with.
>> The long answer is that I have been designing a framework for rapid
>> GUI development (and more), sort of a REPL among GUIs. The sources are
>> available on bitbucket's mercurial, if you want to see:
>> hg clone https://bitbucket.org/panicz/slayer
>
> Would it be an option for the GUI to open the source file at the right
> location, when the user wants to see the source?

Well, the idea for now is that the associated .spec file containing
the state of GUI is loaded on startup, and the state of the
interpreter is dumped to that file on exit (or at GUI's request).
Viewing the file will obviously be an option (for the curious user),
but any modifications would probably be overwritten eventually (unless
the file is write-protected).

This approach allows avoiding the design of any specific file format
to store information about the GUI -- everything is just scheme. The
only requirement is that all used object (images etc.) can be dumped
to scheme expressions, evaluation of which would re-create them. (This
isn't yet fully implemented, but I'm on a good way)

I hope this answers your question :)
M

Re: procedure-source availability

[Daniel Hartwig]

On 3 October 2012 03:29, Panicz Maciej Godek <godek.maciek@gmail.com> wrote:
> Well, the idea for now is that the associated .spec file containing
> the state of GUI is loaded on startup, and the state of the
> interpreter is dumped to that file on exit (or at GUI's request).
> Viewing the file will obviously be an option (for the curious user),
> but any modifications would probably be overwritten eventually (unless
> the file is write-protected).

You may be interested to see how Smalltalk handles this kind of thing.
 IIRC it dumps the complete VM state to disk and reloads.  Even
pre-packaged programs are simply VM dumps.  To deconstruct and then
reconstruct the running state of a system is quite a complex task.

Keep your own record of source information, etc. when your objects are
instantiated.  Associate this with each using object properties (or
similar construct).  This way you have any required environment and
precisely the source which generated the object, the user can inspect
this if interested and it can be used to create new instances.  As
someone mentioned earlier, the way macros are expanded there is no
single point to define (internal to guile) what is the “source” of a
procedure.  However, your system can define this as it has a clear
point of object instantiation.

I have not worked on such a system myself, so am poking around in the
dark here.  Good luck :-)

Re: procedure-source availability

[Panicz Maciej Godek]

> On 3 October 2012 03:29, Panicz Maciej Godek <godek.maciek@gmail.com> wrote:
>> Well, the idea for now is that the associated .spec file containing
>> the state of GUI is loaded on startup, and the state of the
>> interpreter is dumped to that file on exit (or at GUI's request).
>> Viewing the file will obviously be an option (for the curious user),
>> but any modifications would probably be overwritten eventually (unless
>> the file is write-protected).
>
> You may be interested to see how Smalltalk handles this kind of thing.
>  IIRC it dumps the complete VM state to disk and reloads.  Even
> pre-packaged programs are simply VM dumps.

I recall someone having the idea of adding smalltalk-like images to guile.
I'd love to see such feature one day, but if guile is supposed to be
an extension language, I think the C interface would need to be
redesigned, because there would be a need to somehow dump the smobs
from the heap to re-load them later.
I don't know much about the implementation of the GOOPS objects, but I
suspect that they would also require some means of serialization

>  To deconstruct and then
> reconstruct the running state of a system is quite a complex task.

I would use the word 'interesting' instead :)
And if it works, the advantage is that the result can be modified by humans.

> Keep your own record of source information, etc. when your objects are
> instantiated.  Associate this with each using object properties (or
> similar construct).  This way you have any required environment and
> precisely the source which generated the object, the user can inspect
> this if interested and it can be used to create new instances.

Well, this is more or less the way I do it. There is a well-defined
hierarchy of objects which can be dumped. But I want the system to
remain flexible, so that the user will be able to create his or her
own methods and functions (especially that anonymous functions are so
common in scheme), and therefore obtaining procedure names is not
enough.

> As someone mentioned earlier, the way macros are expanded there is no
> single point to define (internal to guile) what is the “source” of a
> procedure.  However, your system can define this as it has a clear
> point of object instantiation.

As I said, I don't want to loose flexibility. If I could rename lambda, like
(define primitive-lambda lambda)

then I could easily implement this functionality myself (making 'self'
a reserved keyword)

(define-syntax lambda
  (syntax-rules ()
    ((_ args body ...)
     (let ((self (primitive-lambda args body ...)))
       (set-procedure-property! self 'source (quote (primitive-lambda
args body ...)))
       self))))

> I have not worked on such a system myself, so am poking around in the
> dark here.  Good luck :-)

:) thanks

Re: procedure-source availability

[Ludovic Courtès]

Hi,

Panicz Maciej Godek <godek.maciek@gmail.com> skribis:

> I recall someone having the idea of adding smalltalk-like images to guile.
> I'd love to see such feature one day, but if guile is supposed to be
> an extension language, I think the C interface would need to be
> redesigned, because there would be a need to somehow dump the smobs
> from the heap to re-load them later.

At the C level, that seems somewhat ambitious.  :-)

> I don't know much about the implementation of the GOOPS objects, but I
> suspect that they would also require some means of serialization

The (oop goops save) module implements parts of a serialization
framework.  When GOOPS is loaded, any Scheme object is a GOOPS instance,
so one could define methods to serialize any type of object.

[...]

> then I could easily implement this functionality myself (making 'self'
> a reserved keyword)
>
> (define-syntax lambda
>   (syntax-rules ()
>     ((_ args body ...)
>      (let ((self (primitive-lambda args body ...)))
>        (set-procedure-property! self 'source (quote (primitive-lambda
> args body ...)))
>        self))))

Note that this doesn’t make ‘self’ a reserved keyword; instead, it’s
just a local variable that cannot be referred to by name in BODY, thanks
to the macro hygiene rules.

Thanks,
Ludo’.

Re: procedure-source availability

[Panicz Maciej Godek]

Howdie

> Instead of storing actual code that recreates the GUI, how about storing
> high-level declarations that describe that GUI?
[...]

Well, the problem is that I don't yet know what I am doing, so I'm
trying to keep the system as general as possible. One of the features
that is certainly going to be needed anyway, is a way to store and
restore lambdas, because this is the essential abstraction mechanism
of Scheme. Maybe some separate layer for GUI description will emerge
later on, but the task that I assigned to myself seems worth exploring

>> I recall someone having the idea of adding smalltalk-like images to guile.
>> I'd love to see such feature one day, but if guile is supposed to be
>> an extension language, I think the C interface would need to be
>> redesigned, because there would be a need to somehow dump the smobs
>> from the heap to re-load them later.
>
> At the C level, that seems somewhat ambitious.  :-)

Yep. :)

>> I don't know much about the implementation of the GOOPS objects, but I
>> suspect that they would also require some means of serialization
>
> The (oop goops save) module implements parts of a serialization
> framework.  When GOOPS is loaded, any Scheme object is a GOOPS instance,
> so one could define methods to serialize any type of object.

It looks interesting, but I somehow can't get it to run, even for very
simple cases:
> (use-modules (oop goops) (oop goops save))
> (save-objects '((a . 1)(b . 2)) (current-output-port))
(define a '1)
(define b '2)
ERROR: In procedure hashq-get-handle:
ERROR: In procedure hashq-get-handle: Handle access not permitted on weak table

> (version)
$1 = "2.0.5-deb+1-1"

Besides this, it looks similar to the thing I'm working on, so if I
get far enough, then perhaps my results could be incorporated into
that framework? (It requires some labour though, as there are many
types to be considered

>> then I could easily implement this functionality myself (making 'self'
>> a reserved keyword)
>>
>> (define-syntax lambda
>>   (syntax-rules ()
>>     ((_ args body ...)
>>      (let ((self (primitive-lambda args body ...)))
>>        (set-procedure-property! self 'source (quote (primitive-lambda
>> args body ...)))
>>        self))))
>
> Note that this doesn’t make ‘self’ a reserved keyword; instead, it’s
> just a local variable that cannot be referred to by name in BODY, thanks
> to the macro hygiene rules.

Yes, you're right.
I even went a little further with that and now I also capture lexical
environment:
(use-modules (system syntax) (ice-9 local-eval))

(define-macro (function args . body)
  `(let ((environment (the-environment))
         (lexical-names (lexical-names))
         (procedure (lambda ,args ,@body)))
     (set-procedure-property! procedure 'source '(function ,args ,@body))
     (set-procedure-property! procedure 'environment environment)
     (set-procedure-property! procedure 'lexical-names lexical-names)
     procedure))

(where ``lexical-names'' returns the car-s of ``lexicals'', as defined
at the bottom of
http://www.gnu.org/software/guile/manual/html_node/Syntax-Transformer-Helpers.html#Syntax-Transformer-Helpers
)

So in addition to the source of the procedure, the lexical environment
can also be retrieved:

(define (procedure-lexicals proc)
  (map (lambda(symbol)
         (cons symbol
               (local-eval symbol (procedure-property proc 'environment))))
   (procedure-property proc 'lexical-names)))

The strange thing was that I had to define the macro ``function''
using define-macro -- the define-syntax counterpart for some reason
wouldn't work. So for example, if I wrote
(define f (let ((y 5)) (function x (set! y (apply + y x)) y))
then if ``function'' was defined by means of define-syntax/syntax-rules, ie

(define-syntax function
  (syntax-rules ()
    ((_ args body ...)
     (let ((environment (the-environment))
           (lexical-names (lexical-names))
           (procedure (lambda args body ...)))
       (set-procedure-property! procedure 'source '(function args body ...))
       (set-procedure-property! procedure 'environment environment)
       (set-procedure-property! procedure 'lexical-names lexical-names)
       procedure))))

then the ``environment'' variable wouldn't capture the ``y'' (or
anything else, for that matter). I find that kinda weird, so I'm
sharing my doubts. Fortunately, the define-macro version behaves as
one could expect, so I can move on with my work

Thanks&regards
Maciek

Re: procedure-source availability

[Ludovic Courtès]

Hello!

Panicz Maciej Godek <godek.maciek@gmail.com> skribis:

> Well, the problem is that I don't yet know what I am doing, so I'm
> trying to keep the system as general as possible. One of the features
> that is certainly going to be needed anyway, is a way to store and
> restore lambdas, because this is the essential abstraction mechanism
> of Scheme. Maybe some separate layer for GUI description will emerge
> later on, but the task that I assigned to myself seems worth exploring

Yeah.  Scheme allows you to create the “essential abstraction mechanism”
that you like best anyway.  :-)

[...]

>> The (oop goops save) module implements parts of a serialization
>> framework.  When GOOPS is loaded, any Scheme object is a GOOPS instance,
>> so one could define methods to serialize any type of object.
>
> It looks interesting, but I somehow can't get it to run, even for very
> simple cases:

[...]

> Besides this, it looks similar to the thing I'm working on, so if I
> get far enough, then perhaps my results could be incorporated into
> that framework? (It requires some labour though, as there are many
> types to be considered

Yes, that’s what I had in mind.  :-)

[...]

>> Note that this doesn’t make ‘self’ a reserved keyword; instead, it’s
>> just a local variable that cannot be referred to by name in BODY, thanks
>> to the macro hygiene rules.
>
> Yes, you're right.
> I even went a little further with that and now I also capture lexical
> environment:
> (use-modules (system syntax) (ice-9 local-eval))
>
> (define-macro (function args . body)
>   `(let ((environment (the-environment))
>          (lexical-names (lexical-names))
>          (procedure (lambda ,args ,@body)))
>      (set-procedure-property! procedure 'source '(function ,args ,@body))
>      (set-procedure-property! procedure 'environment environment)
>      (set-procedure-property! procedure 'lexical-names lexical-names)
>      procedure))
>
> (where ``lexical-names'' returns the car-s of ``lexicals'', as defined
> at the bottom of
> http://www.gnu.org/software/guile/manual/html_node/Syntax-Transformer-Helpers.html#Syntax-Transformer-Helpers
> )
>
> So in addition to the source of the procedure, the lexical environment
> can also be retrieved:
>
> (define (procedure-lexicals proc)
>   (map (lambda(symbol)
>          (cons symbol
>                (local-eval symbol (procedure-property proc 'environment))))
>    (procedure-property proc 'lexical-names)))

I don’t want to sound too dogmatic, but I think you really don’t want to
take that route.  ;-)

The issues with ‘local-eval’ have been discussed at length a year ago or
so on guile-devel.  Basically, the problem is that it plays badly with
compilation, and there are usually nicer way to achieve what you want.

HTH,
Ludo’.

Re: procedure-source availability

[Panicz Maciej Godek]

Hellu!

>> Yes, you're right.
>> I even went a little further with that and now I also capture lexical
>> environment:
>> (use-modules (system syntax) (ice-9 local-eval))
>>
>> (define-macro (function args . body)
>>   `(let ((environment (the-environment))
>>          (lexical-names (lexical-names))
>>          (procedure (lambda ,args ,@body)))
>>      (set-procedure-property! procedure 'source '(function ,args ,@body))
>>      (set-procedure-property! procedure 'environment environment)
>>      (set-procedure-property! procedure 'lexical-names lexical-names)
>>      procedure))
>>
>> (where ``lexical-names'' returns the car-s of ``lexicals'', as defined
>> at the bottom of
>> http://www.gnu.org/software/guile/manual/html_node/Syntax-Transformer-Helpers.html#Syntax-Transformer-Helpers
>> )
>>
>> So in addition to the source of the procedure, the lexical environment
>> can also be retrieved:
>>
>> (define (procedure-lexicals proc)
>>   (map (lambda(symbol)
>>          (cons symbol
>>                (local-eval symbol (procedure-property proc 'environment))))
>>    (procedure-property proc 'lexical-names)))
>
> I don’t want to sound too dogmatic, but I think you really don’t want to
> take that route.  ;-)
>
> The issues with ‘local-eval’ have been discussed at length a year ago or
> so on guile-devel.  Basically, the problem is that it plays badly with
> compilation, and there are usually nicer way to achieve what you want.

Fortunately, calling local-eval isn't strictly necessary, as long as
the exciting and powerful `lexicals' macro is allowed. Then, the
`function' macro can be defined easily as:

(define-syntax function
  (lambda (x)
    (syntax-case x ()
      ((function args body ...)
       #'(let ((procedure (lambda args body ...))
               (get-lexicals (lambda()(lexicals function))))
           (set-procedure-property! procedure 'source '(function args body ...))
           (set-procedure-property! procedure 'get-lexicals get-lexicals)
           procedure)))))

(define (procedure-lexicals proc)
  ((procedure-property proc 'get-lexicals)))

I don't fully understand why it works, as syntax-case is still a
mystery to me, but I wrote it and it works perfectly, not to mention
that the expanded code is much cleaner now, without the-environment.

Re: procedure-source availability

[Mark H Weaver]

Panicz Maciej Godek <godek.maciek@gmail.com> writes:
> The strange thing was that I had to define the macro ``function''
> using define-macro -- the define-syntax counterpart for some reason
> wouldn't work. So for example, if I wrote
> (define f (let ((y 5)) (function x (set! y (apply + y x)) y))
> then if ``function'' was defined by means of define-syntax/syntax-rules, ie
>
> (define-syntax function
>   (syntax-rules ()
>     ((_ args body ...)
>      (let ((environment (the-environment))
>            (lexical-names (lexical-names))
>            (procedure (lambda args body ...)))
>        (set-procedure-property! procedure 'source '(function args body ...))
>        (set-procedure-property! procedure 'environment environment)
>        (set-procedure-property! procedure 'lexical-names lexical-names)
>        procedure))))
>
> then the ``environment'' variable wouldn't capture the ``y'' (or
> anything else, for that matter).

(the-environment) captures the lexical environment where it is found,
and in this case that means the lexical environment of this macro
definition.

Although it is not documented, 'the-environment' accepts an optional
argument, which must be an identifier.  If provided, the lexical
environment captured is the one where that identifier was created.

In theory, this means that this should do what you want:

--8<---------------cut here---------------start------------->8---
(define-syntax function
  (syntax-rules ()
    ((function args body ...)
     (let ((environment (the-environment function))
           (lexical-names (map car (lexicals function)))
           (procedure (lambda args body ...)))
       (set-procedure-property! procedure 'source '(function args body ...))
       (set-procedure-property! procedure 'environment environment)
       (set-procedure-property! procedure 'lexical-names lexical-names)
       procedure))))
--8<---------------cut here---------------end--------------->8---

Unfortunately, our definition of 'syntax-rules' needlessly discards the
keyword identifier.  This should probably be fixed, but in the meantime
you can use 'syntax-case' instead:

--8<---------------cut here---------------start------------->8---
(define-syntax function
  (lambda (x)
    (syntax-case x ()
      ((function args body ...)
       #'(let ((environment (the-environment function))
               (lexical-names (map car (lexicals function)))
               (procedure (lambda args body ...)))
           (set-procedure-property! procedure 'source '(function args body ...))
           (set-procedure-property! procedure 'environment environment)
           (set-procedure-property! procedure 'lexical-names lexical-names)
           procedure)))))
--8<---------------cut here---------------end--------------->8---

Having said all this, I agree with Ludovic that this is probably not a
good approach.  'the-environment' inhibits almost all optimizations in
the compiler.

     Regards,
       Mark

Re: procedure-source availability

[Ludovic Courtès]

Hi,

Panicz Maciej Godek <godek.maciek@gmail.com> skribis:

> Well, the idea for now is that the associated .spec file containing
> the state of GUI is loaded on startup, and the state of the
> interpreter is dumped to that file on exit (or at GUI's request).
> Viewing the file will obviously be an option (for the curious user),
> but any modifications would probably be overwritten eventually (unless
> the file is write-protected).
>
> This approach allows avoiding the design of any specific file format
> to store information about the GUI -- everything is just scheme. The
> only requirement is that all used object (images etc.) can be dumped
> to scheme expressions, evaluation of which would re-create them. (This
> isn't yet fully implemented, but I'm on a good way)

This sounds like an interesting approach.

Instead of storing actual code that recreates the GUI, how about storing
high-level declarations that describe that GUI?

For instance, Ratpoison & co. store declarations like this:

  (frame :number 0 :x 0 :y 0 :width 1320 :height 1200 :screenw 1920
         :screenh 2000
         :window 12582930 :last-access 6 :dedicated 0)

They could instead store actual code (say, a procedure that creates a
window of the right size, using the X11 API, etc.).  However, the
advantage of declarative code is that it’s easy to maintain
backward-compatibility (compared to regular API churn), and it also
allows for portability between implementations (for instance, Ratpoison
vs. StumpWM vs. RPX.)

And, it’s also easily serialized.

Hope I’m not too off-topic.  ;-)

Ludo’.