Emacs Lisp, macros

Emacs Lisp, macros

[Daniel Kraft]

Hi Andy and all,

yesterday and today I implemented lambda expressions and 
defvar/defconst/defun for elisp as well as found a (hopefully good) 
solution to automatically create fluids not yet present on reference. 
With these additions and some basic built-ins (mainly arithmetic/numbers 
so far, but trivial to extend), I think elisp is already quite usable as 
a language for simple tasks.  I just pushed the elisp branch with my 
additions.

For instance, this simple code works in guile's elisp language:

; Check if p is prime, but start looking for divisors at from if
; present.  It seems that recursion using anonymous functions is
; not that nicely to do in elisp as in scheme (at least not before
; defining funcall/apply), so this is done using while.
(defun primep (p &optional from)
   (let ((i (if from from 2)))
     (while (and (/= (% p i) 0) (<= (* i i) p))
       (setq i (1+ i)))
     (/= (% p i) 0)))

(primep 1283939) -> #t
(primep 1283937) -> #f
(primep 1283937 1000) -> #t

While of course the #f should be %nil, this is simply because I use #f 
at the moment for nil (but that will be redefined once the falsity of 
%nil is corrected).

Besides some "important" built-ins like makunbound, boundp, set, 
funcall, apply, fset and others, I think the next big thing to work on 
will be macro support (for a more extensive list of implemented things 
and stuff still missing see module/language/elisp/README) as well as a 
real testsuite.

For macros (and also for funcall/apply, which ought to be able to 
execute functions in form of an uncompiled list like '(lambda (a b) (+ a 
b))) I will probably need some means of compiling and executing a 
certain piece of code 'by hand' from within compilation.  It seems that 
I can do something along

((compile <my code here> #:from 'elisp) arguments to compiled code)

to do this; or is there some other way to do this that would be better 
for this situation?

Regarding the test-suite:  It seems to me there's no 'very extensive' 
and 'complete' single test-suite for guile at the moment, but I might be 
mistaken.  So maybe I should create testsuite/elisp and a 
run-elisp-tests.scm like the one for vm which loads/compiles/runs and 
all that the individual tests?  Or is there some existing framework I 
can plug my tests into?

Good night!

Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri

Re: Emacs Lisp, macros

[Ludovic Courtès]

Hi Daniel!

Daniel Kraft <d@domob.eu> writes:

> yesterday and today I implemented lambda expressions and
> defvar/defconst/defun for elisp

Great news!

> as well as found a (hopefully good) solution to automatically create
> fluids not yet present on reference. 

Is each Elisp variable mapped to a fluid?  Eventually, you may need VM
instructions for fluid-{ref,set!}, to speed things up.

[...]

> Regarding the test-suite:  It seems to me there's no 'very extensive'
> and 'complete' single test-suite for guile at the moment, but I might
> be mistaken.  So maybe I should create testsuite/elisp and a
> run-elisp-tests.scm like the one for vm which loads/compiles/runs and
> all that the individual tests?  Or is there some existing framework I
> can plug my tests into?

There's 1.5 framework.  The first one is under `test-suite/tests', used
to test the API.  The remaining 0.5 is under `testsuite': it's inherited
from Guile-VM and is used to make sure interpreted and compiled code
yield the same result.

I think the best way would be to add, say,
`test-suite/tests/elisp-compiler.test' (there's already `elisp.test'),
and from there use a macro akin to Andy's `c&e' in `bytevectors.test'.

Thanks,
Ludo'.

Re: Emacs Lisp, macros

[Daniel Kraft]

Ludovic Courtès wrote:
>> as well as found a (hopefully good) solution to automatically create
>> fluids not yet present on reference. 
> 
> Is each Elisp variable mapped to a fluid?  Eventually, you may need VM
> instructions for fluid-{ref,set!}, to speed things up.

Yes it is; and I agree that variable references surely have terrible 
performance at the moment, because of other things that have to go on 
for each reference:

* see if there's a fluid with that name; create if not (but creation is 
only done once of course, so the performance hit is from the check only)

* reference the fluid to get the current value

* check if the value is the special elisp void, in which case an error 
is reported

So taking this all together, I think there's a lot of potential for 
optimization here!  Unfortunatly I think that all this is necessary to 
ensure the semantics, but two ideas:

1) As you suggested, try doing some parts of this with new VM 
operations.  Like all of these in one op, or maybe just a "reference and 
error on void" as one op, and/or "lookup the variable in some module, 
and create it if not there" as another.  I think we need some planning 
to get the right choice, but it certainly is an important thing for 
performance.  But for now I'd suggest to keep on with the current 
implementation and later see what the real performance bottle-neck is 
when running real-world code.

2) I already mentioned some ideas like that before, but if we do careful 
analysis of the elisp code, we can probably get rid of some 
fluid-references; for instance:

(let ((a 5))
   (setq a 1)
   (while ... something which references a a lot)
   (foobar 1 2 3))

In order to preserve dynamic scoping for a in the call to foobar, we 
still have to save a in a fluid as well as do the setq on the fluid 
value.  But all references to a within the while-loop (if they are 
reads) are known to refer to the let-bound a, so we could optimize them 
and not take a fluid-ref every now and then.

I think, however, that this can get fairly complicated and need a lot of 
tweeking in order to get a sensible optimization.  So for now I think we 
should try getting the real bottle-necks into VM opterations and see 
what that can buy.

> There's 1.5 framework.  The first one is under `test-suite/tests', used
> to test the API.  The remaining 0.5 is under `testsuite': it's inherited
> from Guile-VM and is used to make sure interpreted and compiled code
> yield the same result.
> 
> I think the best way would be to add, say,
> `test-suite/tests/elisp-compiler.test' (there's already `elisp.test'),
> and from there use a macro akin to Andy's `c&e' in `bytevectors.test'.

Ok thanks for the hint!  So far I did mainly look into testsuite, but 
then I'll take a look at the framework in test-suite and see if I can do 
something useful there :)

cheers,
Daniel

Re: Emacs Lisp, macros

[Andy Wingo]

Hello!

On Tue 14 Jul 2009 21:48, Daniel Kraft <d@domob.eu> writes:

> (defun primep (p &optional from)
>   (let ((i (if from from 2)))
>     (while (and (/= (% p i) 0) (<= (* i i) p))
>       (setq i (1+ i)))
>     (/= (% p i) 0)))
>
> (primep 1283939) -> #t

This is fantastic.

> While of course the #f should be %nil, this is simply because I use #f
> at the moment for nil (but that will be redefined once the falsity of
> %nil is corrected).

Yes. Hopefully we can pull in Mark's patch by the next release.

> For macros (and also for funcall/apply, which ought to be able to
> execute functions in form of an uncompiled list like '(lambda (a b) (+ a
> b))) I will probably need some means of compiling and executing a
> certain piece of code 'by hand' from within compilation.  It seems that
> I can do something along
>
> ((compile <my code here> #:from 'elisp) arguments to compiled code)

Yes this is exactly right.

> Regarding the test-suite:  It seems to me there's no 'very extensive'
> and 'complete' single test-suite for guile at the moment, but I might be
> mistaken.  So maybe I should create testsuite/elisp and a
> run-elisp-tests.scm like the one for vm which loads/compiles/runs and
> all that the individual tests?  Or is there some existing framework I
> can plug my tests into?

You should add tests under test-suite/test/elisp.test (note the dash),
and add your file to TESTS in test/Makefile.am. But you've probably
figured that out already :)

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: Emacs Lisp, macros

[Andy Wingo]

On Wed 15 Jul 2009 10:36, Daniel Kraft <d@domob.eu> writes:

> Ludovic Courtès wrote:
>>> as well as found a (hopefully good) solution to automatically create
>>> fluids not yet present on reference. 
>>
>> Is each Elisp variable mapped to a fluid?  Eventually, you may need VM
>> instructions for fluid-{ref,set!}, to speed things up.
>
> Yes it is; and I agree that variable references surely have terrible
> performance at the moment, because of other things that have to go on
> for each reference:
>
> * see if there's a fluid with that name; create if not (but creation is
> only done once of course, so the performance hit is from the check only)
>
> * reference the fluid to get the current value
>
> * check if the value is the special elisp void, in which case an error
> is reported
>
> So taking this all together, I think there's a lot of potential for
> optimization here!  Unfortunatly I think that all this is necessary to
> ensure the semantics, but two ideas:
>
> 1) As you suggested, try doing some parts of this with new VM
> operations.  Like all of these in one op, or maybe just a "reference and
> error on void" as one op, and/or "lookup the variable in some module,
> and create it if not there" as another.  I think we need some planning
> to get the right choice, but it certainly is an important thing for
> performance.  But for now I'd suggest to keep on with the current
> implementation and later see what the real performance bottle-neck is
> when running real-world code.

So, agreed wrt perspective on when to optimize. I still think that we
should be able to cache fluid locations in the same way that Scheme
caches variable locations, and add dynamic-ref / dynamic-set analogs to
toplevel-ref / toplevel-set. See their implementations for details on
what I mean.

> 2) I already mentioned some ideas like that before, but if we do careful
> analysis of the elisp code, we can probably get rid of some
> fluid-references; for instance:
>
> (let ((a 5))
>   (setq a 1)
>   (while ... something which references a a lot)
>   (foobar 1 2 3))

I wouldn't bother, to be honest. dynamic-ref will be plenty fast, and a
has to be on the heap anyway, unless the compiler does a lot more
analysis than it currently does.

> I think, however, that this can get fairly complicated and need a lot of
> tweeking in order to get a sensible optimization.  So for now I think we
> should try getting the real bottle-necks into VM opterations and see
> what that can buy.

Yes :)

Peace,

Andy
-- 
http://wingolog.org/

Re: Emacs Lisp, macros

[Daniel Kraft]

Andy Wingo wrote:
>> 1) As you suggested, try doing some parts of this with new VM
>> operations.  Like all of these in one op, or maybe just a "reference and
>> error on void" as one op, and/or "lookup the variable in some module,
>> and create it if not there" as another.  I think we need some planning
>> to get the right choice, but it certainly is an important thing for
>> performance.  But for now I'd suggest to keep on with the current
>> implementation and later see what the real performance bottle-neck is
>> when running real-world code.
> 
> So, agreed wrt perspective on when to optimize. I still think that we
> should be able to cache fluid locations in the same way that Scheme
> caches variable locations, and add dynamic-ref / dynamic-set analogs to
> toplevel-ref / toplevel-set. See their implementations for details on
> what I mean.

You mean, (dynamic-ref fluid) equvialent to (fluid-ref fluid) but just 
VM optimized, and the same for dynamic-set!?  Sounds nice!

Well, just some notes about variable references and performance;  there 
are still those three things that have to be done I mentioned already 
some time before (for setting a variable, 1) and 2) will be relevant):

1) Make sure the fluid itself is there for each variable needed and 
create a new one if not.

I did this once on every access, but now I'm scanning each compiled 
expression for variable references and ensure all fluids for them 
beforehand, so that this is only done once per compiled expression and 
not, say, on each iteration in a loop.  This brought runtime for one 
test down from 2.8s to 0.1s...  This pretty amazed me, and I think the 
current solution is quite ok.

2) Reference the fluid.

This is what we could do with VM operations, but I've not yet done tests 
to find out how much time this really costs compared to 3).

3) Check if value is void and error if it is.

Maybe we could also do this in a VM operation (and then it would 
probably cost nearly nothing at all?), but I think that maybe just 
optionally getting rid of this test at all (see my remark about compiler 
options) is the better solution.

When I've implemented the options to disable 2) and/or 3), I'll do some 
timings and let you know about that.  Then we can decide if VM ops for 
the fluids are worth it and how much we can gain by them.

Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri