Writing a function/macro in Elisp that generates a function at runtime

Writing a function/macro in Elisp that generates a function at runtime

[Toby Cubitt]

Hi,

I'm trying to write a function in Elisp that generates a function at
runtime, but hitting the limits of my Lisp abilities. Here's a first
attempt at the kind of thing I'm trying to do:

(defun wrap-insert-function (insfun)
  `(lambda (new-data old-cell)
     (setf (cell-data old-cell)
           (funcall ,insfun new-data (cell-data old-cell)))
     old-cell))

I want to use this to wrap functions that don't know anything about the
"cell" data structures, so they can operate on them:

(setq wrapped-insert-function
  (wrap-insert-function
   (some-complicated-code-that-returns-a-suitable-function)))

(funcall wrapped-insert-function data cell)


The problem is, because it's quoted, the `setf' macro never gets
expanded by the byte-compiler. And that's no good, because the cl
package where `setf' is defined shouldn't be loaded at runtime, and
anyway we want it to be expanded at compile-time, not at run-time as
that would be ugly and inefficient.

There has got to be some canonical way to do this in Lisp, but I've
tried playing around with various macro/function definitions, searching
the web, and reading Lisp books, without getting any good ideas. The
vital clue might well be in one of the chapters on advanced Lisp macro
magic (e.g. Paul Graham's "On Lisp"). But if so, then spotting the clue
is beyond my current Lisp skills, and I could use a guru to point me in
the right direction.

Any gurus out there?

Toby


PS: I've tried to simplify my requirements down to a concise example,
but I apologize in advance if there's some additional constraint I've
forgotten to mention, and which only occurs to me later when I realise a
proposed solution won't work.

Re: Writing a function/macro in Elisp that generates a function at runtime

[Barry Margolin]

In article <mailman.2216.1225168079.25473.help-gnu-emacs@gnu.org>,
 Toby Cubitt <tsc25@cantab.net> wrote:

> Hi,
> 
> I'm trying to write a function in Elisp that generates a function at
> runtime, but hitting the limits of my Lisp abilities. Here's a first
> attempt at the kind of thing I'm trying to do:
> 
> (defun wrap-insert-function (insfun)
>   `(lambda (new-data old-cell)
>      (setf (cell-data old-cell)
>            (funcall ,insfun new-data (cell-data old-cell)))
>      old-cell))
> 
> I want to use this to wrap functions that don't know anything about the
> "cell" data structures, so they can operate on them:
> 
> (setq wrapped-insert-function
>   (wrap-insert-function
>    (some-complicated-code-that-returns-a-suitable-function)))
> 
> (funcall wrapped-insert-function data cell)
> 
> 
> The problem is, because it's quoted, the `setf' macro never gets
> expanded by the byte-compiler. And that's no good, because the cl
> package where `setf' is defined shouldn't be loaded at runtime, and

To do that you'd need lexical closures, but Elisp doesn't have them.  
Since you're not generating the function until runtime, there's nothing 
to compile at compile time.

> anyway we want it to be expanded at compile-time, not at run-time as
> that would be ugly and inefficient.

You don't need to generate a function on the fly for this.  Do something 
like this:

(defun insert-with-function (insfun new-data old-cell)
  (setf (cell-data old-cell)
        (funcall insfun new-data (cell-data old-cell)))
  old-cell)

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Re: Writing a function/macro in Elisp that generates a function at runtime

[Toby Cubitt]

Barry Margolin wrote:
> In article <mailman.2216.1225168079.25473.help-gnu-emacs@gnu.org>,
>  Toby Cubitt <tsc25@cantab.net> wrote:
>> I'm trying to write a function in Elisp that generates a function at
>> runtime, but hitting the limits of my Lisp abilities. Here's a first
>> attempt at the kind of thing I'm trying to do:
>>
>> (defun wrap-insert-function (insfun)
>>   `(lambda (new-data old-cell)
>>      (setf (cell-data old-cell)
>>            (funcall ,insfun new-data (cell-data old-cell)))
>>      old-cell))
[snip]
>> The problem is, because it's quoted, the `setf' macro never gets
>> expanded by the byte-compiler. And that's no good, because the cl
>> package where `setf' is defined shouldn't be loaded at runtime, and
> 
> To do that you'd need lexical closures, but Elisp doesn't have them.  
> Since you're not generating the function until runtime, there's nothing 
> to compile at compile time.

I can start to see how to do it with lexical closures. (How many times
have I wished for them in Elisp! Bring on the lexbind branch...). But is
there really no way to do this without closures? That seems strange. The
above definition works fine, except for the setf issue. As you say, the
run-time generated function obviously can't be compiled, because it
doesn't exist at compile-time. But that's not what I asked for. I want
the setf macro to be *expanded* at compile time into the setf method for
`cell-data', and there *is* enough information at compile-time for that.

I had another go, and came up with the following attempt (I can hear the
Elisp experts whincing already - I know it's ugly!)

(defmacro wrap-insfun-2 (f-2)
  (let ((comma-f `(nil ,f-2)))
    (setcar comma-f ',)
    (macroexpand-all
     `(lambda (new old)
	(setf (cell-data old)
	      (,comma-f (cell-data new)
			(cell-data old)))))))

(defmacro wrap-insfun-1 (f-1)
  `(eval (backquote ,(macroexpand-all `(wrap-insfun-2 ,f-1)))))

(defun wrap-insfun (insfun)
  (wrap-insfun-1 insfun))

This works as long as `wrap-insfun-1' is not byte-compiled (I don't
completely understand why it fails when byte-compiled, but I guess it's
because `backquote' does't work on the byte-compiled expansion of
`wrap-insfun-2'.)

I was trying to have macros generate the body of the wrap-insfun
function, so that I could force setf to be macro-expanded when compiled.
 Getting macros to generate code containing a backquote construct seems
to be tricky, I guess because backquote expansion gets confused by the
commas that are supposed to be part of the generated code, hence hiding
this inside `comma-f'.

Is generating a backquote construct possible using macros, in a robust
way that byte-compiles properly, and without the ugly hack of making use
of Emacs's internal old-style backquote implementation as I've done? Or
is this simply impossible? Or is Elisp lacking some Lisp feature that
would allow this? (reader macros?)


> You don't need to generate a function on the fly for this.  Do something 
> like this:
> 
> (defun insert-with-function (insfun new-data old-cell)
>   (setf (cell-data old-cell)
>         (funcall insfun new-data (cell-data old-cell)))
>   old-cell)

Hmmmm...the simplest ideas are always best. This does sound like a
better approach. Thanks!

I'm still curious about the above macro shenanigans though, since it's
pushing at the boundaries of my understanding of Lisp and I'd like to
understand these issues more fully.

Toby

Re: Writing a function/macro in Elisp that generates a function at runtime

[Toby Cubitt]

Toby Cubitt wrote:
> Barry Margolin wrote:
>> You don't need to generate a function on the fly for this.  Do something 
>> like this:
>>
>> (defun insert-with-function (insfun new-data old-cell)
>>   (setf (cell-data old-cell)
>>         (funcall insfun new-data (cell-data old-cell)))
>>   old-cell)
> 
> Hmmmm...the simplest ideas are always best. This does sound like a
> better approach. Thanks!

I've now remembered why this won't work, at least not straightforwardly.

I have a higher-level data structure built on top of a lower-level one.
Internally, the higher-level structure stores data "cells" in the
lower-level structure. The lower-level structure of course knows nothing
about this. It just stores any kind of data, without knowing anything
about its structure. And users of the higher-level structure also know
nothing about the data cells, which are part of the internal
implementation. As far as users are concerned, they can just store data
of whatever kind they like in the higher-level structure.

For the lower-level structure, I have defined exactly this kind of
insert-with-function, called say `low-level-insert-with-function'. What
I'm trying to do is to define a `high-level-insert-with-function' for
the higher-level data structure. Clearly, it has to call
`low-level-insert-with-function' to insert the data into the lower-level
structure on which it is build, something like:

(defun high-level-insert-with-function (higher-structure data insfun)
  (low-level-insert-with-function
   (higher-structure--get-lower-structure higher-structure)
   data ??insfun??))

Now insfun knows nothing about the data cells, since it was passed in by
the user. And `low-level-insert-with-function' knows nothing about data
cells, since it acts on the lower-level data structure.

So it would seem I need to wrap insfun on the fly in the above function,
to make it aware that the data is actually stored within "cells". And
we're back to the problem from my previous mail: how to generate a
wrapped insfun on the fly, in such a way that the setf macro required to
enter data into a "cell" is expanded into the cell-data accessor
function at compile-time.

I still don't see why this should be fundamentally impossible. The
function can easily be generated on the fly by a backquote construct (as
in my first attempt), and all the necessary information for
macro-expanding setf is there at compile-time. It's 'just' a question of
getting macro-expansion to be carried out within the backquote construct
(which is what I was trying in my second attempt).

Probably there's a different way of approaching the problem that I'm not
seeing.

Thanks for your help (and sorry for the lengthy explanation),

Toby

Re: Writing a function/macro in Elisp that generates a function at runtime

[Ted Zlatanov]

On Tue, 28 Oct 2008 13:42:57 +0100 Toby Cubitt <tsc25@cantab.net> wrote: 

TC> I have a higher-level data structure built on top of a lower-level one.
TC> Internally, the higher-level structure stores data "cells" in the
TC> lower-level structure. The lower-level structure of course knows nothing
TC> about this. It just stores any kind of data, without knowing anything
TC> about its structure. And users of the higher-level structure also know
TC> nothing about the data cells, which are part of the internal
TC> implementation. As far as users are concerned, they can just store data
TC> of whatever kind they like in the higher-level structure.

TC> For the lower-level structure, I have defined exactly this kind of
TC> insert-with-function, called say `low-level-insert-with-function'. What
TC> I'm trying to do is to define a `high-level-insert-with-function' for
TC> the higher-level data structure. Clearly, it has to call
TC> `low-level-insert-with-function' to insert the data into the lower-level
TC> structure on which it is build, something like:

TC> (defun high-level-insert-with-function (higher-structure data insfun)
TC>   (low-level-insert-with-function
TC>    (higher-structure--get-lower-structure higher-structure)
TC>    data ??insfun??))

TC> Now insfun knows nothing about the data cells, since it was passed in by
TC> the user. And `low-level-insert-with-function' knows nothing about data
TC> cells, since it acts on the lower-level data structure.

TC> So it would seem I need to wrap insfun on the fly in the above function,
TC> to make it aware that the data is actually stored within "cells". And
TC> we're back to the problem from my previous mail: how to generate a
TC> wrapped insfun on the fly, in such a way that the setf macro required to
TC> enter data into a "cell" is expanded into the cell-data accessor
TC> function at compile-time.

I don't understand why you're trying to cross between the storage and
the logic (essentially, that's what your "low level" and "high level"
layers are).  The logic layer should know how to convert its data to and
from a neutral data format (in Lisp, that's usually a list).  The
storage layer should just store and retrieve the neutral data format.

Your approach makes sense, IMO, in an OOP environment where every object
knows how to serialize itself.  I got the following from
http://dorophone.blogspot.com/ and it may help you to sort-of-do
serializable objects if that's your goal...  You just print the value of
each object you create, and later you can eval them back in.

HTH
Ted

;;; from J.V. Toups http://dorophone.blogspot.com/
(defun toups-bang (sym)
  (intern (format "%s!" sym)))
(defun toups-s-cat (sym1 sym2)
  (intern (format "%s-%s" sym1 sym2)))
(defun toups-ques (sym)
  (intern (format "%s?" sym)))

(defmacro defstruquine (name &rest slots)
  (let* ((n-fields (length slots))
   (i 1)
   (out `(progn
     (defun ,(toups-bang name) ,slots
       (list ',(toups-bang name) ,@slots)) 
     (defun ,(toups-ques name) (item)
       (eq (car item) ',(toups-bang name))))))
 (loop for slot in slots do
    (setf out 
    (append out
      (list `(defun ,(toups-s-cat name slot) (item) (elt item ,i)))))
    (setf i (+ i 1)))
 (append out (list nil))))

;; Which can be used thusly:

;; (defstruquine person first-name last-name age)

;; (let ((p (person! "Edward" "Olmos" 61)))
;;   (person-first-name p) ;; is "Edward"
;;   (person-age p) ;; is 62
;;   p) ;; returns (person! "Edward" "Olmos" 61)

Re: Writing a function/macro in Elisp that generates a function at runtime

[Toby Cubitt]

Hi Ted,

Thanks for taking the time to reply.

Ted Zlatanov wrote:
> On Tue, 28 Oct 2008 13:42:57 +0100 Toby Cubitt <tsc25@cantab.net> wrote: 
> 
> TC> I have a higher-level data structure built on top of a lower-level one.
> TC> Internally, the higher-level structure stores data "cells" in the
> TC> lower-level structure. The lower-level structure of course knows nothing
> TC> about this. It just stores any kind of data, without knowing anything
> TC> about its structure. And users of the higher-level structure also know
> TC> nothing about the data cells, which are part of the internal
> TC> implementation. As far as users are concerned, they can just store data
> TC> of whatever kind they like in the higher-level structure.
[snip]

> I don't understand why you're trying to cross between the storage and
> the logic (essentially, that's what your "low level" and "high level"
> layers are).  The logic layer should know how to convert its data to and
> from a neutral data format (in Lisp, that's usually a list).  The
> storage layer should just store and retrieve the neutral data format.

It's difficult to motivate it whilst trying to avoid explaining the
entire code :)  I don't really want to get into a discussion here about
general design issues, since it's not appropriate to this list. The
trouble is, it's difficult to explain why I can't just use the simple
solution proposed by Barry Margolin without explaining a bit of the
background.

I'm not too sure what you mean by "logic layer", but I'm not doing
anything as complicated as that sounds. The data structures are just
that: data structures (they're a kind of tree, with some extra gubbins
on top), along with a bunch of functions for using them. (In OOP these
"higher-level" and "lower-level" data structures would be two different
classes, the functions would be class methods, and the "high-level"
class would contain an instance of the "low-level" one. The 'has-a'
relationship, i.e. one containing the other, is the reason I was calling
them "higher-level" and "lower-level", nothing more complicated than that.)


> Your approach makes sense, IMO, in an OOP environment where every object
> knows how to serialize itself.

Off-topic, one thing I love about Lisp is that it's so easy to take the
good parts of OOP and use them directly, without any of the OOP
complexity and mess. The good bits are already there as soon as
functions are first-class objects!.....Any replies to these comments
should be directed off-list, though :)


> I got the following from
> http://dorophone.blogspot.com/ and it may help you to sort-of-do
> serializable objects if that's your goal...  You just print the value of
> each object you create, and later you can eval them back in.

Interesting, but this is certainly overkill for what I'm doing.

Toby

Re: Writing a function/macro in Elisp that generates a function at runtime

[Toby Cubitt]

Johan Bockgård wrote:
> Toby Cubitt <tsc25@cantab.net> writes:
> 
>> (defun wrap-insert-function (insfun)
>>   `(lambda (new-data old-cell)
>>      (setf (cell-data old-cell)
>>            (funcall ,insfun new-data (cell-data old-cell)))
>>      old-cell))
> 
> (lexical-let ((insfun insfun))
>   (lambda ...))

Hah! I knew I could do it with closures, and I knew elisp didn't have
them, and I completely forgot that the CL package provided a way of
simulating them. Thanks a lot!

I expect anything else I could come up with would end up being an
ad-hoc, informally-specified, bug-ridden, slow implementation of half of
lexical-let :)

Out of sheer stubborn curiosity, I'd still like to know whether macros
can be used to generate backquote constructs programmatically...

Toby

Re: Writing a function/macro in Elisp that generates a function at runtime

[Johan Bockgård]

Toby Cubitt <tsc25@cantab.net> writes:

> (defun wrap-insert-function (insfun)
>   `(lambda (new-data old-cell)
>      (setf (cell-data old-cell)
>            (funcall ,insfun new-data (cell-data old-cell)))
>      old-cell))

(lexical-let ((insfun insfun))
  (lambda ...))