interactive closure — variables not bound

interactive closure — variables not bound

[Ricardo Wurmus]

Hi emacs-devel,

I’m currently attempting to rewrite some of the procedures and commands
in xwidget.el to do without the “title hack”.  Currently, the return
value of any JavaScript snippet that is run in WebKit is stored in the
title and then read out by running JavaScript to retrieve the title.

With the WebKit2 API all JavaScript is run asynchronously.  I’ve changed
“xwidget-webkit-execute-script” such that it takes an optional third
argument holding a Lisp callback, which is invoked with the return value
from JavaScript using “call1”.  This works fine for the most part,
although using callbacks is a little less convenient than doing things
synchronously (which is not supported by WebKit2).

Now there is at least one instance where things aren’t as smooth.  Take
“xwidget-webkit-insert-string” for example.  It is a command using
either “read-string” or “read-passwd” dependent on the input field
type.  The input field type is retrieved using JavaScript.  Using the
title hack this can be implemented in a faux synchronous fashion:

 (defun xwidget-webkit-insert-string (xw str)
   …
  (interactive
   (let* ((xww …)
          (field-value …)
          (field-type (xwidget-webkit-execute-script-rv
                       xww
                       "findactiveelement(document).type;")))
     (list xww
           (cond ((equal "text" field-type)
                  (read-string "Text: " field-value))
                 ((equal "password" field-type)
                  (read-passwd "Password: " nil field-value))
                 ((equal "textarea" field-type)
                  …)))))
   … ; do things with “xw” and “str”
   )

Using callbacks instead we have a problem, because the return value of
the JavaScript is only available in the callback procedure, not in the
“interactive” form.  To go around this problem I’m now trying to split
up “xwidget-webkit-insert-string” in two parts:

* “xwidget-webkit-insert-string”, which remains the interactive command
  exposed to the user.  It runs the JavaScript expression and in the
  callback passes the return value to …

* … a second interactive command, which takes care of prompting the user
  for a string to input.

This looks something like this:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(defun xwidget-webkit-insert-string ()
  (interactive)
  (let ((xww (xwidget-webkit-current-session)))
    (xwidget-webkit-execute-script
     xww
     (concat xwidget-webkit-activeelement-js
             "var res = findactiveelement(document); [res.value, res.type];")

     ;; This is called with “call1” from within “xwidget.c” when the
     ;; JavaScript return value is available
     (lambda (field)
       (let* ((field-value (car field))
              (field-type  (cadr field)))
         (call-interactively
          (lambda (str)
            (interactive
             (list (cond ((equal "text" field-type)
                          (read-string "Text: " field-value))
                         ((equal "password" field-type)
                          (read-passwd "Password: " nil field-value))
                         ((equal "textarea" field-type)
                          (xwidget-webkit-begin-edit-textarea xww field-value)))))
            (xwidget-webkit-execute-script
             xww
             (format "findactiveelement(document).value='%s'" str)))))))))
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The problem with this definition is that it doesn’t work (the other
problem is that I’m replacing one ugly hack with another).  At runtime
Emacs says that “field-type” is undefined.  At compile time Emacs says
that in the callback “xww”, “field-value”, and “field-type” are
references to free variables.

But why?  Isn’t this a closure?  “field-value” and “field-type” are
let-bound inside of the callback, so they should be available in the
scope of the interactive lambda’s “cond”.

I’ve also tried to do without “interactive” and just use “read-string”
directly, but while this gives me a prompt at runtime it also freezes
Emacs and eats up all my memory (which might be normal when using
“read-string” in a lambda that’s called from C with “call1”, dunno).

~~ Ricardo


PS: I’ve sent an email to assign@gnu.org a week ago with a completed
request form template, but haven’t received any response yet.

Re: interactive closure — variables not bound

[Stefan Monnier]

> The problem with this definition is that it doesn’t work (the other
> problem is that I’m replacing one ugly hack with another).  At runtime
> Emacs says that “field-type” is undefined.  At compile time Emacs says
> that in the callback “xww”, “field-value”, and “field-type” are
> references to free variables.

Indeed, currently, the `interactive' spec can't be re-created
individually for every closure.  IOW the spec is built once and forall
for a given lambda expression and hence can't refer to surrounding
non-global variables.

I suggest you M-x report-emacs-bug.

This said, in your example, I don't see what benefit you expect to get
from writing

         (call-interactively
          (lambda (str)
            (interactive
             (list (cond ((equal "text" field-type)
                          (read-string "Text: " field-value))
                         ((equal "password" field-type)
                          (read-passwd "Password: " nil field-value))
                         ((equal "textarea" field-type)
                          (xwidget-webkit-begin-edit-textarea xww field-value)))))
            (xwidget-webkit-execute-script
             xww
             (format "findactiveelement(document).value='%s'" str)))))))))

instead of

         (let ((str (cond ((equal "text" field-type)
                           (read-string "Text: " field-value))
                          ((equal "password" field-type)
                           (read-passwd "Password: " nil field-value))
                          ((equal "textarea" field-type)
                           (xwidget-webkit-begin-edit-textarea xww field-value)))))
          (xwidget-webkit-execute-script
           xww
           (format "findactiveelement(document).value='%s'" str)))))))))

[ Oh, and while I'm here, let me advertize pcase:

     (let ((str (pcase field-type
                  ("text" (read-string "Text: " field-value))
                  ("password" (read-passwd "Password: " nil field-value))
                  ("textarea" ...

]

        Stefan

Re: interactive closure — variables not bound

[Ricardo Wurmus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The problem with this definition is that it doesn’t work (the other
>> problem is that I’m replacing one ugly hack with another).  At runtime
>> Emacs says that “field-type” is undefined.  At compile time Emacs says
>> that in the callback “xww”, “field-value”, and “field-type” are
>> references to free variables.
>
> Indeed, currently, the `interactive' spec can't be re-created
> individually for every closure.  IOW the spec is built once and forall
> for a given lambda expression and hence can't refer to surrounding
> non-global variables.
>
> I suggest you M-x report-emacs-bug.

Okay, I will report a bug.

> This said, in your example, I don't see what benefit you expect to get
> from writing
>
>          (call-interactively
>           (lambda (str)
>             (interactive
>              (list (cond ((equal "text" field-type)
>                           (read-string "Text: " field-value))
>                          ((equal "password" field-type)
>                           (read-passwd "Password: " nil field-value))
>                          ((equal "textarea" field-type)
>                           (xwidget-webkit-begin-edit-textarea xww field-value)))))
>             (xwidget-webkit-execute-script
>              xww
>              (format "findactiveelement(document).value='%s'" str)))))))))
>
> instead of
>
>          (let ((str (cond ((equal "text" field-type)
>                            (read-string "Text: " field-value))
>                           ((equal "password" field-type)
>                            (read-passwd "Password: " nil field-value))
>                           ((equal "textarea" field-type)
>                            (xwidget-webkit-begin-edit-textarea xww field-value)))))
>           (xwidget-webkit-execute-script
>            xww
>            (format "findactiveelement(document).value='%s'" str)))))))))

I’ve tried this first and I’m getting unexpected behaviour.  Emacs
prompts for a string in the minibuffer as expected, but the process
becomes *very* busy, stays at 100% CPU usage, grows in memory, and key
presses are registered extremely slowly.  Emacs becomes unusable at this
point.  I cannot even abort this with C-g and have to kill Emacs.

The C code is pretty straight forward:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DEFUN ("xwidget-webkit-execute-script", …)
  (Lisp_Object xwidget, Lisp_Object script, Lisp_Object fun)
{
  …
  CHECK_STRING (script);
  if (!NILP (fun) && (!FUNCTIONP (fun)))
    wrong_type_argument (Qinvalid_function, fun);
  …

  // This runs the JavaScript SCRIPT and then calls the C procedure
  // `webkit_javascript_finished_cb', passing FUN.
  webkit_web_view_run_javascript (WEBKIT_WEB_VIEW (xw->widget_osr),
                                  SSDATA (script),
                                  NULL,
                                  &webkit_javascript_finished_cb,
                                  (gpointer) fun);
  return Qnil;                                  
}

…

static void
webkit_javascript_finished_cb (GObject      *webview,
                               GAsyncResult *result,
                               gpointer      lisp_callback)
{
  …
  Lisp_Object lisp_value = … /* JavaScript return value */
  …
  
  // Run the Lisp callback, passing the converted JavaScript return value
  call1 ((Lisp_Object)lisp_callback, lisp_value);
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I.e., a user calls `xwidget-webkit-execute-script' with a script and a
lisp callback, the script is executed.  When the script is done, a C
callback will be invoked with a reference to the Lisp_Object holding the
lisp callback.  All that does is retrieve the JS return value and pass
it to the lisp callback.

I guessed that this odd behaviour must be a result of calling the
callback procedure from C with “call1”.  I thought that maybe this is
expected behaviour when calling a procedure that interacts with the user
through the mini-buffer (as read-string does) when it is not using the
“interactive” form.

Is this a bug?  (It looks and buzzes like one…)
Or am I doing something stupid?

> [ Oh, and while I'm here, let me advertize pcase:
>
>      (let ((str (pcase field-type
>                   ("text" (read-string "Text: " field-value))
>                   ("password" (read-passwd "Password: " nil field-value))
>                   ("textarea" ...
>
> ]

Nice!  I’ll keep it in mind.

~~ Ricardo

Re: interactive closure — variables not bound

[Stefan Monnier]

> I guessed that this odd behaviour must be a result of calling the
> callback procedure from C with “call1”.  I thought that maybe this is
> expected behaviour when calling a procedure that interacts with the user
> through the mini-buffer (as read-string does) when it is not using the
> “interactive” form.

read-string works just as well from outside an interactive spec.

My crystal ball tells me that it suspects the culprit is that the code
calls the Elisp callback asynchronously, i.e. from a different thread.
IOW it thinks that the code doesn't bother to go through the Elisp event
queue to synchronize with the Elisp engine.


        Stefan

Re: interactive closure — variables not bound

[Ricardo Wurmus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I guessed that this odd behaviour must be a result of calling the
>> callback procedure from C with “call1”.  I thought that maybe this is
>> expected behaviour when calling a procedure that interacts with the user
>> through the mini-buffer (as read-string does) when it is not using the
>> “interactive” form.
>
> read-string works just as well from outside an interactive spec.
>
> My crystal ball tells me that it suspects the culprit is that the code
> calls the Elisp callback asynchronously, i.e. from a different thread.
> IOW it thinks that the code doesn't bother to go through the Elisp event
> queue to synchronize with the Elisp engine.

Ah, this indeed seems to be the case here!  I wasn’t aware of the Elisp
event queue.  Now that I am, I see that xwidget.c already contains code
to create events of kind “XWIDGET_EVENT”, so I’ll try to use that.

Thanks for the good hint!

~~ Ricardo