Returning variable "references" under lexical binding

Returning variable "references" under lexical binding

[Sean McAfee]

I recently tried writing my first non-toy code that employs lexical
binding.  It's a routine that sets up a series of idle timers, storing
each successive timer object into the same lexical variable.

I want my routine to return an object that can be used to cancel the
most recently set timer.  If I were writing this code in the days prior
to lexical binding, I might have dono something like this:

(defun start-my-timer ()
  (let ((timer (gensym)))
    ;; ... (set timer (make-timer ...)) ...
    timer))

(defun cancel-my-timer (timer)
  (cancel-timer (symbol-value timer)))

The documenentation for lexical variables cautions against treating them
as symbols, specifically stating that functions like symbol-value will
not work.  So I wrote my routine to return a closure:

(defun start-my-timer ()
  (let (timer)
    ;; ... (setq timer (make-timer ...)) ...
    (lambda () timer)))

(defun cancel-my-timer (timer)
  (cancel-timer (funcall timer)))

This works, but is it the "correct" way to do this?

Re: Returning variable "references" under lexical binding

[Barry Margolin]

In article <bp8sj1hjsoh.fsf@usca1uw-JZWWPM1.sanmateo.corp.akamai.com>,
 Sean McAfee <eefacm@gmail.com> wrote:

> I recently tried writing my first non-toy code that employs lexical
> binding.  It's a routine that sets up a series of idle timers, storing
> each successive timer object into the same lexical variable.
> 
> I want my routine to return an object that can be used to cancel the
> most recently set timer.  If I were writing this code in the days prior
> to lexical binding, I might have dono something like this:
> 
> (defun start-my-timer ()
>   (let ((timer (gensym)))
>     ;; ... (set timer (make-timer ...)) ...
>     timer))
> 
> (defun cancel-my-timer (timer)
>   (cancel-timer (symbol-value timer)))
> 
> The documenentation for lexical variables cautions against treating them
> as symbols, specifically stating that functions like symbol-value will
> not work.  So I wrote my routine to return a closure:
> 
> (defun start-my-timer ()
>   (let (timer)
>     ;; ... (setq timer (make-timer ...)) ...
>     (lambda () timer)))
> 
> (defun cancel-my-timer (timer)
>   (cancel-timer (funcall timer)))
> 
> This works, but is it the "correct" way to do this?

What's wrong with using ordinary variable values:

(defun start-my-timer ()
  (make-timer ...))

(setq my-timer (start-my-timer))

(cancel-timer my-timer)

What do you gain by indirecting through a gensym or a closure?

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Returning variable "references" under lexical binding

[Stefan Monnier]

> (defun start-my-timer ()
>   (let ((timer (gensym)))
>     ;; ... (set timer (make-timer ...)) ...
>     timer))
[...]
> (defun cancel-my-timer (timer)
>   (cancel-timer (symbol-value timer)))

Why not

  (defun start-my-timer ()
    (let ((timer (make-timer ...))
      ...
      timer))
  (defun cancel-my-timer (timer)
    (cancel-timer timer))

> The documenentation for lexical variables cautions against treating them
> as symbols, specifically stating that functions like symbol-value will
> not work.

In your above code, you're not treating variables as symbols.
You're just storing a symbol inside a variable, which is fine.
E.g. a problematic example would have been (set 'timer ...) or
(symbol-value 'timer).


        Stefan

Re: Returning variable "references" under lexical binding

[Sean McAfee]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> (defun start-my-timer ()
>>   (let ((timer (gensym)))
>>     ;; ... (set timer (make-timer ...)) ...
>>     timer))
> [...]
>> (defun cancel-my-timer (timer)
>>   (cancel-timer (symbol-value timer)))

> Why not
>
>   (defun start-my-timer ()
>     (let ((timer (make-timer ...))
>       ...
>       timer))
>   (defun cancel-my-timer (timer)
>     (cancel-timer timer))

Because start-my-timer sets up callbacks that may repeatedly change the
value of the "timer" variable:

;; -*- lexical-binding: t; -*-

(defun start-my-timer ()
  (let (timer)
    (setq timer (make-timer ...))
    ;; ... (later (occasionally (setq timer (make-timer ...)))) ...
    (lambda () timer))

Like I said in my original article:

> It's a routine that sets up a series of idle timers, storing
> each successive timer object into the same lexical variable.

I should have made that more explicit.

>> The documenentation for lexical variables cautions against treating them
>> as symbols, specifically stating that functions like symbol-value will
>> not work.

> In your above code, you're not treating variables as symbols.
> You're just storing a symbol inside a variable, which is fine.

Yes, I was explaining why I assumed this wouldn't work:

;; -*- lexical-binding: t; -*-

(defun start-my-timer ()
  (let (timer)
    (setq timer (make-timer ...))
    ;; ... (later (occasionally (setq timer (make-timer ...)))) ...
    'timer)

(defun cancel-timer (timer)
  (cancel-timer (symbol-value timer)))

If I were not using lexical binding, though, this should work:

(defun start-my-timer ()
  (let ((timer (gensym)))
    (set timer (make-timer ...))
    ;; ... (later (occasionally (set timer (make-timer ...)))) ...
    timer))

;; cancel-timer as before

The question still stands if using a closure as a handle to a varying
lexical variable is the right way to go.

Re: Returning variable "references" under lexical binding

[Stefan Monnier]

> The question still stands if using a closure as a handle to a varying
> lexical variable is the right way to go.

There's no "right way".  There are various ways which work and either of
them is fine.


        Stefan

Re: Returning variable "references" under lexical binding

[Barry Margolin]

In article <87fvxgc2mz.fsf@gmail.com>, Sean McAfee <eefacm@gmail.com> 
wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> >> (defun start-my-timer ()
> >>   (let ((timer (gensym)))
> >>     ;; ... (set timer (make-timer ...)) ...
> >>     timer))
> > [...]
> >> (defun cancel-my-timer (timer)
> >>   (cancel-timer (symbol-value timer)))
> 
> > Why not
> >
> >   (defun start-my-timer ()
> >     (let ((timer (make-timer ...))
> >       ...
> >       timer))
> >   (defun cancel-my-timer (timer)
> >     (cancel-timer timer))
> 
> Because start-my-timer sets up callbacks that may repeatedly change the
> value of the "timer" variable:
> 
> ;; -*- lexical-binding: t; -*-
> 
> (defun start-my-timer ()
>   (let (timer)
>     (setq timer (make-timer ...))
>     ;; ... (later (occasionally (setq timer (make-timer ...)))) ...
>     (lambda () timer))
> 
> Like I said in my original article:
> 
> > It's a routine that sets up a series of idle timers, storing
> > each successive timer object into the same lexical variable.

But it's not the same lexical variable in your example. Each time you 
call start-my-timer you're creating a new closure over that variable. 
The caller has to save that closure somewhere, so that it can pass it to 
cancel-my-timer later. There's no functional difference between that and 
saving the timer itself.

What you're talking about would be using a closure to implement an 
object-oriented approach:

(defun new-timer ()
  (let (timer)
    #'(lambda (operation)
        (case operation
          ((start)
           (setq timer (make-timer ...)))
          ((update)
           (if timer (stop-timer timer))
           (setq timer (make-timer ...)))
          ((stop)
           (if timer (stop-timer timer)))))))

(setq timer (new-timer))
(funcall timer 'start)
(funcall timer 'stop)

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Returning variable "references" under lexical binding

[Barry OReilly]

>> Why not
>>
>>   (defun start-my-timer ()
>>     (let ((timer (make-timer ...))
>>       ...
>>       timer))
>>   (defun cancel-my-timer (timer)
>>     (cancel-timer timer))
>
> Because start-my-timer sets up callbacks that may repeatedly change the
> value of the "timer" variable:

But you said:

> I want my routine to return an object that can be used to cancel the
> most recently set timer.

How is the above start-my-timer inadequate then?

> (defun start-my-timer ()
>   (let (timer)
>     ;; ... (setq timer (make-timer ...)) ...
>     (lambda () timer)))
>
> (defun cancel-my-timer (timer)
>   (cancel-timer (funcall timer)))

Each invocation of this start-my-timer will return a different closure.
Whether start-my-timer returns a timer or a closure, you still have to keep
track of which one to pass to cancel-my-timer. So the use of closures
doesn't facilitate anything that I can see.

Re: Returning variable "references" under lexical binding

[Sean McAfee]

Barry Margolin <barmar@alum.mit.edu> writes:
> But it's not the same lexical variable in your example. Each time you 
> call start-my-timer you're creating a new closure over that variable. 
> The caller has to save that closure somewhere, so that it can pass it to 
> cancel-my-timer later. There's no functional difference between that and 
> saving the timer itself.

Sure there is.  OK, forget about timers, let's go even simpler:

;; -*- lexical-binding: t; -*-

(defun return-variable-n ()
  (let ((n 1))
    (in-five-seconds (lambda () (setq n 2)))
    SOMETHING))

(setq foo (return-variable-n))

I want to return SOMETHING such that if I inspect it immediately, I'll
get 1, but if I save it and inspect it after five seconds have passed,
I'll get 2.  SOMETHING can't be just "n", because the function returns
by value and foo would only ever contain 1.

Similarly, in my original code, I can't just return "timer", because the
calling code would only ever see the very first timer created, and would
not be able to see the new values for timer that the callbacks store
later.

Without lexical variables, I could do this:

(defun return-variable-n ()
  (let ((n (gensym)))
    (set n 1)
    (in-five-seconds (lambda () (set n 2)))
    n))

...and then get the current value of n using symbol-value.

Using lexical variables, SOMETHING can be (lambda () n), which can be
called to get the current value of n at all times.

My question was whether other methods exist, and which is best, for
reasonable definitions of "best."

Re: Returning variable "references" under lexical binding

[Barry Margolin]

In article <87bo84b85k.fsf@gmail.com>, Sean McAfee <eefacm@gmail.com> 
wrote:

> Barry Margolin <barmar@alum.mit.edu> writes:
> > But it's not the same lexical variable in your example. Each time you 
> > call start-my-timer you're creating a new closure over that variable. 
> > The caller has to save that closure somewhere, so that it can pass it to 
> > cancel-my-timer later. There's no functional difference between that and 
> > saving the timer itself.
> 
> Sure there is.  OK, forget about timers, let's go even simpler:
> 
> ;; -*- lexical-binding: t; -*-
> 
> (defun return-variable-n ()
>   (let ((n 1))
>     (in-five-seconds (lambda () (setq n 2)))
>     SOMETHING))
> 
> (setq foo (return-variable-n))
> 
> I want to return SOMETHING such that if I inspect it immediately, I'll
> get 1, but if I save it and inspect it after five seconds have passed,
> I'll get 2.  SOMETHING can't be just "n", because the function returns
> by value and foo would only ever contain 1.

This is different, because now you have something with state that can 
change, and want to be able to inspect. it.

> 
> Similarly, in my original code, I can't just return "timer", because the
> calling code would only ever see the very first timer created, and would
> not be able to see the new values for timer that the callbacks store
> later.

What "new values"? Your code didn't have any way to change the value of 
timer.  All it did was wrap it in a closure, but that closure doesn't do 
anything other than return the initial value. Did you leave something 
out of the original example?

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Returning variable "references" under lexical binding

[Sean McAfee]

Barry Margolin <barmar@alum.mit.edu> writes:
> In article <87bo84b85k.fsf@gmail.com>, Sean McAfee <eefacm@gmail.com> 
> wrote:
>> Similarly, in my original code, I can't just return "timer", because the
>> calling code would only ever see the very first timer created, and would
>> not be able to see the new values for timer that the callbacks store
>> later.
>
> What "new values"? Your code didn't have any way to change the value of 
> timer.  All it did was wrap it in a closure, but that closure doesn't do 
> anything other than return the initial value. Did you leave something 
> out of the original example?

I said:

> It's a routine that sets up a series of idle timers, storing
> each successive timer object INTO THE SAME LEXICAL VARIABLE.

> I want my routine to return an object that can be used to cancel THE
> MOST RECENTLY SET TIMER.

Emphasis added.  And later I elaborated on my example:

> (defun start-my-timer ()
>   (let (timer)
>     (setq timer (make-timer ...))
>     ;; ... (later (occasionally (setq timer (make-timer ...)))) ...
>     (lambda () timer))

"later" was meant to indicate "later in time, after the function
returns."

Because it's not too long, here's the actual code that inspired my
question.  It manages a number of idle timers to implement a new kind of
timer that repeats at regular intervals as long as Emacs remains idle,
and as long as the callback it's given returns true:

--------------------
;; -*- lexical-binding: t; -*-

(defun run-periodically-when-idle (seconds func)
  (let (timer count)
    (labels ((on-first-idle ()
               (when timer
                 (cancel-timer timer))
               (setq count 1)
               (run-and-schedule-next))
             (run-and-schedule-next ()
               (setq timer
                 (and (funcall func count)
                      (run-with-idle-timer
                       (* seconds (incf count))
                       nil
                       #'run-and-schedule-next)))))
      (list (run-with-idle-timer seconds t #'on-first-idle)
            (lambda () timer)))))

(defun cancel-periodic-idle-timer (timer)
  (cancel-timer (first timer))
  (aif (funcall (second timer))
    (cancel-timer it)))
--------------------

I've been using it to automatically refresh my Gnus group buffer every
two minutes:

(setq watch-mail-timer
      (run-periodically-when-idle 120
        (lambda (n)
          (when (string= (buffer-name) "*Group*")
            (let ((message-log-max nil))
              (gnus-group-get-new-news)
              (message "Checked mail #%d" n)
            t))))

Re: Returning variable "references" under lexical binding

[Stefan Monnier]

> (defun run-periodically-when-idle (seconds func)
>   (let (timer count)
>     (labels ((on-first-idle ()
>                (when timer
>                  (cancel-timer timer))
>                (setq count 1)
>                (run-and-schedule-next))
>              (run-and-schedule-next ()
>                (setq timer
>                  (and (funcall func count)
>                       (run-with-idle-timer
>                        (* seconds (incf count))
>                        nil
>                        #'run-and-schedule-next)))))
>       (list (run-with-idle-timer seconds t #'on-first-idle)
>             (lambda () timer)))))

BTW, for the curious, the byte-compiler will turn the above code into
something similar to:

    (defun run-periodically-when-idle (seconds func)
      (let ((timer (list nil))
            (count (list nil)))
        (letrec ((on-first-idle
                  `(lambda ()
                     (when (car ',timer)
                       (cancel-timer (car ',timer)))
                     (setcar ',count 1)
                     (funcall ',run-and-schedule-next)))
                 (run-and-schedule-next
                  `(lambda ()
                     (setcar ',timer
                             (and (funcall ',func (car ',count))
                                  (run-with-idle-timer
                                   (* ',seconds (incf (car ',count)))
                                   nil
                                   ',run-and-schedule-next))))))
          (list (run-with-idle-timer seconds t on-first-idle)
                `(lambda () (car ',timer))))))
   
So if I were you I'd be tempted to use something like

   (defun run-periodically-when-idle (seconds func)
     (let ((timers (list nil))
           count)
       (labels ((on-first-idle ()
                  (when (cdr timers)
                    (cancel-timer (cdr timers)))
                  (setq count 1)
                  (run-and-schedule-next))
                (run-and-schedule-next ()
                  (setcdr timers
                    (and (funcall func count)
                         (run-with-idle-timer
                          (* seconds (incf count))
                          nil
                          #'run-and-schedule-next)))))
         (setcar timers (run-with-idle-timer seconds t #'on-first-idle))
         timers)))
   
   (defun cancel-periodic-idle-timer (timers)
     (cancel-timer (car timers))
     (aif (cdr timers)
       (cancel-timer it)))

BTW, as you have discovered there is often the need to do what your code
does, and in the packages that need it, they often use
a run-with-timer (i.e. a non-idle timer) for the repetition (which they
usually cancel in something like a pre-command-hook).

In jit-lock, the repetition is done with an idle timer (like here), but
instead of creating a new timer each time, the old timer is re-armed for
a different idle time.

I'm not very satisfied with either of the solutions.  It would be good
to add the functionality directly in timer.el (even if using one of the
existing techniques: at least clients could be later upgraded to a better
technique by improving timer.el).  IOW, I'd welcome a patch to timer.el
providing this feature.


        Stefan