Issues with named-led and pcase inside iter-defun.

Issues with named-led and pcase inside iter-defun.

[Max Brieiev]

Hello,

I've been trying to build some of my code on top of generator.el
library. I've found that functions defined with iter-defun have some
issues with named-let and pcase.

This is a macro I use to wrap iter-defun and advance the iterator to the
first yield expression:

    (defmacro co-defun (name args &rest body)
      "Define a coroutine.  Automatically advance it to the first yield
    expression."
      (declare (indent defun))
      (let ((co-name (gensym (concat (symbol-name name) "-co-"))))
        `(progn
           (iter-defun ,co-name ,args ,@body)
           (defun ,name (&rest passthrough-args)
             (let ((co (apply #',co-name passthrough-args)))
               (iter-next co)
               co)))))

Now, here is a superficial example that involves both named-let and pcase:

    (co-defun lazy-sum ()
      (named-let sum ((running-sum 0))
        (pcase running-sum
          ((pred (< 10)) running-sum)
          (_ (sum (+ running-sum (iter-yield nil)))))))
    
    (setq it (lazy-sum))
    (iter-next it 3)
    (iter-next it 5)
    (iter-next it 7) ; raises (iter-end-of-sequence . 15)

So far so good. But let's make the example a bit more practical:

    (co-defun curling-parser/headers (chunk)
      "Parse HTTP headers. Returns alist of headers and the remaining data."
      (named-let parse-chunk ((chunk chunk) (aheaders (list)))
        (pcase chunk
          ;; header line
          ((rx line-start
               (let name (+ (not ":")))
               ":"
               (* blank)
               (let value (+ (not "\r")))
               "\r\n"
               (let tail (* anything)))
           (parse-chunk tail (cons (cons name value) aheaders)))
    
          ;; end of headers section
          ((rx line-start
               "\r\n"
               (let tail (* anything)))
           (list (nreverse aheaders) tail))
    
          ;; suspend. Resumes parsing after the next chunk comes in
          (_ (parse-chunk (concat chunk (iter-yield nil)) aheaders)))))
    
    ;; Send chunks into the coroutine
    (setq parser (curling-parser/headers "Content-Ty"))
    (iter-next parser "pe: text/plain\r\nCache-Control: no-ca")
    (iter-next parser "che\r\n\r\nHello World!")

The above has essentially the same structure as lazy-sum function,
however this code fails with the error:

    (void-function cps-internal-yield)

Is this a bug related to iterators in Elisp?

After some trial and error, I found that iter-yield sometimes doesn't
work inside pcase, so I moved it outside of pcase like this:

    (named-let parse-chunk ((chunk chunk) (aheaders (list)))
      (let ((result (pcase ...)))
        (if result
            result
          (parse-chunk (concat chunk (iter-yield nil)) aheaders))))

But now it started to fail with the same error outside pcase, but inside
named-let!

So my question is how do I work around this? There is no obvious way to
tell whether iter-yield will succeed or fail inside pcase/named-let
construct. Is it a fundamental limitation of iterators or just a bug
that needs to be fixed?

I have rewritten my code with a plain while loop, and it works without
issues, but I like named-let/pcase approach much more.

Re: Issues with named-led and pcase inside iter-defun.

[Michael Heerdegen]

Max Brieiev <max.brieiev@gmail.com> writes:

> The above has essentially the same structure as lazy-sum function,
> however this code fails with the error:
>
>     (void-function cps-internal-yield)
>
> Is this a bug related to iterators in Elisp?

This looks like bug#59140 ("iter-yield from lambda"), a known but not
yet fixed limitation.

> So my question is how do I work around this?

AFAIU you must avoid yielding from inside a lambda expression in
the expanded code.

Michael.

Re: Issues with named-led and pcase inside iter-defun.

[Max Brieiev]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> AFAIU you must avoid yielding from inside a lambda expression in
> the expanded code.

I see. The hard part is to know in advance what macros expand to lambda
expressions.

In pcase, probably some handlers expand to lambda, but some don't. So
iter-yield may work in some cases, but not in others.

In named-let, it probably should work when a recursive call happens in
tail position. In my code, though, it is always a tail call, and yet it
still fails sporadically.

Re: Issues with named-led and pcase inside iter-defun.

[Michael Heerdegen]

Max Brieiev <max.brieiev@gmail.com> writes:

> In pcase, probably some handlers expand to lambda, but some don't. So
> iter-yield may work in some cases, but not in others.

AFAIU it depends on the tests: if a leaf can be reached in different
ways, the code is wrapped into a lambda to avoid duplication (in the
expansion).

Michael.