Is it possible to get ijp's js branch working anymore?

Is it possible to get ijp's js branch working anymore?

[Christine Lemmer-Webber]

I've moved this thread from guile-user to guile-devel.

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> Christine Lemmer-Webber <cwebber@dustycloud.org> writes:
>
>> I have pushed one more merge with master to the compile-to-js-merge
>> branch.  I've taken the bold move of pushing this to origin... I think
>> it's a good idea to try to get this in, so it's worth it not just
>> sitting in my own personal repo.
>
> wow — thank you!
>
> Is there a short example on how to use this branch for the features that
> already work?
>
> Best wishes,
> Arne

ijp's email, at the start of this thread (see 2017-08-28 in the
guile-user mail archive) explains everything better than I could.
However, the branch does not really work.  I'm worried it might not be
able to, see below.

I've actually made it past the last hurdle I mentioned in my thread;
turns out I was closer on track when I last tried to merge this than in
the efforts I made within the last week.  And so I'm now at the point
where I think I'm dealing with the real merge issues.  Wingo, I've cc'ed
you... I think you're our main hope here.

Here's the problem: CPS changed a lot since ijp wrote this... and it may
have gotten advanced enough that merging has become very difficult.
The problem started out with records that look like so:

  #<cps (primcall allocate-words/immediate (pair . 2))>

There are two weird things here:

 - $primcall used to only have args that were a list; now there are some
   weird ones with cons cells, like 'allocate-words/immediate.  I see
   that compile-bytecode.scm has grown to pattern match against a
   significant number of these.  So, ok, we have to do the same in
   compile-js.scm.

 - Except that wait a minute... 'allocate-words/immediate ?!
   This sounds *really* low-level.  Wait, it *is* really low-level!
   Right now ijp's branch had a runtime.js branch which translated
   a bunch of these primitives, and that could be done quasi-reasonably,
   but allocate-words/immediate.... what the heck do we do with that
   in javascript?!

The commit where things seem to have gotten low-level in CPS land is
9f98b4a5b1067b177c700d27abf4ed477f013951.  Here's the big relevant
change:

#+BEGIN_SRC diff
diff --git a/module/language/cps/closure-conversion.scm b/module/language/cps/closure-conversion.scm
index 4f9296397..746e5cebe 100644
--- a/module/language/cps/closure-conversion.scm
+++ b/module/language/cps/closure-conversion.scm
@@ -19,9 +19,8 @@
 ;;; Commentary:
 ;;;
 ;;; This pass converts a CPS term in such a way that no function has any
-;;; free variables.  Instead, closures are built explicitly with
-;;; make-closure primcalls, and free variables are referenced through
-;;; the closure.
+;;; free variables.  Instead, closures are built explicitly as heap
+;;; objects, and free variables are referenced through the closure.
 ;;;
 ;;; Closure conversion also removes any $rec expressions that
 ;;; contification did not handle.  See (language cps) for a further
@@ -520,10 +519,36 @@ term."
     (define (allocate-closure cps k src label known? nfree)
       "Allocate a new closure, and pass it to $var{k}."
       (match (vector known? nfree)
+        (#(#f 0)
+         ;; The call sites cannot be enumerated, but the closure has no
+         ;; identity; statically allocate it.
+         (with-cps cps
+           (build-term ($continue k src ($closure label 0)))))
         (#(#f nfree)
          ;; The call sites cannot be enumerated; allocate a closure.
          (with-cps cps
-           (build-term ($continue k src ($closure label nfree)))))
+           (letv closure tag code)
+           (letk k* ($kargs () ()
+                      ($continue k src ($values (closure)))))
+           (letk kinit ($kargs ('code) (code)
+                         ($continue k* src
+                           ($primcall 'word-set!/immediate '(closure . 1)
+                                      (closure code)))))
+           (letk kcode ($kargs () ()
+                         ($continue kinit src ($code label))))
+           (letk ktag1
+                 ($kargs ('tag) (tag)
+                   ($continue kcode src
+                     ($primcall 'word-set!/immediate '(closure . 0)
+                                (closure tag)))))
+           (letk ktag0
+                 ($kargs ('closure) (closure)
+                   ($continue ktag1 src
+                     ($primcall 'load-u64 (+ %tc7-program (ash nfree 16)) ()))))
+           (build-term
+             ($continue ktag0 src
+               ($primcall 'allocate-words/immediate `(closure . ,(+ nfree 2))
+                          ())))))
         (#(#t 2)
          ;; Well-known closure with two free variables; the closure is a
          ;; pair.
#+END_SRC

I'm not really sure what to do.  I don't mean to complain; I, along with
everyone here, has benefitted from Wingo's constant tweaking of this
stuff, making it all more and more efficient.  But can it still be
compatible with ijp's vision of javascript emerging the cps layer?

Here's my gut sense though: I wonder if CPS can be broken into phases,
one that does these smarter low-level memory tweaks, and one that
doesn't, so that ijp's lovely branch has a chance of being merged.

Or maybe there's a more obvious path?

What to do?!  Wingo, we need your help!

 - Christine

Re: Is it possible to get ijp's js branch working anymore?

[Christine Lemmer-Webber]

Christine Lemmer-Webber <cwebber@dustycloud.org> writes:

> I've moved this thread from guile-user to guile-devel.
>
> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>
>> Christine Lemmer-Webber <cwebber@dustycloud.org> writes:
>>
>>> I have pushed one more merge with master to the compile-to-js-merge
>>> branch.  I've taken the bold move of pushing this to origin... I think
>>> it's a good idea to try to get this in, so it's worth it not just
>>> sitting in my own personal repo.
>>
>> wow — thank you!
>>
>> Is there a short example on how to use this branch for the features that
>> already work?
>>
>> Best wishes,
>> Arne
>
> ijp's email, at the start of this thread (see 2017-08-28 in the
> guile-user mail archive) explains everything better than I could.
> However, the branch does not really work.  I'm worried it might not be
> able to, see below.
>
> I've actually made it past the last hurdle I mentioned in my thread;
> turns out I was closer on track when I last tried to merge this than in
> the efforts I made within the last week.  And so I'm now at the point
> where I think I'm dealing with the real merge issues.  Wingo, I've cc'ed
> you... I think you're our main hope here.
>
> Here's the problem: CPS changed a lot since ijp wrote this... and it may
> have gotten advanced enough that merging has become very difficult.
> The problem started out with records that look like so:
>
>   #<cps (primcall allocate-words/immediate (pair . 2))>
>
> There are two weird things here:
>
>  - $primcall used to only have args that were a list; now there are some
>    weird ones with cons cells, like 'allocate-words/immediate.  I see
>    that compile-bytecode.scm has grown to pattern match against a
>    significant number of these.  So, ok, we have to do the same in
>    compile-js.scm.
>
>  - Except that wait a minute... 'allocate-words/immediate ?!
>    This sounds *really* low-level.  Wait, it *is* really low-level!
>    Right now ijp's branch had a runtime.js branch which translated
>    a bunch of these primitives, and that could be done quasi-reasonably,
>    but allocate-words/immediate.... what the heck do we do with that
>    in javascript?!
>
> The commit where things seem to have gotten low-level in CPS land is
> 9f98b4a5b1067b177c700d27abf4ed477f013951.  Here's the big relevant
> change:
>
> #+BEGIN_SRC diff
> diff --git a/module/language/cps/closure-conversion.scm b/module/language/cps/closure-conversion.scm
> index 4f9296397..746e5cebe 100644
> --- a/module/language/cps/closure-conversion.scm
> +++ b/module/language/cps/closure-conversion.scm
> @@ -19,9 +19,8 @@
>  ;;; Commentary:
>  ;;;
>  ;;; This pass converts a CPS term in such a way that no function has any
> -;;; free variables.  Instead, closures are built explicitly with
> -;;; make-closure primcalls, and free variables are referenced through
> -;;; the closure.
> +;;; free variables.  Instead, closures are built explicitly as heap
> +;;; objects, and free variables are referenced through the closure.
>  ;;;
>  ;;; Closure conversion also removes any $rec expressions that
>  ;;; contification did not handle.  See (language cps) for a further
> @@ -520,10 +519,36 @@ term."
>      (define (allocate-closure cps k src label known? nfree)
>        "Allocate a new closure, and pass it to $var{k}."
>        (match (vector known? nfree)
> +        (#(#f 0)
> +         ;; The call sites cannot be enumerated, but the closure has no
> +         ;; identity; statically allocate it.
> +         (with-cps cps
> +           (build-term ($continue k src ($closure label 0)))))
>          (#(#f nfree)
>           ;; The call sites cannot be enumerated; allocate a closure.
>           (with-cps cps
> -           (build-term ($continue k src ($closure label nfree)))))
> +           (letv closure tag code)
> +           (letk k* ($kargs () ()
> +                      ($continue k src ($values (closure)))))
> +           (letk kinit ($kargs ('code) (code)
> +                         ($continue k* src
> +                           ($primcall 'word-set!/immediate '(closure . 1)
> +                                      (closure code)))))
> +           (letk kcode ($kargs () ()
> +                         ($continue kinit src ($code label))))
> +           (letk ktag1
> +                 ($kargs ('tag) (tag)
> +                   ($continue kcode src
> +                     ($primcall 'word-set!/immediate '(closure . 0)
> +                                (closure tag)))))
> +           (letk ktag0
> +                 ($kargs ('closure) (closure)
> +                   ($continue ktag1 src
> +                     ($primcall 'load-u64 (+ %tc7-program (ash nfree 16)) ()))))
> +           (build-term
> +             ($continue ktag0 src
> +               ($primcall 'allocate-words/immediate `(closure . ,(+ nfree 2))
> +                          ())))))
>          (#(#t 2)
>           ;; Well-known closure with two free variables; the closure is a
>           ;; pair.
> #+END_SRC
>
> I'm not really sure what to do.  I don't mean to complain; I, along with
> everyone here, has benefitted from Wingo's constant tweaking of this
> stuff, making it all more and more efficient.  But can it still be
> compatible with ijp's vision of javascript emerging the cps layer?
>
> Here's my gut sense though: I wonder if CPS can be broken into phases,
> one that does these smarter low-level memory tweaks, and one that
> doesn't, so that ijp's lovely branch has a chance of being merged.
>
> Or maybe there's a more obvious path?
>
> What to do?!  Wingo, we need your help!
>
>  - Christine

Some further thinking for the moment... I see three possible paths:

 - Actually separate out CPS into two languages: high-cps, and low-cps.
   high-cps is the CPS version pre memory fiddling stuff, something
   possible to transform into js, for instance.

 - Wingo mentioned this "wip-tailify" branch which I really haven't
   looked at but sounds like it's actually covering this path:

    https://lists.gnu.org/archive/html/guile-devel/2021-06/msg00005.html

   We decide okay, you know what, that's the direction that ijp's stuff
   should be put on top of.  Let's do that.  (Or, maybe the energy moves
   towards WASM instead, though it might be nicer if it was
   in-addition-to, but hack time does tend to be limited.)

 - A goofy middle ground is to actually just compile from tree-il to
   js-il instead (and then to javascript... so this could still use some
   of the work from ijp's branch).  This could probably work, but only
   certain scheme programs could be converted: multiple value return
   will turn out to not be possible, for instance.  (I guess it won't be
   in any generated-from-scheme code that is exposed to javascript
   anyway...?)

No idea what's the right path.  The middle one does sound promising
though.  I haven't looked at the branch much yet...