[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Danny Milosavljevic]

* gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/emacs-xyz.scm (emacs-julia-snail)[arguments]<#:phases>[vendor]:
New phase.
[inputs]: Add julia-cstparser, julia-tokenize.
[propagated-inputs]: Remove julia-cstparser, julia-tokenize.
[source]: Add patch.

Change-Id: I61c0a4493fcfd219809bcc35f8765be0928a7e81
---
 gnu/local.mk                                  |  1 +
 gnu/packages/emacs-xyz.scm                    | 25 ++++++--
 .../emacs-julia-snail-1.3.1-vendor.patch      | 57 +++++++++++++++++++
 3 files changed, 78 insertions(+), 5 deletions(-)
 create mode 100644 gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 0c4ab96bf3..8647956f2f 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1179,6 +1179,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/emacs-git-email-missing-parens.patch	\
   %D%/packages/patches/emacs-helpful-fix-tests.patch	\
   %D%/packages/patches/emacs-highlight-stages-add-gexp.patch	\
+  %D%/packages/patches/emacs-julia-snail-1.3.1-vendor.patch	\
   %D%/packages/patches/emacs-json-reformat-fix-tests.patch	\
   %D%/packages/patches/emacs-kv-fix-tests.patch	\
   %D%/packages/patches/emacs-lispy-fix-thread-last-test.patch   \
diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 370daaf50e..1837c759d2 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -14656,22 +14656,37 @@ (define-public emacs-julia-snail
               (file-name (git-file-name name version))
               (sha256
                (base32
-                "0h5lwc2hsk4rc82idhf9qj9553v7x76wmy9x2z1h60pdd93ilcyr"))))
+                "0h5lwc2hsk4rc82idhf9qj9553v7x76wmy9x2z1h60pdd93ilcyr"))
+              (patches
+               (search-patches "emacs-julia-snail-1.3.1-vendor.patch"))))
     (build-system emacs-build-system)
     (arguments
      (list
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'validate-compiled-autoloads 'vendor
+            (lambda* (#:key inputs outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (base (dirname (car (find-files out "JuliaSnail.jl"))))
+                     (vendor (string-append base "/vendor")))
+                (mkdir-p vendor)
+                (copy-recursively (string-append (assoc-ref inputs "julia-cstparser")
+                                                 "/share/julia/loadpath/CSTParser")
+                                  (string-append vendor "/CSTParser"))
+                (copy-recursively (string-append (assoc-ref inputs "julia-tokenize")
+                                                 "/share/julia/loadpath/Tokenize")
+                                  (string-append vendor "/Tokenize"))))))
       #:include
       #~(cons* "^JuliaSnail\\.jl" "extensions" %default-include)))
     (inputs
-     (list emacs-dash emacs-s emacs-spinner emacs-xref))
+     (list emacs-dash emacs-s emacs-spinner emacs-xref julia-cstparser
+           julia-tokenize))
     (propagated-inputs
      (list libvterm
            emacs-julia-mode             ;required by parser
            emacs-parsec                 ;required by parser
            emacs-popup
-           emacs-vterm
-           julia-tokenize
-           julia-cstparser))
+           emacs-vterm))
     (home-page "https://github.com/gcv/julia-snail")
     (synopsis "Development environment and REPL interaction package for Julia")
     (description "This package provides a development environment and REPL
diff --git a/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch b/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch
new file mode 100644
index 0000000000..eaa38fb868
--- /dev/null
+++ b/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch
@@ -0,0 +1,57 @@
+Author: Danny Milosavljevic <dannym@friendly-machines.com>
+Date: 2024-09-03
+License: GPL3+
+
+Previously, we propagated julia-cstparser and julia-tokenize. This would only
+work if the user had installed julia in their profile (because only then
+JULIA_LOAD_PATH of julia-cstparser would be propagated).
+That doesn't seem likely or desireable.
+
+It turns out that it's not difficult to make julia read vendored packages.
+Let's do that (since there's a comment inside JuliaSnail.jl that warns about
+CSTParser APIs being easily incompatible, vendoring is better anyway).
+
+diff -ru orig/4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl 4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl
+--- orig/4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl	2024-09-03 00:02:03.798373662 +0200
++++ 4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl	2024-09-03 01:05:26.310766009 +0200
+@@ -30,12 +30,12 @@
+    catch err
+       if isa(err, ArgumentError)
+          if isfile(joinpath($dir, "Project.toml"))
+-            # force dependency installation
+-            Main.Pkg.activate($dir)
+-            Main.Pkg.instantiate()
+-            Main.Pkg.precompile()
+-            # activate what was the first entry before Snail was pushed to the head of LOAD_PATH
+-            Main.Pkg.activate(LOAD_PATH[2])
++             # force dependency installation
++             if isa(err, ArgumentError)
++                 error("Vendored dependencies not found. Please make sure the vendored packages are located in the 'vendor' directory.")
++             else
++                 rethrow(err)
++             end
+          end
+       end
+    finally
+@@ -51,11 +51,16 @@
+ end
+ 
+ @with_pkg_env (@__DIR__) begin
+-   # list all external dependency imports here (from the appropriate Project.toml, either Snail's or an extension's):
+-   import CSTParser
+-   # check for dependency API compatibility
+-   !isdefined(CSTParser, :iscall) &&
+-     throw(ArgumentError("CSTParser API not compatible, must install Snail-specific version"))
++    @with_pkg_env (joinpath(@__DIR__, "vendor", "CSTParser")) begin
++        @with_pkg_env (joinpath(@__DIR__, "vendor", "Tokenize")) begin
++            println(LOAD_PATH)
++            # list all external dependency imports here (from the appropriate Project.toml, either Snail's or an extension's):
++            import CSTParser
++            # check for dependency API compatibility
++            !isdefined(CSTParser, :iscall) &&
++                throw(ArgumentError("CSTParser API not compatible, must install Snail-specific version"))
++        end
++    end
+ end
+ 
+ 

base-commit: b833aaaee7c95ec0339428a6b602f26831494798
-- 
2.45.2

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Ludovic Courtès]

Hey Danny,

Danny Milosavljevic <dannym@friendly-machines.com> skribis:

> * gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/packages/emacs-xyz.scm (emacs-julia-snail)[arguments]<#:phases>[vendor]:
> New phase.
> [inputs]: Add julia-cstparser, julia-tokenize.
> [propagated-inputs]: Remove julia-cstparser, julia-tokenize.
> [source]: Add patch.
>
> Change-Id: I61c0a4493fcfd219809bcc35f8765be0928a7e81

[...]

> +          (add-after 'validate-compiled-autoloads 'vendor
> +            (lambda* (#:key inputs outputs #:allow-other-keys)
> +              (let* ((out (assoc-ref outputs "out"))
> +                     (base (dirname (car (find-files out "JuliaSnail.jl"))))
> +                     (vendor (string-append base "/vendor")))
> +                (mkdir-p vendor)
> +                (copy-recursively (string-append (assoc-ref inputs "julia-cstparser")
> +                                                 "/share/julia/loadpath/CSTParser")
> +                                  (string-append vendor "/CSTParser"))
> +                (copy-recursively (string-append (assoc-ref inputs "julia-tokenize")
> +                                                 "/share/julia/loadpath/Tokenize")

Use ‘search-input-directory’ rather than ‘assoc-ref’.

> +++ b/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch
> @@ -0,0 +1,57 @@
> +Author: Danny Milosavljevic <dannym@friendly-machines.com>
> +Date: 2024-09-03
> +License: GPL3+
> +
> +Previously, we propagated julia-cstparser and julia-tokenize. This would only
> +work if the user had installed julia in their profile (because only then
> +JULIA_LOAD_PATH of julia-cstparser would be propagated).
> +That doesn't seem likely or desireable.
> +
> +It turns out that it's not difficult to make julia read vendored packages.
> +Let's do that (since there's a comment inside JuliaSnail.jl that warns about
> +CSTParser APIs being easily incompatible, vendoring is better anyway).

I’m not sure I understand.  For most programming languages (Python,
etc.), we use propagated inputs like this.  How does that differ?

Normally we unbundle (“unvendor”?) libraries and this seems to go in the
opposite direction, right?

Thanks,
Ludo’.

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[dannym]

Hi Ludo,

On 2024-09-16 10:24, Ludovic Courtès wrote:
>> +                (copy-recursively (string-append (assoc-ref inputs 
>> "julia-tokenize")
>> +                                                 
>> "/share/julia/loadpath/Tokenize")
> 
> Use ‘search-input-directory’ rather than ‘assoc-ref’.

Will do!

> I’m not sure I understand.  For most programming languages (Python,
> etc.), we use propagated inputs like this.  How does that differ?
> 
> Normally we unbundle (“unvendor”?) libraries and this seems to go in 
> the
> opposite direction, right?

The idea is to make Guix more user-friendly.

I touched this only after I myself got bitten by the following:

I have guix home (with only a few packages--notably no julia (!) or 
gcc-toolchain
or anything like that in it),
and I have emacs set up with emacs-buffer-env and the following 
configuration:

   (setq buffer-env-script-name '("manifest.scm" ".envrc"))

In a julia project, manifest.scm contains:

   (specifications->manifest
     (list "julia"))

The idea is that emacs-bufferenv will automatically load that manifest 
when I
open a file in that directory within emacs--and then have julia-snail 
REPL
(including sending expression on point to the REPL) work.

Since the emacs julia-snail REPL integration needs to find the bounds of 
the expression,
it has to have a parser for the Julia language.

However, what happens now is that because I don't have julia-cstparser 
and julia-tokenize
in guix home, it won't work.

But from the standpoint of the profile I shouldn't have julia-cstparser 
and julia-tokenize
in my home profile since I don't personally directly use it (it's a 
requirement of emacs-julia-snail--I didn't even know what 
julia-cstparser is a few days ago :) ).

But nevermind that.  Even if I added julia-cstparser and julia-tokenize 
to my
guix home profile, it would NOT work.  The reason is because julia 
itself is not part of
the guix home profile, and the "julia" package would have a search-path 
specification that
would set some environment variables that julia needs to find 
julia-cstparser and
julia-tokenize.
Since the package is not there (in guix home) it won't set the 
variables.

The idea of this patch is that emacs will automatically set up whatever 
it needs to in order
to make the REPL work (including the finding of the boundaries of the 
current expression
that the emacs package needs for "send expression at point to REPL") 
without weird
dependencies bleeding explicitly into the user profile or into the local 
directory's manifest.

In my opinion, the user should not need to be concerned with what 
julia-cstparser is in this case.

Emacs should just magically load whatever it needs for the Julia REPL to 
work (whether
that's written in Julia or Python or whatever is an implementation 
detail--you COULD
have a Julia parser written in Python. Notably, most parsers for any 
language in
IntelliJ IDEA are written in Java--the Python parser, Rust parser etc).

That said, I'm totally open to other ways to do it.

The obvious alternative would be for the user to put julia-cstparser and 
julia-tokenize
into every Julia project's manifest.scm . But the error message is 
inscrutable if you
don't have those in there[1]--and I have no idea how any normal person 
would figure out
that julia-cstparser is needed in the first place.

[1] julia-snail--send-to-server: No Julia REPL buffer *julia* found; run 
julia-snail
Starting Julia process and loading Snail...
julia-snail--repl-enable: The REPL terminal buffer is inactive; 
double-check julia-snail-executable path

(In buffer "*scratch*" : )
julia-snail-executable
"julia"

(note: that "julia" excutable IS available in the manifest of the buffer 
env--and launching "julia" does work when launched manually within 
emacs)

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Liliana Marie Prikler]

Hi Danny,

Am Montag, dem 16.09.2024 um 20:01 +0200 schrieb
dannym@friendly-machines.com:
> The idea of this patch is that emacs will automatically set up
> whatever it needs to in order to make the REPL work (including the
> finding of the boundaries of the current expression that the emacs
> package needs for "send expression at point to REPL")  without weird
> dependencies bleeding explicitly into the user profile or into the
> local directory's manifest.
You could write a meta package à la "julia stuff for emacs-julia-snail"
and mention that in the description, no?  I think bundling (or
propagating) these dependencies from the package is likely not going to
do us favours in weird edge cases where you actually want more julia
packages, e.g. in the julia package that you may or may not be hacking
on, which has other dependencies :)

Cheers

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Ludovic Courtès]

Hi Danny & Liliana,

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> Am Montag, dem 16.09.2024 um 20:01 +0200 schrieb
> dannym@friendly-machines.com:
>> The idea of this patch is that emacs will automatically set up
>> whatever it needs to in order to make the REPL work (including the
>> finding of the boundaries of the current expression that the emacs
>> package needs for "send expression at point to REPL")  without weird
>> dependencies bleeding explicitly into the user profile or into the
>> local directory's manifest.

Thanks for explaining!

> You could write a meta package à la "julia stuff for emacs-julia-snail"
> and mention that in the description, no?

Yes, sounds like a good idea to me.  WDYT, Danny?

Ludo’.