[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Danny Milosavljevic]

* gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/emacs-xyz.scm (emacs-julia-snail)[arguments]<#:phases>[vendor]:
New phase.
[inputs]: Add julia-cstparser, julia-tokenize.
[propagated-inputs]: Remove julia-cstparser, julia-tokenize.
[source]: Add patch.

Change-Id: I61c0a4493fcfd219809bcc35f8765be0928a7e81
---
 gnu/local.mk                                  |  1 +
 gnu/packages/emacs-xyz.scm                    | 25 ++++++--
 .../emacs-julia-snail-1.3.1-vendor.patch      | 57 +++++++++++++++++++
 3 files changed, 78 insertions(+), 5 deletions(-)
 create mode 100644 gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 0c4ab96bf3..8647956f2f 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1179,6 +1179,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/emacs-git-email-missing-parens.patch	\
   %D%/packages/patches/emacs-helpful-fix-tests.patch	\
   %D%/packages/patches/emacs-highlight-stages-add-gexp.patch	\
+  %D%/packages/patches/emacs-julia-snail-1.3.1-vendor.patch	\
   %D%/packages/patches/emacs-json-reformat-fix-tests.patch	\
   %D%/packages/patches/emacs-kv-fix-tests.patch	\
   %D%/packages/patches/emacs-lispy-fix-thread-last-test.patch   \
diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 370daaf50e..1837c759d2 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -14656,22 +14656,37 @@ (define-public emacs-julia-snail
               (file-name (git-file-name name version))
               (sha256
                (base32
-                "0h5lwc2hsk4rc82idhf9qj9553v7x76wmy9x2z1h60pdd93ilcyr"))))
+                "0h5lwc2hsk4rc82idhf9qj9553v7x76wmy9x2z1h60pdd93ilcyr"))
+              (patches
+               (search-patches "emacs-julia-snail-1.3.1-vendor.patch"))))
     (build-system emacs-build-system)
     (arguments
      (list
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'validate-compiled-autoloads 'vendor
+            (lambda* (#:key inputs outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (base (dirname (car (find-files out "JuliaSnail.jl"))))
+                     (vendor (string-append base "/vendor")))
+                (mkdir-p vendor)
+                (copy-recursively (string-append (assoc-ref inputs "julia-cstparser")
+                                                 "/share/julia/loadpath/CSTParser")
+                                  (string-append vendor "/CSTParser"))
+                (copy-recursively (string-append (assoc-ref inputs "julia-tokenize")
+                                                 "/share/julia/loadpath/Tokenize")
+                                  (string-append vendor "/Tokenize"))))))
       #:include
       #~(cons* "^JuliaSnail\\.jl" "extensions" %default-include)))
     (inputs
-     (list emacs-dash emacs-s emacs-spinner emacs-xref))
+     (list emacs-dash emacs-s emacs-spinner emacs-xref julia-cstparser
+           julia-tokenize))
     (propagated-inputs
      (list libvterm
            emacs-julia-mode             ;required by parser
            emacs-parsec                 ;required by parser
            emacs-popup
-           emacs-vterm
-           julia-tokenize
-           julia-cstparser))
+           emacs-vterm))
     (home-page "https://github.com/gcv/julia-snail")
     (synopsis "Development environment and REPL interaction package for Julia")
     (description "This package provides a development environment and REPL
diff --git a/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch b/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch
new file mode 100644
index 0000000000..eaa38fb868
--- /dev/null
+++ b/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch
@@ -0,0 +1,57 @@
+Author: Danny Milosavljevic <dannym@friendly-machines.com>
+Date: 2024-09-03
+License: GPL3+
+
+Previously, we propagated julia-cstparser and julia-tokenize. This would only
+work if the user had installed julia in their profile (because only then
+JULIA_LOAD_PATH of julia-cstparser would be propagated).
+That doesn't seem likely or desireable.
+
+It turns out that it's not difficult to make julia read vendored packages.
+Let's do that (since there's a comment inside JuliaSnail.jl that warns about
+CSTParser APIs being easily incompatible, vendoring is better anyway).
+
+diff -ru orig/4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl 4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl
+--- orig/4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl	2024-09-03 00:02:03.798373662 +0200
++++ 4l8xr3ldffb30a44x157lj3asaj3ykls-emacs-julia-snail-1.3.1-checkout/JuliaSnail.jl	2024-09-03 01:05:26.310766009 +0200
+@@ -30,12 +30,12 @@
+    catch err
+       if isa(err, ArgumentError)
+          if isfile(joinpath($dir, "Project.toml"))
+-            # force dependency installation
+-            Main.Pkg.activate($dir)
+-            Main.Pkg.instantiate()
+-            Main.Pkg.precompile()
+-            # activate what was the first entry before Snail was pushed to the head of LOAD_PATH
+-            Main.Pkg.activate(LOAD_PATH[2])
++             # force dependency installation
++             if isa(err, ArgumentError)
++                 error("Vendored dependencies not found. Please make sure the vendored packages are located in the 'vendor' directory.")
++             else
++                 rethrow(err)
++             end
+          end
+       end
+    finally
+@@ -51,11 +51,16 @@
+ end
+ 
+ @with_pkg_env (@__DIR__) begin
+-   # list all external dependency imports here (from the appropriate Project.toml, either Snail's or an extension's):
+-   import CSTParser
+-   # check for dependency API compatibility
+-   !isdefined(CSTParser, :iscall) &&
+-     throw(ArgumentError("CSTParser API not compatible, must install Snail-specific version"))
++    @with_pkg_env (joinpath(@__DIR__, "vendor", "CSTParser")) begin
++        @with_pkg_env (joinpath(@__DIR__, "vendor", "Tokenize")) begin
++            println(LOAD_PATH)
++            # list all external dependency imports here (from the appropriate Project.toml, either Snail's or an extension's):
++            import CSTParser
++            # check for dependency API compatibility
++            !isdefined(CSTParser, :iscall) &&
++                throw(ArgumentError("CSTParser API not compatible, must install Snail-specific version"))
++        end
++    end
+ end
+ 
+ 

base-commit: b833aaaee7c95ec0339428a6b602f26831494798
-- 
2.45.2

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Ludovic Courtès]

Hey Danny,

Danny Milosavljevic <dannym@friendly-machines.com> skribis:

> * gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/packages/emacs-xyz.scm (emacs-julia-snail)[arguments]<#:phases>[vendor]:
> New phase.
> [inputs]: Add julia-cstparser, julia-tokenize.
> [propagated-inputs]: Remove julia-cstparser, julia-tokenize.
> [source]: Add patch.
>
> Change-Id: I61c0a4493fcfd219809bcc35f8765be0928a7e81

[...]

> +          (add-after 'validate-compiled-autoloads 'vendor
> +            (lambda* (#:key inputs outputs #:allow-other-keys)
> +              (let* ((out (assoc-ref outputs "out"))
> +                     (base (dirname (car (find-files out "JuliaSnail.jl"))))
> +                     (vendor (string-append base "/vendor")))
> +                (mkdir-p vendor)
> +                (copy-recursively (string-append (assoc-ref inputs "julia-cstparser")
> +                                                 "/share/julia/loadpath/CSTParser")
> +                                  (string-append vendor "/CSTParser"))
> +                (copy-recursively (string-append (assoc-ref inputs "julia-tokenize")
> +                                                 "/share/julia/loadpath/Tokenize")

Use ‘search-input-directory’ rather than ‘assoc-ref’.

> +++ b/gnu/packages/patches/emacs-julia-snail-1.3.1-vendor.patch
> @@ -0,0 +1,57 @@
> +Author: Danny Milosavljevic <dannym@friendly-machines.com>
> +Date: 2024-09-03
> +License: GPL3+
> +
> +Previously, we propagated julia-cstparser and julia-tokenize. This would only
> +work if the user had installed julia in their profile (because only then
> +JULIA_LOAD_PATH of julia-cstparser would be propagated).
> +That doesn't seem likely or desireable.
> +
> +It turns out that it's not difficult to make julia read vendored packages.
> +Let's do that (since there's a comment inside JuliaSnail.jl that warns about
> +CSTParser APIs being easily incompatible, vendoring is better anyway).

I’m not sure I understand.  For most programming languages (Python,
etc.), we use propagated inputs like this.  How does that differ?

Normally we unbundle (“unvendor”?) libraries and this seems to go in the
opposite direction, right?

Thanks,
Ludo’.

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[dannym]

Hi Ludo,

On 2024-09-16 10:24, Ludovic Courtès wrote:
>> +                (copy-recursively (string-append (assoc-ref inputs 
>> "julia-tokenize")
>> +                                                 
>> "/share/julia/loadpath/Tokenize")
> 
> Use ‘search-input-directory’ rather than ‘assoc-ref’.

Will do!

> I’m not sure I understand.  For most programming languages (Python,
> etc.), we use propagated inputs like this.  How does that differ?
> 
> Normally we unbundle (“unvendor”?) libraries and this seems to go in 
> the
> opposite direction, right?

The idea is to make Guix more user-friendly.

I touched this only after I myself got bitten by the following:

I have guix home (with only a few packages--notably no julia (!) or 
gcc-toolchain
or anything like that in it),
and I have emacs set up with emacs-buffer-env and the following 
configuration:

   (setq buffer-env-script-name '("manifest.scm" ".envrc"))

In a julia project, manifest.scm contains:

   (specifications->manifest
     (list "julia"))

The idea is that emacs-bufferenv will automatically load that manifest 
when I
open a file in that directory within emacs--and then have julia-snail 
REPL
(including sending expression on point to the REPL) work.

Since the emacs julia-snail REPL integration needs to find the bounds of 
the expression,
it has to have a parser for the Julia language.

However, what happens now is that because I don't have julia-cstparser 
and julia-tokenize
in guix home, it won't work.

But from the standpoint of the profile I shouldn't have julia-cstparser 
and julia-tokenize
in my home profile since I don't personally directly use it (it's a 
requirement of emacs-julia-snail--I didn't even know what 
julia-cstparser is a few days ago :) ).

But nevermind that.  Even if I added julia-cstparser and julia-tokenize 
to my
guix home profile, it would NOT work.  The reason is because julia 
itself is not part of
the guix home profile, and the "julia" package would have a search-path 
specification that
would set some environment variables that julia needs to find 
julia-cstparser and
julia-tokenize.
Since the package is not there (in guix home) it won't set the 
variables.

The idea of this patch is that emacs will automatically set up whatever 
it needs to in order
to make the REPL work (including the finding of the boundaries of the 
current expression
that the emacs package needs for "send expression at point to REPL") 
without weird
dependencies bleeding explicitly into the user profile or into the local 
directory's manifest.

In my opinion, the user should not need to be concerned with what 
julia-cstparser is in this case.

Emacs should just magically load whatever it needs for the Julia REPL to 
work (whether
that's written in Julia or Python or whatever is an implementation 
detail--you COULD
have a Julia parser written in Python. Notably, most parsers for any 
language in
IntelliJ IDEA are written in Java--the Python parser, Rust parser etc).

That said, I'm totally open to other ways to do it.

The obvious alternative would be for the user to put julia-cstparser and 
julia-tokenize
into every Julia project's manifest.scm . But the error message is 
inscrutable if you
don't have those in there[1]--and I have no idea how any normal person 
would figure out
that julia-cstparser is needed in the first place.

[1] julia-snail--send-to-server: No Julia REPL buffer *julia* found; run 
julia-snail
Starting Julia process and loading Snail...
julia-snail--repl-enable: The REPL terminal buffer is inactive; 
double-check julia-snail-executable path

(In buffer "*scratch*" : )
julia-snail-executable
"julia"

(note: that "julia" excutable IS available in the manifest of the buffer 
env--and launching "julia" does work when launched manually within 
emacs)

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Liliana Marie Prikler]

Hi Danny,

Am Montag, dem 16.09.2024 um 20:01 +0200 schrieb
dannym@friendly-machines.com:
> The idea of this patch is that emacs will automatically set up
> whatever it needs to in order to make the REPL work (including the
> finding of the boundaries of the current expression that the emacs
> package needs for "send expression at point to REPL")  without weird
> dependencies bleeding explicitly into the user profile or into the
> local directory's manifest.
You could write a meta package à la "julia stuff for emacs-julia-snail"
and mention that in the description, no?  I think bundling (or
propagating) these dependencies from the package is likely not going to
do us favours in weird edge cases where you actually want more julia
packages, e.g. in the julia package that you may or may not be hacking
on, which has other dependencies :)

Cheers

[bug#72994] [PATCH] gnu: emacs-julia-snail: Vendor julia libraries.

[Ludovic Courtès]

Hi Danny & Liliana,

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> Am Montag, dem 16.09.2024 um 20:01 +0200 schrieb
> dannym@friendly-machines.com:
>> The idea of this patch is that emacs will automatically set up
>> whatever it needs to in order to make the REPL work (including the
>> finding of the boundaries of the current expression that the emacs
>> package needs for "send expression at point to REPL")  without weird
>> dependencies bleeding explicitly into the user profile or into the
>> local directory's manifest.

Thanks for explaining!

> You could write a meta package à la "julia stuff for emacs-julia-snail"
> and mention that in the description, no?

Yes, sounds like a good idea to me.  WDYT, Danny?

Ludo’.

[bug#72994] manifest.scm and emacs-specific things

[Danny Milosavljevic]

Hi!

> > On Mon, 16 Sep 2024 20:01 +0200 
> > dannym@friendly-machines.com wrote:
> >> The idea of this patch is that emacs will automatically set up
> >> whatever it needs to in order to make the REPL work (including the
> >> finding of the boundaries of the current expression that the emacs
> >> package needs for "send expression at point to REPL") without weird
> >> dependencies bleeding explicitly into the user profile or into the
> >> local directory's manifest.
> 
> Thanks for explaining!
> 
> > You could write a meta package à la "julia stuff for emacs-julia-snail"
> > and mention that in the description, no?
> 
> Yes, sounds like a good idea to me. WDYT, Danny?

I'm not yet sure.

I can of course add a "julia-toolchain" or something but it's really NOT
a julia toolchain thing. It's a local emacs-julia-snail thing that emacs
punted to some other language (as emacs is wont to do--delegate
things to other processes/languages/...).

A lot of dynamic languages do something like this in emacs and will
run into similar problems. Whatever the official way to do, we should
decide what to do and document it.

(I do think it's a good design to ask the actual Julia REPL about things
instead of reimplementing a Julia parser in elisp. That way it stays
current with reality)

The propagated inputs I would have added is so it WOULD conflict and
fail if you tried to add another Julia parser library (fail guix shell
on purpose!).

That's because, like most dynamic languages, you really can't load
two different modules that are named the same into the same
runtime without doing unholy things first. Otherwise, it would just
not do anything and use the existing one--which is probably not
what you wanted.

For the odd case when you are hacking on cstparser itself, you can
always not use the REPL and not use buffer-env. As you saw before,
julia-cstparser and julia-tokenize are currently NOT optional in
emacs anyway--the REPL won't start without--and with a totally
inscrutable error message that tries to make it julia's fault (it isn't).

So what I'm envisioning is documenting emacs-buffer-env like that:

(define-public emacs-buffer-env
  (package
    ...
    (description "In emacs, emacs-buffer-env can pick up manifest.scm from a
project directory automatically.

The contents of manifest.scm should be as follows:
(specifications->manifest (list

...
@item for Julia, it is recommended to add "julia-emacs-toolchain"
...
))
")))
Or something?

Because the following would leak emacs implementation details (arguably in a lot of cases!):

    (description "In emacs, emacs-buffer-env can pick up manifest.scm from a
project directory automatically.

The contents of manifest.scm should be as follows:
(specifications->manifest (list

@item for C and C++, it is recommended to add "gcc-toolchain" "binutils" "ccls" "bear"
@item for Python, it is recommended to add "python" "python-debugpy" "python-lsp-server"
@item for LaTeX, it is recommended to add "texlive-minted" "texlive-latex-bin" "dvisvgm"
@item for Prolog, it is recommended to add "swi-prolog" "emacs-ediprolog"
@item for Julia, it is recommended to add "julia" "julia-cstparser" "julia-tokenize"
@item for Vala, it is recommended to add "vala" "gcc-toolchain" "binutils" "vala-language-server" "gobject-introspection" "pkg-config"
        "meson" "ninja"
        ;; Note: That's the wrong glib
        "glib"
        ;; Note: That's the wrong gtk+
        "gtk+"
@item for Go, it is recommended to add "go"
@item for Rust, it is recommended to add "rust" "rust:cargo" "gcc-toolchain"
@item for R, it is recommended to add "r"
@item for Rust, it is recommendeed to add "ruby"
@item for Haskell, it is recommended to add "ghc" "gcc-toolchain"
@item for Java, it is recommended to add "openjdk" "openjdk:jdk"
@item for Zig, it is recommended to add "zig" "zig-zls"
@item for Perl, it is recommended to add "perl"
@item for SBCL, it is recommended to add "sbcl"
@item for Node, it is recommended to add "node"
@item for FPC, it is recommended to add "fpc"
@item for Ocaml, it is recommended to add "ocaml"
@item for Racket, it is recommended to add "racket"
@item for Chicken, it is recommended to add "chicken"
@item for D, it is recommended to add "dub" "ldc" "gcc-toolchain"
@item for Nim, it is recommended to add "nim"

))
")))

Adding julia-cstparser julia-tokenize inside julia-toolchain would be weird, no?

Also, for example for Python, worse, there's a python LSP server emacs thing, OR
there's a non-LSP emacs thing that asks the Python REPL directly  (!). I actually
prefer the latter because C-M-x works in emacs Python then (but C-M-x doesn't
work with python-debugpy and python-lsp-server in emacs--at least not for me).

I think making a public package "julia-emacs-toolchain" would be less weird.

(define-public julia-emacs-toolchain
  (package
    ...
    (propagated-inputs (list julia julia-cstparser julia-tokenize))))

Like that?

Which variant exactly do we want? Do we want end users to create
"manifest.scm"s like that, i.e. should those "xxx-emacs-toolchain"
package names be part of a Guix stable interface ?

emacs-julia-snail would not propagate julia-emacs-toolchain
then? Or would it?

(Personally, I think in an ideal world, *emacs* would run in its own
guix profile and when you do M-x list-packages in emacs and install
a package it would just install it into its emacs profile using
"guix package"[1]. Somehow, the emacs environment so configured
should not bleed into processes started by the user inside emacs.
But that's a long way off maybe and not sure whether it would be
a good idea.
In any case it should be *almost* independent of this here)

[bug#72994] manifest.scm and emacs-specific things

[Liliana Marie Prikler]

Hi,

Am Samstag, dem 21.12.2024 um 19:33 +0100 schrieb Danny Milosavljevic:
> I think making a public package "julia-emacs-toolchain" would be less
> weird.
> 
> (define-public julia-emacs-toolchain
>   (package
>     ...
>     (propagated-inputs (list julia julia-cstparser julia-tokenize))))
> 
> Like that?
> 
> Which variant exactly do we want? Do we want end users to create
> "manifest.scm"s like that, i.e. should those "xxx-emacs-toolchain"
> package names be part of a Guix stable interface ?
> 
> emacs-julia-snail would not propagate julia-emacs-toolchain
> then? Or would it?
Weird naming choice aside (commented upon below), I don't think it
would even be an input to emacs-julia-snail, much less a propagated
one.  The package, if we need one at all, would need to be public so
that users can actually use it.

I don't think referring to a manifest as necessarily "manifest.scm"
helps us here.  If you only have one "guix.scm" and one "manifest.scm"
at most, then yes, whatever you need for your hacking setup to work
will have to bleed into the latter.  But it wouldn't be too weird
putting emacs, emacs-snail, julia, julia-cstparser and julia-tokenize
into one "snail-manifest.scm", that you can reuse for multiple
projects.

If you want to look at a comparable situation, look at emacs-geiser and
its implementations like emacs-geiser-guile.  Here, the reference to
guile in geiser is patched to point at the correct guile – load paths
pose no issue.  If we can do something similar for julia, that'd be
great, but… I think leaking cstparser might be a no-no, would it?

> (Personally, I think in an ideal world, *emacs* would run in its own
> guix profile and when you do M-x list-packages in emacs and install
> a package it would just install it into its emacs profile using
> "guix package"[1]. Somehow, the emacs environment so configured
> should not bleed into processes started by the user inside emacs.
> But that's a long way off maybe and not sure whether it would be
> a good idea.  In any case it should be *almost* independent of this
> here)
I mean, you could start pure guix shells as emacs subprocesses.  I
think there are also packages from setting emacs internals within some
specified scope – such as buffer-env.  "something-emacs-toolchain" is
imho a weird name for a(n implied) dependency for a particular emacs
package.  

Other than that you could also spawn a special emacs for julia
development from its own manifest.  Would be a separate emacs process,
sure, but helps isolation :)

Cheers

[bug#72994] manifest.scm and emacs-specific things

[Danny Milosavljevic]

Hi Liliana

> I don't think referring to a manifest as necessarily "manifest.scm"
> helps us here.

It's the one that guix shell and emacs-buffer-env pick up by default.

> If you only have one "guix.scm" and one "manifest.scm"
> at most, then yes, whatever you need for your hacking setup to work
> will have to bleed into the latter. But it wouldn't be too weird
> putting emacs, emacs-snail, julia, julia-cstparser and julia-tokenize
> into one "snail-manifest.scm", that you can reuse for multiple
> projects.

Tomorrow, emacs-julia-snail could drop cstparser for xyzparser.

And "snail-manifest.scm" is not what buffer-env uses. We'd have to
patch buffer-env so it tries 10 different names in priority order or
something.

Or every programmer would have to manually set up init.el to do:

(setq buffer-env-script-name
 '("julia-snail-manifest.scm"
   "python-lsp-manifest.scm"
   "python-repl-manifest.scm"
   "c-manifest.scm"
   ...
   "xyz-manifest.scm"
   "manifest.scm"
   ".envrc"))

I think that's a cop out.
Instead of fixing it, make the user fiddle with it manually. It's the unix way.
Let's not do that if possible.

> If you want to look at a comparable situation, look at emacs-geiser and
> its implementations like emacs-geiser-guile. Here, the reference to
> guile in geiser is patched to point at the correct guile – load paths
> pose no issue.

I had a look at it now. Not sure it's the same situation. Is it?

I think in the guile situation it's not that bad because parsing guile is not
exactly difficult enough to need a library (or two :) ).

In the case of julia the situation is:

emacs-julia-snail launches the executable "julia" and uses
julia's "cstparser" and "tokenize"--both of which are non-standard
julia modules--to figure out what of the communication buffer to use.

(It has to do that because the julia grammar is complicated)

Note: The julia REPL SHOULD load any other user modules from
the guix environment of the manifest! That is, the current guix
environment is the correct environment for the REPL for almost
every module (except two--and even those maybe it should pick
up in all non-automatic-emacs situations).

-----------------

In my earlier attempts using julia-snail on guix master was even worse:

If "julia" is not in the manifest but "julia-cstparser" is in the manifest,
then the search path for julia modules will not be set up and
consequently cstparser will not be found.
That is the case even if "julia" would be in the guix profile anyway (!!).

I fixed it by installing "don't do that then" into my head.

But my goal is not fixing it just for myself (I did work around this
stuff already) but to make the experience of guix users better.

Do you agree that this specific situation is terrible? I mean I know
why it is like that technically--but it's still bad. Can't we have nested
guix environments somehow pick up search paths even in that
situation?
Or warn about the situation?
Anything but what it does currently.
-----------------

> If we can do something similar for julia, that'd be
> great, but… I think leaking cstparser might be a no-no, would it?

What do you mean, exactly?

If you mean this:

If the user has to explicitly specify the word "cstparser" it will become an official
interface and will break if emacs-julia-snail ever uses xyzparser instead of cstparser
in the future. Therefore, emacs-julia-snail must not switch those out in the next
10 years or so.

I mean we can do that but in my opinion it's not the best solution.

If you mean that:

The only problem with bundling cstparser is that you can't load another
cstparser as the end user in the same (snail) repl. You will just get the
first one if you try.
That's it.

Maybe there's even something like (@ (cstparser) x) that wouldn't even
keep the module loaded. If so, would also be a possibility to use that.

Would it be so bad? For me, it's important that guix doesn't silently
do nothing when I put another cstparser in the same environment.
I have no problem even with an error message and failure that says
that there's already a cstparser and why am I trying to add another
cstparser.

For that matter, we could rename the emacs julia cstparser module to
some random string and use it like that. That's ... actually my
favorite solution now!

It's all nice and good to have all those possibilities but which one do
we take, or at least recommend? The situation in guix master surely is
not acceptable--see my older mail where I described just how bad it
is in vivid detail.

> I mean, you could start pure guix shells as emacs subprocesses.

Good idea!  This is an excellent observation and I can think of
multiple other bad guix situations (end user programs pick the
wrong gtk up etc) that can (and probably should) be fixed like
that for everyone!

(For using a supposedly purely functional package manager I'm
using a surprising amount of my lifetime to prevent
environmental leakage from breaking my stuff.
I *should* do "guix shell --pure" right after login so the gdm
env var leakage into my desktop session is gone, for example)

For building projects I use

  guix build -f guix.scm

(yes, from within emacs) and those are isolated anyway.

So I guess if emacs did some extra weird stuff when the user
is not building the project yet it would not be that bad anyway.

[bug#72994] manifest.scm and emacs-specific things

[Liliana Marie Prikler]

Am Samstag, dem 21.12.2024 um 23:58 +0100 schrieb Danny Milosavljevic:
> Or every programmer would have to manually set up init.el to do:
> 
> (setq buffer-env-script-name
>  '("julia-snail-manifest.scm"
>    "python-lsp-manifest.scm"
>    "python-repl-manifest.scm"
>    "c-manifest.scm"
>    ...
>    "xyz-manifest.scm"
>    "manifest.scm"
>    ".envrc"))
> 
> I think that's a cop out.  Instead of fixing it, make the user fiddle
> with it manually. It's the unix way.  Let's not do that if possible.
Guess I'm not a programmer™ then.  

> > If you want to look at a comparable situation, look at emacs-geiser
> > and its implementations like emacs-geiser-guile. Here, the
> > reference to guile in geiser is patched to point at the correct
> > guile – load paths pose no issue.
> 
> I had a look at it now. Not sure it's the same situation. Is it?
> 
> I think in the guile situation it's not that bad because parsing
> guile is not exactly difficult enough to need a library (or two :) ).
> 
> In the case of julia the situation is:
> 
> emacs-julia-snail launches the executable "julia" and uses
> julia's "cstparser" and "tokenize"--both of which are non-standard
> julia modules--to figure out what of the communication buffer to use.
> 
> (It has to do that because the julia grammar is complicated)
Geiser also has to connect to a running REPL at least for parts of its
functionality.  And it achieves a "good default" by having latest guile
as an input.  For Julia one could also solve this with an LSP probably
(which would require having that LSP running, tho), or tree-sitter, but
let's look at emacs-julia-snail in isolation, because that's the
package we're aiming to fix, no?

> Note: The julia REPL SHOULD load any other user modules from
> the guix environment of the manifest! That is, the current guix
> environment is the correct environment for the REPL for almost
> every module (except two--and even those maybe it should pick
> up in all non-automatic-emacs situations).
> 
> -----------------
> 
> In my earlier attempts using julia-snail on guix master was even
> worse:
> 
> If "julia" is not in the manifest but "julia-cstparser" is in the
> manifest, then the search path for julia modules will not be set up
> and consequently cstparser will not be found.
> That is the case even if "julia" would be in the guix profile anyway
> (!!).
You're free to find a general fix to 20255, but I think that's outside
of scope given the workaround should well-known and well-documented by
now.  Quoting the manual on search paths

     Note: Notice that ‘GUIX_PYTHONPATH’ is specified as part of the
     definition of the ‘python’ package, and _not_ as part of that of
     ‘python-numpy’.  This is because this environment variable
     “belongs” to Python, not NumPy: Python actually reads the value of
     that variable and honors it.

     Corollary: if you create a profile that does not contain ‘python’,
     ‘GUIX_PYTHONPATH’ will _not_ be defined, even if it contains
     packages that provide ‘.py’ files:

          $ guix shell python-numpy --search-paths --pure
          export PATH="/gnu/store/...-profile/bin"

     This makes a lot of sense if we look at this profile in isolation:
     no software in this profile would read ‘GUIX_PYTHONPATH’.

> > If we can do something similar for julia, that'd be
> > great, but… I think leaking cstparser might be a no-no, would it?
> 
> What do you mean, exactly?
> 
> If you mean this:
> 
> If the user has to explicitly specify the word "cstparser" it will
> become an official interface and will break if emacs-julia-snail ever
> uses xyzparser instead of cstparser in the future. Therefore, emacs-
> julia-snail must not switch those out in the next 10 years or so.
I really don't think your conclusion follows from your premise.

> I mean we can do that but in my opinion it's not the best solution.
> 
> If you mean that:
> 
> The only problem with bundling cstparser is that you can't load
> another cstparser as the end user in the same (snail) repl. You will
> just get the first one if you try.  That's it.
> 
> Maybe there's even something like (@ (cstparser) x) that wouldn't
> even keep the module loaded. If so, would also be a possibility to
> use that.
The problem with propagating another cstparser would be that Guix yells
at you for propagated inputs.  That's fine enough for the elisp stuff –
there's no real way around it – but we tend to avoid it for non-elisp
stuff in emacs packages.  Again: look at geiser.

> Would it be so bad? For me, it's important that guix doesn't silently
> do nothing when I put another cstparser in the same environment.
> I have no problem even with an error message and failure that says
> that there's already a cstparser and why am I trying to add another
> cstparser.
> 
> For that matter, we could rename the emacs julia cstparser module to
> some random string and use it like that. That's ... actually my
> favorite solution now!
> 
> It's all nice and good to have all those possibilities but which one
> do we take, or at least recommend? The situation in guix master
> surely is not acceptable--see my older mail where I described just
> how bad it is in vivid detail.
Describing how bad things are "in vivid detail" tends to only work for
those who already agree with you.  For me, all the extra emotional
baggage sounds like noise that I'd rather not deal with.  Mind you, I
can also be quite emotional on subjects, so I understand how you feel,
but I think appealing to emotions will not help us find workable
solutions.

As for those workable solutions, I think you could provide a julia that
is wrapped with a suffix env to include cstparser and tokenizer.  Let's
name that prospective package "julia-for-emacs-julia-snail" for now. 
You would replace the julia binary in emacs-julia-snail with bin/julia
from julia-for-emacs-julia-snail just as you would with any other emacs
package (cf. geiser).

Alternatively, if julia-snail was a standalone package that offers the
julia side of things, we could install a "julia-snail-repl" to bin or
libexec and refer to that.  As far as I can see, julia-snail is not a
standalone julia package, though.

> > I mean, you could start pure guix shells as emacs subprocesses.
> 
> Good idea!  This is an excellent observation and I can think of
> multiple other bad guix situations (end user programs pick the
> wrong gtk up etc) that can (and probably should) be fixed like
> that for everyone!
> 
> (For using a supposedly purely functional package manager I'm
> using a surprising amount of my lifetime to prevent
> environmental leakage from breaking my stuff.
> I *should* do "guix shell --pure" right after login so the gdm
> env var leakage into my desktop session is gone, for example)
I don't know how functional package management is supposed to interfere
with processes in your opinion.  I know that systemd goes long ways to
provide "clean" environments (which also break stuff unless you find
that one meaningful config file to edit, mind you), but we're not in a
position where we force everyone to use shepherd for everything.¹

Anyway, let's look at the actual problem within its actual scope. 
Extrapolating from "emacs-julia-snail" not working as intended that
"actually, Guix is broken" is – in my eyes at least – a huge leap I
don't want to take.

Cheers

¹ Yes, I know it's included in Guix Home.  No, that's not relevant to
my point.