bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Jimmy Yuen Ho Wong]

This is a bit of a strange one.

I'm testing 2 custom compiled emacs on macOS, both were built via [this
recipe](https://github.com/macports/macports-ports/blob/master/editors/emacs/Portfile)
but with the latest emacs-29 commit and its corresponding checksums.

The NS port built from the emacs-app-devel port (I'm submitting this
report from the TTY port), is able to find the tree-sitter language
libraries, but not the TTY port, despite both were configured using the
exact same prefix, CFLAGS, LDFLAGS and the --with-tree-sitter flag on.


## TTY emacs 29 on macOS:

ESC M-: (treesit-language-available-p 'java t) RET

```
(nil not-found ("libtree-sitter-java.so" "libtree-sitter-java.so.0" "libtree-sitter-java.so.0.0" "libtree-sitter-java.dylib" "libtree-sitter-java.dylib.0" "libtree-sitter-java.dylib.0.0") "No such file or directory")
```

## NS emacs 29:

ESC M-: (treesit-language-available-p 'java t) RET

```
t
```


## Installed MacPorts tree-sitter library versions:

```
tree-sitter @0.20.8_1 (active)
tree-sitter-bash @0.19.0_0 (active)
tree-sitter-c @0.20.2_0 (active)
tree-sitter-c-sharp @0.20.0_0 (active)
tree-sitter-cmake @0.1.0_0 (active)
tree-sitter-cpp @0.20.0_0 (active)
tree-sitter-css @0.19.0_0 (active)
tree-sitter-dockerfile @0.1.2_0 (active)
tree-sitter-go @0.19.1_0 (active)
tree-sitter-go-mod @20220517_0 (active)
tree-sitter-html @0.19.0_0 (active)
tree-sitter-java @0.20.1_0 (active)
tree-sitter-javascript @0.20.0_0 (active)
tree-sitter-json @0.19.0_0 (active)
tree-sitter-python @0.20.0_0 (active)
tree-sitter-ruby @0.19.0_0 (active)
tree-sitter-rust @0.20.3_0 (active)
tree-sitter-toml @0.5.1_0 (active)
tree-sitter-tsx @0.20.2_0 (active)
tree-sitter-typescript @0.20.2_0 (active)
tree-sitter-yaml @0.5.0_0 (active)
```


## TTY emacs configuration

In GNU Emacs 29.0.91 (build 1, aarch64-apple-darwin22.4.0) of 2023-05-27
 built on MobileCat.localdomain
System Description:  macOS 13.3.1

Configured using:
 'configure --prefix=/opt/local --disable-silent-rules --without-ns
 --without-x --without-dbus --without-gconf --without-libotf
 --without-m17n-flt --with-libgmp --with-gnutls --with-json --with-xml2
 --with-modules --infodir /opt/local/share/info/emacs --with-sqlite3
 --with-webp --with-tree-sitter --with-native-compilation=aot
 'CFLAGS=-pipe -Os -Wno-attributes
 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -arch
 arm64' 'CPPFLAGS=-I/opt/local/include
 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk'
 'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath
 /opt/local/lib/gcc12 -Wl,-no_pie
 -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk
 -arch arm64''

Configured features:
ACL GMP GNUTLS JSON LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY KQUEUE
PDUMPER SQLITE3 THREADS TREE_SITTER ZLIB

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix


## NS emacs configuration

In GNU Emacs 29.0.91 (build 1, aarch64-apple-darwin22.4.0, NS
 appkit-2299.50 Version 13.3.1 (a) (Build 22E772610a)) of 2023-05-26
 built on MobileCat.localdomain
Windowing system distributor 'Apple', version 10.3.2299
System Description:  macOS 13.3.1

Configured using:
 'configure --prefix=/opt/local --disable-silent-rules --without-dbus
 --without-gconf --without-libotf --without-m17n-flt --with-libgmp
 --with-gnutls --with-json --with-xml2 --with-modules --infodir
 /opt/local/share/info/emacs --with-sqlite3 --with-webp --with-ns
 --with-lcms2 --without-harfbuzz --without-imagemagick --without-xaw3d
 --with-tree-sitter --with-rsvg --with-xwidgets
 --with-native-compilation=aot 'CFLAGS=-pipe -Os -Wno-attributes
 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -arch
 arm64' 'CPPFLAGS=-I/opt/local/include
 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk'
 'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath
 /opt/local/lib/gcc12 -Wl,-no_pie
 -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk
 -arch arm64''

Configured features:
ACL GIF GLIB GMP GNUTLS JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XWIDGETS ZLIB

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Eli Zaretskii]

> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> Date: Sun, 28 May 2023 00:46:08 +0100
> 
> 
> This is a bit of a strange one.
> 
> I'm testing 2 custom compiled emacs on macOS, both were built via [this
> recipe](https://github.com/macports/macports-ports/blob/master/editors/emacs/Portfile)
> but with the latest emacs-29 commit and its corresponding checksums.
> 
> The NS port built from the emacs-app-devel port (I'm submitting this
> report from the TTY port), is able to find the tree-sitter language
> libraries, but not the TTY port, despite both were configured using the
> exact same prefix, CFLAGS, LDFLAGS and the --with-tree-sitter flag on.
> 
> 
> ## TTY emacs 29 on macOS:
> 
> ESC M-: (treesit-language-available-p 'java t) RET
> 
> ```
> (nil not-found ("libtree-sitter-java.so" "libtree-sitter-java.so.0" "libtree-sitter-java.so.0.0" "libtree-sitter-java.dylib" "libtree-sitter-java.dylib.0" "libtree-sitter-java.dylib.0.0") "No such file or directory")
> ```
> 
> ## NS emacs 29:
> 
> ESC M-: (treesit-language-available-p 'java t) RET
> 
> ```
> t
> ```

This probably means the TTY version uses a different list of
directories to look for shared libraries, in which case this should be
fixed on your end by a suitable system configuration.  Emacs just uses
the "normal" system procedures for loading shared libraries, AFAIK.

Any macOS expert out there who could explain what is going on?

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Jimmy Yuen Ho Wong]

[-- Attachment #1: Type: text/plain, Size: 3901 bytes --]

I only have 1 copy of those tree-sitter libraries in my entire system, and
on both the TTY and NS ports, (treesit-available-p) returns t. All of the
tree-sitter libraries are in /opt/local/lib, it's just the TTY port isn't
able to pick them up. Here's a directory listing

```
rwxr-xr-x    1 root      admin   471K 24 Feb 21:06
libtree-sitter-bash.dylib*
-rwxr-xr-x    1 root      admin   3.8M 24 Feb 21:05
libtree-sitter-c-sharp.dylib*
-rwxr-xr-x    1 root      admin   372K 24 Feb 21:05 libtree-sitter-c.dylib*
-rwxr-xr-x    1 root      admin    98K 24 Feb 21:06
libtree-sitter-cmake.dylib*
-rwxr-xr-x    1 root      admin   1.5M 24 Feb 21:05
libtree-sitter-cpp.dylib*
-rwxr-xr-x    1 root      admin    82K 24 Feb 21:05
libtree-sitter-css.dylib*
-rwxr-xr-x    1 root      admin    65K 24 Feb 21:06
libtree-sitter-dockerfile.dylib*
-rwxr-xr-x    1 root      admin   194K 24 Feb 21:06 libtree-sitter-go.dylib*
-rwxr-xr-x    1 root      admin    33K 24 Feb 21:06
libtree-sitter-gomod.dylib*
-rwxr-xr-x    1 root      admin    72K 27 Feb 14:04
libtree-sitter-html.dylib*
-rwxr-xr-x    1 root      admin   372K 24 Feb 21:05
libtree-sitter-java.dylib*
-rwxr-xr-x    1 root      admin   324K 27 Feb 14:04
libtree-sitter-javascript.dylib*
-rwxr-xr-x    1 root      admin    33K 24 Feb 21:05
libtree-sitter-json.dylib*
-rwxr-xr-x    1 root      admin   277K 24 Feb 21:05
libtree-sitter-python.dylib*
-rwxr-xr-x    1 root      admin   1.0M 27 Feb 14:04
libtree-sitter-ruby.dylib*
-rwxr-xr-x    1 root      admin   760K 24 Feb 21:06
libtree-sitter-rust.dylib*
-rwxr-xr-x    1 root      admin    34K 24 Feb 21:06
libtree-sitter-toml.dylib*
-rwxr-xr-x    1 root      admin   1.1M 24 Feb 21:05
libtree-sitter-tsx.dylib*
-rwxr-xr-x    1 root      admin   1.1M 24 Feb 21:05
libtree-sitter-typescript.dylib*
-rwxr-xr-x    1 root      admin   213K 24 Feb 21:06
libtree-sitter-yaml.dylib*
-rwxr-xr-x    1 root      admin   184K 27 Apr 22:31
libtree-sitter.0.0.dylib*
lrwxr-xr-x    1 root      admin    24B 27 Apr 22:31 libtree-sitter.0.dylib@
-> libtree-sitter.0.0.dylib
-rwxr-xr-x    1 root      admin   199K 27 Apr 22:31 libtree-sitter.a*
lrwxr-xr-x    1 root      admin    24B 27 Apr 22:31 libtree-sitter.dylib@
-> libtree-sitter.0.0.dylib
```

Jimmy


On Sun, May 28, 2023 at 6:36 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> > Date: Sun, 28 May 2023 00:46:08 +0100
> >
> >
> > This is a bit of a strange one.
> >
> > I'm testing 2 custom compiled emacs on macOS, both were built via [this
> > recipe](
> https://github.com/macports/macports-ports/blob/master/editors/emacs/Portfile
> )
> > but with the latest emacs-29 commit and its corresponding checksums.
> >
> > The NS port built from the emacs-app-devel port (I'm submitting this
> > report from the TTY port), is able to find the tree-sitter language
> > libraries, but not the TTY port, despite both were configured using the
> > exact same prefix, CFLAGS, LDFLAGS and the --with-tree-sitter flag on.
> >
> >
> > ## TTY emacs 29 on macOS:
> >
> > ESC M-: (treesit-language-available-p 'java t) RET
> >
> > ```
> > (nil not-found ("libtree-sitter-java.so" "libtree-sitter-java.so.0"
> "libtree-sitter-java.so.0.0" "libtree-sitter-java.dylib"
> "libtree-sitter-java.dylib.0" "libtree-sitter-java.dylib.0.0") "No such
> file or directory")
> > ```
> >
> > ## NS emacs 29:
> >
> > ESC M-: (treesit-language-available-p 'java t) RET
> >
> > ```
> > t
> > ```
>
> This probably means the TTY version uses a different list of
> directories to look for shared libraries, in which case this should be
> fixed on your end by a suitable system configuration.  Emacs just uses
> the "normal" system procedures for loading shared libraries, AFAIK.
>
> Any macOS expert out there who could explain what is going on?
>

[-- Attachment #2: Type: text/html, Size: 4968 bytes --]

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Eli Zaretskii]

> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> Date: Sun, 28 May 2023 09:15:15 +0100
> Cc: 63762@debbugs.gnu.org
> 
> I only have 1 copy of those tree-sitter libraries in my entire system, and on both the TTY and NS ports,
> (treesit-available-p) returns t. All of the tree-sitter libraries are in /opt/local/lib, it's just the TTY port isn't
> able to pick them up. Here's a directory listing

What I meant to say is that by macOS conventions, a console program
looks for shared libraries in different places.  But that's a guess; I
don't really know how this works on macOS.

All I can suggest is step in a debugger through the code in
treesit_load_language, and see why the TTY version fails to find the
grammar libraries.  I don't have this problem on MS-Windows, FWIW.  So
it is something macOS-specific, and we need a macOS expert to
investigate.

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Jimmy Yuen Ho Wong]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

gdb doesn't work with clang and I have a hard time using lldb, all I can
say for sure is, for some reason, `treesit-extra-load-path' is nil when
running TTY emacs on macOS, whereas it is set to '("/opt/local/lib") on the
NS port.

Can Alan help?

Jimmy


On Sun, May 28, 2023 at 9:31 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> > Date: Sun, 28 May 2023 09:15:15 +0100
> > Cc: 63762@debbugs.gnu.org
> >
> > I only have 1 copy of those tree-sitter libraries in my entire system,
> and on both the TTY and NS ports,
> > (treesit-available-p) returns t. All of the tree-sitter libraries are in
> /opt/local/lib, it's just the TTY port isn't
> > able to pick them up. Here's a directory listing
>
> What I meant to say is that by macOS conventions, a console program
> looks for shared libraries in different places.  But that's a guess; I
> don't really know how this works on macOS.
>
> All I can suggest is step in a debugger through the code in
> treesit_load_language, and see why the TTY version fails to find the
> grammar libraries.  I don't have this problem on MS-Windows, FWIW.  So
> it is something macOS-specific, and we need a macOS expert to
> investigate.
>

[-- Attachment #2: Type: text/html, Size: 1872 bytes --]

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Eli Zaretskii]

> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> Date: Sun, 28 May 2023 10:27:16 +0100
> Cc: 63762@debbugs.gnu.org
> 
> gdb doesn't work with clang and I have a hard time using lldb, all I can say for sure is, for some
> reason, `treesit-extra-load-path' is nil when running TTY emacs on macOS, whereas it is set to '
> ("/opt/local/lib") on the NS port.

treesit-extra-load-path is nil in my session, and that is its normal
value.  How did you get it set to something non-nil in the NS build?
I see nothing in the sources, so maybe it's your customizations?

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Jimmy Yuen Ho Wong]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

Nope, it's all automatic. Something somewhere in the NS port is setting
`treesit-extra-load-path' correctly, despite both ports were compiled with
LDFLAGS='-L/opt/local/lib'

Jimmy


On Sun, May 28, 2023 at 10:42 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> > Date: Sun, 28 May 2023 10:27:16 +0100
> > Cc: 63762@debbugs.gnu.org
> >
> > gdb doesn't work with clang and I have a hard time using lldb, all I can
> say for sure is, for some
> > reason, `treesit-extra-load-path' is nil when running TTY emacs on
> macOS, whereas it is set to '
> > ("/opt/local/lib") on the NS port.
>
> treesit-extra-load-path is nil in my session, and that is its normal
> value.  How did you get it set to something non-nil in the NS build?
> I see nothing in the sources, so maybe it's your customizations?
>

[-- Attachment #2: Type: text/html, Size: 1463 bytes --]

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Jimmy Yuen Ho Wong]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

Actually, I take it back, it appears to be sit in this patch.
https://github.com/macports/macports-ports/blob/master/editors/emacs/files/site-start.el

This appears to be a MacPorts packaging issue, you can close this.

Jimmy


On Sun, May 28, 2023 at 10:47 AM Jimmy Yuen Ho Wong <wyuenho@gmail.com>
wrote:

> Nope, it's all automatic. Something somewhere in the NS port is setting
> `treesit-extra-load-path' correctly, despite both ports were compiled with
> LDFLAGS='-L/opt/local/lib'
>
> Jimmy
>
>
> On Sun, May 28, 2023 at 10:42 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
>> > Date: Sun, 28 May 2023 10:27:16 +0100
>> > Cc: 63762@debbugs.gnu.org
>> >
>> > gdb doesn't work with clang and I have a hard time using lldb, all I
>> can say for sure is, for some
>> > reason, `treesit-extra-load-path' is nil when running TTY emacs on
>> macOS, whereas it is set to '
>> > ("/opt/local/lib") on the NS port.
>>
>> treesit-extra-load-path is nil in my session, and that is its normal
>> value.  How did you get it set to something non-nil in the NS build?
>> I see nothing in the sources, so maybe it's your customizations?
>>
>

[-- Attachment #2: Type: text/html, Size: 2281 bytes --]

bug#63762: 29.0.91; tty emacs on macOS fail to load tree-sitter

[Eli Zaretskii]

> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> Date: Sun, 28 May 2023 10:50:32 +0100
> Cc: 63762@debbugs.gnu.org
> 
> Actually, I take it back, it appears to be sit in this patch.
> https://github.com/macports/macports-ports/blob/master/editors/emacs/files/site-start.el

Now everything is clear.  Thanks.

> This appears to be a MacPorts packaging issue, you can close this.

Done.