bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Jim Myhrberg]

Commit 5dd2d50 seems to have moved the native-lisp folder within self-contained
Emacs.app bundles:

- from: "Contents/Resources/native-lisp"
- to:   "Contents/MacOS/lib/emacs/28.0.50/native-lisp"

Unfortunately, Apple's codesign utility blows up with an error if there is any
folder within "Contents/MacOS" (recursively) which contains two dots within its
name. Here is an example of what the error looks like:

    /Users/runner/work/emacs-builds/emacs-builds/builds/Emacs.2021-06-26.b8f9e58.master.macOS-10-15.x86_64/Emacs.app:
bundle format unrecognized, invalid, or unsuitable
    In subcomponent:
/Users/runner/work/emacs-builds/emacs-builds/builds/Emacs.2021-06-26.b8f9e58.master.macOS-10-15.x86_64/Emacs.app/Contents/MacOS/lib/emacs/28.0.50

The above error is from this GitHub Actions workflow run:
https://github.com/jimeh/emacs-builds/runs/2923284812?check_suite_focus=true

The same issue happens locally for me too.

I have so far not found a way to make codesign not blow up on such folders, so
for now I've had to workaround the issue in my build script with the following
changes:

- https://github.com/jimeh/build-emacs-for-macos/commit/eeca7b798de236a3ffc1ab04b0f7735a37ce5af4
- https://github.com/jimeh/build-emacs-for-macos/commit/23b8236e0a66fb09810e8422bedf02f7192a53e4

In summary, I rename:

- from: Emacs.app/Contents/MacOS/lib/emacs/28.0.50/native-lisp/28.0.50-852ecda2
- to:   Emacs.app/Contents/MacOS/lib/emacs/28-0-50/native-lisp/28-0-50-852ecda2

(Note that I had to rename both "28.0.50" and "28.0.50-852ecda2"
folders, as both
trigger the error from codesign.)

I create a symlink from "Contents/native-lisp" to
"Contents/MacOS/lib/emacs/28-0-50/native-lisp".

And finally I patch Emacs.pdmp (Emacs.app/Contents/MacOS/libexec/Emacs.pdmp),
performing the following raw text replacements:

- "lib/emacs/28.0.50/native-lisp/28.0.50-852ecda2/" -->
"lib/emacs/28-0-50/native-lisp/28-0-50-852ecda2/"
- "../native-lisp/28.0.50-852ecda2/" --> "../native-lisp/28-0-50-852ecda2/"

This yields a seemingly actually working Emacs.app bundle, but it re-compiles
all *.eln files in the bundle into the user cache folder. However, there's a
separate bug causing re-compiling which I've reported as bug#49270, so for now I
don't know if my patching would prevent Emacs from dynamically finding existing
*.eln which are not referenced from Emacs.pdmp.

This patching/workaround to get code signing to work feels very hacky and
fragile. It would be great if we could avoid the need for this kind of patch to
be able to codesign the .app bundle.

The way I see it, there's three possible solutions:

- Figure out a way to make Apple's codesign play nice.
- Move the native-lisp folder back to "Contents/Resources/native-lisp", as
  codesign does not complain about folders with two dots in their names within
  the Resources directory.
- Change paths used to store *.eln files within Contents/MacOS so no single
  folder name contains two dots.

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

On Tue, Jun 29, 2021 at 12:58:44PM +0100, Jim Myhrberg wrote:
> Commit 5dd2d50 seems to have moved the native-lisp folder within self-contained
> Emacs.app bundles:
> 
> - from: "Contents/Resources/native-lisp"
> - to:   "Contents/MacOS/lib/emacs/28.0.50/native-lisp"
> 
> Unfortunately, Apple's codesign utility blows up with an error if there is any
> folder within "Contents/MacOS" (recursively) which contains two dots within its
> name. Here is an example of what the error looks like:
> 
>     /Users/runner/work/emacs-builds/emacs-builds/builds/Emacs.2021-06-26.b8f9e58.master.macOS-10-15.x86_64/Emacs.app:
> bundle format unrecognized, invalid, or unsuitable
>     In subcomponent:
> /Users/runner/work/emacs-builds/emacs-builds/builds/Emacs.2021-06-26.b8f9e58.master.macOS-10-15.x86_64/Emacs.app/Contents/MacOS/lib/emacs/28.0.50

Bummer.

I had three options:

  * Contents/MacOS/lib
  * Contents/Resources/<something>
  * Contents/lib

and a close reading of the Apple documentation left me none-the-wiser
as to which option I should use. Executable binaries go under MacOS,
but these aren't executables. Framework libraries go somewhere else
entirely, but these aren't framework libraries. Resources is intended
for images and things, not libs. Lib is entirely non-standard.

I really don't know where the best place is. I'm still thinking
Resources is definitely not the right place, but none of the other
existing places make any sense, so perhaps the non-standard
/Contents/lib is the best option...

Can you try that? In order to sort it edit configure.ac, search for
the first occurrence of "ns_applibdir" and change the path.

If that fails then I guess we move them back under Resources. Unless
you have any better ideas.
-- 
Alan Third

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Jim Myhrberg]

I've just tried "Contents/lib", and it allowed me to sign and notarize
the .app bundle. And combined with your patch from bug#49270, all
bundled *.eln files are also correctly located and loaded :)

I did a bit of searching myself for alternatives and found
"Contents/PlugIns" as a potentially suitable place, but a quick test
revealed codesign fails the same way with it as it does with
Contents/MacOS.

Personally I think Contents/lib is probably fine, as both codesign and
Apple's notarization process are happy with it. And the notarization
process seems very picky. For example, when *.eln files were in
Resources/native-lisp, my initial notarization attempts failed because
it considered the *.eln files to be binaries, and they had not been
signed by codesign despite the --deep flag being used. Hence I'm
individually signing all the *.eln files before signing the app bundle
itself to get the app through notarization.

Also, I should probably mention that I generally know little about
macOS app development, and have mostly just tested my way forward with
figuring out how to sign and notarize Emacs :)

On Tue, Jun 29, 2021 at 8:19 PM Alan Third <alan@idiocy.org> wrote:
>
> On Tue, Jun 29, 2021 at 12:58:44PM +0100, Jim Myhrberg wrote:
> > Commit 5dd2d50 seems to have moved the native-lisp folder within self-contained
> > Emacs.app bundles:
> >
> > - from: "Contents/Resources/native-lisp"
> > - to:   "Contents/MacOS/lib/emacs/28.0.50/native-lisp"
> >
> > Unfortunately, Apple's codesign utility blows up with an error if there is any
> > folder within "Contents/MacOS" (recursively) which contains two dots within its
> > name. Here is an example of what the error looks like:
> >
> >     /Users/runner/work/emacs-builds/emacs-builds/builds/Emacs.2021-06-26.b8f9e58.master.macOS-10-15.x86_64/Emacs.app:
> > bundle format unrecognized, invalid, or unsuitable
> >     In subcomponent:
> > /Users/runner/work/emacs-builds/emacs-builds/builds/Emacs.2021-06-26.b8f9e58.master.macOS-10-15.x86_64/Emacs.app/Contents/MacOS/lib/emacs/28.0.50
>
> Bummer.
>
> I had three options:
>
>   * Contents/MacOS/lib
>   * Contents/Resources/<something>
>   * Contents/lib
>
> and a close reading of the Apple documentation left me none-the-wiser
> as to which option I should use. Executable binaries go under MacOS,
> but these aren't executables. Framework libraries go somewhere else
> entirely, but these aren't framework libraries. Resources is intended
> for images and things, not libs. Lib is entirely non-standard.
>
> I really don't know where the best place is. I'm still thinking
> Resources is definitely not the right place, but none of the other
> existing places make any sense, so perhaps the non-standard
> /Contents/lib is the best option...
>
> Can you try that? In order to sort it edit configure.ac, search for
> the first occurrence of "ns_applibdir" and change the path.
>
> If that fails then I guess we move them back under Resources. Unless
> you have any better ideas.
> --
> Alan Third

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Eli Zaretskii]

> From: Jim Myhrberg <contact@jimeh.me>
> Date: Wed, 30 Jun 2021 11:04:43 +0100
> 
> I've just tried "Contents/lib", and it allowed me to sign and notarize
> the .app bundle. And combined with your patch from bug#49270, all
> bundled *.eln files are also correctly located and loaded :)
> 
> I did a bit of searching myself for alternatives and found
> "Contents/PlugIns" as a potentially suitable place, but a quick test
> revealed codesign fails the same way with it as it does with
> Contents/MacOS.
> 
> Personally I think Contents/lib is probably fine, as both codesign and
> Apple's notarization process are happy with it. And the notarization
> process seems very picky. For example, when *.eln files were in
> Resources/native-lisp, my initial notarization attempts failed because
> it considered the *.eln files to be binaries, and they had not been
> signed by codesign despite the --deep flag being used. Hence I'm
> individually signing all the *.eln files before signing the app bundle
> itself to get the app through notarization.

The *.eln files are shared libraries.  What is the canonical place to
install shared libraries specific to an application?

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Jim Myhrberg]

On Wed, Jun 30, 2021 at 1:20 PM Eli Zaretskii <eliz@gnu.org> wrote:

> The *.eln files are shared libraries.  What is the canonical place to
> install shared libraries specific to an application?

This page indicates that "Contents/Frameworks" is the place for shared
libraries: https://wiki.lazarus.freepascal.org/Application_Bundle

I just poked around the internals of a bunch of Apple's apps and a few
third party apps, and it looks like "Contents/Frameworks" is where
they're storing *.dylib files among other things. So it might be the
sensible place for shared libraries. However, the same "two dots in
folder name" issue applies with codesign there too, so to use it we'd
need to drop the emacs version (28.0.50) from all folder paths.

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

On Wed, Jun 30, 2021 at 01:39:10PM +0100, Jim Myhrberg wrote:
> On Wed, Jun 30, 2021 at 1:20 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > The *.eln files are shared libraries.  What is the canonical place to
> > install shared libraries specific to an application?
> 
> This page indicates that "Contents/Frameworks" is the place for shared
> libraries: https://wiki.lazarus.freepascal.org/Application_Bundle
> 
> I just poked around the internals of a bunch of Apple's apps and a few
> third party apps, and it looks like "Contents/Frameworks" is where
> they're storing *.dylib files among other things. So it might be the
> sensible place for shared libraries. However, the same "two dots in
> folder name" issue applies with codesign there too, so to use it we'd
> need to drop the emacs version (28.0.50) from all folder paths.

I just found this page which explains the deal with the dots in
directory names:

https://developer.apple.com/library/archive/technotes/tn2206/_index.html

search for "period".

It also explains why you need to put code in certain directories. I
don't know why that page has been so hard to find.

Anyway, I guess we do need to change the install location so it
doesn't have the dots. It shouldn't really matter for the application
bundle since you can't have multiple versions of Emacs installed in
one bundle (without some hardcore messing around like the
emacsformacosx.com builds).
-- 
Alan Third

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Jim Myhrberg]

On Wed, Jun 30, 2021 at 1:53 PM Alan Third <alan@idiocy.org> wrote:
> I just found this page which explains the deal with the dots in
> directory names:
>
> https://developer.apple.com/library/archive/technotes/tn2206/_index.html
>
> search for "period".
>
> It also explains why you need to put code in certain directories. I
> don't know why that page has been so hard to find.

Damn, yeah it makes sense now :)

> Anyway, I guess we do need to change the install location so it
> doesn't have the dots. It shouldn't really matter for the application
> bundle since you can't have multiple versions of Emacs installed in
> one bundle (without some hardcore messing around like the
> emacsformacosx.com builds).

For multi-arch builds I think special care will need to be taken
anyway. The emacsformacosx.com builds for example have duplicate
shared libraries for each supported architecture. Personally though
I'm planning to just create separate builds/dmgs for each when I'm
able to support anything other than x86_64. The main self-contained
Emacs.app my builds are producing at the moment are already 500MB
thanks to full native AoT and essentially copying most of GCC into the
bundle for libgccjit to work.

Also, I'd obviously be more than happy to help with testing any
further changes to eln file locations and related things :)

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Wed, Jun 30, 2021 at 02:10:53PM +0100, Jim Myhrberg wrote:
> 
> Also, I'd obviously be more than happy to help with testing any
> further changes to eln file locations and related things :)

Please try the attached patch.

Eli, is this an acceptable way of modifying a lisp string, or is there
a better way?

I considered just removing the version since it doesn't matter much
for the app bundle, but then I realised that would also remove it in
the eln cache in .emacs.d, which isn't ideal, so I ended up replacing
the dots with underscores.
-- 
Alan Third

[-- Attachment #2: 0001-Fix-NS-self-contained-eln-location-bug-49271.patch --]
[-- Type: text/plain, Size: 2746 bytes --]

From 2d408e5db6f4d111bdf687eea939b0897dad58cd Mon Sep 17 00:00:00 2001
From: Alan Third <alan@idiocy.org>
Date: Wed, 30 Jun 2021 19:58:13 +0100
Subject: [PATCH] Fix NS self contained eln location (bug#49271)

* Makefile.in:
* configure.ac: Change eln file install location to
Contents/Frameworks.
* src/comp.c (hash_native_abi): Replace dots with underscores in the
eln install location as the macOS code-signing tool won't sign the
files if the parent directories have dots.
---
 Makefile.in  |  2 +-
 configure.ac |  4 ++--
 src/comp.c   | 20 +++++++++++++++++++-
 3 files changed, 22 insertions(+), 4 deletions(-)

diff --git a/Makefile.in b/Makefile.in
index 8fccdf7580..8c14c5cc7d 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -333,7 +333,7 @@ BIN_DESTDIR=
 ELN_DESTDIR = $(DESTDIR)${libdir}/emacs/${version}/
 else
 BIN_DESTDIR='${ns_appbindir}/'
-ELN_DESTDIR = ${ns_applibdir}/emacs/${version}/
+ELN_DESTDIR = ${ns_applibdir}/
 endif
 
 all: ${SUBDIR} info
diff --git a/configure.ac b/configure.ac
index 6e2cda947a..c924634d5b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1895,7 +1895,7 @@ AC_DEFUN
      ns_appdir=`pwd`/nextstep/Emacs.app
      ns_appbindir=${ns_appdir}/Contents/MacOS
      ns_applibexecdir=${ns_appdir}/Contents/MacOS/libexec
-     ns_applibdir=${ns_appdir}/Contents/MacOS/lib
+     ns_applibdir=${ns_appdir}/Contents/Frameworks
      ns_appresdir=${ns_appdir}/Contents/Resources
      ns_appsrc=Cocoa/Emacs.base
      ns_fontfile=macfont.o
@@ -1954,7 +1954,7 @@ AC_DEFUN
      ns_appdir=`pwd`/nextstep/Emacs.app
      ns_appbindir=${ns_appdir}
      ns_applibexecdir=${ns_appdir}/libexec
-     ns_applibdir=${ns_appdir}/lib
+     ns_applibdir=${ns_appdir}/Frameworks
      ns_appresdir=${ns_appdir}/Resources
      ns_appsrc=GNUstep/Emacs.base
      ns_fontfile=nsfont.o
diff --git a/src/comp.c b/src/comp.c
index ea05952627..ac77ae704d 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -744,8 +744,26 @@ hash_native_abi (void)
 			Vsystem_configuration_options),
 	       Fmapconcat (intern_c_string ("comp--subr-signature"),
 			   Vcomp_subr_list, build_string (""))));
+
+  Lisp_Object version = Vemacs_version;
+
+#ifdef NS_SELF_CONTAINED
+  /* MacOS self contained app bundles do not like having dots in the
+     directory names under the Contents/Frameworks directory, so
+     convert them to underscores.  */
+
+  char *str = xstrdup (SSDATA(version));
+
+  for (ptrdiff_t c = 0 ; c < SBYTES (version) ; c++)
+    if (*(str + c) == '.')
+      *(str + c) = '_';
+
+  version = build_string (str);
+  xfree (str);
+#endif
+
   Vcomp_native_version_dir =
-    concat3 (Vemacs_version, build_string ("-"), Vcomp_abi_hash);
+    concat3 (version, build_string ("-"), Vcomp_abi_hash);
 }
 
 static void
-- 
2.29.2

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Eli Zaretskii]

> Date: Wed, 30 Jun 2021 20:05:50 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, 49271@debbugs.gnu.org
> 
> Eli, is this an acceptable way of modifying a lisp string, or is there
> a better way?
> [...]
> +#ifdef NS_SELF_CONTAINED
> +  /* MacOS self contained app bundles do not like having dots in the
> +     directory names under the Contents/Frameworks directory, so
> +     convert them to underscores.  */
> +
> +  char *str = xstrdup (SSDATA(version));
> +
> +  for (ptrdiff_t c = 0 ; c < SBYTES (version) ; c++)
> +    if (*(str + c) == '.')
> +      *(str + c) = '_';
> +
> +  version = build_string (str);
> +  xfree (str);
> +#endif

It should work, although I'd prefer writing such code the other way
around: first create an uninitialized Lisp string (with
make_uninit_multibyte_string or make_uninit_string), then copy the
bytes while making the conversions.  The reason for this preference is
that you could then make sure the produced string has the same
multibyte-ness as the original, whereas the way you did it relies on
whatever build_string decides, which is not necessarily the same.

Thanks.

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

On Thu, Jul 01, 2021 at 10:13:04AM +0300, Eli Zaretskii wrote:
> It should work, although I'd prefer writing such code the other way
> around: first create an uninitialized Lisp string (with
> make_uninit_multibyte_string or make_uninit_string), then copy the
> bytes while making the conversions.  The reason for this preference is
> that you could then make sure the produced string has the same
> multibyte-ness as the original, whereas the way you did it relies on
> whatever build_string decides, which is not necessarily the same.

Thanks. I think the attached is doing what you suggested.
-- 
Alan Third

[-- Attachment #2: v2-0001-Fix-NS-self-contained-eln-location-bug-49271.patch --]
[-- Type: text/plain, Size: 3155 bytes --]

From 8de89ce0d2ae7f84ab2df025ae405b15994dfdb6 Mon Sep 17 00:00:00 2001
From: Alan Third <alan@idiocy.org>
Date: Wed, 30 Jun 2021 19:58:13 +0100
Subject: [PATCH v2] Fix NS self contained eln location (bug#49271)

* Makefile.in:
* configure.ac: Change eln file install location to
Contents/Frameworks.
* src/comp.c (hash_native_abi): Replace dots with underscores in the
eln install location as the macOS code-signing tool won't sign the
files if the parent directories have dots.
---
 Makefile.in  |  2 +-
 configure.ac |  4 ++--
 src/comp.c   | 35 ++++++++++++++++++++++++++++++++++-
 3 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/Makefile.in b/Makefile.in
index 8fccdf7580..8c14c5cc7d 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -333,7 +333,7 @@ BIN_DESTDIR=
 ELN_DESTDIR = $(DESTDIR)${libdir}/emacs/${version}/
 else
 BIN_DESTDIR='${ns_appbindir}/'
-ELN_DESTDIR = ${ns_applibdir}/emacs/${version}/
+ELN_DESTDIR = ${ns_applibdir}/
 endif
 
 all: ${SUBDIR} info
diff --git a/configure.ac b/configure.ac
index 6e2cda947a..c924634d5b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1895,7 +1895,7 @@ AC_DEFUN
      ns_appdir=`pwd`/nextstep/Emacs.app
      ns_appbindir=${ns_appdir}/Contents/MacOS
      ns_applibexecdir=${ns_appdir}/Contents/MacOS/libexec
-     ns_applibdir=${ns_appdir}/Contents/MacOS/lib
+     ns_applibdir=${ns_appdir}/Contents/Frameworks
      ns_appresdir=${ns_appdir}/Contents/Resources
      ns_appsrc=Cocoa/Emacs.base
      ns_fontfile=macfont.o
@@ -1954,7 +1954,7 @@ AC_DEFUN
      ns_appdir=`pwd`/nextstep/Emacs.app
      ns_appbindir=${ns_appdir}
      ns_applibexecdir=${ns_appdir}/libexec
-     ns_applibdir=${ns_appdir}/lib
+     ns_applibdir=${ns_appdir}/Frameworks
      ns_appresdir=${ns_appdir}/Resources
      ns_appsrc=GNUstep/Emacs.base
      ns_fontfile=nsfont.o
diff --git a/src/comp.c b/src/comp.c
index ea05952627..c45f9e3a7f 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -744,8 +744,41 @@ hash_native_abi (void)
 			Vsystem_configuration_options),
 	       Fmapconcat (intern_c_string ("comp--subr-signature"),
 			   Vcomp_subr_list, build_string (""))));
+
+  Lisp_Object version = Vemacs_version;
+
+#ifdef NS_SELF_CONTAINED
+  /* MacOS self contained app bundles do not like having dots in the
+     directory names under the Contents/Frameworks directory, so
+     convert them to underscores.  */
+  unsigned char *to;
+  const unsigned char *from = SDATA (Vemacs_version);
+
+  int multibyte = STRING_MULTIBYTE (Vemacs_version);
+
+  version = multibyte
+    ? make_uninit_multibyte_string (SCHARS (Vemacs_version),
+				    SBYTES (Vemacs_version))
+    : make_uninit_string (SBYTES (Vemacs_version));
+
+  to = SDATA (version);
+
+  while (from < SDATA (Vemacs_version) + SBYTES (Vemacs_version))
+    {
+      int c = multibyte ? string_char_advance (&from) : *from++;
+
+      if (c == '.')
+	c = '_';
+
+      if (multibyte)
+	to += CHAR_STRING (c, to);
+      else
+	*to++ = c;
+    }
+#endif
+
   Vcomp_native_version_dir =
-    concat3 (Vemacs_version, build_string ("-"), Vcomp_abi_hash);
+    concat3 (version, build_string ("-"), Vcomp_abi_hash);
 }
 
 static void
-- 
2.29.2

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Eli Zaretskii]

> Date: Thu, 1 Jul 2021 19:45:46 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: contact@jimeh.me, 49271@debbugs.gnu.org
> 
> > It should work, although I'd prefer writing such code the other way
> > around: first create an uninitialized Lisp string (with
> > make_uninit_multibyte_string or make_uninit_string), then copy the
> > bytes while making the conversions.  The reason for this preference is
> > that you could then make sure the produced string has the same
> > multibyte-ness as the original, whereas the way you did it relies on
> > whatever build_string decides, which is not necessarily the same.
> 
> Thanks. I think the attached is doing what you suggested.

Yes.  But I think you can simplify (and make it a tad faster) if you
just copy bytes in either case.  Since a multibyte string is
represented by a superset of UTF-8, you are guaranteed that no
multibyte sequence will ever include '.'.  So converting from
multibyte sequences to characters and back, as well as calls to
string_char_advance, can be avoided.

Thanks.

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

On Thu, Jul 01, 2021 at 10:03:09PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 1 Jul 2021 19:45:46 +0100
> > From: Alan Third <alan@idiocy.org>
> > Cc: contact@jimeh.me, 49271@debbugs.gnu.org
> > 
> > > It should work, although I'd prefer writing such code the other way
> > > around: first create an uninitialized Lisp string (with
> > > make_uninit_multibyte_string or make_uninit_string), then copy the
> > > bytes while making the conversions.  The reason for this preference is
> > > that you could then make sure the produced string has the same
> > > multibyte-ness as the original, whereas the way you did it relies on
> > > whatever build_string decides, which is not necessarily the same.
> > 
> > Thanks. I think the attached is doing what you suggested.
> 
> Yes.  But I think you can simplify (and make it a tad faster) if you
> just copy bytes in either case.  Since a multibyte string is
> represented by a superset of UTF-8, you are guaranteed that no
> multibyte sequence will ever include '.'.  So converting from
> multibyte sequences to characters and back, as well as calls to
> string_char_advance, can be avoided.

I'm glad I asked about this because I had completely misunderstood how
multibyte strings are represented. This makes a lot more sense.

Thanks!
-- 
Alan Third

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Jim Myhrberg]

> On 30 Jun 2021, at 20:05, Alan Third <alan@idiocy.org> wrote:
> 
> Please try the attached patch.

I've successfully built, signed and notarized a full AoT build with this patch. I also applied the patch from bug#49270 to ensure *.eln files are located correctly within the bundle.

Everything is working as expected, and in fact, codesign even automatically finds and signs all *.eln files within Contents/Frameworks when signing the app bundle itself, so I no longer need to individually sign each *.eln file before signing the whole app bundle :)

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

On Thu, Jul 01, 2021 at 03:53:13PM +0100, Jim Myhrberg wrote:
> I've successfully built, signed and notarized a full AoT build with
> this patch. I also applied the patch from bug#49270 to ensure *.eln
> files are located correctly within the bundle.
> 
> Everything is working as expected, and in fact, codesign even
> automatically finds and signs all *.eln files within
> Contents/Frameworks when signing the app bundle itself, so I no
> longer need to individually sign each *.eln file before signing the
> whole app bundle :)

I'm curious what you're actually doing. We have some code that runs on
ARM based Macs only that does this

    codesign -s - -f $@.tmp

where I think $@.tmp is... something... executable?

I'm wondering if we should just install everything into the app bundle
and sign the whole damn thing.

I don't really understand this whole codesigning thing. It seems to
make no difference here, but I'm not trying to run the app on another
Mac or on an ARM Mac, so I think it's normal that I shouldn't see any
difference.

-- 
Alan Third

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Jim Myhrberg]

> On 1 Jul 2021, at 21:13, Alan Third <alan@idiocy.org> wrote:
> 
> I'm curious what you're actually doing. We have some code that runs on
> ARM based Macs only that does this
> 
>    codesign -s - -f $@.tmp
> 
> where I think $@.tmp is... something... executable?
> 
> I'm wondering if we should just install everything into the app bundle
> and sign the whole damn thing.
> 
> I don't really understand this whole codesigning thing. It seems to
> make no difference here, but I'm not trying to run the app on another
> Mac or on an ARM Mac, so I think it's normal that I shouldn't see any
> difference.

I can't say I know what the `$@.tmp` thing is either, or really understand
codesign for that matter. I've managed to make it work through trial and
error... lol

To explain a bit more of what I'm doing though; I basically got bored of making
my laptop sound like a jet engine for 25 minutes every other day as I made new
builds from the master branch, so I set out to make nightly builds using GitHub
Actions which I could just download and/or install/upgrade via brew cask:
https://github.com/jimeh/emacs-builds

With the heavy lifting done by my custom build script:
https://github.com/jimeh/build-emacs-for-macos

The signing and notarizing stuff also lives in the build script repo, but as
part of a new (and somewhat hacky) "emacs-builder" CLI tool written in Go, which
is designed to automated a bunch of the steps.

Because of this, you can see the full build process I'm doing on GitHub Actions:
https://github.com/jimeh/emacs-builds/runs/2957583013?check_suite_focus=true

As for the *.eln file, previously when they were in Contents/Resources I had to
sign each one individually before signing the bundle as a whole. But I didn't
check if that was still needed after they were moved to Contents/MacOS a few
days ago.

But I have checked it with them under Contents/Frameworks, and signing the *.eln
files is no longer needed as codesign finds them when signing the .app bundle
itself..

As for exactly what I'm doing, this is essentially the main command which signs
the .app bundle:

    codesign --sign <cert-id> --deep --timestamp --force --verbose \
             --options runtime --entitlements <path-to-entitlements.plist> \
             /path/to/Emacs.app

And the entitlements I use are:

- com.apple.security.automation.apple-events
- com.apple.security.cs.allow-jit
- com.apple.security.cs.disable-library-validation
- com.apple.security.network.client

Which seem to be enough for everything I do with Emacs to work correctly.

I do also sign a custom little "emacs" shell-script in Contents/MacOS/bin, which
I add to make it easy to expose a "emacs" CLI tool, all it does is correctly
resolve the path to the .app bundle, and execute the main MacOS/Emacs binary
from its absolute real path so it can find everything it depends on within the
app bundle even when symlinked to somewhere else.

I'm happy to answer any questions you might have :)

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

On Thu, Jul 01, 2021 at 03:53:13PM +0100, Jim Myhrberg wrote:
> > On 30 Jun 2021, at 20:05, Alan Third <alan@idiocy.org> wrote:
> > 
> > Please try the attached patch.
> 
> I've successfully built, signed and notarized a full AoT build with
> this patch. I also applied the patch from bug#49270 to ensure *.eln
> files are located correctly within the bundle.
> 
> Everything is working as expected, and in fact, codesign even
> automatically finds and signs all *.eln files within
> Contents/Frameworks when signing the app bundle itself, so I no
> longer need to individually sign each *.eln file before signing the
> whole app bundle :)

I've pushed an updated version of this patch to master, so I'll close
this bug report.

(I'm going to reply to your other email, but I'll redirect it to
emacs-devel.)
-- 
Alan Third

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Alan Third]

On Wed, Jun 30, 2021 at 03:20:41PM +0300, Eli Zaretskii wrote:
> The *.eln files are shared libraries.  What is the canonical place to
> install shared libraries specific to an application?

AFAICT there isn't one. Apple don't expect you to use any, or at least
not to copy them into your application bundle.

The Objective C equivalent are Frameworks, and there's a place for
them, but they have a specific file structure and I don't think we
want to try and emulate that for standard shared libraries.

There also doesn't appear to be any consensus in the community as I've
seen several places suggested.

I'm inclined to towards Contents/lib. It works for Jim and I think it
should be fairly clear to outside observers what's going on.

Although I've just done another search and found multiple people
suggesting Contents/Frameworks is the correct place to put them.

Jim, does putting them in Frameworks work or does that have signing
implications too?

(None of this seems to matter at all on my machine, so I can't test it
myself.)
-- 
Alan Third

bug#49271: 28.0.50: native-comp: Signing macOS self-contained .app bundle fails due to new *.eln location

[Jim Myhrberg]

On Wed, Jun 30, 2021 at 1:42 PM Alan Third <alan@idiocy.org> wrote:

> Jim, does putting them in Frameworks work or does that have signing
> implications too?

You must not have seen my email from a couple of minutes ago, I'm
afraid I tested Frameworks and it has the same folder naming issues.
From the apps I poked around in, they've just put all their *.dylib
files directly in the root of the Frameworks directory.

> (None of this seems to matter at all on my machine, so I can't test it
> myself.)

What version of macOS/Xcode do you have? Case I've seen slight
differences in behavior with codesign on Catalina and Big Sur at least
(unrelated to the two dots issue though). I've written a small CLI
tool to make life easier for myself when it comes to signing, dmg
packaging, and notarizing Emacs. I'm happy to point you towards it if
you want.