bug#41994: Emacs on Apple ARM devices

bug#41994: Emacs on Apple ARM devices

[Roland Kaufmann]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

Code generation on Apple's ARM devices is mostly similar to MacOS, 
except that they expect a code signature section, which to avoid 
wasting too many bytes, are not aligned on page size as with code, 
but rather to nearest 16 bytes. In addition, the ARM compiler will 
also emit some relative pointer relocation records, which it seems 
can be safely ignored as sections aren't rearranged.

This changeset implements those changes on the Mach-O dumper so 
that Emacs can be compiled for ARM devices. With the ongoing 
convergence between the MacOS and iOS platform, and the increasing 
importance of code-signing in later MacOS versions, I think it 
could be nice to mainline these changes.

Since these sections do not occur in current MacOS builds - 
otherwise they would generate errors for being unknown - the 
changes should not impact existing places where Emacs build, and 
it would not work on iOS without to begin with.

Hat tip to Aaron Griffith for sharing his knowledge.


-- 
Sincerely,

Roland Kaufmann







[-- Attachment #2: 0001-Let-us-know-unknown-number-being-complained-about.patch --]
[-- Type: text/x-patch, Size: 721 bytes --]

>From 82a5aed44299db11cee3f97731a906e280cfb114 Mon Sep 17 00:00:00 2001
From: Roland Kaufmann <rlndkfmn+emacs@gmail.com>
Date: Sat, 13 Jun 2020 01:36:52 +0200
Subject: [PATCH 1/6] Let us know unknown number being complained about

This will let us discover new sections emitted by the compiler so we can
add support for them.
---
 src/unexmacosx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/unexmacosx.c b/src/unexmacosx.c
index 59cbe3c..844fc87 100644
--- a/src/unexmacosx.c
+++ b/src/unexmacosx.c
@@ -628,7 +628,7 @@ print_load_command_name (int lc)
       break;
 #endif
     default:
-      printf ("unknown          ");
+      printf ("unknown (0x%x)   ", lc);
     }
 }
 
-- 
2.25.2




[-- Attachment #3: 0002-Make-function-generally-available-for-sections.patch --]
[-- Type: text/x-patch, Size: 983 bytes --]

>From e27fe51106a00f6c10e3ac2ab51cb4d12e028491 Mon Sep 17 00:00:00 2001
From: Roland Kaufmann <rlndkfmn+emacs@gmail.com>
Date: Sat, 13 Jun 2020 02:07:57 +0200
Subject: [PATCH 2/6] Make function generally available for sections

---
 src/unexmacosx.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/unexmacosx.c b/src/unexmacosx.c
index 844fc87..d274ab7 100644
--- a/src/unexmacosx.c
+++ b/src/unexmacosx.c
@@ -1121,7 +1121,6 @@ copy_dyld_info (struct load_command *lc, long delta)
 }
 #endif
 
-#ifdef LC_FUNCTION_STARTS
 /* Copy a LC_FUNCTION_STARTS/LC_DATA_IN_CODE/LC_DYLIB_CODE_SIGN_DRS
    load command from the input file to the output file, adjusting the
    data offset field.  */
@@ -1142,7 +1141,6 @@ copy_linkedit_data (struct load_command *lc, long delta)
 
   curr_header_offset += lc->cmdsize;
 }
-#endif
 
 /* Copy other kinds of load commands from the input file to the output
    file, ones that do not require adjustments of file offsets.  */
-- 
2.25.2




[-- Attachment #4: 0003-Allow-padding-of-sections-to-other-than-page-size.patch --]
[-- Type: text/x-patch, Size: 1780 bytes --]

>From 5b9c1a0ebab3825dce85bc81139c36787923eec0 Mon Sep 17 00:00:00 2001
From: Roland Kaufmann <rlndkfmn+emacs@gmail.com>
Date: Thu, 27 Dec 2018 15:57:42 +0100
Subject: [PATCH 3/6] Allow padding of sections to other than page size

---
 src/unexmacosx.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/src/unexmacosx.c b/src/unexmacosx.c
index d274ab7..561a509 100644
--- a/src/unexmacosx.c
+++ b/src/unexmacosx.c
@@ -191,7 +191,8 @@ static unsigned long curr_header_offset = sizeof (struct mach_header);
 static unsigned long curr_file_offset = 0;
 
 static unsigned long pagesize;
-#define ROUNDUP_TO_PAGE_BOUNDARY(x)	(((x) + pagesize - 1) & ~(pagesize - 1))
+#define ROUNDUP_TO_BOUNDARY(x,size)	(((x) + size - 1) & ~(size - 1))
+#define ROUNDUP_TO_PAGE_BOUNDARY(x)	ROUNDUP_TO_BOUNDARY(x,pagesize)
 
 static int infd, outfd;
 
@@ -1125,13 +1126,16 @@ copy_dyld_info (struct load_command *lc, long delta)
    load command from the input file to the output file, adjusting the
    data offset field.  */
 static void
-copy_linkedit_data (struct load_command *lc, long delta)
+copy_linkedit_data (struct load_command *lc, long delta, long alignment)
 {
   struct linkedit_data_command *ldp = (struct linkedit_data_command *) lc;
 
   if (ldp->dataoff > 0)
     ldp->dataoff += delta;
 
+  /* add bytes necessary to be at given alignment */
+  ldp->dataoff = ROUNDUP_TO_BOUNDARY (ldp->dataoff, alignment);
+
   printf ("Writing ");
   print_load_command_name (lc->cmd);
   printf (" command\n");
@@ -1222,7 +1226,7 @@ dump_it (void)
 #ifdef LC_DYLIB_CODE_SIGN_DRS
       case LC_DYLIB_CODE_SIGN_DRS:
 #endif
-	copy_linkedit_data (lca[i], linkedit_delta);
+	copy_linkedit_data (lca[i], linkedit_delta, 1);
 	break;
 #endif
       default:
-- 
2.25.2




[-- Attachment #5: 0004-Output-code-signature-sections-align-on-16-bytes.patch --]
[-- Type: text/x-patch, Size: 1210 bytes --]

>From a4718afd4f5a9174e7f4f10b212bfebca6661cbe Mon Sep 17 00:00:00 2001
From: Roland Kaufmann <rlndkfmn+emacs@gmail.com>
Date: Thu, 27 Dec 2018 16:13:56 +0100
Subject: [PATCH 4/6] Output code signature sections; align on 16 bytes

---
 src/unexmacosx.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/unexmacosx.c b/src/unexmacosx.c
index 561a509..653ecc0 100644
--- a/src/unexmacosx.c
+++ b/src/unexmacosx.c
@@ -628,6 +628,9 @@ print_load_command_name (int lc)
       printf ("LC_DYLIB_CODE_SIGN_DRS");
       break;
 #endif
+    case LC_CODE_SIGNATURE:
+      printf ("LC_CODE_SIGNATURE");
+      break;
     default:
       printf ("unknown (0x%x)   ", lc);
     }
@@ -1229,6 +1232,13 @@ dump_it (void)
 	copy_linkedit_data (lca[i], linkedit_delta, 1);
 	break;
 #endif
+      case LC_CODE_SIGNATURE:
+	/* as is quasi-documented in Technical Note TN2206, the code */
+	/* signature section is not aligned on page boundary, but    */
+	/* rather on a 16 bytes boundary (it is not large enough to  */
+	/* waste most of a page).                                    */
+	copy_linkedit_data (lca[i], linkedit_delta, 16);
+	break;
       default:
 	copy_other (lca[i]);
 	break;
-- 
2.25.2




[-- Attachment #6: 0005-Make-relocation-of-relative-pointers-a-no-op.patch --]
[-- Type: text/x-patch, Size: 930 bytes --]

>From a02638d9459d53f909593d4c6c9e17d4f1548c64 Mon Sep 17 00:00:00 2001
From: Roland Kaufmann <rlndkfmn+emacs@gmail.com>
Date: Thu, 27 Dec 2018 16:14:38 +0100
Subject: [PATCH 5/6] Make relocation of relative pointers a no-op

---
 src/unexmacosx.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/unexmacosx.c b/src/unexmacosx.c
index 653ecc0..286749c 100644
--- a/src/unexmacosx.c
+++ b/src/unexmacosx.c
@@ -1024,6 +1024,11 @@ unrelocate (const char *name, off_t reloff, int nrel, vm_address_t base)
 		unreloc_count++;
 	      }
 	    break;
+	  case GENERIC_RELOC_LOCAL_SECTDIFF:
+	    /* nothing to do for relative pointer, as section order */
+	    /* is fixed as we dump the image; we won't need to move */
+	    /* them around later.                                   */
+	    break;
 	  default:
 	    unexec_error ("unrelocate: %s:%d cannot handle type = %d",
 			  name, i, reloc_info.r_type);
-- 
2.25.2




[-- Attachment #7: 0006-Allow-build-configuration-on-Apple-ARM-devices.patch --]
[-- Type: text/x-patch, Size: 708 bytes --]

>From b1bd2b5cc076fcb1b4d2b2a790fb94bc860fb52b Mon Sep 17 00:00:00 2001
From: Roland Kaufmann <rlndkfmn+emacs@gmail.com>
Date: Thu, 27 Dec 2018 15:51:49 +0100
Subject: [PATCH 6/6] Allow build configuration on Apple ARM devices

---
 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index b1b8c84..0006d75 100644
--- a/configure.ac
+++ b/configure.ac
@@ -708,7 +708,7 @@ case "${canonical}" in
   *-apple-darwin* )
     case "${canonical}" in
       *-apple-darwin[0-9].*) unported=yes ;;
-      i[3456]86-* | x86_64-* )  ;;
+      i[3456]86-* | x86_64-* | arm-* )  ;;
       * )            unported=yes ;;
     esac
     opsys=darwin
-- 
2.25.2

bug#41994: Emacs on Apple ARM devices

[Stefan Kangas]

Roland Kaufmann <rlndkfmn+emacs@gmail.com> writes:

> Code generation on Apple's ARM devices is mostly similar to MacOS, except that
> they expect a code signature section, which to avoid wasting too many bytes, are
> not aligned on page size as with code, but rather to nearest 16 bytes. In
> addition, the ARM compiler will also emit some relative pointer relocation
> records, which it seems can be safely ignored as sections aren't rearranged.
>
> This changeset implements those changes on the Mach-O dumper so that Emacs can
> be compiled for ARM devices. With the ongoing convergence between the MacOS and
> iOS platform, and the increasing importance of code-signing in later MacOS
> versions, I think it could be nice to mainline these changes.
>
> Since these sections do not occur in current MacOS builds - otherwise they would
> generate errors for being unknown - the changes should not impact existing
> places where Emacs build, and it would not work on iOS without to begin with.
>
> Hat tip to Aaron Griffith for sharing his knowledge.

This series of patches was submitted 8 weeks ago, but never got a reply
at the time.

Could someone with more knowledge about this please help review it?

Best regards,
Stefan Kangas

bug#41994: Emacs on Apple ARM devices

[Robert Pluim]

>>>>> On Fri, 14 Aug 2020 02:17:18 -0700, Stefan Kangas <stefan@marxist.se> said:

    Stefan> Roland Kaufmann <rlndkfmn+emacs@gmail.com> writes:
    >> Code generation on Apple's ARM devices is mostly similar to MacOS, except that
    >> they expect a code signature section, which to avoid wasting too many bytes, are
    >> not aligned on page size as with code, but rather to nearest 16 bytes. In
    >> addition, the ARM compiler will also emit some relative pointer relocation
    >> records, which it seems can be safely ignored as sections aren't rearranged.
    >> 
    >> This changeset implements those changes on the Mach-O dumper so that Emacs can
    >> be compiled for ARM devices. With the ongoing convergence between the MacOS and
    >> iOS platform, and the increasing importance of code-signing in later MacOS
    >> versions, I think it could be nice to mainline these changes.
    >> 
    >> Since these sections do not occur in current MacOS builds - otherwise they would
    >> generate errors for being unknown - the changes should not impact existing
    >> places where Emacs build, and it would not work on iOS without to begin with.
    >> 
    >> Hat tip to Aaron Griffith for sharing his knowledge.

    Stefan> This series of patches was submitted 8 weeks ago, but never got a reply
    Stefan> at the time.

    Stefan> Could someone with more knowledge about this please help review it?

Are these patches even still needed now that emacs-27 has a portable
dumper?

Robert

bug#41994: Emacs on Apple ARM devices

[Alan Third]

On Fri, Aug 14, 2020 at 04:15:40PM +0200, Robert Pluim wrote:
> >>>>> On Fri, 14 Aug 2020 02:17:18 -0700, Stefan Kangas <stefan@marxist.se> said:
> 
>     Stefan> Roland Kaufmann <rlndkfmn+emacs@gmail.com> writes:
>     >> Code generation on Apple's ARM devices is mostly similar to MacOS, except that
>     >> they expect a code signature section, which to avoid wasting too many bytes, are
>     >> not aligned on page size as with code, but rather to nearest 16 bytes. In
>     >> addition, the ARM compiler will also emit some relative pointer relocation
>     >> records, which it seems can be safely ignored as sections aren't rearranged.
>     >> 
>     >> This changeset implements those changes on the Mach-O dumper so that Emacs can
>     >> be compiled for ARM devices. With the ongoing convergence between the MacOS and
>     >> iOS platform, and the increasing importance of code-signing in later MacOS
>     >> versions, I think it could be nice to mainline these changes.
>     >> 
>     >> Since these sections do not occur in current MacOS builds - otherwise they would
>     >> generate errors for being unknown - the changes should not impact existing
>     >> places where Emacs build, and it would not work on iOS without to begin with.
>     >> 
>     >> Hat tip to Aaron Griffith for sharing his knowledge.
> 
>     Stefan> This series of patches was submitted 8 weeks ago, but never got a reply
>     Stefan> at the time.
> 
>     Stefan> Could someone with more knowledge about this please help review it?
> 
> Are these patches even still needed now that emacs-27 has a portable
> dumper?

That's what I was wondering (although to be honest I never even
noticed there were patches attached).

I suspect the patch to configure.ac might be needed even for the
pdumpder?
-- 
Alan Third

bug#41994: Emacs on Apple ARM devices

[Roland Kaufmann]

RK> Code generation on Apple's ARM devices ... expect a code
RK> signature section ... This changeset implements those changes
RK> on the Mach-O dumper

RP> Are these ... still needed now that emacs-27 has a portable
RP> dumper?

This may be a case of incredible bad timing, as I haven't kept up 
on the development of the pdumper. Doing some light reading up on 
it now, it seems to me that it does the loading from a custom file 
format instead of writing a native executable.

Evidently, this renders any improvements on the unexec code moot, 
although it's still in the tree, I guess for platforms where the 
pdumper is not yet fully tested, so I understand the reluctance to 
spend any more scarce time on it.

AT> I suspect the patch to configure.ac might be needed even for 
AT> the pdumpder?

Yes, although one could discuss if the concept of ported/unported 
status still applies in the same way once there is the pdumper.


-- 
Yours sincerely,

Roland Kaufmann

bug#41994: Emacs on Apple ARM devices

[Alan Third]

On Sat, Aug 15, 2020 at 12:10:22AM +0200, Roland Kaufmann wrote:
> RK> Code generation on Apple's ARM devices ... expect a code
> RK> signature section ... This changeset implements those changes
> RK> on the Mach-O dumper
> 
> RP> Are these ... still needed now that emacs-27 has a portable
> RP> dumper?
> 
> This may be a case of incredible bad timing, as I haven't kept up on the
> development of the pdumper. Doing some light reading up on it now, it seems
> to me that it does the loading from a custom file format instead of writing
> a native executable.
> 
> Evidently, this renders any improvements on the unexec code moot, although
> it's still in the tree, I guess for platforms where the pdumper is not yet
> fully tested, so I understand the reluctance to spend any more scarce time
> on it.

Yes, unfortunately I think it's unlikely anyone will be wanting to run
unexec on macOS from now on.

It may be worth seeing if the gccemacs (native elisp compilation)
project will need something like this. I don't know enough about
either these requirements or gccemacs to say.

> AT> I suspect the patch to configure.ac might be needed even for
> AT> the pdumpder?
> 
> Yes, although one could discuss if the concept of ported/unported status
> still applies in the same way once there is the pdumper.

Well, I may as well commit it. Have you contributed to Emacs before
and if so have you signed the copyright paperwork? This is small
enough that it should be exempt, but I'd rather get the notation
correct. :)
-- 
Alan Third

bug#41994: Emacs on Apple ARM devices

[Roland Kaufmann]

AT> It may be worth seeing if the gccemacs (native elisp
AT> compilation) project will need something like this.

It would be nice if they could serve as some "notes of things to 
consider"; I guess it depends on whether gccemacs ever spill a 
Mach-O executable or if all compilation happens in memory.

AT> Well, I may as well commit it. Have you contributed to Emacs 
AT> before and if so have you signed the copyright paperwork?

No, but please send me the forms (or links) and I'll be glad to 
fill them, in that case.

-- 
Yours sincerely,

Roland Kaufmann

bug#41994: Emacs on Apple ARM devices

[Robert Pluim]

>>>>> On Fri, 14 Aug 2020 22:11:46 +0200 (CEST), Alan Third <alan@idiocy.org> said:
    Alan> That's what I was wondering (although to be honest I never even
    Alan> noticed there were patches attached).

    Alan> I suspect the patch to configure.ac might be needed even for the
    Alan> pdumpder?

Yes, I think so. If someone sends me an arm-based mac Iʼm willing to
test that hypothesis :-)

Seriously, itʼs a one-liner. I think we can commit it as obvious.

Robert

bug#41994: Emacs on Apple ARM devices

[Alan Third]

On Sat, Aug 15, 2020 at 01:45:18PM +0200, Robert Pluim wrote:
> >>>>> On Fri, 14 Aug 2020 22:11:46 +0200 (CEST), Alan Third <alan@idiocy.org> said:
>     Alan> That's what I was wondering (although to be honest I never even
>     Alan> noticed there were patches attached).
> 
>     Alan> I suspect the patch to configure.ac might be needed even for the
>     Alan> pdumpder?
> 
> Yes, I think so. If someone sends me an arm-based mac Iʼm willing to
> test that hypothesis :-)
> 
> Seriously, itʼs a one-liner. I think we can commit it as obvious.

I've pushed it with the copyright exempt tag.

4cba236749aafade7bd88cf2a10be48f44983faa

I don't think there's any point keeping this bug report open, so I'll
close it now.

Thanks.
-- 
Alan Third