Emacs on QNX

Emacs on QNX

[Elad Lahav]

[-- Attachment #1: Type: text/plain, Size: 2968 bytes --]

Hello all,

I built Emacs on QNX 7.0. For the most part, the process was straight-
forward and required only minimal changes to the source (see below for
a diff). However, I ran into a problem when trying to run Emacs. The
investigation pointed to the use of sbrk(), which QNX dropped many
years ago. There is still a stub that returns -1 and sets errno to
ENOMEM, but the Emacs code doesn't check for that, and interprets the
result as the address 0xffffffffffffffff. Hilarity ensues.

To overcome the problem, I wrote a naive implementation of sbrk() in a
dynamically-linked library (attached). It doesn't fully adhere to the
original semantics of sbrk(), but is enough to get Emacs running
properly.

Why is sbrk() still used? With 64-bit address spaces and ASLR it is
hard to imagine this API being useful. I am surprised that other
operating systems still support it. Is there a different route I can
take that doesn't require the emulated sbrk()?

Thanks,
--Elad


diff -u -r emacs-25.2.orig/configure.ac emacs-25.2/configure.ac
--- emacs-25.2.orig/configure.ac	2017-02-03 09:34:30.000000000 -0500
+++ emacs-25.2/configure.ac	2017-10-22 03:01:12.000000000 -0400
@@ -698,6 +698,14 @@
     esac
   ;;
 
+  ## QNX Neutrino 
+  *-nto-qnx* )
+    echo Found QNX
+    opsys=qnxnto
+    CFLAGS="$CFLAGS -D__NO_EXT_QNX"
+    LDFLAGS="$LDFLAGS -L/usr/lib -lsocket -lsbrk"
+  ;;
+
   ## Intel 386 machines where we don't care about the manufacturer.
   i[3456]86-*-* )
     case "${canonical}" in
@@ -4465,7 +4473,7 @@
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [])
     ;;
 
-  gnu | openbsd )
+  gnu | openbsd | qnxnto )
     AC_DEFINE(FIRST_PTY_LETTER, ['p'])
     ;;
 
diff -u -r emacs-25.2.orig/src/unexelf.c emacs-25.2/src/unexelf.c
--- emacs-25.2.orig/src/unexelf.c	2017-02-03 05:25:45.000000000 -0500
+++ emacs-25.2/src/unexelf.c	2017-10-22 03:11:10.000000000 -0400
@@ -58,7 +58,9 @@
 #include <sys/types.h>
 #include <unistd.h>
 
-#if !defined (__NetBSD__) && !defined (__OpenBSD__)
+#if defined(__QNX__)
+#include <sys/elf.h>
+#elif !defined (__NetBSD__) && !defined (__OpenBSD__)
 #include <elf.h>
 #endif /* not __NetBSD__ and not __OpenBSD__ */
 #include <sys/mman.h>

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: libsbrk.c --]
[-- Type: text/x-csrc; name="libsbrk.c", Size: 2191 bytes --]

/**
 * @file    libsbrk.c
 * @brief   Naive implementation of sbrk()
 *
 * QNX doesn't support sbrk(), and heap-allocated data can reside anywhere in
 * the virtual address space. However, Emacs depdends on sbrk() for its own
 * allocator implementation, as well as for dumping a pre-loaded image.
 * This implementation reserves a non-backed region in the virtual address
 * space. A call to sbrk() that moves the break point causes the sub range
 * between the old and new break points to be backed by virtual memory.
 */


#include <sys/mman.h>
#include <stdint.h>
#include <errno.h>

#define SEGMENT_SIZE    1024 * 1024 * 1024

static void         *base_addr = MAP_FAILED;
static uintptr_t    break_addr;
static uintptr_t    max_addr;

/**
 * Moves the break point of the emulated data segment.
 * @param   increment   Amount of memory to add to the data segment
 * @return  Previous break point, if successful, -1 on failure
 */
void *
sbrk(intptr_t increment)
{
    if (base_addr == MAP_FAILED) {
        // Allocate a non-backed region of the address space.
        base_addr = mmap(0, SEGMENT_SIZE, PROT_NONE, MAP_PRIVATE | MAP_ANON,
                         -1, 0);
        if (base_addr == MAP_FAILED) {
            errno = ENOMEM;
            return (void *)-1;
        }

        break_addr = (uintptr_t)base_addr;
        max_addr = (uintptr_t)base_addr + SEGMENT_SIZE;
    }

    // Round up to a page size.
    increment = (increment + __PAGESIZE - 1) & ~(__PAGESIZE - 1);

    // Check for overflow.
    uintptr_t   new_addr = break_addr + increment;
    if (new_addr >= max_addr) {
        errno = ENOMEM;
        return (void *)-1;
    }

    // Only support incrementing the break point (though it's not hard to use
    // mprotect(PROT_NONE) to handle decrementing as well).
    if (new_addr <= break_addr) {
        return (void *)break_addr;
    }

    // Back the new sub-range.
    if (mmap((void *)break_addr, increment, PROT_READ | PROT_WRITE,
             MAP_FIXED | MAP_PRIVATE | MAP_ANON, -1, 0) == MAP_FAILED) {
        errno = ENOMEM;
        return (void *)-1;
    }

    void    *result = (void *)break_addr;
    break_addr = new_addr;
    return result;
}

Re: Emacs on QNX

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

Thanks for looking into a QNX port.

Elad Lahav wrote:

> There is still a stub that returns -1 and sets errno to
> ENOMEM, but the Emacs code doesn't check for that, and interprets the
> result as the address 0xffffffffffffffff.

Which part of the Emacs code is that? Is this the call in src/unexelf.c? If so, 
I can fix that.

> To overcome the problem, I wrote a naive implementation of sbrk() in a
> dynamically-linked library

Although that's clever, I'm hoping that we can port to QNX the same way we port 
to Cygwin, as Cygwin already uses a fake sbrk. (Eventually Emacs has got to stop 
using sbrk, but we don't have an adequate solution to that bigger problem yet.)

Does the attached patch work for you instead?

> +    CFLAGS="$CFLAGS -D__NO_EXT_QNX"

Could you fill us on on what __NO_EXT_QNX does? The symbol I see mentioned in 
public discussions is _QNX_SOURCE. What is the difference between those two 
symbols, and why is __NO_EXT_QNX preferable here? What happens if you leave it out?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: qnx.diff --]
[-- Type: text/x-patch; name="qnx.diff", Size: 1714 bytes --]

diff --git a/configure.ac b/configure.ac
index 63324c2c7c..0e5ffb806e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -699,6 +699,12 @@ AC_DEFUN
     esac
   ;;
 
+  ## QNX Neutrino
+  *-nto-qnx* )
+    opsys=qnxnto
+    CFLAGS="$CFLAGS -D__NO_EXT_QNX"
+  ;;
+
   ## Intel 386 machines where we don't care about the manufacturer.
   i[3456]86-*-* )
     case "${canonical}" in
@@ -1507,6 +1513,8 @@ AC_DEFUN
 
   hpux*) LIBS_SYSTEM="-l:libdld.sl" ;;
 
+  qnxnto) LIBS_SYSTEM="-L/usr/lib -lsocket" ;;
+
   sol2*) LIBS_SYSTEM="-lsocket -lnsl" ;;
 
   ## Motif needs -lgen.
@@ -2210,7 +2218,8 @@ AC_DEFUN
 case "$opsys" in
   ## darwin ld insists on the use of malloc routines in the System framework.
   darwin | mingw32 | nacl | sol2-10) ;;
-  cygwin) hybrid_malloc=yes
+  cygwin | qnxnto)
+	  hybrid_malloc=yes
           system_malloc= ;;
   *) test "$ac_cv_func_sbrk" = yes && system_malloc=$emacs_cv_sanitize_address;;
 esac
@@ -4604,7 +4613,7 @@ AC_DEFUN
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [])
     ;;
 
-  gnu | openbsd )
+  gnu | openbsd | qnxnto )
     AC_DEFINE(FIRST_PTY_LETTER, ['p'])
     ;;
 
diff --git a/src/unexelf.c b/src/unexelf.c
index 1cdcfeb44e..200ddf4ae2 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -58,9 +58,12 @@ what you give them.   Help stamp out software-hoarding!  */
 #include <sys/types.h>
 #include <unistd.h>
 
-#if !defined (__NetBSD__) && !defined (__OpenBSD__)
+#ifdef __QNX__
+#include <sys/elf.h>
+#elif !defined __NetBSD__ && !defined __OpenBSD__
 #include <elf.h>
-#endif /* not __NetBSD__ and not __OpenBSD__ */
+#endif
+
 #include <sys/mman.h>
 #if defined (_SYSTYPE_SYSV)
 #include <sys/elf_mips.h>

Re: Emacs on QNX

[Elad Lahav]

Hi Paul,

> Which part of the Emacs code is that? Is this the call in
> src/unexelf.c? If so, 
> I can fix that.
Yes, it's the call in unexelf.c. It ended up with a huge value for the
break point, and the dump only stopped when it filled up the disk...

> Does the attached patch work for you instead?
Not out of the box, but I'll see if I can make it work. The hybrid
malloc certainly looks like the right direction.

> Could you fill us on on what __NO_EXT_QNX does? The symbol I see
> mentioned in 
> public discussions is _QNX_SOURCE. What is the difference between
> those two 
> symbols, and why is __NO_EXT_QNX preferable here? What happens if you
> leave it out?

There is a compilation flag called _EXT_QNX that is used by default and
causes symbols that are neither in ANSI C nor POSIX to be exposed by
header files. There are several such symbols in the QNX malloc.h header
that conflict with symbols in Emacs' gmalloc.c. Using __NO_EXT_QNX was
a quick'n'dirty way to avoid this conflict, but I am far from certain
that it is the best choice. I'll see if I can do better.

--Elad

Re: Emacs on QNX

[Elad Lahav]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

The attached patch seems to work.
I changed a couple of places that where code is conditionally-compiled
on CYGWIN instead of HYBRID_MALLOC, and added support for hybrid malloc
to the ELF version of unexec(). I also removed the -L/usr/lib linker
flag, which is not needed.

--Elad
---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: qnx.diff --]
[-- Type: text/x-patch; name="qnx.diff", Size: 4566 bytes --]

diff --git a/configure b/configure
index 8783b80..1e77e47 100755
--- a/configure
+++ b/configure
@@ -5358,6 +5358,12 @@ case "${canonical}" in
     esac
   ;;
 
+  ## QNX Neutrino
+  *-nto-qnx* )
+    opsys=qnxnto
+    CFLAGS="$CFLAGS -D__NO_EXT_QNX"
+  ;;
+
   ## Intel 386 machines where we don't care about the manufacturer.
   i[3456]86-*-* )
     case "${canonical}" in
@@ -9607,6 +9613,8 @@ case "$opsys" in
 
   hpux*) LIBS_SYSTEM="-l:libdld.sl" ;;
 
+  qnxnto) LIBS_SYSTEM="-lsocket" ;;
+
   sol2*) LIBS_SYSTEM="-lsocket -lnsl" ;;
 
   ## Motif needs -lgen.
@@ -11475,7 +11483,7 @@ hybrid_malloc=
 case "$opsys" in
   ## darwin ld insists on the use of malloc routines in the System framework.
   darwin | mingw32 | nacl | sol2-10) system_malloc=yes ;;
-  cygwin) hybrid_malloc=yes;;
+  cygwin | qnxnto ) hybrid_malloc=yes;;
 esac
 
 GMALLOC_OBJ=
@@ -18475,7 +18483,7 @@ case $opsys in
 
     ;;
 
-  gnu | openbsd )
+  gnu | openbsd | qnxnto )
     $as_echo "#define FIRST_PTY_LETTER 'p'" >>confdefs.h
 
     ;;
@@ -19254,6 +19262,8 @@ elif test "$opsys" = "mingw32"; then
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=
   POST_ALLOC_OBJ=lastfile.o
+elif test "$opsys" = "qnxnto"; then
+  CYGWIN_OBJ=sheap.o
 else
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=lastfile.o
diff --git a/configure.ac b/configure.ac
index 2a4e0c1..6ba7db6 100644
--- a/configure.ac
+++ b/configure.ac
@@ -697,6 +697,12 @@ case "${canonical}" in
       *) ;;
     esac
   ;;
+ 
+  ## QNX Neutrino
+  *-nto-qnx* )
+    opsys=qnxnto
+    CFLAGS="$CFLAGS -D__NO_EXT_QNX"
+  ;;
 
   ## Intel 386 machines where we don't care about the manufacturer.
   i[3456]86-*-* )
@@ -1440,6 +1446,8 @@ case "$opsys" in
 
   hpux*) LIBS_SYSTEM="-l:libdld.sl" ;;
 
+  qnxnto) LIBS_SYSTEM="-lsocket" ;;
+
   sol2*) LIBS_SYSTEM="-lsocket -lnsl" ;;
 
   ## Motif needs -lgen.
@@ -2137,7 +2145,7 @@ hybrid_malloc=
 case "$opsys" in
   ## darwin ld insists on the use of malloc routines in the System framework.
   darwin | mingw32 | nacl | sol2-10) system_malloc=yes ;;
-  cygwin) hybrid_malloc=yes;;
+  cygwin | qnxnto ) hybrid_malloc=yes;;
 esac
 
 GMALLOC_OBJ=
@@ -4465,7 +4473,7 @@ case $opsys in
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [])
     ;;
 
-  gnu | openbsd )
+  gnu | openbsd | qnxnto )
     AC_DEFINE(FIRST_PTY_LETTER, ['p'])
     ;;
 
@@ -5045,6 +5053,8 @@ elif test "$opsys" = "mingw32"; then
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=
   POST_ALLOC_OBJ=lastfile.o
+elif test "$opsys" = "qnxnto"; then
+  CYGWIN_OBJ=sheap.o
 else
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=lastfile.o
diff --git a/src/gmalloc.c b/src/gmalloc.c
index a63927e..3489277 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -67,7 +67,7 @@ extern _Noreturn void emacs_abort (void) NO_INLINE;
 #define aligned_alloc galigned_alloc
 #define free gfree
 
-#ifdef CYGWIN
+#ifdef HYBRID_MALLOC
 extern void *bss_sbrk (ptrdiff_t size);
 extern int bss_sbrk_did_unexec;
 extern char bss_sbrk_buffer[];
@@ -1516,7 +1516,7 @@ void *
 __default_morecore (ptrdiff_t increment)
 {
   void *result;
-#if defined (CYGWIN)
+#if defined (HYBRID_MALLOC)
   if (!DUMPED)
     {
       return bss_sbrk (increment);
diff --git a/src/unexelf.c b/src/unexelf.c
index 5a56cbd..bf05e29 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -58,9 +58,11 @@ what you give them.   Help stamp out software-hoarding!  */
 #include <sys/types.h>
 #include <unistd.h>
 
-#if !defined (__NetBSD__) && !defined (__OpenBSD__)
+#ifdef __QNX__
+#include <sys/elf.h>
+#elif !defined __NetBSD__ && !defined __OpenBSD__
 #include <elf.h>
-#endif /* not __NetBSD__ and not __OpenBSD__ */
+#endif
 #include <sys/mman.h>
 #if defined (_SYSTYPE_SYSV)
 #include <sys/elf_mips.h>
@@ -191,6 +193,10 @@ verify ((! TYPE_SIGNED (ElfW (Half))
 # define DEBUG_LOG(expr) fprintf (stderr, #expr " 0x%jx\n", (uintmax_t) (expr))
 #endif
 
+#ifdef HYBRID_MALLOC
+extern int bss_sbrk_did_unexec;
+#endif
+
 /* Get the address of a particular section or program header entry,
  * accounting for the size of the entries.
  */
@@ -329,8 +335,15 @@ unexec (const char *new_name, const char *old_name)
   if (old_bss_index == -1)
     fatal ("no bss section found");
 
+#ifdef HYBRID_MALLOC
+  /* The pre-dump hybrid malloc uses an area in the BSS as a heap, 
+     which means that allocations have no impact on the final size.  */
+  new_bss_addr = old_bss_addr + old_bss_size;
+  bss_sbrk_did_unexec = 1;
+#else
   new_break = sbrk (0);
   new_bss_addr = (ElfW (Addr)) new_break;
+#endif
   bss_size_growth = new_bss_addr - old_bss_addr;
   new_data2_size = bss_size_growth;
   new_data2_size += alignof (ElfW (Shdr)) - 1;

Re: Emacs on QNX

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

On 10/23/2017 06:52 PM, Elad Lahav wrote:
> The attached patch seems to work.

Thanks. That appears to be against an older version of Emacs, though. I 
adapted it to emacs-26 (this simplified it) and installed the attached. 
Please give a try, e.g., by running these shell commands:

     git clone -b emacs-26 git://git.sv.gnu.org/emacs.git
     cd emacs
     ./autogen.sh
     ./configure
     make

assuming you have the relevant build tools.


[-- Attachment #2: 0001-Port-to-QNX.patch --]
[-- Type: text/x-patch, Size: 3466 bytes --]

From c95cc1e19eaab9df5789e3b07e2cf16298f06078 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 24 Oct 2017 12:54:28 -0700
Subject: [PATCH] Port to QNX

Simplified version of a patch proposed by Elad Lahav in:
https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00716.html
which is based on a previous patch I proposed in:
https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00707.html
* configure.ac (opsys, CFLAGS, LIBS_SYSTEM, hybrid_malloc)
(system_alloc, FIRST_PTY_LETTER, CYGWIN_OBJ):
Set appropriately for QNX.
* src/unexelf.c [__QNX__]: Include <sys/elf.h> instead of <elf.h>.
(unexec): Check for sbrk failure, and fall back on old BSS end.
---
 configure.ac  | 15 +++++++++++++--
 src/unexelf.c | 15 +++++++++------
 2 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/configure.ac b/configure.ac
index 646a637cf3..ca75136474 100644
--- a/configure.ac
+++ b/configure.ac
@@ -699,6 +699,12 @@ AC_DEFUN
     esac
   ;;
 
+  ## QNX Neutrino
+  *-nto-qnx* )
+    opsys=qnxnto
+    CFLAGS="$CFLAGS -D__NO_EXT_QNX"
+  ;;
+
   ## Intel 386 machines where we don't care about the manufacturer.
   i[3456]86-*-* )
     case "${canonical}" in
@@ -1507,6 +1513,8 @@ AC_DEFUN
 
   hpux*) LIBS_SYSTEM="-l:libdld.sl" ;;
 
+  qnxnto) LIBS_SYSTEM="-lsocket" ;;
+
   sol2*) LIBS_SYSTEM="-lsocket -lnsl" ;;
 
   ## Motif needs -lgen.
@@ -2210,7 +2218,8 @@ AC_DEFUN
 case "$opsys" in
   ## darwin ld insists on the use of malloc routines in the System framework.
   darwin | mingw32 | nacl | sol2-10) ;;
-  cygwin) hybrid_malloc=yes
+  cygwin | qnxto)
+	  hybrid_malloc=yes
           system_malloc= ;;
   *) test "$ac_cv_func_sbrk" = yes && system_malloc=$emacs_cv_sanitize_address;;
 esac
@@ -4603,7 +4612,7 @@ AC_DEFUN
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [])
     ;;
 
-  gnu | openbsd )
+  gnu | openbsd | qnxnto )
     AC_DEFINE(FIRST_PTY_LETTER, ['p'])
     ;;
 
@@ -5144,6 +5153,8 @@ AC_DEFUN
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=
   POST_ALLOC_OBJ=lastfile.o
+elif test "$opsys" = "qnxnto"; then
+  CYGWIN_OBJ=sheap.o
 else
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=lastfile.o
diff --git a/src/unexelf.c b/src/unexelf.c
index 1cdcfeb44e..756de5835c 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -58,9 +58,11 @@ what you give them.   Help stamp out software-hoarding!  */
 #include <sys/types.h>
 #include <unistd.h>
 
-#if !defined (__NetBSD__) && !defined (__OpenBSD__)
-#include <elf.h>
-#endif /* not __NetBSD__ and not __OpenBSD__ */
+#ifdef __QNX__
+# include <sys/elf.h>
+#elif !defined __NetBSD__ && !defined __OpenBSD__
+# include <elf.h>
+#endif
 #include <sys/mman.h>
 #if defined (_SYSTYPE_SYSV)
 #include <sys/elf_mips.h>
@@ -222,7 +224,6 @@ unexec (const char *new_name, const char *old_name)
 {
   int new_file, old_file;
   off_t new_file_size;
-  void *new_break;
 
   /* Pointers to the base of the image of the two files.  */
   caddr_t old_base, new_base;
@@ -326,11 +327,13 @@ unexec (const char *new_name, const char *old_name)
   if (old_bss_index == -1)
     fatal ("no bss section found");
 
+  void *no_break = (void *) (intptr_t) -1;
+  void *new_break = no_break;
 #ifdef HAVE_SBRK
   new_break = sbrk (0);
-#else
-  new_break = (byte *) old_bss_addr + old_bss_size;
 #endif
+  if (new_break == no_break)
+    new_break = (byte *) old_bss_addr + old_bss_size;
   new_bss_addr = (ElfW (Addr)) new_break;
   bss_size_growth = new_bss_addr - old_bss_addr;
   new_data2_size = bss_size_growth;
-- 
2.13.6

Re: Emacs on QNX

[Elad Lahav]

On Tue, 2017-10-24 at 13:14 -0700, Paul Eggert wrote:
> Thanks. That appears to be against an older version of Emacs, though.
> I adapted it to emacs-26 (this simplified it) and installed the
> attached. 

1. The line that adds sheap.o in configure.ac is not needed in the
latest version of the source, as the file is already added if
HYBRID_MALLOC is defined in src/Makefile.in.

2. bootstrap-emacs crashes during build due to a stack overflow when
compiling elisp-mode.elc. Perhaps the result of infinite recursion. Has
this been observed elsewhere? It doesn't happen on 25.3.

--Elad

Re: Emacs on QNX

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

Elad Lahav wrote:

> 1. The line that adds sheap.o in configure.ac is not needed in the
> latest version of the source, as the file is already added if
> HYBRID_MALLOC is defined in src/Makefile.in.

I installed the attached patch to fix that.

> 2. bootstrap-emacs crashes during build due to a stack overflow when
> compiling elisp-mode.elc. Perhaps the result of infinite recursion. Has
> this been observed elsewhere? It doesn't happen on 25.3.

I don't recall seeing it, no. Although this shouldn't prevent building on QNX, 
it does suggest a problem in the QNX implementation, which you should probably 
look into.

[-- Attachment #2: 0001-Fix-duplicate-.o-file-on-QNX.patch --]
[-- Type: text/x-patch, Size: 727 bytes --]

From 685fd779592db0019b8489a06d72ec4bebef3c9a Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Wed, 25 Oct 2017 20:47:48 -0700
Subject: [PATCH] Fix duplicate .o file on QNX

* configure.ac (CYGWIN_OBJ): Leave empty on QNX.
Problem reported by Elad Lahav in:
https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00750.html
---
 configure.ac | 2 --
 1 file changed, 2 deletions(-)

diff --git a/configure.ac b/configure.ac
index ca75136..d397e8f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -5153,8 +5153,6 @@ AC_DEFUN
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=
   POST_ALLOC_OBJ=lastfile.o
-elif test "$opsys" = "qnxnto"; then
-  CYGWIN_OBJ=sheap.o
 else
   CYGWIN_OBJ=
   PRE_ALLOC_OBJ=lastfile.o
-- 
2.7.4

Re: Emacs on QNX

[Elad Lahav]

On Wed, 2017-10-25 at 20:52 -0700, Paul Eggert wrote:
> I don't recall seeing it, no. Although this shouldn't prevent
> building on QNX, 
> it does suggest a problem in the QNX implementation, which you should
> probably 
> look into.

I forgot to mention that I did investigate the problem. The recursion
is not infinite, just very deep. In QNX all stack sizes are fixed,
including that of the main thread. The default size for the main thread
is 512K, but some elisp modules (e.g., verilog) require more in version
26 to compile. There is no such problem in the 25.x code. I wonder if
the modules themselves have changed, or something else is causing more
stack space to be used.
I can work around the problem by increasing the stack size of the main
thread, but it would be nice to understand the increase in stack usage.

--Elad

Re: Emacs on QNX

[Eli Zaretskii]

> From: Elad Lahav <elahav@blackberry.com>
> Date: Thu, 9 Nov 2017 17:17:43 +0000
> Cc: "Emacs-devel@gnu.org" <Emacs-devel@gnu.org>
> 
> I forgot to mention that I did investigate the problem. The recursion
> is not infinite, just very deep. In QNX all stack sizes are fixed,
> including that of the main thread. The default size for the main thread
> is 512K, but some elisp modules (e.g., verilog) require more in version
> 26 to compile. There is no such problem in the 25.x code. I wonder if
> the modules themselves have changed, or something else is causing more
> stack space to be used.
> I can work around the problem by increasing the stack size of the main
> thread, but it would be nice to understand the increase in stack usage.

I think you definitely need to increase the stack.  Emacs needs at
least 2MB of stack space; on MS-Windows we use 8MB, and for a good
reason.  So 512K is definitely too little.

Re: Emacs on QNX

[Noam Postavsky]

On Thu, Nov 9, 2017 at 12:17 PM, Elad Lahav <elahav@blackberry.com> wrote:

> I forgot to mention that I did investigate the problem. The recursion
> is not infinite, just very deep. In QNX all stack sizes are fixed,
> including that of the main thread. The default size for the main thread
> is 512K, but some elisp modules (e.g., verilog) require more in version
> 26 to compile. There is no such problem in the 25.x code. I wonder if
> the modules themselves have changed, or something else is causing more
> stack space to be used.

Could be [1: f0a1e9ec3f], which makes read1 try to use a
stack-allocated buffer instead of a heap-allocated one.

[1: f0a1e9ec3f]: 2016-12-08 13:00:32 -0800
  Make read1 more reentrant
  http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=f0a1e9ec3fba3d5bea5bd62f525dba3fb005d1b1

Re: Emacs on QNX

[Elad Lahav]

I definitely saw read1() in the backtrace.

--Elad

On Thu, 2017-11-09 at 12:30 -0500, Noam Postavsky wrote:
> On Thu, Nov 9, 2017 at 12:17 PM, Elad Lahav <elahav@blackberry.com>
> wrote:
> 
> > 
> > I forgot to mention that I did investigate the problem. The
> > recursion
> > is not infinite, just very deep. In QNX all stack sizes are fixed,
> > including that of the main thread. The default size for the main
> > thread
> > is 512K, but some elisp modules (e.g., verilog) require more in
> > version
> > 26 to compile. There is no such problem in the 25.x code. I wonder
> > if
> > the modules themselves have changed, or something else is causing
> > more
> > stack space to be used.
> Could be [1: f0a1e9ec3f], which makes read1 try to use a
> stack-allocated buffer instead of a heap-allocated one.
> 
> [1: f0a1e9ec3f]: 2016-12-08 13:00:32 -0800
>   Make read1 more reentrant
>   http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=f0a1e9ec3fba3
> d5bea5bd62f525dba3fb005d1b1

Re: Emacs on QNX

[Elad Lahav]

On Thu, 2017-11-09 at 19:23 +0200, Eli Zaretskii wrote:
> I think you definitely need to increase the stack.  Emacs needs at
> least 2MB of stack space; on MS-Windows we use 8MB, and for a good
> reason.  So 512K is definitely too little.

Sorry, it took me a while to get back to this.
The following patch fixes the problem, and emacs/HEAD now builds and
runs on QNX:

=====
diff --git a/configure.ac b/configure.ac
index 469ad00eaa..e0944dd12b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -702,7 +702,11 @@ AC_DEFUN
   ## QNX Neutrino
   *-nto-qnx* )
     opsys=qnxnto
+    ## Need to use qcc and the -N flag to increase the default stack
+    ## size for the main thread.
+    CC=qcc
     CFLAGS="$CFLAGS -D__NO_EXT_QNX"
+    LDFLAGS=-N2MB
   ;;
 
   ## Intel 386 machines where we don't care about the manufacturer.
=====

qcc is a simple wrapper around gcc. It takes some extra arguments,
including -N, which sets the stack size of the main thread.

For the record, this is the command line I used to configure emacs:

# LIBXML2_CFLAGS=-I/usr/include LIBXML2_LIBS=-lxml2 -lm ./configure 
  --without-makeinfo --with-gnutls=no --with-x-toolkit=no

The LIBXML2_CFLAGS variable is bogus, as no special flags are required.
However, unless this variable being is set to a non-empty string the
configure script tries to use pkg-config for libxml2, which doesn't
work.

--Elad

Re: Emacs on QNX

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

On 11/30/2017 09:43 AM, Elad Lahav wrote:
> The following patch fixes the problem

Thanks, I installed the attached slightly-more-conservative patch into 
the emacs-26 branch.

Hmm, there appears to have been a typo in configure.ac, which I also 
fixed with the attached patch. Why didn't this typo prevent your build 
from working?

> The LIBXML2_CFLAGS variable is bogus, as no special flags are required.
> However, unless this variable being is set to a non-empty string the
> configure script tries to use pkg-config for libxml2, which doesn't
> work.

In what sense doesn't it work? Can we work around this problem 
automatically? After all, if we were willing to live with undocumented 
manual settings, we could have left 'configure' alone and expected 
installers to figure out a command like this one:

./configure --without-makeinfo --with-gnutls=no --with-x-toolkit=no \
   CC=qcc \
   CFLAGS='-D__NO_EXT_QNX' \
   LDFLAGS='-N2MB' \
   LIBXML2_CFLAGS=-I/usr/include \
   LIBXML2_LIBS='-lxml2 -lm'



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Port-better-to-QNX.patch --]
[-- Type: text/x-patch; name="0001-Port-better-to-QNX.patch", Size: 1360 bytes --]

From e899fb9c69b1b8d6a37c6454746315ca8a0f1d2f Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 30 Nov 2017 15:22:46 -0800
Subject: [PATCH] Port better to QNX
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Problem reported by Elad Lahav on emacs-devel.
* configure.ac: On QNX, default CC to qcc (a GCC wrapper),
and default LDFLAGS to -N2MB so that the initial stack size
is not too small.  Also, fix misspelling of ‘qnxnto’.
---
 configure.ac | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index f1ca7cc81b..2df5679f1d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -702,7 +702,9 @@ AC_DEFUN
   ## QNX Neutrino
   *-nto-qnx* )
     opsys=qnxnto
+    test -z "$CC" && CC=qcc
     CFLAGS="$CFLAGS -D__NO_EXT_QNX"
+    LDFLAGS="-N2MB $LDFLAGS"
   ;;
 
   ## Intel 386 machines where we don't care about the manufacturer.
@@ -2218,7 +2220,7 @@ AC_DEFUN
 case "$opsys" in
   ## darwin ld insists on the use of malloc routines in the System framework.
   darwin | mingw32 | nacl | sol2-10) ;;
-  cygwin | qnxto | freebsd)
+  cygwin | qnxnto | freebsd)
 	  hybrid_malloc=yes
           system_malloc= ;;
   *) test "$ac_cv_func_sbrk" = yes && system_malloc=$emacs_cv_sanitize_address;;
-- 
2.14.3

Re: Emacs on QNX

[Elad Lahav]

On Thu, 2017-11-30 at 15:41 -0800, Paul Eggert wrote:
> Hmm, there appears to have been a typo in configure.ac, which I also 
> fixed with the attached patch. Why didn't this typo prevent your
> build from working?

Excellent question. I'll see if I can figure it out.

> In what sense doesn't it work? Can we work around this problem 
> automatically? 

pkg-config doesn't exist.

> After all, if we were willing to live with undocumented 
> manual settings, we could have left 'configure' alone and expected 
> installers to figure out a command like this one:
> 
> ./configure --without-makeinfo --with-gnutls=no --with-x-toolkit=no \
>    CC=qcc \
>    CFLAGS='-D__NO_EXT_QNX' \
>    LDFLAGS='-N2MB' \
>    LIBXML2_CFLAGS=-I/usr/include \
>    LIBXML2_LIBS='-lxml2 -lm'
> 

That's a valid point. I could have added -lm to LDFLAFS in
configure.ac. Nevertheless, I find the requirement to use pkg-config
for detecting libxml limiting, and I certainly don't like having to set
an unnecessary environment variable to force its inclusion. Is it not
possible to test for libxml with a simple test program, the way other
components are detected?

--Elad

Re: Emacs on QNX

[Paul Eggert]

Elad Lahav wrote:

>> In what sense doesn't it work? Can we work around this problem
>> automatically?
> 
> pkg-config doesn't exist. 

Oh, by "don't work" you meant that you wanted Emacs linked to libxml2 even 
though you lack the pkg-config setup that ordinarily comes with libxml2 
development environments. That is, Emacs still builds and runs, it's just that 
it doesn't automatically have an optional feature that you'd like to have, and 
you have to set two environment variables to get it.

> I find the requirement to use pkg-config
> for detecting libxml limiting, and I certainly don't like having to set
> an unnecessary environment variable to force its inclusion. Is it not
> possible to test for libxml with a simple test program, the way other
> components are detected?

It's of course possible, but I'm not sure it's worth the maintenance hassle for 
us. We used to configure all libraries by hand but found that this was a real 
maintenance burden, and it was easier to rely on pkg-config to solve the problem 
(at least for newer libraries like libxml2). You have a solution that works for 
QNX and, I think it's unlikely that people on other platforms will run into the 
problem. If they do, we can revisit this issue.

Alternatively, you might consider installing pkg-config on QNX. That might 
simplify things for both of us.

http://wiki.ros.org/groovy/Installation/QNX#Install_pkg-config_via_pkgsrc

Re: Emacs on QNX

[Elad Lahav]

On Thu, 2017-11-30 at 19:37 -0800, Paul Eggert wrote:
> It's of course possible, but I'm not sure it's worth the maintenance
> hassle for 
> us. We used to configure all libraries by hand but found that this
> was a real 
> maintenance burden, and it was easier to rely on pkg-config to solve
> the problem 
> (at least for newer libraries like libxml2). You have a solution that
> works for 
> QNX and, I think it's unlikely that people on other platforms will
> run into the 
> problem. If they do, we can revisit this issue.

Don't get me wrong - I have no problem building with the command line I
cited - still better than the one I had to use to build vim ;-) The
number of people currently using Emacs on QNX can probably be counted
on two digits (as in fingers), so there is really no reason to go out
of your way to support it. The only thing I can suggest is to not
require both LIBXML2_LIBS and LIBXML2_CFLAGS to be specified.

--Elad

RE: Emacs on QNX

[Elad Lahav]

> Hmm, there appears to have been a typo in configure.ac, which I also
> fixed with the attached patch. Why didn't this typo prevent your build
> from working?

We got lucky, due to the following lines in configure.ac:

===
  cygwin | qnxto)
	  hybrid_malloc=yes
          system_malloc= ;;
  *) test "$ac_cv_func_sbrk" = yes && system_malloc=$emacs_cv_sanitize_address;;
esac

if test "${system_malloc}" != yes && test "${doug_lea_malloc}" != yes \
   && test "${UNEXEC_OBJ}" = unexelf.o; then
  hybrid_malloc=yes
fi
===

Since 'qnxnto' doesn't match any other case, it goes to the default, and since $emacs_cv_sanitize_address happens to be "no" hybrid_malloc still gets defined.

As a side note, porting Emacs finally drove me to implement openat()/fstatat(), which have been on the to-do list for a very long time (not as trivial as you may think in a micro-kernel environment, where different files within the same directory may be provided by different external servers).

--Elad