bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[Paul Eggert]

[moving this to bug-gnu-emacs, so that it gets a proper bug number]

Georges Ko wrote:
> What were the changes done on Emacs 24.4 regarding subshells (or
> processes), compared to version 24.2?

Quite a few, I suspect.  You can use 'diff' to find out the details....

> When compiled in the same environment, "tty" in a subshell
> (M-x shell) has different outputs:
>
>    - in Emacs 24.2, "tty" outputs the tty,
>    - in Emacs 24.4, "tty" outputs "not a tty".
>
> That is, if I run the command below to spawn a shell:
>
>    (apply 'start-process "shell" "*shell*" "/bin/ksh" '("-i"))
>
>    - in Emacs 24.2, "ps" shows the shell has a normal tty,
>    - in Emacs 24.4, "ps" shows the shell has a tty with "?".

I don't observe either problem when running Emacs 24.5 under Solaris 11 x86-64 
compiled with the bundled GCC (4.5.2).  However, I do observe them when running 
Emacs 24.5 under Solaris 10 sparc compiled with Sun C 5.13.

> I couldn't compile 24.3 due to some errors, so I don't know its behavior.

Emacs 24.3 is bundled within Solaris 11.2, for what it's worth.

If you're still running into problems, I suggest replying with details of how 
you built Emacs (version of OS and compiler, 'configure' arguments, log from 
'configure' and 'make', etc.).

PS.  This looks a bit like Bug#9150 <http://bugs.gnu.org/9150>, but that bug was 
in an older Emacs, which probably means it's independent.

bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[Glenn Morris]

Paul Eggert wrote:

> [moving this to bug-gnu-emacs, so that it gets a proper bug number]

It already has one: http://debbugs.gnu.org/19191

bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[Georges Ko]

Glenn Morris <rgm@gnu.org> writes:

> Paul Eggert wrote:
>
>> [moving this to bug-gnu-emacs, so that it gets a proper bug number]
>
> It already has one: http://debbugs.gnu.org/19191

I have been able to "fix" the problem by trial and error.

On my system, the function allocate_pty (process.c) calls the macro 
PTY_TTY_NAME_SPRINTF, generated in config.h, which is: 

  #define PTY_TTY_NAME_SPRINTF
  {
    char *ptsname (int), *ptyname;
    int grantpt_result;
    sigset_t blocked;
    sigemptyset (&blocked);
    sigaddset (&blocked, SIGCHLD);
    pthread_sigmask (SIG_BLOCK, &blocked, 0);
    grantpt_result = grantpt (fd);
    pthread_sigmask (SIG_UNBLOCK, &blocked, 0);
    if (grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname (fd)))
      {
        emacs_close (fd);
        return -1;
      }
    snprintf (pty_name, PTY_NAME_SIZE, "%s", ptyname);
  } 

It turned out that grantpt(fd) returns -1, with errno = 13 (EACCES), 
so -1 is returned by allocate_pty.

After checking EACCES of grantpt():

  The slave pseudoterminal was opened before grantpt(), or a grantpt() was 
  already issued. In either case, slave pseudoterminal permissions and 
  ownership have already been updated. If you use grantpt() to change slave 
  pseudoterminal permissions, you must issue grantpt() between the master 
  open and the first pseudoterminal open, and grantpt() can only be issued 
  once. 
  
I changed the macro to:

    if ( /* grantpt_result == -1 || */ unlockpt (fd) == -1 || !(ptyname = ptsname (fd)))

and it "fixed" the problem. I'll let the experts figure out the root cause...

The diff of configure (24.5):

$ diff configure.24.5 configure.24.5.ttyfix
17105c17105
<                 $as_echo "#define PTY_TTY_NAME_SPRINTF { char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if (grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname (fd))) { emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, \"%s\", ptyname); }" >>confdefs.h
---
>                 $as_echo "#define PTY_TTY_NAME_SPRINTF { char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if ( /* grantpt_result == -1 || */ unlockpt (fd) == -1 || !(ptyname = ptsname (fd))) { emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, \"%s\", ptyname); }" >>confdefs.h

Georges
-- 
 Georges Ko                     gko@gko.net                      2015-05-13

bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[白井彰]

I wrote a small program my-emacs-24.5.c, and found that grantpt(fd)
returned -1 after fcntl(fd, F_SETFD, FD_CLOEXEC).
Without fcntl(fd, F_SETFD, FD_CLOEXEC), grantpt(fd) returns 0.

aaa% sh emacs-24.5/build-aux/config.guess
i386-pc-solaris2.10

aaa% ls my-emacs-24.5.c
my-emacs-24.5.c

aaa% make my-emacs-24.5 && ./my-emacs-24.5
cc    -o my-emacs-24.5 my-emacs-24.5.c
call_fcntl_fd_cloexec: 1
fcntl_result: 0
fd: 3
grantpt_result: -1
errno: 13
grantpt: Permission denied

aaa% make my-emacs-24.5 && ./my-emacs-24.5 call_fcntl_fd_cloexec=0
`my-emacs-24.5' is up to date.
call_fcntl_fd_cloexec: 0
fd: 3
grantpt_result: 0

bbb% sh emacs-24.5/build-aux/config.guess
sparc-sun-solaris2.8

bbb% rm my-emacs-24.5
bbb% make my-emacs-24.5 && ./my-emacs-24.5
cc    -o my-emacs-24.5 my-emacs-24.5.c
call_fcntl_fd_cloexec: 1
fcntl_result: 0
fd: 3
grantpt_result: -1
errno: 0
grantpt: Error 0

bbb% make my-emacs-24.5 && ./my-emacs-24.5 call_fcntl_fd_cloexec=0
`my-emacs-24.5' is up to date.
call_fcntl_fd_cloexec: 0
fd: 3
grantpt_result: 0

% cat -n my-emacs-24.5.c
     1  /* 24.5 */
     2
     3  #include <errno.h>
     4  #include <fcntl.h>
     5  #include <stdio.h>
     6  #include <stdlib.h>
     7  #include <string.h>
     8  #include <sys/stat.h>
     9  #include <sys/types.h>
    10  #include <unistd.h>
    11
    12  int call_fcntl_fd_cloexec;
    13
    14  enum { PTY_NAME_SIZE = 24 };
    15
    16  int emacs_open(const char *file, int oflags, int mode) {
    17  #define O_CLOEXEC 0
    18    int fd;
    19    oflags |= O_CLOEXEC;
    20    while ((fd = open(file, oflags, mode)) < 0 && errno == EINTR) abort();
    21    if (! O_CLOEXEC && 0 <= fd) {
    22      fprintf(stderr, "call_fcntl_fd_cloexec: %d\n", call_fcntl_fd_cloexec);
    23      if (call_fcntl_fd_cloexec) {
    24        int fcntl_result = fcntl(fd, F_SETFD, FD_CLOEXEC);
    25        fprintf(stderr, "fcntl_result: %d\n", fcntl_result);
    26      }
    27    }
    28    return fd;
    29  }
    30
    31  static int allocate_pty(char pty_name[PTY_NAME_SIZE]) {
    32    int fd;
    33    strcpy(pty_name, "/dev/ptmx");
    34    fd = emacs_open(pty_name, O_RDWR | O_NONBLOCK, 0);
    35    fprintf(stderr, "fd: %d\n", fd);
    36    if (fd >= 0) {
    37      int grantpt_result = grantpt(fd);
    38      fprintf(stderr, "grantpt_result: %d\n", grantpt_result);
    39      if (grantpt_result != 0) {
    40        fprintf(stderr, "errno: %d\n", errno);
    41        perror("grantpt");
    42      }
    43    }
    44    return fd;
    45  }
    46
    47  int main(int argc, char **argv) {
    48    char pty_name[PTY_NAME_SIZE];
    49    if (argc == 2) call_fcntl_fd_cloexec = 0;
    50    else           call_fcntl_fd_cloexec = 1;
    51    allocate_pty(pty_name);
    52    return 0;
    53  }

bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[白井彰]

Today I found the same problem on Ruby:

| grantpt() doesn't work with CLOEXEC on Solaris 10
| http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?revision=33519&view=revision

They say that Solaris grantpt() internally uses a setuid root program
/usr/lib/pt_chmod which receives an fd and calls ptsname(), chown(),
and chmod().  If FD_CLOEXEC is set for the fd, the fd passed for
/usr/lib/pt_chmod is already closed and /usr/lib/pt_chmod does not
work.

The following is their fix for the problem:

% gzcat ruby-2.2.2.tar.gz | tar tvf - | grep ruby-2.2.2/ext/pty/pty.c
-rw-r--r-- 1044/1044  22152 Nov 15 20:49 2014 ruby-2.2.2/ext/pty/pty.c

% cat -n ruby-2.2.2/ext/pty/pty.c | sed -n '221,223p;234,256p;330,341p;450p'
   221  static int
   222  get_device_once(int *master, int *slave, char SlaveName[DEVICELEN], int nomesg, int fail)
   223  {
   234  #if defined(__sun) || (defined(__FreeBSD__) && __FreeBSD_version < 902000)
   235      /* workaround for Solaris 10: grantpt() doesn't work if FD_CLOEXEC is set.  [ruby-dev:44688] */
   236      /* FreeBSD 9.2 or later supports O_CLOEXEC
   237       * http://www.freebsd.org/cgi/query-pr.cgi?pr=162374 */
   238      if ((masterfd = posix_openpt(O_RDWR|O_NOCTTY)) == -1) goto error;
   239      if (sigaction(SIGCHLD, &dfl, &old) == -1) goto error;
   240      if (grantpt(masterfd) == -1) goto grantpt_error;
   241      rb_fd_fix_cloexec(masterfd);
   242  #else
   243      {
   244          int flags = O_RDWR|O_NOCTTY;
   245  # if defined(O_CLOEXEC)
   246          /* glibc posix_openpt() in GNU/Linux calls open("/dev/ptmx", flags) internally.
   247           * So version dependency on GNU/Linux is same as O_CLOEXEC with open().
   248           * O_CLOEXEC is available since Linux 2.6.23.  Linux 2.6.18 silently ignore it. */
   249          flags |= O_CLOEXEC;
   250  # endif
   251          if ((masterfd = posix_openpt(flags)) == -1) goto error;
   252      }
   253      rb_fd_fix_cloexec(masterfd);
   254      if (sigaction(SIGCHLD, &dfl, &old) == -1) goto error;
   255      if (grantpt(masterfd) == -1) goto grantpt_error;
   256  #endif
   330  #if defined(__sun)
   331      /* workaround for Solaris 10: grantpt() doesn't work if FD_CLOEXEC is set.  [ruby-dev:44688] */
   332      if((masterfd = open("/dev/ptmx", O_RDWR, 0)) == -1) goto error;
   333      s = signal(SIGCHLD, SIG_DFL);
   334      if(grantpt(masterfd) == -1) goto error;
   335      rb_fd_fix_cloexec(masterfd);
   336  #else
   337      if((masterfd = rb_cloexec_open("/dev/ptmx", O_RDWR, 0)) == -1) goto error;
   338      rb_update_max_fd(masterfd);
   339      s = signal(SIGCHLD, SIG_DFL);
   340      if(grantpt(masterfd) == -1) goto error;
   341  #endif
   450  }

bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[Glenn Morris]

Paul, you are the go-to-guy for these kind of problems.
Any thoughts on

http://debbugs.gnu.org/20555#13
http://debbugs.gnu.org/20555#20
http://debbugs.gnu.org/20555#23


?

bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 234 bytes --]

Akira Shirai, thanks for the info about grantpt and its internal setuid program; 
that made the problem obvious.  I installed the attached patch into the Emacs 
master sources and am (perhaps optimistically) marking this bug as done.

[-- Attachment #2: 0001-Fix-not-a-tty-bug-on-Solaris-10.patch --]
[-- Type: text/x-diff, Size: 3904 bytes --]

From 099cb9f7b8b1d00293298d947fc77b8c96120820 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 11 Jun 2015 16:41:36 -0700
Subject: [PATCH] Fix "not a tty" bug on Solaris 10

* configure.ac (PTY_OPEN): Define to plain 'open'
on SVR4-derived hosts, so that the O_CLOEXEC flag isn't set.
* src/process.c (allocate_pty): Set the O_CLOEXEC flag after
calling PTY_TTY_NAME_SPRINTF, for the benefit of SVR4-derived
hosts that call grantpt which does its work via a setuid subcommand
(Bug#19191, Bug#19927, Bug#20555, Bug#20686).
Also, set O_CLOEXEC even if PTY_OPEN is not defined, since it
seems relevant in that case too.
---
 configure.ac  |  5 ++++-
 src/process.c | 20 +++++++++++---------
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/configure.ac b/configure.ac
index 9c6a74a..070b061 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4397,14 +4397,17 @@ case $opsys in
     ;;
 
   sol2* )
-    dnl On SysVr4, grantpt(3) forks a subprocess, so keep sigchld_handler()
+    dnl On SysVr4, grantpt(3) forks a subprocess, so do not use
+    dnl O_CLOEXEC when opening the pty, and keep the SIGCHLD handler
     dnl from intercepting that death.  If any child but grantpt's should die
     dnl within, it should be caught after sigrelse(2).
+    AC_DEFINE(PTY_OPEN, [fd = open (pty_name, O_RDWR | O_NONBLOCK)])
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [{ char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if (grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname (fd))) { emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, "%s", ptyname); }])
     ;;
 
   unixware )
     dnl Comments are as per sol2*.
+    AC_DEFINE(PTY_OPEN, [fd = open (pty_name, O_RDWR | O_NONBLOCK)])
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [{ char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if (grantpt_result == -1) fatal("could not grant slave pty"); if (unlockpt(fd) == -1) fatal("could not unlock slave pty"); if (!(ptyname = ptsname(fd))) fatal ("could not enable slave pty"); snprintf (pty_name, PTY_NAME_SIZE, "%s", ptyname); }])
     ;;
 esac
diff --git a/src/process.c b/src/process.c
index 17fe708..b4f979f 100644
--- a/src/process.c
+++ b/src/process.c
@@ -658,22 +658,24 @@ allocate_pty (char pty_name[PTY_NAME_SIZE])
 
 	if (fd >= 0)
 	  {
-#ifdef PTY_OPEN
+#ifdef PTY_TTY_NAME_SPRINTF
+	    PTY_TTY_NAME_SPRINTF
+#else
+	    sprintf (pty_name, "/dev/tty%c%x", c, i);
+#endif /* no PTY_TTY_NAME_SPRINTF */
+
 	    /* Set FD's close-on-exec flag.  This is needed even if
 	       PT_OPEN calls posix_openpt with O_CLOEXEC, since POSIX
 	       doesn't require support for that combination.
+	       Do this after PTY_TTY_NAME_SPRINTF, which on some platforms
+	       doesn't work if the close-on-exec flag is set (Bug#20555).
 	       Multithreaded platforms where posix_openpt ignores
 	       O_CLOEXEC (or where PTY_OPEN doesn't call posix_openpt)
 	       have a race condition between the PTY_OPEN and here.  */
 	    fcntl (fd, F_SETFD, FD_CLOEXEC);
-#endif
-	    /* Check to make certain that both sides are available
-	       this avoids a nasty yet stupid bug in rlogins.  */
-#ifdef PTY_TTY_NAME_SPRINTF
-	    PTY_TTY_NAME_SPRINTF
-#else
-	    sprintf (pty_name, "/dev/tty%c%x", c, i);
-#endif /* no PTY_TTY_NAME_SPRINTF */
+
+	    /* Check to make certain that both sides are available.
+	       This avoids a nasty yet stupid bug in rlogins.  */
 	    if (faccessat (AT_FDCWD, pty_name, R_OK | W_OK, AT_EACCESS) != 0)
 	      {
 		emacs_close (fd);
-- 
2.1.0