emacs-24.2.93 build problems

emacs-24.2.93 build problems

[Nelson H. F. Beebe]

I have spent much of today in build attempts for emacs-24.2.93 on
about 25 flavors of Unix in our lab.  After a lot of restarts, I have
mostly succeeded, except for two platforms: OpenBSD 4.9 on x86, and
MirOS 10 on x86.  On those two systems, compilation of src/sysdep.c
fails like this:

    sysdep.c: In function 'list_system_processes':
    sysdep.c:2656: error: 'KERN_PROC' undeclared (first use in this function)
    sysdep.c:2656: error: (Each undeclared identifier is reported only once
    sysdep.c:2656: error: for each function it appears in.)
    sysdep.c:2684: error: 'struct kinfo_proc' has no member named 'p_pid'

Both systems previously had emacs-24.1 installed.  A similar build of
emacs-24.2.93 on OpenBSD 5.1 was successful.

Examination of the source code reveals the problem: function
list_system_processes() is delving into kernel data structures defined
in <kvm.h> which in turn includes <sys/sysctl.h>, and that header file
defines kernel data structures that list_system_processes() attempts
to reference. 

In OpenBSD 5.1, p_pid is in struct kinfo_proc, but on the other two
systems, p_pid is instead in struct kinfo_proc2.  There is no support
code in list_system_processes() for that alternative, and no test for
it in configure either.

By contrast, the emacs-24.1 version of list_system_processes() has no
reference to the kinfo_proc* structs.

On request, I can provide off-list pointers to copies of sysctl.h for
the two failing systems.

Although some might argue that because OpenBSD 5.x is available,
OpenBSD 4.x is no longer of interest, I do not hold that view.  Emacs
has been around now for almost 40 years, and for the last 30 or so,
its portability has been held in high regard.  As a result, it is
available on, or has been ported to, all common desktop and mobile
platforms, even non-Unix ones.  I prefer to see that portability
continue.  Emacs is what makes computers usable in much the same way,
no matter what the underlying O/S platform is.

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                    FAX: +1 801 581 4148                  -
- Department of Mathematics, 110 LCB    Internet e-mail: beebe@math.utah.edu  -
- 155 S 1400 E RM 233                       beebe@acm.org  beebe@computer.org -
- Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe/ -
-------------------------------------------------------------------------------

Re: emacs-24.2.93 build problems

[Paul Eggert]

On 03/02/2013 03:00 PM, Nelson H. F. Beebe wrote:
>     sysdep.c:2656: error: 'KERN_PROC' undeclared (first use in this function)

Thanks for the heads-up.  Yes, we should fix the problem.
Does the following little patch fix the bug?  (I don't have
any OpenBSD hosts to play with, unfortunately.)

=== modified file 'src/sysdep.c'
--- src/sysdep.c	2013-01-11 07:47:57 +0000
+++ src/sysdep.c	2013-03-03 00:04:08 +0000
@@ -2649,6 +2649,13 @@ list_system_processes (void)
 
 #elif defined BSD_SYSTEM
 
+/* OpenBSD 4.9 and earlier do not have KERN_PROC.  Approximate it with
+   KERN_PROC2.  */
+# ifndef KERN_PROC
+#  define KERN_PROC KERN_PROC2
+#  define kinfo_proc kinfo_proc2
+# endif
+
 Lisp_Object
 list_system_processes (void)
 {

Re: emacs-24.2.93 build problems

[Glenn Morris]

If we don't get a response soon, it seems to me that we can just restore
the 24.2 behaviour for this platform with the following. Is that correct?
(Assuming of course that there is no other build problem that we have
not heard about yet.)


*** src/sysdep.c	2013-01-11 07:47:57 +0000
--- src/sysdep.c	2013-03-04 20:31:27 +0000
***************
*** 2647,2653 ****
    return proclist;
  }
  
! #elif defined BSD_SYSTEM
  
  Lisp_Object
  list_system_processes (void)
--- 2647,2653 ----
    return proclist;
  }
  
! #elif defined (BSD_SYSTEM) && defined (KERN_PROC)
  
  Lisp_Object
  list_system_processes (void)

Re: emacs-24.2.93 build problems

[Jérémie Courrèges-Anglas]

Glenn Morris <rgm@gnu.org> writes:

> If we don't get a response soon, it seems to me that we can just restore
> the 24.2 behaviour for this platform with the following. Is that correct?
> (Assuming of course that there is no other build problem that we have
> not heard about yet.)

I have no idea whether it is correct, but M-: (list-system-processes)
returns nil here on OpenBSD-current (5.3).  So you might as well disable
all support for this function on OpenBSD, unless someone starts whining...

I'll try to take a look at it and at the compat bits, even though I find
it funny to try support all ABIs of all versions of all OSes ever made
on earth.

-- 
Jérémie Courrèges-Anglas (OpenBSD Emacs port maintainer)
GPG Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

Re: emacs-24.2.93 build problems

[Glenn Morris]

I tested this on a virtual OpenBSD 4.9, and it fixes the problem for me.
Please install (otherwise I will in a few hours).

list-system-processes still returns nil, so it does't do anything
useful, but it doesn't crash or do anything obviously bad.

Paul Eggert wrote:

> === modified file 'src/sysdep.c'
> --- src/sysdep.c	2013-01-11 07:47:57 +0000
> +++ src/sysdep.c	2013-03-03 00:04:08 +0000
> @@ -2649,6 +2649,13 @@ list_system_processes (void)
>  
>  #elif defined BSD_SYSTEM
>  
> +/* OpenBSD 4.9 and earlier do not have KERN_PROC.  Approximate it with
> +   KERN_PROC2.  */
> +# ifndef KERN_PROC
> +#  define KERN_PROC KERN_PROC2
> +#  define kinfo_proc kinfo_proc2
> +# endif
> +
>  Lisp_Object
>  list_system_processes (void)
>  {

Re: emacs-24.2.93 build problems

[Paul Eggert]

On 03/04/2013 03:51 PM, Glenn Morris wrote:
> I tested this on a virtual OpenBSD 4.9, and it fixes the problem for me.
> Please install (otherwise I will in a few hours).

Done, as emacs-24 bzr 111316.  Thanks for testing it.

http://bzr.savannah.gnu.org/lh/emacs/emacs-24/revision/111316

Re: emacs-24.2.93 build problems

[Glenn Morris]

Jérémie Courrèges-Anglas wrote:

> I have no idea whether it is correct, but M-: (list-system-processes)
> returns nil here on OpenBSD-current (5.3). 

Too late to make it work for 24.3, but could you make a bug report about
this please?

Re: emacs-24.2.93 build problems

[Jérémie Courrèges-Anglas]

[-- Attachment #1: Type: text/plain, Size: 2162 bytes --]

Glenn Morris <rgm@gnu.org> writes:

> Jérémie Courrèges-Anglas wrote:
>
>> I have no idea whether it is correct, but M-: (list-system-processes)
>> returns nil here on OpenBSD-current (5.3).
>
> Too late to make it work for 24.3, but could you make a bug report about
> this please?

Fine, I think I have a diff to make it work on all OpenBSD systems, that
could be included in the trunk, I just need to test it.

But as far as 24.3 is concerned, I *really* suggest making this
function a stub on OpenBSD (just returning Qnil).

Here's why:

There are two different APIs involved:
- old KERN_PROC, similar to NetBSD
- new KERN_PROC, completely different

Here's how they have evolved:

1. old KERN_PROC API present and visible (usable)
   sysctl.h rev 1.1.1.1 (1995/10/18) -> rev 1.67 (2003/12/23)

2. new KERN_PROC API added as KERN_PROC2, old KERN_PROC still available
   sysctl.h rev 1.68 (2004/01/07) -> rev 1.100 (2009/06/15)

3. old KERN_PROC becomes hidden behind #ifdef's
   sysctl rev 1.101 (2010/01/10) -> rev 1.108 (2011/03/07)

4. old KERN_PROC is removed, the new KERN_PROC replaces it, KERN_PROC2
   aliases still present for backward compat
   sysctl.h rev 1.109 (2011/03/12) -> rev 1.119 (2011/12/14)

5. KERN_PROC2 aliases removed, only the new API remains
   sysctl.h rev 1.120 (2012/01/07) -> now


I've done some _light_ testing and here are the results:

- with or without the patch adding fallback to KERN_PROC2, build fails
  for cases 1 and 2. (This is because p_pid field wasn't present in
  struct kinfo_proc at that time.)

- with the patch, build doesn't fail for case 3

- with the patch, cases 3, 4 and 5 will have list-system-processes
  returning nil, since sysctl() is called with a wrong mib.

So supporting list-system-processes (as-is) on OpenBSD for 24.3 leads to
build failing with 1995/10/18 < OpenBSD < 2010/01/10 and to a useless
(and potentially harmful) function with OpenBSD >= 2010/01/10.

PS: I did not and I will not investigate the case of MirOS.
-- 
Jérémie Courrèges-Anglas
GPG Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: emacs-24.2.93 build problems (on OpenBSD)

[Jérémie Courrèges-Anglas]

[-- Attachment #1.1: Type: text/plain, Size: 854 bytes --]

jca+emacs@wxcvbn.org (Jérémie Courrèges-Anglas) writes:

> Glenn Morris <rgm@gnu.org> writes:
>
>> Jérémie Courrèges-Anglas wrote:
>>
>>> I have no idea whether it is correct, but M-: (list-system-processes)
>>> returns nil here on OpenBSD-current (5.3).
>>
>> Too late to make it work for 24.3, but could you make a bug report about
>> this please?

By the way, I don't have the time to make a bugreport right now, but
here's the diff (against 24.2.93) I would submit (after I have time to
test it thoroughly).

People with an access to OpenBSD and / or MirOS boxes (any version),
please test with diff.

This takes the #ifdef dance out of the function, doesn't hardcode the
mib size and uses the correct mib for (somewhat) recent OpenBSD versions.
list-system-processes doesn't return nil any more (OpenBSD 5.3 -current).


[-- Attachment #1.2: patch-sysdep.c --]
[-- Type: application/octet-stream, Size: 2601 bytes --]

--- sysdep.c.orig	Tue Mar  5 15:09:25 2013
+++ sysdep.c	Tue Mar  5 15:09:30 2013
@@ -2649,42 +2649,61 @@
 
 #elif defined BSD_SYSTEM
 
+/* Sysctl-based access to the list of processes. */
+
+/* Compatibility macros */
+#if defined __OpenBSD__
+# if defined KERN_PROC && KERN_PROC == 14 /* Old API */
+#  define MIB_INITIALIZER {CTL_KERN, KERN_PROC, KERN_PROC_ALL}
+#  define PID_FROM_KINFO_PROC(k) (k).kp_proc.p_pid
+# else /* New API */
+#  if !defined KERN_PROC
+#   define KERN_PROC KERN_PROC2
+#   define kinfo_proc kinfo_proc2
+#  endif
+#  define MIB_INITIALIZER \
+    {CTL_KERN, KERN_PROC, KERN_PROC_ALL, 0, sizeof (struct kinfo_proc), 4096}
+#  define PID_FROM_KINFO_PROC(k) (k).p_pid
+# endif
+/* Darwin, NetBSD, OSX, ... */
+#elif defined DARWIN_OS || defined __NetBSD__
+# define MIB_INITIALIZER {CTL_KERN, KERN_PROC, KERN_PROC_ALL}
+# define PID_FROM_KINFO_PROC(k) (k).kp_proc.p_pid
+#else
+/* FreeBSD, DragonFlyBSD, ... */
+# define MIB_INITIALIZER {CTL_KERN, KERN_PROC, KERN_PROC_PROC}
+# define PID_FROM_KINFO_PROC(k) (k).ki_pid
+#endif
+
 Lisp_Object
 list_system_processes (void)
 {
-#if defined DARWIN_OS || defined __NetBSD__ || defined __OpenBSD__
-  int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_ALL};
-#else
-  int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_PROC};
-#endif
-  size_t len;
+  int mib[] = MIB_INITIALIZER;
+  size_t len, mibsize = sizeof mib / sizeof mib[0];
   struct kinfo_proc *procs;
   size_t i;
+  FILE *f;
 
   struct gcpro gcpro1;
   Lisp_Object proclist = Qnil;
 
-  if (sysctl (mib, 3, NULL, &len, NULL, 0) != 0)
+  if (sysctl (mib, mibsize, NULL, &len, NULL, 0) != 0)
     return proclist;
 
   procs = xmalloc (len);
-  if (sysctl (mib, 3, procs, &len, NULL, 0) != 0)
+  if (sysctl (mib, mibsize, procs, &len, NULL, 0) != 0)
     {
       xfree (procs);
       return proclist;
     }
-
+  
   GCPRO1 (proclist);
-  len /= sizeof (struct kinfo_proc);
+  len /= sizeof procs[0];
   for (i = 0; i < len; i++)
     {
-#if defined DARWIN_OS || defined __NetBSD__
-      proclist = Fcons (make_fixnum_or_float (procs[i].kp_proc.p_pid), proclist);
-#elif defined __OpenBSD__
-      proclist = Fcons (make_fixnum_or_float (procs[i].p_pid), proclist);
-#else
-      proclist = Fcons (make_fixnum_or_float (procs[i].ki_pid), proclist);
-#endif
+      proclist =
+	Fcons (make_fixnum_or_float (PID_FROM_KINFO_PROC (procs[i])),
+	       proclist);
     }
   UNGCPRO;
 
@@ -2692,6 +2711,9 @@
 
   return  proclist;
 }
+
+#undef MIB_INITIALIZER
+#undef PID_FROM_KINFO_PROC
 
 /* The WINDOWSNT implementation is in w32.c.
    The MSDOS implementation is in dosfns.c.  */

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: emacs-24.2.93 build problems

[Glenn Morris]

Jérémie Courrèges-Anglas wrote:

> So supporting list-system-processes (as-is) on OpenBSD for 24.3 leads to
> build failing with 1995/10/18 < OpenBSD < 2010/01/10 and to a useless
> (and potentially harmful) function with OpenBSD >= 2010/01/10.

I get the feeling you are probably right, but since no-one reported any
build problems in the year since since this was introduced, or through
several months of pretest (until just now), we can only conclude that
(almost) no-one is using latest Emacs on older OpenBSD, so it doesn't
matter that much in practice.

list-system-processes on OpenBSD [was Re: emacs-24.2.93 build problems]

[Glenn Morris]

Jérémie Courrèges-Anglas wrote:

> But as far as 24.3 is concerned, I *really* suggest making this
> function a stub on OpenBSD (just returning Qnil).

I'm inclined to agree with this. Any other opinions?

Ref
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11797
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=5725
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=5243

> Here's why:
>
> There are two different APIs involved:
> - old KERN_PROC, similar to NetBSD
> - new KERN_PROC, completely different
>
> Here's how they have evolved:
>
> 1. old KERN_PROC API present and visible (usable)
>    sysctl.h rev 1.1.1.1 (1995/10/18) -> rev 1.67 (2003/12/23)
>
> 2. new KERN_PROC API added as KERN_PROC2, old KERN_PROC still available
>    sysctl.h rev 1.68 (2004/01/07) -> rev 1.100 (2009/06/15)
>
> 3. old KERN_PROC becomes hidden behind #ifdef's
>    sysctl rev 1.101 (2010/01/10) -> rev 1.108 (2011/03/07)
>
> 4. old KERN_PROC is removed, the new KERN_PROC replaces it, KERN_PROC2
>    aliases still present for backward compat
>    sysctl.h rev 1.109 (2011/03/12) -> rev 1.119 (2011/12/14)
>
> 5. KERN_PROC2 aliases removed, only the new API remains
>    sysctl.h rev 1.120 (2012/01/07) -> now
>
>
> I've done some _light_ testing and here are the results:
>
> - with or without the patch adding fallback to KERN_PROC2, build fails
>   for cases 1 and 2. (This is because p_pid field wasn't present in
>   struct kinfo_proc at that time.)
>
> - with the patch, build doesn't fail for case 3
>
> - with the patch, cases 3, 4 and 5 will have list-system-processes
>   returning nil, since sysctl() is called with a wrong mib.
>
> So supporting list-system-processes (as-is) on OpenBSD for 24.3 leads to
> build failing with 1995/10/18 < OpenBSD < 2010/01/10 and to a useless
> (and potentially harmful) function with OpenBSD >= 2010/01/10.
>
> PS: I did not and I will not investigate the case of MirOS.

Re: list-system-processes on OpenBSD [was Re: emacs-24.2.93 build problems]

[Paul Eggert]

On 03/05/13 09:24, Glenn Morris wrote:
>> > But as far as 24.3 is concerned, I *really* suggest making this
>> > function a stub on OpenBSD (just returning Qnil).
> I'm inclined to agree with this.

Likewise.  MirBSD should be a stub too.

Re: emacs-24.2.93 build problems (on OpenBSD)

[Glenn Morris]

Jérémie Courrèges-Anglas wrote:

> By the way, I don't have the time to make a bugreport right now, but

http://debbugs.gnu.org/13881

Please send any further followup on this to 13881@debbugs.gnu.org.

Re: list-system-processes on OpenBSD

[Glenn Morris]

Paul Eggert wrote:

> On 03/05/13 09:24, Glenn Morris wrote:
>>> > But as far as 24.3 is concerned, I *really* suggest making this
>>> > function a stub on OpenBSD (just returning Qnil).
>> I'm inclined to agree with this.
>
> Likewise.  MirBSD should be a stub too.

If you know the right #ifdefs, could you go ahead and do this in
emacs-24, please?

AFAICS, it is supposed to work on FreeBSD, Mac OS X, ...?
But not on OpenBSD.
Not sure about NetBSD.

Re: list-system-processes on OpenBSD

[Paul Eggert]

On 03/05/13 12:42, Glenn Morris wrote:
> If you know the right #ifdefs, could you go ahead and do this in
> emacs-24, please?

Done, as emacs-24 bzr 111322.

Re: list-system-processes on OpenBSD

[Jérémie Courrèges-Anglas]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 03/05/13 12:42, Glenn Morris wrote:
>> If you know the right #ifdefs, could you go ahead and do this in
>> emacs-24, please?
>
> Done, as emacs-24 bzr 111322.

Nice, thanks.

-- 
Jérémie Courrèges-Anglas
GPG Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

[24.3] possible build failure for list-system-processes on NetBSD

[Jérémie Courrèges-Anglas]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]


Well, NetBSD might deserve the same treatment:

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/sys/sysctl.h?rev=1.205&content-type=text/x-cvsweb-markup&only_with_tag=MAIN

I'm keeping this in the same thread, feel free to break it you prefer.

-- 
Jérémie Courrèges-Anglas
GPG Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

[24.3] possible build failure for list-system-processes on DragonFlyBSD

[Jérémie Courrèges-Anglas]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]


Same problem with DragonFlyBSD, which doesn't have KERN_PROC_PROC but
KERN_PROC_ALL and has switched to a flat kinfo_proc structure.

sys/sysctl.h (KERN_PROC_ALL)
http://gitweb.dragonflybsd.org/dragonfly.git/blob/HEAD:/sys/sys/sysctl.h

sys/kinfo.h (struct kinfo_proc)
http://gitweb.dragonflybsd.org/dragonfly.git/blob/HEAD:/sys/sys/kinfo.h

"Change kinfo_proc interface between kernel and userland."
http://gitweb.dragonflybsd.org/dragonfly.git/commit/5dfd06ac148512faf075c4e399e8485fd955578f

Perhaps should the list-system-processes function be enabled only on
targets where it is known to work?
-- 
Jérémie Courrèges-Anglas
GPG Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: list-system-processes on OpenBSD [was Re: emacs-24.2.93 build problems]

[Leo Liu]

On 2013-03-06 01:24 +0800, Glenn Morris wrote:
>> But as far as 24.3 is concerned, I *really* suggest making this
>> function a stub on OpenBSD (just returning Qnil).
>
> I'm inclined to agree with this. Any other opinions?
>
> Ref
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11797
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=5725
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=5243

Sounds like a good plan since it makes openbsd users happy.

Leo

Re: [24.3] possible build failure for list-system-processes on DragonFlyBSD

[Glenn Morris]

Jérémie Courrèges-Anglas wrote:

> Perhaps should the list-system-processes function be enabled only on
> targets where it is known to work?

What a crazy idea...

AFAICS, it was tested on FreeBSD and Max OS X?
So yes, as far as I'm concerned it can be restricted to just those rather
than BSD_SYSTEM.

Re: [24.3] possible build failure for list-system-processes on DragonFlyBSD

[Paul Eggert]

On 03/05/2013 07:12 PM, Glenn Morris wrote:
> AFAICS, it was tested on FreeBSD and Max OS X?
> So yes, as far as I'm concerned it can be restricted to just those rather
> than BSD_SYSTEM.

From what I can gather,
the NetBSD problem will come up only in the next NetBSD release, as
the current release should be OK; still, if nobody's tested it we're
probably better off omitting it for now.

How about the following patch?

=== modified file 'src/sysdep.c'
--- src/sysdep.c	2013-03-05 21:56:36 +0000
+++ src/sysdep.c	2013-03-06 05:02:30 +0000
@@ -2647,12 +2647,12 @@
   return proclist;
 }
 
-#elif defined BSD_SYSTEM && !defined __OpenBSD__ && !defined __MirBSD__
+#elif defined DARWIN_OS || defined __FreeBSD__
 
 Lisp_Object
 list_system_processes (void)
 {
-#if defined DARWIN_OS || defined __NetBSD__
+#ifdef DARWIN_OS
   int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_ALL};
 #else
   int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_PROC};
@@ -2678,7 +2678,7 @@
   len /= sizeof (struct kinfo_proc);
   for (i = 0; i < len; i++)
     {
-#if defined DARWIN_OS || defined __NetBSD__
+#ifdef DARWIN_OS
       proclist = Fcons (make_fixnum_or_float (procs[i].kp_proc.p_pid), proclist);
 #else
       proclist = Fcons (make_fixnum_or_float (procs[i].ki_pid), proclist);

Re: [24.3] possible build failure for list-system-processes on DragonFlyBSD

[Glenn Morris]

Paul Eggert wrote:

> How about the following patch?

Looks good to me, thanks.