[PATCH] support for accessing CPU/core count (processor-count)

[PATCH] support for accessing CPU/core count (processor-count)

[Campbell Barton]

[-- Attachment #1: Type: text/plain, Size: 577 bytes --]

Hi, this patch adds support for accessing the number of CPU's / cores on 
a system, matching CPython's multiprocessing.cpu_count() [0].

I've only tested this for Linux, this includes code that should work on 
other platforms, although that would need to be double-checked of course.
For reference I checked CPython / Blender & Stack-overflow [1]

Accessing this information can be useful to automatically detect the 
number of jobs to run.


[0]: 
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count
[1]: https://stackoverflow.com/a/3006416/432509

[-- Attachment #2: 0001-processor-count.patch --]
[-- Type: text/x-patch, Size: 1664 bytes --]

commit 7be53f10f3df3c3183cc97d6bbadb78ebb61e8d2
Author: Campbell Barton <ideasman42@gmail.com>
Date:   Sun Oct 10 10:16:47 2021 +1100

    Support accessing the number of CPU's.

    Add (processor-count) for accessing the number of cores/CPU's.

diff --git a/src/emacs.c b/src/emacs.c
index 866e43fda9..26e2f6b1f2 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -3156,6 +3156,38 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0,
   return Qt;
 }

+DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
+       doc: /* Return the number of CPUs in the system.
+
+The value will always be above zero, 1 for unsupported systems.  */)
+  (void)
+{
+  int nproc = -1;
+#ifdef WINDOWSNT
+  SYSTEM_INFO info;
+  GetSystemInfo(&info);
+  nproc = (int)info.dwNumberOfProcessors;
+#elif defined (__APPLE__) || \
+      defined (__OpenBSD__) || \
+      defined (__FreeBSD__) || \
+      defined (__NetBSD__) || \
+      defined (__DragonFly__)
+  int mib[2];
+  size_t len;
+
+  mib[0] = CTL_HW;
+  mib[1] = HW_NCPU;
+  len = sizeof(nproc);
+  sysctl(mib, 2, &nproc, &len, nullptr, 0);
+#elif defined (__hpux)
+  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
+#elif defined (_SC_NPROCESSORS_ONLN)
+  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
+#endif
+
+  return make_fixnum (MAX(nproc, 1));
+}
+
 void
 syms_of_emacs (void)
 {
@@ -3176,6 +3208,7 @@ syms_of_emacs (void)
   defsubr (&Sinvocation_directory);
   defsubr (&Sdaemonp);
   defsubr (&Sdaemon_initialized);
+  defsubr (&Sprocessor_count);

   DEFVAR_LISP ("command-line-args", Vcommand_line_args,
 	       doc: /* Args passed by shell to Emacs, as a list of strings.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

[-- Attachment #1: Type: text/plain, Size: 3265 bytes --]


Campbell Barton <ideasman42@gmail.com> writes:

> Hi, this patch adds support for accessing the number of CPU's / cores
> on a system, matching CPython's multiprocessing.cpu_count() [0].
>
> I've only tested this for Linux, this includes code that should work
> on other platforms, although that would need to be double-checked of
> course.
> For reference I checked CPython / Blender & Stack-overflow [1]

I can confirm it works on OpenBSD too, but needs tweaking.  Some
comments inline and attaching an updated patch

> Accessing this information can be useful to automatically detect the
> number of jobs to run.
>
>
> [0]:
> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count
> [1]: https://stackoverflow.com/a/3006416/432509
>
> commit 7be53f10f3df3c3183cc97d6bbadb78ebb61e8d2
> Author: Campbell Barton <ideasman42@gmail.com>
> Date:   Sun Oct 10 10:16:47 2021 +1100
> 
>     Support accessing the number of CPU's.
> 
>     Add (processor-count) for accessing the number of cores/CPU's.
> 
> diff --git a/src/emacs.c b/src/emacs.c
> index 866e43fda9..26e2f6b1f2 100644
> --- a/src/emacs.c
> +++ b/src/emacs.c
> @@ -3156,6 +3156,38 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0,
>    return Qt;
>  }
> 
> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
> +       doc: /* Return the number of CPUs in the system.
> +
> +The value will always be above zero, 1 for unsupported systems.  */)
> +  (void)
> +{
> +  int nproc = -1;
> +#ifdef WINDOWSNT
> +  SYSTEM_INFO info;
> +  GetSystemInfo(&info);
> +  nproc = (int)info.dwNumberOfProcessors;
> +#elif defined (__APPLE__) || \
> +      defined (__OpenBSD__) || \
> +      defined (__FreeBSD__) || \
> +      defined (__NetBSD__) || \
> +      defined (__DragonFly__)
> +  int mib[2];
> +  size_t len;
> +
> +  mib[0] = CTL_HW;

the #include <sys/sysctl.h> is missing.

> +  mib[1] = HW_NCPU;

at least on OpenBSD this should be HW_NCPUONLINE.

OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
misleading.  For example, on my machine

	% uname -a
	OpenBSD venera 7.0 GENERIC.MP#221 amd64
	% sysctl hw.ncpu
	hw.ncpu=8
	% sysctl hw.ncpuonline
	hw.ncpuonline=4

and this has been the case for a while already (I mean, a couple of
years if not more.)

I don't have access to other BSDs other than OpenBSD, but judging from
the manpages online.

 - NetBSD has hw.ncpuonline
 - DragonFly and FreeBSD have only hw.ncpu
 - apple I don't know how to check.  man.apple.com doesn't seem to
   exists ^^"

> +  len = sizeof(nproc);
> +  sysctl(mib, 2, &nproc, &len, nullptr, 0);
                                  ^^^^^^^
shouldn't this be NULL?

> +#elif defined (__hpux)
> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
> +#elif defined (_SC_NPROCESSORS_ONLN)
> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
> +#endif
> +
> +  return make_fixnum (MAX(nproc, 1));

emacs.c:3188:23: warning: implicit declaration of function 'MAX' is invalid in C99 [-Wimplicit-function-declaration]

I'm attaching an updated patch.  Note that I'm not sure if including
sys/types.h and sys/sysctl.h breaks the build on some OS.

if someone knows how to reduce the number of #ifdefs I'll be glad :)

> +}
> +

Cheers,

Omar Polo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-add-function-to-access-the-number-of-CPU.patch --]
[-- Type: text/x-patch, Size: 2011 bytes --]

From 5fe26a73abc60eae7dd2528e3854f2f379bcb4c7 Mon Sep 17 00:00:00 2001
From: Campbell Barton <ideasman42@gmail.com>
Date: Sun, 10 Oct 2021 08:21:01 +0000
Subject: [PATCH] add function to access the number of CPU

* emacs.c (Fprocessor_count): add processor-count function to access
  the number of CPU
---
 src/emacs.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/src/emacs.c b/src/emacs.c
index 866e43fda9..5bd10a7586 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -25,6 +25,8 @@ along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.  */
 #include <fcntl.h>
 #include <stdlib.h>
 
+#include <sys/types.h>
+#include <sys/sysctl.h>
 #include <sys/file.h>
 #include <sys/stat.h>
 #include <unistd.h>
@@ -3156,6 +3158,38 @@ from the parent process and its tty file descriptors.  */)
   return Qt;
 }
 
+DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
+       doc: /* Return the number of CPUs in the system.
+
+The value will always be above zero, 1 for unsupported systems.  */)
+  (void)
+{
+  int nproc = -1;
+#ifdef WINDOWSNT
+  SYSTEM_INFO info;
+  GetSystemInfo(&info);
+  nproc = (int)info.dwNumberOfProcessors;
+#elif defined (HW_NCPU) || defined (HW_NCPUONLINE)
+  int mib[2];
+  size_t len;
+
+  mib[0] = CTL_HW;
+#ifdef HW_NCPUONLINE
+  mib[1] = HW_NCPUONLINE;
+#else
+  mib[1] = HW_NCPU;
+#endif
+  len = sizeof(nproc);
+  sysctl(mib, 2, &nproc, &len, NULL, 0);
+#elif defined (__hpux)
+  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
+#elif defined (_SC_NPROCESSORS_ONLN)
+  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
+#endif
+
+  return make_fixnum (nproc > 1 ? nproc : 1);
+}
+
 void
 syms_of_emacs (void)
 {
@@ -3176,6 +3210,7 @@ syms_of_emacs (void)
   defsubr (&Sinvocation_directory);
   defsubr (&Sdaemonp);
   defsubr (&Sdaemon_initialized);
+  defsubr (&Sprocessor_count);
 
   DEFVAR_LISP ("command-line-args", Vcommand_line_args,
 	       doc: /* Args passed by shell to Emacs, as a list of strings.
-- 
2.33.0

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Omar Polo <op@omarpolo.com> writes:

> Campbell Barton <ideasman42@gmail.com> writes:
>
>> Hi, this patch adds support for accessing the number of CPU's / cores
>> on a system, matching CPython's multiprocessing.cpu_count() [0].
>>
>> I've only tested this for Linux, this includes code that should work
>> on other platforms, although that would need to be double-checked of
>> course.
>> For reference I checked CPython / Blender & Stack-overflow [1]
>
> I can confirm it works on OpenBSD too, but needs tweaking.  Some
> comments inline and attaching an updated patch
>
>> Accessing this information can be useful to automatically detect the
>> number of jobs to run.
>>
>>
>> [0]:
>> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count
>> [1]: https://stackoverflow.com/a/3006416/432509
>>
>> commit 7be53f10f3df3c3183cc97d6bbadb78ebb61e8d2
>> Author: Campbell Barton <ideasman42@gmail.com>
>> Date:   Sun Oct 10 10:16:47 2021 +1100
>> 
>>     Support accessing the number of CPU's.
>> 
>>     Add (processor-count) for accessing the number of cores/CPU's.
>> 
>> diff --git a/src/emacs.c b/src/emacs.c
>> index 866e43fda9..26e2f6b1f2 100644
>> --- a/src/emacs.c
>> +++ b/src/emacs.c
>> @@ -3156,6 +3156,38 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0,
>>    return Qt;
>>  }
>> 
>> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
>> +       doc: /* Return the number of CPUs in the system.
>> +
>> +The value will always be above zero, 1 for unsupported systems.  */)
>> +  (void)
>> +{
>> +  int nproc = -1;
>> +#ifdef WINDOWSNT
>> +  SYSTEM_INFO info;
>> +  GetSystemInfo(&info);
>> +  nproc = (int)info.dwNumberOfProcessors;
>> +#elif defined (__APPLE__) || \
>> +      defined (__OpenBSD__) || \
>> +      defined (__FreeBSD__) || \
>> +      defined (__NetBSD__) || \
>> +      defined (__DragonFly__)
>> +  int mib[2];
>> +  size_t len;
>> +
>> +  mib[0] = CTL_HW;
>
> the #include <sys/sysctl.h> is missing.
>
>> +  mib[1] = HW_NCPU;
>
> at least on OpenBSD this should be HW_NCPUONLINE.
>
> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
> misleading.  For example, on my machine
>
> 	% uname -a
> 	OpenBSD venera 7.0 GENERIC.MP#221 amd64
> 	% sysctl hw.ncpu
> 	hw.ncpu=8
> 	% sysctl hw.ncpuonline
> 	hw.ncpuonline=4
>
> and this has been the case for a while already (I mean, a couple of
> years if not more.)
>
> I don't have access to other BSDs other than OpenBSD, but judging from
> the manpages online.
>
>  - NetBSD has hw.ncpuonline
>  - DragonFly and FreeBSD have only hw.ncpu
>  - apple I don't know how to check.  man.apple.com doesn't seem to
>    exists ^^"
>
>> +  len = sizeof(nproc);
>> +  sysctl(mib, 2, &nproc, &len, nullptr, 0);
>                                   ^^^^^^^
> shouldn't this be NULL?
>
>> +#elif defined (__hpux)
>> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
>> +#elif defined (_SC_NPROCESSORS_ONLN)
>> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
>> +#endif
>> +
>> +  return make_fixnum (MAX(nproc, 1));
>
> emacs.c:3188:23: warning: implicit declaration of function 'MAX' is invalid in C99 [-Wimplicit-function-declaration]
>
> I'm attaching an updated patch.  Note that I'm not sure if including
> sys/types.h and sys/sysctl.h breaks the build on some OS.
>
> if someone knows how to reduce the number of #ifdefs I'll be glad :)
>
>> +}
>> +
>
> Cheers,
>
> Omar Polo
>
> From 5fe26a73abc60eae7dd2528e3854f2f379bcb4c7 Mon Sep 17 00:00:00 2001
> From: Campbell Barton <ideasman42@gmail.com>
> Date: Sun, 10 Oct 2021 08:21:01 +0000
> Subject: [PATCH] add function to access the number of CPU
>
> * emacs.c (Fprocessor_count): add processor-count function to access
>   the number of CPU
> ---
>  src/emacs.c | 35 +++++++++++++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
>
> diff --git a/src/emacs.c b/src/emacs.c
> index 866e43fda9..5bd10a7586 100644
> --- a/src/emacs.c
> +++ b/src/emacs.c
> @@ -25,6 +25,8 @@ along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.  */
>  #include <fcntl.h>
>  #include <stdlib.h>
>  
> +#include <sys/types.h>
> +#include <sys/sysctl.h>
>  #include <sys/file.h>
>  #include <sys/stat.h>
>  #include <unistd.h>
> @@ -3156,6 +3158,38 @@ from the parent process and its tty file descriptors.  */)
>    return Qt;
>  }
>  
> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
> +       doc: /* Return the number of CPUs in the system.
> +
> +The value will always be above zero, 1 for unsupported systems.  */)
> +  (void)
> +{
> +  int nproc = -1;
> +#ifdef WINDOWSNT
> +  SYSTEM_INFO info;
> +  GetSystemInfo(&info);
> +  nproc = (int)info.dwNumberOfProcessors;
> +#elif defined (HW_NCPU) || defined (HW_NCPUONLINE)
> +  int mib[2];
> +  size_t len;
> +
> +  mib[0] = CTL_HW;
> +#ifdef HW_NCPUONLINE
> +  mib[1] = HW_NCPUONLINE;
> +#else
> +  mib[1] = HW_NCPU;
> +#endif
> +  len = sizeof(nproc);
> +  sysctl(mib, 2, &nproc, &len, NULL, 0);
> +#elif defined (__hpux)
> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
> +#elif defined (_SC_NPROCESSORS_ONLN)
> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
> +#endif
> +
> +  return make_fixnum (nproc > 1 ? nproc : 1);
> +}
> +
>  void
>  syms_of_emacs (void)
>  {
> @@ -3176,6 +3210,7 @@ syms_of_emacs (void)
>    defsubr (&Sinvocation_directory);
>    defsubr (&Sdaemonp);
>    defsubr (&Sdaemon_initialized);
> +  defsubr (&Sprocessor_count);
>  
>    DEFVAR_LISP ("command-line-args", Vcommand_line_args,
>  	       doc: /* Args passed by shell to Emacs, as a list of strings.

What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?

Windows has "wmic cpu get NumberOfCores,NumberOfLogicalProcessors" which works
on cmd prompt, which means executable from elisp as a process. I dont know for
apple and *bsds.

It is just text that can be parsed directly with elisp, so it is easier to
maintain does not need to be compiled etc. I don't see cpu queries as a
performance crucial query. Just a thought.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Arthur Miller <arthur.miller@live.com> writes:

> Omar Polo <op@omarpolo.com> writes:
>
>> Campbell Barton <ideasman42@gmail.com> writes:
>>
>>> Hi, this patch adds support for accessing the number of CPU's / cores
>>> on a system, matching CPython's multiprocessing.cpu_count() [0].
>>>
>>> I've only tested this for Linux, this includes code that should work
>>> on other platforms, although that would need to be double-checked of
>>> course.
>>> For reference I checked CPython / Blender & Stack-overflow [1]
>>
>> I can confirm it works on OpenBSD too, but needs tweaking.  Some
>> comments inline and attaching an updated patch
>>
>>> Accessing this information can be useful to automatically detect the
>>> number of jobs to run.
>>>
>>>
>>> [0]:
>>> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count
>>> [1]: https://stackoverflow.com/a/3006416/432509
>>>
>>> commit 7be53f10f3df3c3183cc97d6bbadb78ebb61e8d2
>>> Author: Campbell Barton <ideasman42@gmail.com>
>>> Date:   Sun Oct 10 10:16:47 2021 +1100
>>> 
>>>     Support accessing the number of CPU's.
>>> 
>>>     Add (processor-count) for accessing the number of cores/CPU's.
>>> 
>>> diff --git a/src/emacs.c b/src/emacs.c
>>> index 866e43fda9..26e2f6b1f2 100644
>>> --- a/src/emacs.c
>>> +++ b/src/emacs.c
>>> @@ -3156,6 +3156,38 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0,
>>>    return Qt;
>>>  }
>>> 
>>> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
>>> +       doc: /* Return the number of CPUs in the system.
>>> +
>>> +The value will always be above zero, 1 for unsupported systems.  */)
>>> +  (void)
>>> +{
>>> +  int nproc = -1;
>>> +#ifdef WINDOWSNT
>>> +  SYSTEM_INFO info;
>>> +  GetSystemInfo(&info);
>>> +  nproc = (int)info.dwNumberOfProcessors;
>>> +#elif defined (__APPLE__) || \
>>> +      defined (__OpenBSD__) || \
>>> +      defined (__FreeBSD__) || \
>>> +      defined (__NetBSD__) || \
>>> +      defined (__DragonFly__)
>>> +  int mib[2];
>>> +  size_t len;
>>> +
>>> +  mib[0] = CTL_HW;
>>
>> the #include <sys/sysctl.h> is missing.
>>
>>> +  mib[1] = HW_NCPU;
>>
>> at least on OpenBSD this should be HW_NCPUONLINE.
>>
>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>> misleading.  For example, on my machine
>>
>> 	% uname -a
>> 	OpenBSD venera 7.0 GENERIC.MP#221 amd64
>> 	% sysctl hw.ncpu
>> 	hw.ncpu=8
>> 	% sysctl hw.ncpuonline
>> 	hw.ncpuonline=4
>>
>> and this has been the case for a while already (I mean, a couple of
>> years if not more.)
>>
>> I don't have access to other BSDs other than OpenBSD, but judging from
>> the manpages online.
>>
>>  - NetBSD has hw.ncpuonline
>>  - DragonFly and FreeBSD have only hw.ncpu
>>  - apple I don't know how to check.  man.apple.com doesn't seem to
>>    exists ^^"
>>
>>> +  len = sizeof(nproc);
>>> +  sysctl(mib, 2, &nproc, &len, nullptr, 0);
>>                                   ^^^^^^^
>> shouldn't this be NULL?
>>
>>> +#elif defined (__hpux)
>>> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
>>> +#elif defined (_SC_NPROCESSORS_ONLN)
>>> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
>>> +#endif
>>> +
>>> +  return make_fixnum (MAX(nproc, 1));
>>
>> emacs.c:3188:23: warning: implicit declaration of function 'MAX' is invalid in C99 [-Wimplicit-function-declaration]
>>
>> I'm attaching an updated patch.  Note that I'm not sure if including
>> sys/types.h and sys/sysctl.h breaks the build on some OS.
>>
>> if someone knows how to reduce the number of #ifdefs I'll be glad :)
>>
>>> +}
>>> +
>>
>> Cheers,
>>
>> Omar Polo
>>
>> From 5fe26a73abc60eae7dd2528e3854f2f379bcb4c7 Mon Sep 17 00:00:00 2001
>> From: Campbell Barton <ideasman42@gmail.com>
>> Date: Sun, 10 Oct 2021 08:21:01 +0000
>> Subject: [PATCH] add function to access the number of CPU
>>
>> * emacs.c (Fprocessor_count): add processor-count function to access
>>   the number of CPU
>> ---
>>  src/emacs.c | 35 +++++++++++++++++++++++++++++++++++
>>  1 file changed, 35 insertions(+)
>>
>> diff --git a/src/emacs.c b/src/emacs.c
>> index 866e43fda9..5bd10a7586 100644
>> --- a/src/emacs.c
>> +++ b/src/emacs.c
>> @@ -25,6 +25,8 @@ along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.  */
>>  #include <fcntl.h>
>>  #include <stdlib.h>
>>  
>> +#include <sys/types.h>
>> +#include <sys/sysctl.h>
>>  #include <sys/file.h>
>>  #include <sys/stat.h>
>>  #include <unistd.h>
>> @@ -3156,6 +3158,38 @@ from the parent process and its tty file descriptors.  */)
>>    return Qt;
>>  }
>>  
>> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
>> +       doc: /* Return the number of CPUs in the system.
>> +
>> +The value will always be above zero, 1 for unsupported systems.  */)
>> +  (void)
>> +{
>> +  int nproc = -1;
>> +#ifdef WINDOWSNT
>> +  SYSTEM_INFO info;
>> +  GetSystemInfo(&info);
>> +  nproc = (int)info.dwNumberOfProcessors;
>> +#elif defined (HW_NCPU) || defined (HW_NCPUONLINE)
>> +  int mib[2];
>> +  size_t len;
>> +
>> +  mib[0] = CTL_HW;
>> +#ifdef HW_NCPUONLINE
>> +  mib[1] = HW_NCPUONLINE;
>> +#else
>> +  mib[1] = HW_NCPU;
>> +#endif
>> +  len = sizeof(nproc);
>> +  sysctl(mib, 2, &nproc, &len, NULL, 0);
>> +#elif defined (__hpux)
>> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
>> +#elif defined (_SC_NPROCESSORS_ONLN)
>> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
>> +#endif
>> +
>> +  return make_fixnum (nproc > 1 ? nproc : 1);
>> +}
>> +
>>  void
>>  syms_of_emacs (void)
>>  {
>> @@ -3176,6 +3210,7 @@ syms_of_emacs (void)
>>    defsubr (&Sinvocation_directory);
>>    defsubr (&Sdaemonp);
>>    defsubr (&Sdaemon_initialized);
>> +  defsubr (&Sprocessor_count);
>>  
>>    DEFVAR_LISP ("command-line-args", Vcommand_line_args,
>>  	       doc: /* Args passed by shell to Emacs, as a list of strings.
>
> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?
>
> Windows has "wmic cpu get NumberOfCores,NumberOfLogicalProcessors" which works
> on cmd prompt, which means executable from elisp as a process. I dont know for
> apple and *bsds.
>
> It is just text that can be parsed directly with elisp, so it is easier to
> maintain does not need to be compiled etc. I don't see cpu queries as a
> performance crucial query. Just a thought.

You're right

	syscstl -n hw.ncpuonline

would do it.  I guess that a rationale for doing this in C would be to
make it easier on the elisp side to use it.  We can try to support every
OS here instead of leaving the burden on the elisp package authors.

(the same thing applies to proced.  You could implement it by parsing ps
output I guess, but there are elisp API that calls into C for that.)

But I don't know, I don't really have strong opinions.  I've read a bit
of code that didn't look right and tried to improve it :)

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Omar Polo <op@omarpolo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Omar Polo <op@omarpolo.com> writes:
>>
>>> Campbell Barton <ideasman42@gmail.com> writes:
>>>
>>>> Hi, this patch adds support for accessing the number of CPU's / cores
>>>> on a system, matching CPython's multiprocessing.cpu_count() [0].
>>>>
>>>> I've only tested this for Linux, this includes code that should work
>>>> on other platforms, although that would need to be double-checked of
>>>> course.
>>>> For reference I checked CPython / Blender & Stack-overflow [1]
>>>
>>> I can confirm it works on OpenBSD too, but needs tweaking.  Some
>>> comments inline and attaching an updated patch
>>>
>>>> Accessing this information can be useful to automatically detect the
>>>> number of jobs to run.
>>>>
>>>>
>>>> [0]:
>>>> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count
>>>> [1]: https://stackoverflow.com/a/3006416/432509
>>>>
>>>> commit 7be53f10f3df3c3183cc97d6bbadb78ebb61e8d2
>>>> Author: Campbell Barton <ideasman42@gmail.com>
>>>> Date:   Sun Oct 10 10:16:47 2021 +1100
>>>> 
>>>>     Support accessing the number of CPU's.
>>>> 
>>>>     Add (processor-count) for accessing the number of cores/CPU's.
>>>> 
>>>> diff --git a/src/emacs.c b/src/emacs.c
>>>> index 866e43fda9..26e2f6b1f2 100644
>>>> --- a/src/emacs.c
>>>> +++ b/src/emacs.c
>>>> @@ -3156,6 +3156,38 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0,
>>>>    return Qt;
>>>>  }
>>>> 
>>>> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
>>>> +       doc: /* Return the number of CPUs in the system.
>>>> +
>>>> +The value will always be above zero, 1 for unsupported systems.  */)
>>>> +  (void)
>>>> +{
>>>> +  int nproc = -1;
>>>> +#ifdef WINDOWSNT
>>>> +  SYSTEM_INFO info;
>>>> +  GetSystemInfo(&info);
>>>> +  nproc = (int)info.dwNumberOfProcessors;
>>>> +#elif defined (__APPLE__) || \
>>>> +      defined (__OpenBSD__) || \
>>>> +      defined (__FreeBSD__) || \
>>>> +      defined (__NetBSD__) || \
>>>> +      defined (__DragonFly__)
>>>> +  int mib[2];
>>>> +  size_t len;
>>>> +
>>>> +  mib[0] = CTL_HW;
>>>
>>> the #include <sys/sysctl.h> is missing.
>>>
>>>> +  mib[1] = HW_NCPU;
>>>
>>> at least on OpenBSD this should be HW_NCPUONLINE.
>>>
>>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>>> misleading.  For example, on my machine
>>>
>>> 	% uname -a
>>> 	OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>> 	% sysctl hw.ncpu
>>> 	hw.ncpu=8
>>> 	% sysctl hw.ncpuonline
>>> 	hw.ncpuonline=4
>>>
>>> and this has been the case for a while already (I mean, a couple of
>>> years if not more.)
>>>
>>> I don't have access to other BSDs other than OpenBSD, but judging from
>>> the manpages online.
>>>
>>>  - NetBSD has hw.ncpuonline
>>>  - DragonFly and FreeBSD have only hw.ncpu
>>>  - apple I don't know how to check.  man.apple.com doesn't seem to
>>>    exists ^^"
>>>
>>>> +  len = sizeof(nproc);
>>>> +  sysctl(mib, 2, &nproc, &len, nullptr, 0);
>>>                                   ^^^^^^^
>>> shouldn't this be NULL?
>>>
>>>> +#elif defined (__hpux)
>>>> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
>>>> +#elif defined (_SC_NPROCESSORS_ONLN)
>>>> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
>>>> +#endif
>>>> +
>>>> +  return make_fixnum (MAX(nproc, 1));
>>>
>>> emacs.c:3188:23: warning: implicit declaration of function 'MAX' is invalid in C99 [-Wimplicit-function-declaration]
>>>
>>> I'm attaching an updated patch.  Note that I'm not sure if including
>>> sys/types.h and sys/sysctl.h breaks the build on some OS.
>>>
>>> if someone knows how to reduce the number of #ifdefs I'll be glad :)
>>>
>>>> +}
>>>> +
>>>
>>> Cheers,
>>>
>>> Omar Polo
>>>
>>> From 5fe26a73abc60eae7dd2528e3854f2f379bcb4c7 Mon Sep 17 00:00:00 2001
>>> From: Campbell Barton <ideasman42@gmail.com>
>>> Date: Sun, 10 Oct 2021 08:21:01 +0000
>>> Subject: [PATCH] add function to access the number of CPU
>>>
>>> * emacs.c (Fprocessor_count): add processor-count function to access
>>>   the number of CPU
>>> ---
>>>  src/emacs.c | 35 +++++++++++++++++++++++++++++++++++
>>>  1 file changed, 35 insertions(+)
>>>
>>> diff --git a/src/emacs.c b/src/emacs.c
>>> index 866e43fda9..5bd10a7586 100644
>>> --- a/src/emacs.c
>>> +++ b/src/emacs.c
>>> @@ -25,6 +25,8 @@ along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.  */
>>>  #include <fcntl.h>
>>>  #include <stdlib.h>
>>>  
>>> +#include <sys/types.h>
>>> +#include <sys/sysctl.h>
>>>  #include <sys/file.h>
>>>  #include <sys/stat.h>
>>>  #include <unistd.h>
>>> @@ -3156,6 +3158,38 @@ from the parent process and its tty file descriptors.  */)
>>>    return Qt;
>>>  }
>>>  
>>> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
>>> +       doc: /* Return the number of CPUs in the system.
>>> +
>>> +The value will always be above zero, 1 for unsupported systems.  */)
>>> +  (void)
>>> +{
>>> +  int nproc = -1;
>>> +#ifdef WINDOWSNT
>>> +  SYSTEM_INFO info;
>>> +  GetSystemInfo(&info);
>>> +  nproc = (int)info.dwNumberOfProcessors;
>>> +#elif defined (HW_NCPU) || defined (HW_NCPUONLINE)
>>> +  int mib[2];
>>> +  size_t len;
>>> +
>>> +  mib[0] = CTL_HW;
>>> +#ifdef HW_NCPUONLINE
>>> +  mib[1] = HW_NCPUONLINE;
>>> +#else
>>> +  mib[1] = HW_NCPU;
>>> +#endif
>>> +  len = sizeof(nproc);
>>> +  sysctl(mib, 2, &nproc, &len, NULL, 0);
>>> +#elif defined (__hpux)
>>> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
>>> +#elif defined (_SC_NPROCESSORS_ONLN)
>>> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
>>> +#endif
>>> +
>>> +  return make_fixnum (nproc > 1 ? nproc : 1);
>>> +}
>>> +
>>>  void
>>>  syms_of_emacs (void)
>>>  {
>>> @@ -3176,6 +3210,7 @@ syms_of_emacs (void)
>>>    defsubr (&Sinvocation_directory);
>>>    defsubr (&Sdaemonp);
>>>    defsubr (&Sdaemon_initialized);
>>> +  defsubr (&Sprocessor_count);
>>>  
>>>    DEFVAR_LISP ("command-line-args", Vcommand_line_args,
>>>  	       doc: /* Args passed by shell to Emacs, as a list of strings.
>>
>> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?
>>
>> Windows has "wmic cpu get NumberOfCores,NumberOfLogicalProcessors" which works
>> on cmd prompt, which means executable from elisp as a process. I dont know for
>> apple and *bsds.
>>
>> It is just text that can be parsed directly with elisp, so it is easier to
>> maintain does not need to be compiled etc. I don't see cpu queries as a
>> performance crucial query. Just a thought.
>
> You're right
>
> 	syscstl -n hw.ncpuonline
>
> would do it.  I guess that a rationale for doing this in C would be to
> make it easier on the elisp side to use it.  We can try to support every
> OS here instead of leaving the burden on the elisp package authors.
>
> (the same thing applies to proced.  You could implement it by parsing ps
> output I guess, but there are elisp API that calls into C for that.)
>
> But I don't know, I don't really have strong opinions.  I've read a bit
> of code that didn't look right and tried to improve it :)

By the way; I just realize also that native compiler does this, and there is
also a note about exporting already written c function to lisp:

#begin_src emacs-lisp
(declare-function w32-get-nproc "w32.c")
(defvar comp-num-cpus nil)
(defun comp-effective-async-max-jobs ()
  "Compute the effective number of async jobs."
  (if (zerop native-comp-async-jobs-number)
      (or comp-num-cpus
          (setf comp-num-cpus
                ;; FIXME: we already have a function to determine
                ;; the number of processors, see get_native_system_info in w32.c.
                ;; The result needs to be exported to Lisp.
                (max 1 (/ (cond ((eq 'windows-nt system-type)
                                 (w32-get-nproc))
                                ((executable-find "nproc")
                                 (string-to-number
                                  (shell-command-to-string "nproc")))
                                ((eq 'berkeley-unix system-type)
                                 (string-to-number
                                  (shell-command-to-string "sysctl -n hw.ncpu")))
                                (t 1))
                          2))))
    native-comp-async-jobs-number))
#end_src

Maybe you can just extract the cpu number part from Andrea's function and rename
into some public API independent of native comp?

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Arthur Miller <arthur.miller@live.com> writes:

> Omar Polo <op@omarpolo.com> writes:
>[...]
>>> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?
>>>
>>> Windows has "wmic cpu get NumberOfCores,NumberOfLogicalProcessors" which works
>>> on cmd prompt, which means executable from elisp as a process. I dont know for
>>> apple and *bsds.
>>>
>>> It is just text that can be parsed directly with elisp, so it is easier to
>>> maintain does not need to be compiled etc. I don't see cpu queries as a
>>> performance crucial query. Just a thought.
>>
>> You're right
>>
>> 	syscstl -n hw.ncpuonline
>>
>> would do it.  I guess that a rationale for doing this in C would be to
>> make it easier on the elisp side to use it.  We can try to support every
>> OS here instead of leaving the burden on the elisp package authors.
>>
>> (the same thing applies to proced.  You could implement it by parsing ps
>> output I guess, but there are elisp API that calls into C for that.)
>>
>> But I don't know, I don't really have strong opinions.  I've read a bit
>> of code that didn't look right and tried to improve it :)
>
> By the way; I just realize also that native compiler does this, and there is
> also a note about exporting already written c function to lisp:
>
> #begin_src emacs-lisp
> (declare-function w32-get-nproc "w32.c")
> (defvar comp-num-cpus nil)
> (defun comp-effective-async-max-jobs ()
>   "Compute the effective number of async jobs."
>   (if (zerop native-comp-async-jobs-number)
>       (or comp-num-cpus
>           (setf comp-num-cpus
>                 ;; FIXME: we already have a function to determine
>                 ;; the number of processors, see get_native_system_info in w32.c.
>                 ;; The result needs to be exported to Lisp.
>                 (max 1 (/ (cond ((eq 'windows-nt system-type)
>                                  (w32-get-nproc))
>                                 ((executable-find "nproc")
>                                  (string-to-number
>                                   (shell-command-to-string "nproc")))
>                                 ((eq 'berkeley-unix system-type)
>                                  (string-to-number
>                                   (shell-command-to-string "sysctl -n hw.ncpu")))
>                                 (t 1))
>                           2))))
>     native-comp-async-jobs-number))
> #end_src
>
> Maybe you can just extract the cpu number part from Andrea's function and rename
> into some public API independent of native comp?

Thanks for bringing up that bit, as is something that would need to be
tweaked on OpenBSD (refer to my previous mail regarding ncpu vs
ncpuonline.)

Hope to remember to tweak this next time I'll try to get nativecomp on
OpenBSD...

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Omar Polo <op@omarpolo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Omar Polo <op@omarpolo.com> writes:
>>[...]
>>>> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?
>>>>
>>>> Windows has "wmic cpu get NumberOfCores,NumberOfLogicalProcessors" which works
>>>> on cmd prompt, which means executable from elisp as a process. I dont know for
>>>> apple and *bsds.
>>>>
>>>> It is just text that can be parsed directly with elisp, so it is easier to
>>>> maintain does not need to be compiled etc. I don't see cpu queries as a
>>>> performance crucial query. Just a thought.
>>>
>>> You're right
>>>
>>> 	syscstl -n hw.ncpuonline
>>>
>>> would do it.  I guess that a rationale for doing this in C would be to
>>> make it easier on the elisp side to use it.
There is no difference in usage on Lisp side if you implement a function as a C
built-in or a Lisp function.

However, difference in maintaining it is huge. For any change, entire Emacs has
to be recompiled, but if you do it in Lisp you can tweak it whenever, advice it,
etc.

I can't imagine number of CPUs change under the running Emacs process; at least
not on platforms I am aware off, so quering cpu number will probably be done in
some 'with-eval-after-load', i.e. not performance critical code.

>>> make it easier on the elisp side to use it.  We can try to support every
>>> OS here instead of leaving the burden on the elisp package authors.
Sorry for answering this as aftertought; I think there is a little bit of
missunderstanding going on. What I meant, is not that every user should do this
on their own. I just meant you should implement this in Lisp rather than in
C. An API to query number of physical cores is definitely a good thing to have.

>>> (the same thing applies to proced.  You could implement it by parsing ps
>> make it easier on the elisp side to use it.  We can try to support every
>>> output I guess, but there are elisp API that calls into C for that.)
>>>
>>> But I don't know, I don't really have strong opinions.  I've read a bit
>>> of code that didn't look right and tried to improve it :)
>>
>> By the way; I just realize also that native compiler does this, and there is
>> also a note about exporting already written c function to lisp:
>>
>> #begin_src emacs-lisp
>> (declare-function w32-get-nproc "w32.c")
>> (defvar comp-num-cpus nil)
>> (defun comp-effective-async-max-jobs ()
>>   "Compute the effective number of async jobs."
>>   (if (zerop native-comp-async-jobs-number)
>>       (or comp-num-cpus
>>           (setf comp-num-cpus
>>                 ;; FIXME: we already have a function to determine
>>                 ;; the number of processors, see get_native_system_info in w32.c.
>>                 ;; The result needs to be exported to Lisp.
>>                 (max 1 (/ (cond ((eq 'windows-nt system-type)
>>                                  (w32-get-nproc))
>>                                 ((executable-find "nproc")
>>                                  (string-to-number
>>                                   (shell-command-to-string "nproc")))
>>                                 ((eq 'berkeley-unix system-type)
>>                                  (string-to-number
>>                                   (shell-command-to-string "sysctl -n hw.ncpu")))
>>                                 (t 1))
>>                           2))))
>>     native-comp-async-jobs-number))
>> #end_src
>>
>> Maybe you can just extract the cpu number part from Andrea's function and rename
>> into some public API independent of native comp?
>
> Thanks for bringing up that bit, as is something that would need to be
> tweaked on OpenBSD (refer to my previous mail regarding ncpu vs
> ncpuonline.)
>
> Hope to remember to tweak this next time I'll try to get nativecomp on
> OpenBSD...
And that is the beauty of having it as a Lisp function. You can just tweak it,
don't need to recompile entire Emacs :). 

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Arthur Miller <arthur.miller@live.com> writes:

> [...]
> And that is the beauty of having it as a Lisp function. You can just tweak it,
> don't need to recompile entire Emacs :).

I know I'm getting off-topic, but I just don't understand your point.  I
don't see how spawning a bunch of commands, checking their return code
and parsing their output is better than a couple of lines of C that do
the right thing depending on the platform (decided at compile time!) and
get directly an int.

I love lisp, don't get me wrong, and I actually prefer writing elisp
rather than following the GNU C coding style (I love C too but GNU style
hurts my eyes.)

Sure, checking the number of cpus is not something that is done a lot,
and I can't imagine a situation where it would be a bottleneck, but on
the other hand, for the same argument, it's not something that needs to
be tweaked often, so the "recompile emacs" argument doesn't apply
really.  Better still, as Stefan suggested, we could reuse the glib
function which is better than the OP patch and the function in native
comp.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Omar Polo <op@omarpolo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> [...]
>> And that is the beauty of having it as a Lisp function. You can just tweak it,
>> don't need to recompile entire Emacs :).
>
> I know I'm getting off-topic, but I just don't understand your point.  I
> don't see how spawning a bunch of commands, checking their return code
> and parsing their output is better than a couple of lines of C that do
> the right thing depending on the platform (decided at compile time!) and
> get directly an int.
I don't undestand what you don't udnerstand :-)

I don't know my man; what do you mean with "bunch of commands" and how you would
achieve this for all platforms with "couple of lines of C".

Here you have it; based on Andreas code from comp.el. I have just chagned part
shell command on gnu/linux since it can fail dependning on flags. Of course you
get an int back, "directly" :).

#+begin_src emacs-lisp
(declare-function w32-get-nproc "w32.c")

(defun processor-count ()
  (cond ((executable-find "nproc")
         (with-temp-buffer
           (call-process (executable-find "nproc") nil t nil)
           (string-to-number (buffer-string))))
        ((eq 'windows-nt system-type)
         (w32-get-nproc))
        ((eq 'berkeley-unix system-type)
         (string-to-number
          (shell-command-to-string "sysctl -n hw.ncpu")))
        (t 1)))
#+end_src

Compare to original patch in C, and tell me how is doing same in C better than
doing it in Lisp? Your Lisp routine should 
return an int directly. I don't see what is different there and what advantage C
will give you here; more than extra work to implement it and maintain it later on.

To note here is that 'shell-command-to-string' is not recommended since it can
return "more", than what expected, depending on what flags are used to pass to
bash. I am not sure if it can also differ if user uses some other
shell. call-process should be fine. I don't have a bsd system to test though.

I haven't used /proc/cpuinfo. It is a bit dependning on what is goal here: is it
to get number of "usable" cpus for spawning threads, or is it to get real
hardware number of cpus. The reason is that Emacs can run in a "restricted"
system such as a Docker environement where number of CPUs available can be
limited. /proc/cpuinfo (on linux kernel) records hardware number of cores but
nproc return "available" number. So you could have something like this:

#+begin_src emacs-lisp
(declare-function w32-get-nproc "w32.c")

(defun processor-hardware-count ()
  (cond ((eq 'gnu/linux system-type)
         (with-current-buffer (find-file-noselect "/proc/cpuinfo")
           (if (re-search-forward "cpu cores.*: " nil t)
               (string-to-number (current-word))
             1)))
        ((eq 'windows-nt system-type)
         (w32-get-nproc))
        ((eq 'berkeley-unix system-type)
         (string-to-number
          (shell-command-to-string "sysctl -n hw.ncpu")))
        (t 1)))
#+end_src

Could be done with "-all" flag to nproc too, but I think reading /proc/cpuinfo
is faster.

> I love lisp, don't get me wrong, and I actually prefer writing elisp
> rather than following the GNU C coding style (I love C too but GNU style
> hurts my eyes.)

Trust me; if anyone I always vote for doing it in C; but this one is probably
not worth doing in C. I have no idea how suggested posix sysconf deals with
restricted environements either.

> Sure, checking the number of cpus is not something that is done a lot,
> and I can't imagine a situation where it would be a bottleneck, but on
> the other hand, for the same argument, it's not something that needs to
> be tweaked often

Do you want hardware count; logical cores (think hyperthreading); should it work
in restricted environments? Quite a few things to take into consideration, isn't
it?

Hope you understand what I mean better after examples. Something tells me you
won't agree :-), but that is OK. I just present my opinion.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Arthur Miller <arthur.miller@live.com> writes:

> Omar Polo <op@omarpolo.com> writes:
>
>> Arthur Miller <arthur.miller@live.com> writes:
>>
>>> [...]
>>> And that is the beauty of having it as a Lisp function. You can just tweak it,
>>> don't need to recompile entire Emacs :).
>>
>> I know I'm getting off-topic, but I just don't understand your point.  I
>> don't see how spawning a bunch of commands, checking their return code
>> and parsing their output is better than a couple of lines of C that do
>> the right thing depending on the platform (decided at compile time!) and
>> get directly an int.
> I don't undestand what you don't udnerstand :-)
>
> I don't know my man; what do you mean with "bunch of commands" and how you would
> achieve this for all platforms with "couple of lines of C".
>
> Here you have it; based on Andreas code from comp.el. I have just chagned part
> shell command on gnu/linux since it can fail dependning on flags. Of course you
> get an int back, "directly" :).
>
> #+begin_src emacs-lisp
> (declare-function w32-get-nproc "w32.c")
>
> (defun processor-count ()
>   (cond ((executable-find "nproc")
>          (with-temp-buffer
>            (call-process (executable-find "nproc") nil t nil)
>            (string-to-number (buffer-string))))
>         ((eq 'windows-nt system-type)
>          (w32-get-nproc))
>         ((eq 'berkeley-unix system-type)
>          (string-to-number
>           (shell-command-to-string "sysctl -n hw.ncpu")))
>         (t 1)))
> #+end_src
>
>
> Compare to original patch in C, and tell me how is doing same in C better than
> doing it in Lisp? Your Lisp routine should 
> return an int directly. I don't see what is different there and what advantage C
> will give you here; more than extra work to implement it and maintain it later on.
>
> To note here is that 'shell-command-to-string' is not recommended since it can
> return "more", than what expected, depending on what flags are used to pass to
> bash. I am not sure if it can also differ if user uses some other
> shell. call-process should be fine. I don't have a bsd system to test though.
>
> I haven't used /proc/cpuinfo. It is a bit dependning on what is goal here: is it
> to get number of "usable" cpus for spawning threads, or is it to get real
> hardware number of cpus. The reason is that Emacs can run in a "restricted"
> system such as a Docker environement where number of CPUs available can be
> limited. /proc/cpuinfo (on linux kernel) records hardware number of cores but
> nproc return "available" number. So you could have something like this:
>
> #+begin_src emacs-lisp
> (declare-function w32-get-nproc "w32.c")
>
> (defun processor-hardware-count ()
>   (cond ((eq 'gnu/linux system-type)
>          (with-current-buffer (find-file-noselect "/proc/cpuinfo")
>            (if (re-search-forward "cpu cores.*: " nil t)
>                (string-to-number (current-word))
>              1)))
>         ((eq 'windows-nt system-type)
>          (w32-get-nproc))
>         ((eq 'berkeley-unix system-type)
>          (string-to-number
>           (shell-command-to-string "sysctl -n hw.ncpu")))
>         (t 1)))
> #+end_src
>
> Could be done with "-all" flag to nproc too, but I think reading /proc/cpuinfo
> is faster.
>
>> I love lisp, don't get me wrong, and I actually prefer writing elisp
>> rather than following the GNU C coding style (I love C too but GNU style
>> hurts my eyes.)
>
> Trust me; if anyone I always vote for doing it in C; but this one is probably
> not worth doing in C. I have no idea how suggested posix sysconf deals with
> restricted environements either.
>
>> Sure, checking the number of cpus is not something that is done a lot,
>> and I can't imagine a situation where it would be a bottleneck, but on
>> the other hand, for the same argument, it's not something that needs to
>> be tweaked often
>
> Do you want hardware count; logical cores (think hyperthreading); should it work
> in restricted environments? Quite a few things to take into consideration, isn't
> it?
>
> Hope you understand what I mean better after examples. Something tells me you
> won't agree :-), but that is OK. I just present my opinion.

I don't really want to start a pointless thread, so I hope I didn't
sound annoying.  If that's the case, I'm sorry.

I kind of get your point, and as I sad before, I don't have opinions on
this particular case.

I'm still not sure how C can be more difficult to maintain than an
elisp, as to my eyes they're equal.  (yes, elisp has enormous advantages
in most cases, I won't be ever writing a major mode in C for example,
but this is not one of those IMHO).  But I've never really contributed
something significant to Emacs, and I spend almost all my free time
hacking in C, so I'm kinda biased ;-)

But I'd like to add a small correction to your example.  The sysctl is
not correct on OpenBSD (and maybe NetBSD too?  I can't check.)  It
should read

(shell-command-to-string "sysctl -n hw.ncpuonline || sysctl -n hw.ncp")

or something equivalent, please refer to my reply to the OP for the
HW_NCPUONLINE vs HW_NCPU on OpenBSD.

Cheers :)

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Omar Polo <op@omarpolo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Omar Polo <op@omarpolo.com> writes:
>>
>>> Arthur Miller <arthur.miller@live.com> writes:
>>>
>>>> [...]
>>>> And that is the beauty of having it as a Lisp function. You can just tweak it,
>>>> don't need to recompile entire Emacs :).
>>>
>>> I know I'm getting off-topic, but I just don't understand your point.  I
>>> don't see how spawning a bunch of commands, checking their return code
>>> and parsing their output is better than a couple of lines of C that do
>>> the right thing depending on the platform (decided at compile time!) and
>>> get directly an int.
>> I don't undestand what you don't udnerstand :-)
>>
>> I don't know my man; what do you mean with "bunch of commands" and how you would
>> achieve this for all platforms with "couple of lines of C".
>>
>> Here you have it; based on Andreas code from comp.el. I have just chagned part
>> shell command on gnu/linux since it can fail dependning on flags. Of course you
>> get an int back, "directly" :).
>>
>> #+begin_src emacs-lisp
>> (declare-function w32-get-nproc "w32.c")
>>
>> (defun processor-count ()
>>   (cond ((executable-find "nproc")
>>          (with-temp-buffer
>>            (call-process (executable-find "nproc") nil t nil)
>>            (string-to-number (buffer-string))))
>>         ((eq 'windows-nt system-type)
>>          (w32-get-nproc))
>>         ((eq 'berkeley-unix system-type)
>>          (string-to-number
>>           (shell-command-to-string "sysctl -n hw.ncpu")))
>>         (t 1)))
>> #+end_src
>>
>>
>> Compare to original patch in C, and tell me how is doing same in C better than
>> doing it in Lisp? Your Lisp routine should 
>> return an int directly. I don't see what is different there and what advantage C
>> will give you here; more than extra work to implement it and maintain it later on.
>>
>> To note here is that 'shell-command-to-string' is not recommended since it can
>> return "more", than what expected, depending on what flags are used to pass to
>> bash. I am not sure if it can also differ if user uses some other
>> shell. call-process should be fine. I don't have a bsd system to test though.
>>
>> I haven't used /proc/cpuinfo. It is a bit dependning on what is goal here: is it
>> to get number of "usable" cpus for spawning threads, or is it to get real
>> hardware number of cpus. The reason is that Emacs can run in a "restricted"
>> system such as a Docker environement where number of CPUs available can be
>> limited. /proc/cpuinfo (on linux kernel) records hardware number of cores but
>> nproc return "available" number. So you could have something like this:
>>
>> #+begin_src emacs-lisp
>> (declare-function w32-get-nproc "w32.c")
>>
>> (defun processor-hardware-count ()
>>   (cond ((eq 'gnu/linux system-type)
>>          (with-current-buffer (find-file-noselect "/proc/cpuinfo")
>>            (if (re-search-forward "cpu cores.*: " nil t)
>>                (string-to-number (current-word))
>>              1)))
>>         ((eq 'windows-nt system-type)
>>          (w32-get-nproc))
>>         ((eq 'berkeley-unix system-type)
>>          (string-to-number
>>           (shell-command-to-string "sysctl -n hw.ncpu")))
>>         (t 1)))
>> #+end_src
>>
>> Could be done with "-all" flag to nproc too, but I think reading /proc/cpuinfo
>> is faster.
>>
>>> I love lisp, don't get me wrong, and I actually prefer writing elisp
>>> rather than following the GNU C coding style (I love C too but GNU style
>>> hurts my eyes.)
>>
>> Trust me; if anyone I always vote for doing it in C; but this one is probably
>> not worth doing in C. I have no idea how suggested posix sysconf deals with
>> restricted environements either.
>>
>>> Sure, checking the number of cpus is not something that is done a lot,
>>> and I can't imagine a situation where it would be a bottleneck, but on
>>> the other hand, for the same argument, it's not something that needs to
>>> be tweaked often
>>
>> Do you want hardware count; logical cores (think hyperthreading); should it work
>> in restricted environments? Quite a few things to take into consideration, isn't
>> it?
>>
>> Hope you understand what I mean better after examples. Something tells me you
>> won't agree :-), but that is OK. I just present my opinion.
>
> I don't really want to start a pointless thread, so I hope I didn't
> sound annoying.  If that's the case, I'm sorry.
Neither do I :). I was just trying to explain why I reason the way I do.

> I kind of get your point, and as I sad before, I don't have opinions on
> this particular case.
>
> I'm still not sure how C can be more difficult to maintain than an
> elisp, as to my eyes they're equal.

You can just open a lisp file in Emacs, hack it and eval the function; no need
to recompile and restart Emacs. Also the code is easier to read, type etc. In my
eyes it is a winner for the maintance.

> in most cases, I won't be ever writing a major mode in C for example,
> but this is not one of those IMHO).  But I've never really contributed
> something significant to Emacs, and I spend almost all my free time
> hacking in C, so I'm kinda biased ;-)

I had fingers in one patch to dired.c and in some smaller lisp patches, but
nothing of sifnicance either. I really contribute mostly discussions and rarely
code. I hope to change it in the future .. :)

> But I'd like to add a small correction to your example.  The sysctl is
> not correct on OpenBSD (and maybe NetBSD too?  I can't check.)  It
> should read
>
> (shell-command-to-string "sysctl -n hw.ncpuonline || sysctl -n hw.ncp")
It wasn't my; I copy that from comp.el; please report it as a bug to Andrea so
he can correct it. I don't have access to neither *bsd nor mac, so I can't tell.

I also think it should be wrapped into call-process rather than
shell-command-to-string, since shell-command-to-string lumps stdout and stderr
into same buffer, which might result in erronous output. call-process seems to
avoid that.

> or something equivalent, please refer to my reply to the OP for the
> HW_NCPUONLINE vs HW_NCPU on OpenBSD.
>
> Cheers :)

Cheers!

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Omar Polo <op@omarpolo.com> writes:

> But I'd like to add a small correction to your example.  The sysctl is
> not correct on OpenBSD (and maybe NetBSD too?  I can't check.)  It
> should read
>
> (shell-command-to-string "sysctl -n hw.ncpuonline || sysctl -n hw.ncp")

There you have a patch!

Check the function 'comp-effective-async-max-jobs' in comp.el, line 3881.

Fix the bsd-case in cond-statement and give Andrea a patch, so he does not have
to bother :).

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Arthur Miller <arthur.miller@live.com> writes:

> Omar Polo <op@omarpolo.com> writes:
>
>> But I'd like to add a small correction to your example.  The sysctl is
>> not correct on OpenBSD (and maybe NetBSD too?  I can't check.)  It
>> should read
>>
>> (shell-command-to-string "sysctl -n hw.ncpuonline || sysctl -n hw.ncp")
>
> There you have a patch!
>
> Check the function 'comp-effective-async-max-jobs' in comp.el, line 3881.
>
> Fix the bsd-case in cond-statement and give Andrea a patch, so he does not have
> to bother :).

Isn't this just fixed (or about to get fixed) by Paul Eggert?  In a
previous mail he attached two patches, one already installed in the
gnulib repo to fix this ncpu/ncpuonline issue (thanks!) and another to
use that gnulib function in Emacs (and specifically in comp.el.)

or am I missing something?

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Omar Polo <op@omarpolo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Omar Polo <op@omarpolo.com> writes:
>>
>>> But I'd like to add a small correction to your example.  The sysctl is
>>> not correct on OpenBSD (and maybe NetBSD too?  I can't check.)  It
>>> should read
>>>
>>> (shell-command-to-string "sysctl -n hw.ncpuonline || sysctl -n hw.ncp")
>>
>> There you have a patch!
>>
>> Check the function 'comp-effective-async-max-jobs' in comp.el, line 3881.
>>
>> Fix the bsd-case in cond-statement and give Andrea a patch, so he does not have
>> to bother :).
>
> Isn't this just fixed (or about to get fixed) by Paul Eggert?  In a
> previous mail he attached two patches, one already installed in the
> gnulib repo to fix this ncpu/ncpuonline issue (thanks!) and another to
> use that gnulib function in Emacs (and specifically in comp.el.)
>
> or am I missing something?

Ok, no idea; sorry I didn't noticed; I am sorry than. I pulled my source like
yesterday or day before, so I was just looking at the sources; I have obiously
missed his patch.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andreas Schwab]

On Okt 10 2021, Arthur Miller wrote:

>   (cond ((eq 'gnu/linux system-type)
>          (with-current-buffer (find-file-noselect "/proc/cpuinfo")
>            (if (re-search-forward "cpu cores.*: " nil t)
>                (string-to-number (current-word))

Why do you think this does anything useful?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Okt 10 2021, Arthur Miller wrote:
>
>>   (cond ((eq 'gnu/linux system-type)
>>          (with-current-buffer (find-file-noselect "/proc/cpuinfo")
>>            (if (re-search-forward "cpu cores.*: " nil t)
>>                (string-to-number (current-word))
>
> Why do you think this does anything useful?
>
use nproc -all if you don't like "/proc/cpuinfo"

Why is it not useful? 

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andreas Schwab]

On Okt 11 2021, Arthur Miller wrote:

> Why is it not useful? 

Because it doesn't work.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Okt 11 2021, Arthur Miller wrote:
>
>> Why is it not useful? 
>
What does not work? It gives me number of physical cores on my kernel.

Cmon Andreas, you can do better than this; what constructive does this way of
communication add?

Tell me why is it no portable on gnu/linux? This is first time I hear that proc
system is not portable on Linux kernel? I would be interested to hear myself.

By the way all was just examples and illustrations to illustrate a point of
using lisp ocer C; nothing is meant to be "use this!"; there is no error
checking nowhere there, I think the API could be better etc. 

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andreas Schwab]

On Okt 11 2021, Arthur Miller wrote:

> What does not work? It gives me number of physical cores on my kernel.

You need to leave your tiny bubble.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Okt 11 2021, Arthur Miller wrote:
>
>> What does not work? It gives me number of physical cores on my kernel.
>
> You need to leave your tiny bubble.
>
Dude I want number of cores, not number of logical cores. I don't give darn
about Intel's "logcial cpus". num of cores is 4; num of logical cores is 8 that
would be siblings in /proc/cpuinfo.

And what tiny bubble? :) Jesus christ dude.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Campbell Barton]

On Sun, Oct 10, 2021 at 8:38 PM Arthur Miller <arthur.miller@live.com> wrote:
>
> Omar Polo <op@omarpolo.com> writes:
>
> > Campbell Barton <ideasman42@gmail.com> writes:
> >
> >> Hi, this patch adds support for accessing the number of CPU's / cores
> >> on a system, matching CPython's multiprocessing.cpu_count() [0].
> >>
> >> I've only tested this for Linux, this includes code that should work
> >> on other platforms, although that would need to be double-checked of
> >> course.
> >> For reference I checked CPython / Blender & Stack-overflow [1]
> >
> > I can confirm it works on OpenBSD too, but needs tweaking.  Some
> > comments inline and attaching an updated patch
> >
> >> Accessing this information can be useful to automatically detect the
> >> number of jobs to run.
> >>
> >>
> >> [0]:
> >> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count
> >> [1]: https://stackoverflow.com/a/3006416/432509
> >>
> >> commit 7be53f10f3df3c3183cc97d6bbadb78ebb61e8d2
> >> Author: Campbell Barton <ideasman42@gmail.com>
> >> Date:   Sun Oct 10 10:16:47 2021 +1100
> >>
> >>     Support accessing the number of CPU's.
> >>
> >>     Add (processor-count) for accessing the number of cores/CPU's.
> >>
> >> diff --git a/src/emacs.c b/src/emacs.c
> >> index 866e43fda9..26e2f6b1f2 100644
> >> --- a/src/emacs.c
> >> +++ b/src/emacs.c
> >> @@ -3156,6 +3156,38 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0,
> >>    return Qt;
> >>  }
> >>
> >> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
> >> +       doc: /* Return the number of CPUs in the system.
> >> +
> >> +The value will always be above zero, 1 for unsupported systems.  */)
> >> +  (void)
> >> +{
> >> +  int nproc = -1;
> >> +#ifdef WINDOWSNT
> >> +  SYSTEM_INFO info;
> >> +  GetSystemInfo(&info);
> >> +  nproc = (int)info.dwNumberOfProcessors;
> >> +#elif defined (__APPLE__) || \
> >> +      defined (__OpenBSD__) || \
> >> +      defined (__FreeBSD__) || \
> >> +      defined (__NetBSD__) || \
> >> +      defined (__DragonFly__)
> >> +  int mib[2];
> >> +  size_t len;
> >> +
> >> +  mib[0] = CTL_HW;
> >
> > the #include <sys/sysctl.h> is missing.
> >
> >> +  mib[1] = HW_NCPU;
> >
> > at least on OpenBSD this should be HW_NCPUONLINE.
> >
> > OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
> > misleading.  For example, on my machine
> >
> >       % uname -a
> >       OpenBSD venera 7.0 GENERIC.MP#221 amd64
> >       % sysctl hw.ncpu
> >       hw.ncpu=8
> >       % sysctl hw.ncpuonline
> >       hw.ncpuonline=4
> >
> > and this has been the case for a while already (I mean, a couple of
> > years if not more.)
> >
> > I don't have access to other BSDs other than OpenBSD, but judging from
> > the manpages online.
> >
> >  - NetBSD has hw.ncpuonline
> >  - DragonFly and FreeBSD have only hw.ncpu
> >  - apple I don't know how to check.  man.apple.com doesn't seem to
> >    exists ^^"
> >
> >> +  len = sizeof(nproc);
> >> +  sysctl(mib, 2, &nproc, &len, nullptr, 0);
> >                                   ^^^^^^^
> > shouldn't this be NULL?
> >
> >> +#elif defined (__hpux)
> >> +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
> >> +#elif defined (_SC_NPROCESSORS_ONLN)
> >> +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
> >> +#endif
> >> +
> >> +  return make_fixnum (MAX(nproc, 1));
> >
> > emacs.c:3188:23: warning: implicit declaration of function 'MAX' is invalid in C99 [-Wimplicit-function-declaration]
> >
> > I'm attaching an updated patch.  Note that I'm not sure if including
> > sys/types.h and sys/sysctl.h breaks the build on some OS.
> >
> > if someone knows how to reduce the number of #ifdefs I'll be glad :)
> >
> >> +}
> >> +
> >
> > Cheers,
> >
> > Omar Polo
> >
> > From 5fe26a73abc60eae7dd2528e3854f2f379bcb4c7 Mon Sep 17 00:00:00 2001
> > From: Campbell Barton <ideasman42@gmail.com>
> > Date: Sun, 10 Oct 2021 08:21:01 +0000
> > Subject: [PATCH] add function to access the number of CPU
> >
> > * emacs.c (Fprocessor_count): add processor-count function to access
> >   the number of CPU
> > ---
> >  src/emacs.c | 35 +++++++++++++++++++++++++++++++++++
> >  1 file changed, 35 insertions(+)
> >
> > diff --git a/src/emacs.c b/src/emacs.c
> > index 866e43fda9..5bd10a7586 100644
> > --- a/src/emacs.c
> > +++ b/src/emacs.c
> > @@ -25,6 +25,8 @@ along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.  */
> >  #include <fcntl.h>
> >  #include <stdlib.h>
> >
> > +#include <sys/types.h>
> > +#include <sys/sysctl.h>
> >  #include <sys/file.h>
> >  #include <sys/stat.h>
> >  #include <unistd.h>
> > @@ -3156,6 +3158,38 @@ from the parent process and its tty file descriptors.  */)
> >    return Qt;
> >  }
> >
> > +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
> > +       doc: /* Return the number of CPUs in the system.
> > +
> > +The value will always be above zero, 1 for unsupported systems.  */)
> > +  (void)
> > +{
> > +  int nproc = -1;
> > +#ifdef WINDOWSNT
> > +  SYSTEM_INFO info;
> > +  GetSystemInfo(&info);
> > +  nproc = (int)info.dwNumberOfProcessors;
> > +#elif defined (HW_NCPU) || defined (HW_NCPUONLINE)
> > +  int mib[2];
> > +  size_t len;
> > +
> > +  mib[0] = CTL_HW;
> > +#ifdef HW_NCPUONLINE
> > +  mib[1] = HW_NCPUONLINE;
> > +#else
> > +  mib[1] = HW_NCPU;
> > +#endif
> > +  len = sizeof(nproc);
> > +  sysctl(mib, 2, &nproc, &len, NULL, 0);
> > +#elif defined (__hpux)
> > +  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
> > +#elif defined (_SC_NPROCESSORS_ONLN)
> > +  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
> > +#endif
> > +
> > +  return make_fixnum (nproc > 1 ? nproc : 1);
> > +}
> > +
> >  void
> >  syms_of_emacs (void)
> >  {
> > @@ -3176,6 +3210,7 @@ syms_of_emacs (void)
> >    defsubr (&Sinvocation_directory);
> >    defsubr (&Sdaemonp);
> >    defsubr (&Sdaemon_initialized);
> > +  defsubr (&Sprocessor_count);
> >
> >    DEFVAR_LISP ("command-line-args", Vcommand_line_args,
> >              doc: /* Args passed by shell to Emacs, as a list of strings.
>
> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?
>
> Windows has "wmic cpu get NumberOfCores,NumberOfLogicalProcessors" which works
> on cmd prompt, which means executable from elisp as a process. I dont know for
> apple and *bsds.
>
> It is just text that can be parsed directly with elisp, so it is easier to
> maintain does not need to be compiled etc. I don't see cpu queries as a
> performance crucial query. Just a thought.

Agree - in many cases the overhead of launching a process isn't likely
to be an issue,
although some corner cases could end up calling this more often than
expected, causing slowdowns.

In general I've found this kind of code (while it doesn't read so
nicely with all the ifdef's), doesn't cause much maintenance overhead
since it's just querying a value. Once this is working, it only needs
to be touched if API's are ever deprecated (for example).
Also, it seems it's not so trivial to get this info from the command
line on all systems, hpux docs [0] for e.g. suggest calling:
/usr/sbin/ioscan -kf | grep processor | wc -l
... while this can be replicated in elisp, the C version seems more concise.

[0]: https://www.ibm.com/support/pages/determining-how-many-cpus-you-have-hp-ux




--
- Campbell

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andreas Schwab]

On Okt 10 2021, Arthur Miller wrote:

> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?

If you mean /proc/cpuinfo, it's highly architecture dependent.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Okt 10 2021, Arthur Miller wrote:
>
>> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux?
>
> If you mean /proc/cpuinfo, it's highly architecture dependent.

Oh sorry; no idea how 'cat' got in there; of course I ment /proc/cpuinfo

Does it differ between kernel versions or in which way is it architecture
dependent on Linux kernel? I suggested this only for gnu/Linux.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andreas Schwab]

On Okt 10 2021, Arthur Miller wrote:

> Does it differ between kernel versions or in which way is it architecture
> dependent on Linux kernel? I suggested this only for gnu/Linux.

Every architecture is different.  Why reinventing the wheel if the
standard library has already worked out all the details?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Okt 10 2021, Arthur Miller wrote:
>
>> Does it differ between kernel versions or in which way is it architecture
>> dependent on Linux kernel? I suggested this only for gnu/Linux.
>
> Every architecture is different.  Why reinventing the wheel if the
> standard library has already worked out all the details?
>
? Which standard library? C? Or what?

I think there is some missunderstanding going on here. I didn't oppose to the
OPs idea with patch; I just said it would be more practical to implement it in
Lisp than in C.

I gave him /proc/cpuinfo for gnu/Linux and windows call just as pointers what he
could use in a Lisp function, and later on I posted him Corrallo's take on same problem.

I am not sure though, what you are talking about :). Sorry. I think you have
missinterpretted what I wrote to him. Sorry if I was unclear. But hope this
message clears it up.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andreas Schwab]

On Okt 10 2021, Arthur Miller wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> On Okt 10 2021, Arthur Miller wrote:
>>
>>> Does it differ between kernel versions or in which way is it architecture
>>> dependent on Linux kernel? I suggested this only for gnu/Linux.
>>
>> Every architecture is different.  Why reinventing the wheel if the
>> standard library has already worked out all the details?
>>
> ? Which standard library? C? Or what?
>
> I think there is some missunderstanding going on here. I didn't oppose to the
> OPs idea with patch; I just said it would be more practical to implement it in
> Lisp than in C.
>
> I gave him /proc/cpuinfo for gnu/Linux

That is not portable.  It is much easier to let sysconf do the heavy
lifting, instead of reimplementing it.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Okt 10 2021, Arthur Miller wrote:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>> On Okt 10 2021, Arthur Miller wrote:
>>>
>>>> Does it differ between kernel versions or in which way is it architecture
>>>> dependent on Linux kernel? I suggested this only for gnu/Linux.
>>>
>>> Every architecture is different.  Why reinventing the wheel if the
>>> standard library has already worked out all the details?
>>>
>> ? Which standard library? C? Or what?
>>
>> I think there is some missunderstanding going on here. I didn't oppose to the
>> OPs idea with patch; I just said it would be more practical to implement it in
>> Lisp than in C.
>>
>> I gave him /proc/cpuinfo for gnu/Linux
>
> That is not portable.  It is much easier to let sysconf do the heavy
> lifting, instead of reimplementing it.

We seem to be talking past each other. I don't see why would want this as a C
primitive.

Sysconf is portable only on posix systems. It covers linux and *bsd kernels, but
you still have windows left, and some hypotetical future os.

I think it is better to implement it as a Lisp function, as Andrea did in native
comp. Less to maintain and don't need to be recompiled if some future non-posix
system need to be added.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Stefan Monnier]

> We seem to be talking past each other. I don't see why would want this as a C
> primitive.

There are typically 2 reasons to use C:
- For speed
- To use someone else's code which is easily available from C but not
  from ELisp.


        Stefan

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We seem to be talking past each other. I don't see why would want this as a C
>> primitive.
>
> There are typically 2 reasons to use C:
> - For speed
> - To use someone else's code which is easily available from C but not
>   from ELisp.

Of course, but I don't see any of those apply to this case :).

As I see nproc from core-utils is already doing suggested, and more than so, so
it is just to call it :).

https://github.com/coreutils/gnulib/blob/master/lib/nproc.c

PS: it is not directly "a couple of lines" either as someone suggested :).

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Campbell Barton]

On 10/11/21 06:50, Arthur Miller wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
>>> We seem to be talking past each other. I don't see why would want this as a C
>>> primitive.
>>
>> There are typically 2 reasons to use C:
>> - For speed
>> - To use someone else's code which is easily available from C but not
>>    from ELisp.
> 
> Of course, but I don't see any of those apply to this case :).

The Elisp version of this function posted was incomplete by comparison 
in that it didn't support OpenBSD's ncpuonline feature or HPUX.

Also, this is not simply a call to an external process (executable-find 
"nproc") has some additional overhead.

I don't see this so much a case of ELisp vs C, more a case of OS-level 
API's being more appropriate than searching around for commands and 
parsing their output.

> As I see nproc from core-utils is already doing suggested, and more than so, so
> it is just to call it :).
> 
> https://github.com/coreutils/gnulib/blob/master/lib/nproc.c
> 
> PS: it is not directly "a couple of lines" either as someone suggested :).

Right, it's quite involved, as mentioned in a previous reply, this seems 
more appropriate for detecting OS-level threads (not spawning new 
processes).

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Campbell Barton <ideasman42@gmail.com> writes:

> On 10/11/21 06:50, Arthur Miller wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>>>> We seem to be talking past each other. I don't see why would want this as a C
>>>> primitive.
>>>
>>> There are typically 2 reasons to use C:
>>> - For speed
>>> - To use someone else's code which is easily available from C but not
>>>    from ELisp.
>> Of course, but I don't see any of those apply to this case :).
>
> The Elisp version of this function posted was incomplete by comparison in that
> it didn't support OpenBSD's ncpuonline feature or HPUX.
But then complete it :). I don't have openbsd installed.

> Also, this is not simply a call to an external process (executable-find "nproc")
> has some additional overhead.
You can skip that search if you want; it was just to be on the safe side. That
overhead is not importnat. As I wrote to Omar, you will probably ask for number
of cpu's in some eval-after-load or similar, not in some performance critical
code.

> I don't see this so much a case of ELisp vs C, more a case of OS-level API's
> being more appropriate than searching around for commands and parsing their
> output.
nproc.c has already sorted all that for you; so you can just call the process
instead of doing it all over.

>> As I see nproc from core-utils is already doing suggested, and more than so, so
>> it is just to call it :).
>> https://github.com/coreutils/gnulib/blob/master/lib/nproc.c
>> PS: it is not directly "a couple of lines" either as someone suggested :).
>
> Right, it's quite involved, as mentioned in a previous reply, this seems more
> appropriate for detecting OS-level threads (not spawning new processes).

Why does os threads matter if you can't access them. Or you mean "logical cores"
a.k.a "hyperthreading"?

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Stefan Kangas]

Omar Polo <op@omarpolo.com> writes:

> > +  mib[1] = HW_NCPU;
>
> at least on OpenBSD this should be HW_NCPUONLINE.
>
> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
> misleading.  For example, on my machine
>
>         % uname -a
>         OpenBSD venera 7.0 GENERIC.MP#221 amd64
>         % sysctl hw.ncpu
>         hw.ncpu=8
>         % sysctl hw.ncpuonline
>         hw.ncpuonline=4
>
> and this has been the case for a while already (I mean, a couple of
> years if not more.)

BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
Maybe that should be reported to the Gnulib developers?

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Stefan Kangas <stefan@marxist.se> writes:

> Omar Polo <op@omarpolo.com> writes:
>
>> > +  mib[1] = HW_NCPU;
>>
>> at least on OpenBSD this should be HW_NCPUONLINE.
>>
>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>> misleading.  For example, on my machine
>>
>>         % uname -a
>>         OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>         % sysctl hw.ncpu
>>         hw.ncpu=8
>>         % sysctl hw.ncpuonline
>>         hw.ncpuonline=4
>>
>> and this has been the case for a while already (I mean, a couple of
>> years if not more.)
>
> BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
> Maybe that should be reported to the Gnulib developers?

Thanks for letting me know, it looks wrong indeed.  I've never used
gnulib (directly at least) but I'll try to send a patch.

Thanks

Omar Polo

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On 10/10/21 5:21 AM, Stefan Kangas wrote:
> Omar Polo <op@omarpolo.com> writes:
>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>> misleading.  For example, on my machine
>>
>>          % uname -a
>>          OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>          % sysctl hw.ncpu
>>          hw.ncpu=8
>>          % sysctl hw.ncpuonline
>>          hw.ncpuonline=4
>>
>> and this has been the case for a while already (I mean, a couple of
>> years if not more.)
> 
> BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
> Maybe that should be reported to the Gnulib developers?


No need for a bug report as I'm one of those developers. To fix the 
Gnulib nproc OpenBSD issue I installed the first attached patch into 
Gnulib, and to port Emacs's recently-added processor-count code to more 
platforms I installed the second attached patch into Emacs's emacs-28 
branch. Although I think the latter patch means that 
emacs/src/w32proc.c's w32-get-nproc function can be removed, I'll let 
the Emacs MS-Windows experts opine on that.

[-- Attachment #2: 0001-nproc-port-better-to-OpenBSD.patch --]
[-- Type: text/x-patch, Size: 1988 bytes --]

From 7fc3219bcc2bf448ef26cf30a2e5770fdda3f2b4 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 10 Oct 2021 10:43:47 -0700
Subject: [PATCH] nproc: port better to OpenBSD

Problem reported by Omar Polo in:
https://lists.gnu.org/r/emacs-devel/2021-10/msg00692.html
* lib/nproc.c (num_processors_ignoring_omp): Prefer HW_NCPUONLINE
to HW_NCPU, for OpenBSD.  Also, make mib const.
---
 ChangeLog   |  8 ++++++++
 lib/nproc.c | 19 +++++++++++++------
 2 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 232d2dec3..da5b570ee 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2021-10-10  Paul Eggert  <eggert@cs.ucla.edu>
+
+	nproc: port better to OpenBSD
+	Problem reported by Omar Polo in:
+	https://lists.gnu.org/r/emacs-devel/2021-10/msg00692.html
+	* lib/nproc.c (num_processors_ignoring_omp): Prefer HW_NCPUONLINE
+	to HW_NCPU, for OpenBSD.  Also, make mib const.
+
 2021-10-02  Paul Eggert  <eggert@cs.ucla.edu>
 
 	timer-time: port better to OpenBSD 6.9
diff --git a/lib/nproc.c b/lib/nproc.c
index e3ddb9288..a9e369dd3 100644
--- a/lib/nproc.c
+++ b/lib/nproc.c
@@ -310,12 +310,19 @@ num_processors_ignoring_omp (enum nproc_query query)
   { /* This works on Mac OS X, FreeBSD, NetBSD, OpenBSD.  */
     int nprocs;
     size_t len = sizeof (nprocs);
-    static int mib[2] = { CTL_HW, HW_NCPU };
-
-    if (sysctl (mib, ARRAY_SIZE (mib), &nprocs, &len, NULL, 0) == 0
-        && len == sizeof (nprocs)
-        && 0 < nprocs)
-      return nprocs;
+    static int const mib[][2] = {
+# ifdef HW_NCPUONLINE
+      { CTL_HW, HW_NCPUONLINE },
+# endif
+      { CTL_HW, HW_NCPU }
+    };
+    for (int i = 0; i < ARRAY_SIZE (mib); i++)
+      {
+        if (sysctl (mib[i], ARRAY_SIZE (mib[i]), &nprocs, &len, NULL, 0) == 0
+            && len == sizeof (nprocs)
+            && 0 < nprocs)
+          return nprocs;
+      }
   }
 #endif
 
-- 
2.30.2


[-- Attachment #3: 0001-New-function-num-processors.patch --]
[-- Type: text/x-patch, Size: 26157 bytes --]

From 96278de8ac2166c37925f2dfbc0eeb6d368142b9 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 10 Oct 2021 13:59:16 -0700
Subject: [PATCH] New function num-processors
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This addresses a FIXME comment in lisp/emacs-lisp/comp.el,
relating to the number of subsidiary processes used by
comp-run-async-workers in native compilation.
* admin/merge-gnulib (GNULIB_MODULES): Add nproc.
* doc/lispref/processes.texi (Process Information), etc/NEWS:
Document num-processors.
* lib/gnulib.mk.in, m4/gnulib-comp.m4: Regenerate.
* lib/nproc.c, lib/nproc.h, m4/nproc.m4:
New files, copied from Gnulib by admin/merge-gnulib.
* lisp/emacs-lisp/comp.el (w32-get-nproc): Remove decl.
(comp-effective-async-max-jobs): Use num-processors.
* src/process.c: Include nproc.h.
(Fnum_processors): New function.
(syms_of_process): Define ‘all’, ‘current’, ‘num-processors’.
* src/w32proc.c (Fw32_get_nproc): Add FIXME comment.
* test/src/process-tests.el (process-num-processors): New test.
---
 admin/merge-gnulib         |   3 +-
 doc/lispref/processes.texi |  13 ++
 etc/NEWS                   |   4 +
 lib/gnulib.mk.in           |  11 +
 lib/nproc.c                | 403 +++++++++++++++++++++++++++++++++++++
 lib/nproc.h                |  46 +++++
 lisp/emacs-lisp/comp.el    |  15 +-
 m4/gnulib-comp.m4          |   5 +
 m4/nproc.m4                |  54 +++++
 src/process.c              |  18 ++
 src/w32proc.c              |   1 +
 test/src/process-tests.el  |   6 +
 12 files changed, 564 insertions(+), 15 deletions(-)
 create mode 100644 lib/nproc.c
 create mode 100644 lib/nproc.h
 create mode 100644 m4/nproc.m4

diff --git a/admin/merge-gnulib b/admin/merge-gnulib
index 886f37e28c..c9fe3b2f95 100755
--- a/admin/merge-gnulib
+++ b/admin/merge-gnulib
@@ -39,7 +39,8 @@ GNULIB_MODULES=
   free-posix fstatat fsusage fsync futimens
   getloadavg getopt-gnu getrandom gettime gettimeofday gitlog-to-changelog
   ieee754-h ignore-value intprops largefile libgmp lstat
-  manywarnings memmem-simple mempcpy memrchr minmax mkostemp mktime nstrftime
+  manywarnings memmem-simple mempcpy memrchr minmax mkostemp mktime
+  nproc nstrftime
   pathmax pipe2 pselect pthread_sigmask
   qcopy-acl readlink readlinkat regex
   sig2str sigdescr_np socklen stat-time std-gnu11 stdalign stddef stdio
diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 90c4215637..d90097d0b0 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -1047,6 +1047,19 @@ Process Information
 @end smallexample
 @end defun
 
+@defun num-processors &optional query
+This function returns the number of processors, a positive integer.
+Each usable thread execution unit counts as a processor.
+By default, the count includes the number of available processors,
+which you can override by setting the
+@url{https://www.openmp.org/spec-html/5.1/openmpse59.html,
+@env{OMP_NUM_THREADS} environment variable of OpenMP}.
+If the optional argument @var{query} is @code{current},
+this function ignores @env{OMP_NUM_THREADS};
+if @var{query} is @code{all}, this function also counts processors
+that are on the system but are not available to the current process.
+@end defun
+
 @defun get-process name
 This function returns the process named @var{name} (a string), or
 @code{nil} if there is none.  The argument @var{name} can also be a
diff --git a/etc/NEWS b/etc/NEWS
index 09537d7d31..791248f7dc 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -4094,6 +4094,10 @@ Parse a string as a mail address-like string.
 ** New function 'make-separator-line'.
 Make a string appropriate for usage as a visual separator line.
 
++++
+** New function 'num-processors'.
+Return the number of processors on the system.
+
 +++
 ** New function 'object-intervals'.
 This function returns a copy of the list of intervals (i.e., text
diff --git a/lib/gnulib.mk.in b/lib/gnulib.mk.in
index e9a1a5dc02..c7c7eb455b 100644
--- a/lib/gnulib.mk.in
+++ b/lib/gnulib.mk.in
@@ -129,6 +129,7 @@
 #  minmax \
 #  mkostemp \
 #  mktime \
+#  nproc \
 #  nstrftime \
 #  pathmax \
 #  pipe2 \
@@ -2378,6 +2379,16 @@ EXTRA_libgnu_a_SOURCES += mktime.c
 endif
 ## end   gnulib module mktime-internal
 
+## begin gnulib module nproc
+ifeq (,$(OMIT_GNULIB_MODULE_nproc))
+
+libgnu_a_SOURCES += nproc.c
+
+EXTRA_DIST += nproc.h
+
+endif
+## end   gnulib module nproc
+
 ## begin gnulib module nstrftime
 ifeq (,$(OMIT_GNULIB_MODULE_nstrftime))
 
diff --git a/lib/nproc.c b/lib/nproc.c
new file mode 100644
index 0000000000..a9e369dd3f
--- /dev/null
+++ b/lib/nproc.c
@@ -0,0 +1,403 @@
+/* Detect the number of processors.
+
+   Copyright (C) 2009-2021 Free Software Foundation, Inc.
+
+   This file is free software: you can redistribute it and/or modify
+   it under the terms of the GNU Lesser General Public License as
+   published by the Free Software Foundation; either version 2.1 of the
+   License, or (at your option) any later version.
+
+   This file is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public License
+   along with this program.  If not, see <https://www.gnu.org/licenses/>.  */
+
+/* Written by Glen Lenker and Bruno Haible.  */
+
+#include <config.h>
+#include "nproc.h"
+
+#include <limits.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#if HAVE_PTHREAD_GETAFFINITY_NP && 0
+# include <pthread.h>
+# include <sched.h>
+#endif
+#if HAVE_SCHED_GETAFFINITY_LIKE_GLIBC || HAVE_SCHED_GETAFFINITY_NP
+# include <sched.h>
+#endif
+
+#include <sys/types.h>
+
+#if HAVE_SYS_PSTAT_H
+# include <sys/pstat.h>
+#endif
+
+#if HAVE_SYS_SYSMP_H
+# include <sys/sysmp.h>
+#endif
+
+#if HAVE_SYS_PARAM_H
+# include <sys/param.h>
+#endif
+
+#if HAVE_SYS_SYSCTL_H && ! defined __GLIBC__
+# include <sys/sysctl.h>
+#endif
+
+#if defined _WIN32 && ! defined __CYGWIN__
+# define WIN32_LEAN_AND_MEAN
+# include <windows.h>
+#endif
+
+#include "c-ctype.h"
+
+#include "minmax.h"
+
+#define ARRAY_SIZE(a) (sizeof (a) / sizeof ((a)[0]))
+
+/* Return the number of processors available to the current process, based
+   on a modern system call that returns the "affinity" between the current
+   process and each CPU.  Return 0 if unknown or if such a system call does
+   not exist.  */
+static unsigned long
+num_processors_via_affinity_mask (void)
+{
+  /* glibc >= 2.3.3 with NPTL and NetBSD 5 have pthread_getaffinity_np,
+     but with different APIs.  Also it requires linking with -lpthread.
+     Therefore this code is not enabled.
+     glibc >= 2.3.4 has sched_getaffinity whereas NetBSD 5 has
+     sched_getaffinity_np.  */
+#if HAVE_PTHREAD_GETAFFINITY_NP && defined __GLIBC__ && 0
+  {
+    cpu_set_t set;
+
+    if (pthread_getaffinity_np (pthread_self (), sizeof (set), &set) == 0)
+      {
+        unsigned long count;
+
+# ifdef CPU_COUNT
+        /* glibc >= 2.6 has the CPU_COUNT macro.  */
+        count = CPU_COUNT (&set);
+# else
+        size_t i;
+
+        count = 0;
+        for (i = 0; i < CPU_SETSIZE; i++)
+          if (CPU_ISSET (i, &set))
+            count++;
+# endif
+        if (count > 0)
+          return count;
+      }
+  }
+#elif HAVE_PTHREAD_GETAFFINITY_NP && defined __NetBSD__ && 0
+  {
+    cpuset_t *set;
+
+    set = cpuset_create ();
+    if (set != NULL)
+      {
+        unsigned long count = 0;
+
+        if (pthread_getaffinity_np (pthread_self (), cpuset_size (set), set)
+            == 0)
+          {
+            cpuid_t i;
+
+            for (i = 0;; i++)
+              {
+                int ret = cpuset_isset (i, set);
+                if (ret < 0)
+                  break;
+                if (ret > 0)
+                  count++;
+              }
+          }
+        cpuset_destroy (set);
+        if (count > 0)
+          return count;
+      }
+  }
+#elif HAVE_SCHED_GETAFFINITY_LIKE_GLIBC /* glibc >= 2.3.4 */
+  {
+    cpu_set_t set;
+
+    if (sched_getaffinity (0, sizeof (set), &set) == 0)
+      {
+        unsigned long count;
+
+# ifdef CPU_COUNT
+        /* glibc >= 2.6 has the CPU_COUNT macro.  */
+        count = CPU_COUNT (&set);
+# else
+        size_t i;
+
+        count = 0;
+        for (i = 0; i < CPU_SETSIZE; i++)
+          if (CPU_ISSET (i, &set))
+            count++;
+# endif
+        if (count > 0)
+          return count;
+      }
+  }
+#elif HAVE_SCHED_GETAFFINITY_NP /* NetBSD >= 5 */
+  {
+    cpuset_t *set;
+
+    set = cpuset_create ();
+    if (set != NULL)
+      {
+        unsigned long count = 0;
+
+        if (sched_getaffinity_np (getpid (), cpuset_size (set), set) == 0)
+          {
+            cpuid_t i;
+
+            for (i = 0;; i++)
+              {
+                int ret = cpuset_isset (i, set);
+                if (ret < 0)
+                  break;
+                if (ret > 0)
+                  count++;
+              }
+          }
+        cpuset_destroy (set);
+        if (count > 0)
+          return count;
+      }
+  }
+#endif
+
+#if defined _WIN32 && ! defined __CYGWIN__
+  { /* This works on native Windows platforms.  */
+    DWORD_PTR process_mask;
+    DWORD_PTR system_mask;
+
+    if (GetProcessAffinityMask (GetCurrentProcess (),
+                                &process_mask, &system_mask))
+      {
+        DWORD_PTR mask = process_mask;
+        unsigned long count = 0;
+
+        for (; mask != 0; mask = mask >> 1)
+          if (mask & 1)
+            count++;
+        if (count > 0)
+          return count;
+      }
+  }
+#endif
+
+  return 0;
+}
+
+
+/* Return the total number of processors.  Here QUERY must be one of
+   NPROC_ALL, NPROC_CURRENT.  The result is guaranteed to be at least 1.  */
+static unsigned long int
+num_processors_ignoring_omp (enum nproc_query query)
+{
+  /* On systems with a modern affinity mask system call, we have
+         sysconf (_SC_NPROCESSORS_CONF)
+            >= sysconf (_SC_NPROCESSORS_ONLN)
+               >= num_processors_via_affinity_mask ()
+     The first number is the number of CPUs configured in the system.
+     The second number is the number of CPUs available to the scheduler.
+     The third number is the number of CPUs available to the current process.
+
+     Note! On Linux systems with glibc, the first and second number come from
+     the /sys and /proc file systems (see
+     glibc/sysdeps/unix/sysv/linux/getsysstats.c).
+     In some situations these file systems are not mounted, and the sysconf call
+     returns 1 or 2 (<https://sourceware.org/bugzilla/show_bug.cgi?id=21542>),
+     which does not reflect the reality.  */
+
+  if (query == NPROC_CURRENT)
+    {
+      /* Try the modern affinity mask system call.  */
+      {
+        unsigned long nprocs = num_processors_via_affinity_mask ();
+
+        if (nprocs > 0)
+          return nprocs;
+      }
+
+#if defined _SC_NPROCESSORS_ONLN
+      { /* This works on glibc, Mac OS X 10.5, FreeBSD, AIX, OSF/1, Solaris,
+           Cygwin, Haiku.  */
+        long int nprocs = sysconf (_SC_NPROCESSORS_ONLN);
+        if (nprocs > 0)
+          return nprocs;
+      }
+#endif
+    }
+  else /* query == NPROC_ALL */
+    {
+#if defined _SC_NPROCESSORS_CONF
+      { /* This works on glibc, Mac OS X 10.5, FreeBSD, AIX, OSF/1, Solaris,
+           Cygwin, Haiku.  */
+        long int nprocs = sysconf (_SC_NPROCESSORS_CONF);
+
+# if __GLIBC__ >= 2 && defined __linux__
+        /* On Linux systems with glibc, this information comes from the /sys and
+           /proc file systems (see glibc/sysdeps/unix/sysv/linux/getsysstats.c).
+           In some situations these file systems are not mounted, and the
+           sysconf call returns 1 or 2.  But we wish to guarantee that
+           num_processors (NPROC_ALL) >= num_processors (NPROC_CURRENT).  */
+        if (nprocs == 1 || nprocs == 2)
+          {
+            unsigned long nprocs_current = num_processors_via_affinity_mask ();
+
+            if (/* nprocs_current > 0 && */ nprocs_current > nprocs)
+              nprocs = nprocs_current;
+          }
+# endif
+
+        if (nprocs > 0)
+          return nprocs;
+      }
+#endif
+    }
+
+#if HAVE_PSTAT_GETDYNAMIC
+  { /* This works on HP-UX.  */
+    struct pst_dynamic psd;
+    if (pstat_getdynamic (&psd, sizeof psd, 1, 0) >= 0)
+      {
+        /* The field psd_proc_cnt contains the number of active processors.
+           In newer releases of HP-UX 11, the field psd_max_proc_cnt includes
+           deactivated processors.  */
+        if (query == NPROC_CURRENT)
+          {
+            if (psd.psd_proc_cnt > 0)
+              return psd.psd_proc_cnt;
+          }
+        else
+          {
+            if (psd.psd_max_proc_cnt > 0)
+              return psd.psd_max_proc_cnt;
+          }
+      }
+  }
+#endif
+
+#if HAVE_SYSMP && defined MP_NAPROCS && defined MP_NPROCS
+  { /* This works on IRIX.  */
+    /* MP_NPROCS yields the number of installed processors.
+       MP_NAPROCS yields the number of processors available to unprivileged
+       processes.  */
+    int nprocs =
+      sysmp (query == NPROC_CURRENT && getuid () != 0
+             ? MP_NAPROCS
+             : MP_NPROCS);
+    if (nprocs > 0)
+      return nprocs;
+  }
+#endif
+
+  /* Finally, as fallback, use the APIs that don't distinguish between
+     NPROC_CURRENT and NPROC_ALL.  */
+
+#if HAVE_SYSCTL && ! defined __GLIBC__ && defined HW_NCPU
+  { /* This works on Mac OS X, FreeBSD, NetBSD, OpenBSD.  */
+    int nprocs;
+    size_t len = sizeof (nprocs);
+    static int const mib[][2] = {
+# ifdef HW_NCPUONLINE
+      { CTL_HW, HW_NCPUONLINE },
+# endif
+      { CTL_HW, HW_NCPU }
+    };
+    for (int i = 0; i < ARRAY_SIZE (mib); i++)
+      {
+        if (sysctl (mib[i], ARRAY_SIZE (mib[i]), &nprocs, &len, NULL, 0) == 0
+            && len == sizeof (nprocs)
+            && 0 < nprocs)
+          return nprocs;
+      }
+  }
+#endif
+
+#if defined _WIN32 && ! defined __CYGWIN__
+  { /* This works on native Windows platforms.  */
+    SYSTEM_INFO system_info;
+    GetSystemInfo (&system_info);
+    if (0 < system_info.dwNumberOfProcessors)
+      return system_info.dwNumberOfProcessors;
+  }
+#endif
+
+  return 1;
+}
+
+/* Parse OMP environment variables without dependence on OMP.
+   Return 0 for invalid values.  */
+static unsigned long int
+parse_omp_threads (char const* threads)
+{
+  unsigned long int ret = 0;
+
+  if (threads == NULL)
+    return ret;
+
+  /* The OpenMP spec says that the value assigned to the environment variables
+     "may have leading and trailing white space".  */
+  while (*threads != '\0' && c_isspace (*threads))
+    threads++;
+
+  /* Convert it from positive decimal to 'unsigned long'.  */
+  if (c_isdigit (*threads))
+    {
+      char *endptr = NULL;
+      unsigned long int value = strtoul (threads, &endptr, 10);
+
+      if (endptr != NULL)
+        {
+          while (*endptr != '\0' && c_isspace (*endptr))
+            endptr++;
+          if (*endptr == '\0')
+            return value;
+          /* Also accept the first value in a nesting level,
+             since we can't determine the nesting level from env vars.  */
+          else if (*endptr == ',')
+            return value;
+        }
+    }
+
+  return ret;
+}
+
+unsigned long int
+num_processors (enum nproc_query query)
+{
+  unsigned long int omp_env_limit = ULONG_MAX;
+
+  if (query == NPROC_CURRENT_OVERRIDABLE)
+    {
+      unsigned long int omp_env_threads;
+      /* Honor the OpenMP environment variables, recognized also by all
+         programs that are based on OpenMP.  */
+      omp_env_threads = parse_omp_threads (getenv ("OMP_NUM_THREADS"));
+      omp_env_limit = parse_omp_threads (getenv ("OMP_THREAD_LIMIT"));
+      if (! omp_env_limit)
+        omp_env_limit = ULONG_MAX;
+
+      if (omp_env_threads)
+        return MIN (omp_env_threads, omp_env_limit);
+
+      query = NPROC_CURRENT;
+    }
+  /* Here query is one of NPROC_ALL, NPROC_CURRENT.  */
+  {
+    unsigned long nprocs = num_processors_ignoring_omp (query);
+    return MIN (nprocs, omp_env_limit);
+  }
+}
diff --git a/lib/nproc.h b/lib/nproc.h
new file mode 100644
index 0000000000..d7659a5cad
--- /dev/null
+++ b/lib/nproc.h
@@ -0,0 +1,46 @@
+/* Detect the number of processors.
+
+   Copyright (C) 2009-2021 Free Software Foundation, Inc.
+
+   This file is free software: you can redistribute it and/or modify
+   it under the terms of the GNU Lesser General Public License as
+   published by the Free Software Foundation; either version 2.1 of the
+   License, or (at your option) any later version.
+
+   This file is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public License
+   along with this program.  If not, see <https://www.gnu.org/licenses/>.  */
+
+/* Written by Glen Lenker and Bruno Haible.  */
+
+/* Allow the use in C++ code.  */
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* A "processor" in this context means a thread execution unit, that is either
+   - an execution core in a (possibly multi-core) chip, in a (possibly multi-
+     chip) module, in a single computer, or
+   - a thread execution unit inside a core
+     (hyper-threading, see <https://en.wikipedia.org/wiki/Hyper-threading>).
+   Which of the two definitions is used, is unspecified.  */
+
+enum nproc_query
+{
+  NPROC_ALL,                 /* total number of processors */
+  NPROC_CURRENT,             /* processors available to the current process */
+  NPROC_CURRENT_OVERRIDABLE  /* likewise, but overridable through the
+                                OMP_NUM_THREADS environment variable */
+};
+
+/* Return the total number of processors.  The result is guaranteed to
+   be at least 1.  */
+extern unsigned long int num_processors (enum nproc_query query);
+
+#ifdef __cplusplus
+}
+#endif /* C++ */
diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 63d4a74b54..0052fd0f8d 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -3876,26 +3876,13 @@ comp-async-runnings
    do (remhash file-name comp-async-compilations))
   (hash-table-count comp-async-compilations))
 
-(declare-function w32-get-nproc "w32.c")
 (defvar comp-num-cpus nil)
 (defun comp-effective-async-max-jobs ()
   "Compute the effective number of async jobs."
   (if (zerop native-comp-async-jobs-number)
       (or comp-num-cpus
           (setf comp-num-cpus
-                ;; FIXME: we already have a function to determine
-                ;; the number of processors, see get_native_system_info in w32.c.
-                ;; The result needs to be exported to Lisp.
-                (max 1 (/ (cond ((eq 'windows-nt system-type)
-                                 (w32-get-nproc))
-                                ((executable-find "nproc")
-                                 (string-to-number
-                                  (shell-command-to-string "nproc")))
-                                ((eq 'berkeley-unix system-type)
-                                 (string-to-number
-                                  (shell-command-to-string "sysctl -n hw.ncpu")))
-                                (t 1))
-                          2))))
+		(max 1 (/ (num-processors) 2))))
     native-comp-async-jobs-number))
 
 (defvar comp-last-scanned-async-output nil)
diff --git a/m4/gnulib-comp.m4 b/m4/gnulib-comp.m4
index a795fe7651..e314edcfb5 100644
--- a/m4/gnulib-comp.m4
+++ b/m4/gnulib-comp.m4
@@ -139,6 +139,7 @@ AC_DEFUN
   # Code from module mktime-internal:
   # Code from module multiarch:
   # Code from module nocrash:
+  # Code from module nproc:
   # Code from module nstrftime:
   # Code from module open:
   # Code from module openat-h:
@@ -413,6 +414,7 @@ AC_DEFUN
   fi
   gl_TIME_MODULE_INDICATOR([mktime])
   gl_MULTIARCH
+  gl_NPROC
   gl_FUNC_GNU_STRFTIME
   gl_PATHMAX
   gl_FUNC_PIPE2
@@ -1221,6 +1223,8 @@ AC_DEFUN
   lib/mkostemp.c
   lib/mktime-internal.h
   lib/mktime.c
+  lib/nproc.c
+  lib/nproc.h
   lib/nstrftime.c
   lib/open.c
   lib/openat-priv.h
@@ -1370,6 +1374,7 @@ AC_DEFUN
   m4/mode_t.m4
   m4/multiarch.m4
   m4/nocrash.m4
+  m4/nproc.m4
   m4/nstrftime.m4
   m4/off_t.m4
   m4/open-cloexec.m4
diff --git a/m4/nproc.m4 b/m4/nproc.m4
new file mode 100644
index 0000000000..887c66bee8
--- /dev/null
+++ b/m4/nproc.m4
@@ -0,0 +1,54 @@
+# nproc.m4 serial 5
+dnl Copyright (C) 2009-2021 Free Software Foundation, Inc.
+dnl This file is free software; the Free Software Foundation
+dnl gives unlimited permission to copy and/or distribute it,
+dnl with or without modifications, as long as this notice is preserved.
+
+AC_DEFUN([gl_NPROC],
+[
+  gl_PREREQ_NPROC
+])
+
+# Prerequisites of lib/nproc.c.
+AC_DEFUN([gl_PREREQ_NPROC],
+[
+  dnl Persuade glibc <sched.h> to declare CPU_SETSIZE, CPU_ISSET etc.
+  AC_REQUIRE([AC_USE_SYSTEM_EXTENSIONS])
+
+  AC_CHECK_HEADERS([sys/pstat.h sys/sysmp.h sys/param.h],,,
+    [AC_INCLUDES_DEFAULT])
+  dnl <sys/sysctl.h> requires <sys/param.h> on OpenBSD 4.0.
+  AC_CHECK_HEADERS([sys/sysctl.h],,,
+    [AC_INCLUDES_DEFAULT
+     #if HAVE_SYS_PARAM_H
+     # include <sys/param.h>
+     #endif
+    ])
+
+  AC_CHECK_FUNCS([sched_getaffinity sched_getaffinity_np \
+                  pstat_getdynamic sysmp sysctl])
+
+  dnl Test whether sched_getaffinity has the expected declaration.
+  dnl glibc 2.3.[0-2]:
+  dnl   int sched_getaffinity (pid_t, unsigned int, unsigned long int *);
+  dnl glibc 2.3.3:
+  dnl   int sched_getaffinity (pid_t, cpu_set_t *);
+  dnl glibc >= 2.3.4:
+  dnl   int sched_getaffinity (pid_t, size_t, cpu_set_t *);
+  if test $ac_cv_func_sched_getaffinity = yes; then
+    AC_CACHE_CHECK([for glibc compatible sched_getaffinity],
+      [gl_cv_func_sched_getaffinity3],
+      [AC_COMPILE_IFELSE(
+         [AC_LANG_PROGRAM(
+            [[#include <errno.h>
+              #include <sched.h>]],
+            [[sched_getaffinity (0, 0, (cpu_set_t *) 0);]])],
+         [gl_cv_func_sched_getaffinity3=yes],
+         [gl_cv_func_sched_getaffinity3=no])
+      ])
+    if test $gl_cv_func_sched_getaffinity3 = yes; then
+      AC_DEFINE([HAVE_SCHED_GETAFFINITY_LIKE_GLIBC], [1],
+        [Define to 1 if sched_getaffinity has a glibc compatible declaration.])
+    fi
+  fi
+])
diff --git a/src/process.c b/src/process.c
index 221d4c7f6c..746cdc0428 100644
--- a/src/process.c
+++ b/src/process.c
@@ -90,6 +90,7 @@ #define HAVE_LOCAL_SOCKETS
 
 #include <c-ctype.h>
 #include <flexmember.h>
+#include <nproc.h>
 #include <sig2str.h>
 #include <verify.h>
 
@@ -8212,6 +8213,20 @@ DEFUN ("process-attributes", Fprocess_attributes,
   return system_process_attributes (pid);
 }
 
+DEFUN ("num-processors", Fnum_processors, Snum_processors, 0, 1, 0,
+       doc: /* Return the number of processors, a positive integer.
+Each usable thread execution unit counts as a processor.
+By default, count the number of available processors,
+overridable via the OMP_NUM_THREADS environment variable.
+If optional argument QUERY is `current', ignore OMP_NUM_THREADS.
+If QUERY is `all', also count processors not available.  */)
+  (Lisp_Object query)
+{
+  return make_uint (num_processors (EQ (query, Qall) ? NPROC_ALL
+				    : EQ (query, Qcurrent) ? NPROC_CURRENT
+				    : NPROC_CURRENT_OVERRIDABLE));
+}
+
 #ifdef subprocesses
 /* Arrange to catch SIGCHLD if this hasn't already been arranged.
    Invoke this after init_process_emacs, and after glib and/or GNUstep
@@ -8472,6 +8487,8 @@ syms_of_process (void)
   DEFSYM (Qpcpu, "pcpu");
   DEFSYM (Qpmem, "pmem");
   DEFSYM (Qargs, "args");
+  DEFSYM (Qall, "all");
+  DEFSYM (Qcurrent, "current");
 
   DEFVAR_BOOL ("delete-exited-processes", delete_exited_processes,
 	       doc: /* Non-nil means delete processes immediately when they exit.
@@ -8633,4 +8650,5 @@ #define ADD_SUBFEATURE(key, val) \
   defsubr (&Sprocess_inherit_coding_system_flag);
   defsubr (&Slist_system_processes);
   defsubr (&Sprocess_attributes);
+  defsubr (&Snum_processors);
 }
diff --git a/src/w32proc.c b/src/w32proc.c
index 702ea122e6..3b7d92a2aa 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -3878,6 +3878,7 @@ w32_compare_strings (const char *s1, const char *s2, char *locname,
   return val - 2;
 }
 
+/* FIXME: Remove, merging any of its special features into num-processors.  */
 DEFUN ("w32-get-nproc", Fw32_get_nproc,
        Sw32_get_nproc, 0, 0, 0,
        doc: /* Return the number of system's processor execution units.  */)
diff --git a/test/src/process-tests.el b/test/src/process-tests.el
index e39f57d23b..44f3ea2fbb 100644
--- a/test/src/process-tests.el
+++ b/test/src/process-tests.el
@@ -946,5 +946,11 @@ process-async-https-with-delay
       (when buf
         (kill-buffer buf)))))
 
+(ert-deftest process-num-processors ()
+  "Sanity checks for num-processors."
+  (should (equal (num-processors) (num-processors)))
+  (should (integerp (num-processors)))
+  (should (< 0 (num-processors))))
+
 (provide 'process-tests)
 ;;; process-tests.el ends here
-- 
2.30.2

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/10/21 5:21 AM, Stefan Kangas wrote:
>> Omar Polo <op@omarpolo.com> writes:
>>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>>> misleading.  For example, on my machine
>>>
>>>          % uname -a
>>>          OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>>          % sysctl hw.ncpu
>>>          hw.ncpu=8
>>>          % sysctl hw.ncpuonline
>>>          hw.ncpuonline=4
>>>
>>> and this has been the case for a while already (I mean, a couple of
>>> years if not more.)
>> BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
>> Maybe that should be reported to the Gnulib developers?
>
>
> No need for a bug report as I'm one of those developers. To fix the
> Gnulib nproc OpenBSD issue I installed the first attached patch into
> Gnulib, and to port Emacs's recently-added processor-count code to
> more platforms I installed the second attached patch into Emacs's
> emacs-28 branch. Although I think the latter patch means that
> emacs/src/w32proc.c's w32-get-nproc function can be removed, I'll let
> the Emacs MS-Windows experts opine on that.

I had just cloned glib when I saw your mail :)

FWIW the first patch reads fine to me.  Thanks!

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/10/21 5:21 AM, Stefan Kangas wrote:
>> Omar Polo <op@omarpolo.com> writes:
>>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>>> misleading.  For example, on my machine
>>>
>>>          % uname -a
>>>          OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>>          % sysctl hw.ncpu
>>>          hw.ncpu=8
>>>          % sysctl hw.ncpuonline
>>>          hw.ncpuonline=4
>>>
>>> and this has been the case for a while already (I mean, a couple of
>>> years if not more.)
>> BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
>> Maybe that should be reported to the Gnulib developers?
>
>
> No need for a bug report as I'm one of those developers. To fix the Gnulib nproc
> OpenBSD issue I installed the first attached patch into Gnulib, and to port
> Emacs's recently-added processor-count code to more platforms I installed the
> second attached patch into Emacs's emacs-28 branch. Although I think the latter
> patch means that emacs/src/w32proc.c's w32-get-nproc function can be removed,
> I'll let the Emacs MS-Windows experts opine on that.

So you just included entire nproc into Emacs :).  Ok. Anyway; it only gives
number of "logical cores" when "hyperthreading" is on not physical. I don't
think nproc can give you that either.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andy Moreton]

On Sun 10 Oct 2021, Campbell Barton wrote:

> Hi, this patch adds support for accessing the number of CPU's / cores on a
> system, matching CPython's multiprocessing.cpu_count() [0].
>
> I've only tested this for Linux, this includes code that should work on other
> platforms, although that would need to be double-checked of course.
> For reference I checked CPython / Blender & Stack-overflow [1]
>
> Accessing this information can be useful to automatically detect the number of
> jobs to run.

This logic already exists in `comp-effective-async-max-jobs' from
comp.el, as an internal implementation detail. Perhaps it would be
better to refactor that code.

    AndyM

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Sun 10 Oct 2021, Campbell Barton wrote:
>
>> Hi, this patch adds support for accessing the number of CPU's / cores on a
>> system, matching CPython's multiprocessing.cpu_count() [0].
>>
>> I've only tested this for Linux, this includes code that should work on other
>> platforms, although that would need to be double-checked of course.
>> For reference I checked CPython / Blender & Stack-overflow [1]
>>
>> Accessing this information can be useful to automatically detect the number of
>> jobs to run.
>
> This logic already exists in `comp-effective-async-max-jobs' from
> comp.el, as an internal implementation detail. Perhaps it would be
> better to refactor that code.
>
We must have been typing at the same time :).

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Stefan Kangas]

Campbell Barton <ideasman42@gmail.com> writes:

> Hi, this patch adds support for accessing the number of CPU's / cores on
> a system, matching CPython's multiprocessing.cpu_count() [0].
>
> I've only tested this for Linux, this includes code that should work on
> other platforms, although that would need to be double-checked of course.
> For reference I checked CPython / Blender & Stack-overflow [1]
>
> Accessing this information can be useful to automatically detect the
> number of jobs to run.

Could we reuse the num_processors_ignoring_omp function in nproc.c in Gnulib?

From a cursory look, it seems to support also e.g. HP-UX, IRIX.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Campbell Barton]

On 10/10/21 23:09, Stefan Kangas wrote:
> Campbell Barton <ideasman42@gmail.com> writes:
> 
>> Hi, this patch adds support for accessing the number of CPU's / cores on
>> a system, matching CPython's multiprocessing.cpu_count() [0].
>>
>> I've only tested this for Linux, this includes code that should work on
>> other platforms, although that would need to be double-checked of course.
>> For reference I checked CPython / Blender & Stack-overflow [1]
>>
>> Accessing this information can be useful to automatically detect the
>> number of jobs to run.
> 
> Could we reuse the num_processors_ignoring_omp function in nproc.c in Gnulib?
> 
>  From a cursory look, it seems to support also e.g. HP-UX, IRIX.
> 

This makes use of CPU affinity (where possible), which would be 
appropriate for Emacs spawning operating system level threads.

Since Python's multiprocessing is straightforward and widely used, 
following the same behavior seems reasonable.

If CPU affinity becomes an important this could be supported if/when 
it's needed.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Po Lu]

Campbell Barton <ideasman42@gmail.com> writes:

> Hi, this patch adds support for accessing the number of CPU's / cores
> on a system, matching CPython's multiprocessing.cpu_count() [0].

Forgive me if I'm wrong, but doesn't this belong in sysdep.c, and not
emacs.c?

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Campbell Barton]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

On 10/11/21 12:34, Po Lu wrote:
> Campbell Barton <ideasman42@gmail.com> writes:
> 
>> Hi, this patch adds support for accessing the number of CPU's / cores
>> on a system, matching CPython's multiprocessing.cpu_count() [0].
> 
> Forgive me if I'm wrong, but doesn't this belong in sysdep.c, and not
> emacs.c?
> 

This seems like a better location. Updated the patch, including 
OpenBSD's ncpuonline support from Omar Polo.

[-- Attachment #2: 0001-processor-count.patch --]
[-- Type: text/x-patch, Size: 2546 bytes --]

commit 336affe9277fd0dee1d6bec028bb17f080a841af
Author: Campbell Barton <ideasman42@gmail.com>
Date:   Sun Oct 10 10:16:47 2021 +1100

    Support accessing the number of processors
    
    Add (processor-count) for accessing the number of cores/CPU's.

diff --git a/src/emacs.c b/src/emacs.c
index 866e43fda9..5715325ffe 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -3156,6 +3156,15 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0,
   return Qt;
 }
 
+DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0,
+       doc: /* Return the number of processors on this system.
+
+The value will always be above zero, 1 for unsupported systems.  */)
+  (void)
+{
+  return make_fixnum (get_processor_count());
+}
+
 void
 syms_of_emacs (void)
 {
@@ -3176,6 +3185,7 @@ syms_of_emacs (void)
   defsubr (&Sinvocation_directory);
   defsubr (&Sdaemonp);
   defsubr (&Sdaemon_initialized);
+  defsubr (&Sprocessor_count);
 
   DEFVAR_LISP ("command-line-args", Vcommand_line_args,
 	       doc: /* Args passed by shell to Emacs, as a list of strings.
diff --git a/src/lisp.h b/src/lisp.h
index 480c389a3b..b74bcae7fb 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4618,6 +4618,7 @@ maybe_disable_address_randomization (int argc, char **argv)
 extern EMACS_INT get_random (void);
 extern void seed_random (void *, ptrdiff_t);
 extern void init_random (void);
+extern int get_processor_count (void);
 extern void emacs_backtrace (int);
 extern AVOID emacs_abort (void) NO_INLINE;
 extern int emacs_fstatat (int, char const *, void *, int);
diff --git a/src/sysdep.c b/src/sysdep.c
index 8eaee22498..a46e3ce200 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2182,6 +2182,38 @@ get_random (void)
   return val & INTMASK;
 }
 
+/*
+ * Return the number of CPU's / cores (>= 1).
+ */
+int
+get_processor_count (void)
+{
+  int nproc = -1;
+#ifdef WINDOWSNT
+  SYSTEM_INFO info;
+  GetSystemInfo(&info);
+  nproc = (int)info.dwNumberOfProcessors;
+#elif defined (HW_NCPUONLINE) || defined (HW_NCPU)
+  int mib[2];
+  size_t len;
+
+  mib[0] = CTL_HW;
+# ifdef HW_NCPUONLINE
+  mib[1] = HW_NCPUONLINE;
+# else
+  mib[1] = HW_NCPU;
+# endif
+  len = sizeof nproc;
+  sysctl(mib, 2, &nproc, &len, NULL, 0);
+#elif defined (__hpux)
+  nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL);
+#elif defined (_SC_NPROCESSORS_ONLN)
+  nproc = (int)sysconf(_SC_NPROCESSORS_ONLN);
+#endif
+
+  return (nproc > 0) ? nproc : 1;
+}
+
 #ifndef HAVE_SNPRINTF
 /* Approximate snprintf as best we can on ancient hosts that lack it.  */
 int

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Po Lu]

Campbell Barton <ideasman42@gmail.com> writes:

> This seems like a better location. Updated the patch, including
> OpenBSD's ncpuonline support from Omar Polo.

Doesn't gnulib already have a portable mechanism to obtain the number of
CPUs in a system?  And if not, wouldn't it be useful there?

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Campbell Barton]

On 10/11/21 14:04, Po Lu wrote:
> Campbell Barton <ideasman42@gmail.com> writes:
> 
>> This seems like a better location. Updated the patch, including
>> OpenBSD's ncpuonline support from Omar Polo.
> 
> Doesn't gnulib already have a portable mechanism to obtain the number of
> CPUs in a system?  And if not, wouldn't it be useful there?

Yes, `num_processors (NPROC_ALL)` is an equivalent that would work.

Correction to my previous mail which mentioned CPU affinity being an 
issue, that's only used for `NPROC_CURRENT`.

If the additional complexity of GNULib's `lib/nproc.c` isn't viewed as a 
major down-side (~279-SLOC ~27-SLOC from the patch I submitted), both 
can work.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Lars Ingebrigtsen]

Campbell Barton <ideasman42@gmail.com> writes:

> If the additional complexity of GNULib's `lib/nproc.c` isn't viewed as
> a major down-side (~279-SLOC ~27-SLOC from the patch I submitted),
> both can work.

Note that Paul already pushed the gnulib nproc solution to emacs-28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 11 Oct 2021 10:20:49 +0200
> Cc: Po Lu <luangruo@yahoo.com>, emacs-devel@gnu.org
> 
> Campbell Barton <ideasman42@gmail.com> writes:
> 
> > If the additional complexity of GNULib's `lib/nproc.c` isn't viewed as
> > a major down-side (~279-SLOC ~27-SLOC from the patch I submitted),
> > both can work.
> 
> Note that Paul already pushed the gnulib nproc solution to emacs-28.

It's water under the bridge, but this kind of change shouldn't have
been installed without a discussion.  We have just pushed the first
pretest at least a week forward...

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Stefan Monnier]

> Note that Paul already pushed the gnulib nproc solution to emacs-28.

Why emacs-28?
This is purely a new feature, and I can't see any urgency that would
justify bypassing the general rule to only push bug fixes to emacs-28.


        Stefan

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Campbell Barton <ideasman42@gmail.com>,  Po Lu <luangruo@yahoo.com>,
>  emacs-devel@gnu.org
> Date: Mon, 11 Oct 2021 11:12:22 -0400
> 
> > Note that Paul already pushed the gnulib nproc solution to emacs-28.
> 
> Why emacs-28?
> This is purely a new feature, and I can't see any urgency that would
> justify bypassing the general rule to only push bug fixes to emacs-28.

100% agreement.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Andy Moreton]

On Mon 11 Oct 2021, Eli Zaretskii wrote:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Campbell Barton <ideasman42@gmail.com>,  Po Lu <luangruo@yahoo.com>,
>>  emacs-devel@gnu.org
>> Date: Mon, 11 Oct 2021 11:12:22 -0400
>> 
>> > Note that Paul already pushed the gnulib nproc solution to emacs-28.
>> 
>> Why emacs-28?
>> This is purely a new feature, and I can't see any urgency that would
>> justify bypassing the general rule to only push bug fixes to emacs-28.
>
> 100% agreement.

Indeed.

The followup changes in commit 1a1b206a8b33 result in `w32-get-nproc'
being removed, so should there be an obsolete alias for this function ?
Otherwise users will see it removed from emacs without any deprecation
period.

    AndyM

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Ken Brown]

On 10/11/2021 5:14 PM, Andy Moreton wrote:
> The followup changes in commit 1a1b206a8b33 result in `w32-get-nproc'
> being removed, so should there be an obsolete alias for this function ?
> Otherwise users will see it removed from emacs without any deprecation
> period.

I don't think that function was ever in any released version of emacs.

Ken

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Mon, 11 Oct 2021 22:14:18 +0100
> 
> The followup changes in commit 1a1b206a8b33 result in `w32-get-nproc'
> being removed, so should there be an obsolete alias for this function ?
> Otherwise users will see it removed from emacs without any deprecation
> period.

That function was introduced for Emacs 28 (early this year), so it has
no history worthy of an alias.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Why emacs-28?
>> This is purely a new feature, and I can't see any urgency that would
>> justify bypassing the general rule to only push bug fixes to emacs-28.
>
> 100% agreement.

Me too.  Would it make sense to revert it and reapply on master, or
would that just slow the release down even more?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  ideasman42@gmail.com,
>   luangruo@yahoo.com,  emacs-devel@gnu.org, Paul Eggert
>  <eggert@cs.ucla.edu>
> Date: Tue, 12 Oct 2021 12:39:03 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Why emacs-28?
> >> This is purely a new feature, and I can't see any urgency that would
> >> justify bypassing the general rule to only push bug fixes to emacs-28.
> >
> > 100% agreement.
> 
> Me too.  Would it make sense to revert it and reapply on master, or
> would that just slow the release down even more?

The latter, I think.  It's too late to fix that, sorry.  Especially as
Paul installed quite a few more changes, and I have no idea whether
they somehow assume that one.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Paul Eggert]

On 10/12/21 07:09, Eli Zaretskii wrote:

> The latter, I think.  It's too late to fix that, sorry.  Especially as
> Paul installed quite a few more changes, and I have no idea whether
> they somehow assume that one.

They should be independent.

I thought it was a bugfix since it addressed a FIXME and it fixed a 
performance bug on OpenBSD. But if you'd rather I reverted on emacs-28 
and installed on master instead, I can volunteer to do that.

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Eli Zaretskii]

> Date: Tue, 12 Oct 2021 12:58:46 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: luangruo@yahoo.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>  ideasman42@gmail.com
> 
> I thought it was a bugfix since it addressed a FIXME and it fixed a 
> performance bug on OpenBSD. But if you'd rather I reverted on emacs-28 
> and installed on master instead, I can volunteer to do that.

Thanks, but I think it's too late for that.