From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Arthur Miller Newsgroups: gmane.emacs.devel Subject: Re: [PATCH] support for accessing CPU/core count (processor-count) Date: Sun, 10 Oct 2021 12:52:17 +0200 Message-ID: References: <87h7dpgues.fsf@omarpolo.com> <871r4tgp9o.fsf@omarpolo.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="19389"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) Cc: Campbell Barton , emacs-devel@gnu.org To: Omar Polo Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Oct 10 12:54:39 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mZWTV-0004nJ-On for ged-emacs-devel@m.gmane-mx.org; Sun, 10 Oct 2021 12:54:38 +0200 Original-Received: from localhost ([::1]:55816 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mZWTU-0003Vb-LR for ged-emacs-devel@m.gmane-mx.org; Sun, 10 Oct 2021 06:54:36 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:56762) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZWRN-00010g-At for emacs-devel@gnu.org; Sun, 10 Oct 2021 06:52:26 -0400 Original-Received: from mail-oln040092074095.outbound.protection.outlook.com ([40.92.74.95]:46914 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZWRJ-0001t1-Ig for emacs-devel@gnu.org; Sun, 10 Oct 2021 06:52:24 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bSV5II8G3dB9OH/cUaQSb7Fvp8zQOGhdSlIBhdHU6HlvVWFRz2Cs5l4LzGbDSuIxwBXvQ8+yM1XQB4rFU0WobmBmfltZY6NpZhuFpnoBdv6nqyYydsN/1IplsusmArshteJnboiPCH9owEBAiGLnwIZFi21WvDfJlLbVate8yTC2dR9Mw75lXDjx78B9XlkfyhawL463GBMwZ6agmuXOf6gxjqZfHyjumTrqRVdCuq5yRdUjNS+erYT/5+cJOHFDo/SD+810jgIlwbyspgWNAX3DyLFjr7vzQRRMI6nFNZH4eZw9SXlz3yfd2ZbChgzrtywxfV+Yfy8XvAqXYChrZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qNQrKB58Si8lmP2ouOKpL3FRpE7yKDzzEaOa+pzrbO0=; b=dRoPC1IyaK1RIoDUAou2G3AruOWFG2j2sMwzsml3+3ASdcXVdHejNZwkJWEe9MmNXB9+oi+0PwInuqZo+BYd/3fFVXTuqAmAI0RdSqntVaPl2S4mJfS0Qm/7rsLerZacN2qlEQ67+SRJKzUPcjYE0Y8zLZAkuwhbgdPmGv2kYCNCfLH/DO6UrhEZ6bX2egoizzQRkm1bSdhJddBfzKJfUxxM9ooz/Wg0CXikAAESn2PR9Gs84Ec0uvUIS3NGsNfrGNpDbP7khp+mBx2X8arobw2eJRnMLzWR9qIRpRVmDj6xOOYMnFAFUocVpUcx6z0LY7kWwrHOz6CT4jEwvRxXJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qNQrKB58Si8lmP2ouOKpL3FRpE7yKDzzEaOa+pzrbO0=; b=EzBtvVQI8NGPfZKho9wQDhIPcp+hjsO00KccrdOkYmIV1UmizD+5L25IOAwXXEur4q9IgPtc32k3S72SHKOE5dgIX+C1zF71+SHlPgcKbMJa6TLqWxw/wUO81P/KaO53zyZm+9inqYUZuvohtdlPeuuFJjc3YyOwAd+xrQZEa6HCtO4JgGHGOIQjDifTgaEJmW3C/JXmIyMi5GQN63pyRfu8CvlO002hiRUp/eXAUwv3dDtxkrA1kWK63KCKswE5f6h9v2osKdZNa/+NKXWzUxna6QZbwvps62WJeH34R3NWavoDPkZacqmOM1eZlDjKGUPkb4/AeAsIGI2lf/G0Hw== Original-Received: from AM9PR09MB4977.eurprd09.prod.outlook.com (2603:10a6:20b:304::20) by AM0PR09MB2770.eurprd09.prod.outlook.com (2603:10a6:208:128::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.25; Sun, 10 Oct 2021 10:52:17 +0000 Original-Received: from AM9PR09MB4977.eurprd09.prod.outlook.com ([fe80::c05b:e2ca:5b8a:56d2]) by AM9PR09MB4977.eurprd09.prod.outlook.com ([fe80::c05b:e2ca:5b8a:56d2%9]) with mapi id 15.20.4587.025; Sun, 10 Oct 2021 10:52:17 +0000 In-Reply-To: <871r4tgp9o.fsf@omarpolo.com> (Omar Polo's message of "Sun, 10 Oct 2021 11:43:29 +0200") X-TMN: [MjIccggQTVcUg4OZrQzdLef0ExIxzShD] X-ClientProxiedBy: AS9PR06CA0281.eurprd06.prod.outlook.com (2603:10a6:20b:45a::30) To AM9PR09MB4977.eurprd09.prod.outlook.com (2603:10a6:20b:304::20) X-Microsoft-Original-Message-ID: <87v9252lum.fsf@live.com> X-MS-Exchange-MessageSentRepresentingType: 1 Original-Received: from pascal.homepc (81.232.177.30) by AS9PR06CA0281.eurprd06.prod.outlook.com (2603:10a6:20b:45a::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.21 via Frontend Transport; Sun, 10 Oct 2021 10:52:17 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7be861f4-f1ba-418a-edf1-08d98bdc06ec X-MS-TrafficTypeDiagnostic: AM0PR09MB2770: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0nJXbbSb4THUXC3hgFFEbURDfC0csGH/5SqHc4Okm4hP2esEE16q3hJN6lICkE+ELurD8RV0W8d+iB6QwrT3Q3VgbLcazLFHXEOHxH9M9FT0JEHdQjUc8/1Km9VTV1c+62Snt6ewOHtyoIn9yNnx1QQK22PcTUYyTLQUEP7n1si94TiS+16CDg76eBHd4Cb1rv3VbGjnyARBLvI9IT2BLrqP8RYkxzAO5v9ovHzVuFpp6bUS+PfzMKYXvUIIZNX3VBq/WAtdzNb96LHJCA/XspO6e2SpRUfGz2UK2VHlTsVcmpNgNLJPLWWItvup+Mt0QBr33q0+NjU7xTZohodyLm6UQqOLKsuolx5ytvIMTTobwlklbnvfI7WKzCFWRu5LR5wavYUUBDN7hCnJi0nxAVB4TDVG9t8utoe3DntqqFJh+c5sA7kfhcdSlBnlRhLP+8nWWXsoNWN1nsvr5s5aCjK9tlk/3VpgcF62rGvX5/XFujfO2P6/BVcsnu/XOLExpBOIXYqSDj4ytk8GneupiQ== X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NfUZlUrVlNLycQKrgqI8Yr4iTcZUVNiiNHbujH1PuHl9wUtAQ5U0TEItRpPK/VXilKky3anPY1f+/77/CGYWA7hNu2pQPYtUuY/UZuhmD5LpV6NLYFuTbZtiR/o7ooQ3OxRdOJ7bQPxtKDTYn0VGzQ== X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-72e6e.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: 7be861f4-f1ba-418a-edf1-08d98bdc06ec X-MS-Exchange-CrossTenant-AuthSource: AM9PR09MB4977.eurprd09.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2021 10:52:17.7899 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR09MB2770 Received-SPF: pass client-ip=40.92.74.95; envelope-from=arthur.miller@live.com; helo=EUR04-DB3-obe.outbound.protection.outlook.com X-Spam_score_int: -1 X-Spam_score: -0.2 X-Spam_bar: / X-Spam_report: (-0.2 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:276655 Archived-At: Omar Polo writes: > Arthur Miller writes: > >> Omar Polo writes: >> >>> Campbell Barton writes: >>> >>>> Hi, this patch adds support for accessing the number of CPU's / cores >>>> on a system, matching CPython's multiprocessing.cpu_count() [0]. >>>> >>>> I've only tested this for Linux, this includes code that should work >>>> on other platforms, although that would need to be double-checked of >>>> course. >>>> For reference I checked CPython / Blender & Stack-overflow [1] >>> >>> I can confirm it works on OpenBSD too, but needs tweaking. Some >>> comments inline and attaching an updated patch >>> >>>> Accessing this information can be useful to automatically detect the >>>> number of jobs to run. >>>> >>>> >>>> [0]: >>>> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count >>>> [1]: https://stackoverflow.com/a/3006416/432509 >>>> >>>> commit 7be53f10f3df3c3183cc97d6bbadb78ebb61e8d2 >>>> Author: Campbell Barton >>>> Date: Sun Oct 10 10:16:47 2021 +1100 >>>> >>>> Support accessing the number of CPU's. >>>> >>>> Add (processor-count) for accessing the number of cores/CPU's. >>>> >>>> diff --git a/src/emacs.c b/src/emacs.c >>>> index 866e43fda9..26e2f6b1f2 100644 >>>> --- a/src/emacs.c >>>> +++ b/src/emacs.c >>>> @@ -3156,6 +3156,38 @@ DEFUN ("daemon-initialized", Fdaemon_initialized, Sdaemon_initialized, 0, 0, 0, >>>> return Qt; >>>> } >>>> >>>> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0, >>>> + doc: /* Return the number of CPUs in the system. >>>> + >>>> +The value will always be above zero, 1 for unsupported systems. */) >>>> + (void) >>>> +{ >>>> + int nproc = -1; >>>> +#ifdef WINDOWSNT >>>> + SYSTEM_INFO info; >>>> + GetSystemInfo(&info); >>>> + nproc = (int)info.dwNumberOfProcessors; >>>> +#elif defined (__APPLE__) || \ >>>> + defined (__OpenBSD__) || \ >>>> + defined (__FreeBSD__) || \ >>>> + defined (__NetBSD__) || \ >>>> + defined (__DragonFly__) >>>> + int mib[2]; >>>> + size_t len; >>>> + >>>> + mib[0] = CTL_HW; >>> >>> the #include is missing. >>> >>>> + mib[1] = HW_NCPU; >>> >>> at least on OpenBSD this should be HW_NCPUONLINE. >>> >>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always >>> misleading. For example, on my machine >>> >>> % uname -a >>> OpenBSD venera 7.0 GENERIC.MP#221 amd64 >>> % sysctl hw.ncpu >>> hw.ncpu=8 >>> % sysctl hw.ncpuonline >>> hw.ncpuonline=4 >>> >>> and this has been the case for a while already (I mean, a couple of >>> years if not more.) >>> >>> I don't have access to other BSDs other than OpenBSD, but judging from >>> the manpages online. >>> >>> - NetBSD has hw.ncpuonline >>> - DragonFly and FreeBSD have only hw.ncpu >>> - apple I don't know how to check. man.apple.com doesn't seem to >>> exists ^^" >>> >>>> + len = sizeof(nproc); >>>> + sysctl(mib, 2, &nproc, &len, nullptr, 0); >>> ^^^^^^^ >>> shouldn't this be NULL? >>> >>>> +#elif defined (__hpux) >>>> + nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL); >>>> +#elif defined (_SC_NPROCESSORS_ONLN) >>>> + nproc = (int)sysconf(_SC_NPROCESSORS_ONLN); >>>> +#endif >>>> + >>>> + return make_fixnum (MAX(nproc, 1)); >>> >>> emacs.c:3188:23: warning: implicit declaration of function 'MAX' is invalid in C99 [-Wimplicit-function-declaration] >>> >>> I'm attaching an updated patch. Note that I'm not sure if including >>> sys/types.h and sys/sysctl.h breaks the build on some OS. >>> >>> if someone knows how to reduce the number of #ifdefs I'll be glad :) >>> >>>> +} >>>> + >>> >>> Cheers, >>> >>> Omar Polo >>> >>> From 5fe26a73abc60eae7dd2528e3854f2f379bcb4c7 Mon Sep 17 00:00:00 2001 >>> From: Campbell Barton >>> Date: Sun, 10 Oct 2021 08:21:01 +0000 >>> Subject: [PATCH] add function to access the number of CPU >>> >>> * emacs.c (Fprocessor_count): add processor-count function to access >>> the number of CPU >>> --- >>> src/emacs.c | 35 +++++++++++++++++++++++++++++++++++ >>> 1 file changed, 35 insertions(+) >>> >>> diff --git a/src/emacs.c b/src/emacs.c >>> index 866e43fda9..5bd10a7586 100644 >>> --- a/src/emacs.c >>> +++ b/src/emacs.c >>> @@ -25,6 +25,8 @@ along with GNU Emacs. If not, see . */ >>> #include >>> #include >>> >>> +#include >>> +#include >>> #include >>> #include >>> #include >>> @@ -3156,6 +3158,38 @@ from the parent process and its tty file descriptors. */) >>> return Qt; >>> } >>> >>> +DEFUN ("processor-count", Fprocessor_count, Sprocessor_count, 0, 0, 0, >>> + doc: /* Return the number of CPUs in the system. >>> + >>> +The value will always be above zero, 1 for unsupported systems. */) >>> + (void) >>> +{ >>> + int nproc = -1; >>> +#ifdef WINDOWSNT >>> + SYSTEM_INFO info; >>> + GetSystemInfo(&info); >>> + nproc = (int)info.dwNumberOfProcessors; >>> +#elif defined (HW_NCPU) || defined (HW_NCPUONLINE) >>> + int mib[2]; >>> + size_t len; >>> + >>> + mib[0] = CTL_HW; >>> +#ifdef HW_NCPUONLINE >>> + mib[1] = HW_NCPUONLINE; >>> +#else >>> + mib[1] = HW_NCPU; >>> +#endif >>> + len = sizeof(nproc); >>> + sysctl(mib, 2, &nproc, &len, NULL, 0); >>> +#elif defined (__hpux) >>> + nproc = mpctl(MPC_GETNUMSPUS, NULL, NULL); >>> +#elif defined (_SC_NPROCESSORS_ONLN) >>> + nproc = (int)sysconf(_SC_NPROCESSORS_ONLN); >>> +#endif >>> + >>> + return make_fixnum (nproc > 1 ? nproc : 1); >>> +} >>> + >>> void >>> syms_of_emacs (void) >>> { >>> @@ -3176,6 +3210,7 @@ syms_of_emacs (void) >>> defsubr (&Sinvocation_directory); >>> defsubr (&Sdaemonp); >>> defsubr (&Sdaemon_initialized); >>> + defsubr (&Sprocessor_count); >>> >>> DEFVAR_LISP ("command-line-args", Vcommand_line_args, >>> doc: /* Args passed by shell to Emacs, as a list of strings. >> >> What is wrong on just reading this from /cat/cpuinfo on GNU/Linux? >> >> Windows has "wmic cpu get NumberOfCores,NumberOfLogicalProcessors" which works >> on cmd prompt, which means executable from elisp as a process. I dont know for >> apple and *bsds. >> >> It is just text that can be parsed directly with elisp, so it is easier to >> maintain does not need to be compiled etc. I don't see cpu queries as a >> performance crucial query. Just a thought. > > You're right > > syscstl -n hw.ncpuonline > > would do it. I guess that a rationale for doing this in C would be to > make it easier on the elisp side to use it. We can try to support every > OS here instead of leaving the burden on the elisp package authors. > > (the same thing applies to proced. You could implement it by parsing ps > output I guess, but there are elisp API that calls into C for that.) > > But I don't know, I don't really have strong opinions. I've read a bit > of code that didn't look right and tried to improve it :) By the way; I just realize also that native compiler does this, and there is also a note about exporting already written c function to lisp: #begin_src emacs-lisp (declare-function w32-get-nproc "w32.c") (defvar comp-num-cpus nil) (defun comp-effective-async-max-jobs () "Compute the effective number of async jobs." (if (zerop native-comp-async-jobs-number) (or comp-num-cpus (setf comp-num-cpus ;; FIXME: we already have a function to determine ;; the number of processors, see get_native_system_info in w32.c. ;; The result needs to be exported to Lisp. (max 1 (/ (cond ((eq 'windows-nt system-type) (w32-get-nproc)) ((executable-find "nproc") (string-to-number (shell-command-to-string "nproc"))) ((eq 'berkeley-unix system-type) (string-to-number (shell-command-to-string "sysctl -n hw.ncpu"))) (t 1)) 2)))) native-comp-async-jobs-number)) #end_src Maybe you can just extract the cpu number part from Andrea's function and rename into some public API independent of native comp?