From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Anders Lindgren Newsgroups: gmane.emacs.bugs Subject: bug#22814: 25.0.91; Emacs runs out of file descriptors on OS X Date: Sat, 27 Feb 2016 20:00:30 +0100 Message-ID: References: <87twkupoq9.fsf@gmx.de> <83d1ricxyp.fsf@gnu.org> <87lh66nv22.fsf@gmx.de> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=001a114389d4936433052cc50792 X-Trace: ger.gmane.org 1456599691 31004 80.91.229.3 (27 Feb 2016 19:01:31 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sat, 27 Feb 2016 19:01:31 +0000 (UTC) Cc: 22814@debbugs.gnu.org To: Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Sat Feb 27 20:01:11 2016 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aZk75-0000g0-0D for geb-bug-gnu-emacs@m.gmane.org; Sat, 27 Feb 2016 20:01:11 +0100 Original-Received: from localhost ([::1]:55924 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZk74-0002dF-Kt for geb-bug-gnu-emacs@m.gmane.org; Sat, 27 Feb 2016 14:01:10 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57756) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZk70-0002d3-KN for bug-gnu-emacs@gnu.org; Sat, 27 Feb 2016 14:01:07 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aZk6w-0007Jy-Ab for bug-gnu-emacs@gnu.org; Sat, 27 Feb 2016 14:01:06 -0500 Original-Received: from debbugs.gnu.org ([208.118.235.43]:51933) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZk6w-0007Ju-7Y for bug-gnu-emacs@gnu.org; Sat, 27 Feb 2016 14:01:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84) (envelope-from ) id 1aZk6v-0005FR-UT for bug-gnu-emacs@gnu.org; Sat, 27 Feb 2016 14:01:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Anders Lindgren Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 27 Feb 2016 19:01:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 22814 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 22814-submit@debbugs.gnu.org id=B22814.145659963820139 (code B ref 22814); Sat, 27 Feb 2016 19:01:01 +0000 Original-Received: (at 22814) by debbugs.gnu.org; 27 Feb 2016 19:00:38 +0000 Original-Received: from localhost ([127.0.0.1]:49060 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZk6X-0005Ek-KV for submit@debbugs.gnu.org; Sat, 27 Feb 2016 14:00:37 -0500 Original-Received: from mail-vk0-f44.google.com ([209.85.213.44]:35272) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZk6W-0005EY-CF for 22814@debbugs.gnu.org; Sat, 27 Feb 2016 14:00:36 -0500 Original-Received: by mail-vk0-f44.google.com with SMTP id e6so103182793vkh.2 for <22814@debbugs.gnu.org>; Sat, 27 Feb 2016 11:00:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=ZqPOfLczM59UFJy5jSTzGMaE7iF2I0/H1YcRgrqTbB8=; b=CDJOx84D59O9lIUWAVzbX6ldPuT/0Vtn0BYLLthurFCguR9t7xEwGsTYkE6Y7v9I5t ttynQLW0z1tdUUrOy9S2aiA5OlOhY1Us2EJvvVMWe9OkqPRZkRBJlIMqBhRGTREqSUXk Gn9HiYfvnUKNNPHkcgFxPjtQQpNHKr2iiwvOynlX99G1LTiprmKlFMxYp9Ua0pQCGJf/ XWJuStZR39cEWg8RJOTTwcl5EGncpvoSmw7Bzy1Q5K91lFVmlIy9SOeOa/inusw/tDe0 ntAJnz8QrPDIyHUYTSZN0exu6eqf2WV0Dm15uN+LoUm9ztIiuNq6bBCf3J2S0odK2oWh kgvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=ZqPOfLczM59UFJy5jSTzGMaE7iF2I0/H1YcRgrqTbB8=; b=TYizYAkpnZ+OxjGKa7hUPM7CkbP1dShjUvV0KpAJaIXOkmrPvJswbjMfZFCOZUCCc7 knUYDssmTymMBgYv1uY4qRX6hZd12ILZjcp8EUkFLOwov9FK4LLFR0YKRJQKMp1cjbM1 fYiL2cyYBS+Oz6xm4ow09bT976mIRmmSQWSMitvrH5eu4mWHgWfSsNGQlZdWT3NQHgAT Xkd9uSE6moDoB7G+myXuK+p+18WH5RmKBWkbntDWqYlTtlVWduGlEF8ilF1ZawubLNF1 3u9O6hVWII3se/JR9bXOueI6kUuxlK0bq5jZGvSlZklW+0Uo4k5PeuTnqUqjenbTkYyj lfCQ== X-Gm-Message-State: AD7BkJKzGp7NCG+7fRwn7XeYWAs+b46MyOHXwz6tfjn8SYpwd9PKcDariakxWR521z4p5oFJht1FwOESY2xu/g== X-Received: by 10.31.58.83 with SMTP id h80mr5726507vka.149.1456599630830; Sat, 27 Feb 2016 11:00:30 -0800 (PST) Original-Received: by 10.31.214.131 with HTTP; Sat, 27 Feb 2016 11:00:30 -0800 (PST) In-Reply-To: <87lh66nv22.fsf@gmx.de> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:113963 Archived-At: --001a114389d4936433052cc50792 Content-Type: text/plain; charset=UTF-8 Hi! I just gave this some thought (without knowing anything about the kqueue notifications). I don't think the problem is what kqueue does when it run out of file descriptors, but how the rest of the Emacs process acts when this happens. For example, can it even read and write files? Can subprocesses be started? The question, then, is how should the kqueue system work so that it doesn't run Emacs out of file descriptors? For example, if it would be possible to check how many are available, it could stop when there are, say, 50 remaining. Effectively, this would give a user 200 files that are auto-reverted using the notification system, and the rest would be handled by the old timer system. -- Anders On Sat, Feb 27, 2016 at 2:26 PM, Michael Albinus wrote: > Eli Zaretskii writes: > > >> > One simple way to handle this is to define a variable with "max" > >> > number of files the notification system can handle. We can set this > >> > to, say, 200 on OS X and unlimited on other systems. > >> > >> Would be possible, yes. I would prefer to set the limit to a system > >> related value. Does there exist a portable way to detect, how many file > >> descriptors / processes Emacs is able to consume? > > > > Not portably, AFAIK. > > On POSIX systems, we could propagate the result of > > if !(getrlimit (RLIMIT_NOFILE, &rlim)) > return make_number (rlim.rlim_cur); > else > return Qnil; > > Maybe we add a function like `-max-descriptors' to the > libraries. Or maybe not, and the backends check this themselves, and > cease to work when reaching an internal limit. Such an internal limit > could be half the number of soft RLIMIT_NOFILE on POSIX systems, for > example. > > > Also, different implementations use different > > resources for receiving file notifications. For example, w32notify > > uses one handle and one thread per watched file/directory. The number > > of handles a process can have on Windows is very large, and the > > theoretical max number of threads is 32K, but both are limited by the > > resources already consumed by the Emacs process. So determining the > > practical maximum at any given point will require a non-trivial > > function. > > There must be different scenarios for different file notification > libraries. But using RLIMIT_NOFILE as basis when possible gives us > better results on systems like OS X and FreeBSD, which both use kqueue, > but provide different ressource limits. > > Best regards, Michael. > --001a114389d4936433052cc50792 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi!

I just gave this some thought= (without knowing anything about the kqueue notifications).

<= div>I don't think the problem is what kqueue does when it run out of fi= le descriptors, but how the rest of the Emacs process acts when this happen= s. For example, can it even read and write files? Can subprocesses be start= ed?

The question, then, is how should the kqueue s= ystem work so that it doesn't run Emacs out of file descriptors? For ex= ample, if it would be possible to check how many are available, it could st= op when there are, say, 50 remaining. Effectively, this would give a user 2= 00 files that are auto-reverted using the notification system, and the rest= would be handled by the old timer system.

=C2=A0 = =C2=A0 -- Anders

On Sat, Feb 27, 2016 at 2:26 PM, Michael Albinus <michael.= albinus@gmx.de> wrote:
Eli Zaretskii <eliz@gnu.org= > writes:

>> > One simple way to handle this is to define a variable with &q= uot;max"
>> > number of files the notification system can handle. We can se= t this
>> > to, say, 200 on OS X and unlimited on other systems.
>>
>> Would be possible, yes. I would prefer to set the limit to a syste= m
>> related value. Does there exist a portable way to detect, how many= file
>> descriptors / processes Emacs is able to consume?
>
> Not portably, AFAIK.

On POSIX systems, we could propagate the result of

if !(getrlimit (RLIMIT_NOFILE, &rlim))
=C2=A0 =C2=A0return make_number (rlim.rlim_cur);
else
=C2=A0 =C2=A0return Qnil;

Maybe we add a function like `<library>-max-descriptors' to the libraries. Or maybe not, and the backends check this themselves, and
cease to work when reaching an internal limit. Such an internal limit
could be half the number of soft RLIMIT_NOFILE on POSIX systems, for
example.

> Also, different implementations use different
> resources for receiving file notifications.=C2=A0 For example, w32noti= fy
> uses one handle and one thread per watched file/directory.=C2=A0 The n= umber
> of handles a process can have on Windows is very large, and the
> theoretical max number of threads is 32K, but both are limited by the<= br> > resources already consumed by the Emacs process.=C2=A0 So determining = the
> practical maximum at any given point will require a non-trivial
> function.

There must be different scenarios for different file notification libraries. But using RLIMIT_NOFILE as basis when possible gives us
better results on systems like OS X and FreeBSD, which both use kqueue,
but provide different ressource limits.

Best regards, Michael.

--001a114389d4936433052cc50792--