From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Daniel Colascione Newsgroups: gmane.emacs.devel Subject: Re: When should ralloc.c be used? Date: Fri, 28 Oct 2016 01:11:08 -0700 Message-ID: <6350b2df-fde9-e716-d279-9f29438f8ee5@dancol.org> References: <838ttfnmev.fsf@gnu.org> <837f8znk8f.fsf@gnu.org> <83zilvm2ud.fsf@gnu.org> <83r377m0i8.fsf@gnu.org> <83eg36n6v5.fsf@gnu.org> <83shrl523p.fsf@gnu.org> <83eg354ux3.fsf@gnu.org> <4f0c2868-d408-a5c4-d5a8-90dae750eb33@dancol.org> <878tt9ggdk.fsf@ritchie.wxcvbn.org> <83k2cssypt.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1477642298 19542 195.159.176.226 (28 Oct 2016 08:11:38 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 28 Oct 2016 08:11:38 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 Cc: eggert@cs.ucla.edu, monnier@iro.umontreal.ca, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Fri Oct 28 10:11:30 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1c02G6-0003cS-DR for ged-emacs-devel@m.gmane.org; Fri, 28 Oct 2016 10:11:26 +0200 Original-Received: from localhost ([::1]:47363 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c02G6-0007Md-QC for ged-emacs-devel@m.gmane.org; Fri, 28 Oct 2016 04:11:26 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57491) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c02Fx-0007LZ-O7 for emacs-devel@gnu.org; Fri, 28 Oct 2016 04:11:18 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c02Fu-00045d-FF for emacs-devel@gnu.org; Fri, 28 Oct 2016 04:11:17 -0400 Original-Received: from dancol.org ([2600:3c01::f03c:91ff:fedf:adf3]:43738) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1c02Fu-00045E-61; Fri, 28 Oct 2016 04:11:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=dancol.org; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:Cc:References:To:Subject; bh=+73QiKRRilvqZGesw2YWcisgY6at7cWZsHWzu6wZ5+c=; b=c9Fuv6gucitJGswLceyHWzV4oI8P5GnMzqK0D7Fmt8oP3LlqJ2clDpC6E9KozGllT9K6onVPHNOQNsB/yrMQfiwWEbeRU/KSwC47oCFzXeg80QCBwyaIEDLnCfHhn/AGy3jEaMv2Q9hdxKBzBrwmnkeOTQdssqNF/oppF4p+GM0Xfde9UverS/NVhDVqq3SVxTkM5oe8xJHigADzQpfCkcd3uxQ6RwxLgeptJxaECjc+gzVzzdsofRRWd5jsXR+KnM/XBmq7VwFZ7RdDoER4ITGYbnmqsl1rcu9+9oYz8AM27ayJ0c6ATOF6hf8PPSkUEghoO5EGhZBRKQZzguh1IQ==; Original-Received: from c-73-97-199-232.hsd1.wa.comcast.net ([73.97.199.232] helo=[192.168.1.173]) by dancol.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1c02Fs-0004pw-56; Fri, 28 Oct 2016 01:11:12 -0700 In-Reply-To: <83k2cssypt.fsf@gnu.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2600:3c01::f03c:91ff:fedf:adf3 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:208929 Archived-At: On 10/28/2016 12:46 AM, Eli Zaretskii wrote: >> From: Daniel Colascione >> Cc: Stefan Monnier , eggert@cs.ucla.edu, emacs-devel@gnu.org >> Date: Thu, 27 Oct 2016 23:23:05 -0700 >> >> We're talking about setting aside address space only, not asking >> the OS to make guarantees about future memory availability. All major >> operating systems, even ones like Windows that don't do overcommit, >> provide ways to reserve address space without asking the OS to guarantee >> availability of memory. > > Not sure I understand you: if a portion of the address space has been > reserved, how come these addresses won't be available when we try to > commit them later? There might not be physical pages available for > that, but virtual memory for those addresses must be available, no? I'm not sure I understand what you're confused about, so I'll try a broader explanation. Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the initial mapping, that address space is unavailable for other uses. But because the page protections are PROT_NONE, my program has no legal right to access that page, so the OS doesn't have to guarantee that it can find a physical page to back that page I've mmaped. In this state, the memory is reserved. The 20GB PROT_NONE address space reservation itself requires very little memory. It's just a note in the kernel's VM interval tree that says "the addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is Now imagine I change the protections to PROT_READ|PROT_WRITE --- once the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right to access that page; under a strict accounting scheme (that is, without overcommit), the OS has to guarantee that it'll be able to go find a physical page to back that virtual page. In this state, the memory is committed -- the kernel has committed to finding backing storage for that page at some point when the current process tries to access it. Say you have a strict-accounting system with 1GB of RAM and 1GB of swap. I can write a program that reserves 20GB of address space. That's fine. The kernel isn't promising to give you 20GB of memory: it's setting address space. Now if I attempt to map 20GB PROT_READ|PROT_WRITE, on any reasonable (i..e, not overcommit) system, mmap should fail, since there's no way a system with 1GB of RAM and 1GB of swap can promise to provide 20GB of private memory. Overcommit confuses the issue: the kernel will _commit_ to as much memory as you ask it for and then renege on that commitment when it finds it convenient. An overcommit system with 1GB of RAM and 1GB of swap will happily let you make that 20GB PROT_READ|PROT_WRITE mapping. It'll just kill you after you use more than 2GB of that mapping. A non-overcommit system understands how to say "sorry, I can't let you do that" up front. On a non-overcommit system, your process will never be killed for accessing memory that the kernel told the process in advance that it could use. I think Jérémie is working with a mental model where every memory mapping is commit, and only an overcommit system allows you to commit more memory than the system actually has. Any system will let you reserve more address space than you have available commit: reservations are cheap. (In Windows, the corresponding concepts MEM_RESERVE and MEM_COMMIT. Windows is much more explicit about the difference between memory reservations and memory commitments than Linux is, because most of the time, Linux users don't care about the difference.)