From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Daniel Colascione Newsgroups: gmane.emacs.devel Subject: Re: When should ralloc.c be used? Date: Fri, 28 Oct 2016 01:44:33 -0700 Message-ID: <7ab47b94-c662-1351-0dd3-ed5269842438@dancol.org> References: <838ttfnmev.fsf@gnu.org> <837f8znk8f.fsf@gnu.org> <83zilvm2ud.fsf@gnu.org> <83r377m0i8.fsf@gnu.org> <83eg36n6v5.fsf@gnu.org> <83shrl523p.fsf@gnu.org> <83eg354ux3.fsf@gnu.org> <4f0c2868-d408-a5c4-d5a8-90dae750eb33@dancol.org> <878tt9ggdk.fsf@ritchie.wxcvbn.org> <83k2cssypt.fsf@gnu.org> <6350b2df-fde9-e716-d279-9f29438f8ee5@dancol.org> <83d1ikswsf.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Trace: blaine.gmane.org 1477645529 29430 195.159.176.226 (28 Oct 2016 09:05:29 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 28 Oct 2016 09:05:29 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 Cc: eggert@cs.ucla.edu, monnier@iro.umontreal.ca, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Fri Oct 28 11:05:24 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1c0364-0005AH-Mz for ged-emacs-devel@m.gmane.org; Fri, 28 Oct 2016 11:05:09 +0200 Original-Received: from localhost ([::1]:47605 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c0367-0001Pw-8Y for ged-emacs-devel@m.gmane.org; Fri, 28 Oct 2016 05:05:11 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:37815) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c02mI-0002Js-Tu for emacs-devel@gnu.org; Fri, 28 Oct 2016 04:44:44 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c02mE-0001AB-Ul for emacs-devel@gnu.org; Fri, 28 Oct 2016 04:44:42 -0400 Original-Received: from dancol.org ([2600:3c01::f03c:91ff:fedf:adf3]:44094) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1c02mE-00019j-LR; Fri, 28 Oct 2016 04:44:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=dancol.org; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:Cc:References:To:Subject; bh=eXC+zWar4lb4miAh5L7llnCYMxdrrD3gtLmLQ0T6MOs=; b=g12uMwnAX0Bufm0O0Aj047v3mjEkk5A4FUTan64mzES8It8ZaW9WRaxFtRdGHXN9in9Xi7fPdAkPoaku30f7TWbN+Sps8KqFVYvFZi8+ESIvDwhNkbmyQGr6ngxu1+zvMgC5eGqcH7nVuKcYv6XXnfgUUHs1p+YRpLU9FuukBtseQYwwdfnxIogz41ELvw6HHeL72yo4R2QkdModC2Jrzr+t55p0fl7l7gxr3dk+KTHCuopFk3/AUzDuF1CXrYtlVjV9A8V/mMhZelLf65GMPzNC8yFQC+mC13nG7cjUwGRwyamQN24SJMWcZ0QwKFL1Np5E45S7c+QVyMRHY6hkzA==; Original-Received: from c-73-97-199-232.hsd1.wa.comcast.net ([73.97.199.232] helo=[192.168.1.173]) by dancol.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1c02mD-000565-Br; Fri, 28 Oct 2016 01:44:37 -0700 In-Reply-To: <83d1ikswsf.fsf@gnu.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2600:3c01::f03c:91ff:fedf:adf3 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:208932 Archived-At: On 10/28/2016 01:27 AM, Eli Zaretskii wrote: >> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org >> From: Daniel Colascione >> Date: Fri, 28 Oct 2016 01:11:08 -0700 >> >> Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the >> initial mapping, that address space is unavailable for other uses. But >> because the page protections are PROT_NONE, my program has no legal >> right to access that page, so the OS doesn't have to guarantee that it >> can find a physical page to back that page I've mmaped. In this state, >> the memory is reserved. >> >> The 20GB PROT_NONE address space reservation itself requires very little >> memory. It's just a note in the kernel's VM interval tree that says "the >> addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is >> >> Now imagine I change the protections to PROT_READ|PROT_WRITE --- once >> the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right >> to access that page; under a strict accounting scheme (that is, without >> overcommit), the OS has to guarantee that it'll be able to go find a >> physical page to back that virtual page. In this state, the memory is >> committed -- the kernel has committed to finding backing storage for >> that page at some point when the current process tries to access it. > > I'm with you up to here. My question is whether PROT_READ|PROT_WRITE > call could fail after PROT_NONE succeeded. You seem to say it could; > I thought it couldn't. Yes, it can fail. This program just failed on my system, which is a strict accounting (echo 2 > /proc/sys/vm/overcommit_memory) Linux box with much less than 100GB total commit available. #include #include #include #include size_t GB = (size_t) 1024 * 1024 * 1024; int main() { size_t sz = 100*GB; void* mem = mmap(NULL, sz, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (mem == MAP_FAILED) { fprintf(stderr, "map failed: %s\n", strerror(errno)); return 1; } if (mprotect(mem, sz, PROT_READ|PROT_WRITE)) { fprintf(stderr, "mprotect failed: %s\n", strerror(errno)); return 1; } fprintf(stderr, "mprotect worked\n"); return 0; } >> Say you have a strict-accounting system with 1GB of RAM and 1GB of swap. >> I can write a program that reserves 20GB of address space. > > I thought such a reservation should fail, because you don't have > enough virtual memory for 20GB of addresses. IOW, I thought the > ability to reserve address space is restricted by the actual amount of > virtual memory available on the system at the time of the call. You > seem to say I was wrong. I'm not sure you're even wrong :-) What does "virtual memory" mean to you? I'm not sure what you have in mind maps to any of the concepts I'm using. When we allocate memory, we can consume two resources: address space and commit. That 100GB mmap above doesn't consume virtual memory, but it does consume address space. Address space is a finite resource, but usually much larger than commit, which is the sum of RAM and swap space. When you commit a page, the resource you're consuming is commit. (Technically, the 100GB mapping consumes real memory enough for the OS to remember you've set aside that address space, but it's usually a negligible book-keeping note. On my system, I can make sz equal to 80TB or so before the mmap starts to fail: that's about the size of the address space range dictated by amd64 processor design.) (In a 32-bit process on modern systems, it's frequently the case that you have more commit on the system than any one process has address space.)