From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Steve Fink Newsgroups: gmane.emacs.devel Subject: Re: [EXPERIMENT] Emacs with the SpiderMonkey garbage collector Date: Sat, 2 Dec 2017 21:24:50 -0800 Message-ID: References: <8fa934aa-9bbd-ca8e-2468-d6370324e6a5@mozilla.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Trace: blaine.gmane.org 1512278733 5546 195.159.176.226 (3 Dec 2017 05:25:33 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sun, 3 Dec 2017 05:25:33 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Thunderbird/57.0 Cc: emacs-devel@gnu.org To: Pip Cet Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Dec 03 06:25:25 2017 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eLMmK-0000zj-VF for ged-emacs-devel@m.gmane.org; Sun, 03 Dec 2017 06:25:25 +0100 Original-Received: from localhost ([::1]:37915 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eLMmR-0006jL-20 for ged-emacs-devel@m.gmane.org; Sun, 03 Dec 2017 00:25:31 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:39105) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eLMlt-0006jE-H3 for emacs-devel@gnu.org; Sun, 03 Dec 2017 00:24:58 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eLMlq-0002XC-BU for emacs-devel@gnu.org; Sun, 03 Dec 2017 00:24:57 -0500 Original-Received: from mail-pg0-x233.google.com ([2607:f8b0:400e:c05::233]:38894) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eLMlq-0002Wx-28 for emacs-devel@gnu.org; Sun, 03 Dec 2017 00:24:54 -0500 Original-Received: by mail-pg0-x233.google.com with SMTP id f12so6206607pgo.5 for ; Sat, 02 Dec 2017 21:24:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mozilla.com; s=google; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=SdO82HeIIUeB5PG0mwaH0IO+ghcY6wt+IqeL8e1dCww=; b=S6T4v1pBX+aO5p1VJStN9YtZn8DcQUE9DdKqXne0kty1uRlzEip+rncKgv0eLa/psq qXqJ9fBaN5NtkF7CcOaQqr0n/cuL1CjMAtEXikQEUWjHStNII/O0sb28zC5HzbXM9J9Y 2Sg9liRYHpKiXJPTQ2NpB3g8NxHd1JLlLBZKs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=SdO82HeIIUeB5PG0mwaH0IO+ghcY6wt+IqeL8e1dCww=; b=hBdzscQhqppdhdBhls2Oxh/9Hres666DvEjSPIAvIBJUCdJwIROw5D45e/bRI5+c8a Y7zkg0eY9Yae8e4R5wiSAOWRVIY9iCiFCuXNOExDJE14b4BdNx35X78qJNX1/asvMgNP RcEFnt6w6JF8Gp49ykt/rsvBdU56nBkp3Ax/8H3mzhsxkwTzFni1PYEta6OPA3M09zQY QSpGXTPASRLWeABybJDFNZFNMKlAmoqGzPP2a3BdBlJ3hXcihsn7emyGXcLCkfaO38jv pCUbnl4nsqvy9mtduq2CZvZhj0mUPdzPklgdi3G47zkgYejOJ3oCOanuBx46FgNIwv2m sTMA== X-Gm-Message-State: AJaThX52BiePB37DmbDaHBZs8v0x9pMAQhUSQ/IdwqU3P/REcwBDWKk4 e1yE2tm1JRmB7+N13ROQvptiFhvjYl4= X-Google-Smtp-Source: AGs4zMY9n8ToiI4ia/1+bwaZhNQKw66kPwBkJ9F63Z7L1kRLhqk4I46RlhdfSiCK6fJLJ11bSkBOuw== X-Received: by 10.99.49.81 with SMTP id x78mr10748866pgx.35.1512278692338; Sat, 02 Dec 2017 21:24:52 -0800 (PST) Original-Received: from [192.168.1.58] (107-207-38-202.lightspeed.sntcca.sbcglobal.net. [107.207.38.202]) by smtp.gmail.com with ESMTPSA id s68sm19318168pfj.81.2017.12.02.21.24.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Dec 2017 21:24:51 -0800 (PST) In-Reply-To: Content-Language: en-US X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::233 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:220644 Archived-At: On 12/2/17 4:37 PM, Pip Cet wrote: > On Fri, Dec 1, 2017 at 5:57 PM, Steve Fink wrote: > >> The one big thing that the analysis doesn't handle particularly well is >> internal pointers. For now, it sounds like you're mostly safe from these >> moving because all your data is hanging off of the JS_GetPrivate >> indirection, so you wouldn't have any pointers internal to GC things. But >> you can still run into issues keeping things alive. We mostly have this >> problem with strings, since small strings are stored inline in the GC things >> and so it's very easy to end up with a char* that will move. We tend to work >> around this by either (1) passing around the GC pointer instead of the >> char*; (2) making functions that accept such a char* to also declare that >> they won't GC by requiring a reference to an 'AutoRequireCannotGC&' token, >> which is validated by the static analysis; or (3) forcing the contents to be >> stored in the malloc heap if they aren't already, and just being careful >> with keeping the owning JSString* in a Rooted somewhere higher on >> the stack. >> >> Collections are also an issue, if you want to index them by GC pointer >> value. > It seems I have two options: rewrite all hashed collections whenever > something moves, or make up a hash value and store it in a private > slot for each object upon creation. My understanding is SpiderMonkey > does the former for WeakMaps, and those seem to perform okay, so that > might be the better option long-term, but I haven't given much thought > to this and the made-up hash value seems easier to implement... We've done it two ways, gradually shifting almost everything over to the second. The first was what we called "rekeying", which is just removing and re-inserting anything whose pointer changed (or more generally, when any pointer that formed part of the key changed.) We had to do some careful dancing in our hashtable implementation to be certain rekeying can't cause the table to grow (we have tombstones, so the naive approach accumulates tombstones.) Most things are now switching over to the second approach, which is to key tables off of unique ids. We have a separate table mapping anything that needs one to a unique id. But that's still just a space optimization; if all of your objects are going to end up needing a unique id, then you're paying the extra memory cost for everything anyway and you may as well store a hash (or unique id), as you say. If these tables aren't holding strong references (if being a key in the table shouldn't keep the key alive), then you need to sweep them too, to throw out all of the dead stuff. Even if nothing ever looks up those keys again, hash collisions will have you comparing their dead memory with lookup keys. And if you have code to sweep the table, I guess you can always reuse it during the moving part of the collection to rekey everything that is moving. (And if they *are* holding strong references, they should be traced.) The embedder API to hook into this can be seen at https://searchfox.org/mozilla-central/source/js/public/GCAPI.h#926 We use GC-aware data structures within spidermonkey (GCHashMap, GCHashSet, GCVector) to automate most of this. See eg https://searchfox.org/mozilla-central/source/js/public/GCHashTable.h#28 though even those still require something to call their sweep() methods. The WeakCache at https://searchfox.org/mozilla-central/source/js/public/SweepingAPI.h#54 sets itself up to be swept automatically at the right time. And like I said, those generally use unique IDs. https://searchfox.org/mozilla-central/source/js/public/GCHashTable.h#105 is the hashtable that rekeys instead. (Note that our hashtable implementation isn't great. It uses double hashing, and probably ought to be replaced with one of the Robin Hood variants, which would change rekeying.)