From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?UTF-8?Q?Linus_Bj=C3=B6rnstam?= Newsgroups: gmane.lisp.guile.devel Subject: Re: Hatables are slow Date: Wed, 23 Feb 2022 08:40:57 +0100 Message-ID: <4dccd80b-18f2-40e3-b6b2-c1d97bd91224@www.fastmail.com> References: Mime-Version: 1.0 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="28141"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Cyrus-JMAP/3.5.0-alpha0-4778-g14fba9972e-fm-20220217.001-g14fba997 To: "Stefan Israelsson Tampe" , guile-devel Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Wed Feb 23 08:42:06 2022 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nMmHj-0006zs-Kr for guile-devel@m.gmane-mx.org; Wed, 23 Feb 2022 08:42:05 +0100 Original-Received: from localhost ([::1]:37816 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nMmHi-0008HR-Bl for guile-devel@m.gmane-mx.org; Wed, 23 Feb 2022 02:42:02 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:58204) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nMmHQ-0008Gm-5Q for guile-devel@gnu.org; Wed, 23 Feb 2022 02:41:44 -0500 Original-Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:59975) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nMmHL-0005Cc-D5 for guile-devel@gnu.org; Wed, 23 Feb 2022 02:41:43 -0500 Original-Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 4DCBA3200BD2; Wed, 23 Feb 2022 02:41:19 -0500 (EST) Original-Received: from imap51 ([10.202.2.101]) by compute5.internal (MEProxy); Wed, 23 Feb 2022 02:41:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=veryfast.biz; h= cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm2; bh=sWWXHKGZGNzy8m IOKrSBVFWuWlD7s0Wm3wiOh1vtZRg=; b=xINUEHWwIfrdPl5c9teN5gjytzhXjP ITkLpw82KNc/UD6evz9/T1BUKvSEhvxOdGo1JxlKKLVGy/aEuDMQj9SVi3heUT8m ieK5+Kgeg1cgxBwOQGxdEjHjOWt0ElMZSPdebxMTAo33zjGYsxuFIsIcDVgYm1PO gRYHIPv/D0JYteS/y7zKyWQHaPCftCQTG3ecqBLXsL1+cJci8dk+Mb9TP17Zhr1C DrhfYr6xj0lQz4pB51gHKEr7/e1dJOLMRozY2UZHlbMtT63kS0cnWvIhZ49GggCR LbE2c659us8CSsOs/gXuwIfO9BbWb8IglaSm2sek7FOBPqQskkM6IOkg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:date:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=sWWXHKGZGNzy8mIOKrSBVFWuWlD7s0Wm3wiOh1vtZRg=; b=SGZZhnST FxAyfbmkqJuRhT8ZwN2u9r18lciLBGjunCy8vzwur9NoslM4fIvgfYn6prUhvGIJ 8slyZY54o0NZiBfC5yj1fIv0kWvw0Xc8QytUdQfNA17dnuwqg96YSytxzo3WIGtr pDZodhvtBRS4IouQNi6gXOJQ6ky9EnJcorPntHJohAIhfisa+7+mdjC+/igREmmw M1wLSVo8AdXzjwz6mZ/2tNX1F05jWb6z+mf/usGlhZ54XkK3b3wqfbF4rdVgvXYu Dmfp20FqZSFdQM7N+p7t4wxjLw6j7KvsitxcBj6VdjVz2EQKDKL3WkSHyHamW7LL 2v4oIgv1CEFhsg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrkeelgddutdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgfgsehtqhertderreejnecuhfhrohhmpefnihhn uhhspgeujhpnrhhnshhtrghmuceolhhinhhushdrsghjohhrnhhsthgrmhesvhgvrhihfh grshhtrdgsihiiqeenucggtffrrghtthgvrhhnpeetvdeuteehueektefhuddvjeelteff uedvvdeuudeiuedvtedtueehhedvjeeiffenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehlihhnuhhsrdgsjhhorhhnshhtrghmsehvvghrhihf rghsthdrsghiii X-ME-Proxy: Original-Received: by mailuser.nyi.internal (Postfix, from userid 501) id 979D41BC007D; Wed, 23 Feb 2022 02:41:18 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface In-Reply-To: Received-SPF: pass client-ip=64.147.123.20; envelope-from=linus.bjornstam@veryfast.biz; helo=wout4-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.io gmane.lisp.guile.devel:21145 Archived-At: Hej! I would also propose a hash table based on a more sane interface. The eq= uality and hash procedures should be associated with the hash table at c= reation rather than every time the hash table is used. Like in R6RS, srf= i-69, or srfi-12X (intermediate hash tables).=20 Maybe the current HT could be relegated to some kind of compat or deprec= ated library to be removed in 3.4... I am no maintainer, but I think we = can all agree that the current API, while fine in the context of guile 1= .6, is somewhat clunky by today's standards. It is also commonplace enou= gh that regular deprecation might become rough. Just the simple fact that hash-set! and hashq-set! can be used interchan= geably while you at the same time NEVER EVER should mix them is somewhat= unnerving. I would say a hash table that specifies everything at creation time (wit= h maybe an opportunity to use something like the hashx-* functions for d= aredevils and for future srfi needs) is the way to go. Best regards Linus Bj=C3=B6rnstam On Mon, 21 Feb 2022, at 14:18, Stefan Israelsson Tampe wrote: > A datastructure I fancy is hash tables. But I found out that hashtable= s=20 > in guile are really slow, How? First of all we make a hash table > > (define h (make-hash-table)) > > Then add values > (for-each (lambda (i) (hash-set! h i i)) (iota 20000000)) > > Then the following operation cost say 5s > (hash-fold (lambda (k v s) (+ k v s)) 0 h) > > It is possible with the foreign interface to speedt this up to 2s usin= g=20 > guiles internal interface. But this is slow for such a simple=20 > application. Now let's change focus. Assume the in stead an assoc, > > (define l (map (lambda (i) (cons i i)) (iota 20000000))) > > Then > ime (let lp ((l ll) (s 0)) (if (pair? l) (lp (cdr l) (+ s (caar l))) s= ))=20 > $5 =3D 199999990000000=20 > ;; 0.114530s real time, 0.114391s run time. 0.000000s spent in GC. > > That's 20X faster. What have happened?, Well hashmaps has terrible=20 > memory layout for scanning. So essentially keeping a list of the=20 > created values consed on a list not only get you an ordered hashmap,=20 > you also have 20X increase in speed, you sacrifice memory, say about=20 > 25-50% extra. The problem actually more that when you remove elements=20 > updating the ordered list is very expensive. In python-on-guile I have=20 > solved this by moving to a doubly linked list when people start's to=20 > delete single elements. For small hashmap things are different. > > I suggest that guile should have a proper faster standard hashmap=20 > implemention of such kind in scheme. > > Stefan