From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 63idG2/GomBOPwEAgWs5BA (envelope-from ) for ; Mon, 17 May 2021 21:39:27 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id oBqJFm/GomCQHwAAbx9fmQ (envelope-from ) for ; Mon, 17 May 2021 19:39:27 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [IPv6:2607:5300:201:3100::1657]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EC22225F2A for ; Mon, 17 May 2021 21:39:26 +0200 (CEST) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 9B3512AA39; Mon, 17 May 2021 15:39:21 -0400 (EDT) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) by mail.notmuchmail.org (Postfix) with ESMTPS id B1F0A26874 for ; Mon, 17 May 2021 15:39:19 -0400 (EDT) Received: by mail-ot1-x32e.google.com with SMTP id r26-20020a056830121ab02902a5ff1c9b81so6567023otp.11 for ; Mon, 17 May 2021 12:39:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=przZWurSeVKRpIyYEmtaN9CdppPYTZwzuvvZ+9aNqpg=; b=i2D8Bc0Z4rQOYNkin4PjzUIyfsy9VgEB8p6qnjdrH/gEo91hD0NZ7dMH1t7Dqx61Tw VnibdHHVw2RPIdz0shFV2kuUwbB3rcP63KUtaNTGg4HKRuZtOsxug32/vy3Fx9Grz3hj Nu5lkXEwJ7UJCYaQTy3/wiDavUcEw6/XpxiKHYp38zYyqxHPSh8t1JTHbuxQaiGcpu3+ MVMsPAGxy7Y2HUTTK0vS7JE9sc1aylcWS04p/POEVx0huFYF4SrR+o/7tR5/vQRNrW0I nuVJX8WvcBUpE8cy2SyqfjiM69Zfol0/GIK6WTlzj25x7RftaXTDzwH/Nv1QhaV5u82P a4Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=przZWurSeVKRpIyYEmtaN9CdppPYTZwzuvvZ+9aNqpg=; b=R5X8prIkrRwxPBP6CckRjM4ZUuRg6ZPZYwgVdGuXAG7aQFs6UmbtUQG/5zcEB/Viim TI0uSglBT125iXFisHkHzYJ7p2We1AtFs3IxJx+o8w5SmTYMYaVl9iVWxBRlGDnfn2Rt NKBS0l4vxlZDWj2pcfNYrX28Hv6GBRIXDz9BXw0glWem+14nLJqUBys3Kp7prFXgaDUt NWnzN0Pvh67d8KM6yXMuAu4Zbrm4WqMbJ374iX0FHED0xOGaXpT+QrZkkwZMEnIvKFwF 9LExnIHtl2YoIJJGwVY8eZMmXWwxxeh9gRrY1Zi34HfGDdSp4WWUyXmd47DOPICVBihC GWVw== X-Gm-Message-State: AOAM530B6BmcTwDuHRxASqw/dDbTK2VsDKYSpqOLDxjgmSllNDOZPMqQ d9fqJYgBjGA0HxLz9ni5P5MNlp+ny7kgog== X-Google-Smtp-Source: ABdhPJxWKBcd2Me6wj0zQLTEryox4V/Na9cpG9T8GbRoZaZTXG8L3mLNRMPixUr36bNcV8arNsU9+g== X-Received: by 2002:a05:6830:1386:: with SMTP id d6mr1043266otq.122.1621280357624; Mon, 17 May 2021 12:39:17 -0700 (PDT) Received: from localhost (fixed-187-190-78-172.totalplay.net. [187.190.78.172]) by smtp.gmail.com with ESMTPSA id f30sm3250911ooh.10.2021.05.17.12.39.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 May 2021 12:39:17 -0700 (PDT) From: Felipe Contreras To: notmuch@notmuchmail.org Subject: [PATCH 0/2] ruby: enable garbage collection Date: Mon, 17 May 2021 14:39:13 -0500 Message-Id: <20210517193915.1220114-1-felipe.contreras@gmail.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Message-ID-Hash: EPZXQ3V54UPVWRHZHQNCVBIWP6FDSGV7 X-Message-ID-Hash: EPZXQ3V54UPVWRHZHQNCVBIWP6FDSGV7 X-MailFrom: felipe.contreras@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Tomi Ollila X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1621280367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=1zEs5jHBvSNZAwwnhglUImuDx3C9bxCFxTZnYS+VYHQ=; b=Pi0gbNmO7g6xMLJ82xeZFU81MoP/4ejcX8rsR2alwvklZCHptYEJ5nnBMJbOAQ/IrjxY8v Oj7/EPU1nlGUwCa6ZQ7mp4YrghCkUXVSqO+xXWomDQ88IMYwWI0TXq0D60IoCO5KnoQ4DQ NKXdZopaI+etowzKZXaIvzgfxNsEqbbIVcj292SqGfsEAprdAJAdMvBid3DxuJTWLZV7YH 7UG6U9joKoiBeb3Afwb4KkEhOEIsGYSd8ZkG+VPcT7NqsG26FnVujOF34tk2KcBT9yw8Nj Txzj/6m0bXYSyvkJcXsSEx6Skb3Vk6BaS6WeZ0IvMngo6KGkNVAPWIazYabE4A== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1621280367; a=rsa-sha256; cv=none; b=NI8QwnXvKC+/ni6YuCBcY/a/R2BQEIJmzc/VujPuwJsIe8RJWtyT5Pobi95BqejFpC7n/K Kzt7QuHjkSVbUvqA18Qn11GMpQvmFyD89RttNv2RPlFl2mEtPXqprYTDXiM3Vi+Fiqz/nb xAdBOcdomHnUN/AuuOhEx0lOhorNyMJp4teFIWXwz9yma6WJTyRXoEhyRCRjalvFdsx+8f 1N508EXtRI5ESisJ9oToObrRr8X1qbF4K5QBRmjPZ8kET7TdOmNmffVCwhCdxPXzsZP/Qn gM/x1yHqQuUJovKHvYSO8R+B+pIyivClhFnqEJM5YBw76XqxoZ4uBTpTbynOYg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20161025 header.b=i2D8Bc0Z; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -0.96 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20161025 header.b=i2D8Bc0Z; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: EC22225F2A X-Spam-Score: -0.96 X-Migadu-Scanner: scn0.migadu.com X-TUID: YbBw6V8d8s77 Ruby is a gc language, we shouldn't be doing workaround to free memory when Ruby is perfectly capable of doing so. The problem is that talloc wants to be smart, and Ruby and talloc both fight to free memory. We can let Ruby win by stealing all the objects from talloc control. Thanks to the previous cleanup patches it's now possible to easily do this. In order to test this series I've used the following script: require 'notmuch' $db = Notmuch::Database.new(ENV['HOME'] + '/mail') $do_destroy = true while true query = $db.query('') threads = query.search_threads threads.each do |thread| puts '%s: %s' % [thread.thread_id, thread.subject] thread.destroy! if $do_destroy end threads.destroy! if $do_destroy query.destroy! if $do_destroy end $db.close All threads from the database are fetched over and over with no significant increase in memory. The old method of destroying objects with destroy! still works, but now it's not necessary. I tried other methods, like increasing the reference counter and adding a second parent to talloc objects, but for some reason they didn't work. I investiged why but couldn't reach any conclussion. On the other hand the talloc_steal approach works perfectly fine. Felipe Contreras (2): ruby: create an actual wrapper struct ruby: enable garbage collection using talloc bindings/ruby/database.c | 2 +- bindings/ruby/defs.h | 42 +++++++++++++++++++++++++++++++++++----- bindings/ruby/extconf.rb | 1 + bindings/ruby/init.c | 6 ++++++ 4 files changed, 45 insertions(+), 6 deletions(-) -- 2.31.1