From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id D02DD6DE026B for ; Sat, 7 Apr 2018 08:09:36 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.062 X-Spam-Level: X-Spam-Status: No, score=0.062 tagged_above=-999 required=5 tests=[AWL=0.182, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7MeUpOWecLRL for ; Sat, 7 Apr 2018 08:09:36 -0700 (PDT) Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by arlo.cworth.org (Postfix) with ESMTPS id D71806DE025B for ; Sat, 7 Apr 2018 08:09:35 -0700 (PDT) Received: by mail-wm0-f41.google.com with SMTP id f125so7931508wme.4 for ; Sat, 07 Apr 2018 08:09:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=CC4N0HOheGaXf+K2r3EtrzTnYNFNixCREfHk6MTJEHg=; b=PaQfM/TRi00sEasks27EDUKQhqdUa/Jqa3jAahyKVdhMO+nCOa+aPMjAjch6EVcQIs i99x5H9QZo64qFha37ijfY+tAa+VB8oBOnGXA0VxMB4sAUR8z3zquLuCfn4+FGHS39Vs m/Pg3gIGaAopumnBHOOH1EibHrKcebDLGFRT6i8+/DdbeeIZTn1JjbE8isMS6Kdf40EI 8CHVzoh3agBXwz6FEREapjPahSPh93Kk3dZxM11C+6yk0WyTY7lg9mcuImj91VtzAILU tPTWSs9SsOqO2yOCFt9/tx/762/+2aaBb5hIbxvQodgaEB2k2hF6tetj8tHdm1hmv4Bb /sXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=CC4N0HOheGaXf+K2r3EtrzTnYNFNixCREfHk6MTJEHg=; b=bpe5WlMzqZAVtVArYV/g85IfyGFHqtJ66epPHy1zc7TrICqMcvoQhf0G1hTf3Q4jFU c81TQGlLrJbhE7r2sgAOS4HFWYqWi9TOOrZP/LepDdq3gkSdLAfpZG5h8v67VNY/T6Jg zW/g9DUV3uvi8++ptI5cP0m+hBJFcOcieOWQguDQkVpfo2QHTDZL/Of4fFP47tPgaZRv 4HXwGSXg0wpFxWNJgZlKfPVVRvvryhmZi4esEnJNdz6pV+nP+BGx4rYZQ3ao6ItTTjri LFaKpgOjatLDWYQlYsEH/DveOvtwImxfKkCwq1d6PUdJMTG6nt+BF1seCwkAoqYOVP3H 0GpQ== X-Gm-Message-State: ALQs6tB4rggPZLSr7aizxeez5C0U3GOHsTV2yXerCjzFOHbHTiwNAqWf E3KpRcUr+T+r8yF9miejd4E= X-Google-Smtp-Source: AIpwx4+2bNxgEeKHkmHWT8e8zdyFOoNDG8797Pl2sQPlkxH+XbmptrXF9NSgV3zt6oRS/QcsB7Qaog== X-Received: by 10.28.225.193 with SMTP id y184mr15212055wmg.9.1523113774089; Sat, 07 Apr 2018 08:09:34 -0700 (PDT) Received: from localhost.localdomain ([87.124.229.17]) by smtp.gmail.com with ESMTPSA id 24sm18883215wrt.60.2018.04.07.08.09.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Apr 2018 08:09:33 -0700 (PDT) Subject: Re: Database corruption after clean rebuild To: David Bremner , notmuch@notmuchmail.org Cc: xapian-discuss@lists.xapian.org, Patrick Totzke , Justus Winter References: <8a311263-a46f-3327-4054-d8d863fa0169@gmail.com> <874lknwztk.fsf@tethera.net> <87y3hzuw5z.fsf@tethera.net> From: Javier Garcia Message-ID: <5bde652e-8ffe-d43d-106d-a43c7a1b4126@gmail.com> Date: Sat, 7 Apr 2018 17:09:31 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <87y3hzuw5z.fsf@tethera.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Apr 2018 15:09:36 -0000 Unfortunately I can't share my emails without the approval of other parties. The minimum subsets that trigger the error are in the range of 1000-5000 mails, so asking each and everyone of them is out of my reach. I tried to replicate the problem using just spam folders without success. The following is a solid workaround I've stumbled upon. Afew no longer complains and database corruption is gone. $ notmuch compact $ xapian-check ~/.mail/.notmuch/xapian       No errors found I built xapian-core 1.50 but I can't compile notmuch 0.26.1 against it. I will wait and test again in a few weeks. If you are interested in my setup, the error happens with this minimal configuration. #~/.config/afew/config [Filter.1] query = 'folder:"//(INBOX|Inbox|inbox)$/" AND (NOT tag:inbox)' tags = +inbox;-new message = Messages in INBOX folder are tagged as inbox [Filter.2] query = '(NOT folder:"//(INBOX|Inbox|inbox)$/") AND (tag:inbox)' tags = -inbox message = Messages not in INBOX folder cannot be inbox #~/.notmuch-config [database] path=/home-path/.mail [new] tags=new On 07/04/18 12:51, David Bremner wrote: > Javier Garcia writes: > >> I've applied the path to notmuch 0.26.1 without success. >> >> $ rm -rf ~/.mail/.notmuch >> $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH >> ./notmuch new >>    Found 20065 total files (that's not much mail). >>    Processed 20065 total files in 58s (341 files/sec.). >>    Added 19605 new messages to the database. >> >> $ xapian-check .mail/.notmuch/xapian/ >>    docdata: >>    blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 >>    B-tree checked okay >>    docdata table structure checked OK >>    termlist: >>    blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748 >>    xapian-check: DatabaseError: 1 unused block(s) missing from the free >> list, first is 0 > OK, so probably not related to reference loops (although that patch is > not very well tested). It's not clear how notmuch is triggering it, but > this looks like the same bug in Xapian that olly fixed recently [1]. > > A possible next step is to try building xapian master, and linking > notmuch against that. > > Maybe Patrick or Justus (in copy) has some idea why you're only seeing > problems in afew. > > Another debugging direction is to try to duplicate your problem with > some subset of mail that you're willing to share (bisection is the usual > strategy). > > [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html