From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 6VhXLARIrmIigQAAbAwnHQ (envelope-from ) for ; Sat, 18 Jun 2022 23:47:48 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id EG9pKwRIrmKiOAEA9RJhRA (envelope-from ) for ; Sat, 18 Jun 2022 23:47:48 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 225CA24F23 for ; Sat, 18 Jun 2022 23:47:48 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id AB5D65F029; Sat, 18 Jun 2022 21:47:44 +0000 (UTC) Received: from fethera.tethera.net (fethera.tethera.net [IPv6:2607:5300:60:c5::1]) by mail.notmuchmail.org (Postfix) with ESMTP id 2F33C5E226 for ; Sat, 18 Jun 2022 21:47:42 +0000 (UTC) Received: by fethera.tethera.net (Postfix, from userid 1001) id 9D7395FBD7; Sat, 18 Jun 2022 17:47:40 -0400 (EDT) Received: (nullmailer pid 26220 invoked by uid 1000); Sat, 18 Jun 2022 21:47:39 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: RFC: directory layout for notmuch git Date: Sat, 18 Jun 2022 18:47:33 -0300 Message-Id: <20220618214734.25789-1-david@tethera.net> X-Mailer: git-send-email 2.35.2 MIME-Version: 1.0 Message-ID-Hash: 2UA7I3556WIWX2QHVYXT2OP542FVD3CJ X-Message-ID-Hash: 2UA7I3556WIWX2QHVYXT2OP542FVD3CJ X-MailFrom: bremner@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1655588868; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=xKZJ514eKx3obTkXGWIGltVvZQyZj3Hq0CG+lEKn8ZY=; b=VYYVsT2pjVkuNxSQwylwXHVRSeqrYLLKpWdEd5bNBogyrkGaok5gO9yHAkTg0pbn3k9Sda 8ZXfpuAUp0pg24JNus4/jkz0iYuWZHMPt97mUWitFadxwNKe4F+XTNbCVqed1ttITe/Htd r53MmfqjgriOhZcY1bgIXPLKGJqAn9AtI72rmTPYnpbgNv1gHrbCtePDDXJLYonIWUkzLw AJUjFG6gTOQcfxcXw9clNiHrQnrzMadQytIVUWK8sBEd5pBv074KUYTOTXf7g31NRudyE/ dJ3dJZojg1ELS0ATYHZ1E0Abip+PFO9n3mG1m89JFh8phuLije8hB1OVcJAL4w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1655588868; a=rsa-sha256; cv=none; b=lbvP6rAT3eq0FAoYBqErPl5/29d/HRMWBaVy9eMjGDTAjduo7r6kcEvA5OnkPJmNw/Pb53 lRJGY4122Zmdv/fxKFablHMtucO8zfIcnxHhtKb1GHS620klnTON6CR37mNqM8EhK+c6oR Bxm04bNbW1eqRxnwaaCGpoic7lUEdsF/cEw1iE+l+Cr0EfcRnqmd+hFUUmh1YKdKEltxYC Zomx3aIt8v8My5xFnGUzIBg3irbNWTaSPuHRhLeKsvI9Uwia5TtO3toWkxetT/vTrgznU2 siNdEjl6ubQOkJPmjX17mtY9DbLj8M2tZKMLKBWWEKPmmyinNtiGrB1zRN0PoQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.66 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 225CA24F23 X-Spam-Score: -1.66 X-Migadu-Scanner: scn0.migadu.com X-TUID: VfHsyEENBB3s It turns out that if you have more than about 150k messages _and_ your /tmp is on ext4, then the merge operation for notmuch-git can fail (either hash collisions in the directory indexing, or running out of inodes). There are few possible solutions. The simplest is just to make /tmp tmpfs. Another solution is to implement a hashing scheme like the patch that follows. This is not much work, but we have to figure out what to do with existing repos. Do we write migration code (considerably more complex than the hashing code itself), or just have a flag day? This would involve everybody re-cloning from nmbug, but I guess there are not that many active users. Other possibilities include using the non-hashed layout when in "nmbug mode".