From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id u0BYJOu8Ml+uFQAA0tVLHw (envelope-from ) for ; Tue, 11 Aug 2020 15:44:43 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id sFNWHuu8Ml9hMAAA1q6Kng (envelope-from ) for ; Tue, 11 Aug 2020 15:44:43 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E40FF9409D6 for ; Tue, 11 Aug 2020 15:44:42 +0000 (UTC) Received: from localhost ([::1]:43288 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k5WS9-0007ys-Lj for larch@yhetil.org; Tue, 11 Aug 2020 11:44:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60250) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k5WQw-0006pY-AW for guix-devel@gnu.org; Tue, 11 Aug 2020 11:43:26 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21175) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k5WQu-0004Ax-4z for guix-devel@gnu.org; Tue, 11 Aug 2020 11:43:25 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1597160594; cv=none; d=zohomail.com; s=zohoarc; b=NuftYFl+gWCOXwUkgOsuq6Lwn7gVVH3qVbK2Km6sQ24tFPgGX/UK7WAbt9rgi+G2C+0nJp+/4fgRfD8Bcar3sSc/lrq9juNOyuMziO2Z2HtKF9B7Y5sHEIKi2KiGeEChlI9jVFGGqNQwDEq7lFXnvTHEqplyTgP7UZxqhPes//o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1597160594; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=Qgn2M394C8OHRP2qKy6aBp43ZBnW5K4kgL+U/HaX6zc=; b=YO2f3GoWp1tyLn1yM7IrMqUPIC0VNoVtziDvNhmAUzcF7RHTQz0KNZQkmlO6H5aPfTrwMrPDJDhFRTAl2SCJI9HskiVfgK+371pnHnT3UZ9bNVH9jVyp+dwvSoJab/jV/EuoAJG+PbPeZ+alUD8xOTepsZVerXnPhg7bPoHVj3w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1597160594; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=Qgn2M394C8OHRP2qKy6aBp43ZBnW5K4kgL+U/HaX6zc=; b=TG2Cf88/ibr2VRg5i7dtHxzEvISLOxc2ijylmNL3s82u7cnQhfH+Di/UBfOL/9uo v4K/6g44K+biKE8Q5XbHhzgRarkQdkWHDF4F/wtvvALjHzJfphwpxAq+C5A+CrARI3+ Wya8X9/FojsPtOWQpxb6NSQ9isNWnWYbyxtJdCgM= Received: from localhost (p54ad4b86.dip0.t-ipconnect.de [84.173.75.134]) by mx.zohomail.com with SMTPS id 1597160590019213.48329456758597; Tue, 11 Aug 2020 08:43:10 -0700 (PDT) References: <87sgcuh8rb.fsf@ambrevar.xyz> User-agent: mu4e 1.4.10; emacs 26.3 From: Ricardo Wurmus To: Pierre Neidhardt Subject: Re: File search progress: database review and question on triggers In-reply-to: <87sgcuh8rb.fsf@ambrevar.xyz> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 11 Aug 2020 17:43:05 +0200 Message-ID: <87y2ml429i.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/11 11:43:16 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=TG2Cf88/; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: -2.21 X-TUID: KxZndQbsnlsi Pierre Neidhardt writes: > 3. Size of the database: > I've persisted all locally-present store items for my current Guix ver= sion > and it produced a database of 72 MiB. It compresses down to 8 MiB > in zstd. For comparison, my laptop=E2=80=99s store contains 1,103,543 files, excludi= ng .links 691,994. The updatedb database for all of them is 86MB and takes ~6 seconds to generate: time updatedb \ --localpaths=3D/gnu/store \ --findoptions=3D'( -path /gnu/store/.links -o -name *.drv -o -name= *.chroot ) -prune -o -type f -print' \ --output=3D/tmp/dbfile locate -d /tmp/dbfile ecxc0800 (This could be further tweaked to exclude links=E2=80=A6) > The worse case is around (number of guix generations) x ~100 MiB. This seems a little excessive. > 4. Indexing speed: > The above items took some 20 minutes to complete (on my rather > powerful machine). Oof. The updatedb hack above takes 6 seconds on my i7-6500U CPU @ 2.50GHz with SSD. I=E2=80=99m not suggesting to use updatedb, but I think it can be instructi= ve to look at how the file database is implemented there. We don=E2=80=99t have = to use SQlite if it is much slower and heavier than a custom inverted index. --=20 Ricardo