From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id mHuGFSoKk2O3HAAAbAwnHQ (envelope-from ) for ; Fri, 09 Dec 2022 11:12:58 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id MA+NFSoKk2MEuAAA9RJhRA (envelope-from ) for ; Fri, 09 Dec 2022 11:12:58 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7BBA144C3F for ; Fri, 9 Dec 2022 11:12:57 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p3aHA-0001yh-Fz; Fri, 09 Dec 2022 05:06:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p3aGd-0001pW-HV for guix-devel@gnu.org; Fri, 09 Dec 2022 05:06:09 -0500 Received: from mail-wm1-f42.google.com ([209.85.128.42]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1p3aGX-0000gK-5Y; Fri, 09 Dec 2022 05:06:07 -0500 Received: by mail-wm1-f42.google.com with SMTP id f13-20020a1cc90d000000b003d08c4cf679so3032801wmb.5; Fri, 09 Dec 2022 02:05:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2CuBisyMHHepekLgiwsZquL7gXKLFA2rPfdqtbJFCqI=; b=CKFlunlm7YjSWIWOJUxAVhJld3d8tXOkkIFByPXq5x+f6xWK1B/7vyR9Iq0EFDnGAV vn/EQ+pIFtSsAtk86bp0vX3WqWcmRZ/9CERo1+A420t3ETKsrJ2QITmZZJO5t9XTYlXG fffs0Sxwr6iXm6Y4wR/YmbF015/hWeJrrNcrf2042zyQ2sG1sK7GEmQPL00ijfVbySpx WPq0yBUvwRQBTGEiD8ycDNThfOjxJbd4BUGRJlM/A4KKnhLi9zpojZwenxddJfpnBnIG p1gCTyKTcqVkC7WkK6bmB4JXMR1ruvYYFy1LxzGcaavoQ+2WDesFeUVxdar/FyaaDpNR +Zcw== X-Gm-Message-State: ANoB5pn5FiLQIeaKFJ7ixfRnT63IuB/GhH9+snxwGxpYlaEQ7ib63DKR gyH4h0OZk7dHhLBe6t3W1QyF5y2Sjmg= X-Google-Smtp-Source: AA0mqf4h0FlNl9ezIM4hwbJaaPfQg/4NIIsM0SS/rT1kPxBgNoANmiH0lCixWQ21O2QeOPPgt4htFQ== X-Received: by 2002:a7b:c30e:0:b0:3cf:735c:9d54 with SMTP id k14-20020a7bc30e000000b003cf735c9d54mr4508098wmj.1.1670580355752; Fri, 09 Dec 2022 02:05:55 -0800 (PST) Received: from yavin4 ([2a01:e0a:1c8:8a40:2394:a743:1507:85ab]) by smtp.gmail.com with ESMTPSA id d4-20020adfe2c4000000b00236488f62d6sm957267wrj.79.2022.12.09.02.05.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Dec 2022 02:05:55 -0800 (PST) From: "Antoine R. Dumont (@ardumont)" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Subject: Re: File search In-Reply-To: <878rjkn949.fsf@gnu.org> References: <87pmd1r8kt.fsf@gmail.com> <87mt85r7h8.fsf@gmail.com> <87lenonydz.fsf@gnu.org> <877cz7qg80.fsf@gmail.com> <878rjkn949.fsf@gnu.org> Date: Fri, 09 Dec 2022 11:05:53 +0100 Message-ID: <87pmcsriwu.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=209.85.128.42; envelope-from=antoine.romain.dumont@gmail.com; helo=mail-wm1-f42.google.com X-Spam_score_int: -13 X-Spam_score: -1.4 X-Spam_bar: - X-Spam_report: (-1.4 / 5.0 requ) BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.229, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1670580777; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=2CuBisyMHHepekLgiwsZquL7gXKLFA2rPfdqtbJFCqI=; b=W9Q11EczxgvScVYbADiZc5luDA0ARSwyG/v0QEC/wg02ntlr7HKe0ee+ShVVEEugobZKBe r0Np11e0JZTY14/ojjJUi5rw5z8C/3mipd10uIEpIJpYlU4IlfD8PIKVoOTI+34Z9MH51H M/YpLx99kNV97CnE/mNAbMrZsNyxaVnT0XO4f2Ztr2QeYdWjVptQQKFJUlRq8NVCmZ6zl6 eOsYbMXiP6Td4DKMhP4Mi1bd1MOyLGSJQ0kyv8CPGovA0D8Y53xtMpgM7XQSh52CisjozO yYf29o0qg9v8ow1jnjBKLwfLqUbEda/9XIuqcrCAKiRZ4SKauT9+qDNRqjnq4g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1670580777; a=rsa-sha256; cv=none; b=XATGbH9ZrPfMFd4+HYVvVmlK4zedizWksKxIGO1dpVVR5tfddDrbo9h26wAfkXSkJF4AWT DKhZP8jdZ7EPp+yD9cKH4CHCTpUk0+wViRd3otZf/ITVrk8sZCD2rKtOj49rD1QD/5ESmG fNrsfk8LJnpM78ZAESFDY1H7/vbqxcZa1XXJptJqtX36BvyzgkJe3ozNm/2rszWcEhcbc/ PKK1cSmV2lkTvUYYTSWEYsy+QA6uDjvGH1ZKEyEYtb/7/DRbNIbvVvFu4croagsc4QMFn5 RhPfFqSEZQXMGTreyIzSidmjrsPHryyUHaWjO93w3cwmYE3O+GJPTgv+jp1aGg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.35 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 7BBA144C3F X-Spam-Score: -3.35 X-Migadu-Scanner: scn1.migadu.com X-TUID: dJ+KlQHDVOxt --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, > So we went from 413s to 11s (on the Guix System node) for only 6% fewer > files in the latter case? Do I get that right? That=E2=80=99s pretty co= ol. Not 6% of loss, a bit more, around half is only detected between the first and second round. Here is the summary [1] (org-mode) table I should have sent to ease reading. I don't have a better dataset to test against so you tell me if it is still worth pushing in that direction ;) [1] |-----------+-------------+----------+----------| | Iteration | Host System | Time (s) | Packages | |-----------+-------------+----------+----------| | 1st | Debian | 121.88 | 284 | | | Guix System | 413.55 | 749 | |-----------+-------------+----------+----------| | 2nd | Debian | 1.3 | 101 | | | Guix System | 11.54 | 354 | |-----------+-------------+----------+----------| > It should instead show =E2=80=9Cgit@2.38.1:send-email=E2=80=9D. We proba= bly need an > =E2=80=98output=E2=80=99 field in the =E2=80=98Packages=E2=80=99 table. Why must that be "git@2.38.1:send-email", what does it mean? Providing we continue on that direction (see first question above), I'll check what i can do (I'm not sure how to properly do that just yet). > Also going forward we=E2=80=99ll need a schema version, as in: yes, that I see how to do. > Oh, and progress bars too. I'm a bit unsettled on this. Hopefully it was mostly a joke ;) If it's serious, how can we implement this as we don't know in advance how many packages we'll discover locally (if we don't change the current approach, that is, to avoid incurring too much time penalty). And also, what implem should be used, I know the "pv" package provide some pipe utility for that but that's about it. Do you have some example in the guix codebase that does some progress bar already? > And a pony. :-) ;p Cheers, =2D- tony / Antoine R. Dumont (@ardumont) =2D---------------------------------------------------------------- gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8 Ludovic Court=C3=A8s writes: > Howdy! > > "Antoine R. Dumont (@ardumont)" > skribis: > >> Please, find enclosed the latest implementation as a patch (somewhat vcs >> code ;). I've edited commits to mark Ludo as author with his >> started/amended implementations first [0] (that should be in the patch). > > Nice! > >> For information, I extracted some number from runs to compare our >> iterations (see the org-file attachment). The first iteration being >> "extracts packages from the store" and the second one "extracts packages >> from the system manifest". Those runs happened both on a guixified >> debian host and a raw guix host (more packages). > > So we went from 413s to 11s (on the Guix System node) for only 6% fewer > files in the latter case? Do I get that right? That=E2=80=99s pretty co= ol. > > The implementation based on manifests can of course miss packages, so > it=E2=80=99s a tradeoff. Purely local indexing will only find packages y= ou > already have anyway, so eventually we=E2=80=99ll need a second mode that = would > download a database. > > BTW, I noticed outputs are not properly handled so far, as in this > example: > > --8<---------------cut here---------------start------------->8--- > $ GUIX_EXTENSIONS_PATH=3D$HOME/tmp/guix-index guix index search git-send-= email > git@2.38.1 /gnu/store/g3lgyzr749l76qma7srycclgsm0f78iq-git-2.38= .1-send-email/libexec/git-core/git-send-email > git@2.37.1 /gnu/store/n3hkzz5ydm0qm1c2jja2pwy2v19mq1k0-git-2.37= .1-send-email/libexec/git-core/git-send-email > --8<---------------cut here---------------end--------------->8--- > > It should instead show =E2=80=9Cgit@2.38.1:send-email=E2=80=9D. We proba= bly need an > =E2=80=98output=E2=80=99 field in the =E2=80=98Packages=E2=80=99 table. > > Also going forward we=E2=80=99ll need a schema version, as in: > > --8<---------------cut here---------------start------------->8--- > create table SchemaVersion ( > version integer not null; > ); > --8<---------------cut here---------------end--------------->8--- > > so that the tool can upgrade or discard databases that have the wrong > version. > > Oh, and progress bars too. And a pony. :-) > > Thanks for your work! > > Ludo=E2=80=99. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJSBAEBCgA8FiEEvwAgPXQaydVGqL4HUuLphA0Qw7gFAmOTCIEeHGFyZHVtb250 QHNvZnR3YXJlaGVyaXRhZ2Uub3JnAAoJEFLi6YQNEMO4Op8P/12Hf8x0igxWR4u9 8ZA1tTwXsq0QHGPMVljTMZhEZUUjElcji+jSC9sOgT6kckcjckWeu3059bNMF2l/ ocbgR9oQ+AzeVmZ4eTQlEZ+0R5TY/B9P7AYlhmy0FgUDH3xDB3DVYf/0NH7g2WPb TlD31Zmyla1hJUW5WQmUTT3Yem+br1luv3ahbyddr3XFCLCDvdV8YljTAePj9mcU iy4TcXXieK/JrYIo+DjfLE0nimtinEAiA2xqLTI3/pga+9auTeM10EyFdswxfetN 1lYLeVZYR3zUxqAlHmdQZVrOYjknShkMoSKH6xX8ZuE2/hdNem0MrzDvNy8WBK7e Sw5YDeV4WTLgH/rY0KoLy4RJJK5/cis2GReoAK8F5PhigtUgE6SZSYhRxuU7iK0n xthaOmmytRSF9QB2meQnAZvW0phM1ZwJv4qmHkJ8VBnk/lcuKuf64GaCgFUDLecB fuerRZJh0YCgrZRjX61M/Jwovj/nWhguAhD1zYA2bwgkgxzlbueVnByjw7kDiyYV hq2XLki/xqigKzt1iSB8TkPOdUxze7XfU/hP7N7sjxHrimwiqmIqysrT/mp9Exvo IIBfHiADV6H065hsBq3ZISE9KRdSmM4tqBHhOKGsMz1LKcK1S8uqLD1DpOrisCpe tiDUPQ/2rAQ8oHXTcX3ooHKamC4p =K5aL -----END PGP SIGNATURE----- --=-=-=--