From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Amirouche Boubekki Newsgroups: gmane.lisp.guile.user Subject: Re: babelia Date: Fri, 6 Dec 2019 14:19:59 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="70040"; mail-complaints-to="usenet@blaine.gmane.org" To: Guile User Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Fri Dec 06 16:06:59 2019 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1idFC6-000HwD-Ue for guile-user@m.gmane.org; Fri, 06 Dec 2019 16:06:59 +0100 Original-Received: from localhost ([::1]:39348 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1idFC5-0005vp-4f for guile-user@m.gmane.org; Fri, 06 Dec 2019 10:06:57 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:49740) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1idEZP-0004ra-Mq for guile-user@gnu.org; Fri, 06 Dec 2019 09:27:04 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1idEZN-00038B-Nv for guile-user@gnu.org; Fri, 06 Dec 2019 09:26:59 -0500 Original-Received: from mail-pj1-x1030.google.com ([2607:f8b0:4864:20::1030]:47064) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1idEZN-0002z2-DA for guile-user@gnu.org; Fri, 06 Dec 2019 09:26:57 -0500 Original-Received: by mail-pj1-x1030.google.com with SMTP id z21so2800775pjq.13 for ; Fri, 06 Dec 2019 06:26:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=+lfc8vrzpnt552KC4bctjtTm8FcE4kthbcmqnNsxwWU=; b=OKeqV5XJaOkluArMPi3//5mMge1cWId9R4P/SSeh+pXpLGAbAFMZAIqN69lm4TVyAv CTIdodzO0GIn0NtLfuv9cczJNEThFmrjbbI98rP/WM3xYQNjhYOaqa962ZvN6xIkZdFq amSUoZIDCBMKXmh0pBSM3G5D2ou56kZQasY0pf9wFVsVvSddx7HU4aAf5UnOIQnszxRR Q9cS7ONV23ZlK3iGCbuxNcoM4KU3FfqouWkgBJkZfxQdjkYAugQ5FN6ppkaSyTVQ2B94 Ma3YPwzeOgpX0sB/IYRWUVTuVgm3pBcI4pY7zd/A64dimTnLqriZDX+qs0sTE1kJ40Ft oCQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=+lfc8vrzpnt552KC4bctjtTm8FcE4kthbcmqnNsxwWU=; b=jZRszv8c7Ar+NcOizx9yz6nqQ5amNYlNADDw1B/XoxRNCN/Ne3lMok+RhSJJKFOzFC AzzfbYWSoNeDjJDNUUu46zLOXFzYouo+Sab7Up+Dt7hyBeZ/Ovi9erf77209minS3lrV qkyPaDUgbZ088EkBdgJLWO0X7JgHhBSlbmiKcINDMY+76Zr6l3eejKyKdHVoVjrhumWh mntqGMfzrocMDbwJFa3uciini/JcfQo3442erO7MXvq1pQdae1zOzanQtxR92FrusjQZ DQokU0tZPk+RyDrGD7xAkY9+SuR35IsMkj+XX5D4LxKXy+poNghxd11H21yPCa+2w4XA QqyA== X-Gm-Message-State: APjAAAUQ/QhHmJ8LXVA6Hb6QDARmp6G5lcCIwofzvPfvzvFkqZON++6b PyUpZsNloeuBRErEK3iul1orgUIMWGQT/cH3WBDKplrM X-Google-Smtp-Source: APXvYqw9AMR4FqgIVT1gaBraqVSmZWXyK0xIbl+QsNjr9LgihjU/G2WGIjOFPvi1+HgsfgL8GixxoRDUbWj8Kv6eBAY= X-Received: by 2002:a67:d88b:: with SMTP id f11mr9831441vsj.212.1575638410695; Fri, 06 Dec 2019 05:20:10 -0800 (PST) In-Reply-To: X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1030 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:15938 Archived-At: Hello all! Le sam. 16 nov. 2019 =C3=A0 11:06, Amirouche Boubekki < amirouche.boubekki@gmail.com> a =C3=A9crit : > I restarted working on my personal search engine. > After two weeks of work, 41 files changed, 1845 insertions(+), 441 deletions(-) and 97 commits, I tagged a v0.2.0 in the repository at: https://git.sr.ht/~amz3/guile-babelia The babelia index and babelia search subcommands were removed. Instead, one has to `make web` to spawn a server and then hit the /api/search?query=3Dfoobar to make a search. To index stuff, one can POST a file like test.scm to /api/index or rely on babelia crawler subcommands. The crawler is still a work-in-progress. Do not expect the index to be compatible with future releases. > > The last iteration, gotofish, was not too bad even if it has bitrot. > Based on my research and practical experiment, it seems very clear > that there is no workaround the use of map-reduce, that might be known > as n-par-for-each [3]. > > [3] > https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html#i= ndex-n_002dpar_002dfor_002deach > > I made a prototype similar to that n-par-for-each, except it works > with guile-fibers, is asynchronous and works with a shared pool of > threads instead of spawning N threads for each incoming query like > gotofish does. Actually, what I need is n-for-each-par-map where map happens in parallel. The implementation can be found in the babelia/pool.scm file [4]. [4] https://git.sr.ht/~amz3/guile-babelia/tree/v0.2.0/babelia/pool.scm The installation process is still a little bit akward, because one needs to change the path to wiredtiger-3.2.0-0 shared library in the source. Add my channel [5] and do `make init` to get started. [5] https://git.sr.ht/~amz3/guix-amz3-channel Happy hacking!