Re: [HELP] a search engine in GNU Guile

[Amirouche Boubekki <amirouche@hypermove.net>]

Héllo,


I made some progress regarding culturia. Now there is a web
interface available at hypermove.net [0].

[0] http://hypermove.net/?query=guile+algorithms+-wingolog

Now you can:

- search a bunch of website related to Guile I quickly selected
- use minus char "-" to exclude some keyword from the result set
- index small-ish domains

There is no UI yet to add your website to the index, if you want
your website to be part of the experiment reach me.

I tried two search backends. One based on a graph database which
is more versatile but much slower and another based on inverted
index. The latter is the one that is used currently. It's fast
enough right now.

What will be done next:

- Add wikipedia, wiktionary, stackoverflow and hackernews to the
   database

- Improve how results are displayed

- Remove feeds from results

- Index pdf

- Remove duplicate entries from the results

- Paginate

- Make use of fibers to run the http server

- Better crawling algorithm that support updating the index

At some point there will be a 0.1 release, not sure when.

The code is also available at framagit [1] the web interface
is implemented in a single module [2].

[1] https://framagit.org/a-guile-mind/culturia
[2] https://framagit.org/a-guile-mind/culturia/blob/master/src/web.scm

Happy hacking!

On 2016-08-13 17:25, Amirouche Boubekki wrote:
> Héllo,
> 
> 
> The goal of Culturia is to create a framework that makes it easy
> to tape into Natural Language Understanding algorithms (and NLP)
> and provide an interface for common tasks.
> 
> Culturia is an intelligence augmentation software.
> 
> It's primary interface is a search engine. Another important aspect
> of the project is that it wants to be useable offline as such it will
> come with infrastructure to dump, load and store dataset for offline 
> use.
> 
> The current state of the project can be described as a big ball of mud.
> There is a tiny search engine with crawling skills and that's basically
> all of it.
> 
> The immediate changes that should happen are in order of preference:
> 
> - offline stackoverflow (cf. sotoki.scm) and use the generated
>   website to create a zim for kiwix [0]. This is great occasion to
>   show how great GNU Guile is!
> - port whoosh/lucene to guile to improve text search
> - offline hackernews, wikidata, wikipedia, wiktionary
> - implement BM25f
> 
> Culturia is a reference to _Culture and Empire_ by Pieter Hintjens.
> 
> It has a sparse documentation is available online [1].
> It's hosted on github [2] (This can change, if contributors
> don't want to use github).
> 
> The TODO list is big, here is some stuff that needs to be done:
> 
> - finish GrammarLink bindings
> - create sophia [3] bindings
> - implement TextRank
> - implement PageRank
> - create a GUI using sly or html
> - explore ways to easily share database among several processus
> 
> And many other things! Newbies are accepted obviously!
> 
> Send me a mail or use #guile @ irc.freenode.net, I am amz3.
> 
> 
> Happy hacking!
> 
> 
> [0] http://www.kiwix.org/wiki/Main_Page
> [1] https://amirouche.github.io/Culturia/doc/
> [2] https://github.com/amirouche/Culturia
> [3] http://sophia.systems/

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr