Re: Introducing GNUPaste (and guile-wiredtiger future)

[Amirouche <amirouche@hypermove.net>]

Le 17/12/2017 à 18:35, Kristofer Buffington a écrit :
> Amirouche Boubekki <amirouche@hypermove.net> writes:
>
>> Héllo,
>>
>> On 2017-12-14 19:37, Kristofer Buffington wrote:
>>> Hello!
>>>
>>> I am excited to share GNUPaste! This is a really simple web app
>>> similar to paste.lisp.org built with Guile. I have a linode running it
>>> from git on GuixSD.
>>>
>>> https://paste.freshbakedyams.com (Please use it!)
>>> Source: https://github.com/kristoferbuffington/gnupaste.git
>>>
>>> Currently the frontend uses twitter bootstrap + jquery and highlightjs
>>> from a CDN. It really doesn't need all that boilerplate. It will
>>> definitely change in the future. GNUPaste depends on guile-wiredtiger
>>> and guile-fibers to compile.
>> Thanks for considering guile-wiredtiger.
>>
>> Basically, guile-wiredtiger is not compatible yet with fiber
>> in the general case, because fiber will spawn several threads
>> and several fibers in each thread (and I think that fibers
>> can be stolen by other threads but I am not sure).
>>
>> The way wiredtiger works is that there is one
>> *connexion* (called *environment* in guile wiredtiger)
>> per database and one *session* per "thread" of execution
>> (called *context* in guile wiredtiger).
>>
>> I changed the naming because they are different from the
>> original things. Both environment and context are backed
>> by fluids.
>>
>> In a pre-fork thread model, one must use with-context [1]
>> after the fork to populate the current fuild with a specific
>> context.
>>
>> [1]
>> https://framagit.org/a-guile-mind/guile-wiredtiger/blob/master/wiredtiger/extra.scm#L328
>>
>> The thing is that when using fibers, the thread of execution
>> is a fiber, hence simply said the wiredtiger extra abstraction
>> called context fails. Because context/session must not be
>> shared between different fibers even if they are executed in
>> the same thread.
> I have noticed that I get "Resource busy" issues if multiple requests
> are handled by the server simultaneously. I tried running (with-env ...)
> in the request-handler instead, but it is the same problem. Multiple
> environments can't open the database simultaneously either.

I will try to make the 0.8 before the end of the year with a fix.

>
>> Otherwise said, guile-wiredtiger extra abstraction context
>> and environment are handy in single thread context because
>> it avoids passing the context around. It's also handy in
>> simple multithread settings where you don't need to pass
>> environment around (to create a new session per thread).
>> But it fails in the advanced use case of guile fibers.
>>
>> This won't trigger a crash under low traffic, but will fail
>> under load and advanced use of guile wiredtiger, like multiple
>> statments transactions. This can be mitigated by turning off
>> preemption in guile-fibers but again it's not perfect solution.
>>
>> I failed to create an API that makes simple things simple and
>> complex things possible. There might be an escape if fibers
>> implemented fluids for fibers something like PEP 550 [2].
>> But anyway, it won't be perfect, so I will rework the current
>> databases in guile wiredtiger to completly avoid the use of
>> fluids and instead pass around database cnx and session.
>>
>> [2] https://www.python.org/dev/peps/pep-0550/
>>
>> I started doing this in culturia [3] but it's far from being
>> complete. Since I need to convert all databases grf3, feature-space
>> and ix to that style.
>>
>> Also, it's a backward incompatible change.
>>
>> [3]
>> https://github.com/a-guile-mind/culturia.one/commit/15328e53fcb51d43f14de2f9f21d0b309237969a
>>
>> At the end of the day, I don't think I want to maintain grf3,
>> feature-space and ix inside guile-wiredtiger because they are
>> much more than wiredtiger bindings. So I am pondering the fact
>> that I will drop those databases abstraction from guile wiredtiger
>> and focus on improving the core bindings (like actionable exceptions)
>> and maybe improve bindings coverage.
> IMHO I think feature-space, ix and grf3  don't really belong in
> guile-wiredtiger.

agree

>> For the next release 0.8 of guile-wiredtiger, the abstractions grf3,
>> feature-space and ix will be deprecated. And after for 0.9 release
>> they will be moved to the example folder (or better, forked by
>> other people to be maintained separatly).
> I really like the advanced abstractions. I'm not sure I'm knowledgeable
> enough to fork and improve them yet.

This is not happening before I have a replacement for them.
When I think replacement, I think about a database (server?)
that is hopefully both easy to use (in the spirit of mongodb)
but that is also ACID across documents (aka. that cares about
your data).

>
>>> Ultimately I want to be able to deploy with something like:
>>>
>>> $ guix system disk-image gnupaste-system.scm
>>>
>>> Then boot it up in a VPS.
>> That will be neat!
>>
>>> Thanks!
>>> Kris
> I really don't want to rely on a database server. It would be easy
> enough to store pastes on the filesystem, maybe even with (guix store)
> or use git like tekuti and get the benefit of revision history with
> paste modifications.

I don't like either the idea of the database server. That said
without one, you have to code a secure API to allow
third parties to access the database ie. in a web context,
setup a RESTful API with credentials with secure tokens etc...

For instance, how will you delete old pastes in your application?

Anyway, the next step is to make guile-wiredtiger abstractions
work flawlessly with fiber.

>
> Thanks for the update!
> Kris