From: Amirouche <amirouche@hypermove.net>
To: Kristofer Buffington <kristoferbuffington@gmail.com>
Cc: Andy Wingo <wingo@igalia.com>,
guile-user@gnu.org, help-guix@gnu.org,
Help-Guix <help-guix-bounces+amirouche=hypermove.net@gnu.org>
Subject: Re: Introducing GNUPaste (and guile-wiredtiger future)
Date: Mon, 18 Dec 2017 20:38:28 +0100 [thread overview]
Message-ID: <9862af37-ef60-167f-8660-e9dd586de55e@hypermove.net> (raw)
In-Reply-To: <87lgi1z282.fsf@gmail.com>
Le 17/12/2017 à 18:35, Kristofer Buffington a écrit :
> Amirouche Boubekki <amirouche@hypermove.net> writes:
>
>> Héllo,
>>
>> On 2017-12-14 19:37, Kristofer Buffington wrote:
>>> Hello!
>>>
>>> I am excited to share GNUPaste! This is a really simple web app
>>> similar to paste.lisp.org built with Guile. I have a linode running it
>>> from git on GuixSD.
>>>
>>> https://paste.freshbakedyams.com (Please use it!)
>>> Source: https://github.com/kristoferbuffington/gnupaste.git
>>>
>>> Currently the frontend uses twitter bootstrap + jquery and highlightjs
>>> from a CDN. It really doesn't need all that boilerplate. It will
>>> definitely change in the future. GNUPaste depends on guile-wiredtiger
>>> and guile-fibers to compile.
>> Thanks for considering guile-wiredtiger.
>>
>> Basically, guile-wiredtiger is not compatible yet with fiber
>> in the general case, because fiber will spawn several threads
>> and several fibers in each thread (and I think that fibers
>> can be stolen by other threads but I am not sure).
>>
>> The way wiredtiger works is that there is one
>> *connexion* (called *environment* in guile wiredtiger)
>> per database and one *session* per "thread" of execution
>> (called *context* in guile wiredtiger).
>>
>> I changed the naming because they are different from the
>> original things. Both environment and context are backed
>> by fluids.
>>
>> In a pre-fork thread model, one must use with-context [1]
>> after the fork to populate the current fuild with a specific
>> context.
>>
>> [1]
>> https://framagit.org/a-guile-mind/guile-wiredtiger/blob/master/wiredtiger/extra.scm#L328
>>
>> The thing is that when using fibers, the thread of execution
>> is a fiber, hence simply said the wiredtiger extra abstraction
>> called context fails. Because context/session must not be
>> shared between different fibers even if they are executed in
>> the same thread.
> I have noticed that I get "Resource busy" issues if multiple requests
> are handled by the server simultaneously. I tried running (with-env ...)
> in the request-handler instead, but it is the same problem. Multiple
> environments can't open the database simultaneously either.
I will try to make the 0.8 before the end of the year with a fix.
>
>> Otherwise said, guile-wiredtiger extra abstraction context
>> and environment are handy in single thread context because
>> it avoids passing the context around. It's also handy in
>> simple multithread settings where you don't need to pass
>> environment around (to create a new session per thread).
>> But it fails in the advanced use case of guile fibers.
>>
>> This won't trigger a crash under low traffic, but will fail
>> under load and advanced use of guile wiredtiger, like multiple
>> statments transactions. This can be mitigated by turning off
>> preemption in guile-fibers but again it's not perfect solution.
>>
>> I failed to create an API that makes simple things simple and
>> complex things possible. There might be an escape if fibers
>> implemented fluids for fibers something like PEP 550 [2].
>> But anyway, it won't be perfect, so I will rework the current
>> databases in guile wiredtiger to completly avoid the use of
>> fluids and instead pass around database cnx and session.
>>
>> [2] https://www.python.org/dev/peps/pep-0550/
>>
>> I started doing this in culturia [3] but it's far from being
>> complete. Since I need to convert all databases grf3, feature-space
>> and ix to that style.
>>
>> Also, it's a backward incompatible change.
>>
>> [3]
>> https://github.com/a-guile-mind/culturia.one/commit/15328e53fcb51d43f14de2f9f21d0b309237969a
>>
>> At the end of the day, I don't think I want to maintain grf3,
>> feature-space and ix inside guile-wiredtiger because they are
>> much more than wiredtiger bindings. So I am pondering the fact
>> that I will drop those databases abstraction from guile wiredtiger
>> and focus on improving the core bindings (like actionable exceptions)
>> and maybe improve bindings coverage.
> IMHO I think feature-space, ix and grf3 don't really belong in
> guile-wiredtiger.
agree
>> For the next release 0.8 of guile-wiredtiger, the abstractions grf3,
>> feature-space and ix will be deprecated. And after for 0.9 release
>> they will be moved to the example folder (or better, forked by
>> other people to be maintained separatly).
> I really like the advanced abstractions. I'm not sure I'm knowledgeable
> enough to fork and improve them yet.
This is not happening before I have a replacement for them.
When I think replacement, I think about a database (server?)
that is hopefully both easy to use (in the spirit of mongodb)
but that is also ACID across documents (aka. that cares about
your data).
>
>>> Ultimately I want to be able to deploy with something like:
>>>
>>> $ guix system disk-image gnupaste-system.scm
>>>
>>> Then boot it up in a VPS.
>> That will be neat!
>>
>>> Thanks!
>>> Kris
> I really don't want to rely on a database server. It would be easy
> enough to store pastes on the filesystem, maybe even with (guix store)
> or use git like tekuti and get the benefit of revision history with
> paste modifications.
I don't like either the idea of the database server. That said
without one, you have to code a secure API to allow
third parties to access the database ie. in a web context,
setup a RESTful API with credentials with secure tokens etc...
For instance, how will you delete old pastes in your application?
Anyway, the next step is to make guile-wiredtiger abstractions
work flawlessly with fiber.
>
> Thanks for the update!
> Kris
next prev parent reply other threads:[~2017-12-18 19:38 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-17 14:39 Introducing GNUPaste (and guile-wiredtiger future) Amirouche Boubekki
2017-12-17 17:35 ` Kristofer Buffington
2017-12-18 19:38 ` Amirouche [this message]
2017-12-18 9:06 ` Andy Wingo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9862af37-ef60-167f-8660-e9dd586de55e@hypermove.net \
--to=amirouche@hypermove.net \
--cc=guile-user@gnu.org \
--cc=help-guix-bounces+amirouche=hypermove.net@gnu.org \
--cc=help-guix@gnu.org \
--cc=kristoferbuffington@gmail.com \
--cc=wingo@igalia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).