From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Amirouche Newsgroups: gmane.lisp.guile.user,gmane.comp.gnu.guix.user Subject: Re: Introducing GNUPaste (and guile-wiredtiger future) Date: Mon, 18 Dec 2017 20:38:28 +0100 Message-ID: <9862af37-ef60-167f-8660-e9dd586de55e@hypermove.net> References: <11c569c155672a6da5fce60c67b358f4@hypermove.net> <87lgi1z282.fsf@gmail.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1513625755 14808 195.159.176.226 (18 Dec 2017 19:35:55 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 18 Dec 2017 19:35:55 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 Cc: Andy Wingo , guile-user@gnu.org, help-guix@gnu.org, Help-Guix To: Kristofer Buffington Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Mon Dec 18 20:35:50 2017 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eR1CY-0003P9-2r for guile-user@m.gmane.org; Mon, 18 Dec 2017 20:35:50 +0100 Original-Received: from localhost ([::1]:34687 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eR1EW-0002Sz-9x for guile-user@m.gmane.org; Mon, 18 Dec 2017 14:37:52 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57880) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eR1D4-0001rH-5D for guile-user@gnu.org; Mon, 18 Dec 2017 14:37:13 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eR1CI-0001YR-BN for guile-user@gnu.org; Mon, 18 Dec 2017 14:36:22 -0500 Original-Received: from relay4-d.mail.gandi.net ([2001:4b98:c:538::196]:58103) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eR1CH-0001TX-Py; Mon, 18 Dec 2017 14:35:34 -0500 Original-Received: from [IPv6:2a01:e35:2ef3:d930:3508:87d:1aa9:111] (unknown [IPv6:2a01:e35:2ef3:d930:3508:87d:1aa9:111]) (Authenticated sender: amirouche@hypermove.net) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id A2A49172097; Mon, 18 Dec 2017 20:35:29 +0100 (CET) In-Reply-To: <87lgi1z282.fsf@gmail.com> Content-Language: en-US X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4b98:c:538::196 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:14383 gmane.comp.gnu.guix.user:2782 Archived-At: Le 17/12/2017 à 18:35, Kristofer Buffington a écrit : > Amirouche Boubekki writes: > >> Héllo, >> >> On 2017-12-14 19:37, Kristofer Buffington wrote: >>> Hello! >>> >>> I am excited to share GNUPaste! This is a really simple web app >>> similar to paste.lisp.org built with Guile. I have a linode running it >>> from git on GuixSD. >>> >>> https://paste.freshbakedyams.com (Please use it!) >>> Source: https://github.com/kristoferbuffington/gnupaste.git >>> >>> Currently the frontend uses twitter bootstrap + jquery and highlightjs >>> from a CDN. It really doesn't need all that boilerplate. It will >>> definitely change in the future. GNUPaste depends on guile-wiredtiger >>> and guile-fibers to compile. >> Thanks for considering guile-wiredtiger. >> >> Basically, guile-wiredtiger is not compatible yet with fiber >> in the general case, because fiber will spawn several threads >> and several fibers in each thread (and I think that fibers >> can be stolen by other threads but I am not sure). >> >> The way wiredtiger works is that there is one >> *connexion* (called *environment* in guile wiredtiger) >> per database and one *session* per "thread" of execution >> (called *context* in guile wiredtiger). >> >> I changed the naming because they are different from the >> original things. Both environment and context are backed >> by fluids. >> >> In a pre-fork thread model, one must use with-context [1] >> after the fork to populate the current fuild with a specific >> context. >> >> [1] >> https://framagit.org/a-guile-mind/guile-wiredtiger/blob/master/wiredtiger/extra.scm#L328 >> >> The thing is that when using fibers, the thread of execution >> is a fiber, hence simply said the wiredtiger extra abstraction >> called context fails. Because context/session must not be >> shared between different fibers even if they are executed in >> the same thread. > I have noticed that I get "Resource busy" issues if multiple requests > are handled by the server simultaneously. I tried running (with-env ...) > in the request-handler instead, but it is the same problem. Multiple > environments can't open the database simultaneously either. I will try to make the 0.8 before the end of the year with a fix. > >> Otherwise said, guile-wiredtiger extra abstraction context >> and environment are handy in single thread context because >> it avoids passing the context around. It's also handy in >> simple multithread settings where you don't need to pass >> environment around (to create a new session per thread). >> But it fails in the advanced use case of guile fibers. >> >> This won't trigger a crash under low traffic, but will fail >> under load and advanced use of guile wiredtiger, like multiple >> statments transactions. This can be mitigated by turning off >> preemption in guile-fibers but again it's not perfect solution. >> >> I failed to create an API that makes simple things simple and >> complex things possible. There might be an escape if fibers >> implemented fluids for fibers something like PEP 550 [2]. >> But anyway, it won't be perfect, so I will rework the current >> databases in guile wiredtiger to completly avoid the use of >> fluids and instead pass around database cnx and session. >> >> [2] https://www.python.org/dev/peps/pep-0550/ >> >> I started doing this in culturia [3] but it's far from being >> complete. Since I need to convert all databases grf3, feature-space >> and ix to that style. >> >> Also, it's a backward incompatible change. >> >> [3] >> https://github.com/a-guile-mind/culturia.one/commit/15328e53fcb51d43f14de2f9f21d0b309237969a >> >> At the end of the day, I don't think I want to maintain grf3, >> feature-space and ix inside guile-wiredtiger because they are >> much more than wiredtiger bindings. So I am pondering the fact >> that I will drop those databases abstraction from guile wiredtiger >> and focus on improving the core bindings (like actionable exceptions) >> and maybe improve bindings coverage. > IMHO I think feature-space, ix and grf3 don't really belong in > guile-wiredtiger. agree >> For the next release 0.8 of guile-wiredtiger, the abstractions grf3, >> feature-space and ix will be deprecated. And after for 0.9 release >> they will be moved to the example folder (or better, forked by >> other people to be maintained separatly). > I really like the advanced abstractions. I'm not sure I'm knowledgeable > enough to fork and improve them yet. This is not happening before I have a replacement for them. When I think replacement, I think about a database (server?) that is hopefully both easy to use (in the spirit of mongodb) but that is also ACID across documents (aka. that cares about your data). > >>> Ultimately I want to be able to deploy with something like: >>> >>> $ guix system disk-image gnupaste-system.scm >>> >>> Then boot it up in a VPS. >> That will be neat! >> >>> Thanks! >>> Kris > I really don't want to rely on a database server. It would be easy > enough to store pastes on the filesystem, maybe even with (guix store) > or use git like tekuti and get the benefit of revision history with > paste modifications. I don't like either the idea of the database server. That said without one, you have to code a secure API to allow third parties to access the database ie. in a web context, setup a RESTful API with credentials with secure tokens etc... For instance, how will you delete old pastes in your application? Anyway, the next step is to make guile-wiredtiger abstractions work flawlessly with fiber. > > Thanks for the update! > Kris