From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Kristofer Buffington Newsgroups: gmane.comp.gnu.guix.user,gmane.lisp.guile.user Subject: Re: Introducing GNUPaste (and guile-wiredtiger future) Date: Sun, 17 Dec 2017 12:35:57 -0500 Message-ID: <87lgi1z282.fsf@gmail.com> References: <11c569c155672a6da5fce60c67b358f4@hypermove.net> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: blaine.gmane.org 1513532064 2785 195.159.176.226 (17 Dec 2017 17:34:24 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sun, 17 Dec 2017 17:34:24 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) Cc: Andy Wingo , guile-user@gnu.org, help-guix@gnu.org, Help-Guix To: Amirouche Boubekki Original-X-From: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sun Dec 17 18:34:20 2017 Return-path: Envelope-to: gcggh-help-guix@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eQcpP-0000St-Gh for gcggh-help-guix@m.gmane.org; Sun, 17 Dec 2017 18:34:19 +0100 Original-Received: from localhost ([::1]:55020 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQcrN-0001PB-NS for gcggh-help-guix@m.gmane.org; Sun, 17 Dec 2017 12:36:21 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:54841) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQcrB-0001Oo-W8 for help-guix@gnu.org; Sun, 17 Dec 2017 12:36:11 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eQcrA-0005OS-LC for help-guix@gnu.org; Sun, 17 Dec 2017 12:36:10 -0500 Original-Received: from mail-ua0-x243.google.com ([2607:f8b0:400c:c08::243]:38784) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eQcr4-0005Lm-8Y; Sun, 17 Dec 2017 12:36:02 -0500 Original-Received: by mail-ua0-x243.google.com with SMTP id i4so9237471uab.5; Sun, 17 Dec 2017 09:36:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=w9eA3CIT18bSpoXc1hOGsGdN5TfHhTGPjTks8BlOkXo=; b=iNp6TNF4JUdtmdOwarXZ//I40W0iGmD/9MLxF9OekPb5Dtej7HlLYg8Cr8vRJjxSgY NrhonTTk/7cSDDAVIMIezmpMgNWbRY9yKP/9xlgAl9Cu18xBxgN55NJ2UV1l5i3lv/R0 7rlBduH9Lp9DVVSx19x0diSSeFeBEy47F1T8OIhfjr6jVEHjlB9gYMX0/tTtQpGVVpfq PU8f1aYQoGHrURdMVBZY5uGVYz5dbrGftWqoRkH9pYl7KJtI1BDl7omDGleWjjpLmD2m 9mgiqmR5fD/COsvm9Rh4cqnGaGecPR6DhS+nY3qfwaMF0KXiI1t4/kgKH33krx+4msNl ZoTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=w9eA3CIT18bSpoXc1hOGsGdN5TfHhTGPjTks8BlOkXo=; b=tqVcrn+agiIGSKix9fIMdnlywh2MoKFMefNDPk305Pt3kDs23MITbK6uGgnz7FYlDG WoZS692gzbUOKtCU9v6jU01RUZ+krBEWj/c2Ks49lY/5YBRzMliLM0fUAghaTyBtFyIA ww5xyyfa+pF/c9MdbdoHsBHdfvmVrCHDH/aWM3gy1o3TvEybH19heYQCSNvmWRWLR4Kh o+NojtqzZYsTVdlVIlXhu7iwzLeIvMWQmA81xORZbYMj/y2bgW9ekab4rU0UrKR4/EE8 hFJDtpAscV5MvsSELczh8+iPlEUlAmXj7DSiUftF2cZq7oM+QGbXLcHweTqsPK9ugrsK zFBQ== X-Gm-Message-State: AKGB3mI90LUWWPSoCYoHcSkWU9dgTnPiJeyJWDNuLWUD4osdMMIIn53i lDH+ADohktZPSMdAl9upwzlUIiYikj0= X-Google-Smtp-Source: ACJfBoub7tNXKai/xfJJD/Y4hQaeg+t/jRMJ2kpKgficSkrD8KDMXQW8M4csgZFhpPZlOIgxDsqhRQ== X-Received: by 10.176.0.73 with SMTP id 67mr21085348uai.132.1513532161220; Sun, 17 Dec 2017 09:36:01 -0800 (PST) Original-Received: from libre ([172.56.20.196]) by smtp.gmail.com with ESMTPSA id u195sm3597128vkb.23.2017.12.17.09.35.58 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 17 Dec 2017 09:35:59 -0800 (PST) In-Reply-To: <11c569c155672a6da5fce60c67b358f4@hypermove.net> (Amirouche Boubekki's message of "Sun, 17 Dec 2017 15:39:52 +0100") X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400c:c08::243 X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Original-Sender: "Help-Guix" Xref: news.gmane.org gmane.comp.gnu.guix.user:2778 gmane.lisp.guile.user:14380 Archived-At: Amirouche Boubekki writes: > H=C3=A9llo, > > On 2017-12-14 19:37, Kristofer Buffington wrote: >> Hello! >> >> I am excited to share GNUPaste! This is a really simple web app >> similar to paste.lisp.org built with Guile. I have a linode running it >> from git on GuixSD. >> >> https://paste.freshbakedyams.com (Please use it!) >> Source: https://github.com/kristoferbuffington/gnupaste.git >> >> Currently the frontend uses twitter bootstrap + jquery and highlightjs >> from a CDN. It really doesn't need all that boilerplate. It will >> definitely change in the future. GNUPaste depends on guile-wiredtiger >> and guile-fibers to compile. > > Thanks for considering guile-wiredtiger. > > Basically, guile-wiredtiger is not compatible yet with fiber > in the general case, because fiber will spawn several threads > and several fibers in each thread (and I think that fibers > can be stolen by other threads but I am not sure). > > The way wiredtiger works is that there is one > *connexion* (called *environment* in guile wiredtiger) > per database and one *session* per "thread" of execution > (called *context* in guile wiredtiger). > > I changed the naming because they are different from the > original things. Both environment and context are backed > by fluids. > > In a pre-fork thread model, one must use with-context [1] > after the fork to populate the current fuild with a specific > context. > > [1] > https://framagit.org/a-guile-mind/guile-wiredtiger/blob/master/wiredtiger= /extra.scm#L328 > > The thing is that when using fibers, the thread of execution > is a fiber, hence simply said the wiredtiger extra abstraction > called context fails. Because context/session must not be > shared between different fibers even if they are executed in > the same thread. I have noticed that I get "Resource busy" issues if multiple requests are handled by the server simultaneously. I tried running (with-env ...) in the request-handler instead, but it is the same problem. Multiple environments can't open the database simultaneously either. > Otherwise said, guile-wiredtiger extra abstraction context > and environment are handy in single thread context because > it avoids passing the context around. It's also handy in > simple multithread settings where you don't need to pass > environment around (to create a new session per thread). > But it fails in the advanced use case of guile fibers. > > This won't trigger a crash under low traffic, but will fail > under load and advanced use of guile wiredtiger, like multiple > statments transactions. This can be mitigated by turning off > preemption in guile-fibers but again it's not perfect solution. > > I failed to create an API that makes simple things simple and > complex things possible. There might be an escape if fibers > implemented fluids for fibers something like PEP 550 [2]. > But anyway, it won't be perfect, so I will rework the current > databases in guile wiredtiger to completly avoid the use of > fluids and instead pass around database cnx and session. > > [2] https://www.python.org/dev/peps/pep-0550/ > > I started doing this in culturia [3] but it's far from being > complete. Since I need to convert all databases grf3, feature-space > and ix to that style. > > Also, it's a backward incompatible change. > > [3] > https://github.com/a-guile-mind/culturia.one/commit/15328e53fcb51d43f14de= 2f9f21d0b309237969a > > At the end of the day, I don't think I want to maintain grf3, > feature-space and ix inside guile-wiredtiger because they are > much more than wiredtiger bindings. So I am pondering the fact > that I will drop those databases abstraction from guile wiredtiger > and focus on improving the core bindings (like actionable exceptions) > and maybe improve bindings coverage. IMHO I think feature-space, ix and grf3 don't really belong in guile-wiredtiger. > For the next release 0.8 of guile-wiredtiger, the abstractions grf3, > feature-space and ix will be deprecated. And after for 0.9 release > they will be moved to the example folder (or better, forked by > other people to be maintained separatly). I really like the advanced abstractions. I'm not sure I'm knowledgeable enough to fork and improve them yet. >> Ultimately I want to be able to deploy with something like: >> >> $ guix system disk-image gnupaste-system.scm >> >> Then boot it up in a VPS. > > That will be neat! > >> >> Thanks! >> Kris I really don't want to rely on a database server. It would be easy enough to store pastes on the filesystem, maybe even with (guix store) or use git like tekuti and get the benefit of revision history with paste modifications. Thanks for the update! Kris