[ANN] gzochi project development release 0.10

[ANN] gzochi project development release 0.10

[Julian Graham]

Hi everyone,

I'm pleased to announce the tenth development release of the gzochi
game development framework.

The project description, from Savannah: gzochi (/zoʊ-tʃiː/) is a
framework for developing massively multiplayer online games. A server
container provides services to deployed games, which are written in
Guile Scheme, that abstract and simplify some of the most challenging
and error-prone aspects of online game development: Concurrency, data
persistence, and network communications. A very thin client library
can be embedded to provide connectivity for client applications
written in any language.

This the largest release so far, including more than 150 commits since
0.9. The major highlight of this release is the introduction of the
gzochi meta server: a new, experimental server component that allows
multiple running instances of the gzochid container to share game
application data in a high-availability configuration. (Future
enhancements are planned to have the meta server perform load
balancing of client connections and transactional task execution, and
for it to coordinate ordered message delivery across the cluster.)

In addition to the regular slate of bugfixes and stability
improvements, this release also includes the following notable
changes:

* The built-in B+tree-based storage engine supports much higher
transaction throughput and concurrency

* Client message processing is now fully asynchronous, yielding higher
task throughput and enabling much lower response latency

* A Dockerfile for a complete, Debian-based build environment is
provided to make it easier to get started

* ...and more! As usual, see the NEWS files in the distribution for details

This is a development release; the framework is not bug-free.
Nonetheless, there's extensive server and client documentation, and
the distribution includes three example games with heavily-annotated
source code. For more information, visit the web site at
http://www.nongnu.org/gzochi/ or check out the project page on
Savannah, at http://savannah.nongnu.org/projects/gzochi/ -- you can
pick up the release from the downloads section.


Regards,
Julian

Re: [ANN] gzochi project development release 0.10

[Amirouche Boubekki]

Hi Julian!

On 2016-08-13 04:29, Julian Graham wrote:
> 
> * The built-in B+tree-based storage engine supports much higher
> transaction throughput and concurrency
> 

Can you explain in more details what this B+tree based storage engine 
is?
And what is used for?


Thanks!

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: [ANN] gzochi project development release 0.10

[Julian Graham]

Hi Amirouche!

> Can you explain in more details what this B+tree based storage engine is?
> And what is used for?

Sure!

Some context: gzochi is an application server for games written in
Guile. It provides various services to the applications that it hosts,
including data storage. The underlying implementation of the data
store (the "storage engine") is configurable - the framework comes
with a Berkeley DB storage engine and an in-memory storage engine.
(And you can write your own, if you wish.)

All of the data structures manipulated by a game running in the gzochi
container - the game's "object graph" - are serialized and persisted
to the data store. The container guarantees a consistent view of the
object graph to each "task" that runs as part of the game, and it
handles changes to the graph transactionally. It's up to the storage
engine implementation to make sure that every transaction is isolated
from every other transaction, that deadlocks are resolved properly,
that each transaction either commits or rolls back atomically, etc.
The Berkeley DB-based storage engine relies on BDB for those things.
The in-memory engine uses a B+tree and intentional (R/W) locking to
implement transactions.

The B+tree storage engine has a second function when running gzochi in
a distributed / high-availability configuration: It's used to cache a
subset of the object graph on each node in the cluster while that node
is executing game code that manipulates that part of the graph. (In
this configuration, the centralized "meta server" is responsible for
storing and retrieving the canonical version of the object graph.)

Does that answer your question?


Regards,
Julian

Re: [ANN] gzochi project development release 0.10

[Amirouche Boubekki]

On 2016-08-13 20:33, Julian Graham wrote:
> Hi Amirouche!
> 
>> Can you explain in more details what this B+tree based storage engine 
>> is?
>> And what is used for?
> 
> Sure!
> 
> Some context: gzochi is an application server for games written in
> Guile. It provides various services to the applications that it hosts,
> including data storage. The underlying implementation of the data
> store (the "storage engine") is configurable - the framework comes
> with a Berkeley DB storage engine and an in-memory storage engine.
> (And you can write your own, if you wish.)
> 
> All of the data structures manipulated by a game running in the gzochi
> container - the game's "object graph" - are serialized and persisted
> to the data store. The container guarantees a consistent view of the
> object graph to each "task" that runs as part of the game, and it
> handles changes to the graph transactionally. It's up to the storage
> engine implementation to make sure that every transaction is isolated
> from every other transaction, that deadlocks are resolved properly,
> that each transaction either commits or rolls back atomically, etc.
> The Berkeley DB-based storage engine relies on BDB for those things.
> The in-memory engine uses a B+tree and intentional (R/W) locking to
> implement transactions.
> 
> The B+tree storage engine has a second function when running gzochi in
> a distributed / high-availability configuration: It's used to cache a
> subset of the object graph on each node in the cluster while that node
> is executing game code that manipulates that part of the graph. (In
> this configuration, the centralized "meta server" is responsible for
> storing and retrieving the canonical version of the object graph.)
> 
> Does that answer your question?

Yes thanks!

Based on a few tests wiredtiger is faster than berkeley db. You might
consider having a look at it. 
https://framagit.org/a-guile-mind/guile-wiredtiger

> 
> 
> Regards,
> Julian

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: [ANN] gzochi project development release 0.10

[Julian Graham]

On Sat, Aug 13, 2016 at 3:33 PM, Amirouche Boubekki
<amirouche@hypermove.net> wrote:
> Based on a few tests wiredtiger is faster than berkeley db. You might
> consider having a look at it.
> https://framagit.org/a-guile-mind/guile-wiredtiger

I've looked at WiredTiger before - and this Guile integration looks
very nice - but I decided not to go any further with an integration
because it only supports (as far as I can tell) an optimistic
transactional concurrency model, and my belief is that game
application workloads are likely to benefit more from a pessimistic
model in which concurrent write locks on the same record are
serialized. That is to say, the restart-on-conflict workflow in an
optimistic model is expensive, so it's only worth it if you expect
write contention to be rare. If you expect it to be frequent, you're
going to waste a lot of CPU cycles re-executing transactional code.

However, that's just my intuition: If you'd like to integrate it and
test it out, the storage engine SPI is easy to implement!

Re: [ANN] gzochi project development release 0.10

[Amirouche Boubekki]

On 2016-08-14 01:23, Julian Graham wrote:
> On Sat, Aug 13, 2016 at 3:33 PM, Amirouche Boubekki
> <amirouche@hypermove.net> wrote:
>> Based on a few tests wiredtiger is faster than berkeley db. You might
>> consider having a look at it.
>> https://framagit.org/a-guile-mind/guile-wiredtiger
> 
> I've looked at WiredTiger before - and this Guile integration looks
> very nice - but I decided not to go any further with an integration
> because it only supports (as far as I can tell) an optimistic
> transactional concurrency model, and my belief is that game
> application workloads are likely to benefit more from a pessimistic
> model in which concurrent write locks on the same record are
> serialized. That is to say, the restart-on-conflict workflow in an
> optimistic model is expensive, so it's only worth it if you expect
> write contention to be rare. If you expect it to be frequent, you're
> going to waste a lot of CPU cycles re-executing transactional code.

Ah, you are probably right. I did not know what optimistic locking is.

> However, that's just my intuition: If you'd like to integrate it and
> test it out, the storage engine SPI is easy to implement!

TBH, gzochi seems awesome but I don't have an immediate need for it.
Except maybe something. You do store task that must be executed in the 
database
and somehow when it's time for them to execute you schedule them? Is 
that correct?
Is that correct that the task queue is persisted to disk?


-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: [ANN] gzochi project development release 0.10

[Julian Graham]

On Sun, Aug 14, 2016 at 4:15 AM, Amirouche Boubekki
<amirouche@hypermove.net> wrote:
> TBH, gzochi seems awesome but I don't have an immediate need for it.
> Except maybe something. You do store task that must be executed in the
> database
> and somehow when it's time for them to execute you schedule them? Is that
> correct?
> Is that correct that the task queue is persisted to disk?

Yes. Application tasks can be scheduled "durably" so that they survive
the restart of individual (or all) nodes in the cluster. The task
execution schedule is rebuilt from the database on startup.