Checkpoint

Checkpoint

[Amirouche Boubekki]

Hello schemers,

tl;dr. Finally I accept that I am a megalomaniac

It's been a long time since I did not start a topic about the status
of my projects and plans.

Here is the two last topic I've started:

- Smarter cooperative scheduler (maybe fiber related)
   http://lists.gnu.org/archive/html/guile-user/2018-03/msg00024.html

- neon: git for structured data [Was: Functional database]
   http://lists.gnu.org/archive/html/guile-user/2018-02/msg00011.html

First, there is no more projects and plans. There will be one single
project and one single plan, at the very least it will appear as a
single project because there will be a single repository. The purpose
of the project is explained at the very end of the mail.

Basically, all my ideas converged and doing small experiments each on
their own repository doesn't deliver. I need to build and learn to 
refactor.

monorepo for the win like others have said [0] and used. It has been I
believe easing the progress of guix itself.  Also, my attempts at
building guile-wiredtiger as its own repository [1] did not help
enough people in my mind.  Let's write it down.  We agree that we
disagree that guile-wiredtiger is the way forward.  I won't try to
convince people again [1].  I agree It's not the conventional wisdom
aka. it is not mainstream until now in free software.  And only a
secret weapon, but it's not a secret anymore.  I won't try to argue
anymore against no feedback.  Because I agree with what I read in
between silent IRC channel full of join and leave ``Show me quality
code'', ``Build something useful'' and ``Scratch your own
itch''. That's what I have been doing, even if seemed erratic and
unpredictable.

[0] https://danluu.com/monorepo/
[1] search for wiredtiger @ http://hyperdev.fr/

My last project, neon is also known as zeheyu93600 was a moon shot.  Me,
Myself and I took upon the task of building versioned in a
Direct-Acyclic-Graph quad store!  WAT!  This was for me like an
``Eureka!'' kind of moment. The best all things seemed too obvious and
almost trivial.  I had picked a more difficult task that the defacto
standard functional database (defacto because there is only one).  But
with a dire consequence: the ability to apply git workflow to
virtually any work of knowledge. Long story short, I was catching up
on Audit Trail, Event Source and CQRS. And still, I see how versioned
database can be useful.

That being said, I don't think one should ``version all the things'', I
still, think multi-paradigm disk persistence (``multi-paradigm
database'' is the term that is catching up) is the best choice for the
foreseeable future. Any, all this poor data management. I will not add
a paragraph about the disaster of the king-god Model class as in
Model-View-Controller architecture, because the correct solution is
blurry to me, but for sure it doesn't involve a spagetti ORM code.

So I changed the topic. And looked into the serverless dubbed
revolution. That's how I came up with 'Smarter cooperative scheduler'
topic which then leads to fill the following issue against Celery,
a distributed background task queue scheduler, for help designing a
distributed single task based on priority and age [0].

[2] https://github.com/celery/celery/issues/4901

You already prolly agree with me but serverless is just a way to be
locked in to a specific platform, it's only useful when you don't know
better.  It's only a problem when you applied the "micro services"
dogma blindlessly and need an escape hatch too shy to admit the old
way of the monolith can work. It creates more dev and ops issues. How
do you think you test and develop those serverless based information
systems?

(By the way docker is a neat way to monitor and easily kill the
business of those who not do pay the bill. It's not necessary for
reproducible builds that is just a side effect of building an image
that embeds all the things. Again the winner is the big corp selling
software nobody wanted but is forced to use because ``industry
standards'', you know, catch the train before the train is too late.)

Then Apple free FoundationDB. Their promise is simple as distributed
consistent fault tolerant ACID multi-paradigm database thanks to an 
ordered
key-value store. Simply said, it's wiredtiger distributed. There's more
to say about the subject but I will stop here.

Anyway, how does this related to me and my project. What is that 
project.

Euraka! I am at the intersection of many ideas that look promising,
useful, liberating for the first billion of Internet users and the next.

The name of the project is 'Earth Software System'.

Because I don't know which application will be the revolution I will 
just
copy everything that is useful. I will start with (micro) blogging,
wiki, forum and real-time discussion.  To keep thing simple, I will not
support interop with other Internet protocols in the first versions. And
since the goal is to create a civilization bootstrap kit (code +
data), I don't plan to interop with legacy protocols. I want it to be
used today to build micro-communities hence I will rely on web
technologies supported by IceCat or any other ethical browser. I will
say that web browsers are too costly to bootstrap. GTK is a much
better path. The step after that is to rely on GTK.

I think a small set of powerful primitives are the way to go. I want
to prove two things:

a) ordered key-value stores are such a primitive
b) priority and age based distributed task queue is another

To my mind, the architecture is innovative, the applications are not.


NB: I don't have a solution for global warming


Note for future me: write about Korg paper and expert systems.

My view on bootstrappable effort (Re: Checkpoint)

[Amirouche Boubekki]

On 2018-08-28 19:06, Amirouche Boubekki wrote:
> Hello schemers,
> 
> tl;dr. Finally I accept that I am a megalomaniac

aka. I want to change the world for less evil and more good :)

> 
> The name of the project is 'Earth Software System'.
> 

And its contrived civilization bootstrap kit is what I call the π goal
(they are goals 0, 1 and 2 which I described before without naming them)
is still fuzzy and a moving target. You never reach the exact pi number
you only approach it. That's why I choose to call that goal π instead of 
3.

People might not be familiar with the civilization bootstrap problem.

We do not have the knowledge of a sustainable civilization, granted, 
that is
not the point of the π goal.

I will illustrate that π solves several problems:

   a) It has to do with our way of life of today, the things we take for
      granted like weather, food, water, computers doing what we want.
      The simplest example I think is the food chain. To build our 
civilization,
      people we required to create farms, before that they required to 
teach chickens
      to behave, etc.. to build 999 chicken farms it required smaller 
farms before
      that. But imagine everything disappears. What would happen?! The 
farmer of today
      know how to handle mega farms of polite chickens. But what if 
tomorrow he
      has to deal with 3 mutated chickens alone in the desert after a 
global nuclear war?!

      Boom no more civilization, our bare hands and few scrapes and the 
few neurons
      that remains that may not have the skills to survive in that new 
but predictable
      environment. These fictif neurons could have a chance with a 
civkit!

Damn, not even in the movies they only rebuild our civilization two 
thousand years.
Battle Star Galactica Spoiler: You discover that at the very last 30 
seconds of the
last episode!

This is a lot of extreme far-fetched post-apocalyptic paranoia and 
stuff!
But bear with me, the next example is less dramatic.

Our civilization is not bootstrappable, we do not know how we got here,
we can not _reproduce_ our way of life.

   b) Let's now imagine that we have interstellar space travel. Do you 
think
      it's possible for a family of five to create a _living_. We can 
imagine they
      come with a notice something like Wikipedia (!) and say a 3d 
printer... and
      that is a civkit.

That's what I want to build.

Janneke and others working on bootstrappable and reproducible guix are 
not
only important for "near" term security reasons cap based or not. It's 
also
a requirement for many bad situations that hopefully will never happen 
like
nuclear war or hopefully will happen like an interstellar family trip.

Last story:

   c) Imagine you are on a planet several years away from Earth, you hit
   a bug in production. Do you think that you will wait 1 year for a 
software
   update or what? No, like any good space corpocrop it delivers the 
source
   code of their corpocropware wikipedia based civkit because that's the 
only thing
   they could put a hand on for free and gratis. Bad luck, PHP person has 
a flu.
   And die.

     The end.

Maybe you see where I am going. It's not just about the civkit. guix
will be part of the civkit, and prolly so does Wikipedia. Do you 
believe,
in your Scheme mind that would be the best solution?! The most 
sustainable,
the most boostrappable? Do you think a web browser is bootstrappable?
autotools?!

There is many things to think of. Many things to experiment. The future 
is
exciting nowdays in guile universe.

The most exciting stuff, is that the more bootstrappable code we have,
the easier it is  for people to jump into the train. Otherwise said,
bootstrappable projects will do good before we know it.

 From r7rs.

   Programming languages should be designed not by piling feature on top 
of feature,
   but by removing the weaknesses and restrictions that make additional 
features
   appear necessary.


> I think a small set of powerful primitives are the way to go. I want
> to prove two things:
> 
> a) ordered key-value stores are such a primitive
> b) priority and age-based distributed task queue is another

Any help will be much appreciated.


It's an exciting time to be in guile universe and would like to thank
every people taking part it.

Happy hacking!

Re: My view on bootstrappable effort (Re: Checkpoint)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]


Amirouche Boubekki <amirouche@hypermove.net> writes:

> On 2018-08-28 19:06, Amirouche Boubekki wrote:
>> a) ordered key-value stores are such a primitive

why ordered?

>> b) priority and age-based distributed task queue is another

How do you prioritize? What’s your source of truth? Does truth have to
be defined centrally or does it center around each individual (hopefully
with eventual consistency within society)?

I’m asking because the minimum truth you need for any communication
system is (spam? item).

For reference: https://www.draketo.de/english/freenet/deterministic-load-decentralized-spam-filter

Best wishes,
Arne

PS: sorry for keeping this short.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Technical problems [was: Re: My view on bootstrappable effort (Re: Checkpoint)]

[Amirouche Boubekki]

On 2018-08-29 10:26, Arne Babenhauserheide wrote:
> Amirouche Boubekki <amirouche@hypermove.net> writes:
> 
>> On 2018-08-28 19:06, Amirouche Boubekki wrote:
>>> a) ordered key-value stores are such a primitive
> 
> why ordered?

Sometime people call them "sorted". Ordered is required
to have more assumptions about how the data is structured.
Because keys are ordered, you know that "similar" keys
will be "collocated". That what makes _range_ scan possible
(in opposition to full scan). range scan will return a slice
of key that start with a given prefix. By construction (at insertion
time) it might be all the keys that represent a given document.

There is various pattern you can implement with ordered key value
store the one I know and experimented with are for instance
space filling curve. But to build a leaderboard for instance,
you can implement a probabilistic data structure inside the ordered
key value store.

If you remove the oredered, you end up with something like gdbm.
In that situation there is no "extra structure" that you can take
advantage to speed up queries. It always have to be a full scan.
Or single key get.

> 
>>> b) priority and age-based distributed task queue is another

Indeed, it's a distributed peer to peer network (in the sens every node
can do anything). But it's a trusted network. Untrusted peers can not
access the task queue directly.

> How do you prioritize?

That's the question! I will start another topic with that question.

> What’s your source of truth?

The database, that is FoundationDB in the distributed setup.

> Does truth have to be defined centrally or does it center around each 
> individual

Yes truth is defined centrally, but with fault tolerance. Every peer (or 
node)
has a heartbeat to check every minute that every role is fulfilled.
It's fulfilled if a lock is taken in FDB. If it's not there (or expired) 
there is a
race for taking on that role. FDB is atomic and consistent. So only one 
"peer"
will take a given role at a given moment. Until the peer goes down
they will fulfill their duty for the role they have taken.
Like keeping the task queue up-to-date.

In practice, the role is fulfilled by a fiber of a peer.

> (hopefully with eventual consistency within society)?

When it comes to FDB, it's consistent. Every peer see the same version 
of the truth.

> I’m asking because the minimum truth you need for any communication
> system is (spam? item).

Like I said earlier, this is a controlled environment. Some kind of peer 
to peer
architecture for scaling centralized systems.

> 
> For reference:
> https://www.draketo.de/english/freenet/deterministic-load-decentralized-spam-filter

thanks for the link.

> 
> Best wishes,
> Arne
> 
> PS: sorry for keeping this short.

Thanks for replying.

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: Checkpoint

[Amirouche Boubekki]

On 2018-08-28 19:06, Amirouche Boubekki wrote:
> Hello schemers,
> 

> 
> My last project, neon is also known as zeheyu93600 was a moon shot.  
> Me,
> Myself and I took upon the task of building versioned in a
> Direct-Acyclic-Graph quad store!  WAT!  This was for me like an
> ``Eureka!'' kind of moment.

LiteTree: SQLite with Branches

see https://news.ycombinator.com/item?id=17865687

Re: Technical problems [was: Re: My view on bootstrappable effort (Re: Checkpoint)]

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]


Amirouche Boubekki <amirouche@hypermove.net> writes:

> On 2018-08-29 10:26, Arne Babenhauserheide wrote:
>>>> a) ordered key-value stores are such a primitive
…
>> why ordered?
…
> If you remove the ordered, you end up with something like gdbm.
> In that situation there is no "extra structure" that you can take
> advantage to speed up queries.

That sounds convincing.

>>>> b) priority and age-based distributed task queue is another
…
>> Does truth have to be defined centrally or does it center around
>> each individual
…
> In practice, the role is fulfilled by a fiber of a peer.
…
> peer to peer architecture for scaling centralized systems.

That sounds interesting, too. It goes a different direction than my
current main interest, but it definitely sounds promising.

Thank you for your detailed answer! And good luck in implementing this!
I wish I could join in, but my time is already fully allocated with
other projects (and sadly some procrastination).

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]