Hackathon: Mumi and Cuirass

Hackathon: Mumi and Cuirass

[Ricardo Wurmus]

Hey there,

now that Guix 1.0 is out I think it’s a good time to improve our tools
to allow us to better identify problems with package builds and to
better respond to bug reports and patch submissions.

1) Mumi

We are using Debbugs for bug reports (bug-guix@gnu.org) and patches
(guix-patches@gnu.org), and one of the interfaces to the bug tracker is
the web interface at https://issues.guix.info.  This is backed by a
little Guile tool called “Mumi”.  The code is hosted here:

    https://git.elephly.net/software/mumi.git

Mumi uses guile-email and guile-debbugs to talk to the Debbugs SOAP API
(that’s a verbose XML API) and render the results as a web page.
Currently, the search isn’t really working so well due to limitations on
the Debbugs side, and things are a little slower than they should be.
Also, the interface only provides read access — you cannot comment on
issues with a web browser.

I’d like us to change these two things.  Instead of using the Debbugs
API for searching I’d like Mumi to keep a local append-only database of
email messages and a mutable table of issue meta data (such as the issue
status, the submitter, the owner, etc).  For write access I’d like us to
offer a form that allows people to submit comments right there.  We’d
need to think about ways to prevent SPAM, obviously.  (Options are: spam
assassin, user logins, GPG signed text that’s checked against a keyring,
etc…)

If you’d like to work on this with me, please get in touch!


2) Cuirass

Cuirass has a web interface: https://ci.guix.gnu.org/

Unfortunately, it doesn’t yet enable us to answer questions like:

- What packages are currently failing to build on aarch64?
- What is the history of icecat builds on i686?
- Can you show me all builds matching “rust” across all architectures?
- What builds are blocked because of this one failing build?

Implementing features that would allow us to better see the build status
for all of our supported architectures would be very welcome.  It would
be even better if we could also see an overview of our build nodes and
what they are currently doing – how many build slots are still
available?  Why is only one of the nodes doing any work?

We have the freedom to integrate Cuirass more closely with the rest of
Guix and the offloading framework, so this seems like a good target.

If you find this kind of project interesting please get in touch!  You
don’t need to know how to do any of these tasks as long as you are
motivated to discuss this and help us find solutions together.

Let’s do this!

--
Ricardo

Re: Hackathon: Mumi and Cuirass

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3914 bytes --]


Ricardo Wurmus <rekado@elephly.net> writes:

> Hey there,
>
> now that Guix 1.0 is out I think it’s a good time to improve our tools
> to allow us to better identify problems with package builds and to
> better respond to bug reports and patch submissions.

...

> 2) Cuirass
>
> Cuirass has a web interface: https://ci.guix.gnu.org/
>
> Unfortunately, it doesn’t yet enable us to answer questions like:
>
> - What packages are currently failing to build on aarch64?
> - What is the history of icecat builds on i686?
> - Can you show me all builds matching “rust” across all architectures?
> - What builds are blocked because of this one failing build?
>
> Implementing features that would allow us to better see the build status
> for all of our supported architectures would be very welcome.  It would
> be even better if we could also see an overview of our build nodes and
> what they are currently doing – how many build slots are still
> available?  Why is only one of the nodes doing any work?
>
> We have the freedom to integrate Cuirass more closely with the rest of
> Guix and the offloading framework, so this seems like a good target.
>
> If you find this kind of project interesting please get in touch!  You
> don’t need to know how to do any of these tasks as long as you are
> motivated to discuss this and help us find solutions together.
>
> Let’s do this!

I mentioned some things which seem related to this in this email [1],
but one of the enhancements I'm specifically interested in getting in to
Cuirass is a way to provide subscriptions to the build information. This
would enable "listening" to Cuirass, to get a near real time stream of
events, build information, things like that.

I previously tried scraping build information in to the Guix Data
Service, and you can see some data here [2], but that was slow, and I
was potentially missing builds. I think some kind of publish/subscribe
model would work out much better.

1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html
2: https://prototype-guix-data-service.cbaines.net/builds

> - What packages are currently failing to build on aarch64?
> - What is the history of icecat builds on i686?
> - Can you show me all builds matching “rust” across all architectures?
> - What builds are blocked because of this one failing build?

On the subject of questions like these, I would hope that if the Guix
Data Service had up to date build information, it wouldn't be too
difficult to add the necessary pages to make answering these questions
easy.

At the moment, as I say, there's no up to date information on builds,
and the only page I've added a form for comparing derivations between
two revisions ([3] for example).

3: https://prototype-guix-data-service.cbaines.net/compare/derivations?base_commit=094aaa1a1e81b6c5461c1a2aa8ccd9019faa97ec&target_commit=4ee55938de315ac965c12a99f5dc98f48ac2cc83

Now I'm not saying that I think it would be bad if Cuirass had a more
detailed web interface, but even if it did, I'd still be interested in
gathering up all the build information, as one thing I'm excited about
for the future is the possibility of having multiple independent build
farms.

In my mind, reproducible builds + multiple independant build farms is
the path towards trusted substitutes. From a users perspective, I'm
imaginging setting some policy where I trust substitutes if berlin, and
some other build farm agree on the output. But for a overall view of
what's going on, this is where I see the value of gathering up build
information from potentially many build farms in to one place.

Now that I've got some updated patches for Patchwork up [4], I'll
hopefully be able to make some time to work on this area over the coming
weeks.

4: https://issues.guix.info/issue/33185#38

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Hackathon: Mumi and Cuirass

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

Hi Ricardo,

do you mean by "Hackathon" a specific day where all
participants will meet personally/virtual and work together on these
project ideas? Or do you want to generally encourage people to work on
these themes?

On Fri, 03 May 2019 23:45:00 +0200
Ricardo Wurmus <rekado@elephly.net> wrote:

> 1) Mumi

With debbugs I'm missing a feature to easily add/remove tags to bugs,
like the github bugtracker has. For example, it is not possible to
search for all bugs for cuirass, or get a list of all "beginners" bug
(very helpful for Outreachy).

Do you think that feature could be added with Mumi?


> 2) Cuirass

On Guix days, we discussed the idea of using Postgresql as a backend.
Could that also be a task for hacking?

Chris, have you worked on that? I think I saw you adding some
guile-postgres bindings as a Guix-package.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Hackathon: Mumi and Cuirass

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]


Björn Höfling <bjoern.hoefling@bjoernhoefling.de> writes:

>> 2) Cuirass
>
> On Guix days, we discussed the idea of using Postgresql as a backend.
> Could that also be a task for hacking?
>
> Chris, have you worked on that? I think I saw you adding some
> guile-postgres bindings as a Guix-package.

I've been using PostgreSQL from Guile for the Guix Data Service [1]
stuff, guile-dbd-postgresql didn't work out, I can't remember exactly
why, but I think it could have had something to do with error
handling/reporting. Anyway, guile-squee seems to do the trick.

1: https://git.cbaines.net/guix/data-service/

I'm interested in doing some Cuirass stuff, but I'm still a little
uncertain about how you might go about that. I think there's probably
some easier related tasks though.

One thing that I saw related to Sqlite is that the rowid's that Cuirass
does use behave differently if there's an INTEGER PRIMARY KEY column in
the table [2]. This is important I think, because without this,
operations like VACUUM can change the rowid's [3]. I think Cuirass uses
rowid's as the identitifers for builds for example, but not in this
way.

2: https://www.sqlite.org/draft/lang_createtable.html#rowid
3: https://www.sqlite.org/draft/lang_vacuum.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Hackathon: Mumi and Cuirass

[Ricardo Wurmus]

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> writes:

> Hi Ricardo,
>
> do you mean by "Hackathon" a specific day where all
> participants will meet personally/virtual and work together on these
> project ideas? Or do you want to generally encourage people to work on
> these themes?

Either way is fine I think.  My hope is that a group of people will
emerge and coordinate a few hack sessions.

> On Fri, 03 May 2019 23:45:00 +0200
> Ricardo Wurmus <rekado@elephly.net> wrote:
>
>> 1) Mumi
>
> With debbugs I'm missing a feature to easily add/remove tags to bugs,
> like the github bugtracker has. For example, it is not possible to
> search for all bugs for cuirass, or get a list of all "beginners" bug
> (very helpful for Outreachy).
>
> Do you think that feature could be added with Mumi?

Yes.  Debbugs has support for arbitrary tags, but it is a little
strange.  It supports so-called user tags, which are not global but
scoped under a user name or email address.  We could save these tags
under the Mumi user and retrieve them the same way.

We would probably need to restrict access to tagging bugs, though, so
that not every visitor to the bug tracker can modify them.  We would
need some form of authentication: a shared password for admins, or a
admin accounts with their own password, or authentication through GPG
signed commands…

--
Ricardo

postgresql vs sqlite for Cuirass (was Re: Hackathon: Mumi and Cuirass)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1637 bytes --]

Hello Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> writes:

[...]

>> 2) Cuirass
>
> On Guix days, we discussed the idea of using Postgresql as a backend.

Please do you remember the rationale behind this idea?

What is the issue with using sqlite?

Why a client/server approach vs an embedded database?

I don't have much experience with web applications backed by sqlite, but
AFAIU sqlite is sometime "misunderstood"

Some interesting arguments from sqlite developer:
https://sqlite.org/whentouse.html#website
--8<---------------cut here---------------start------------->8---

Dynamic content uses about 200 SQL statements per webpage. This setup
runs on a single VM that shares a physical server with 23 others and yet
still keeps the load average below 0.1 most of the time.

--8<---------------cut here---------------end--------------->8---

https://sqlite.org/np1queryprob.html
--8<---------------cut here---------------start------------->8---

200 SQL statements per webpage is excessive for client/server database
engines like MySQL, PostgreSQL, or SQL Server.

But with SQLite, 200 or more SQL statement per webpage is not a problem.

SQLite can also do large and complex queries efficiently, just like
client/server databases. But SQLite can do many smaller queries
efficiently too. Application developers can use whichever technique
works best for the task at hand.

[...]

N+1 Queries Are Not A Problem With SQLite

--8<---------------cut here---------------end--------------->8---

[...]

Thanks! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: postgresql vs sqlite for Cuirass (was Re: Hackathon: Mumi and Cuirass)

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2093 bytes --]

Hello,

Giovanni Biscuolo <g@xelera.eu> ezt írta (időpont: 2019. máj. 16., Cs,
16:23):

> Hello Björn,
>
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> writes:
>
> [...]
>
> >> 2) Cuirass
> >
> > On Guix days, we discussed the idea of using Postgresql as a backend.
>
> Please do you remember the rationale behind this idea?
>
>
This was just an idea to test if we can make the database scalability
problem we
currently have solved.


> What is the issue with using sqlite?
>

Currently the database response times on sqlite are not acceptable.
Any suggestions to overcome this would be appreciated.


> Why a client/server approach vs an embedded database?
>
> I don't have much experience with web applications backed by sqlite, but
> AFAIU sqlite is sometime "misunderstood"
>
> Some interesting arguments from sqlite developer:
> https://sqlite.org/whentouse.html#website
> --8<---------------cut here---------------start------------->8---
>
> Dynamic content uses about 200 SQL statements per webpage. This setup
> runs on a single VM that shares a physical server with 23 others and yet
> still keeps the load average below 0.1 most of the time.
>
> --8<---------------cut here---------------end--------------->8---
>
> https://sqlite.org/np1queryprob.html
> --8<---------------cut here---------------start------------->8---
>
> 200 SQL statements per webpage is excessive for client/server database
> engines like MySQL, PostgreSQL, or SQL Server.
>
> But with SQLite, 200 or more SQL statement per webpage is not a problem.
>
> SQLite can also do large and complex queries efficiently, just like
> client/server databases. But SQLite can do many smaller queries
> efficiently too. Application developers can use whichever technique
> works best for the task at hand.
>
> [...]
>
> N+1 Queries Are Not A Problem With SQLite
>
> --8<---------------cut here---------------end--------------->8---
>
> [...]
>
> Thanks! Gio'
>
> --
> Giovanni Biscuolo
>
> Xelera IT Infrastructures
>
Best regards,
g_bor

[-- Attachment #2: Type: text/html, Size: 3213 bytes --]

Re: postgresql vs sqlite for Cuirass (was Re: Hackathon: Mumi and Cuirass)

[Aljosha Papsch]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

Hi,

Am Freitag, den 17.05.2019, 10:26 +0200 schrieb Gábor Boskovits:
> Any suggestions to overcome this would be appreciated.

I would go with the client/server database. It helps scaling in the
long term and Guix must take over the world, no? With SQLite you'll be
stuck with the single file on a single server. You could implement your
custom replication using rsync or whatever, but you'll have to ensure
that the receivers do not write to the file (readonly "slaves"). In the
end it will be a problem already solved by Postgres and other
server/client databases.

I don't know about multi-master replication in Postgres, with MySQL it
would be possible.

> 200 SQL statements per webpage is excessive for client/server
> database like MySQL [...] But with SQLite [it's] not a problem.

The main problem the author sees is the roundtrip of the message to the
DB server. It's a common problem that you will get into when moving
from single machine to cluster. IMHO you shouldn't shy away by sticking
to the single machine, but optimize the application to get the benefits
that a cluster provides you (mainly availability).

Best regards
Aljosha
(checking Guix out for web development)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: postgresql vs sqlite for Cuirass (was Re: Hackathon: Mumi and Cuirass)

[Ricardo Wurmus]

Aljosha Papsch <aljosha.papsch@vinexus.eu> writes:

> Am Freitag, den 17.05.2019, 10:26 +0200 schrieb Gábor Boskovits:
>> Any suggestions to overcome this would be appreciated.
>
> I would go with the client/server database. It helps scaling in the
> long term and Guix must take over the world, no? With SQLite you'll be
> stuck with the single file on a single server. You could implement your
> custom replication using rsync or whatever, but you'll have to ensure
> that the receivers do not write to the file (readonly "slaves"). In the
> end it will be a problem already solved by Postgres and other
> server/client databases.

Our current problem is not replication and we’re far from needing it at
this point.  I think there are better ways to improve Cuirass than to
switch to Postgres.

--
Ricardo

Re: postgresql vs sqlite for Cuirass (was Re: Hackathon: Mumi and Cuirass)

[Ricardo Wurmus]

Gábor Boskovits <boskovits@gmail.com> writes:

> Currently the database response times on sqlite are not acceptable.
> Any suggestions to overcome this would be appreciated.

I started playing with Cuirass again to add a search feature and while
looking through the code I was astonished to see how complicated the SQL
queries seem.

Are there any queries in particular that you consider not acceptable?

-- 
Ricardo

Re: postgresql vs sqlite for Cuirass (was Re: Hackathon: Mumi and Cuirass)

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Gábor Boskovits <boskovits@gmail.com> writes:
>
>> Currently the database response times on sqlite are not acceptable.
>> Any suggestions to overcome this would be appreciated.
>
> I started playing with Cuirass again to add a search feature and while
> looking through the code I was astonished to see how complicated the SQL
> queries seem.
>
> Are there any queries in particular that you consider not acceptable?

At the Guix Days, I think there was consensus that the big query of
‘db-get-builds’, although it was initially introduced for performance
reasons, was “problematic.”  I’m sure Gábor, Björn, or Danny would have
a clearer explanation.  :-)

Ludo’.