GSoC: Adding a web interface similar to the Hydra web interface

GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 291 bytes --]

Hello!

I'm Tatiana, I'm a GSoC intern. Now, I'm pursuing my master degree
in Computer Science at Moscow State University.

My project in GSoC is implementation web interface analogous to Hydra
interface.

I am really happy to be accepted for GSoC and excited about the project!

--
Tatiana

[-- Attachment #2: Type: text/html, Size: 2226 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Maxim Cournoyer]

Hello!

Tatiana Sholokhova <tanja201396@gmail.com> writes:

> Hello!
>
> I'm Tatiana, I'm a GSoC intern. Now, I'm pursuing my master degree
> in Computer Science at Moscow State University.

Welcome :)

> My project in GSoC is implementation web interface analogous to Hydra
> interface.

Sounds interesting and useful!

> I am really happy to be accepted for GSoC and excited about the project!

See you in #guix; enjoy your SoC!

Maxim

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hello Tatiana,

Tatiana Sholokhova <tanja201396@gmail.com> skribis:

> I'm Tatiana, I'm a GSoC intern. Now, I'm pursuing my master degree
> in Computer Science at Moscow State University.
>
> My project in GSoC is implementation web interface analogous to Hydra
> interface.

Thanks for your message, and welcome to Guix!

A good starting point for you will be to take a look at the basic ideas
behind Cuirass: what it does, how it structures information about builds
that it performs (which can be inferred from its database¹).

In terms of Web interface, Danny Milosavljevic wrote a simple HTML page
with JavaScript code that queries Cuirass using its HTTP interface².
You can see it here:

  https://berlin.guixsd.org/status/

It’s rough on the edges but it gives you an idea.

Anyway, these are the entry points.  Please do get in touch with us as
you start digging into it.  People on the mailing list and on IRC can
help (I’m “civodul” on IRC.)

Happy hacking!  :-)

Ludo’.

¹ https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/schema.sql
² https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/cuirass/http.scm

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

welcome to Guix!

> A good starting point for you will be to take a look at the basic ideas
> behind Cuirass: what it does, how it structures information about builds
> that it performs (which can be inferred from its database¹).

Some time ago I tried to document the database schema.  This can be
found in the manual.  The source file for the manual is here:

  https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/doc/cuirass.texi

This may not be 100% accurate, but it may be helpful in getting to
understand what Cuirass stores and why.

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Hi Tatjana and welcome!

On Fri, 04 May 2018 14:55:01 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> In terms of Web interface, Danny Milosavljevic wrote a simple HTML page
> with JavaScript code that queries Cuirass using its HTTP interface².
> You can see it here:
> 
>   https://berlin.guixsd.org/status/

For a very quick way to get started, you can click "View Source" in your web
browser and you'll see its source code.

It queries the Cuirass CI server.

If you have any questions, I'm reachable via the list - but not via IRC.

There's a more current version of the program:

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/html/status/index.html

@Ludo: Can you update it on berlin.guixsd.org ?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Danny,

> Hi Tatjana and welcome!
>
> On Fri, 04 May 2018 14:55:01 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> In terms of Web interface, Danny Milosavljevic wrote a simple HTML page
>> with JavaScript code that queries Cuirass using its HTTP interface².
>> You can see it here:
>> 
>>   https://berlin.guixsd.org/status/
>
> For a very quick way to get started, you can click "View Source" in your web
> browser and you'll see its source code.
>
> It queries the Cuirass CI server.
>
> If you have any questions, I'm reachable via the list - but not via IRC.
>
> There's a more current version of the program:
>
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/html/status/index.html
>
> @Ludo: Can you update it on berlin.guixsd.org ?

I’ll see if I can do that today.

Thanks!

-- 
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

we haven’t heard from you since about a week, nor have I seen you on
IRC.  (Granted, I was offline for some time, so I probably just missed
your introduction.)

The official so-called “community bonding period” of GSoC is soon coming
to an end, and it would be advisable to use the remaining time before
coding begins for some community bonding :)

I’d like to repeat that frequent communication is the key to a
successful GSoC project.  If you have any questions about the project,
the code, or how to proceed, please don’t hesitate to let us know.

Thanks!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 2128 bytes --]

Hi all,

Thank you for your help and the provided resources which are very useful
for me at the first stage of the project.

I have built Cuirass on my local computer but I have encountered a few
problems while doing it. It turned out that I had an old version of
guile-sqlite3 installed by guix. I have spent some time discovering this
issue. Eventually, I've found that all I had to do is to update guix
packages list invoking "guix pull" and then update the packages.

Unfortunately, I have no experience of IRC usage and I'm not sure how to do
in a proper way. Of course, I would like to communicate with the community,
but I'm not sure which discussions are relevant in the IRC. For instance,
could I ask questions on building issues similar to ones I've encountered?

I have taken a look at Danny's Cuirass frontend application. Now I try to
run it locally. I have already figured out that I need to change URLPREFIX
and name of the repository and the branch in the code. But I still can't
get it working. According to the browser console, all the queries to
localhost sent by js receive following error:

"No 'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'null' is therefore not allowed access."

But similar queries work well if I enter them manually via the browser or
curl. Could you give me any hints on that?

Best regards,
Tatiana

2018-05-09 20:21 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:

> Hi Tatiana,
>
> we haven’t heard from you since about a week, nor have I seen you on
> IRC.  (Granted, I was offline for some time, so I probably just missed
> your introduction.)
>
> The official so-called “community bonding period” of GSoC is soon coming
> to an end, and it would be advisable to use the remaining time before
> coding begins for some community bonding :)
>
> I’d like to repeat that frequent communication is the key to a
> successful GSoC project.  If you have any questions about the project,
> the code, or how to proceed, please don’t hesitate to let us know.
>
> Thanks!
>
> --
> Ricardo
>
>

[-- Attachment #2: Type: text/html, Size: 2710 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2475 bytes --]

2018-05-13 20:45 GMT+02:00 Tatiana Sholokhova <tanja201396@gmail.com>:

> Hi all,
>
> Thank you for your help and the provided resources which are very useful
> for me at the first stage of the project.
>
> I have built Cuirass on my local computer but I have encountered a few
> problems while doing it. It turned out that I had an old version of
> guile-sqlite3 installed by guix. I have spent some time discovering this
> issue. Eventually, I've found that all I had to do is to update guix
> packages list invoking "guix pull" and then update the packages.
>
> Unfortunately, I have no experience of IRC usage and I'm not sure how to
> do in a proper way. Of course, I would like to communicate with the
> community, but I'm not sure which discussions are relevant in the IRC. For
> instance, could I ask questions on building issues similar to ones I've
> encountered?
>
> I have taken a look at Danny's Cuirass frontend application. Now I try to
> run it locally. I have already figured out that I need to change URLPREFIX
> and name of the repository and the branch in the code. But I still can't
> get it working. According to the browser console, all the queries to
> localhost sent by js receive following error:
>
> "No 'Access-Control-Allow-Origin' header is present on the requested
> resource. Origin 'null' is therefore not allowed access."
>
> But similar queries work well if I enter them manually via the browser or
> curl. Could you give me any hints on that?
>
>
Hello Tatiana,

by any chance do you run this code through a file:///... url? Origin null
usually means that. If so, it would be better to test in a locally
installed webserver.


> Best regards,
> Tatiana
>
> 2018-05-09 20:21 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:
>
>> Hi Tatiana,
>>
>> we haven’t heard from you since about a week, nor have I seen you on
>> IRC.  (Granted, I was offline for some time, so I probably just missed
>> your introduction.)
>>
>> The official so-called “community bonding period” of GSoC is soon coming
>> to an end, and it would be advisable to use the remaining time before
>> coding begins for some community bonding :)
>>
>> I’d like to repeat that frequent communication is the key to a
>> successful GSoC project.  If you have any questions about the project,
>> the code, or how to proceed, please don’t hesitate to let us know.
>>
>> Thanks!
>>
>> --
>> Ricardo
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3507 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tonton]

[-- Attachment #1: Type: text/plain, Size: 3331 bytes --]

Hi!

As far as IRC is concerned you're welcome to take any questions or
conversations that you have there. When people are online it tends to be
faster, and some problems - like live troubleshooting - is better suited to
IRC.

Personally I ask on IRC when I have a question. If it isn't answered or if my
problem is more complex, or maybe it requires a longer explanation, I'll send
it to the mailing list.

All you really need to try is an IRC client (there are also ones you can run
in your browser). And join server chat.freenode.net (port 6697 (or 6667
without ssl)) and channel #guix

Drop by and say hello if you like.

I'm not familiar with cuirass, so I can't help there, sorry. But good luck!

On Sun, 13 May 2018 21:45:04 +0300
Tatiana Sholokhova <tanja201396@gmail.com> wrote:

> Hi all,
> 
> Thank you for your help and the provided resources which are very useful
> for me at the first stage of the project.
> 
> I have built Cuirass on my local computer but I have encountered a few
> problems while doing it. It turned out that I had an old version of
> guile-sqlite3 installed by guix. I have spent some time discovering this
> issue. Eventually, I've found that all I had to do is to update guix
> packages list invoking "guix pull" and then update the packages.
> 
> Unfortunately, I have no experience of IRC usage and I'm not sure how to do
> in a proper way. Of course, I would like to communicate with the community,
> but I'm not sure which discussions are relevant in the IRC. For instance,
> could I ask questions on building issues similar to ones I've encountered?
> 
> I have taken a look at Danny's Cuirass frontend application. Now I try to
> run it locally. I have already figured out that I need to change URLPREFIX
> and name of the repository and the branch in the code. But I still can't
> get it working. According to the browser console, all the queries to
> localhost sent by js receive following error:
> 
> "No 'Access-Control-Allow-Origin' header is present on the requested
> resource. Origin 'null' is therefore not allowed access."
> 
> But similar queries work well if I enter them manually via the browser or
> curl. Could you give me any hints on that?
> 
> Best regards,
> Tatiana
> 
> 2018-05-09 20:21 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:
> 
> > Hi Tatiana,
> >
> > we haven’t heard from you since about a week, nor have I seen you on
> > IRC.  (Granted, I was offline for some time, so I probably just missed
> > your introduction.)
> >
> > The official so-called “community bonding period” of GSoC is soon coming
> > to an end, and it would be advisable to use the remaining time before
> > coding begins for some community bonding :)
> >
> > I’d like to repeat that frequent communication is the key to a
> > successful GSoC project.  If you have any questions about the project,
> > the code, or how to proceed, please don’t hesitate to let us know.
> >
> > Thanks!
> >
> > --
> > Ricardo
> >
> >  



-- 
I use gpg to sign my emails. All the symbols you may see at the bottom
of this mail is my cryptographic signature. It can be ignored, or used
to check that it really is me sending this email. Learn more by asking
me or see: https://u.fsf.org/zb or https://ssd.eff.org/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

> I have taken a look at Danny's Cuirass frontend application. Now I try to
> run it locally. I have already figured out that I need to change URLPREFIX
> and name of the repository and the branch in the code. But I still can't
> get it working. According to the browser console, all the queries to
> localhost sent by js receive following error:
> 
> "No 'Access-Control-Allow-Origin' header is present on the requested
> resource. Origin 'null' is therefore not allowed access."
> 
> But similar queries work well if I enter them manually via the browser or
> curl. Could you give me any hints on that?

You're trying to run it as a file from your local computer, right?

Then it's security settings in your web browser preventing cross-site scripting
attacks.

If you use icecat, please install this extension:

  https://addons.mozilla.org/de/firefox/addon/cors-everywhere/

There's an square icon at the top right of the icecat window then (it says "cors").

If you click on it it will toggle between green and red.

When it's green it means that it will let all requests pass.  WHEN IT'S GREEN, DON'T USE ONLINE BANKING or similar in the same session.

But the index.html frontend application will work just fine then :)

When it's red it means that cross-site scripting attack protection is online.  If in doubt, leave it red.  It should be red most of the time.

I myself develop like that:

For testing,

* Switch CorsE to green
* Refresh Cuirass frontend index.html page
* Try some stuff
* Close Cuirass frontend index.html page
* Switch CorsE to red

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

Tatiana Sholokhova <tanja201396@gmail.com> writes:

> Unfortunately, I have no experience of IRC usage and I'm not sure how to do
> in a proper way. Of course, I would like to communicate with the community,
> but I'm not sure which discussions are relevant in the IRC. For instance,
> could I ask questions on building issues similar to ones I've encountered?

If you need a specific recommendation, I'd suggest you try HexChat for
IRC.  It's available in Guix via "guix package -i hexchat".  There are
other clients, but I've found this one works quite nicely for my simple
needs.

Another popular option I know of is Pidgin, which is also packaged in
Guix.  I haven't used the Guix-installed Pidgin before, though, so I
don't know how well it works at this time.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> Unfortunately, I have no experience of IRC usage and I'm not sure how to do
> in a proper way. Of course, I would like to communicate with the community,
> but I'm not sure which discussions are relevant in the IRC. For instance,
> could I ask questions on building issues similar to ones I've encountered?

Yes, anything relating to the project that is easier discussed live is
appropriate for the IRC channel.

If you don’t have an IRC client you can use

    https://webchat.freenode.net/

Input any nickname and #guix as the channel.

-- 
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 2070 bytes --]

Hello everyone,

Thank you for the proposed ways of dealing with the problem I've faced.
Eventually, I have managed to run Danny's application locally. I have
learned all the queries to Cuirass web API which the application sends and
the general workflow of the interface.

I have started thinking about the type of web interface we want to have for
Cuirass in this project. As far as I see, there are two options:

   - a web application served by Cuirass web server;
   - a standalone static site which sends queries to the Cuirass web API
   (this is similar to Danny's application);

I suppose that the first option has more benefits since it allows to choose
the exact type of information required to be extracted from the database by
a specific part of the web interface. What do think regarding these options?

Since I am a newcomer to the guix developers team, I have a question
related to the interface development workflow. Once we choose the desired
type of the web interface, I will start implementing it. How will we
organize the development process? More precisely, where should I place the
implemented code in order to provide access to it for our team? In my
experience, I have used to create the separate branch in the git
repository. I would like to know which way of doing this you would prefer.

Best regards,
Tatiana

2018-05-14 7:20 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Hi Tatiana,
>
> > Unfortunately, I have no experience of IRC usage and I'm not sure how to
> do
> > in a proper way. Of course, I would like to communicate with the
> community,
> > but I'm not sure which discussions are relevant in the IRC. For instance,
> > could I ask questions on building issues similar to ones I've
> encountered?
>
> Yes, anything relating to the project that is easier discussed live is
> appropriate for the IRC channel.
>
> If you don’t have an IRC client you can use
>
>     https://webchat.freenode.net/
>
> Input any nickname and #guix as the channel.
>
> --
> Ricardo
>
>
>

[-- Attachment #2: Type: text/html, Size: 2715 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I have started thinking about the type of web interface we want to have for
> Cuirass in this project. As far as I see, there are two options:
>
>    - a web application served by Cuirass web server;
>    - a standalone static site which sends queries to the Cuirass web API
>    (this is similar to Danny's application);
>
> I suppose that the first option has more benefits since it allows to choose
> the exact type of information required to be extracted from the database by
> a specific part of the web interface. What do think regarding these options?

You are free to extend the Cuirass web API to suit your application’s
needs.  Having a standalone site is a valid way of providing a web
interface, but it doesn’t have to be the only way of accessing the
information.

Even if you go for the first route, the HTML you serve could talk to the
API.  These two options don’t have to be mutually exclusive.

FWIW, I expect the first approach to be easier because you can use
Scheme for the most part; the pages it serves could be progressively
enhanced with JavaScript, if the client supports it.  I’m sure there are
many users who would prefer a system that would work fine even without
running JavaScript in the browser.

> How will we organize the development process? More precisely, where
> should I place the implemented code in order to provide access to it
> for our team? In my experience, I have used to create the separate
> branch in the git repository. I would like to know which way of doing
> this you would prefer.

I forgot how we did this for past GSoC, but my preference is to do this
in a separate branch of the Cuirass git repository.  Do you have an
account on Savannah yet?  Once you do we could give you permissions to
push your work to a separate branch on the repository.

(You are free to host the code elsewhere as long as we have read
access via git.)

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 3516 bytes --]

Hi Ricardo,

I agree with your point of view on the interface implementation approaches.
Indeed, the approaches are not supposed to be mutually exclusive. As you
said the first approach is easier and I have started working closer to this
approach.

I have already created a small module implementing basic HTML templates in
Scheme. Also, I have made an extension to the Cuirass Web API. It responds
on the "/status" request and generates a page with a minimalistic HTML
table displaying the list of specifications stored in the database.

My login on Savannah is "TSholokhova". I am looking forward to making my
first commit. I would be glad to receive comments on my code to be sure
that I am moving in the right direction.

You have mentioned that many users would prefer an interface working
without javascript running. Am I right that we would like to have a
non-interactive (js-free) interface working and also add some functionality
(e. g. search tools for tables) via javascript?

Also, I have a couple of questions regarding the frontend part. What
frontend framework we would prefer? If I get it right, Hydra uses
Bootstrap. For the frontend implementation, we need to include some static
css&js files in the interface and serve them somehow. Is it a good idea to
serve the static files by Cuirass web server in Scheme?

Best regards,
Tatiana

2018-05-18 23:35 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Hi Tatiana,
>
> > I have started thinking about the type of web interface we want to have
> for
> > Cuirass in this project. As far as I see, there are two options:
> >
> >    - a web application served by Cuirass web server;
> >    - a standalone static site which sends queries to the Cuirass web API
> >    (this is similar to Danny's application);
> >
> > I suppose that the first option has more benefits since it allows to
> choose
> > the exact type of information required to be extracted from the database
> by
> > a specific part of the web interface. What do think regarding these
> options?
>
> You are free to extend the Cuirass web API to suit your application’s
> needs.  Having a standalone site is a valid way of providing a web
> interface, but it doesn’t have to be the only way of accessing the
> information.
>
> Even if you go for the first route, the HTML you serve could talk to the
> API.  These two options don’t have to be mutually exclusive.
>
> FWIW, I expect the first approach to be easier because you can use
> Scheme for the most part; the pages it serves could be progressively
> enhanced with JavaScript, if the client supports it.  I’m sure there are
> many users who would prefer a system that would work fine even without
> running JavaScript in the browser.
>
> > How will we organize the development process? More precisely, where
> > should I place the implemented code in order to provide access to it
> > for our team? In my experience, I have used to create the separate
> > branch in the git repository. I would like to know which way of doing
> > this you would prefer.
>
> I forgot how we did this for past GSoC, but my preference is to do this
> in a separate branch of the Cuirass git repository.  Do you have an
> account on Savannah yet?  Once you do we could give you permissions to
> push your work to a separate branch on the repository.
>
> (You are free to host the code elsewhere as long as we have read
> access via git.)
>
> --
> Ricardo
>
>
>

[-- Attachment #2: Type: text/html, Size: 4188 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I have already created a small module implementing basic HTML templates in
> Scheme. Also, I have made an extension to the Cuirass Web API. It responds
> on the "/status" request and generates a page with a minimalistic HTML
> table displaying the list of specifications stored in the database.

Excellent.

> My login on Savannah is "TSholokhova". I am looking forward to making my
> first commit. I would be glad to receive comments on my code to be sure
> that I am moving in the right direction.

The Cuirass repository is available at
http://git.savannah.gnu.org/cgit/guix/guix-cuirass.git

I have added your account to the group, so you should have write access
to the repository now.  Please push your commits to a new branch on that
repository.  Please also read the section “Coding Style” in the Guix
manual.

You are welcome to rewrite your own published commits in your own
branch, but not in other branches.  When you want us to comment on your
work, please let us know the range of commits that you would like us to
review.  Ideally, we would be able to merge your work into the “master”
branch regularly.

> You have mentioned that many users would prefer an interface working
> without javascript running. Am I right that we would like to have a
> non-interactive (js-free) interface working and also add some functionality
> (e. g. search tools for tables) via javascript?

Yes, this sounds like a good idea.  It is not absolutely necessary, but
it would be nice if basic features of the interface would still be
usable even when JavaScript is disabled.

> Also, I have a couple of questions regarding the frontend part. What
> frontend framework we would prefer? If I get it right, Hydra uses
> Bootstrap.

It is fine to use Bootstrap.  Please include the unminified Bootstrap
CSS and JS files to the repository.  It is easy to minify them later,
but it is virtually impossible to turn minified code into something
readable.

> For the frontend implementation, we need to include some static
> css&js files in the interface and serve them somehow. Is it a good idea to
> serve the static files by Cuirass web server in Scheme?

Yes, this can be done by the Cuirass web server in Scheme.

In the web interface for RCAS I’m dispatching to “render-static-asset”
in the controller:

    https://github.com/BIMSBbioinfo/rcas-web/blob/master/rcas/web/controller.scm

“render-static-asset” is defined here:

    https://github.com/BIMSBbioinfo/rcas-web/blob/master/rcas/web/render.scm#L65

As you can see, this module defines a bunch of mime types and includes
the appropriate type in the headers along with the contents of the file
it serves.

Hope this helps!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 3965 bytes --]

Hi Ricardo,

Thanks for the helpful answer. Now I almost figured out how to serve static
files.

I have committed the first HTML template page (with table of specifications
stored in the database) to web-interface branch. Can you please review it?

I am a bit confused about the database structure. As far as I understand,
there are project_name (project) and branch_name (jobset) properties, but
project_name is a primary key, so a project can't have several branches?

I'm working on static file serving but I don't know how to set the path of
the static file directory. Now I just wrote my absolute path to the
style.css file. So, I have two questions. 1. Where should I place the
static files? 2. How can I execute getcwd function (as you do it in
rcas-web/rcas/config.scm.in)? I tried to add something like

(define-public %current-directory
`(,(getcwd)))

to my config.scm.in but it does not work.

Best regards,
Tatiana Sholokhova

2018-05-22 8:33 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Hi Tatiana,
>
> > I have already created a small module implementing basic HTML templates
> in
> > Scheme. Also, I have made an extension to the Cuirass Web API. It
> responds
> > on the "/status" request and generates a page with a minimalistic HTML
> > table displaying the list of specifications stored in the database.
>
> Excellent.
>
> > My login on Savannah is "TSholokhova". I am looking forward to making my
> > first commit. I would be glad to receive comments on my code to be sure
> > that I am moving in the right direction.
>
> The Cuirass repository is available at
> http://git.savannah.gnu.org/cgit/guix/guix-cuirass.git
>
> I have added your account to the group, so you should have write access
> to the repository now.  Please push your commits to a new branch on that
> repository.  Please also read the section “Coding Style” in the Guix
> manual.
>
> You are welcome to rewrite your own published commits in your own
> branch, but not in other branches.  When you want us to comment on your
> work, please let us know the range of commits that you would like us to
> review.  Ideally, we would be able to merge your work into the “master”
> branch regularly.
>
> > You have mentioned that many users would prefer an interface working
> > without javascript running. Am I right that we would like to have a
> > non-interactive (js-free) interface working and also add some
> functionality
> > (e. g. search tools for tables) via javascript?
>
> Yes, this sounds like a good idea.  It is not absolutely necessary, but
> it would be nice if basic features of the interface would still be
> usable even when JavaScript is disabled.
>
> > Also, I have a couple of questions regarding the frontend part. What
> > frontend framework we would prefer? If I get it right, Hydra uses
> > Bootstrap.
>
> It is fine to use Bootstrap.  Please include the unminified Bootstrap
> CSS and JS files to the repository.  It is easy to minify them later,
> but it is virtually impossible to turn minified code into something
> readable.
>
> > For the frontend implementation, we need to include some static
> > css&js files in the interface and serve them somehow. Is it a good idea
> to
> > serve the static files by Cuirass web server in Scheme?
>
> Yes, this can be done by the Cuirass web server in Scheme.
>
> In the web interface for RCAS I’m dispatching to “render-static-asset”
> in the controller:
>
>     https://github.com/BIMSBbioinfo/rcas-web/blob/
> master/rcas/web/controller.scm
>
> “render-static-asset” is defined here:
>
>     https://github.com/BIMSBbioinfo/rcas-web/blob/
> master/rcas/web/render.scm#L65
>
> As you can see, this module defines a bunch of mime types and includes
> the appropriate type in the headers along with the contents of the file
> it serves.
>
> Hope this helps!
>
> --
> Ricardo
>
>
>

[-- Attachment #2: Type: text/html, Size: 5359 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I have committed the first HTML template page (with table of specifications
> stored in the database) to web-interface branch. Can you please review it?

That’s great!  Congratulations on your first commit!

I’ll make a couple of extra comments on style and conventions because
this is your first commit.  You aren’t expected to remember all of these
conventions right away.

> commit a4fe6dd0d0c82c84a810d3368dd60fea3aa1b2b0 (HEAD -> web-interface, origin/web-interface)
> Author: TSholokhova <tanja201396@gmail.com>
> Date:   Wed May 23 16:37:23 2018 +0300
>
>     basic html templates

Please remember to make expressive commit messages.  We normally use a
format similar to the ChangeLog convention, which consists of a one-line
summary that is usually a sentence, and a listing of all changes.

In this case it would be something like:

--8<---------------cut here---------------start------------->8---
Add basic HTML templates.

* src/cuirass/templates.scm: New file.
* Makefile.am (dist_pkgmodule_DATA): Add it.
* src/cuirass/http.scm (url-handler): Add handler for “status”
endpoint.
--8<---------------cut here---------------end--------------->8---


> diff --git a/src/cuirass/http.scm b/src/cuirass/http.scm
> index e911b9b..f5e3ac1 100644
> --- a/src/cuirass/http.scm
> +++ b/src/cuirass/http.scm
> @@ -1,3 +1,4 @@
> +
>  ;;;; http.scm -- HTTP API
>  ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
>  ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>

Please make sure to add a copyright line for yourself to the bottom of
the copyright block.  Also try to avoid pure whitespace changes like the
addition of an empty line at the top of the file.

> @@ -135,6 +139,12 @@ Hydra format."
>       #:body
>       (object->json-string
>        `((error . ,message)))))
> +
> +  (define (respond-html body)
> +    (respond '((content-type . (text/html)))
> +             #:body (lambda (port)
> +                      (sxml->xml body port)
> +                      )))

Please don’t leave closing parentheses on a line by itself; they feel
lonely and prefer to be on the previous line.

> @@ -223,6 +233,11 @@ Hydra format."
>                                        ,@params
>                                        (order status+submission-time)))))
>             (respond-json-with-error 500 "Parameter not defined!"))))
> +    (("status")
> +     (respond-html (templatize
> +		    "Status"
> +		    (specifications-table
> +		     (with-critical-section db-channel (db) (db-get-specifications db))))))

Here and in other places you used tabs, but we’re using spaces for
indentation in all source files.  Please configure your editor to use
spaces instead of tabs.

I feel that the “url-handler” procedure is already very large, so it may
be a good idea to break out the handler to a separate procedure instead
of having the details inline.  This is not a problem yet, but it may be
good to keep in mind in case you need to grow this handler in the
future.  As long as it stays this small it’s fine to keep it there.

> diff --git a/src/cuirass/templates.scm b/src/cuirass/templates.scm
> new file mode 100644
> index 0000000..ff63469
> --- /dev/null
> +++ b/src/cuirass/templates.scm
> @@ -0,0 +1,32 @@
> +(define-module (cuirass templates)
> +  #:export (templatize
> +            specifications-table))
> +
> +

Please add the usual copyright header to the top of this module.

> +(define (templatize title body)
> +  `(html
> +    ,(head title)
> +    (body ,body)))

Please also add a docstring to every toplevel definition to explain what
the procedure is supposed to do.  “templatize” is quite a fancy name for
what I’d call “html-page” :)

> +
> +(define (head title)
> +  `(head
> +    (meta (@ (charset "utf-8")))
> +    (title ,title)))

This could become part of “templatize” instead.  It’s generally good to
have small independent procedures, but in this case I don’t see us ever
using “head” without “templatize”.

> +(define (specifications-table specs)
> +  `(table
> +    (@ (class "table-fill"))
> +    (thead
> +     (tr
> +      (th (@ (class "text-left")) Name)
> +      (th (@ (class "text-left")) Branch)))
> +    (tbody
> +     (@ (class "table-fill"))
> +     ,@(map
> +        (lambda (spec)
> +          `(tr
> +            (td ,(assq-ref spec #:name))
> +            (td ,(assq-ref spec #:branch))))
> +        specs))))

Please also add a docstring to this procedure.  What is the result of
this procedure if “specs” is empty?  Should that case be covered to
communicate this to the viewer?

> I am a bit confused about the database structure. As far as I understand,
> there are project_name (project) and branch_name (jobset) properties, but
> project_name is a primary key, so a project can't have several branches?

I share your confusion.  Maybe Ludovic or Mathieu can shed some more
light on this.

It is true that “repo_name” (not project_name) in the Specifications
table is the primary key and it is used as the only foreign key on other
tables.  On the other hand, “repo_name” is really just an arbitrary
name.  You could have “Guix (master branch)” as the value for repo_name
with “branch” as “master”, and you could have another specification with
the repo_name “Guix (next)” with “branch” as “core-updates”.

I feel it would be nicer to have a composite primary key consisting of
both the repo_name and the branch, but since the “branch” is an optional
field maybe that’s not so easy.

But I don’t really know if there are other reasons for doing it this
way.

> I'm working on static file serving but I don't know how to set the path of
> the static file directory. Now I just wrote my absolute path to the
> style.css file. So, I have two questions. 1. Where should I place the
> static files? 2. How can I execute getcwd function (as you do it in
> rcas-web/rcas/config.scm.in)? I tried to add something like
>
> (define-public %current-directory
> `(,(getcwd)))

Do you really need the current directory, or could you use %datadir
instead?  Note that the value of %datadir is provided at configuration
time, i.e. when the “configure” script is run.  “configure” generates
“config.scm” from “config.scm.in” by substituting all placeholders
(strings wrapped in “@…@”) with the configured values.

If you want to avoid the need to reconfigure and install cuirass every
time, you could respond to variables set in the “pre-inst-env” script.
For example, you can see the following in src/cuirass/database.scm:

--8<---------------cut here---------------start------------->8---
(define %package-schema-file
  ;; Define to the database schema file of this package.
  (make-parameter (string-append (or (getenv "CUIRASS_DATADIR")
                                     (string-append %datadir "/" %package))
                                 "/schema.sql")))
--8<---------------cut here---------------end--------------->8---

CUIRASS_DATADIR is specified by “pre-inst-env”, so when Cuirass is run
this way, it looks up the schema.sql file from the source directory.  If
Cuirass has been installed, however, and is not run with the
“pre-inst-env” script it looks up the file in the %datadir.

Does this help?

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I've just made a new commit. I tried to fix codestyle mistakes but I'm not
> sure that I managed to cover them all.

Thank you for making the extra effort!

One more thing I’ve noticed is that your patches add trailing whitespace
to some lines (i.e. a line that ends on a space, or an empty line that
only contains a space character), which we try to avoid.  This is just a
minor problem, but we would probably edit these commits to remove these
whitespace changes before merging the commits into the “master” branch.

You are welcome to rewrite history in the “web-interface” branch,
i.e. you can fix the previous commits in your local repository and then
force push them to “web-interface”.  (Please be extra careful not to
force push to the “master” branch.)

Another thing I saw is things like this:

    '(
      something else …)

i.e. an opening parenthesis followed by a line break.  Please try to
avoid those.

> I've added static files support and an example of css style file.
>
> Also, I've added new html templates displaying builds of a specification.
> It is available via "/status/<repo_name>" or via a link from the main page.

Excellent.

For your latest commit please use this commit message body instead:

--8<---------------cut here---------------start------------->8---
* src/cuirass/http.scm (%static-directory, file-mime-types): New variables.
(url-handler): Add handler for “/status/<repo_name>”; add handler for
static files.
* src/static/style.css: New file.
…
--8<---------------cut here---------------end--------------->8---

I recommend merging this commit with the previous commit.  You can use
“git rebase -i $start” to interactively rebase commits and mark the last
two commits to be squashed into one commit.  Please reword the commit
message as well.

I know that this may seem like nitpicking, but it’s a good habit to
acquire early — fixing up commits at the very end is much harder in my
experience.


> Now I'm not sure what I should display in the tables and what pages to
> implement in the interface? I think it would be helpful if I have a more
> realistic database for understanding database structure and testing
> purposes. But I don't know how can I get one.

You can download a copy of the Cuirass database as it is used on
berlin.guixsd.org, one of the build farms of the Guix project.  I have
copied it here:

    http://bootstrappable.org/cuirass.db

It is 12G(!), which indicates that Cuirass adds way too many entries
than absolutely needed.  Ludovic wrote on IRC that we don’t seem to
check if a record already exists when two subsequent evaluations yield
the same build.

I have also put up a smaller database at

    http://bootstrappable.org/cuirass-small.db

which also came from berlin.guixsd.org.  I don’t know if that one would
be useful to you, though, as it is only 48kB in size.

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hello Tatiana & all,

Ricardo Wurmus <rekado@elephly.net> skribis:

>> I am a bit confused about the database structure. As far as I understand,
>> there are project_name (project) and branch_name (jobset) properties, but
>> project_name is a primary key, so a project can't have several branches?
>
> I share your confusion.  Maybe Ludovic or Mathieu can shed some more
> light on this.

It’s confusing indeed, I think it’s a mistake that has yet to be fixed.
Basically what we do now is that we use a different ‘repo_name’ when we
just want to add a branch…

We should fix it at some point.  Suggestions welcome!

I would encourage you to write commits in a way to minimize friction
when we are to merge them—that is, following the conventions that
Ricardo outlined.  That way Mathieu, Ricardo, or myself can take a look
and quickly cherry-pick to master.

Anyway, kudos on what you’ve already achieved!  Getting started on an
existing code base is never easy, so I think you’ve done a good job.

Thank you,
Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

2018-05-29 18:07 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:

> Hello Tatiana & all,
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
> >> I am a bit confused about the database structure. As far as I
> understand,
> >> there are project_name (project) and branch_name (jobset) properties,
> but
> >> project_name is a primary key, so a project can't have several branches?
> >
> > I share your confusion.  Maybe Ludovic or Mathieu can shed some more
> > light on this.
>
> It’s confusing indeed, I think it’s a mistake that has yet to be fixed.
> Basically what we do now is that we use a different ‘repo_name’ when we
> just want to add a branch…
>
> We should fix it at some point.  Suggestions welcome!
>
>
I believe the confusion arises from the fact, that there is a naming
inconsistency.
If we could come up with good names to the things involved, then that might
suggest a clean solution. It seems that the problem is caused by a project
and
a repository is somehow intermixed. That might suggest a solution to keep
project-name as is, as primary key, and add a separate repository field...
or not.
Does that make sense?


> I would encourage you to write commits in a way to minimize friction
> when we are to merge them—that is, following the conventions that
> Ricardo outlined.  That way Mathieu, Ricardo, or myself can take a look
> and quickly cherry-pick to master.
>
> Anyway, kudos on what you’ve already achieved!  Getting started on an
> existing code base is never easy, so I think you’ve done a good job.
>
> Thank you,
> Ludo’.
>
>

[-- Attachment #2: Type: text/html, Size: 2250 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

did you find the database files useful?

Could you please let us know what the current state of your project is?

Thanks,
Ricardo


Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Tatiana,
>
>> I've just made a new commit. I tried to fix codestyle mistakes but I'm not
>> sure that I managed to cover them all.
>
> Thank you for making the extra effort!
>
> One more thing I’ve noticed is that your patches add trailing whitespace
> to some lines (i.e. a line that ends on a space, or an empty line that
> only contains a space character), which we try to avoid.  This is just a
> minor problem, but we would probably edit these commits to remove these
> whitespace changes before merging the commits into the “master” branch.
>
> You are welcome to rewrite history in the “web-interface” branch,
> i.e. you can fix the previous commits in your local repository and then
> force push them to “web-interface”.  (Please be extra careful not to
> force push to the “master” branch.)
>
> Another thing I saw is things like this:
>
>     '(
>       something else …)
>
> i.e. an opening parenthesis followed by a line break.  Please try to
> avoid those.
>
>> I've added static files support and an example of css style file.
>>
>> Also, I've added new html templates displaying builds of a specification.
>> It is available via "/status/<repo_name>" or via a link from the main page.
>
> Excellent.
>
> For your latest commit please use this commit message body instead:
>
> --8<---------------cut here---------------start------------->8---
> * src/cuirass/http.scm (%static-directory, file-mime-types): New variables.
> (url-handler): Add handler for “/status/<repo_name>”; add handler for
> static files.
> * src/static/style.css: New file.
> …
> --8<---------------cut here---------------end--------------->8---
>
> I recommend merging this commit with the previous commit.  You can use
> “git rebase -i $start” to interactively rebase commits and mark the last
> two commits to be squashed into one commit.  Please reword the commit
> message as well.
>
> I know that this may seem like nitpicking, but it’s a good habit to
> acquire early — fixing up commits at the very end is much harder in my
> experience.
>
>
>> Now I'm not sure what I should display in the tables and what pages to
>> implement in the interface? I think it would be helpful if I have a more
>> realistic database for understanding database structure and testing
>> purposes. But I don't know how can I get one.
>
> You can download a copy of the Cuirass database as it is used on
> berlin.guixsd.org, one of the build farms of the Guix project.  I have
> copied it here:
>
>     http://bootstrappable.org/cuirass.db
>
> It is 12G(!), which indicates that Cuirass adds way too many entries
> than absolutely needed.  Ludovic wrote on IRC that we don’t seem to
> check if a record already exists when two subsequent evaluations yield
> the same build.
>
> I have also put up a smaller database at
>
>     http://bootstrappable.org/cuirass-small.db
>
> which also came from berlin.guixsd.org.  I don’t know if that one would
> be useful to you, though, as it is only 48kB in size.


--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Hello Ricardo,

I fixed my previous commits as you adviced. I have tested some features on
examples databases and it was correct. But I can't try your database
(small-cuirass.db) file, the queries just return empty 'queue' and 'latest
build' lists, and I haven't tried other requests yet.

Now we have a web application with two pages: main page with current
specifications status and pages for each specification. I think, for the
next stage I should display some more information from the database but I'm
not sure what else I should display.

Now I'm not sure what features I should implement next? What else do you
want to see by the first deadline?

Best regards,
Tatiana

[-- Attachment #2: Type: text/html, Size: 837 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I fixed my previous commits as you adviced. I have tested some features on
> examples databases and it was correct. But I can't try your database
> (small-cuirass.db) file, the queries just return empty 'queue' and 'latest
> build' lists, and I haven't tried other requests yet.

Oh, maybe there’s something wrong with that file.  I’m sorry.

> Now we have a web application with two pages: main page with current
> specifications status and pages for each specification. I think, for the
> next stage I should display some more information from the database but I'm
> not sure what else I should display.

Have you taken a look at https://hydra.nixos.org yet?  This is the hydra
instance of the Nix project.  (We use an older variant of the same
software on https://hydra.gnu.org, but it’s not very responsive.)

On https://hydra.nixos.org/jobset/nixos/staging you can see a particular
branch of the nixos project.  It lists evaluations, displays the number
of successes, failures, and pending builds, and it links to a detailed
overview of the evaluation.

For example, at https://hydra.nixos.org/eval/1459429 we see the list of
builds that are associated with a particular evaluation and we can
follow a link to a description of that build.  The build page shows us
some information about the derivation/package and links to the build
logs.

It also shows us when the build first failed, what change in the
repository lead to the build failure, and so on.

> Now I'm not sure what features I should implement next? What else do you
> want to see by the first deadline?

We are not just looking for a status page that displays the database
contents.  Some of the bits of information have to be derived from more
than one database record.  When you compare the current state of the
Cuirass web interface to that of Hydra, what do you see is still missing
and should be implemented next?  Could you identify the top 5 features
that you think are missing and could be added to Cuirass?

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 4170 bytes --]

Hello, Ricardo!

Yes, I've taken a look at Hydra. I think my problem is that I don't totally
understand what means some pages at Hydra and how it is related to Cuirass
database tables. Hydra has Projects, Jobsets for each project, Evaluations
and Jobs for each Jobset. And Cuirass has Specifications, Evaluations and
Derivations. Could you please clarify how these entities are related to
each other? I formulate more precise questions below.

I have come up with the following list of features:

1. Add the list of all evaluations of a specification displaying numbers of
successful and failed builds for each evaluation (like this list
https://hydra.nixos.org/jobset/gnu/guile-2-0/evals).
2. Add the table with status information of each job-evaluation pair like
https://hydra.nixos.org/jobset/gnu/guile-2-0/#tabs-jobs.  Am I right that
in terms of Cuirass database derivations correspond to jobs? So, we need to
display "derivations" table with derivations as rows and evaluations as
columns and fill the cells using information from the "builds" table? Also,
it is not clear to me how to order evaluations by their creation time like
it's done in Hydra.
3. Add page displaying information about a build similar to
https://hydra.nixos.org/build/74870692. Am I right that here we should
display the information stored in hydra-like build description (defined
in http.scm) alongside with links to the build log and outputs?
4. Add additional information about previous builds (latest successful,
first broken, etc) on this build page. For this feature, we need to extend
database requests functionality.
5. Add job information page containing the list of job's latest builds like
https://hydra.nixos.org/job/gnu/guile-2-0/build.x86_64-linux.

What do you think about these features? I afraid that I am not familiar
with typical Hydra use cases, so I may miss some important features.

Best regards,
Tatiana


2018-06-03 22:40 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:

> Hi Tatiana,
>
> > I fixed my previous commits as you adviced. I have tested some features
> on
> > examples databases and it was correct. But I can't try your database
> > (small-cuirass.db) file, the queries just return empty 'queue' and
> 'latest
> > build' lists, and I haven't tried other requests yet.
>
> Oh, maybe there’s something wrong with that file.  I’m sorry.
>
> > Now we have a web application with two pages: main page with current
> > specifications status and pages for each specification. I think, for the
> > next stage I should display some more information from the database but
> I'm
> > not sure what else I should display.
>
> Have you taken a look at https://hydra.nixos.org yet?  This is the hydra
> instance of the Nix project.  (We use an older variant of the same
> software on https://hydra.gnu.org, but it’s not very responsive.)
>
> On https://hydra.nixos.org/jobset/nixos/staging you can see a particular
> branch of the nixos project.  It lists evaluations, displays the number
> of successes, failures, and pending builds, and it links to a detailed
> overview of the evaluation.
>
> For example, at https://hydra.nixos.org/eval/1459429 we see the list of
> builds that are associated with a particular evaluation and we can
> follow a link to a description of that build.  The build page shows us
> some information about the derivation/package and links to the build
> logs.
>
> It also shows us when the build first failed, what change in the
> repository lead to the build failure, and so on.
>
> > Now I'm not sure what features I should implement next? What else do you
> > want to see by the first deadline?
>
> We are not just looking for a status page that displays the database
> contents.  Some of the bits of information have to be derived from more
> than one database record.  When you compare the current state of the
> Cuirass web interface to that of Hydra, what do you see is still missing
> and should be implemented next?  Could you identify the top 5 features
> that you think are missing and could be added to Cuirass?
>
> --
> Ricardo
>
>

[-- Attachment #2: Type: text/html, Size: 7712 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> Yes, I've taken a look at Hydra. I think my problem is that I don't totally
> understand what means some pages at Hydra and how it is related to Cuirass
> database tables. Hydra has Projects, Jobsets for each project, Evaluations
> and Jobs for each Jobset. And Cuirass has Specifications, Evaluations and
> Derivations. Could you please clarify how these entities are related to
> each other?

This is a good question and I can relate to your confusion about these
terms :)

A Cuirass “Specification” roughly combines what Hydra calls a “Project”
with a “Jobset”.  On Hydra you would have a Project “gnu” (arbitrary
name) with a Jobset for building all packages from the “master” branch,
and another Jobset for building all packages from the “core-updates”
branch.  A Cuirass specification looks something like this (taken from
the manual):

--8<---------------cut here---------------start------------->8---
 `((#:name . "hello")
   (#:url . "git://git.savannah.gnu.org/guix.git")
   (#:branch . "master")
   (#:no-compile? . #t)
   (#:load-path . ".")
   (#:proc . cuirass-jobs)
   (#:file . "/tmp/drv-file.scm")
   (#:arguments (subset . "hello")))
--8<---------------cut here---------------end--------------->8---

As you can see it includes a name and the repository URL (~= “Project”),
as well as the branch and a procedure “proc” that specifies how this is
to be built.

Evaluations in Hydra and Cuirass are the same concept.  The Cuirass
manual says:

    An evaluation relates a specification with the revision of the
    repository specified therein.  Derivations and builds (see below)
    each belong to a specific evaluation.

When an evaluation is started, we take a look at the current state of
the repository and then determine all build processes that need to be
performed.

Derivations are a fundamental concept in Guix.  The Guix manual says
this about derivations:

--8<---------------cut here---------------start------------->8---
Low-level build actions and the environment in which they are performed
are represented by “derivations”.  A derivation contains the following
pieces of information:

   • The outputs of the derivation—derivations produce at least one file
     or directory in the store, but may produce more.

   • The inputs of the derivations, which may be other derivations or
     plain files in the store (patches, build scripts, etc.)

   • The system type targeted by the derivation—e.g., ‘x86_64-linux’.

   • The file name of a build script in the store, along with the
     arguments to be passed.

   • A list of environment variables to be defined.

   Derivations allow clients of the daemon to communicate build actions
to the store.
--8<---------------cut here---------------end--------------->8---

When the build actions that are prescribed by a derivation are
performed, the result is a build.  This could be a failed build or a
directory containing the files that were generated by compiling a
package.

In the Hydra web interface “derivations” are not mentioned; instead they
speak of “Jobs”, which is roughly the same thing.  After all, a
derivation needs to be processed to get a build, so we can consider the
collection of derivations in a particular evaluation of a specification
to be a collection of build jobs.

> I have come up with the following list of features: […]

The list looks very good.  Please see my comments below.

> 1. Add the list of all evaluations of a specification displaying numbers of
> successful and failed builds for each evaluation (like this list
> https://hydra.nixos.org/jobset/gnu/guile-2-0/evals).

Just a note: The example URL you picked may be a little misleading
because hydra.nixos.org is used slightly differently from hydra.gnu.org
— for our specifications the number of successful, failed, and ongoing
builds would be considerably larger than in this example.

This would be a very useful feature, because we could observe a trend:
as the repository is changed and an increasing number of evaluations are
completed, do more builds fail now than before, or are we improving…?

Clicking on the number of failed/successful/pending builds we should get
a list of those builds/derivations.

> 2. Add the table with status information of each job-evaluation pair like
> https://hydra.nixos.org/jobset/gnu/guile-2-0/#tabs-jobs.

A table like this would be very large, because in our case evaluations
result in thousands of derivations (= jobs).  So you need to design this
to use pagination.

With a very long table, collecting all this past information (on the
columns) can be expensive, so I’d suggest to only display the *current*
state, not the state over a range of past days.

> Am I right that
> in terms of Cuirass database derivations correspond to jobs?

Correct.

> So, we need to
> display "derivations" table with derivations as rows and evaluations as
> columns and fill the cells using information from the "builds" table?

Correct.

> Also,
> it is not clear to me how to order evaluations by their creation time like
> it's done in Hydra.

The “Evaluations” table has an “id” field, which contains an
incrementing numeric identifier.  While you may not be able to sort them
by creation time you could sort them by “id”.

> 3. Add page displaying information about a build similar to
> https://hydra.nixos.org/build/74870692. Am I right that here we should
> display the information stored in hydra-like build description (defined
> in http.scm) alongside with links to the build log and outputs?

Correct.

Most important for us is to see the state of the build, the derivation
that corresponds to the build and the build log.

> 4. Add additional information about previous builds (latest successful,
> first broken, etc) on this build page. For this feature, we need to extend
> database requests functionality.

Yes, obtaining this information may require some new procedures in
Cuirass to perform SQL queries efficiently.

> 5. Add job information page containing the list of job's latest builds like
> https://hydra.nixos.org/job/gnu/guile-2-0/build.x86_64-linux.

Yes, this would be good.

> What do you think about these features? I afraid that I am not familiar
> with typical Hydra use cases, so I may miss some important features.

Your selection of features is very good.

What we want to know when looking at the Cuirass web interface is:

1) is the build farm building something?
2) if so, what is it building?
3) how many more things does it need to build in this round (=
   “evaluation”)?
4) has any build failed?
5) if so, how did the builds fail?
6) what build failure is blocking a certain package from being built?
7) is the percentage of failing builds decreasing over time as we modify
   the repository?
…

The Cuirass web interface should allow us to answer questions of this
kind.  The features you have selected to work on next would already be
very useful to answer some of these questions.  Good job!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1680 bytes --]

Hi Tatiana,

> I afraid that I am not familiar with typical Hydra use cases

Generally, the continuous integration process should enable developers
to get feedback about the effects of their changes.

This means that as soon as a commit is made, usually an evaluation of
the build source on the continuous integration server starts.

(Sometimes there are exceptions to this (for example in order to not overload
the build servers) - but generally it's true)

For a new evaluation, as a developer I'd like to know:

* Are now more packages broken than before?  Which ones?
* Are now more packages working than before?  Which ones?
* Do some packages work on more architectures than before? Fewer?
* Is the build server still building my change?  Or is it done and I
can trust that the information I see is now complete?  If not,
what is it building now or later?

"before" means "with the previous evaluation" or "with some specific past
evaluation" or "in another branch".

I think this would be the most basic functionality.

More advanced functionalities would include automatic tracking on the reason
of the failure:

* If it's dependency failure, specifically mark this package so I know I don't
have to fix this package - I have to fix a package this one depends on (which one?).
* What kind of failure is it? What's the latest non-noise error message in the
build log?  Display suggestions on what to do about it.

What do you think?

>4. Add additional information about previous builds (latest successful,
>first broken, etc) on this build page. For this feature, we need to extend
>database requests functionality.

Sounds good.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1.1: Type: text/plain, Size: 3730 bytes --]

Hello!

Thank you all for very useful comments describing the database and Hydra
use cases! They were very helpful for me on this week.

I've just committed a new version of the interface. I've implemented the
first feature and create a more friendly interface based on bootstrap.
I had to add new database requests: db-get-evaluations-count,
db-get-evaluations-info for the feature. I have added new endpoints:
("jobset" name), ("eval" id) and changed "status" endpoint to "/". Now,
when you launch Cuirass you can see all specifications on the main page
(localhost:PORT/);
when you click on the specification name you can see a list of all
evaluations of a specification displaying numbers of successful, failed and
pending builds for each evaluation; when you click on the evaluation ID you
can see a list of builds with their statuses. The evaluation list is broken
down into a set of pages with 20 evaluations on each page. I have
implemented a page navigation tool which may be used for other pages of
this kind that we will implement later.

Could you please take a look at the commit and new functions?

I am still facing the local testing issue. When I tried to launch Cuirass
with the large database you sent before it crashed with some git error. For
now, I change my local database manually for testing. I still do not have
an idea how I should fix this problem. Maybe you could recommend me some
specifications to add to my local database? Also, maybe you have some
remote server with working Cuirass where I would be able to test the
interface? I've attached some illustrations of the interface pages I have
locally.

Now I am going to implement separate pages for builds with different
statuses and implementation of the first feature will be finished. Also, I
think It will be useful if I add some more navigation buttons to the
header. Now it has only one link to the main page with Guix logo.

Best regards,
Tatiana

2018-06-06 21:02 GMT+03:00 Danny Milosavljevic <dannym@scratchpost.org>:

> Hi Tatiana,
>
> > I afraid that I am not familiar with typical Hydra use cases
>
> Generally, the continuous integration process should enable developers
> to get feedback about the effects of their changes.
>
> This means that as soon as a commit is made, usually an evaluation of
> the build source on the continuous integration server starts.
>
> (Sometimes there are exceptions to this (for example in order to not
> overload
> the build servers) - but generally it's true)
>
> For a new evaluation, as a developer I'd like to know:
>
> * Are now more packages broken than before?  Which ones?
> * Are now more packages working than before?  Which ones?
> * Do some packages work on more architectures than before? Fewer?
> * Is the build server still building my change?  Or is it done and I
> can trust that the information I see is now complete?  If not,
> what is it building now or later?
>
> "before" means "with the previous evaluation" or "with some specific past
> evaluation" or "in another branch".
>
> I think this would be the most basic functionality.
>
> More advanced functionalities would include automatic tracking on the
> reason
> of the failure:
>
> * If it's dependency failure, specifically mark this package so I know I
> don't
> have to fix this package - I have to fix a package this one depends on
> (which one?).
> * What kind of failure is it? What's the latest non-noise error message in
> the
> build log?  Display suggestions on what to do about it.
>
> What do you think?
>
> >4. Add additional information about previous builds (latest successful,
> >first broken, etc) on this build page. For this feature, we need to extend
> >database requests functionality.
>
> Sounds good.
>

[-- Attachment #1.2: Type: text/html, Size: 4388 bytes --]

[-- Attachment #2: Screenshot of Builds for second evaluation.png --]
[-- Type: image/png, Size: 89467 bytes --]

[-- Attachment #3: Screenshot of guix Evaluations page.png --]
[-- Type: image/png, Size: 109899 bytes --]

[-- Attachment #4: Screenshot of Main page.png --]
[-- Type: image/png, Size: 24941 bytes --]

[-- Attachment #5: Screenshot of random Evaluations page.png --]
[-- Type: image/png, Size: 38303 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I've just committed a new version of the interface. I've implemented the
> first feature and create a more friendly interface based on bootstrap.

I’ve looked at the screenshots and have to say that this is really
looking good already.  Exciting!

> I had to add new database requests: db-get-evaluations-count,
> db-get-evaluations-info for the feature. I have added new endpoints:
> ("jobset" name), ("eval" id) and changed "status" endpoint to "/".

Okay.

> Now, when you launch Cuirass you can see all specifications on the
> main page (localhost:PORT/); when you click on the specification name
> you can see a list of all evaluations of a specification displaying
> numbers of successful, failed and pending builds for each evaluation;
> when you click on the evaluation ID you can see a list of builds with
> their statuses.

Excellent.

> The evaluation list is broken down into a set of pages with 20
> evaluations on each page. I have implemented a page navigation tool
> which may be used for other pages of this kind that we will implement
> later.

Sounds good.

> Could you please take a look at the commit and new functions?

I will take a look today and reply with some comments on your commits.

> I am still facing the local testing issue. When I tried to launch Cuirass
> with the large database you sent before it crashed with some git
> error.

Could you share the error message with us?

> Maybe you could recommend me some
> specifications to add to my local database?

I wasn’t sure about this, so I asked on the #guix IRC channel.  Ludovic
replied there that the Cuirass repository contains a “random”
specification in “examples/random.scm”.  It uses
“examples/random-jobs.scm” to generate … random jobs :)

Note that you’ll need to have Guix installed to use this, and you need
to start Cuirass from your source checkout.

> Now I am going to implement separate pages for builds with different
> statuses and implementation of the first feature will be finished. Also, I
> think It will be useful if I add some more navigation buttons to the
> header. Now it has only one link to the main page with Guix logo.

These sound like good next steps.  Thank you, Tatiana!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hello Tatiana & Ricardo!

Ricardo Wurmus <rekado@elephly.net> skribis:

> I wasn’t sure about this, so I asked on the #guix IRC channel.  Ludovic
> replied there that the Cuirass repository contains a “random”
> specification in “examples/random.scm”.  It uses
> “examples/random-jobs.scm” to generate … random jobs :)

Specifically, here’s how I would launch Cuirass for testing purposes:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env cuirass -D cuirass.db -I 10 -S examples/random.scm
2018-06-11T13:20:58 running Fibers on 4 kernel threads
2018-06-11T13:20:58 marking stale builds as "scheduled"...
2018-06-11T13:20:58 listening on 127.0.0.1:8080
2018-06-11T13:20:58 retrieving list of pending builds...
2018-06-11T13:20:58 heap: 11.82 MiB; threads: 10; file descriptors: 55
2018-06-11T13:20:58 considering spec 'random', URL 'file:///data/src/cuirass'
2018-06-11T13:20:58 canceling 3 stale builds
2018-06-11T13:20:58 restarting 0 pending builds
2018-06-11T13:20:58 building 0 derivations in batches of 200
2018-06-11T13:20:58 done with 0 derivations
2018-06-11T13:20:58 done with restarted builds
2018-06-11T13:20:58 spec 'random': fetched commit "238f856e48ee333ed3e19fa32ce5e1742c650c67" (stamp was "43be95c40a433d21f65c9e6bfb04ccc9fa8e2db4")
2018-06-11T13:20:58 next evaluation in 10 seconds
2018-06-11T13:20:58 evaluating 'random' with commit "238f856e48ee333ed3e19fa32ce5e1742c650c67"
evaluating random jobs from directory "/gnu/store/bb7x9wgc91h9jndyd9k36dysqnamjmyl-cuirass-238f856", commit "238f856e48ee333ed3e19fa32ce5e1742c650c67"
2018-06-11T13:20:59 created evaluation 5 for random, commit 238f856e48ee333ed3e19fa32ce5e1742c650c67
2018-06-11T13:20:59 building 11 jobs for 'random'
2018-06-11T13:20:59 building 11 derivations in batches of 200
2018-06-11T13:20:59 building batch of 200 derivations (0/11)
2018-06-11T13:21:00 build started: '/gnu/store/npkk2v9n3lrs99j6hfm2sa7z839q00lz-random0.drv'
2018-06-11T13:21:00 build started: '/gnu/store/xbsa9sk4aipcvkqpxai73pzad523mwnc-random1.drv'
2018-06-11T13:21:08 considering spec 'random', URL 'file:///data/src/cuirass'
2018-06-11T13:21:08 spec 'random': fetched commit "238f856e48ee333ed3e19fa32ce5e1742c650c67" (stamp was "238f856e48ee333ed3e19fa32ce5e1742c650c67")
2018-06-11T13:21:08 next evaluation in 10 seconds
--8<---------------cut here---------------end--------------->8---

This example instructs Cuirass to populate the ‘cuirass.db’ file from
the current directory, to check the repo in the current directory every
10 seconds, and to use the job specification from ‘examples/random.scm’.
The HTTP server of Cuirass is listening on ‘localhost’, port 8080.

Let me know if you have any questions!  (I’m civodul on #guile.)

HTH,
Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 3238 bytes --]

Hi Tatiana,

nice work!

I have a few comments:

db-get-builds looks fine and we could merge this change to master.  But you have other changes in the same commit, so we can't directly cherry-pick it.  (not so bad, but somewhat cumbersome now)

I'd suggest to rename "db-get-evaluations-info" to "db-get-evaluations-build-summary".  Please don't use nonsensical names like "succ" or "que" in public interfaces (f.e. #:que ).

Nitpick: You have a few variables with "_" like "eval_cnt".  This is not customary in Scheme.  Also, having funny abbreviations like "cnt" isn't either.  I suggest "evaluation-count".

For db-get-evaluations-count: Please add a short docstring documenting what it's for (i.e. "Return the number of evaluations of the given specification SPEC").

In src/cuirass/http.scm :

I don't think that HTML5 is XML-compatible.  So if not, please use XHTML preamble and content-type (I think that that is even broken on master - but it's not difficult to fix).  It's also easier on web browsers if they can assume well-formedness once their XML parser is done parsing.

respond-static-file: We should not second-guess the VFS layer.  Checking for ".." gives an illusion of security when in fact random things could be mounted and also the VFS could have funny syntax for who knows what on the filesystem.  Let's rather have a static list of permissible names and allow those (whitelist).  That's the intention of the check anyway, right?

Also, in light of an ever-changing backing store (cuirass continusly evaluates things), the way you are doing pagination is not the correct way to do it because the data set will scroll underneath you and you will miss items (or see duplicate items) as an user.  Also, it's slow and the DBMS can't reuse anything because you are cutting it off and offseting it over and over again and the transaction isolation level is too low for the DBMS to be able to do anything about it[1].

See also https://use-the-index-luke.com/blog/2013-07/pagination-done-the-postgresql-way for a much better way.  Basically, remember the last value and use it as a limit in the WHERE condition by this one (also, sorting always).

That means if you have the (sorted) list:

A
B
C
E
G
H

and for some reason only 3 fit the page, print first page:

A
B
C <--- print, and remember this one for the boundary of the next page

Second page is "after C", so "... WHERE ... > 'C'":

E
G
H <--- print; and remember that no next page exists.

Note that this also works if, in the mean time, "D" appeared in the backing store.

Then second page is "after C", so "... WHERE ... > 'C'":

D
E
G <--- print, and remember this one for the boundary of the next page

This would only cause a problem if items vanished, so we should keep this in mind.  And jumps to random pages by page number would be difficult now (but who needs that in this case?  Much better to be able to jump by actual timestamp, which then would be easily possible).

In the case of evaluations, it would be good to sort by timestamp and then by evaluation id - since no matter what this would be a stable sort criteria.
(maybe revision, too)

[1] We should fix the latter eventually, too.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

I have only little to add to Danny’s comments.

> db-get-builds looks fine and we could merge this change to master.
> But you have other changes in the same commit, so we can't directly
> cherry-pick it.  (not so bad, but somewhat cumbersome now)

It is good practise to make small commits, one for every set of
logically connected changes.  This makes the review simpler and it makes
it easier to merge some parts while leaving others for later.

As you work on the changes that Danny’s comments imply, please take the
opportunity to group related changes and commit only those together.  It
is fine and desirable to have many independent small commits.

Thanks again for your excellent work!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 3141 bytes --]

Hello,

Thank you for your reviews!

I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.


respond-static-file: We should not second-guess the VFS layer.  Checking
> for ".." gives an illusion of security when in fact random things could be
> mounted and also the VFS could have funny syntax for who knows what on the
> filesystem.  Let's rather have a static list of permissible names and allow
> those (whitelist).  That's the intention of the check anyway, right?
>

I adopted the static file serving procedure from code shared by Ricardo.

https://github.com/BIMSBbioinfo/rcas-web/blob/22a114a0f281845117ed0ab105267f132fc525e4/rcas/web/render.scm#L68

I am considering the following possible implementation of a whitelist. We
can create association list with allowed file names and their mime types (to
replace file-mime-type list). What do you think about it?

Also, in light of an ever-changing backing store (cuirass continusly
> evaluates things), the way you are doing pagination is not the correct way
> to do it because the data set will scroll underneath you and you will miss
> items (or see duplicate items) as an user.  Also, it's slow and the DBMS
> can't reuse anything because you are cutting it off and offseting it over
> and over again and the transaction isolation level is too low for the DBMS
> to be able to do anything about it[1].


I understand how to fix the pagination on the database level. However, it
is not completely clear to me how to incorporate the solution in the
web-request handling. Now we have the only one parameter of the request
which is related to pagination. It is the page number. Should we add more
parameters to the request in order to request evaluations with specific
timestamps and IDs? Or there is some other way of doing that? I have
checked the Hydra pagination request structure. It uses the same form of
the request path for pagination buttons that we have now:
"?page=<page-id>".

It is good practise to make small commits, one for every set of
> logically connected changes.  This makes the review simpler and it makes
> it easier to merge some parts while leaving others for later.


Okay, I will follow this commit strategy.

Best regards,
Tatiana



2018-06-13 0:52 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Hi Tatiana,
>
> I have only little to add to Danny’s comments.
>
> > db-get-builds looks fine and we could merge this change to master.
> > But you have other changes in the same commit, so we can't directly
> > cherry-pick it.  (not so bad, but somewhat cumbersome now)
>
> It is good practise to make small commits, one for every set of
> logically connected changes.  This makes the review simpler and it makes
> it easier to merge some parts while leaving others for later.
>
> As you work on the changes that Danny’s comments imply, please take the
> opportunity to group related changes and commit only those together.  It
> is fine and desirable to have many independent small commits.
>
> Thanks again for your excellent work!
>
> --
> Ricardo
>
>

[-- Attachment #2: Type: text/html, Size: 6926 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3653 bytes --]

2018-06-13 0:43 GMT+02:00 Tatiana Sholokhova <tanja201396@gmail.com>:

> Hello,
>
> Thank you for your reviews!
>
> I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
>
> respond-static-file: We should not second-guess the VFS layer.  Checking
>> for ".." gives an illusion of security when in fact random things could be
>> mounted and also the VFS could have funny syntax for who knows what on the
>> filesystem.  Let's rather have a static list of permissible names and allow
>> those (whitelist).  That's the intention of the check anyway, right?
>>
>
> I adopted the static file serving procedure from code shared by Ricardo.
>
> https://github.com/BIMSBbioinfo/rcas-web/blob/
> 22a114a0f281845117ed0ab105267f132fc525e4/rcas/web/render.scm#L68
>
> I am considering the following possible implementation of a whitelist. We
> can create association list with allowed file names and their mime types (to
> replace file-mime-type list). What do you think about it?
>
> Also, in light of an ever-changing backing store (cuirass continusly
>> evaluates things), the way you are doing pagination is not the correct way
>> to do it because the data set will scroll underneath you and you will miss
>> items (or see duplicate items) as an user.  Also, it's slow and the DBMS
>> can't reuse anything because you are cutting it off and offseting it over
>> and over again and the transaction isolation level is too low for the DBMS
>> to be able to do anything about it[1].
>
>
> I understand how to fix the pagination on the database level. However, it
> is not completely clear to me how to incorporate the solution in the
> web-request handling. Now we have the only one parameter of the request
> which is related to pagination. It is the page number.
>

Actually one parameter is still enough, but it will be the last item on the
previous page. This style of pagination supports sequential visiting of
pages from the first
page. It is later possible to extend on this for example by interpolating
on a counter, but on the first run this also needs only one parameter...


> Should we add more parameters to the request in order to
> request evaluations with specific timestamps and IDs? Or there is some
> other way of doing that? I have checked the Hydra pagination request
> structure. It uses the same form of the request path for pagination buttons
> that we have now: "?page=<page-id>".
>
> It is good practise to make small commits, one for every set of
>> logically connected changes.  This makes the review simpler and it makes
>> it easier to merge some parts while leaving others for later.
>
>
> Okay, I will follow this commit strategy.
>
> Best regards,
> Tatiana
>
>
>
> 2018-06-13 0:52 GMT+03:00 Ricardo Wurmus <rekado@elephly.net>:
>
>>
>> Hi Tatiana,
>>
>> I have only little to add to Danny’s comments.
>>
>> > db-get-builds looks fine and we could merge this change to master.
>> > But you have other changes in the same commit, so we can't directly
>> > cherry-pick it.  (not so bad, but somewhat cumbersome now)
>>
>> It is good practise to make small commits, one for every set of
>> logically connected changes.  This makes the review simpler and it makes
>> it easier to merge some parts while leaving others for later.
>>
>> As you work on the changes that Danny’s comments imply, please take the
>> opportunity to group related changes and commit only those together.  It
>> is fine and desirable to have many independent small commits.
>>
>> Thanks again for your excellent work!
>>
>> --
>> Ricardo
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 8369 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 3092 bytes --]

Hi Tatiana,

On Wed, 13 Jun 2018 01:43:31 +0300
Tatiana Sholokhova <tanja201396@gmail.com> wrote:

> I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.

Yeah, looks much more regular now.

> I adopted the static file serving procedure from code shared by Ricardo.
> 
> https://github.com/BIMSBbioinfo/rcas-web/blob/22a114a0f281845117ed0ab105267f132fc525e4/rcas/web/render.scm#L68

I understand.

It's common to do stuff like that - but it's just an unnecessary risk.

Especially since we have cuirass build (and presumably sign) everything for us,
it would be very very bad for an attacker to be able to read out the private key
used to sign everything.

Blacklists are a whack-a-mole approach (maintaining it will never end) - and in
this case nothing in POSIX states that these characters are the only
problematic ones (and I know of quite a few filesystems where there are a lot
more problematic strings).

Much better to have a whitelist.

> I am considering the following possible implementation of a whitelist. We
> can create association list with allowed file names and their mime types (to
> replace file-mime-type list). What do you think about it?

Yes, that would be a simple way - and easily maintainable, too.  Let's do it
like that.

> Now we have the only one parameter of the request
> which is related to pagination. It is the page number. Should we add more
> parameters to the request in order to request evaluations with specific
> timestamps and IDs? Or there is some other way of doing that?

I don't think we would need the parameter for the page number anymore.

I think we should add a parameter like "boundary" or something
(or multiple ones - I defer to your judgement).

It would also be possible to use "boundary=a&boundary=b" (note: same name).
Not sure whether that would be obtuse or not.

From a technical standpoint it doesn't matter - you just have to have get a
tuple of data cells (of the last row) into the query string somehow.
Also, it has to be able to be entirely optional (in order to get the first page).

The tuple of data cells should uniquely identify one row in the result.  (If it
didn't, you'd skip the other same-value rows when going to the next page)

(There would be an alternative possible to be able to dispense of this
requirement, but I think it would be too obtuse to maintain)

For the evaluations, the relevant data cells would be (starttime, id,
revision) or so.

I suggest adding starttime to the "Evaluations" database table to
improve usability.

(If we wanted to eventually provide a way for the user to sort columns,
we'd have to also adapt what columns this cell list contains - although
it doesn't make sense to stay on page 243 when you change the sort order :) )

What do you think?

> I have
> checked the Hydra pagination request structure. It uses the same form of
> the request path for pagination buttons that we have now:
> "?page=<page-id>".

Yeah, the practice is widespread.  Let's do better than that if we can.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Joshua Branson]

Tatiana Sholokhova <tanja201396@gmail.com> writes:

> Hello,
>
> Thank you for your reviews!
>
> I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.

Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
(This question comes from non guix developer by the way.  Just an enthusiast).

>
>
>  Thanks again for your excellent work!
>
>  --
>  Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

Joshua Branson <jbranso@fastmail.com> ezt írta (időpont: 2018. jún. 13.,
Sze 15:52):

> Tatiana Sholokhova <tanja201396@gmail.com> writes:
>
> > Hello,
> >
> > Thank you for your reviews!
> >
> > I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
> Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
> (This question comes from non guix developer by the way.  Just an
> enthusiast).
>
> >
> >
> >  Thanks again for your excellent work!
> >
> >  --
> >  Ricardo
>
Joshua, both formats have advantages, and also disadvantages. I think this
issue deserves its own discussion. It would be nice to collect the
opinions, but I think that currently we have a good quality and well
integrated xml generator, and we do not have a html5 generator. Please
correct me if I am wrong.

>

[-- Attachment #2: Type: text/html, Size: 1514 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Joshua Branson]

Gábor Boskovits <boskovits@gmail.com> writes:

> Joshua Branson <jbranso@fastmail.com> ezt írta (időpont: 2018. jún. 13., Sze 15:52):
>
>  Tatiana Sholokhova <tanja201396@gmail.com> writes:
>
>  > Hello,
>  >
>  > Thank you for your reviews!
>  >
>  > I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
>  Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
>  (This question comes from non guix developer by the way.  Just an enthusiast).
>
>  >
>  >
>  >  Thanks again for your excellent work!
>  >
>  >  --
>  >  Ricardo
>
> Joshua, both formats have advantages, and also disadvantages. I think this issue deserves its own discussion. It would be nice to collect the opinions, but I think that currently we have a good
> quality and well integrated xml generator, and we do not have a html5 generator. Please correct me if I am wrong.

Nope you sound right.  I'm not arguing in favor or html5 over
xhtml. Both are widely supported by all browsers.  Just trying to
learn.  Thanks!

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 50 bytes --]

Can you please send us an update on your project?

[-- Attachment #2: Type: text/html, Size: 71 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

Hello all,

On the last week, I had fallen out of the process. I had been having exams
at the university since the beginning of June. The last exam was
rescheduled and that has affected my plans. I am sorry for that. Now the
semester is finished and I am having much more time to focus on our
project. I am very happy to pass the first evaluation. It is a pleasure for
me to work with such a great team!

In a few days, I am going to implement the whitelist for the static files
and improve pagination tool performance as we discussed before. Also, I
intend to make the pagination tool universal and add it to the page which
displays all builds of a selected evaluation. Do you think it would be okay
if I add auxiliary pagination filters to the request which retrieves builds
from the database?

Best regards,
Tatiana


2018-06-25 13:46 GMT+03:00 Gábor Boskovits <boskovits@gmail.com>:

> Can you please send us an update on your project?
>

[-- Attachment #2: Type: text/html, Size: 1388 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hi Tatiana,

Tatiana Sholokhova <tanja201396@gmail.com> skribis:

> On the last week, I had fallen out of the process. I had been having exams
> at the university since the beginning of June. The last exam was
> rescheduled and that has affected my plans. I am sorry for that. Now the
> semester is finished and I am having much more time to focus on our
> project. I am very happy to pass the first evaluation. It is a pleasure for
> me to work with such a great team!

Thank you, and thanks for letting us know.

> In a few days, I am going to implement the whitelist for the static files
> and improve pagination tool performance as we discussed before. Also, I
> intend to make the pagination tool universal and add it to the page which
> displays all builds of a selected evaluation.

Sounds cool!  I haven’t followed your work as closely as I was hoping
for, but it seems like we could merge it quickly, notably because
everyone wants these enhancements.  :-)

> Do you think it would be okay if I add auxiliary pagination filters to
> the request which retrieves builds from the database?

Why not.  Depending on the filter, We may need to add indexes to the
database (see the bottom of ‘schema.sql’) so that requests aren’t too
slow.

Thank you for the update!

Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 2505 bytes --]

Hi all,

I just committed the code I wrote trying to improve pagination. I screwed
up a bit with the new pagination.
The problem I encountered is following. If we want to maintain a link to
the previous page we have to filter the database table entries with to
types of filters: one with lower bound on the id, and the other with the
upper bound. As we do so simultaneously with limiting the query output to
the maximal page size, we need to change the ordering type depending on the
type of the filter. At the moment I am not sure, what it the best way to
implement database query with such properties. Could you please take a look
on the commit and share your ideas on that?

The current implementation of pagination works correctly but it does not
support link to the previous page (first, and next only).

I have been trying to improve pagination for a while, and I now am thinking
about starting the parallel work on the implementation of the features we
listed before. What do you think about it?

Best regards,
Tatiana

ср, 27 июн. 2018 г. в 21:56, Ludovic Courtès <ludo@gnu.org>:

> Hi Tatiana,
>
> Tatiana Sholokhova <tanja201396@gmail.com> skribis:
>
> > On the last week, I had fallen out of the process. I had been having
> exams
> > at the university since the beginning of June. The last exam was
> > rescheduled and that has affected my plans. I am sorry for that. Now the
> > semester is finished and I am having much more time to focus on our
> > project. I am very happy to pass the first evaluation. It is a pleasure
> for
> > me to work with such a great team!
>
> Thank you, and thanks for letting us know.
>
> > In a few days, I am going to implement the whitelist for the static files
> > and improve pagination tool performance as we discussed before. Also, I
> > intend to make the pagination tool universal and add it to the page which
> > displays all builds of a selected evaluation.
>
> Sounds cool!  I haven’t followed your work as closely as I was hoping
> for, but it seems like we could merge it quickly, notably because
> everyone wants these enhancements.  :-)
>
> > Do you think it would be okay if I add auxiliary pagination filters to
> > the request which retrieves builds from the database?
>
> Why not.  Depending on the filter, We may need to add indexes to the
> database (see the bottom of ‘schema.sql’) so that requests aren’t too
> slow.
>
> Thank you for the update!
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 3011 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 2237 bytes --]

2018-07-04 22:54 GMT+02:00 Tatiana Sholokhova <tanja201396@gmail.com>:

> Hi all,
>
>
Hi Tatiana,


> I just committed the code I wrote trying to improve pagination. I screwed
> up a bit with the new pagination.
> The problem I encountered is following. If we want to maintain a link to
> the previous page we have to filter the database table entries with to
> types of filters: one with lower bound on the id, and the other with the
> upper bound. As we do so simultaneously with limiting the query output to
> the maximal page size, we need to change the ordering type depending on the
> type of the filter. At the moment I am not sure, what it the best way to
> implement database query with such properties. Could you please take a look
> on the commit and share your ideas on that?
>

> The current implementation of pagination works correctly but it does not
> support link to the previous page (first, and next only).
>

It has been some time since I last had a look at databases, so you have my
apologies
in advance if what I am saying does not really apply, or is even not
entirely correct.

You could perhaps have a look at reversing the sort order, and replace ">"
with "<" and "<"
with "<" in your WHERE clauses. The query to the previous page would be
similar to
retrieving the next page, but basically reversing the order you would page
through the results,
if that makes any sense.

If this works, you could also hack together a maybe-efficient query to
retrieve the items
for the last page; simply insert the maximum possible value in your query,
and retrieve the
previous page with regards to that maximum value. In the current case, you
could enter the
highest possible value any id can have.

If it is possible for new items to show up on previous pages as well (so
with id's that are lower
than the highest id), you would need to replace your sorting and filtering
to act on a composite
value of e.g. <id, timestamp>, instead of on only the id value.


>
> I have been trying to improve pagination for a while, and I now am
> thinking about starting the parallel work on the implementation of the
> features we listed before. What do you think about it?
>

> Best regards,
> Tatiana
>
> Good luck, and HTH!

- Jelle

[-- Attachment #2: Type: text/html, Size: 3531 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

Hi Tatiana,

On Wed, 4 Jul 2018 22:54:46 +0200
Tatiana Sholokhova <tanja201396@gmail.com> wrote:

> If we want to maintain a link to
> the previous page we have to filter the database table entries with to
> types of filters: one with lower bound on the id, and the other with the
> upper bound. 

Yeah, I know what you mean.

I'd suggest one of the following alternatives for implementing "Previous page":

(1) Remember all the page boundaries in the query string (or maybe hidden
form elements).  So instead of finding out where the beginning of the
previous page was all over again just remember it from before.
(2) Reverse the ordering in the query and the boundary check and run
the query, reverse the result of the query.  Handle finished result as before.
(3) Just use the browser's back button.  In fact, you can just put a
"Previous" link that presses the back button via Javascript for the time being.

I suggest (3) - and implement one of the others later.

> The current implementation of pagination works correctly but it does not
> support link to the previous page (first, and next only).

Cool!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1628 bytes --]

Danny Milosavljevic <dannym@scratchpost.org> ezt írta (időpont: 2018. júl.
5., Cs 10:28):

> Hi Tatiana,
>
> On Wed, 4 Jul 2018 22:54:46 +0200
> Tatiana Sholokhova <tanja201396@gmail.com> wrote:
>
> > If we want to maintain a link to
> > the previous page we have to filter the database table entries with to
> > types of filters: one with lower bound on the id, and the other with the
> > upper bound.
>
> Yeah, I know what you mean.
>
> I'd suggest one of the following alternatives for implementing "Previous
> page":
>
> (1) Remember all the page boundaries in the query string (or maybe hidden
> form elements).  So instead of finding out where the beginning of the
> previous page was all over again just remember it from before.
> (2) Reverse the ordering in the query and the boundary check and run
> the query, reverse the result of the query.  Handle finished result as
> before.
> (3) Just use the browser's back button.  In fact, you can just put a
> "Previous" link that presses the back button via Javascript for the time
> being.
>
> I suggest (3) - and implement one of the others later.
>
> I agree that going with (3) now makes sense. The most flexible solution is
(2). I usually do that, as it does not rely on having the earlier pages
seen. I usually abstract this away in a cursor, which has the first, the
last, the current page first and last ids, and a gotofirst, gotolast, next
and previous operators.WDYT?

> > The current implementation of pagination works correctly but it does not
> > support link to the previous page (first, and next only).
>
> Cool!
>

[-- Attachment #2: Type: text/html, Size: 2249 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

 Hello all!

Thank you for your helpful comments and ideas!

I've committed an improved version of the pagination. As you advised I
chose and implemented (2) variant. I alter sorting order in SQL query
depending on the type of the current page border. So, now all
operators (gotofirst, gotolast, next and previous) are working.

Also, I added pagination for builds page ("eval" query). Here I face a
problem that Denny mentioned before.

> The tuple of data cells should uniquely identify one row in the result.
> (If it
> didn't, you'd skip the other same-value rows when going to the next page)

I order builds by stoptime and there are some builds with identical
stoptime timestamps. I am not sure what is the best way to support
pagination filtering by multiple columns. Do you have ideas on how to
implement tuple comparison and other routines in SQL and guile in a
convenient and flexible way?

Could you please review the last 3 commits and maybe find some more issues
besides that?

Best regards,
Tatiana

[-- Attachment #2: Type: text/html, Size: 1359 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Hi Tatiana,

On Sun, 8 Jul 2018 21:48:32 +0200
Tatiana Sholokhova <tanja201396@gmail.com> wrote:

> Do you have ideas on how to
> implement tuple comparison and other routines in SQL and guile in a
> convenient and flexible way?

sqlite3 supports row values, so the comparison can be
written like this:

  select * from foo where (a,b,c) = (2,'foo',3);

It even supports NULLs for wildcards, though it's a little more complicated:

  select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;

The sqlite C interface doesn't support parameter bindings for the entire
row, though, so you'd have to specify 3 values.

This works:

  (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" "," 3 ");")

but this doesn't work, unfortunately:

  (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")

See also https://www.sqlite.org/rowvalue.html

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

Tatiana Sholokhova <tanja201396@gmail.com> ezt írta (időpont: 2018. júl.
8., V, 21:48):

>  Hello all!
>
> Thank you for your helpful comments and ideas!
>
> I've committed an improved version of the pagination. As you advised I
> chose and implemented (2) variant. I alter sorting order in SQL query
> depending on the type of the current page border. So, now all
> operators (gotofirst, gotolast, next and previous) are working.
>
> Also, I added pagination for builds page ("eval" query). Here I face a
> problem that Denny mentioned before.
>
>> The tuple of data cells should uniquely identify one row in the result.
>> (If it
>> didn't, you'd skip the other same-value rows when going to the next page)
>
> I order builds by stoptime and there are some builds with identical
> stoptime timestamps. I am not sure what is the best way to support
> pagination filtering by multiple columns. Do you have ideas on how to
> implement tuple comparison and other routines in SQL and guile in a
> convenient and flexible way?
>
>
Please have a look at this. https://www.sqlite.org/rowvalue.html. I did not
check the sqlite version.


> Could you please review the last 3 commits and maybe find some more issues
> besides that?
>
> Best regards,
> Tatiana
>
>

[-- Attachment #2: Type: text/html, Size: 2069 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Ricardo Wurmus <rekado@elephly.net> writes:

> You can download a copy of the Cuirass database as it is used on
> berlin.guixsd.org, one of the build farms of the Guix project.  I have
> copied it here:
>
>     http://bootstrappable.org/cuirass.db
>
> It is 12G(!), which indicates that Cuirass adds way too many entries
> than absolutely needed.  Ludovic wrote on IRC that we don’t seem to
> check if a record already exists when two subsequent evaluations yield
> the same build.

Adding bug-guix@gnu.org to keep track of that bug.

bug#32190: Cuirass doesn't check if two subsequent jobs yield the same derivation

[Clément Lassieur]

Consider the following table:

--8<---------------cut here---------------start------------->8---
CREATE TABLE Derivations (
  derivation    TEXT NOT NULL,
  evaluation    INTEGER NOT NULL,
  job_name      TEXT NOT NULL,
  system        TEXT NOT NULL,
  nix_name      TEXT NOT NULL,
  PRIMARY KEY (derivation, evaluation),
  FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
);
--8<---------------cut here---------------end--------------->8---

And the following code:

--8<---------------cut here---------------start------------->8---
(define (db-add-derivation db job)
  "Store a derivation result in database DB and return its ID."
  (catch 'sqlite-error
    (lambda ()
      (sqlite-exec db "\
INSERT INTO Derivations (derivation, job_name, system, nix_name, evaluation)\
  VALUES ("
                   (assq-ref job #:derivation) ", "
                   (assq-ref job #:job-name) ", "
                   (assq-ref job #:system) ", "
                   (assq-ref job #:nix-name) ", "
                   (assq-ref job #:eval-id) ");")
      (last-insert-rowid db))
    (lambda (key who code message . rest)
      ;; If we get a unique-constraint-failed error, that means we have
      ;; already inserted the same (derivation,eval-id) tuple.  That happens
      ;; when several jobs produce the same derivation, and we can ignore it.
      (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
          (sqlite-exec db "SELECT * FROM Derivations WHERE derivation="
                       (assq-ref job #:derivation) ";")
          (apply throw key who code rest)))))
--8<---------------cut here---------------end--------------->8---

I think the above constraint can't happen because by definition a new
job (for the same job_name) is produced at each evaluation.  So eval-id
will be incremented every time.

Also, the docs (and a comment in schema.sql) says:

    Builds are not in a one to one relationship with derivations in
    order to keep track of non deterministic compilations.

But I think it doesn't make sense, because Guix won't try to build twice
the same thing unless '--check' is used (which obviously isn't the
case).

So not only we have a huge Derivations table full of identical items,
but we also ask Guix to build them and we store the results in the
Builds table...

Maybe the solution is to replace the (derivation, evaluation) primary
key with (derivation), and only build the newly added derivations.
WDYT?

Clément

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Dear all,

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Tatiana & all,
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>>> I am a bit confused about the database structure. As far as I understand,
>>> there are project_name (project) and branch_name (jobset) properties, but
>>> project_name is a primary key, so a project can't have several branches?
>>
>> I share your confusion.  Maybe Ludovic or Mathieu can shed some more
>> light on this.
>
> It’s confusing indeed, I think it’s a mistake that has yet to be fixed.
> Basically what we do now is that we use a different ‘repo_name’ when we
> just want to add a branch…

The notion of "project" has been removed[1].  It was previously the
specification name, which is a primary key indeed, so it didn't make
sense because one project couldn't have several branches.

Now, Hydra's jobsets are the exact same thing as Cuirass'
specifications.  So if you want to build the "master" and "core-update"
branches of Guix, you need two specifications.

However it wasn't even possible, before, to build several branches,
because specifications names were used by the evaluator: they had to be
"guix" or "guix-modular".  Since the name was a primary key, we could
only have two specifications.  It is now[2] possible, because the
evaluator uses the input name instead of the specification name.

If you think there is a need for the notion of "Project" in Cuirass, we
could add it, but it needs to be a new SQL table.  And each
specification would be associated with one project.

Clément

[1]: https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=be713f8a30788861806a74865b07403aa6774117
[2]: https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=7b2f9e0de1ad2d320973b7aea132a8afcad8bece

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Hi Tatiana,

Tatiana Sholokhova <tanja201396@gmail.com> writes:

> Am I right that in terms of Cuirass database derivations correspond to
> jobs?

Yes, but to be more precise, a job is a structure containing:
  - derivation
  - job-name
  - system
  - nix-name
  - eval-id

The database table called "Derivations" should be called "Jobs", so the
name is confusing indeed.

A derivation, as Ricardo explained, is a file (.drv) representing
low-level build actions and the environment in which they are performed.

At each evaluation, there is a new set of jobs returned by the
evaluator, each job having its 'eval-id' incremented.  That means that
two different jobs for the same job-name (i.e. linux-libre-4.17.6-job)
could embed the same derivation.  In that case, it's useless to build
that job in my opinion, see that bug[1].

I hope it's clearer,
Clément

[1]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32190

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Hi Tatiana,

Tatiana Sholokhova <tanja201396@gmail.com> writes:

> Could you please review the last 3 commits and maybe find some more issues
> besides that?

I've integrated your work onto my Cuirass instance[1], and I really like
it!  I had to fix a few things and adapt it[2] so that it works with
multiple inputs.

I will do a review as soon as possible, and then we can merge it.  I'm a
bit late: going through the whole conversation history took more time
than I expected.

Clément

[1]: https://cuirass.lassieur.org:8081/
[2]: https://git.lassieur.org/cgit/cuirass.git/

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

Hi Clément,

Thank you for the clarifications on the job structure!

I have read your changes to the web interface and everything looks good to
me. Also, it works nicely on your server. So, let's prepare for the merge.
Let me know if you want me to make some fixes before the merge. In the
meantime, I continue working on the next features for the interface locally
and then integrate them into the updated codebase.

Best regards,
Tatiana

ср, 18 июл. 2018 г. в 12:37, Clément Lassieur <clement@lassieur.org>:

> Hi Tatiana,
>
> Tatiana Sholokhova <tanja201396@gmail.com> writes:
>
> > Could you please review the last 3 commits and maybe find some more
> issues
> > besides that?
>
> I've integrated your work onto my Cuirass instance[1], and I really like
> it!  I had to fix a few things and adapt it[2] so that it works with
> multiple inputs.
>
> I will do a review as soon as possible, and then we can merge it.  I'm a
> bit late: going through the whole conversation history took more time
> than I expected.
>
> Clément
>
> [1]: https://cuirass.lassieur.org:8081/
> [2]: https://git.lassieur.org/cgit/cuirass.git/
>

[-- Attachment #2: Type: text/html, Size: 1868 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

Fwiw, you can use bootstrap without js.

Causal reply,

Amirouche

Le jeu. 19 juil. 2018 à 22:11, Tatiana Sholokhova <tanja201396@gmail.com> a
écrit :

> Hi Clément,
>
> Thank you for the clarifications on the job structure!
>
> I have read your changes to the web interface and everything looks good to
> me. Also, it works nicely on your server. So, let's prepare for the merge.
> Let me know if you want me to make some fixes before the merge. In the
> meantime, I continue working on the next features for the interface locally
> and then integrate them into the updated codebase.
>
> Best regards,
> Tatiana
>
> ср, 18 июл. 2018 г. в 12:37, Clément Lassieur <clement@lassieur.org>:
>
>> Hi Tatiana,
>>
>> Tatiana Sholokhova <tanja201396@gmail.com> writes:
>>
>> > Could you please review the last 3 commits and maybe find some more
>> issues
>> > besides that?
>>
>> I've integrated your work onto my Cuirass instance[1], and I really like
>> it!  I had to fix a few things and adapt it[2] so that it works with
>> multiple inputs.
>>
>> I will do a review as soon as possible, and then we can merge it.  I'm a
>> bit late: going through the whole conversation history took more time
>> than I expected.
>>
>> Clément
>>
>> [1]: https://cuirass.lassieur.org:8081/
>> [2]: https://git.lassieur.org/cgit/cuirass.git/
>>
>

[-- Attachment #2: Type: text/html, Size: 2467 bytes --]

bug#32190: Cuirass doesn't check if two subsequent jobs yield the same derivation

[Ludovic Courtès]

Hi Clément,

Clément Lassieur <clement@lassieur.org> skribis:

> Consider the following table:
>
> CREATE TABLE Derivations (
>   derivation    TEXT NOT NULL,
>   evaluation    INTEGER NOT NULL,
>   job_name      TEXT NOT NULL,
>   system        TEXT NOT NULL,
>   nix_name      TEXT NOT NULL,
>   PRIMARY KEY (derivation, evaluation),
>   FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
> );
>
>
> And the following code:
>
> (define (db-add-derivation db job)
>   "Store a derivation result in database DB and return its ID."
>   (catch 'sqlite-error
>     (lambda ()
>       (sqlite-exec db "\
> INSERT INTO Derivations (derivation, job_name, system, nix_name, evaluation)\
>   VALUES ("
>                    (assq-ref job #:derivation) ", "
>                    (assq-ref job #:job-name) ", "
>                    (assq-ref job #:system) ", "
>                    (assq-ref job #:nix-name) ", "
>                    (assq-ref job #:eval-id) ");")
>       (last-insert-rowid db))
>     (lambda (key who code message . rest)
>       ;; If we get a unique-constraint-failed error, that means we have
>       ;; already inserted the same (derivation,eval-id) tuple.  That happens
>       ;; when several jobs produce the same derivation, and we can ignore it.
>       (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
>           (sqlite-exec db "SELECT * FROM Derivations WHERE derivation="
>                        (assq-ref job #:derivation) ";")
>           (apply throw key who code rest)))))
>
> I think the above constraint can't happen because by definition a new
> job (for the same job_name) is produced at each evaluation.  So eval-id
> will be incremented every time.

I added it at the time because it did happen.  In a given eval, there
can be two jobs producing the same derivation (for instance a job called
“gcc” produces xyz-gcc-5.5.0.drv, and a job called “gcc-5.5.0” produces
the very same xyz-gcc-5.5.0.drv.)

> Also, the docs (and a comment in schema.sql) says:
>
>     Builds are not in a one to one relationship with derivations in
>     order to keep track of non deterministic compilations.
>
> But I think it doesn't make sense, because Guix won't try to build twice
> the same thing unless '--check' is used (which obviously isn't the
> case).

The rationale (that was back in Mathieu’s GSoC) was that sometimes, you
can have several builds logs for one derivation.  In Hydra this happens
if a build fails for some non-deterministic reason and then you click on
“Restart” in the hope that it’ll succeed this time.  ;-)  In this
situation Hydra keeps both build logs IIRC.

Anyway, I lean towards keeping only one build log, at least for now,
which is what guix-daemon does.

> So not only we have a huge Derivations table full of identical items,
> but we also ask Guix to build them and we store the results in the
> Builds table...
>
> Maybe the solution is to replace the (derivation, evaluation) primary
> key with (derivation), and only build the newly added derivations.
> WDYT?

I agree, we don’t need all these identical items, it makes no sense.

You can go ahead and clean that up!  ;-)

Thank you,
Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi Tatiana,
>
> On Sun, 8 Jul 2018 21:48:32 +0200
> Tatiana Sholokhova <tanja201396@gmail.com> wrote:
>
>> Do you have ideas on how to
>> implement tuple comparison and other routines in SQL and guile in a
>> convenient and flexible way?
>
> sqlite3 supports row values, so the comparison can be
> written like this:
>
>   select * from foo where (a,b,c) = (2,'foo',3);
>
> It even supports NULLs for wildcards, though it's a little more complicated:
>
>   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
>
> The sqlite C interface doesn't support parameter bindings for the entire
> row, though, so you'd have to specify 3 values.
>
> This works:
>
>   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" "," 3 ");")
>
> but this doesn't work, unfortunately:
>
>   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
>
> See also https://www.sqlite.org/rowvalue.html

With the '<' operator, it doesn't give the results we are looking for, I
think.

For example:

select (0,1) < (1,0); -- returns 1
select (0,0) < (0,1); -- returns 1

In both cases, we'd want it to return 0.

I think we should use:

select (0 < 1) and (1 < 0); -- returns 0
select (0 < 0) and (0 < 1); -- returns 0

instead, for the pagination borders code.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]

P

Clément Lassieur <clement@lassieur.org> ezt írta (időpont: 2018. júl. 29.,
V 14:01):

> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
> > Hi Tatiana,
> >
> > On Sun, 8 Jul 2018 21:48:32 +0200
> > Tatiana Sholokhova <tanja201396@gmail.com> wrote:
> >
> >> Do you have ideas on how to
> >> implement tuple comparison and other routines in SQL and guile in a
> >> convenient and flexible way?
> >
> > sqlite3 supports row values, so the comparison can be
> > written like this:
> >
> >   select * from foo where (a,b,c) = (2,'foo',3);
> >
> > It even supports NULLs for wildcards, though it's a little more
> complicated:
> >
> >   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
> >
> > The sqlite C interface doesn't support parameter bindings for the entire
> > row, though, so you'd have to specify 3 values.
> >
> > This works:
> >
> >   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" ","
> 3 ");")
> >
> > but this doesn't work, unfortunately:
> >
> >   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
> >
> > See also https://www.sqlite.org/rowvalue.html
>
> With the '<' operator, it doesn't give the results we are looking for, I
> think.
>
> For example:
>
> select (0,1) < (1,0); -- returns 1
> select (0,0) < (0,1); -- returns 1
>

This is working as expected. Actually this:
(a,b)<(c,d) is a shortcut for a<c or (a=c and b<d).

In both cases, we'd want it to return 0.
>

How do we use it? Why this is the expected result?


> I think we should use:
>
> select (0 < 1) and (1 < 0); -- returns 0
> select (0 < 0) and (0 < 1); -- returns 0
>

Could you please clarify which numbers are the placeholders for which
quantities?

>
> instead, for the pagination borders code.
>

[-- Attachment #2: Type: text/html, Size: 3392 bytes --]

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Gábor Boskovits <boskovits@gmail.com> writes:

> P
>
> Clément Lassieur <clement@lassieur.org> ezt írta (időpont: 2018. júl. 29.,
> V 14:01):
>
>> Danny Milosavljevic <dannym@scratchpost.org> writes:
>>
>> > Hi Tatiana,
>> >
>> > On Sun, 8 Jul 2018 21:48:32 +0200
>> > Tatiana Sholokhova <tanja201396@gmail.com> wrote:
>> >
>> >> Do you have ideas on how to
>> >> implement tuple comparison and other routines in SQL and guile in a
>> >> convenient and flexible way?
>> >
>> > sqlite3 supports row values, so the comparison can be
>> > written like this:
>> >
>> >   select * from foo where (a,b,c) = (2,'foo',3);
>> >
>> > It even supports NULLs for wildcards, though it's a little more
>> complicated:
>> >
>> >   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
>> >
>> > The sqlite C interface doesn't support parameter bindings for the entire
>> > row, though, so you'd have to specify 3 values.
>> >
>> > This works:
>> >
>> >   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" ","
>> 3 ");")
>> >
>> > but this doesn't work, unfortunately:
>> >
>> >   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
>> >
>> > See also https://www.sqlite.org/rowvalue.html
>>
>> With the '<' operator, it doesn't give the results we are looking for, I
>> think.
>>
>> For example:
>>
>> select (0,1) < (1,0); -- returns 1
>> select (0,0) < (0,1); -- returns 1
>>
>
> This is working as expected. Actually this:
> (a,b)<(c,d) is a shortcut for a<c or (a=c and b<d).
>
> In both cases, we'd want it to return 0.
>>
>
> How do we use it? Why this is the expected result?
>
>
>> I think we should use:
>>
>> select (0 < 1) and (1 < 0); -- returns 0
>> select (0 < 0) and (0 < 1); -- returns 0
>>
>
> Could you please clarify which numbers are the placeholders for which
> quantities?
>
>>
>> instead, for the pagination borders code.

Actually, forget what I said, I was wrong ;-)

bug#32190: [PATCH] database: Merge Derivations into Builds table.

[Clément Lassieur]

Fixes <https://bugs.gnu.org/32190>.

* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-2.sql'.
* src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table.
* src/cuirass/database.scm (db-add-derivation, db-get-derivation): Remove
exported procedures.
(db-add-build): Catch SQLITE_CONSTRAINT_PRIMARYKEY error, which means that two
jobs produced the same derivation, and return #f in that case.  Add columns
that were in the Derivations table before.  Use 'derivation' as primary key
for the Outputs table.
(db-get-outputs): Use 'derivation' as identifier, instead of 'build-id'.
(filters->order): Replace 'id' with 'rowid'.
(db-get-builds): Add a 'derivation' filter.  Replace 'id' with 'rowid'.
Remove the 'INNER JOIN Derivations'.  Replace Derivations with Builds.  Return
'derivation' in first position to make it clear that it's the primary key.
Pass DERIVATION instead of ID to DB-GET-OUTPUTS.
(db-get-build): Allow to take a derivation as argument.  Use NUMBER? to
differentiate between derivations and ids.
(db-get-pending-derivations): Remove the 'SELECT DISTINCT' clause now that
derivations are unique.  Remove the 'INNER JOIN Builds'.
(db-get-evaluations-build-summary, db-get-builds-min, db-get-builds-max):
Replace 'id' with 'rowid'.
* src/schema.sql (Derivations): Remove table.
(Outputs): Replace Builds.id with Builds.derivation.
(Builds): Use 'derivation' as primary key.  Remove the 'id' column.  Add
'job_name', 'system', 'nix_name' columns that were in the Derivations table
before.
(Builds_Derivations_index): Rename to Builds_index.  Update accordingly.
(Derivations_index): Remove index.
* src/sql/upgrade-2.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-job, make-dummy-derivation): Remove
procedures.
(make-dummy-build): Add columns that were in MAKE-DUMMY-DERIVATION.  Get the
DRV parameter to be mandatory because it's a primary key.
(%id): Remove parameter.
(db-add-derivation, db-get-derivation): Remove tests.
(db-add-build): Expect #f, because it adds twice the same derivation.  Pass
the derivation argument to MAKE-DUMMY-BUILD.
(db-update-build-status!): Rename 'id' to 'derivation'.  Pass the derivation
argument to MAKE-DUMMY-BUILD.  Remove the DB-ADD-DERIVATION call.
(db-get-builds, db-get-pending-derivations): Pass the derivation argument to
MAKE-DUMMY-BUILD.  Remove the DB-ADD-DERIVATION calls.
* tests/http.scm (fill-db): Remove DERIVATION1 and DERIVATION2, and put
their content in BUILD1 and BUILD2.  Remove the DB-ADD-DERIVATION calls.
---
 Makefile.am              |   3 +-
 src/cuirass/base.scm     |  23 +++---
 src/cuirass/database.scm | 174 +++++++++++++++++----------------------
 src/schema.sql           |  28 ++-----
 src/sql/upgrade-2.sql    |  49 +++++++++++
 tests/database.scm       |  78 ++++++------------
 tests/http.scm           |  20 ++---
 7 files changed, 176 insertions(+), 199 deletions(-)
 create mode 100644 src/sql/upgrade-2.sql

diff --git a/Makefile.am b/Makefile.am
index ac22601..db56165 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -65,7 +65,8 @@ nodist_webobject_DATA =				\
 dist_pkgdata_DATA = src/schema.sql
 
 dist_sql_DATA = 				\
-  src/sql/upgrade-1.sql
+  src/sql/upgrade-1.sql				\
+  src/sql/upgrade-2.sql
 
 dist_css_DATA =					\
   src/static/css/bootstrap.css			\
diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm
index 82f49a4..26a5996 100644
--- a/src/cuirass/base.scm
+++ b/src/cuirass/base.scm
@@ -280,11 +280,9 @@ Return a list of jobs."
                         db `((#:specification . ,spec-name)
                              (#:commits . ,commits)))))
          (log-message "created evaluation ~a for '~a'" eval-id spec-name)
-         (let ((jobs (map (lambda (job)
-                            (augment-job job eval-id))
-                          jobs)))
-           (for-each (cut db-add-derivation db <>) jobs)
-           jobs))))))
+         (map (lambda (job)
+                (augment-job job eval-id))
+              jobs))))))
 
 \f
 ;;;
@@ -546,6 +544,9 @@ procedure is meant to be called at startup."
     (let* ((name     (assq-ref job #:job-name))
            (drv      (assq-ref job #:derivation))
            (eval-id  (assq-ref job #:eval-id))
+           (job-name (assq-ref job #:job-name))
+           (system   (assq-ref job #:system))
+           (nix-name (assq-ref job #:nix-name))
            ;; XXX: How to keep logs from several attempts?
            (log      (log-file store drv))
            (outputs  (filter-map (lambda (res)
@@ -556,6 +557,9 @@ procedure is meant to be called at startup."
            (cur-time (time-second (current-time time-utc))))
       (let ((build `((#:derivation . ,drv)
                      (#:eval-id . ,eval-id)
+                     (#:job-name . ,job-name)
+                     (#:system . ,system)
+                     (#:nix-name . ,nix-name)
 
                      ;; XXX: We'd leave LOG to #f (i.e., NULL) but that
                      ;; currently violates the non-NULL constraint.
@@ -568,13 +572,12 @@ procedure is meant to be called at startup."
                      (#:stoptime . 0))))
         (db-add-build db build))))
 
-  (define build-ids
-    (map register jobs))
+  (define derivations
+    (filter-map register jobs))
 
-  (spawn-builds store db
-                (map (cut assq-ref <> #:derivation) jobs))
+  (spawn-builds store db derivations)
 
-  (let* ((results (filter-map (cut db-get-build db <>) build-ids))
+  (let* ((results (filter-map (cut db-get-build db <>) derivations))
          (status (map (cut assq-ref <> #:status) results))
          (success (count (lambda (status)
                            (= status (build-status succeeded)))
diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm
index b4b1652..7788ac9 100644
--- a/src/cuirass/database.scm
+++ b/src/cuirass/database.scm
@@ -41,8 +41,6 @@
             db-add-stamp
             db-get-stamp
             db-add-evaluation
-            db-add-derivation
-            db-get-derivation
             db-get-pending-derivations
             build-status
             db-add-build
@@ -312,32 +310,6 @@ package_path_inputs, proc_input, proc_file, proc, proc_args) \
                      (#:inputs . ,(db-get-inputs db name)))
                    specs))))))
 
-(define (db-add-derivation db job)
-  "Store a derivation result in database DB and return its ID."
-  (catch 'sqlite-error
-    (lambda ()
-      (sqlite-exec db "\
-INSERT INTO Derivations (derivation, job_name, system, nix_name, evaluation)\
-  VALUES ("
-                   (assq-ref job #:derivation) ", "
-                   (assq-ref job #:job-name) ", "
-                   (assq-ref job #:system) ", "
-                   (assq-ref job #:nix-name) ", "
-                   (assq-ref job #:eval-id) ");")
-      (last-insert-rowid db))
-    (lambda (key who code message . rest)
-      ;; If we get a unique-constraint-failed error, that means we have
-      ;; already inserted the same (derivation,eval-id) tuple.  That happens
-      ;; when several jobs produce the same derivation, and we can ignore it.
-      (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
-          (sqlite-exec db "SELECT * FROM Derivations WHERE derivation="
-                       (assq-ref job #:derivation) ";")
-          (apply throw key who code rest)))))
-
-(define (db-get-derivation db id)
-  "Retrieve a job in database DB which corresponds to ID."
-  (car (sqlite-exec db "SELECT * FROM Derivations WHERE derivation=" id ";")))
-
 (define (db-add-evaluation db eval)
   (sqlite-exec db "\
 INSERT INTO Evaluations (specification, commits) VALUES ("
@@ -384,27 +356,39 @@ string."
 (define (db-add-build db build)
   "Store BUILD in database DB. BUILD eventual outputs are stored
 in the OUTPUTS table."
-  (let* ((build-exec
-          (sqlite-exec db "\
-INSERT INTO Builds (derivation, evaluation, log, status, timestamp, starttime, stoptime)\
-  VALUES ("
-                       (assq-ref build #:derivation) ", "
-                       (assq-ref build #:eval-id) ", "
-                       (assq-ref build #:log) ", "
-                       (or (assq-ref build #:status)
-                           (build-status scheduled)) ", "
-                       (or (assq-ref build #:timestamp) 0) ", "
-                       (or (assq-ref build #:starttime) 0) ", "
-                       (or (assq-ref build #:stoptime) 0) ");"))
-         (build-id (last-insert-rowid db)))
-    (for-each (lambda (output)
-                (match output
-                  ((name . path)
-                   (sqlite-exec db "\
-INSERT INTO Outputs (build, name, path) VALUES ("
-                                build-id ", " name ", " path ");"))))
-              (assq-ref build #:outputs))
-    build-id))
+  (catch 'sqlite-error
+    (lambda ()
+      (sqlite-exec db "
+INSERT INTO Builds (derivation, evaluation, job_name, system, nix_name, log,
+status, timestamp, starttime, stoptime)
+VALUES ("
+                   (assq-ref build #:derivation) ", "
+                   (assq-ref build #:eval-id) ", "
+                   (assq-ref build #:job-name) ", "
+                   (assq-ref build #:system) ", "
+                   (assq-ref build #:nix-name) ", "
+                   (assq-ref build #:log) ", "
+                   (or (assq-ref build #:status)
+                       (build-status scheduled)) ", "
+                   (or (assq-ref build #:timestamp) 0) ", "
+                   (or (assq-ref build #:starttime) 0) ", "
+                   (or (assq-ref build #:stoptime) 0) ");")
+      (let ((derivation (assq-ref build #:derivation)))
+        (for-each (lambda (output)
+                    (match output
+                      ((name . path)
+                       (sqlite-exec db "\
+INSERT INTO Outputs (derivation, name, path) VALUES ("
+                                    derivation ", " name ", " path ");"))))
+                  (assq-ref build #:outputs))
+        derivation))
+    (lambda (key who code message . rest)
+      ;; If we get a unique-constraint-failed error, that means we have
+      ;; already inserted the same build.  That happens when several jobs
+      ;; produce the same derivation, and we can ignore it.
+      (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
+          #f
+          (apply throw key who code rest)))))
 
 (define* (db-update-build-status! db drv status #:key log-file)
   "Update DB so that DRV's status is STATUS.  This also updates the
@@ -429,11 +413,11 @@ log file for DRV."
                        ", status=" status
                        "WHERE derivation=" drv " AND status != " status ";"))))
 
-(define (db-get-outputs db build-id)
-  "Retrieve the OUTPUTS of the build identified by BUILD-ID in DB database."
+(define (db-get-outputs db derivation)
+  "Retrieve the OUTPUTS of the build identified by DERIVATION in DB database."
   (let loop ((rows
-              (sqlite-exec db "SELECT name, path FROM Outputs WHERE build="
-                           build-id ";"))
+              (sqlite-exec db "SELECT name, path FROM Outputs
+WHERE derivation =" derivation ";"))
              (outputs '()))
     (match rows
       (() outputs)
@@ -445,56 +429,56 @@ log file for DRV."
 
 (define (filters->order filters)
   (match (assq 'order filters)
-    (('order . 'build-id) "id ASC")
-    (('order . 'decreasing-build-id) "id DESC")
+    (('order . 'build-id) "rowid ASC")
+    (('order . 'decreasing-build-id) "rowid DESC")
     (('order . 'finish-time) "stoptime DESC")
-    (('order . 'finish-time+build-id) "stoptime DESC, id DESC")
+    (('order . 'finish-time+build-id) "stoptime DESC, rowid DESC")
     (('order . 'start-time) "starttime DESC")
     (('order . 'submission-time) "timestamp DESC")
     ;; With this order, builds in 'running' state (-1) appear
     ;; before those in 'scheduled' state (-2).
     (('order . 'status+submission-time) "status DESC, timestamp DESC")
-    (_ "id DESC")))
+    (_ "rowid DESC")))
 
 (define (db-get-builds db filters)
   "Retrieve all builds in database DB which are matched by given FILTERS.
-FILTERS is an assoc list whose possible keys are 'id | 'jobset | 'job |
-'system | 'nr | 'order | 'status | 'evaluation."
+FILTERS is an assoc list whose possible keys are 'derivation | 'id | 'jobset |
+'job | 'system | 'nr | 'order | 'status | 'evaluation."
   (let* ((order (filters->order filters))
          (stmt-text (format #f "SELECT * FROM (
-SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime,
-Builds.log, Builds.status, Builds.derivation, Derivations.job_name,
-Derivations.system, Derivations.nix_name, Specifications.name
+SELECT Builds.derivation, Builds.rowid, Builds.timestamp, Builds.starttime,
+Builds.stoptime, Builds.log, Builds.status, Builds.job_name, Builds.system,
+Builds.nix_name, Specifications.name
 FROM Builds
-INNER JOIN Derivations ON Builds.derivation = Derivations.derivation
-AND Builds.evaluation = Derivations.evaluation
-INNER JOIN Evaluations ON Derivations.evaluation = Evaluations.id
+INNER JOIN Evaluations ON Builds.evaluation = Evaluations.id
 INNER JOIN Specifications ON Evaluations.specification = Specifications.name
-WHERE (:id IS NULL OR (:id = Builds.id))
+WHERE (:id IS NULL OR (:id = Builds.rowid))
+AND (:derivation IS NULL OR (:derivation = Builds.derivation))
 AND (:jobset IS NULL OR (:jobset = Specifications.name))
-AND (:job IS NULL OR (:job = Derivations.job_name))
-AND (:system IS NULL OR (:system = Derivations.system))
+AND (:job IS NULL OR (:job = Builds.job_name))
+AND (:system IS NULL OR (:system = Builds.system))
 AND (:evaluation IS NULL OR (:evaluation = Builds.evaluation))
 AND (:status IS NULL OR (:status = 'done' AND Builds.status >= 0)
                      OR (:status = 'pending' AND Builds.status < 0))
 AND (:borderlowtime IS NULL OR :borderlowid IS NULL
- OR ((:borderlowtime, :borderlowid) < (Builds.stoptime, Builds.id)))
+ OR ((:borderlowtime, :borderlowid) < (Builds.stoptime, Builds.rowid)))
 AND (:borderhightime IS NULL OR :borderhighid IS NULL
- OR ((:borderhightime, :borderhighid) > (Builds.stoptime, Builds.id)))
+ OR ((:borderhightime, :borderhighid) > (Builds.stoptime, Builds.rowid)))
 ORDER BY
 CASE WHEN :borderlowtime IS NULL
        OR :borderlowid IS NULL THEN Builds.stoptime
                                ELSE -Builds.stoptime
 END DESC,
 CASE WHEN :borderlowtime IS NULL
-       OR :borderlowid IS NULL THEN Builds.id
-                               ELSE -Builds.id
+       OR :borderlowid IS NULL THEN Builds.rowid
+                               ELSE -Builds.rowid
 END DESC
 LIMIT :nr)
-ORDER BY ~a, id ASC;" order))
+ORDER BY ~a, rowid ASC;" order))
          (stmt (sqlite-prepare db stmt-text #:cache? #t)))
     (sqlite-bind-arguments
      stmt
+     #:derivation (assq-ref filters 'derivation)
      #:id (assq-ref filters 'id)
      #:jobset (assq-ref filters 'jobset)
      #:job (assq-ref filters 'job)
@@ -513,45 +497,37 @@ ORDER BY ~a, id ASC;" order))
                (builds '()))
       (match rows
         (() (reverse builds))
-        ((#(id timestamp starttime stoptime log status derivation job-name
-               system nix-name specification) . rest)
+        ((#(derivation id timestamp starttime stoptime log status job-name
+                       system nix-name specification) . rest)
          (loop rest
-               (cons `((#:id . ,id)
+               (cons `((#:derivation . ,derivation)
+                       (#:id . ,id)
                        (#:timestamp . ,timestamp)
                        (#:starttime . ,starttime)
                        (#:stoptime . ,stoptime)
                        (#:log . ,log)
                        (#:status . ,status)
-                       (#:derivation . ,derivation)
                        (#:job-name . ,job-name)
                        (#:system . ,system)
                        (#:nix-name . ,nix-name)
                        (#:specification . ,specification)
-                       (#:outputs . ,(db-get-outputs db id)))
+                       (#:outputs . ,(db-get-outputs db derivation)))
                      builds)))))))
 
-(define (db-get-build db id)
-  "Retrieve a build in database DB which corresponds to ID."
-  (match (db-get-builds db `((id . ,id)))
-    ((build)
-     build)
-    (() #f)))
+(define (db-get-build db derivation-or-id)
+  "Retrieve a build in database DB which corresponds to DERIVATION-OR-ID."
+  (let ((key (if (number? derivation-or-id) 'id 'derivation)))
+    (match (db-get-builds db `((,key . ,derivation-or-id)))
+      ((build)
+       build)
+      (() #f))))
 
 (define (db-get-pending-derivations db)
   "Return the list of derivation file names corresponding to pending builds in
 DB.  The returned list is guaranteed to not have any duplicates."
-  ;; This is of course much more efficient than calling 'delete-duplicates' on
-  ;; a list of results obtained without DISTINCT, both in space and time.
-  ;;
-  ;; Here we use a subquery so that sqlite can use two indexes instead of
-  ;; creating a "TEMP B-TREE" when doing a single flat query, as "EXPLAIN
-  ;; QUERY PLAN" shows.
   (map (match-lambda (#(drv) drv))
        (sqlite-exec db "
-SELECT DISTINCT derivation FROM (
-  SELECT Derivations.derivation FROM Derivations INNER JOIN Builds
-  WHERE Derivations.derivation = Builds.derivation AND Builds.status < 0
-);")))
+SELECT derivation FROM Builds WHERE Builds.status < 0;")))
 
 (define (db-get-stamp db spec)
   "Return a stamp corresponding to specification SPEC in database DB."
@@ -596,7 +572,7 @@ AND (" border-high "IS NULL OR (id <" border-high "))
 ORDER BY CASE WHEN " border-low "IS NULL THEN id ELSE -id END DESC
 LIMIT " limit ") E
 LEFT JOIN
-(SELECT id, evaluation, SUM(status=0) as succeeded,
+(SELECT rowid, evaluation, SUM(status=0) as succeeded,
 SUM(status>0) as failed, SUM(status<0) as scheduled
 FROM Builds
 GROUP BY evaluation) B
@@ -632,8 +608,8 @@ WHERE specification=" spec)))
   "Return the min build (stoptime, id) pair for
    the given evaluation EVAL."
   (let ((rows (sqlite-exec db "
-SELECT stoptime, MIN(id) FROM
-(SELECT id, stoptime FROM Builds
+SELECT stoptime, MIN(rowid) FROM
+(SELECT rowid, stoptime FROM Builds
 WHERE evaluation=" eval " AND
 stoptime = (SELECT MIN(stoptime)
 FROM Builds WHERE evaluation=" eval "))")))
@@ -643,8 +619,8 @@ FROM Builds WHERE evaluation=" eval "))")))
   "Return the max build (stoptime, id) pair for
    the given evaluation EVAL."
   (let ((rows (sqlite-exec db "
-SELECT stoptime, MAX(id) FROM
-(SELECT id, stoptime FROM Builds
+SELECT stoptime, MAX(rowid) FROM
+(SELECT rowid, stoptime FROM Builds
 WHERE evaluation=" eval " AND
 stoptime = (SELECT MAX(stoptime)
 FROM Builds WHERE evaluation=" eval "))")))
diff --git a/src/schema.sql b/src/schema.sql
index eb0f7e9..0452495 100644
--- a/src/schema.sql
+++ b/src/schema.sql
@@ -37,43 +37,31 @@ CREATE TABLE Evaluations (
   FOREIGN KEY (specification) REFERENCES Specifications (name)
 );
 
-CREATE TABLE Derivations (
-  derivation    TEXT NOT NULL,
-  evaluation    INTEGER NOT NULL,
-  job_name      TEXT NOT NULL,
-  system        TEXT NOT NULL,
-  nix_name      TEXT NOT NULL,
-  PRIMARY KEY (derivation, evaluation),
-  FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
-);
-
 CREATE TABLE Outputs (
-  build INTEGER NOT NULL,
+  derivation TEXT NOT NULL,
   name TEXT NOT NULL,
   path TEXT NOT NULL,
-  PRIMARY KEY (build, name),
-  FOREIGN KEY (build) REFERENCES Builds (id)
+  PRIMARY KEY (derivation, name),
+  FOREIGN KEY (derivation) REFERENCES Builds (derivation)
 );
 
--- Builds are not in a one to one relationship with derivations in order to
--- keep track of non deterministic compilations.
 CREATE TABLE Builds (
-  id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
-  derivation    TEXT NOT NULL,
+  derivation    TEXT NOT NULL PRIMARY KEY,
   evaluation    INTEGER NOT NULL,
+  job_name      TEXT NOT NULL,
+  system        TEXT NOT NULL,
+  nix_name      TEXT NOT NULL,
   log           TEXT NOT NULL,
   status        INTEGER NOT NULL,
   timestamp     INTEGER NOT NULL,
   starttime     INTEGER NOT NULL,
   stoptime      INTEGER NOT NULL,
-  FOREIGN KEY (derivation) REFERENCES Derivations (derivation),
   FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
 );
 
 -- Create indexes to speed up common queries, in particular those
 -- corresponding to /api/latestbuilds and /api/queue HTTP requests.
-CREATE INDEX Builds_Derivations_index ON Builds(status ASC, timestamp ASC, id, derivation, evaluation, stoptime DESC);
+CREATE INDEX Builds_index ON Builds(job_name, system, status ASC, timestamp ASC, derivation, evaluation, stoptime DESC);
 CREATE INDEX Inputs_index ON Inputs(specification, name, branch);
-CREATE INDEX Derivations_index ON Derivations(derivation, evaluation, job_name, system);
 
 COMMIT;
diff --git a/src/sql/upgrade-2.sql b/src/sql/upgrade-2.sql
new file mode 100644
index 0000000..dfb919b
--- /dev/null
+++ b/src/sql/upgrade-2.sql
@@ -0,0 +1,49 @@
+BEGIN TRANSACTION;
+
+DROP INDEX Derivations_index;
+DROP INDEX Builds_Derivations_index;
+
+ALTER TABLE Outputs RENAME TO tmp_Outputs;
+ALTER TABLE Builds RENAME TO tmp_Builds;
+
+CREATE TABLE Builds (
+  derivation    TEXT NOT NULL PRIMARY KEY,
+  evaluation    INTEGER NOT NULL,
+  job_name      TEXT NOT NULL,
+  system        TEXT NOT NULL,
+  nix_name      TEXT NOT NULL,
+  log           TEXT NOT NULL,
+  status        INTEGER NOT NULL,
+  timestamp     INTEGER NOT NULL,
+  starttime     INTEGER NOT NULL,
+  stoptime      INTEGER NOT NULL,
+  FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
+);
+
+CREATE TABLE Outputs (
+  derivation TEXT NOT NULL,
+  name TEXT NOT NULL,
+  path TEXT NOT NULL,
+  PRIMARY KEY (derivation, name),
+  FOREIGN KEY (derivation) REFERENCES Builds (derivation)
+);
+
+INSERT OR IGNORE INTO Builds (derivation, evaluation, job_name, system, nix_name, log, status, timestamp, starttime, stoptime)
+SELECT Derivations.derivation, Derivations.evaluation, Derivations.job_name, Derivations.system, Derivations.nix_name,
+       tmp_Builds.log, tmp_Builds.status, tmp_Builds.timestamp, tmp_Builds.starttime, tmp_Builds.stoptime
+FROM Derivations
+INNER JOIN tmp_Builds ON tmp_Builds.derivation = Derivations.derivation
+                     AND tmp_Builds.evaluation = Derivations.evaluation;
+
+INSERT OR IGNORE INTO Outputs (derivation, name, path)
+SELECT tmp_Builds.derivation, tmp_Outputs.name, tmp_Outputs.path
+FROM tmp_Outputs
+INNER JOIN tmp_Builds on tmp_Builds.id = tmp_Outputs.build;
+
+CREATE INDEX Builds_index ON Builds(job_name, system, status ASC, timestamp ASC, derivation, evaluation, stoptime DESC);
+
+DROP TABLE tmp_Builds;
+DROP TABLE tmp_Outputs;
+DROP TABLE Derivations;
+
+COMMIT;
diff --git a/tests/database.scm b/tests/database.scm
index 17d48f5..af518bd 100644
--- a/tests/database.scm
+++ b/tests/database.scm
@@ -50,27 +50,14 @@
   `((#:specification . "guix")
     (#:commits . ,commits)))
 
-(define* (make-dummy-job #:optional (name "foo"))
-  `((#:name . ,name)
-    (#:job-name . "job")
-    (#:system . "x86_64-linux")
-    (#:derivation . ,(string-append name ".drv"))
-    (#:nix-name . "foo")
-    (#:specification 0)
-    (#:eval-id . 42)))
-
-(define* (make-dummy-derivation drv #:optional (eval-id 0))
+(define* (make-dummy-build drv
+                           #:optional (eval-id 42)
+                           #:key (outputs '(("foo" . "/foo"))))
   `((#:derivation . ,drv)
+    (#:eval-id . ,eval-id)
     (#:job-name . "job")
     (#:system . "x86_64-linux")
-    (#:nix-name . ,(basename drv ".drv"))
-    (#:eval-id . ,eval-id)))
-
-(define* (make-dummy-build #:optional (eval-id 42)
-                           #:key (drv "/foo.drv")
-                           (outputs '(("foo" . "/foo"))))
-  `((#:derivation . ,drv)
-    (#:eval-id . ,eval-id)
+    (#:nix-name . "foo")
     (#:log . "log")
     (#:outputs . (("foo" . "/foo")))))
 
@@ -86,10 +73,6 @@
   ;; Global Slot for a database object.
   (make-parameter #t))
 
-(define %id
-  ;; Global Slot for a job ID in the database.
-  (make-parameter #t))
-
 (define database-name
   ;; Use an empty and temporary database for the tests.
   (string-append (getcwd) "/" (number->string (getpid)) "-tmp.db"))
@@ -114,21 +97,13 @@ INSERT INTO Evaluations (specification, commits) VALUES (3, 3);")
       (db-add-specification (%db) example-spec)
       (car (db-get-specifications (%db)))))
 
-  (test-assert "db-add-derivation"
-    (let* ((job (make-dummy-job))
-           (key (assq-ref job #:derivation)))
-      (db-add-derivation (%db) job)
-      (db-add-derivation (%db) job)               ;idempotent
-      (%id key)))
-
-  (test-assert "db-get-derivation"
-    (db-get-derivation (%db) (%id)))
-
-  (test-assert "db-add-build"
-    (let ((build (make-dummy-build)))
+  (test-equal "db-add-build"
+    #f
+    (let ((build (make-dummy-build "/foo.drv")))
       (db-add-build (%db) build)
 
-      ;; This should be idempotent, see <https://bugs.gnu.org/28094>.
+      ;; Should return #f when adding a build whose derivation is already
+      ;; there, see <https://bugs.gnu.org/28094>.
       (db-add-build (%db) build)))
 
   (test-equal "db-update-build-status!"
@@ -137,13 +112,12 @@ INSERT INTO Evaluations (specification, commits) VALUES (3, 3);")
           (build-status succeeded)
           "/foo.drv.log")
     (with-temporary-database db
-      (let* ((id (db-add-build
-                  db
-                  (make-dummy-build 1 #:drv "/foo.drv"
-                                    #:outputs '(("out" . "/foo")))))
+      (let* ((derivation (db-add-build
+                          db
+                          (make-dummy-build "/foo.drv" 1
+                                            #:outputs '(("out" . "/foo")))))
              (get-status (lambda* (#:optional (key #:status))
-                           (assq-ref (db-get-build db id) key))))
-        (db-add-derivation db (make-dummy-derivation "/foo.drv" 1))
+                           (assq-ref (db-get-build db derivation) key))))
         (db-add-evaluation db (make-dummy-eval))
         (db-add-specification db example-spec)
 
@@ -171,18 +145,15 @@ INSERT INTO Evaluations (specification, commits) VALUES (3, 3);")
       ((3 "/baz.drv"))                               ;nr = 1
       ((2 "/bar.drv") (1 "/foo.drv") (3 "/baz.drv"))) ;status+submission-time
     (with-temporary-database db
-      ;; Populate the 'Builds', 'Derivations', 'Evaluations', and
+      ;; Populate the 'Builds'', 'Evaluations', and
       ;; 'Specifications' tables in a consistent way, as expected by the
       ;; 'db-get-builds' query.
-      (db-add-build db (make-dummy-build 1 #:drv "/foo.drv"
+      (db-add-build db (make-dummy-build "/foo.drv" 1
                                          #:outputs `(("out" . "/foo"))))
-      (db-add-build db (make-dummy-build 2 #:drv "/bar.drv"
+      (db-add-build db (make-dummy-build "/bar.drv" 2
                                          #:outputs `(("out" . "/bar"))))
-      (db-add-build db (make-dummy-build 3 #:drv "/baz.drv"
+      (db-add-build db (make-dummy-build "/baz.drv" 3
                                          #:outputs `(("out" . "/baz"))))
-      (db-add-derivation db (make-dummy-derivation "/foo.drv" 1))
-      (db-add-derivation db (make-dummy-derivation "/bar.drv" 2))
-      (db-add-derivation db (make-dummy-derivation "/baz.drv" 3))
       (db-add-evaluation db (make-dummy-eval))
       (db-add-evaluation db (make-dummy-eval))
       (db-add-evaluation db (make-dummy-eval))
@@ -206,19 +177,16 @@ INSERT INTO Evaluations (specification, commits) VALUES (3, 3);")
   (test-equal "db-get-pending-derivations"
     '("/bar.drv" "/foo.drv")
     (with-temporary-database db
-      ;; Populate the 'Builds', 'Derivations', 'Evaluations', and
+      ;; Populate the 'Builds', 'Evaluations', and
       ;; 'Specifications' tables.  Here, two builds map to the same derivation
       ;; but the result of 'db-get-pending-derivations' must not contain any
       ;; duplicate.
-      (db-add-build db (make-dummy-build 1 #:drv "/foo.drv"
+      (db-add-build db (make-dummy-build "/foo.drv" 1
                                          #:outputs `(("out" . "/foo"))))
-      (db-add-build db (make-dummy-build 2 #:drv "/bar.drv"
+      (db-add-build db (make-dummy-build "/bar.drv" 2
                                          #:outputs `(("out" . "/bar"))))
-      (db-add-build db (make-dummy-build 3 #:drv "/foo.drv"
+      (db-add-build db (make-dummy-build "/foo.drv" 3
                                          #:outputs `(("out" . "/foo"))))
-      (db-add-derivation db (make-dummy-derivation "/foo.drv" 1))
-      (db-add-derivation db (make-dummy-derivation "/bar.drv" 2))
-      (db-add-derivation db (make-dummy-derivation "/foo.drv" 3))
       (db-add-evaluation db (make-dummy-eval))
       (db-add-evaluation db (make-dummy-eval))
       (db-add-evaluation db (make-dummy-eval))
diff --git a/tests/http.scm b/tests/http.scm
index e05fdc5..a9fc3ef 100644
--- a/tests/http.scm
+++ b/tests/http.scm
@@ -142,6 +142,9 @@
     (let* ((build1
             `((#:derivation . "/gnu/store/fake.drv")
               (#:eval-id . 1)
+              (#:job-name . "fake-job")
+              (#:system . "x86_64-linux")
+              (#:nix-name . "fake-1.0")
               (#:log . "unused so far")
               (#:status . ,(build-status succeeded))
               (#:outputs . (("out" . "/gnu/store/fake-1.0")))
@@ -151,24 +154,15 @@
            (build2
             `((#:derivation . "/gnu/store/fake2.drv")
               (#:eval-id . 1)
+              (#:job-name . "fake-job")
+              (#:system . "x86_64-linux")
+              (#:nix-name . "fake-2.0")
               (#:log . "unused so far")
               (#:status . ,(build-status scheduled))
               (#:outputs . (("out" . "/gnu/store/fake-2.0")))
               (#:timestamp . 1501347493)
               (#:starttime . 0)
               (#:stoptime . 0)))
-           (derivation1
-            '((#:derivation . "/gnu/store/fake.drv")
-              (#:job-name . "fake-job")
-              (#:system . "x86_64-linux")
-              (#:nix-name . "fake-1.0")
-              (#:eval-id . 1)))
-           (derivation2
-            '((#:derivation . "/gnu/store/fake2.drv")
-              (#:job-name . "fake-job")
-              (#:system . "x86_64-linux")
-              (#:nix-name . "fake-2.0")
-              (#:eval-id . 1)))
            (specification
             '((#:name . "guix")
               (#:load-path-inputs . ("savannah"))
@@ -192,8 +186,6 @@
               (#:commits . ("fakesha2" "fakesha3")))))
       (db-add-build (%db) build1)
       (db-add-build (%db) build2)
-      (db-add-derivation (%db) derivation1)
-      (db-add-derivation (%db) derivation2)
       (db-add-specification (%db) specification)
       (db-add-evaluation (%db) evaluation1)
       (db-add-evaluation (%db) evaluation2)))
-- 
2.18.0

bug#32190: [PATCH] database: Merge Derivations into Builds table.

[Clément Lassieur]

Clément Lassieur <clement@lassieur.org> writes:

> Fixes <https://bugs.gnu.org/32190>.

[...]

> ---
>  Makefile.am              |   3 +-
>  src/cuirass/base.scm     |  23 +++---
>  src/cuirass/database.scm | 174 +++++++++++++++++----------------------
>  src/schema.sql           |  28 ++-----
>  src/sql/upgrade-2.sql    |  49 +++++++++++
>  tests/database.scm       |  78 ++++++------------
>  tests/http.scm           |  20 ++---
>  7 files changed, 176 insertions(+), 199 deletions(-)
>  create mode 100644 src/sql/upgrade-2.sql

I'll update the doc as well.

bug#32190: [PATCH] database: Merge Derivations into Builds table.

[Clément Lassieur]

Clément Lassieur <clement@lassieur.org> writes:

>  Makefile.am              |   3 +-
>  src/cuirass/base.scm     |  23 +++---
>  src/cuirass/database.scm | 174 +++++++++++++++++----------------------
>  src/schema.sql           |  28 ++-----
>  src/sql/upgrade-2.sql    |  49 +++++++++++
>  tests/database.scm       |  78 ++++++------------
>  tests/http.scm           |  20 ++---
>  7 files changed, 176 insertions(+), 199 deletions(-)
>  create mode 100644 src/sql/upgrade-2.sql

You can see how it looks like here:

https://cuirass.lassieur.org:8081/jobset/guix-master

bug#32190: [PATCH] database: Merge Derivations into Builds table.

[Clément Lassieur]

I forgot:

Clément Lassieur <clement@lassieur.org> writes:

> Fixes <https://bugs.gnu.org/32190>.
>
> * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-2.sql'.
> * src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table.

(build-packages): Add columns that were in the Derivations table before.  Only
build the derivations that were successfully registered, that is, those that
didn't exist in the Builds table.  Give a derivation instead of a build id to
DB-GET-BUILD.  Compute the number of failed jobs based on the derivations that
were added to the table, instead of the jobs.

> * src/cuirass/database.scm (db-add-derivation, db-get-derivation): Remove
> exported procedures.
> (db-add-build): Catch SQLITE_CONSTRAINT_PRIMARYKEY error, which means that two
> jobs produced the same derivation, and return #f in that case.  Add columns

[...]

And (in base.scm)

@@ -584,7 +587,7 @@ procedure is meant to be called at startup."
                              (((_ (#:path . (? string? outputs))) ...)
                               outputs))
                            outputs))
-         (fail (- (length jobs) success)))
+         (fail (- (length derivations) success)))
     (log-message "outputs:\n~a" (string-join outs "\n"))
     (log-message "success: ~a, fail: ~a" success fail)
     results))

bug#32190: [PATCH] database: Merge Derivations into Builds table.

[Ricardo Wurmus]

Hi Clément,

> Fixes <https://bugs.gnu.org/32190>.

Woo!  Thank you for this patch.

> * src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table.

I see that you’ve mentioned your changes to “build-packages” in a later
email.

I have two general questions about this: why was the change from “id” to
“rowid” necessary?  And: could you please also update the documentation
so that is reflects the changes?

> diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm
> index b4b1652..7788ac9 100644
> --- a/src/cuirass/database.scm
> +++ b/src/cuirass/database.scm
[…]
>  (define (db-add-evaluation db eval)
>    (sqlite-exec db "\
>  INSERT INTO Evaluations (specification, commits) VALUES ("
> @@ -384,27 +356,39 @@ string."
>  (define (db-add-build db build)
>    "Store BUILD in database DB. BUILD eventual outputs are stored
>  in the OUTPUTS table."
> -  (let* ((build-exec
> -          (sqlite-exec db "\
> -INSERT INTO Builds (derivation, evaluation, log, status, timestamp, starttime, stoptime)\
> -  VALUES ("
> -                       (assq-ref build #:derivation) ", "
> -                       (assq-ref build #:eval-id) ", "
> -                       (assq-ref build #:log) ", "
> -                       (or (assq-ref build #:status)
> -                           (build-status scheduled)) ", "
> -                       (or (assq-ref build #:timestamp) 0) ", "
> -                       (or (assq-ref build #:starttime) 0) ", "
> -                       (or (assq-ref build #:stoptime) 0) ");"))
> -         (build-id (last-insert-rowid db)))
> -    (for-each (lambda (output)
> -                (match output
> -                  ((name . path)
> -                   (sqlite-exec db "\
> -INSERT INTO Outputs (build, name, path) VALUES ("
> -                                build-id ", " name ", " path ");"))))
> -              (assq-ref build #:outputs))
> -    build-id))
> +  (catch 'sqlite-error
> +    (lambda ()
> +      (sqlite-exec db "
> +INSERT INTO Builds (derivation, evaluation, job_name, system, nix_name, log,
> +status, timestamp, starttime, stoptime)
> +VALUES ("
> +                   (assq-ref build #:derivation) ", "
> +                   (assq-ref build #:eval-id) ", "
> +                   (assq-ref build #:job-name) ", "
> +                   (assq-ref build #:system) ", "
> +                   (assq-ref build #:nix-name) ", "
> +                   (assq-ref build #:log) ", "
> +                   (or (assq-ref build #:status)
> +                       (build-status scheduled)) ", "
> +                   (or (assq-ref build #:timestamp) 0) ", "
> +                   (or (assq-ref build #:starttime) 0) ", "
> +                   (or (assq-ref build #:stoptime) 0) ");")
> +      (let ((derivation (assq-ref build #:derivation)))
> +        (for-each (lambda (output)
> +                    (match output
> +                      ((name . path)
> +                       (sqlite-exec db "\
> +INSERT INTO Outputs (derivation, name, path) VALUES ("
> +                                    derivation ", " name ", " path ");"))))
> +                  (assq-ref build #:outputs))
> +        derivation))

This procedure is called when a build is scheduled, isn’t it?  The
docstring says “BUILD eventual outputs are stored in the OUTPUTS table.”
– does this mean the names of the *expected* output directories are
recorded in Outputs?  They don’t exist at this point, right?

> +    (lambda (key who code message . rest)
> +      ;; If we get a unique-constraint-failed error, that means we have
> +      ;; already inserted the same build.  That happens when several jobs
> +      ;; produce the same derivation, and we can ignore it.
> +      (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
> +          #f
> +          (apply throw key who code rest)))))

Okay.

Can we prevent this from happening in the first place?  I feel a little
uncomfortable about performing an operation that we expect to cause
primary key errors regularly.

> diff --git a/src/schema.sql b/src/schema.sql
> index eb0f7e9..0452495 100644
> --- a/src/schema.sql
> +++ b/src/schema.sql
[…]
> --- Builds are not in a one to one relationship with derivations in order to
> --- keep track of non deterministic compilations.

Is this comment still correct considering that the derivation is now the
primary key of the Builds table?

>  CREATE TABLE Builds (
> -  id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
> -  derivation    TEXT NOT NULL,
> +  derivation    TEXT NOT NULL PRIMARY KEY,
>    evaluation    INTEGER NOT NULL,
> +  job_name      TEXT NOT NULL,
> +  system        TEXT NOT NULL,
> +  nix_name      TEXT NOT NULL,
>    log           TEXT NOT NULL,
>    status        INTEGER NOT NULL,
>    timestamp     INTEGER NOT NULL,
>    starttime     INTEGER NOT NULL,
>    stoptime      INTEGER NOT NULL,
> -  FOREIGN KEY (derivation) REFERENCES Derivations (derivation),
>    FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
>  );

Thanks again!

--
Ricardo

bug#32190: [PATCH] database: Merge Derivations into Builds table.

[Clément Lassieur]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> I see that you’ve mentioned your changes to “build-packages” in a later
> email.
>
> I have two general questions about this: why was the change from “id” to
> “rowid” necessary?  And: could you please also update the documentation
> so that is reflects the changes?

I removed the 'id' column from the Builds table because it was used as
the primary key, and I wanted the primary key to be 'derivation'.  But
we still need to display the build id in the web interface, and the API
allows to get the builds by id.  We can't use the 'id' column anymore
because AUTOINCREMENT isn't allowed without PRIMARY KEY[1], but we can
use the rowid[2] implicit column instead.

I updated the documentation on my personal branch[3], but I didn't send
it to the ml, because I have sent too many things already ;)

[1]: https://www.sqlite.org/autoinc.html
[2]: https://www.sqlite.org/rowidtable.html
[3]: https://git.lassieur.org/cgit/cuirass.git/

>> diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm
>> index b4b1652..7788ac9 100644
>> --- a/src/cuirass/database.scm
>> +++ b/src/cuirass/database.scm

[...]

>>  (define (db-add-build db build)
>>    "Store BUILD in database DB. BUILD eventual outputs are stored
>>  in the OUTPUTS table."

[...]

> This procedure is called when a build is scheduled, isn’t it?

Yes.

> The docstring says “BUILD eventual outputs are stored in the OUTPUTS
> table.”  – does this mean the names of the *expected* output
> directories are recorded in Outputs?  They don’t exist at this point,
> right?

Indeed, it's just the output paths written in the derivation files.

>> +    (lambda (key who code message . rest)
>> +      ;; If we get a unique-constraint-failed error, that means we have
>> +      ;; already inserted the same build.  That happens when several jobs
>> +      ;; produce the same derivation, and we can ignore it.
>> +      (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
>> +          #f
>> +          (apply throw key who code rest)))))
>
> Okay.
>
> Can we prevent this from happening in the first place?  I feel a little
> uncomfortable about performing an operation that we expect to cause
> primary key errors regularly.

We would need to manually check that the derivation doesn't already
exist.  That would be an extra query so I think it would be less
efficient and it would require more code.  I believe the 'constraint' is
a pretty good mechanism to describe things that shouldn't happen in the
database, I don't know any better way.

>> diff --git a/src/schema.sql b/src/schema.sql
>> index eb0f7e9..0452495 100644
>> --- a/src/schema.sql
>> +++ b/src/schema.sql
> […]
>> --- Builds are not in a one to one relationship with derivations in order to
>> --- keep track of non deterministic compilations.
>
> Is this comment still correct considering that the derivation is now the
> primary key of the Builds table?

No, I forgot to remove it.

>>  CREATE TABLE Builds (
>> -  id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>> -  derivation    TEXT NOT NULL,
>> +  derivation    TEXT NOT NULL PRIMARY KEY,
>>    evaluation    INTEGER NOT NULL,
>> +  job_name      TEXT NOT NULL,
>> +  system        TEXT NOT NULL,
>> +  nix_name      TEXT NOT NULL,
>>    log           TEXT NOT NULL,
>>    status        INTEGER NOT NULL,
>>    timestamp     INTEGER NOT NULL,
>>    starttime     INTEGER NOT NULL,
>>    stoptime      INTEGER NOT NULL,
>> -  FOREIGN KEY (derivation) REFERENCES Derivations (derivation),
>>    FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
>>  );
>
> Thanks again!

:-) you're welcome!  Thanks for reviewing.

Clément

bug#32190: [PATCH] database: Merge Derivations into Builds table.

[Clément Lassieur]

Pushed, thank you for the review!

Clément

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Clément,
>
>> Fixes <https://bugs.gnu.org/32190>.
>
> Woo!  Thank you for this patch.
>
>> * src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table.
>
> I see that you’ve mentioned your changes to “build-packages” in a later
> email.
>
> I have two general questions about this: why was the change from “id” to
> “rowid” necessary?  And: could you please also update the documentation
> so that is reflects the changes?
>
>> diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm
>> index b4b1652..7788ac9 100644
>> --- a/src/cuirass/database.scm
>> +++ b/src/cuirass/database.scm
> […]
>>  (define (db-add-evaluation db eval)
>>    (sqlite-exec db "\
>>  INSERT INTO Evaluations (specification, commits) VALUES ("
>> @@ -384,27 +356,39 @@ string."
>>  (define (db-add-build db build)
>>    "Store BUILD in database DB. BUILD eventual outputs are stored
>>  in the OUTPUTS table."
>> -  (let* ((build-exec
>> -          (sqlite-exec db "\
>> -INSERT INTO Builds (derivation, evaluation, log, status, timestamp, starttime, stoptime)\
>> -  VALUES ("
>> -                       (assq-ref build #:derivation) ", "
>> -                       (assq-ref build #:eval-id) ", "
>> -                       (assq-ref build #:log) ", "
>> -                       (or (assq-ref build #:status)
>> -                           (build-status scheduled)) ", "
>> -                       (or (assq-ref build #:timestamp) 0) ", "
>> -                       (or (assq-ref build #:starttime) 0) ", "
>> -                       (or (assq-ref build #:stoptime) 0) ");"))
>> -         (build-id (last-insert-rowid db)))
>> -    (for-each (lambda (output)
>> -                (match output
>> -                  ((name . path)
>> -                   (sqlite-exec db "\
>> -INSERT INTO Outputs (build, name, path) VALUES ("
>> -                                build-id ", " name ", " path ");"))))
>> -              (assq-ref build #:outputs))
>> -    build-id))
>> +  (catch 'sqlite-error
>> +    (lambda ()
>> +      (sqlite-exec db "
>> +INSERT INTO Builds (derivation, evaluation, job_name, system, nix_name, log,
>> +status, timestamp, starttime, stoptime)
>> +VALUES ("
>> +                   (assq-ref build #:derivation) ", "
>> +                   (assq-ref build #:eval-id) ", "
>> +                   (assq-ref build #:job-name) ", "
>> +                   (assq-ref build #:system) ", "
>> +                   (assq-ref build #:nix-name) ", "
>> +                   (assq-ref build #:log) ", "
>> +                   (or (assq-ref build #:status)
>> +                       (build-status scheduled)) ", "
>> +                   (or (assq-ref build #:timestamp) 0) ", "
>> +                   (or (assq-ref build #:starttime) 0) ", "
>> +                   (or (assq-ref build #:stoptime) 0) ");")
>> +      (let ((derivation (assq-ref build #:derivation)))
>> +        (for-each (lambda (output)
>> +                    (match output
>> +                      ((name . path)
>> +                       (sqlite-exec db "\
>> +INSERT INTO Outputs (derivation, name, path) VALUES ("
>> +                                    derivation ", " name ", " path ");"))))
>> +                  (assq-ref build #:outputs))
>> +        derivation))
>
> This procedure is called when a build is scheduled, isn’t it?  The
> docstring says “BUILD eventual outputs are stored in the OUTPUTS table.”
> – does this mean the names of the *expected* output directories are
> recorded in Outputs?  They don’t exist at this point, right?
>
>> +    (lambda (key who code message . rest)
>> +      ;; If we get a unique-constraint-failed error, that means we have
>> +      ;; already inserted the same build.  That happens when several jobs
>> +      ;; produce the same derivation, and we can ignore it.
>> +      (if (= code SQLITE_CONSTRAINT_PRIMARYKEY)
>> +          #f
>> +          (apply throw key who code rest)))))
>
> Okay.
>
> Can we prevent this from happening in the first place?  I feel a little
> uncomfortable about performing an operation that we expect to cause
> primary key errors regularly.
>
>> diff --git a/src/schema.sql b/src/schema.sql
>> index eb0f7e9..0452495 100644
>> --- a/src/schema.sql
>> +++ b/src/schema.sql
> […]
>> --- Builds are not in a one to one relationship with derivations in order to
>> --- keep track of non deterministic compilations.
>
> Is this comment still correct considering that the derivation is now the
> primary key of the Builds table?
>
>>  CREATE TABLE Builds (
>> -  id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>> -  derivation    TEXT NOT NULL,
>> +  derivation    TEXT NOT NULL PRIMARY KEY,
>>    evaluation    INTEGER NOT NULL,
>> +  job_name      TEXT NOT NULL,
>> +  system        TEXT NOT NULL,
>> +  nix_name      TEXT NOT NULL,
>>    log           TEXT NOT NULL,
>>    status        INTEGER NOT NULL,
>>    timestamp     INTEGER NOT NULL,
>>    starttime     INTEGER NOT NULL,
>>    stoptime      INTEGER NOT NULL,
>> -  FOREIGN KEY (derivation) REFERENCES Derivations (derivation),
>>    FOREIGN KEY (evaluation) REFERENCES Evaluations (id)
>>  );
>
> Thanks again!