From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mathieu Lirzin Subject: Re: [GSoC] Development of Cuirass. Date: Mon, 20 Mar 2017 23:43:46 +0100 Message-ID: <87h92ny3fh.fsf@gnu.org> References: <87tw6yim7o.fsf@gnu.org> <87inndblxq.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:60008) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cq61o-0005S7-IZ for guix-devel@gnu.org; Mon, 20 Mar 2017 18:43:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cq61n-0003W3-Al for guix-devel@gnu.org; Mon, 20 Mar 2017 18:43:52 -0400 In-Reply-To: <87inndblxq.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 13 Mar 2017 09:49:53 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Hi, ludo@gnu.org (Ludovic Court=C3=A8s) writes: >> 2.1 The testing infrastructure >> =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80 >> >> Currently Cuirass is providing a small test suite consisting only of >> unit tests. The integration tests are done manually with basic >> examples consisting in building small packages such as GNU Hello. >> This is not good, because actual packages take a consequent time to >> compile, and testing only basic examples leaves the interesting >> complex case out of the scope of testing. The worst is that the >> result depend on the phase of the moon since every commit in Guix can >> potentially break the build. >> >> The idea is to take inspiration from the Guix test suite by providing >> tiny mocked packages that will help having lightweight build >> specifications. Those tests require launching another instance of the >> `guix-daemon' with a local temporary store. > > Sounds like a good idea. > > It would be nice to refine what it is we=E2=80=99re trying to test and th= at=E2=80=99s > not addressed by unit tests. At one end of the spectrum, I imagine we > could be spawning a couple of GuixSD VMs connected together, one serving > a Git repo, and the other one running Cuirass pulling from that repo (a > system test). > > At the middle of the spectrum is something like you describe, I think: > run Cuirass directly upon =E2=80=9Cmake check=E2=80=9D, though there are = complications: > what repo do we pull from, do we really want to build packages and if > not how to we mock builds, etc. The idea I had was to create fake vcs repositories programmatically. I imagine that it is possible to associate some dummy build script that we can deterministically make fail. > There are several components we=E2=80=99d like to test more closely IMO: = SCM > polling, evaluation, and job scheduling. Perhaps some of that can also > be achieved through unit tests? Yeah, that would be better. >> 2.2 HTTP API >> =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80 >> >> Since commit [12d71ee098c3eb84a9a5dc2419bd37a3b9e55dd2] Cuirass has a >> web server which is running in a separate thread of the `cuirass' >> process. However the resources it provides are pretty limited since >> you can only query the lists of specifications currently watched. >> This is done with the `/specifications' or `/jobsets' resource names. >> "Jobsets" is the terminology used by Hydra and is used here for >> compatibility with its HTTP API. > > For a start, I think we should implement all or a subset of the Hydra > API, as-is (it=E2=80=99s currently used by Emacs-Guix): > > https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Controller/API= .pm > > A large part of it is about monitoring the status of Hydra: queued > builds, recently-completed builds, jobs, etc. That is the most pressing > need, I think. OK. >> This roadmap lacks details about the concrete tasks and its associated >> timeline. However I prefer having some feedback about this proposal >> first, before providing a complete roadmap. >> >> =E2=80=A2 From May 30 to June 26, I will work on the testing infrastru= cture,. >> =E2=80=A2 From June 27 to August 29, I will work on the HTTP API. > > You may find that you=E2=80=99ll want to interleave tasks. I don=E2=80= =99t know about > you, but sometimes I get bored if I keep working on the same task or > domain for too long, so switching to something else for a while can be a > relief. :-) I don't know, I have been rarely confronted to that issue personnaly (more the opposite: too much context switching). But I will preventively adapt my roadmap. > There are a couple of isolated tasks that come to mind, which do not > really fit in the proposal but are worth looking into as time permits: > > =E2=80=A2 improved error reporting upon evaluation failures and so on; > > =E2=80=A2 improved error recovery; I totally forgot to precise that the error handling issue is exactly the reason I think we need a better test infrastructure for Cuirass. Being able to reproduce various errors or failues allows a better confidence in the error being properly handled. I find it hard to just imagine and write the appropriate handler. > =E2=80=A2 use of Guile-Git instead of Git. I Will add this to the plan. Thanks. --=20 Mathieu Lirzin GPG: F2A3 8D7E EB2B 6640 5761 070D 0ADE E100 9460 4D37