From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:45653) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ihEdn-0001rA-9H for guix-patches@gnu.org; Tue, 17 Dec 2019 10:20:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ihEdm-0005kK-0h for guix-patches@gnu.org; Tue, 17 Dec 2019 10:20:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:36222) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ihEdl-0005jj-Tv for guix-patches@gnu.org; Tue, 17 Dec 2019 10:20:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ihEdl-0005j6-Oj for guix-patches@gnu.org; Tue, 17 Dec 2019 10:20:01 -0500 Subject: [bug#38649] [PATCH] Parallelize `guix package` Resent-Message-ID: Message-ID: <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> From: Leo Prikler Date: Tue, 17 Dec 2019 16:19:34 +0100 In-Reply-To: <87tv5zrpjp.fsf@gnu.org> References: <87tv5zrpjp.fsf@gnu.org> Content-Type: text/plain; charset="UTF-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: julien lepiller , 38649@debbugs.gnu.org Hi Ludo’, Am Dienstag, den 17.12.2019, 15:32 +0100 schrieb Ludovic Courtès: > Hi Leo, > > (Cc: Julien, who worked on this part.) > > Leo Prikler skribis: > > > Yesterday I had an interesting conversation on IRC about the > > behaviour > > of multiple `guix package` processes running in parallel. > > Specifically, when two transactions target the same profile > > (usually > > /var/guix/profiles/per-user/$USER/guix-profile) at the same time, > > one > > of them will fail to claim the lock and abort. 0001 makes it so > > that > > the process waits for the lock. 0002 makes it so that packages > > specified via -i can be built in parallel. > > I actually like the current behavior, FWIW. Julien came up with this > locking mostly so that people do not inadvertently attempt to perform > several operations concurrently. Fair enough and that is an improvement over non-locking behaviour, where one could spawn multiple profile generations from one, neither of which is complete. Perhaps my attempt at doing this in a somewhat controlled manner is equally harmful, but I will still try my best arguing for it, as I believe it can make a positive impact. > The key word here is “inadvertently”: IMO, there’s no reason to run > multiple ‘guix package’ on the same profile concurrently. With a > wait-for-lock policy, the result would be non-deterministic: you > cannot > tell which one of the two processes will complete first. > > WDYT? I think the current policy is wait-for-lock deferred to the user. The user has to let the first task complete before they can start the second. In this setup, the user can simply launch the setup and trust, that it will complete later while taking into account the changes the first one has made. Let's talk about three classes of operations – installations, removals and upgrades – and their interactions. I will not take into account roll-back, switch-generation and delete-generation, as it is nonsensical to perform these in parallel to any other action. Perhaps we could check for their presence first and acquire the lock with no- wait semantics in that case. - any operation on different packages: Either succeeds first and the other builds on the profile it generates. As there is no collision in the packages themselves, there will be no harm. - install same package twice: Either succeeds first, the other will be a no-op. - install vs. remove same package: Non-deterministic, but why would you do that? - install vs. upgrade same package: Upgrade will be a no-op in either case. - remove vs. upgrade same package: Upgrade may inadvertently upgrade the old package if it happens to come first, but in the final package it will be removed either way. Of course, any operation can also fail midway due to some step not succeeding. In that case it would be as if one had issued the other command right after that, which may perhaps not be what one wanted to do (assuming I install package A, and some guide suggests to also build related, but not dependency-connected package B, so I end up installing B without A). However, such cases can easily be fixed by either installing a fixed version of A later, using B on its own if it can be, or rolling back. Of course, both solutions are flawed in the way that they assume user intent either way. Perhaps a better one would be to let the user specify whether they want to wait or not through a command line parameter, using the current behaviour as the default approach. WDYT? Regards, Leo