From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id UOqDMkrwomG0WAAAgWs5BA (envelope-from ) for ; Sun, 28 Nov 2021 03:58:18 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id kEk+LkrwomFZcQAAB5/wlQ (envelope-from ) for ; Sun, 28 Nov 2021 02:58:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 720DB2446B for ; Sun, 28 Nov 2021 03:58:18 +0100 (CET) Received: from localhost ([::1]:47804 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mrAOP-0001Np-Jj for larch@yhetil.org; Sat, 27 Nov 2021 21:58:17 -0500 Received: from eggs.gnu.org ([209.51.188.92]:52696) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mrAOA-0001Nd-PE for guix-patches@gnu.org; Sat, 27 Nov 2021 21:58:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:51473) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mrAOA-00075f-Gz for guix-patches@gnu.org; Sat, 27 Nov 2021 21:58:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mrAOA-0000ux-3c for guix-patches@gnu.org; Sat, 27 Nov 2021 21:58:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52117] [core-updates-frozen] [PATCH 0/6] Fix Julia packages. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 28 Nov 2021 02:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52117 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: zimoun Cc: 52117@debbugs.gnu.org Received: via spool by 52117-submit@debbugs.gnu.org id=B52117.16380682803519 (code B ref 52117); Sun, 28 Nov 2021 02:58:02 +0000 Received: (at 52117) by debbugs.gnu.org; 28 Nov 2021 02:58:00 +0000 Received: from localhost ([127.0.0.1]:34786 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mrAO7-0000uh-QE for submit@debbugs.gnu.org; Sat, 27 Nov 2021 21:58:00 -0500 Received: from mail-qt1-f169.google.com ([209.85.160.169]:44988) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mrAO6-0000uV-FY for 52117@debbugs.gnu.org; Sat, 27 Nov 2021 21:57:58 -0500 Received: by mail-qt1-f169.google.com with SMTP id a2so12792053qtx.11 for <52117@debbugs.gnu.org>; Sat, 27 Nov 2021 18:57:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=jaIPA/l01t1TSTpclrwqQwQGmJNQI5hBcltQawr3Bl4=; b=D4AdjUHWT7faZBCURDR16RsJcYJrGPtPLgj8Pzgjy41KqSrTCElMcHRXnAaGIUJiDK 7iOBTzU4DKC+VpouheJZ/rugBpoSJOrPl38pfnfdNw7+xFiMpTjlJWnRUNxoCNshar0p rejMl+kGD3rxLc4n320dvvSNuCSNcsGzii2iskrm7TH4q/ovihbQOqWb+QLP9Mwwn0EK VfbWYouOHAwadMeTeD055ilCsi29t15KnhCvWXSU8IIrSppraMztrlITEp3fLdEW1LzP EsSJuhks92bppM0sL/zrMMfCg6NADdQv8/8B93JPVSC1FOfc+6TQU8+NN9jo5cy3iKPJ LMQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=jaIPA/l01t1TSTpclrwqQwQGmJNQI5hBcltQawr3Bl4=; b=vWUfGhEsNTeM4RVimSMLcKBppQtUnjZV33rr58iMHJpWQ7e4FZP0xyyTHk4F+MhnAL 1db80bPSdVxHmRseGFHrxEz8oHvr165WvZTK2yqUmPlDcsBR+5p23xvVNP1Tp5fDizDw STgV4Wc6tZpkeEFsCu0/9yo5Qg1B4d503zRGizaakYVMhnYtBkyTvBkjNXHWaYMnWi8A azd7QajbdS9JGNhiPL7cGBX7LRTNkywpUdtzT+JYXk9v3fpyrbKME9dcNSqLxjPJoLYt x8YlTR8Z9Uq32yjlxtsVu12qEhdrdPVOKx1ewPkjXYnAfMTJHv5hcORNaiOUX3EGlWZX bILA== X-Gm-Message-State: AOAM531bfikCKJECspxNPzU4ijgy2qx+a2EFSPlgnYZT+2hYUhpNfXXM 7SX9dm6JS2qJz7/Mu1s091TbkuZgzTbuFw== X-Google-Smtp-Source: ABdhPJzSrQh13HWqIhH5rmk3ZIZijxAf6INHw12cIm43uXVo1eQcJiwG73Wfw/vE/66qZHIgoQhk7g== X-Received: by 2002:a05:622a:18e:: with SMTP id s14mr34911751qtw.203.1638068272447; Sat, 27 Nov 2021 18:57:52 -0800 (PST) Received: from hurd (dsl-148-169.b2b2c.ca. [66.158.148.169]) by smtp.gmail.com with ESMTPSA id u21sm6479448qtw.29.2021.11.27.18.57.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 27 Nov 2021 18:57:52 -0800 (PST) From: Maxim Cournoyer References: <20211125233559.34575-1-zimon.toutoune@gmail.com> <20211125233559.34575-2-zimon.toutoune@gmail.com> <87fsri8ez5.fsf_-_@gmail.com> <861r31kc2m.fsf@gmail.com> Date: Sat, 27 Nov 2021 21:57:51 -0500 In-Reply-To: <861r31kc2m.fsf@gmail.com> (zimoun's message of "Sat, 27 Nov 2021 13:38:57 +0100") Message-ID: <87tufx6l74.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1638068298; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=jaIPA/l01t1TSTpclrwqQwQGmJNQI5hBcltQawr3Bl4=; b=QWwjOdQqbms6EyIvo+zo71AN0fwNDL4RMfESJrhr2gPBubcFBkzUgquTaEPNYNBSD+fnNb N/F5Zc9wXXX6uxKTmoqAjJVj3QEC4w7zmhxRbdIpjPi5vLcc9GG69feZo5dARZig8xTVGW GXlCMiQ+Mm52Fl3oSnIbWLKdzTAyOwecLQXDODYqzfdi9IyvI8kXm8tKJw2U+ddhVe10hy Ct54IBZ4p3+XkOzzTLf7ecuAhr+eWbrg8I3Gh6dmIJDS6UOSfWoEBaqY7tvDWfAwelOaNW wnKumDOONV4cUfKa33C6hAVgZhPAFOfqZ/1AMWpFztccq4WjQBH4avL+DmjM+w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1638068298; a=rsa-sha256; cv=none; b=akR3gthXlsPfLNcp6jFO0GdZvL9R0gX7psf7uRySyk95d8GLmVTAuTxtOgjhZD/sM6zAnU OIoprRWzQHo9jNGfEyg4es+a1y+ZvbAuma2eYXFNPYEbcIN8bf/bvRcJopATcxvdb630Qv Gj754ZUDJ13teVrFZTk5ZzUF0lFzySpolHvdGRjAGc9casmG0QUKZiy4Qin33u2e0RBSU4 tna7WlaGo5Gpe+ux7rBTzmiAR59st3eztXRIlZcMGvyXx96NNJXGs8O7mr6IXEH+2SoJQo x+v3Cl1jGLXTDR+YHaNtwvpA5dbRg2cHSDXEAOHP7zSEAY4NoUaiWScLaRGsPA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=D4AdjUHW; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.80 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=D4AdjUHW; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 720DB2446B X-Spam-Score: -1.80 X-Migadu-Scanner: scn0.migadu.com X-TUID: Pg7Oz9yFvHju Hello Simon, zimoun writes: > Hi Maxim, > > Thanks for the review and the improved patch. > > I am sorry if the commit message and/or changelog I provided was badly > worded, but somehow it was an attempt to explain the odd behaviour =E2=80= =93 at > least counter-intuitive since I initially felt into when sending the > very first patch allowing parallel tests and you felt too, I guess. :-) No worries. Communicating changes (or anything) is always one of the greatest challenges in programming and elsewhere, it seems :-). The nice thing about it is that it can be improved with perseverance and some feedback. > > On Fri, 26 Nov 2021 at 22:17, Maxim Cournoyer = wrote: > >>> --- >>> guix/build/julia-build-system.scm | 8 +++++--- >>> 1 file changed, 5 insertions(+), 3 deletions(-) >>> >>> diff --git a/guix/build/julia-build-system.scm b/guix/build/julia-build= -system.scm >>> index f0dc419c17..af478fd4a3 100644 >>> --- a/guix/build/julia-build-system.scm >>> +++ b/guix/build/julia-build-system.scm >>> @@ -112,7 +112,10 @@ (define* (check #:key tests? source inputs outputs= julia-package-name >>> (builddir (string-append out "/share/julia/")) >>> (jobs (if parallel-tests? >>> (number->string (parallel-job-count)) >>> - "1"))) >>> + "1")) >>> + (nprocs (if parallel-tests? >>> + (string-append "--procs=3D" jobs) >>> + ""))) >>> ;; With a patch, SOURCE_DATE_EPOCH is honored >>> (setenv "SOURCE_DATE_EPOCH" "1") >>> (setenv "JULIA_DEPOT_PATH" builddir) >>> @@ -122,8 +125,7 @@ (define* (check #:key tests? source inputs outputs = julia-package-name >>> ""))) >>> (setenv "JULIA_CPU_THREADS" jobs) >>> (setenv "HOME" "/tmp") >>> - (invoke "julia" "--depwarn=3Dyes" >>> - (string-append "--procs=3D" jobs) >>> + (invoke "julia" "--depwarn=3Dyes" nprocs >> >> Here nprocs can be ""; is it really OK to pass an empty string argument >> to julia? > > Yes it is OK. When #:parallel-tests? sets to #f, my patch leads to > the call =E2=80=9Cjulia --depwarn=3Dyes=E2=80=9D which is valid. Your mo= dified patch > adds another test but leads to the same call =E2=80=9Cjulia --depwarn=3Dy= es=E2=80=9D. No, it would invoke julia with the following argv list: "julia" "-depwarn=3Dyes" "" [...]; My point is that invoke is equivalent to doing an execlp system call; and the arguments get passed as a list (including that empty string argument when parallel-tests? is #f). Whether this works or not is up to the application, so I'd suggest not relying on it. Consider for example: --8<---------------cut here---------------start------------->8--- (execlp "python" "python" "" "-c" "print('hello')") /gnu/store/cwqv4z5bvb5x6i0zvqgc1j1dnr6w9vp8-profile/bin/python: can't find '__main__' module in '/home/maxim/src/guix-core-updates-next/gnu/packages/' --8<---------------cut here---------------end--------------->8--- It fails because it interprets the empty string argument as the current directory, apparently. If that works with the above Julia invocation, that's great, but it doesn't make it cleaner in my opinion :-). > + (job-count (if parallel-tests? > + (parallel-job-count) > + 1)) > + ;; The --proc argument of Julia *adds* extra processors rathe= r than > + ;; specify the exact count to use, so zero must be specified = to > + ;; disable parallel processing... > > [..] > > + (apply invoke "julia" > + `("--depwarn=3Dyes" > + ,@(if parallel-tests? > + ;; XXX: ... but '--procs' doesn't accept 0 as a val= id > + ;; value, so just omit the argument entirely. > + (list (string-append "--procs=3D" > + (number->string additional-pr= ocs))) > + '()) > > > So because of 2 tests. I think your modified patch is more > =E2=80=9Ccomplicated=E2=80=9D. :-) It is slightly more complex indeed; but I think it provides the reader with useful knowledge of julia's quirks and is more correct. > About this, > > + (additional-procs (max 0 (1- job-count)))) > > I considered that it was not a big deal; initially, I did something > similar in =E2=80=99let=E2=80=99 but remove it because it changes nothing= from my > experiments. In fact, because =E2=80=99--procs=E2=80=99 overrides JULIA_= CPU_THREADS and > run Julia with n or n+1 is more or less the same for the Julia land, > IMHO. Well, it is not clear what is the load for the main worker. And > JULIA_CPU_THREADS=3D1 is required for running using only one worker. > Anyway, this changes nothing, practically speaking. :-) Indeed, it is > better and more consistent. Yeah, I don't like that the behavior of --procs is to *add* workers, rather than set its exact count; which make thing awkward. Even upstream get tricked by that by erroneously *adding* Sys.CPU_THREADS workers because of this in test/runtest.jl [0] [0] https://github.com/JuliaLang/julia/pull/43217#pullrequestreview-817102= 530 Thanks, Maxim