From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id qLNPNsLG+2H6EAAAgWs5BA (envelope-from ) for ; Thu, 03 Feb 2022 13:12:50 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id KGEpL8LG+2H0MQAAG6o9tA (envelope-from ) for ; Thu, 03 Feb 2022 13:12:50 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5318427BD3 for ; Thu, 3 Feb 2022 13:12:50 +0100 (CET) Received: from localhost ([::1]:37068 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nFayn-0003yP-EC for larch@yhetil.org; Thu, 03 Feb 2022 07:12:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nFaw9-0003xt-1B for guix-devel@gnu.org; Thu, 03 Feb 2022 07:10:05 -0500 Received: from [2a00:1450:4864:20::434] (port=36797 helo=mail-wr1-x434.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nFaw3-0003Pe-HI for guix-devel@gnu.org; Thu, 03 Feb 2022 07:10:02 -0500 Received: by mail-wr1-x434.google.com with SMTP id u15so4660388wrt.3 for ; Thu, 03 Feb 2022 04:09:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:in-reply-to:references:date:message-id:mime-version :content-transfer-encoding; bh=TcsgqDYw6DcWpIhMtIFJFz146xmLU3CJ1f1B0zFEN2s=; b=FHDvFN6zTtmsy/TQbQKjI/PjXxVx6lQPl34YNm55Yz2TUQEPMnS1aNc8+NwlIH2hyj ybfCSmqcvJXRnllEMVXlziIA91epUFg17aHn6kLB7OymIbdXy2wUXE36dNfkaEXi60gW tpQ7QgFrBw4Dyx7A0hU6bGW2gciMza+Ikd7xFAmijGl/t9ZorLrhCV2fvR9JZqIkxzTi wpg01Aa8ZSea5l1CNTLoF9vcGKsSorVQ55x0lcFU7wcmvz4r1L/ElMMkUoX5zr2OlakT F51LpLIWapFfJDWmlYTV2n8AVI0t+FARFOQD/LvB4SphwAx7oZNKP//9pf9gqb12T1uR mGiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=TcsgqDYw6DcWpIhMtIFJFz146xmLU3CJ1f1B0zFEN2s=; b=fMG5UFAbDA2RrOtA6DEYZtn69glpfvvgXwrgroTNDENDydrQoBZ03cWENzsS95vWt3 MyG0WhTW2p8P5nOtdd2JiyC9RWHcvztMdmVAECNqZHCl8PLxLG83BmhNTBNRA+PbpTzM +oebBOTnNumKgT9p17Mn+inLtfURV1C7kB6eCR8Reih3KKKDhWXaL/KKdxpKTrxA92OA 1Jx5bmYze/KAkq4Jqtird5a5VYt86B5oHiSSMlw1jkweYbSDJRnMRaWG2hResw+ycB+j CTfHoAqyRyiQzeGJB6rvxdtW9R3yIXN3YIESishU0tOQs7lU4ade4CYqf2/K4fSV0Nqm fz7w== X-Gm-Message-State: AOAM532wECtET6qicHFf+H70oKQTTTYQbuadPL3NWN0gZ4kJMtCN2mME QHn0kF3lz9QyVE5EQQkAAyfkflaw290= X-Google-Smtp-Source: ABdhPJyZ3L6w67mKiwpw1lO2ndIu9ZTGxXfMRNrfIZGoBEomN4h0SrQk2BUHh6e50iVvKZ++7kiyEQ== X-Received: by 2002:adf:e8ce:: with SMTP id k14mr20710253wrn.284.1643890185747; Thu, 03 Feb 2022 04:09:45 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id h4sm23803519wre.0.2022.02.03.04.09.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 04:09:45 -0800 (PST) From: zimoun To: Konrad Hinsen , Guix Devel , Ricardo Wurmus Subject: Re: Investigating a reproducibility failure In-Reply-To: References: <871r0l9fd1.fsf@gmail.com> Date: Thu, 03 Feb 2022 13:07:37 +0100 Message-ID: <86o83oywza.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::434 (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::434; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x434.google.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PDS_HP_HELO_NORDNS=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1643890370; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=TcsgqDYw6DcWpIhMtIFJFz146xmLU3CJ1f1B0zFEN2s=; b=IeXVhcj4hS4BHyHbFve+RokppJh51O/BKnIwbiVxpKRFA4iZScG8fbmWLnRrcpu/Qji3KM EK745ph/AOzHYNTcsN0zodRcDC9fKGL5lMeh19MCjRqS87M9M8Cdw1QogoQ7Km1CNl6agZ TkEr+VOyeEGmmNxG5GBDgJY+acU8Jz25wwX/YI/vjRitWvvShhvqjGg8UwKV95hRjGNDcl 6gu7YBm+ZdYrWThtOCDfcTYjoPWp2ohNH9RDxSxdJ0FMDl0lYivzKZ+6E97uQH1ro4OyKS pxSdO3tLonoOqELXIAVM6rJIIwdJOU63DjYZckZjFrGkR8pXmShslswCoTttdw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1643890370; a=rsa-sha256; cv=none; b=JUXQvdDytW7J62b20ycAELHobqFX/1zzmMgVJcdyw7tYpS3GhkG6b7yXpFovQm+bxQw+7M kZ4uJe+7YdNELt9e/7iJsUlzdGrUPAe+To7vjj7CC628Mxzj5QgZ4Xrd4bQPaZ7UVn+v8o acG6Gj5g/UzKbG9nczHu89k2+7yd/SDky7v/Y0fTIq098G8yx6n5mnGYNOVrNop4apRlos a0QyeHis3KA6AoznqMyJVG7z/QWdMrQs+U5nqSHNWGnBIFKnn4skXu9GOQtgd5A6F0iprq w8b1jp8dDORCZyngOtzhvp7/IcJBsYHwCn4hKG3Odym3G91YebXdEaAEIIEOZw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FHDvFN6z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.33 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FHDvFN6z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 5318427BD3 X-Spam-Score: -4.33 X-Migadu-Scanner: scn1.migadu.com X-TUID: hb3aCo4HmO7e Hi Konrad, On Thu, 03 Feb 2022 at 10:16, Konrad Hinsen wr= ote: >> CPU detection is a bottomless can of worms. > > That sounds very credible. But what can we do about this? Well, I do not know what could be done about this. Today, the picture for OpenBLAS@0.3.6 build looks like: * Fail i7-1185G7E (Tiger Lake) i7-10700K (Comet Lake) * Build i7-6500U (Skylake) E7-4870V2 (Ivy Bridge) 5218 (Cascade Lake) Somehow, =E2=80=9Crecent=E2=80=9D processors cannot build old versions. > There is obviously a trade-off between reproducibility and performance > here. Can we support both, in a way that users can understand and manage? Usually both [1]. However, it is not clear for me why OpenBLAS v0.3.6 does not build on some =E2=80=9Crecent=E2=80=9C processors; even in poor pe= rformance mode with as much as possible generic code. 1: > The OpenBlas package in Guix is (or at least was, back then) written for > performance. Can I, as a user, ask for a reproducible version? That > could either be a generic version for any x86 architecture (preferably), > or one that always builds for a given sub-architecture and then fails at > runtime if the CPU doesn't match. > > Next: can I, as a user of dependent code, ask for reproducible versions > of all my dependencies? In my case, I was packaging Python code that > calls OpenBlas via NumPy. Many people in that situation don't even know > what OpenBlas is. I did know, but wasn't aware of the build-time CPU > detection. > > There is of course the issue that we can never be sure if a build will > be reproducible in the future. But we can at least take care of the > cases where the packager is aware of non-reproducibility issues, and > make them transparent and manageable. The answer of your concerns is the transformation --tune, I guess. This transformation is providing micro-optimizations for high performance while preserving provenance tracking. Here the issue seems different. OpenBLAS v0.3.6 seems to fail to fallback to generic processor when it does not find the processor=E2=80=93probably because the microarchitecture was not existing or supported at the time. (Note that =E2=80=99Comet Lake=E2=80=99 is not in the list %gcc-10-x86_64-micro-architectures, so --tune would probably be inefficient; I do not know.) Cheers, simon