From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id EKyTChpd4V6qaQAA0tVLHw (envelope-from ) for ; Wed, 10 Jun 2020 22:22:18 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 6D9PBhpd4V7CBgAAbx9fmQ (envelope-from ) for ; Wed, 10 Jun 2020 22:22:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4B0AA9408FC for ; Wed, 10 Jun 2020 22:22:17 +0000 (UTC) Received: from localhost ([::1]:36234 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jj96s-0005jt-M9 for larch@yhetil.org; Wed, 10 Jun 2020 18:22:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42512) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jj95i-0004Dy-VB for bug-guix@gnu.org; Wed, 10 Jun 2020 18:21:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:52264) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jj95i-00022l-Ac for bug-guix@gnu.org; Wed, 10 Jun 2020 18:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jj95i-0005p8-6i for bug-guix@gnu.org; Wed, 10 Jun 2020 18:21:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible Resent-From: Bengt Richter Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 10 Jun 2020 22:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41669 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Chris Marusich Received: via spool by 41669-submit@debbugs.gnu.org id=B41669.159182762422293 (code B ref 41669); Wed, 10 Jun 2020 22:21:02 +0000 Received: (at 41669) by debbugs.gnu.org; 10 Jun 2020 22:20:24 +0000 Received: from localhost ([127.0.0.1]:35566 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jj955-0005nV-IK for submit@debbugs.gnu.org; Wed, 10 Jun 2020 18:20:23 -0400 Received: from imta-36.everyone.net ([216.200.145.36]:36906 helo=imta-38.everyone.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jj954-0005nM-17 for 41669@debbugs.gnu.org; Wed, 10 Jun 2020 18:20:23 -0400 Received: from pps.filterd (omta002.sj2.proofpoint.com [127.0.0.1]) by imta-38.everyone.net (8.16.0.27/8.16.0.27) with SMTP id 05AMH0QI004431; Wed, 10 Jun 2020 15:20:21 -0700 X-Eon-Originating-Account: H4SzW8FaSkwGSif7jeDnM_m1YTv3GA2Qd-qjf20FW3M X-Eon-Dm: m0116293.ppops.net Received: by m0116293.mta.everyone.net (EON-AUTHRELAY2 - 5a81c94d) id m0116293.5e67f91c.81a880; Wed, 10 Jun 2020 15:20:18 -0700 X-Eon-Sig: AQMHrIJe4VyiMk/tzgIAAAAF,f938924eb8f51a25ce799e06f480d947 X-Eip: WAdbK_sdcq8v4IO9A5Md1wWTkjF1uhrNQrNtDrvbinY Date: Thu, 11 Jun 2020 00:20:08 +0200 From: Bengt Richter Message-ID: <20200610222008.GB3238@LionPure> References: <874krtnvk8.fsf@gmail.com> <87y2p4mqe2.fsf@gmail.com> <87imfzcuqi.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <87imfzcuqi.fsf@gmail.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.687 definitions=2020-06-10_13:2020-06-10, 2020-06-10 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1034 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2004280000 definitions=main-2006100159 X-Spam-Score: -0.4 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.4 (-) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bengt Richter Cc: 41669@debbugs.gnu.org, =?UTF-8?Q?L=C3=A9o?= Le Bouter , Maxim Cournoyer , Vincent Legoll Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: 2.49 X-TUID: tHltE6vbb86/ Hi Chris, et al, On +2020-06-09 23:15:01 -0700, Chris Marusich wrote: > Hi Vincent and everyone, > > Vincent Legoll writes: > > > Is that showing the same (or a similar) problem : > > > > https://data.guix-patches.cbaines.net/gnu/store/0lcbxpw1vrca02dzpzw2rxhad7pn4zw7-gcc-objc-5.5.0 > > > > ? > > Can you clarify what you mean? I'm not sure what you're referring to. > > Chris Marusich writes: > > > At present, it seems possible that within the context of a single > > machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a > > different machine, it may (reproducibly) build a different output. > > I'm a bit paranoid about making mistakes, so I'll perform another full > > GC and then try yet again to build gcc-stripped-tarball-5.5.0.drv in > > order to verify whether it truly produces the same output when all (or > > nearly all) of its inputs are rebuilt from scratch. > > I repeated the experiment on the same machine (it took a day or two to > build), and the result was the same: on my machine, > gcc-stripped-tarball-5.5.0.drv builds identical output to what it built > before. To be clear, using Guix 8159ce1970d91567468cf1bacac313099a009d2a > on an x86_64-linux machine, I tried (yet again) the following steps: > [...] > Efraim's diff looks a little different in statx.h, even though he used > the same Guix commit as me. Maybe this is because he cross-compiled on > an aarch64-linux machine, while I cross-compiled on an x86_64-linux > machine. In the other cases, it looks like the binary files differ in > basically the same ways. I will share some examples below. > > Here is some diffoscope output between my c++ and Efraim's (many other > sections also differed in similarly cryptic ways): > [...] > > If I'm reading this correctly, one problem seems to be that our GCC > toolchains are putting symbols at different locations. This issue (and > maybe others) could be trickling down, causing other aspects of the > binaries to differ (e.g., in length). Nothing really stands out, but > when we discussed this on IRC, we thought perhaps factors like the > following might contribute to the non-reproducibility: > > - Perhaps we are all running different Linux kernel versions? In some > cases, the kernel version can unfortunately influence the build > output, so this might be worth testing. > > - Perhaps the GCC Makefiles etc. are doing something non-deterministic? > Questions triggered in my mind: Where are respective machines getting their rules for packing and aligning structs and unions? Is any struct or rule/flags source dynamically generated, where different rules could come from different defaults, or .configs, or even invalid memoizations jumping domains? Could pointer arithmetic get done in one domain and the offset be misused in another? Wrong C preprocessor? Difference in sort key comparisons for canonicalization of ordering? Hope that's not all red herrings :) Sorry for the noise otherwise. > - Something else? Hm, some race condition between processes that should be order-independent but are not. Then if different hardware components on different systems -- disks, memory, processors -- cause different but repeatable patterns of waits (convoying?) you could get repeatable but different builds. I guess you'd have to figure out which order was really right, and force the order of processing explicitly to that order, so all systems would do it that way. > > Avenues of investigation: > > - If anything obvious stands out from the diffoscope output, please > leave a comment. > > - Try building with different kernel versions on the same machine, to > see if they differ. > > - If somebody else could please confirm that running the following > command reports no difference on their own machine (i.e., exit code > 0), that would be good to know, since it would help further solidify > the theory that on a single machine, the build of gcc-static-5.5.0.drv > is reproducible, even if it is not reproducible across machines: > > guix build --no-substitutes --check --target=powerpc64-linux-gnu \ > -e '(@@ (gnu packages make-bootstrap) %gcc-static)' > > - Try building two different versions of gcc-7.5.0 (maybe by hand?), and > then use them to build a simple reproduction case and compare results. > If we're lucky, maybe this will help us understand the problem better. > > We'll get there! > > -- > Chris -- Regards, Bengt Richter