From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id sM1gEnrNhF9lfQAA0tVLHw (envelope-from ) for ; Mon, 12 Oct 2020 21:41:14 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id cA9UDnrNhF8pQwAA1q6Kng (envelope-from ) for ; Mon, 12 Oct 2020 21:41:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DCA809404D1 for ; Mon, 12 Oct 2020 21:41:11 +0000 (UTC) Received: from localhost ([::1]:58658 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kS5Z8-0007tf-Ic for larch@yhetil.org; Mon, 12 Oct 2020 17:41:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55092) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kS5Yz-0007tW-Vs for guix-devel@gnu.org; Mon, 12 Oct 2020 17:41:02 -0400 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]:38707) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kS5Yx-0002mm-JH for guix-devel@gnu.org; Mon, 12 Oct 2020 17:41:01 -0400 Received: by mail-wm1-x335.google.com with SMTP id b127so6394786wmb.3 for ; Mon, 12 Oct 2020 14:40:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=jCqHeVFn4moMwkHO9NIZGlymiFfy02gmVIwvfHL1E24=; b=TXE4PYM4mS4f/DPFllxW0BdE9RYdWv3o/rBmlgIJn3xL+a7hrK7x1bkfdI3hTwfxbj cT1xLtaQHviAlmQxsz6ornKAijoeMAxfu82H7dOWg50ZItPDn06RC1+1fGA9AYrCMLFl t4ao13Z7bbU0LGPN5AU1+end9df+XjTy9jVp1AbmTcqHnhnc8cX4F0/AMaHRtyHZLAzV Fi96xxJ1JE1XV81fmkkeUW4uV5PZ7nvuOkUxic3xYoj+HVRHi/L4yjgrf0wIHnjaw3gi CYuET0D5hEkqRIswTPM7IXNwI+cTBnD3NQlgih1u+lB3iDGpP0+NgwOsMQiaW0F96sLj kdRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=jCqHeVFn4moMwkHO9NIZGlymiFfy02gmVIwvfHL1E24=; b=Tr3aiQSWehWT7fvAxberdiIo/i1LmT/H+AZW4Knn2iN4A9PRz9jiV47wFmd9vR6w/C MIJlvKFD3RHlo6th8d8L7lgs69m8OB85FSHUewGx1ylkB1yDfenD2K0inG6RHgs7x95M +AI6RgKyduajH4a3kfCyYmY62sLyOnNFm6awns4KNC4tkSqUUjMaKYE6LHyWeOIYU+e5 ht3gFlAcU46orPsf39IKkDMpT+9LBsRFyqaDjetdpBI4oL1298EynCJopDLjLZJQQ+Zu KSvhR1uj1kFYH8EI9f7iWhXlRJZmy0W0Lqxrc8GKRym3UulbrZ4jskSB9K0qfMXbUzxr 5YmA== X-Gm-Message-State: AOAM532/ZdQV+yCr+5qXy0lpYgcnwTtnvcDNHmKegZ/+3YGGiBRIBcLU 0rXHeCG2M7dMRrDLYDWqxSw= X-Google-Smtp-Source: ABdhPJx2NiDYWq+EmZeWF1p+xO9cj/lrNSUMzMM718mSmDVAAMwPjRwSBzBwyo9UngDB3XDtTznO4w== X-Received: by 2002:a7b:c250:: with SMTP id b16mr8169879wmj.161.1602538857671; Mon, 12 Oct 2020 14:40:57 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id t6sm29573523wre.30.2020.10.12.14.40.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Oct 2020 14:40:56 -0700 (PDT) From: zimoun To: Guix Devel , Christopher Baines Subject: Reproductibility, Data Services, guix weather Date: Mon, 12 Oct 2020 23:40:55 +0200 Message-ID: <86mu0rt95k.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2a00:1450:4864:20::335; envelope-from=zimon.toutoune@gmail.com; helo=mail-wm1-x335.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=TXE4PYM4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: 3.79 X-TUID: auOJn6n5J6V4 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Dear, Recently, we discovered a regression in the Haskell build system: introducing unreproducible builds. Well, it was a kind of luck: I was testing =E2=80=99git-annex=E2=80=99 with the willing to have =E2=80=99git-a= nnex-assitant=E2=80=99 building it several times (--check) [1]. Aside this particular issue, ~10% of packages are not reproducible and I am not convinced that =E2=80=9C--check=E2=80=9D is done by submitter/commit= ter at each update or new package. Otherwise the case of unreproducible Mesa [2] would raised before than June. :-) (That=E2=80=99s fine, we need package a= fter all and we cannot fix the world all in the same time. :-)) The issue is to be able to find them. I proposed (below) to run cron task doing =E2=80=99--check=E2=80=99 on the build farms and then report by = email the failure. Chris indicated me the work they is doing [3] and instead of a cron task, they is proposing to parse the JSON. That=E2=80=99s what the ti= ny script attached is doing. guix repl -L . -- weather-repro.scm For example, I run: guix repl -L . -- weather-repro.scm | sort | grep ghc to list (almost) all the unreproducible Haskell packages. What I would like is to be able to filter by build system for example. First, Chris could you add the fields package name and version? Because it is hard to automatically reconstruct them by parsing the output-path. Second, the revision of does not match the Guix commit. Is it possible to have a bridge? Other said, how is computed this revision hash? (A working revision is 6cf35799dec60723f37d83a559429aa8b90482d5 which does not seems founding in Guix repo.) Third, this tiny script is better than nothing but *far far away* form perfect. The question about tooling is: does it make sense to include something like that directly in =E2=80=9Cguix weather=E2=80=9D? For exampl= e, guix weather --reproducible or maybe under =E2=80=9Cguix challenge=E2=80=9D? WDYT? Feedback and ideas are very welcome. :-) All the best, simon PS: Below my question and the Chris=E2=80=99s answer. Both deserve to be p= ublic as Chris told me. :-) 1: 2: 3: -------------------- Start of forwarded message -------------------- From: zimoun Subject: [guix-sysadmin] whishlist: Hook on the build-farm? Date: Sun, 11 Oct 2020 17:19:26 +0200 Hi, Currently, it is hard to catch: 1. which commit breaks which package 2. if the package builds reproductibly Even if the Data services helps, *a lot!*. There are still a lot of manual actions to spot one or the other. And I fully agree that the work initiated by Chris is The Right Thing=C2=A9. However it is not ready and the man power is not extensible. For the #1, Danny have started a discussion.=20 For the #2, I am proposing to add a cron task on one build-farm. To be concrete, let=E2=80=99s *randomly* pick 100 packages once a week, rebuild w= ith =E2=80=9C--check=E2=80=9C and send by email the unreproducible packages. Even, I am proposing: 1rst week 100 packages of build-system =E2=80=9Cfoo= =E2=80=9D, 2nd week 100 packages of build-system =E2=80=9Cbar=E2=80=9D, 3rd week=E2=80=A6 It is far from perfect but it seems a good heuristic to catch regression, spot packages with reproducibility troubles, etc. Note that it should not happen since the committer should catch the reproducibility issue; but as a matter of fact it is not the case. Somehow, I am proposing a workaround. I volunteer to be the recipient of these automatic emails, then I can do some triage (remove false-positive, check what=E2=80=99s going, etc.) and = open a bug report if there is an issue. Currently, I do not have the CPU power to do so. So I am asking if it possible to put something like that on one of the building machines. I totally understand an answer as: =C2=AB Simon, you are enthusiast and that= =E2=80=99s nice but no and go to hell! =C2=BB :-) Cheers, simon -------------------- End of forwarded message -------------------- -------------------- Start of forwarded message -------------------- From: Christopher Baines Subject: Re: [guix-sysadmin] whishlist: Hook on the build-farm? Date: Sun, 11 Oct 2020 17:38:03 +0100 > Currently, I do not have the CPU power to do so. So I am asking if it > possible to put something like that on one of the building machines. I > totally understand an answer as: =C2=AB Simon, you are enthusiast and tha= t=E2=80=99s > nice but no and go to hell! =C2=BB :-) I too would really like to be able to identify/prevent regressions, including with respect to build reproducibility, and although the work I'm doing on this is going slowly I'm hoping that with the Guix Build Coordinator now I'll be able to get something sort of working. I've just made a few tweaks to the Guix Data Service to make the data it has on this a little easier to use. This URL [1] should show you package reproducibility stats for each architecture, computed from substitute data from ci.guix.gnu.org, bayfront.guix.gnu.org, guix.tobias.gr and guix.cbaines.net. Currently there seem to be 1515 outputs (so not exactly packages, but close) that don't seem to have built reproducibly. 1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-= processed-revision/package-reproducibility Clicking through to the "Not matching" ones for x86_64-linux should give you this URL [2]. In case it's useful to have this data in a more machine readable form, I've added a JSON output option. 2 :https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-= processed-revision/package-derivation-outputs?output_consistency=3Dnot-matc= hing&system=3Dx86_64-linux You mention triage, and that's probably the biggest blocker to being able to methodically try and reduce the "Not matching" numbers on [1]. As far as I know, Debian has things like [3] and [4] to help with that. 3: https://tests.reproducible-builds.org/debian/index_issues.html 4: https://salsa.debian.org/reproducible-builds/reproducible-notes/-/blob/m= aster/issues.yml Going back to the issue of a cron job to run guix build --check on some random packages and send emails, if you're looking for a list of packages (well actually outputs) which don't build reproducibly, then [2] might do? The JSON output doesn't contain the package names, but it probably could with only a little effort. If it did, you could download the JSON file for all the non-matching package outputs, and record the package names in a sorted list in a Git repository. If you do that every day, then you could read the git log to spot potential patterns/regressions. I don't think your email hit a mailing list, feel free to send my reply to one though, maybe guix-devel as this discussion probably deserves a wide audience. Thanks, Chris -------------------- End of forwarded message -------------------- --=-=-= Content-Type: application/octet-stream Content-Disposition: inline; filename=weather-repro.scm Content-Transfer-Encoding: base64 Content-Description: weather-repro.scm KGRlZmluZS1tb2R1bGUgKHdlYXRoZXItcmVwcm8pCiAgIzp1c2UtbW9kdWxlIChqc29uKQoKICAj OnVzZS1tb2R1bGUgKChndWl4IGkxOG4pICM6c2VsZWN0IChHXykpCiAgIzp1c2UtbW9kdWxlICgo Z3VpeCBkaWFnbm9zdGljcykgIzpzZWxlY3QgKGxlYXZlKSkKICAjOnVzZS1tb2R1bGUgKChndWl4 IGRlc2NyaWJlKSAjOnNlbGVjdCAoY3VycmVudC1wcm9maWxlKSkKICAjOnVzZS1tb2R1bGUgKGd1 aXggY2hhbm5lbHMpCgogICM6dXNlLW1vZHVsZSAoZ3VpeCB1dGlscykgICAgICAgICAgICAgO3Bh Y2thZ2UtbmFtZS0+bmFtZSt2ZXJzaW9uLCB2ZXJzaW9uLSoKICAjOnVzZS1tb2R1bGUgKChndWl4 IGJ1aWxkIGRvd25sb2FkKSAjOnNlbGVjdCAodXJsLWZldGNoKSkKCiAgOyM6dXNlLW1vZHVsZSAo c3JmaSBzcmZpLTExKSAgICAgICAgICAgOyBsZXQtdmFsdWVzCiAgIzp1c2UtbW9kdWxlIChpY2Ut OSBtYXRjaCkKICAjOnVzZS1tb2R1bGUgKHNyZmkgc3JmaS0xKSAgICAgICAgICAgIDsgZm9sZAog ICM6dXNlLW1vZHVsZSAoc3JmaSBzcmZpLTM3KSAgICAgICAgICAgOyBwYXJzZSBjb21tYW5kIGxp bmUgKG9wdGlvbikKICApCgoKDAooZGVmaW5lICV0ZW1wb3JhcnktZGlyZWN0b3J5CiAgOzsgVGVt cG9yYXJ5IGRpcmVjdG9yeS4KICAob3IgKGdldGVudiAiVE1QRElSIikgIi90bXAiKSkKCgooZGVm aW5lICVwcmVmaXgtdXJsCiAgImh0dHBzOi8vZGF0YS5ndWl4LXBhdGNoZXMuY2JhaW5lcy5uZXQv cmV2aXNpb24iKQoKKGRlZmluZSAlc3VmZml4LXVybAogICJvdXRwdXRfY29uc2lzdGVuY3k9bm90 LW1hdGNoaW5nJnRhcmdldD1ub25lJmFsbF9yZXN1bHRzPW9uIikKCihkZWZpbmUgJWpzb24tbmFt ZSAicGFja2FnZS1kZXJpdmF0aW9uLW91dHB1dHMuanNvbiIpCgoMCihkZWZpbmUqIChqc29uLXVy bCByZXZpc2lvbgogICAgICAgICAgICAgICM6b3B0aW9uYWwKICAgICAgICAgICAgICAoc3lzdGVt ICglY3VycmVudC1zeXN0ZW0pKQogICAgICAgICAgICAgIChqc29uICVqc29uLW5hbWUpKQogICJS ZXR1cm4gdGhlIFVSTCBjb3JyZXNwb25kaW5nIHRvIFJFVklTSU9OLiIKICAoc3RyaW5nLWFwcGVu ZAogICAlcHJlZml4LXVybCAiLyIgcmV2aXNpb24gIi8iIGpzb24gIj8iICVzdWZmaXgtdXJsIiZz eXN0ZW09IiBzeXN0ZW0pKQoKKGRlZmluZSogKGpzb24tZmlsZSByZXZpc2lvbgogICAgICAgICAg ICAgICAgICAgICM6b3B0aW9uYWwKICAgICAgICAgICAgICAgICAgICAoc3lzdGVtICglY3VycmVu dC1zeXN0ZW0pKQogICAgICAgICAgICAgICAgICAgIChuYW1lICVqc29uLW5hbWUpCiAgICAgICAg ICAgICAgICAgICAgKHRtcCAldGVtcG9yYXJ5LWRpcmVjdG9yeSkpCiAgIlBhdGggd2hlcmUgdGhl IEpTT04gaXMgc3RvcmVkLgoKQnkgZGVmYXVsdCBpbiAlVEVNUE9SQVJZLURJUkVDVE9SWS9SRVYt JUpTT04tTkFNRS4iCiAgKGxldCAoKGhhc2ggKHN1YnN0cmluZyByZXZpc2lvbiAwIDYpKSkKICAg IChzdHJpbmctYXBwZW5kIHRtcCAiLyIgaGFzaCAiLSIgbmFtZSAiLSIgc3lzdGVtKSkpCgooZGVm aW5lIChmZXRjaC1qc29uIHJldmlzaW9uIHN5c3RlbSkKICAiRmV0Y2ggdGhlIEpTT04gZmlsZSBm cm9tIHRoZSBEYXRhIFNlcnZpY2UgY29ycmVzcG9uZGluZyB0byBSRVZJU0lPTi4KClN0b3JlIHRo ZSByZXN1bHQgaW4gJVRFTVBPUkFSWS1ESVJFQ1RPUlkuIgogIChsZXQqICgob3V0IChqc29uLWZp bGUgcmV2aXNpb24gc3lzdGVtKSkKICAgICAgICAgKHVybCAoanNvbi11cmwgcmV2aXNpb24gc3lz dGVtKSkpCiAgICAodXJsLWZldGNoIHVybCBvdXQpKSkKCihkZWZpbmUgKGpzb24tZmlsZSEgcmV2 aXNpb24gc3lzdGVtKQogICJSZXR1cm4gdGhlIEpTT04gZmlsZW5hbWUgY29ycmVzcG9uZGluZyB0 byBSRVZJU0lPTi4KCklmIHRoZSBKU09OIGZpbGUgZG9lcyBub3QgZXhpc3QgbG9jYWxseSwgdGhl biBmZXRjaCBpdC4iCiAgKGxldCAoKGpzb24gKGpzb24tZmlsZSByZXZpc2lvbiBzeXN0ZW0pKSkK ICAgICh3aGVuIChub3QgKGFjY2Vzcz8ganNvbiBGX09LKSkKICAgICAgKGZldGNoLWpzb24gcmV2 aXNpb24gc3lzdGVtKSkKICAgIGpzb24pKQoKDAooZGVmaW5lIChyZWFkLXBhdGhzIHJldmlzaW9u IHN5c3RlbSkKICAiUmV0dXJuIHRoZSBsaXN0IGNvcnJlc3BvbmRpbmcgdG8gUkVWSVNJT04gb2Yg YWxsIHRoZSBzdG9yZSBpdGVtcyBmcm9tIEpTT04Kbm90IG1hdGNoaW5nIGJldHdlZW4gdGhlIGJ1 aWxkIGZhcm1zLiIKICAobWFwIChsYW1iZGEgKGVsZW0pIChhc3NvYy1yZWYgZWxlbSAicGF0aCIp KQogICAgICAgKHZlY3Rvci0+bGlzdAogICAgICAgIChhc3NvYy1yZWYKICAgICAgICAgKGNhbGwt d2l0aC1pbnB1dC1maWxlIChqc29uLWZpbGUhIHJldmlzaW9uIHN5c3RlbSkganNvbi0+c2NtKQog ICAgICAgICAic3RvcmVfcGF0aHMiKSkpKQoKKGRlZmluZSAocGF0aC0+cGFja2FnZS12ZXJzaW9u IGdudS1zdG9yZS1oYXNoLXBhY2thZ2UtdmVyc2lvbikKICAiUmV0dXJuIFBBQ0tBR0UtVkVSU0lP TiBmcm9tIC9nbnUvc3RvcmUvaGFzaC1wYWNrYWdlLXZlcnNpb24uIgogIChsZXQqICgoc3BsaXQg KHN0cmluZy1zcGxpdCBnbnUtc3RvcmUtaGFzaC1wYWNrYWdlLXZlcnNpb24gI1wtKSkKICAgICAg ICAgKHBhY2thZ2UtdmVyc2lvbi1saXN0IChjZHIgc3BsaXQpKQogICAgICAgICAocGFja2FnZS12 ZXJzaW9uIChzdHJpbmctam9pbiBwYWNrYWdlLXZlcnNpb24tbGlzdCAiLSIpKSkKICAgIHBhY2th Z2UtdmVyc2lvbikpCgooZGVmaW5lIChsaXN0LXBhY2thZ2VzIHJldmlzaW9uIHN5c3RlbSkKICAo bWFwIHBhdGgtPnBhY2thZ2UtdmVyc2lvbiAocmVhZC1wYXRocyByZXZpc2lvbiBzeXN0ZW0pKSkK CgoMCihkZWZpbmUgKHNob3ctaGVscCkKICAoZGlzcGxheSAoR18gIlVzYWdlOiBndWl4IHJlcGwg LUwgL3BhdGgvdG8vbW9kdWxlcyAtLSB3ZWF0aGVyLXJlcHJvLnNjbSBbT1BUSU9OXS4uLgpUaW55 IHRvb2xzIHRvIGZpbmQgdW5yZXByb2R1Y2libGUgcGFja2FnZXMuIikpCiAgKG5ld2xpbmUpCiAg KGRpc3BsYXkgKEdfICIKICAtaCwgLS1oZWxwICAgICAgICAgICAgIGRpc3BsYXkgdGhpcyBoZWxw IGFuZCBleGl0IikpCiAgKGRpc3BsYXkgKEdfICIKICAtYywtLWNvbW1pdCAgICAgICAgICAgIHJl dmlzaW9uIHRvIGNoZWNrIikpCiAgKGRpc3BsYXkgKEdfICIKICAtcywgLS1zeXN0ZW0gICAgICAg ICAgIHN5c3RlbSB0byBjaGVjayIpKQogIChuZXdsaW5lKSkKCihkZWZpbmUgJW9wdGlvbnMKICAo bGlzdCAob3B0aW9uICcoI1xoICJoZWxwIikgI2YgI2YKICAgICAgICAgICAgICAgIChsYW1iZGEg YXJncwogICAgICAgICAgICAgICAgICAoc2hvdy1oZWxwKQogICAgICAgICAgICAgICAgICAoZXhp dCAwKSkpCiAgICAgICAgKG9wdGlvbiAnKCNcYyAiY29tbWl0IikgI3QgI2YKICAgICAgICAgICAg ICAgIChsYW1iZGEgKG9wdCBuYW1lIGFyZyByZXN1bHQgLiByZXN0KQogICAgICAgICAgICAgICAg ICAoZm9ybWF0ICN0IChHXyAiY29tbWl0OiB+QX4lIikgYXJnKQogICAgICAgICAgICAgICAgICAo YWxpc3QtY29ucyAnY29tbWl0IGFyZyByZXN1bHQpKSkKICAgICAgICAob3B0aW9uICcoI1xzICJz eXN0ZW0iKSAjdCAjZgogICAgICAgICAgICAgICAgKGxhbWJkYSAob3B0IG5hbWUgYXJnIHJlc3Vs dCAuIHJlc3QpCiAgICAgICAgICAgICAgICAgIChmb3JtYXQgI3QgKEdfICJzeXN0ZW06IH5BfiUi KSBhcmcpCiAgICAgICAgICAgICAgICAgIChhbGlzdC1jb25zICdzeXN0ZW0gYXJnIHJlc3VsdCkp KSkpCgoMCjs7Owo7OzsgRW50cnkgcG9pbnQKOzs7CgooZGVmaW5lIChtYWluIC4gYXJncykKICAo ZGVmaW5lIG9wdHMKICAgIChhcmdzLWZvbGQgYXJncyAlb3B0aW9ucwogICAgICAgICAgICAgICAo bGFtYmRhIChvcHQgbmFtZSBhcmcgbG9hZHMpCiAgICAgICAgICAgICAgICAgKGxlYXZlIChHXyAi fkE6IHVucmVjb2duaXplZCBvcHRpb25+JSIpIG5hbWUpKQogICAgICAgICAgICAgICAobGFtYmRh IChvcCBsb2FkcykgKGNvbnMgb3AgbG9hZHMpKQogICAgICAgICAgICAgICAnKCkpKQoKICAoZm9y bWF0ICN0ICJ+YVxuIiBvcHRzKQoKICAobGV0ICgoY29tbWl0IChvciAoYXNzb2MtcmVmIG9wdHMg J2NvbW1pdCkKICAgICAgICAgICAgICAgICAgICA7OyBOb3QgdGhlIGNvbW1pdCBoYXNoIGFzIGlu IEd1aXgKICAgICAgICAgICAgICAgICAgICAiNmNmMzU3OTlkZWM2MDcyM2YzN2Q4M2E1NTk0Mjlh YThiOTA0ODJkNSIpKQogICAgICAgICAgICAgICAgICAgIDs7IChtYXRjaCAoY3VycmVudC1wcm9m aWxlKQogICAgICAgICAgICAgICAgICAgIDs7ICAgKCNmICVndWl4LXZlcnNpb24pICAgO2ZvciBs YWNrIG9mIGEgYmV0dGVyIElECiAgICAgICAgICAgICAgICAgICAgOzsgICAocHJvZmlsZQogICAg ICAgICAgICAgICAgICAgIDs7ICAgIChsZXQgKChjaGFubmVsCiAgICAgICAgICAgICAgICAgICAg OzsgICAgICAgICAgIChmaW5kIGd1aXgtY2hhbm5lbD8gKHByb2ZpbGUtY2hhbm5lbHMgcHJvZmls ZSkpKSkKICAgICAgICAgICAgICAgICAgICA7OyAgICAgIChjaGFubmVsLWNvbW1pdCBjaGFubmVs KSkpKQogICAgICAgIChzeXN0ZW0gKG9yIChhc3NvYy1yZWYgb3B0cyAnc3lzdGVtKQogICAgICAg ICAgICAgICAgICAgICAoJWN1cnJlbnQtc3lzdGVtKSkpKQoKICAgIChtYXAgKGxhbWJkYSAobmFt ZSkKICAgICAgICAgICAoZm9ybWF0ICN0ICJ+YVxuIiBuYW1lKSkKICAgICAgICAgKGxpc3QtcGFj a2FnZXMgY29tbWl0IHN5c3RlbSkpKQoKICAoZXhpdCAwKSkKCihhcHBseSBtYWluIChjZHIgKGNv bW1hbmQtbGluZSkpKQoKCg== --=-=-=--