From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QIvKBes6Q1+rIAAA0tVLHw (envelope-from ) for ; Mon, 24 Aug 2020 03:58:35 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 4FJxAes6Q1+iIwAAbx9fmQ (envelope-from ) for ; Mon, 24 Aug 2020 03:58:35 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8D1809404D2 for ; Mon, 24 Aug 2020 03:58:34 +0000 (UTC) Received: from localhost ([::1]:56058 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kA3cv-0003cI-If for larch@yhetil.org; Sun, 23 Aug 2020 23:58:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kA3cm-0003bt-0R for guix-devel@gnu.org; Sun, 23 Aug 2020 23:58:24 -0400 Received: from fsfla.org ([217.69.89.140]:48224) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kA3cj-0007UP-SN for guix-devel@gnu.org; Sun, 23 Aug 2020 23:58:23 -0400 Received: from free.home (unknown [201.82.51.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by fsfla.org (Postfix) with ESMTPSA id DDAD81DEF02; Mon, 24 Aug 2020 03:58:18 +0000 (UTC) Received: from livre.home (livre.home [172.31.160.2]) by free.home (8.15.2/8.15.2) with ESMTPS id 07O3wA2Q117696 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Aug 2020 00:58:10 -0300 From: Alexandre Oliva To: Mark H Weaver Subject: Re: Linux-libre 5.8 and beyond Organization: Free thinker, not speaking for FSF Latin America References: <87d03vv0nm.fsf@netris.org> <875z9kv41h.fsf@netris.org> Date: Mon, 24 Aug 2020 00:58:10 -0300 In-Reply-To: <875z9kv41h.fsf@netris.org> (Mark H. Weaver's message of "Sat, 15 Aug 2020 02:03:27 -0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.84 Received-SPF: none client-ip=217.69.89.140; envelope-from=lxoliva@fsfla.org; helo=fsfla.org X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/23 23:45:31 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org, Jason Self Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: -1.01 X-TUID: cazLMNWUKAlT On Aug 15, 2020, Mark H Weaver wrote: > Alexandre Oliva wrote: >> On Aug 12, 2020, Mark H Weaver wrote: >> >>>>> It may be useful for users with newer hardware devices, which are >>>>> not yet well supported by the latest stable release, to use an >>>>> arbitrary commit from either Linus' mainline git repository or some >>>>> other subsystem tree. >>>> >>>> The cleaning up scripts are version-specific and won't work on an >>>> "arbitrary commit from Linus's mainline git repository" (i.e., someone >>>> wanting to get today's most recent commit going into 5.9.) The scripts >>>> would fall over and die in such a scenario, >> >>> Okay, perhaps this was wishful thinking on my part. >> >> Yup. If you ran a deblob-check in verify mode on the resulting >> tarballs, you'd see how error-prone this is. You'd at least stop >> non-Free code from silently sneaking in and finding its way into running >> on users' machines. That's the *least* someone who runs the >> deblob-scripts on their own should do to smoke-test the result WRT >> *known* freedom issues. > What is this "verify mode" that you're referring to, and where is it > documented? I'm talking about the --list-blobs (default) option of deblob-check, that tests whether an input file (source file, patch file, or tarball) contains any suspicious patterns. Running deblob-check with --help prints a significant amount of documentation, though it is mostly aimed at the internal purposes that the scripts serve. The cleaning up scripts are not really meant to be blindly used by third parties to clean up anything but releases they're associated with; they're provided for documentation and transparency purposes, but they're not even something whose existence you should count on. E.g., once we realize the long-term vision of having a git repo with the entire history, manually cleaned-up, there won't be a script to clean things up any more, though there will surely still be something to help us identify anything that needs cleaning up. > The word "verify" does not occur in either of the deblob > scripts that I know about That's what the 'check' in deblob-check stands for. Originally, it would only scan for blobs. Later on, it was extended with other actions for use in cleaning up. > I don't see anything like a verification mode mentioned > in the options documented at the top of those two scripts. Indeed, deblob-, which is what you use, and deblob-check, as used by it, do not perform any verification whatsoever. They're not meant to. They automate and document what we intend to clean up. The verifications are steps we take once we have a candidate release, that well us whether or not it's fit for release. If it isn't, we adjust the scripts and start over. You'd have to run deblob-check linux-libre--guix.tar on the cleaned up tarball to check that none of the suspicious patterns known by deblob-check have survived in the resulting tarball. It would have caught the errors that Vagrant hit the other day, and it would have reported the deblobbing errors you'd have got this week had you not waited for the updated scripts. Running this script in -B or -C modes is part of our development process for new releases, and it is also one of our safety nets to stop us from releasing non-Free Software: we run it for every release before putting it out. > For the record, it was not my intent to skip any automated checking > provided by these scripts. I understand it was not your intent, but using the scripts in environments it wasn't tested, with upstream releases or commits it wasn't meant for, the expectation that it will do the job you wish without any of the verification steps we perform is misplaced. > If we're running the scripts in a suboptimal > way, please tell me a better way. > FYI, right now we're simply running the main 'deblob-' script > with no arguments in the unpacked Linux source directory, with the > corresponding 'deblob-check' script in $PATH and $PYTHON pointing to > python 2.x. If 'deblob-' exits abnormally or with a non-zero > result, the Guix build process fails. > Last I checked, 'deblob-check' was certainly being run by > 'deblob-' as a subprocess, because I had to make several > substitutions of hard-coded paths before it would work in Guix > (e.g. /bin/sed and /usr/bin/python). The expected use of the scripts, for people who wish to verify that our releases have been cleaned up as specified in the scripts, is to do just what you do, and then compare the resulting source tree with that of our release. If they match, you know we haven't sneaked in any unintended changes. If they don't, something went wrong on either end. Given our amount of experience and automation in the release and verification processes, that scan the resulting source tree and also compare the changes with those made by an earlier known good recent release, a platform-specific bug in the underlying tools, an unexpected change to regexp engines (as in some recent version of python3), the use of mismatched scripts are more likely sources of differences than our failing to notice an unexpected change on our ends. Now, if you wanted to use the scripts for purposes other than verification, e.g., to clean up releases before we check them, or even after we put them out but without any attempt to verify that the result you get is indeed what we put out, you should take responsibility for verifying the releases at least as much as we do, otherwise any freedom issues arising from your not catching a problem we would have caught would unfairly reflect negatively on our project. -- Alexandre Oliva, happy hacker https://FSFLA.org/blogs/lxo/ Free Software Activist GNU Toolchain Engineer