From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id iKudLpPLDmakGAEAqHPOHw:P1 (envelope-from ) for ; Thu, 04 Apr 2024 17:47:31 +0200 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id iKudLpPLDmakGAEAqHPOHw (envelope-from ) for ; Thu, 04 Apr 2024 17:47:31 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1712245651; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=yaV3INGmGTxTCKRERPhqjXqxKNNkx8GoBRqct1fDxDg=; b=MegwDQfdZrAWJwH4wI57EUyQQJHUhi24+hFW2Bif1PtW9+22gSUP29py4al4Jf26HTRCer hNzB31FFT8e9bctHc4AyMmeBQElhOITWtgzF/82dF4hViFSQJoAy0AKnSWj9++83sLAsaR BgbT+QiUkM7ca464QODLAP/rvaQILGqeyj8uJue416stpzwY+ovAGx/Zw1j/GSRHeJMP1m bFNM3JlisYObqGtY4shPaWy5gyou8OJGZDfcnjvMQzia+651adiZecRwMEA52y7ob+XXCm 3+8mbO5hQBGEEoKGEq4B/+A500f8vE2E4HVPxh6zYFquUvv7r+QE2VJlYFtr7Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1712245651; a=rsa-sha256; cv=none; b=iY+i1oQNCp9816kfmgrrAkHUi6CPHluYkZkJZNzFOTPyUEh6iC1Wcpga5crKw75GSM10Mp ECec0CXpq+9/RnB2dwptXzbZm8UrA2ZlAWl0WFZulIhxeWDU+tlWDuIHZHpTmesFObK+2g QOVBSn5oeeI+qJTz76End7cpEIWV7cilohP0pH9LRz6Dyte5KYjqZCoX7kRpsHKOtbZmeO A1LaVUlfaF+wjNmmvKkibBy0ZMw8TdUYe6voq+5HxwsjuwWwLEv6dTajE/qgM+lWH5r69A ecwQ/W9jTb1CXPaQdNdgIcJOswPEhNoWhyK88rgbRjZNSjVT0HJyv/z5cKfZIA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A03F918892 for ; Thu, 4 Apr 2024 17:47:30 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rsPJ6-0001kS-Vw; Thu, 04 Apr 2024 11:47:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rsPJ5-0001kH-JT for guix-devel@gnu.org; Thu, 04 Apr 2024 11:47:15 -0400 Received: from ns13.heimat.it ([46.4.214.66]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rsPJ3-0002oc-4M for guix-devel@gnu.org; Thu, 04 Apr 2024 11:47:15 -0400 Received: from localhost (ip6-localhost [127.0.0.1]) by ns13.heimat.it (Postfix) with ESMTP id 1679130081F for ; Thu, 4 Apr 2024 15:47:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at ns13.heimat.it Received: from ns13.heimat.it ([127.0.0.1]) by localhost (ns13.heimat.it [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4sQhK-KnANa5 for ; Thu, 4 Apr 2024 15:47:09 +0000 (UTC) Received: from bourrache.mug.xelera.it (unknown [93.56.171.217]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by ns13.heimat.it (Postfix) with ESMTPSA id 06B6B30081A for ; Thu, 4 Apr 2024 15:47:09 +0000 (UTC) Received: from roquette.mug.biscuolo.net (roquette [10.38.2.14]) by bourrache.mug.xelera.it (Postfix) with SMTP id 14454307E2AF for ; Thu, 4 Apr 2024 17:47:08 +0200 (CEST) Received: (nullmailer pid 25632 invoked by uid 1000); Thu, 04 Apr 2024 15:47:07 -0000 From: Giovanni Biscuolo To: Guix Devel Subject: Re: backdoor injection via release tarballs combined with binary artifacts (was Re: Backdoor in upstream xz-utils) In-Reply-To: <8734s1mn5p.fsf@xelera.eu> Organization: Xelera.eu References: <87ttkon4c4.fsf@protonmail.com> <8734s1mn5p.fsf@xelera.eu> Date: Thu, 04 Apr 2024 17:47:06 +0200 Message-ID: <87zfu9ku4l.fsf@xelera.eu> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=46.4.214.66; envelope-from=g@xelera.eu; helo=ns13.heimat.it X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -8.46 X-Spam-Score: -8.46 X-Migadu-Queue-Id: A03F918892 X-Migadu-Scanner: mx12.migadu.com X-TUID: 5nEyUfs0coyq --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, a couple of additional (IMO) useful resources... Giovanni Biscuolo writes: [...] > Let me highlight this: =C2=ABIt is pragmatically impossible [...] to peer > review a tarball prepared in this manner.=C2=BB > > There is no doubt that the release tarball is a very weak "trusted > source" (trusted by peer review, not by authority) than the upstream > DVCS repository. This kind of attack was described by Daniel Stenberg in his =C2=ABHOWTO backdoor curl=C2=BB article in 2021.03.30 as "skip-git-altogether" method: https://daniel.haxx.se/blog/2021/03/30/howto-backdoor-curl/ =2D-8<---------------cut here---------------start------------->8--- The skip-git-altogether methods As I=E2=80=99ve described above, it is really hard even for a skilled devel= oper to write a backdoor and have that landed in the curl git repository and stick there for longer than just a very brief period. If the attacker instead can just sneak the code directly into a release archive then it won=E2=80=99t appear in git, it won=E2=80=99t get tested an= d it won=E2=80=99t get easily noticed by team members! curl release tarballs are made by me, locally on my machine. After I=E2=80= =99ve built the tarballs I sign them with my GPG key and upload them to the curl.se origin server for the world to download. (Web users don=E2=80=99t actually hit my server when downloading curl. The user visible web site and downloads are hosted by Fastly servers.) An attacker that would infect my release scripts (which btw are also in the git repository) or do something to my machine could get something into the tarball and then have me sign it and then create the =E2=80=9Cperf= ect backdoor=E2=80=9D that isn=E2=80=99t detectable in git and requires someone= to diff the release with git in order to detect =E2=80=93 which usually isn=E2=80=99t d= one by anyone that I know of. [...] I of course do my best to maintain proper login sanitation, updated operating systems and use of safe passwords and encrypted communications everywhere. But I=E2=80=99m also a human so I=E2=80=99m boun= d to do occasional mistakes. Another way could be for the attacker to breach the origin download server and replace one of the tarballs there with an infected version, and hope that people skip verifying the signature when they download it or otherwise notice that the tarball has been modified. I do my best at maintaining server security to keep that risk to a minimum. Most people download the latest release, and then it=E2=80=99s enough if a subset check= s the signature for the attack to get revealed sooner rather than later. =2D-8<---------------cut here---------------end--------------->8--- Unfortunately Stenberg in that section misses one attack vector he mentioned in a previous article section named "The tricking a user method": =2D-8<---------------cut here---------------start------------->8--- We can even include more forced =E2=80=9Cconvincing=E2=80=9D such as direct= threats against persons or their families: =E2=80=9Cpush this code or else=E2=80=A6= =E2=80=9D. This way of course cannot be protected against using 2fa, better passwords or things like that. =2D-8<---------------cut here---------------end--------------->8--- ...and an attack vector involving more subltle ways (let's call it distributed social engineering) to convince the upstream developer and other contributors and/or third parties they need a project co-maintainer authorized to publish _official_ release tarballs. Following Stenberg's attacks classification, since the supply-chain attack was intended to install a backdoor in the _sshd_ service, and _not_ in xz-utils or liblzma, we can classify this attack as: skip-git-altogether to install a backdoor further-down-the-chain, precisely in a _dependency_ of the attacked one, durind a period of "weakness" of the upstream maintainers Stenberg closes his article with this update and one related reply to a comment: =2D-8<---------------cut here---------------start------------->8--- Dependencies Added after the initial post. Lots of people have mentioned that curl can get built with many dependencies and maybe one of those would be an easier or better target. Maybe they are, but they are products of their own individual projects and an attack on those projects/products would not be an attack on curl or backdoor in curl by my way of looking at it. In the curl project we ship the source code for curl and libcurl and the users, the ones that builds the binaries from that source code will get the dependencies too. [...] Jean Hominal says:=20 April 1, 2021 at 14:04=20 I think the big difference why you =E2=80=9Cmissed=E2=80=9D dependencies a= s an attack vector is because today, most application developers ship their dependencies in their application binaries (by linking statically or shipping a container) =E2=80=93 in such a case, I would definitely count an attack on such a dependency, that is then shipped as part of the project=E2=80=99s artifacts, as a successful attack on the project. However, as you only ship a source artifact =E2=80=93 of course, dependenc= ies *are* out of scope in your case. Daniel Stenberg says:=20 April 1, 2021 at 15:05=20 Jean: Right. I don=E2=80=99t want to dismiss the risk or the danger of an attack to a curl dependency. However, it is not possible for me or the curl project to keep them safe! =2D-8<---------------cut here---------------end--------------->8--- That lets a number of open questions about some developers attitude towards _distributing_ their software, but it's off-topic here IMO. Anyway, let me highlight, again, the "pragmatically impossible peer review of release tarballs" argument; Stenberg says: =C2=ABthe =E2=80=9Cper= fect backdoor=E2=80=9D that isn=E2=80=99t detectable in git and requires someone= to diff the release with git in order to detect =E2=80=93 which usually isn=E2=80=99t d= one by anyone that I know of.=C2=BB [...] > Is it possible to enhance our build-system(s) (e.g. gnu-build-system) so > thay can /ignore/ pre-built .m4 or similar script and rebuild them > during the build process? There is a related security issue for PHP [1], with an interesting thread on the php.internals mailing list (via externals.io [2]): =2D-8<---------------cut here---------------start------------->8--- Consider removing autogenerated files from tarballs [...] I believe that it would be a good idea to remove the huge attack surface offered by the pre-generated autoconf build scripts and lexers, offered in the release tarballs. [...] this injection mode makes sense, as extra files in the tarball not present in the git repo would raise suspicions, but machine-generated configure scripts containing hundreds of thousands of lines of code not present in the upstream VCS are the norm, and are usually not checked before execution. [...] Specifically in the case of PHP, along from the configure script, the tarball also bundles generated lexer files which contain actual C code, which is an additional attack vector [...] To prevent attacks from malevolent/compromised RMs, I propose completely removing all autogenerated files from the release tarballs, and ensuring their content exactly matches the content of the associated git tag [...] Of course this means that users will have to generate the build scripts when compiling PHP, as when installing PHP from the VCS repo. [...] Distros like arch linux already re-generate the configure scripts from scratch, but I believe that no distinction should be made, everyone should get a tarball containing only the bare source code, without leaving to the user the choice to re-generate the build files, or use a potentially compromised build script. [...] The current standard way of distributing generated configure files in tarballs is precisely what allowed the xz supply chain attack to go unnoticed for so long. I strongly believe all projects using autotools, including PHP, should switch away from this "standard" way of doing things. [...] when a user downloads a source tarball, there's a false sense of security rooted in the mistaken belief that the source code in the tarball matches the one distributed in the VCS, but in reality, the tarball also contains potentially malicious semi-compiled blobs, not present in the VCS. =2D-8<---------------cut here---------------end--------------->8--- Are really "configure scripts containing hundreds of thousands of lines of code not present in the upstream VCS" the norm? If so, can we consider hundreds of thousand of lines of configure scripts and other (auto)generated files bundled in release tarballs "pragmatically impossible" to be peer reviewed? Can we consider that artifacts as sort-of-binary and "force" our build-systems to _regenerate_ *all* them? ...or is it better to completely avoid release tarballs as our sources uris? [...] Thanks, Gio' [1] https://github.com/php/php-src/issues/13838 [2] https://externals.io/message/122811 =2D-=20 Giovanni Biscuolo Xelera IT Infrastructures --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJABAEBCgAqFiEERcxjuFJYydVfNLI5030Op87MORIFAmYOy3sMHGdAeGVsZXJh LmV1AAoJENN9DqfOzDkSuMoP/juA1NU/KD5QFu3BzZIE+S/tFrPosvh7f17gWLbV tH5lGcXf4KpU4/YRQGIiHdMYUcuNmdzCorJR3qdlcObaidWeRrncGt90nPydVIN+ BpG/TgdD0xccuqvvvP/ZSRw+l1dQhL995u5abaxcSd7IwfKQvAHnLyBAm+Av5yrG 29IRXNESRLaAxN6+QC7QCoevJhTUPhzXu/Qi1cHc0yMgKVFq4t2lu+aPkNPky+Rq dU1dg9tPpV7HiU+/Px2H3ehor3R9awGqiKrF8gYRlWnGRvMl1LGSmeV7DX5XGaDy FmQxk+Gtj4UdkHSjYJeDpUWUTeaxk7uk5c5/zP4WEbSdeKjuRwuWpIZ8dZZC1eTl OVJLP6tNFNbevjKqJgFpxHmBLO+5yCiz7dR4wMKo6n//Ghg41G7hI0wMW+LmHwZ0 OkgQ6rYQaSLFwN0sp4/NXcRBR0pAlSGVRsZG46Hxk168+mAOqw8v/QWrRKXqYZa0 Py9cSrn3k819vScFlHgb4Ru63zKsoCfe9Pi1Emj5bkO7jLbUXopr0LDjKhyczLH7 3Qf/+Lv9kBjT9Nq/B3fQ16fYKxnf9sLVySrxqiCjhoiMhYDxBz4eDx9KWv+Tp3aR MyQmsEEtNzKMYPvtKu/ptTJp8dDfDOVjcrkehAIfyAJElrXZ8OGaK7B8Oi2HfRny iCJl =d19a -----END PGP SIGNATURE----- --=-=-=--