From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id pc8NN45PWmDTOgAA0tVLHw (envelope-from ) for ; Tue, 23 Mar 2021 20:29:02 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 8Dg7Mo5PWmCPQgAAbx9fmQ (envelope-from ) for ; Tue, 23 Mar 2021 20:29:02 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E7ADED72D for ; Tue, 23 Mar 2021 21:29:01 +0100 (CET) Received: from localhost ([::1]:57908 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOne9-0008NY-1Q for larch@yhetil.org; Tue, 23 Mar 2021 16:29:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48522) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOmH0-0003fC-LM for bug-guix@gnu.org; Tue, 23 Mar 2021 15:01:05 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:49694) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOmGz-0004HW-Ni for bug-guix@gnu.org; Tue, 23 Mar 2021 15:01:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOmGz-00072P-KZ; Tue, 23 Mar 2021 15:01:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#47229: Hardlink mitigation limits Resent-From: Nathan Nye Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 19:01:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47229 X-GNU-PR-Package: guix X-GNU-PR-Keywords: security fixed To: 47229@debbugs.gnu.org Received: via spool by 47229-submit@debbugs.gnu.org id=B47229.161652603027007 (code B ref 47229); Tue, 23 Mar 2021 19:01:01 +0000 Received: (at 47229) by debbugs.gnu.org; 23 Mar 2021 19:00:30 +0000 Received: from localhost ([127.0.0.1]:33007 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOmGQ-00071U-I1 for submit@debbugs.gnu.org; Tue, 23 Mar 2021 15:00:30 -0400 Received: from mail-qk1-f182.google.com ([209.85.222.182]:36743) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOlbm-0005zu-Gb for 47229@debbugs.gnu.org; Tue, 23 Mar 2021 14:18:28 -0400 Received: by mail-qk1-f182.google.com with SMTP id c4so15347973qkg.3 for <47229@debbugs.gnu.org>; Tue, 23 Mar 2021 11:18:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=whitebeamsec.com; s=google; h=references:to:from:organization:subject:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=Y9LUfY/800DANE7OF7b2/GwfJWxirD6xrL/HqcZuMb8=; b=lrLrv8QU1v6AsI8gFF0ISARvVqmjlPXvuLYOFGvLAY/BAIE8VUPxLEeKTAYtMON3nt HMAURVn09cEpbGXZ2dSdxElyN95k6tfoAYHMsTjQDGW5a9zfOd8NltM2FbbjduWglW1M 4i1JWzUURoE97QR2VBAm0rAsP38XDSnWb6COKWiCuqDSHQ7e3OCu92fnwANvpESZlaS1 NwcX+xblWKEH2LYWOOZGShm427mO/jBzyChR6z9lqLQrUlJlwwko9UUI/uI6j50sBdcM kxyFj41yS36uHvA7X8y40bk7ugDtaPsR0RzT3/hcBYtwF34D32Cs8CPF4IY3UvS6s0gu xZeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:to:from:organization:subject :message-id:date:user-agent:mime-version:in-reply-to :content-language; bh=Y9LUfY/800DANE7OF7b2/GwfJWxirD6xrL/HqcZuMb8=; b=U2jadky+Jmr6+/cvcZ+q5C5o5lnAMaZh90KiIGsysMkC0q7C1bakzxTyy770/tiRGg bomEz2eQDZIr5YVWujkHVHGzO0qiuEuybcBxldSsvPtP28FlAwYKyufiEKtvs/am2qB6 P2vXDgNUDsOxjLpPY6D+ul+n1zq1XzDRkbjxPeZkVjh6G+drFTfh0OW9AnQC9mthYpUa pVANh/lbfORdApVfH952jqo+DZ5CZeYC44B7PPvH3NZVpFW0m83Qar8U5Sp+z6I5ii1B s7/cSRX0UGegBBnu6fxKBHORZGo0CAOrTC3C1OWiVpayo24hzEmYjsFGTDjCNPg7qWWx SKZA== X-Gm-Message-State: AOAM531iiH8fr+pFW3VfDojweT7Sk2ZC5bfAun6pTTmF3o1sRQmCvf6H umXG1XZAIkN7GmdIoYeTtx/AJPiy/w89Momi0Ls= X-Google-Smtp-Source: ABdhPJySEJkVYzuUu019qNtK3lARwkZKUAff4taLD/7eRcy6Tkkzyv1X2+VMVN0D7WhB9QyIgxkX/w== X-Received: by 2002:a37:a9cf:: with SMTP id s198mr6432295qke.143.1616523500268; Tue, 23 Mar 2021 11:18:20 -0700 (PDT) Received: from [10.0.2.15] (93-94-240-221.mobileaccess.monzoon.net. [93.94.240.221]) by smtp.gmail.com with ESMTPSA id z5sm13796671qkz.2.2021.03.23.11.18.18 for <47229@debbugs.gnu.org> (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 23 Mar 2021 11:18:19 -0700 (PDT) References: From: Nathan Nye Organization: WhiteBeam Security, Inc. X-Forwarded-Message-Id: Message-ID: <8f95179a-5574-98bd-c44e-f5ee74638dc3@whitebeamsec.com> Date: Tue, 23 Mar 2021 14:18:14 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------56E58B699AC93465FA9C8949" Content-Language: en-US X-Mailman-Approved-At: Tue, 23 Mar 2021 15:00:25 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Mailman-Approved-At: Tue, 23 Mar 2021 16:28:53 -0400 X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616531342; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=Y9LUfY/800DANE7OF7b2/GwfJWxirD6xrL/HqcZuMb8=; b=FM3z+MK5e7HflCbge/0qmxjlJkQFNy9NB2/VX/6ybmj/9G1y2lY9UpUvbQiKylkfgmREsj fJBIn+9lqic/F/GGTCMA39r2+EPDAQykqeMdsoKIwKNVXMffG3Theeadw7ECZclypZFB+J 9438x2USxWD3Fm9gfBsbBNVxzzC5MgfMpoOAX0Nb2c/au2Rsw6oTNBBH+uQn2Xn1ib5sXl hXSx9RK3iOLSidkU9tDkacotQis3BW1yGVPiiefvpJSJv4ympRyTm/Il+uYNg8KbS3FQng RxIQKplWNQozKzRu6Km7RRR1T+ZWvNofRbvmTw/Gp+WM7xF2ipp0Ey3SLTavuQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616531342; a=rsa-sha256; cv=none; b=cSFvrFqiv4K33Ah9x6YsueDSziAKBEoUtJ3Nhp0aux7OOWzLB+z7FFI910Bt4USpK8vd38 TpTFM0ywTDzbLrCs/8KLlL3LfyCVVMLe/oQGLX68FIP8all6njZLz4okAOKLcS2HpOlEoq VocUlh/pMaTudFFWBtMCpe/uLarMQ4TtWK36kkMEIYNe60IcDl7mO6C3tNTefYLg57UDKl JL0beA6g8L4pqYJZZDTqL83F/lqRVmGGWLePAdWKZu2FlADYv1V6eYx9lH53nDyznnB7xF bvUV60T8Y4s70lkmk7v/eVjibeUg1WfAveOKi9CNwgLEHKaduRTJLJKuh66MpQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=whitebeamsec.com header.s=google header.b=lrLrv8QU; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -1.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=whitebeamsec.com header.s=google header.b=lrLrv8QU; dmarc=fail reason="SPF not aligned (relaxed)" header.from=whitebeamsec.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: E7ADED72D X-Spam-Score: -1.32 X-Migadu-Scanner: scn0.migadu.com X-TUID: MtJ10D1onRaH This is a multi-part message in MIME format. --------------56E58B699AC93465FA9C8949 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hello, I'm sharing here for future reference why protected hardlinks alone did=20 not mitigate the recent LPE security advisory, pre-patch: "The reasons why are lines 2633 and 2637 of nix/libstore/build.cc: * https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc= #n2633 * https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc= #n2637 When a package fails to build and the keep failed flag is set=20 (-K/--keep-failed), it runs a recursive chown on the build directory=20 (which is writable following guixbuilder01 changing the permissions to=20 777). It starts at the top level and chowns downwards. The first important thing to notice here is that at any point (even=20 pre-chown) the build user has been compromised. The build user can write = a SUID /bin/sh to the build path, and because a normal user can traverse = into the directory before and during the chown, they can run a SUID=20 shell (allowing them to become guixbuilder01 even after the build user=20 processes are terminated). Becoming the build user allows multiple paths = to privilege escalation, but in this scenario we have faster ways of=20 becoming root. Moving on to getting root, we're choosing not to use a hardlink to show=20 why it isn't necessary. Instead, we create a directory under the build=20 directory with thousands of sequentially named files, the final entry=20 being "passwd" or "shadow". Then we terminate the build and watch for=20 the first entry to be chowned to our user ID (possibly with the inotify=20 API). This way, we have opened a lengthy window of time where it is=20 enumerating over a list of file paths in our chosen directory and=20 chowning each of them. Now we can execute our TOCTOU race condition=20 vulnerability. At the time of check (TOC), the guix-daemon has a list of file paths to=20 chown under what it assumes is a regular directory (because it ran=20 S_ISDIR on the directory). But we can swap out the directory from under=20 it with a symlink to /etc (most efficiently with renameat2() and using=20 the RENAME_EXCHANGE flag to atomically exchange the paths). At the time=20 of use (TOU) lchown() only checks if the file /itself/ that is being=20 chowned is a symlink, not if the path components are, as can be=20 demonstrated with Python: $ mkdir td;touch td/tf;python3 -c 'import os;os.lchown("/home/example/td/= tf", 1000, 4)';ls -lahtrd td td/tf -rw-rw-r-- 1 example adm=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 Mar 19 = 19:20 td/tf drwxrwxr-x 2 example example 4.0K Mar 19 19:20 td $ rm -rf td $ mkdir td; ln -s td td2;touch td2/tf;python3 -c 'import os;os.lchown("/h= ome/example/td2/tf", 1000, 4)';ls -lahtrd td2 td2/tf lrwxrwxrwx 1 example example 2 Mar 19 19:21 td2 -> td -rw-rw-r-- 1 example adm=C2=A0=C2=A0=C2=A0 0 Mar 19 19:21 td2/tf So lchown can blindly chown /etc/passwd to our user by following the=20 directory symlink and subsequently verifying that passwd itself is not a = symlink. I hope this explains the TOCTOU race condition and why=20 protected hardlinks help (forcing an attacker to get root using this=20 race condition), but they are not a solution to the problem (alone)." - Nathan --------------56E58B699AC93465FA9C8949 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Hello,

I'm sharing here for future reference why protected hardlinks alone did not mitigate the recent LPE security advisory, pre-patch:

"The reasons why are lines 2633 and 2637 of nix/libstore/build.cc:

When a package fails to build and the keep failed flag is set (-K/--keep-failed), it runs a recursive chown on the build directory (which is writable following guixbuilder01 changing the permissions to 777). It starts at the top level and chowns downwards.

The first important thing to notice here is that at any point (even pre-chown) the build user has been compromised. The build user can write a SUID /bin/sh to the build path, and because a normal user can traverse into the directory before and during the chown, they can run a SUID shell (allowing them to become guixbuilder01 even after the build user processes are terminated). Becoming the build user allows multiple paths to privilege escalation, but in this scenario we have faster ways of becoming root.

Moving on to getting root, we're choosing not to use a hardlink to show why it isn't necessary. Instead, we create a directory under the build directory with thousands of sequentially named files, the final entry being "passwd" or "shadow". Then we terminate the build and watch for the first entry to be chowned to our user ID (possibly with the inotify API). This way, we have opened a lengthy window of time where it is enumerating over a list of file paths in our chosen directory and chowning each of them. Now we can execute our TOCTOU race condition vulnerability.

At the time of check (TOC), the guix-daemon has a list of file paths to chown under what it assumes is a regular directory (because it ran S_ISDIR on the directory). But we can swap out the directory from under it with a symlink to /etc (most efficiently with renameat2() and using the RENAME_EXCHANGE flag to atomically exchange the paths). At the time of use (TOU) lchown() only checks if the file itself that is being chowned is a symlink, not if the path components are, as can be demonstrated with Python:

$ mkdir td;touch td/tf;python3 -c 'import os;os.lchown("/home/example/td/tf", 1000, 4)';ls -lahtrd td td/tf
-rw-rw-r-- 1 example adm       0    Mar 19 19:20 td/tf
drwxrwxr-x 2 example example   4.0K Mar 19 19:20 td
$ rm -rf td
$ mkdir td; ln -s td td2;touch td2/tf;python3 -c 'import os;os.lchown("/home/example/td2/tf", 1000, 4)';ls -lahtrd td2 td2/tf
lrwxrwxrwx 1 example example 2 Mar 19 19:21 td2 -> td
-rw-rw-r-- 1 example adm     0 Mar 19 19:21 td2/tf

So lchown can blindly chown /etc/passwd to our user by following the directory symlink and subsequently verifying that passwd itself is not a symlink. I hope this explains the TOCTOU race condition and why protected hardlinks help (forcing an attacker to get root using this race condition), but they are not a solution to the problem (alone)."

- Nathan

--------------56E58B699AC93465FA9C8949--