From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id OAlCHQSG02HZeQAAgWs5BA (envelope-from ) for ; Tue, 04 Jan 2022 00:25:56 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 8J66GgSG02FHVwEA9RJhRA (envelope-from ) for ; Tue, 04 Jan 2022 00:25:56 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2021247601 for ; Tue, 4 Jan 2022 00:25:56 +0100 (CET) Received: from localhost ([::1]:36482 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n4WiB-00039t-61 for larch@yhetil.org; Mon, 03 Jan 2022 18:25:55 -0500 Received: from eggs.gnu.org ([209.51.188.92]:60622) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n4Taq-0000tG-TI for guix-devel@gnu.org; Mon, 03 Jan 2022 15:06:08 -0500 Received: from [2607:f8b0:4864:20::b29] (port=42631 helo=mail-yb1-xb29.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n4Tam-0007Yt-0X for guix-devel@gnu.org; Mon, 03 Jan 2022 15:06:08 -0500 Received: by mail-yb1-xb29.google.com with SMTP id m19so84391760ybf.9 for ; Mon, 03 Jan 2022 12:06:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google-2018; h=mime-version:from:date:message-id:subject:to:cc; bh=NW6wKcCcOxeYBo8Yjn3f0pG6ieVh2v4pdJDXVHfbd78=; b=OX7lTg+EJZpIalo/aJVaN5YXMuyc0mJJM3HpU5JC6AQOvMIydghHQmvXkwNdoPfE4D 2apW41XuIdLbUBVs/Z3bIQnvn0iDVf4GWLYigVUj78l+RVwDyiEewE5HFd+Z9+jjO+Pi +RwnLBzbJwTXD4BIvGtdCNEd2O46BMRs6PMQnMY6J2MPMsDNHSDSGZizJnZvagmwGaqQ 4kekSkA3BAWjSNEEKbxJHjfKp5cqhXaTllqhihT6od3E/6xa9pAJjxFUXqS74I8MnM2Q b3YIcafVd08BcPkt1cBxqtteXCl331iilDP4Mqe0Rc7/OcCINB8BSaJ3h9jg68w71zGA gApw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=NW6wKcCcOxeYBo8Yjn3f0pG6ieVh2v4pdJDXVHfbd78=; b=qmWrFlsZo2Qwne9WJ7LVKWUG0+qeLVOK3Ysam1/hwggGMPiHnWuImP3R2CVMh7iZri UKNIp9bORjZjCK7ukFFxuSnrwLRR5UiWO35F3acsPad/eaiDrY4vOu1IcV9n8PGmXTRV +LbacGl87KXPjqWtu+8u5/Lz3hVYuEzXaA6Fsh/iB0fPGKS32mtlvARbu4d2qpGoHelI nZdH5Ye0kAx8UVFuNow/yYGlc/PZ4WS75FBkAn2UUjDGeM7XlqfeOszJwmQro2rmEy8A bS+/PjZU4iYAAZY5Z5tvxmsmet9NZrnd992lpqypi5AMM+lds61LKfHpco+uymr6o+Qv NF+Q== X-Gm-Message-State: AOAM533yaywm0guioLOI5JGoNb4s6c6a6gfeAsuLf6VD+yBSFyQivFpJ 3jI1sIZIftgpD0MKDPVxaM25piYmt7Ya3bpNuUEpy2qgjJ1HNw== X-Google-Smtp-Source: ABdhPJx6NXi7cFUTki3/Y8IlKi68Ssw/QQCcGzEQpqjlyM5AqgXqMlQRJ8c2Nj0ksSsLZ4YT345BDuNV9J3pxdSEi+M= X-Received: by 2002:a25:b981:: with SMTP id r1mr50476870ybg.520.1641240361625; Mon, 03 Jan 2022 12:06:01 -0800 (PST) MIME-Version: 1.0 From: Farid Zakaria Date: Mon, 3 Jan 2022 12:05:51 -0800 Message-ID: Subject: Alternative solution to stat storm problem To: guix-devel@gnu.org Cc: "Scogland, Tom" , Carlos Maltzahn Content-Type: text/plain; charset="UTF-8" X-Host-Lookup-Failed: Reverse DNS lookup failed for 2607:f8b0:4864:20::b29 (failed) Received-SPF: pass client-ip=2607:f8b0:4864:20::b29; envelope-from=fmzakari@ucsc.edu; helo=mail-yb1-xb29.google.com X-Spam_score_int: 6 X-Spam_score: 0.6 X-Spam_bar: / X-Spam_report: (0.6 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Mon, 03 Jan 2022 18:25:45 -0500 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641252356; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=NW6wKcCcOxeYBo8Yjn3f0pG6ieVh2v4pdJDXVHfbd78=; b=rxNLbBRrz/lweVc0CMJoNzY01MPJO3H15F5VRhy+HdBfhs6f9beubE7rckKb/T6WHot6eP R7aT5MuavwhXHvIBENV8UK4QlpSWRNVWrw1uQmrrpiIWE3akT7EtcsYr6UgGpB38PK6zvL AL1FJbTfPMuwPJcBc5kV1ujh7WYshOhN1lAZw3BjAguwiVBlnBq7kAimECfasYnRRTp7z6 1nbKg0MzQJXQetQaSonk3Cb+lvK6NVKu6rc6ScoqiigHbAhJh3XDSWXc9AGPlM1FCGr8lQ mObe90hEL764XXJU0CjEbizpO2tTYGAwgm6LLceUHRta/OONlNWEF1Eq28gZrg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641252356; a=rsa-sha256; cv=none; b=qF+zV+UZvuR75m/PKTZhYA+7IdFBZzETcaZI2Xu0CLVn1Bg0lDW14nBNz26tp+EwiWn55A ME6pBqJvoU6P+8gM4H7Xer0ukTOt9y8Jc/uCZ0+832hl/HiIaTzkpX3zLB6BArnI7UVR/j /SbcniyqQ65mif5Ed5gBmtvRbGTpqKV9U21WRe+NbcDcmo9RKMSWiZk2n+LUDvc/sOSMB9 Q1EiQ27aJWCREKATZVyJic+P2egn1lZAWQISUWDFckFRd1xoQ8TMT0ev7sO6ZL96WEZ01B E2WDCtVikMLSapYcbViydaLrtgnmUseWp8aK1JYZYufsH1mx2IKKWBPdRrl7Aw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=ucsc.edu header.s=ucsc-google-2018 header.b=OX7lTg+E; dmarc=pass (policy=quarantine) header.from=ucsc.edu; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -6.29 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=ucsc.edu header.s=ucsc-google-2018 header.b=OX7lTg+E; dmarc=pass (policy=quarantine) header.from=ucsc.edu; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 2021247601 X-Spam-Score: -6.29 X-Migadu-Scanner: scn1.migadu.com X-TUID: xL8oqMU8eVqn Hi! I was very inspired by the blog post on a per-application ld.so.conf.cache to solve the stat-storm problem[1]. I wanted to share here another approach I am pursuing and seek to eventually try to merge into NixOS however I thought starting a discussion here on Guix would be fruitful since it was the genesis of the idea. I have written a tool _shrinkwrap_ [2] that takes all transitive dynamic shared object dependencies (only those listed in DT_NEEDED) and turns them into an absolute path. This has the same result as caching the entries and avoids the unnecessary failed attempts at trying each RUNPATH entry. Using the same demo application _emacs_ shows as much as well: $strace -e openat,stat -c ./emacs_stamped --version GNU Emacs 27.2 Copyright (C) 2021 Free Software Foundation, Inc. GNU Emacs comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GNU Emacs under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING. % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000950 9 104 1 openat ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000950 9 104 1 total $strace -e openat,stat -c /nix/store/vvxcs4f8x14gyahw50ssff3sk2dij2b3-emacs-27.2/bin/.emacs-27.2-wrapped --version GNU Emacs 27.2 Copyright (C) 2021 Free Software Foundation, Inc. GNU Emacs comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GNU Emacs under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING. % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.034121 18 1823 1720 openat ------ ----------- ----------- --------- --------- ---------------- 100.00 0.034121 18 1823 1720 total Happy to hear some thoughts on this approach. [1] https://guix.gnu.org/blog/2021/taming-the-stat-storm-with-a-loader-cache/ [2] https://github.com/fzakaria/shrinkwrap