From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id fTyWNz3MZWClrwAAgWs5BA (envelope-from ) for ; Thu, 01 Apr 2021 15:35:57 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 6HOqJj3MZWA8EwAAbx9fmQ (envelope-from ) for ; Thu, 01 Apr 2021 13:35:57 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 816462220B for ; Thu, 1 Apr 2021 15:35:56 +0200 (CEST) Received: from localhost ([::1]:35650 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lRxUJ-0008KR-IE for larch@yhetil.org; Thu, 01 Apr 2021 09:35:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41724) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lRxOV-0002x5-CP for guix-science@gnu.org; Thu, 01 Apr 2021 09:29:55 -0400 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]:41612) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lRxOR-0004u5-Ku for guix-science@gnu.org; Thu, 01 Apr 2021 09:29:54 -0400 Received: by mail-qk1-x72f.google.com with SMTP id v70so2115623qkb.8 for ; Thu, 01 Apr 2021 06:29:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=gtxRpWtxnpbVwxKypHCeWWRd/QTjBcva/DmIgsM2qac=; b=fmGLs3cHL0yfkRch1BhTrz/TE4oABHHCJ/I+68JRsMLJAG7fVYLQjndsKejxgmCq7b +VosdSfBj86yQdMo9XRYN5QFtVqIL8NP79YVH9sAWZJ2Fke/k2ZAM+Unvgbd79ZSMfGW /0rdfmCzEMe4zBTAU6cle2z0tJJ87TTFglIx7hNJQBrcIoRb9gnV5E9yskRbCAACB8Yw q0rN6NkNNxXdM3SpqGvoJTlQdUqJ6ZgrHrcLKqfeiX7NhIlRlJrjpxzH+9gPLQToKNny g55IDjctAWwcLnO6fPNZz4JbyMn5qI7WrwU5/6agNy3cn2DabnGICNWokxakHkdL1Pu9 EODA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=gtxRpWtxnpbVwxKypHCeWWRd/QTjBcva/DmIgsM2qac=; b=ShTCGKEc5+Cmzgl24qZqnYfPnlj4YAY3+5qmtiuU8kV6qHt6JOe+EEuxjXDR3/pCmk UcLyAlUX6NVEZhtuL+Y4tQSCZ1e2LBhxbKfjZzHOO7UzdSkSO6q5O9HPExZJOnzUwglc XzLETO526cTE/zMdvwadhHjxY8ZojK2lsM0BFTMy+eC09iZKFiFkMy1fG8YE/CQ8WMkV SgCQDwifqrNfpiSfZJ9c12Jj9BbpA+oO08ZwLRbVGAfnWP55k1XrCw1pdklrhLl1TDZq ELY6b2CyrKL5shad2h1YnC0wvWyNUAKh5KxMgddUg+nYRkx+lJrdvz4mrymG0ZfrcCBs KYwg== X-Gm-Message-State: AOAM530y7wx0oOLJpd+TRKRf4ywXE7Xzn3miRExZnYCK6+nxXtCmH/qN ozBFq89cRICTN+sN4Jlcq9ejRZbsh+zHiC3bbgY= X-Google-Smtp-Source: ABdhPJy5NJG5/nYa3coAJiCiWWcr2bvrct2+K8PxKM1kWiNLPFl2SzIk4eWBzNbDXypd8pTE+wa/d1VVbY4RuOW28ug= X-Received: by 2002:a37:b807:: with SMTP id i7mr8179441qkf.126.1617283789536; Thu, 01 Apr 2021 06:29:49 -0700 (PDT) MIME-Version: 1.0 References: <87v9969uml.fsf@inria.fr> <87wntm89lp.fsf@inria.fr> In-Reply-To: <87wntm89lp.fsf@inria.fr> From: zimoun Date: Thu, 1 Apr 2021 15:29:38 +0200 Message-ID: Subject: Re: GC strategy on clusters To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Cc: Efraim Flashner , guix-science@gnu.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2607:f8b0:4864:20::72f; envelope-from=zimon.toutoune@gmail.com; helo=mail-qk1-x72f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-science@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-science-bounces+larch=yhetil.org@gnu.org Sender: "Guix-Science" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1617284156; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=gtxRpWtxnpbVwxKypHCeWWRd/QTjBcva/DmIgsM2qac=; b=SlbdD0wk/ohmtEvu8E4FNmajp+u52syhD0cdDKSwbjBJxKioLWu2kds9LbAxgndtG4qa5f J71gv8DbY3cJjoeLMF/6mOuyW5XFyVE66vhwlPmnz/c/QkVr0BUfN2FaQbHjkTDyYMIQdm bQBebAfJhhE87r3xC/yqiRwKTCoRaKuOfksQyfC5Z6ZcpipW4+NLjMBHwycy2NkDakeTRV CXM7Yy5nfBjSgObF8TEMd3XJxwiGA3nNlJwxtcYgaGDU8gunMwvrpGpQR13uWIeufdi3bF bNVlMtpEIjXkBlJ6ijs5/6ClUfjvH0ACIHygGMdAs2gWZ9QljKTyxDvogKGvGA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1617284156; a=rsa-sha256; cv=none; b=E3X1UTdb7xYStC1GHXJfLAVNWrxP64nBAsOYqxF8NU7ICDWCwkkrifZ18ui6TMArJvs5+V lrGd1fG9yEeSCW+peuYKl8LvNQDnNAa4og4M1va2cIUcBwCXHa94KJtqVNQccWxHaRRiN7 XVtkUsi7fFcwhRdGTx7/UYBvb5ZMBSCTQA//oy1URIDSP3DcGZdZ4hp0YmROYHCt6zmJr+ iua7uhpLYBaabcWjOzmh7ee7azuQQsRjSpZmEhKZnyGaSZ+X/Teb0aY3KIEOo6s8M6Cqnz Ksv1eX9yuyCHZVg1LC/SO6JKBanv9BdVMPh8m5wwGCI1wfkywszU7IHAh8y4/Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=fmGLs3cH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-science-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-science-bounces@gnu.org X-Migadu-Spam-Score: -3.13 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=fmGLs3cH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-science-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-science-bounces@gnu.org X-Migadu-Queue-Id: 816462220B X-Spam-Score: -3.13 X-Migadu-Scanner: scn0.migadu.com X-TUID: Hg2zDYewxDH0 Hi, On Thu, 1 Apr 2021 at 14:36, Ludovic Court=C3=A8s wrote: > Efraim Flashner skribis: > > > On Thu, Apr 01, 2021 at 12:17:22PM +0200, Ludovic Court=C3=A8s wrote: > > [...] > > >> guix package --delete-generations=3D4m > >> > >> or similar, which was enough to free more space. For now, I am waiting to reach the full capacity at hand to ask how to deal with it. ;-) > > I feel like 4-6 months should be plenty for anything active. Even if it > > were run automatically for them it wouldn't remove the last generation. > > It depends. A practical use case I have in mind: you run experiments, > you submit a paper including its results, you get initial reviews months > later, and even later it=E2=80=99s published and you get to present it. = At that > point, you want to answer questions and to reproduce it. 4=E2=80=936 mon= ths is > not a lot in that context. To add another data point. Even it is hard to have a good overview, I would say the average is 2-4 years for a typical project. It is hard because currently all is not done with the same tools for the same task. For instance, some data is cleaned with some tools, then an partial analysis is done, months later another data is added so another partial analysis with probably different tools, then months later a full toolsuite as Bioconductor (or whatever) is updated and some analysis are re-done. The final publication is a mix of all over the 2-4 years project with details at various level. Other said, it depends on the level we are looking. > (Though of course, ideally you=E2=80=99d save channels.scm + manifest.scm= and > share it with reviewers and readers in the first place=E2=80=A6) It is what I am tirelessly explaining in my lab. ;-) > >> Longer term, I think Guix should automatically delete old generations > >> and instead store the channels + manifest to reproduce them, when > >> possible. > >> > > > > This seems to help a bit less when we run into issues about dates being > > wrong on SSL tests, or when sources go missing. > > Good points. Hopefully =E2=80=9Csources go missing=E2=80=9D can soon be = considered > addressed. Really, failing TLS tests is the most worrisome issue to me > because we don=E2=80=99t have any idea on how to address it systematicall= y. By "soon", you mean the bricks are there and it is missing to glue them together. From my opinion, some details need to be addressed to have a full end-to-end sources fallback, since evil is hidden inside the details. ;-) About the TLS, you proposed to setup a machine ahead of clock. Maybe it is worth to try. > > I guess I'm not really sure if its a technology problem or a people > > problem. Figuring out if someone is the only one pulling in a copy of > > glibc-2.25 is doable but how many copies of diffoscope is too many? > > > > On a practical note, 'guix package --list-profiles' as root currently > > lists everyone's profiles so it can be easier to see who has older > > profiles hanging around. > > Actually, as non-root, I walked /var/guix/profiles/per-user on the > cluster to see the number of generations per user, which allowed us to > target those with a lot of generations. :-) Well, when we discussed the '--list-profiles', it was initially for my personal purposes. Then, I have tried to use it to monitor the few users that I have. Well, in Biology they have the concept of "-80 fridge". It is a big and very cold fridge where you keep samples, potentially for a long time. Everybody put in until it is full and once it is full, there is endless discussion on what to throw... until the fridge is broken because shutdown or unexpected failures and then it is obvious to everybody what needs to be taken or thrown. I am using such analogy to explain the hygiene to have on shared machines Hum, once written I do not know if it relevant. ;-) >From my point of view, having a channel+manifest "backup" for old profiles seems something to try. It cost nothing with ''--export-manifest" and "--export-channels". Cheers, simon