From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id sFPaCDMp92WHfQEAe85BDQ:P1 (envelope-from ) for ; Sun, 17 Mar 2024 18:32:35 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id sFPaCDMp92WHfQEAe85BDQ (envelope-from ) for ; Sun, 17 Mar 2024 18:32:35 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=abesis.fr header.s=mail header.b=xbEORrmY; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1710696754; a=rsa-sha256; cv=none; b=FcTayqD6sqVyutf2K8p1tAQanmAXBHjIjHu+QziL9t/DFUNce776fcEmgr65cwoRM+wyRs 5GfQ75G9rMvYVBExoDaeXJtklgwgaNA+Uv8PJdzMPg0ub9uGmFUIhsaU0TVE6hBqtfTsty 2OjaCfJIw7SYBx4RJJHUbK5r85FEm27U63O7CehGFxD1YjkrzFGtYTDxTo+VqpVeQYwSIY Hqs5dNvEnj2UZfCC4+4PhfMfCeAMzbhDvERmHRUjAZlHtix3naR6TJit61VFlZiJy7luyf T1ufrfmmu3fnqve866Mfts64EqtkiGy4u44vhHFFeqIXIWLw0Kx9FuI9OrzUfQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=abesis.fr header.s=mail header.b=xbEORrmY; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1710696754; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=Ik5swqYWSl3fC9+uL+ShRBJOTyykZeb7inWpS8R2fQE=; b=WZBgDiE+xF+PicYXsPdZazaYAB5x1dkrtCGFe528tEMFIlth1UDU7b2942zA3TXz98KfJ+ ksNWl5VfpIQF1GK6y7TQqp0QuEigb6Y1vUpF0kdA2flt+M+yuoWzFrGUK7c96QB5FaKVnj KzTKrfMkMMf36AnNtRJUQVS3yhePDCWWq8m18PuvDvMMqBnc5mND5lu3MGY1OOEZpCrYOM 2Sh3Vnp7AMzZ82c76n9BDqweaHm9y/UU66QAWhdDmDiRbmt+4eNMxVw1ZA+Gr6L3u75MIJ 5KfII+2P5NgArVDm2pZTNdunKe9y1seIUspSCiqulKUvh20xTukLcI0kWMZaOA== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D7B626C8E5 for ; Sun, 17 Mar 2024 18:32:34 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rluMu-0004X6-Qo; Sun, 17 Mar 2024 13:32:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rluMt-0004Wx-It for guix-devel@gnu.org; Sun, 17 Mar 2024 13:32:19 -0400 Received: from kordia.abesis.fr ([37.187.96.121]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rluMr-00050Y-83 for guix-devel@gnu.org; Sun, 17 Mar 2024 13:32:18 -0400 From: Antoine Eiche DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=abesis.fr; s=mail; t=1710696733; bh=Ik5swqYWSl3fC9+uL+ShRBJOTyykZeb7inWpS8R2fQE=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=xbEORrmYFgPJiYWgOdHRdk1f++/L1vfossEqM9wAUhMG3v8k0IcXaa9MjGZ5Zc7iR dj4mBh8Hs+VSY6ZY1hK7Z2sT1RUuqlOz9EPbJUaHe1zt3LYgt+js26nZzcWYPAhQqA jfP0ghp8WXBLKMjNr6xfuvXhPXJxg0A2uKTz80Wg= To: Simon Tournier , Ricardo Wurmus Cc: guix-devel@gnu.org Subject: Re: Building container images with nix2container In-Reply-To: <87cys5817b.fsf@gmail.com> References: <8734tijjpy.fsf@tilia> <87wmqszwif.fsf@elephly.net> <87plwjilke.fsf@tilia> <87o7c36sgu.fsf@gmail.com> <87cys5817b.fsf@gmail.com> Date: Sun, 17 Mar 2024 18:32:10 +0100 Message-ID: <8734sokbmt.fsf@tilia> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=37.187.96.121; envelope-from=lewo@abesis.fr; helo=kordia.abesis.fr X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -6.50 X-Spam-Score: -6.50 X-Migadu-Queue-Id: D7B626C8E5 X-TUID: hdIQaIoC8ymB Simon Tournier writes: >> Well, I have not followed on which strategy Guix relies. What is the >> one of nix2container? The one described here: >> >> https://grahamc.com/blog/nix-and-layered-docker-images/ > > To answer to my question, the way to build the container image is > different, hence it does not make much sense to speak about a > =E2=80=9Cstrategy=E2=80=9C. :-) nix2container actually uses this strategy. To build an image, nix2container builds several derivations. Each of these derivation contains a set of layers. The layers of each set are determined by using the algorithm mentionned above. At runtime, all of these derivations are packed to build the final image. > However, the blog post says: > > To address this issue, we could add a nonReproducible option in > the containerTools.buildLayer function. Instead of only storing > the digest, we would also store the tar. Note in practice, an > important part of nixpkgs is bit reproducible and this would > rarely be needed. > > And so the question is how do you know beforehand if the flag > =E2=80=99nonReproducible=E2=80=99 must be applied or not? Sorry, i have missed your mails:/ So, here is the answer I provided on Mastodon few days ago. Generally, i don't think it's possible to know when this flag is needed before encountering an issue at image push time: Skopeo would then say layer hashes mismatch. However, in practice, it seems this flag is not often used. To hit this issue, I think you need to substitute the JSON file from a binary cache while building non bit reproducible store paths locally. In this case, the hash specified in the JSON file could be different from the hash of the derivation locally built. But, the locally built store path is a dependency of the JSON file, they should also be in the binary cache and would generally also be substituted. I admit this is not rock-solid science... but it seems to make the job in practice. And if it occurs, a strategy could be to isolate this storepath into a layer with the non-reproducible flag. There is also another option which is currently used by nixpkgs.dockerTools.streamLayeredImage. To avoid the reproducibility issue while still avoiding writing layer tarballs, we could introduce another option to let nix2container generating the layer hashes on the fly, just before pushing the image. This would then require more IOs since nix2container would need to read layer store paths on each image push. > Indeed, the approach of nix2container could be helpful in addition to > =E2=80=98guix pack=E2=80=99. Maybe an extension=E2=80=A6 :-) >From my point of view, another advantage of the nix2container approach is to delegate a important part of the build image mecanism to Skopeo, a tool well maintained and used by a lot of people. lewo.