From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id aDUtJrrXXWCO2gAAgWs5BA (envelope-from ) for ; Fri, 26 Mar 2021 13:46:50 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id yLj6H7rXXWA5MwAA1q6Kng (envelope-from ) for ; Fri, 26 Mar 2021 12:46:50 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D92F43793B for ; Fri, 26 Mar 2021 13:46:49 +0100 (CET) Received: from localhost ([::1]:36704 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lPlrU-00040v-Qa for larch@yhetil.org; Fri, 26 Mar 2021 08:46:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58208) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lPlrS-00040N-I7 for gwl-devel@gnu.org; Fri, 26 Mar 2021 08:46:46 -0400 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:36325) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lPlrQ-0001y6-Q5 for gwl-devel@gnu.org; Fri, 26 Mar 2021 08:46:46 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id EEBF05C0750; Fri, 26 Mar 2021 08:46:43 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 26 Mar 2021 08:46:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.net; h= from:to:subject:in-reply-to:references:date:message-id :mime-version:content-type:content-transfer-encoding; s=fm3; bh= wokZCYeNRVSZkund/OE+euNwAi1KUdVhq+9UZa4DVoU=; b=FY6Yx4S+560fCbQl Kv3sm+RBcpNJSByblk6DCE8i6fa7el9mY8v4XG2ET0/BhvEjspgOW0csL9zOeEtv JW735CiZCHX5MW8fFDAeW7qcxALfWU5B68dC2jDGaE7rDIQoCh2vXhrRJTjtgirL lMoeYJBzbqRbgmTodcHYEJz6NhUe1kU621FUmiwnEVNmCQbPtcJxkZIFi9E46j3P 584sU3NNVZ9IncQLcN1JPl8incGFWeqF7DMkVzmFL19TSI5z5SBY0wJhUfUf2okN X/6uta6AlvFjDOPWR7wqfqNfuDaLuDZXGwoAiHcwA+iG6pYVpoDWX392l1z/MFez s26bPQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=wokZCYeNRVSZkund/OE+euNwAi1KUdVhq+9UZa4DV oU=; b=AIQF1pGJNfmnIozDuvaUG7W4I4WrpNdqdom2vfxs/TzSnppK7eSKrUzmY U535PHBXb5mG9CZ1tmFx4rxGvDpTZcRDWdZWDT/CyrY0+uQHDKoibkOfHXO6zY78 vDQRX8CDhHg4wSZGcZbwrSYkxxnQ3OfiuUNVh3oj4TkWJkxnWl/YsOswcsd0YWwS ySD4++HHuBZBL4so5A6H+BImfjqm768lk3JV7sDcthJ+78BjUvKH/i7zl7bYk82j YB0aUlMQYUhph0Fa/RsD9Has/ydq40jJ1D2T111IdG3a05qIYaU2yOTV0cE0MJn9 Rd/PG3yBvcT/ONhUVYlb3bR6EsqcQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudehvddggeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffujghffffkgggtgfesthhqre dttddtjeenucfhrhhomhepmfhonhhrrgguucfjihhnshgvnhcuoehkohhnrhgrugdrhhhi nhhsvghnsehfrghsthhmrghilhdrnhgvtheqnecuggftrfgrthhtvghrnhepveehleevge ejuddvgfduieefveetffdvteegfeekgfduueejfeegvdehgeekhfejnecukfhppeekiedr vdegjedrgeekrdehkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehkohhnrhgrugdrhhhinhhsvghnsehfrghsthhmrghilhdrnhgvth X-ME-Proxy: Received: from ordinateur-de-catherine--konrad.home (lfbn-idf2-1-840-58.w86-247.abo.wanadoo.fr [86.247.48.58]) by mail.messagingengine.com (Postfix) with ESMTPA id 39FED240054; Fri, 26 Mar 2021 08:46:43 -0400 (EDT) From: Konrad Hinsen To: zimoun , gwl-devel@gnu.org Subject: Re: Managing data files in workflows In-Reply-To: <86v99ebdnz.fsf@gmail.com> References: <86v99ebdnz.fsf@gmail.com> Date: Fri, 26 Mar 2021 13:46:42 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=66.111.4.29; envelope-from=konrad.hinsen@fastmail.net; helo=out5-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616762810; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=wokZCYeNRVSZkund/OE+euNwAi1KUdVhq+9UZa4DVoU=; b=ckhLn/XNmFy3Po5cznuPlP5YowTdyZbgsYovJE09giOPjkeZx1JBbCcKBfYgPtyw+6oxr4 9jHwK81zt15b3cNLwoKb9gOloi6RNMPUamPQ1KEbyZYtHrYl7ULBq5QuXe/QiOFL1T3QE8 kkOdwPOV4ZMx76V09Xwy1xxRQkUoAoHVcodhjWuT5G/ooceS905311z41UtnT9Kjq9T3qi H0F8l+KowGmgZcWPSQGoiOYJPsJIoPFy7QcmtJAID3OgON74CepOw+4aqJXySSCKZqU9Tl Y+ezJGwoticwzt5qdKGgj/hdP2G08t06hsBA5Jn02a48FOwMmpBS0+v2pcip1Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616762810; a=rsa-sha256; cv=none; b=fLRL2/qFpcg7ajqIRFW6/br0k4PoxDFrf7UEC7S5SOMyUReC7qhlZIxEvO2mnKEHiIvlXV Pu9ggjUG+m4YGgUwVsnaBnbmT4Rk+xi8A4kSUbrPfNcW53ayfkUpC9KfM+v8vmD4UqeicO 9mG/5c4UHgeYnFlDOAqjGNFb3X5nkW/aG7e/JFn6DSENzRIt2H7jE7+NyN/WBbQI2Yj8+X 40PGMQuzjdXLwSyQFfG1QSshPbBtfBmdlVxzpZck/uqcbq2lQnNWycVPmbwCbkD9W69fGs eIQZG/PGL15Q0FJd0UsZad9NKebWwj0tRpXcgPS8m8hVJ4GnGMzAGxpy9yPMcg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=fastmail.net header.s=fm3 header.b=FY6Yx4S+; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=AIQF1pGJ; dmarc=pass (policy=none) header.from=fastmail.net; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.12 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=fastmail.net header.s=fm3 header.b=FY6Yx4S+; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=AIQF1pGJ; dmarc=pass (policy=none) header.from=fastmail.net; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: D92F43793B X-Spam-Score: -3.12 X-Migadu-Scanner: scn0.migadu.com X-TUID: D+HwgK/0QT8V Hi Simon, > It does not answer your concrete question but instead open a new > one. :-) And a good one! > 1. how to deal with data? > 2. on which does the workflow trigger a recomputation? Number 2 was what I had in mind with my question. And I still wonder how GWL handles it now and/or in some near future. > There is 3 levels: > > 1- the methods for fetching: URL (http or ftp), Git, IPFS, Dat, etc. > 2- the record representing a =E2=80=9Cdata=E2=80=9D > 3- how to effectively locally store and deal with it > > And if it makes sense that a =E2=80=99data=E2=80=99 is an input of a > =E2=80=99package=E2=80=99, and conversely, is a question. > > Long time ago, with GWL folks we discussed =E2=80=9Cbackend=E2=80=9D, as = git-annex or > something else, but from my understanding, it would answer about #3 and > what git-annex accepts as protocol would answer to #1. Remaining #2. Perhaps a good first step is to actually use git-annex for big files, and then integrate it more and more into Guix and/or GWL. Multiple backends will certainly be required in the near future, because data storage is not yet sufficiently standardized to pick one specific technology. So why not profit from the work already done in git-annex? One answer to #2 would be to use a git repository. Managed by git-annex, with remotes pointing to the repositories that actually hold the data. Not very elegant, but as a first step, why not. Cheers, Konrad.