From: Julien Lepiller <julien@lepiller.eu>
To: guix-devel@gnu.org, zimoun <zimon.toutoune@gmail.com>,
"Miguel Ángel Arruga Vivas" <rosen644835@gmail.com>,
"Ludovic Courtès" <ludo@gnu.org>
Cc: Guix Devel <guix-devel@gnu.org>
Subject: Re: Identical files across subsequent package revisions
Date: Wed, 23 Dec 2020 10:40:00 -0500 [thread overview]
Message-ID: <077ECD6C-AB0D-4FEA-ABBA-82550834265E@lepiller.eu> (raw)
In-Reply-To: <86wnx8r4ys.fsf@gmail.com>
Le 23 décembre 2020 09:07:23 GMT-05:00, zimoun <zimon.toutoune@gmail.com> a écrit :
>Hi,
>
>On Wed, 23 Dec 2020 at 14:10, Miguel Ángel Arruga Vivas
><rosen644835@gmail.com> wrote:
>
>> Probably you're already aware of it, but I want to mention that
>> Tridgell's thesis[1] contains a very neat approach to this problem.
>
>This thesis is a must to read! :-)
>
>
>> A naive prototype would be copying of the latest available nar of the
>> package on the client side and using it as the destination for a copy
>> using rsync. Either the protocol used by the rsync application, or a
>> protocol based on those ideas, could be implemented over the HTTP
>layer;
>> client and server implementation and cooperation would be needed
>> though.
>
>I could misunderstand and miss something, one part of the problem is
>how
>to detect “latest”; other said how to know it is different. From my
>memories, and I have drunk couple of beers since I read the thesis :-),
>the ’rsync’ approach uses timestamp and size. And if you switch to
>checksum instead, the performances are poor, because of IO. Well, it
>depends on the number of files and their size, if this checksum are
>computed ahead, etc.
>
>
>> Another idea that might fit well into that kind of protocol---with
>> harder impact on the design, and probably with a high cost on the
>> runtime---would be the "upgrade" of the deduplication process towards
>a
>> content-based file system as git does[2]. This way the a description
>of
>> the nar contents (size, hash) could trigger the retrieval only of the
>> needed files not found in the current store.
>
>Is it not related to Content-Addressed Store? i.e, «intensional
>model»?
>
>Chap. 6: <https://edolstra.github.io/pubs/phd-thesis.pdf>
>Nix FRC:
><https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md>
I think this is different, because we're talking about sub-element content-addressing. The intensional model is about content-addressing whole store elements. I think the idea would be to save individual files in, say, /gnu/store/.links, and let nar or narinfo files describe the files to retrieve. If we are missing some, we'd download them, then create hardlinks. This could even help our deduplication I think :)
>
>
>Cheers,
>simon
next prev parent reply other threads:[~2020-12-23 15:41 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-22 22:01 Identical files across subsequent package revisions Ludovic Courtès
2020-12-23 9:08 ` Taylan Kammer
2020-12-23 10:50 ` Pierre Neidhardt
2020-12-23 22:06 ` Mark H Weaver
2020-12-23 22:47 ` Jonathan Brielmaier
2020-12-23 23:42 ` Mark H Weaver
2020-12-23 10:19 ` zimoun
2020-12-23 10:41 ` zimoun
2020-12-27 22:22 ` Ludovic Courtès
2020-12-23 11:48 ` Christopher Baines
2020-12-23 13:10 ` Miguel Ángel Arruga Vivas
2020-12-23 14:07 ` zimoun
2020-12-23 15:40 ` Julien Lepiller [this message]
2020-12-23 19:07 ` Miguel Ángel Arruga Vivas
2020-12-27 22:29 ` Ludovic Courtès
2021-01-06 9:39 ` Ludovic Courtès
2020-12-29 20:01 ` pukkamustard
2021-01-06 9:27 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=077ECD6C-AB0D-4FEA-ABBA-82550834265E@lepiller.eu \
--to=julien@lepiller.eu \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
--cc=rosen644835@gmail.com \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).