unofficial mirror of guix-patches@gnu.org 
 help / color / mirror / code / Atom feed
From: "Ludovic Courtès" <ludo@gnu.org>
To: Caleb Ristvedt <caleb.ristvedt@cune.org>
Cc: bug-guix@gnu.org
Subject: [bug#42023] [PATCH] Retry deduplication on ENOENT
Date: Wed, 16 Sep 2020 22:37:05 +0200	[thread overview]
Message-ID: <87k0wtlaq6.fsf@gnu.org> (raw)
In-Reply-To: <87sgbikclv.fsf_-_@cune.org> (Caleb Ristvedt's message of "Tue, 15 Sep 2020 15:29:32 -0500")

Hi!

Caleb Ristvedt <caleb.ristvedt@cune.org> skribis:

[...]

>> There’s an issue with this patch.  I gave it a spin (offloading a few
>> builds) and it got stuck in a infinite loop:
>>
>> stat("/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhrk", 0x7ffe43898cd0) = -1 ENOENT (Dosiero aŭ dosierujo ne ekzistas)
>> link("/gnu/store/83jy739bn644w3pnwgb5kwjig0kzs92f-libwps-0.4.12/lib/libwps-0.4.so", "/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhrk") = -1 EEXIST (Dosiero jam ekzistas)
>> stat("/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhrk", 0x7ffe43898cd0) = -1 ENOENT (Dosiero aŭ dosierujo ne ekzistas)
>> link("/gnu/store/83jy739bn644w3pnwgb5kwjig0kzs92f-libwps-0.4.12/lib/libwps-0.4.so", "/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhrk") = -1 EEXIST (Dosiero jam ekzistas)
>> stat("/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhrk", 0x7ffe43898cd0) = -1 ENOENT (Dosiero aŭ dosierujo ne ekzistas)
>> link("/gnu/store/83jy739bn644w3pnwgb5kwjig0kzs92f-libwps-0.4.12/lib/libwps-0.4.so", "/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhrk") = -1 EEXIST (Dosiero jam ekzistas)
>>
>
> I believe I can explain this. In 'deduplicate' we currently treat
> anything that isn't a directory as a hardlinkable thing. This includes
> symlinks (although it's implementation-defined whether symlinks can be
> hardlinked to - we use CAN_LINK_SYMLINK to test this in
> nix/libstore/optimise-store.cc). This means that at present we
> unconditionally attempt to deduplicate symlinks (which happens to work
> on linux). However, 'file-exists?' uses stat, not lstat, to check for
> file existence. Thus, if there is a dangling symlink, 'file-exists?'
> will return #f when passed it, but of course attempting to call link()
> to create it will fail with EEXIST. Attached is a modified patch that
> tests for file existence with lstat instead. I expect that will fix the
> problem.

Ah ha!

> We should probably still add a test in 'deduplicate' for whether
> symlinks can be hardlinked to.

If GNU/Linux and GNU/Hurd support it, it’s unnecessary.

> Tangent: I was curious why libwps-0.4.so would be a dangling symlink,
> and it turns out that it's actually a relative symlink, so when
> accessing it via /gnu/store/...-libwps-0.4.12/lib/libwps-0.4.so it isn't
> dangling, but when accessing it via /gnu/store/.links/0k63r... it is.

I see, good catch!

> From 12f5848e79b0ede95babebea240264b32e39812c Mon Sep 17 00:00:00 2001
> From: Caleb Ristvedt <caleb.ristvedt@cune.org>
> Date: Sat, 8 Aug 2020 11:25:57 -0500
> Subject: [PATCH] deduplication: retry on ENOENT.
>
> It's possible for the garbage collector to remove the "canonical" link after
> it's been detected as existing by 'deduplicate'.  This would cause an ENOENT
> error when replace-with-link attempts to create the temporary link.  This
> changes it so that it will properly handle that by retrying.
>
> * guix/store/deduplication.scm (replace-with-link): renamed to
>   canonicalize-with-link, now also handles the case where the target link
>   doesn't exist yet, and retries on ENOENT.  Also modified to support
>   canonicalizing symbolic links, though it is the caller's responsibility to
>   ensure that the system supports hardlinking to a symbolic link (on Linux it
>   does).
>   (deduplicate): modified to use canonicalize-with-link.

[...]

> +            (lambda args
> +              (let ((errno (system-error-errno args)))
> +                (cond
> +                 ((= errno ENOENT)
> +                  ;; either SWAP-DIRECTORY has missing directory
> +                  ;; components or TARGET was deleted - this is a
> +                  ;; fundamental ambiguity to the errno produced by
> +                  ;; link()
> +                  (if (file-exists? swap-directory)
> +                      ;; we must assume link failed because target doesn't
> +                      ;; exist, so create it.

Nitpick: Please capitalize sentences, add a period at the end, and write
“'link'” instead of “link()” or “link” for clarity.

Otherwise LGTM.

I think we’ll have to stress-test it through offloading to catch any
remaining issues.

Thank you!

Ludo’.




      reply	other threads:[~2020-09-16 20:38 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-23 16:36 [bug#42023] [PATCH] database: register-items: reduce transaction scope Christopher Baines
2020-06-23 22:15 ` Ludovic Courtès
2020-06-24 17:51   ` Caleb Ristvedt
2020-06-25  7:45     ` Ludovic Courtès
2020-07-26 15:31       ` Ludovic Courtès
2020-08-09  4:13         ` Caleb Ristvedt
2020-09-14  8:58           ` Ludovic Courtès
2020-09-15 20:29             ` [bug#42023] [PATCH] Retry deduplication on ENOENT Caleb Ristvedt
2020-09-16 20:37               ` Ludovic Courtès [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87k0wtlaq6.fsf@gnu.org \
    --to=ludo@gnu.org \
    --cc=bug-guix@gnu.org \
    --cc=caleb.ristvedt@cune.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).