Move switch-symlinks to (guix build utils)

Move switch-symlinks to (guix build utils)

[Arun Isaac]

Hi Guix,

The switch-symlinks function from (guix utils) is often required in
activation-service G-expressions. But, only (guix build utils) and not
(guix utils) is available to such G-expressions. See, for example, the
pcscd-activation G-expression in (gnu services security-token) where
switch-symlinks gets
redefined. https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/security-token.scm#n70

Could we move switch-symlinks to (guix build utils)? Would such a change
require a core-updates cycle?

Thanks!
Arun

Re: Move switch-symlinks to (guix build utils)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

Arun Isaac schreef op wo 01-06-2022 om 13:04 [+0530]:
> Hi Guix,
> 
> The switch-symlinks function from (guix utils) is often required in
> activation-service G-expressions. But, only (guix build utils) and not
> (guix utils) is available to such G-expressions. See, for example, the
> pcscd-activation G-expression in (gnu services security-token) where
> switch-symlinks gets
> redefined. https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/security-token.scm#n70
> 
> Could we move switch-symlinks to (guix build utils)? Would such a change
> require a core-updates cycle?

To avoid a world-rebuild, you could for now make a module (guix build
symlinks) or such?  An alternative is (gnu build activation), but then
some (guix ...) modules would depend on (gnu ...).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Move switch-symlinks to (guix build utils)

[Arun Isaac]

Hi Maxime,

> To avoid a world-rebuild, you could for now make a module (guix build
> symlinks) or such?  An alternative is (gnu build activation), but then
> some (guix ...) modules would depend on (gnu ...).

I don't really mind waiting for the next core-updates cycle. This
problem has been around for a long time. It wouldn't hurt much to wait a
bit longer. If there is consensus on this change, I can send a
patch. WDYT?

Thanks,
Arun

Re: Move switch-symlinks to (guix build utils)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

Arun Isaac schreef op do 02-06-2022 om 20:36 [+0530]:
> Hi Maxime,
> 
> > To avoid a world-rebuild, you could for now make a module (guix build
> > symlinks) or such?  An alternative is (gnu build activation), but then
> > some (guix ...) modules would depend on (gnu ...).
> 
> I don't really mind waiting for the next core-updates cycle. This
> problem has been around for a long time. It wouldn't hurt much to wait a
> bit longer. If there is consensus on this change, I can send a
> patch. WDYT?

Downside of (guix build utils): it makes it harder to modify switch-
symlinks.  I don't know if that's too much of a downside in practice. 
Also, there is a bug in switch-symlinks:

(use-modules (guix utils))
;; simulate an interrupted switch-symlinks (C-c, power interruption, ...)
(symlink "target" "link.new")
;; now try again
(switch-symlinks "link" "target")

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure symlink: Bestand bestaat al: "link.new"

If we move it to (guix build utils), I'd prefer the bug to be addressed first.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Move switch-symlinks to (guix build utils)

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Arun Isaac schreef op do 02-06-2022 om 20:36 [+0530]:
>> Hi Maxime,
>> 
>> > To avoid a world-rebuild, you could for now make a module (guix build
>> > symlinks) or such?  An alternative is (gnu build activation), but then
>> > some (guix ...) modules would depend on (gnu ...).
>> 
>> I don't really mind waiting for the next core-updates cycle. This
>> problem has been around for a long time. It wouldn't hurt much to wait a
>> bit longer. If there is consensus on this change, I can send a
>> patch. WDYT?
>
> Downside of (guix build utils): it makes it harder to modify switch-
> symlinks.  I don't know if that's too much of a downside in practice. 
> Also, there is a bug in switch-symlinks:
>
> (use-modules (guix utils))
> ;; simulate an interrupted switch-symlinks (C-c, power interruption, ...)
> (symlink "target" "link.new")
> ;; now try again
> (switch-symlinks "link" "target")
>
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> In procedure symlink: Bestand bestaat al: "link.new"

That’s EACCES?

> If we move it to (guix build utils), I'd prefer the bug to be addressed first.

Yes, better be cautious before “setting it in stone”.  Do you have a fix
in mind?

With this issue addressed, I think it’s fine to move to (guix build
utils) on ‘core-updates’.  We’ll have to run ‘make check’ to make sure
all users are updated, probably with #:select to clarify what binding
they need from (guix build utils).

Thanks,
Ludo’.

Re: Move switch-symlinks to (guix build utils)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

Ludovic Courtès schreef op vr 03-06-2022 om 18:38 [+0200]:
> That’s EACCES?

It's EEXIST.

> > If we move it to (guix build utils), I'd prefer the bug to be addressed first.
> 
> Yes, better be cautious before “setting it in stone”.  Do you have a fix
> in mind?

Maybe replace

   (symlink target pivot)

by (symlink/remove-old target pivot)

where

(define (symlink/remove-old target link)
  "Make a symbolic link named LINK pointing to TARGET.
If LINK already exists, it will be removed first.
This is not an atomic operation."
  (catch 'system-error
    (lambda ()
      (symlink target link))
    (lambda stuff
      (if (= (system-error-errno stuff) EEXIST)
          (begin
            ;; remove old link and retry
            (delete-file link)
            (symlink/remove-old link target))
          (apply throw stuff)))))

?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Move switch-symlinks to (guix build utils)

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Maybe replace
>
>    (symlink target pivot)
>
> by (symlink/remove-old target pivot)
>
> where
>
> (define (symlink/remove-old target link)
>   "Make a symbolic link named LINK pointing to TARGET.
> If LINK already exists, it will be removed first.
> This is not an atomic operation."
>   (catch 'system-error
>     (lambda ()
>       (symlink target link))
>     (lambda stuff
>       (if (= (system-error-errno stuff) EEXIST)
>           (begin
>             ;; remove old link and retry
>             (delete-file link)
>             (symlink/remove-old link target))
>           (apply throw stuff)))))

Alright, SGTM (this procedure would be kept private).

So Arun, the floor is yours!  :-)

Ludo’.

Re: Move switch-symlinks to (guix build utils)

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Maybe replace
>
>    (symlink target pivot)
>
> by (symlink/remove-old target pivot)
>
> where
>
> (define (symlink/remove-old target link)
>   "Make a symbolic link named LINK pointing to TARGET.
> If LINK already exists, it will be removed first.
> This is not an atomic operation."
>   (catch 'system-error
>     (lambda ()
>       (symlink target link))
>     (lambda stuff
>       (if (= (system-error-errno stuff) EEXIST)
>           (begin
>             ;; remove old link and retry
>             (delete-file link)
>             (symlink/remove-old link target))
>           (apply throw stuff)))))

Alright, SGTM (this procedure would be kept private).

So Arun, the floor is yours!  :-)

Ludo’.

Re: Move switch-symlinks to (guix build utils)

[bokr]

Hi,

I am not expert on kernel link internals, but if
you need/prefer atomic change to a specific link,
does my log [1] below suggest a way?

On +2022-06-03 21:30:39 +0200, Ludovic Courtès wrote:
> Hi,
> 
> Maxime Devos <maximedevos@telenet.be> skribis:
> 
> >, Maybe replace
> >
> >    (symlink target pivot)
> >
> > by (symlink/remove-old target pivot)
> >
> > where
> >
> > (define (symlink/remove-old target link)
> >   "Make a symbolic link named LINK pointing to TARGET.
> > If LINK already exists, it will be removed first.
> > This is not an atomic operation."
> >   (catch 'system-error
> >     (lambda ()
> >       (symlink target link))
> >     (lambda stuff
> >       (if (= (system-error-errno stuff) EEXIST)
> >           (begin
> >             ;; remove old link and retry
> >             (delete-file link)
> >             (symlink/remove-old link target))
> >           (apply throw stuff)))))
> 
> Alright, SGTM (this procedure would be kept private).
> 
> So Arun, the floor is yours!  :-)
> 
> Ludo’.
>

[0]: (some "<--= note" notes edited in)
BTW: notice that Size: is number of chars in TARGET name, not the LINK name
--8<---------------cut here---------------start------------->8---
$ ln -sT TARGET LINK
$ stat LINK
  File: LINK -> TARGET
  Size: 6               Blocks: 0          IO Block: 4096   symbolic link
Device: fe00h/65024d    Inode: 19401553    Links: 1   # <--= note inode number: mv will not change it
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/    bokr)   Gid: ( 1000/    bokr)
Access: 2022-06-04 00:57:47.680832044 +0200
Modify: 2022-06-04 00:57:47.680832044 +0200
Change: 2022-06-04 00:57:47.680832044 +0200
 Birth: -
$ mv -v LINK WAS-LINK
renamed 'LINK' -> 'WAS-LINK'
$ stat WAS-LINK
  File: WAS-LINK -> TARGET
  Size: 6               Blocks: 0          IO Block: 4096   symbolic link # <--= TARGET is still 6 chars :)
Device: fe00h/65024d    Inode: 19401553    Links: 1    # <--= note inode number: did not change with mv
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/    bokr)   Gid: ( 1000/    bokr)
Access: 2022-06-04 00:57:47.680832044 +0200
Modify: 2022-06-04 00:57:47.680832044 +0200
Change: 2022-06-04 00:58:33.617240104 +0200
 Birth: -
$ ln -sT NEW-TARGET LINK
$ stat LINK 
  File: LINK -> NEW-TARGET
  Size: 10              Blocks: 0          IO Block: 4096   symbolic link # <--= NEW-TARGET is 10 chars :)
Device: fe00h/65024d    Inode: 19401638    Links: 1    <--= note inode number: new for new use of LINK name
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/    bokr)   Gid: ( 1000/    bokr)
Access: 2022-06-04 01:00:10.630121441 +0200
Modify: 2022-06-04 01:00:10.630121441 +0200
Change: 2022-06-04 01:00:10.630121441 +0200
 Birth: -
$ rm -v WAS-LINK      <--= note no OS objection to removing renamed old LINK
removed 'WAS-LINK'
$ stat *LINK*
  File: LINK -> NEW-TARGET
  Size: 10              Blocks: 0          IO Block: 4096   symbolic link
Device: fe00h/65024d    Inode: 19401638    Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/    bokr)   Gid: ( 1000/    bokr)
Access: 2022-06-04 01:00:10.630121441 +0200
Modify: 2022-06-04 01:00:10.630121441 +0200
Change: 2022-06-04 01:00:10.630121441 +0200
 Birth: -
$ 
--8<---------------cut here---------------end--------------->8---

I'm guessing the inode rewrite involved in mv -v LINK WAS-LINK should be atomic?
Maybe some kernels have to struggle without good hardware support, but still atomic?

Maybe a kernel insider will chime in?
HTH some way.
--
Regards,
Bengt Richter

Re: Move switch-symlinks to (guix build utils)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

bokr@bokr.com schreef op za 04-06-2022 om 01:55 [+0200]:
> I am not expert on kernel link internals, but if
> you need/prefer atomic change to a specific link,
> does my log [1] below suggest a way?
> [...]

Only the replacing of an old by a new symlink needs to be atomic, and
this is already the case.  The only non-atomicity is that if
interrupted, an extra symlink [...].new can be created, but except for
tidyness this is not really a problem AFAICT.

Speaking of atomicity, I wonder if 'fsync'ing the symlink or the
directory is required?  E.g., is it possible after a power interruption
for the contents (= target) of the new symlink to be truncated without
doing a 'fsync' before the rename?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Move switch-symlinks to (guix build utils)

[Arun Isaac]

> Maybe replace
>
>    (symlink target pivot)
>
> by (symlink/remove-old target pivot)
>
> where
>
> (define (symlink/remove-old target link)
>   "Make a symbolic link named LINK pointing to TARGET.
> If LINK already exists, it will be removed first.
> This is not an atomic operation."
>   (catch 'system-error
>     (lambda ()
>       (symlink target link))
>     (lambda stuff
>       (if (= (system-error-errno stuff) EEXIST)
>           (begin
>             ;; remove old link and retry
>             (delete-file link)
>             (symlink/remove-old link target))
>           (apply throw stuff)))))
>
> ?

Could we implement symlink/remove-old without catch and throw? Something
like:

--8<---------------cut here---------------start------------->8---
(define (symlink/remove-old target link)
  "Make a symbolic link named LINK pointing to TARGET. If LINK already
exists, it will be removed first.  This is not an atomic operation."
  (when (file-exists? link)
    (delete-file link))
  (symlink target link))
--8<---------------cut here---------------end--------------->8---

We might also inline this into switch-symlinks like so.

--8<---------------cut here---------------start------------->8---
(define (switch-symlinks link target)
  "Atomically switch LINK, a symbolic link, to point to TARGET.  Works
both when LINK already exists and when it does not."
  (let ((pivot (string-append link ".new")))
    ;; Delete pivot link in case it already exists. This can happen if
    ;; a previous switch-symlinks was interrupted.
    (when (file-exists? pivot)
      (delete-file pivot))
    (symlink target pivot)
    (rename-file pivot link)))
--8<---------------cut here---------------end--------------->8---

WDYT?

Re: Move switch-symlinks to (guix build utils)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

Arun Isaac schreef op ma 06-06-2022 om 16:57 [+0530]:
> Could we implement symlink/remove-old without catch and throw? Something
> like:
> 
> --8<---------------cut here---------------start------------->8---
> (define (symlink/remove-old target link)
>   "Make a symbolic link named LINK pointing to TARGET. If LINK already
> exists, it will be removed first.  This is not an atomic operation."
>   (when (file-exists? link)
>     (delete-file link))
>   (symlink target link))
> --8<---------------cut here---------------end--------------->8---

file-exists? does not check if the file exists -- for a broken symlink,
it will return #false instead of #true.

Also, this introduces additional non-atomicity (though in this
particular case, this does not seem to be a problem).  It would also
make things a tiny bit slower, because file-exists? makes an additional
syscall.

For these three reasons, I would go for the more robust catch+throw.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Move switch-symlinks to (guix build utils)

[Arun Isaac]

Hi Maxime,

Thanks for the explanation! I am working on a patch. I'll send something
soon.

Regards,
Arun