Too many levels of symbolic links

Too many levels of symbolic links

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]


Hi Guix,

after upgrading my system it becomes unusable. The guix system
reconfigure completes, but after that, running any command fails with an
"Too many levels of symbolic links" error.

Here are some examples:

```
bash: /root/.config/guix/current/bin/guix:
/gnu/store/dlivmgf8xikvfi6422dggra77lay9g8y-guile-wrapper/bin/guile: bad
interpreter: Too many levels of symbolic links
bash: /run/current-system/profile/bin/sh: Too many levels of symbolic links
bash: /run/current-system/profile/bin/ls: Too many levels of symbolic links
bash: /run/current-system/profile/bin/whoami: Too many levels of symbolic links
bash: /run/current-system/profile/bin/hostname: Too many levels of symbolic links
```

I'm not sure what's going on with the files in
/run/current-system/profile/bin (I can't run ls on them), but after
booting a previous working system, running the guile interpreter in
/gnu/store/dlivmgf8xikvfi6422dggra77lay9g8y-guile-wrapper/bin/guile
seems to be working fine.

I first thought I have too many symlinks on my system. I cleaned up old
generations and collected the garbage, but the problem still
persists. So I think this is not the issue.

Any ideas?

Thanks, Roman.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

Re: Too many levels of symbolic links

[Felix Lechner via]

Hi Roman,

On Sun, Mar 5, 2023 at 8:34 AM Roman Scherer
<roman.scherer@burningswell.com> wrote:
>
> "Too many levels of symbolic links" error.

I think you have circular symbolic links (or possibly just a
poorly-placed link to a folder). [1] Unfortunately, I am not sure how
to examine the issue unless you have basic tools like 'find'
available.

Perhaps you can run the absolute path from the store.

    find /run/current-system/ -follow -printf ""

Kind regards
Felix Lechner

[1] https://serverfault.com/questions/265598/how-do-i-find-circular-symbolic-links/265614#265614

Re: Too many levels of symbolic links

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 1516 bytes --]


Hi Felix,

thanks for your help! Running

  find /run/current-system/ -follow -printf ""

on my current system does not find anything. When inside the broken
system, find fails with the same error message.

However, running

  find /gnu/store/ -follow -printf ""
  find: File system loop detected; ‘/gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo/posix’ is part of the same file system loop as ‘/gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo’.

detects that the file

  /gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo/posix

links to itself. Since I read I should not mess with the store, any
ideas how to get rid of that file system loop in a safe way?

I already tried re-installing it, but that had no effect.

Roman.

Felix Lechner <felix.lechner@lease-up.com> writes:

> Hi Roman,
>
> On Sun, Mar 5, 2023 at 8:34 AM Roman Scherer
> <roman.scherer@burningswell.com> wrote:
>>
>> "Too many levels of symbolic links" error.
>
> I think you have circular symbolic links (or possibly just a
> poorly-placed link to a folder). [1] Unfortunately, I am not sure how
> to examine the issue unless you have basic tools like 'find'
> available.
>
> Perhaps you can run the absolute path from the store.
>
>     find /run/current-system/ -follow -printf ""
>
> Kind regards
> Felix Lechner
>
> [1] https://serverfault.com/questions/265598/how-do-i-find-circular-symbolic-links/265614#265614

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

Re: Too many levels of symbolic links

[Felix Lechner via]

Hi Roman,

On Sun, Mar 5, 2023 at 9:03 AM Roman Scherer
<roman.scherer@burningswell.com> wrote:
>
> the file
>
>   /gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo/posix
>
> links to itself.

I also have that loop in
/gnu/store/7pjzwj9d4fnyzp9x7k8cc4hazypyrk0p-tzdata-2022a/ but do not
believe it is the source of your problems. I would keep looking around
your profile, which are the symbolic links into the store that provide
a traditional file system layout for user interactions.

Kind regards
Felix Lechner

Re: Too many levels of symbolic links

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

Hi Roman,

Roman Scherer 写道：
>   /gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo/posix
>
> links to itself.

FYI, it should link to its parent:

  /gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo

(So, yes, you could …/posix/posix/posix/posix/… all night long.)

> Since I read I should not mess with the store, any
> ideas how to get rid of that file system loop in a safe way?

You could try

  # guix gc --verify=contents,repair

and Guix should detect the mismatch and refresh tzdata from a 
substitute server.

The unsupported alternative would be to bind-mount /gnu/store 
somewhere and manually relink the file.

> I already tried re-installing it, but that had no effect.

It wouldn't.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Too many levels of symbolic links

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 2604 bytes --]


Hi Tobias and Felix,

the posix file was actually pointing to the correct parent directory (my bad)
and as Felix mentioned probably not the root of my issue.

The `guix gc --verify=contents,repair` unfortunatly did not help either.

I looked a bit around and found this in /var/guix/profiles

drwxr-xr-x 4 root root 4096 Jan 21 16:28 per-user/
lrwxrwxrwx 1 root root   14 Mar  8 16:20 system -> system-53-link
lrwxrwxrwx 1 root root   50 Feb 12 14:58 system-49-link -> /gnu/store/r048wnvv5wb792d0h29147j4q2m874d0-system
lrwxrwxrwx 1 root root   50 Mar  8 15:17 system-50-link -> /gnu/store/99kgb2jh7jdjcvqac81dj0gd2nkhanmw-system
lrwxrwxrwx 1 root root   50 Feb 20 12:30 system-51-link -> /gnu/store/cwk04xkh3zf7vb5bg3cpjr8jyviv5b2f-system
lrwxrwxrwx 1 root root   50 Feb 25 14:16 system-52-link -> /gnu/store/aqxlpw468wvnardbb8dmyxj1qghv68ah-system
lrwxrwxrwx 1 root root   50 Mar  3 16:33 system-53-link -> /gnu/store/1xdmrafzscrpi34rynag71my6nw3w8w2-system
lrwxrwxrwx 1 root root   50 Mar  8 15:53 system-54-link -> /gnu/store/cjdabnlk74irjh3xwqh9d31kza3c78a0-system

Right now I rolled back to generation 53, which is still
working. Generation 54 is the broken one, which does boot up, but
logging in fails with the too many levels of symbolic link issue.

Is it correct to look for the problematic files in the directory to
which system-54-link links to?

I run the problematic commands from that directory and they seem to run
from the non-broken system

I'm thinking of re-installing my system completely. Do I have any other
option, even an unsupported one?

Thanks again for your help,

Roman.

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> [[PGP Signed Part:Undecided]]
> Hi Roman,
>
> Roman Scherer 写道：
>>   /gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo/posix
>>
>> links to itself.
>
> FYI, it should link to its parent:
>
>  /gnu/store/097dmm40lhcf777acqh5i660j4i09k85-tzdata-2022a/share/zoneinfo
>
> (So, yes, you could …/posix/posix/posix/posix/… all night long.)
>
>> Since I read I should not mess with the store, any
>> ideas how to get rid of that file system loop in a safe way?
>
> You could try
>
>  # guix gc --verify=contents,repair
>
> and Guix should detect the mismatch and refresh tzdata from a
> substitute server.
>
> The unsupported alternative would be to bind-mount /gnu/store
> somewhere and manually relink the file.
>
>> I already tried re-installing it, but that had no effect.
>
> It wouldn't.
>
> Kind regards,
>
> T G-R
>
> [[End of PGP Signed Part]]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

Re: Too many levels of symbolic links

[Felix Lechner via]

Hi Roman,

On Fri, Mar 10, 2023 at 7:32 AM Roman Scherer
<roman.scherer@burningswell.com> wrote:
>
> Right now I rolled back to generation 53, which is still
> working.

Great!

> Is it correct to look for the problematic files in the directory to
> which system-54-link links to?

Yes, probably. Unfortunately, profiles are large bundles of symbolic
links, so finding the offending cycle may not be easy.

> Generation 54 is the broken one, which does boot up, but
> logging in fails with the too many levels of symbolic link issue.

Since the issue occurs at login, there may also be a bad interaction
with your user profile. What if you log in as another user?

> I run the problematic commands from that directory and they seem to run
> from the non-broken system

My prime list of suspects include something like a system-wide Bash
profile (or whichever shell you use).

> I'm thinking of re-installing my system completely.

Please don't do that. It is not necessary.

Instead, you can delete generation 54 and try again. Make sure to
'guix pull' before reconfiguring again.

> Do I have any other option, even an unsupported one?

Please post the config.scm for your generation 53, plus a diff to
generation 54. You find copies via 'guix system describe'. It may also
be helpful to attach your home.scm, if you use Guix Home.

Together with the changes in Guix since the commit on which generation
53 is based, we will find the issue. The process is more or less
deterministic.

Please keep up hope. All needed software is already installed. Guix
has extraordinary ways to recover—even though they can be hard to
figure out.

Kind regards
Felix

Re: Too many levels of symbolic links

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 2952 bytes --]


Felix, I'm back on track! I found what triggered this horrible problem :)

At some point I added the qemu-binfmt-service-type to my system. I
copied a snipped from the manual and finally "adjusted" it to this:

```
(define %qemu-service
  (service qemu-binfmt-service-type
           (qemu-binfmt-configuration
            (platforms (lookup-qemu-platforms "aarch64" "x86_64")))))
```

I don't know much about quem and I thought, well let's choose the
"aarch64" and "x86_64" qemu platforms on which I run Guix, and maybe
re-use this service on multiple machines.

My Guix system runs on the aarch64 architecture. It turns out that
having "aarch64" in lookup-qemu-platforms causes the "Too many levels of
symbolic links" error. As soon as I remove it, everything is good again.

Now, maybe it does not even makes sense to add "aarch64" to that list,
but I think it should not fail that catastrophical. I still don't know
which symlink was causing this issue, though.

That's where I am. Just happy that I survived my first serious Guix
crash. :)

Thanks for your help so far,

Roman

Felix Lechner <felix.lechner@lease-up.com> writes:

> Hi Roman,
>
> On Fri, Mar 10, 2023 at 7:32 AM Roman Scherer
> <roman.scherer@burningswell.com> wrote:
>>
>> Right now I rolled back to generation 53, which is still
>> working.
>
> Great!
>
>> Is it correct to look for the problematic files in the directory to
>> which system-54-link links to?
>
> Yes, probably. Unfortunately, profiles are large bundles of symbolic
> links, so finding the offending cycle may not be easy.
>
>> Generation 54 is the broken one, which does boot up, but
>> logging in fails with the too many levels of symbolic link issue.
>
> Since the issue occurs at login, there may also be a bad interaction
> with your user profile. What if you log in as another user?
>
>> I run the problematic commands from that directory and they seem to run
>> from the non-broken system
>
> My prime list of suspects include something like a system-wide Bash
> profile (or whichever shell you use).
>
>> I'm thinking of re-installing my system completely.
>
> Please don't do that. It is not necessary.
>
> Instead, you can delete generation 54 and try again. Make sure to
> 'guix pull' before reconfiguring again.
>
>> Do I have any other option, even an unsupported one?
>
> Please post the config.scm for your generation 53, plus a diff to
> generation 54. You find copies via 'guix system describe'. It may also
> be helpful to attach your home.scm, if you use Guix Home.
>
> Together with the changes in Guix since the commit on which generation
> 53 is based, we will find the issue. The process is more or less
> deterministic.
>
> Please keep up hope. All needed software is already installed. Guix
> has extraordinary ways to recover—even though they can be hard to
> figure out.
>
> Kind regards
> Felix

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

Re: Too many levels of symbolic links

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]

On Tue, Mar 14, 2023 at 08:16:52PM +0100, Roman Scherer wrote:
> 
> Felix, I'm back on track! I found what triggered this horrible problem :)
> 
> At some point I added the qemu-binfmt-service-type to my system. I
> copied a snipped from the manual and finally "adjusted" it to this:
> 
> ```
> (define %qemu-service
>   (service qemu-binfmt-service-type
>            (qemu-binfmt-configuration
>             (platforms (lookup-qemu-platforms "aarch64" "x86_64")))))
> ```
> 
> I don't know much about quem and I thought, well let's choose the
> "aarch64" and "x86_64" qemu platforms on which I run Guix, and maybe
> re-use this service on multiple machines.
> 
> My Guix system runs on the aarch64 architecture. It turns out that
> having "aarch64" in lookup-qemu-platforms causes the "Too many levels of
> symbolic links" error. As soon as I remove it, everything is good again.
> 
> Now, maybe it does not even makes sense to add "aarch64" to that list,
> but I think it should not fail that catastrophical. I still don't know
> which symlink was causing this issue, though.
> 
> That's where I am. Just happy that I survived my first serious Guix
> crash. :)
> 
> Thanks for your help so far,
> 
> Roman

I'm surprised that it worked for you. One of the unspoken rules of qemu
binfmt emulation is you don't emulate your own hardware. I'm also
surprised because in the past I tried to enable binfmt emulation on my
pinebook pro and the qemu-binfmt-service-type was the difference between
booting and not booting.



-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Too many levels of symbolic links

[Tobias Geerinckx-Rice]

Roman,

Nice.

> I still don't know which symlink was causing this issue, though.

Don't take unix error messages seriously, they probably weren't intended that way.

The #define for the error string you saw is ELOOP, so that should answer your question.

Kind regards,

T G-R

Sent on the go.  Excuse or enjoy my brevity.

Re: Too many levels of symbolic links

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]


Good to know about that unspoken rule. :) Should we do something about
that? I thought about adding a note to the manual. Or even not allowing
the user to re-configure in such a case?

WDYT?

Efraim Flashner <efraim@flashner.co.il> writes:

> [[PGP Signed Part:Undecided]]
> On Tue, Mar 14, 2023 at 08:16:52PM +0100, Roman Scherer wrote:
>>
>> Felix, I'm back on track! I found what triggered this horrible problem :)
>>
>> At some point I added the qemu-binfmt-service-type to my system. I
>> copied a snipped from the manual and finally "adjusted" it to this:
>>
>> ```
>> (define %qemu-service
>>   (service qemu-binfmt-service-type
>>            (qemu-binfmt-configuration
>>             (platforms (lookup-qemu-platforms "aarch64" "x86_64")))))
>> ```
>>
>> I don't know much about quem and I thought, well let's choose the
>> "aarch64" and "x86_64" qemu platforms on which I run Guix, and maybe
>> re-use this service on multiple machines.
>>
>> My Guix system runs on the aarch64 architecture. It turns out that
>> having "aarch64" in lookup-qemu-platforms causes the "Too many levels of
>> symbolic links" error. As soon as I remove it, everything is good again.
>>
>> Now, maybe it does not even makes sense to add "aarch64" to that list,
>> but I think it should not fail that catastrophical. I still don't know
>> which symlink was causing this issue, though.
>>
>> That's where I am. Just happy that I survived my first serious Guix
>> crash. :)
>>
>> Thanks for your help so far,
>>
>> Roman
>
> I'm surprised that it worked for you. One of the unspoken rules of qemu
> binfmt emulation is you don't emulate your own hardware. I'm also
> surprised because in the past I tried to enable binfmt emulation on my
> pinebook pro and the qemu-binfmt-service-type was the difference between
> booting and not booting.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]