bug#24703: fontconfig keeps obfuscated reference to itself, not grafted

bug#24703: fontconfig keeps obfuscated reference to itself, not grafted

[Mark H Weaver]

After running "guix gc", I've found that my fonts are broken.

Gnome-terminal, Xfce4-terminal, and xterm are no longer able to find my
preferred "Dejavu Sans Mono" font.  IceCat also uses a different
selection of fonts.  Of the programs I've tried, only Emacs is able to
find them.

The failing programs output the following message to stderr:

  Fontconfig error: Cannot load default config file

Strace reveals:

  access("/gnu/store/b484nvn9nnr3ddclpz2fma9yxmimg2jj-fontconfig-2.11.94/etc/fonts/fonts.conf", R_OK) = -1 ENOENT (No such file or directory)
  write(2, "Fontconfig error: ", 18Fontconfig error: )      = 18
  write(2, "Cannot load default config file", 31Cannot load default config file) = 31

On my system, which includes a graft for 'bash' (not yet pushed to
Savannah), 'fontconfig' is grafted.  After running 'guix gc', the
original ungrafted 'fontconfig' was removed, but libfontconfig is trying
to access it above.

It turns out there's an obfuscated self-reference to fontconfig's store
directory.  Here's an excerpt of the output of "hexdump -C
libfontconfig.so.1.9.0":

0000cca0  00 48 b9 2f 67 6e 75 2f  73 74 6f c6 40 48 00 45  |.H./gnu/sto.@H.E|
0000ccb0  31 e4 48 89 08 48 b9 72  65 2f 62 34 38 34 6e 48  |1.H..H.re/b484nH|
0000ccc0  89 48 08 48 b9 76 6e 39  6e 6e 72 33 64 48 89 48  |.H.H.vn9nnr3dH.H|
0000ccd0  10 48 b9 64 63 6c 70 7a  32 66 6d 48 89 48 18 48  |.H.dclpz2fmH.H.H|
0000cce0  b9 61 39 79 78 6d 69 6d  67 48 89 48 20 48 b9 32  |.a9yxmimgH.H H.2|
0000ccf0  6a 6a 2d 66 6f 6e 74 48  89 48 28 48 b9 63 6f 6e  |jj-fontH.H(H.con|
0000cd00  66 69 67 2d 32 48 89 48  30 48 b9 2e 31 31 2e 39  |fig-2H.H0H..11.9|
0000cd10  34 2f 65 48 89 48 38 48  b9 74 63 2f 66 6f 6e 74  |4/eH.H8H.tc/font|
0000cd20  73 48 89 48 40 48 8b 04  24 48 8b 18 48 89 c5 48  |sH.H@H..$H..H..H|

To be continued...

     Mark

bug#24703: fontconfig keeps obfuscated reference to itself, not grafted

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> It turns out there's an obfuscated self-reference to fontconfig's store
> directory.  Here's an excerpt of the output of "hexdump -C
> libfontconfig.so.1.9.0":
>
> 0000cca0  00 48 b9 2f 67 6e 75 2f  73 74 6f c6 40 48 00 45  |.H./gnu/sto.@H.E|
> 0000ccb0  31 e4 48 89 08 48 b9 72  65 2f 62 34 38 34 6e 48  |1.H..H.re/b484nH|
> 0000ccc0  89 48 08 48 b9 76 6e 39  6e 6e 72 33 64 48 89 48  |.H.H.vn9nnr3dH.H|
> 0000ccd0  10 48 b9 64 63 6c 70 7a  32 66 6d 48 89 48 18 48  |.H.dclpz2fmH.H.H|
> 0000cce0  b9 61 39 79 78 6d 69 6d  67 48 89 48 20 48 b9 32  |.a9yxmimgH.H H.2|
> 0000ccf0  6a 6a 2d 66 6f 6e 74 48  89 48 28 48 b9 63 6f 6e  |jj-fontH.H(H.con|
> 0000cd00  66 69 67 2d 32 48 89 48  30 48 b9 2e 31 31 2e 39  |fig-2H.H0H..11.9|
> 0000cd10  34 2f 65 48 89 48 38 48  b9 74 63 2f 66 6f 6e 74  |4/eH.H8H.tc/font|
> 0000cd20  73 48 89 48 40 48 8b 04  24 48 8b 18 48 89 c5 48  |sH.H@H..$H..H..H|

It turns out that this is part of the compiled x86_64 code for
'FcConfigFilename' in src/fccfg.c, which copies a compile-time string
constant, 8 bytes at a time, into a buffer:

--8<---------------cut here---------------start------------->8---
$ objdump -d libfontconfig.so.1.9.0 | grep -B1 -A35 '48 b9 2f 67 6e 75 2f'
    cc9b:	0f 84 3f 01 00 00    	je     cde0 <FcConfigFilename+0x2d0>
    cca1:	48 b9 2f 67 6e 75 2f 	movabs $0x6f74732f756e672f,%rcx
    cca8:	73 74 6f 
    ccab:	c6 40 48 00          	movb   $0x0,0x48(%rax)
    ccaf:	45 31 e4             	xor    %r12d,%r12d
    ccb2:	48 89 08             	mov    %rcx,(%rax)
    ccb5:	48 b9 72 65 2f 62 34 	movabs $0x6e343834622f6572,%rcx
    ccbc:	38 34 6e 
    ccbf:	48 89 48 08          	mov    %rcx,0x8(%rax)
    ccc3:	48 b9 76 6e 39 6e 6e 	movabs $0x6433726e6e396e76,%rcx
    ccca:	72 33 64 
    cccd:	48 89 48 10          	mov    %rcx,0x10(%rax)
    ccd1:	48 b9 64 63 6c 70 7a 	movabs $0x6d66327a706c6364,%rcx
    ccd8:	32 66 6d 
    ccdb:	48 89 48 18          	mov    %rcx,0x18(%rax)
    ccdf:	48 b9 61 39 79 78 6d 	movabs $0x676d696d78793961,%rcx
    cce6:	69 6d 67 
    cce9:	48 89 48 20          	mov    %rcx,0x20(%rax)
    cced:	48 b9 32 6a 6a 2d 66 	movabs $0x746e6f662d6a6a32,%rcx
    ccf4:	6f 6e 74 
    ccf7:	48 89 48 28          	mov    %rcx,0x28(%rax)
    ccfb:	48 b9 63 6f 6e 66 69 	movabs $0x322d6769666e6f63,%rcx
    cd02:	67 2d 32 
    cd05:	48 89 48 30          	mov    %rcx,0x30(%rax)
    cd09:	48 b9 2e 31 31 2e 39 	movabs $0x652f34392e31312e,%rcx
    cd10:	34 2f 65 
    cd13:	48 89 48 38          	mov    %rcx,0x38(%rax)
    cd17:	48 b9 74 63 2f 66 6f 	movabs $0x73746e6f662f6374,%rcx
    cd1e:	6e 74 73 
    cd21:	48 89 48 40          	mov    %rcx,0x40(%rax)
    cd25:	48 8b 04 24          	mov    (%rsp),%rax
    cd29:	48 8b 18             	mov    (%rax),%rbx
    cd2c:	48 89 c5             	mov    %rax,%rbp
    cd2f:	48 85 db             	test   %rbx,%rbx
    cd32:	48 89 df             	mov    %rbx,%rdi
    cd35:	75 16                	jne    cd4d <FcConfigFilename+0x23d>
    cd37:	eb 44                	jmp    cd7d <FcConfigFilename+0x26d>
--8<---------------cut here---------------end--------------->8---

So far, I've not been able to find any evidence of the fontconfig code
doing anything strange here.  I strongly suspect that GCC is generating
this code, most likely due to an inlinable string/memory copy function
where the source is a string literal.

Obviously, this could be a serious problem for Guix (and Nix), since it
suggests that we may not be able to continue with our simplistic
assumption that references to the store in compiled code will be easy to
find and replace.

      Mark

bug#24703: fontconfig keeps obfuscated reference to itself, not grafted

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> It turns out there's an obfuscated self-reference to fontconfig's store
>> directory.  Here's an excerpt of the output of "hexdump -C
>> libfontconfig.so.1.9.0":
>>
>> 0000cca0  00 48 b9 2f 67 6e 75 2f  73 74 6f c6 40 48 00 45  |.H./gnu/sto.@H.E|
>> 0000ccb0  31 e4 48 89 08 48 b9 72  65 2f 62 34 38 34 6e 48  |1.H..H.re/b484nH|
>> 0000ccc0  89 48 08 48 b9 76 6e 39  6e 6e 72 33 64 48 89 48  |.H.H.vn9nnr3dH.H|
>> 0000ccd0  10 48 b9 64 63 6c 70 7a  32 66 6d 48 89 48 18 48  |.H.dclpz2fmH.H.H|
>> 0000cce0  b9 61 39 79 78 6d 69 6d  67 48 89 48 20 48 b9 32  |.a9yxmimgH.H H.2|
>> 0000ccf0  6a 6a 2d 66 6f 6e 74 48  89 48 28 48 b9 63 6f 6e  |jj-fontH.H(H.con|
>> 0000cd00  66 69 67 2d 32 48 89 48  30 48 b9 2e 31 31 2e 39  |fig-2H.H0H..11.9|
>> 0000cd10  34 2f 65 48 89 48 38 48  b9 74 63 2f 66 6f 6e 74  |4/eH.H8H.tc/font|
>> 0000cd20  73 48 89 48 40 48 8b 04  24 48 8b 18 48 89 c5 48  |sH.H@H..$H..H..H|
>
> It turns out that this is part of the compiled x86_64 code for
> 'FcConfigFilename' in src/fccfg.c, which copies a compile-time string
> constant, 8 bytes at a time, into a buffer:
>
> $ objdump -d libfontconfig.so.1.9.0 | grep -B1 -A35 '48 b9 2f 67 6e 75 2f'
>     cc9b:	0f 84 3f 01 00 00    	je     cde0 <FcConfigFilename+0x2d0>
>     cca1:	48 b9 2f 67 6e 75 2f 	movabs $0x6f74732f756e672f,%rcx
>     cca8:	73 74 6f 
>     ccab:	c6 40 48 00          	movb   $0x0,0x48(%rax)
>     ccaf:	45 31 e4             	xor    %r12d,%r12d
>     ccb2:	48 89 08             	mov    %rcx,(%rax)
>     ccb5:	48 b9 72 65 2f 62 34 	movabs $0x6e343834622f6572,%rcx
>     ccbc:	38 34 6e 
>     ccbf:	48 89 48 08          	mov    %rcx,0x8(%rax)
>     ccc3:	48 b9 76 6e 39 6e 6e 	movabs $0x6433726e6e396e76,%rcx
>     ccca:	72 33 64 
>     cccd:	48 89 48 10          	mov    %rcx,0x10(%rax)
>     ccd1:	48 b9 64 63 6c 70 7a 	movabs $0x6d66327a706c6364,%rcx
>     ccd8:	32 66 6d 
>     ccdb:	48 89 48 18          	mov    %rcx,0x18(%rax)
>     ccdf:	48 b9 61 39 79 78 6d 	movabs $0x676d696d78793961,%rcx
>     cce6:	69 6d 67 
>     cce9:	48 89 48 20          	mov    %rcx,0x20(%rax)
>     cced:	48 b9 32 6a 6a 2d 66 	movabs $0x746e6f662d6a6a32,%rcx
>     ccf4:	6f 6e 74 
>     ccf7:	48 89 48 28          	mov    %rcx,0x28(%rax)
>     ccfb:	48 b9 63 6f 6e 66 69 	movabs $0x322d6769666e6f63,%rcx
>     cd02:	67 2d 32 
>     cd05:	48 89 48 30          	mov    %rcx,0x30(%rax)
>     cd09:	48 b9 2e 31 31 2e 39 	movabs $0x652f34392e31312e,%rcx
>     cd10:	34 2f 65 
>     cd13:	48 89 48 38          	mov    %rcx,0x38(%rax)
>     cd17:	48 b9 74 63 2f 66 6f 	movabs $0x73746e6f662f6374,%rcx
>     cd1e:	6e 74 73 
>     cd21:	48 89 48 40          	mov    %rcx,0x40(%rax)
>     cd25:	48 8b 04 24          	mov    (%rsp),%rax
>     cd29:	48 8b 18             	mov    (%rax),%rbx
>     cd2c:	48 89 c5             	mov    %rax,%rbp
>     cd2f:	48 85 db             	test   %rbx,%rbx
>     cd32:	48 89 df             	mov    %rbx,%rdi
>     cd35:	75 16                	jne    cd4d <FcConfigFilename+0x23d>
>     cd37:	eb 44                	jmp    cd7d <FcConfigFilename+0x26d>
>
> So far, I've not been able to find any evidence of the fontconfig code
> doing anything strange here.  I strongly suspect that GCC is generating
> this code, most likely due to an inlinable string/memory copy function
> where the source is a string literal.

I've confirmed this.  After building this package manually, "objdump -d
--source src/.libs/fccfg.o" reveals that the corresponding source code
is:

    dir = (FcChar8 *) FONTCONFIG_PATH;
    path[i] = malloc (strlen ((char *) dir) + 1);
    if (!path[i])
	goto bail1;
    strcpy ((char *) path[i], (const char *) dir);

It is part of 'FcConfigGetPath', inlined into 'FcConfigFilename', in
src/fccfg.c.  -DFONTCONFIG_PATH='"$(BASECONFIGDIR)"' is one of the flags
passed to GCC, via AM_CPPFLAGS in src/Makefile.am.

> Obviously, this could be a serious problem for Guix (and Nix), since it
> suggests that we may not be able to continue with our simplistic
> assumption that references to the store in compiled code will be easy to
> find and replace.

      Mark

bug#24703: Store references in 8-byte chunks in compiled code

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> It turns out there's an obfuscated self-reference to fontconfig's store
>>> directory.  Here's an excerpt of the output of "hexdump -C
>>> libfontconfig.so.1.9.0":
>>>
>>> 0000cca0  00 48 b9 2f 67 6e 75 2f  73 74 6f c6 40 48 00 45  |.H./gnu/sto.@H.E|
>>> 0000ccb0  31 e4 48 89 08 48 b9 72  65 2f 62 34 38 34 6e 48  |1.H..H.re/b484nH|
>>> 0000ccc0  89 48 08 48 b9 76 6e 39  6e 6e 72 33 64 48 89 48  |.H.H.vn9nnr3dH.H|
>>> 0000ccd0  10 48 b9 64 63 6c 70 7a  32 66 6d 48 89 48 18 48  |.H.dclpz2fmH.H.H|
>>> 0000cce0  b9 61 39 79 78 6d 69 6d  67 48 89 48 20 48 b9 32  |.a9yxmimgH.H H.2|
>>> 0000ccf0  6a 6a 2d 66 6f 6e 74 48  89 48 28 48 b9 63 6f 6e  |jj-fontH.H(H.con|
>>> 0000cd00  66 69 67 2d 32 48 89 48  30 48 b9 2e 31 31 2e 39  |fig-2H.H0H..11.9|
>>> 0000cd10  34 2f 65 48 89 48 38 48  b9 74 63 2f 66 6f 6e 74  |4/eH.H8H.tc/font|
>>> 0000cd20  73 48 89 48 40 48 8b 04  24 48 8b 18 48 89 c5 48  |sH.H@H..$H..H..H|
>>
>> It turns out that this is part of the compiled x86_64 code for
>> 'FcConfigFilename' in src/fccfg.c, which copies a compile-time string
>> constant, 8 bytes at a time, into a buffer:
>>
>> $ objdump -d libfontconfig.so.1.9.0 | grep -B1 -A35 '48 b9 2f 67 6e 75 2f'
>>     cc9b:	0f 84 3f 01 00 00    	je     cde0 <FcConfigFilename+0x2d0>
>>     cca1:	48 b9 2f 67 6e 75 2f 	movabs $0x6f74732f756e672f,%rcx
>>     cca8:	73 74 6f 
>>     ccab:	c6 40 48 00          	movb   $0x0,0x48(%rax)
>>     ccaf:	45 31 e4             	xor    %r12d,%r12d
>>     ccb2:	48 89 08             	mov    %rcx,(%rax)
>>     ccb5:	48 b9 72 65 2f 62 34 	movabs $0x6e343834622f6572,%rcx
>>     ccbc:	38 34 6e 
>>     ccbf:	48 89 48 08          	mov    %rcx,0x8(%rax)
>>     ccc3:	48 b9 76 6e 39 6e 6e 	movabs $0x6433726e6e396e76,%rcx
>>     ccca:	72 33 64 
>>     cccd:	48 89 48 10          	mov    %rcx,0x10(%rax)
>>     ccd1:	48 b9 64 63 6c 70 7a 	movabs $0x6d66327a706c6364,%rcx
>>     ccd8:	32 66 6d 
>>     ccdb:	48 89 48 18          	mov    %rcx,0x18(%rax)
>>     ccdf:	48 b9 61 39 79 78 6d 	movabs $0x676d696d78793961,%rcx
>>     cce6:	69 6d 67 
>>     cce9:	48 89 48 20          	mov    %rcx,0x20(%rax)
>>     cced:	48 b9 32 6a 6a 2d 66 	movabs $0x746e6f662d6a6a32,%rcx
>>     ccf4:	6f 6e 74 
>>     ccf7:	48 89 48 28          	mov    %rcx,0x28(%rax)
>>     ccfb:	48 b9 63 6f 6e 66 69 	movabs $0x322d6769666e6f63,%rcx
>>     cd02:	67 2d 32 
>>     cd05:	48 89 48 30          	mov    %rcx,0x30(%rax)
>>     cd09:	48 b9 2e 31 31 2e 39 	movabs $0x652f34392e31312e,%rcx
>>     cd10:	34 2f 65 
>>     cd13:	48 89 48 38          	mov    %rcx,0x38(%rax)
>>     cd17:	48 b9 74 63 2f 66 6f 	movabs $0x73746e6f662f6374,%rcx
>>     cd1e:	6e 74 73 
>>     cd21:	48 89 48 40          	mov    %rcx,0x40(%rax)
>>     cd25:	48 8b 04 24          	mov    (%rsp),%rax
>>     cd29:	48 8b 18             	mov    (%rax),%rbx
>>     cd2c:	48 89 c5             	mov    %rax,%rbp
>>     cd2f:	48 85 db             	test   %rbx,%rbx
>>     cd32:	48 89 df             	mov    %rbx,%rdi
>>     cd35:	75 16                	jne    cd4d <FcConfigFilename+0x23d>
>>     cd37:	eb 44                	jmp    cd7d <FcConfigFilename+0x26d>
>>
>> So far, I've not been able to find any evidence of the fontconfig code
>> doing anything strange here.  I strongly suspect that GCC is generating
>> this code, most likely due to an inlinable string/memory copy function
>> where the source is a string literal.
>
> I've confirmed this.  After building this package manually, "objdump -d
> --source src/.libs/fccfg.o" reveals that the corresponding source code
> is:
>
>     dir = (FcChar8 *) FONTCONFIG_PATH;
>     path[i] = malloc (strlen ((char *) dir) + 1);
>     if (!path[i])
> 	goto bail1;
>     strcpy ((char *) path[i], (const char *) dir);
>
> It is part of 'FcConfigGetPath', inlined into 'FcConfigFilename', in
> src/fccfg.c.  -DFONTCONFIG_PATH='"$(BASECONFIGDIR)"' is one of the flags
> passed to GCC, via AM_CPPFLAGS in src/Makefile.am.
>
>> Obviously, this could be a serious problem for Guix (and Nix), since it
>> suggests that we may not be able to continue with our simplistic
>> assumption that references to the store in compiled code will be easy to
>> find and replace.

To get an idea of how widespread this problem is, I searched for
occurrences of /gnu/sto[^r] on my GNOME/Xfce desktop system:

$ LC_ALL=C grep '/gnu/sto[^r]' --recursive /gnu/store/
[.../gnu/store/.links hits removed...]
Binary file /gnu/store/0010wvgs40kdq8chzsh403qm7la9jxq7-bash-static-4.3.42/bin/bash matches
Binary file /gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/sbin/sln matches
Binary file /gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/lib/libc-2.23.so matches
Binary file /gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/lib/libc.a matches
Binary file /gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/bin/locale matches
Binary file /gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/bin/localedef matches
Binary file /gnu/store/4fs8dg5jhf99xl3ikl093dn1va4wlv79-fontconfig-2.11.94/lib/libfontconfig.so.1.9.0 matches
Binary file /gnu/store/b484nvn9nnr3ddclpz2fma9yxmimg2jj-fontconfig-2.11.94/lib/libfontconfig.so.1.9.0 matches
Binary file /gnu/store/80m11l8km7bwi9iljfyr82hmscaq4xk2-unixodbc-2.3.4/lib/libodbcinst.so.2.0.0 matches
Binary file /gnu/store/80m11l8km7bwi9iljfyr82hmscaq4xk2-unixodbc-2.3.4/lib/libodbc.so.2.0.0 matches
Binary file /gnu/store/ld16jy012l3jpkj6azynzmldxn28cspw-ncurses-6.0/lib/libncursesw.so.6.0 matches
Binary file /gnu/store/ld16jy012l3jpkj6azynzmldxn28cspw-ncurses-6.0/lib/libncursesw.a matches
Binary file /gnu/store/lxv20br9ic8abvqd7fipszbs4mg8hkjv-ncurses-6.0/lib/libncursesw.so.6.0 matches
Binary file /gnu/store/lxv20br9ic8abvqd7fipszbs4mg8hkjv-ncurses-6.0/lib/libncursesw.a matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/gcc matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-gcc-4.9.3 matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-c++ matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-gcc matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/g++ matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-g++ matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/cpp matches
Binary file /gnu/store/frrj3bfbmg5vrd0flh9cf8j64h7cr2v4-gcc-4.9.3/bin/c++ matches
Binary file /gnu/store/sxa3b6l28ckcnyg3g7f4brvl2rdpncy1-gnuplot-5.0.4-1/bin/gnuplot matches
Binary file /gnu/store/2jvvwqz16hj2c5ws0xa46x98fvl9m90m-fontconfig-2.11.94/lib/libfontconfig.so.1.9.0 matches
Binary file /gnu/store/bxy1hwriqzdw6xk7bl28qfsw4s1s5xdq-bash-static-4.3.48/bin/bash matches
Binary file /gnu/store/3chnjjkazbv0fqbshrwahq7c3zfg42s8-ncurses-6.0/lib/libncursesw.so.6.0 matches
Binary file /gnu/store/3chnjjkazbv0fqbshrwahq7c3zfg42s8-ncurses-6.0/lib/libncursesw.a matches
Binary file /gnu/store/my7f7fq2ca5rqq4wyyrg20cw2bjrj2l4-ncurses-6.0/lib/libncursesw.so.6.0 matches
Binary file /gnu/store/my7f7fq2ca5rqq4wyyrg20cw2bjrj2l4-ncurses-6.0/lib/libncursesw.a matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/gcc matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-gcc-4.9.3 matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-c++ matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-gcc matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/g++ matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-g++ matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/cpp matches
Binary file /gnu/store/dmz9v8bmxd3davz77s4b10pmpmjnv98a-gcc-4.9.3/bin/c++ matches
Binary file /gnu/store/fjsaprcdmdn39pk39jrhbby1jl5i8rp5-xorg-server-1.18.1/bin/Xorg matches
Binary file /gnu/store/mz301kb7wqvyl9kxil4bpn8ng99ikgqy-glibc-2.23/sbin/sln matches
Binary file /gnu/store/mz301kb7wqvyl9kxil4bpn8ng99ikgqy-glibc-2.23/lib/libc-2.23.so matches
Binary file /gnu/store/mz301kb7wqvyl9kxil4bpn8ng99ikgqy-glibc-2.23/lib/libc.a matches
Binary file /gnu/store/mz301kb7wqvyl9kxil4bpn8ng99ikgqy-glibc-2.23/bin/locale matches
Binary file /gnu/store/mz301kb7wqvyl9kxil4bpn8ng99ikgqy-glibc-2.23/bin/localedef matches
Binary file /gnu/store/slhwk75g7d8bywpq2hifs7g9fxr6jx3d-bash-static-4.3.48/bin/bash matches
Binary file /gnu/store/qjsmp5s85qrba18fxf319m5lv7f8awf8-graphviz-2.38.0/bin/lefty matches
Binary file /gnu/store/6ddfp4pwlrxc5jdgf17ddh4m5wi1cldy-xorg-server-1.18.1/bin/Xorg matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/gcc matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-gcc-4.9.3 matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-c++ matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-gcc matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/g++ matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/x86_64-unknown-linux-gnu-g++ matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/cpp matches
Binary file /gnu/store/zav1zqwmzzz5xk71v22i7n6qidwh49in-gcc-4.9.3/bin/c++ matches
Binary file /gnu/store/xl19qrfzga52vrvp4ncccwjlnrjqwj95-ncurses-6.0/lib/libncursesw.so.6.0 matches
Binary file /gnu/store/xl19qrfzga52vrvp4ncccwjlnrjqwj95-ncurses-6.0/lib/libncursesw.a matches
Binary file /gnu/store/kh3awka9xslyp52dldb3gma47rr0kp2x-xorg-server-1.18.1/bin/Xorg matches
Binary file /gnu/store/121596cgx25s8zcl3yznyh2vh1f842ni-babl-0.1.18/lib/libbabl-0.1.so.0.117.1 matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-gcc-6.2.0 matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/gcc matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-c++ matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-gcc matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/g++ matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-g++ matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/cpp matches
Binary file /gnu/store/srfjnfkmjkc4xcld311xsdvhng08mmpi-gcc-6.2.0/bin/c++ matches
Binary file /gnu/store/j4q20kwzd5g1d3gv419692k66ghfzymz-gnuplot-5.0.4-1/bin/gnuplot matches
Binary file /gnu/store/mdc84lh0mfzw9n404cnzi9l1l8qr7a4r-gnuplot-5.0.4-1/bin/gnuplot matches
Binary file /gnu/store/8mj5yd1z936j64sdpx3hbqi3qkdif0c4-alsa-lib-1.0.27.1/lib/libasound.so.2.0.0 matches
Binary file /gnu/store/ninvaqcyhm6s11yp97m17h2i1q3aj24s-recode-3.7.0.201402/lib/librecode.a matches
Binary file /gnu/store/ninvaqcyhm6s11yp97m17h2i1q3aj24s-recode-3.7.0.201402/lib/librecode.so.0.0.0 matches
Binary file /gnu/store/kjybzn7az86n1qzxcm8zdz2gaypp4az6-fontconfig-2.11.94/lib/libfontconfig.so.1.9.0 matches
Binary file /gnu/store/cl2vwkvmk60s7vpamivpclzgyfxlb7wx-graphviz-2.38.0/bin/lefty matches
Binary file /gnu/store/qc8qg71k1b7gizqxa785c6ls71i8qk6d-units-2.13/bin/units matches
Binary file /gnu/store/sj8ygx2yz58hn1142yjjsb34sql4b9xr-unixodbc-2.3.4/lib/libodbcinst.so.2.0.0 matches
Binary file /gnu/store/sj8ygx2yz58hn1142yjjsb34sql4b9xr-unixodbc-2.3.4/lib/libodbc.so.2.0.0 matches
Binary file /gnu/store/571c58j8f06x8svykg4n5s0ip36kna5c-util-linux-2.27/lib/libuuid.so.1.3.0 matches
Binary file /gnu/store/1i3xmm18dw9kq6wi46f6sj9nxy9pckjl-alsa-lib-1.0.27.1/lib/libasound.so.2.0.0 matches
Binary file /gnu/store/x1kh4kkifc3f4gnlwvfgk53fm2zhfhmm-gnuplot-5.0.4-1/bin/gnuplot matches
Binary file /gnu/store/fnw55giyr1gnqyyw5yx4hf96mlrkp603-gnuplot-5.0.4-1/bin/gnuplot matches
Binary file /gnu/store/mj727bz0l8afmn97l8gsc5wh30jnql8s-babl-0.1.18/lib/libbabl-0.1.so.0.117.1 matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-gcc-6.2.0 matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/gcc matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-c++ matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-gcc matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/g++ matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/x86_64-unknown-linux-gnu-g++ matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/cpp matches
Binary file /gnu/store/0m908gszml9bb6vkikbzdqpslah2a1db-gcc-6.2.0/bin/c++ matches
Binary file /gnu/store/lqybibdq19q6ypm29p9p9s9ns0s66n0b-recode-3.7.0.201402/lib/librecode.a matches
Binary file /gnu/store/lqybibdq19q6ypm29p9p9s9ns0s66n0b-recode-3.7.0.201402/lib/librecode.so.0.0.0 matches
Binary file /gnu/store/pxdl28wbikj2jy0jjldmii245gcsh5fq-fontconfig-2.11.94/lib/libfontconfig.so.1.9.0 matches
Binary file /gnu/store/lhxgw5ynrccx9vvk9jx2k9pdsgq0yxm3-units-2.13/bin/units matches
Binary file /gnu/store/cji1mwp7n6zhdnpg73gighf3fvrh2gdi-ncurses-6.0/lib/libncursesw.so.6.0 matches
Binary file /gnu/store/cji1mwp7n6zhdnpg73gighf3fvrh2gdi-ncurses-6.0/lib/libncursesw.a matches
Binary file /gnu/store/1rcsqn8h2xwmgdra3zr33xv73d44wf1s-fontconfig-2.11.94/lib/libfontconfig.so.1.9.0 matches
Binary file /gnu/store/iiakyhsw889rvv3ghk6jl3mfdrs1wfan-fontconfig-2.11.94/lib/libfontconfig.so.1.9.0 matches
Binary file /gnu/store/zb3kh929v8yhkyg3sn8405zj4x49aza4-gnuplot-5.0.4-1/bin/gnuplot matches
Binary file /gnu/store/mkq7h60l1kx4hjclsrd0nbz8v4mnx4lv-gnuplot-5.0.4-1/bin/gnuplot matches
Binary file /gnu/store/4vhgx5xkbx0x7gnph9fq3c581rnj5ynq-libgphoto2-2.5.2/lib/libgphoto2/2.5.2/konica.so matches
Binary file /gnu/store/r6kwxzban57ghxsy8p8dqjvym6vnb2nz-graphviz-2.38.0/bin/lefty matches
/gnu/store/lwxifldgdyyzd83510nzf1qffpzxbdyl-guix-0.11.0-1.4420/share/guile/site/2.0/gnu/packages/package-management.scm:              (uri (string-append "mirror://gnu/stow/stow-"
Binary file /gnu/store/lwxifldgdyyzd83510nzf1qffpzxbdyl-guix-0.11.0-1.4420/share/guile/site/2.0/gnu/packages/package-management.go matches
Binary file /gnu/store/zslzcw2wg6xylwwr9hcx2sbal0bm6kjx-util-linux-2.27/lib/libuuid.so.1.3.0 matches

bug#24703: Store references in 8-byte chunks in compiled code

[Mark H Weaver]

The 8-byte chunks may appear out of order.  For example, in
'bash-static' we have this reference to:

  /gnu/store/7z3hpynjsbidxkq78xi5qi6lbcm8ndhp-glibc-intermediate-2.23

where the chunks are found in the following order:

1_/gnu/sto
2_________________ynjsbidx
3_________re/7z3hp
4_________________________________i6lbcm8n
5_________________________kq78xi5q
6_________________________________________dhp-glib
7_________________________________________________c-interm
8_________________________________________________________ediate-2
9_________________________________________________________________.23/lib/

Here's an excerpt of "objdump -d" output, annotated to show the 8-byte
constant as a string, and its position within the larger string.

======== /gnu/store/0010wvgs40kdq8chzsh403qm7la9jxq7-bash-static-4.3.42/bin/bash
  46b2c3:	48 b9 2f 67 6e 75 2f 	movabs $0x6f74732f756e672f,%rcx ; "/gnu/sto" (1)
  46b2ca:	73 74 6f 
  46b2cd:	48 01 d8             	add    %rbx,%rax
  46b2d0:	48 bb 79 6e 6a 73 62 	movabs $0x78646962736a6e79,%rbx ; "ynjsbidx" (3)
  46b2d7:	69 64 78 
  46b2da:	48 89 48 01          	mov    %rcx,0x1(%rax)
  46b2de:	48 b9 72 65 2f 37 7a 	movabs $0x7068337a372f6572,%rcx ; "re/7z3hp" (2)
  46b2e5:	33 68 70 
  46b2e8:	48 89 58 11          	mov    %rbx,0x11(%rax)
  46b2ec:	48 89 48 09          	mov    %rcx,0x9(%rax)
  46b2f0:	48 bb 69 36 6c 62 63 	movabs $0x6e386d63626c3669,%rbx ; "i6lbcm8n" (5)
  46b2f7:	6d 38 6e 
  46b2fa:	48 b9 6b 71 37 38 78 	movabs $0x713569783837716b,%rcx ; "kq78xi5q" (4)
  46b301:	69 35 71 
  46b304:	48 89 48 19          	mov    %rcx,0x19(%rax)
  46b308:	48 89 58 21          	mov    %rbx,0x21(%rax)
  46b30c:	48 b9 64 68 70 2d 67 	movabs $0x62696c672d706864,%rcx ; "dhp-glib" (6)
  46b313:	6c 69 62 
  46b316:	48 bb 63 2d 69 6e 74 	movabs $0x6d7265746e692d63,%rbx ; "c-interm" (7)
  46b31d:	65 72 6d 
  46b320:	48 89 48 29          	mov    %rcx,0x29(%rax)
  46b324:	ba 76 00 00 00       	mov    $0x76,%edx
  46b329:	48 89 58 31          	mov    %rbx,0x31(%rax)
  46b32d:	48 b9 65 64 69 61 74 	movabs $0x322d657461696465,%rcx ; "ediate-2" (8)
  46b334:	65 2d 32 
  46b337:	48 bb 2e 32 33 2f 6c 	movabs $0x2f62696c2f33322e,%rbx ; ".23/lib/" (9)
  46b33e:	69 62 2f 
  46b341:	48 89 58 41          	mov    %rbx,0x41(%rax)
  46b345:	31 f6                	xor    %esi,%esi
  46b347:	31 ff                	xor    %edi,%edi
  46b349:	c6 00 3a             	movb   $0x3a,(%rax)
  46b34c:	48 89 48 39          	mov    %rcx,0x39(%rax)
  46b350:	c7 40 49 67 63 6f 6e 	movl   $0x6e6f6367,0x49(%rax)
  46b357:	66 89 50 4d          	mov    %dx,0x4d(%rax)

bug#24703: Store references in 8-byte chunks in compiled code

[Mark H Weaver]

Here's a complex example of a reference to:

  /gnu/store/80m11l8km7bwi9iljfyr82hmscaq4xk2-unixodbc-2.3.4/etc

This also illustrates what can happen near the end of a reference.  In
this case, the last two characters are found at the end, within a 'mov'
instruction, and the previous 4 characters are found near the beginning,
after the first 8-byte chunk.

======== /gnu/store/80m11l8km7bwi9iljfyr82hmscaq4xk2-unixodbc-2.3.4/lib/libodbcinst.so.2.0.0
    8238:	48 b8 2f 67 6e 75 2f 	movabs $0x6f74732f756e672f,%rax   ; "/gnu/sto"
    823f:	73 74 6f 
    8242:	c7 05 6c fa 20 00 2e 	movl   $0x652f342e,0x20fa6c(%rip) ; ".4/e"
    8249:	34 2f 65 
    824c:	c6 05 6b fa 20 00 00 	movb   $0x0,0x20fa6b(%rip)
    8253:	48 89 05 26 fa 20 00 	mov    %rax,0x20fa26(%rip)
    825a:	48 b8 72 65 2f 38 30 	movabs $0x31316d30382f6572,%rax   ; "re/80m11"
    8261:	6d 31 31 
    8264:	c7 05 16 0a 21 00 01 	movl   $0x1,0x210a16(%rip)
    826b:	00 00 00 
    826e:	48 89 05 13 fa 20 00 	mov    %rax,0x20fa13(%rip)
    8275:	48 b8 6c 38 6b 6d 37 	movabs $0x697762376d6b386c,%rax   ; "l8km7bwi"
    827c:	62 77 69 
    827f:	48 8d 1d 42 94 00 00 	lea    0x9442(%rip),%rbx
    8286:	48 89 05 03 fa 20 00 	mov    %rax,0x20fa03(%rip)
    828d:	48 b8 39 69 6c 6a 66 	movabs $0x387279666a6c6939,%rax   ; "9iljfyr8"
    8294:	79 72 38 
    8297:	48 89 05 fa f9 20 00 	mov    %rax,0x20f9fa(%rip)
    829e:	48 b8 32 68 6d 73 63 	movabs $0x34716163736d6832,%rax   ; "2hmscaq4"
    82a5:	61 71 34 
    82a8:	48 89 05 f1 f9 20 00 	mov    %rax,0x20f9f1(%rip)
    82af:	48 b8 78 6b 32 2d 75 	movabs $0x78696e752d326b78,%rax   ; "xk2-unix"
    82b6:	6e 69 78 
    82b9:	48 89 05 e8 f9 20 00 	mov    %rax,0x20f9e8(%rip)
    82c0:	48 b8 6f 64 62 63 2d 	movabs $0x332e322d6362646f,%rax   ; "odbc-2.3"
    82c7:	32 2e 33 
    82ca:	48 89 05 df f9 20 00 	mov    %rax,0x20f9df(%rip)
    82d1:	b8 74 63 00 00       	mov    $0x6374,%eax               ; "tc"
    82d6:	66 89 05 df f9 20 00 	mov    %ax,0x20f9df(%rip)

Now imagine what will be required to graft cases like this.  Consider
what will happen for strings with an odd number of characters.  The last
character might be all by itself.

When grafting, how will we achieve confidence that we've found the
correct occurrence of the last character?  I think we will have to give
up our recently added feature of being able to change the version number
of grafts.

Oh, one more thing: I forgot to mention in my previous email, where
'bash-static' refers to 'glibc-intermediate', that on my system the
referenced 'glibc-intermediate' has been garbage-collected from my
system, so that's a real-world example of a broken link, and it happened
to be the first one I investigated :-(

       Mark

bug#24703: Store references in 8-byte chunks in compiled code

[Mark H Weaver]

Here's what happens with a variant of 'unixodbc' with one character
removed from its name, to make an odd number of characters.  The last
character 'c' is all by itself.  The chunks of the reference

  /gnu/store/il1bn7n0l4yj3idrii23fhvzg4nn939i-unxodbc-2.3.4/etc

are found in the following order:

1_/gnu/sto
2_________________________________________________________4/et
3_________re/il1bn
4_________________7n0l4yj3
5_________________________idrii23f
6_________________________________hvzg4nn9
7_________________________________________39i-unxo
8_________________________________________________dbc-2.3.
9_____________________________________________________________c

======== /gnu/store/y2dwj5vkbxka7y5rlakprbk4k8vrd5ij-unxodbc-2.3.4/lib/libodbcinst.so.2.0.0
    8238:	48 b8 2f 67 6e 75 2f 	movabs $0x6f74732f756e672f,%rax   ; "/gnu/sto"
    823f:	73 74 6f 
    8242:	c7 05 6c fa 20 00 34 	movl   $0x74652f34,0x20fa6c(%rip) ; "4/et"
    8249:	2f 65 74 
    824c:	c7 05 2e 0a 21 00 01 	movl   $0x1,0x210a2e(%rip)
    8253:	00 00 00 
    8256:	48 89 05 23 fa 20 00 	mov    %rax,0x20fa23(%rip)
    825d:	48 b8 72 65 2f 69 6c 	movabs $0x6e62316c692f6572,%rax   ; "re/il1bn"
    8264:	31 62 6e 
    8267:	48 8d 1d 5a 94 00 00 	lea    0x945a(%rip),%rbx
    826e:	48 89 05 13 fa 20 00 	mov    %rax,0x20fa13(%rip)
    8275:	48 b8 37 6e 30 6c 34 	movabs $0x336a79346c306e37,%rax   ; "7n0l4yj3"
    827c:	79 6a 33 
    827f:	48 89 05 0a fa 20 00 	mov    %rax,0x20fa0a(%rip)
    8286:	48 b8 69 64 72 69 69 	movabs $0x6633326969726469,%rax   ; "idrii23f"
    828d:	32 33 66 
    8290:	48 89 05 01 fa 20 00 	mov    %rax,0x20fa01(%rip)
    8297:	48 b8 68 76 7a 67 34 	movabs $0x396e6e34677a7668,%rax   ; "hvzg4nn9"
    829e:	6e 6e 39 
    82a1:	48 89 05 f8 f9 20 00 	mov    %rax,0x20f9f8(%rip)
    82a8:	48 b8 33 39 69 2d 75 	movabs $0x6f786e752d693933,%rax   ; "39i-unxo"
    82af:	6e 78 6f 
    82b2:	48 89 05 ef f9 20 00 	mov    %rax,0x20f9ef(%rip)
    82b9:	48 b8 64 62 63 2d 32 	movabs $0x2e332e322d636264,%rax   ; "dbc-2.3."
    82c0:	2e 33 2e 
    82c3:	48 89 05 e6 f9 20 00 	mov    %rax,0x20f9e6(%rip)
    82ca:	b8 63 00 00 00       	mov    $0x63,%eax                 ; "c"
    82cf:	66 89 05 e6 f9 20 00 	mov    %ax,0x20f9e6(%rip)

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> When grafting, how will we achieve confidence that we've found the
> correct occurrence of the last character?  I think we will have to give
> up our recently added feature of being able to change the version number
> of grafts.

Wait, don’t jump to the conclusions.  :-)

> Oh, one more thing: I forgot to mention in my previous email, where
> 'bash-static' refers to 'glibc-intermediate', that on my system the
> referenced 'glibc-intermediate' has been garbage-collected from my
> system, so that's a real-world example of a broken link, and it happened
> to be the first one I investigated :-(

‘bash-static’ is purposefully designed to stand alone: it must have zero
references:

--8<---------------cut here---------------start------------->8---
$ guix size bash-static
store item                                                       total    self
/gnu/store/nxysmfi9ybchnxfpxpmpgvz4f0nh7mxs-bash-static-4.3.42       1.4     1.4 100.0%
total: 1.4 MiB
--8<---------------cut here---------------end--------------->8---

So this one is fine.

Ludo’.

bug#24703: Re: bug#24703: Store references in 8-byte chunks in compiled code

[Török Edwin]

On 2016-10-16 22:04, Ludovic Courtès wrote:
> Mark H Weaver <mhw@netris.org> skribis:
> 
>> When grafting, how will we achieve confidence that we've found the
>> correct occurrence of the last character?  I think we will have to give
>> up our recently added feature of being able to change the version number
>> of grafts.
> 
> Wait, don’t jump to the conclusions.  :-)

I've just encountered the same problem with fontconfig (after installing GuixSD, running guix pull and guix system reconfigure, --no-grafts was required).
Would it be possible for the grafts to keep a symlink (somehow registered to be part of the grafted fontconfig so that guix gc doesn't remove it) instead of patching the binaries?
/gnu/store/<old-hash>-fontconfig-2.11.94 -> /gnu/store/<grafted-hash>-fontconfig-2.11.94

Best regards,
-- 
Edwin Török | Co-founder and Lead Developer

Skylable open-source object storage: reliable, fast, secure
http://www.skylable.com

bug#24703: Re: bug#24703: Store references in 8-byte chunks in compiled code

[Mark H Weaver]

Török Edwin <edwin+ml-guix@etorok.net> writes:

> On 2016-10-16 22:04, Ludovic Courtès wrote:
>> Mark H Weaver <mhw@netris.org> skribis:
>> 
>>> When grafting, how will we achieve confidence that we've found the
>>> correct occurrence of the last character?  I think we will have to give
>>> up our recently added feature of being able to change the version number
>>> of grafts.
>> 
>> Wait, don’t jump to the conclusions.  :-)
>
> I've just encountered the same problem with fontconfig (after
> installing GuixSD, running guix pull and guix system reconfigure,
> --no-grafts was required).

Here's how to recover, for now:

  guix build --no-grafts -e '(@@ (gnu packages fontutils) fontconfig/fixed)'

> Would it be possible for the grafts to keep a symlink (somehow
> registered to be part of the grafted fontconfig so that guix gc
> doesn't remove it) instead of patching the binaries?
> /gnu/store/<old-hash>-fontconfig-2.11.94 -> /gnu/store/<grafted-hash>-fontconfig-2.11.94

This would effectively mutate the store, and we must never do this.

      Mark

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Török Edwin <edwin+ml-guix@etorok.net> skribis:

> On 2016-10-16 22:04, Ludovic Courtès wrote:
>> Mark H Weaver <mhw@netris.org> skribis:
>> 
>>> When grafting, how will we achieve confidence that we've found the
>>> correct occurrence of the last character?  I think we will have to give
>>> up our recently added feature of being able to change the version number
>>> of grafts.
>> 
>> Wait, don’t jump to the conclusions.  :-)
>
> I've just encountered the same problem with fontconfig (after installing GuixSD, running guix pull and guix system reconfigure, --no-grafts was required).
> Would it be possible for the grafts to keep a symlink (somehow registered to be part of the grafted fontconfig so that guix gc doesn't remove it) instead of patching the binaries?
> /gnu/store/<old-hash>-fontconfig-2.11.94 -> /gnu/store/<grafted-hash>-fontconfig-2.11.94

We could use a self symlink, or we could use something like
<http://git.savannah.gnu.org/cgit/guix.git/commit/?id=9d50da70608de32d9db0c29859caec6f2cddb95f>.

Mark, WDYT?

What remains to be seen is how many packages are affected by this issue,
and whether a generic solution needs to be found.

Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Török Edwin <edwin+ml-guix@etorok.net> skribis:
>
>> On 2016-10-16 22:04, Ludovic Courtès wrote:
>>> Mark H Weaver <mhw@netris.org> skribis:
>>> 
>>>> When grafting, how will we achieve confidence that we've found the
>>>> correct occurrence of the last character?  I think we will have to give
>>>> up our recently added feature of being able to change the version number
>>>> of grafts.
>>> 
>>> Wait, don’t jump to the conclusions.  :-)
>>
>> I've just encountered the same problem with fontconfig (after installing GuixSD, running guix pull and guix system reconfigure, --no-grafts was required).
>> Would it be possible for the grafts to keep a symlink (somehow
>> registered to be part of the grafted fontconfig so that guix gc
>> doesn't remove it) instead of patching the binaries?
>> /gnu/store/<old-hash>-fontconfig-2.11.94 -> /gnu/store/<grafted-hash>-fontconfig-2.11.94
>
> We could use a self symlink, or we could use something like
> <http://git.savannah.gnu.org/cgit/guix.git/commit/?id=9d50da70608de32d9db0c29859caec6f2cddb95f>.
>
> Mark, WDYT?
>
> What remains to be seen is how many packages are affected by this issue,
> and whether a generic solution needs to be found.

Unfortunately, it is too widespread.  As I just pointed out in

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24712#13

Among the many packages that include these obfuscated store references,
one is 'glibc-final'.  My attempt to graft 'bash' in 'master' to fix
CVE-2016-0634 and CVE-2016-7543 has resulted in a system where I cannot
build *any* derivation, because 'guile-final' crashes during boot while
its 'glibc-final' tries to find its 'gconv' modules in the ungrafted
'glibc-final', which is not available in the build environment.

So, if our approach is to use -fno-builtin-strcpy, then we will have to
apply it system-wide, and rebuild all of 'core-updates' from scratch.

I've been investigating another approach: to enhance our scanner and
grafter to handle these 8-byte-chunked references.  I believe it is
feasible, but only if we abandon the ability to change the file names of
grafts outside of the hash.  The reason is that the hash portion of
store references are surrounded by enough other known characters on both
sides that the hash portion is almost always contained entirely within
8-byte chunks.

To be continued...

     Mark

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> We could use a self symlink, or we could use something like
>> <http://git.savannah.gnu.org/cgit/guix.git/commit/?id=9d50da70608de32d9db0c29859caec6f2cddb95f>.
>>
>> Mark, WDYT?
>>
>> What remains to be seen is how many packages are affected by this issue,
>> and whether a generic solution needs to be found.
>
> Unfortunately, it is too widespread.  As I just pointed out in
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24712#13
>
> Among the many packages that include these obfuscated store references,
> one is 'glibc-final'.  My attempt to graft 'bash' in 'master' to fix
> CVE-2016-0634 and CVE-2016-7543 has resulted in a system where I cannot
> build *any* derivation, because 'guile-final' crashes during boot while
> its 'glibc-final' tries to find its 'gconv' modules in the ungrafted
> 'glibc-final', which is not available in the build environment.

Out of curiosity, Guile crashes while loading iconv modules, right?

This is surprising because the guile-for-build is always the ungrafted
‘guile-final’ (see ‘gnu-build’ in (guix build-system gnu)).

> So, if our approach is to use -fno-builtin-strcpy, then we will have to
> apply it system-wide, and rebuild all of 'core-updates' from scratch.

Another approach would be to patch GCC, specifically ‘expand_movstr’ in
gcc/builtins.c, which is the part responsible for this optimization,
along these lines (untested):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1155 bytes --]

--- gcc-5.3.0/gcc/builtins.c.orig	2016-10-18 10:45:35.042826368 +0200
+++ gcc-5.3.0/gcc/builtins.c	2016-10-18 10:50:46.080616285 +0200
@@ -3470,6 +3470,19 @@ expand_builtin_mempcpy_args (tree dest,
 # define CODE_FOR_movstr CODE_FOR_nothing
 #endif
 
+/* Return true if STR is a string denoting a "/gnu/store" file name.  */
+
+static bool
+store_reference_p (tree str)
+{
+  const char *store;
+
+  store = getenv ("NIX_STORE") ?: "/gnu/store";
+
+  return (TREE_STRING_LENGTH (str) > strlen (store)
+	  && strncmp (TREE_STRING_POINTER (str), store, strlen (store)));
+}
+
 /* Expand into a movstr instruction, if one is available.  Return NULL_RTX if
    we failed, the caller should emit a normal call, otherwise try to
    get the result in TARGET, if convenient.  If ENDP is 0 return the
@@ -3484,7 +3497,9 @@ expand_movstr (tree dest, tree src, rtx
   rtx dest_mem;
   rtx src_mem;
 
-  if (!HAVE_movstr)
+  /* When SRC denotes a store item, do not perform any optimization.  See
+     <http://bugs.gnu.org/24703> for background.  */
+  if (!HAVE_movstr || store_reference_p (src))
     return NULL_RTX;
 
   dest_mem = get_memory_rtx (dest, NULL);

[-- Attachment #3: Type: text/plain, Size: 160 bytes --]


WDYT?

In the meantime, we need a workaround.  The only option I can think of
is to retain a reference to the ungrafted item by adding a symlink to
it, like:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: Type: text/x-patch, Size: 696 bytes --]

diff --git a/guix/grafts.scm b/guix/grafts.scm
index 80ae27e..4de19ae 100644
--- a/guix/grafts.scm
+++ b/guix/grafts.scm
@@ -127,7 +127,14 @@ recursively applied to dependencies of DRV."
                       files))
                    (match %outputs
                      (((names . files) ...)
-                      files))))))
+                      files)))
+
+         (for-each (match-lambda
+                     ((name . file)
+                      (symlink file
+                               (string-append (assoc-ref %outputs name)
+                                              "/ungrafted"))))
+                   old-outputs))))
 
   (define add-label
     (cut cons "x" <>))

[-- Attachment #5: Type: text/plain, Size: 712 bytes --]


> I've been investigating another approach: to enhance our scanner and
> grafter to handle these 8-byte-chunked references.  I believe it is
> feasible, but only if we abandon the ability to change the file names of
> grafts outside of the hash.  The reason is that the hash portion of
> store references are surrounded by enough other known characters on both
> sides that the hash portion is almost always contained entirely within
> 8-byte chunks.

I think that would add complexity, would make grafting slower, and
abandoning the ability to change file names would be a regression.

So I’m more in favor of a GCC patch like above, or another compilation
tweak.

WDYT?

Thanks,
Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> Unfortunately, it is too widespread.  As I just pointed out in
>>
>>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24712#13
>>
>> Among the many packages that include these obfuscated store references,
>> one is 'glibc-final'.  My attempt to graft 'bash' in 'master' to fix
>> CVE-2016-0634 and CVE-2016-7543 has resulted in a system where I cannot
>> build *any* derivation, because 'guile-final' crashes during boot while
>> its 'glibc-final' tries to find its 'gconv' modules in the ungrafted
>> 'glibc-final', which is not available in the build environment.
>
> Out of curiosity, Guile crashes while loading iconv modules, right?
>
> This is surprising because the guile-for-build is always the ungrafted
> ‘guile-final’ (see ‘gnu-build’ in (guix build-system gnu)).

Indeed.  The derivations that crashed were using a grafted 'guile'.
These were not 'gnu-build' derivations, but simpler derivations such as
'module-import-compiled' derivations and some others involved with
building a 'system'.

>> So, if our approach is to use -fno-builtin-strcpy, then we will have to
>> apply it system-wide, and rebuild all of 'core-updates' from scratch.
>
> Another approach would be to patch GCC, specifically ‘expand_movstr’ in
> gcc/builtins.c, which is the part responsible for this optimization,
> along these lines (untested):
>
> --- gcc-5.3.0/gcc/builtins.c.orig	2016-10-18 10:45:35.042826368 +0200
> +++ gcc-5.3.0/gcc/builtins.c	2016-10-18 10:50:46.080616285 +0200
> @@ -3470,6 +3470,19 @@ expand_builtin_mempcpy_args (tree dest,
>  # define CODE_FOR_movstr CODE_FOR_nothing
>  #endif
>  
> +/* Return true if STR is a string denoting a "/gnu/store" file name.  */
> +
> +static bool
> +store_reference_p (tree str)
> +{
> +  const char *store;
> +
> +  store = getenv ("NIX_STORE") ?: "/gnu/store";
> +
> +  return (TREE_STRING_LENGTH (str) > strlen (store)
> +	  && strncmp (TREE_STRING_POINTER (str), store, strlen (store)));
> +}

[...]

> WDYT?

I think it's not sufficient to apply this workaround only for string
literals that _begin_ with the store directory.  In some cases, the
store name may appear only in the middle of a string.

> In the meantime, we need a workaround.  The only option I can think of
> is to retain a reference to the ungrafted item by adding a symlink to
> it, like:

I consider it a potentially serious security problem that ungrafted
outputs are being used.  Papering over the problem by preventing this
buggy software from being deleted is, in my opinion, not acceptable.

I would suggest instead that we'll need to add grafts for all packages
that include these chunked references.  However, due to bug 24832, it
may be that we'll need to rebuild all of 'core-updates' from scratch
anyway.

>> I've been investigating another approach: to enhance our scanner and
>> grafter to handle these 8-byte-chunked references.  I believe it is
>> feasible, but only if we abandon the ability to change the file names of
>> grafts outside of the hash.  The reason is that the hash portion of
>> store references are surrounded by enough other known characters on both
>> sides that the hash portion is almost always contained entirely within
>> 8-byte chunks.
>
> I think that would add complexity, would make grafting slower, and
> abandoning the ability to change file names would be a regression.
>
> So I’m more in favor of a GCC patch like above, or another compilation
> tweak.
>
> WDYT?

The GCC approach is okay with me in the short term, but I'll likely want
to revisit this issue in the future.

     Thanks,
       Mark

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>>> So, if our approach is to use -fno-builtin-strcpy, then we will have to
>>> apply it system-wide, and rebuild all of 'core-updates' from scratch.
>>
>> Another approach would be to patch GCC, specifically ‘expand_movstr’ in
>> gcc/builtins.c, which is the part responsible for this optimization,
>> along these lines (untested):
>>
>> --- gcc-5.3.0/gcc/builtins.c.orig	2016-10-18 10:45:35.042826368 +0200
>> +++ gcc-5.3.0/gcc/builtins.c	2016-10-18 10:50:46.080616285 +0200
>> @@ -3470,6 +3470,19 @@ expand_builtin_mempcpy_args (tree dest,
>>  # define CODE_FOR_movstr CODE_FOR_nothing
>>  #endif
>>  
>> +/* Return true if STR is a string denoting a "/gnu/store" file name.  */
>> +
>> +static bool
>> +store_reference_p (tree str)
>> +{
>> +  const char *store;
>> +
>> +  store = getenv ("NIX_STORE") ?: "/gnu/store";
>> +
>> +  return (TREE_STRING_LENGTH (str) > strlen (store)
>> +	  && strncmp (TREE_STRING_POINTER (str), store, strlen (store)));
>> +}
>
> [...]
>
>> WDYT?
>
> I think it's not sufficient to apply this workaround only for string
> literals that _begin_ with the store directory.  In some cases, the
> store name may appear only in the middle of a string.

Do you have examples?  I think this is unlikely: the common case here is
that we’re capturing the installation prefix as in:

  #define PREFIX "/gnu/store/…"
  strcpy (file, PREFIX);

>> In the meantime, we need a workaround.  The only option I can think of
>> is to retain a reference to the ungrafted item by adding a symlink to
>> it, like:
>
> I consider it a potentially serious security problem that ungrafted
> outputs are being used.  Papering over the problem by preventing this
> buggy software from being deleted is, in my opinion, not acceptable.

In practice, only data from the ungrafted input would be used, AFAICS.

I’m not saying that this is a good solution, I’m just trying to think of
solutions that we can deploy now while waiting for something better,
which may involve a full rebuild.

> The GCC approach is okay with me in the short term, but I'll likely want
> to revisit this issue in the future.

From your message, it’s unclear to me what you are proposing:

  1. as the short term solution;

  2. as the long term solution.

To me short-term is symlink and long-term is GCC patch.

Thanks for your feedback,
Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Leo Famulari]

On Mon, Oct 17, 2016 at 11:36:57PM -0400, Mark H Weaver wrote:
> I've been investigating another approach: to enhance our scanner and
> grafter to handle these 8-byte-chunked references.  I believe it is
> feasible, but only if we abandon the ability to change the file names of
> grafts outside of the hash.  The reason is that the hash portion of
> store references are surrounded by enough other known characters on both
> sides that the hash portion is almost always contained entirely within
> 8-byte chunks.
> 
> To be continued...

Any progress? If not, I think we should start making a list of packages
affected by the bug in order to prepare a -fno-builtin-strcpy branch.

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Mon, Oct 17, 2016 at 11:36:57PM -0400, Mark H Weaver wrote:
>> I've been investigating another approach: to enhance our scanner and
>> grafter to handle these 8-byte-chunked references.  I believe it is
>> feasible, but only if we abandon the ability to change the file names of
>> grafts outside of the hash.  The reason is that the hash portion of
>> store references are surrounded by enough other known characters on both
>> sides that the hash portion is almost always contained entirely within
>> 8-byte chunks.
>> 
>> To be continued...
>
> Any progress?

I built GCC 5.3 with the patch I posted just to notice that it had no
effect (according to ‘ltrace’ it never goes through the
getenv("NIX_STORE") call that the patch adds.)

I plan to investigate further Real Soon.

> If not, I think we should start making a list of packages affected by
> the bug in order to prepare a -fno-builtin-strcpy branch.

Yeah it would be nice to identify these packages.

I’d prefer the GCC patch over -fno-builtin-strcpy because the latter is
hacky and potentially misses the issue when it occurs with other
builtins.

Either way, we’ll have to rebuild a lot of things.

Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

I’ve fiddled a bit with GCC and read some code.  No success yet, but
here’s a status update.

AIUI, ‘strcpy’ declarations are immediately (in the front-end)
recognized as “built-ins” as they are read (IOW, there’s no explicit
conversion from function call to built-in call.)

‘__builtin_strcpy’ calls are then converted to ‘__builtin_memcpy’ calls
(‘handle_builtin_strcpy’ in tree-ssa-strlen.c), which in turn leads to
code that uses the ‘movabs’ instruction that’s causing us problems on
x86:

--8<---------------cut here---------------start------------->8---
$ echo 'extern char *strcpy(char*,const char*);void foo(char*x){ strcpy(x, "foo"); }'> t.c
$ gcc -c -fdump-tree-optimized t.c
$ cat t.c.211t.optimized 

;; Function foo (foo, funcdef_no=0, decl_uid=1759, cgraph_uid=0, symbol_order=0)

foo (char * x)
{
  <bb 2>:
  __builtin_memcpy (x_2(D), "foo", 4);
  return;

}

--8<---------------cut here---------------end--------------->8---

(Strangely, when using ‘memcpy’ in the source, the ‘movabs’ optimization
does not take place at -O0, unlike for ‘strcpy’.)

So it seems there’s no place on the front-end side where we could do
what I tried to do in my initial patch, which is to test the contents of
the literal string passed to strcpy/memcpy and turn off the
‘movabs’-producing optimization.

Instead, the knobs we have are (1) global flag to enable/disable each
built-in function (like -fno-builtin-… does), and (2) an x86-specific
knob to determine whether to use ‘movabs’ or not (‘-mmemcpy-strategy’
supposedly controls that, but ‘-mmemcpy-strategy=libcall:-1:noalign’
doesn’t seem to have any effect for instance.)

These knobs are not great because that would lead us to disable the
optimization wholesale, which is not desirable.

Other than that, I’ve also looked at all the movabs-related things in
gcc/config/i386/*.md, but to no avail.

To be continued…

Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Leo Famulari]

On Sat, Nov 05, 2016 at 12:15:25AM +0100, Ludovic Courtès wrote:
> I’ve fiddled a bit with GCC and read some code.  No success yet, but
> here’s a status update.

Thanks for writing this out!

> Instead, the knobs we have are (1) global flag to enable/disable each
> built-in function (like -fno-builtin-… does), and (2) an x86-specific
> knob to determine whether to use ‘movabs’ or not (‘-mmemcpy-strategy’
> supposedly controls that, but ‘-mmemcpy-strategy=libcall:-1:noalign’
> doesn’t seem to have any effect for instance.)

Please correct me if I paraphrase the choices incorrectly:
(1) Completely disable the strcpy optimization for all architectures
(2) Ostensibly change how strcpy is optimized on x86, except the knob
seems to have no effect

> These knobs are not great because that would lead us to disable the
> optimization wholesale, which is not desirable.

What are the costs of (1)? Should we report (2) upstream?

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Sat, Nov 05, 2016 at 12:15:25AM +0100, Ludovic Courtès wrote:

[...]

>> Instead, the knobs we have are (1) global flag to enable/disable each
>> built-in function (like -fno-builtin-… does), and (2) an x86-specific
>> knob to determine whether to use ‘movabs’ or not (‘-mmemcpy-strategy’
>> supposedly controls that, but ‘-mmemcpy-strategy=libcall:-1:noalign’
>> doesn’t seem to have any effect for instance.)
>
> Please correct me if I paraphrase the choices incorrectly:
> (1) Completely disable the strcpy optimization for all architectures
> (2) Ostensibly change how strcpy is optimized on x86, except the knob
> seems to have no effect

Correct.

>> These knobs are not great because that would lead us to disable the
>> optimization wholesale, which is not desirable.
>
> What are the costs of (1)? Should we report (2) upstream?

-fno-builtin-{strcpy,memcpy} disables not just use of ‘movabs’ but also
all the optimizations that GCC can make when the built-ins are being
used (see the doc for ‘-mmemcpy-strategy’).  It wouldn’t feel right to
disable that completely.

Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Hello!

I read much more code than I wanted just to end up in gcc/builtins.c.
In 8033772363b287ca529461e575ceb4a69d7af942 I added a patch for GCC 5.x
and 6.x that disables the ‘movabs’ optimization for strcpy/memcpy when
the source is a string constant that contains “/gnu/store” (I followed
Mark’s advice to disable the optimization for any string that contains
“/gnu/store”, rather than just for strings that start with
“/gnu/store”.)

This can be tested by compiling a file like this one (comment or
uncomment the lines that you want):

--8<---------------cut here---------------start------------->8---
#include <string.h>
void foo (char *x, char *y)
{
  /* memcpy(x, "this is a long string, a very long string", 42); */
  /* strcpy(x, "STRCPY THIS IS A LONG STRING, A VERY LONG STRING"); */
  strcpy(x, "STRCPY /gnu/store/THIS IS A LONG  STRING, A VERY LONG STRING");
  /* __builtin_strcpy(x, "THIS IS A LONG STRING, A VERY LONG STRING"); */
  /* strcpy(y, "/gnu/store/THIS IS A LONG STRING, A VERY LONG STRING"); */
  /* memcpy(y, "MEMCPY THIS IS A LONG STRING, A VERY LONG STRING", 30); */
  memcpy(y, "MEMCPY /gnu/store/THIS IS A LONG STRING, A VERY LONG STRING", 30);
  /* __builtin_memcpy(y, "/gnu/store/THIS IS A LONG STRING, A VERY LONG STRING", 30); */
}
--8<---------------cut here---------------end--------------->8---

… and then running “objdump -S foo.o | grep movabs”, for instance.

Now we need a plan to actually fix the bug.

The long-term goal is to rebuild everything with a compiler that has
this patch, in the next ‘core-updates’.  We might as well switch to GCC
5 as the default compiler.

In the meantime, the only approach I can think of is to (1) ungraft more
frequently than we’ve done so far, and (2) when we ungraft a package,
add gcc@5 as an input such that it gets rebuilt without the problem.

Thoughts?

Thanks,
Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Leo Famulari]

On Wed, Nov 09, 2016 at 09:40:05PM +0100, Ludovic Courtès wrote:
> I read much more code than I wanted just to end up in gcc/builtins.c.
> In 8033772363b287ca529461e575ceb4a69d7af942 I added a patch for GCC 5.x
> and 6.x that disables the ‘movabs’ optimization for strcpy/memcpy when
> the source is a string constant that contains “/gnu/store” (I followed
> Mark’s advice to disable the optimization for any string that contains
> “/gnu/store”, rather than just for strings that start with
> “/gnu/store”.)
> 
> This can be tested by compiling a file like this one (comment or
> uncomment the lines that you want):
> 
> --8<---------------cut here---------------start------------->8---
> #include <string.h>
> void foo (char *x, char *y)
> {
>   /* memcpy(x, "this is a long string, a very long string", 42); */
>   /* strcpy(x, "STRCPY THIS IS A LONG STRING, A VERY LONG STRING"); */
>   strcpy(x, "STRCPY /gnu/store/THIS IS A LONG  STRING, A VERY LONG STRING");
>   /* __builtin_strcpy(x, "THIS IS A LONG STRING, A VERY LONG STRING"); */
>   /* strcpy(y, "/gnu/store/THIS IS A LONG STRING, A VERY LONG STRING"); */
>   /* memcpy(y, "MEMCPY THIS IS A LONG STRING, A VERY LONG STRING", 30); */
>   memcpy(y, "MEMCPY /gnu/store/THIS IS A LONG STRING, A VERY LONG STRING", 30);
>   /* __builtin_memcpy(y, "/gnu/store/THIS IS A LONG STRING, A VERY LONG STRING", 30); */
> }
> --8<---------------cut here---------------end--------------->8---
> 
> … and then running “objdump -S foo.o | grep movabs”, for instance.

Awesome. Thank you!

> Now we need a plan to actually fix the bug.
> 
> The long-term goal is to rebuild everything with a compiler that has
> this patch, in the next ‘core-updates’.  We might as well switch to GCC
> 5 as the default compiler.
> 
> In the meantime, the only approach I can think of is to (1) ungraft more
> frequently than we’ve done so far, and (2) when we ungraft a package,
> add gcc@5 as an input such that it gets rebuilt without the problem.
> 
> Thoughts?

Sounds good to me. I wonder, with our current build farm, how often can
we do the full rebuilds required by ungrafting? I don't yet have a sense
of how long it takes.

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Wed, Nov 09, 2016 at 09:40:05PM +0100, Ludovic Courtès wrote:

[...]

>> The long-term goal is to rebuild everything with a compiler that has
>> this patch, in the next ‘core-updates’.  We might as well switch to GCC
>> 5 as the default compiler.
>> 
>> In the meantime, the only approach I can think of is to (1) ungraft more
>> frequently than we’ve done so far, and (2) when we ungraft a package,
>> add gcc@5 as an input such that it gets rebuilt without the problem.
>> 
>> Thoughts?
>
> Sounds good to me. I wonder, with our current build farm, how often can
> we do the full rebuilds required by ungrafting? I don't yet have a sense
> of how long it takes.

I don’t know exactly.  I have a feeling that the build farm has been
underused over the last couple of weeks for instance, but no precise
stats.  I’d like to be able to gather usage stats so we can see when the
right time for rebuilds comes.

For now I think we’ll merge core-updates Real Soon, and then start a
staging branch or the python-updates one.

Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

The patch to fix this bug was committed in ‘core-updates’ in
8033772363b287ca529461e575ceb4a69d7af942, which has now been merged.

Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> In the meantime, the only approach I can think of is to (1) ungraft more
> frequently than we’ve done so far, and (2) when we ungraft a package,
> add gcc@5 as an input such that it gets rebuilt without the problem.

As an example, I did that in commit
2c5ab05bffe5f89092ef60c3743b3941dcf92af0 for WordNet.

Ludo’.

bug#24703: Store references in 8-byte chunks in compiled code

[Török Edwin]

On 2016-10-17 12:42, Mark H Weaver wrote:
> 
> Here's how to recover, for now:
> 
>   guix build --no-grafts -e '(@@ (gnu packages fontutils) fontconfig/fixed)'

Thanks!

On 2016-10-17 15:09, Ludovic Courtès wrote:
> Török Edwin <edwin+ml-guix@etorok.net> skribis:
> 
>> On 2016-10-16 22:04, Ludovic Courtès wrote:
>>> Mark H Weaver <mhw@netris.org> skribis:
>>>
>>>> When grafting, how will we achieve confidence that we've found the
>>>> correct occurrence of the last character?  I think we will have to give
>>>> up our recently added feature of being able to change the version number
>>>> of grafts.
>>>
>>> Wait, don’t jump to the conclusions.  :-)
>>
>> I've just encountered the same problem with fontconfig (after installing GuixSD, running guix pull and guix system reconfigure, --no-grafts was required).
>> Would it be possible for the grafts to keep a symlink (somehow registered to be part of the grafted fontconfig so that guix gc doesn't remove it) instead of patching the binaries?
>> /gnu/store/<old-hash>-fontconfig-2.11.94 -> /gnu/store/<grafted-hash>-fontconfig-2.11.94
> 
> We could use a self symlink

I'm new here [*] and I'd like to understand what solutions would be possible, could you please explain how the self symlink would work?

I was thinking about bind mounts (the list of needed bind mounts would be maintained as a derivation, and initialized from initrd maybe?):
/gnu/store/<new-hash>-fontconfig-2.11.94 on /gnu/store/<old-hash>-fontconfig-2.11.94

or equivalently another layer of symlinks (where /gnu/store is the mutable symlink graft location, and /gnu/immutablestore is the real destination):
/gnu/store/<old-hash>-fontconfig-2.11.94 -> /gnu/immutablestore/<new-hash>-fontconfig-2.11.94
/gnu/store/<new-hash>-fontconfig-2.11.94 -> /gnu/immutablestore/<new-hash>-fontconfig-2.11.94

However bit worried of what happens to running applications: they might end up loading libs/data from both old-hash and new-hash, but
it wouldn't be worse than what happens on a traditional distribution when you upgrade say libc, you need to restart things using it eventually.

> or we could use something like
> <http://git.savannah.gnu.org/cgit/guix.git/commit/?id=9d50da70608de32d9db0c29859caec6f2cddb95f>.

IMHO binary rewriting should be a last resort, it is too low-level and depends on the compiler/version to get it right.

[*] I've briefly tried NixOS and GuixSD in the past, but I'm using Debian and Gentoo as my main OS.
IIUC what the current grafting process does is that it reads all files of packages that depend on X from the store,
and writes them under a new hash with all references to hash of X replaced by hash of X-fixed, a bit like patchELF
except strings remain same size.

Best regards,

bug#24703: Store references in 8-byte chunks in compiled code

[Ludovic Courtès]

Török Edwin <edwin+ml-guix@etorok.net> skribis:

> On 2016-10-17 12:42, Mark H Weaver wrote:
>> 
>> Here's how to recover, for now:
>> 
>>   guix build --no-grafts -e '(@@ (gnu packages fontutils) fontconfig/fixed)'
>
> Thanks!
>
> On 2016-10-17 15:09, Ludovic Courtès wrote:
>> Török Edwin <edwin+ml-guix@etorok.net> skribis:
>> 
>>> On 2016-10-16 22:04, Ludovic Courtès wrote:
>>>> Mark H Weaver <mhw@netris.org> skribis:
>>>>
>>>>> When grafting, how will we achieve confidence that we've found the
>>>>> correct occurrence of the last character?  I think we will have to give
>>>>> up our recently added feature of being able to change the version number
>>>>> of grafts.
>>>>
>>>> Wait, don’t jump to the conclusions.  :-)
>>>
>>> I've just encountered the same problem with fontconfig (after installing GuixSD, running guix pull and guix system reconfigure, --no-grafts was required).
>>> Would it be possible for the grafts to keep a symlink (somehow registered to be part of the grafted fontconfig so that guix gc doesn't remove it) instead of patching the binaries?
>>> /gnu/store/<old-hash>-fontconfig-2.11.94 -> /gnu/store/<grafted-hash>-fontconfig-2.11.94
>> 
>> We could use a self symlink
>
> I'm new here [*] and I'd like to understand what solutions would be possible, could you please explain how the self symlink would work?

The grafted variant of each store item would have a symlink pointing to
its ungrafted variant.

The problem with this approach is that it would induce some storage
overhead (although deduplication probably mitigates that), and it would
make it harder to check whether a given item is using only grafted
things because ‘guix gc -R something’ would always show both the grafted
and the ungrafted variant of each dependency.

> I was thinking about bind mounts (the list of needed bind mounts would be maintained as a derivation, and initialized from initrd maybe?):
> /gnu/store/<new-hash>-fontconfig-2.11.94 on /gnu/store/<old-hash>-fontconfig-2.11.94

That would only work on GuixSD, not on foreign distros, and it’s a
“stateful” approach, which defeats the whole reproducible approach.

> IMHO binary rewriting should be a last resort, it is too low-level and depends on the compiler/version to get it right.

I agree.  However, for fast security updates, it seems to be the best
option:

  https://savannah.gnu.org/forum/forum.php?forum_id=8470

Thanks for your feedback,
Ludo’.

bug#24703: fontconfig keeps obfuscated reference to itself, not grafted

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> It turns out there's an obfuscated self-reference to fontconfig's store
> directory.  Here's an excerpt of the output of "hexdump -C
> libfontconfig.so.1.9.0":
>
> 0000cca0  00 48 b9 2f 67 6e 75 2f  73 74 6f c6 40 48 00 45  |.H./gnu/sto.@H.E|
> 0000ccb0  31 e4 48 89 08 48 b9 72  65 2f 62 34 38 34 6e 48  |1.H..H.re/b484nH|
> 0000ccc0  89 48 08 48 b9 76 6e 39  6e 6e 72 33 64 48 89 48  |.H.H.vn9nnr3dH.H|
> 0000ccd0  10 48 b9 64 63 6c 70 7a  32 66 6d 48 89 48 18 48  |.H.dclpz2fmH.H.H|
> 0000cce0  b9 61 39 79 78 6d 69 6d  67 48 89 48 20 48 b9 32  |.a9yxmimgH.H H.2|
> 0000ccf0  6a 6a 2d 66 6f 6e 74 48  89 48 28 48 b9 63 6f 6e  |jj-fontH.H(H.con|
> 0000cd00  66 69 67 2d 32 48 89 48  30 48 b9 2e 31 31 2e 39  |fig-2H.H0H..11.9|
> 0000cd10  34 2f 65 48 89 48 38 48  b9 74 63 2f 66 6f 6e 74  |4/eH.H8H.tc/font|
> 0000cd20  73 48 89 48 40 48 8b 04  24 48 8b 18 48 89 c5 48  |sH.H@H..$H..H..H|

See commit 9d50da70608de32d9db0c29859caec6f2cddb95f in WordNet:
-fno-builtin-strcpy is the solution I found.

Weird, no?  :-)

Ludo’.

bug#24703: fontconfig keeps obfuscated reference to itself, not grafted

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

ludo@gnu.org (Ludovic Courtès) skribis:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> It turns out there's an obfuscated self-reference to fontconfig's store
>> directory.  Here's an excerpt of the output of "hexdump -C
>> libfontconfig.so.1.9.0":
>>
>> 0000cca0  00 48 b9 2f 67 6e 75 2f  73 74 6f c6 40 48 00 45  |.H./gnu/sto.@H.E|
>> 0000ccb0  31 e4 48 89 08 48 b9 72  65 2f 62 34 38 34 6e 48  |1.H..H.re/b484nH|
>> 0000ccc0  89 48 08 48 b9 76 6e 39  6e 6e 72 33 64 48 89 48  |.H.H.vn9nnr3dH.H|
>> 0000ccd0  10 48 b9 64 63 6c 70 7a  32 66 6d 48 89 48 18 48  |.H.dclpz2fmH.H.H|
>> 0000cce0  b9 61 39 79 78 6d 69 6d  67 48 89 48 20 48 b9 32  |.a9yxmimgH.H H.2|
>> 0000ccf0  6a 6a 2d 66 6f 6e 74 48  89 48 28 48 b9 63 6f 6e  |jj-fontH.H(H.con|
>> 0000cd00  66 69 67 2d 32 48 89 48  30 48 b9 2e 31 31 2e 39  |fig-2H.H0H..11.9|
>> 0000cd10  34 2f 65 48 89 48 38 48  b9 74 63 2f 66 6f 6e 74  |4/eH.H8H.tc/font|
>> 0000cd20  73 48 89 48 40 48 8b 04  24 48 8b 18 48 89 c5 48  |sH.H@H..$H..H..H|
>
> See commit 9d50da70608de32d9db0c29859caec6f2cddb95f in WordNet:
> -fno-builtin-strcpy is the solution I found.

Simple test case:


[-- Attachment #2: the code --]
[-- Type: text/plain, Size: 113 bytes --]

#include <string.h>
void foo (char *bar)
{
  strcpy (bar, "this is a very long string that's gonna be split");
}

[-- Attachment #3: Type: text/plain, Size: 74 bytes --]


Run “gcc -c t.c” with and without -fno-builtin-strcpy.

Ludo’.