* bug#32942: nss-certs not deterministic @ 2018-10-05 8:17 Julien Lepiller 2018-12-05 14:01 ` Ludovic Courtès 0 siblings, 1 reply; 4+ messages in thread From: Julien Lepiller @ 2018-10-05 8:17 UTC (permalink / raw) To: 32942 While updating a profile, I found that nss-certs was not deterministic. From ludo: $ wget -O - -q https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | grep Hash NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla $ wget -O - -q https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | grep Hash NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s ^ permalink raw reply [flat|nested] 4+ messages in thread
* bug#32942: nss-certs not deterministic 2018-10-05 8:17 bug#32942: nss-certs not deterministic Julien Lepiller @ 2018-12-05 14:01 ` Ludovic Courtès 2018-12-09 23:29 ` Ludovic Courtès 2018-12-19 17:42 ` swedebugia 0 siblings, 2 replies; 4+ messages in thread From: Ludovic Courtès @ 2018-12-05 14:01 UTC (permalink / raw) To: Julien Lepiller; +Cc: 32942 [-- Attachment #1: Type: text/plain, Size: 2702 bytes --] Hello, Julien Lepiller <julien@lepiller.eu> skribis: > While updating a profile, I found that nss-certs was not > deterministic. From ludo: > > $ wget -O - -q > https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo| grep Hash > NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla > $ wget -O - -q > https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | > grep Hash > NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s As shown above, berlin and hydra disagree on nss-certs. The difference is an encoding bug: --8<---------------cut here---------------start------------->8--- $ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin $ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra $ diff -ru /tmp/nss-certs.{hydra,berlin} Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem --8<---------------cut here---------------end--------------->8--- The problem was already reported as <https://bugs.gnu.org/26948> and since commit 412701b0e5e073e6767eed162c14698db99df69c (July 2017) ‘guix publish’ on GuixSD runs in a UTF-8 locale to avoid that problem. The faulty narinfo/nar on berlin were generated on Oct. 17, 2018, so clearly the above commit was in effect. Indeed, after removing them and regenerating them, I’m still getting 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s (aka. the wrong hash). On closer inspection the problem is elsewhere: the /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on berlin has question marks in file names, so ‘guix publish’ is not to blame; instead the problem likely comes from ‘guix offload’. Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run with an empty environment, and thus in the C locale. Specifically, ‘restore-file-set’ on the build farm front-end must be the one substituting question marks to the non-ASCII characters. If this analysis is correct, the patch below should fix it. I’ll try it later. Thanks, Ludo’. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 972 bytes --] diff --git a/gnu/services/base.scm b/gnu/services/base.scm index cee9898d79..9fe64e8087 100644 --- a/gnu/services/base.scm +++ b/gnu/services/base.scm @@ -1603,7 +1603,15 @@ failed to register public key '~a': ~a~%" key status)))))))) '()) #$@(if tmpdir (list (string-append "TMPDIR=" tmpdir)) - '())) + '()) + + ;; Make sure we run in a UTF-8 locale so that 'guix + ;; offload' correctly restores nars that contain UTF-8 + ;; file names such as 'nss-certs'. See + ;; <https://bugs.gnu.org/32942>. + (string-append "GUIX_LOCPATH=" + #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") #:log-file #$log-file)) (stop #~(make-kill-destructor)))))) ^ permalink raw reply related [flat|nested] 4+ messages in thread
* bug#32942: nss-certs not deterministic 2018-12-05 14:01 ` Ludovic Courtès @ 2018-12-09 23:29 ` Ludovic Courtès 2018-12-19 17:42 ` swedebugia 1 sibling, 0 replies; 4+ messages in thread From: Ludovic Courtès @ 2018-12-09 23:29 UTC (permalink / raw) To: Julien Lepiller; +Cc: 32942-done Hello, ludo@gnu.org (Ludovic Courtès) skribis: > The difference is an encoding bug: > > $ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin > $ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra > $ diff -ru /tmp/nss-certs.{hydra,berlin} > Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem > Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem > Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem > Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem [...] > On closer inspection the problem is elsewhere: the > /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on > berlin has question marks in file names, so ‘guix publish’ is not to > blame; instead the problem likely comes from ‘guix offload’. > > Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run > with an empty environment, and thus in the C locale. Specifically, > ‘restore-file-set’ on the build farm front-end must be the one > substituting question marks to the non-ASCII characters. > > If this analysis is correct, the patch below should fix it. I’ll try it > later. Pushed as 7e4bc215098f334bc2a11737f2665dd4992fc2da. Thanks, Ludo'. ^ permalink raw reply [flat|nested] 4+ messages in thread
* bug#32942: nss-certs not deterministic 2018-12-05 14:01 ` Ludovic Courtès 2018-12-09 23:29 ` Ludovic Courtès @ 2018-12-19 17:42 ` swedebugia 1 sibling, 0 replies; 4+ messages in thread From: swedebugia @ 2018-12-19 17:42 UTC (permalink / raw) To: Ludovic Courtès, Julien Lepiller; +Cc: 32942 On 2018-12-05 15:01, Ludovic Courtès wrote: > Hello, > > Julien Lepiller <julien@lepiller.eu> skribis: > >> While updating a profile, I found that nss-certs was not >> deterministic. From ludo: >> >> $ wget -O - -q >> https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo| grep Hash >> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla >> $ wget -O - -q >> https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | >> grep Hash >> NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s > > As shown above, berlin and hydra disagree on nss-certs. > > The difference is an encoding bug: > > --8<---------------cut here---------------start------------->8--- > $ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin > $ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra > $ diff -ru /tmp/nss-certs.{hydra,berlin} > Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem > Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem > Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem > Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem > --8<---------------cut here---------------end--------------->8--- > > The problem was already reported as <https://bugs.gnu.org/26948> and > since commit 412701b0e5e073e6767eed162c14698db99df69c (July 2017) ‘guix > publish’ on GuixSD runs in a UTF-8 locale to avoid that problem. > > The faulty narinfo/nar on berlin were generated on Oct. 17, 2018, so > clearly the above commit was in effect. Indeed, after removing them and > regenerating them, I’m still getting > 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s (aka. the wrong > hash). > > On closer inspection the problem is elsewhere: the > /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on > berlin has question marks in file names, so ‘guix publish’ is not to > blame; instead the problem likely comes from ‘guix offload’. > > Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run > with an empty environment, and thus in the C locale. Specifically, > ‘restore-file-set’ on the build farm front-end must be the one > substituting question marks to the non-ASCII characters. > > If this analysis is correct, the patch below should fix it. I’ll try it > later. > > Thanks, > Ludo’. > > > > diff --git a/gnu/services/base.scm b/gnu/services/base.scm > index cee9898d79..9fe64e8087 100644 > --- a/gnu/services/base.scm > +++ b/gnu/services/base.scm > @@ -1603,7 +1603,15 @@ failed to register public key '~a': ~a~%" key status)))))))) > '()) > #$@(if tmpdir > (list (string-append "TMPDIR=" tmpdir)) > - '())) > + '()) > + > + ;; Make sure we run in a UTF-8 locale so that 'guix > + ;; offload' correctly restores nars that contain UTF-8 > + ;; file names such as 'nss-certs'. See > + ;; <https://bugs.gnu.org/32942>. > + (string-append "GUIX_LOCPATH=" > + #$glibc-utf8-locales "/lib/locale") > + "LC_ALL=en_US.utf8") > > #:log-file #$log-file)) > (stop #~(make-kill-destructor)))))) > Congratulations with the succeded hunt and thanks a lot for showing all the steps you took so I can improve my hunting skills and eventually begin helping by hunting on my own :D -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-12-19 17:36 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-05 8:17 bug#32942: nss-certs not deterministic Julien Lepiller 2018-12-05 14:01 ` Ludovic Courtès 2018-12-09 23:29 ` Ludovic Courtès 2018-12-19 17:42 ` swedebugia
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).