From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: bug#40525: inferior process on core-updates crashes: mmap(PROT_NONE) failed Date: Thu, 16 Apr 2020 18:29:59 +0100 Message-ID: <87pnc75ofc.fsf@cbaines.net> References: <87369c8mus.fsf@cbaines.net> <87a73jy8y9.fsf@gnu.org> <87y2r37dy8.fsf@cbaines.net> <87h7xqrug3.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:37858) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jP8Lw-0003ck-R7 for bug-guix@gnu.org; Thu, 16 Apr 2020 13:31:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jP8Lu-0001Aj-TG for bug-guix@gnu.org; Thu, 16 Apr 2020 13:31:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:55999) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jP8Lt-0001AU-Vq for bug-guix@gnu.org; Thu, 16 Apr 2020 13:31:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-reply-to: <87h7xqrug3.fsf@gnu.org> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane-mx.org@gnu.org Sender: "bug-Guix" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 40525@debbugs.gnu.org --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi Christopher, > > Christopher Baines skribis: > >> I've attached a script that when run should reproduce the issue. I >> extracted the code relating to lint warnings from the Guix Data >> Service. The script attached runs this code twice against the inferior, >> once will often be enough to cause it to crash, but twice should >> reproduce it more reliably. > > Thanks a lot. > > Here=E2=80=99s a backtrace from the core dumped by the inferior: ... > It could be an unbounded growth of libgc=E2=80=99s finalizer table or our= weak > tables as we experienced in . > > We should be able to reproduce it with something like: > > guix time-machine --commit=3Dd523eb5c9c2659cbbaf4eeef3691234ae527ee6a -= - \ > lint -c inputs-should-be-native,license,mirror-url,source-file-name,s= ource-unstable-tarball,derivation,patch-file-names,formatting,synopsis > > In top one can see that heap usage keeps growing, which may well be a > bug in Guix proper rather than in Guile=E2=80=A6 but it doesn=E2=80=99t c= rash. > > I would propose three actions here: > > 1. Run linters un =E2=80=98gcprof=E2=80=99 to see what=E2=80=99s eating= memory and hopefully > find and address the leak. As a start, maybe just start reducing > the list of checkers to see if there=E2=80=99s one of them that=E2= =80=99s causing > it. > > The =E2=80=98derivation=E2=80=99 checker is definitely responsible f= or a lot of the > heap consumption because of the various caches in (guix packages) & > co. Perhaps add calls to =E2=80=98invalidate-derivation-caches!=E2= =80=99 as in > (gnu ci). > > 2. Work around the problem in Guix Data Service by running, say, one > inferior per checker instead of one inferior for all checkers for > all packages. > > 3. If #1 didn=E2=80=99t help, let=E2=80=99s see if we can isolate a Gui= le weak-table > bug or something like that. > > Thoughts? Thanks, that's useful to know. I think I've now managed to find a way of reproducing this without the inferior getting in the way. I was testing if triggering garbage collection in Guile would help avoid the problem, but actually it seems to cause it. I guess given the mentions of GC in the above stacktrace, and the major version change of libgc, some GC related bug seems quite likely here. I've been testing with a checkout of Guix built with Guix from the core-updates branch. I think that provides the same broken Guile that the guix repl is using. When trying to just use a checkout of the core-updates branch, and guile built from that branch I get the following odd error: =E2=86=92 ./pre-inst-env /gnu/store/18hp7flyb3yid3yp49i6qcdq0sbi5l1n-guile-= 3.0.2/bin/guile ./reproduce-core-updates-mmap-PROT_NONE-failed.scm guile: warning: failed to install locale warning: failed to load '(gnu packages abiword)': Function not implemented error: git-fetch: unbound variable hint: Did you forget `(use-modules (guix git-download))'? error: git-version: unbound variable No idea what's happening there, but when I ./configure and make with packages from core-updates, I seem to end up with a setup that works: This is the guile I'm using: /gnu/store/18hp7flyb3yid3yp49i6qcdq0sbi5l1n-gu= ile-3.0.2/bin/guile If you just run the script, you should see: =E2=86=92 ./pre-inst-env guile ./reproduce-core-updates-mmap-PROT_NONE-fail= ed.scm ;;; ("%package-table-setup" #) mmap(PROT_NONE) failed Aborted For more information, you can pipe the script to the REPL. What you should see is that it's slow to compute the lint warnings the first time, but the subsequent times are quick, and it crashes in one of the (gc) calls. I'm going to try and continue looking in to this, at least it'll be easier to delve in to guile now that I can directly control what guile is used. Thanks, Chris --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=reproduce-core-updates-mmap-PROT_NONE-failed.scm Content-Description: Guile script (use-modules (srfi srfi-1) (ice-9 match) (gnu packages) (guix store) (guix grafts) (guix packages) (guix lint) (guix utils)) (define %package-table (make-hash-table)) (begin (fold-packages (lambda (package result) (let ((id (object-address package))) (hashv-set! %package-table id package) (cons (list (package-name package) (package-version package) id) result))) '()) (peek "%package-table-setup" %package-table)) (define lint-warnings-for-checker (lambda (checker-name store) (let* ((checker (find (lambda (checker) (eq? (lint-checker-name checker) checker-name)) %local-checkers)) (check (lint-checker-check checker))) (define (process-lint-warning lint-warning) (list (match (lint-warning-location lint-warning) (($ file line column) (list (if (string-prefix? "/gnu/store/" file) (string-join (drop (string-split file #\/) 8) "/") file) line column))) (let* ((source-locale "en_US.utf8") (source-message (begin (setlocale LC_MESSAGES source-locale) (lint-warning-message lint-warning))) (messages-by-locale (filter-map (lambda (locale) (catch 'system-error (lambda () (setlocale LC_MESSAGES locale)) (lambda (key . args) (error (simple-format #f "error changing locale to ~A: ~A ~A" locale key args)))) (let ((message (lint-warning-message lint-warning))) (setlocale LC_MESSAGES source-locale) (if (string=? message source-message) #f (cons locale message)))) '("cs_CZ.utf8" "da_DK.utf8" "de_DE.utf8" "eo_EO.utf8" "es_ES.utf8" "fr_FR.utf8" "hu_HU.utf8" "pl_PL.utf8" "pt_BR.utf8" ;;"sr_SR.utf8" "sv_SE.utf8" "vi_VN.utf8" "zh_CN.utf8")))) (cons (cons source-locale source-message) messages-by-locale)))) (filter (match-lambda ((package-id . warnings) (not (null? warnings)))) (hash-map->list (lambda (package-id package) (cons package-id (catch #t (lambda () (map process-lint-warning (check package #:store store))) (lambda (key . args) '())))) %package-table))))) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (gc) (with-store store (lint-warnings-for-checker 'derivation store)) (simple-format #t "finished successfully\n\n") --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6YlhdfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9Xdzlg/+OZw55c5R2aMtO9hryDVm7dvKDS8NMbLz+lg5CVhnRf+TrnMY/oxRcVAw /ngf6YuOIcnL737CAUncPASVCql5AYHoS9yS+SOrTlMIlJvenIA9qFgaIjBEPvvK c9yBw6x9Ie5k1ofgcav1Adydz6bBQ6SfYsFfhsD47ifG1cuTcdeRhL6Ypajz+twJ gPO5p/guaV1+iBPvZmCDSPEoqRdR3EDYbpGVy4VeM/+vEtKJLnfvonsW1criw6aF I8ja4V4L0aVBhjjHwcKdaHK8MYvg82rJR3CyajiklrvY1SlYY7mhcUfvZAMbILJS xMkRSdYDNX8XrZjs9FyPmzhlXGs4Wx1yKrRMNV46orGxUJBMGobPAWPXwIjZVGAU IUVCLItJuFcn7b5HlNwvdSfi8GH8+HZhirsAGQ8vk+U4hDI8afTAyvo4bE1MEZ0O EaxoAZ+Bz2MbFMeaig6WIc35Ki7UxPaXhQsPoUOZLJrqbBzawSbPF/jz6siF2CLp aZHBE6QQGpEJ/mM62fxm8PF4480+B4y9MNaeH09Pq1f0zEIaiFIqYL6yglreiU4e K9T2x+a+FZ6qcfXHKi0fvmjwPrM3BLA0PrOZJczRByTS9iOVYyqNzxzxXfA/rU76 yU643JxmV8F58zKgma6PEhRBb17kscwf4E13jRTGAcQLIB8XwKk= =4JbI -----END PGP SIGNATURE----- --==-=-=--