From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: Regression in dump-emacs-portable Date: Thu, 16 Feb 2023 20:29:34 -0500 Message-ID: References: <83ttzocomk.fsf@gnu.org> <834jrncd6a.fsf@gnu.org> <83o7pt395q.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36730"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Feb 17 02:30:44 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pSpaG-0009OE-Fl for ged-emacs-devel@m.gmane-mx.org; Fri, 17 Feb 2023 02:30:44 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSpZQ-0004sN-TD; Thu, 16 Feb 2023 20:29:52 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSpZP-0004rw-E9 for emacs-devel@gnu.org; Thu, 16 Feb 2023 20:29:51 -0500 Original-Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pSpZN-00024j-1y; Thu, 16 Feb 2023 20:29:50 -0500 Original-Received: by mail-pg1-x531.google.com with SMTP id a23so2359067pga.13; Thu, 16 Feb 2023 17:29:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=tJutegACy6aks2Yj/G0aW/8yA5Dlrw7Y7b/dZ2vIZjA=; b=O5tq0OV9Xtw3ji8N6Co3tNOhFVPh2W7ykNtBdFLJUkYvQNcRtkP+A/BqfDgu8tYjmI VVuEUrhKviB1qDRlQTaT5WmZrG6tBrHBtrxTvTlwyIkQc19fNFpXzH5ivxYQ40eDF7Dv D1Ebas0VfRgNrwq8vfBfZe9/VneJXIZ2LC/i3WDMBN/ScK5zEW2DCveuQ6/EzjkdsO7Y nW4y3bCrmLAgfLzLPUcjXuZZJ2WrozrlJEUuK8YlUSOx9SBGnlUJUkOTci9cL9oG8rCe QPYdGyyUP5u8zhOUIPAHw2CIobpeb5uD0FkBAMuXmpatao1tCkSl62juWNPqkcYi4CM8 dUJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tJutegACy6aks2Yj/G0aW/8yA5Dlrw7Y7b/dZ2vIZjA=; b=56VKj4kxHsGsRNRsC3IXb0gm7fLRMTdbViWimBuVG9wQ9i+S9CaJuI2ySlppnu//UG Psa7CrCugW0mxH2EYuXRG9ds1iY9KOHu++XvM4fiY6OXokjAbI+riGrn4U/1GD+gmQhi qtNg2anZzPIsp/Vm+BeV4ynlq+XoEwDfQ3eheLGEXXGIMygmj/tR83uw26ETKWfzaubU Hsmm1CHw92bM4KzB5S8ND3ABTtoRbEsYRaT63gpBe0Loe3V6MTDMPp2uTOr86+STmk26 tnolxaf39es1zudsItuIB5SUQg8o9dg4/xE+0tYA608t0Wrqqx4RggVJaSnDeJLBk5eT 07KA== X-Gm-Message-State: AO0yUKU4jOHQX1Yx0ALTc+2QHxcPctoIOOujsLkELoRqVjsyab/Z8vM2 YwFFpXAiQiPRJmMnrKnL4DiBQ5X+l2hCeXJdhgzhZCren8I= X-Google-Smtp-Source: AK7set/IhXUzvpugriWB8p2ACqqn3Nyr9CCSkGVnFyytc0pPYSfUjbMWSomnBg7KEiDIeGgU4tyK5eescR/tz2iGJeM= X-Received: by 2002:a62:ab07:0:b0:594:2681:dd5d with SMTP id p7-20020a62ab07000000b005942681dd5dmr47537pff.4.1676597386236; Thu, 16 Feb 2023 17:29:46 -0800 (PST) In-Reply-To: <83o7pt395q.fsf@gnu.org> Received-SPF: pass client-ip=2607:f8b0:4864:20::531; envelope-from=owinebar@gmail.com; helo=mail-pg1-x531.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:303444 Archived-At: On Thu, Feb 16, 2023 at 10:46 AM Eli Zaretskii wrote: > > > Almost every library in 28.2 could be redumped, excepting those which > > > simply failed to load for whatever reason. > > > > Don't we have Lisp objects that cannot be dumped? If we do, then not > > every library could be dumped even in principle. > > > > In 28.2, using dump-emacs-portable, the answer is, not many in the libraries in included in the Emacs > > source distribution. I excluded the term and obsolete subdirectories from generating the set of libraries to > > dump (but not from the final set determined from load-history). > > Is exclusion really the way to go, if we want eventually to support > re-dumping at any given moment? At the moment, my concern is just with keeping the functionality 28.2 has for redumping, unless there is some deliberate tradeoff being made. I'm just using exclusion until I can automate a more precise determination of what is dumpable and what isn't. Also, some of those libraries may be included in the dump - I'm only excluding them from the set used to generate the list of libraries to include in the dump. I probably could have narrowed down the problematic libraries, but I didn't want to spend a lot of time manually investigating why libraries classified as obsolete were causing problems. On the term side, it seemed that the ones causing issues were either due to incompatibility (win32 on linux, say) or perhaps required some feature not available in batch mode, so just attempting to load them caused problems. More precise lists of expected failures can be created. > > > Another potential issue with this is (assuming you suggest to actually > > try dumping every library) that it will take too long, and thus will > > be likely to be skipped in any "normal" run of the test suite, thus > > missing the point. > > > > My 2017-vintage laptop dumps the 1252 files, including all of leim, in 34 seconds, for a 135MB dump file. > > When I added leim to the exclusions list, 1172 libraries are dumped in 24 seconds for a 83MB dump file, > > which explains why my effort with 30.0.50 produces a 75MB dump. > > That's a lot. And I presume your build is optimized? So an > unoptimized build will be slower. Running such slow tests routinely > is a PITA, so such a test will probably moved to a category that > doesn't run by default. It's fine with me if the tests aren't run directly by the developer, unless they're particularly concerned about it. I'd expect pdumper functionality to require testing on multiple platforms anyway, not just whatever system the developer is using. I can pick some that are causing failures now, but I have no idea why they stress the current implementation. icomplete causes an abort in 131ms. sh-script generates a "weird pseudovector" message in 376ms > time emacs -Q -batch --eval '(load "icomplete")' --eval '(dump-emacs-portable "test-icomplete.pdmp")' Loading icomplete... Dumping fingerprint: 30c4a01f07c30c32913a3c578aa6871d7af188212b14e2f9c5e3717f80b9ab43 Fatal error 6: Aborted Backtrace: emacs(emacs_backtrace+0x37)[0x55ec89e6de67] emacs(terminate_due_to_signal+0x6f)[0x55ec89d33514] emacs(nsberror+0x0)[0x55ec89d33726] emacs(+0x21f9da)[0x55ec89ebb9da] emacs(+0x21efc7)[0x55ec89ebafc7] emacs(+0x2203a1)[0x55ec89ebc3a1] emacs(Fdump_emacs_portable+0xb70)[0x55ec89ebd070] emacs(eval_sub+0x9b3)[0x55ec89edc1d3] emacs(Feval+0x3f)[0x55ec89ede22f] emacs(exec_byte_code+0x482)[0x55ec89f1e182] emacs(+0x2412fc)[0x55ec89edd2fc] emacs(eval_sub+0x2bc)[0x55ec89edbadc] emacs(Feval+0x3f)[0x55ec89ede22f] emacs(internal_condition_case+0x3e)[0x55ec89ed6c7e] emacs(+0x1b4618)[0x55ec89e50618] emacs(internal_catch+0x3d)[0x55ec89ed6bed] emacs(+0x1b2998)[0x55ec89e4e998] emacs(recursive_edit_1+0x86)[0x55ec89e54d56] emacs(Frecursive_edit+0xd0)[0x55ec89e550f0] emacs(main+0x190f)[0x55ec89d35c5f] /lib64/libc.so.6(__libc_start_main+0xef)[0x7fb49dd4129d] emacs(_start+0x2a)[0x55ec89d369ea] Aborted (core dumped) real 0m0.131s user 0m0.102s sys 0m0.029s =================================== > time emacs -Q -batch --eval '(load "sh-script")' --eval '(dump-emacs-portable "test-sh-script.pdmp")' Loading sh-script... Dumping fingerprint: 30c4a01f07c30c32913a3c578aa6871d7af188212b14e2f9c5e3717f80b9ab43 Error: error ("unsupported object type in dump: weird pseudovector") mapbacktrace(#f(compiled-function (evald func args flags) #)) debug-early-backtrace() debug-early(error (error "unsupported object type in dump: weird pseudovector")) dump-emacs-portable("test-sh-script.pdmp") eval((dump-emacs-portable "test-sh-script.pdmp") t) command-line-1(("--eval" "(load \"sh-script\")" "--eval" "(dump-emacs-portable \"test-sh-script.pdmp\")")) command-line() normal-top-level() unsupported object type in dump: weird pseudovector real 0m0.376s user 0m0.172s sys 0m0.040s =================================== > > Aside from testing on a per-commit basis, isn't there a > > more comprehensive set of regression tests run pre-release? Does emacs have a CI process regularly > > running the test suite, or is it more ad hoc? > > There's EMBA. Ok, that's the kind of thing I was thinking of. I'll read over the relevant admin/notes before asking more questions. I'm going to put some more work into automating generation of comprehensive redumping scripts this weekend. Once I have that I can look into how to use that process to generate regression tests for specified sets of libraries to redump. Thanks, Lynn