From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: Regression in dump-emacs-portable Date: Thu, 16 Feb 2023 10:05:00 -0500 Message-ID: References: <83ttzocomk.fsf@gnu.org> <834jrncd6a.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="0000000000000355dc05f4d2865b" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="1984"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Feb 16 16:05:54 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pSfpa-0000Ku-00 for ged-emacs-devel@m.gmane-mx.org; Thu, 16 Feb 2023 16:05:54 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSfpA-00029m-7T; Thu, 16 Feb 2023 10:05:28 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSfp5-0001ud-8I for emacs-devel@gnu.org; Thu, 16 Feb 2023 10:05:25 -0500 Original-Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pSfoy-0001oc-UK; Thu, 16 Feb 2023 10:05:20 -0500 Original-Received: by mail-pg1-x531.google.com with SMTP id c29so1451633pgm.5; Thu, 16 Feb 2023 07:05:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=HOGyv+MG7SvXokoyUgueBBMygyzN9CI/jfLMQjRHJE0=; b=DNT5NGsV3BrGboIshNBpsprPJKElKvMA65+V2qioOzCp3hEW98G0xfLtM7vJVKpifh vwHN4aWhjVfIIT33vpBaW9OLWnOLpuWdtGdWWNbvOYCG5lLRDJjJCRMZK69r7w1wTzJX lrBdmFYTvl3yeyInP+TJ/PpkOZcAagiH90ClYWmAtooGBN66yCq+1LczFbr9vWIdqfYq jQwxNZGV+tWNenWF3z8nmQ/d/35UC0j4+rpPrzlzHv/JD/Z+aWNfso87Oujx8qqo0XYl lFBZNmlo7lfeihJM0i26nJxfpmiOyI/NZAhgNuSDNqkJn4H0u4/6whTPgqXDjDxsoz0X mReQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HOGyv+MG7SvXokoyUgueBBMygyzN9CI/jfLMQjRHJE0=; b=zkDHG1CF4BtK8WYN/4e5+VQWRfSAAgaNpa1rVeu6QEQSU4h5eY7vOC5axiCbtobAqp nO+7SWTbUNUEpPQrzDcojH1L3nRLEFTjkioNPf40wi6splhSVD1tBNcpAQuQ3v/hn0PZ krleDIJZ1Z0J6tKeeRI7R9fLRpuuEPPfw+oyXm9N+r60ZFKuiyT/evvtFMx09XE+1qam p70A6iPIDg1KrKpl3DY0FJr6Ba/UsWit0nk/h+dKzmbAz9v0D+dIP9YQa8uIBOwIFcPq OKKU7ZfxhBDaKP2fbEKeWfTe4x1MeildZkjcBFHPz942mT6BaphKwWZR5ruzK4ElFRBm 741g== X-Gm-Message-State: AO0yUKWGwYQW6TIYmBnA3havzWacBMgCyslwBjkbK6uOHOU+j6V5/T5M rUoXoBkaEJtnBOybFUxVWtKyg2g+F+rbULfeWCIj4zEz X-Google-Smtp-Source: AK7set/PU4cW02Y6XcDM7+15HiXVUmW7nvhfFOOPmre8rWIjlsZcoHlLfJOleDWQzRo8XE4wiJtkwlyL2BsGVOTGTTw= X-Received: by 2002:a63:6e41:0:b0:4d3:459a:2258 with SMTP id j62-20020a636e41000000b004d3459a2258mr924468pgc.3.1676559912226; Thu, 16 Feb 2023 07:05:12 -0800 (PST) In-Reply-To: Received-SPF: pass client-ip=2607:f8b0:4864:20::531; envelope-from=owinebar@gmail.com; helo=mail-pg1-x531.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:303419 Archived-At: --0000000000000355dc05f4d2865b Content-Type: text/plain; charset="UTF-8" On Thu, Feb 16, 2023, 4:54 AM Lynn Winebarger wrote: > On Thu, Feb 16, 2023 at 4:31 AM Lynn Winebarger > wrote: > > On Wed, Feb 15, 2023, 7:43 AM Eli Zaretskii wrote: > >> > From: Lynn Winebarger > >> > Date: Tue, 14 Feb 2023 18:26:07 -0500 > >> > Cc: emacs-devel@gnu.org > >> > > >> > On Tue, Feb 14, 2023 at 9:23 AM Eli Zaretskii wrote: > >> > > > >> > > What do these tests actually test? > >> > > >> > Whether libraries expected to be redumpable are in fact redumpable. > >> > >> That's the goal, not the actual testing algorithm. I asked about the > >> latter. How do you intend to test that a dump succeeded (assuming > >> there's no crash)? > > > > > > The dump will have to be performed in a separate emacs process. The > easiest criteria to judge is whether the dump file exists and is greater > than 0 bytes. Emacs appears to create a 0 byte file when > dump-emacs-portable is invoked, which is just not updated if the dump > terminates unsuccessfully. > > A second criteria is then to invoke emacs with the dump-file and > evaluate some simple expression to verify no unexpected errors were > encountered on load. > > I've started automating my process with some simple shell scripting > tracked at https://github.com/owinebar/emacs-redumping. It's not much > yet, but at least I was able to align my efforts between 28.2 and 30.0.50. > The next step will be to create a proper load-time dependency graph, so I > can automate the calculation of the minimal list of features that need to > be provided so that the maximum number of libraries can be loaded for the > dump, with the artificially provided features loaded on an after-init hook > (because before-init happens prior to the X frame initialization). > > Once these dependencies are identified and lists are calculated, then > creating a set of canned tests should be straightforward. Some > makefile-based approach should be adequate for determining which parts of > the dependency graph need to be recalculated after an update. > > I want to calculate these dependencies (and compile-time dependencies) > to construct a more robust native-compilation build process anyway. > > For a regression test, I would want to record the results from 28.2 as a > basis for measuring 29 and 30, at least as a starting point. In any case, > I never see an "abort signal" termination in 28, or even a "weird > pseudovector" message. It's either something incompatible (because I > blindly attempted to load the world) as in the "term" subdirectory or > dos/w32 libraries under linux, or some redefinition of a character table > (which is why I calculate the files loaded in the baseline dump and exclude > them). I got some very lengthy error messages printing out explicit > objects from some obsolete libraries, so I exclude them as well. > > And viper demands user input at startup when it's loaded, so it has to > be excluded from dumping. There might be some variable to turn off that > annoying behavior, I'm just not interested in investigating. > > > >> > Almost every library in 28.2 could be redumped, excepting those which > >> > simply failed to load for whatever reason. > >> > >> Don't we have Lisp objects that cannot be dumped? If we do, then not > >> every library could be dumped even in principle. > > > > > > In 28.2, using dump-emacs-portable, the answer is, not many in the > libraries in included in the Emacs source distribution. I excluded the > term and obsolete subdirectories from generating the set of libraries to > dump (but not from the final set determined from load-history). Even > outside of the emacs distribution, the only problematic objects are dynamic > modules. I assume this is due to dumping in batch-mode. My exclusion on > wid-edit.el is because dumping it in batch-mode appears to bar it from ever > subsequently creating proper buttons in a graphic terminal. But dumping > it still succeeds. > > > >> Another potential issue with this is (assuming you suggest to actually > >> try dumping every library) that it will take too long, and thus will > >> be likely to be skipped in any "normal" run of the test suite, thus > >> missing the point. > > > > My 2017-vintage laptop dumps the 1252 files, including all of leim, in > 34 seconds, for a 135MB dump file. > > When I added leim to the exclusions list, 1172 libraries are dumped in > 24 seconds for a 83MB dump file, which explains why my effort with 30.0.50 > produces a 75MB dump. I excluded leim for 30.0.50 because I was > encountering too many errors to deal with manually, which explains most of > the size reduction. > > I'm not sure how the tests are normally run, but I would think anyone > working on pdumper should be interested in a comprehensive test at some > point. Aside from testing on a per-commit basis, isn't there a more > comprehensive set of regression tests run pre-release? Does emacs have a CI > process regularly running the test suite, or is it more ad hoc? If nothing > else, failures reported from such a routine run could be used to create a > more targeted test set for someone actively working on pdumper. > > Just to finish this thought - dumping the full set of libraries, > excluding a few expected to fail, should be the "normal" test. If > 24-34 seconds is too long, there are probably other large subsets that > provide substantial coverage in less time. The more comprehensive > file-by-file approach should be reserved for tracking down the cause > of failures in the normal test. Theoretically, pdumper might be able > to indicate the source library(ies) associated with particular error, > but the file-by-file approach is always available. > I do see something in the redumped emacs that seems like a bug to me. The process I use for creating the dump uses the -Q flag. But some of the settings I see in "emacs -Q --dump-file ..." are not the ones I see with just "emacs -Q". Some are pretty basic - menu-bar-mode, tool-bar-mode, global-font-lock-mode, transient-mark-mode are all nil in the redumped process but not the baseline. I should check if that happens even when the only expression evaluated is the call to dump-emacs-portable, with nothing additional loaded. A general test would be load the additional files to be dumped, write out all the symbol properties, variable values, and function values, then load the dump file and compare everything with equal, including the set of symbols, variables and functions defined. Is it fair to say that is the correct expectation of the dumping procedure? --0000000000000355dc05f4d2865b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Thu, Feb 16, 2023, 4:54 AM Lynn Winebarger <owinebar@gma= il.com> wrote:
On Thu, Feb 1= 6, 2023 at 4:31 AM Lynn Winebarger <owinebar@gmail.com>= ; wrote:
> On Wed, Feb 15, 2023, 7:43 AM Eli Zaretskii <eliz@gnu.org&= gt; wrote:
>> > From: Lynn Winebarger <owinebar@gmail.com&g= t;
>> > Date: Tue, 14 Feb 2023 18:26:07 -0500
>> > Cc: emacs-devel@gnu.org
>> >
>> > On Tue, Feb 14, 2023 at 9:23 AM Eli Zaretskii <eliz@g= nu.org> wrote:
>> > >
>> > > What do these tests actually test?
>> >
>> > Whether libraries expected to be redumpable are in fact redum= pable.
>>
>> That's the goal, not the actual testing algorithm.=C2=A0 I ask= ed about the
>> latter.=C2=A0 How do you intend to test that a dump succeeded (ass= uming
>> there's no crash)?
>
>
> The dump will have to be performed in a separate emacs process. The ea= siest criteria to judge is whether the dump file exists and is greater than= 0 bytes.=C2=A0 Emacs appears to create a 0 byte file when dump-emacs-porta= ble is invoked, which is just not updated if the dump terminates unsuccessf= ully.
> A second criteria is then to invoke emacs with the dump-file and evalu= ate some simple expression to verify no unexpected=C2=A0 errors were encoun= tered on load.
> I've started automating my process with some simple shell scriptin= g tracked at https://github.com/owi= nebar/emacs-redumping.=C2=A0 It's not much yet, but at least I was = able to align my efforts between 28.2 and 30.0.50.=C2=A0 The next step will= be to create a proper load-time dependency graph, so I can automate the ca= lculation of the minimal list of features that need to be provided so that = the maximum number of libraries can be loaded for the dump, with the artifi= cially provided features loaded on an after-init hook (because before-init = happens prior to the X frame initialization).
> Once these dependencies are identified and lists are calculated, then = creating a set of canned tests should be straightforward.=C2=A0 Some makefi= le-based approach should be adequate for determining which parts of the dep= endency graph need to be recalculated after an update.
> I want to calculate these dependencies (and compile-time dependencies)= to construct a more robust native-compilation build process anyway.
> For a regression test, I would want to record the results from 28.2 as= a basis for measuring 29 and 30, at least as a starting point.=C2=A0 =C2= =A0In any case, I never see an "abort signal" termination in 28, = or even a "weird pseudovector" message.=C2=A0 It's either som= ething incompatible (because I blindly attempted to load the world) as in t= he "term" subdirectory or dos/w32 libraries under linux, or some = redefinition of a character table (which is why I calculate the files loade= d in the baseline dump and exclude them).=C2=A0 I got some very lengthy err= or messages printing out explicit objects from some obsolete libraries, so = I exclude them as well.
> And viper demands user input at startup when it's loaded, so it ha= s to be excluded from dumping.=C2=A0 There might be some variable to turn o= ff that annoying behavior, I'm just not interested in investigating. >
>> > Almost every library in 28.2 could be redumped, excepting tho= se which
>> > simply failed to load for whatever reason.
>>
>> Don't we have Lisp objects that cannot be dumped?=C2=A0 If we = do, then not
>> every library could be dumped even in principle.
>
>
> In 28.2, using dump-emacs-portable, the answer is, not many in the lib= raries in included in the Emacs source distribution.=C2=A0 I excluded the t= erm and obsolete subdirectories from generating the set of libraries to dum= p (but not from the final set determined from load-history).=C2=A0 Even out= side of the emacs distribution, the only problematic objects are dynamic mo= dules.=C2=A0 I assume this is due to dumping in batch-mode.=C2=A0 My exclus= ion on wid-edit.el is because dumping it in batch-mode appears to bar it fr= om ever subsequently creating proper buttons in a graphic terminal.=C2=A0 = =C2=A0But dumping it still succeeds.
>
>> Another potential issue with this is (assuming you suggest to actu= ally
>> try dumping every library) that it will take too long, and thus wi= ll
>> be likely to be skipped in any "normal" run of the test = suite, thus
>> missing the point.
>
> My 2017-vintage laptop dumps the 1252 files, including all of leim, in= 34 seconds, for a 135MB dump file.
> When I added leim to the exclusions list,=C2=A0 1172 libraries are dum= ped in 24 seconds for a 83MB dump file, which explains why my effort with 3= 0.0.50 produces a 75MB dump.=C2=A0 I excluded leim for 30.0.50 because I wa= s encountering too many errors to deal with manually, which explains most o= f the size reduction.
> I'm not sure how the tests are normally run, but I would think any= one working on pdumper should be interested in a comprehensive test at some= point.=C2=A0 Aside from testing on a per-commit basis, isn't there a m= ore comprehensive set of regression tests run pre-release? Does emacs have = a CI process regularly running the test suite, or is it more ad hoc?=C2=A0 = If nothing else, failures reported from such a routine run could be used to= create a more targeted test set for someone actively working on pdumper.
Just to finish this thought - dumping the full set of libraries,
excluding a few expected to fail, should be the "normal" test.=C2= =A0 If
24-34 seconds is too long, there are probably other large subsets that
provide substantial coverage in less time. The more comprehensive
file-by-file approach should be reserved for tracking down the cause
of failures in the normal test.=C2=A0 Theoretically, pdumper might be able<= br> to indicate the source library(ies) associated with particular error,
but the file-by-file approach is always available.

I do see something in the= redumped emacs that seems like a bug to me.=C2=A0 The process I use for cr= eating the dump uses the -Q flag.=C2=A0 But some of the settings I see in &= quot;emacs -Q --dump-file ..." are not the ones I see with just "= emacs -Q".=C2=A0 Some are pretty basic - menu-bar-mode, tool-bar-mode,= global-font-lock-mode, transient-mark-mode are all nil in the redumped pro= cess but not the baseline.=C2=A0 I should check if that happens even when t= he only expression evaluated is the call to dump-emacs-portable, with nothi= ng additional loaded.
A general test would be load t= he additional files to be dumped, write out all the symbol properties, vari= able values, and function values, then load the dump file and compare every= thing with equal, including the set of symbols, variables and functions def= ined.
Is it fair to say that is the correct expectat= ion of the dumping procedure?
--0000000000000355dc05f4d2865b--