From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: Loading tramp for dump goes into infinite regress Date: Mon, 25 Jul 2022 08:49:27 -0400 Message-ID: References: <8735erhrlg.fsf@gmx.de> <83wnc2g0n8.fsf@gnu.org> <83sfmqfxcb.fsf@gnu.org> <83a68yfp6j.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000fc55e205e4a09dde" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35763"; mail-complaints-to="usenet@ciao.gmane.io" Cc: michael.albinus@gmx.de, emacs-devel To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Jul 25 14:52:24 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oFxZN-00096l-J7 for ged-emacs-devel@m.gmane-mx.org; Mon, 25 Jul 2022 14:52:21 +0200 Original-Received: from localhost ([::1]:41702 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oFxZM-0001c3-HG for ged-emacs-devel@m.gmane-mx.org; Mon, 25 Jul 2022 08:52:20 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:55270) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oFxWq-0007Af-Rt for emacs-devel@gnu.org; Mon, 25 Jul 2022 08:49:44 -0400 Original-Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]:44658) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oFxWo-0002sX-W6; Mon, 25 Jul 2022 08:49:44 -0400 Original-Received: by mail-pg1-x532.google.com with SMTP id bf13so10258521pgb.11; Mon, 25 Jul 2022 05:49:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sm/zXMaOpcp2RBT/SgLf5yxHjX7hLo01BAuZhM5h8JA=; b=mhgErC7dWq7qLe0/bWGdKwpDHU/ledJvCnlmyCNqalT3fm1kumrpTmJ2mmodO/EERs EVOCtSkzgnp9dBhLafzUeK1O1iYkhRwun5hNP79bG+z1aaXcCJhe1T8LG8dqoXC70Ugr fiVOcjPHGGySN5dF/7UyazOPuijaYoiSUBWxW2Ra7lDmmnMxGXaxlofNa5iCsYRSquZs SoBqSPXhC3kqRNsGysrREYaNwerMS2PqPl1Oj5K2k43k0cNpcgVkCG6nsSDOigljvJaa mckb43TGetP6fqTjgf9wIGDiD6Cp7VE2HCrkl5XQBpiUcDihpQvYiUyiv7dhmVAHVgo4 BHMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sm/zXMaOpcp2RBT/SgLf5yxHjX7hLo01BAuZhM5h8JA=; b=uBvMUQ8TvpVLqFgmoW073NSzghNAosYmFm5AufMLTCkN7pLgQrQSq6d1JTilvDBzBD Dp1eDJQydLnaWtgZbCOyAx6wOaRub/N+IrGiAMdeHhPIr1WM9kQwU9ys5d41EoZ9tusE d5LwOdZ/gvRVRtWn9IryFsSE1hw36aT/Cn5ZuFWlI3ZKxFsVz4jZjjZtFT/w8Vz0e6W0 6CjS0K8wRQpBouaIPPgsONiVGVMXfwWjhvZmEqBcHiwp7l/ngZQwVSVNvJ0QaA2FMm15 /vcgEndYdeIL654VWUDUpGMYFVQXwQDbTLMLp3iaI7JsdL70DeeaMkkXsVFdDS8SEIHl mZAQ== X-Gm-Message-State: AJIora+rB1RJICU2Bhj49YuOV946dbu0Z2PRlOIDrdgUoN/PAliLKRnV zK1jHGhHg4wN0oBubEwBP39F5WD/WnnLFr7gDFVcYvlf X-Google-Smtp-Source: AGRyM1vPZzs8x6XleJKLQR5ryCMXz7YaTp9cH+a81pns+6j9I4MdPCv/TKYcdTdTq4RlNf0yP1CYoaWd0dxWFanowCM= X-Received: by 2002:a05:6a00:298d:b0:52b:cf1f:5738 with SMTP id cj13-20020a056a00298d00b0052bcf1f5738mr12693866pfb.0.1658753380012; Mon, 25 Jul 2022 05:49:40 -0700 (PDT) In-Reply-To: <83a68yfp6j.fsf@gnu.org> Received-SPF: pass client-ip=2607:f8b0:4864:20::532; envelope-from=owinebar@gmail.com; helo=mail-pg1-x532.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FREEMAIL_REPLY=1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:292626 Archived-At: --000000000000fc55e205e4a09dde Content-Type: text/plain; charset="UTF-8" On Sun, Jul 24, 2022 at 12:31 PM Eli Zaretskii wrote: > > > From: Lynn Winebarger > > Date: Sun, 24 Jul 2022 12:20:47 -0400 > > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org > > > > it seems like something undesirable is going on between dumping and > > the native compilation unit. > > I find this hard to believe. If I can recreate it in a more direct way on my personal machines, I'll put in a bug with details including the required modifications to tramp.el (to remove the problematic require of tramp-loaddefs). I also observed the same problem/fix with: gnus/message progmodes/gdb-mi calc/calc-ext gnus/gnus-sum gnus/gnus-art More troublesome, though is that emacs segfaulted when it attempted to load nxml/rng-pttrn, whether loading native-compiled, byte-compiled, or source. Otherwise, I was able to resolve the various issues (including some extensive surgery on vc/ediff-X files to stamp out the circular loading). I am going to have to do a real bootstrap to avoid the "incoherent eln" error, though, and somehow force native compilation for all the libraries loaded in the dump (that are not on the compiler's "forbidden" list, anyway). > Btw, you are aware that dumping *.eln files basically dumps just their > names, not their code? And that, when you start Emacs after dumping, > it will load all of those *.eln files one by one, which takes time > (*.eln files are just shared libraries, like *.so files), and use up > shared-library and handle slots of the Emacs process? So I'm not even > sure doing this would make sense from the performance POV: it could be > that startup will slower, not faster. I am aware that dumping the eln files produces an indirection to the shared library, but not the details of the implementation. I would expect (or at least hope) that the effect of the evaluation that is done on loading to be stored in the dump and for the shared libraries to get loaded into the process without repeating the evaluation. So things like the order of customization groups should be fixed. I would also expect that whatever the effect is on the emacs process with .eln files loaded by the dumped file, it is no worse than it would be if loaded as part of my .emacs file. Another benefit I expect from native-compilation, dumped or not, is more efficient memory use when running multiple emacs processes. With dumping, I would expect (or hope for) better garbage collector behavior since the amount of allocation required for the loaded modules should be pre-determined (whether byte- or native-compiled). If the image is 300MB (including the shared libraries), so be it, as long as the memory is shared between multiple processes. I'd also like a baseline expectation of performance with native-compiled libraries in the dumped image. To the extent that dumping verifies that there is no cyclic loading, I can always try the approach to linking I inquired about on this list about a month ago: just catenating the source files together with some minor alterations. I'm not sure how large a source file the native compiler (or libgccjit) will handle, but it would be interesting to see. > In any case, I'd suggest to get this working with *.elc files in a > build without native compilation support, before you try it with > native compilation. First I'll see if I can get the ordering straightened out with just the core emacs files required by these packages, then do a test between byte-compiled and native-compiled on those. Lynn --000000000000fc55e205e4a09dde Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sun, Jul 24, 2022 at 12:31 PM Eli Zar= etskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sun, 24 Jul 2022 12:20:47 -0400
> > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> >
> > it seems like something undesirable is going on between dumping a= nd
> > the native compilation unit.
>
> I find this hard to believe.

If I can recreate it in a more direct way on my personal machines, I'll= put in a bug with details including the required modifications to tramp.el= (to remove the problematic require of tramp-loaddefs).
I also obse= rved the same problem/fix with:
=C2=A0 gnus/message
=C2= =A0 progmodes/gdb-mi
=C2=A0 calc/calc-ext
=C2=A0 gnus/g= nus-sum
=C2=A0 gnus/gnus-art
More troublesome, though i= s that emacs segfaulted=C2=A0when it attempted to load nxml/rng-pttrn, whet= her loading native-compiled, byte-compiled, or source.

Otherwise, I was able to resolve the various issues (incl= uding some extensive surgery on vc/ediff-X files to stamp out the circular = loading).=C2=A0 I am going to have to do a real bootstrap to avoid the &quo= t;incoherent eln" error, though, and somehow force native compilation = for all the libraries loaded in the dump (that are not on the compiler'= s "forbidden" list, anyway).

> Btw, you are aware that dumping *.eln files basically dumps just the= ir
> names, not their code?=C2=A0 And that, when you start Emacs after dump= ing,
> it will load all of those *.eln files one by one, which takes time
> (*.eln files are just shared libraries, like *.so files), and use up > shared-library and handle slots of the Emacs process?=C2=A0 So I'm= not even
> sure doing this would make sense from the performance POV: it could be=
> that startup will slower, not faster.

I am aware that dumping the eln files produces an indirection to the shared= library, but not the details of the implementation.
I would expect (or at least hope) that the effect of the evaluation that is= done on loading to be stored in the dump and for the shared libraries to g= et loaded into the process without repeating the evaluation.=C2=A0 So thing= s=C2=A0like=C2=A0the order of customization groups should be fixed.
I would also expect that whatever the effect is on the emacs process with .= eln files loaded by the dumped file, it is no worse than it would be if loa= ded as part of my .emacs file.
Another benefit I expect from native-comp= ilation, dumped or not, is more efficient memory use=C2=A0when running mult= iple emacs processes.=C2=A0 With dumping, I would expect (or hope for) bett= er garbage collector behavior since the amount of allocation required for t= he loaded modules should be pre-determined (whether byte- or native-compile= d).=C2=A0 If the image is 300MB (including the shared libraries), so be it,= as long as the=C2=A0memory is shared between multiple processes.=C2=A0=C2= =A0

I'd also like a baseline expectation of pe= rformance with native-compiled libraries in the dumped=C2=A0image.=C2=A0 To= the extent that dumping=C2=A0verifies that there is no cyclic loading, I c= an always try the approach to linking I inquired about on this list about a= month ago: just catenating the source files together with some minor alter= ations.=C2=A0 I'm not sure how large a source file the native compiler = (or libgccjit) will handle, but it would be interesting to see.

> In any case, I'd suggest to get this working with *.elc files in a=
> build without native compilation support, before you try it with
> native compilation.

First I'll see if I can get the ordering straightened out with just the= core emacs files required by these packages, then do a test between byte-c= ompiled and native-compiled on those.

=
Lynn

--000000000000fc55e205e4a09dde--