From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: native compilation units Date: Sun, 5 Jun 2022 08:16:25 -0400 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="0000000000009f1a1905e0b25382" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="20772"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Andrea Corallo , emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Jun 05 14:58:02 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nxppR-00059O-BT for ged-emacs-devel@m.gmane-mx.org; Sun, 05 Jun 2022 14:58:01 +0200 Original-Received: from localhost ([::1]:34460 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nxppQ-0003ZK-CD for ged-emacs-devel@m.gmane-mx.org; Sun, 05 Jun 2022 08:58:00 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:45786) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nxpBR-00087k-1m for emacs-devel@gnu.org; Sun, 05 Jun 2022 08:16:41 -0400 Original-Received: from mail-lj1-x235.google.com ([2a00:1450:4864:20::235]:39792) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nxpBO-0008LS-6r for emacs-devel@gnu.org; Sun, 05 Jun 2022 08:16:40 -0400 Original-Received: by mail-lj1-x235.google.com with SMTP id b7so553708ljr.6 for ; Sun, 05 Jun 2022 05:16:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=E7x+MoxUb6/OH/XGKBUuq5uvQx1hiAt+aSQ2+4YgHtU=; b=Ei73xM1oWgjUkfBGVRtQqnLP58EMeI5XSyzOviJKzasfDRB4z5XcrI8DwyF1aihcYp 0ho0w6AqHpiesMi4Uxv/ds3dKuP6qgYf/pI6Ff1kM01reDTO6K6zN2nO0zg9pbZE3jHR Zuz3pxHRvYESrF9eLjuyqmSEjcoTcGSdkGDvohhpKQ//atP+oc3QhrSdS2bu2o0b3fYm jUSQ2jz6hBUG7iZJV3bAAWFrjh6TXmW62XEr/+4t6EEj+F0HEW5F7Qkhb5CUMItXoP9/ kebZtTjkCk9cXNf0ujXPmoN8pVehwQVPxiEXS7lL00IhZXCngpNkcp2s4726lg3lKY4q WWmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=E7x+MoxUb6/OH/XGKBUuq5uvQx1hiAt+aSQ2+4YgHtU=; b=B0pZD1c8zE55Y9e0PtcZlohpbQp7+OA0Hl8KRk4IBkU+TyxAGXNW6r6h2lYc3uhoBk J+cgXoBBeIRyZodzA/606ErtzFm//6mgtk7Ff/D7GqliKvaAZ1xklbcnn+B8B1MwqhGj v62tuFzuiBj/XFuZzIYfu3Itw9c1HNmZG2oHPe6Ch3EAmeHXa9KoxqzeWWgE3fh/y98A wMIMTa4OJ80HWEsEL2/WL9SNeCewmHNwlsbS5kr2ZWQ74MOsOruq5iHC8HlhEm/rfkVR A+ALEWekbEEMQEAg9d1D3NXnqL533eCV6qzItnblFeNZK7m4xVzEZ1Cb4xgYLR2n5P0Q pZSA== X-Gm-Message-State: AOAM531q1mfbdFIW1r6/B36DwITYw1iVDQhC0yESzJo9Rtx6Lvy5F0Vy S/4H7Y9BaljjFW5yv92hhQjB4dPpPxF6LORc9xI= X-Google-Smtp-Source: ABdhPJzZBkmaS8m/4MHVA8galofZT9nYQQIrDnw4jQCyC+mEE3Jl1q2+FM3HtmKCjCSNG0iN4O8CDwc8TU8Kh/h6vTw= X-Received: by 2002:a2e:82c4:0:b0:255:4672:5599 with SMTP id n4-20020a2e82c4000000b0025546725599mr23053865ljh.26.1654431395288; Sun, 05 Jun 2022 05:16:35 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2a00:1450:4864:20::235; envelope-from=owinebar@gmail.com; helo=mail-lj1-x235.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sun, 05 Jun 2022 08:56:29 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:290703 Archived-At: --0000000000009f1a1905e0b25382 Content-Type: text/plain; charset="UTF-8" On Sat, Jun 4, 2022, 10:32 AM Stefan Monnier wrote: > >> Performance issues with read access to directories containing less than > >> 10K files seems like something that was solved last century, so > >> I wouldn't worry very much about it. > > Per my response to Eli, I see (network) directories become almost > unusable > > somewhere around 1000 files, > > I don't doubt there are still (in the current century) cases where > largish directories get slow, but what I meant is that it's now > considered as a problem that should be solved by making those > directories fast rather than by avoiding making them so large. > Unfortunately sometimes we have to cope with environment we use. And for all I know some of the performance penalties may be inherent in the (security related) infrastructure requirements in a highly regulated industry. Not that that should be a primary concern for the development team, but it is something a local packager might be stuck with. > >> [ But that doesn't mean we shouldn't try to compile several ELisp files > >> into a single ELN file, especially since the size of ELN files seems > >> to be proportionally larger for small ELisp files than for large > >> ones. ] > > > > Since I learned of the native compiler in 28.1, I decided to try it out > and > > also "throw the spaghetti at the wall" with a bunch of packages that > > provide features similar to those found in more "modern" IDEs. In terms > of > > startup time, the normal package system does not deal well with hundreds > of > > directories on the load path, regardless of AOR native compilation, so > I'm > > tranforming the packages to install in the version-specific load path, > and > > compiling that ahead of time. At least for the ones amenable to such > > treatment. > > There are two load-paths at play (`load-path` and > `native-comp-eln-load-path`) and I'm not sure which one you're taking > about. OT1H `native-comp-eln-load-path` should not grow with the number > of packages so it typically contains exactly 2 entries, and definitely > not hundreds. OTOH `load-path` is unrelated to native compilation. > Not entirely - as I understand it, the load system first finds the source file and computers a hash before determining if there is an ELN file corresponding to it. Although I do wonder if there is some optimization for ELN files in the system directory as opposed to the user's cache. I have one build where I native compiled (but not byte compiled) all the el files in the lisp directory, and another where I byte compiled and then native compiled the same set of files. In both cases I used the flag to batch-native-compile to put the ELN file in the system cache. In the first case a number of files failed to compile, and in the second, they all compiled. I've also observed another situation where a file will only (bye or native) compile if one of its required files has been byte compiled ahead of time - but only native compiling that dependency resulted in the same behavior as not compiling it at all. I planned to send a separate mail to the list asking whether it was intended behavior once I had reduced it to a simple case, or if it should be submitted as a bug. In any case, I noticed that the "browse customization groups" buffer is noticeable faster in the second case. I need to try it again to confirm that it wasn't just waiting on the relevant source files to compile in the first case. I also don't understand what you mean by "version-specific load path". > In the usual unix installation, there will be a "site-lisp" one directory above the version specific installation directory, and another site-lisp in the version-specific installation directory. I'm referring to installing the source (ultimately) in ..../emacs/28.1/site-lisp. During the build it's just in the site-lisp subdirectory of the source root path. > Also, what kind of startup time are you talking about? > E.g., are you using `package-quickstart`? > That was the first alternative I tried. With 1250 packages, it did not work. First, the file consisted of a series of "let" forms corresponding to the package directories, and apparently the autoload forms are ignored if they appear anywhere below top-level. At least I got a number of warnings to that effect. The other problem was that I got a "bytecode overflow error". I only got the first error after chopping off the file approximately after the first 10k lines. Oddly enough, when I put all the files in the site-lisp directory, and collect all the autoloads for that directory in a single file, it has no problem with the 80k line file that results. > > Given I'm compiling all the files AOT for use in a common installation > > (this is on Linux, not Windows), the natural question for me is whether > > larger compilation units would be more efficient, particularly at > startup. > > It all depends where the slowdown comes from :-) > > E.g. `package-quickstart` follows a similar idea to the one you propose > by collecting all the `-autoloads.el` into one bug file, which > saves us from having to load separately all those little files. It also > saves us from having to look for them through those hundreds > of directories. > > I suspect a long `load-path` can itself be a source of slow down > especially during startup, but I haven't bumped into that yet. > There are ways we could speed it up, if needed: > > - create "meta packages" (or just one containing all your packages), > which would bring together in a single directory the files of several > packages (and presumably also bring together their > `-autoloads.el` into a larger combined one). Under GNU/Linux we > could have this metapackage be made of symlinks, making it fairly > efficient an non-obtrusive (e.g. `C-h o` could still get you to the > actual file rather than its metapackage-copy). > - Manage a cache of where are our ELisp files (i.e. a hash table > mapping relative ELisp file names to the absolute file name returned > by looking for them in `load-path`). This way we can usually avoid > scanning those hundred directories to find the .elc file we need, and > go straight to it. > I'm pretty sure the load-path is an issue with 1250 packages, even if half of them consist of single files. Since I'm preparing this for a custom installation that will be accessible for multiple users, I decided to try putting everything in site-lisp and native compile everything AOT. Most of the other potential users are not experienced Unix users, which is why I'm trying to make everything work smoothly up front and have features they would find familiar from other editors. One issue with this approach is that the package selection mechanism doesn't recognize the modules as being installed, or provide any assistance in selectively activating modules. Other places where there is a noticeable slowdown with large numbers of packages: * Browsing customization groups - just unfolding a single group can take minutes (this is on fast server hardware with a lot of free memory) * Browsing custom themes with many theme packages installed I haven't gotten to the point that I can test the same situation by explicitly loading the same modules from the site-lisp directory that had been activated as packages. Installing the themes in the system directory does skip the "suspicious files" check that occurs when loading them from the user configuration. > > I posed the question to the list mostly to see if the approach (or > similar) > > had already been tested for viability or effectiveness, so I can avoid > > unnecessary experimentation if the answer is already well-understood. > > I don't think it has been tried, no. > > > I don't know enough about modern library loading to know whether you'd > > expect N distinct but interdependent dynamic libraries to be loaded in as > > compact a memory region as a single dynamic library formed from the same > > underlying object code. > > I think you're right here, but I'd expect the effect to be fairly small > except when the .elc/.eln files are themselves small. > There are a lot of packages that have fairly small source files, just because they've factored their code the same way it would be in languages where the shared libraries are not in 1-1 correspondence with source files. > > > It's not clear to me whether those points are limited to call > > sites or not. > > I believe it is: the optimization is to replace a call via `Ffuncall` to > a "symbol" (which looks up the value stored in the `symbol-function` > cell), with a direct call to the actual C function contained in the > "subr" object itself (expected to be) contained in the > `symbol-function` cell. > > Andrea would know if there are other semantic-non-preserving > optimizations in the level 3 of the optimizations, but IIUC this is very > much the main one. > > >> IIUC the current native-compiler will actually leave those > >> locally-defined functions in their byte-code form :-( > > That's not what I understood from > > https://akrl.sdf.org/gccemacs.html#org0f21a5b > > As you deduce below, I come from a Scheme background - cl-flet is the > form > > I should have referenced, not let. > > Indeed you're right that those functions can be native compiled, tho only > if > they're closed (i.e. if they don't refer to surrounding lexical > variables). > [ I always forget that little detail :-( ] > I would expect this would apply to most top-level defuns in elisp packages/modules. From my cursory review, it looks like the ability to redefine these defuns is mostly useful when developing the packages themselves, and "sealing" them for use would be appropriate. I'm not clear on whether this optimization is limited to the case of calling functions defined in the compilation unit, or applied more broadly. Thanks, Lynn > --0000000000009f1a1905e0b25382 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sat, Jun 4, 2022, 10:32 AM Stefan Monnier <monnier= @iro.umontreal.ca> wrote:
&g= t;> Performance issues with read access to directories containing less t= han
>> 10K files seems like something that was solved last century, so >> I wouldn't worry very much about it.
> Per my response to Eli, I see (network) directories become almost unus= able
> somewhere around 1000 files,

I don't doubt there are still (in the current century) cases where
largish directories get slow, but what I meant is that it's now
considered as a problem that should be solved by making those
directories fast rather than by avoiding making them so large.
Unfortunately sometimes we have to cope w= ith environment we use.=C2=A0 And for all I know some of the performance pe= nalties may be inherent in the (security related) infrastructure requiremen= ts in a highly regulated industry.
Not that that sho= uld be a primary concern for the development team, but it is something a lo= cal packager might be stuck with.


>> [ But that doesn't mean we shouldn't try to compile severa= l ELisp files
>>=C2=A0 =C2=A0into a single ELN file, especially since the size of E= LN files seems
>>=C2=A0 =C2=A0to be proportionally larger for small ELisp files than= for large
>>=C2=A0 =C2=A0ones.=C2=A0 ]
>
> Since I learned of the native compiler in 28.1, I decided to try it ou= t and
> also "throw the spaghetti at the wall" with a bunch of packa= ges that
> provide features similar to those found in more "modern" IDE= s.=C2=A0 In terms of
> startup time, the normal package system does not deal well with hundre= ds of
> directories on the load path, regardless of AOR native compilation, so= I'm
> tranforming the packages to install in the version-specific load path,= and
> compiling that ahead of time.=C2=A0 At least for the ones amenable to = such
> treatment.

There are two load-paths at play (`load-path` and
`native-comp-eln-load-path`) and I'm not sure which one you're taki= ng
about.=C2=A0 OT1H `native-comp-eln-load-path` should not grow with the numb= er
of packages so it typically contains exactly 2 entries, and definitely
not hundreds.=C2=A0 OTOH `load-path` is unrelated to native compilation.

Not= entirely - as I understand it, the load system first finds the source file= and computers a hash before determining if there is an ELN file correspond= ing to it.
Although I do wonder if there is some opt= imization for ELN files in the system directory as opposed to the user'= s cache.=C2=A0 I have one build where I native compiled (but not byte compi= led) all the el files in the lisp directory, and another where I byte compi= led and then native compiled the same set of files.=C2=A0 In both cases I u= sed the flag to batch-native-compile to put the ELN file in the system cach= e.=C2=A0 In the first case a number of files failed to compile, and in the = second, they all compiled.=C2=A0 I've also observed another situation w= here a file will only (bye or native) compile if one of its required files = has been byte compiled ahead of time - but only native compiling that depen= dency resulted in the same behavior as not compiling it at all.=C2=A0 I pla= nned to send a separate mail to the list asking whether it was intended beh= avior once I had reduced it to a simple case, or if it should be submitted = as a bug.
In any case, I noticed that the "brow= se customization groups" buffer is noticeable faster in the second cas= e.=C2=A0 I need to try it again to confirm that it wasn't just waiting = on the relevant source files to compile in the first case.

I also don't understand what you mean by "version-specific load pa= th".
In the usual unix i= nstallation, there will be a "site-lisp" one directory above the = version specific installation directory, and another site-lisp in the versi= on-specific installation directory.=C2=A0 I'm referring to installing t= he source (ultimately) in ..../emacs/28.1/site-lisp.=C2=A0 During the build= it's just in the site-lisp subdirectory of the source root path.
=


Also, what kind of startup time are you talking about?
E.g., are you using `package-quickstart`?
That was the first alternative I tried.=C2=A0 With 1250 packag= es, it did not work.=C2=A0 First, the file consisted of a series of "l= et" forms corresponding to the package directories, and apparently the= autoload forms are ignored if they appear anywhere below top-level.=C2=A0 = At least I got a number of warnings to that effect.
= The other problem was that I got a "bytecode overflow error".=C2= =A0 I only got the first error after chopping off the file approximately af= ter the first 10k lines.=C2=A0 Oddly enough, when I put all the files in th= e site-lisp directory, and collect all the autoloads for that directory in = a single file, it has no problem with the 80k line file that results.
=


> Given I'm compiling all the files AOT for use in a common installa= tion
> (this is on Linux, not Windows), the natural question for me is whethe= r
> larger compilation units would be more efficient, particularly at star= tup.

It all depends where the slowdown comes from :-)

E.g. `package-quickstart` follows a similar idea to the one you propose
by collecting all the `<pkg>-autoloads.el` into one bug file, which saves us from having to load separately all those little files.=C2=A0 It al= so
saves us from having to look for them through those hundreds
of directories.

I suspect a long `load-path` can itself be a source of slow down
especially during startup, but I haven't bumped into that yet.
There are ways we could speed it up, if needed:

- create "meta packages" (or just one containing all your package= s),
=C2=A0 which would bring together in a single directory the files of severa= l
=C2=A0 packages (and presumably also bring together their
=C2=A0 `<pkg>-autoloads.el` into a larger combined one).=C2=A0 Under = GNU/Linux we
=C2=A0 could have this metapackage be made of symlinks, making it fairly =C2=A0 efficient an non-obtrusive (e.g. `C-h o` could still get you to the<= br> =C2=A0 actual file rather than its metapackage-copy).
- Manage a cache of where are our ELisp files (i.e. a hash table
=C2=A0 mapping relative ELisp file names to the absolute file name returned=
=C2=A0 by looking for them in `load-path`).=C2=A0 This way we can usually a= void
=C2=A0 scanning those hundred directories to find the .elc file we need, an= d
=C2=A0 go straight to it.
I&#= 39;m pretty sure the load-path is an issue with 1250 packages, even if half= of them consist of single files.

Since I'm preparing this for a cust= om installation that will be accessible for multiple users, I decided to tr= y putting everything in site-lisp and native compile everything AOT.=C2=A0 = Most of the other potential users are not experienced Unix users, which is = why I'm trying to make everything work smoothly up front and have featu= res they would find familiar from other editors.=C2=A0=C2=A0

One issue with this approach is that = the package selection mechanism doesn't recognize the modules as being = installed, or provide any assistance in selectively activating modules.=C2= =A0=C2=A0

Other places w= here there is a noticeable slowdown with large numbers of packages:
=C2=A0 * Browsing customization groups - just unfolding a s= ingle group can take minutes (this is on fast server hardware with a lot of= free memory)
=C2=A0 * Browsing custom themes with m= any theme packages installed
I haven't gotten to= the point that I can test the same situation by explicitly loading the sam= e modules from the site-lisp directory that had been activated as packages.= =C2=A0 Installing the themes in the system directory does skip the "su= spicious files" check that occurs when loading them from the user conf= iguration.


> I posed the question to the list mostly to see if the approach (or sim= ilar)
> had already been tested for viability or effectiveness, so I can avoid=
> unnecessary experimentation if the answer is already well-understood.<= br>
I don't think it has been tried, no.

> I don't know enough about modern library loading to know whether y= ou'd
> expect N distinct but interdependent dynamic libraries to be loaded in= as
> compact a memory region as a single dynamic library formed from the sa= me
> underlying object code.

I think you're right here, but I'd expect the effect to be fairly s= mall
except when the .elc/.eln files are themselves small.

There are a lot of pac= kages that have fairly small source files, just because they've factore= d their code the same way it would be in languages where the shared librari= es are not in 1-1 correspondence with source files.
=

> It's not clear to me whether those points are limited to call
> sites or not.

I believe it is: the optimization is to replace a call via `Ffuncall` to a "symbol" (which looks up the value stored in the `symbol-functi= on`
cell), with a direct call to the actual C function contained in the
"subr" object itself (expected to be) contained in the
`symbol-function` cell.

Andrea would know if there are other semantic-non-preserving
optimizations in the level 3 of the optimizations, but IIUC this is very much the main one.

>> IIUC the current native-compiler will actually leave those
>> locally-defined functions in their byte-code form :-(
> That's not what I understood from
> https://akrl.sdf.org/gccemac= s.html#org0f21a5b
> As you deduce below, I come from a Scheme background - cl-flet is the = form
> I should have referenced, not let.

Indeed you're right that those functions can be native compiled, tho on= ly if
they're closed (i.e. if they don't refer to surrounding lexical
variables).
[ I always forget that little detail :-(=C2=A0 ]

I would expect this would a= pply to most top-level defuns in elisp packages/modules.=C2=A0 From my curs= ory review, it looks like the ability to redefine these defuns is mostly us= eful when developing the packages themselves, and "sealing" them = for use would be appropriate.=C2=A0=C2=A0
I'm no= t clear on whether this optimization is limited to the case of calling func= tions defined in the compilation unit, or applied more broadly.

Thanks,
Lynn=


--0000000000009f1a1905e0b25382--