From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: native compilation units Date: Sun, 5 Jun 2022 10:08:35 -0400 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000ddbf1205e0b3e411" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36278"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Andrea Corallo , emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Jun 05 16:18:59 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nxr5k-00095H-QZ for ged-emacs-devel@m.gmane-mx.org; Sun, 05 Jun 2022 16:18:57 +0200 Original-Received: from localhost ([::1]:58616 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nxr5j-0008OE-Rg for ged-emacs-devel@m.gmane-mx.org; Sun, 05 Jun 2022 10:18:55 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:58486) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nxqw0-0007od-RW for emacs-devel@gnu.org; Sun, 05 Jun 2022 10:08:53 -0400 Original-Received: from mail-lf1-x12d.google.com ([2a00:1450:4864:20::12d]:40803) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nxqvx-0008KB-8i for emacs-devel@gnu.org; Sun, 05 Jun 2022 10:08:52 -0400 Original-Received: by mail-lf1-x12d.google.com with SMTP id t25so19622188lfg.7 for ; Sun, 05 Jun 2022 07:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=F0PkwUBREw2cguP+6poXjAf5hqU4BujMUwaSgoTrS88=; b=QSiyAkjl0gewtqKtPWbNpBMt6jtZZWAwia1j4YBZonSeXAnBibF6hsA7eOIIq820AE 8BOkBuNkXd0b516Jbc/H4pNCklcbT5/T6RP6Iu6u/wyqF6OILvLYtNlE8s9op5BgU4AN Jp7b1qrnwhk5uQh2W1laQs11/ozzxzhmDlXuyH5KmSiho0ri8YoYhBQVjWzfybQeLRgT Cokp1rtUoIpSLmvkfe+7b3AbI6a7VNnXeV+ZLeoTxVLGTZPFqkAP31hbYMGS40vokq/n g4WpHfXfFiy6lgpWb196/fQfYD+wtygp5W4rpRL9+pfHJNFC8wn7MQsJ3WEQDBKCscWQ dI9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=F0PkwUBREw2cguP+6poXjAf5hqU4BujMUwaSgoTrS88=; b=PJVncvKftr54THyTFpN38dTUfJbCCk1WjUSTuOCn0isb8N3ahNFFG3Mrefc8m19RLj bt1o/v8ttyj0vh1jEnp5Z7B0HzfFn+xS98yE3YfvJIHnfbLvzdQVigB08/ohByHF0wjw MB4LhKkH5hYRfMfxakTjuGDzYDHipQdZLlUjbXXmbUkh2GnkVIUxeuuNJdX9NgRgXl1U NO4Z9M+l6RvCff1irM04+3K2iyAzsHBx6j4QYY3i9Sott7DRzOdJvUxzPuoyBu40CGXv fKWxbH651o/h6zBr1JzNLVTetXAfnVLlRbZCzD6zdvxAV6ZnhgxVE0OaGeJEZkXtvMhN wj1A== X-Gm-Message-State: AOAM533RSegi2wO7Io4Kz/zjiXqFjf//y42n275S8djwSD5ok0YYt6i+ TUX8vi6flI5ACRBOLczNZoSdW7U/xBspHosPXoE= X-Google-Smtp-Source: ABdhPJxc1cVADwqpw7vUo5vxuExj6moFyJazf8h4idO+RrxYHbA7rdNEOqEq5z34mxaIj+MNQtSJaOINMnkjn+03RM0= X-Received: by 2002:a05:6512:1041:b0:478:afc6:5846 with SMTP id c1-20020a056512104100b00478afc65846mr12412171lfb.132.1654438127057; Sun, 05 Jun 2022 07:08:47 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2a00:1450:4864:20::12d; envelope-from=owinebar@gmail.com; helo=mail-lf1-x12d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sun, 05 Jun 2022 10:17:09 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:290715 Archived-At: --000000000000ddbf1205e0b3e411 Content-Type: text/plain; charset="UTF-8" On Sun, Jun 5, 2022 at 8:16 AM Lynn Winebarger wrote: > On Sat, Jun 4, 2022, 10:32 AM Stefan Monnier > wrote: > >> >> [ But that doesn't mean we shouldn't try to compile several ELisp files >> > >> into a single ELN file, especially since the size of ELN files seems >> >> to be proportionally larger for small ELisp files than for large >> >> ones. ] >> > >> > Since I learned of the native compiler in 28.1, I decided to try it out >> and >> > also "throw the spaghetti at the wall" with a bunch of packages that >> > provide features similar to those found in more "modern" IDEs. In >> terms of >> > startup time, the normal package system does not deal well with >> hundreds of >> > directories on the load path, regardless of AOR native compilation, so >> I'm >> > tranforming the packages to install in the version-specific load path, >> and >> > compiling that ahead of time. At least for the ones amenable to such >> > treatment. >> >> There are two load-paths at play (`load-path` and >> `native-comp-eln-load-path`) and I'm not sure which one you're taking >> about. OT1H `native-comp-eln-load-path` should not grow with the number >> of packages so it typically contains exactly 2 entries, and definitely >> not hundreds. OTOH `load-path` is unrelated to native compilation. >> > > Not entirely - as I understand it, the load system first finds the source > file and computers a hash before determining if there is an ELN file > corresponding to it. > Although I do wonder if there is some optimization for ELN files in the > system directory as opposed to the user's cache. I have one build where I > native compiled (but not byte compiled) all the el files in the lisp > directory, and another where I byte compiled and then native compiled the > same set of files. In both cases I used the flag to batch-native-compile > to put the ELN file in the system cache. In the first case a number of > files failed to compile, and in the second, they all compiled. I've also > observed another situation where a file will only (bye or native) compile > if one of its required files has been byte compiled ahead of time - but > only native compiling that dependency resulted in the same behavior as not > compiling it at all. I planned to send a separate mail to the list asking > whether it was intended behavior once I had reduced it to a simple case, or > if it should be submitted as a bug. > Unrelated, but the one type of file I don't seem to be able to produce AOT (because I have no way to specify them) in the system directory are the subr/trampoline files. Any hints on how to make those AOT in the system directory? > >> Also, what kind of startup time are you talking about? >> E.g., are you using `package-quickstart`? >> > That was the first alternative I tried. With 1250 packages, it did not > work. First, the file consisted of a series of "let" forms corresponding > to the package directories, and apparently the autoload forms are ignored > if they appear anywhere below top-level. At least I got a number of > warnings to that effect. > The other problem was that I got a "bytecode overflow error". I only got > the first error after chopping off the file approximately after the first > 10k lines. Oddly enough, when I put all the files in the site-lisp > directory, and collect all the autoloads for that directory in a single > file, it has no problem with the 80k line file that results. > >> >> Also, I should have responded to the first question - "minutes" on recent server-grade hardware with 24 cores and >100GB of RAM. That was with 1193 enabled packages in my .emacs file. On Sun, Jun 5, 2022 at 8:16 AM Lynn Winebarger wrote: > On Sat, Jun 4, 2022, 10:32 AM Stefan Monnier > wrote: > >> >> Performance issues with read access to directories containing less than >> >> 10K files seems like something that was solved last century, so >> >> I wouldn't worry very much about it. >> > Per my response to Eli, I see (network) directories become almost >> unusable >> > somewhere around 1000 files, >> >> I don't doubt there are still (in the current century) cases where >> largish directories get slow, but what I meant is that it's now >> considered as a problem that should be solved by making those >> directories fast rather than by avoiding making them so large. >> > Unfortunately sometimes we have to cope with environment we use. And for > all I know some of the performance penalties may be inherent in the > (security related) infrastructure requirements in a highly regulated > industry. > Not that that should be a primary concern for the development team, but it > is something a local packager might be stuck with. > > >> >> [ But that doesn't mean we shouldn't try to compile several ELisp files >> >> into a single ELN file, especially since the size of ELN files seems >> >> to be proportionally larger for small ELisp files than for large >> >> ones. ] >> > >> > Since I learned of the native compiler in 28.1, I decided to try it out >> and >> > also "throw the spaghetti at the wall" with a bunch of packages that >> > provide features similar to those found in more "modern" IDEs. In >> terms of >> > startup time, the normal package system does not deal well with >> hundreds of >> > directories on the load path, regardless of AOR native compilation, so >> I'm >> > tranforming the packages to install in the version-specific load path, >> and >> > compiling that ahead of time. At least for the ones amenable to such >> > treatment. >> >> There are two load-paths at play (`load-path` and >> `native-comp-eln-load-path`) and I'm not sure which one you're taking >> about. OT1H `native-comp-eln-load-path` should not grow with the number >> of packages so it typically contains exactly 2 entries, and definitely >> not hundreds. OTOH `load-path` is unrelated to native compilation. >> > > Not entirely - as I understand it, the load system first finds the source > file and computers a hash before determining if there is an ELN file > corresponding to it. > Although I do wonder if there is some optimization for ELN files in the > system directory as opposed to the user's cache. I have one build where I > native compiled (but not byte compiled) all the el files in the lisp > directory, and another where I byte compiled and then native compiled the > same set of files. In both cases I used the flag to batch-native-compile > to put the ELN file in the system cache. In the first case a number of > files failed to compile, and in the second, they all compiled. I've also > observed another situation where a file will only (bye or native) compile > if one of its required files has been byte compiled ahead of time - but > only native compiling that dependency resulted in the same behavior as not > compiling it at all. I planned to send a separate mail to the list asking > whether it was intended behavior once I had reduced it to a simple case, or > if it should be submitted as a bug. > In any case, I noticed that the "browse customization groups" buffer is > noticeable faster in the second case. I need to try it again to confirm > that it wasn't just waiting on the relevant source files to compile in the > first case. > > I also don't understand what you mean by "version-specific load path". >> > In the usual unix installation, there will be a "site-lisp" one directory > above the version specific installation directory, and another site-lisp in > the version-specific installation directory. I'm referring to installing > the source (ultimately) in ..../emacs/28.1/site-lisp. During the build > it's just in the site-lisp subdirectory of the source root path. > > >> Also, what kind of startup time are you talking about? >> E.g., are you using `package-quickstart`? >> > That was the first alternative I tried. With 1250 packages, it did not > work. First, the file consisted of a series of "let" forms corresponding > to the package directories, and apparently the autoload forms are ignored > if they appear anywhere below top-level. At least I got a number of > warnings to that effect. > The other problem was that I got a "bytecode overflow error". I only got > the first error after chopping off the file approximately after the first > 10k lines. Oddly enough, when I put all the files in the site-lisp > directory, and collect all the autoloads for that directory in a single > file, it has no problem with the 80k line file that results. > > >> > Given I'm compiling all the files AOT for use in a common installation >> > (this is on Linux, not Windows), the natural question for me is whether >> > larger compilation units would be more efficient, particularly at >> startup. >> >> It all depends where the slowdown comes from :-) >> >> E.g. `package-quickstart` follows a similar idea to the one you propose >> by collecting all the `-autoloads.el` into one bug file, which >> saves us from having to load separately all those little files. It also >> saves us from having to look for them through those hundreds >> of directories. >> >> I suspect a long `load-path` can itself be a source of slow down >> especially during startup, but I haven't bumped into that yet. >> There are ways we could speed it up, if needed: >> >> - create "meta packages" (or just one containing all your packages), >> which would bring together in a single directory the files of several >> packages (and presumably also bring together their >> `-autoloads.el` into a larger combined one). Under GNU/Linux we >> could have this metapackage be made of symlinks, making it fairly >> efficient an non-obtrusive (e.g. `C-h o` could still get you to the >> actual file rather than its metapackage-copy). >> - Manage a cache of where are our ELisp files (i.e. a hash table >> mapping relative ELisp file names to the absolute file name returned >> by looking for them in `load-path`). This way we can usually avoid >> scanning those hundred directories to find the .elc file we need, and >> go straight to it. >> > I'm pretty sure the load-path is an issue with 1250 packages, even if half > of them consist of single files. > > Since I'm preparing this for a custom installation that will be accessible > for multiple users, I decided to try putting everything in site-lisp and > native compile everything AOT. Most of the other potential users are not > experienced Unix users, which is why I'm trying to make everything work > smoothly up front and have features they would find familiar from other > editors. > > One issue with this approach is that the package selection mechanism > doesn't recognize the modules as being installed, or provide any assistance > in selectively activating modules. > > Other places where there is a noticeable slowdown with large numbers of > packages: > * Browsing customization groups - just unfolding a single group can take > minutes (this is on fast server hardware with a lot of free memory) > * Browsing custom themes with many theme packages installed > I haven't gotten to the point that I can test the same situation by > explicitly loading the same modules from the site-lisp directory that had > been activated as packages. Installing the themes in the system directory > does skip the "suspicious files" check that occurs when loading them from > the user configuration. > > >> > I posed the question to the list mostly to see if the approach (or >> similar) >> > had already been tested for viability or effectiveness, so I can avoid >> > unnecessary experimentation if the answer is already well-understood. >> >> I don't think it has been tried, no. >> >> > I don't know enough about modern library loading to know whether you'd >> > expect N distinct but interdependent dynamic libraries to be loaded in >> as >> > compact a memory region as a single dynamic library formed from the same >> > underlying object code. >> >> I think you're right here, but I'd expect the effect to be fairly small >> except when the .elc/.eln files are themselves small. >> > > There are a lot of packages that have fairly small source files, just > because they've factored their code the same way it would be in languages > where the shared libraries are not in 1-1 correspondence with source files. > >> >> > It's not clear to me whether those points are limited to call >> > sites or not. >> >> I believe it is: the optimization is to replace a call via `Ffuncall` to >> a "symbol" (which looks up the value stored in the `symbol-function` >> cell), with a direct call to the actual C function contained in the >> "subr" object itself (expected to be) contained in the >> `symbol-function` cell. >> >> Andrea would know if there are other semantic-non-preserving >> optimizations in the level 3 of the optimizations, but IIUC this is very >> much the main one. >> >> >> IIUC the current native-compiler will actually leave those >> >> locally-defined functions in their byte-code form :-( >> > That's not what I understood from >> > https://akrl.sdf.org/gccemacs.html#org0f21a5b >> > As you deduce below, I come from a Scheme background - cl-flet is the >> form >> > I should have referenced, not let. >> >> Indeed you're right that those functions can be native compiled, tho only >> if >> they're closed (i.e. if they don't refer to surrounding lexical >> variables). >> [ I always forget that little detail :-( ] >> > > I would expect this would apply to most top-level defuns in elisp > packages/modules. From my cursory review, it looks like the ability to > redefine these defuns is mostly useful when developing the packages > themselves, and "sealing" them for use would be appropriate. > I'm not clear on whether this optimization is limited to the case of > calling functions defined in the compilation unit, or applied more broadly. > > Thanks, > Lynn > > >> --000000000000ddbf1205e0b3e411 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Jun 5, 2022 at 8:16 AM Lynn W= inebarger <owinebar@gmail.com&= gt; wrote:
On Sat, Jun 4, 2022, 10:32 AM Stefan Monnier <monnier@iro.= umontreal.ca> wrote:
>> [ But that doesn't mean we shouldn't try to co= mpile several ELisp files
>>=C2=A0 =C2=A0into a single ELN file, especially since the size of E= LN files seems
>>=C2=A0 =C2=A0to be proportionally larger for small ELisp files than= for large
>>=C2=A0 =C2=A0ones.=C2=A0 ]
>
> Since I learned of the native compiler in 28.1, I decided to try it ou= t and
> also "throw the spaghetti at the wall" with a bunch of packa= ges that
> provide features similar to those found in more "modern" IDE= s.=C2=A0 In terms of
> startup time, the normal package system does not deal well with hundre= ds of
> directories on the load path, regardless of AOR native compilation, so= I'm
> tranforming the packages to install in the version-specific load path,= and
> compiling that ahead of time.=C2=A0 At least for the ones amenable to = such
> treatment.

There are two load-paths at play (`load-path` and
`native-comp-eln-load-path`) and I'm not sure which one you're taki= ng
about.=C2=A0 OT1H `native-comp-eln-load-path` should not grow with the numb= er
of packages so it typically contains exactly 2 entries, and definitely
not hundreds.=C2=A0 OTOH `load-path` is unrelated to native compilation.

Not= entirely - as I understand it, the load system first finds the source file= and computers a hash before determining if there is an ELN file correspond= ing to it.
Although I do wonder if there is some opt= imization for ELN files in the system directory as opposed to the user'= s cache.=C2=A0 I have one build where I native compiled (but not byte compi= led) all the el files in the lisp directory, and another where I byte compi= led and then native compiled the same set of files.=C2=A0 In both cases I u= sed the flag to batch-native-compile to put the ELN file in the system cach= e.=C2=A0 In the first case a number of files failed to compile, and in the = second, they all compiled.=C2=A0 I've also observed another situation w= here a file will only (bye or native) compile if one of its required files = has been byte compiled ahead of time - but only native compiling that depen= dency resulted in the same behavior as not compiling it at all.=C2=A0 I pla= nned to send a separate mail to the list asking whether it was intended beh= avior once I had reduced it to a simple case, or if it should be submitted = as a bug.

Unrelated, but the on= e type of file I don't seem to be able to produce AOT (because I have n= o way to specify them) in the system directory are the subr/trampoline file= s.=C2=A0 Any hints on how to make those AOT in the system directory?=C2=A0<= /div>



Also, what kind of startup time are you talking about?
E.g., are you using `package-quickstart`?
That was the first alternative I tried.=C2=A0 With 1250 packag= es, it did not work.=C2=A0 First, the file consisted of a series of "l= et" forms corresponding to the package directories, and apparently the= autoload forms are ignored if they appear anywhere below top-level.=C2=A0 = At least I got a number of warnings to that effect.
= The other problem was that I got a "bytecode overflow error".=C2= =A0 I only got the first error after chopping off the file approximately af= ter the first 10k lines.=C2=A0 Oddly enough, when I put all the files in th= e site-lisp directory, and collect all the autoloads for that directory in = a single file, it has no problem with the 80k line file that results.
=

Al= so, I should have responded to the first question - "minutes" on = recent server-grade hardware with 24 cores and >100GB of RAM.=C2=A0 That= was with 1193 enabled packages in my .emacs file.


=C2=A0=

On Sun, Jun 5, 2022 at 8:16 AM Lynn Winebarger <owinebar@gmail.com> wrote:
On Sat, Jun 4, 202= 2, 10:32 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote= :
>> Perfo= rmance issues with read access to directories containing less than
>> 10K files seems like something that was solved last century, so >> I wouldn't worry very much about it.
> Per my response to Eli, I see (network) directories become almost unus= able
> somewhere around 1000 files,

I don't doubt there are still (in the current century) cases where
largish directories get slow, but what I meant is that it's now
considered as a problem that should be solved by making those
directories fast rather than by avoiding making them so large.
Unfortunately sometimes we have to cope w= ith environment we use.=C2=A0 And for all I know some of the performance pe= nalties may be inherent in the (security related) infrastructure requiremen= ts in a highly regulated industry.
Not that that sho= uld be a primary concern for the development team, but it is something a lo= cal packager might be stuck with.


>> [ But that doesn't mean we shouldn't try to compile severa= l ELisp files
>>=C2=A0 =C2=A0into a single ELN file, especially since the size of E= LN files seems
>>=C2=A0 =C2=A0to be proportionally larger for small ELisp files than= for large
>>=C2=A0 =C2=A0ones.=C2=A0 ]
>
> Since I learned of the native compiler in 28.1, I decided to try it ou= t and
> also "throw the spaghetti at the wall" with a bunch of packa= ges that
> provide features similar to those found in more "modern" IDE= s.=C2=A0 In terms of
> startup time, the normal package system does not deal well with hundre= ds of
> directories on the load path, regardless of AOR native compilation, so= I'm
> tranforming the packages to install in the version-specific load path,= and
> compiling that ahead of time.=C2=A0 At least for the ones amenable to = such
> treatment.

There are two load-paths at play (`load-path` and
`native-comp-eln-load-path`) and I'm not sure which one you're taki= ng
about.=C2=A0 OT1H `native-comp-eln-load-path` should not grow with the numb= er
of packages so it typically contains exactly 2 entries, and definitely
not hundreds.=C2=A0 OTOH `load-path` is unrelated to native compilation.

Not= entirely - as I understand it, the load system first finds the source file= and computers a hash before determining if there is an ELN file correspond= ing to it.
Although I do wonder if there is some opt= imization for ELN files in the system directory as opposed to the user'= s cache.=C2=A0 I have one build where I native compiled (but not byte compi= led) all the el files in the lisp directory, and another where I byte compi= led and then native compiled the same set of files.=C2=A0 In both cases I u= sed the flag to batch-native-compile to put the ELN file in the system cach= e.=C2=A0 In the first case a number of files failed to compile, and in the = second, they all compiled.=C2=A0 I've also observed another situation w= here a file will only (bye or native) compile if one of its required files = has been byte compiled ahead of time - but only native compiling that depen= dency resulted in the same behavior as not compiling it at all.=C2=A0 I pla= nned to send a separate mail to the list asking whether it was intended beh= avior once I had reduced it to a simple case, or if it should be submitted = as a bug.
In any case, I noticed that the "brow= se customization groups" buffer is noticeable faster in the second cas= e.=C2=A0 I need to try it again to confirm that it wasn't just waiting = on the relevant source files to compile in the first case.

I also don't understand what you mean by "version-specific load pa= th".
In the usual unix i= nstallation, there will be a "site-lisp" one directory above the = version specific installation directory, and another site-lisp in the versi= on-specific installation directory.=C2=A0 I'm referring to installing t= he source (ultimately) in ..../emacs/28.1/site-lisp.=C2=A0 During the build= it's just in the site-lisp subdirectory of the source root path.
=


Also, what kind of startup time are you talking about?
E.g., are you using `package-quickstart`?
That was the first alternative I tried.=C2=A0 With 1250 packag= es, it did not work.=C2=A0 First, the file consisted of a series of "l= et" forms corresponding to the package directories, and apparently the= autoload forms are ignored if they appear anywhere below top-level.=C2=A0 = At least I got a number of warnings to that effect.
= The other problem was that I got a "bytecode overflow error".=C2= =A0 I only got the first error after chopping off the file approximately af= ter the first 10k lines.=C2=A0 Oddly enough, when I put all the files in th= e site-lisp directory, and collect all the autoloads for that directory in = a single file, it has no problem with the 80k line file that results.
=


> Given I'm compiling all the files AOT for use in a common installa= tion
> (this is on Linux, not Windows), the natural question for me is whethe= r
> larger compilation units would be more efficient, particularly at star= tup.

It all depends where the slowdown comes from :-)

E.g. `package-quickstart` follows a similar idea to the one you propose
by collecting all the `<pkg>-autoloads.el` into one bug file, which saves us from having to load separately all those little files.=C2=A0 It al= so
saves us from having to look for them through those hundreds
of directories.

I suspect a long `load-path` can itself be a source of slow down
especially during startup, but I haven't bumped into that yet.
There are ways we could speed it up, if needed:

- create "meta packages" (or just one containing all your package= s),
=C2=A0 which would bring together in a single directory the files of severa= l
=C2=A0 packages (and presumably also bring together their
=C2=A0 `<pkg>-autoloads.el` into a larger combined one).=C2=A0 Under = GNU/Linux we
=C2=A0 could have this metapackage be made of symlinks, making it fairly =C2=A0 efficient an non-obtrusive (e.g. `C-h o` could still get you to the<= br> =C2=A0 actual file rather than its metapackage-copy).
- Manage a cache of where are our ELisp files (i.e. a hash table
=C2=A0 mapping relative ELisp file names to the absolute file name returned=
=C2=A0 by looking for them in `load-path`).=C2=A0 This way we can usually a= void
=C2=A0 scanning those hundred directories to find the .elc file we need, an= d
=C2=A0 go straight to it.
I&#= 39;m pretty sure the load-path is an issue with 1250 packages, even if half= of them consist of single files.

Since I'm prepar= ing this for a custom installation that will be accessible for multiple use= rs, I decided to try putting everything in site-lisp and native compile eve= rything AOT.=C2=A0 Most of the other potential users are not experienced Un= ix users, which is why I'm trying to make everything work smoothly up f= ront and have features they would find familiar from other editors.=C2=A0= =C2=A0

One issue with th= is approach is that the package selection mechanism doesn't recognize t= he modules as being installed, or provide any assistance in selectively act= ivating modules.=C2=A0=C2=A0

Other places where there is a noticeable slowdown with large numbers o= f packages:
=C2=A0 * Browsing customization groups -= just unfolding a single group can take minutes (this is on fast server har= dware with a lot of free memory)
=C2=A0 * Browsing c= ustom themes with many theme packages installed
I ha= ven't gotten to the point that I can test the same situation by explici= tly loading the same modules from the site-lisp directory that had been act= ivated as packages.=C2=A0 Installing the themes in the system directory doe= s skip the "suspicious files" check that occurs when loading them= from the user configuration.


> I posed the question to the list mostly to see if the approach (or sim= ilar)
> had already been tested for viability or effectiveness, so I can avoid=
> unnecessary experimentation if the answer is already well-understood.<= br>
I don't think it has been tried, no.

> I don't know enough about modern library loading to know whether y= ou'd
> expect N distinct but interdependent dynamic libraries to be loaded in= as
> compact a memory region as a single dynamic library formed from the sa= me
> underlying object code.

I think you're right here, but I'd expect the effect to be fairly s= mall
except when the .elc/.eln files are themselves small.

There are a lot of pac= kages that have fairly small source files, just because they've factore= d their code the same way it would be in languages where the shared librari= es are not in 1-1 correspondence with source files.
=
https://akrl.sdf.org/gccemac= s.html#org0f21a5b
> As you deduce below, I come from a Scheme background - cl-flet is the = form
> I should have referenced, not let.

Indeed you're right that those functions can be native compiled, tho on= ly if
they're closed (i.e. if they don't refer to surrounding lexical
variables).
[ I always forget that little detail :-(=C2=A0 ]

I would expect this would a= pply to most top-level defuns in elisp packages/modules.=C2=A0 From my curs= ory review, it looks like the ability to redefine these defuns is mostly us= eful when developing the packages themselves, and "sealing" them = for use would be appropriate.=C2=A0=C2=A0
I'm no= t clear on whether this optimization is limited to the case of calling func= tions defined in the compilation unit, or applied more broadly.

Thanks,
Lynn=


--000000000000ddbf1205e0b3e411--