bug#45858: 28.0.50; [native-comp] compilation very slow

bug#45858: 28.0.50; [native-comp] compilation very slow

[Óscar Fuentes]

After starting a new instance of Emacs with an empty eln-cache,
producing .eln files takes a long time. Any typical file takes more than
five minutes, with some files (like lsp-protocol.el, right now)
approaching two hours. Memory consumption is reasonable (155 MB for the
process that is compiling lsp-protocol.el, about 60 MB for a typical
compilation process).

Right now, after 9 hours, there are 272 .eln files in the cache.

The machine is a high-end desktop from 2015 and has dozens of GB of free
memory and remains perfectly responsible, this is not the problem about
excessive memory use that was reported previously. There is no other
significant CPU activity.

There are no nativecomp-related customizations on my config.


In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0)
 of 2021-01-13 built on sky
Repository revision: 79b9a262ffab37296a39c2d69cdabae153db10a7
Repository branch: feature/native-comp
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure --without-toolkit-scroll-bars --with-x-toolkit=lucid
 --with-modules --without-imagemagick --with-nativecomp'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND DBUS GSETTINGS GLIB NOTIFY
INOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB LUCID
X11 XDBE XIM MODULES NATIVE_COMP THREADS JSON PDUMPER

Important settings:
  value of $LANG: C
  locale-coding-system: nil

Major mode: Group

Minor modes in effect:
  cursor-sensor-mode: t
  gnus-undo-mode: t
  dap-tooltip-mode: t
  dap-ui-many-windows-mode: t
  dap-ui-controls-mode: t
  dap-ui-mode: t
  treemacs-filewatch-mode: t
  treemacs-follow-mode: t
  treemacs-git-mode: deferred
  treemacs-fringe-indicator-mode: t
  dap-auto-configure-mode: t
  dap-mode: t
  window-highlight-mode: t
  global-git-commit-mode: t
  evil-owl-mode: t
  buffer-flip-mode: t
  evil-leader-mode: t
  shell-dirtrack-mode: t
  evil-mode: t
  evil-local-mode: t
  pdf-occur-global-minor-mode: t
  TeX-PDF-mode: t
  show-paren-mode: t
  ido-grid-mode: t
  flx-ido-mode: t
  ido-hacks-mode: t
  ido-everywhere: t
  which-key-mode: t
  global-anzu-mode: t
  anzu-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t

Load-path shadows:
/home/oscar/elisp/singles/avy hides /home/oscar/.emacs.d/elpa/avy-20201226.1734/avy
/home/oscar/elisp/singles/s hides /home/oscar/.emacs.d/elpa/s-20180406.808/s
/home/oscar/elisp/singles/dash hides /home/oscar/.emacs.d/elpa/dash-20210111.1247/dash

Features:
(shadow emacsbug flow-fill mm-archive cl-print help-fns radix-tree
misearch multi-isearch ol-eww eww xdg url-queue mm-url ol-rmail ol-mhe
ol-irc ol-info ol-gnus nnselect gnus-search eieio-opt speedbar ezimage
dframe ol-docview doc-view ol-bibtex bibtex ol-bbdb ol-w3m qp sort
smiley gnus-cite mail-extr gnus-async gnus-bcklg gnus-ml cursor-sensor
pop3 nndraft nnmh nnml nnfolder gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime
smime dig nntp gnus-cache gnus-sum shr kinsoku svg gnus-group gnus-undo
smtpmail sendmail gnus-start gnus-dbus gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo parse-time iso8601 gnus-spec gnus-int
gnus-range gnus-win gnus nnheader lsp-diagnostics company-oddmuse
company-keywords company-etags etags fileloop company-gtags
company-dabbrev-code company-dabbrev company-files company-clang
company-capf company-cmake company-semantic company-template
company-bbdb company lsp-headerline lsp-icons lsp-modeline dap-mouse
dap-ui gdb-mi gud bui bui-list bui-info bui-entry bui-core bui-history
bui-button bui-utils lsp-ui lsp-ui-flycheck lsp-ui-doc goto-addr
lsp-ui-imenu lsp-ui-peek lsp-ui-sideline flycheck view pkg-info url-http
url-auth url-gw epl lsp-steep lsp-svelte lsp-sqls lsp-yaml lsp-xml
lsp-vimscript lsp-vhdl lsp-vetur lsp-html lsp-verilog lsp-vala
lsp-terraform lsp-tex lsp-sorbet lsp-solargraph lsp-rust lsp-rf lsp-r
lsp-purescript lsp-pyls lsp-pwsh lsp-php lsp-perl lsp-ocaml lsp-nix
lsp-nim lsp-lua lsp-kotlin lsp-json lsp-javascript lsp-haxe lsp-groovy
lsp-hack lsp-go lsp-completion lsp-gdscript lsp-fsharp lsp-fortran
lsp-eslint lsp-erlang lsp-elixir lsp-elm lsp-dockerfile lsp-dhall
lsp-css lsp-csharp gnutls lsp-crystal lsp-cmake lsp-clojure lsp-clangd
dom lsp-bash lsp-angular lsp-ada lsp-dart lsp-dart-commands
lsp-dart-flutter-widget-guide lsp-dart-flutter-fringe-colors
lsp-dart-flutter-colors lsp-dart-outline lsp-dart-code-lens lsp-lens
lsp-dart-test-tree lsp-treemacs lsp-treemacs-themes treemacs
treemacs-header-line treemacs-compatibility treemacs-mode
treemacs-bookmarks treemacs-interface treemacs-extensions
treemacs-mouse-interface treemacs-tags treemacs-persistence
treemacs-filewatch-mode treemacs-follow-mode treemacs-rendering
treemacs-async treemacs-workspaces treemacs-dom treemacs-visuals
treemacs-fringe-indicator treemacs-scope pulse treemacs-faces
treemacs-icons treemacs-themes treemacs-core-utils pfuture hl-line
treemacs-logging treemacs-customization treemacs-macros
lsp-dart-test-output lsp-dart-test-support lsp-dart-dap
lsp-dart-devtools lsp-dart-flutter-daemon dap-utils dap-mode dap-launch
posframe dap-overlays lsp-dart-closing-labels lsp-dart-utils
lsp-dart-protocol lsp-mode lsp-protocol xref project spinner
network-stream nsm markdown-mode lv inline ht f ewoc dash-functional
bindat elec-pair dart-mode window-highlight color face-remap nord-theme
lp0-mode aggressive-indent lisp-mnt deft org-noter org-element avl-tree
generator org-protocol org-capture org-refile org-crypt orgit org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src
ob-comint org-pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp ob-core ob-eval org-table ol org-keys
org-compat org-macs org-loaddefs magit-bookmark magit-imenu git-rebase
magit-extras magit-gitignore magit-ediff ediff magit-subtree magit-patch
magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
diff magit-core magit-autorevert autorevert filenotify magit-margin
magit-transient magit-process magit-mode git-commit recentf tree-widget
transient magit-git magit-section benchmark magit-utils which-func
vc-git with-editor term ehelp eshell esh-cmd esh-ext esh-opt esh-proc
esh-io esh-arg esh-module esh-groups esh-util evil-owl buffer-flip
key-chord evil-leader evil-paredit evil-anzu evil evil-keybindings
evil-integration evil-maps evil-commands ffap reveal flyspell ispell
evil-jumps evil-command-window evil-types evil-search evil-ex shell
pcomplete evil-macros evil-repeat evil-states evil-core evil-common
windmove rect evil-digraphs evil-vars wgrep grep ag vc-svn find-dired s
dash pdf-occur ibuf-ext ibuffer ibuffer-loaddefs tablist tablist-filter
semantic/wisent/comp semantic/wisent semantic/wisent/wisent
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw mode-local find-func cedet pdf-isearch let-alist pdf-misc
imenu pdf-tools cus-edit cus-start cus-load pdf-view bookmark jka-compr
pdf-cache pdf-info tq pdf-util format-spec image-mode exif preview
reporter desktop frameset tex-buf latex latex-flymake flymake-proc
flymake compile comint ansi-color tex-ispell tex-style tex dbus xml crm
texmathp paren cmake-mode paredit-menu paredit server yasnippet psvn
wid-edit log-edit message rmc puny rfc822 mml mml-sec epa epg epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log diff-mode pp elp ediff-merg ediff-mult ediff-wind
ediff-diff ediff-help ediff-init ediff-util dired dired-loaddefs
display-fill-column-indicator ido-grid-mode flx-ido flx ido-hacks ido
goto-chg edmacro kmacro avy ring ws-butler which-key diminish cl iscroll
anzu easy-mmode thingatpt solar cal-dst cal-menu calendar cal-loaddefs
finder-inf advice disp-table tex-site info package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap url-handlers url-parse auth-source eieio
eieio-core eieio-loaddefs password-cache json map url-vars comp derived
comp-cstr warnings subr-x rx cl-seq cl-macs cl-extra help-mode easymenu
seq byte-opt gv cl-loaddefs cl-lib bytecomp byte-compile cconv
iso-transl tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face pcase macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
x-toolkit x multi-tty make-network-process nativecomp emacs)

Memory information:
((conses 16 1700382 1332106)
 (symbols 48 66229 3)
 (strings 32 245718 141658)
 (string-bytes 1 8290552)
 (vectors 16 251783)
 (vector-slots 8 6718813 929848)
 (floats 8 1600 1786)
 (intervals 56 4199 1262)
 (buffers 984 33))

bug#45858: 28.0.50; [native-comp] compilation very slow

[akrl--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Óscar Fuentes <ofv@wanadoo.es> writes:

> After starting a new instance of Emacs with an empty eln-cache,
> producing .eln files takes a long time. Any typical file takes more than
> five minutes, with some files (like lsp-protocol.el, right now)
> approaching two hours. Memory consumption is reasonable (155 MB for the
> process that is compiling lsp-protocol.el, about 60 MB for a typical
> compilation process).
>
> Right now, after 9 hours, there are 272 .eln files in the cache.
>
> The machine is a high-end desktop from 2015 and has dozens of GB of free
> memory and remains perfectly responsible, this is not the problem about
> excessive memory use that was reported previously. There is no other
> significant CPU activity.

This is behaviour is unexpected, but is kind of hard if not impossible
to guess the issue from here.

I guess both Emacs and and libgccjit are not debug build correct?

  Andrea

bug#45858: 28.0.50; [native-comp] compilation very slow

[Óscar Fuentes]

Andrea Corallo <akrl@sdf.org> writes:

> This is behaviour is unexpected, but is kind of hard if not impossible
> to guess the issue from here.
>
> I guess both Emacs and and libgccjit are not debug build correct?

The build was performed with

../emacs/configure --without-toolkit-scroll-bars --with-x-toolkit=lucid --with-modules --without-imagemagick --with-nativecomp && make -j8

libgccjit was installed from Debian's package (gcc 10.2.1-6):

lrwxrwxrwx 1 root root       40 ene 10 12:35 /usr/lib/gcc/x86_64-linux-gnu/10/libgccjit.so -> ../../../x86_64-linux-gnu/libgccjit.so.0
lrwxrwxrwx 1 root root       18 ene 10 12:35 /usr/lib/x86_64-linux-gnu/libgccjit.so.0 -> libgccjit.so.0.0.1
-rw-r--r-- 1 root root 26103760 ene 10 12:35 /usr/lib/x86_64-linux-gnu/libgccjit.so.0.0.1

Even supposing that libgccjit was built by mistake on debug mode, I
guess that it would not take so much time to compile an .eln.

I'll try building gccemacs from a pristine checkout on a few days. Let's
see if the problem persists.

bug#45858: 28.0.50; [native-comp] compilation very slow

[Óscar Fuentes]

Óscar Fuentes <ofv@wanadoo.es> writes:

> I'll try building gccemacs from a pristine checkout on a few days. Let's
> see if the problem persists.

After `git pull' and a clean build, the problems seems gone. Closing.