bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 21854 bytes --]

I installed 940+ packages, with the package-quickstart option set to t.
During the last set of packages I attempted to install, a message stating
that attempting to byte-compile package-quickstart.el generated the
error "bytecode overflow" appeared, and the installation process failed
I don't know if the transaction was left in an incomplete state or not.

I examined the package-quickstart.el file, and by trial and error found
that cutting out the last few packages rendered a file that
byte-compiled, and then verified that this happened whether I cut from
the beginning or the end.

I disassembled the byte code and it's simply a result of the byte-code
interpreter being limited to 16-bit addresses.

Whether this is a bug in the package system code for generating the
package-quickstart.el file or a bug in the compiler for not handling the
limitation more robustly, I leave to you.


In GNU Emacs 28.1 (build 1, x86_64-pc-cygwin, GTK+ Version 3.22.28, cairo
version 1.17.4)
 of 2022-04-05 built on fv-az282-442
Repository revision: 402e460d4c4ab116ebe2aebea5606ac4ac4a9e21
Repository branch: master
Windowing system distributor 'The Cygwin/X Project', version 11.0.12101003
Configured using:
 'configure
 --srcdir=/cygdrive/d/a/scallywag/emacs/emacs-28.1-1.x86_64/src/emacs-28.1
 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc
 --infodir=/usr/share/info --mandir=/usr/share/man -C 'CFLAGS=-ggdb -O2
 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -fstack-protector-strong --param=ssp-buffer-size=4
 -fdebug-prefix-map=/cygdrive/d/a/scallywag/emacs/emacs-28.1-1.x86_64/build=/usr/src/debug/emacs-28.1-1
 -fdebug-prefix-map=/cygdrive/d/a/scallywag/emacs/emacs-28.1-1.x86_64/src/emacs-28.1=/usr/src/debug/emacs-28.1-1'
 CPPFLAGS= LDFLAGS='

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBXML2 M17N_FLT MODULES NOTIFY GFILENOTIFY PDUMPER PNG
RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Compilation

Minor modes in effect:
  erefactor-check-eval-mode: t
  async-bytecomp-package-mode: t
  sly-symbol-completion-mode: t
  treemacs-filewatch-mode: t
  treemacs-follow-mode: t
  treemacs-git-mode: t
  treemacs-fringe-indicator-mode: t
  csv-field-index-mode: t
  TeX-PDF-mode: t
  delete-selection-mode: t
  highlight-changes-visible-mode: t
  shell-dirtrack-mode: t
  global-tab-line-mode: t
  tab-line-mode: t
  windmove-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  context-menu-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/owine/.emacs.d/elpa/evil-1.15.0/evil hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-vars hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-vars
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-types hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-types
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-states hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-states
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-search hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-search
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-repeat hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-repeat
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-pkg hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-pkg
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-maps hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-maps
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-macros hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-macros
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-keybindings hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-keybindings
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-jumps hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-jumps
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-integration hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-integration
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-ex hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-ex
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-digraphs hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-digraphs
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-development hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-development
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-core hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-core
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-common hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-common
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-commands hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-commands
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-command-window hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-command-window
/home/owine/.emacs.d/elpa/evil-1.15.0/evil-autoloads hides
/home/owine/.emacs.d/elpa/evil-20220518.859/evil-autoloads
/home/owine/.emacs.d/elpa/lispy-20220209.1138/elpa hides
/home/owine/.emacs.d/elpa/ivy-20220529.1130/elpa
/home/owine/.emacs.d/elpa/xcscope-20210719.828/xcscope hides
/usr/share/emacs/site-lisp/xcscope
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-site hides
/usr/share/emacs/site-lisp/tex-site
/home/owine/.emacs.d/elpa/maxima-20220531.1751/maxima hides
/usr/share/emacs/site-lisp/maxima
/home/owine/.emacs.d/elpa/maxima-20220531.1751/maxima-font-lock hides
/usr/share/emacs/site-lisp/maxima-font-lock
/home/owine/.emacs.d/elpa/cmake-mode-20220322.1258/cmake-mode hides
/usr/share/emacs/site-lisp/cmake-mode
/home/owine/.emacs.d/elpa/clhs-20210428.1911/clhs hides
/usr/share/emacs/site-lisp/clhs
/home/owine/.emacs.d/elpa/clang-format-20191106.950/clang-format hides
/usr/share/emacs/site-lisp/clang-format
/home/owine/.emacs.d/elpa/auctex-13.1.3/auctex hides
/usr/share/emacs/site-lisp/auctex
/home/owine/.emacs.d/elpa/auctex-13.1.3/toolbar-x hides
/usr/share/emacs/site-lisp/auctex/toolbar-x
/home/owine/.emacs.d/elpa/auctex-13.1.3/texmathp hides
/usr/share/emacs/site-lisp/auctex/texmathp
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex hides
/usr/share/emacs/site-lisp/auctex/tex
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-style hides
/usr/share/emacs/site-lisp/auctex/tex-style
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-mik hides
/usr/share/emacs/site-lisp/auctex/tex-mik
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-jp hides
/usr/share/emacs/site-lisp/auctex/tex-jp
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-ispell hides
/usr/share/emacs/site-lisp/auctex/tex-ispell
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-info hides
/usr/share/emacs/site-lisp/auctex/tex-info
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-font hides
/usr/share/emacs/site-lisp/auctex/tex-font
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-fold hides
/usr/share/emacs/site-lisp/auctex/tex-fold
/home/owine/.emacs.d/elpa/auctex-13.1.3/tex-bar hides
/usr/share/emacs/site-lisp/auctex/tex-bar
/home/owine/.emacs.d/elpa/auctex-13.1.3/preview hides
/usr/share/emacs/site-lisp/auctex/preview
/home/owine/.emacs.d/elpa/auctex-13.1.3/plain-tex hides
/usr/share/emacs/site-lisp/auctex/plain-tex
/home/owine/.emacs.d/elpa/auctex-13.1.3/multi-prompt hides
/usr/share/emacs/site-lisp/auctex/multi-prompt
/home/owine/.emacs.d/elpa/auctex-13.1.3/latex hides
/usr/share/emacs/site-lisp/auctex/latex
/home/owine/.emacs.d/elpa/auctex-13.1.3/latex-flymake hides
/usr/share/emacs/site-lisp/auctex/latex-flymake
/home/owine/.emacs.d/elpa/auctex-13.1.3/font-latex hides
/usr/share/emacs/site-lisp/auctex/font-latex
/home/owine/.emacs.d/elpa/auctex-13.1.3/context hides
/usr/share/emacs/site-lisp/auctex/context
/home/owine/.emacs.d/elpa/auctex-13.1.3/context-nl hides
/usr/share/emacs/site-lisp/auctex/context-nl
/home/owine/.emacs.d/elpa/auctex-13.1.3/context-en hides
/usr/share/emacs/site-lisp/auctex/context-en
/home/owine/.emacs.d/elpa/auctex-13.1.3/bib-cite hides
/usr/share/emacs/site-lisp/auctex/bib-cite
/home/owine/.emacs.d/elpa/transient-20220527.2213/transient hides
/usr/share/emacs/28.1/lisp/transient
/home/owine/.emacs.d/elpa/xref-1.4.1/xref hides
/usr/share/emacs/28.1/lisp/progmodes/xref
/home/owine/.emacs.d/elpa/dictionary-20201001.1727/dictionary hides
/usr/share/emacs/28.1/lisp/net/dictionary
/home/owine/.emacs.d/elpa/hierarchy-20190425.842/hierarchy hides
/usr/share/emacs/28.1/lisp/emacs-lisp/hierarchy
/home/owine/.emacs.d/elpa/cl-print-1.0/cl-print hides
/usr/share/emacs/28.1/lisp/emacs-lisp/cl-print

Features:
(shadow mail-extr geiser-impl geiser-custom geiser-base disass emacsbug
two-column misearch multi-isearch erefactor helm-for-files helm-comint
helm-imenu helm-bookmark helm-external helm-mode helm-misc helm-elisp
helm-files image-dired helm-buffers helm-tags helm-locate helm-eval
helm-occur helm-grep helm-regexp helm-net helm-adaptive helm-info
helm-utils helm-types helm-help helm helm-global-bindings helm-easymenu
helm-core async-bytecomp slime sly sly-completion sly-buttons
sly-messages sly-common hyperspec ace-jump-mode dired-subtree
dired-hacks-utils dired-x windata helm-make django-snippets pythonic
magit-popup docbook-snippets docstr docstr-key docstr-faces
docstr-writers docstr-util doom-modeline-segments doom-modeline-env
doom-modeline-core shrink-path doom-themes-base dynamic-ruler compat
checkdoc ebib ebib-reading-list ebib-notes ebib-filters ebib-keywords
ebib-utils ebib-db ecb-symboldef ecb-analyse ecb-compatibility
ecb-winman-support ecb-autogen ecb-tod ecb-cycle ecb-eshell ecb-help
ecb-jde ecb-upgrade ecb-file-browser ecb-method-browser
ecb-semantic-wrapper ecb-semantic ecb-speedbar ecb-layout
ecb-create-layout ecb-compilation ecb-common-browser ecb-navigate
eieio-compat ecb-cedet-wrapper semantic/db-mode semantic/db-find
semantic/db-ref semantic/analyze semantic/sort semantic/scope
semantic/analyze/fcn semantic/format semantic/tag-ls semantic/find
semantic/ctxt ecb-mode-line ecb-face tree-buffer ecb-util silentcomp
vc-mtn vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs eglot array
julia-mode-latexsubs eide-vc xcscope eide-windows eide-search
eide-project eide-keys eide-compare eide-popup eide-edit eide-menu
eide-display eide-config js ein-process ein-jupyter ein-dev ein-notebook
ein-python-send ein-traceback ein-pytools ein-pager ein-completer
ein-notification ein-scratchsheet ein-worksheet poly-ein
display-line-numbers ein-kill-ring ein-cell ein-kernelinfo
ein-notebooklist ein-contents-api ein-file ein-shared-output
ein-output-area ein-kernel ein-ipdb ein-query ein-events ein-websocket
websocket ein-node ein-log ein-classes ein-core request ein-utils
anaphora ein polymode polymode-base polymode-weave polymode-export
polymode-compat polymode-methods trace poly-lock polymode-core
polymode-classes eldev profiler eldev-util jsonrpc electric-operator elp
benchmark electric-spacing-text-mode electric-spacing-emacs-lisp-mode
electric-spacing-haskell-mode electric-spacing-org-mode
electric-spacing-rust-mode electric-spacing-python-mode
electric-spacing-cc-mode electric-spacing lsp-pwsh lsp-kotlin dap-ui
lsp-treemacs gdb-mi gud bui lsp-lens dap-utils dap-mode dap-launch
dap-overlays lsp-treemacs-themes treemacs treemacs-header-line
treemacs-compatibility treemacs-mode treemacs-bookmarks thunk
treemacs-interface treemacs-extensions treemacs-mouse-interface
treemacs-tags treemacs-faces treemacs-persistence
treemacs-filewatch-mode treemacs-follow-mode treemacs-rendering
treemacs-annotations treemacs-async treemacs-visuals
treemacs-fringe-indicator treemacs-icons treemacs-themes
treemacs-workspaces treemacs-scope treemacs-dom treemacs-core-utils
treemacs-macros treemacs-logging treemacs-customization pfuture
dashboard git dashboard-widgets recentf extmap dotassoc kv debbugs-gnu
add-log debbugs-compat debbugs soap-client rng-xsd xsd-regexp ob-shell
em-hist em-pred esh-mode detached autorevert trie tNFA heap link
connection diff-hl-show-hunk diff-hl-inline-popup diff-hl-dired diff
diff-hl vc-hg log-view pcvs-util vc-dir treeview dash-docs ediff
ediff-merg ediff-mult ediff-wind ediff-diff ediff-help ediff-init
ediff-util projectile counsel xdg swiper skeleton ibuf-macs find-dired
grep ibuf-ext ibuffer ibuffer-loaddefs cpio-dired cpio-odc cpio-hpodc
cpio-hpbin cpio-crc cpio-bin cpio-affiliated-buffers cpio-modes
cpio-newc cpio-generic bindat cquery-inheritance-hierarchy
cquery-call-hierarchy cquery-code-lens cquery-semantic-highlighting
cquery-tree cquery-common lsp autothemer csharp-compilation
css-eldoc-hash-table csv-mode hsluv ctxmenu common-lisp-snippets
company-c-headers axiom-environment company-box-doc company-box-icons
frame-local company-coq-utils company-coq-tg company-coq-abbrev smie shr
kinsoku svg hideshow company-math flx ffap irony-completion
irony-snippet irony-cdb-json irony-cdb irony irony-iotask jedi-core epc
ctable concurrent python-environment company-etags lean-dev
lean-right-click lean-message-boxes lean-type lean-hole lean-flycheck
lean-input lean-eri lean-info lean-syntax lean-server lean-leanpkg
lean-debug lean-settings lean-util math-symbol-lists maxima
maxima-font-lock posframe rtags repeat asm-mode company-web company-css
web-completion-data levenshtein find-file python helm-source
helm-multi-match helm-lib clj-refactor cap-words superword subword
inflections cider paredit mc-separate-operations rectangular-region-mode
mc-mark-pop mc-edit-lines mc-hide-unmatched-lines-mode mc-mark-more
mc-cycle-cursors multiple-cursors-core oc-csl org-id citar citar-file
clang-format tramp-sh cider-debug cider-repl-history cider-scratch pulse
cider-mode cider-completion cider-profile cider-inspector cider-eval
cider-repl cider-test cider-stacktrace cider-overlays cider-jar
cider-browse-ns cider-eldoc apropos cider-find cider-resolve cider-doc
cider-browse-spec cider-clojuredocs cider-client cider-connection
sesman-browser nrepl-client parseedn cider-popup cider-common cider-util
sesman clojure-mode nrepl-dict align parseclj-ast parseclj-parser
parseclj-lex parseclj-alist tls lui-track circe lui-irc-colors irc lcs
lui-logging lui-format lui tracking shorten flyspell circe-compat
citeproc-org org-ref org-ref-core org-ref-glossary org-ref-bibtex
org-ref-export ox-pandoc org-ref-misc-links org-ref-label-link
org-ref-ref-links org-ref-citation-links org-ref-bibliography-links
bibtex-completion doi-utils org-ref-utils citeproc citeproc-itemgetters
citeproc-cite citeproc-generic-elements citeproc-choose
string-inflection citeproc-biblatex citeproc-subbibs citeproc-sort
citeproc-name citeproc-macro citeproc-formatters citeproc-proc
citeproc-disamb citeproc-itemdata citeproc-number queue citeproc-date
citeproc-context citeproc-style citeproc-locale citeproc-prange
citeproc-term citeproc-rt citeproc-lib citeproc-bibtex citeproc-s
ol-bibtex ox-org ox-odt ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox citre-peek citre-basic-tools citre-ctags citre-util
citre-core citre-tag citre-core-tables citre-common cl-format-builtins
clisp-format cl-format-def tree-mode hierarchy cg-global which-func caml
view rfc1345 cape ccls-member-hierarchy ccls-inheritance-hierarchy
ccls-call-hierarchy ccls-code-lens ccls-semantic-highlight ccls-tree
ccls-common lsp-javascript lsp-html lsp-icons lsp-completion
lsp-semantic-tokens lsp-mode lsp-protocol markdown-mode ht inline
centaur-tabs-interactive centaur-tabs-functions centaur-tabs-elements
powerline powerline-separators powerline-themes config-manage
config-manage-mode config-manage-prop config-manage-base choice-program
time-stamp config-manage-declare choice-program-complete bug-hunter
bui-list bui-info bui-entry bui-core bui-history bui-button bui-utils
org-agenda org-colview org-element avl-tree boxy benchmark-init
reftex-sel org-capture org-refile filenotify biblio parsebib
biblio-download biblio-dissemin biblio-ieee biblio-hal biblio-dblp
biblio-crossref biblio-arxiv biblio-doi biblio-core url-queue ido
hl-line f f-shortdoc tempo make-mode etags fileloop company-dabbrev
company-template company avy axiom-selector axiom-buffer-menu
axiom-boot-mode axiom-spad-mode axiom-input-mode imenu
axiom-process-mode axiom-help-mode axiom-base mm-url vc-git diff-mode vc
vc-dispatcher macrostep-c cmacexp macrostep lua-mode texinfo
texinfo-loaddefs toolbar-x reporter desktop frameset context plain-tex
latex latex-flymake flymake-proc flymake tex-ispell tex-style tex crm
texmathp auto-loads tex-site auto-complete-config rng-nxml nxml-mode
nxml-outln nxml-rap rng-valid rng-loc rng-uri rng-parse nxml-parse
nxml-enc rng-match rng-dt rng-util rng-pttrn nxml-ns xmltok nxml-util
yaxception auto-complete popup sage-shell-mode deferred ede/srecode
srecode semantic/db semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw mode-local ede/speedbar ede/files
ede ede/detect ede/base ede/source eieio-speedbar cedet cc-langs spinner
ede/auto eieio-base arxiv-query arxiv-vars org ob ob-tangle ob-ref
ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint
org-pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp ob-core ob-eval org-table oc-basic bibtex ol
org-keys oc org-compat org-macs org-loaddefs hydra lv smtpmail sendmail
dired-aux async async-await iter2 promise promise-rejection-tracking
promise-finally promise-done promise-es6-extensions promise-core
generator enriched pces-e20 pces-20 pccl-20 broken ccl path-util
invisible inv-23 mcs-e20 mcs-20 mcharset poem poem-e20 poem-e20_3 pym
poe alist static apel-ver product cl yasnippet-snippets angular-snippets
yasnippet log4e notifications gntp almost-mono-themes sort reftex-dcr
reftex-parse reftex-cite reftex reftex-loaddefs reftex-vars s
all-the-icons ivy-rich ivy delsel ivy-faces ivy-overlay colir
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons edit-indirect ssass-mode
vue-html-mode mmm-cmds mmm-mode mmm-univ sgml-mode dom mmm-class
mmm-region mmm-auto mmm-vars mmm-utils mmm-compat face-remap term
disp-table ehelp term/xterm xterm reformatter timezone xmlunicode quail
xmlunicode-character-list ert ewoc debug flycheck dash let-alist
easy-mmode epl rx warnings yaml-mode tar-mode arc-mode archive-mode
names edebug backtrace autoload lisp-mnt mm-archive network-stream
url-http url-gw url-auth cl-print eieio-opt cl-extra shortdoc package-x
help-fns radix-tree mule-util vhdl-mode hippie-exp cus-theme tmm
find-func time color shadowfile xwidget bookmark wid-browse tree-widget
eieio-custom woman speedbar ezimage dframe rst compile proced mh-e
mh-compat mh-buffers mh-loaddefs message rfc822 mml mml-sec epa derived
epg rfc6068 mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader hilit-chg hi-lock gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums text-property-search
mail-utils mm-util mail-prsvr facemenu cwarn cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs ses
unsafep rcirc newst-backend mpc ispell htmlfontify htmlfontify-loaddefs
eshell esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg esh-module
esh-groups esh-util erc-goodies erc erc-backend erc-loaddefs doc-view
jka-compr image-mode exif cal-menu calendar cal-loaddefs calculator
edmacro kmacro calc calc-loaddefs rect calc-macs tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat parse-time iso8601
ls-lisp format-spec ntlm hmac-md5 hex-util md4 nsm rmc netrc net-utils
ldap kermit shell pcomplete gravatar dns url-cache goto-addr gnutls puny
eudc-vars dig dbus xml bug-reference ange-ftp comint server man
ansi-color locate dired dired-loaddefs epg-config ebnf2ps ps-print
ps-print-loaddefs ps-def lpr bib-mode help-mode novice thingatpt xref
project ring cus-edit pp wid-edit cus-start time-date
modus-operandi-theme modus-themes pcase tab-line cus-load windmove
finder-inf bnf-mode-autoloads names-autoloads advice
namespaces-autoloads realgud-recursive-autoloads slime-autoloads
with-editor-autoloads info package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
gfilenotify lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 3122102 613628)
 (symbols 48 167919 62)
 (strings 32 759155 206619)
 (string-bytes 1 22732645)
 (vectors 16 252152)
 (vector-slots 8 4169882 371529)
 (floats 8 3882 287)
 (intervals 56 204409 17430)
 (buffers 992 249))

[-- Attachment #2: Type: text/html, Size: 23009 bytes --]

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

If, as seems to be the case, byte-compile-keep-pending is only used for top-level forms, then this patch may even be correct. Does it solve your problems?
It still seems to generate far bigger bytecode chunks than the 300 cutoff would imply but that's perhaps just a matter of calling the function in more places.


[-- Attachment #2: bytecomp-flush-big.diff --]
[-- Type: application/octet-stream, Size: 924 bytes --]

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index ab21fba8a2..156a38ec2d 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -2521,13 +2521,12 @@ byte-compile-output-docform
 (defun byte-compile-keep-pending (form &optional handler)
   (if (memq byte-optimize '(t source))
       (setq form (byte-optimize-one-form form t)))
+  ;; To avoid consing up monstrously large forms at load time, we split
+  ;; the output regularly.
+  (when (nthcdr 300 byte-compile-output)
+    (byte-compile-flush-pending))
   (if handler
       (let ((byte-compile--for-effect t))
-	;; To avoid consing up monstrously large forms at load time, we split
-	;; the output regularly.
-	(and (memq (car-safe form) '(fset defalias))
-	     (nthcdr 300 byte-compile-output)
-	     (byte-compile-flush-pending))
 	(funcall handler form)
 	(if byte-compile--for-effect
 	    (byte-compile-discard)))

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Mattias Engdegård]

> It still seems to generate far bigger bytecode chunks than the 300 cutoff would imply

That is simply because of some very large top-level forms: each package encloses its autoload definitions in a single `let` form at the top level. We could change the generation of package-quickstart.el to limit the number of forms in each top-level `let`.

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2142 bytes --]

On Tue, Jun 14, 2022 at 12:29 PM Mattias Engdegård <mattiase@acm.org> wrote:

> If, as seems to be the case, byte-compile-keep-pending is only used for
> top-level forms, then this patch may even be correct. Does it solve your
> problems?
> It still seems to generate far bigger bytecode chunks than the 300 cutoff
> would imply but that's perhaps just a matter of calling the function in
> more places.
>
>
Thanks, Mattias, it does work.

I was going to ask about directly addressing the underlying problem by
tail-calling
or trampolining to a byte-code vector in the constants array, but then
realized you would have to
either make sure there could be no "gotos" between the segments or do a
real trampoline to
an explicit label.  And in either case you would have to save the contents
of the stack frame
and reinstate them in the continuation byte-code call, and I don't see any
byte-codes that would
support that.   Otherwise you could only do it when you know there is no
stack in use, which is what I believe
your solution effectively does.

On the other hand, given the code for patching up byte-code in
byte-compile-lapcode, you could
explicitly byte-compile a thunk for every top-level expression, then glue
them together until they would exceed
the 65K pc limit, then do another segment, etc, and do a simple trampoline
between the resulting byte-code vectors,
or no trampoline if there were only one required.  Strictly speaking,
gluing them all together is really an optimization
of creating byte-code vectors (thunks) for each top-level expression, and
looping over the collection of them, invoking
each one in turn.

As long as I'm looking at the compile log, I also see a lot of errors of
the form:
     package-quickstart2.el:14739:39: Warning: The compiler ignores
‘autoload’ except at top level.  You should
         probably put the autoload of the macro ‘bind-map-for-minor-mode’
at top-level.
This message is only reported for macros - there are plenty of autoload
expressions that do not generate this
message despite being in the same kind of "let" form.

Lynn

[-- Attachment #2: Type: text/html, Size: 2740 bytes --]

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I was going to ask about directly addressing the underlying problem by
> tail-calling or trampolining to a byte-code vector in the constants
> array, but then realized you would have to either make sure there
> could be no "gotos" between the segments or do a real trampoline to an
> explicit label.  And in either case you would have to save the
> contents of the stack frame and reinstate them in the continuation
> byte-code call, and I don't see any byte-codes that would support
> that.   Otherwise you could only do it when you know there is no stack
> in use, which is what I believe your solution effectively does.

Indeed, the 64k limit can bite hard in some cases where working around
it would take a lot of work and we don't have any of the infrastructure
needed for it.

But luckily, in practice those hard cases tend not to occur at all.
Instead we tend to bump into the limit just because we're too naive in
how we combine together several top-level expressions.

> As long as I'm looking at the compile log, I also see a lot of errors of
> the form:
>      package-quickstart2.el:14739:39: Warning: The compiler ignores
> ‘autoload’ except at top level.  You should
>          probably put the autoload of the macro ‘bind-map-for-minor-mode’
> at top-level.

Could you make a separate bug report for this one?
I'm not sure what's the better fix for this one.  It should be a harmless
warning, but even when it's harmless it's annoying.


        Stefan

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Mattias Engdegård]

17 juni 2022 kl. 23.27 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> Indeed, the 64k limit can bite hard in some cases where working around
> it would take a lot of work and we don't have any of the infrastructure
> needed for it.

Right and actually we don't want even to get near that limit because of bad scaling in the compiler.

Stefan, do you agree with the patch (for byte-compile-keep-pending)? I think it's sound but would welcome a second pair of eyes on it.

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Stefan, do you agree with the patch (for byte-compile-keep-pending)? I think
> it's sound but would welcome a second pair of eyes on it.

Looks OK to me, yes.


        Stefan

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Mattias Engdegård]

18 juni 2022 kl. 14.59 skrev Stefan Monnier <monnier@iro.umontreal.ca>:
> 
>> Stefan, do you agree with the patch (for byte-compile-keep-pending)? I think
>> it's sound but would welcome a second pair of eyes on it.
> 
> Looks OK to me, yes.

Thank you, pushed.

To solve the problem entirely, it's possible that we also need to split the top-level `let` forms in package-quickstart.el. For instance, `magit` generates a 5 KiB bytecode chunk and 750-element constant vector which is perfectly manageable, but a package with more autoloads may be troublesome. Do you know a package that would fit the description?

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> To solve the problem entirely, it's possible that we also need to split the
> top-level `let` forms in package-quickstart.el.

I think the other bug (about `autoload` calls that define macros
leading to warnings because they're not at top-level) suggests that we
should move the calls to `autoload` outside of the `let`, which would
probably make this problem even less likely.

> For instance, `magit` generates a 5 KiB bytecode chunk and 750-element
> constant vector which is perfectly manageable, but a package with more
> autoloads may be troublesome. Do you know a package that would fit
> the description?

I suspect none.  I suspect in the case of Magit the magit-autoloads.el
file is actually too large for its own good: it should probably be split
into autoloads that "face outside" (i.e. those that need to be defined
before Magit is used) from those facing inside (those that are needed
for internal lazy loading).

The old autoload.el didn't support this conveniently, but the new one
does, so a few years from now we'll have a clean solution :-)


        Stefan

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> For instance, `magit` generates a 5 KiB bytecode chunk and 750-element
>> constant vector which is perfectly manageable, but a package with more
>> autoloads may be troublesome. Do you know a package that would fit
>> the description?
>
> I suspect none.  I suspect in the case of Magit the magit-autoloads.el
> file is actually too large for its own good: it should probably be split
> into autoloads that "face outside" (i.e. those that need to be defined
> before Magit is used) from those facing inside (those that are needed
> for internal lazy loading).

Copying in Jonas, in case he has any comments.

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Jonas Bernoulli via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Kangas <stefankangas@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> For instance, `magit` generates a 5 KiB bytecode chunk and 750-element
>>> constant vector which is perfectly manageable, but a package with more
>>> autoloads may be troublesome. Do you know a package that would fit
>>> the description?
>>
>> I suspect none.  I suspect in the case of Magit the magit-autoloads.el
>> file is actually too large for its own good: it should probably be split
>> into autoloads that "face outside" (i.e. those that need to be defined
>> before Magit is used) from those facing inside (those that are needed
>> for internal lazy loading).
>
> Copying in Jonas, in case he has any comments.

I have no idea how I would split the autoloads into two parts.  Is there
some mechanism, say ;;;###autoload[external] / ;;;###autoload[internal],
that would be respected by `loaddefs-generate' and/or code that uses that?
Does package.el know how to use it, and what about other package managers
and homegrown makefiles and scripts?

Then there is also the question of how much work it would mean for me to
go over all autoloaded things and decide which bucket they belong into.

bug#55972: 28.1; Package quickstart generated for large number of packages generates byte-code string larger than 64K, triggering bytecode overflow error

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I have no idea how I would split the autoloads into two parts.  Is there
> some mechanism, say ;;;###autoload[external] / ;;;###autoload[internal],
> that would be respected by `loaddefs-generate' and/or code that uses that?

The new `loaddefs-gen.el` does offer something for that, yes, but that's
only for Emacs≥29, so until Org support older Emacsen you'd have to
pregenerate that "internal autoloads" file "by hand" (and even keep it
in the Git until `elpa.gnu.org` upgrades to a version of Debian that has
Emacs-29, i.e. presumably Debian 13 aka Trixie around 2025).


        Stefan